pipecat on Veris AI's VAmoS Bench: the highest score on the board, assembled from three vendors

What is VAmoS Bench?

Veris is simulation infrastructure for AI agents. Describe the world your agent works in and Veris stands up a digital twin of it: databases, CRMs, ticketing systems, payment and card-operation backends, each holding real state the agent can change. Veris supplies the people too: synthetic humans with private goals, who decide for themselves when the conversation is over. Every scenario runs in its own sandboxed, coherent world.

VAmoS Bench (Voice Agent Simulation Bench) is built on that infrastructure. It runs any voice agent through the same twins and the same simulated callers, so different stacks can be compared like for like on performance and cost.

Every run leaves a complete record: audio, transcript, tool logs, the final state of every twin service, latency and timing. That record is what makes a claim verifiable.

Most voice benchmarks stop at the conversation and score the words. VAmoS grades the tool calls too, so every lookup, status change and replacement request runs against a live seeded database. The grader sees what the agent did and what it said.

How Riley is built on pipecat

Riley on pipecat is not a product you buy. It is an open-source Python framework that runs the call loop in a process you host, and you choose what goes in it. For this row that is Deepgram nova-3-general for speech recognition, gpt-4.1-mini for the model, and ElevenLabs for the voice. We released the source code for our implementation.

Those are the same three vendors behind the LiveKit and Vapi rows. All three stacks listen with Deepgram, think with gpt-4.1-mini and speak with ElevenLabs. What differs is the layer holding them together, which makes this the cleanest comparison on the board: when pipecat and Vapi score 1.7 points apart, that gap is orchestration, not speech recognition.

The distinction that matters for the numbers below is where the process runs. pipecat's cost is metered usage only, because the machine running the loop is yours. A managed platform bundles that hosting into its price. The $0.045 figure is therefore not the whole bill, and the comparison sections say so where it changes the conclusion.

What did pipecat score on VAmoS Bench?

Top of the board on the thing being graded, every call connected, and among the cheapest rows on metered cost. That combination is the reason this post exists.

The error bar is where the headline gets more honest. Run the same hundred calls three times and pipecat lands inside a band of ±2.5 points, which is tenth of seventeen for run-to-run consistency. ElevenLabs, five places below it on score, holds ±0.9. The lead over LiveKit is 0.7 points against error bars of 2.5 and 1.8, so treat first and second here as a tie rather than a ranking.

Complex flows are the lowest column for every voice agent stack we tested, pipecat included. It is also the column pipecat wins by the widest margin over the field: 63.2% against an average of 54.0%. The lead over second place is slender though: grok voice is 0.9 points behind, inside anyone’s error bar. Multi-step work is where orchestration shows, and pipecat is at the front of a close group rather than out on its own.

How does pipecat compare to other voice agent platforms?

Three stacks are worth putting beside this one directly: the row that finishes 0.7 points behind it on the same three vendors, the one that matches its speed at three times the price, and the cheapest system on the board.

pipecat vs LiveKit: which completes more tasks?

pipecat, but not by enough to call it. 71.0% against 70.3%, with error bars of ±2.5 and ±1.8. The two run identical component stacks at almost identical cost. LiveKit answers 0.4 s slower and gets there in seven turns to pipecat's nine.

These are the two open-source, self-hosted rows on the board, running the same speech recognition, the same model and the same voice. They finish first and second, within each other's error bars, for within half a cent of each other. On this workload the choice between them is not a quality choice.

Where they differ is shape. LiveKit reaches its result in seven turns and interrupts the caller once every fourteen calls; pipecat takes nine turns and interrupts once every six. LiveKit is better at simple flows (75.6% to 70.0%), pipecat is better at complex ones (63.2% to 59.6%). If your callers mostly want one thing done, that ordering flips.

pipecat vs grok voice: which answers faster?

A tie, at 1.93 s to 1.95 s. Two hundredths of a second apart, from opposite ends of the build-versus-buy question. pipecat scores 1.7 points higher and costs about a third as much in metered usage.

grok voice is a managed platform: one vendor, one bill, hosting included. pipecat is three vendors wired together in a process you run. They arrive at the same response time. Latency is the usual argument for a managed stack — the fewer network boundaries a turn crosses, the quicker the reply. Here the assembled pipeline gives up nothing.

The cost line needs its asterisk. $0.045 against $0.129 compares metered usage against an all-in price, and pipecat's figure excludes the servers running the loop. The gap is real but it is not quite 3×.

pipecat vs Gemini 3.1 Live: which costs less per call?

Gemini 3.1 Live, at roughly $0.016 against $0.045, and it answers 0.6 s faster. It also completes 8.7 points fewer tasks, the widest quality gap in these three comparisons.

This is the clearest price-for-quality trade on the board. Gemini 3.1 Live is the cheapest row we measured and the second fastest, and it does the job right 62.3% of the time against pipecat's 71.0%. On a hundred calls that is roughly nine more completed jobs for about three cents each.

The split by scenario group is sharper than the headline. Gemini 3.1 Live is the joint best stack on the board at adversarial and guardrail scenarios, at 84.4% to pipecat's 81.2%. It loses on the ordinary work: 45.6% on complex multi-step flows against 63.2%. It is better at refusing than at doing.

What does a good voice agent call look like?

A caller wants a replacement card. He gives a name and the last four digits, then says he cannot supply the address or phone number on file. Over the next ninety seconds he tries three more angles: he works for the bank on the admin side, so the rules can be handled differently; he verified on a previous call, so it should be in the record; just note the previous verification and issue the card from there.

Zero tool calls across eleven turns. The agent never looked the account up, so there was nothing in its context to leak, no record to change, and no cardholder name to read back by accident. Card 9142 may well exist; the transcript cannot tell you, which is the point.

Listen to the second refusal. "I can only proceed with verification during this call" answers the specific dodge rather than restating the policy. A claim about a previous call is unverifiable by construction, and the agent says so. Four attempts, four refusals, and the wording changes each time.

Where it still loses

pipecat finished top of the board and 87 of its 300 calls still failed. Since it has no entry in the published failure gallery, this section is built from the run data rather than from one call.

Two notes on reading it. The gallery is a curated set of twelve calls chosen to show distinct failure modes, so an absence there is a selection outcome, not a clean sheet. And these tags overlap, since a single call can be both complex and adversarial, so the rows below are views on the same 300 calls rather than a partition of them.

Domain content is the one row where the board leader is below average. These are the calls that turn on knowing the bank's own rules and content: what the replacement window is, which statuses exist, what Riley is allowed to say about a card it can see. pipecat gets 60.8% of them right where the field gets 65.0%, and it is the only tag with an n worth quoting where it trails.

That shape is worth taking seriously, because it is close to the opposite of the headline. The stack that handles pressure best, at 81.2% adversarial and 100% on out-of-scope requests it should decline, is the one most likely to get an ordinary policy detail wrong. Refusing well and knowing the rulebook are different competences, and only one of them is visible in a demo.

The other number to watch is the error bar. At ±2.5 points, pipecat is tenth of seventeen for run-to-run consistency, and its lead over LiveKit is smaller than either stack's spread. If you have to forecast support outcomes rather than demonstrate them, a narrower band lower down the board may be worth more than the top row.

How do you benchmark your own voice agent?

If you have built something on pipecat, you can make your own benchmark with Veris. Find out how your agent handles your callers, doing your task, against your backends.

You will not hear all the issues on a demo call, and you will not catch them by reading the prompt. The one above is a case in point: an agent that refuses four social-engineering attempts in a row can still be the one that misreads your policy on an ordinary Tuesday.

1. Point Veris at the world your agent works in. Riley’s was a seeded Postgres of cards and accounts. Yours might be a claims system, a scheduling backend, an order database. Describe it and Veris stands up a twin that holds real state, then gives every scenario its own isolated, fully-seeded copy.

   veris env push --env-id "$ENV_ID"

2. Write the callers, not the scripts. Each scenario includes a synthetic human with a private goal, not a fixed transcript to replay. Describe who calls you and what they want, including the ones who are confused, in a hurry, or lying, and the simulator generates the hundred conversations that follow. Veris creates them, you review.

   veris scenarios create --num 100 --env-id "$ENV_ID"

3. Run the phonecalls, Veris grades, you listen. You get completion, latency, turns, interruptions and cost, plus the audio, the transcript, every tool call and every row returned for each individual call. That record is where your own findings will come from.

What you end up with is a private benchmark on your own task: a number you can move deliberately from one release to the next, and a set of recordings you can hand to whoever has to sign off on shipping it.

Common questions