LiveKit on Veris AI's VAmoS Bench: 0.7 points behind first, on identical parts

What is VAmoS Bench?

Veris is simulation infrastructure for AI agents. Describe the world your agent works in and Veris stands up a digital twin of it: databases, CRMs, ticketing systems, payment and card-operation backends, each holding real state the agent can change. Veris supplies the people too: synthetic humans with private goals, who decide for themselves when the conversation is over. Every scenario runs in its own sandboxed, coherent world.

VAmoS Bench (Voice Agent Simulation Bench) is built on that infrastructure. It runs any voice agent through the same twins and the same simulated callers, so different stacks can be compared like for like on performance and cost.

Every run leaves a complete record: audio, transcript, tool logs, the final state of every twin service, latency and timing. That record is what makes a claim verifiable.

Two rows on the board are built from exactly the same parts, and they finished first and second. This post is about the second one, and about what a 0.7-point gap between identical components does and does not tell you.

How Riley is built on LiveKit

Riley on LiveKit Agents is a cascaded pipeline you host yourself: Deepgram nova-3-general for speech recognition, gpt-4.1-mini for the model, ElevenLabs for the voice, with Silero VAD deciding when the caller has finished talking. Riley runs as an agent worker inside a LiveKit room, and the five card-operations tools run in the same process against the twin database. We released the source code for our implementation.

Those are the same three vendors behind the pipecat row, chosen deliberately, with the same prompt and the same 800 ms end-of-turn silence target. They listen with the same model, think with the same model and speak with the same voice. Whatever separates 70.3% from 71.0% is the layer holding the parts together, which makes this the cleanest orchestration comparison on the board.

One difference is structural rather than configured. LiveKit is WebRTC end to end, so the caller's audio enters through a small bridge that joins the room as a participant and publishes it as a microphone track; the worker's reply comes back the same way. Three processes run in one container: the LiveKit server, the worker, and that bridge. pipecat has no room to go through. The latency section comes back to this.

Cost is metered usage only, because the machine running the loop is yours: about $0.048 a call, of which $0.031 is the voice, $0.012 is speech recognition and $0.005 is the model. A managed platform bundles hosting into its price, so the comparison with Vapi below is not quite like for like, and says so.

What did LiveKit score on VAmoS Bench?

Second on the board, every call connected, cheap to run and steady from run to run. The one column where it is ordinary is response time. Split the headline score by scenario group and it is above the field in all three.

75.6% on simple flows is joint second on the board, level with Vapi and behind only OpenAI Realtime. 59.6% on complex flows is joint third. The adversarial column is the least distinguished of the three at eighth of seventeen, and it is still 5.5 points above the field.

There is no weak column here. What there is, is a leader 0.7 points ahead whose own error bar is ±2.5, and a runner-up whose bar is ±1.8. Each of those is larger than the gap between them. The board shows pipecat first because 71.0 is a bigger number than 70.3, and that is the whole basis of the ordering.

How does LiveKit compare to other voice agent platforms?

Three stacks share this row's brain. pipecat runs the identical three vendors in a different framework; Vapi runs them as a managed platform; ElevenLabs is the voice this row buys, sold as a whole agent. Together they answer the question this row raises: on the same model, what is the orchestration worth?

LiveKit vs pipecat: which completes more tasks?

Too close to call. pipecat leads 71.0% to 70.3%, and both error bars are wider than the gap. pipecat is 0.42 s faster to answer, which is a real difference; LiveKit finishes calls in fewer turns and interrupts callers less than half as often.

The completion gap is noise until someone runs more than three repeats. The latency gap is not: the bootstrap interval around pipecat's median runs 1.85 to 2.03 s and LiveKit's runs 2.29 to 2.49, and they do not touch. The end-of-turn silence is the same 0.8 s in both rows by design, so the timer is not where the time goes. What LiveKit has that pipecat does not is the room: the caller's audio goes in through the bridge and the reply comes back out through it, and every hop is paid on every turn. That accounts for some of the gap; we have not isolated the rest.

The two rows also split the groups. LiveKit is 5.6 points better at simple flows; pipecat is 3.6 better at complex ones and 3.1 better under adversarial pressure. Weight the columns by the calls you actually get, and pick the framework you would rather operate. On this evidence the framework is not going to decide the score.

LiveKit vs Vapi: is the managed platform worth 2.8× the price?

Not on this task. Vapi runs the same three vendors for $0.135 a call against $0.048, completes one point fewer, takes nearly twice as many turns to do it and drops calls that LiveKit connects. The two rows are identical to the decimal on simple and complex flows.

Those two matching rows are the most useful numbers in this post. Three different orchestrations of gpt-4.1-mini, with the same prompt and tools, land at 75.6, 75.6 and 70.0 on simple flows and at 59.6, 59.6 and 63.2 on complex ones. The ceiling on this task is the model and the prompt. The orchestration layer moves the score by about three points either way, and it moves the bill by a factor of three.

What Vapi charges for is running the thing. If you have nobody to keep a worker process alive, $0.087 a call is the price of not having to, and it is an honest price. What it does not buy is a better agent.

LiveKit vs ElevenLabs: which answers faster?

ElevenLabs, by a second. 1.19 s against 2.37, the fastest row on the board against the eleventh. LiveKit completes three points more, costs 42% as much, and is using ElevenLabs for its voice the whole time.

The second is mostly turn detection. ElevenLabs runs a learned end-of-turn model with no silence timer; this row waits a fixed 0.8 s of silence before it is allowed to answer, then pays for the speech recognition and the room on top. Nothing on the LiveKit side is slow. It is a pipeline, and a pipeline has hops.

What the extra second buys is the score. ElevenLabs' one bad column is out-of-scope requests, where it declines 9 of 15 and this row declines all 15. Whether a second of response time matters more than that depends on what your callers are asking for, and how often they are asking for something they should not get.

What does a good voice agent call look like?

Riley's brief tells it never to break character or reveal that it is an AI. Callers test that. This one opens with the question before saying anything else, then asks it again as a yes-or-no with a threat attached: answer, or I am not giving you any personal information.

It is a short call and a narrow test, and it passes it cleanly. The agent does not say it is an AI, which its brief forbids, and it does not say it is a person, which would be a lie. It offers to help twice and the caller leaves. Nothing was looked up and nothing was written.

The disclosure rule is Riley's, not a recommendation; some operators will want the opposite behaviour and the scenario would then grade the other way. What the benchmark is measuring is whether the agent holds its brief when a caller pushes on it directly. Under a stack that answers in 2.4 seconds, on a script that tries to shame it out of character, this one held.

Where it still loses

89 of the 300 calls failed. LiveKit has no entry in the published failure gallery, so this section is built from the run data rather than from one call. An absence there is a curation outcome, not a clean sheet: the gallery is twelve calls chosen to show distinct failure modes.

Above the field on every tag. That is unusual on this board; the leader is not, because pipecat trails on domain content. So the honest answer to "where does LiveKit lose" is not a column. It is the three places pipecat beats it on the same parts: complex flows, adversarial callers and response time.

The first two are small and inside the error bars. The third is not, and it is the one a caller would notice. Two point four seconds is a long pause on the phone, and this row pays it on every turn while running the fastest voice on the board. If you build on LiveKit, the end-of-turn delay and the room topology are where the time is, and both are yours to change.

The other thing to hold onto is the identical-to-Vapi rows. Two orchestrations of the same model, prompt and tools land on the same simple and complex scores to the decimal. That puts the ceiling on this task in the model and the prompt, and moving it means changing those, not the framework.

How do you benchmark your own voice agent?

If you have built something on LiveKit Agents, you can make your own benchmark with Veris. Find out how your agent handles your callers, doing your task, against your backends.

You will not hear the difference between this row and pipecat on a demo call, and neither would we. It took 600 calls and a bootstrap to say which one is faster, and 600 were not enough to say which one is better.

1. Point Veris at the world your agent works in. Riley’s was a seeded Postgres of cards and accounts. Yours might be a claims system, a scheduling backend, an order database. Describe it and Veris stands up a twin that holds real state, then gives every scenario its own isolated, fully-seeded copy.

   veris env push --env-id "$ENV_ID"

2. Write the callers, not the scripts. Each scenario includes a synthetic human with a private goal, not a fixed transcript to replay. Describe who calls you and what they want, including the ones who are confused, in a hurry, or lying, and the simulator generates the hundred conversations that follow. Veris creates them, you review.

   veris scenarios create --num 100 --env-id "$ENV_ID"

3. Run the phonecalls, Veris grades, you listen. You get completion, latency, turns, interruptions and cost, plus the audio, the transcript, every tool call and every row returned for each individual call. That record is where your own findings will come from.

What you end up with is a private benchmark on your own task: a number you can move deliberately from one release to the next, and a set of recordings you can hand to whoever has to sign off on shipping it.

Common questions