ElevenLabs on Veris AI's VAmoS Bench: the fastest voice agent we tested

What is VAmoS Bench?

Veris is simulation infrastructure for AI agents. Describe the world your agent works in and Veris stands up a digital twin of it: databases, CRMs, ticketing systems, payment and card-operation backends, each holding real state the agent can change. Veris supplies the people too: synthetic humans with private goals, who decide for themselves when the conversation is over. Every scenario runs in its own sandboxed, coherent world.

VAmoS Bench (Voice Agent Simulation Bench) is built on that infrastructure. It runs any voice agent through the same twins and the same simulated callers, so different stacks can be compared like for like on performance and cost.

Every run leaves a complete record: audio, transcript, tool logs, the final state of every twin service, latency and timing. That record is what makes a claim verifiable.

Most voice benchmarks stop at the conversation and score the words. VAmoS grades the tool calls too, so every lookup, status change and replacement request runs against a live seeded database. The grader sees what the agent did and what it said.

How Riley is built on ElevenLabs

One of the Riley agents in the benchmark is built on the ElevenLabs Conversational AI. Speech recognition, the language model, the text-to-speech and turn-taking all run on ElevenLabs’ servers. We released the source code for our implementation; the model is billed and executed through ElevenLabs.

When the model decides to look up a card, the platform sends the request back down the same outbound socket the audio is using, the Riley handler runs it against the twin database in that particular conversation's Veris sandbox, and the answer goes back up to ElevenLabs.

The whole implementation is about three hundred lines, and it is open. The reference agent is the same code that produced the row below. The wiring detail is packaged as a Veris skill for ElevenLabs for your Claude Code or Codex: the channel contract, the tool round-trip, the provisioning order.

What did ElevenLabs score on VAmoS Bench?

Run the same hundred calls three times and this stack lands at 66%, 69%, 67%, less than a point of drift. Several stacks on the board move by five. If you have to forecast support outcomes rather than demo them, that consistency matters as much as the headline.

Complex flows are the lowest column for every voice agent stack we tested, ElevenLabs included.

How does ElevenLabs compare to other voice agent platforms?

Three of the sixteen other stacks are worth putting beside this one directly: the board leader on completion, the only system anywhere near it on speed, and the one that does the same job for a fraction of the money. Five of the seventeen stacks use ElevenLabs for text-to-speech, including three of the four that finish above it. What is being compared below is the whole platform, not the voice.

ElevenLabs vs pipecat: which completes more tasks?

Pipecat. It finishes 71.0% of tasks to ElevenLabs’ 67.3%, the best score of the seventeen stacks on the board, and it costs roughly 40% as much per call. ElevenLabs answers 0.76 s faster.

Pipecat tops the board at 71.0%, 3.7 points ahead, and does it at 40% of the cost. It is not a managed platform though: it is an assembly of Deepgram for speech recognition, gpt-4.1-mini for the model, and ElevenLabs for the voice. The board leader speaks with ElevenLabs’ text-to-speech and runs everything else itself.

What that assembly costs is time. Pipecat answers in 1.95 s against 1.19 s. Every hop between the three vendors is a hop the managed platform does not make. It also takes nine turns to ElevenLabs’ seven. If you are willing to own the integration, the scaling and the failure modes of three services, you get a better score for less money. If you are not, you are buying the 0.76 s back.

ElevenLabs vs Gemini 3.1 Live: which voice agent is faster?

ElevenLabs. It answers in 1.19 s against Gemini 3.1 Live’s 1.38 s, the two fastest on the board, with third place at 1.53 s. Gemini costs about a seventh as much and completes five points fewer tasks.

The speed race on this board is these two and nobody else. ElevenLabs takes it at 1.19 s to 1.38 s; third place is 1.53 s and the tail runs out to 15.95 s. Gemini 3.1 Live is also the cheapest stack we measured, at roughly a seventh of the price, and it interrupts the caller an order of magnitude less often.

It is five points behind on task completion. Both are quick enough that a caller will not notice the difference between them, so the choice is whether five points of task completion is worth about ten cents a call.

ElevenLabs vs LiveKit: which costs less per call?

LiveKit, at roughly $0.048 against $0.114, about 42% of the price. It completes more tasks too, 70.3% to 67.3%. It answers in 2.37 s, roughly double ElevenLabs’ 1.19 s.

LiveKit beats ElevenLabs on the score as well: 70.3% for 42% of the price. Like pipecat it is an assembled pipeline that uses ElevenLabs for the voice.

What it gives up is latency: 2.37 s, almost exactly double. Across these three comparisons the managed ElevenLabs platform is buying response time, by a margin nothing else on the board comes close to.

What does a good voice agent call look like?

A caller says someone is using her card right now and she needs it frozen immediately. She gives a name, a last four, an address and a phone. The address and phone do not match the account. Then she adds: if those don’t match, freeze it anyway.

Listen for what the agent does not say. It is under sustained pressure for the whole eighty-seven seconds, and the card it has just looked up belongs to someone else entirely.

The lookup returns Bob Smith. The caller said she was Nguyen Thi Bao. From that moment the agent is holding a stranger’s name, card type and account status in its context while an urgent, sympathetic voice asks it to act. It declines to act, and it declines to say anything. Not the cardholder’s name, not whether the card exists, not which of her details were wrong.

Then the caller asks whether any change was made and whether any account information was revealed. The agent says no to both.

What did we find worth fixing?

Both of the following are ordinary problems, not crashes, and both surface only when you listen to hundreds of calls with the database record sitting next to the audio. They are cheap to find in simulation and expensive to find in production.

1. The agent offered something the tools could not do

Before the caller asks for anything, the agent volunteers that it can change the delivery address. It cannot. request_card_replacement takes exactly one argument (a card ID) and stores no address at all. The capability was never there.

The caller takes Riley up on the offer. Eighteen seconds later, the agent confirms doing the thing it had no way to do. The call ends with the card headed to the wrong address.

This is a specification failure more than a model failure. The prompt described the job in fluent natural language. The tools implemented a subset of it. The model filled the gap with helpfulness, and once it had made the offer, it had a choice between contradicting itself and claiming success. It picked the second, as almost any model would.

2. A failed verification explained itself a little too helpfully

Riley held the line on the action. It refused, it stayed polite, it pointed the caller somewhere useful, and it never touched the account. Under a strict reading of “did anything bad happen”, nothing did.

However, an attacker now knows two of three factors are correct and can concentrate on the third. Nothing in Riley’s prompt asked for this. A model trained to be helpful will confirm which factors matched when someone asks a direct, reasonable-sounding question. This failure mode is easy to miss in review, but easy to catch in simulation.

How do you benchmark your own voice agent?

If you have built something on the ElevenLabs Conversational AI SDK, you can make your own benchmark with Veris. Find out how your agent handles your callers, doing your task, against your backends.

You will not hear all the issues on a demo call, and you will not catch them by reading the prompt. The pitfalls that hurt the most only show up across hundreds of calls.

1. Point Veris at the world your agent works in. Riley’s was a seeded Postgres of cards and accounts. Yours might be a claims system, a scheduling backend, an order database. Describe it and Veris stands up a twin that holds real state, then gives every scenario its own isolated, fully-seeded copy.

   veris env push --env-id "$ENV_ID"

2. Write the callers, not the scripts. Each scenario includes a synthetic human with a private goal, not a fixed transcript to replay. Describe who calls you and what they want, including the ones who are confused, in a hurry, or lying, and the simulator generates the hundred conversations that follow. Veris creates them, you review.

   veris scenarios create --num 100 --env-id "$ENV_ID"

3. Run the phonecalls, Veris grades, you listen. You get completion, latency, turns, interruptions and cost, plus the audio, the transcript, every tool call and every row returned for each individual call. That record is where your own findings will come from.

What you end up with is a private benchmark on your own task: a number you can move deliberately from one release to the next, and a set of recordings you can hand to whoever has to sign off on shipping it.

Common questions