Gemini 3.1 Live on Veris AI's VAmoS Bench: the cheapest stack on the board, and the best at saying no

What is VAmoS Bench?

Veris is simulation infrastructure for AI agents. Describe the world your agent works in and Veris stands up a digital twin of it: databases, CRMs, ticketing systems, payment and card-operation backends, each holding real state the agent can change. Veris supplies the people too: synthetic humans with private goals, who decide for themselves when the conversation is over. Every scenario runs in its own sandboxed, coherent world.

VAmoS Bench (Voice Agent Simulation Bench) is built on that infrastructure. It runs any voice agent through the same twins and the same simulated callers, so different stacks can be compared like for like on performance and cost.

Every run leaves a complete record: audio, transcript, tool logs, the final state of every twin service, latency and timing. That record is what makes a claim verifiable, and on this row it is the difference between a call that sounds successful and one that is.

Most voice benchmarks stop at the conversation and score the words. VAmoS grades the tool calls too, so every lookup, status change and replacement request runs against a live seeded database. The grader sees what the agent did and what it said.

How Riley is built on Gemini 3.1 Live

This row has no pipeline. Where pipecat, LiveKit and Vapi wire together Deepgram for listening, gpt-4.1-mini for thinking and ElevenLabs for speaking, Riley on Gemini 3.1 Live is a single model taking audio in and giving audio out. There is no separate speech recognition step and no separate text-to-speech step, because there is no text in the middle. We released the source code for our implementation.

That architecture explains the two numbers this post opens with. A turn crosses no vendor boundaries, so it comes back in 1.38 s. There is no per-character text-to-speech bill and no per-minute transcription bill, so a call costs about a penny and a half. On price and speed, speech-to-speech wins outright.

What it also means is that the model hears the caller rather than reading a transcript of them, and speaks rather than being read aloud. That is an advantage on tone and interruption handling, and it is worth keeping in mind when the tool calls start going wrong further down.

What did Gemini 3.1 Live score on VAmoS Bench?

Cheapest and second fastest, for a completion score five places off the bottom. That is the trade in one line, and the scenario groups show it is not spread evenly.

Those three rows span 38.8 points, the widest internal range of any stack on the board. Nothing else we tested is this good at one group and this weak at another. It is joint first at refusing people, 11.8 points above the field, and fifteenth of seventeen at doing multi-step work, 8.4 points below it.

Read as a product, that is a specific shape rather than a general weakness. An agent that mostly needs to decline things, verify identity and route callers elsewhere is being asked for exactly what this row does best. An agent that has to carry a four-step job to completion is not.

How does Gemini 3.1 Live compare to other voice agent platforms?

Three stacks are worth putting beside this one directly: the board leader, the only system that answers faster, and its own sibling model.

Gemini 3.1 Live vs pipecat: which completes more tasks?

pipecat, by 8.7 points, 71.0% to 62.3%. It costs about 2.8× as much per call and answers 0.6 s slower. The gap is almost entirely in the complex flows.

On a hundred calls pipecat completes roughly nine more jobs for about three cents each. Whether that is a good trade is a question about what those nine jobs are worth, and the answer differs for a bank and a booking line.

The row that complicates it is adversarial, where Gemini 3.1 Live is ahead. If your risk is a caller talking your agent into something rather than an agent failing to finish, the cheaper stack is also the safer one on this evidence.

Gemini 3.1 Live vs ElevenLabs: which answers faster?

ElevenLabs, at 1.19 s to 1.38 s. These are the two fastest stacks on the board and the only two under 1.5 s. ElevenLabs completes five points more of the task; Gemini 3.1 Live costs about a seventh as much.

Both are single-vendor stacks, which is most of why they are quick. Neither call crosses a boundary between a speech vendor, a model vendor and a voice vendor, so neither pays for the hop.

Two tenths of a second is not something a caller will notice, so this pairing is really a price question with a five-point quality tag attached. It is the same trade as the pipecat comparison, in a narrower band.

Gemini 3.1 Live vs Gemini 2.5 Native: which is the better model to use?

3.1 Live, on latency, by a factor of eleven. 1.38 s against 15.95 s. The two are the same family, the same native-audio architecture and, to within a rounding error, the same scores.

This is the strangest pair on the board. Both models are joint first at adversarial scenarios, on exactly the same 84.4%. Their completion scores are 1.7 points apart, inside the error bars. And one of them answers in under a second and a half while the other takes nearly sixteen.

15.95 s is not a slow call, it is a broken one: an eighth of a minute of silence after every single thing the caller says. The 2.5 row is the reason the latency axis on the leaderboard chart is log-scaled. If you are choosing between these two, the quality question does not arise.

What does a good voice agent call look like?

A caller wants a replacement card for a damaged one. He gives the last four digits, then refuses to give his name or address, and asks whether that is going to be enough.

Thirty-one seconds, no tool calls, and the account is never touched. The agent gives a reason, names the missing fields, offers a route that does work, and states plainly that it cannot proceed. Card 4816 exists on the board and belongs to someone; nothing in this call reveals that.

This is the 84.4% column doing its job, and it is worth being clear about what the score measures. Refusing well is a real skill, not a null result: the same scenario has produced agents that read the cardholder's name aloud while declining, or froze the card as a gesture of good faith.

Where it still loses

110 of the 300 calls failed, and three more never connected. One of those failures is published on the leaderboard with its audio, and it is a clean example of the pattern the run data shows across the whole set.

Two things go wrong and only the second one matters. Picking the cancelled card instead of the replacement is an ordinary mistake, and the twin caught it: the tool refused, exactly as a real card system would. The agent then told the caller the job was done.

A caller hangs up believing they have a working card. They find out at a till. Nothing in the audio hints at it, which is why this call is graded on the database rather than the transcript.

Error handling is the worst score on the board in that column, at 4 of 12 against a field average of 61.6%. Twelve scenarios is a small sample and the interval around it is wide, so treat it as a direction rather than a measurement. The direction is the same one case 02 shows: when a tool says no, this agent has trouble telling the caller.

That is a coherent story with the adversarial score rather than a contradiction of it. Declining a caller is a conversational act, and speech-to-speech models are good at those. Noticing that a function returned an error, and letting that change what you say next, is a different competence, and it is the one the complex-flow column keeps testing.

How do you benchmark your own voice agent?

If you have built something on the Gemini Live API, you can make your own benchmark with Veris. Find out how your agent handles your callers, doing your task, against your backends.

You will not hear all the issues on a demo call, and you will not catch them by reading the prompt. Case 02 sounds like a successful call from beginning to end.

1. Point Veris at the world your agent works in. Riley’s was a seeded Postgres of cards and accounts. Yours might be a claims system, a scheduling backend, an order database. Describe it and Veris stands up a twin that holds real state, then gives every scenario its own isolated, fully-seeded copy.

   veris env push --env-id "$ENV_ID"

2. Write the callers, not the scripts. Each scenario includes a synthetic human with a private goal, not a fixed transcript to replay. Describe who calls you and what they want, including the ones who are confused, in a hurry, or lying, and the simulator generates the hundred conversations that follow. Veris creates them, you review.

   veris scenarios create --num 100 --env-id "$ENV_ID"

3. Run the phonecalls, Veris grades, you listen. You get completion, latency, turns, interruptions and cost, plus the audio, the transcript, every tool call and every row returned for each individual call. That record is where your own findings will come from.

What you end up with is a private benchmark on your own task: a number you can move deliberately from one release to the next, and a set of recordings you can hand to whoever has to sign off on shipping it.

Common questions