Your voice agent eval is grading the transcript, not the outcome

How do you know a voice agent is ready for customers?

August 2026 | Veris AI

There are good practical guides to this now. Coval published one of the better ones: start simple with a spreadsheet and an LLM judge, iterate on prompts, add audio quality and workflow completion, then move to continuous monitoring across cohorts. It is right about the things that make voice hard. Latency compounds across turns in a way it never does in chat. Speech recognition and synthesis introduce failure modes that text evals cannot see. Conversations branch, so a single golden path proves almost nothing. Interruption handling is real work.

We agree with all of it. It is also, on its own, not enough to sign off a deployment, and the reason is structural rather than a matter of adding more metrics.

Every metric in that stack is measured on the conversation. Latency, transcription accuracy, function calling, workflow completion, success rate: each one is read off the call. But the customer does not live in the call. They live in the billing system, the card ledger, the CRM. What matters is not whether the agent said it froze the card. It is whether the card is frozen, exactly once, on the right account.

A transcript will absorb any fiction the agent tells about it.

1. The agent that says it did the thing

Here is a failure we found on our own voice benchmark, in a stack that scored above the field average in every scenario group.

No amount of audio quality scoring finds that. Function calling accuracy does not find it either, because the function was called correctly, with well-formed arguments, and returned a success. The only thing that finds it is checking the end state of the card ledger against what should have happened, and having an account there to check.

This is what we mean when we say a benchmark should grade the world rather than the reply. Every scenario carries assertions written against the state of the environment and fixed before the run: the card is frozen, exactly once, on the account that was verified. The verdict comes from the database, not from the agent's account of the database.

2. You simulated the user. Did you simulate the systems?

Most voice testing tools simulate the caller. That is the visible half of the problem and the easier half. The other half is everything the agent reaches for once the caller asks for something: the card operations backend, the payments rail, the CRM, the ticketing system.

If those are mocks, they return canned responses forever. A mock cannot be double-charged. A mock cannot have a webhook arrive late. A mock cannot hold a row you wrote a minute ago and hand it back inconsistently. The failures that reach production are precisely the ones that need state to exist, and they are invisible to a test rig where the backend is a fixture.

Veris stands up stateful twins of those services, over seventy of them off the shelf, with a new one built from an API spec in about forty-eight hours. They hold state, run business logic, and fail the way production fails. You can inject a fault mid-call and see what the agent does when the payment API times out on the retry.

3. Nobody checked whether the simulated caller sounds like a person

Synthetic users are standard in every one of these guides. What is not standard is measuring them.

This matters more than it sounds. If your simulated caller opens with a tidy 157-word statement of their problem, in complete sentences, with every detail volunteered up front, then you have built an evaluation of how your agent handles a caller who does not exist. Real people interrupt. They withhold the account number until asked. They change their mind halfway. They say "wait, actually" and reorder their own priorities.

We scored our simulated users against τ²-bench-retail on a five-check human-likeness rubric, judged by the same model on byte-identical prompts: 74.2% against 60.9%, ahead on all five checks. The largest gap was on never sounding like an assistant, where the baseline passed 12.5% of the time and ours passed 45.5%.

Whether or not you use our numbers, the question is the one to ask of any voice testing tool: how human-like are the callers, and how do you know?

4. Who grades the grader?

"Use an LLM as a judge" is sound advice and incomplete advice. A judge is a model with a prompt, which means it has a failure rate, and an eval whose judge is unreliable is worse than no eval because it produces confident numbers.

Two things make judging defensible. First, run it more than once: we score every check on three independent passes and discard any check where agreement falls below eighty percent, rather than reporting it. A check the judges cannot agree on is not a result. Second, prefer state assertions over judgement wherever the question admits one. Whether the card is frozen is not a matter of opinion, and it should not be handed to a model to have an opinion about.

The related question is reproducibility. If the same benchmark gives a different answer on Tuesday, a failure is just flake you retry away. Runs are seed-pinned and replayable; in a repeat study across two freshly built deployments the benchmark returned 238 identical verdicts out of 238.

5. One score hides the failure you are buying

The guides talk about picking a success-rate threshold, eighty percent here, ninety-nine point nine there. The threshold is not the problem. Aggregating to a single number is.

On one of our benchmarks, seven models finished within five points of each other. Pull out the error-handling tasks alone and the same seven ran from 14.7% to 80.6%. A sixty-six point spread, invisible in the headline. On another, the model leading four of eight categories failed 85% of its error-handling checks.

If you are choosing a vendor on an aggregate score, you are choosing on the average of the tasks that would break you and the tasks that would not. Scores need to come back per category, as distributions with error bars, or they are not telling you what you think they are.

6. Cost is a quality metric

Cost is missing from most voice evaluation writing, which is strange, because it is the axis where the answer most often flips.

Across the seventeen voice stacks on our public board, cost per call spans $0.016 to $0.208, a thirteen-fold range, and the three stacks on the quality-cost frontier are not the three most expensive. In a separate bake-off on a production telecom assistant, one model spent 24× more per scenario than another on knowledge grounding and scored twenty points lower. The cheapest model in that pool won the category outright at 96%.

Quality, latency and cost belong on the same scoreboard, because a stack that is two points better and four times the price is a different decision than a stack that is two points better and free.

7. Is the sandbox itself right?

One last question, and the one most likely to be skipped: if you test against a simulated environment, how do you know the simulation behaves like the real thing? A confidently wrong sandbox produces confidently wrong results, and nothing in the eval will tell you.

The only honest answer is to measure it against an instrument you do not control. Every test is first proven against the live vendor API, then run unmodified against the twin, and every divergence is counted: 431 of 436 third-party verdicts agree with production, with no fake passes caused by a twin defect.

What to ask

If you are evaluating voice agents, or evaluating a tool that evaluates them, the checklist in the guides is the right starting point. These are the questions that come after it.

None of this is an argument against measuring latency or transcription accuracy. Those matter, and a team that is not measuring them yet should start there. It is an argument that passing every conversational metric is compatible with freezing the wrong customer's card, and that the gap between those two facts is where the risk of a deployment actually lives.

We built VAmoS Bench to demonstrate what this looks like in public: seventeen voice stacks, the same hundred scenarios, three runs each, graded on end state with cost and latency on the same board. If you want the same thing built around your own policies, systems and callers rather than our fictional bank, that is what Veris does.