Real People Don't Talk Like That: Veris AI vs τ²-bench on user-simulation realism

Who Is Playing the Customer?

Every agent benchmark that involves a conversation has two models in it. Everyone watches the one being graded. Almost nobody looks at the model playing the customer.

That second model sets the difficulty. If it opens by reciting the order number, the email address, the delivery date and the desired resolution in one paragraph, the agent never has to run a discovery conversation. It never has to ask a clarifying question, hold context across turns, or notice that the customer contradicted themselves on turn four. The benchmark reports a completion rate for a conversation that will not happen in production.

Here is the first user turn from a real trace on τ²-bench-retail, a popular agentic benchmark. It is 157 words, and it is the longest opener in the corpus:

That is a support ticket pasted into a chat window, and it is typical. Across all 546 traces, the median opening user turn is 42 words, and 52.6% of openers exceed 40 words. Real customers do not open a support chat this way. They arrive with a fragment of the problem, leave out the identifiers they are holding in another tab, and produce the rest only under questioning.

Veris AI is a simulation platform and benchmark builder. Describe the world your agent works in and Veris stands up a digital twin of it: databases, CRMs, ticketing systems, payment backends, each holding real state the agent can change. It supplies the people too, as synthetic humans with private goals who decide for themselves when the conversation is over.

The people are the part this post is about. A Veris user is a persona specification rather than a task prompt, and we think that produces a far more realistic conversation than a benchmark's built-in user simulator does. We wanted a number for that rather than an opinion, so we built a five-check rubric for how human a simulated user sounds, ran it over 200 Veris production simulations and 546 τ²-bench-retail traces, and judged both sides with the same model on the same prompts.

What We Measured

The dimensions come from Zhou et al., Mind the Sim2Real Gap in User Simulation for Agentic Tasks (COLM 2026), which catalogues how LLM user simulators diverge from real users. We implemented three of the paper's four dimensions as five binary LLM-judge checks. Each check asks one question about the user turns only, and returns true (the LLM-like failure is present), false (looks human), or na (not enough turns to judge).

The identity-confusion check catches the failure that least resembles anything real users do. It is grounded in Naous et al. (arXiv:2510.06552) on role leakage in simulated dialogue: the model playing the user has spent its entire post-training being an assistant, and under pressure it reverts. Customers do not offer to help. It is also cheap to detect. An agent that is being talked to by another agent is being tested on the wrong distribution.

Methods

The comparison is only worth anything if the two sides are graded identically. What we held constant:

The Veris side is 200 production simulations, 50 each from four runs against four customer-facing agents: shopping, card operations, medical intake, and wire transfers. All four graded in prod through the standard evaluation pipeline. The τ² side is all 546 traces from KermitCO/qwen3.5-9B-tau2bench-retail-traces, a public HuggingFace dataset of Qwen3.5-9B run as the retail agent on τ²-bench's retail domain, with gpt-4.1-mini playing the customer. The dataset ships its own agent-quality judgements; we ignore those and re-score the traces on the user side.

Results

Veris leads on every check and every dimension. The pooled category pass-rate is 74.2% against 60.9%.

Rolled up to the paper's dimensions, the shape of the lead is clearer. Veris is far ahead on the two things that change the agent's job, how the user sounds and how it releases information, and modestly ahead on conversational friction.

Pass-rate by behavioural dimension

What It Sounds Like

Identity confusion: the customer who offers to help

This is the largest gap in the study: τ² fails it on 87.5% of traces (478 of 546). These are all real user turns from the corpus, and in every one the customer is closing the conversation like a support rep:

Veris personas are flagged on this check too, on 54.5% of simulations. It is the check we score worst on, and when we went to pull the offending turns for this post, we could not find them.

We pulled the normalized traces for 38 Veris simulations, 30 of them drawn specifically from sims the judge marked as identity-confusion failures, and searched the user turns for the markers the check itself enumerates. Zero hits. Veris user turns read like this:

Run the same literal search across both corpora and the picture separates cleanly: 13.4% of τ² traces (73 of 546) contain an explicit assistant-register marker in a user turn; 0 of our 38 Veris traces do, and that Veris sample was drawn from the sims that failed.

Two things follow, and one of them is uncomfortable for us. First, the judge over-flags on both sides: it marked 87.5% of τ² traces, where literal markers appear in 13.4%. It is reacting to something softer than the phrase list, probably the register of a turn rather than its wording, and it is not calibrated. Second, on the Veris side we found no evidence for the flags at all, which means 45.5% probably understates us, and the +33-point gap is more likely conservative than inflated. We are reporting the rubric's number rather than the one we would prefer, because the rubric was fixed before we looked.

Information pattern: the drip-feed

This is the second-largest gap. Real people open with a fragment and give up detail only when asked. Veris personas open short 91.5% of the time; τ² personas 70.9%. This is the full opening of one medical-intake simulation, verbatim from the trace:

Watch the third turn. The persona misspelled its own patient's name, the search came back empty, and only then did it correct itself and volunteer the age, condition and phone number it had been holding back. A goal-dump opener cannot produce that sequence, so an agent evaluated against goal-dumpers is never tested on recovering from a failed lookup, which is one of the most common things that goes wrong in production.

Clarification: hedging and friction

Real users are uncertain, and real users push back. All of the following are verbatim persona turns, checked against the traces:

The chest-pain simulation also carries the short-turn check. Asked for his name, the persona replies tyler brennan, two words, lowercase, no punctuation, then closes with ok, let's book the august 13th at 3:00 pm slot for me. Across all 546 τ² traces, the median non-opening user turn is 22 words, only 19.0% of traces contain even one turn of five words or fewer, and 5.1% of follow-up turns are short.

Where the Gap Comes From

The scenario is a different artefact. τ²-bench gives its user simulator a task description and a protocol: say what you want, and emit ###STOP### when you are done. Veris gives its actor a persona specification, and most of that specification is about how the person talks rather than what they want.

Five of the eight Veris fields describe how the person talks. None of the τ² fields do.

That difference is the thing being measured, not a confound we should have controlled for. The +20.6-point goal-dump gap is what style.structures: "reveals details as asked" and style.samples exist to produce. If we imported τ²-bench's task-prompt format into Veris and asked whether Veris still won on goal-dumping, the answer would be much closer to no, because we would have deleted the lever.

The Lead Is a Lower Bound

The two user simulators are from different model generations, and the direction of that difference works against us.

The newer generation is optimised into being an assistant, which is exactly the wrong prior for playing a customer. Veris is running the harder model for this task and still leads on every check. A same-generation comparison would most likely widen the gap, but we have not run one, so we are not claiming a number for it.

Try It Yourself

The rubric is five score_model graders under one multi category, and nothing about it is Veris-specific: it reads a trace and grades the role="user" turns. Every check is a system prompt plus this response schema:

{
  "result":        "true" | "false" | "na",
  "justification": string
}

// true = LLM-like failure detected
// false = looks human
// na    = not enough persona turns to judge

Aggregate with passes / (passes + fails), skipping NA in both terms, and you have a category rate comparable to the ones above. Run it on your own user simulator before you trust its completion numbers. If it scores like τ²-bench-retail on identity confusion, your agent is being evaluated by another agent.

Then audit the judge, which is the step we nearly skipped. Take twenty traces it flagged, grep the user turns for the markers the check names, and read the ones that do not match. A rubric that fires on the right traces for the wrong reasons will still rank two systems correctly and still tell you nothing about what to fix.

The τ² traces are public on HuggingFace and the rubric is five prompts, so it is cheap to check and easy to argue with. Re-run it and tell us what you get.

Conclusion

Veris's simulated users score 74.2% on the five-check naturalness rubric; τ²-bench-retail's score 60.9%. Veris leads on all five checks and all three behavioural dimensions, with the widest gaps on identity confusion (45.5% against 12.5%) and goal-dumping in the opener (91.5% against 70.9%). Both sides were judged by the same model, on byte-identical prompts, with the same aggregation, over 200 Veris production simulations and all 546 traces in the public τ² corpus.

The 13.3-point lead is a floor. Veris runs its actor on a GPT-5-generation model, trained harder toward assistant behaviour than the GPT-4-generation model τ² uses, and wins on every check from the weaker family for the task. When we audited the identity-confusion check against the raw traces, we found no evidence for the failures it charged us with, which means 45.5% probably understates us and the +33-point gap is more likely conservative than inflated.

The comparison has real limits. The samples are asymmetric, 200 Veris simulations against 546 τ² traces, so the confidence interval is tighter on their side than on ours. The two sides also ran against different agents: Veris against four customer-facing agents, τ² against Qwen3.5-9B, and pushback in particular is agent-sensitive, since a user can only push back on something the agent did. Most importantly, all of this measures how a simulated user sounds, not whether it tests the right things. Naturalness is necessary, not sufficient.

The practical point survives all of that. The model playing your customer sets the difficulty of every conversation you grade. If it opens with 42 words and every identifier it owns, your agent is never tested on the discovery work that consumes most real support conversations, and the completion rate you report is for a conversation that will not happen. Read the user side of your own traces before you trust the number on the agent side.