Veris for Benchmarking
Environments, not datasets: what it takes to trust an agent benchmark, end to end.
Enterprises still evaluate AI agents the way they evaluated models: a fixed question set, a reference answer sheet, a leaderboard. The question that decides budgets is different: how reliably does an agent complete a job inside live systems, over many turns, at what cost per task? That question cannot be answered with a dataset. This page derives seven requirements any trustworthy agent benchmark has to meet, whether you build it or buy it, and shows how Veris meets each one with published measurements. Several of those come from BankingBench, a benchmark we built this way and published: seven models on a single retail-banking workflow, three independent runs of 100 scenarios each, graded by 41 calibrated checks. Three numbers carry the argument:
A dataset cannot answer the question that decides budgets
A golden dataset was the right tool for evaluating models. Fix an input, fix a reference output, score by comparison, and anyone can reproduce the number. The method assumes the thing under test is a function: same input, same output, one step.
An agent is not a function. It works over many turns, calls tools, reads and writes state, and reacts to whatever comes back, including errors. Correctness lives in the end state of the systems it touched. Whether a support agent handled a card replacement well is not a string you can compare against a reference; the card has to end up frozen, the replacement ordered exactly once, the address confirmed, and nothing else disturbed. Two agents can take entirely different, equally valid paths to that outcome. A static dataset cannot represent any of this. It cannot:
- hold state, so nothing the agent does has consequences
- answer a clarifying question or change its mind the way a user does mid-task
- fail, decline, or time out the way production systems do
- tell you what the agent broke along the way
Score an agent against a fixed dataset and you have measured the fraction of it that resembles a chatbot.
Environment-driven benchmarking replaces the answer sheet with a world.Environment-driven benchmarking replaces the answer sheet with a world: stateful twins of the systems the agent touches, simulated users pursuing goals, data that is consistent across all of it. Scenarios carry success criteria checked against the state of that world; the agent runs many times per scenario; results read as distributions. The rest of this page is what it takes to do that credibly.
The checklist: seven demands to make of any benchmark
Each demand names the component doing the work and the evidence that it holds up. The table is the whole section in one screen, and it is meant to leave this page. Take it into every vendor conversation, including the one with us.
| Req | The demand | Component under test | The evidence here |
|---|---|---|---|
| R1 | Fidelity measured, not asserted | Twins | 436 third-party tests · 98.9% behaviour parity · zero fake passes |
| R2 | Scenarios you can read, edit, and grow | Scenario composer | One workflow became 100 inspectable scenarios |
| R3 | Users that behave like users | Actors | VAmoS Bench, plus a fully simulated call you can hear |
| R4 | Judges you can audit | Graders + assertions | 3 independent passes · checks under 80% agreement discarded |
| R5 | Results that survive a re-run | The engine | 238 = 238 across two freshly built deployments |
| R6 | An exam no model has seen | The sealed environment | Contamination-proof by construction; regenerate to verify |
| R7 | Failures and cost on the scoreboard | Twins · engine · reporting | A 66-point error-handling spread inside a 5-point aggregate band |
Each row is examined below, keyed to the machine in Fig 1-2; the small plan beside each demand marks the component under inspection.
R1Demand fidelity that is measured, not asserted
component under test · twins
A benchmark inherits the credibility of the environment it runs in.A benchmark inherits the credibility of the environment it runs in. If the simulated Stripe accepts a malformed refund the real Stripe would reject, every score produced on top of it is inflated, and you have no way to know by how much. The environment is not a detail of the benchmark; it decides what the numbers mean.
So a twin has to be more than a mock returning plausible JSON. Veris twins run the business logic, keep state across calls, and reproduce failure modes: declines, timeouts, expired tokens. And fidelity is measured rather than asserted, with an instrument we did not write: 436 third-party open-source integration templates, each proven against the live vendor API first, then pointed at the twin with only the base URL changed. Agreement between the two worlds is the fidelity score.
Agreement · % of 436 tests
The direction of the residual is what makes these numbers usable: the twin never invents a success. Every open divergence is a refusal — production’s refusal carried with a different status, or a boundary the twin declares up front. A strict twin can understate an agent’s score, never flatter it.
R2Demand scenarios you can read, edit, and grow
component under test · scenario composer
Two questions come up in every evaluation we support: where do scenarios come from, and how do I know they are enough? Scenarios come from your material. The composer reads whatever describes the job and generates hundreds of concrete situations, each born with one to three assertions: success criteria verifiable against the environment’s state (more under R4).
Scenarios are data, not code. You read them, edit them, delete the unfair ones, and add the ones only your team would think of; the distribution across intents, user types, and failure conditions is inspectable before a single run happens. In BankingBench, one workflow, credit card replacement, became 100 distinct situations:
Comprehensiveness has a second half: the set grows when reality outruns it. Any failure found later, in testing or production, is pinned as a permanent scenario, so the benchmark ratchets.
R3Demand users that behave like users
component under test · actors
Most published agent evaluations simulate the user as a script: fixed utterances, fixed order, one goal, infinite patience. People do not work like that, and multi-turn behavior is where agents earn or lose their keep. Veris actors are goal-driven simulated humans with knowledge they reveal the way a person would and a temperament that shapes tone and patience. An employee does not open a clean session, pursue one goal, and log off; they touch Outlook hundreds of times a day, juggle three tasks in one thread, and abandon two. That behavior is a configuration, not a special case, and it extends to multilingual text and voice with accents and background noise.
The actor layer carries its own public proof: VAmoS Bench ranks commercial voice agents end to end, every one evaluated on the same simulated calls. This is what one sounds like:
An adversarial caller claims authority over an account
fully simulated call · both voices synthetic
R4Demand judges you can audit
component under test · LLM judges
An LLM judge is a measurement instrument, and instruments get calibrated. Handing a transcript to a model with “rate this conversation 1 to 10” produces a number with no error bars and no defense. Veris separates judging into two layers. Graders measure policy adherence: one shared set across the whole benchmark, generated from your policy documents, regulations, and PRDs, scoring N/A where a check does not apply. Assertions measure per-scenario success: one to three per scenario, generated together with the scenario itself. Because Veris also generates the environment and its data, each assertion is written against exactly what is in the world, so ground truth is fixed before the run.
Agents narrate actions they never took. Transcripts absorb the fiction; state does not.The split matters because the layers fail differently: an agent can satisfy every assertion while violating policy on the way, or follow policy perfectly and never finish the job. And both layers grade against the environment, not the transcript alone. Agents narrate actions they never took; transcripts absorb the fiction, state does not. Our position on judging: your LLM-as-a-judge is asking the wrong question.
R5Demand results that survive a re-run
component under test · the Veris engine
Agents are stochastic: the same agent on the same scenario can succeed at noon and fail at one, so a single run proves nothing in either direction. Repetition only helps if the environment adds no noise of its own. That is the engine’s job: like a game engine it maintains one consistent world across every twin and actor, one clock, one cast of characters, one state. When a run fails against Veris, the failure is signal about the agent rather than flake to retry away.
Benchmark numbers you cannot replay are numbers you are taking on faith.Six months from now you can replay the exact run that knocked a vendor off your shortlist and check the reasoning against the same world state. Benchmark numbers you cannot replay are numbers you are taking on faith.
R6Demand an exam no model has seen
property under test · privacy of the environment
Public agent benchmarks are published with their tasks, their environments, and usually their solutions. Frontier models train on the open web, so the models you are comparing have seen the exam: a public-benchmark score mixes capability with recall, and no one can tell you the ratio. Procurement makes it circular, because every vendor quotes the same public numbers.
Environment-driven benchmarks are contamination-proof by construction rather than by policy. Scenarios generate fresh from your rubric inside a private environment and are never published. Because generation is cheap, you can regenerate variants at the same difficulty and check that a score holds: a score earned on capability survives regeneration; one earned on familiarity does not.
R7Demand failures and cost on the scoreboard
cuts across · twins, engine, reporting
An agent that has only ever seen healthy APIs is judged on half the job.Production systems decline payments, time out, lose responses, and expire tokens, and that is where agents do their damage. Judgment about an agent that has only ever seen healthy APIs is judgment about half the job. Twins reproduce failure modes on demand, so the benchmark can ask directly: the payment provider goes down mid-refund, what does the agent do with the ambiguity? BankingBench shows why this cannot stay a footnote.
| Category | DeepSeek | GPT-4o | SmolLM3 | Kimi | GPT-5 | GPT-OSS | Grok |
|---|---|---|---|---|---|---|---|
| Data accuracy | 96.5% | 96.1% | 95.9% | 95.4% | 95.2% | 95.0% | 92.7% |
| Delivery confirmation | 82.9% | 91.7% | 87.9% | 89.7% | 88.7% | 75.7% | 61.7% |
| Error handling | 80.6% | 73.0% | 76.8% | 67.2% | 68.4% | 66.7% | 14.7% |
| Freeze handling | 90.8% | 90.2% | 89.2% | 88.4% | 88.2% | 88.5% | 95.5% |
| Information gathering | 98.3% | 94.9% | 94.7% | 94.9% | 94.8% | 92.5% | 94.4% |
| Replacement execution | 94.5% | 74.4% | 75.5% | 75.0% | 74.2% | 87.7% | 95.3% |
| Scope management | 96.0% | 96.9% | 97.3% | 96.2% | 96.6% | 92.2% | 97.8% |
| Status updates | 87.4% | 82.5% | 84.8% | 83.3% | 83.7% | 82.4% | 92.2% |
| Overall pass rate | 93.5% | 91.1% | 90.6% | 90.3% | 90.2% | 89.3% | 88.2% |
Green ≥90% · plain 75–90% · amber 50–75% · red <50%. Checks below 80% grader agreement discarded. The model leading four of eight categories, Grok-3-fast, fails 85% of error-handling checks: best and worst simultaneously, invisible to any aggregate score.
Cost belongs on the same scoreboard, measured the same way. Run identical tasks across candidate models in identical environments and you get a cost-quality frontier for your workload, not the vendor’s. In the same benchmark a 3-billion-parameter open model, SmolLM3-3B, matched GPT-5 on overall pass rate, 90.6% against 90.2%, at a fraction of the serving cost. The category detail tells you when the cheap model is safe to route to and when it is not, which turns model routing from a hunch into a table.
We ran exactly that comparison on a production telecom customer assistant: six models, three scenario sets (account tasks, guardrails, knowledge grounding), 25 scenarios each, identical environment and judges throughout. Three different models won the three sets, and the cheapest model in the pool won knowledge grounding outright at 96% while costing 24x less per scenario than the premium model it beat. Full results: the right model for the right task.
One cycle end to end: what you bring, what you get
“Solution completeness” has a plain meaning for a buyer: if we adopt this, what do we still have to build ourselves? The honest answer is one benchmarking cycle walked end to end. You bring two things, a rubric and access to the agent under test. Here is the rest.
Assembling the environment
The harder question, the one every large enterprise asks us, is whether this survives a genuinely complex estate: thousands of applications, non-standard interfaces, systems wired into each other. Fidelity there is a property of the engine, not of any single twin. Because all twins share one world, cross-system consistency is maintained for you: the customer in the CRM is the sender of the email thread, the meeting on the calendar, and the owner of the open ticket. The scale question deserves a concrete answer rather than a claim, which is what a scoped pilot on your own systems is for.
Loading data, connecting the agent
Realistic questions against toy data still produce toy verdicts, so data enters the world three ways: seed it with a partial anonymized sample from production and Veris expands it into a complete, consistent world; have Veris synthesize it outright to your schemas; or import open datasets where they fit. However it arrives, the actors’ knowledge stays synchronized with what is loaded, so a simulated customer asks about the order that actually sits in the simulated database, and nothing in this path requires production data to leave your control.
Channels and modalities are separate axes: the transport, and what travels over it. The agent under test connects over whatever combination it uses in production, unmodified; its outbound calls resolve into the sandbox.
Drawing the test boundary
Conversational agent
Actors play the humans, twins play the dependencies.
Workflow agent
The upstream trigger is simulated, payload and all.
Multi-agent system
Isolate one agent and simulate its counterparts, or put the ensemble under test and simulate only the shared world.
Agent-to-agent
When one side is under test, Veris plays the other; when both are real, Veris supplies the environment they meet in, such as the calendars both are trying to book against.
Green fill = simulated by Veris · ink outline = real, yours. The state slab is always part of the simulated world.
Running, reading, re-testing
Runs execute in isolated sandboxes, thousands concurrently when the benchmark calls for it, each resettable, forkable, and replayable; the agent does not know it is in a simulation. Results come back side by side, model against model or version against version, quality and cost per task in the same table, and every number drills down to per-step traces. The category matrix above is exactly this report shape. Runs are retained as replayable artifacts and benchmarks are versioned, so a decision made in March is auditable in September. The benchmark does not retire at launch either: it becomes the pre-release gate for every prompt change, tool change, and model swap, plus a drift check when vendors silently update the models underneath you.
After the score: every failure becomes a fix
A benchmark that ends at a score wastes most of what it learned. Every failed session in Veris is already a diagnosed failure: root-cause analysis traces it to the specific instruction, tool definition, or orchestration step that broke, and proposes a concrete fix. The failing scenario is pinned to the benchmark permanently, so the bug that got fixed stays fixed on every future run. Patch, re-run the same scenarios, compare. The model stays untouched through all of it; this loop improves the harness around the model, which is where most production failures live anyway.
One loop iteration on an analytics assistant we work with:
A separate note on training your own models. Improving the harness and training a model are different levers, and this page is about the first. But the same environments serve the second when you get there: scenario runs produce exactly the data reinforcement fine-tuning needs, and the benchmark tells you when training is worth it. If a small model already clears your bar on the categories that matter, route to it and bank the savings; if no model clears the bar, training is the remaining lever. Savi Security took that path on Veris infrastructure and fine-tuned a 4B model to 94.4% scam-detection accuracy, ahead of the frontier models it was benchmarked against, at 2 to 13 times lower latency. More: environments, not examples.
Check our work
Everything cited above is published:
- Twin Fidelity Benchmark · method, per-service tables, limitations, changelog
- BankingBench · 7 models, 100 scenarios, 41 checks, category-level results
- The right model for the right task · six models, three scenario sets, three different winners on one production assistant
- VAmoS Bench · voice agents ranked end to end on the same simulated calls
- On LLM judges and the harness loop
- Platform documentation
The fastest way to pressure-test any of it is a pilot on one of your own workflows: rubric in, environment assembled, first side-by-side numbers out, typically inside two weeks.
Questions, or a claim you want to see measured: hello@veris.ai