Stop grading your agents on someone else’s exam
Veris is a simulation sandbox for building custom AI agent benchmarks around your own policies, workflows, and systems. Evaluate the agents you buy and the agents you build, end to end, against the job they actually have to do.
A public leaderboard is not your acceptance test
Useful for comparing models in the abstract. Useless for deciding whether a specific agent will do your job, in your systems, under your rules.
It grades a reply, not a job
A dataset fixes one input and compares one answer. An agent works over many turns, calls tools, and writes state — none of which a string comparison can see.
The model has already seen it
A public set is public to the training pipeline too. A strong score stops being evidence of capability.
It measures a generic job
Someone else's task, someone else's definition of done. Not your disclosure rules, your thresholds, or your systems.
The average hides where it breaks
A leaderboard is one score per model, so the tasks that would break you are averaged in with the ones that would not. Seven models finished within five points of each other. Pull out the error-handling tasks alone and the same seven ran from 14.7% to 80.6%.
Every piece a benchmark needs, and none of it generic
The agent under test is the candidate. Everything around it is the benchmark, and every layer of it is a place a benchmark quietly goes wrong. You bring the candidate, your policies, and your systems. Veris builds the other seven.
Environment
The world the candidate acts in: the services it calls, the data seeded behind them, and the clock they run on.
Actors
Whoever is on the other side. Simulated humans and upstream triggers that drive the candidate through its real interface.
Task corpus
The exam questions. Scenarios generated from your rubrics, policies, PRDs, and production traces, each with success criteria fixed before the run.
Harness
What actually executes the runs. Isolated, deterministic, and replayable, so a failure is signal about the agent rather than flake to retry away.
Verification
How a run is graded. Against the end state of the world rather than the transcript, because a transcript will absorb any fiction the agent tells about it.
Bench
The benchmark itself, versioned as one object. A version pins the scenario set, the grader spec, the protocol, and the world, so two candidates months apart are still comparable.
Reporting
What you take to the decision. Scores per category as distributions, the failures named and traceable, and cost per task on the same scoreboard.
We build these in public first
Before asking anyone to trust a benchmark we built for them, we published ones anybody can check, with the papers, the agent, and the failure cases in the open.
VAmoS Bench
Seventeen voice-agent configurations run the same credit card support task through the same simulated phone calls, compared on observed end-to-end task completion, with barge-in, latency, and cost on the same scoreboard.
BankingBench
Seven models, three runs of a hundred scenarios each, scored per category. The result that matters: the model leading four of eight categories failed 85% of error-handling checks, invisible to any aggregate score.
Bring the job. We will build the exam
Tell us what your agents have to do and which systems they touch. Or take a sample report with you first.
Questions we get asked
$Where can I read the methodology in full?
Veris for benchmarking sets out the seven requirements any trustworthy agent benchmark has to meet, how each layer is built, and the published measurements behind each claim. The twin fidelity numbers have their own report at veris.ai/fidelity.
$How is this different from a public agent benchmark?
A public benchmark scores a generic job and is public to the training pipeline as well as to you. Veris builds the benchmark around your policies, your workflows, and twins of your systems, so the exam is one no model has seen and the result answers your question rather than someone else’s.
$Can we benchmark a vendor's agent we cannot see inside?
Yes. A vendor’s system is evaluated as a black box through its API endpoint, or through a phone number for a voice agent. It runs inside the twin against simulated systems and synthetic records, so it never learns your production topology and takes nothing home.
$Do you need our production data?
No. The default is synthetic data carrying the shape, logic, and messiness of yours without containing anyone’s actual information. If you would rather use your own, Veris deploys into your cloud or on-prem, so the data stays inside your existing controls.
$How do we know the twin behaves like the real system?
Because we measure it and publish the result rather than asserting it. Every test is first proven against the live vendor API, then run unmodified against the twin, and every divergence is counted: 96.8% outcome agreement across 372 integration templates, with no fake passes caused by a twin defect. Read the fidelity report.
$Who writes the scenarios, and why should we trust the grading?
Veris generates scenarios and their success criteria from your code, prompts, and production traces, and the graders from your policy documents. You review and approve before anything runs. Judges are calibrated as a panel, and any check falling below the agreement threshold is discarded rather than reported.
$Will the same benchmark give the same answer twice?
Yes. Runs are seed-pinned and reproducible, with the full conversation and tool-call trail behind every verdict. In a repeat study across two fresh deployments the benchmark returned 238 identical verdicts out of 238.
$Does this work for voice agents as well as chat and tool-calling?
Yes, and both are published. VAmoS Bench ranks voice agents on simulated phone calls, with barge-in and latency on the scoreboard; BankingBench scores tool-calling agents per category across seven models. The same personas, scenarios, and verifiers apply across modalities.
$What do we actually walk away with?
A decision packet rather than a dashboard: per-category scores as distributions, the failures found and exactly where, cost per task at your volume, and a reproducible record of every run behind it. The artifact your risk committee and your auditor both need.