Why an agent passes its eval set and fails in production
An eval set fixes one input and compares one answer. Production is a customer who changes their mind mid-call, a tool that times out on the second attempt, a record that someone else already updated. None of that exists in a dataset, so an agent can score well and still fail the moment a user goes off script.
The pattern shows up whenever agents are measured on outcomes rather than replies. On VAmoS Pro Bench, a utility billing agent run through chained phone calls, one stack's task completion fell from 38.7% to 8.6% when a television was playing in the caller's room, and stacks built on the same language model landed anywhere between 17% and 36% depending on everything around the model.
Three things break an agent that a static test never exercises.
- The user interrupts, withholds a detail until asked, asks for two things at once, or gets angry.
- The tools return errors, rate limits, partial data and stale records, and sometimes succeed after the agent has already given up.
- State carries across turns and across tools. The payment arrangement the agent promised has to exist in the billing system afterwards, not only in the transcript.
The five checks
Whatever tools you use, a pre-production test has to answer five questions.
- Task success. Did the agent complete the job, measured on what changed in the systems rather than on whether the reply sounded right?
- Tool use and trajectories. Did it call the right tools with valid arguments, in an order that makes sense, and stop when it should?
- Simulation and edge cases. What happens with an ambiguous user, missing data, an API timeout, a rate limit, a tool failure, a record that changed underneath it?
- Safety and security. Prompt injection, unauthorized actions, data leakage, permission boundaries, and the policies your industry imposes.
- Regression and performance. Did this version get worse than the last on task success, latency, cost per task or failure rate, and can you tell a real change from noise?
Most stacks cover the first, second and fifth with datasets and scorers, and the fourth with red-teaming. The third is where the sandbox comes in, and it is the one that predicts production.
The approaches, and the tools for each
| Approach | What it answers | Tools |
|---|---|---|
| Golden sets and LLM-as-judge | Does the output match an expected answer or a rubric? | Braintrust, LangSmith, Langfuse, Arize Phoenix, promptfoo |
| Red-teaming | Can the agent be talked into something it should not do? | promptfoo; adversarial scenarios on a Veris bench |
| Tracing and monitoring | What did the agent do in a real run, and what did it cost? | Arize, Langfuse, LangSmith, Datadog |
| User simulation | How does the agent handle a realistic multi-turn user? | openevals with LangSmith, promptfoo's simulated user, Sierra and Decagon simulations for their own agents, Google Agent Platform's user simulator, Coval and Cekura for voice, Maxim |
| A stateful environment | What happens to the systems behind the agent, under faults, at scale, repeatably? | Veris, or an in-house sandbox |
The rows are not alternatives. A production team usually runs one from each of the first three, and the last two are what separate a test from a rehearsal.
The controlled sandbox, concretely
When a guide tells you to build a controlled sandbox with mocked APIs, synthetic users and reproducible state, this is what each phrase means in practice.
Mocked APIs become twins
A mock returns what you told it to. A twin behaves like the service: it holds state between calls, enforces the vendor's lifecycle rules, returns the vendor's error shapes, and nothing it does is real. Veris ships twins of more than 70 enterprise and SaaS services and builds a custom one from an API spec in 24 to 48 hours.
Synthetic users become simulated people
A simulated user has a goal, facts it knows and facts it will only reveal when asked, and a way of speaking, and it stays in character for the whole conversation. On the phone it has an accent, a background and a tendency to interrupt. Voice, noise and persona vary while the task and its checks stay fixed.
Reproducible state becomes a seeded world
Before every attempt the twins are restored to that task's starting data and the clock is fixed, so repeats start from the same place and two candidates face the same situation. The result is a pass rate with a confidence interval, not a single run.
Tool failures become injected faults
Timeouts, errors and stale records are put into the twins on purpose: a rate limit on the second call, a record that changed underneath the agent, a service that hangs. The question is whether the agent copes, recovers, or quietly does the wrong thing.
Grading moves from the transcript to the world
Each task carries pass criteria that describe outcomes: the arrangement exists in the billing system, the case was closed with the right disposition, the review happened before the filing. Code checks what it can, and a narrow judge decides the rest from the transcript and the agent's trace.
Build it or use it
Teams do build this themselves, and the first version is not hard: a few mocks, a scripted user, a handful of scenarios. What is hard is the second year. Mocks drift as vendors change. Each incident needs to become a family of tests rather than a one-off. Grading on outcomes needs code checks and a judge you can trust. And the people maintaining the sandbox are the people you hired to build the agent. The teams we work with reached the same conclusion from different directions: testing was taking longer than building.
Veris is that sandbox as a product: the twins, the simulated users, the seeded worlds, the faults, the grading and the reports, run nightly or on every pull request, as SaaS or fully on-prem inside your VPC. How it sits next to your eval and tracing tools has its own page.
A starting checklist
- Pick one workflow that writes to a system and matters when it goes wrong.
- Write pass criteria as outcomes, and have the people who own the policy sign them before the first run.
- Cover happy paths, edge cases and adversarial cases, with at least a few dozen tasks and more than one user persona.
- Seed the systems the agent touches, and inject the faults production will produce: timeouts, errors, stale records.
- Run every task more than once and set a floor on the pass rate, judged on the confidence interval rather than a single number.
- Run it nightly, run a smaller set on each pull request, and turn every production incident into a new task.