Evals vs. Benchmarks: Every Agent Attempt Needs Its Own World

We think AI agents that act should be benchmarked, not just evaluated. The two words often get used interchangeably. They describe different setups, and for agents the difference decides whether a score can be trusted.

An eval, as most teams build one, is a golden dataset: a list of inputs, each with an expected output. You send every input to the agent and score what comes back. When the agent only answers questions, that takes you a long way.

It stops being enough once the agent acts: it updates a record, cancels an order, opens a ticket. Two things break.

A benchmark, in the sense we argue for here, fixes both. Every attempt runs in its own isolated copy of a defined world, starting from the same state, and is graded on what it did there: the calls it made and the state it left behind.

The golden-dataset model

Evals are useful, so it is worth being precise about what they do. The system under test might be a model, or a model connected to tools and external services, and the scoring can take several forms.

These are useful checks. But in this setup the dataset is the center of the system, and every run still uses the same underlying services, integrations and state.

For a read-only task, such as answering questions against a fixed retrieval corpus, that shared state may be perfectly reasonable: one attempt does not change what the next one can read. Once the agent can write, that assumption no longer holds.

Shared state makes independent tests interfere

Imagine testing an agent that processes an order cancellation. You launch two attempts against the same test order. The first attempt cancels it. The second finds an order that has already been canceled.

The second attempt is no longer solving the same task under the same conditions. Its outcome depends partly on what the first attempt did, and possibly on which attempt reached the service first.

This is why running more test inputs concurrently is not enough. When attempts can modify the same records, they can interfere with one another. You either prevent those collisions, serialize the conflicting work, or accept that the results are entangled.

Not every pair of tasks has to conflict. The point is that a shared, mutable environment does not give you independence by default.

A retry needs a fresh starting point

Concurrency is only half the problem. An agent can take a different path, and reach a different outcome, on repeated attempts at the same task. One successful run does not tell you how consistently it will succeed.

To measure that variation you need repeated attempts. But repeating the prompt is not the same as repeating the experiment. If the first attempt changed an order, wrote a database record or created a ticket, the next attempt inherits those changes unless you reset them. You are now varying both the agent's behavior and the environment it meets.

So each repetition needs the same defined starting state, unless you are deliberately testing a different condition. And the reset has to cover the relevant state across all of the agent's dependencies, not just its conversation history.

With the world held fixed, the spread is the agent's. Claude Sonnet 5.5 solves 64.7% of cases on all three attempts and 68.6% on at least one, so very little of its score is luck. Gemini 3.8 Flash solves 42.5% every time and 67.3% at least once. That 25-point gap is invisible to a benchmark that runs each case once, and meaningless in one that lets each attempt inherit the last one's writes.

A benchmark needs a world, not just a list of inputs

This is where simulation-based benchmarking comes in. Instead of treating a test as an input and an expected output, treat it as a task inside a defined world.

Each attempt gets its own sandbox, holding the task-relevant dependencies and their starting state. The agent can read, write, call tools and change that world without changing the world of any other attempt.

The unit that matters is the attempt, not just the task. Run the same task ten times, and each attempt should begin in its own fresh instance of the intended world.

That world has to include the state behind the integrations the agent uses. Putting the agent in a container while its tools still write to a shared database does not solve the problem. The boundary has to extend to the task-relevant mutable state, whether the dependencies are sandboxed services or simulated equivalents.

A note on words: I use "benchmarking" here for this controlled, simulation-based setup. I am not claiming that every benchmark is isolated, or that evals cannot use sandboxes. The distinction is architectural. What matters is whether the testing system controls the environment of each attempt.

What isolation unlocks

Take the same request to cancel an order. It can be tested in a world where the order is still pending, another where it has already shipped, and another where it has already been canceled.

Grade what the agent did, not what it said

This is the heart of the difference. An eval scores what comes back: an answer, a transcript, a trace. For an agent that acts, what comes back is a claim about what it did. The evidence lives in the systems: whether the payment was recorded, the case was moved, the ticket was opened, once, on the right account.

A benchmark can grade the evidence because it owns the world the attempt ran in. When the attempt ends, every record and every request the agent made is still there to read, so the pass criteria are written against what happened. That also gives quality and compliance checks a real context: define the workflow that applies, create the conditions that exercise it, then inspect both the agent's actions and the state they left.

A convincing answer is not proof the work was done. Neither is a satisfied customer.

On VAmoS Pro Bench, our voice-agent benchmark on a utility's billing line, the simulated caller hung up satisfied on 2,072 calls: everything it had asked for was done, as far as it could tell from what it heard. More than half of those calls, 1,122, failed the grade, and on 412 of them the agent had left the account wrong. The caller cannot see the account. The benchmark can.

Evals still matter. They need a controlled environment.

None of this removes the need for evaluators. Exact-match checks, LLM judges, tool-call validation and trace inspection all still have a role. They tell you how to assess what happened.

The isolated world answers a different question: under what conditions did it happen? Without that control, a score can reflect the changes other attempts introduced as much as the behavior of the agent being tested.

At Veris, this is the distinction we make with simulation-based benchmarking: moving from scoring runs against shared integrations to measuring agent behavior through controlled, isolated attempts. The goal is not to give a golden dataset a different label. It is to make the environment part of the test.

Read next

Why enterprises need a centralized AI evaluation platform Vendor-provided evals help you test one product. Independent benchmarking helps you set the bar for every AI system you buy or build.