Can you benchmark a vendor that will not cooperate?

The bench reaches the agent the way a customer would, through its endpoint, chat channel or phone number, and grades what it says and what it does. Grading on the state of your systems needs the agent's integrations pointed at the twins for the test instance, which is the same configuration vendors use for a customer sandbox. Without that, the bench still scores the conversation and the agent's stated outcomes, and the report says which kind of evidence each number rests on.

Do we need the vendor's prompts or model?

No. Candidates are black boxes. The same scenarios, budget caps and judges apply to each, and the report puts quality, cost and latency side by side without seeing inside any of them.

Can the vendor's own simulation results be imported?

Transcripts and incidents from any source can become scenarios on the bench; that is how production failures join the exam. Scores from one vendor's simulator are not comparable with another's, because the exam, the judge and the environment differ. Removing that difference is the point of the bench.

Does this work for voice agents built on these platforms?

Yes. Candidates answer a phone number, simulated callers bring accents, background noise and interruptions, and the call is graded on what changed in the billing or account systems. VAmoS Pro Bench is the public example: sixteen voice stacks on the same hundred calls.

The vendor's simulator vs an independent bench: Sierra, Google Agent Platform, Decagon, Salesforce

What the vendor simulators do well

Every serious agent platform now tests its own agents with simulated users, and the buyer gets it with the licence. That is a real advance over manual QA, and the four most often named in enterprise deals are worth describing accurately.

Sierra Simulations and Voice Sims. Simulated customers inside Agent Studio put a Sierra agent through difficult conversations, by chat and by voice, on shared evaluation infrastructure. The runs plug into a CI/CD pipeline and gate releases the way unit tests do.
Google Gemini Enterprise Agent Platform evaluations. Generally available since July 2026: a case generator, a user simulator that plays a persona through a multi-turn exchange, and an environment simulator that stands in for the systems the agent calls, with metrics such as tool-use quality and trajectory quality. Reached from the SDK, the command line, the console and ADK.
Decagon Simulations. Mock users across channels, built around the agent's operating procedures, with voice conditions such as accents, speaking speed, background noise and emotional tone. Historical transcripts become test cases, tests can be generated from procedures and production conversations, and the runs integrate with CI/CD.
Salesforce Agentforce Testing Center. Hundreds of synthetic utterances generated from an agent's topics and actions, with a chosen mix of positive and negative cases, run in batch inside a sandbox and scored on whether the agent picked the right topic and action.

Use them. If you are shipping on one of these platforms, its simulator is the fastest way to catch a regression in your own configuration. The gap is not quality. It is structural.

What they cannot do by construction

A vendor's simulator is built to test agents on that vendor's platform, scored by that vendor's evaluation, inside that vendor's account. Five things follow, and none of them is a flaw the vendor can fix.

It cannot grade two vendors on the same exam. Sierra's simulator does not run a Decagon agent, and neither runs the build your own team made. A selection decision needs every candidate on the same scenarios, under the same budget caps, judged the same way.
The pass criteria are the vendor's. Each simulator generates its tests from the agent's own configuration: operating procedures, topics and actions, conversation plans. Your policies, your regulations and your escalation rules have to be compiled into criteria your risk function signs before anything runs, and the vendor being graded cannot be the one holding the pen.
The systems behind the agent are the vendor's sandbox, or yours. Where a tool call lands during a simulation is rarely stated. An independent bench runs the agent against stateful twins of your systems, seeded fresh for every attempt, so writes persist, reads reflect them, nothing real is touched and no production data is needed.
The evidence stays in the platform. Transcripts, traces and scores live in the vendor's account. When the contract ends or a second vendor is added, the record of how the first one behaved on your work does not come with you.
It cannot run a champion and a challenger. Buyers want the incumbent agent and a challenger to face the same calls every quarter, with the option to switch, hold, or route part of the traffic. One buyer called this a champion-challenger setup. It needs a referee neither side owns.

Side by side

	Veris	Sierra Simulations	Google Agent Platform evaluations	Decagon Simulations	Agentforce Testing Center
Agents it can run	Any: your build, a vendor's agent through its endpoint, or a managed harness as a baseline	Agents built on Sierra	Agents on the Agent Platform and ADK	Agents built on Decagon	Agents built on Agentforce
Who writes the pass criteria	Generated from your policies, process maps, traces and incidents; you sign them before a run	The platform, from your configuration	Case generator plus your metrics	Generated from operating procedures and transcripts	Generated from topics and actions, or uploaded
Simulated users	Yes, with goals, knowledge they reveal only when asked, and voice	Yes, chat and voice	Yes, a persona and a conversation plan	Yes, with voice conditions	Single utterances, positive and negative
Systems behind the agent during a test	Stateful twins of your systems, seeded fresh per attempt	Not documented	Environment simulator	Not documented	Salesforce sandbox
Grading	The end state of the twins plus the transcript, three judge passes, disagreement discarded	The platform's evaluation	Tool-use and trajectory metrics	The platform's evaluation	Topic and action selection
Same exam across vendors	Yes	No	No	No	No
Evidence after you switch vendors	Yours: transcripts, traces, ledgers and verdicts, exportable	Stays in the platform	Stays in your Google Cloud project	Stays in the platform	Stays in your org
CI/CD	A small trial on each pull request, the full trial nightly	Yes	Yes, SDK and CLI	Yes	Batch runs
Where it runs	SaaS, or fully on-prem inside your VPC	Vendor-hosted	Your Google Cloud project	Vendor-hosted	Your Salesforce org

From each vendor's public documentation as of October 2026. "Not documented" means we could not find it stated publicly, not that it is absent. If something here is out of date, write to hello@veris.ai and we will correct it.

How a bake-off runs on Veris

Any vendor, or your own build, on the same exam. Each candidate connects through its own endpoint, a container you ship, or a managed harness used as a baseline, and runs the same scenarios under the same budget caps, so the comparison is between the candidates and not between the demos they chose to show you. Connectors exist for Genesys Chat bots and for Google CX Agent Studio apps, text and voice, so an agent built there joins a bench without changes to the agent.

Days 0 to 2: personas and scenarios. Composed from templates and your policy documents, process maps, traces and incidents. The world is seeded from sanitized exports, or from synthetic data when nothing can be shared. Scenarios that would sink a deployment are tagged as dealbreakers, and your team approves the exam before anything runs.
Days 3 to 5: connect candidates. Each agent gets its own parallel environment, and for voice, a phone number to answer. Vendors plug in through the interfaces they already expose. None can see the estate.
Weeks 1 to 2: run and grade. Three or more runs per candidate against the identical exam, each attempt in a fresh sealed copy of the world, graded on the state the systems are left in as well as the transcript, with error bars on the pass rate.
About week 2: the side-by-side report. Quality, latency and cost on one table, a failure taxonomy per candidate, and replayable evidence behind every number: the transcript, the agent's trace, the ledger of every request the twins received, and the verdict on each criterion.

The rubric is never published and scenarios regenerate on demand, so a candidate cannot train on the exam. The same exam re-runs when a vendor ships a new version or a provider changes a model, which is how a selection turns into a standing champion-challenger review. Deployment is SaaS, or fully on-prem inside your VPC; see security and deployment and how Veris is licensed.

What to ask a vendor about its simulator

Six questions separate a release check from selection evidence. They are fair to ask any platform, including Veris.

Can it run a competitor's agent, or our own build, on the same scenarios?
Who writes the pass criteria, and can our risk function sign them before the run?
During a simulation, which systems do the agent's tool calls reach, and is anything real touched?
Is the grade on the transcript, or on the records the agent left behind?
Can we export every transcript, trace and verdict, and keep them when the contract ends?
Can the same scenarios run next quarter against a different vendor?

When the vendor's simulator is enough

If you have one vendor, the agent only reads, and the question is whether yesterday's change broke anything, the platform's own simulator is the right tool and the cheapest one. An independent bench earns its place at the decision points: choosing between vendors, approving a go-live, a major release, a change of model or provider, and the recurring review that keeps the incumbent honest.