What each tool is for
The five tools most often named for agent testing solve two problems well: seeing what happened in a run, and scoring outputs against expectations.
- LangSmith, from LangChain: tracing and evaluation for LLM applications, closest to LangChain and LangGraph but not limited to them. Datasets, experiments, annotation queues, and multi-turn simulation through the open-source openevals package, where a language model plays the user. Self-hosting on the enterprise tier.
- Braintrust: evaluation first. Experiments against datasets, code and LLM scorers, side-by-side comparison of versions, and production logs turned into new test cases. A hybrid deployment keeps the data plane in your own cloud.
- Arize Phoenix and Arize AX: open-source tracing on OpenTelemetry, evaluation and annotation on top of the traces, and a hosted enterprise service. The natural choice when the rest of your observability already speaks OpenTelemetry.
- Langfuse: open-source tracing, prompt management, datasets and experiments, self-hosted or in their cloud. Its cookbook shows simulated multi-turn conversations built on openevals and scored afterwards by an LLM judge.
- promptfoo: declarative evaluations from a YAML file with assertions and a CI step, red-teaming for prompt injection, jailbreaks and data leakage, and a simulated-user provider for multi-turn tests. Open source.
All five are good at what they do, and Veris does not try to replace them. The gap is somewhere else.
What none of them provides
An agent that can act, on a CRM, a billing system or a ticket queue, is only as testable as the systems behind it. In every tool above those systems are the live vendor, a staging copy shared with everyone else, or a stub you wrote yourself. A simulated user, as in openevals or promptfoo, makes the conversation realistic. It does not make the world behind the agent real. Veris supplies that world:
- Stateful twins of the services the agent calls. A write persists, the next read reflects it, lifecycle rules are enforced the way the vendor enforces them, and nothing touches production.
- Simulated users with a goal, knowledge they reveal only when asked, and a way of speaking, who stay in character across every turn. On the phone, an accent, a background and a tendency to interrupt.
- A seeded world per attempt, so every candidate and every repeat starts from the same accounts, records and clock, and a result can be reproduced.
- Injected faults: timeouts, errors and stale records put into the twins to see how the agent copes and recovers.
- Grading on the state of the world, not the transcript. A refund that was promised and never written is a failure, however good the conversation sounded.
- Repeats with a confidence interval, so a change in pass rate can be told from noise.
- The same exam for every candidate, whether it is your build, a vendor's agent reached through its endpoint, or a managed harness used as a baseline.
Side by side
| Veris | LangSmith | Braintrust | Arize Phoenix | Langfuse | promptfoo | |
|---|---|---|---|---|---|---|
| Production tracing | Of every bench attempt, not of production | Yes | Yes | Yes | Yes | Limited |
| Datasets and scorers | Tasks with pass criteria, code checks and a judge | Yes | Yes | Yes | Yes | Yes |
| Red-teaming | Adversarial scenarios and guardrail tests on the same bench | Limited | Limited | No | No | Yes |
| Simulated users | Yes, with goals, knowledge and voice | Via openevals | Limited | No | Via cookbook | Simulated-user provider |
| Stateful service twins | Yes | No | No | No | No | No |
| Injected faults | Yes | No | No | No | No | No |
| Same exam across vendors | Yes, bought or built | Your versions | Your versions | Your versions | Your versions | Your versions |
| Runs in your VPC | SaaS, or fully on-prem | Enterprise tier | Hybrid data plane | Yes, open source | Yes, open source | Yes, open source |
From each vendor's public documentation as of October 2026. If something here is out of date, write to hello@veris.ai and we will correct it.
How they fit together
The usual shape is a loop. Tracing catches an incident in production. The incident becomes a scenario in Veris, then a family of scenarios around it. The bench runs against twins, nightly and on pull requests, and gates the release. Veris reads incidents from Langfuse, Arize, Datadog and Braintrust to seed scenarios, and every attempt keeps its transcript, its trace and its ledger of service requests for whichever tool you review them in.
- Trace a failure in production with the tool you already have.
- Reproduce it in Veris: the same seeded state, the same tool responses, a simulated user who does what the real one did.
- Expand it into variants: other personas, a timeout on the second call, a record that changed underneath the agent.
- Add the family to the bench and gate the next release on it.
When you do not need Veris
If the agent only reads, a retrieval or chat assistant with few side effects, then a dataset, a judge and a tracing tool cover most of the risk. The same holds early in prototyping, when the agent changes daily and the question is whether it works at all. Veris earns its place when the agent writes: refunds, replacements, account changes, filings, messages sent on a customer's behalf, and a wrong write costs more than the test.