Veris Twin Fidelity Benchmark

The obvious objection to a vendor-run fidelity study is that the vendor wrote the tests. This study removes it by borrowing its instrument from an open-source project that has nothing to do with us: NangoHQ/integration-templates, a public repository of production integration code written for Nango’s own customers to run against live vendor APIs. It runs here unmodified through Nango’s own command-line tool: 436 templates across nine services, exercising 194 distinct operations. Every test, every assertion, and every pass-or-fail decision in this study belongs to them. We supplied only the environment the tests point at.

A test written by looking at the twin can only prove that the twin behaves the way its author expected. It cannot detect the cases where the author’s understanding of the real system was wrong, and those are exactly the cases that matter. So the real system defines what counts as a correct answer, and the twin never does. Every test is first proven against the live vendor API on a dedicated account; only then is it pointed at the twin, with only the base URL changed.

Agreement is scored strictest first. The exact-status lens is the raw measurement: both worlds must return the same verdict down to the HTTP status, and the number is published before anything is set aside. The behaviour lens then removes exactly seven units that differ only in what the two worlds hold — seed data, not behaviour, each one named. The population never changes: all 436 units, nothing excluded.

Twins of large APIs are scoped to their working core, and every operation outside a twin’s published contract refuses cleanly by policy, so scope is a countable property of each twin, not an estimate.

The twin never invents a success. In every open divergence it refuses: on three units with a different status than production’s own refusal, on two by declaring an operation out of scope. A permissive twin inflates every score run against it; a strict one only produces conservative results. That is what makes twin-based testing usable in an SDLC: a green integration run is never the twin flattering the agent’s code, and a red one means the agent fixes something real or hits a documented divergence.

Rev 1.0 carried one divergence that pointed the wrong way — cross-run memory of spent ids, flagged then as the structural exception. It is closed: the twin now takes seedable registries of ids its world does not own. What remains on the permissive side is bookkeeping, not behaviour: two Attio slugs are held by archived leftovers of the recorded account, so the recording refuses a create that a clean production workspace — and the twin — would accept. Both units are counted in the raw 424, set aside by the behaviour lens, and said plainly here rather than rounded away.

The study produces a named divergence list rather than a score, and the list is now five units long — each with the mechanism behind it, read off the wire logs. Three need work: two wait on a mechanism production has and the twin lacks, one on a state the twin’s world cannot yet express. Two are published boundaries the twin declares up front rather than defects. Rev 1.0’s ledger shrank the right way: the 26-unit Asana status rule and the structural exception closed outright, and the small status-code families are down to the five units below.

The seven set-aside units, named too: four are the instrument’s own parser refusing response shapes its vendor documents, reachable only because the twin’s world has rows where the recorded account was empty; three are leftovers in the recorded account itself — a calendar that still resolves ids it was told to clear, and the two archived Attio slugs of FIG 3. Data, not behaviour: counted in the raw 424, set aside in the 431.

A repeat over 238 units across two freshly built deployments returned every verdict identical, down to the HTTP status, with zero flips. Over the same repeats, four verdicts on the real arm moved, on vendor-side memory of spent ids. For agents, a red test against the twin is not flake: no cycles burned retrying nondeterministic failures, and two agents testing the same change get the same answer. The oracle is held to the same standard: the recorded real-arm corpus is verified byte-stable before every scoring, and has not moved since it was taped.

A typical request is the same order of magnitude in both worlds; the twin runs about 3.6x faster at the median and its p90 stays near 130 ms on every service. The tail is where the two worlds separate: real vendors regularly take half a second and occasionally tens of seconds (Attio’s worst request sat through 32 s of rate-limit backoff), while the twin has no throttling and no multi-second stalls on someone else’s infrastructure. For agent loops this is the difference between running an integration suite once and running it hundreds of times per session. The cost is that same missing tail: timeout, retry, and backoff behavior under real-world latency is not exercised by default. Latency and fault injection exist in the product; twin behavior under injected conditions is not covered by this study.

• The instrument probes a focused slice intensely rather than sweeping the twin’s whole declared surface; what it measures, it measures independently, but it does not measure everything.
• Twin behavior under injected latency and faults is not covered by this study.
• This revision freshly measured the five services that carried open divergences, 214 units; the four services already exact-perfect were re-scored against their recorded results, not re-run, so a change in them under the newest build would not have been seen.
• All measurements ran through an off-the-shelf integration gateway; behavior behind a gateway with mutual TLS or DLP inspection is not evidenced here.
• Results are stamped per build; numbers from unstamped or mixed builds are not comparable.

Every number on this page is reproducible from retained artifacts: a frozen claim manifest, a per-operation ledger, captured request and response corpora from both arms, per-operation diffs, per-test logs, latency tapes, the analyzers, and an identity trail binding each result to the software build that produced it. Each figure carries the command that produced it alongside the raw output. This is the artifact set to demand from any benchmarking vendor, including us.

Pinned build

The environment is rebuilt from a pinned software build immediately before each measurement, and every result is stamped with that build. Comparisons are only ever made between stamped worlds.

Fixed oracle

The instrument and its real-arm recording are pinned; between revisions only the twin build moves, so every delta on this page belongs to the twin and nothing else.

Sealed credentials

Credentials on the twin arm are minted by the environment’s own identity twin, so no real traffic can leak into a simulated measurement.

Fresh world

Each measurement starts from a clean world with no memory, which is why determinism holds.

The benchmark re-runs on the same protocol as twins and instrument grow; new versions append here, and the tables above always show the current revision’s results. Questions, or a system you want to see measured: hello@veris.ai