last updated 2026-10-01

VAmoS Pro Bench

Voice Agent Simulation Bench for utility billing and payment assistance. Every call chains two to four caller requests, over street, café or television noise, to a calm or angry caller. Every world effect and spoken figure is graded against the trace.

Leaderboard

Voice agentTask completionLatencyTurnsBarge-in/callCostTypeModel
phonic57.5%1.42s191.87~$0.211speech‑to‑speechPhonic phonic-1
grok voice44.7%2.64s161.07~$0.210speech‑to‑speechGrok Voice Think Fast 2.0
dialt fast41.1%1.60s140.04~$0.182speech‑to‑speechDialt Fast
gemini 3.8 live40.1%1.66s120.13~$0.193speech‑to‑speechGemini 3.8 Live
gpt-live 139.2%1.62s171.05~$0.192speech‑to‑speechGPT-Live 1
openai realtime 2.137.9%2.12s140.44~$0.633speech‑to‑speechGPT Realtime 2.1
dialt smart37.8%1.96s170.13~$0.239speech‑to‑speechDialt Smart
elevenlabs36.4%2.18s120.04~$0.272cascadeElevenLabs ASR → gpt-4.1-mini → eleven_flash_v2
openai realtime33.6%2.32s140.35~$0.607speech‑to‑speechGPT Realtime 2
livekit29.4%2.96s120.04~$0.077cascadeDeepgram nova-3 → gpt-4.1-mini → ElevenLabs eleven_flash_v2
smallest ai26.8%3.58s100.36~$0.048cascadePulse → Electron → Lightning v3.1 Pro
gemini 3.1 live25.6%2.66s120.10~$0.160speech‑to‑speechGemini 3.1 Flash Live
polyai25.3%1.24s180.56~$0.119speech‑to‑speechPolyAI Dialog-RSN-1
mistral22.7%3.75s120.08~$0.279cascadeVoxtral realtime → mistral-medium-2604 → Voxtral TTS
gradium21.3%2.42s141.18~$0.120cascadeGradium STT → gpt-4.1-mini → Gradium TTS
vapi20.5%2.08s200.71~$0.271cascadeDeepgram → gpt-4.1-mini → ElevenLabs
deepgram19.8%2.04s130.34~$0.193cascadenova-3 → gpt-4.1-mini → aura-2
eleven realtime mini19.6%2.12s110.02~$0.262speech‑to‑speechEleven Realtime v1 Mini
hugging face19.0%5.28s129.86~$0.053cascadeWhisper large-v3 → gpt-oss-120b → Kokoro-82M
pipecat17.3%1.96s187.94~$0.109cascadeDeepgram nova-3 → gpt-4.1-mini → ElevenLabs eleven_flash_v2

Cost vs. completed tasks

10% 20% 30% 40% 50% 60% $0.05 $0.10 $0.20 $0.50 Mean cost per call (USD, log scale) · Y: task completion (% of calls) Pareto frontier phonic: 57.5% completion, ~$0.211 per call grok voice: 44.7% completion, ~$0.210 per call dialt fast: 41.1% completion, ~$0.182 per call 3.8 Livegemini 3.8 live: 40.1% completion, ~$0.193 per call GPT-Livegpt-live 1: 39.2% completion, ~$0.192 per call Realtime 2.1openai realtime 2.1: 37.9% completion, ~$0.633 per call dialt smart: 37.8% completion, ~$0.239 per call elevenlabs: 36.4% completion, ~$0.272 per call Realtime 2openai realtime: 33.6% completion, ~$0.607 per call livekit: 29.4% completion, ~$0.077 per call smallest ai: 26.8% completion, ~$0.048 per call 3.1 Livegemini 3.1 live: 25.6% completion, ~$0.160 per call polyai: 25.3% completion, ~$0.119 per call mistral: 22.7% completion, ~$0.279 per call gradium: 21.3% completion, ~$0.120 per call vapi: 20.5% completion, ~$0.271 per call deepgram: 19.8% completion, ~$0.193 per call eleven realtime mini: 19.6% completion, ~$0.262 per call hugging face: 19.0% completion, ~$0.053 per call pipecat: 17.3% completion, ~$0.109 per call phonic: 57.5% completion, ~$0.211 per callphonic grok voice: 44.7% completion, ~$0.210 per callgrok voice dialt fast: 41.1% completion, ~$0.182 per calldialt fast gemini 3.8 live: 40.1% completion, ~$0.193 per callgemini 3.8 live gpt-live 1: 39.2% completion, ~$0.192 per callgpt-live 1 openai realtime 2.1: 37.9% completion, ~$0.633 per callopenai realtime 2.1 dialt smart: 37.8% completion, ~$0.239 per calldialt smart elevenlabs: 36.4% completion, ~$0.272 per callelevenlabs openai realtime: 33.6% completion, ~$0.607 per callopenai realtime livekit: 29.4% completion, ~$0.077 per calllivekit smallest ai: 26.8% completion, ~$0.048 per callsmallest ai gemini 3.1 live: 25.6% completion, ~$0.160 per callgemini 3.1 live polyai: 25.3% completion, ~$0.119 per callpolyai mistral: 22.7% completion, ~$0.279 per callmistral gradium: 21.3% completion, ~$0.120 per callgradium vapi: 20.5% completion, ~$0.271 per callvapi deepgram: 19.8% completion, ~$0.193 per calldeepgram eleven realtime mini: 19.6% completion, ~$0.262 per calleleven realtime mini hugging face: 19.0% completion, ~$0.053 per callhugging face pipecat: 17.3% completion, ~$0.109 per callpipecat

Build this around your own systems

This board is one exam, on one utility’s policies. Veris builds the same thing around yours: your workflows, your systems, your rules, and the calls your agents actually get. Simulated callers, twinned services, and a verdict from the world the agent leaves behind rather than the transcript it produced.

Book a demo

Running a voice stack of your own? We’ll put it through the same 100 tasks and publish the result. Contact us at hello@veris.ai

Unlock the full VAmoS Pro Bench