last updated 2026-11-10

VAmoS Guardrails Bench

A guardrail benchmark for voice agents. Rory, Acme Energy's billing agent, takes calls that carry common attacks alongside ordinary billing calls. The bench measures how well guardrails protect Rory from the attacks, and how much they hurt its ordinary calls.

Leaderboard

Guarded agenttype · guardrailΔ OverallcombinedΔ Adversarialguard rateΔ Ordinarysuccess rateDetection F1precision · recallGuardrail costavg per call
ElevenLabs Agents Guardrailsspeech‑to‑speechGuardrails on, Standard preset+5.0%67.0% → 72.0%+10.0%72.0% → 82.0%±0.0%62.0% → 62.0%0.44P 1.00 · R 0.28included
GPT-6 Luna DecisionscascadeOur policy VG1–VG8+5.0%61.3% → 66.3%+7.3%75.3% → 82.7%+2.7%47.3% → 50.0%0.65P 1.00 · R 0.49$0.0012
Clef OmnicascadeOur policy VG1–VG8+3.0%61.3% → 64.3%+3.3%75.3% → 78.7%+2.7%47.3% → 50.0%0.76P 0.77 · R 0.76$0.0020
Jev 1.14cascadeOur policy VG1–VG8+2.0%61.3% → 63.3%+9.3%75.3% → 84.7%−5.3%47.3% → 42.0%0.71P 0.80 · R 0.64$0.0008
Gemini Live Safety Settingsspeech‑to‑speechStandard safety settings+1.5%65.5% → 67.0%±0.0%75.0% → 75.0%+3.0%56.0% → 59.0%0.04P 1.00 · R 0.02included
Mercury DecidecascadeOur policy VG1–VG8+1.3%61.3% → 62.7%+11.3%75.3% → 86.7%−8.7%47.3% → 38.7%0.69P 0.63 · R 0.75$0.0004
Gemini Decision 1cascadeOur policy VG1–VG8+1.2%61.3% → 62.5%+10.7%75.3% → 86.0%−9.0%47.3% → 38.4%0.64P 0.63 · R 0.65$0.0012
Bedrock GuardrailscascadeVendor categories + prompt attack+0.7%61.3% → 62.0%±0.0%75.3% → 75.3%+1.3%47.3% → 48.7%0.42P 0.71 · R 0.29$0.0053
OpenAI moderationcascadeVendor categories−2.0%61.3% → 59.3%±0.0%75.3% → 75.3%−4.0%47.3% → 43.3%0.26P 0.75 · R 0.16free
kettei-fatcascadeOur policy VG1–VG8−4.7%61.3% → 56.7%+10.0%75.3% → 85.3%−19.3%47.3% → 28.0%0.62P 0.61 · R 0.63$0.0011
Llama Guard 4 12BcascadeMeta default categories−11.0%61.3% → 50.3%+4.0%75.3% → 79.3%−26.0%47.3% → 21.3%0.61P 0.50 · R 0.80$0.0012
Nemotron 3.5 Content SafetycascadeVendor default−15.0%61.3% → 46.3%+11.3%75.3% → 86.7%−41.3%47.3% → 6.0%0.64P 0.49 · R 0.93$0.0028

Each figure is the change against the same agent with no guardrails: the cascade with no guardrail added, or the same platform with its built-in guardrails off. The line under it reads baseline score → guarded score. Higher is better on every score; lower is better on cost.

Detection F1 against guardrail cost

0.00.10.20.30.40.50.60.70.8$0$0.001$0.002$0.003$0.004$0.005$0.006Guardrail cost per callDetection F1PARETO FRONTIERElevenLabs Agents Guardrails: F1 0.44, $0.0000 per call (included)ElevenLabs Agents GuardrailsGPT-6 Luna Decisions: F1 0.65, $0.0012 per call (billed)GPT-6 Luna DecisionsClef Omni: F1 0.76, $0.0020 per call (billed)Clef OmniJev 1.14: F1 0.71, $0.0008 per call (billed)Jev 1.14Gemini Live Safety Settings: F1 0.04, $0.0000 per call (included)Gemini Live Safety SettingsMercury Decide: F1 0.69, $0.0004 per call (estimate)Mercury DecideGemini Decision 1: F1 0.64, $0.0012 per call (billed)Gemini Decision 1Bedrock Guardrails: F1 0.42, $0.0053 per call (list price)Bedrock GuardrailsOpenAI moderation: F1 0.26, $0.0000 per call (free)OpenAI moderationkettei-fat: F1 0.62, $0.0011 per call (estimate)kettei-fatLlama Guard 4 12B: F1 0.61, $0.0012 per call (billed)Llama Guard 4 12BNemotron 3.5 Content Safety: F1 0.64, $0.0028 per call (billed)Nemotron 3.5 Content Safety

Up is a better classifier of attack calls; right is a guardrail that costs more per call. The dashed line is the Pareto frontier: no guardrail is both cheaper and a better classifier than one on it. Hover a mark for its figures.

Change in adversarial guard rate against change in ordinary success rate

better on both−5%0+5%+10%+15%−50%−40%−30%−20%−10%0+10%Change in ordinary success rateChange in adversarial guard rateElevenLabs Agents Guardrails: attacks stopped +10.0%, ordinary calls passed +0.0%, against its baselineElevenLabs Agents GuardrailsGPT-6 Luna Decisions: attacks stopped +7.3%, ordinary calls passed +2.7%, against its baselineGPT-6 Luna DecisionsClef Omni: attacks stopped +3.3%, ordinary calls passed +2.7%, against its baselineClef OmniJev 1.14: attacks stopped +9.3%, ordinary calls passed -5.3%, against its baselineJev 1.14Gemini Live Safety Settings: attacks stopped +0.0%, ordinary calls passed +3.0%, against its baselineGemini Live Safety SettingsMercury Decide: attacks stopped +11.3%, ordinary calls passed -8.7%, against its baselineMercury DecideGemini Decision 1: attacks stopped +10.7%, ordinary calls passed -9.0%, against its baselineGemini Decision 1Bedrock Guardrails: attacks stopped +0.0%, ordinary calls passed +1.3%, against its baselineBedrock GuardrailsOpenAI moderation: attacks stopped +0.0%, ordinary calls passed -4.0%, against its baselineOpenAI moderationkettei-fat: attacks stopped +10.0%, ordinary calls passed -19.3%, against its baselinekettei-fatLlama Guard 4 12B: attacks stopped +4.0%, ordinary calls passed -26.0%, against its baselineLlama Guard 4 12BNemotron 3.5 Content Safety: attacks stopped +11.3%, ordinary calls passed -41.3%, against its baselineNemotron 3.5 Content Safety

Each mark is a guarded agent against its own baseline. Up is more attacks stopped than the baseline; right of the zero line is more ordinary calls passed, left is ordinary calls lost. The green area is better than the baseline on both.

Unlock the full VAmoS Guardrails Bench