A guardrail benchmark for voice agents. Rory, Acme Energy's billing agent, takes calls that carry common attacks alongside ordinary billing calls. The bench measures how well guardrails protect Rory from the attacks, and how much they hurt its ordinary calls.
Leaderboard
Guarded agenttype · guardrail
Δ Overallcombined
Δ Adversarialguard rate
Δ Ordinarysuccess rate
Detection F1precision · recall
Guardrail costavg per call
ElevenLabs Agents Guardrailsspeech‑to‑speechGuardrails on, Standard preset
+5.0%67.0% → 72.0%
+10.0%72.0% → 82.0%
±0.0%62.0% → 62.0%
0.44P 1.00 · R 0.28
included
GPT-6 Luna DecisionscascadeOur policy VG1–VG8
+5.0%61.3% → 66.3%
+7.3%75.3% → 82.7%
+2.7%47.3% → 50.0%
0.65P 1.00 · R 0.49
$0.0012
Clef OmnicascadeOur policy VG1–VG8
+3.0%61.3% → 64.3%
+3.3%75.3% → 78.7%
+2.7%47.3% → 50.0%
0.76P 0.77 · R 0.76
$0.0020
Jev 1.14cascadeOur policy VG1–VG8
+2.0%61.3% → 63.3%
+9.3%75.3% → 84.7%
−5.3%47.3% → 42.0%
0.71P 0.80 · R 0.64
$0.0008
Gemini Live Safety Settingsspeech‑to‑speechStandard safety settings
Each figure is the change against the same agent with no guardrails: the cascade with no guardrail added, or the same platform with its built-in guardrails off. The line under it reads baseline score → guarded score. Higher is better on every score; lower is better on cost.
Detection F1 against guardrail cost
Up is a better classifier of attack calls; right is a guardrail that costs more per call. The dashed line is the Pareto frontier: no guardrail is both cheaper and a better classifier than one on it. Hover a mark for its figures.
Change in adversarial guard rate against change in ordinary success rate
Each mark is a guarded agent against its own baseline. Up is more attacks stopped than the baseline; right of the zero line is more ordinary calls passed, left is ordinary calls lost. The green area is better than the baseline on both.