AML Bench
An agent benchmark for anti-money-laundering investigation . The agent is the second-level investigator on a case handed over by a colleague: it reads the ledgers, re-screens the parties, applies the bank’s written procedure and files a disposition. The grade is the state it leaves the case system in, never what it says in chat.
Leaderboard
Leaderboard
Agent pass@1every criterion 95% CI pass³all 3 repeats Single-turn (105)everything is on file Multi-turn (48)must ask the colleague Askedon multi-turn tasks $/case Harness Model idno aliases
GPT-6.1 Sol 70.6% 63.6–77.3 66.7% 90.5% 27.1% 8.3% ~$0.26 Codex gpt-6.1-sol Sonnet 5.5 66.9% 59.7–74.1 64.7% 88.9% 18.8% 0.0% ~$0.21 Claude Code claude-sonnet-5-5 Opus 5.5 66.9% 59.5–74.1 64.7% 83.5% 30.6% 12.5% ~$0.44 Claude Code claude-opus-5-5 GLM 5.3 66.4% 59.5–73.2 58.2% 88.3% 18.8% 0.0% ~$0.14 Pi GLM-5.3 GLM 5.3 Flash 66.4% 59.7–73.2 56.9% 88.6% 18.1% 0.0% ≈$0.02 Pi GLM-5.3-Flash Kimi K3 65.6% 58.6–72.3 57.5% 87.3% 18.1% 0.0% ~$0.31 Pi kimi-k3 Grok 4.7 63.8% 57.1–70.6 53.6% 85.1% 17.4% 0.0% ~$0.46 Grok CLI grok-4.7 Gemini 4 Argon 63.4% 56.4–70.4 54.9% 79.4% 28.5% 7.6% ~$0.47 Gemini CLI gemini-4-argon DeepSeek V4.1 Flash 61.7% 54.5–68.6 52.9% 81.9% 17.4% 0.0% ~$0.06 Pi DeepSeek-V4.1-Flash GPT-5.6 Terra 61.0% 54.0–67.8 48.4% 80.6% 18.1% 0.0% ~$0.29 Codex gpt-5.6-terra Gemini 3.8 Flash 56.9% 50.1–63.6 42.5% 76.2% 14.6% 0.0% ~$0.42 Gemini CLI gemini-3.8-flash
Cost vs. cases passed
50% 55% 60% 65% 70% 75% $0.02 $0.05 $0.10 $0.20 $0.50 Mean agent-side cost per case (USD, log scale) · Y: pass@1 (% of attempts) Gemini 3.8 Flash: 56.9% pass@1 [50.1–63.6], $0.42 per case Gemini 3.8 Flash GPT-5.6 Terra: 61.0% pass@1 [54.0–67.8], $0.29 per case GPT-5.6 Terra DeepSeek V4.1 Flash: 61.7% pass@1 [54.5–68.6], $0.06 per case DeepSeek V4.1 Flash Gemini 4 Argon: 63.4% pass@1 [56.4–70.4], $0.47 per case Gemini 4 Argon Grok 4.7: 63.8% pass@1 [57.1–70.6], $0.46 per case Grok 4.7 Kimi K3: 65.6% pass@1 [58.6–72.3], $0.31 per case Kimi K3 GLM 5.3: 66.4% pass@1 [59.5–73.2], $0.14 per case GLM 5.3 GLM 5.3 Flash: 66.4% pass@1 [59.7–73.2], $0.02 per case (estimate) GLM 5.3 Flash Sonnet 5.5: 66.9% pass@1 [59.7–74.1], $0.21 per case Sonnet 5.5 Opus 5.5: 66.9% pass@1 [59.5–74.1], $0.44 per case Opus 5.5 GPT-6.1 Sol: 70.6% pass@1 [63.6–77.3], $0.26 per case GPT-6.1 Sol GPT-6.1 Sol Sonnet 5.5 Opus 5.5 GLM 5.3 GLM 5.3 Flash Kimi K3 Grok 4.7 Gemini 4 Argon DeepSeek V4.1 Flash GPT-5.6 Terra Gemini 3.8 Flash
