TreasuryQA Bench
An agent benchmark for working an archive of scanned financial records . The agent gets one question and a cloud drive holding 1,435 scanned U.S. Treasury volumes with no text layer. It has to find the right volume, read the right page through a hosted OCR server, and reply with one exact figure. A script, not a model, decides whether that figure is right.
Leaderboard
Leaderboard
Agent pass@1exact figure 95% CI pass³all 3 repeats Clear26 tasks Very blurry20 tasks Answeredcommitted a figure Right thenof those $/task Harness · model idno aliases
Claude Opus 5.5 64.8% 55.7–73.3 45.7% 62% 60% 93.8% 69.0% ~$1.56 Claude Code claude-opus-5-5 Gemini 4 Argon 60.0% 50.0–70.0 45.7% 55% 50% 72.4% 82.9% ~$2.45 Gemini CLI gemini-4-argon GPT-6.1 Sol 51.4% 41.0–61.4 38.6% 73% 25% 90.5% 56.8% ~$0.62 Codex gpt-6.1-sol Gemini 3.8 Flash 51.0% 40.5–61.4 37.1% 55% 37% 84.3% 60.5% ~$1.74 Gemini CLI gemini-3.8-flash Claude Sonnet 5.5 41.9% 31.9–51.9 27.1% 54% 12% 89.0% 47.1% ~$1.19 Claude Code claude-sonnet-5-5 DeepSeek V4.1 Flash 37.1% 28.1–46.2 17.1% 40% 20% 84.3% 44.1% ~$0.25 Pi DeepSeek-V4.1-Flash Grok 4.7 18.1% 11.4–25.2 4.3% 21% 3% 80.5% 22.5% ~$3.55 Grok CLI grok-4.7 GPT-5.6 Terra 16.7% 10.0–24.3 8.6% 24% 3% 93.3% 17.9% ~$1.17 Codex gpt-5.6-terra
Cost vs. questions answered correctly
10% 20% 30% 40% 50% 60% 70% $0.25 $0.50 $1.00 $2.00 $4.00 agent-side cost per task (log scale) GPT-5.6 Terra: 16.7% pass@1 [10.0–24.3], $1.17 per task GPT-5.6 Terra Grok 4.7: 18.1% pass@1 [11.4–25.2], $3.55 per task Grok 4.7 DeepSeek V4.1 Flash: 37.1% pass@1 [28.1–46.2], $0.25 per task DeepSeek V4.1 Flash Claude Sonnet 5.5: 41.9% pass@1 [31.9–51.9], $1.19 per task Claude Sonnet 5.5 Gemini 3.8 Flash: 51.0% pass@1 [40.5–61.4], $1.74 per task Gemini 3.8 Flash GPT-6.1 Sol: 51.4% pass@1 [41.0–61.4], $0.62 per task GPT-6.1 Sol Gemini 4 Argon: 60.0% pass@1 [50.0–70.0], $2.45 per task Gemini 4 Argon Claude Opus 5.5: 64.8% pass@1 [55.7–73.3], $1.56 per task Claude Opus 5.5 Claude Opus 5.5 Gemini 4 Argon GPT-6.1 Sol Gemini 3.8 Flash Claude Sonnet 5.5 DeepSeek V4.1 Flash Grok 4.7 GPT-5.6 Terra
