TreasuryQA Bench
An agent benchmark for office work on scanned records . The agent searches a company Dropbox of 1,435 scanned U.S. Treasury volumes through MCP, reads pages through Reducto’s OCR server, also over MCP, and answers with one exact figure: a value from a table, or a sum, percentage or statistic computed from several.
Leaderboard
Results
Failure Gallery
Methodology
Leaderboard
Cost vs. questions answered correctly
10% 20% 30% 40% 50% 60% 70% $0.25 $0.50 $1.00 $2.00 $4.00 agent-side cost per task (log scale) GPT-5.6 Terra: 16.7% pass@1 [10.0–24.3], $1.17 per task GPT-5.6 Terra Grok 4.7: 18.1% pass@1 [11.4–25.2], $3.55 per task Grok 4.7 GLM 5.3 Flash: 27.1% pass@1 [19.0–35.7], $0.24 per task GLM 5.3 Flash Kimi K3: 27.6% pass@1 [18.6–36.7], $2.55 per task Kimi K3 DeepSeek V4.1 Flash: 37.1% pass@1 [28.1–46.2], $0.25 per task DeepSeek V4.1 Flash Claude Sonnet 5.5: 41.9% pass@1 [31.9–51.9], $1.19 per task Claude Sonnet 5.5 Gemini 3.8 Flash: 51.0% pass@1 [40.5–61.4], $1.74 per task Gemini 3.8 Flash GPT-6.1 Sol: 51.4% pass@1 [41.0–61.4], $0.62 per task GPT-6.1 Sol Gemini 4 Argon: 60.0% pass@1 [50.0–70.0], $2.45 per task Gemini 4 Argon Claude Opus 5.5: 64.8% pass@1 [55.7–73.3], $1.56 per task Claude Opus 5.5 Claude Opus 5.5 Gemini 4 Argon GPT-6.1 Sol Gemini 3.8 Flash Claude Sonnet 5.5 DeepSeek V4.1 Flash Kimi K3 GLM 5.3 Flash Grok 4.7 GPT-5.6 Terra