An agent benchmark for research desk work. The agent is the assistant to a senior analyst at a simulated fiscal-data research firm. It gets one line, finds the details across ten business services, pulls an exact figure out of 1,435 U.S. Treasury statements, and delivers it inside real client and office work.
pass@1 is the share of one agent’s 480 attempts (160 tasks × three repeats) that passed every check. pass³ is the share of the 160 tasks it passed in all three. The board as a whole is 7,680 attempts: 16 agents, each on every task three times.
The 95% confidence interval is a task bootstrap of pass@1; rows a few points apart overlap. Hover a harness tag for the model id and the median minutes per attempt. How each agent did on each kind of check is on the Results tab. Data as of 2026-11-17.
Cost vs. tasks passed
Gemini 4 Carbon
Gemini 4 Hydrogen
Gemini 4 Argon
Gemini 3.8 Flash
Claude Opus 5.5
Claude Sonnet 5.5
GLM 5.3 Flash
Grok 4.7
Mistral Large 4
Inkling-2 Small
GPT-6.1 Sol
GPT-5.6 Terra
Muse Spark Open
DeepSeek V4.1 Flash
Kimi K3
Nemotron 3.5 Super VL
The line joins the agents nothing cheaper beats: GLM 5.3 Flash, DeepSeek V4.1 Flash, GPT-6.1 Sol, Claude Sonnet 5.5, Claude Opus 5.5, Gemini 4 Argon and Gemini 4 Carbon. Runs at max effort are on the Results tab. Cost is the agent’s own model bill at list price, per attempt.