Agent Eval Cost Calculator: Estimate $ to Run SWE-bench, GAIA, TauBench

Order-of-magnitude estimates of what each benchmark costs to run end-to-end at frontier-model pricing.

Estimates assume a single pass-1 run with reasoning mode enabled, priced on Claude Sonnet 4.6 at $3 per million input and $15 per million output tokens (Anthropic list pricing). The dollar figures are computed directly from the token columns and that pricing, so they are only as good as the approximate token counts — treat them as order-of-magnitude, not as a quote. To price a different model, run the DIY formula below with its published rates.

For your own model and benchmark estimate, the formula is straightforward. Cost equals tasks times average input tokens times input price per million plus tasks times average output tokens times output price per million, all divided by one million. Pass^k or pass@k runs multiply by k. Multi-seed runs multiply by the number of seeds. Best-of-N runs multiply by N at the model side then run a verifier (usually cheap) at the end.

SWE-bench Verified, WebArena, and AppWorld dominate the cost budget for agent benchmarking because they involve large per-task context and long agent trajectories. MMLU-Pro is expensive in aggregate because it has 12,000 tasks, even though each task is cheap. GPQA-Diamond is small in task count but burns reasoning tokens on hard problems. The total cost ladder roughly tracks task count times average tokens per task, with reasoning-mode multipliers applying on reasoning-heavy benchmarks.