Agent routing benchmark

Multi-agent tasks: which coding agent

Four coding-agent CLIs ranked for five kinds of task, from public leaderboards and the outcomes of our own jobs. Refreshed weekly.

Generated 2026-10-15
Weekly refresh
5 task classes
6 kinds of coding and agentic work
4 agent CLIs

For agents: /bench.md (compact) · /bench.json (full) · /llms.txt

Which agent to use, by task class and kind of work · score 0–1 · Proven = at least 5 own jobs in the last 28 days
Task class	First choice	Score	Runner-up	Evidence
Codingrepo-level code changes	Claude CodeClaude Sonnet 5.5 high cost tier 3 of 4	0.86	Codex CLIGPT-6.1 Sol xhigh (config) cost tier 3 of 4 0.65	Proven · 46 jobs
Coding: Bug fixingfinding and fixing defects in existing code	Claude CodeClaude Sonnet 5.5 high cost tier 3 of 4	0.55	Antigravity CLIGPT-OSS 120B medium cost tier 1 of 4 0.57	Proven · 13 jobs
Coding: Features and refactorsnew features, tools and restructuring	Claude CodeClaude Sonnet 5.5 high cost tier 3 of 4	0.79	Antigravity CLIClaude Opus 5.5 high cost tier 4 of 4 0.67	Proven · 18 jobs
Coding: Frontend and UIweb pages, UI components and styling	Claude CodeClaude Opus 5.5 high cost tier 4 of 4	0.68	Antigravity CLIClaude Opus 5.5 high cost tier 4 of 4 0.66	Unproven
Coding: Tests and reviewcode review, audits and tests	Claude CodeClaude Sonnet 5.5 high cost tier 3 of 4	0.72	Codex CLIGPT-6.1 Sol xhigh (config) cost tier 3 of 4 0.52	Proven · 13 jobs
Agenticterminal and multi-step tool work	Claude CodeClaude Opus 5.5 high cost tier 4 of 4	0.72	Claude CodeClaude Sonnet 5.5 high cost tier 3 of 4 0.74	Unproven
Agentic: Terminal and sysadminshell, installs, configuration, server upkeep	Claude CodeClaude Sonnet 5.5 high cost tier 3 of 4	0.74	Claude CodeClaude Opus 5.5 high cost tier 4 of 4 0.70	Unproven
Agentic: Tools and connectorsmulti-step work through APIs and tool connectors	Codex CLIGPT-6.1 Sol xhigh (config) cost tier 3 of 4	0.71	Claude CodeClaude Opus 5.5 high cost tier 4 of 4 0.57	Unproven
Researchweb research and fact-finding	Claude CodeClaude Fable 5.1 medium cost tier 4 of 4	0.56	Claude CodeClaude Sonnet 5.5 high cost tier 3 of 4 0.51	Proven · 33 jobs
Readinglong documents, transcripts, summaries, extraction	Claude CodeClaude Fable 5.1 medium cost tier 4 of 4	0.90	Claude CodeClaude Sonnet 5.5 high cost tier 3 of 4 0.45	Proven · 12 jobs
Writingdrafting and editing prose	Claude CodeClaude Opus 5.5 high cost tier 4 of 4	0.72	Antigravity CLIClaude Opus 5.5 high cost tier 4 of 4 0.66	Unproven · 2 jobs

Proven at least 5 own jobs of this kind in the last 28 days
Unproven fewer than 5 own jobs: ranked on public scores
cost tier 1 of 4 to cost tier 4 of 4 list-price cost tier, cheapest to dearest

Full ranking by task class

Top 6 per class. Score is the priority (0–1); Proven = at least 5 own jobs in 28 days, and an unproven entry leads only on a clear public margin; quality comes from a weekly random sample of jobs graded against their own instructions. Rows tagged tie or evidence were reordered by the rules in Method.

Coding
repo-level code changesswe_bench_verified_resolvedswe_bench_multilingual_resolvedterminal_bench_4_accuracyaa_coding
#	Engine	Model	Evidence	Score	Public	Own success	Jobs	Median min	Speed	Cost
1	Claude Code	Claude Sonnet 5.5 high	Proven	0.86	0.94	0.91	46	4.0	1.00	cost tier 3 of 4 0.41
2	Codex CLI	GPT-6.1 Sol xhigh (config)	Proven tie	0.65	0.87	0.75	12	9.9	0.00	cost tier 3 of 4 0.45
3	Claude Code	Claude Opus 5.5 high	Unproven	0.68	1.00	0.50 default	2	9.0	0.33	cost tier 4 of 4 0.35
4	Claude Code	Claude Fable 5.1 medium	Unproven tie	0.62	0.87	0.50 default	4	4.8	0.67	cost tier 4 of 4 0.00
5	Antigravity CLI	Claude Sonnet 5.5 high	Unproven evidence	0.60	0.75 ×0.8	0.50 default	0	–	0.50	cost tier 3 of 4 0.39
6	Antigravity CLI	Claude Opus 5.5 high	Unproven evidence	0.59	0.80 ×0.8	0.50 default	0	–	0.50	cost tier 4 of 4 0.23
	4 more candidates ranked in /bench.json

Kinds of work (ranked in the summary above): Bug fixing swe_bench_verified_resolved · Features and refactors arena_text_coding · Frontend and UI arena_webdev_overall · Tests and review own jobs only

Agentic
terminal and multi-step tool workterminal_bench_4_accuracyaa_agentic
#	Engine	Model	Evidence	Score	Public	Own success	Jobs	Median min	Speed	Cost
1	Claude Code	Claude Opus 5.5 high	Unproven evidence	0.72	1.00	0.50 default	0	–	0.50	cost tier 4 of 4 0.44
2	Claude Code	Claude Sonnet 5.5 high	Unproven	0.74	0.94	0.50 default	0	–	0.50	cost tier 3 of 4 0.77
3	Codex CLI	GPT-6.1 Sol xhigh (config)	Unproven	0.69	0.87	0.50 default	0	–	0.50	cost tier 3 of 4 0.68
4	Claude Code	Claude Fable 5.1 medium	Unproven evidence	0.59	0.87	0.50 default	0	–	0.50	cost tier 4 of 4 0.00
5	Antigravity CLI	Claude Sonnet 5.5 high	Unproven tie	0.62	0.75 ×0.8	0.50 default	0	–	0.50	cost tier 3 of 4 0.53
6	Antigravity CLI	Claude Opus 5.5 high	Unproven	0.59	0.80 ×0.8	0.50 default	0	–	0.50	cost tier 4 of 4 0.20
	3 more candidates ranked in /bench.json

Kinds of work (ranked in the summary above): Terminal and sysadmin terminal_bench_4_accuracy arena_agent_bash_reliability_no_prereq_overall arena_agent_bash_recovery_steps_overall · Tools and connectors arena_agent_tool_hallucination_overall arena_agent_task_outcome_explicit_overall

Research
web research and fact-findingdrb_total_scorearena_search_overall
#	Engine	Model	Evidence	Score	Public	Own success	Jobs	Median min	Speed	Cost
1	Claude Code	Claude Fable 5.1 medium	Proven evidence	0.56	0.50 none	0.88	33	1.9	0.75	cost tier 4 of 4 0.01
2	Claude Code	Claude Sonnet 5.5 high	Proven evidence	0.51	0.50 none	0.83	24	5.0	0.50	cost tier 3 of 4 0.00
3	Codex CLI	GPT-6.1 Sol xhigh (config)	Proven evidence	0.38	0.50 none	0.60	35	12.6	0.00	cost tier 3 of 4 0.03
4	Grok CLI	Grok 4.5 medium (config)	Unproven evidence	0.45	0.50	0.50 default	0	–	0.50	cost tier 3 of 4 0.20
5	Claude Code	Claude Haiku 5.5 high	Unproven	0.65	0.50 none	0.50 default	3	0.2	1.00	cost tier 1 of 4 1.00
6	Claude Code	Claude Opus 5.5 high	Unproven	0.40	0.50 none	0.50 default	2	8.5	0.25	cost tier 4 of 4 0.07

Reading
long documents, transcripts, summaries, extractionarena_document_overall
#	Engine	Model	Evidence	Score	Public	Own success	Jobs	Median min	Speed	Cost
1	Claude Code	Claude Fable 5.1 medium	Proven	0.90	1.00	1.00	12	0.5	1.00	cost tier 4 of 4 0.31
2	Claude Code	Claude Sonnet 5.5 high	Proven evidence	0.45	0.50 none	0.50 default	5	1.3	0.50	cost tier 3 of 4 0.20
3	Codex CLI	GPT-6.1 Sol xhigh (config)	Proven evidence	0.38	0.50 none	0.60	25	8.7	0.00	cost tier 3 of 4 0.00
4	Grok CLI	Grok 4.6 medium (config)	Unproven evidence	0.22	0.01	0.50 default	0	–	0.50	cost tier 3 of 4 0.08
5	Grok CLI	Grok 4.5 medium (config)	Unproven evidence	0.21	0.00	0.50 default	0	–	0.50	cost tier 3 of 4 0.08
6	Claude Code	Claude Haiku 5.5 high	Unproven	0.57	0.50 none	0.50 default	1	–	0.50	cost tier 1 of 4 1.00

Writing
drafting and editing prosearena_text_creative_writingarena_text_overall
#	Engine	Model	Evidence	Score	Public	Own success	Jobs	Median min	Speed	Cost
1	Claude Code	Claude Opus 5.5 high	Unproven	0.72	1.00	0.50 default	2	2.4	0.80	cost tier 4 of 4 0.20
2	Antigravity CLI	Claude Opus 5.5 high	Unproven evidence	0.66	1.00	0.50 default	0	–	0.50	cost tier 4 of 4 0.09
3	Claude Code	Claude Fable 5.1 medium	Unproven tie	0.63	0.93	0.50 default	2	2.6	0.60	cost tier 4 of 4 0.00
4	Claude Code	Claude Sonnet 5.5 high	Proven evidence	0.56	0.72	0.50 default	7	3.3	0.40	cost tier 3 of 4 0.30
5	Antigravity CLI	Gemini 3.8 Flash high	Unproven evidence	0.58	0.87	0.50 default	2	6.9	0.00	cost tier 2 of 4 0.45
6	Antigravity CLI	Claude Sonnet 5.5 high	Unproven evidence	0.56	0.72	0.50 default	0	–	0.50	cost tier 3 of 4 0.24
	6 more candidates ranked in /bench.json

Method

priority = 0.45*public_norm + 0.25*own_success + 0.15*speed_rank + 0.15*cost_rank: weights public 0.45, own success 0.25, speed 0.15, cost 0.15; every input is 0–1.
Public: each class metric is min-max scaled across the engine/model pairs ranked here, then averaged; 0.5 when a model has none.
Harness discount: a score measured in another agent's harness counts at 0.8× (marked ×0.8), for example a Terminal-Bench run in Claude Code credited to Antigravity running the same model. The engine's own harness, or a model-only benchmark, counts in full.
Own success: quality of own jobs from a weekly random sample of jobs graded against their own instructions (pass 1, partial 0.5, fail 0) once a pair has enough graded jobs; until then the share of own jobs in the last 28 days that finished cleanly, or 0.5 with fewer than 10 jobs. Speed: rank by own median minutes, else Artificial Analysis output speed, else 0.5.
Cost: weight 0.15; the cheaper choice scores higher when quality is close. Cost per successful job comes from own jobs (3 or more in the class), else the model's list price times the engine's typical job size, else 0.5; it is ranked on a log scale, cheapest 1. The $ to $$$$ tier is the model's list price alone (3 input : 1 output tokens, four log-spaced bins).
Evidence first: Proven means at least 5 own jobs in the last 28 days. Scores within 0.03 tie (marked tie) and go to more own jobs, then lower cost, then speed. An unproven entry ranks above a proven one only on a clear public margin: more than 0.03 ahead and a higher public score (else marked evidence), never on cost or speed alone. Quality before cost: an unproven entry leads only with a public score within 0.05 of the best.
Effort: the label next to a model is the reasoning effort to run it at: the standing default (Claude Code Fable medium, others high; Codex and Grok their config; Antigravity the -high id), moved only when a public leaderboard scores another level at least 0.03 higher for that class.
Kinds of work: Coding and agentic subclasses are ranked the same way, on the public metrics that measure that kind of work and own jobs of that kind; a subclass with neither is not listed.
Only aggregate job counts, success rates and median minutes are published; no prompts, job details or grades. Model lists for Grok CLI are unverified.

Benchmarks and base data

The public part of each score (see Method) is the mean of these values, scaled 0–1 across the ranked models for each class; the scaled values are in /bench.json.

Sources · every input the score readsScroll sideways for every column →
Source	Measures	Feeds	Weight	Snapshot	Models
Terminal-Bench 4.0	terminal tasks an agent completes	coding, agentic; agentic / terminal	public 0.45 · ×0.8 in another agent's harness	2026-10-13	7 of 10
SWE-bench Verified	real repo issues resolved	coding; coding / bugfix	public 0.45 · ×0.8 in another agent's harness	2026-10-13	1 of 10
SWE-bench Multilingual	repo issues resolved in nine languages	coding; coding / multilang	public 0.45 · ×0.8 in another agent's harness	2026-10-13	No current scores for the models we rank
Artificial Analysis coding index	composite coding score	coding	public 0.45	–	Not fetched (needs an API key)
Arena Text: coding	people's votes on coding prompts	coding / features	public 0.45	2026-10-13	9 of 10
Arena WebDev	people's votes on built web apps	coding / frontend	public 0.45	2026-10-13	9 of 10
Artificial Analysis agentic index	composite agentic score	agentic	public 0.45	–	Not fetched (needs an API key)
Arena Agent: bash reliability	shell work in real agent sessions	agentic / terminal	public 0.45	2026-10-13	8 of 10
Arena Agent: bash recovery	recovering from a failed shell command	agentic / terminal	public 0.45	2026-10-13	8 of 10
Arena Agent: tool calls	not calling tools that do not exist	agentic / tools	public 0.45	2026-10-13	8 of 10
Arena Agent: task outcome	users marking the task done	agentic / tools	public 0.45	2026-10-13	8 of 10
DeepResearch Bench	long research reports	research	public 0.45	2026-10-13	No current scores for the models we rank
Arena Search	people's votes on search answers	research	public 0.45	2026-10-13	1 of 10
Arena Document	people's votes on answers about PDFs	reading	public 0.45	2026-10-13	3 of 10
Arena Text: creative writing	people's votes on creative writing	writing	public 0.45	2026-10-13	9 of 10
Arena Text	people's votes on all text prompts	writing	public 0.45	2026-10-13	9 of 10
Artificial Analysis output speed	output tokens per second	every class	speed 0.15	–	Not fetched (needs an API key)
OpenRouter prices	list price per token	every class	cost 0.15	2026-10-13	10 of 10
Own jobs	our own dispatched jobs over 28 days	every class	own success 0.25	rolling 28 days	–

Terminal-Bench 4.0

Each task gives an agent an instruction and its own environment; the agent works in the terminal, and a verifier's tests decide whether the task was done.
Scored as accuracy over trials, with at least 5 trials per task; errored trials count as failures, and an LLM judge reviews successful trials for reward hacking.
A submission is one agent harness, model and reasoning effort together, so a score says as much about the harness as the model; we count a score from another agent's harness at 0.8x.

SWE-bench Verified

A model, inside an agent scaffold, gets a GitHub issue and its repository and has to produce a patch that fixes it.
Scored as the percentage of tasks resolved, out of 500 SWE-bench tasks that human annotators checked for clear problem statements and correct tests.
The standard test for fixing bugs in existing code. Caveat: entries are a model plus a scaffold, and the board had no recent entries for most models we rank.

SWE-bench Multilingual

SWE-bench-style tasks in other languages: 300 tasks from 42 repositories in C, C++, Go, Java, JavaScript, TypeScript, PHP, Ruby and Rust.
A task counts as resolved when the patch passes two sets of unit tests taken from the original pull request.
Would tell us about repo work outside Python. Caveat: none of the models we rank has an entry yet, so it affects no ranking today.

Artificial Analysis coding index

A single coding score per model that Artificial Analysis publishes in its API, from its own runs of coding evaluations.
Its methodology page does not list which evaluations make up this index, so we do not describe its parts.
Not in use: the API needs a key we do not use, so no model has a value here.

Arena Text: coding

People ask a question in Arena's battle mode, see answers from two anonymous models side by side and vote for the better one; this board keeps only prompts Arena classifies as coding.
Ratings come from a Bradley-Terry fit to those votes, published as the Arena score.
Shows whose coding answers people prefer in chat. Caveat: preference in a chat answer, not whether the code runs in a repository.

Arena WebDev

Code Arena: two anonymous models build a web app or page from the same request and people vote on the result; Arena describes it as full-stack web development with multi-step reasoning, tool use and end-to-end app generation.
Rated from the votes and published as the Arena score; one model can appear at two reasoning efforts.
Our best signal for frontend and UI work. Caveat: it judges the finished app as people see it, not the code behind it.

Artificial Analysis agentic index

A single agentic score per model that Artificial Analysis publishes in its API.
Its methodology page does not list which evaluations make up this index, so we do not describe its parts.
Not in use: the API needs a key we do not use, so no model has a value here.

Arena Agent: bash reliability

A shell-command signal from Agent Arena, which analyses real sessions of people using Agent Mode on arena.ai for their own work.
Published as an IPS score with a 95% confidence interval; Arena's methodology post does not define this particular signal, so we do not describe how it is counted.
Feeds terminal and sysadmin work. Caveat: the sessions run in Arena's own agent, on whatever tasks its users bring.

Arena Agent: bash recovery

After a shell command errors in a real Agent Mode session, how many more turns the agent takes to reach a command that runs; giving up is penalised.
Published as an IPS score with a 95% confidence interval.
Feeds terminal and sysadmin work. Caveat: measured in Arena's own agent, on its users' tasks.

Arena Agent: tool calls

In real Agent Mode sessions, a task is marked a failure when the agent calls a tool that does not exist.
Published as an IPS score with a 95% confidence interval; the spread between models is small.
Feeds tool and connector workflows. Caveat: it catches invented tools, not wrong arguments to real ones.

Arena Agent: task outcome

At the end of a real Agent Mode session the user marks the task a success or a failure in Arena's interface.
Published as an IPS score with a 95% confidence interval.
Feeds tool and connector workflows. Caveat: each user judges their own task, in Arena's agent.

DeepResearch Bench

100 PhD-level research tasks (50 in Chinese, 50 in English) across 22 fields, written by domain experts; a system writes a full research report for each.
Reports are judged by RACE, which compares them with reference reports using an LLM judge, and FACT, which checks how many citations are effective and accurate; we read the leaderboard's total score.
Would tell us about deep research. Caveat: most entries are research products, not the models we rank, and none of ours has an entry.

Arena Search

Arena's leaderboard for searching the web: people compare two anonymous models' answers side by side and vote.
Rated with a Bradley-Terry fit to the votes, published as the Arena score.
Feeds research. Caveat: few of the models we rank have an entry.

Arena Document

People upload a PDF, ask about it, and compare two anonymous models' answers side by side.
Rated with a Bradley-Terry fit to the votes, published as the Arena score.
Feeds reading. Caveat: questions about one uploaded document, not reading across many files.

Arena Text: creative writing

Arena Text battles in the creative writing category: two anonymous answers, and people vote for the one they prefer.
Rated with a Bradley-Terry fit to the votes, published as the Arena score.
Feeds writing. Caveat: taste in short chat answers; it says nothing about long documents or a house style.

Arena Text

All Arena Text battles: people see answers from two anonymous models side by side and vote for the better one.
Rated with a Bradley-Terry fit to the votes, published as the Arena score; we use the raw ratings, not the style-controlled ones.
Feeds writing as a general quality signal. Caveat: a broad mix of everyday prompts, specific to no one task.

Artificial Analysis output speed

How fast a model streams its answer: tokens received per second after the first token, measured on the same endpoint Artificial Analysis uses for price.
Higher is faster; we would scale it across the models we rank.
Only a fallback when we have no timings of our own. Not in use: the API needs a key we do not use.

OpenRouter prices

The per-token prices OpenRouter lists for each model, taken from the top provider for that model, in US dollars.
We blend 3 input tokens to 1 output token and show dollars per million tokens; lower is cheaper.
Sets the $ to $$$$ tier, and the cost estimate when we have fewer than 3 priced jobs of our own. Caveat: one provider's list price, which can differ from the model maker's own price.

Own jobs

The jobs we hand to these agents in our own work, counted per engine and model for each task class and kind.
Own success is the share that finished cleanly or, once a pair has enough graded jobs, the mean grade from a weekly random sample graded against each job's own instructions; median minutes and list-price cost also count.
Only aggregate counts and rates are published, never the jobs themselves. Caveat: our own task mix, not a general benchmark.

Base data · raw published values; effort level in small type when a board names one; – = not scored darkerbrighter = better within the column; price not shadedScroll sideways for every source →

Base data: raw published values, shaded within each column
Model	Coding				Agentic				Research	Reading	Writing		Speed and cost
Model	Terminal-Bench 4.0 %	SWE-bench Verified %	Arena Text: coding score	Arena WebDev score	Arena Agent: bash reliability IPS	Arena Agent: bash recovery IPS	Arena Agent: tool calls IPS	Arena Agent: task outcome IPS	Arena Search score	Arena Document score	Arena Text: creative writing score	Arena Text score	OpenRouter prices $/M
Claude Fable 5.1	57.9% max	–	1528 max	1744 max	0.107 max	0.116 max	0.0056 max	0.16 max	–	1513 max	1507 max	1510 max	$20.00
Claude Opus 5.5	64.8% max	–	1547 high	1813 max	0.151 max	0.136 high	0.0051 max	0.205 max	–	–	1533 high	1515 high	$8.00
Claude Sonnet 5.5	61.8% max	–	1513 xhigh	1715 high / 1774 xhigh	0.13 max	0.15 max	0.0044 max	0.187 max	–	–	1465 xhigh	1471 xhigh	$4.00
Claude Haiku 5.5	–	–	–	1587 high	–	–	–	–	–	–	–	–	$0.20
GPT-6.1 Sol	58.2% max	–	1487 max	1755 max	0.026 max	0.045 max	0.0065 max	0.169 max	–	–	1432 max	1447 max	$4.00
Gemini 3.8 Flash	19.1% high	–	1510 high	1583 high	0.113 high	-0.007 high	0.005 high	0.06 high	–	–	1492 high	1499 high	$1.50
GPT-OSS 120B	–	26.0%	1379	–	–	–	–	–	–	–	1275	1365	$0.26
Grok 4.7	–	–	1427 xhigh	1639 xhigh	0.089 xhigh	0.072 xhigh	0.0055 xhigh	0.056 xhigh	–	–	1391 xhigh	1399 xhigh	$3.00
Grok 4.6	20.3% none	–	1465 high	1617 high	0.093 xhigh	0.057 xhigh	0.0051 xhigh	-0.003 xhigh	–	1452 high	1428 high	1428 high	$3.00
Grok 4.5	12.4% none	–	1474	1553	-0.012	0.051	0.0059	0.008	1213	1452	1442	1448	$3.00

Full ranking by task class

Coding

Agentic

Research

Reading

Writing

Method

Benchmarks and base data