Agent benchmarks · waves 1–21
How well does your
agent get things done?
New in Medium: wave 21 · 10 composite tasks → · Measurements & protocol
Four harnesses × four models · wave 19 results →
Brain + Claude Haiku 5.5: timeout experiment, final-attempt tokens & limitations →
Files, code, tools and memory. Real tasks, mechanically verified.
harness_bench_fast light
Tasks 1–391. Loading published measurements…
| # | Model / harness | Passed | Score | Steps | Tokens |
|---|
Scores compare complete agent configurations; tools and budgets vary. Brain + Claude Haiku is an experimental selected-retry result with a changed deadline and unresolved transport errors; see its protocol label. mean@N is the mean of N runs, not pass@N. Rescored historical runs use v0.18.0 task definitions. Protocols & original results →
About steps and tokens
Totals cover the tasks in each table, including failed tasks. Steps are tool calls where native traces are available; raw counters follow the harness's own accounting. Tokens are recorded input + output usage, including reported cache usage; reasoning is not added twice. These are token counts, not a monetary bill.
Partial means some usage is missing; Raw means the counters have not been audited and may include accounting errors. Task counts show where metrics were recorded, not whether every request was captured. A dash means no usable total for this subset. Hover or focus a value for its exact total and notes. Historical full-run totals are not reused for smaller subsets.
Wave 21 · Medium continuation
10 selected coupled workflows · v0.21.2. Existing runs rescored on this subset; budgets remain separate. Historical Medium measurements are preserved above and below.
This subset was selected adaptively from calibration results. Scores reuse existing runs; this is a post-selection rescore, not a fresh held-out evaluation. The 80-step historical diagnostics and 160-step runs remain separate. Codex’s 160 model-request budget differs from DeepAgents’ 160 graph-step budget. Exact protocols and per-task results →
Opus 5.5 solved the one selected task it was tested on (522); the other nine were not measured. This partial result is excluded from the complete 10-task ranking.
| # | Model / harness | Passed | Score | Steps | Tokens |
|---|
No wave 21 results match your search.
Wave 19 · new Medium tasks
Tasks 432–471 · 40 new tasks. These scores compare the same subset; they are ranked separately from the complete 80-task results above.
| # | Model / harness | Passed | Score | Steps | Tokens |
|---|
No new-task results match your search.
What is in each benchmark?
Light · tasks 1–391: files, code, data, memory, agent workflows, Git and terminal tools.
Medium · tasks 392–471: long context (20), tool reflection (20), and wave 19 feasibility (40). The published v0.18.0 runs cover only the first 40 Medium tasks. Medium shows complete 80-task results first, followed by the new 40-task wave 19 separately. Historical 40-task data stays in the archive. Missing old coverage is never scored as zero.
Medium · wave 21: tasks 512, 518, 520, 522–527 and 530. Each task combines two workflows with shared state and mechanical checks. Wave 20 and the original 20-task calibration results remain available in the measurement history; all existing Light and Medium measurements are preserved.
All tasks use mechanical verification. Partial Medium results are ranked only against the same 40-task subset, separately from the current 80-task suite. Historical scores are offline rescores; four DeepAgents configurations are full runs; the new harness matrix measures only wave 19. Historical + wave 19 rows combine disjoint measurements and are labeled. Repeat variability is unavailable for single runs.
Run the benchmark · Download aggregate data · Wave 19 protocol