Agent benchmarks · Light & Medium
How well does your
agent get things done?
Files, code, tools and memory. Real tasks, mechanically verified.
harness_bench_fast light
Tasks 1–391. Loading published measurements…
| # | Model / harness | Passed | Score | Steps | Tokens |
|---|
Historical Medium · 80 tasks
Tasks 392–471 · long context, tool reflection and feasibility. These measurements cover the earlier 80-task suite and have their own ranking. They are not scores on Medium90.
| # | Model / harness | Passed | Score | Steps | Tokens |
|---|
No historical results match your search.
Medium90 partial result · Claude Code + Haiku 5.5 (OpenRouter) · 59/84 with valid verdicts
Scores compare complete agent configurations; tools and budgets vary. Brain + Claude Haiku is an experimental selected-retry result with a changed deadline and unresolved transport errors; see its protocol label. mean@N is the mean of N runs, not pass@N. Rescored historical runs use v0.18.0 task definitions. Protocols & original results →
About steps and tokens
Totals cover the tasks in each table, including failed tasks. Steps are tool calls where native traces are available; raw counters follow the harness's own accounting. Tokens are recorded input + output usage, including reported cache usage; reasoning is not added twice. These are token counts, not a monetary bill.
Partial means some usage is missing; Raw means the counters have not been audited and may include accounting errors. Task counts show where metrics were recorded, not whether every request was captured. A dash means no usable total for this subset. Hover or focus a value for its exact total and notes. Historical full-run totals are not reused for smaller subsets.
What is in each benchmark?
Light · 391 frozen tasks: files, code, data, memory, agent workflows, Git and terminal tools.
Medium · 90 tasks: long context (20), tool reflection (20), feasibility (40) and coupled workflows (10). The main table scores complete, valid 90-task measurements. The table below it preserves measurements on the earlier 80-task suite. Membership & protocols; historical and partial results remain available in the cumulative data and archive.
Run the benchmark · Light data · Medium data · Measurement archive