harness_bench_fast.

Agent benchmarks · Light & Medium

How well does your
agent get things done?

Files, code, tools and memory. Real tasks, mechanically verified.

harness_bench_fast light

Tasks 1–391. Loading published measurements…

#Model / harnessPassedScoreStepsTokens

Medium90 partial result · Claude Code + Haiku 5.5 (OpenRouter) · 59/84 with valid verdicts

Scores compare complete agent configurations; tools and budgets vary. Brain + Claude Haiku is an experimental selected-retry result with a changed deadline and unresolved transport errors; see its protocol label. mean@N is the mean of N runs, not pass@N. Rescored historical runs use v0.18.0 task definitions. Protocols & original results →

About steps and tokens

Totals cover the tasks in each table, including failed tasks. Steps are tool calls where native traces are available; raw counters follow the harness's own accounting. Tokens are recorded input + output usage, including reported cache usage; reasoning is not added twice. These are token counts, not a monetary bill.

Partial means some usage is missing; Raw means the counters have not been audited and may include accounting errors. Task counts show where metrics were recorded, not whether every request was captured. A dash means no usable total for this subset. Hover or focus a value for its exact total and notes. Historical full-run totals are not reused for smaller subsets.

What is in each benchmark?

Light · 391 frozen tasks: files, code, data, memory, agent workflows, Git and terminal tools.

Medium · 90 tasks: long context (20), tool reflection (20), feasibility (40) and coupled workflows (10). The main table scores complete, valid 90-task measurements. The table below it preserves measurements on the earlier 80-task suite. Membership & protocols; historical and partial results remain available in the cumulative data and archive.

Run the benchmark · Light data · Medium data · Measurement archive