Task-set v0.21.3 · 23 October 2026
Medium · 90 tasks
The current Medium suite contains historical tasks 392–471 and selected wave 21 tasks 512, 518, 520, 522, 523, 524, 525, 526, 527 and 530. Light (Lite) is frozen at tasks 1–391. Task IDs and task contracts are unchanged. The full registry retains 531 tasks, including archived controls and calibration tasks.
Combining existing measurements
The cumulative table reuses public measurements on disjoint subsets within the same published measured configuration. It preserves original source versions, reasoning settings, harness profiles and budgets. Historical 80-task results already include wave 19; those results are counted once. Model display names alone do not establish compatibility.
Historical DeepAgents GLM and DeepSeek profiles enabled native subagents; wave 21 profiles disabled them. These remain separate. Sonnet with pi-mono and Sonnet with DeepAgents remain separate, as do 80/160 graph-step limits and Codex model-request limits. The configured uncensored DeepSeek endpoint is separate from the official model. See the linked original protocols for each measurement.
Only a valid configuration covering all 90 tasks receives an overall result, percentage and rank. Missing tasks are unmeasured, never failures. Partial results show their actual denominator and coverage. There are currently 125 configurations: 19 cover 80 tasks, 89 cover 40, eight cover ten, one covers one task and eight cover all 90. Three complete small-model DeepAgents profiles combine earlier v0.19.1 measurements with a fresh ten-task v0.21.3 run without subagents. Two fresh Haiku 5.5 runs on 26 October measured all 90 tasks with mini-SWE-agent and DeepAgents; their profiles stay separate from historical rows. A fresh Codex + DeepSeek full90 cohort on 27 October is separate from all historical rows. A fresh MiniSWE + DeepSeek full90 cohort adds an independently measured harness configuration. A fresh Claude Code + Sonnet 5.5 full90 cohort adds a separately measured configuration. The main leaderboard ranks complete valid 90-task configurations; historical and partial measurements remain in the cumulative data and archive.
Provenance and limitations
The wave 21 component of the three small-model rows is a fresh run made on 24 October 2026 with the release profile; apart from the two fresh all-90 Haiku runs, other wave 21 components were selected from prior calibration outcomes and reused, so those stay retrospective and are not a fresh held-out evaluation. No repeat-variability interval is inferred from this projection.
Steps and tokens count final retained task attempts only. Native step units differ between harnesses. Missing counters, raw counters and incomplete request accounting remain explicit; token lower bounds are marked. Coverage of task counters does not prove complete request-level accounting.
Cumulative data and source fingerprints · Unchanged historical and Light data · Historical protocols and results · Wave 21 protocols
Run uv run python -m harness_bench list --suite medium for the canonical 90 tasks. medium-historical80 preserves the earlier scope. The forty wave 20 controls and ten unselected composite calibration tasks are excluded from Medium 90.