Wave 21 · v0.21.2 · 10 selected tasks
Wave 21 · Medium continuation
Two coupled workflows per task, with changing service state, dependency constraints and mechanically checked final submissions. No specialist factual knowledge is required. The model must inspect tools, manage state and finish both workflows within the declared limits.
Selected task IDs: 512, 518, 520, 522, 523, 524, 525, 526, 527, 530. The original IDs are preserved. Run uv run python -m harness_bench list --suite composite-hard. The full 20-task calibration is available as composite-hard-calibration; the 531-task registry also retains the 40 original wave 20 controls.
Retrospective calibration rescore, not a fresh held-out evaluation. These ten tasks were selected using existing model outcomes, then the same audited executions were projected onto the selected IDs. There are no new model calls and no best-of mixing between budgets. One retained execution per task; no repeat-variability estimate.
Measurements: 21–22 October 2026; selected-wave projection: 23 October 2026. Exact dates and source fingerprints are retained per configuration.
| Model / harness | Budget | Passed | Score | Steps | Tokens |
|---|---|---|---|---|---|
| Sonnet 5.5 (high)DeepAgents 0.6.12 | 160 graph steps | 9/10 | 90.0% | 484 | 4,766,905 |
| DeepSeek V4.1 Flash 1017 UNCENSORED (high)Codex CLI 0.163.0 | 160 model requests | 9/10 | 90.0% | 679 | 19,225,827 |
| Sonnet 5.5 (high)DeepAgents 0.6.12 | 80 graph steps | 5/10 | 50.0% | ≥262 (8/10 tasks) | 3,507,874 |
| DeepSeek V4.1 Flash 1017 (high)DeepAgents 0.6.12 | 80 graph steps | 0/10 | 0.0% | Unavailable | 4,096,759 |
| Fable 5.1 (medium)DeepAgents 0.6.12 | 80 graph steps | 0/10 | 0.0% | ≥72 (8/10 tasks) | ≥1,186,934 (9/10 tasks) |
| GLM 5.3 (high)DeepAgents 0.6.12 | 80 graph steps | 0/10 | 0.0% | ≥19 (1/10 tasks) | 3,469,103 |
| MiMo 2.6 Pro (high)DeepAgents 0.6.12 | 80 graph steps | 0/10 | 0.0% | ≥67 (3/10 tasks) | ≥908,521 (4/10 tasks) |
| Qwen 3.8 Max (high)DeepAgents 0.6.12 | 80 graph steps | 0/10 | 0.0% | Unavailable | ≥4,712,361 (9/10 tasks) |
| Opus 5.5 (high)DeepAgents 0.6.12 | 160 graph steps | 1/1 | Incomplete · 1/10 measured | 44 | 428,830 |
Protocol and limits
Linux bubblewrap isolation; no subagents; provider-default temperature; 900 seconds, 320 trusted-service calls and 16,384 output tokens. DeepAgents 0.6.12 uses 80 or 160 graph supersteps. Configured Codex CLI 0.163.0 uses 160 model API requests, including transport retries: those units are different. The task prompt states its resource limits; runner prompt additions disclose finite graph limits. Historical 80-step results retain their original task prompts and source version.
Non-censorship API failures are retried and do not count as model failures. Censorship is a terminal failure. Historical Sonnet80 task527 retained a recovered HTTP502; it is not labelled a first-entire-clean execution. Formal run status governs scoring: Qwen task525 remains a failure despite domain goal acceptance after budget exhaustion. Configured Codex task512 remains a failure because the service-call contract was violated; it is not retried as an API failure.
Steps and tokens count only retained executions, passed and failed. Missing metrics remain unavailable; partial sums state coverage. Repeated input context is counted; cached input is already part of input tokens. Reasoning tokens are not added again to reported output totals. The configured uncensored DeepSeek on private servers is distinct from the official OpenRouter DeepSeek model. The default-catalog Codex diagnostic is excluded.
Sources are frozen calibration task sets v0.21.0 (80-step profiles) and v0.21.1 (160-step profiles); v0.21.2 selects IDs without changing the 160-step contracts. The frozen selected suite passed gold, blank-workspace rejection and frozen replay admission on all ten tasks. This release preserves current upstream trusted-service v2 runtime handling; historical results retain their original source/runtime versions and are not presented as new executions of that runtime. Calibration outcome selection and missing-counter coverage are preserved in the downloadable JSON.
Per-task outcomes
| Task | sonnet55_80 | deepseek41_flash_80 | fable51_medium_80 | glm53_high_80 | mimo26pro_high_80 | qwen38max_high_80 | sonnet55_high_160 | opus55_high_160_partial | codex0163_deepseek41_unc_high_160 |
|---|---|---|---|---|---|---|---|---|---|
| 512 · Reconcile incident ownership with a changing delivery policy | FAIL | FAIL | FAIL | FAIL | FAIL | FAIL | PASS | Unmeasured | FAIL |
| 518 · Publish mutually dependent restoration and backup snapshots | PASS | FAIL | CENSOR | FAIL | FAIL | FAIL | PASS | Unmeasured | PASS |
| 520 · Contact groups and client namespace handoff | PASS | FAIL | CENSOR | FAIL | FAIL | FAIL | PASS | Unmeasured | PASS |
| 522 · Latency reports and layered runtime feedback | PASS | FAIL | CENSOR | FAIL | FAIL | FAIL | FAIL | PASS | PASS |
| 523 · Template providers and package artifact feedback | PASS | FAIL | CENSOR | FAIL | FAIL | FAIL | PASS | Unmeasured | PASS |
| 524 · Versioned document recipes and source-pin merge | FAIL | FAIL | FAIL | FAIL | FAIL | FAIL | PASS | Unmeasured | PASS |
| 525 · Delivery compensation and dependency test coverage | FAIL | FAIL | CENSOR | FAIL | FAIL | FAIL | PASS | Unmeasured | PASS |
| 526 · Reconcile a migrated snapshot with a changing database view | FAIL | FAIL | FAIL | FAIL | FAIL | FAIL | PASS | Unmeasured | PASS |
| 527 · Close a live restore and copy-store cutover without stale replicas | PASS | FAIL | FAIL | FAIL | FAIL | FAIL | PASS | Unmeasured | PASS |
| 530 · Reconcile inherited tenant configuration with live profile deployment | FAIL | FAIL | CENSOR | FAIL | FAIL | FAIL | PASS | Unmeasured | PASS |
Historical wave 20 · preserved separately
The 40 original tasks 472–511 remain available, together with all previously published wave 1–19 results. They are not replaced by the selected wave 21 ranking. Different suites, source revisions, reasoning levels or budgets are not summed into an unsupported full-run score.
| Model / harness | Coverage | Recorded pass count | Status | Steps | Tokens |
|---|---|---|---|---|---|
| Claude Haiku 5.5 (high) · DeepAgents 0.6.12 | 40/40 | 29/40 | Historical raw score · 72.5% | ≥1,135 (35/40 tasks) | 9,606,273 |
| Claude Sonnet 5.5 (high) · DeepAgents 0.6.12 | 12/40 | 9/12 | Partial pilot · no full-wave score | ≥241 (11/12 tasks) | 1,843,090 |
Wave20 Haiku retains an ambiguous window-key contract on task473 as a raw failure; its 29/40 is not an unqualified clean difficulty estimate. The Sonnet pilot covers 12/40 only, with two censorship outcomes. Source v8a→v8b clarified task479 quota-capacity wording, so revisions stay explicit. Historical numbers are preserved without rerunning or silently correcting them.
Wave 21 outcomes, coverage and source fingerprints · Historical wave 20 outcomes and caveats · Task-set documentation · Existing Light and Medium results · Original measurement archive