Wave 19 · task-set v0.19.0 · measured 19–20 October 2026
Four harnesses.
Four models.
Only the 40 new tasks (432–471) were measured. Earlier tasks were retained where matching historical results exist.
Wave 19 · passed out of 40
GigaChat 3.5 uses PROM, build 32.9.16.9, default reasoning. Haiku 5.5 uses default reasoning; GLM-5.3 and DeepSeek V4.1 Flash use high + thinking. All model/harness entries below have complete 40-task coverage.
| Harness | GigaChat 3.5 | Haiku 5.5 | GLM-5.3 | DeepSeek |
|---|---|---|---|---|
| Codex CLI 0.160.1 | 0/40 (0.0%) | 31/40 (77.5%) | 14/40 (35.0%) | 31/40 (77.5%) |
| Claude Code 2.1.300 | 2/40 (5.0%) | 26/40 (65.0%) | 17/40 (42.5%) | 30/40 (75.0%) |
| pi-mono 0.73.1 | 0/40 (0.0%) | 29/40 (72.5%) | 24/40 (60.0%) | 26/40 (65.0%) |
| OpenClaw 2026.9.6 | 0/40 (0.0%) | 23/40 (57.5%) | 18/40 (45.0%) | 28/40 (70.0%) |
| DeepAgents · previous full run | 0/40 (0.0%) | 31/40 (77.5%) | 9/40 (22.5%) | 27/40 (67.5%) |
Light and Medium coverage
The Light leaderboard retains measured tasks 1–391. Full Medium uses tasks 392–471. Three new rows combine historical tasks 392–431 with the newly measured wave 19: pi-mono + GigaChat, pi-mono + GLM, and OpenClaw + GLM. They are labeled historical + wave 19, not fresh 80-task runs. Other new configurations appear only in the wave 19 view; missing old tasks are not failures.
Historical pi-mono + GigaChat and OpenClaw + GLM were checked against their complete raw 431-task records. The pi-mono + GLM baseline uses the already published aggregate (396/431); its raw trace/settings were not re-audited, so its composite is an arithmetic extension of that published configuration. Run dates and API transports differ across historical and new batches.
Per-configuration results
95% Wilson intervals describe task outcomes, not repeat variability. A single run per configuration does not establish repeat-based statistical significance.
| Configuration | Wave 19 | 95% interval | Full Medium |
|---|---|---|---|
| GigaChat 3.5 (PROM), build 32.9.16.9 (default) Codex CLI 0.160.1 | 0/40 (0.0%) | 0.0–8.8% | Unmeasured old half |
| Claude Haiku 5.5 (OpenRouter) (default) Codex CLI 0.160.1 | 31/40 (77.5%) | 62.5–87.7% | Unmeasured old half |
| GLM-5.3 (OpenRouter) (high) Codex CLI 0.160.1 | 14/40 (35.0%) | 22.1–50.5% | Unmeasured old half |
| DeepSeek V4.1 Flash (OpenRouter) (high) Codex CLI 0.160.1 | 31/40 (77.5%) | 62.5–87.7% | Unmeasured old half |
| GigaChat 3.5 (PROM), build 32.9.16.9 (default) Claude Code 2.1.300 | 2/40 (5.0%) | 1.4–16.5% | Unmeasured old half |
| Claude Haiku 5.5 (OpenRouter) (default) Claude Code 2.1.300 | 26/40 (65.0%) | 49.5–77.9% | Unmeasured old half |
| GLM-5.3 (OpenRouter) (high) Claude Code 2.1.300 | 17/40 (42.5%) | 28.5–57.8% | Unmeasured old half |
| DeepSeek V4.1 Flash (OpenRouter) (high) Claude Code 2.1.300 | 30/40 (75.0%) | 59.8–85.8% | Unmeasured old half |
| GigaChat 3.5 (PROM), build 32.9.16.9 (default) pi-mono 0.73.1 | 0/40 (0.0%) | 0.0–8.8% | 0/80 · historical + wave 19 |
| Claude Haiku 5.5 (OpenRouter) (default) pi-mono 0.73.1 | 29/40 (72.5%) | 57.2–83.9% | Unmeasured old half |
| GLM-5.3 (OpenRouter) (high) pi-mono 0.73.1 | 24/40 (60.0%) | 44.6–73.7% | 44/80 · historical + wave 19 |
| DeepSeek V4.1 Flash (OpenRouter) (high) pi-mono 0.73.1 | 26/40 (65.0%) | 49.5–77.9% | Unmeasured old half |
| GigaChat 3.5 (PROM), build 32.9.16.9 (default) OpenClaw 2026.9.6 | 0/40 (0.0%) | 0.0–8.8% | Unmeasured old half |
| Claude Haiku 5.5 (OpenRouter) (default) OpenClaw 2026.9.6 | 23/40 (57.5%) | 42.2–71.5% | Unmeasured old half |
| GLM-5.3 (OpenRouter) (high) OpenClaw 2026.9.6 | 18/40 (45.0%) | 30.7–60.2% | 41/80 · historical + wave 19 |
| DeepSeek V4.1 Flash (OpenRouter) (high) OpenClaw 2026.9.6 | 28/40 (70.0%) | 54.6–81.9% | Unmeasured old half |
Protocol and validation
Source 3796b9084093e5f9383077d7bde8f1ce1f0d57b5, Linux/bubblewrap, one attempt per task, hard 120-second task budget, max_tokens=16384. Eight total task slots; one active task per configuration in this matrix. Codex reuses 46 completed wave 19 task records from its interrupted full runs, without score-based selection; original runtime identity and task budgets were retained. No old tasks were rerun for this extension.
Native versions: Codex CLI 0.160.1, Claude Code 2.1.300, pi-mono 0.73.1, OpenClaw 2026.9.6 (eb377ac). Codex uses a Responses-to-Chat adapter; Claude Code uses a Messages-to-Chat adapter; pi-mono and OpenClaw use Chat Completions. Native tools and reasoning replay are retained; there is no SFT tool pruning or single-call restriction. Codex multi-agent/apps/hooks and web search were disabled. Claude uses isolated settings and no external MCP configuration; pi uses its historical offline/no-extension configuration. Client token-count estimates are approximate when requested. These are complete agent configurations, not model-only rankings.
Validation covered 40 unique current task IDs per configuration, frozen task/service protocol, unchanged retained records, native infrastructure counters and terminal API-error screening. Three Claude Code + Haiku attempts were infrastructure-invalid: an adapter placed user continuation before pending tool results, producing API 400. After fixing that order, only those three tasks were rerun; all three passed, changing 23/40 to 26/40. The other 37 outcomes were retained unchanged. Original attempts remain private. Final native infrastructure failures: zero. Model/task timeouts remain ordinary failures. This validation is not an exhaustive qualitative study of every failed solution.
Download aggregates and source fingerprints · Wave 19 task protocol · Earlier DeepAgents measurement