Measured 21 October 2026 · historical v0.19.0 tasks
Brain + Claude Haiku 5.5.
Timeout experiment.
444/471 mechanically verified outcomes after selected retries (94.27%). This is a mixed-protocol experiment with unresolved transport errors, included in the general table with an experimental protocol label.
Final retained attempts
| Harness | Model | Passed | Score | Steps | Tokens |
|---|---|---|---|---|---|
| Brain | Claude Haiku 5.5 · default reasoning | 444/471 | 94.3% | 2,803 | 191,781,602 |
Token usage sums only the final retained attempt of each task, including failures. Superseded recovery attempts are excluded. Input: 184,907,548 tokens; output: 6,874,054 tokens. Counters exist for 471/471 retained tasks. Failed-request usage completeness has not been independently audited, so this is recorded API usage, not a verified bill. Steps use the adapter's recorded tool-call counter.
What changed
The original full run recorded 428/471 verifier passes (90.87%). Repeating 24 failed timeout tasks produced 11 passes, giving 439/471 retained passes (93.21%). Five tasks still hit the 120-second wall-clock limit. A separate retry changed their analysis deadline and prompt from 120 to 7200 seconds and removed the task-specific hard cap, with an outer 7200-second guard. All five passed: tasks 433, 453, 459, 460 and 462. Their elapsed times were 117.1, 224.1, 600.8, 111.1 and 123.0 seconds respectively.
The final retained total is 444/471 (94.27%), 3.40 percentage points above the original observed count. Selective repeats and changed deadlines prevent attributing that difference solely to the timeout setting. Formal resource budgets inside the tasks were unchanged. This is not a v0.19.1 measurement and does not revise historical leaderboard scores.
Settings and limits
Brain revision f65453dd, native Go/A2A adapter and bench preset, model anthropic/claude-haiku-5.5 through OpenRouter. Reasoning default, no temperature override; the upstream weight build was not disclosed. One attempt per selected task at each stage, up to eight concurrent workers, Linux/bubblewrap, Brain loop step limit 120. The first retry raised the A2A client timeout from 900 to 7500 seconds while retaining outer task limits.
Eleven retained task messages contain client timeout or connection-close errors: six of these tasks passed their file verifier and five failed. A passing file verifier does not establish a clean agent termination. These transport failures remain unaudited; native measurement_valid=true and zero classified infrastructure failures do not resolve them. No repeat-based uncertainty estimate or controlled causal comparison is available.
Download final-attempt counts and provenance · Back to results. Raw requests, traces and all-attempt accounting remain private.