471 mechanically verified agent tasks

Secret-700-SFT + native deepagents 0.6.12: completed October 17, 2026; 377/431 (87.5%), 54 failures, no pending tasks or unresolved infrastructure failures. All 431 unique tasks validated. Ordinary 368/391 (94.1%), long context 0/20, tool reflection 9/20 (45.0%). Nineteen long tasks timed out after 7,200 s; the remaining task failed its verifier. k=1, Linux bwrap, concurrency 2; reasoning enabled, effort default, history forwarded; temperature 0.8, top_p 0.95, max output 81,920 tokens per response. Task timeout 900 s with long/reflection floors of 7,200/1,800 s. Interactive-client reasoning limits were not applied.

Secret-700-SFT telemetry: steps cover 402/431 tasks; tokens cover 409/431. Raw token counters contain duplicate usage and omit final telemetry from some interrupted tasks. Native-trace and API totals are not yet reconciled; these are not verified spend or cost. Score validation is independent of the telemetry audit. Completed public aggregate replaces the earlier 409-task snapshot. Raw traces and backend identifiers remain private.

MultiTool CLI 1.14.33: completed 2026-10-14, @mean1 on all 431 tasks, Linux bwrap, concurrency 4. GLM-5.3 (z-ai/glm-5.3, high): ordinary 386/391, long 10/20, reflection 15/20; Claude Haiku 4.5 (anthropic/claude-haiku-4.5, default): 371/391, 1/20, 4/20. Custom evaluation profile: max output 16,384 tokens per response (native CLI default 32,000), timeout 900 s with 7200 s long and 1800 s reflection floors. Steps count native tool attempts; tokens include input, output, reasoning and cache usage across all parent and child sessions, audited from native SQLite telemetry for all 431 tasks. Token totals are not monetary cost. Transport-affected attempts were replaced after fixing streaming; no transient infrastructure failures remain. Six GLM long-task failures exhausted the output budget and were retained. API responses confirmed the requested model IDs but provided no build fingerprint.

cowork + GLM-5.3 (OpenRouter, high): 376/431 (87.2%), k=1, completed in two disjoint batches on October 13–14: retained 367/411 plus 9/20 reflection tasks. Legacy task definitions and the CLI runner were checked unchanged. Mimir local CLI with reduced evaluation tools, Linux bwrap, temperature 0.2, max_tokens 16,384, 80 tool calls, timeout 900 s with task floors. Native-trace usage covers 429/431; steps and tokens are lower bounds when timed-out tasks have no final telemetry. The prior task-397 technical retry remains disclosed in the v0.17 notes.

Scores and ranks here use the full historical 431-task set. @mean3 remains a mean of three runs. Scores use the v0.18.0 definitions of this 431-task result snapshot; wave 19 is not included. Steps and tokens below remain full-run totals.

Completed, validated full runs measured 2026-10-02–14. The PROM matrix is still running; incomplete runs and infrastructure failures are excluded. The PROM matrix uses one run per configuration (k=1), eight task slots across the matrix, bwrap isolation. The separate maestro harness run used four task workers and bwrap.

All listed PROM matrix runs scored 0/20 on the long-context wave. Reflection: GigaChat 2 Reasoning + mini-SWE-agent: 8/20; GigaChat 2 Reasoning + OpenCode: 1/20; GigaChat 2 Reasoning + deepagents: 1/20; GigaChat 2 Pro + mini-SWE-agent: 1/20; GigaChat 2 Reasoning + deepagents (GigaChat profile): 2/20; maestro harness: 1/20 (and 0/20 long); other listed runs 0/20. Tools, profiles and budgets differ; these single runs do not isolate the effect of the harness.

GigaChat 2 and GigaChat 3 Lightning (PROM): build 32.4.16.3. GigaChat 2 Pro and GigaChat 3 Pro (PROM): build 32.4.30.3. GigaChat 2 Max, GigaChat 3 Ultra and GigaChat 3.5 (PROM): build 32.9.16.9. GigaChat 2 Reasoning (PROM): build 3.5.16.9, reasoning medium. For non-Reasoning models, reasoning: default for deepagents, pi-mono, mini-SWE-agent and OpenCode; not requested for OpenHands; medium (native default) for Hermes. Runtime versions and integration details are recorded in the README.

GigaChat 2 Max + Hermes completed with 0/431 and no infrastructure failures recorded by the runner; the cause of the all-fail result remains under investigation.

‡ Three deepagents runs without profile (GigaChat 2 Pro, GigaChat 2 Reasoning and GigaChat 3 Pro) had HTTP 422 CONTEXT_TOO_LONG incorrectly classified as infrastructure errors. These are normal model/harness failures. Corrected derived reports retain the full denominator and original scores; source reports remain unchanged.

Telemetry: 0 means absent from the artifact. * Deepagents tokens are raw, unaudited counters with known duplicate usage accounting, not verified spend or cost. CLI counters await native-trace audit except for recorded mini-SWE-agent and maestro harness metrics.

maestro harness: MAESTRO CARL with a custom shell coding adapter and reasoning-history replay, not an upstream ready-made coding agent. Measured 2026-10-05, k=1; no compaction, self-critic or subagents. Ordinary 247/391 (63.2%), long 0/20, reflection 1/20; no infrastructure failures. Budget: 200 model turns / 900 s with native task-specific floors, max_tokens 16384. Steps count completed model responses; token totals were checked against native traces for all 431 tasks. Elapsed time 63.8 minutes. This result describes this adapter configuration; raw traces remain private.

Earlier task-set versions are historical and cannot be compared directly with v0.18.0.

One complete k=1 run on the full v0.18.0 set, including the long-context and tool-reflection waves. dsh 0.2.0-rc.2 headless, profile hbf, DeepSeek V4.1 Flash (deepseek-v4.1-flash), reasoning high, concurrency 3, Linux bwrap isolation; measured 2026-10-03/04.

Server-outage attempts were replaced by recovery attempts. The subsequent repeat hit another outage and is excluded; this is not mean@3. Steps and tokens cover 430/431 tasks and are lower bounds because usage for the timed-out task is absent. See the README measurement notes.

Free-code 2.1.251-free-code.1 + DeepSeek V4.1 Flash Uncensored FP8 (high): 419/431 (97.2%), @mean3, measured 2026-10-04/05. Individual runs: 421, 416, 420; sample standard deviation 0.6 percentage points. Average ordinary score 99.6%, long 63.3%, reflection 85.0%; pass@3 425/431 (98.6%), pass^3 409/431 (94.9%). No server outages. Steps and tokens are rounded per-run means; telemetry coverage was 430/431, 428/431, 430/431 and tokens are not independently audited.

Pi-mono 0.73.1 + DeepSeek V4.1 Flash Uncensored FP8 (high): 423/431 (98.1%), @mean1, measured 2026-10-05, concurrency 3, bwrap; ordinary 391/391, long 13/20, reflection 19/20. No infrastructure failures recorded. Steps and tokens are absent from the artifact (shown as 0). Both configurations use timeout 900 s with task-specific floors of 7200 s for long tasks and 1800 s for reflection. The dsh result is @mean1.

Additional completed DeepSeek V4.1 Flash Uncensored FP8 (high) runs, measured 2026-10-05/06: mini-SWE-agent 2.4.6, deepagents 0.6.12, OpenCode 1.18.33 and Hermes (revision 5307e93252ac), each @mean1 on all 431 tasks, concurrency 3, Linux bwrap. Interrupted jobs resumed from completed-task checkpoints; no infrastructure failures remain in the final reports. Raw token counters are not independently audited; deepagents telemetry covers 429/431 tasks and has known duplicate usage accounting. Zero denotes absent telemetry.

GLM-5.3 Flash Uncensored (glm-5.3-flash-uncensored), reasoning high: eight completed @mean1 runs, measured 2026-10-06/07 on v0.18.0, Linux bwrap, concurrency 3, with the same frozen harness versions as the DeepSeek matrix. Infrastructure attempts were replaced by retries; final reports contain no infrastructure errors. Token telemetry is raw and not independently audited; zero means absent.

The DeepSeek V4.1 Flash and GLM-5.3 Flash measurements here use dealignai uncensored FP8 snapshots, as identified by the stand owner: dealignai/DeepSeek-V4.1-Flash-UNCENSORED-FP8 and dealignai/GLM-5.3-Flash-UNCENSORED-FP8. These scores describe the modified snapshots and are not measurements of the original base checkpoints.

GigaCode Universal Agent gc-cli 0.4.0 (commit 8c76e8b0c) + DeepSeek V4.1 Flash Uncensored FP8 (high): 419/431 (97.2%), @mean1, measured 2026-10-09, concurrency 3, Linux bwrap via run-cli; ordinary 389/391, long 12/20, reflection 18/20. No infrastructure failures, retries or timeouts. Main loop reasoning high, auxiliary finish-gate/summary calls without thinking; guards and built-in skills disabled. Steps count model responses; tokens are the agent's own usage counters (coverage 431/431), not independently audited. Timeout 900 s with floors of 7200 s for long and 1800 s for reflection.

GigaCode Universal Agent gc-cli 0.4.0 + GLM-5.3 (z-ai/glm-5.3, upstream weights through OpenRouter, reasoning high): 423/431 (98.1%), @mean1, measured 2026-10-09/10 with the same run-cli/bwrap setup; ordinary 388/391, long 16/20, reflection 19/20. No infrastructure failures; two long tasks timed out at 7200 s and count as fails. Steps and tokens are the agent's own counters with coverage 429/431 (lower bounds), not independently audited. This is not the glm-5.3-flash-uncensored FP8 snapshot of the eight-harness GLM matrix and is not comparable to it.

GLM-5.3 (z-ai/glm-5.3, upstream weights through OpenRouter, reasoning high) harness matrix, all six runs completed 2026-10-09/10, each @mean1 on all 431 tasks with the same frozen runtimes as the DeepSeek matrix, Linux bwrap, concurrency 3: OpenCode 413/431 (ordinary 386, long 12, reflection 15), deepagents 408/431 (381/14/13), mini-SWE-agent 407/431 (385/9/13), pi-mono 396/431 (376/6/14), OpenHands 404/431 (384/6/14), Hermes 402/431 (387/3/12). No infrastructure failures remain; timed-out tasks count as fails. Token telemetry is raw and not independently audited; zero means absent. Not comparable with the glm-5.3-flash-uncensored FP8 rows.

GigaCode Universal Agent gc-cli 0.4.0, further OpenRouter runs with the same run-cli/bwrap setup, @mean1: Kimi-K3 (moonshotai/kimi-k3, reasoning high), measured 2026-10-10/11: 417/431 (96.7%), ordinary 387/391, long 13/20, reflection 17/20; six transient gateway errors were replaced by retries, no infrastructure failures remain; five tasks timed out and count as fails; usage coverage 426/431. GPT-6 Luna (openai/gpt-6-luna), measured 2026-10-11: 388/431 (90.0%), ordinary 376/391, long 3/20, reflection 9/20. The gateway accepts tool calls for this model on Chat Completions only with reasoning_effort: none, so this run has no reasoning and is not comparable with the high-effort rows; three tasks timed out; usage coverage 428/431. Tokens are the agent's own counters, not independently audited.

Codex CLI 0.159.3 + GPT-6.1 Sol (gpt-6.1-sol), reasoning low: completed @mean1 on 2026-10-07, 420/431 (97.4%). Ordinary 388/391, long 19/20, reflection 13/20; no infrastructure failures in the final report. Linux bwrap, concurrency 3, ChatGPT OAuth, multi-agent disabled, no user configs. Task timeout 900 s with floors of 7200 s for long and 1800 s for reflection. Native sessions confirm model and low effort. Tokens are raw runner telemetry, not independently audited. Network transport used end-to-end TLS through a Mac CONNECT/SSH tunnel; runner and task execution stayed on bench.