27 October 2026 · task-set v0.21.3
Codex + DeepSeek · Medium90
| Harness | Model | Passed | Score | Steps | Tokens |
|---|---|---|---|---|---|
| Codex CLI 0.164.0 | DeepSeek V4.1 Flash 1017 UNCENSORED (high) | 78/90 | 86.7% | 4,309 | 568,036,191 |
One fresh cohort covers all 90 canonical Medium tasks. No historical model or harness results were combined. The selected ten wave21 tasks began with four workers; after the operator expanded the run, their first clean outcomes were retained unchanged and the disjoint remaining 80 tasks ran under a global limit of ten concurrent tasks. Quality failures were not retried. Only superseded transport attempts were replaced.
Profile and limits
Configured codex-deepseek with its custom model catalog and base instructions; native Codex CLI 0.164.0, requested and returned model deepseek-v4.1-flash-1017-uncensored, reasoning high, reasoning summaries and web search disabled, no subagents. Sampling uses provider defaults without a temperature override. Context window 1,048,576; automatic compaction threshold 900,000; maximum output 16,384 tokens per response. Linux/bubblewrap provides a private home, workspace and scratch directory; source, gold solutions and raw captures are outside the sandbox.
The base budget is 160 model API requests including transport retries, raised by numeric task floors: 3,000 for long-context tasks and 400 for reflection. Execution guards are 7,200 seconds for long context, 3,600 seconds for ordinary/feasibility/reflection tasks, and 900 seconds plus 320 shared service requests for the selected coupled workflows. Model API requests, native tool steps and service calls are different units. Setup and verification are outside agent execution limits.
Verification and accounting
The source is frozen at task-set 0.21.3. Gold verification passed 90/90 on the byte-identical frozen source; the selected ten also passed a fresh gold check. Every retained result has an independent official verifier audit, with frozen host evidence replay for service tasks. Source/runtime fingerprints, emitted high/default-sampling/output settings, actual response model identities and raw request/response hashes were checked. A timely correct host-recorded terminal submission keeps its official credit if a final client reply times out; other timeouts remain failures.
Published counters include final retained attempts for both passed and failed tasks. They exclude superseded infrastructure attempts and separate smoke checks. Steps count completed native action events; a missing runner counter is recovered by replaying saved native JSONL when available. Steps are present for 90/90 tasks; API token counters are present for 90/90 tasks, with complete request-level usage for 89/90 and 1 requests without usage. Missing usage remains partial and is shown as a lower bound in the leaderboard. API input/output usage is authoritative; reported cached input and reasoning output are included once. Native/API counter disagreements, if present, are recorded without rerunning quality outcomes. These token totals are not a monetary bill.
One run, no independent repeats or repeat-variability interval. Different harnesses and budgets do not establish causal model or harness superiority. The older 9/10 calibration on Codex 0.163.0 remains a separate historical measurement.
Public outcomes, counters and source fingerprints · Medium membership and combination policy