27 October 2026 · task-set v0.21.3
MiniSWE + DeepSeek · Medium90
| Harness | Model | Passed | Score | Steps | Tokens |
|---|---|---|---|---|---|
| mini-SWE-agent 2.4.6 | DeepSeek V4.1 Flash 1017 UNCENSORED (high) | 70/90 | 77.8% | 4,364 | 1,262,564,743 |
One fresh full90 cohort, ten concurrent tasks, fixed predeclared order seed20261027. Every task retains its first clean outcome, including failures and timeouts. Quality outcomes were not rerun; only transient infrastructure failures permit replacement attempts, at most three physical attempts. No historical results were joined. Raw traces remain private.
Model and native harness
mini-SWE-agent2.4.6, native DefaultAgent, builtin mini.yaml and LocalEnvironment, LiteLLM Chat Completions transport. Model deepseek-v4.1-flash-1017-uncensored, revision c19ca5ca4eaeb79d29fdf08aea8327da95b1116a, reasoning high, context1,048,576, output16,384, no temperature override. Native tool prompts and error policy are preserved; native observations truncate at10,000 characters and local command timeout is30 seconds. There is no custom context compaction.
A provider compatibility subclass restores content and reasoning from a format-invalid assistant response before the native user correction, removing orphan tool-call IDs without tool results. The shared native runtime is unchanged. Normal and format-error history replay passed unit checks; backend tokenization of a captured next request counted1,468 tokens with reasoning versus1,395 without. Every subsequent benchmark request is audited to preserve all prior reasoning. This compatibility behavior is part of the measured configuration.
Budgets and isolation
Base160 model API requests including transport retries, raised by numeric task floors: long3,000 and reflection400. Native Mini step limits count model queries; reported steps count native bash actions. Long tasks allow7,200 seconds, ordinary/reflection/feasibility3,600 seconds, selected coupled workflows900 seconds and320 shared service requests. Setup and verification are outside execution guards. Bubblewrap gives each task a private home/workspace/scratch and hides source, gold and raw captures.
Verification and counters
Frozen task-set0.21.3, identical to the preceding Codex90 source; byte-identical gold admission90/90. Each task has an independent official verifier audit with frozen trusted-service evidence where required. Source/runtime/API/evidence hashes, model identity, high effort, absence of temperature override and reasoning replay are checked. Timely correct host-recorded terminal submissions retain official credit if the client later times out; other deadlines remain failures.
Retained passed and failed attempts contribute counters; superseded infrastructure attempts and preflight smokes are excluded. Native bash action coverage90/90. Captured API input+output counters are present for90/90 tasks; complete request usage for79/90; 11 requests lack usage. Missing usage remains partial and is shown as a lower bound. Reported cached input and reasoning output are counted once. Token counts are not a monetary bill.
One cohort, no independent repeats or repeat variability interval. The preceding Codex configuration used the same source, model checkpoint and declared effort/budgets, but different native prompts, observation limits, transport and context management. The comparison does not isolate a causal harness effect.
Public outcomes and counters · Medium membership and combination policy