← Medium leaderboard

27 October 2026 · task-set v0.21.3

Claude Code + Sonnet 5.5 · Medium90

HarnessModelPassedScoreStepsTokens
Claude Code 2.1.300Claude Sonnet 5.5 (high)62/9068.9%1,16175,672,259

One fresh cohort covers every canonical Medium task. Sixteen concurrent tasks; fixed predeclared order seed20261027. First clean outcomes, including failures, are retained. No historical results were joined or quality failures retried. Six tasks required one replacement after upstream HTTP502 errors; a transport error invalidated an attempt even when its verifier passed, so earlier passes were not selected over worse clean replacements.

Model and native harness

Native Claude Code2.1.300, requested and returned OpenRouter model anthropic/claude-sonnet-5.5; provider checkpoint revision is not exposed. Native effort high and adaptive thinking; translated requests carry high reasoning with reasoning excluded=false and summarized thinking requested. No temperature override. Native catalog reports a1,000,000-token context window; every captured generation request is limited to16,384 output tokens, regardless of the catalog's larger default output limit.

The native Messages client uses an Anthropic-to-Chat-Completions compatibility bridge. Tool IDs, assistant thinking text and available provider reasoning metadata are preserved; stream framing and transport signatures are synthesized for the client. Native context management is retained; count_tokens uses a characters/4 approximation rather than an exact provider tokenizer. These transport choices are part of the measured configuration. Native tools are used, with web and delegation disabled, empty external settings/MCP, no session persistence or automatic memory. Bubblewrap isolates the home, workspace and scratch, hiding benchmark source, gold and raw captures. The shared installed runtime is unchanged.

Limits and verification

Base160 model generation requests including transport retries, raised by task floors to3,000 for long context and400 for reflection. Native max-turns is set to the same numeric limit; turns, API requests, tool actions and service calls remain different units. Long tasks allow7,200 seconds; ordinary/reflection/feasibility3,600 seconds; selected coupled workflows900 seconds and320 shared service requests. Setup and verification are outside execution limits. Infrastructure replacements are bounded at three physical attempts per task.

Frozen task-set0.21.3 with the same source digest as the preceding full90 runs. Byte-identical gold admission90/90. Every retained answer has an independent official verifier audit; service tasks replay frozen host evidence. Source/runtime/API/evidence hashes, actual model identities, high/default-sampling/output settings and reasoning replay were checked. The final audit verified537 subsequent requests containing previous reasoning. Adaptive thinking need not emit a visible thought on every easy response. A timely correct host-recorded terminal submission retains official credit if the final client response times out; other timeouts remain failures.

Accounting and interpretation

Steps count native tool actions, present for90/90 tasks. Tokens are authoritative captured API input+output from retained passed and failed attempts: complete90/90, no requests without usage. Cached input and reasoning output are included once. Superseded infrastructure attempts and preflight checks are excluded. Native thinking/cache breakdown and native list-price estimates can differ from provider usage; these counters are not a monetary bill.

Some discarded attempts also contained model-generated malformed tool argument JSON. These are format/bridge issues, distinct from upstream network errors; no whole-task replacement was selected solely for those format errors. All six replaced attempts independently contained upstream502 failures.

One cohort, no repeat variability interval or causal model-versus-harness attribution. Different native prompts, transport, context management and models limit comparisons with other rows. Raw traces remain private; only sanitized outcomes and counters are published.

Public outcomes and counters · Medium membership and combination policy