Latest benchmark: task-set v0.13.0

351 mechanically verified coding-agent tasks

A compact benchmark for comparing agent harnesses on file edits, data transforms, pytest bug fixes, memory discipline, synthetic benchmark-like workflows, version-control work, skill discovery, and adversarial / hostile-environment handling.

View results Run the bench
351 tasks
29 runs listed
26 models
100% top score
Claude Code CLI / Opus 4.8 100%
deepagents / grok-4.5 98.6%
Claude Code CLI / Sonnet 4.6 97.2%
1-253 File, code, data, diagnostic, and memory tasks
254-313 Synthetic agentic (Terminal-Bench / tau / SWE-bench-like) and Git / version-control workflows
314-351 Skill-discovery tasks and adversarial / hostile-environment handling (broken versions, unreadable files, contradictions, a ~100 MB log)

Current results only

Leaderboard

Results below use the full 351-task set (v0.13.0). One run per harness + model setup is listed. Filter by model group or a single harness; steps and tokens are shown only when the harness exposes them.

29 of 29 shown

1

Claude Code CLI

Claude Opus 4.8

351/351
100.0%
Steps — · Tokens —
2

deepagents

grok-4.5

346/351
98.6%
Steps 3,164 · Tokens 30,776,418
3

Claude Code CLI

Claude Sonnet 4.6

341/351
97.2%
Steps — · Tokens —
4

Claude Code CLI

Claude Haiku 4.5

340/351
96.9%
Steps — · Tokens —
4

deepagents

GLM-5.2

340/351
96.9%
Steps 3,966 · Tokens 41,664,423
6

deepagents

DeepSeek V4 Pro

339/351
96.6%
Steps 4,014 · Tokens 44,552,076
7

deepagents

GLM-5.1

335/351
95.4%
Steps 3,802 · Tokens 39,320,469
8

deepagents + OpenAI profile

GPT-5.6 Luna

330/351
94.0%
Steps 4,392 · Tokens 40,181,133
9

deepagents + Anthropic profile

Claude Haiku 4.5

328/351
93.4%
Steps 3,682 · Tokens 50,549,085
10

deepagents

Qwen 3.7 Max

326/351
92.9%
Steps 4,154 · Tokens 48,563,241
10

deepagents

DeepSeek V3.2

326/351
92.9%
Steps 6,413 · Tokens 98,708,199
12

deepagents

Qwen 3.6 Flash

325/351
92.6%
Steps 4,334 · Tokens 49,387,938
13

deepagents + GigaChat profile

GigaChat 3.5

312/351
88.9%
Steps 2,887 · Tokens 4,715,017 · mean of 3 runs
14

deepagents

DeepSeek V4 Flash

310/351
88.3%
Steps 4,082 · Tokens 46,732,794
15

deepagents + GigaChat profile

GigaChat 3 Ultra

303/351
86.3%
Steps 2,776 · Tokens 3,424,473
16

deepagents

GPT-4.1

300/351
85.5%
Steps 3,382 · Tokens 34,199,277
17

opencode (gpt2giga)

GigaChat 3.5

298/351
84.9%
Steps — · Tokens —
18

deepagents + GigaChat profile

GigaChat 2 Max

292/351
83.2%
Steps 2,714 · Tokens 3,209,751
19

deepagents

Qwen 3.5 Flash

288/351
82.1%
Steps 3,507 · Tokens 44,553,189
20

deepagents, no profile

GigaChat 3.5

287/351
81.8%
Steps 3,374 · Tokens 8,486,826
21

deepagents

MiniMax M2.7

282/351
80.3%
Steps 3,387 · Tokens 38,231,538
22

deepagents

Qwen3-Coder-30B-A3B

258/351
73.5%
Steps 3,849 · Tokens 57,997,536
23

deepagents + GigaChat profile

GigaChat 3 Pro

241/351
68.7%
Steps 2,823 · Tokens 2,400,507
24

deepagents

yandex/gpt5.1-pro

220/351
62.7%
Steps 4,030 · Tokens 42,623,784
25

deepagents

GPT-OSS-120B

185/351
52.7%
Steps 2,054 · Tokens 22,079,085
26

deepagents

yandex/gpt5-pro

180/351
51.3%
Steps 2,407 · Tokens 20,845,269
27

deepagents + GigaChat profile

GigaChat 3 Lightning

179/351
51.0%
Steps 2,232 · Tokens 1,739,949
28

deepagents

Llama 4 Maverick

57/351
16.2%
Steps 1,391 · Tokens 17,065,878
29

deepagents

yandex/gpt5-lite

37/351
10.5%
Steps 1,666 · Tokens 118,843,500

Verification

No LLM judge

Each task is checked mechanically: exact files, JSON shape and values, pytest outcomes, SQLite/XLSX checks, regex matches, and deterministic command outputs.