#1
Base Model
claude-opus-4-7
79.4%
Pass^3
Thinking
Off
Pass@3
87.3%
Tasks
50/63
Date
2026-04-20
Total Cost
$60.39
Avg Cost
$0.320
Avg Tokens
157k
Leaderboard
All benchmark submissions are ranked by Pass^3, with Pass@3 shown alongside it for the same three-attempt runs.
Two headline scores are used in this benchmark to evaluate model performance.
Pass^3 measures consistency across three runs: a task only counts as a pass if the model passes all three attempts.
Pass@3 measures best-of-three success: a task counts as a pass if the model solves it at least once across three attempts.
Failed attempts are scored as 0 rather than treated as missing, including agent timeouts, non-zero exits, and other run-level exceptions captured in the trial metadata. Each task has a 10-minute timeout per attempt.
Thinking
Source
#2
Base Model
claude-opus-4-7
73.0%
Pass^3
Thinking
High
Pass@3
90.5%
Tasks
46/63
Date
2026-04-21
Total Cost
$74.34
Avg Cost
$0.393
Avg Tokens
170k
#3
Base Model
claude-sonnet-4-6
68.3%
Pass^3
Thinking
High
Pass@3
84.1%
Tasks
43/63
Date
2026-04-21
Total Cost
$36.82
Avg Cost
$0.195
Avg Tokens
119k
#4
Base Model
claude-opus-4-6
66.7%
Pass^3
Thinking
Off
Pass@3
90.5%
Tasks
42/63
Date
2026-04-21
Total Cost
$53.19
Avg Cost
$0.281
Avg Tokens
131k
#5
Base Model
gpt-5.4-2026-03-05
64.9%
Pass^3
Thinking
High
Pass@3
87.7%
Tasks
37/57
Date
2026-04-21
Total Cost
$24.59
Avg Cost
$0.146
Avg Tokens
119k
#6
Base Model
claude-opus-4-7
63.5%
Pass^3
Thinking
Low
Pass@3
85.7%
Tasks
40/63
Date
2026-04-21
Total Cost
$43.20
Avg Cost
$0.229
Avg Tokens
128k
#7
Base Model
gemini-3.1-pro-preview
63.5%
Pass^3
Thinking
High
Pass@3
82.5%
Tasks
40/63
Date
2026-04-21
Total Cost
$34.38
Avg Cost
$0.182
Avg Tokens
290k
#8
Base Model
gemini-3-flash-preview
61.9%
Pass^3
Thinking
High
Pass@3
84.1%
Tasks
39/63
Date
2026-04-20
Total Cost
$13.21
Avg Cost
$0.070
Avg Tokens
410k
#9
Base Model
gemini-3-flash-preview
58.7%
Pass^3
Thinking
Off
Pass@3
82.5%
Tasks
37/63
Date
2026-04-21
Total Cost
$14.57
Avg Cost
$0.077
Avg Tokens
418k
#10
Base Model
claude-sonnet-4-6
58.7%
Pass^3
Thinking
Off
Pass@3
79.4%
Tasks
37/63
Date
2026-04-21
Total Cost
$26.80
Avg Cost
$0.142
Avg Tokens
114k
#11
Base Model
claude-opus-4-6
58.7%
Pass^3
Thinking
High
Pass@3
76.2%
Tasks
37/63
Date
2026-04-21
Total Cost
$68.91
Avg Cost
$0.365
Avg Tokens
153k
#12
Base Model
gemini-3.1-pro-preview
57.1%
Pass^3
Thinking
Low
Pass@3
76.2%
Tasks
36/63
Date
2026-04-21
Total Cost
$20.50
Avg Cost
$0.108
Avg Tokens
154k
#13
Base Model
gpt-5.4-2026-03-05
55.6%
Pass^3
Thinking
Low
Pass@3
82.5%
Tasks
35/63
Date
2026-04-21
Total Cost
$12.49
Avg Cost
$0.066
Avg Tokens
75.1k
#14
Base Model
qwen/qwen3.6-plus Open
55.6%
Pass^3
Thinking
Off
Pass@3
79.4%
Tasks
35/63
Date
2026-04-28
Total Cost
$15.68 *
Avg Cost
$0.083 *
Avg Tokens
0
#15
Base Model
gemini-3.1-pro-preview
54.0%
Pass^3
Thinking
Off
Pass@3
76.2%
Tasks
34/63
Date
2026-04-20
Total Cost
$33.83
Avg Cost
$0.179
Avg Tokens
291k
#16
Base Model
qwen3.6-27b@bf16 Open
52.7%
Pass^3
Thinking
Off
Pass@3
85.5%
Tasks
29/55
Date
2026-05-08
Total Cost
$0.00 *
Avg Cost
$0.000 *
Avg Tokens
0
#17
Base Model
moonshotai/kimi-k2.5 Open
50.8%
Pass^3
Thinking
Off
Pass@3
81.0%
Tasks
32/63
Date
2026-04-28
Total Cost
$8.63 *
Avg Cost
$0.046 *
Avg Tokens
90.0k
#18
Base Model
claude-sonnet-4-6
50.8%
Pass^3
Thinking
Low
Pass@3
76.2%
Tasks
32/63
Date
2026-04-21
Total Cost
$25.92
Avg Cost
$0.137
Avg Tokens
108k
#19
Base Model
claude-opus-4-6
49.2%
Pass^3
Thinking
Low
Pass@3
69.8%
Tasks
31/63
Date
2026-04-21
Total Cost
$37.61
Avg Cost
$0.199
Avg Tokens
117k
#20
Base Model
gpt-5.4-nano
47.6%
Pass^3
Thinking
High
Pass@3
76.2%
Tasks
30/63
Date
2026-04-21
Total Cost
$5.31
Avg Cost
$0.028
Avg Tokens
283k
#21
Base Model
claude-haiku-4-5-20251001
47.6%
Pass^3
Thinking
Off
Pass@3
58.7%
Tasks
30/63
Date
2026-04-20
Total Cost
$21.56
Avg Cost
$0.114
Avg Tokens
199k
#22
Base Model
gemini-3-flash-preview
46.0%
Pass^3
Thinking
Low
Pass@3
84.1%
Tasks
29/63
Date
2026-04-21
Total Cost
$7.15
Avg Cost
$0.038
Avg Tokens
195k
#23
Base Model
gpt-5.1-codex-mini
46.0%
Pass^3
Thinking
High
Pass@3
74.6%
Tasks
29/63
Date
2026-04-20
Total Cost
$3.18
Avg Cost
$0.017
Avg Tokens
114k
#24
Base Model
gpt-5.1-codex-mini
46.0%
Pass^3
Thinking
Off
Pass@3
71.4%
Tasks
29/63
Date
2026-04-21
Total Cost
$2.72
Avg Cost
$0.014
Avg Tokens
113k
#25
Base Model
gpt-5.4-2026-03-05
44.4%
Pass^3
Thinking
Off
Pass@3
76.2%
Tasks
28/63
Date
2026-04-21
Total Cost
$11.89
Avg Cost
$0.063
Avg Tokens
79.5k
#26
Base Model
claude-haiku-4-5-20251001
39.7%
Pass^3
Thinking
High
Pass@3
66.7%
Tasks
25/63
Date
2026-04-21
Total Cost
$14.04
Avg Cost
$0.074
Avg Tokens
151k
#27
Base Model
gemma-4-31b-it@q8_k_xl Open
38.1%
Pass^3
Thinking
Off
Pass@3
71.4%
Tasks
24/63
Date
2026-05-09
Total Cost
$0.00 *
Avg Cost
$0.000 *
Avg Tokens
0
#28
Base Model
minimax-m2.7 Open
36.5%
Pass^3
Thinking
High
Pass@3
73.0%
Tasks
23/63
Date
2026-04-25
Total Cost
$0.00 *
Avg Cost
$0.000 *
Avg Tokens
0
#29
Base Model
gemini-3.1-flash-lite-preview
34.9%
Pass^3
Thinking
High
Pass@3
71.4%
Tasks
22/63
Date
2026-04-21
Total Cost
$5.25
Avg Cost
$0.028
Avg Tokens
239k
#30
Base Model
claude-haiku-4-5-20251001
34.9%
Pass^3
Thinking
Low
Pass@3
61.9%
Tasks
22/63
Date
2026-04-21
Total Cost
$14.01
Avg Cost
$0.074
Avg Tokens
146k
#31
Base Model
gemini-3.1-flash-lite-preview
33.3%
Pass^3
Thinking
Low
Pass@3
66.7%
Tasks
21/63
Date
2026-04-21
Total Cost
$4.79
Avg Cost
$0.025
Avg Tokens
224k
#32
Base Model
gemini-3.1-flash-lite-preview
33.3%
Pass^3
Thinking
Off
Pass@3
58.7%
Tasks
21/63
Date
2026-04-21
Total Cost
$3.61
Avg Cost
$0.019
Avg Tokens
199k
#33
Base Model
x-ai/grok-4.20
31.7%
Pass^3
Thinking
Off
Pass@3
61.9%
Tasks
20/63
Date
2026-04-28
Total Cost
$57.79
Avg Cost
$0.306
Avg Tokens
0
#34
Base Model
gpt-5.4-mini
28.6%
Pass^3
Thinking
High
Pass@3
76.2%
Tasks
18/63
Date
2026-04-21
Total Cost
$9.02
Avg Cost
$0.048
Avg Tokens
112k
#35
Base Model
qwen3.5-9b Open
25.4%
Pass^3
Thinking
Off
Pass@3
61.9%
Tasks
16/63
Date
2026-05-08
Total Cost
$0.00 *
Avg Cost
$0.000 *
Avg Tokens
0
#36
Base Model
gpt-5.4-mini
25.4%
Pass^3
Thinking
Off
Pass@3
44.4%
Tasks
16/63
Date
2026-04-21
Total Cost
$2.83
Avg Cost
$0.015
Avg Tokens
73.0k
#37
Base Model
gpt-5.4-mini
20.6%
Pass^3
Thinking
Low
Pass@3
61.9%
Tasks
13/63
Date
2026-04-21
Total Cost
$2.89
Avg Cost
$0.015
Avg Tokens
61.3k
#38
Base Model
gpt-5.4-nano
19.0%
Pass^3
Thinking
Low
Pass@3
50.8%
Tasks
12/63
Date
2026-04-21
Total Cost
$1.33
Avg Cost
$0.007
Avg Tokens
111k
#39
Base Model
gpt-5.4-nano
11.1%
Pass^3
Thinking
Off
Pass@3
22.2%
Tasks
7/63
Date
2026-04-21
Total Cost
$0.85
Avg Cost
$0.005
Avg Tokens
79.9k
| # | Agent | Model | Thinking | Tasks | Date | |||||
|---|---|---|---|---|---|---|---|---|---|---|
| 1 | Base Model | claude-opus-4-7 | Off | 79.4% | 87.3% | 50/63 | $60.39 | $0.320 | 157k | 2026-04-20 |
| 2 | Base Model | claude-opus-4-7 | High | 73.0% | 90.5% | 46/63 | $74.34 | $0.393 | 170k | 2026-04-21 |
| 3 | Base Model | claude-sonnet-4-6 | High | 68.3% | 84.1% | 43/63 | $36.82 | $0.195 | 119k | 2026-04-21 |
| 4 | Base Model | claude-opus-4-6 | Off | 66.7% | 90.5% | 42/63 | $53.19 | $0.281 | 131k | 2026-04-21 |
| 5 | Base Model | gpt-5.4-2026-03-05 | High | 64.9% | 87.7% | 37/57 | $24.59 | $0.146 | 119k | 2026-04-21 |
| 6 | Base Model | claude-opus-4-7 | Low | 63.5% | 85.7% | 40/63 | $43.20 | $0.229 | 128k | 2026-04-21 |
| 7 | Base Model | gemini-3.1-pro-preview | High | 63.5% | 82.5% | 40/63 | $34.38 | $0.182 | 290k | 2026-04-21 |
| 8 | Base Model | gemini-3-flash-preview | High | 61.9% | 84.1% | 39/63 | $13.21 | $0.070 | 410k | 2026-04-20 |
| 9 | Base Model | gemini-3-flash-preview | Off | 58.7% | 82.5% | 37/63 | $14.57 | $0.077 | 418k | 2026-04-21 |
| 10 | Base Model | claude-sonnet-4-6 | Off | 58.7% | 79.4% | 37/63 | $26.80 | $0.142 | 114k | 2026-04-21 |
| 11 | Base Model | claude-opus-4-6 | High | 58.7% | 76.2% | 37/63 | $68.91 | $0.365 | 153k | 2026-04-21 |
| 12 | Base Model | gemini-3.1-pro-preview | Low | 57.1% | 76.2% | 36/63 | $20.50 | $0.108 | 154k | 2026-04-21 |
| 13 | Base Model | gpt-5.4-2026-03-05 | Low | 55.6% | 82.5% | 35/63 | $12.49 | $0.066 | 75.1k | 2026-04-21 |
| 14 | Base Model | qwen/qwen3.6-plus Open | Off | 55.6% | 79.4% | 35/63 | $15.68 * | $0.083 * | 0 | 2026-04-28 |
| 15 | Base Model | gemini-3.1-pro-preview | Off | 54.0% | 76.2% | 34/63 | $33.83 | $0.179 | 291k | 2026-04-20 |
| 16 | Base Model | qwen3.6-27b@bf16 Open | Off | 52.7% | 85.5% | 29/55 | $0.00 * | $0.000 * | 0 | 2026-05-08 |
| 17 | Base Model | moonshotai/kimi-k2.5 Open | Off | 50.8% | 81.0% | 32/63 | $8.63 * | $0.046 * | 90.0k | 2026-04-28 |
| 18 | Base Model | claude-sonnet-4-6 | Low | 50.8% | 76.2% | 32/63 | $25.92 | $0.137 | 108k | 2026-04-21 |
| 19 | Base Model | claude-opus-4-6 | Low | 49.2% | 69.8% | 31/63 | $37.61 | $0.199 | 117k | 2026-04-21 |
| 20 | Base Model | gpt-5.4-nano | High | 47.6% | 76.2% | 30/63 | $5.31 | $0.028 | 283k | 2026-04-21 |
| 21 | Base Model | claude-haiku-4-5-20251001 | Off | 47.6% | 58.7% | 30/63 | $21.56 | $0.114 | 199k | 2026-04-20 |
| 22 | Base Model | gemini-3-flash-preview | Low | 46.0% | 84.1% | 29/63 | $7.15 | $0.038 | 195k | 2026-04-21 |
| 23 | Base Model | gpt-5.1-codex-mini | High | 46.0% | 74.6% | 29/63 | $3.18 | $0.017 | 114k | 2026-04-20 |
| 24 | Base Model | gpt-5.1-codex-mini | Off | 46.0% | 71.4% | 29/63 | $2.72 | $0.014 | 113k | 2026-04-21 |
| 25 | Base Model | gpt-5.4-2026-03-05 | Off | 44.4% | 76.2% | 28/63 | $11.89 | $0.063 | 79.5k | 2026-04-21 |
| 26 | Base Model | claude-haiku-4-5-20251001 | High | 39.7% | 66.7% | 25/63 | $14.04 | $0.074 | 151k | 2026-04-21 |
| 27 | Base Model | gemma-4-31b-it@q8_k_xl Open | Off | 38.1% | 71.4% | 24/63 | $0.00 * | $0.000 * | 0 | 2026-05-09 |
| 28 | Base Model | minimax-m2.7 Open | High | 36.5% | 73.0% | 23/63 | $0.00 * | $0.000 * | 0 | 2026-04-25 |
| 29 | Base Model | gemini-3.1-flash-lite-preview | High | 34.9% | 71.4% | 22/63 | $5.25 | $0.028 | 239k | 2026-04-21 |
| 30 | Base Model | claude-haiku-4-5-20251001 | Low | 34.9% | 61.9% | 22/63 | $14.01 | $0.074 | 146k | 2026-04-21 |
| 31 | Base Model | gemini-3.1-flash-lite-preview | Low | 33.3% | 66.7% | 21/63 | $4.79 | $0.025 | 224k | 2026-04-21 |
| 32 | Base Model | gemini-3.1-flash-lite-preview | Off | 33.3% | 58.7% | 21/63 | $3.61 | $0.019 | 199k | 2026-04-21 |
| 33 | Base Model | x-ai/grok-4.20 | Off | 31.7% | 61.9% | 20/63 | $57.79 | $0.306 | 0 | 2026-04-28 |
| 34 | Base Model | gpt-5.4-mini | High | 28.6% | 76.2% | 18/63 | $9.02 | $0.048 | 112k | 2026-04-21 |
| 35 | Base Model | qwen3.5-9b Open | Off | 25.4% | 61.9% | 16/63 | $0.00 * | $0.000 * | 0 | 2026-05-08 |
| 36 | Base Model | gpt-5.4-mini | Off | 25.4% | 44.4% | 16/63 | $2.83 | $0.015 | 73.0k | 2026-04-21 |
| 37 | Base Model | gpt-5.4-mini | Low | 20.6% | 61.9% | 13/63 | $2.89 | $0.015 | 61.3k | 2026-04-21 |
| 38 | Base Model | gpt-5.4-nano | Low | 19.0% | 50.8% | 12/63 | $1.33 | $0.007 | 111k | 2026-04-21 |
| 39 | Base Model | gpt-5.4-nano | Off | 11.1% | 22.2% | 7/63 | $0.85 | $0.005 | 79.9k | 2026-04-21 |
* Price can vary for open-weight models; self-hosted runs may report $0 when local serving cost is not captured.
Open-source models were run via OpenRouter, and some models were also run on self-hosted infrastructure.