Leaderboard

Ranked by Pass^3. Every task is run three times.

o11y-bench 1.1o11y-bench 1.0Scores aren't comparable across versions: 1.1 uses an outcome-based LLM judge and adds a task.
Pass^3
Task passes only if all three attempts pass.
Pass@3
Task passes if any of the three attempts passes.

Timeouts (10 min per attempt), crashes and run errors count as failed attempts.

Effort

Pass^3 vs. cost

Each point is a model at one effort level. The dotted line is the Pareto frontier: no other run is both cheaper and higher scoring. Up and left is better.

Scatter plot of o11y-bench model configurations. The vertical axis shows Pass^3 and the horizontal axis shows average cost per trial in USD on a log scale, with a dotted Pareto frontier line through the best configurations.

AnthropicOpenAIGooglexAIOtherPareto frontierEstimated cost

Rankings

#1

Anthropic logoclaude-opus-5-5

93.8%

Pass^3

Effort
medium
Pass@3
98.4%
Solved
60/64
Cost / trial
$0.264
Tokens / trial
232k
Date
2026-10-22

#2

Anthropic logoclaude-sonnet-5-5

90.6%

Pass^3

Effort
xhigh
Pass@3
96.9%
Solved
58/64
Cost / trial
$0.290
Tokens / trial
338k
Date
2026-10-23

#3

Anthropic logoclaude-sonnet-5-5

89.1%

Pass^3

Effort
high
Pass@3
100.0%
Solved
57/64
Cost / trial
$0.106
Tokens / trial
199k
Date
2026-10-20

#4

Anthropic logoclaude-haiku-5-5

87.5%

Pass^3

Effort
xhigh
Pass@3
100.0%
Solved
56/64
Cost / trial
$0.058
Tokens / trial
438k
Date
2026-10-27

#5

Google logogemini-4-argon

87.5%

Pass^3

Effort
low
Pass@3
98.4%
Solved
56/64
Cost / trial
$0.268
Tokens / trial
335k
Date
2026-10-23

#6

OpenAI logogpt-6-astra

87.5%

Pass^3

Effort
low
Pass@3
96.9%
Solved
56/64
Cost / trial
$0.195
Tokens / trial
92.4k
Date
2026-10-22

#7

OpenAI logogpt-6.1-sol

87.5%

Pass^3

Effort
medium
Pass@3
95.3%
Solved
56/64
Cost / trial
$0.037
Tokens / trial
103k
Date
2026-10-22

#8

Anthropic logoclaude-opus-5-5

87.5%

Pass^3

Effort
max
Pass@3
92.2%
Solved
56/64
Cost / trial
$1.691
Tokens / trial
637k
Date
2026-10-22

#9

Anthropic logoclaude-haiku-5-5

85.9%

Pass^3

Effort
high
Pass@3
100.0%
Solved
55/64
Cost / trial
$0.018
Tokens / trial
283k
Date
2026-10-27

#10

Anthropic logoclaude-opus-5-5

85.9%

Pass^3

Effort
low
Pass@3
98.4%
Solved
55/64
Cost / trial
$0.126
Tokens / trial
164k
Date
2026-10-22

#11

OpenAI logogpt-6-astra

85.9%

Pass^3

Effort
medium
Pass@3
93.8%
Solved
55/64
Cost / trial
$0.244
Tokens / trial
105k
Date
2026-10-22

#12

Meta logomuse-spark-openOpen

85.9%

Pass^3

Effort
high
Pass@3
93.8%
Solved
55/64
Cost / trial
~$0.121
Tokens / trial
355k
Date
2026-10-24

#13

OpenAI logogpt-6.1-sol

85.9%

Pass^3

Effort
xhigh
Pass@3
92.2%
Solved
55/64
Cost / trial
$0.062
Tokens / trial
130k
Date
2026-10-23

#14

OpenAI logogpt-6.1-sol

85.9%

Pass^3

Effort
high
Pass@3
92.2%
Solved
55/64
Cost / trial
$0.051
Tokens / trial
121k
Date
2026-10-22

#15

Anthropic logoclaude-opus-5-5

84.4%

Pass^3

Effort
high
Pass@3
98.4%
Solved
54/64
Cost / trial
$0.344
Tokens / trial
267k
Date
2026-10-22

#16

OpenAI logogpt-6.1-sol

84.4%

Pass^3

Effort
low
Pass@3
96.9%
Solved
54/64
Cost / trial
$0.030
Tokens / trial
91.6k
Date
2026-10-22

#17

Google logogemini-4-argon

84.4%

Pass^3

Effort
medium
Pass@3
96.9%
Solved
54/64
Cost / trial
$0.220
Tokens / trial
305k
Date
2026-10-23

#18

Anthropic logoclaude-sonnet-5-5

84.4%

Pass^3

Effort
max
Pass@3
96.9%
Solved
54/64
Cost / trial
$1.336
Tokens / trial
854k
Date
2026-10-23

#19

OpenAI logogpt-6-luna

84.4%

Pass^3

Effort
xhigh
Pass@3
93.8%
Solved
54/64
Cost / trial
$0.007
Tokens / trial
213k
Date
2026-10-23

#20

Anthropic logoclaude-opus-5-5

84.4%

Pass^3

Effort
xhigh
Pass@3
93.8%
Solved
54/64
Cost / trial
$0.656
Tokens / trial
380k
Date
2026-10-22

#21

OpenAI logogpt-6-astra

84.4%

Pass^3

Effort
xhigh
Pass@3
90.6%
Solved
54/64
Cost / trial
$0.356
Tokens / trial
127k
Date
2026-10-22

#22

Anthropic logoclaude-haiku-5-5

82.8%

Pass^3

Effort
max
Pass@3
95.3%
Solved
53/64
Cost / trial
$0.224
Tokens / trial
805k
Date
2026-10-28

#23

OpenAI logogpt-6-astra

82.8%

Pass^3

Effort
high
Pass@3
93.8%
Solved
53/64
Cost / trial
$0.310
Tokens / trial
120k
Date
2026-10-22

#24

Moonshot AI logokimi-k3Open

81.3%

Pass^3

Effort
max
Pass@3
96.9%
Solved
52/64
Cost / trial
~$0.087
Tokens / trial
191k
Date
2026-10-23

#25

Mistral logomistral-large-4-0Open

81.3%

Pass^3

Effort
high
Pass@3
95.3%
Solved
52/64
Cost / trial
$0.052
Tokens / trial
286k
Date
2026-10-27

#26

Z.ai logoglm-5.3Open

81.3%

Pass^3

Effort
max
Pass@3
95.3%
Solved
52/64
Cost / trial
~$0.030
Tokens / trial
257k
Date
2026-10-23

#27

Google logogemini-4-argon

81.3%

Pass^3

Effort
high
Pass@3
93.8%
Solved
52/64
Cost / trial
$0.370
Tokens / trial
693k
Date
2026-10-24

#28

DeepSeek logodeepseek-v4.1-flash-1017Open

81.3%

Pass^3

Effort
max
Pass@3
85.9%
Solved
52/64
Cost / trial
~$0.030
Tokens / trial
917k
Date
2026-10-23

#29

Meta logomuse-spark-openOpen

79.7%

Pass^3

Effort
xhigh
Pass@3
96.9%
Solved
51/64
Cost / trial
~$0.119
Tokens / trial
365k
Date
2026-10-24

#30

DeepSeek logodeepseek-v4.1-flash-1017Open

79.7%

Pass^3

Effort
low
Pass@3
93.8%
Solved
51/64
Cost / trial
~$0.013
Tokens / trial
332k
Date
2026-10-23

#31

Anthropic logoclaude-haiku-5-5

78.1%

Pass^3

Effort
medium
Pass@3
100.0%
Solved
50/64
Cost / trial
$0.010
Tokens / trial
218k
Date
2026-10-20

#32

Mistral logomistral-large-4-0Open

78.1%

Pass^3

Effort
low
Pass@3
96.9%
Solved
50/64
Cost / trial
$0.053
Tokens / trial
291k
Date
2026-10-27

#33

Qwen logoqwen3.8-max

78.1%

Pass^3

Effort
xhigh
Pass@3
93.8%
Solved
50/64
Cost / trial
~$0.248
Tokens / trial
456k
Date
2026-10-24

#34

OpenAI logogpt-6-luna

76.6%

Pass^3

Effort
high
Pass@3
100.0%
Solved
49/64
Cost / trial
$0.005
Tokens / trial
147k
Date
2026-10-23

#35

Z.ai logoglm-5.3Open

76.6%

Pass^3

Effort
high
Pass@3
98.4%
Solved
49/64
Cost / trial
~$0.019
Tokens / trial
210k
Date
2026-10-23

#36

Moonshot AI logokimi-k3Open

76.6%

Pass^3

Effort
high
Pass@3
96.9%
Solved
49/64
Cost / trial
~$0.074
Tokens / trial
200k
Date
2026-10-22

#37

xAI logogrok-4.7

76.6%

Pass^3

Effort
high
Pass@3
93.8%
Solved
49/64
Cost / trial
$0.167
Tokens / trial
228k
Date
2026-10-24

#38

OpenAI logogpt-6-luna

75.0%

Pass^3

Effort
medium
Pass@3
96.9%
Solved
48/64
Cost / trial
$0.004
Tokens / trial
135k
Date
2026-10-23

#39

Anthropic logoclaude-sonnet-5-5

75.0%

Pass^3

Effort
medium
Pass@3
96.9%
Solved
48/64
Cost / trial
$0.064
Tokens / trial
157k
Date
2026-10-23

#40

xAI logogrok-4.7

75.0%

Pass^3

Effort
xhigh
Pass@3
92.2%
Solved
48/64
Cost / trial
$0.192
Tokens / trial
264k
Date
2026-10-24

#41

DeepSeek logodeepseek-v4.1-flash-1017Open

75.0%

Pass^3

Effort
high
Pass@3
84.4%
Solved
48/64
Cost / trial
~$0.015
Tokens / trial
356k
Date
2026-10-23

#42

Qwen logoqwen3.8-max

73.4%

Pass^3

Effort
low
Pass@3
98.4%
Solved
47/64
Cost / trial
~$0.142
Tokens / trial
297k
Date
2026-10-24

#43

Qwen logoqwen3.8-max

73.4%

Pass^3

Effort
medium
Pass@3
95.3%
Solved
47/64
Cost / trial
~$0.143
Tokens / trial
282k
Date
2026-10-24

#44

Anthropic logoclaude-sonnet-5-5

71.9%

Pass^3

Effort
low
Pass@3
95.3%
Solved
46/64
Cost / trial
$0.059
Tokens / trial
149k
Date
2026-10-23

#45

xAI logogrok-4.7

70.3%

Pass^3

Effort
medium
Pass@3
96.9%
Solved
45/64
Cost / trial
$0.135
Tokens / trial
190k
Date
2026-10-24

#46

Meta logomuse-spark-openOpen

70.3%

Pass^3

Effort
medium
Pass@3
92.2%
Solved
45/64
Cost / trial
~$0.087
Tokens / trial
260k
Date
2026-10-24

#47

Anthropic logoclaude-haiku-5-5

68.8%

Pass^3

Effort
low
Pass@3
100.0%
Solved
44/64
Cost / trial
$0.007
Tokens / trial
192k
Date
2026-10-27

#48

xAI logogrok-4.7

68.8%

Pass^3

Effort
low
Pass@3
96.9%
Solved
44/64
Cost / trial
$0.089
Tokens / trial
132k
Date
2026-10-24

#49

Moonshot AI logokimi-k3Open

68.8%

Pass^3

Effort
low
Pass@3
93.8%
Solved
44/64
Cost / trial
~$0.039
Tokens / trial
126k
Date
2026-10-22

#50

Qwen logoqwen3.8-max

65.6%

Pass^3

Effort
off
Pass@3
96.9%
Solved
42/64
Cost / trial
~$0.103
Tokens / trial
245k
Date
2026-10-24

#51

Meta logomuse-spark-openOpen

65.6%

Pass^3

Effort
low
Pass@3
93.8%
Solved
42/64
Cost / trial
~$0.060
Tokens / trial
197k
Date
2026-10-24

#52

OpenAI logogpt-5.6-luna

65.6%

Pass^3

Effort
low
Pass@3
89.1%
Solved
42/64
Cost / trial
$0.005
Tokens / trial
94.7k
Date
2026-10-20

#53

Qwen logoqwen3.8-27bOpen

64.1%

Pass^3

Effort
low
Pass@3
95.3%
Solved
41/64
Cost / trial
$0.064
Tokens / trial
323k
Date
2026-10-27

#54

Z.ai logoglm-5.3Open

64.1%

Pass^3

Effort
low
Pass@3
93.8%
Solved
41/64
Cost / trial
~$0.013
Tokens / trial
157k
Date
2026-10-23

#55

MiniMax logominimax-m3.1-flashOpen

64.1%

Pass^3

Effort
high
Pass@3
89.1%
Solved
41/64
Cost / trial
~$0.027
Tokens / trial
775k
Date
2026-10-27

#56

Anthropic logoclaude-haiku-5-5

62.5%

Pass^3

Effort
off
Pass@3
92.2%
Solved
40/64
Cost / trial
$0.008
Tokens / trial
211k
Date
2026-10-27

#57

Qwen logoqwen3.8-27bOpen

60.9%

Pass^3

Effort
high
Pass@3
87.5%
Solved
39/64
Cost / trial
$0.072
Tokens / trial
366k
Date
2026-10-27

#58

OpenAI logogpt-6-luna

56.3%

Pass^3

Effort
low
Pass@3
93.8%
Solved
36/64
Cost / trial
$0.002
Tokens / trial
99.7k
Date
2026-10-23

#59

Google logogemma-4-31b-itOpen

56.3%

Pass^3

Effort
low
Pass@3
82.8%
Solved
36/64
Cost / trial
$0.015
Tokens / trial
265k
Date
2026-10-27

#60

OpenAI logogpt-5.4-mini

54.7%

Pass^3

Effort
low
Pass@3
92.2%
Solved
35/64
Cost / trial
$0.023
Tokens / trial
124k
Date
2026-10-20

#61

DeepSeek logodeepseek-v4.1-flash-1017Open

54.7%

Pass^3

Effort
off
Pass@3
82.8%
Solved
35/64
Cost / trial
~$0.009
Tokens / trial
281k
Date
2026-10-23

#62

Google logogemma-4-31b-itOpen

53.1%

Pass^3

Effort
high
Pass@3
85.9%
Solved
34/64
Cost / trial
$0.014
Tokens / trial
247k
Date
2026-10-27

#63

MiniMax logominimax-m3.1-flashOpen

51.6%

Pass^3

Effort
low
Pass@3
95.3%
Solved
33/64
Cost / trial
~$0.015
Tokens / trial
478k
Date
2026-10-27

#64

Qwen logoqwen3.6-27bOpen

51.6%

Pass^3

Effort
high
Pass@3
87.5%
Solved
33/64
Cost / trial
$0.048
Tokens / trial
252k
Date
2026-10-27

#65

OpenAI logogpt-6-luna

48.4%

Pass^3

Effort
off
Pass@3
95.3%
Solved
31/64
Cost / trial
$0.005
Tokens / trial
253k
Date
2026-10-23

#66

Qwen logoqwen3.6-35b-a3bOpen

43.8%

Pass^3

Effort
high
Pass@3
78.1%
Solved
28/64
Cost / trial
$0.023
Tokens / trial
307k
Date
2026-10-27

#67

Qwen logoqwen3.6-27bOpen

40.6%

Pass^3

Effort
low
Pass@3
89.1%
Solved
26/64
Cost / trial
$0.043
Tokens / trial
256k
Date
2026-10-27

#68

Qwen logoqwen3.6-35b-a3bOpen

29.7%

Pass^3

Effort
low
Pass@3
81.3%
Solved
19/64
Cost / trial
$0.029
Tokens / trial
369k
Date
2026-10-27

#69

xAI logogrok-4.7

26.6%

Pass^3

Effort
max
Pass@3
92.2%
Solved
17/64
Cost / trial
$0.142
Tokens / trial
192k
Date
2026-10-24

#70

Anthropic logoclaude-haiku-4-5

26.6%

Pass^3

Effort
off
Pass@3
62.5%
Solved
17/64
Cost / trial
$0.080
Tokens / trial
202k
Date
2026-10-20

#71

OpenAI logogpt-5.4-nano

10.9%

Pass^3

Effort
off
Pass@3
32.8%
Solved
7/64
Cost / trial
$0.005
Tokens / trial
111k
Date
2026-10-20

~ Estimated from OpenRouter list prices: the run (Fireworks or self-hosted) reported no cost. — No list price available.

By category

Pass^3 per task category. Green is 90%+, yellow 70%+, red below 70%.

Scroll sideways for all categories.

#ModelDashboardsGrafana APIInvestigationLogsMetricsTraces
1
Anthropic logoclaude-opus-5-5medium
71%100%91%100%94%100%
2
Anthropic logoclaude-sonnet-5-5xhigh
86%100%91%100%94%85%
3
Anthropic logoclaude-sonnet-5-5high
86%100%100%100%94%69%
4
Anthropic logoclaude-haiku-5-5xhigh
86%100%100%100%88%62%
5
Google logogemini-4-argonlow
71%83%100%100%81%92%
6
OpenAI logogpt-6-astralow
57%100%91%90%88%92%
7
OpenAI logogpt-6.1-solmedium
57%100%100%100%94%77%
8
Anthropic logoclaude-opus-5-5max
71%100%91%100%88%85%
9
Anthropic logoclaude-haiku-5-5high
86%83%82%100%100%62%
10
Anthropic logoclaude-opus-5-5low
71%100%82%90%94%85%
11
OpenAI logogpt-6-astramedium
57%100%91%90%88%85%
12
Meta logomuse-spark-openhighOpen
100%100%91%100%81%62%
13
OpenAI logogpt-6.1-solxhigh
57%100%100%100%88%77%
14
OpenAI logogpt-6.1-solhigh
43%100%100%100%88%85%
15
Anthropic logoclaude-opus-5-5high
71%100%91%90%81%85%
16
OpenAI logogpt-6.1-sollow
57%100%91%90%81%92%
17
Google logogemini-4-argonmedium
57%83%91%100%81%92%
18
Anthropic logoclaude-sonnet-5-5max
57%100%82%100%88%85%
19
OpenAI logogpt-6-lunaxhigh
71%100%82%90%88%85%
20
Anthropic logoclaude-opus-5-5xhigh
43%100%91%90%88%92%
21
OpenAI logogpt-6-astraxhigh
43%100%91%100%88%85%
22
Anthropic logoclaude-haiku-5-5max
86%100%82%100%88%62%
23
OpenAI logogpt-6-astrahigh
43%100%100%100%81%77%
24
Moonshot AI logokimi-k3maxOpen
29%100%64%100%94%85%
25
Mistral logomistral-large-4-0highOpen
14%100%100%80%94%85%
26
Z.ai logoglm-5.3maxOpen
29%100%91%100%81%85%
27
Google logogemini-4-argonhigh
71%100%91%90%81%69%
28
DeepSeek logodeepseek-v4.1-flash-1017maxOpen
71%83%100%90%75%77%
29
Meta logomuse-spark-openxhighOpen
100%100%91%90%81%38%
30
DeepSeek logodeepseek-v4.1-flash-1017lowOpen
43%100%91%100%75%77%
31
Anthropic logoclaude-haiku-5-5medium
57%100%82%100%75%62%
32
Mistral logomistral-large-4-0lowOpen
29%100%91%90%81%77%
33
Qwen logoqwen3.8-maxxhigh
43%100%91%80%81%77%
34
OpenAI logogpt-6-lunahigh
57%100%82%80%81%62%
35
Z.ai logoglm-5.3highOpen
43%100%91%80%75%77%
36
Moonshot AI logokimi-k3highOpen
71%100%55%90%69%92%
37
xAI logogrok-4.7high
57%100%100%80%63%77%
38
OpenAI logogpt-6-lunamedium
71%100%64%90%63%77%
39
Anthropic logoclaude-sonnet-5-5medium
71%100%64%80%88%62%
40
xAI logogrok-4.7xhigh
57%100%91%90%63%69%
41
DeepSeek logodeepseek-v4.1-flash-1017highOpen
57%83%73%90%94%54%
42
Qwen logoqwen3.8-maxlow
14%100%82%100%75%69%
43
Qwen logoqwen3.8-maxmedium
43%100%82%80%69%77%
44
Anthropic logoclaude-sonnet-5-5low
71%100%55%80%81%62%
45
xAI logogrok-4.7medium
43%100%91%80%50%77%
46
Meta logomuse-spark-openmediumOpen
71%100%82%80%81%31%
47
Anthropic logoclaude-haiku-5-5low
43%100%55%90%75%62%
48
xAI logogrok-4.7low
43%100%82%70%63%69%
49
Moonshot AI logokimi-k3lowOpen
71%83%73%90%75%38%
50
Qwen logoqwen3.8-maxoff
14%100%64%60%75%77%
51
Meta logomuse-spark-openlowOpen
43%100%45%80%94%38%
52
OpenAI logogpt-5.6-lunalow
43%100%64%70%81%46%
53
Qwen logoqwen3.8-27blowOpen
14%100%73%70%88%38%
54
Z.ai logoglm-5.3lowOpen
29%100%73%80%69%46%
55
MiniMax logominimax-m3.1-flashhighOpen
29%100%82%80%56%54%
56
Anthropic logoclaude-haiku-5-5off
57%100%64%60%69%46%
57
Qwen logoqwen3.8-27bhighOpen
29%100%64%60%69%54%
58
OpenAI logogpt-6-lunalow
43%100%45%30%81%46%
59
Google logogemma-4-31b-itlowOpen
29%83%45%60%69%54%
60
OpenAI logogpt-5.4-minilow
29%100%45%40%75%46%
61
DeepSeek logodeepseek-v4.1-flash-1017offOpen
0%100%45%60%75%46%
62
Google logogemma-4-31b-ithighOpen
43%100%18%70%69%31%
63
MiniMax logominimax-m3.1-flashlowOpen
14%83%55%70%44%54%
64
Qwen logoqwen3.6-27bhighOpen
14%100%45%60%50%54%
65
OpenAI logogpt-6-lunaoff
29%100%27%60%63%31%
66
Qwen logoqwen3.6-35b-a3bhighOpen
14%100%45%40%38%46%
67
Qwen logoqwen3.6-27blowOpen
29%100%36%50%31%31%
68
Qwen logoqwen3.6-35b-a3blowOpen
0%100%27%30%25%15%
69
xAI logogrok-4.7max
29%50%27%10%31%23%
70
Anthropic logoclaude-haiku-4-5off
0%83%9%10%50%15%
71
OpenAI logogpt-5.4-nanooff
0%83%0%10%0%8%

Open-weight models ran viaOpenRouterorFireworks; some also ran self-hosted.