Leaderboard

Ranked by Pass^3. Every task is run three times.

o11y-bench 1.1o11y-bench 1.0Scores aren't comparable across versions: 1.1 uses an outcome-based LLM judge and adds a task.
Pass^3
Task passes only if all three attempts pass.
Pass@3
Task passes if any of the three attempts passes.

Timeouts (10 min per attempt), crashes and run errors count as failed attempts.

Effort

Pass^3 vs. cost

Each point is a model at one effort level. The dotted line is the Pareto frontier: no other run is both cheaper and higher scoring. Up and left is better.

Scatter plot of o11y-bench model configurations. The vertical axis shows Pass^3 and the horizontal axis shows average cost per trial in USD on a log scale, with a dotted Pareto frontier line through the best configurations.

AnthropicOpenAIGooglexAIOtherPareto frontierEstimated cost

Rankings

#1

Anthropic logoclaude-opus-4-8

79.4%

Pass^3

Effort
high
Pass@3
88.9%
Solved
50/63
Cost / trial
$0.436
Tokens / trial
182k
Date
2026-08-21

#2

Anthropic logoclaude-opus-4-7

79.4%

Pass^3

Effort
off
Pass@3
87.3%
Solved
50/63
Cost / trial
$0.320
Tokens / trial
157k
Date
2026-04-20

#3

Anthropic logoclaude-opus-5

76.2%

Pass^3

Effort
high
Pass@3
93.7%
Solved
48/63
Cost / trial
$0.806
Tokens / trial
289k
Date
2026-08-21

#4

Anthropic logoclaude-opus-4-7

73.0%

Pass^3

Effort
high
Pass@3
90.5%
Solved
46/63
Cost / trial
$0.393
Tokens / trial
170k
Date
2026-04-21

#5

Anthropic logoclaude-opus-5

68.3%

Pass^3

Effort
low
Pass@3
92.1%
Solved
43/63
Cost / trial
$0.253
Tokens / trial
140k
Date
2026-08-21

#6

Anthropic logoclaude-sonnet-4-6

68.3%

Pass^3

Effort
high
Pass@3
84.1%
Solved
43/63
Cost / trial
$0.195
Tokens / trial
119k
Date
2026-04-21

#7

OpenAI logogpt-5.6-sol

66.7%

Pass^3

Effort
high
Pass@3
90.5%
Solved
42/63
Cost / trial
$0.267
Tokens / trial
154k
Date
2026-08-21

#8

Anthropic logoclaude-opus-4-6

66.7%

Pass^3

Effort
off
Pass@3
90.5%
Solved
42/63
Cost / trial
$0.281
Tokens / trial
131k
Date
2026-04-21

#9

OpenAI logogpt-6-luna

66.7%

Pass^3

Effort
low
Pass@3
87.3%
Solved
42/63
Cost / trial
$0.004
Tokens / trial
111k
Date
2026-09-22

#10

OpenAI logogpt-5.6-luna

66.7%

Pass^3

Effort
high
Pass@3
87.3%
Solved
42/63
Cost / trial
$0.013
Tokens / trial
187k
Date
2026-08-20

#11

OpenAI logogpt-5.6-terra

65.1%

Pass^3

Effort
high
Pass@3
87.3%
Solved
41/63
Cost / trial
$0.078
Tokens / trial
112k
Date
2026-08-21

#12

OpenAI logogpt-6-sol

65.1%

Pass^3

Effort
low
Pass@3
87.3%
Solved
41/63
Cost / trial
$0.059
Tokens / trial
101k
Date
2026-09-22

#13

Anthropic logoclaude-opus-4-7

63.5%

Pass^3

Effort
low
Pass@3
85.7%
Solved
40/63
Cost / trial
$0.229
Tokens / trial
128k
Date
2026-04-21

#14

Google logogemini-3.1-pro-preview

63.5%

Pass^3

Effort
high
Pass@3
82.5%
Solved
40/63
Cost / trial
$0.182
Tokens / trial
290k
Date
2026-04-21

#15

OpenAI logogpt-5.4-2026-03-05

61.9%

Pass^3

Effort
high
Pass@3
84.1%
Solved
39/63
Cost / trial
$0.153
Tokens / trial
119k
Date
2026-04-21

#16

Google logogemini-3-flash-preview

61.9%

Pass^3

Effort
high
Pass@3
84.1%
Solved
39/63
Cost / trial
$0.070
Tokens / trial
410k
Date
2026-04-20

#17

Grafana logoo11y-qwen3.6-27bOpen

61.9%

Pass^3

Effort
off
Pass@3
79.4%
Solved
39/63
Cost / trial
~$0.085
Tokens / trial
244k
Date
2026-10-03

#18

Anthropic logoclaude-opus-4-8

60.3%

Pass^3

Effort
low
Pass@3
87.3%
Solved
38/63
Cost / trial
$0.289
Tokens / trial
152k
Date
2026-08-21

#19

OpenAI logogpt-5.6-terra

60.3%

Pass^3

Effort
low
Pass@3
85.7%
Solved
38/63
Cost / trial
$0.055
Tokens / trial
86.6k
Date
2026-08-20

#20

Grafana logoo11y-qwen3.6-27bOpen

60.3%

Pass^3

Effort
low
Pass@3
84.1%
Solved
38/63
Cost / trial
~$0.076
Tokens / trial
208k
Date
2026-10-03

#21

OpenAI logogpt-6-sol

58.7%

Pass^3

Effort
high
Pass@3
85.7%
Solved
37/63
Cost / trial
$0.061
Tokens / trial
103k
Date
2026-09-22

#22

Google logogemini-3-flash-preview

58.7%

Pass^3

Effort
off
Pass@3
82.5%
Solved
37/63
Cost / trial
$0.077
Tokens / trial
418k
Date
2026-04-21

#23

Anthropic logoclaude-sonnet-4-6

58.7%

Pass^3

Effort
off
Pass@3
79.4%
Solved
37/63
Cost / trial
$0.142
Tokens / trial
114k
Date
2026-04-21

#24

Anthropic logoclaude-opus-4-6

58.7%

Pass^3

Effort
high
Pass@3
76.2%
Solved
37/63
Cost / trial
$0.365
Tokens / trial
153k
Date
2026-04-21

#25

OpenAI logogpt-5.6-sol

57.1%

Pass^3

Effort
low
Pass@3
81.0%
Solved
36/63
Cost / trial
$0.138
Tokens / trial
86.9k
Date
2026-08-21

#26

Google logogemini-3.1-pro-preview

57.1%

Pass^3

Effort
low
Pass@3
76.2%
Solved
36/63
Cost / trial
$0.108
Tokens / trial
154k
Date
2026-04-21

#27

OpenAI logogpt-6-luna

55.6%

Pass^3

Effort
high
Pass@3
82.5%
Solved
35/63
Cost / trial
$0.004
Tokens / trial
106k
Date
2026-09-22

#28

OpenAI logogpt-5.4-2026-03-05

55.6%

Pass^3

Effort
low
Pass@3
82.5%
Solved
35/63
Cost / trial
$0.066
Tokens / trial
75.1k
Date
2026-04-21

#29

Qwen logoqwen3.6-plus

55.6%

Pass^3

Effort
off
Pass@3
79.4%
Solved
35/63
Cost / trial
$0.083
Tokens / trial
236k
Date
2026-04-28

#30

Qwen logoqwen3.6-27b@bf16Open

54.0%

Pass^3

Effort
off
Pass@3
84.1%
Solved
34/63
Cost / trial
~$0.094
Tokens / trial
270k
Date
2026-05-08

#31

Google logogemini-3.1-pro-preview

54.0%

Pass^3

Effort
off
Pass@3
76.2%
Solved
34/63
Cost / trial
$0.179
Tokens / trial
291k
Date
2026-04-20

#32

OpenAI logogpt-5.6-luna

50.8%

Pass^3

Effort
low
Pass@3
81.0%
Solved
32/63
Cost / trial
$0.005
Tokens / trial
76.9k
Date
2026-08-20

#33

Moonshot AI logokimi-k2.5Open

50.8%

Pass^3

Effort
off
Pass@3
81.0%
Solved
32/63
Cost / trial
$0.046
Tokens / trial
130k
Date
2026-04-28

#34

Anthropic logoclaude-sonnet-4-6

50.8%

Pass^3

Effort
low
Pass@3
76.2%
Solved
32/63
Cost / trial
$0.137
Tokens / trial
108k
Date
2026-04-21

#35

Anthropic logoclaude-opus-4-6

49.2%

Pass^3

Effort
low
Pass@3
69.8%
Solved
31/63
Cost / trial
$0.199
Tokens / trial
117k
Date
2026-04-21

#36

OpenAI logogpt-5.4-nano

47.6%

Pass^3

Effort
high
Pass@3
76.2%
Solved
30/63
Cost / trial
$0.028
Tokens / trial
283k
Date
2026-04-21

#37

Anthropic logoclaude-haiku-4-5-20251001

47.6%

Pass^3

Effort
off
Pass@3
58.7%
Solved
30/63
Cost / trial
$0.114
Tokens / trial
199k
Date
2026-04-20

#38

Google logogemini-3-flash-preview

46.0%

Pass^3

Effort
low
Pass@3
84.1%
Solved
29/63
Cost / trial
$0.038
Tokens / trial
195k
Date
2026-04-21

#39

OpenAI logogpt-5.1-codex-mini

46.0%

Pass^3

Effort
high
Pass@3
74.6%
Solved
29/63
Cost / trial
$0.017
Tokens / trial
114k
Date
2026-04-20

#40

OpenAI logogpt-5.1-codex-mini

46.0%

Pass^3

Effort
off
Pass@3
71.4%
Solved
29/63
Cost / trial
$0.014
Tokens / trial
113k
Date
2026-04-21

#41

OpenAI logogpt-5.4-2026-03-05

44.4%

Pass^3

Effort
off
Pass@3
76.2%
Solved
28/63
Cost / trial
$0.063
Tokens / trial
79.5k
Date
2026-04-21

#42

Anthropic logoclaude-haiku-4-5-20251001

39.7%

Pass^3

Effort
high
Pass@3
66.7%
Solved
25/63
Cost / trial
$0.074
Tokens / trial
151k
Date
2026-04-21

#43

Mistral logomistral-medium-3.5

39.7%

Pass^3

Effort
off
Pass@3
57.1%
Solved
25/63
Cost / trial
—
Tokens / trial
191k
Date
2026-07-03

#44

Google logogemma-4-31b-it@q8_k_xlOpen

38.1%

Pass^3

Effort
off
Pass@3
71.4%
Solved
24/63
Cost / trial
~$0.029
Tokens / trial
145k
Date
2026-05-09

#45

MiniMax logominimax-m2.7Open

36.5%

Pass^3

Effort
high
Pass@3
73.0%
Solved
23/63
Cost / trial
~$0.027
Tokens / trial
323k
Date
2026-04-25

#46

Google logogemini-3.1-flash-lite-preview

34.9%

Pass^3

Effort
high
Pass@3
71.4%
Solved
22/63
Cost / trial
$0.028
Tokens / trial
239k
Date
2026-04-21

#47

Anthropic logoclaude-haiku-4-5-20251001

34.9%

Pass^3

Effort
low
Pass@3
61.9%
Solved
22/63
Cost / trial
$0.074
Tokens / trial
146k
Date
2026-04-21

#48

Google logogemini-3.1-flash-lite-preview

33.3%

Pass^3

Effort
low
Pass@3
66.7%
Solved
21/63
Cost / trial
$0.025
Tokens / trial
224k
Date
2026-04-21

#49

Google logogemini-3.1-flash-lite-preview

33.3%

Pass^3

Effort
off
Pass@3
58.7%
Solved
21/63
Cost / trial
$0.019
Tokens / trial
199k
Date
2026-04-21

#50

xAI logogrok-4.20

31.7%

Pass^3

Effort
off
Pass@3
61.9%
Solved
20/63
Cost / trial
$0.306
Tokens / trial
350k
Date
2026-04-28

#51

OpenAI logogpt-5.4-mini

28.6%

Pass^3

Effort
high
Pass@3
76.2%
Solved
18/63
Cost / trial
$0.048
Tokens / trial
112k
Date
2026-04-21

#52

Mistral logomistral-medium-3.5

27.0%

Pass^3

Effort
high
Pass@3
58.7%
Solved
17/63
Cost / trial
—
Tokens / trial
106k
Date
2026-07-03

#53

Qwen logoqwen3.5-9bOpen

25.4%

Pass^3

Effort
off
Pass@3
61.9%
Solved
16/63
Cost / trial
~$0.068
Tokens / trial
400k
Date
2026-05-08

#54

OpenAI logogpt-5.4-mini

25.4%

Pass^3

Effort
off
Pass@3
44.4%
Solved
16/63
Cost / trial
$0.015
Tokens / trial
73.0k
Date
2026-04-21

#55

Qwen logoqwen3.8-27bOpen

22.2%

Pass^3

Effort
off
Pass@3
65.1%
Solved
14/63
Cost / trial
~$0.021
Tokens / trial
92.1k
Date
2026-08-20

#56

OpenAI logogpt-5.4-mini

20.6%

Pass^3

Effort
low
Pass@3
61.9%
Solved
13/63
Cost / trial
$0.015
Tokens / trial
61.3k
Date
2026-04-21

#57

OpenAI logogpt-5.4-nano

19.0%

Pass^3

Effort
low
Pass@3
50.8%
Solved
12/63
Cost / trial
$0.007
Tokens / trial
111k
Date
2026-04-21

#58

OpenAI logogpt-5.4-nano

11.1%

Pass^3

Effort
off
Pass@3
22.2%
Solved
7/63
Cost / trial
$0.005
Tokens / trial
79.9k
Date
2026-04-21

~ Estimated from OpenRouter list prices: the run (Fireworks or self-hosted) reported no cost. — No list price available.

By category

Pass^3 per task category. Green is 90%+, yellow 70%+, red below 70%.

Scroll sideways for all categories.

#ModelDashboardsGrafana APIInvestigationLogsMetricsTraces
1
Anthropic logoclaude-opus-4-8high
57%100%82%30%100%92%
2
Anthropic logoclaude-opus-4-7off
57%100%73%80%88%77%
3
Anthropic logoclaude-opus-5high
71%100%36%80%81%92%
4
Anthropic logoclaude-opus-4-7high
43%100%45%80%88%77%
5
Anthropic logoclaude-opus-5low
57%100%64%10%81%92%
6
Anthropic logoclaude-sonnet-4-6high
29%100%45%50%94%77%
7
OpenAI logogpt-5.6-solhigh
29%100%45%60%81%77%
8
Anthropic logoclaude-opus-4-6off
43%100%45%60%75%77%
9
OpenAI logogpt-6-lunalow
57%100%45%60%75%69%
10
OpenAI logogpt-5.6-lunahigh
43%100%55%30%88%77%
11
OpenAI logogpt-5.6-terrahigh
43%100%55%30%75%85%
12
OpenAI logogpt-6-sollow
43%100%45%50%88%62%
13
Anthropic logoclaude-opus-4-7low
43%100%36%70%69%69%
14
Google logogemini-3.1-pro-previewhigh
43%100%27%50%94%62%
15
OpenAI logogpt-5.4-2026-03-05high
29%83%64%40%81%62%
16
Google logogemini-3-flash-previewhigh
14%100%64%40%81%62%
17
Grafana logoo11y-qwen3.6-27boffOpen
43%100%45%40%69%77%
18
Anthropic logoclaude-opus-4-8low
43%100%27%30%69%92%
19
OpenAI logogpt-5.6-terralow
29%100%55%30%63%85%
20
Grafana logoo11y-qwen3.6-27blowOpen
57%100%36%40%69%69%
21
OpenAI logogpt-6-solhigh
29%100%55%30%69%69%
22
Google logogemini-3-flash-previewoff
29%100%36%40%88%54%
23
Anthropic logoclaude-sonnet-4-6off
14%100%55%30%81%62%
24
Anthropic logoclaude-opus-4-6high
29%100%64%60%56%54%
25
OpenAI logogpt-5.6-sollow
29%100%36%40%75%62%
26
Google logogemini-3.1-pro-previewlow
29%100%27%30%94%54%
27
OpenAI logogpt-6-lunahigh
43%83%55%30%75%46%
28
OpenAI logogpt-5.4-2026-03-05low
29%83%45%30%75%62%
29
Qwen logoqwen3.6-plusoff
14%100%45%30%75%62%
30
Qwen logoqwen3.6-27b@bf16offOpen
29%100%45%40%63%54%
31
Google logogemini-3.1-pro-previewoff
57%100%36%40%69%38%
32
OpenAI logogpt-5.6-lunalow
14%100%36%20%69%62%
33
Moonshot AI logokimi-k2.5offOpen
29%100%18%20%69%69%
34
Anthropic logoclaude-sonnet-4-6low
29%100%45%30%56%54%
35
Anthropic logoclaude-opus-4-6low
0%100%45%60%63%31%
36
OpenAI logogpt-5.4-nanohigh
0%100%45%50%69%23%
37
Anthropic logoclaude-haiku-4-5-20251001off
14%100%27%10%56%77%
38
Google logogemini-3-flash-previewlow
29%100%27%40%63%31%
39
OpenAI logogpt-5.1-codex-minihigh
14%100%36%20%56%54%
40
OpenAI logogpt-5.1-codex-minioff
29%100%36%10%69%38%
41
OpenAI logogpt-5.4-2026-03-05off
29%100%18%30%63%38%
42
Anthropic logoclaude-haiku-4-5-20251001high
14%100%27%0%50%54%
43
Mistral logomistral-medium-3.5off
14%100%0%30%50%54%
44
Google logogemma-4-31b-it@q8_k_xloffOpen
0%100%18%20%63%31%
45
MiniMax logominimax-m2.7highOpen
0%100%18%40%25%54%
46
Google logogemini-3.1-flash-lite-previewhigh
14%100%27%20%50%15%
47
Anthropic logoclaude-haiku-4-5-20251001low
0%100%36%0%50%31%
48
Google logogemini-3.1-flash-lite-previewlow
0%100%18%10%56%23%
49
Google logogemini-3.1-flash-lite-previewoff
0%100%18%10%63%15%
50
xAI logogrok-4.20off
0%100%9%0%44%46%
51
OpenAI logogpt-5.4-minihigh
14%83%9%10%50%15%
52
Mistral logomistral-medium-3.5high
14%100%0%0%38%31%
53
Qwen logoqwen3.5-9boffOpen
0%100%18%10%13%38%
54
OpenAI logogpt-5.4-minioff
0%83%18%10%38%15%
55
Qwen logoqwen3.8-27boffOpen
0%83%9%20%31%8%
56
OpenAI logogpt-5.4-minilow
0%83%0%0%44%8%
57
OpenAI logogpt-5.4-nanolow
0%100%0%0%38%0%
58
OpenAI logogpt-5.4-nanooff
0%100%9%0%0%0%

Open-weight models ran viaOpenRouterorFireworks; some also ran self-hosted.