#1
79.4%
Pass^3
- Effort
- high
- Pass@3
- 88.9%
- Solved
- 50/63
- Cost / trial
- $0.436
- Tokens / trial
- 182k
- Date
- 2026-08-21
Ranked by Pass^3. Every task is run three times.
Timeouts (10 min per attempt), crashes and run errors count as failed attempts.
Each point is a model at one effort level. The dotted line is the Pareto frontier: no other run is both cheaper and higher scoring. Up and left is better.
Scatter plot of o11y-bench model configurations. The vertical axis shows Pass^3 and the horizontal axis shows average cost per trial in USD on a log scale, with a dotted Pareto frontier line through the best configurations.
AnthropicOpenAIGooglexAIOtherPareto frontierEstimated cost
#1
79.4%
Pass^3
#2
79.4%
Pass^3
#3
76.2%
Pass^3
#4
73.0%
Pass^3
#5
68.3%
Pass^3
#6
68.3%
Pass^3
#7
66.7%
Pass^3
#8
66.7%
Pass^3
#9
66.7%
Pass^3
#10
66.7%
Pass^3
#11
65.1%
Pass^3
#12
65.1%
Pass^3
#13
63.5%
Pass^3
#14
63.5%
Pass^3
#15
61.9%
Pass^3
#16
61.9%
Pass^3
#17
61.9%
Pass^3
#18
60.3%
Pass^3
#19
60.3%
Pass^3
#20
60.3%
Pass^3
#21
58.7%
Pass^3
#22
58.7%
Pass^3
#23
58.7%
Pass^3
#24
58.7%
Pass^3
#25
57.1%
Pass^3
#26
57.1%
Pass^3
#27
55.6%
Pass^3
#28
55.6%
Pass^3
#29
55.6%
Pass^3
#30
54.0%
Pass^3
#31
54.0%
Pass^3
#32
50.8%
Pass^3
#33
50.8%
Pass^3
#34
50.8%
Pass^3
#35
49.2%
Pass^3
#36
47.6%
Pass^3
#37
47.6%
Pass^3
#38
46.0%
Pass^3
#39
46.0%
Pass^3
#40
46.0%
Pass^3
#41
44.4%
Pass^3
#42
39.7%
Pass^3
#43
mistral-medium-3.539.7%
Pass^3
#44
38.1%
Pass^3
#45
36.5%
Pass^3
#46
34.9%
Pass^3
#47
34.9%
Pass^3
#48
33.3%
Pass^3
#49
33.3%
Pass^3
#50
31.7%
Pass^3
#51
28.6%
Pass^3
#52
mistral-medium-3.527.0%
Pass^3
#53
25.4%
Pass^3
#54
25.4%
Pass^3
#55
22.2%
Pass^3
#56
20.6%
Pass^3
#57
19.0%
Pass^3
#58
11.1%
Pass^3
| # | Model | Effort | Solved | Date | ||||
|---|---|---|---|---|---|---|---|---|
| 1 | high | 79.4% | 88.9% | 50/63 | $0.436 | 182k | 2026-08-21 | |
| 2 | off | 79.4% | 87.3% | 50/63 | $0.320 | 157k | 2026-04-20 | |
| 3 | high | 76.2% | 93.7% | 48/63 | $0.806 | 289k | 2026-08-21 | |
| 4 | high | 73.0% | 90.5% | 46/63 | $0.393 | 170k | 2026-04-21 | |
| 5 | low | 68.3% | 92.1% | 43/63 | $0.253 | 140k | 2026-08-21 | |
| 6 | high | 68.3% | 84.1% | 43/63 | $0.195 | 119k | 2026-04-21 | |
| 7 | high | 66.7% | 90.5% | 42/63 | $0.267 | 154k | 2026-08-21 | |
| 8 | off | 66.7% | 90.5% | 42/63 | $0.281 | 131k | 2026-04-21 | |
| 9 | low | 66.7% | 87.3% | 42/63 | $0.004 | 111k | 2026-09-22 | |
| 10 | high | 66.7% | 87.3% | 42/63 | $0.013 | 187k | 2026-08-20 | |
| 11 | high | 65.1% | 87.3% | 41/63 | $0.078 | 112k | 2026-08-21 | |
| 12 | low | 65.1% | 87.3% | 41/63 | $0.059 | 101k | 2026-09-22 | |
| 13 | low | 63.5% | 85.7% | 40/63 | $0.229 | 128k | 2026-04-21 | |
| 14 | high | 63.5% | 82.5% | 40/63 | $0.182 | 290k | 2026-04-21 | |
| 15 | high | 61.9% | 84.1% | 39/63 | $0.153 | 119k | 2026-04-21 | |
| 16 | high | 61.9% | 84.1% | 39/63 | $0.070 | 410k | 2026-04-20 | |
| 17 | off | 61.9% | 79.4% | 39/63 | ~$0.085 | 244k | 2026-10-03 | |
| 18 | low | 60.3% | 87.3% | 38/63 | $0.289 | 152k | 2026-08-21 | |
| 19 | low | 60.3% | 85.7% | 38/63 | $0.055 | 86.6k | 2026-08-20 | |
| 20 | low | 60.3% | 84.1% | 38/63 | ~$0.076 | 208k | 2026-10-03 | |
| 21 | high | 58.7% | 85.7% | 37/63 | $0.061 | 103k | 2026-09-22 | |
| 22 | off | 58.7% | 82.5% | 37/63 | $0.077 | 418k | 2026-04-21 | |
| 23 | off | 58.7% | 79.4% | 37/63 | $0.142 | 114k | 2026-04-21 | |
| 24 | high | 58.7% | 76.2% | 37/63 | $0.365 | 153k | 2026-04-21 | |
| 25 | low | 57.1% | 81.0% | 36/63 | $0.138 | 86.9k | 2026-08-21 | |
| 26 | low | 57.1% | 76.2% | 36/63 | $0.108 | 154k | 2026-04-21 | |
| 27 | high | 55.6% | 82.5% | 35/63 | $0.004 | 106k | 2026-09-22 | |
| 28 | low | 55.6% | 82.5% | 35/63 | $0.066 | 75.1k | 2026-04-21 | |
| 29 | off | 55.6% | 79.4% | 35/63 | $0.083 | 236k | 2026-04-28 | |
| 30 | off | 54.0% | 84.1% | 34/63 | ~$0.094 | 270k | 2026-05-08 | |
| 31 | off | 54.0% | 76.2% | 34/63 | $0.179 | 291k | 2026-04-20 | |
| 32 | low | 50.8% | 81.0% | 32/63 | $0.005 | 76.9k | 2026-08-20 | |
| 33 | off | 50.8% | 81.0% | 32/63 | $0.046 | 130k | 2026-04-28 | |
| 34 | low | 50.8% | 76.2% | 32/63 | $0.137 | 108k | 2026-04-21 | |
| 35 | low | 49.2% | 69.8% | 31/63 | $0.199 | 117k | 2026-04-21 | |
| 36 | high | 47.6% | 76.2% | 30/63 | $0.028 | 283k | 2026-04-21 | |
| 37 | off | 47.6% | 58.7% | 30/63 | $0.114 | 199k | 2026-04-20 | |
| 38 | low | 46.0% | 84.1% | 29/63 | $0.038 | 195k | 2026-04-21 | |
| 39 | high | 46.0% | 74.6% | 29/63 | $0.017 | 114k | 2026-04-20 | |
| 40 | off | 46.0% | 71.4% | 29/63 | $0.014 | 113k | 2026-04-21 | |
| 41 | off | 44.4% | 76.2% | 28/63 | $0.063 | 79.5k | 2026-04-21 | |
| 42 | high | 39.7% | 66.7% | 25/63 | $0.074 | 151k | 2026-04-21 | |
| 43 | mistral-medium-3.5 | off | 39.7% | 57.1% | 25/63 | — | 191k | 2026-07-03 |
| 44 | off | 38.1% | 71.4% | 24/63 | ~$0.029 | 145k | 2026-05-09 | |
| 45 | high | 36.5% | 73.0% | 23/63 | ~$0.027 | 323k | 2026-04-25 | |
| 46 | high | 34.9% | 71.4% | 22/63 | $0.028 | 239k | 2026-04-21 | |
| 47 | low | 34.9% | 61.9% | 22/63 | $0.074 | 146k | 2026-04-21 | |
| 48 | low | 33.3% | 66.7% | 21/63 | $0.025 | 224k | 2026-04-21 | |
| 49 | off | 33.3% | 58.7% | 21/63 | $0.019 | 199k | 2026-04-21 | |
| 50 | off | 31.7% | 61.9% | 20/63 | $0.306 | 350k | 2026-04-28 | |
| 51 | high | 28.6% | 76.2% | 18/63 | $0.048 | 112k | 2026-04-21 | |
| 52 | mistral-medium-3.5 | high | 27.0% | 58.7% | 17/63 | — | 106k | 2026-07-03 |
| 53 | off | 25.4% | 61.9% | 16/63 | ~$0.068 | 400k | 2026-05-08 | |
| 54 | off | 25.4% | 44.4% | 16/63 | $0.015 | 73.0k | 2026-04-21 | |
| 55 | off | 22.2% | 65.1% | 14/63 | ~$0.021 | 92.1k | 2026-08-20 | |
| 56 | low | 20.6% | 61.9% | 13/63 | $0.015 | 61.3k | 2026-04-21 | |
| 57 | low | 19.0% | 50.8% | 12/63 | $0.007 | 111k | 2026-04-21 | |
| 58 | off | 11.1% | 22.2% | 7/63 | $0.005 | 79.9k | 2026-04-21 |
~ Estimated from OpenRouter list prices: the run (Fireworks or self-hosted) reported no cost. — No list price available.
Pass^3 per task category. Green is 90%+, yellow 70%+, red below 70%.
Scroll sideways for all categories.
| # | Model | Dashboards | Grafana API | Investigation | Logs | Metrics | Traces |
|---|---|---|---|---|---|---|---|
| 1 | 57% | 100% | 82% | 30% | 100% | 92% | |
| 2 | 57% | 100% | 73% | 80% | 88% | 77% | |
| 3 | 71% | 100% | 36% | 80% | 81% | 92% | |
| 4 | 43% | 100% | 45% | 80% | 88% | 77% | |
| 5 | 57% | 100% | 64% | 10% | 81% | 92% | |
| 6 | 29% | 100% | 45% | 50% | 94% | 77% | |
| 7 | 29% | 100% | 45% | 60% | 81% | 77% | |
| 8 | 43% | 100% | 45% | 60% | 75% | 77% | |
| 9 | 57% | 100% | 45% | 60% | 75% | 69% | |
| 10 | 43% | 100% | 55% | 30% | 88% | 77% | |
| 11 | 43% | 100% | 55% | 30% | 75% | 85% | |
| 12 | 43% | 100% | 45% | 50% | 88% | 62% | |
| 13 | 43% | 100% | 36% | 70% | 69% | 69% | |
| 14 | 43% | 100% | 27% | 50% | 94% | 62% | |
| 15 | 29% | 83% | 64% | 40% | 81% | 62% | |
| 16 | 14% | 100% | 64% | 40% | 81% | 62% | |
| 17 | 43% | 100% | 45% | 40% | 69% | 77% | |
| 18 | 43% | 100% | 27% | 30% | 69% | 92% | |
| 19 | 29% | 100% | 55% | 30% | 63% | 85% | |
| 20 | 57% | 100% | 36% | 40% | 69% | 69% | |
| 21 | 29% | 100% | 55% | 30% | 69% | 69% | |
| 22 | 29% | 100% | 36% | 40% | 88% | 54% | |
| 23 | 14% | 100% | 55% | 30% | 81% | 62% | |
| 24 | 29% | 100% | 64% | 60% | 56% | 54% | |
| 25 | 29% | 100% | 36% | 40% | 75% | 62% | |
| 26 | 29% | 100% | 27% | 30% | 94% | 54% | |
| 27 | 43% | 83% | 55% | 30% | 75% | 46% | |
| 28 | 29% | 83% | 45% | 30% | 75% | 62% | |
| 29 | 14% | 100% | 45% | 30% | 75% | 62% | |
| 30 | 29% | 100% | 45% | 40% | 63% | 54% | |
| 31 | 57% | 100% | 36% | 40% | 69% | 38% | |
| 32 | 14% | 100% | 36% | 20% | 69% | 62% | |
| 33 | 29% | 100% | 18% | 20% | 69% | 69% | |
| 34 | 29% | 100% | 45% | 30% | 56% | 54% | |
| 35 | 0% | 100% | 45% | 60% | 63% | 31% | |
| 36 | 0% | 100% | 45% | 50% | 69% | 23% | |
| 37 | 14% | 100% | 27% | 10% | 56% | 77% | |
| 38 | 29% | 100% | 27% | 40% | 63% | 31% | |
| 39 | 14% | 100% | 36% | 20% | 56% | 54% | |
| 40 | 29% | 100% | 36% | 10% | 69% | 38% | |
| 41 | 29% | 100% | 18% | 30% | 63% | 38% | |
| 42 | 14% | 100% | 27% | 0% | 50% | 54% | |
| 43 | mistral-medium-3.5off | 14% | 100% | 0% | 30% | 50% | 54% |
| 44 | 0% | 100% | 18% | 20% | 63% | 31% | |
| 45 | 0% | 100% | 18% | 40% | 25% | 54% | |
| 46 | 14% | 100% | 27% | 20% | 50% | 15% | |
| 47 | 0% | 100% | 36% | 0% | 50% | 31% | |
| 48 | 0% | 100% | 18% | 10% | 56% | 23% | |
| 49 | 0% | 100% | 18% | 10% | 63% | 15% | |
| 50 | 0% | 100% | 9% | 0% | 44% | 46% | |
| 51 | 14% | 83% | 9% | 10% | 50% | 15% | |
| 52 | mistral-medium-3.5high | 14% | 100% | 0% | 0% | 38% | 31% |
| 53 | 0% | 100% | 18% | 10% | 13% | 38% | |
| 54 | 0% | 83% | 18% | 10% | 38% | 15% | |
| 55 | 0% | 83% | 9% | 20% | 31% | 8% | |
| 56 | 0% | 83% | 0% | 0% | 44% | 8% | |
| 57 | 0% | 100% | 0% | 0% | 38% | 0% | |
| 58 | 0% | 100% | 9% | 0% | 0% | 0% |
Open-weight models ran viaOpenRouterorFireworks; some also ran self-hosted.