#1
93.8%
Pass^3
- Effort
- medium
- Pass@3
- 98.4%
- Solved
- 60/64
- Cost / trial
- $0.264
- Tokens / trial
- 232k
- Date
- 2026-10-22
Ranked by Pass^3. Every task is run three times.
Timeouts (10 min per attempt), crashes and run errors count as failed attempts.
Each point is a model at one effort level. The dotted line is the Pareto frontier: no other run is both cheaper and higher scoring. Up and left is better.
Scatter plot of o11y-bench model configurations. The vertical axis shows Pass^3 and the horizontal axis shows average cost per trial in USD on a log scale, with a dotted Pareto frontier line through the best configurations.
AnthropicOpenAIGooglexAIOtherPareto frontierEstimated cost
#1
93.8%
Pass^3
#2
90.6%
Pass^3
#3
89.1%
Pass^3
#4
87.5%
Pass^3
#5
87.5%
Pass^3
#6
87.5%
Pass^3
#7
87.5%
Pass^3
#8
87.5%
Pass^3
#9
85.9%
Pass^3
#10
85.9%
Pass^3
#11
85.9%
Pass^3
#12
85.9%
Pass^3
#13
85.9%
Pass^3
#14
85.9%
Pass^3
#15
84.4%
Pass^3
#16
84.4%
Pass^3
#17
84.4%
Pass^3
#18
84.4%
Pass^3
#19
84.4%
Pass^3
#20
84.4%
Pass^3
#21
84.4%
Pass^3
#22
82.8%
Pass^3
#23
82.8%
Pass^3
#24
81.3%
Pass^3
#25
mistral-large-4-0Open81.3%
Pass^3
#26
81.3%
Pass^3
#27
81.3%
Pass^3
#28
81.3%
Pass^3
#29
79.7%
Pass^3
#30
79.7%
Pass^3
#31
78.1%
Pass^3
#32
mistral-large-4-0Open78.1%
Pass^3
#33
78.1%
Pass^3
#34
76.6%
Pass^3
#35
76.6%
Pass^3
#36
76.6%
Pass^3
#37
76.6%
Pass^3
#38
75.0%
Pass^3
#39
75.0%
Pass^3
#40
75.0%
Pass^3
#41
75.0%
Pass^3
#42
73.4%
Pass^3
#43
73.4%
Pass^3
#44
71.9%
Pass^3
#45
70.3%
Pass^3
#46
70.3%
Pass^3
#47
68.8%
Pass^3
#48
68.8%
Pass^3
#49
68.8%
Pass^3
#50
65.6%
Pass^3
#51
65.6%
Pass^3
#52
65.6%
Pass^3
#53
64.1%
Pass^3
#54
64.1%
Pass^3
#55
64.1%
Pass^3
#56
62.5%
Pass^3
#57
60.9%
Pass^3
#58
56.3%
Pass^3
#59
56.3%
Pass^3
#60
54.7%
Pass^3
#61
54.7%
Pass^3
#62
53.1%
Pass^3
#63
51.6%
Pass^3
#64
51.6%
Pass^3
#65
48.4%
Pass^3
#66
43.8%
Pass^3
#67
40.6%
Pass^3
#68
29.7%
Pass^3
#69
26.6%
Pass^3
#70
26.6%
Pass^3
#71
10.9%
Pass^3
| # | Model | Effort | Solved | Date | ||||
|---|---|---|---|---|---|---|---|---|
| 1 | medium | 93.8% | 98.4% | 60/64 | $0.264 | 232k | 2026-10-22 | |
| 2 | xhigh | 90.6% | 96.9% | 58/64 | $0.290 | 338k | 2026-10-23 | |
| 3 | high | 89.1% | 100.0% | 57/64 | $0.106 | 199k | 2026-10-20 | |
| 4 | xhigh | 87.5% | 100.0% | 56/64 | $0.058 | 438k | 2026-10-27 | |
| 5 | low | 87.5% | 98.4% | 56/64 | $0.268 | 335k | 2026-10-23 | |
| 6 | low | 87.5% | 96.9% | 56/64 | $0.195 | 92.4k | 2026-10-22 | |
| 7 | medium | 87.5% | 95.3% | 56/64 | $0.037 | 103k | 2026-10-22 | |
| 8 | max | 87.5% | 92.2% | 56/64 | $1.691 | 637k | 2026-10-22 | |
| 9 | high | 85.9% | 100.0% | 55/64 | $0.018 | 283k | 2026-10-27 | |
| 10 | low | 85.9% | 98.4% | 55/64 | $0.126 | 164k | 2026-10-22 | |
| 11 | medium | 85.9% | 93.8% | 55/64 | $0.244 | 105k | 2026-10-22 | |
| 12 | high | 85.9% | 93.8% | 55/64 | ~$0.121 | 355k | 2026-10-24 | |
| 13 | xhigh | 85.9% | 92.2% | 55/64 | $0.062 | 130k | 2026-10-23 | |
| 14 | high | 85.9% | 92.2% | 55/64 | $0.051 | 121k | 2026-10-22 | |
| 15 | high | 84.4% | 98.4% | 54/64 | $0.344 | 267k | 2026-10-22 | |
| 16 | low | 84.4% | 96.9% | 54/64 | $0.030 | 91.6k | 2026-10-22 | |
| 17 | medium | 84.4% | 96.9% | 54/64 | $0.220 | 305k | 2026-10-23 | |
| 18 | max | 84.4% | 96.9% | 54/64 | $1.336 | 854k | 2026-10-23 | |
| 19 | xhigh | 84.4% | 93.8% | 54/64 | $0.007 | 213k | 2026-10-23 | |
| 20 | xhigh | 84.4% | 93.8% | 54/64 | $0.656 | 380k | 2026-10-22 | |
| 21 | xhigh | 84.4% | 90.6% | 54/64 | $0.356 | 127k | 2026-10-22 | |
| 22 | max | 82.8% | 95.3% | 53/64 | $0.224 | 805k | 2026-10-28 | |
| 23 | high | 82.8% | 93.8% | 53/64 | $0.310 | 120k | 2026-10-22 | |
| 24 | max | 81.3% | 96.9% | 52/64 | ~$0.087 | 191k | 2026-10-23 | |
| 25 | mistral-large-4-0Open | high | 81.3% | 95.3% | 52/64 | $0.052 | 286k | 2026-10-27 |
| 26 | max | 81.3% | 95.3% | 52/64 | ~$0.030 | 257k | 2026-10-23 | |
| 27 | high | 81.3% | 93.8% | 52/64 | $0.370 | 693k | 2026-10-24 | |
| 28 | max | 81.3% | 85.9% | 52/64 | ~$0.030 | 917k | 2026-10-23 | |
| 29 | xhigh | 79.7% | 96.9% | 51/64 | ~$0.119 | 365k | 2026-10-24 | |
| 30 | low | 79.7% | 93.8% | 51/64 | ~$0.013 | 332k | 2026-10-23 | |
| 31 | medium | 78.1% | 100.0% | 50/64 | $0.010 | 218k | 2026-10-20 | |
| 32 | mistral-large-4-0Open | low | 78.1% | 96.9% | 50/64 | $0.053 | 291k | 2026-10-27 |
| 33 | xhigh | 78.1% | 93.8% | 50/64 | ~$0.248 | 456k | 2026-10-24 | |
| 34 | high | 76.6% | 100.0% | 49/64 | $0.005 | 147k | 2026-10-23 | |
| 35 | high | 76.6% | 98.4% | 49/64 | ~$0.019 | 210k | 2026-10-23 | |
| 36 | high | 76.6% | 96.9% | 49/64 | ~$0.074 | 200k | 2026-10-22 | |
| 37 | high | 76.6% | 93.8% | 49/64 | $0.167 | 228k | 2026-10-24 | |
| 38 | medium | 75.0% | 96.9% | 48/64 | $0.004 | 135k | 2026-10-23 | |
| 39 | medium | 75.0% | 96.9% | 48/64 | $0.064 | 157k | 2026-10-23 | |
| 40 | xhigh | 75.0% | 92.2% | 48/64 | $0.192 | 264k | 2026-10-24 | |
| 41 | high | 75.0% | 84.4% | 48/64 | ~$0.015 | 356k | 2026-10-23 | |
| 42 | low | 73.4% | 98.4% | 47/64 | ~$0.142 | 297k | 2026-10-24 | |
| 43 | medium | 73.4% | 95.3% | 47/64 | ~$0.143 | 282k | 2026-10-24 | |
| 44 | low | 71.9% | 95.3% | 46/64 | $0.059 | 149k | 2026-10-23 | |
| 45 | medium | 70.3% | 96.9% | 45/64 | $0.135 | 190k | 2026-10-24 | |
| 46 | medium | 70.3% | 92.2% | 45/64 | ~$0.087 | 260k | 2026-10-24 | |
| 47 | low | 68.8% | 100.0% | 44/64 | $0.007 | 192k | 2026-10-27 | |
| 48 | low | 68.8% | 96.9% | 44/64 | $0.089 | 132k | 2026-10-24 | |
| 49 | low | 68.8% | 93.8% | 44/64 | ~$0.039 | 126k | 2026-10-22 | |
| 50 | off | 65.6% | 96.9% | 42/64 | ~$0.103 | 245k | 2026-10-24 | |
| 51 | low | 65.6% | 93.8% | 42/64 | ~$0.060 | 197k | 2026-10-24 | |
| 52 | low | 65.6% | 89.1% | 42/64 | $0.005 | 94.7k | 2026-10-20 | |
| 53 | low | 64.1% | 95.3% | 41/64 | $0.064 | 323k | 2026-10-27 | |
| 54 | low | 64.1% | 93.8% | 41/64 | ~$0.013 | 157k | 2026-10-23 | |
| 55 | high | 64.1% | 89.1% | 41/64 | ~$0.027 | 775k | 2026-10-27 | |
| 56 | off | 62.5% | 92.2% | 40/64 | $0.008 | 211k | 2026-10-27 | |
| 57 | high | 60.9% | 87.5% | 39/64 | $0.072 | 366k | 2026-10-27 | |
| 58 | low | 56.3% | 93.8% | 36/64 | $0.002 | 99.7k | 2026-10-23 | |
| 59 | low | 56.3% | 82.8% | 36/64 | $0.015 | 265k | 2026-10-27 | |
| 60 | low | 54.7% | 92.2% | 35/64 | $0.023 | 124k | 2026-10-20 | |
| 61 | off | 54.7% | 82.8% | 35/64 | ~$0.009 | 281k | 2026-10-23 | |
| 62 | high | 53.1% | 85.9% | 34/64 | $0.014 | 247k | 2026-10-27 | |
| 63 | low | 51.6% | 95.3% | 33/64 | ~$0.015 | 478k | 2026-10-27 | |
| 64 | high | 51.6% | 87.5% | 33/64 | $0.048 | 252k | 2026-10-27 | |
| 65 | off | 48.4% | 95.3% | 31/64 | $0.005 | 253k | 2026-10-23 | |
| 66 | high | 43.8% | 78.1% | 28/64 | $0.023 | 307k | 2026-10-27 | |
| 67 | low | 40.6% | 89.1% | 26/64 | $0.043 | 256k | 2026-10-27 | |
| 68 | low | 29.7% | 81.3% | 19/64 | $0.029 | 369k | 2026-10-27 | |
| 69 | max | 26.6% | 92.2% | 17/64 | $0.142 | 192k | 2026-10-24 | |
| 70 | off | 26.6% | 62.5% | 17/64 | $0.080 | 202k | 2026-10-20 | |
| 71 | off | 10.9% | 32.8% | 7/64 | $0.005 | 111k | 2026-10-20 |
~ Estimated from OpenRouter list prices: the run (Fireworks or self-hosted) reported no cost. — No list price available.
Pass^3 per task category. Green is 90%+, yellow 70%+, red below 70%.
Scroll sideways for all categories.
| # | Model | Dashboards | Grafana API | Investigation | Logs | Metrics | Traces |
|---|---|---|---|---|---|---|---|
| 1 | 71% | 100% | 91% | 100% | 94% | 100% | |
| 2 | 86% | 100% | 91% | 100% | 94% | 85% | |
| 3 | 86% | 100% | 100% | 100% | 94% | 69% | |
| 4 | 86% | 100% | 100% | 100% | 88% | 62% | |
| 5 | 71% | 83% | 100% | 100% | 81% | 92% | |
| 6 | 57% | 100% | 91% | 90% | 88% | 92% | |
| 7 | 57% | 100% | 100% | 100% | 94% | 77% | |
| 8 | 71% | 100% | 91% | 100% | 88% | 85% | |
| 9 | 86% | 83% | 82% | 100% | 100% | 62% | |
| 10 | 71% | 100% | 82% | 90% | 94% | 85% | |
| 11 | 57% | 100% | 91% | 90% | 88% | 85% | |
| 12 | 100% | 100% | 91% | 100% | 81% | 62% | |
| 13 | 57% | 100% | 100% | 100% | 88% | 77% | |
| 14 | 43% | 100% | 100% | 100% | 88% | 85% | |
| 15 | 71% | 100% | 91% | 90% | 81% | 85% | |
| 16 | 57% | 100% | 91% | 90% | 81% | 92% | |
| 17 | 57% | 83% | 91% | 100% | 81% | 92% | |
| 18 | 57% | 100% | 82% | 100% | 88% | 85% | |
| 19 | 71% | 100% | 82% | 90% | 88% | 85% | |
| 20 | 43% | 100% | 91% | 90% | 88% | 92% | |
| 21 | 43% | 100% | 91% | 100% | 88% | 85% | |
| 22 | 86% | 100% | 82% | 100% | 88% | 62% | |
| 23 | 43% | 100% | 100% | 100% | 81% | 77% | |
| 24 | 29% | 100% | 64% | 100% | 94% | 85% | |
| 25 | mistral-large-4-0highOpen | 14% | 100% | 100% | 80% | 94% | 85% |
| 26 | 29% | 100% | 91% | 100% | 81% | 85% | |
| 27 | 71% | 100% | 91% | 90% | 81% | 69% | |
| 28 | 71% | 83% | 100% | 90% | 75% | 77% | |
| 29 | 100% | 100% | 91% | 90% | 81% | 38% | |
| 30 | 43% | 100% | 91% | 100% | 75% | 77% | |
| 31 | 57% | 100% | 82% | 100% | 75% | 62% | |
| 32 | mistral-large-4-0lowOpen | 29% | 100% | 91% | 90% | 81% | 77% |
| 33 | 43% | 100% | 91% | 80% | 81% | 77% | |
| 34 | 57% | 100% | 82% | 80% | 81% | 62% | |
| 35 | 43% | 100% | 91% | 80% | 75% | 77% | |
| 36 | 71% | 100% | 55% | 90% | 69% | 92% | |
| 37 | 57% | 100% | 100% | 80% | 63% | 77% | |
| 38 | 71% | 100% | 64% | 90% | 63% | 77% | |
| 39 | 71% | 100% | 64% | 80% | 88% | 62% | |
| 40 | 57% | 100% | 91% | 90% | 63% | 69% | |
| 41 | 57% | 83% | 73% | 90% | 94% | 54% | |
| 42 | 14% | 100% | 82% | 100% | 75% | 69% | |
| 43 | 43% | 100% | 82% | 80% | 69% | 77% | |
| 44 | 71% | 100% | 55% | 80% | 81% | 62% | |
| 45 | 43% | 100% | 91% | 80% | 50% | 77% | |
| 46 | 71% | 100% | 82% | 80% | 81% | 31% | |
| 47 | 43% | 100% | 55% | 90% | 75% | 62% | |
| 48 | 43% | 100% | 82% | 70% | 63% | 69% | |
| 49 | 71% | 83% | 73% | 90% | 75% | 38% | |
| 50 | 14% | 100% | 64% | 60% | 75% | 77% | |
| 51 | 43% | 100% | 45% | 80% | 94% | 38% | |
| 52 | 43% | 100% | 64% | 70% | 81% | 46% | |
| 53 | 14% | 100% | 73% | 70% | 88% | 38% | |
| 54 | 29% | 100% | 73% | 80% | 69% | 46% | |
| 55 | 29% | 100% | 82% | 80% | 56% | 54% | |
| 56 | 57% | 100% | 64% | 60% | 69% | 46% | |
| 57 | 29% | 100% | 64% | 60% | 69% | 54% | |
| 58 | 43% | 100% | 45% | 30% | 81% | 46% | |
| 59 | 29% | 83% | 45% | 60% | 69% | 54% | |
| 60 | 29% | 100% | 45% | 40% | 75% | 46% | |
| 61 | 0% | 100% | 45% | 60% | 75% | 46% | |
| 62 | 43% | 100% | 18% | 70% | 69% | 31% | |
| 63 | 14% | 83% | 55% | 70% | 44% | 54% | |
| 64 | 14% | 100% | 45% | 60% | 50% | 54% | |
| 65 | 29% | 100% | 27% | 60% | 63% | 31% | |
| 66 | 14% | 100% | 45% | 40% | 38% | 46% | |
| 67 | 29% | 100% | 36% | 50% | 31% | 31% | |
| 68 | 0% | 100% | 27% | 30% | 25% | 15% | |
| 69 | 29% | 50% | 27% | 10% | 31% | 23% | |
| 70 | 0% | 83% | 9% | 10% | 50% | 15% | |
| 71 | 0% | 83% | 0% | 10% | 0% | 8% |
Open-weight models ran viaOpenRouterorFireworks; some also ran self-hosted.