⚡ August 16, 2026¶
Generated: 2026-08-16 05:24 UTC
Total Duration: 2h 50m 44s
Iterations: 1
Judge (classifier) model: gpt-4.1
Fast Benchmark
Markers: regression or benchmark
Schedule: Weekly (Sunday 2 AM UTC)
Purpose: Quick regression tests to catch breaking changes
HolmesGPT is continuously evaluated against real-world Kubernetes and cloud troubleshooting scenarios.
If you find scenarios that HolmesGPT does not perform well on, please consider adding them as evals to the benchmark.
Model Accuracy Comparison¶
| Model | Pass | Fail | Skip/Error | Total | Success Rate |
|---|---|---|---|---|---|
| deepseek-r1-reasoner | 14 | 7 | 0 | 21 | 🟡 67% (14/21) |
| deepseek-v3.2-chat | 13 | 8 | 0 | 21 | 🟡 62% (13/21) |
| gemini-3.1-pro-preview | 15 | 6 | 0 | 21 | 🟡 71% (15/21) |
| gpt-5.3-codex | 9 | 12 | 0 | 21 | 🟡 43% (9/21) |
| gpt-5.4 | 16 | 5 | 0 | 21 | 🟡 76% (16/21) |
| gpt-5.5 | 15 | 6 | 0 | 21 | 🟡 71% (15/21) |
| haiku-4.5 | 17 | 4 | 0 | 21 | 🟡 81% (17/21) |
| opus-4.6 | 17 | 4 | 0 | 21 | 🟡 81% (17/21) |
| opus-4.7 | 19 | 2 | 0 | 21 | 🟡 90% (19/21) |
| opus-4.8 | 17 | 4 | 0 | 21 | 🟡 81% (17/21) |
| qwen-next-80B-instruct | 12 | 9 | 0 | 21 | 🟡 57% (12/21) |
| qwen-next-80B-thinking | 0 | 21 | 0 | 21 | 🔴 0% (0/21) |
| sonnet-4.6 | 11 | 10 | 0 | 21 | 🟡 52% (11/21) |
Model Cost Comparison¶
| Model | Tests | Avg Cost | Min Cost | Max Cost | Total Cost |
|---|---|---|---|---|---|
| deepseek-r1-reasoner | 18 | $0.01 | $0.00 | $0.06 | $0.13 |
| deepseek-v3.2-chat | 18 | $0.00 | $0.00 | $0.01 | $0.06 |
| gemini-3.1-pro-preview | 19 | $0.13 | $0.02 | $0.57 | $2.55 |
| gpt-5.3-codex | 21 | $0.02 | $0.00 | $0.06 | $0.47 |
| gpt-5.4 | 21 | $0.05 | $0.01 | $0.14 | $1.13 |
| gpt-5.5 | 21 | $0.27 | $0.01 | $1.28 | $5.69 |
| haiku-4.5 | 21 | $0.04 | $0.02 | $0.10 | $0.80 |
| opus-4.6 | 21 | $0.23 | $0.08 | $0.52 | $4.84 |
| opus-4.7 | 21 | $0.25 | $0.10 | $1.01 | $5.29 |
| opus-4.8 | 21 | $0.32 | $0.10 | $1.70 | $6.68 |
| qwen-next-80B-instruct | 21 | $0.02 | $0.00 | $0.06 | $0.36 |
| sonnet-4.6 | 21 | $0.12 | $0.04 | $0.22 | $2.56 |
Model Latency Comparison¶
| Model | Avg (s) | Min (s) | Max (s) | P50 (s) | P95 (s) |
|---|---|---|---|---|---|
| deepseek-r1-reasoner | 86.7 | 1.1 | 869.7 | 36.6 | 188.6 |
| deepseek-v3.2-chat | 34.2 | 1.1 | 113.2 | 22.5 | 110.2 |
| gemini-3.1-pro-preview | 62.5 | 5.6 | 343.2 | 29.2 | 270.0 |
| gpt-5.3-codex | 9.6 | 3.7 | 15.3 | 9.6 | 13.9 |
| gpt-5.4 | 14.2 | 3.3 | 24.8 | 14.9 | 24.1 |
| gpt-5.5 | 52.6 | 5.1 | 245.9 | 38.9 | 116.7 |
| haiku-4.5 | 21.4 | 4.4 | 64.9 | 17.5 | 40.3 |
| opus-4.6 | 45.5 | 5.5 | 109.1 | 30.5 | 93.8 |
| opus-4.7 | 29.0 | 4.5 | 154.4 | 21.4 | 41.7 |
| opus-4.8 | 61.9 | 6.4 | 456.4 | 35.2 | 126.7 |
| qwen-next-80B-instruct | 21.7 | 5.4 | 53.9 | 17.4 | 42.6 |
| sonnet-4.6 | 35.5 | 4.0 | 78.6 | 32.9 | 69.2 |
⚠️ Note: 21 test(s) excluded from latency calculations due to throttling/timeout errors (qwen-next-80B-thinking: 21)
Performance by Tag¶
Success rate by test category and model:
| Tag | deepseek-r1-reasoner | deepseek-v3.2-chat | gemini-3.1-pro-preview | gpt-5.3-codex | gpt-5.4 | gpt-5.5 | haiku-4.5 | opus-4.6 | opus-4.7 | opus-4.8 | qwen-next-80B-instruct | qwen-next-80B-thinking | sonnet-4.6 | Warnings |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| benchmark | 🟡 67% (4/6) | 🟡 50% (3/6) | 🟡 50% (3/6) | 🟡 33% (2/6) | 🟡 67% (4/6) | 🟡 50% (3/6) | 🟡 83% (⅚) | 🟡 67% (4/6) | 🟡 67% (4/6) | 🟡 67% (4/6) | 🟡 17% (⅙) | 🔴 0% (0/6) | 🟡 67% (4/6) | |
| context_window | 🟢 100% (2/2) | 🟡 50% (½) | 🟢 100% (2/2) | 🟡 50% (½) | 🟢 100% (2/2) | 🟡 50% (½) | 🟢 100% (2/2) | 🟢 100% (2/2) | 🟢 100% (2/2) | 🟢 100% (2/2) | 🔴 0% (0/2) | 🔴 0% (0/2) | 🟢 100% (2/2) | |
| counting | 🟡 50% (½) | 🟢 100% (2/2) | 🔴 0% (0/2) | 🔴 0% (0/2) | 🔴 0% (0/2) | 🟡 50% (½) | 🔴 0% (0/2) | 🟡 50% (½) | 🟢 100% (2/2) | 🟡 50% (½) | 🔴 0% (0/2) | 🔴 0% (0/2) | 🔴 0% (0/2) | |
| datetime | 🟢 100% (3/3) | 🟡 67% (⅔) | 🟢 100% (3/3) | 🟡 67% (⅔) | 🟢 100% (3/3) | 🟡 67% (⅔) | 🟢 100% (3/3) | 🟢 100% (3/3) | 🟢 100% (3/3) | 🟢 100% (3/3) | 🟡 33% (⅓) | 🔴 0% (0/3) | 🟢 100% (3/3) | |
| easy | 🟡 75% (6/8) | 🟡 88% (⅞) | 🟡 88% (⅞) | 🟡 38% (⅜) | 🟡 75% (6/8) | 🟡 75% (6/8) | 🟡 88% (⅞) | 🟡 75% (6/8) | 🟢 100% (8/8) | 🟡 75% (6/8) | 🟡 75% (6/8) | 🔴 0% (0/8) | 🟡 25% (2/8) | |
| elasticsearch | 🔴 0% (0/3) | 🔴 0% (0/3) | 🟡 67% (⅔) | 🟡 67% (⅔) | 🟢 100% (3/3) | 🟡 67% (⅔) | 🟡 67% (⅔) | 🟢 100% (3/3) | 🟢 100% (3/3) | 🟢 100% (3/3) | 🟡 67% (⅔) | 🔴 0% (0/3) | 🟢 100% (3/3) | |
| grafana | 🟢 100% (1/1) | 🟢 100% (1/1) | 🟢 100% (1/1) | 🔴 0% (0/1) | 🟢 100% (1/1) | 🟢 100% (1/1) | 🟢 100% (1/1) | 🟢 100% (1/1) | 🟢 100% (1/1) | 🟢 100% (1/1) | 🟢 100% (1/1) | 🔴 0% (0/1) | 🟢 100% (1/1) | |
| hard | 🟡 50% (½) | 🟡 50% (½) | 🟡 50% (½) | 🔴 0% (0/2) | 🟡 50% (½) | 🟡 50% (½) | 🟢 100% (2/2) | 🟡 50% (½) | 🟡 50% (½) | 🟡 50% (½) | 🟡 50% (½) | 🔴 0% (0/2) | 🟡 50% (½) | |
| kubernetes | 🟡 80% (8/10) | 🟡 70% (7/10) | 🟡 80% (8/10) | 🟡 20% (2/10) | 🟡 70% (7/10) | 🟡 80% (8/10) | 🟡 80% (8/10) | 🟡 80% (8/10) | 🟡 90% (9/10) | 🟡 80% (8/10) | 🟡 70% (7/10) | 🔴 0% (0/10) | 🟡 40% (4/10) | |
| logs | 🟡 67% (4/6) | 🟡 33% (2/6) | 🟡 67% (4/6) | 🟡 33% (2/6) | 🟡 67% (4/6) | 🟡 50% (3/6) | 🟡 83% (⅚) | 🟡 67% (4/6) | 🟡 67% (4/6) | 🟡 67% (4/6) | 🟡 33% (2/6) | 🔴 0% (0/6) | 🟡 50% (3/6) | |
| loki | 🟡 50% (½) | 🔴 0% (0/2) | 🟡 50% (½) | 🔴 0% (0/2) | 🟡 50% (½) | 🟡 50% (½) | 🟡 50% (½) | 🟡 50% (½) | 🟡 50% (½) | 🟡 50% (½) | 🟡 50% (½) | 🔴 0% (0/2) | 🟡 50% (½) | |
| medium | 🟡 56% (5/9) | 🟡 44% (4/9) | 🟡 67% (6/9) | 🟡 56% (5/9) | 🟡 89% (8/9) | 🟡 67% (6/9) | 🟡 78% (7/9) | 🟡 89% (8/9) | 🟡 89% (8/9) | 🟡 89% (8/9) | 🟡 44% (4/9) | 🔴 0% (0/9) | 🟡 78% (7/9) | |
| metrics | 🟢 100% (1/1) | 🟢 100% (1/1) | 🟢 100% (1/1) | 🔴 0% (0/1) | 🟢 100% (1/1) | 🟢 100% (1/1) | 🟢 100% (1/1) | 🟢 100% (1/1) | 🟢 100% (1/1) | 🟢 100% (1/1) | 🟢 100% (1/1) | 🔴 0% (0/1) | 🟢 100% (1/1) | |
| multi-cluster | 🔴 0% (0/3) | 🔴 0% (0/3) | 🟡 67% (⅔) | 🟡 67% (⅔) | 🟢 100% (3/3) | 🟡 67% (⅔) | 🟡 67% (⅔) | 🟢 100% (3/3) | 🟢 100% (3/3) | 🟢 100% (3/3) | 🟡 67% (⅔) | 🔴 0% (0/3) | 🟢 100% (3/3) | |
| network | 🔴 0% (0/1) | 🟢 100% (1/1) | 🟢 100% (1/1) | 🔴 0% (0/1) | 🟢 100% (1/1) | 🔴 0% (0/1) | 🟢 100% (1/1) | 🔴 0% (0/1) | 🟢 100% (1/1) | 🔴 0% (0/1) | 🔴 0% (0/1) | 🔴 0% (0/1) | 🔴 0% (0/1) | |
| one-test | 🟢 100% (1/1) | 🟢 100% (1/1) | 🟢 100% (1/1) | 🔴 0% (0/1) | 🟢 100% (1/1) | 🟢 100% (1/1) | 🟢 100% (1/1) | 🟢 100% (1/1) | 🟢 100% (1/1) | 🟢 100% (1/1) | 🟢 100% (1/1) | 🔴 0% (0/1) | 🔴 0% (0/1) | |
| port-forward | 🟡 67% (⅔) | 🟡 33% (⅓) | 🟡 67% (⅔) | 🔴 0% (0/3) | 🟡 67% (⅔) | 🟡 67% (⅔) | 🟡 67% (⅔) | 🟡 67% (⅔) | 🟡 67% (⅔) | 🟡 67% (⅔) | 🟡 67% (⅔) | 🔴 0% (0/3) | 🟡 67% (⅔) | |
| question-answer | 🟢 100% (2/2) | 🟡 50% (½) | 🟢 100% (2/2) | 🟡 50% (½) | 🟢 100% (2/2) | 🟢 100% (2/2) | 🟢 100% (2/2) | 🟢 100% (2/2) | 🟢 100% (2/2) | 🟢 100% (2/2) | 🟢 100% (2/2) | 🔴 0% (0/2) | 🟢 100% (2/2) | |
| regression | 🟡 67% (10/15) | 🟡 67% (10/15) | 🟡 80% (12/15) | 🟡 47% (7/15) | 🟡 80% (12/15) | 🟡 80% (12/15) | 🟡 80% (12/15) | 🟡 87% (13/15) | 🟢 100% (15/15) | 🟡 87% (13/15) | 🟡 73% (11/15) | 🔴 0% (0/15) | 🟡 47% (7/15) | |
| skills | 🟡 83% (10/12) | 🟡 92% (11/12) | 🟡 75% (9/12) | 🟡 42% (5/12) | 🟡 75% (9/12) | 🟡 83% (10/12) | 🟡 83% (10/12) | 🟡 83% (10/12) | 🟢 100% (12/12) | 🟡 83% (10/12) | 🟡 67% (8/12) | 🔴 0% (0/12) | 🟡 33% (4/12) | |
| transparency | 🔴 0% (0/3) | 🔴 0% (0/3) | 🟡 67% (⅔) | 🟡 67% (⅔) | 🟢 100% (3/3) | 🟡 67% (⅔) | 🟡 67% (⅔) | 🟢 100% (3/3) | 🟢 100% (3/3) | 🟢 100% (3/3) | 🟡 67% (⅔) | 🔴 0% (0/3) | 🟢 100% (3/3) | |
| Overall | 🟡 67% (14/21) | 🟡 62% (13/21) | 🟡 71% (15/21) | 🟡 43% (9/21) | 🟡 76% (16/21) | 🟡 71% (15/21) | 🟡 81% (17/21) | 🟡 81% (17/21) | 🟡 90% (19/21) | 🟡 81% (17/21) | 🟡 57% (12/21) | 🔴 0% (0/21) | 🟡 52% (11/21) |
Raw Results¶
Status of all evaluations across models. Color coding:
- 🟢 Passing 100% (stable)
- 🟡 Passing 1-99%
- 🔴 Passing 0% (failing)
- 🔧 Mock data failure (missing or invalid test data)
- ⚠️ Setup failure (environment/infrastructure issue)
- ⏱️ Timeout or rate limit error
- ⏭️ Test skipped (e.g., known issue or precondition not met)
Detailed Raw Results¶
| Eval ID | deepseek-r1-reasoner | deepseek-v3.2-chat | gemini-3.1-pro-preview | gpt-5.3-codex | gpt-5.4 | gpt-5.5 | haiku-4.5 | opus-4.6 | opus-4.7 | opus-4.8 | qwen-next-80B-instruct | qwen-next-80B-thinking | sonnet-4.6 |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| 09_crashpod 🔗 | 🟢 100% (1/1) / ⏱️ 34.4s / 💰 $0.00 | 🟢 100% (1/1) / ⏱️ 23.2s / 💰 $0.00 | 🟢 100% (1/1) / ⏱️ 27.5s / 💰 $0.08 | 🔴 0% (0/1) / ⏱️ 4.0s / 💰 $0.00 | 🟢 100% (1/1) / ⏱️ 9.7s / 💰 $0.03 | 🟢 100% (1/1) / ⏱️ 22.5s / 💰 $0.09 | 🟢 100% (1/1) / ⏱️ 16.3s / 💰 $0.03 | 🟢 100% (1/1) / ⏱️ 26.8s / 💰 $0.17 | 🟢 100% (1/1) / ⏱️ 16.1s / 💰 $0.17 | 🟢 100% (1/1) / ⏱️ 22.0s / 💰 $0.17 | 🟢 100% (1/1) / ⏱️ 18.3s / 💰 $0.01 | ⏱️ 0% (0/1) / ⏱️ 1269.9s | 🔴 0% (0/1) / ⏱️ 23.2s / 💰 $0.10 |
| 100a_loki_historical_logs 🔗 | 🔴 0% (0/1) / ⏱️ 869.7s / 💰 $0.06 | 🔴 0% (0/1) / ⏱️ 113.2s / 💰 $0.01 | 🔴 0% (0/1) / ⏱️ 138.7s / 💰 $0.57 | 🔴 0% (0/1) / ⏱️ 9.2s / 💰 $0.01 | 🔴 0% (0/1) / ⏱️ 19.5s / 💰 $0.05 | 🔴 0% (0/1) / ⏱️ 76.4s / 💰 $0.26 | 🔴 0% (0/1) / ⏱️ 30.1s / 💰 $0.04 | 🔴 0% (0/1) / ⏱️ 109.1s / 💰 $0.52 | 🔴 0% (0/1) / ⏱️ 154.4s / 💰 $1.01 | 🔴 0% (0/1) / ⏱️ 126.7s / 💰 $0.58 | 🔴 0% (0/1) / ⏱️ 42.6s / 💰 $0.04 | ⏱️ 0% (0/1) / ⏱️ 1270.7s | 🔴 0% (0/1) / ⏱️ 78.6s / 💰 $0.22 |
| 101_loki_historical_logs_pod_deleted 🔗 | 🟢 100% (1/1) / ⏱️ 68.7s / 💰 $0.01 | 🔴 0% (0/1) / ⏱️ 110.2s / 💰 $0.01 | 🟢 100% (1/1) / ⏱️ 62.8s / 💰 $0.21 | 🔴 0% (0/1) / ⏱️ 9.4s / 💰 $0.02 | 🟢 100% (1/1) / ⏱️ 23.3s / 💰 $0.09 | 🟢 100% (1/1) / ⏱️ 58.3s / 💰 $0.32 | 🟢 100% (1/1) / ⏱️ 20.0s / 💰 $0.04 | 🟢 100% (1/1) / ⏱️ 59.1s / 💰 $0.27 | 🟢 100% (1/1) / ⏱️ 19.3s / 💰 $0.18 | 🟢 100% (1/1) / ⏱️ 75.5s / 💰 $0.36 | 🟢 100% (1/1) / ⏱️ 22.2s / 💰 $0.02 | ⏱️ 0% (0/1) / ⏱️ 1271.9s | 🟢 100% (1/1) / ⏱️ 36.0s / 💰 $0.12 |
| 108_logs_nearby_lines 🔗 | 🔴 0% (0/1) / ⏱️ 93.2s / 💰 $0.01 | 🔴 0% (0/1) / ⏱️ 83.2s / 💰 $0.01 | 🔴 0% (0/1) / ⏱️ 52.6s / 💰 $0.23 | 🔴 0% (0/1) / ⏱️ 15.3s / 💰 $0.03 | 🔴 0% (0/1) / ⏱️ 21.3s / 💰 $0.08 | 🔴 0% (0/1) / ⏱️ 62.2s / 💰 $0.23 | 🟢 100% (1/1) / ⏱️ 25.6s / 💰 $0.05 | 🔴 0% (0/1) / ⏱️ 68.2s / 💰 $0.34 | 🔴 0% (0/1) / ⏱️ 35.2s / 💰 $0.30 | 🔴 0% (0/1) / ⏱️ 92.2s / 💰 $0.47 | 🔴 0% (0/1) / ⏱️ 53.9s / 💰 $0.06 | ⏱️ 0% (0/1) / ⏱️ 1271.0s | 🔴 0% (0/1) / ⏱️ 41.7s / 💰 $0.17 |
| 112_find_pvcs_by_uuid 🔗 | 🟢 100% (1/1) / ⏱️ 36.6s / 💰 $0.00 | 🟢 100% (1/1) / ⏱️ 8.1s / 💰 $0.00 | 🟢 100% (1/1) / ⏱️ 14.2s / 💰 $0.04 | 🔴 0% (0/1) / ⏱️ 8.4s / 💰 $0.01 | 🟢 100% (1/1) / ⏱️ 10.4s / 💰 $0.03 | 🟢 100% (1/1) / ⏱️ 18.9s / 💰 $0.07 | 🟢 100% (1/1) / ⏱️ 10.0s / 💰 $0.02 | 🟢 100% (1/1) / ⏱️ 25.1s / 💰 $0.15 | 🟢 100% (1/1) / ⏱️ 22.3s / 💰 $0.22 | 🟢 100% (1/1) / ⏱️ 19.6s / 💰 $0.15 | 🟢 100% (1/1) / ⏱️ 9.0s / 💰 $0.01 | ⏱️ 0% (0/1) / ⏱️ 1270.6s | 🟢 100% (1/1) / ⏱️ 28.2s / 💰 $0.09 |
| 12_job_crashing 🔗 | 🟢 100% (1/1) / ⏱️ 38.5s / 💰 $0.00 | 🟢 100% (1/1) / ⏱️ 31.5s / 💰 $0.00 | 🟢 100% (1/1) / ⏱️ 14.2s / 💰 $0.05 | 🟢 100% (1/1) / ⏱️ 12.5s / 💰 $0.02 | 🟢 100% (1/1) / ⏱️ 11.4s / 💰 $0.04 | 🟢 100% (1/1) / ⏱️ 21.6s / 💰 $0.11 | 🟢 100% (1/1) / ⏱️ 19.1s / 💰 $0.04 | 🟢 100% (1/1) / ⏱️ 26.8s / 💰 $0.17 | 🟢 100% (1/1) / ⏱️ 15.2s / 💰 $0.16 | 🟢 100% (1/1) / ⏱️ 37.6s / 💰 $0.19 | 🟢 100% (1/1) / ⏱️ 16.8s / 💰 $0.01 | ⏱️ 0% (0/1) / ⏱️ 1270.2s | 🔴 0% (0/1) / ⏱️ 24.7s / 💰 $0.09 |
| 176_network_policy_blocking_traffic_no_skills 🔗 | 🔴 0% (0/1) / ⏱️ 123.5s / 💰 $0.01 | 🟢 100% (1/1) / ⏱️ 38.3s / 💰 $0.00 | 🟢 100% (1/1) / ⏱️ 38.9s / 💰 $0.10 | 🔴 0% (0/1) / ⏱️ 11.3s / 💰 $0.03 | 🟢 100% (1/1) / ⏱️ 18.9s / 💰 $0.07 | 🔴 0% (0/1) / ⏱️ 102.5s / 💰 $0.57 | 🟢 100% (1/1) / ⏱️ 20.3s / 💰 $0.04 | 🔴 0% (0/1) / ⏱️ 55.9s / 💰 $0.28 | 🟢 100% (1/1) / ⏱️ 23.4s / 💰 $0.26 | 🔴 0% (0/1) / ⏱️ 456.4s / 💰 $1.70 | 🔴 0% (0/1) / ⏱️ 21.4s / 💰 $0.02 | ⏱️ 0% (0/1) / ⏱️ 1270.1s | 🔴 0% (0/1) / ⏱️ 35.0s / 💰 $0.12 |
| 179_grafana_big_dashboard_query 🔗 | 🟢 100% (1/1) / ⏱️ 12.4s / 💰 $0.00 | 🟢 100% (1/1) / ⏱️ 8.2s / 💰 $0.00 | 🟢 100% (1/1) / ⏱️ 15.1s / 💰 $0.07 | 🔴 0% (0/1) / ⏱️ 13.9s / 💰 $0.06 | 🟢 100% (1/1) / ⏱️ 5.9s / 💰 $0.04 | 🟢 100% (1/1) / ⏱️ 10.5s / 💰 $0.09 | 🟢 100% (1/1) / ⏱️ 13.2s / 💰 $0.04 | 🟢 100% (1/1) / ⏱️ 14.9s / 💰 $0.15 | 🟢 100% (1/1) / ⏱️ 13.2s / 💰 $0.20 | 🟢 100% (1/1) / ⏱️ 14.4s / 💰 $0.19 | 🟢 100% (1/1) / ⏱️ 13.2s / 💰 $0.01 | ⏱️ 0% (0/1) / ⏱️ 1271.5s | 🟢 100% (1/1) / ⏱️ 18.4s / 💰 $0.10 |
| 227_count_configmaps_per_namespace[0] 🔗 | 🟢 100% (1/1) / ⏱️ 13.2s / 💰 $0.00 | 🟢 100% (1/1) / ⏱️ 13.5s / 💰 $0.00 | 🔴 0% (0/1) / ⏱️ 270.0s | 🔴 0% (0/1) / ⏱️ 6.4s / 💰 $0.03 | 🔴 0% (0/1) / ⏱️ 7.2s / 💰 $0.05 | 🟢 100% (1/1) / ⏱️ 23.9s / 💰 $0.23 | 🔴 0% (0/1) / ⏱️ 11.4s / 💰 $0.03 | 🟢 100% (1/1) / ⏱️ 30.5s / 💰 $0.17 | 🟢 100% (1/1) / ⏱️ 20.3s / 💰 $0.19 | 🟢 100% (1/1) / ⏱️ 24.2s / 💰 $0.20 | 🔴 0% (0/1) / ⏱️ 16.2s / 💰 $0.01 | ⏱️ 0% (0/1) / ⏱️ 1270.7s | 🔴 0% (0/1) / ⏱️ 20.3s / 💰 $0.10 |
| 243_pod_names_contain_service 🔗 | 🟢 100% (1/1) / ⏱️ 41.1s / 💰 $0.00 | 🟢 100% (1/1) / ⏱️ 22.5s / 💰 $0.00 | 🟢 100% (1/1) / ⏱️ 19.8s / 💰 $0.09 | 🟢 100% (1/1) / ⏱️ 9.6s / 💰 $0.03 | 🟢 100% (1/1) / ⏱️ 14.9s / 💰 $0.06 | 🟢 100% (1/1) / ⏱️ 20.5s / 💰 $0.12 | 🟢 100% (1/1) / ⏱️ 21.5s / 💰 $0.03 | 🟢 100% (1/1) / ⏱️ 30.4s / 💰 $0.15 | 🟢 100% (1/1) / ⏱️ 21.4s / 💰 $0.16 | 🟢 100% (1/1) / ⏱️ 23.0s / 💰 $0.17 | 🟢 100% (1/1) / ⏱️ 17.4s / 💰 $0.01 | ⏱️ 0% (0/1) / ⏱️ 1271.0s | 🔴 0% (0/1) / ⏱️ 31.4s / 💰 $0.09 |
| 24_misconfigured_pvc 🔗 | 🟢 100% (1/1) / ⏱️ 37.1s / 💰 $0.00 | 🟢 100% (1/1) / ⏱️ 23.0s / 💰 $0.00 | 🟢 100% (1/1) / ⏱️ 29.2s / 💰 $0.10 | 🔴 0% (0/1) / ⏱️ 3.7s / 💰 $0.00 | 🔴 0% (0/1) / ⏱️ 15.7s / 💰 $0.05 | 🟢 100% (1/1) / ⏱️ 38.9s / 💰 $0.18 | 🟢 100% (1/1) / ⏱️ 14.1s / 💰 $0.03 | 🟢 100% (1/1) / ⏱️ 28.6s / 💰 $0.17 | 🟢 100% (1/1) / ⏱️ 17.8s / 💰 $0.17 | 🟢 100% (1/1) / ⏱️ 29.5s / 💰 $0.20 | 🟢 100% (1/1) / ⏱️ 12.8s / 💰 $0.01 | ⏱️ 0% (0/1) / ⏱️ 1270.8s | 🔴 0% (0/1) / ⏱️ 31.1s / 💰 $0.10 |
| 254_elasticsearch_dr_test_log_check 🔗 | 🔴 0% (0/1) / ⏱️ 1.1s | 🔴 0% (0/1) / ⏱️ 1.4s | 🟢 100% (1/1) / ⏱️ 46.7s / 💰 $0.15 | 🔴 0% (0/1) / ⏱️ 11.1s / 💰 $0.02 | 🟢 100% (1/1) / ⏱️ 16.2s / 💰 $0.07 | 🔴 0% (0/1) / ⏱️ 245.9s / 💰 $1.28 | 🔴 0% (0/1) / ⏱️ 64.9s / 💰 $0.10 | 🟢 100% (1/1) / ⏱️ 89.8s / 💰 $0.41 | 🟢 100% (1/1) / ⏱️ 30.1s / 💰 $0.21 | 🟢 100% (1/1) / ⏱️ 54.2s / 💰 $0.24 | 🔴 0% (0/1) / ⏱️ 16.8s / 💰 $0.01 | ⏱️ 0% (0/1) / ⏱️ 1271.1s | 🟢 100% (1/1) / ⏱️ 69.2s / 💰 $0.22 |
| 259_wrong_cluster_logs_confusion 🔗 | 🔴 0% (0/1) / ⏱️ 1.2s | 🔴 0% (0/1) / ⏱️ 1.1s | 🔴 0% (0/1) / ⏱️ 74.0s / 💰 $0.28 | 🟢 100% (1/1) / ⏱️ 9.3s / 💰 $0.02 | 🟢 100% (1/1) / ⏱️ 18.0s / 💰 $0.07 | 🟢 100% (1/1) / ⏱️ 87.4s / 💰 $0.43 | 🟢 100% (1/1) / ⏱️ 28.4s / 💰 $0.04 | 🟢 100% (1/1) / ⏱️ 69.9s / 💰 $0.30 | 🟢 100% (1/1) / ⏱️ 33.7s / 💰 $0.22 | 🟢 100% (1/1) / ⏱️ 43.2s / 💰 $0.22 | 🟢 100% (1/1) / ⏱️ 37.7s / 💰 $0.03 | ⏱️ 0% (0/1) / ⏱️ 1270.8s | 🟢 100% (1/1) / ⏱️ 43.4s / 💰 $0.12 |
| 260_global_es_remote_cluster_logs 🔗 | 🔴 0% (0/1) / ⏱️ 1.3s | 🔴 0% (0/1) / ⏱️ 1.1s | 🟢 100% (1/1) / ⏱️ 38.2s / 💰 $0.15 | 🟢 100% (1/1) / ⏱️ 10.0s / 💰 $0.01 | 🟢 100% (1/1) / ⏱️ 24.1s / 💰 $0.07 | 🟢 100% (1/1) / ⏱️ 116.7s / 💰 $0.60 | 🟢 100% (1/1) / ⏱️ 40.3s / 💰 $0.06 | 🟢 100% (1/1) / ⏱️ 75.4s / 💰 $0.31 | 🟢 100% (1/1) / ⏱️ 36.1s / 💰 $0.22 | 🟢 100% (1/1) / ⏱️ 59.0s / 💰 $0.27 | 🟢 100% (1/1) / ⏱️ 17.3s / 💰 $0.01 | ⏱️ 0% (0/1) / ⏱️ 1270.8s | 🟢 100% (1/1) / ⏱️ 56.9s / 💰 $0.15 |
| 283_todowrite_multistep_audit 🔗 | 🟢 100% (1/1) / ⏱️ 21.0s / 💰 $0.00 | 🔴 0% (0/1) / ⏱️ 9.0s / 💰 $0.00 | 🟢 100% (1/1) / ⏱️ 20.9s / 💰 $0.07 | 🟢 100% (1/1) / ⏱️ 13.4s / 💰 $0.05 | 🟢 100% (1/1) / ⏱️ 11.8s / 💰 $0.05 | 🟢 100% (1/1) / ⏱️ 23.0s / 💰 $0.14 | 🟢 100% (1/1) / ⏱️ 17.5s / 💰 $0.03 | 🟢 100% (1/1) / ⏱️ 23.8s / 💰 $0.16 | 🟢 100% (1/1) / ⏱️ 31.2s / 💰 $0.26 | 🟢 100% (1/1) / ⏱️ 27.6s / 💰 $0.19 | 🟢 100% (1/1) / ⏱️ 22.3s / 💰 $0.01 | ⏱️ 0% (0/1) / ⏱️ 1271.7s | 🟢 100% (1/1) / ⏱️ 32.9s / 💰 $0.13 |
| 43_current_datetime_from_prompt 🔗 | 🟢 100% (1/1) / ⏱️ 4.0s / 💰 $0.00 | 🟢 100% (1/1) / ⏱️ 3.4s / 💰 $0.00 | 🟢 100% (1/1) / ⏱️ 5.6s / 💰 $0.02 | 🟢 100% (1/1) / ⏱️ 4.0s / 💰 $0.00 | 🟢 100% (1/1) / ⏱️ 3.3s / 💰 $0.01 | 🟢 100% (1/1) / ⏱️ 5.1s / 💰 $0.01 | 🟢 100% (1/1) / ⏱️ 4.4s / 💰 $0.02 | 🟢 100% (1/1) / ⏱️ 5.5s / 💰 $0.08 | 🟢 100% (1/1) / ⏱️ 4.5s / 💰 $0.10 | 🟢 100% (1/1) / ⏱️ 6.4s / 💰 $0.10 | 🟢 100% (1/1) / ⏱️ 7.3s / 💰 $0.00 | ⏱️ 0% (0/1) / ⏱️ 1270.9s | 🟢 100% (1/1) / ⏱️ 4.0s / 💰 $0.04 |
| 51_logs_summarize_errors 🔗 | 🟢 100% (1/1) / ⏱️ 32.6s / 💰 $0.00 | 🟢 100% (1/1) / ⏱️ 10.2s / 💰 $0.00 | 🟢 100% (1/1) / ⏱️ 14.5s / 💰 $0.05 | 🟢 100% (1/1) / ⏱️ 9.6s / 💰 $0.02 | 🟢 100% (1/1) / ⏱️ 8.8s / 💰 $0.03 | 🟢 100% (1/1) / ⏱️ 14.1s / 💰 $0.06 | 🟢 100% (1/1) / ⏱️ 16.2s / 💰 $0.03 | 🟢 100% (1/1) / ⏱️ 20.6s / 💰 $0.13 | 🟢 100% (1/1) / ⏱️ 15.7s / 💰 $0.18 | 🟢 100% (1/1) / ⏱️ 25.6s / 💰 $0.19 | 🟢 100% (1/1) / ⏱️ 11.8s / 💰 $0.01 | ⏱️ 0% (0/1) / ⏱️ 1270.5s | 🔴 0% (0/1) / ⏱️ 21.6s / 💰 $0.08 |
| 61_exact_match_counting 🔗 | 🔴 0% (0/1) / ⏱️ 10.1s / 💰 $0.00 | 🟢 100% (1/1) / ⏱️ 8.7s / 💰 $0.00 | 🔴 0% (0/1) / ⏱️ 9.6s / 💰 $0.03 | 🔴 0% (0/1) / ⏱️ 5.3s / 💰 $0.01 | 🔴 0% (0/1) / ⏱️ 3.7s / 💰 $0.01 | 🔴 0% (0/1) / ⏱️ 5.4s / 💰 $0.02 | 🔴 0% (0/1) / ⏱️ 12.4s / 💰 $0.02 | 🔴 0% (0/1) / ⏱️ 8.4s / 💰 $0.09 | 🟢 100% (1/1) / ⏱️ 11.4s / 💰 $0.25 | 🔴 0% (0/1) / ⏱️ 10.1s / 💰 $0.21 | 🔴 0% (0/1) / ⏱️ 5.4s / 💰 $0.00 | ⏱️ 0% (0/1) / ⏱️ 1270.7s | 🔴 0% (0/1) / ⏱️ 6.0s / 💰 $0.05 |
| 73a_time_window_anomaly 🔗 | 🟢 100% (1/1) / ⏱️ 96.2s / 💰 $0.01 | 🔴 0% (0/1) / ⏱️ 104.1s / 💰 $0.01 | 🟢 100% (1/1) / ⏱️ 58.1s / 💰 $0.21 | 🟢 100% (1/1) / ⏱️ 12.6s / 💰 $0.02 | 🟢 100% (1/1) / ⏱️ 16.5s / 💰 $0.06 | 🔴 0% (0/1) / ⏱️ 38.9s / 💰 $0.26 | 🟢 100% (1/1) / ⏱️ 16.8s / 💰 $0.03 | 🟢 100% (1/1) / ⏱️ 35.0s / 💰 $0.19 | 🟢 100% (1/1) / ⏱️ 20.9s / 💰 $0.21 | 🟢 100% (1/1) / ⏱️ 35.2s / 💰 $0.24 | 🔴 0% (0/1) / ⏱️ 30.0s / 💰 $0.01 | ⏱️ 0% (0/1) / ⏱️ 1270.9s | 🟢 100% (1/1) / ⏱️ 47.6s / 💰 $0.12 |
| 73b_time_window_anomaly 🔗 | 🟢 100% (1/1) / ⏱️ 95.6s / 💰 $0.01 | 🟢 100% (1/1) / ⏱️ 50.4s / 💰 $0.00 | 🟢 100% (1/1) / ⏱️ 17.9s / 💰 $0.07 | 🔴 0% (0/1) / ⏱️ 8.5s / 💰 $0.02 | 🟢 100% (1/1) / ⏱️ 12.9s / 💰 $0.04 | 🟢 100% (1/1) / ⏱️ 41.5s / 💰 $0.20 | 🟢 100% (1/1) / ⏱️ 16.0s / 💰 $0.03 | 🟢 100% (1/1) / ⏱️ 57.3s / 💰 $0.32 | 🟢 100% (1/1) / ⏱️ 26.1s / 💰 $0.24 | 🟢 100% (1/1) / ⏱️ 38.5s / 💰 $0.24 | 🔴 0% (0/1) / ⏱️ 23.9s / 💰 $0.03 | ⏱️ 0% (0/1) / ⏱️ 1270.6s | 🟢 100% (1/1) / ⏱️ 43.2s / 💰 $0.13 |
| 96_no_matching_skill 🔗 | 🟢 100% (1/1) / ⏱️ 188.6s / 💰 $0.01 | 🟢 100% (1/1) / ⏱️ 53.1s / 💰 $0.01 | 🔴 0% (0/1) / ⏱️ 343.2s | 🟢 100% (1/1) / ⏱️ 13.8s / 💰 $0.05 | 🟢 100% (1/1) / ⏱️ 24.8s / 💰 $0.14 | 🟢 100% (1/1) / ⏱️ 69.8s / 💰 $0.43 | 🟢 100% (1/1) / ⏱️ 30.9s / 💰 $0.06 | 🟢 100% (1/1) / ⏱️ 93.8s / 💰 $0.32 | 🟢 100% (1/1) / ⏱️ 41.7s / 💰 $0.36 | 🟢 100% (1/1) / ⏱️ 79.1s / 💰 $0.42 | 🔴 0% (0/1) / ⏱️ 39.0s / 💰 $0.04 | ⏱️ 0% (0/1) / ⏱️ 1272.0s | 🟢 100% (1/1) / ⏱️ 52.8s / 💰 $0.19 |
Results are automatically generated and updated weekly. View full traces and detailed analysis in Braintrust experiment: ci-benchmark-31922012024.