Contributing

There are two contribution paths: proposing benchmark tasks and submitting benchmark results.

Tasks

Task specs live in the grafana/o11y-bench repository under tasks-spec/. Generated task output is derived from those specs and should not be edited directly.

Open Task Specs
  1. Add or update a task spec under tasks-spec/<category>/<task-id>.yaml.
  2. Regenerate the benchmark tasks from the task specs before opening a PR.
  3. Open a pull request against grafana/o11y-bench.

Task contributions are reviewed for scenario quality, grading correctness, and fit with the public benchmark.

Submissions

Leaderboard submissions are stored in the Hugging Face dataset repo grafanalabs/o11y-bench-leaderboard.

Open Leaderboard Dataset
  1. Run the benchmark and keep the full Harbor job directory for each model variant you want scored.
  2. Add your submission under submissions/o11y-bench/1.1/<agent>__<model>/ with a metadata.yaml.
  3. Include the full job contents: config.json, top-level result.json, all trial directories, and downloaded agent and verifier artifacts.
  4. Open a pull request against the leaderboard dataset repo.

Submissions should preserve the benchmark’s shipped evaluation settings and currently use three attempts per task. Version 1.1 grades with an LLM judge, so the verifier needs ANTHROPIC_API_KEY set when you run the benchmark. Each benchmark version is ranked on its own leaderboard.

Review Process

Task contributions are reviewed in the benchmark repo. Result submissions are reviewed in the leaderboard dataset repo.

In both cases, maintainers are checking that the contribution matches the public benchmark structure and can be compared fairly against other runs.