Devesh
568c04f7c1
* docs: reorganize sidebar navigation and clean up install pages` * feat: enhance index hero styling and update features documentation * docs: update installation guides and enhance table styling * docs: enhance investigation documentation and improve interactive shell descriptions * Update documentation for API, community giveaway, CloudOpsBench, deployment, FAQ, PR review flow, and Python API * Update documentation for background investigations, closed-loop learning, cron scheduling, and integrations overview * Introducing structured flow for the documentation * docs: update integration documentation for various services --------- Co-authored-by: Vaibhav Upreti <vaibhav.upreti16@gmail.com> Co-authored-by: Cursor <cursoragent@cursor.com>
231 lines
7.9 KiB
Plaintext
231 lines
7.9 KiB
Plaintext
---
|
||
title: "CloudOpsBench benchmark"
|
||
description: "Run opensre+LLM against the 452-scenario Cloud-OpsBench corpus and compare to published LLM-alone baselines."
|
||
---
|
||
|
||
## Overview
|
||
|
||
CloudOpsBench is a 452-scenario Kubernetes root-cause analysis benchmark from
|
||
Wang et al
|
||
([arXiv:2603.00468v1](https://arxiv.org/abs/2603.00468), February 2026). Each
|
||
fault case is a frozen JSON repository served through mocked `kubectl`-style
|
||
tool calls, so evaluations are reproducible and do not require a live cluster.
|
||
|
||
OpenSRE evaluates this corpus through a benchmark framework that adds cost
|
||
tracking, integrity checks (pre-registration, per-stratum reporting, negative
|
||
results, conflict-of-interest disclosure), per-LLM dispatch with version
|
||
pinning, and self-contained Markdown and HTML reports. The goal is to publish
|
||
an opensre+LLM column against the paper’s LLM-alone baselines on the same
|
||
scenarios.
|
||
|
||
```text
|
||
Paper baseline opensre+LLM (this benchmark)
|
||
───────────── ──────────────────────────
|
||
DeepSeek-V3.2 0.73 A@1 → target 0.78+
|
||
GPT-5 0.67 A@1 → target 0.78+
|
||
GPT-4o 0.49 A@1 → target 0.65+
|
||
Claude-4-Sonnet 0.50 A@1 → target 0.65+
|
||
```
|
||
|
||
## Prerequisites
|
||
|
||
CloudOpsBench does not require live infrastructure. Frozen snapshots are the
|
||
environment.
|
||
|
||
```bash
|
||
# 1. Python 3.12+ (project standard)
|
||
|
||
# 2. Benchmark-dedicated LLM API keys (keep separate from production keys)
|
||
export ANTHROPIC_API_KEY=... # Claude-4-Sonnet via Anthropic
|
||
export OPENAI_API_KEY=... # GPT-5, GPT-4o
|
||
export DEEPSEEK_API_KEY=... # DeepSeek-V3.2
|
||
|
||
# 3. Download the corpus (one-time; a few hundred MB)
|
||
make download-cloudopsbench-hf
|
||
```
|
||
|
||
You do not need AWS credentials, an EKS cluster, kind/minikube, Bedrock, a GPU,
|
||
Grafana, Datadog, or Prometheus.
|
||
|
||
## Quick start
|
||
|
||
### List adapters
|
||
|
||
```bash
|
||
uv run python -m tests.benchmarks._framework.cli list
|
||
```
|
||
|
||
### Validate a config
|
||
|
||
```bash
|
||
uv run python -m tests.benchmarks._framework.cli validate \
|
||
tests/benchmarks/cloudopsbench/configs/cloudopsbench_smoke.yml
|
||
```
|
||
|
||
Validation fails on anti-patterns such as `runs_per_case < 3`, a missing
|
||
`pre_registration_path`, oversized grids, or a system-path `output_dir`.
|
||
|
||
### Development run
|
||
|
||
`--dev` skips integrity gates so you can smoke-test the wiring without a
|
||
committed pre-registration file. The run ID is prefixed with `dev-` so
|
||
development results are not treated as production results.
|
||
|
||
```bash
|
||
uv run python -m tests.benchmarks._framework.cli run \
|
||
tests/benchmarks/cloudopsbench/configs/cloudopsbench_smoke.yml --dev
|
||
```
|
||
|
||
### Production run
|
||
|
||
A production run requires:
|
||
|
||
- A pre-registration YAML at `pre_registration_path` that lists per-model
|
||
expected deltas and is committed to git before the run starts
|
||
- `seed:` set in the config
|
||
- Adapter declaration of `data_contamination_checked = True`
|
||
- At least one validity metric declared by the adapter
|
||
|
||
```bash
|
||
uv run python -m tests.benchmarks._framework.cli run \
|
||
tests/benchmarks/cloudopsbench/configs/cloudopsbench_v1.yml
|
||
```
|
||
|
||
On completion, the run directory contains `report.json`, `report.md`,
|
||
`report.html` (self-contained), and `cases/*.json` (per-cell artifacts).
|
||
|
||
### Re-render an existing report
|
||
|
||
```bash
|
||
uv run python -m tests.benchmarks._framework.cli report \
|
||
.bench-results/example/<run-dir>/
|
||
```
|
||
|
||
## Config reference
|
||
|
||
Example production config shape (see
|
||
`tests/benchmarks/cloudopsbench/configs/cloudopsbench_v1.yml` for the full
|
||
file):
|
||
|
||
```yaml
|
||
benchmark: cloudopsbench
|
||
|
||
modes:
|
||
- opensre+llm # opensre wrapping the LLM
|
||
# - llm_alone # paper baselines; rerun only if you do not trust them
|
||
|
||
llms:
|
||
- claude-4-sonnet
|
||
- deepseek-v3.2
|
||
- gpt-5
|
||
- gpt-4o
|
||
|
||
model_versions: # pinned provider snapshots
|
||
claude-4-sonnet: claude-sonnet-4-5-20250929
|
||
deepseek-v3.2: deepseek-chat-v3.2
|
||
gpt-5: gpt-5-2025-08-07
|
||
gpt-4o: gpt-4o-2024-11-20
|
||
|
||
runs_per_case: 3
|
||
workers: 4 # serial across LLMs, parallel within
|
||
cost_budget_usd: 500
|
||
seed: 42
|
||
|
||
filters: # optional case subsetting
|
||
systems: [boutique]
|
||
difficulty: [hard, medium]
|
||
|
||
output_dir: .bench-results/cloudopsbench_v1/
|
||
report_formats: [json, markdown, html]
|
||
pre_registration_path: tests/benchmarks/cloudopsbench/configs/preregistrations/cloudopsbench_v1.yml
|
||
```
|
||
|
||
### Environment variable overrides (CI)
|
||
|
||
| Variable | Purpose |
|
||
| --- | --- |
|
||
| `OPENSRE_BENCH_WORKERS` | Override `workers:` |
|
||
| `OPENSRE_BENCH_COST_BUDGET_USD` | Override `cost_budget_usd:` |
|
||
|
||
## Integrity guarantees
|
||
|
||
The framework enforces 11 honest-results mechanisms in code. There is no bypass
|
||
short of editing the framework itself.
|
||
|
||
### Pre-flight (before any case runs)
|
||
|
||
`IntegrityGuard.pre_flight` raises `IntegrityViolation` if any of the following
|
||
hold:
|
||
|
||
- **M1 — Pre-registration**: `pre_registration_path` is unset, missing, or
|
||
empty. Expected deltas must be committed before results are known.
|
||
- **M3 — Validity metrics**: the adapter declares no validity metric.
|
||
- **M6 — Seeded selection**: `seed:` is `None`.
|
||
- **M7 — Contamination check**: the adapter has not declared
|
||
`data_contamination_checked = True`.
|
||
|
||
Violations are reported together so they can be fixed in one pass.
|
||
|
||
### Report validation (before the report is emitted)
|
||
|
||
`IntegrityGuard.report_validation` refuses to publish a report if:
|
||
|
||
- **M3** — Not every adapter-declared metric appears in the report
|
||
- **M4** — Per-stratum breakdown is missing or contains only `all`
|
||
- **M5** — Raw per-case artifacts directory is missing
|
||
- **M9** — `negative_results` is empty
|
||
- **M10** — `coi_disclosure` is empty
|
||
- **M1** — Pre-registration path is not carried into the report
|
||
|
||
### Operational mechanisms (not code-enforced)
|
||
|
||
- **M8 — External replication** of at least one cell by a third party before a
|
||
public claim
|
||
- **M11 — Blinded LLM-as-judge calibration** (BDIL Phase B; tracked separately)
|
||
|
||
## Cost tracking
|
||
|
||
The framework registers a usage hook on the LLM clients in
|
||
`core/llm/transports/sdk/llm_clients.py`. Each successful call records
|
||
`(model, tokens_in, tokens_out)` in a `CostTracker`. When the next call would
|
||
exceed `cost_budget_usd`, the runner raises `CostBudgetExceeded` and stops with
|
||
a partial-completion report.
|
||
|
||
Per-cell `tokens_in` / `tokens_out` / `cost_usd` may currently be `0`
|
||
(aggregate cost in `report.json` is correct; per-cell capture is a follow-up).
|
||
|
||
## Metrics
|
||
|
||
The paper’s 13 deterministic metrics plus three framework validity metrics:
|
||
|
||
| Family | Metric | Source |
|
||
| --- | --- | --- |
|
||
| Outcome | `a1`, `a3`, `tcr`, `exact`, `in_order`, `any_order` | Paper § 4.2.1 |
|
||
| Process — alignment | `rel`, `cov` | Paper § 4.2.2 |
|
||
| Process — efficiency | `steps`, `mtti` | Paper § 4.2.2 |
|
||
| Process — robustness | `iac`, `rar`, `ztdr` | Paper § 4.2.2 |
|
||
| Validity | `citation_grounding_rate`, `entity_existence_rate`, `kubectl_actionability_rate` | Framework |
|
||
|
||
All 16 metrics are deterministic (string or set comparison). Evaluation does
|
||
not use an LLM as judge.
|
||
|
||
## Existing production entry points
|
||
|
||
`make test-cloudopsbench` and `opensre tests cloudopsbench` use
|
||
`tests/benchmarks/cloudopsbench/run_suite.py` (legacy imperative CLI). The
|
||
framework runner is the YAML-config surface. Both share the same adapter,
|
||
scoring code, and replay backend.
|
||
|
||
## Reference
|
||
|
||
- Paper: Wang et al, *Cloud-OpsBench: A Reproducible Benchmark for Agentic Root
|
||
Cause Analysis in Cloud Systems*,
|
||
[arXiv:2603.00468v1](https://arxiv.org/abs/2603.00468), 28 February 2026 —
|
||
[GitHub](https://github.com/LLM4Ops/Cloud-OpsBench)
|
||
- Hugging Face dataset:
|
||
[`tracer-cloud/cloud-ops-bench-dataset`](https://huggingface.co/datasets/tracer-cloud/cloud-ops-bench-dataset)
|
||
- Framework source:
|
||
[`tests/benchmarks/_framework/`](https://github.com/Tracer-Cloud/opensre/tree/main/tests/benchmarks/_framework)
|
||
- Adapter source:
|
||
[`tests/benchmarks/cloudopsbench/`](https://github.com/Tracer-Cloud/opensre/tree/main/tests/benchmarks/cloudopsbench)
|