ap9
cf6376ae9a
* docs(argocd): add a local verification recipe Verified both registered Argo CD tools live against a real local Argo CD install with a real GitOps application synced from a public repo (argoproj/argocd-example-apps guestbook) -- no bugs found. argocd_application_status confirmed with real sync/health status, revision, and operation phase; argocd_application_diff confirmed against real drift (disabled automated sync, then patched a live Deployment's replica count directly, confirming the tool correctly surfaces the resulting diff). Also documents a genuine gotcha hit while building the recipe: the official install.yaml fails under plain `kubectl apply` with `metadata.annotations: Too long: may not be more than 262144 bytes` -- the ApplicationSet CRD is large enough that client-side apply's last-applied-configuration annotation exceeds Kubernetes' own limit. `--server-side --force-conflicts` avoids the annotation entirely. Verified the full recipe twice, end-to-end, against two independent fresh kind clusters. * docs(argocd): fix reproducibility and lifecycle gaps per Greptile - The diff-tool output was attributed to "direct tool invocation" with no actual reproducible command -- and opensre investigate doesn't guarantee argocd_application_diff runs on any given turn (the planner decides). Added the exact invocation as a copy-pasteable command; first draft used bare python3, which fails since importing integrations.argocd.tools needs the project's own venv -- fixed to uv run python3 and re-verified live. - The port-forward was backgrounded with no PID capture or readiness check, so a slow start or leftover process on 8080 could leave the login step targeting a dead endpoint, or leak a stray process past teardown. Added a bounded readiness loop and captured the PID so teardown can actually kill it. - Trimmed a "Confirmed live: ..." verification-narration sentence from the CRD-size Warning box; kept the actionable explanation. Re-verified the changed sections live on a second fresh cluster: PID capture, readiness loop, and the corrected uv run invocation all confirmed working. * docs(argocd): trap-kill the port-forward on any exit path, per Greptile The bounded readiness loops' exit 1 on timeout (or an interrupted run) skipped straight past the teardown block, leaving the backgrounded kubectl port-forward running and occupying port 8080 for the next attempt. Added a trap on EXIT right after starting the port-forward, so normal completion, a readiness-loop timeout, and an interrupted run all reliably kill it. Verified live: a simulated timeout correctly triggers the trap and the background process is gone immediately after. * docs(argocd): make cluster creation idempotent, per Greptile The port-forward trap only covers that one process -- an early exit from the readiness loop, a failed command, or an interrupted run still left the kind cluster itself running, and kind create cluster fails outright on a name collision (node(s) already exist for a cluster with the name "..."), so a retry after any failed attempt needed a manual kind delete first. Deleting any stale cluster of the same name before creating (a no-op on a clean first run) makes retries always work without requiring cleanup on every possible failure path -- confirmed the exact error message live before writing the explanation. * docs(argocd): warn before deleting a pre-existing cluster, per Greptile The unconditional kind delete before create (added for retry-ability) would silently destroy a reader's own cluster if they already had one named opensre-argocd-demo for an unrelated purpose -- no check, no warning. Now only deletes if a cluster with that exact name already exists, and prints a 5-second warning with a name-specific Ctrl-C window before doing so, rather than acting silently. Verified live both paths: no pre-existing cluster skips straight to create, and a pre-existing cluster of that name triggers the warning, waits, then deletes and recreates. * docs(argocd): require affirmative confirmation before deleting a cluster, per Greptile A timed Ctrl-C window is still an automatic delete if the reader isn't watching the terminal at that exact moment. Replaced with a blocking read prompt requiring an explicit y/Y before deleting an already-existing opensre-argocd-demo cluster; anything else (including no input, e.g. a non-interactive run) aborts with exit 1 and leaves the existing cluster untouched. Verified live: no pre-existing cluster skips the prompt entirely, confirming 'y' deletes and recreates, declining aborts and preserves the existing cluster. * docs(argocd): fix store-path pattern per human review on a sibling doc Same fix as docs/splunk.mdx and siblings: "a literal zero-byte file does not work here" doesn't hold on current main -- store.py catches JSONDecodeError from an empty file and falls back to an empty v2 store. Switched to pointing OPENSRE_INTEGRATIONS_STORE_PATH at a path inside a fresh mktemp -d directory instead, which avoids the exception entirely since the file itself never exists. Also trims a verification-narrative sentence to match the corrected sibling docs. * docs(argocd): remove embedded internal-import diff reproduction, per muddlebee Removed the python3 -c snippet that directly imported and invoked ArgoCDApplicationDiffTool -- user-facing docs shouldn't teach or depend on internal integration class paths. Replaced with prose stating the same substance (the planner doesn't guarantee this tool runs every turn, and here's what it reported on a turn where it did) without a reader-facing internal-import command. The opensre investigate flow, empty-directory store setup, and teardown are unchanged.
OpenSRE v0.1: Build Your Own AI SRE Agents
The open-source framework for AI SRE agents, and the training and evaluation environment they need to improve. Connect the 60+ tools you already run, define your own workflows, and investigate incidents on your own infrastructure.
Quickstart · Docs · FAQ · Security
🚧 Public Alpha: Core workflows are usable for early exploration, though not yet fully stable. The project is in active development, and APIs and integrations may evolve
Table of Contents
- Why OpenSRE?
- Install
- Quick Start
- Deployment
- How OpenSRE Works
- Benchmark
- Capabilities & integrations
- Contributing & development
- Security
- Telemetry
- License
- Citations
Why OpenSRE?
When something breaks in production, the evidence is scattered across logs, metrics, traces, runbooks, and Slack threads. OpenSRE is an open-source framework for AI SRE agents that resolve production incidents, built to run on your own infrastructure.
We do that because SWE-bench1 gave coding agents scalable training data and clear feedback. Production incident response still lacks an equivalent.
Distributed failures are slower, noisier, and harder to simulate and evaluate than local code tasks, which is why AI SRE, and AI for production debugging more broadly, remains unsolved.
OpenSRE is building that missing layer:
an open reinforcement learning environment for agentic infrastructure incident response, with end-to-end tests and synthetic incident simulations for realistic production failures
We do that by:
- building easy-to-deploy, customizable AI SRE agents for production incident investigation and response
- running scored synthetic RCA suites that check root-cause accuracy, required evidence, and adversarial red herrings (tests/synthetic)
- running real-world end-to-end tests across cloud-backed scenarios including Kubernetes, EC2, CloudWatch, Lambda, ECS Fargate, and Flink (tests/e2e)
- keeping semantic test-catalog naming so e2e vs synthetic and local vs cloud boundaries stay obvious (tests/README.md)
Our mission is to build AI SRE agents on top of this, scale it to thousands of realistic infrastructure failure scenarios, and establish OpenSRE as the benchmark and training ground for AI SRE.
1 https://arxiv.org/abs/2310.06770
Install
The root installer URL auto-detects Unix shell vs PowerShell and installs the latest build from main. OpenSRE moves quickly, so main is the latest stable version for normal installs.
macOS / Linux:
curl -fsSL https://install.opensre.com | bash
The macOS/Linux installer does not require sudo. If no writable bin directory is already on PATH, it installs to ~/.local/bin and prints the shell command to apply the PATH update.
Equivalent explicit main-channel form:
curl -fsSL https://install.opensre.com | bash -s -- --main
Homebrew:
brew tap tracer-cloud/tap
brew install tracer-cloud/tap/opensre
Windows (PowerShell):
irm https://install.opensre.com | iex
Quick Start
Contributors: start at main.py for the process entrypoint map.
Configure once, then pick how you want to run investigations:
opensre onboard
Interactive shell — with no subcommand, opensre starts a REPL (TTY required). Describe incidents in plain language, stream investigations, and use slash commands for session control (/help, /status, /cost, /sessions, /resume, /compact, /new, /exit), integrations (/integrations list, /integrations verify), local agent fleet monitoring (/agents), and reasoning depth (/effort for OpenAI and Codex — low through max). Ctrl+C cancels an in-flight investigation without losing session state. See interactive shell commands for the full reference.
opensre
Headless CLI — run one agent turn non-interactively from a terminal, script, or CI job:
opensre ask "why is checkout-api slow?"
See Headless CLI for stdin prompts, JSON output, and tool approvals.
One-shot investigation — run the agent once against an alert file:
opensre investigate -i tests/e2e/kubernetes/fixtures/datadog_k8s_alert.json
Remote runtime investigation — investigate a deployed service by name (live health, logs, and deployment status):
opensre investigate --service api-backend
Hermes log watch — tail a Hermes errors.log, classify incidents, and optionally alert on Telegram:
opensre hermes watch
From Python — drive the agent in-process from your own code (source checkout required):
from core.agent_harness import AgentSession
session = AgentSession.start()
result = session.chat("why is checkout-api slow?")
if result.answered:
print(result.primary_response_text)
See Python API for sessions, conversations, and custom output sinks.
For your team's daily loop: embed OpenSRE in the Python services and automations your teammates already use. Start with the in-repo Python API guide, then use it every day to make incident response repeatable.
Other useful commands:
opensre integrations setup
opensre agents scan
opensre update
opensre uninstall # remove opensre and all local data
Deployment
Two primary AWS EC2 paths and a general hosted option:
- Gateway (AMI + systemd):
make build-gateway-imagethenmake deploy-gateway— Telegram gateway only, no Docker; the gateway is installed into a server image that new servers start from. - Hosted (Railway / ECS / Vercel): deploy with the repo
Dockerfile; setLLM_PROVIDERand the matching API key (see.env.example), plusDATABASE_URIandREDIS_URIif persistence is needed.
Full deployment steps and prerequisites → DEPLOYMENT.md
How OpenSRE Works
When an alert fires, OpenSRE automatically:
- Fetches the alert context and correlated logs, metrics, traces, and recent deploys
- Masks sensitive identifiers (optional) before external LLM calls
- Reasons across your connected systems to test hypotheses in a tool-calling loop
- Generates a structured investigation report with probable root cause and linked evidence
- Suggests next steps and, optionally, executes remediation actions
- Posts a summary directly to Slack, PagerDuty, or Telegram — no context switching needed
For the current code-level agent architecture after removing the old graph and chain framework layers, see AGENTS.md.
Benchmark
Regenerate numbers with make benchmark; refresh this table from cached results via make benchmark-update-readme. See docs/DEVELOPMENT.md for details.
No benchmark results yet.
Capabilities & integrations
| 🔍 Structured incident investigation | Correlated root-cause analysis across logs, metrics, traces, deploys, and config |
| 📋 Runbook-aware reasoning | OpenSRE reads your runbooks and applies them automatically |
| 🔗 Evidence-backed root cause | Every conclusion is linked to the data behind it |
| 🛡️ Reversible identifier masking | Redact pods, clusters, and account IDs before external LLM calls; restore in output |
| 📊 Session cost & history | Per-session token tracking (/cost) and resumable REPL sessions (/sessions) |
| 👥 Local agent fleet | Monitor Claude Code, Cursor, Codex, and other coding agents on your machine |
| 🌐 Remote runtime RCA | Investigate deployed services by name with live health probes and recent logs |
| 📡 Hermes log watch | Tail Hermes error logs, classify incidents, and deliver Telegram alerts |
| 🤖 Full LLM flexibility | Bring your own model — Anthropic, OpenAI, Codex, Ollama, Gemini, OpenRouter, TrustedRouter, NVIDIA NIM, Bedrock |
OpenSRE connects to 60+ tools across LLMs, observability, cloud infrastructure, data platforms, incident management, and MCP. The full matrix (with roadmap links) lives in the product docs; a detailed catalog is also maintained in-repo as the project grows.
Integrations
OpenSRE connects to 60+ tools and services across the modern cloud stack, from LLM providers and observability platforms to infrastructure, databases, and incident management.
| Category | Integrations | Roadmap |
|---|---|---|
| AI / LLM Providers | Anthropic · OpenAI · OpenAI Codex · Ollama · Google Gemini · OpenRouter · TrustedRouter · NVIDIA NIM · Bedrock | |
| Observability |  Grafana (Loki · Mimir · Tempo · annotations) ·  Datadog · Honeycomb · Coralogix ·  groundcover ·  CloudWatch ·  Sentry · Elasticsearch · Better Stack · Splunk · Victoria Logs · SignOz · OpenObserve · OpenSearch · Azure Monitor · New Relic · Hermes |
|
| Infrastructure |  Kubernetes ·  AWS (S3 · Lambda · EKS · EC2 · CloudTrail · Bedrock) ·  GCP ·  Azure · Yandex Cloud · ArgoCD · Helm · Jenkins |
|
| Database | MongoDB · ClickHouse · PostgreSQL · MySQL · MariaDB · MongoDB Atlas · Azure SQL · Snowflake · Redis · RDS · Supabase | |
| Data Platform | Apache Airflow · Apache Kafka · Apache Spark · Prefect · RabbitMQ · Dagster | |
| Dev Tools |  GitHub · GitHub MCP · Bitbucket · GitLab |
|
| Incident Management |  PagerDuty · Opsgenie · Jira · Alertmanager · incident.io · ServiceNow |
Trello · Linear |
| Communication |  Slack · Google Docs · Discord · Telegram ·  Rocket.Chat · WhatsApp · Buzz |
Notion · Teams · Confluence |
| Agent Deployment |  Vercel · EC2 · ECS · Railway |
|
| Protocols |  MCP ·  ACP ·  OpenClaw |
OpenSRE is community-built. Looking for a safe first contribution? Browse good first issue tickets or see the Good First Issues guide. See CONTRIBUTING.md for the full workflow.
Local environment: SETUP.md (all platforms, Windows, MCP/OpenClaw).
Developing in this repo: docs/DEVELOPMENT.md (install from source, CI parity checks, dev container, benchmark, deployment detail, telemetry reference).
Thanks goes to these amazing people:
Security
OpenSRE is designed with production environments in mind: structured and auditable LLM prompts, local transcript handling by default, and no silent bulk export of raw logs. See SECURITY.md for responsible disclosure.
Telemetry
PostHog (product analytics) and Sentry (errors) are opt-out. Quick disable:
export OPENSRE_NO_TELEMETRY=1
Full matrix, DSN override, and local event logging → docs/DEVELOPMENT.md
License
Apache 2.0 — see LICENSE.