Awokoya Olawale Davidson
580b3f2cce
* Meter telegram and buzz turns, closing the transport-contract ledger
Slack and Discord consume credits before running a turn; telegram and buzz
did not, so two of the four transports ran every turn for free. Both were
carried in the known-gaps ledger in test_transport_contract.
Add the credit gate to both inbound handlers at the same point Slack and
Discord place it — after the session resolves, before any turn work — and
empty the ledger, so the contract test now requires metering of all four.
The charge goes to scope.principal.id (the silo org), not the chat id or
sender pubkey. Only an explicit 402 blocks the turn: UNCONFIGURED and
UNAVAILABLE proceed, so a billing outage never silences the bot and a config
error never reads to users as "out of credits".
The call is awaited off-thread. The Slack/Discord dispatchers are already on
worker threads, but these two are coroutines, and buzz dispatches turns as
concurrent tasks on one loop — a synchronous POST would park every other
in-flight turn, and the poll loop with them, for the client timeout.
Gateway tests now clear the metering env, since a developer whose shell
exports it would otherwise have the suite POST charges to a live ledger. The
machine secret goes too: the bearer token is minted from Clerk before the URL
is checked, so clearing the URL alone still calls out.
* Charge synchronously so a cancelled turn cannot strand a credit
Greptile P1 on #5388. Awaiting the charge off-thread made it a cancellation
point. Shutdown cancels turns that outlast the drain budget, and _dispatch_turn
catches Exception, which CancelledError is not — so the coroutine unwinds past
the trailing acknowledge(). The POST still completes on its worker thread, the
mention stays uncommitted, and the next start re-delivers it and charges the
organization a second time for one message.
Reverting to the synchronous call closes the window. Nothing suspends between
the charge and the run_in_executor handoff, so the turn body is always queued
once the ledger has been debited, and its on_handled acknowledges from the
executor thread the way the existing docstring describes.
This also puts both handlers back on the Slack/Discord shape the issue asked
for. The loop-parking the off-thread call avoided was speculative; the
surrounding code already blocks the loop with the session resolve and, on the
denial path, a buzz CLI subprocess spawn.
Add a regression test that cancels a Buzz turn while the charge is in flight
and asserts the mention is still acknowledged. It fails on the awaited version
and passes on this one.
* Charge from the turn body so the ledger blocks neither the loop nor the ack
Two review findings pulled in opposite directions. Greptile's P1 said not to
await the charge: a shutdown cancel landing on that await unwinds past
_dispatch_turn's acknowledge() while the POST completes anyway, stranding a
credit on a mention the next start re-delivers and charges again. Devesh36
then flagged the synchronous fix: up to 5s of the credits client timeout on
the event loop, stalling Buzz polling, dispatch and shutdown.
Both hold. Metering now runs inside the turn body, on the executor thread,
which answers each. Off the loop, a slow ledger cannot stall polling or the
other turns sharing it. Fused to the work it pays for, the debit cannot be
stranded: cancelling the awaiting task does not stop the executor thread,
whose finally still acknowledges the mention.
A denied turn now finalizes its placeholder with the out-of-credits message
rather than posting a second one, matching how these two handlers already
surface timeout and user-stop.
Tests: charging off the event loop, and a turn cancelled as the charge lands
still being acknowledged. Each fails against the shape it guards against.
* fix(gateway): meter only admitted chat turns
* fix(gateway): keep admitted turns fused to billing
* test(gateway): pin the pre-admission cancel branch, trim the duplicate
`run` returns None for three reasons now, and only the capacity gate
finalizes a message. Say so on the docstring, and record why admission
deliberately holds a slot it has not used yet.
Add the missing test for the cancel check that runs before admission, so a
turn stopped while queued can never reach a metering hook that debits.
Drop buzz's fail-open duplicate: the parametrized test in
gateway/tests/billing already covers every non-denied outcome. Scope the
transport contract's empty gap ledger to transports, since scheduled runs
reach the agent through scheduler_runners rather than a transport.
* test(telegram): meter on the production dispatch path
Telegram now registers the cancel Event before the turn exists (#5218), so
the metering test was exercising a branch production no longer takes.
* test(cli): pin the generic Anthropic model-not-found message
f3053e2c5 stopped copying str(exc) into the Anthropic 404 user message so
internal model ids stay out of investigation reports, but this older CLI
test still asserted the model id was echoed. Assert the hardening instead.
OpenSRE v0.1: Build Your Own AI SRE Agents
The open-source framework for AI SRE agents, and the training and evaluation environment they need to improve. Connect the 60+ tools you already run, define your own workflows, and investigate incidents on your own infrastructure.
Quickstart · Docs · FAQ · Security
🚧 Public Alpha: Core workflows are usable for early exploration, though not yet fully stable. The project is in active development, and APIs and integrations may evolve
Table of Contents
- Why OpenSRE?
- Install
- Quick Start
- Deployment
- How OpenSRE Works
- Benchmark
- Capabilities & integrations
- Contributing & development
- Security
- Telemetry
- License
- Citations
Why OpenSRE?
When something breaks in production, the evidence is scattered across logs, metrics, traces, runbooks, and Slack threads. OpenSRE is an open-source framework for AI SRE agents that resolve production incidents, built to run on your own infrastructure.
We do that because SWE-bench1 gave coding agents scalable training data and clear feedback. Production incident response still lacks an equivalent.
Distributed failures are slower, noisier, and harder to simulate and evaluate than local code tasks, which is why AI SRE, and AI for production debugging more broadly, remains unsolved.
OpenSRE is building that missing layer:
an open reinforcement learning environment for agentic infrastructure incident response, with end-to-end tests and synthetic incident simulations for realistic production failures
We do that by:
- building easy-to-deploy, customizable AI SRE agents for production incident investigation and response
- running scored synthetic RCA suites that check root-cause accuracy, required evidence, and adversarial red herrings (tests/synthetic)
- running real-world end-to-end tests across cloud-backed scenarios including Kubernetes, EC2, CloudWatch, Lambda, ECS Fargate, and Flink (tests/e2e)
- keeping semantic test-catalog naming so e2e vs synthetic and local vs cloud boundaries stay obvious (tests/README.md)
Our mission is to build AI SRE agents on top of this, scale it to thousands of realistic infrastructure failure scenarios, and establish OpenSRE as the benchmark and training ground for AI SRE.
1 https://arxiv.org/abs/2310.06770
Install
The root installer URL auto-detects Unix shell vs PowerShell and installs the latest build from main. OpenSRE moves quickly, so main is the latest stable version for normal installs.
macOS / Linux:
curl -fsSL https://install.opensre.com | bash
The macOS/Linux installer does not require sudo. If no writable bin directory is already on PATH, it installs to ~/.local/bin and prints the shell command to apply the PATH update.
Equivalent explicit main-channel form:
curl -fsSL https://install.opensre.com | bash -s -- --main
Homebrew:
brew tap tracer-cloud/tap
brew install tracer-cloud/tap/opensre
Windows (PowerShell):
irm https://install.opensre.com | iex
Quick Start
Contributors: start at main.py for the process entrypoint map.
Configure once, then pick how you want to run investigations:
opensre onboard
Interactive shell — with no subcommand, opensre starts a REPL (TTY required). Describe incidents in plain language, stream investigations, and use slash commands for session control (/help, /status, /cost, /sessions, /resume, /compact, /new, /exit), integrations (/integrations list, /integrations verify), local agent fleet monitoring (/agents), and reasoning depth (/effort for OpenAI and Codex — low through max). Ctrl+C cancels an in-flight investigation without losing session state. See interactive shell commands for the full reference.
opensre
Headless CLI — run one agent turn non-interactively from a terminal, script, or CI job:
opensre ask "why is checkout-api slow?"
See Headless CLI for stdin prompts, JSON output, and tool approvals.
One-shot investigation — run the agent once against an alert file:
opensre investigate -i tests/e2e/kubernetes/fixtures/datadog_k8s_alert.json
Remote runtime investigation — investigate a deployed service by name (live health, logs, and deployment status):
opensre investigate --service api-backend
Hermes log watch — tail a Hermes errors.log, classify incidents, and optionally alert on Telegram:
opensre hermes watch
From Python — drive the agent in-process from your own code (source checkout required):
from core.agent_harness import AgentSession
session = AgentSession.start()
result = session.chat("why is checkout-api slow?")
if result.answered:
print(result.primary_response_text)
See Python API for sessions, conversations, and custom output sinks.
For your team's daily loop: embed OpenSRE in the Python services and automations your teammates already use. Start with the in-repo Python API guide, then use it every day to make incident response repeatable.
Other useful commands:
opensre integrations setup
opensre agents scan
opensre update
opensre uninstall # remove opensre and all local data
Deployment
Two primary AWS EC2 paths and a general hosted option:
- Gateway (AMI + systemd):
make build-gateway-imagethenmake deploy-gateway— Telegram gateway only, no Docker; the gateway is installed into a server image that new servers start from. - Hosted (Railway / ECS / Vercel): deploy with the repo
Dockerfile; setLLM_PROVIDERand the matching API key (see.env.example), plusDATABASE_URIandREDIS_URIif persistence is needed.
Full deployment steps and prerequisites → DEPLOYMENT.md
How OpenSRE Works
When an alert fires, OpenSRE automatically:
- Fetches the alert context and correlated logs, metrics, traces, and recent deploys
- Masks sensitive identifiers (optional) before external LLM calls
- Reasons across your connected systems to test hypotheses in a tool-calling loop
- Generates a structured investigation report with probable root cause and linked evidence
- Suggests next steps and, optionally, executes remediation actions
- Posts a summary directly to Slack, PagerDuty, or Telegram — no context switching needed
For the current code-level agent architecture after removing the old graph and chain framework layers, see AGENTS.md.
Benchmark
Regenerate numbers with make benchmark; refresh this table from cached results via make benchmark-update-readme. See docs/DEVELOPMENT.md for details.
No benchmark results yet.
Capabilities & integrations
| 🔍 Structured incident investigation | Correlated root-cause analysis across logs, metrics, traces, deploys, and config |
| 📋 Runbook-aware reasoning | OpenSRE reads your runbooks and applies them automatically |
| 🔗 Evidence-backed root cause | Every conclusion is linked to the data behind it |
| 🛡️ Reversible identifier masking | Redact pods, clusters, and account IDs before external LLM calls; restore in output |
| 📊 Session cost & history | Per-session token tracking (/cost) and resumable REPL sessions (/sessions) |
| 👥 Local agent fleet | Monitor Claude Code, Cursor, Codex, and other coding agents on your machine |
| 🌐 Remote runtime RCA | Investigate deployed services by name with live health probes and recent logs |
| 📡 Hermes log watch | Tail Hermes error logs, classify incidents, and deliver Telegram alerts |
| 🤖 Full LLM flexibility | Bring your own model — Anthropic, OpenAI, Codex, Ollama, Gemini, OpenRouter, TrustedRouter, NVIDIA NIM, Bedrock |
OpenSRE connects to 60+ tools across LLMs, observability, cloud infrastructure, data platforms, incident management, and MCP. The full matrix (with roadmap links) lives in the product docs; a detailed catalog is also maintained in-repo as the project grows.
Integrations
OpenSRE connects to 60+ tools and services across the modern cloud stack, from LLM providers and observability platforms to infrastructure, databases, and incident management.
| Category | Integrations | Roadmap |
|---|---|---|
| AI / LLM Providers | Anthropic · OpenAI · OpenAI Codex · Ollama · Google Gemini · OpenRouter · TrustedRouter · NVIDIA NIM · Bedrock | |
| Observability |  Grafana (Loki · Mimir · Tempo · annotations) ·  Datadog · Honeycomb · Coralogix ·  groundcover ·  CloudWatch ·  Sentry · Elasticsearch · Better Stack · Splunk · Victoria Logs · SignOz · OpenObserve · OpenSearch · Azure Monitor · New Relic · Hermes |
|
| Infrastructure |  Kubernetes ·  AWS (S3 · Lambda · EKS · EC2 · CloudTrail · Bedrock) ·  GCP ·  Azure · Yandex Cloud · ArgoCD · Helm · Jenkins |
|
| Database | MongoDB · ClickHouse · PostgreSQL · MySQL · MariaDB · MongoDB Atlas · Azure SQL · Snowflake · Redis · RDS · Supabase | |
| Data Platform | Apache Airflow · Apache Kafka · Apache Spark · Prefect · RabbitMQ · Dagster | |
| Dev Tools |  GitHub · GitHub MCP · Bitbucket · GitLab |
|
| Incident Management |  PagerDuty · Opsgenie · Jira · Alertmanager · incident.io · ServiceNow |
Trello · Linear |
| Communication |  Slack · Google Docs · Discord · Telegram ·  Rocket.Chat · WhatsApp · Buzz |
Notion · Teams · Confluence |
| Agent Deployment |  Vercel · EC2 · ECS · Railway |
|
| Protocols |  MCP ·  ACP ·  OpenClaw |
OpenSRE is community-built. Looking for a safe first contribution? Browse good first issue tickets or see the Good First Issues guide. See CONTRIBUTING.md for the full workflow.
Local environment: SETUP.md (all platforms, Windows, MCP/OpenClaw).
Developing in this repo: docs/DEVELOPMENT.md (install from source, CI parity checks, dev container, benchmark, deployment detail, telemetry reference).
Thanks goes to these amazing people:
Security
OpenSRE is designed with production environments in mind: structured and auditable LLM prompts, local transcript handling by default, and no silent bulk export of raw logs. See SECURITY.md for responsible disclosure.
Telemetry
PostHog (product analytics) and Sentry (errors) are opt-out. Quick disable:
export OPENSRE_NO_TELEMETRY=1
Full matrix, DSN override, and local event logging → docs/DEVELOPMENT.md
License
Apache 2.0 — see LICENSE.