From a682f1d1718297bc41d14147fa6e8edc761424db Mon Sep 17 00:00:00 2001 From: Matt Aitken Date: Wed, 8 Jul 2026 14:58:37 +0100 Subject: [PATCH] feat(hosting): default self-hosted realtime streams to v2 (s2-lite) (#4185) ## Summary Realtime streams (AI-agent token streaming and run streams) now default to v2 for self-hosters, backed by a bundled [s2-lite](https://s2.dev) service. Self-hosting previously shipped no S2 configuration, so streams ran on the Redis-backed v1 path and there were no docs for wiring up v2. Both the Docker Compose stack and the Helm chart now provision s2-lite with persistent storage and set the stream env vars out of the box. ## What's included - **Docker Compose**: a persistent `s2` service (s2-lite), a basin init spec, and the `REALTIME_STREAMS_S2_*` plus `REALTIME_STREAMS_DEFAULT_VERSION=v2` env on the webapp. `.env.example` documents the v1 fallback and hosted-S2 options. - **Helm**: an `s2` StatefulSet, PVC, Service and ConfigMap (runs as the non-root image user via `fsGroup`), an `s2` values block, and webapp env wiring with an existing-secret path for hosted S2. - **Docs**: the `REALTIME_STREAMS_S2_*` and `REALTIME_STREAMS_DEFAULT_VERSION` vars in the webapp env reference, plus a "Realtime streams" section in the Docker and Kubernetes self-hosting guides. ## Notes - The OSS code default stays `v1`; v2 becomes the default purely through the self-hosting artifacts, so non-self-host deployments are unaffected. Disabling s2, or setting the version back to `v1`, cleanly reverts to Redis-backed v1. - With v2 enabled, the bundled s2 service is a required dependency for streaming: if it is down, streams error while the task itself still runs. That is the intended trade for the better v2 path. - You can point at a hosted S2 at s2.dev instead of the bundled server. --- docs/self-hosting/docker.mdx | 21 +++ docs/self-hosting/env/webapp.mdx | 7 +- docs/self-hosting/kubernetes.mdx | 26 ++++ hosting/docker/.env.example | 16 ++ hosting/docker/webapp/docker-compose.yml | 60 ++++++++ hosting/k8s/helm/templates/_helpers.tpl | 21 +++ hosting/k8s/helm/templates/s2.yaml | 142 ++++++++++++++++++ .../templates/validate-external-config.yaml | 4 + hosting/k8s/helm/templates/webapp.yaml | 21 +++ hosting/k8s/helm/values.yaml | 82 ++++++++++ 10 files changed, 399 insertions(+), 1 deletion(-) create mode 100644 hosting/k8s/helm/templates/s2.yaml diff --git a/docs/self-hosting/docker.mdx b/docs/self-hosting/docker.mdx index 62eea5c61..2bfee7154 100644 --- a/docs/self-hosting/docker.mdx +++ b/docs/self-hosting/docker.mdx @@ -354,6 +354,27 @@ EVENT_REPOSITORY_DEFAULT_STORE=clickhouse_v2 This only affects new runs; existing runs continue to read from wherever their events were originally stored. +## Realtime streams + +Realtime streams power AI-agent token streaming and run streams. They default to **v2**, backed by the bundled `s2` service — [s2-lite](https://s2.dev), the open-source, self-hostable S2 server. It stores stream data in a persistent volume and is preconfigured in the webapp compose file, so no setup is required. + +To fall back to the Redis-backed **v1** streams, set on the webapp in your `.env`: + +```bash +REALTIME_STREAMS_DEFAULT_VERSION=v1 +``` + +To use a hosted S2 at [s2.dev](https://s2.dev) instead of the bundled s2-lite, point the endpoint at your basin and supply an access token: + +```bash +REALTIME_STREAMS_S2_BASIN=your-basin +REALTIME_STREAMS_S2_ENDPOINT=https://your-basin.b.aws.s2.dev/v1 +REALTIME_STREAMS_S2_SKIP_ACCESS_TOKENS=false +REALTIME_STREAMS_S2_ACCESS_TOKEN=your-access-token +``` + +See the [webapp environment variables](/self-hosting/env/webapp) for the full list of realtime stream settings. + ## Troubleshooting - **Deployment fails at the push step.** The machine running `deploy` needs registry access. See the [registry setup](#registry-setup) section for more details. diff --git a/docs/self-hosting/env/webapp.mdx b/docs/self-hosting/env/webapp.mdx index e9851e7fb..db9fc0dbe 100644 --- a/docs/self-hosting/env/webapp.mdx +++ b/docs/self-hosting/env/webapp.mdx @@ -139,9 +139,14 @@ mode: "wide" | **Task events** | | | | | `EVENT_REPOSITORY_DEFAULT_STORE` | No | postgres | Where to store task events. Set to `clickhouse_v2` to store in ClickHouse (recommended for production). | | **Realtime** | | | | -| `REALTIME_STREAM_VERSION` | No | v1 | Realtime stream protocol version. One of `v1`, `v2`. | +| `REALTIME_STREAM_VERSION` | No | v1 | Stream version exposed to tasks via the `TRIGGER_REALTIME_STREAM_VERSION` variable. Distinct from `REALTIME_STREAMS_DEFAULT_VERSION`. One of `v1`, `v2`. | | `REALTIME_STREAM_MAX_LENGTH` | No | 1000 | Realtime stream max length. | | `REALTIME_STREAM_TTL` | No | 86400 (1d) | Realtime stream TTL (s). | +| `REALTIME_STREAMS_DEFAULT_VERSION` | No | v1 | Server-side default the webapp uses when a stream request doesn't pin a version (modern SDKs request `v2`). The Docker and Helm self-hosting defaults set this to `v2`. One of `v1`, `v2`. | +| `REALTIME_STREAMS_S2_BASIN` | No | — | S2 basin that holds v2 realtime streams. Required for `v2`. Must be at least 8 characters. | +| `REALTIME_STREAMS_S2_ENDPOINT` | No | — | Custom S2 API endpoint, including the `/v1` suffix (e.g. `http://s2/v1` for the bundled s2-lite). Omit to use hosted S2 at s2.dev. | +| `REALTIME_STREAMS_S2_SKIP_ACCESS_TOKENS` | No | false | Skip minting per-stream access tokens. Set to `true` for s2-lite, which needs no authentication. | +| `REALTIME_STREAMS_S2_ACCESS_TOKEN` | No | — | S2 access token. Required for hosted S2 unless `REALTIME_STREAMS_S2_SKIP_ACCESS_TOKENS` is `true`. | | **Bootstrap** | | | | | `TRIGGER_BOOTSTRAP_ENABLED` | No | 0 | Trigger bootstrap enabled. | | `TRIGGER_BOOTSTRAP_WORKER_GROUP_NAME` | No | — | Trigger bootstrap worker group name. | diff --git a/docs/self-hosting/kubernetes.mdx b/docs/self-hosting/kubernetes.mdx index 3d6d1bc66..47fc08b4a 100644 --- a/docs/self-hosting/kubernetes.mdx +++ b/docs/self-hosting/kubernetes.mdx @@ -375,6 +375,32 @@ webapp: This only affects new runs; existing runs continue to read from wherever their events were originally stored. +## Realtime streams + +Realtime streams power AI-agent token streaming and run streams. They default to **v2**, backed by the bundled `s2` deployment — [s2-lite](https://s2.dev), the open-source, self-hostable S2 server. The chart deploys it with a persistent volume, so no extra services are required. + +To fall back to the Redis-backed **v1** streams, set the default version to `v1`: + +```yaml +s2: + defaultStreamVersion: "v1" +``` + +To use a hosted S2 at [s2.dev](https://s2.dev) instead of the bundled s2-lite, disable the bundled deployment and point at your basin. Supply the access token via an existing secret: + +```yaml +s2: + deploy: false + skipAccessTokens: false + external: + endpoint: "https://your-basin.b.aws.s2.dev/v1" + existingSecret: "s2-credentials" + existingSecretAccessTokenKey: "access-token" + basin: "your-basin" +``` + +To disable realtime streams v2 entirely and use v1, set `s2.deploy: false` with no external endpoint. See `helm show values` for all `s2` options. + ## Worker token When using the default bootstrap configuration, worker creation and authentication is handled automatically. The webapp generates a worker token and makes it available to the supervisor via a shared volume. diff --git a/hosting/docker/.env.example b/hosting/docker/.env.example index cc30d9c1a..383b6f79d 100644 --- a/hosting/docker/.env.example +++ b/hosting/docker/.env.example @@ -106,6 +106,22 @@ OBJECT_STORE_SECRET_ACCESS_KEY=very-safe-password # MINIO_ROOT_USER=admin # MINIO_ROOT_PASSWORD=very-safe-password +# Realtime streams +# - Realtime streams power AI-agent token streaming and run streams +# - They default to v2, backed by the bundled s2-lite service (open-source S2, https://s2.dev) +# - To fall back to the Redis-backed v1 streams, set REALTIME_STREAMS_DEFAULT_VERSION=v1 +# REALTIME_STREAMS_DEFAULT_VERSION=v2 +# The bundled s2-lite creates whatever basin you set here (min 8 characters) +# REALTIME_STREAMS_S2_BASIN=trigger-realtime +# REALTIME_STREAMS_S2_ENDPOINT=http://s2/v1 +# REALTIME_STREAMS_S2_SKIP_ACCESS_TOKENS=true +# To use a hosted S2 (https://s2.dev) instead of the bundled s2-lite: +# - point the endpoint at your basin, disable token-skipping, and set an access token +# REALTIME_STREAMS_S2_SKIP_ACCESS_TOKENS=false +# REALTIME_STREAMS_S2_ACCESS_TOKEN= +# Pin the s2-lite image in production (full image reference, digest recommended) +# S2_IMAGE=ghcr.io/s2-streamstore/s2:latest@sha256:d6ded5ca7dd619fa7c946f06e39a98f9c95c6883c8bb884e5eaa129f232c920c + # Other image tags # - These are the versions of the other images to use # - You should lock these to a specific version in production diff --git a/hosting/docker/webapp/docker-compose.yml b/hosting/docker/webapp/docker-compose.yml index c969dbd5f..611ce4939 100644 --- a/hosting/docker/webapp/docker-compose.yml +++ b/hosting/docker/webapp/docker-compose.yml @@ -18,6 +18,7 @@ services: - postgres - redis - clickhouse + - s2 networks: - webapp - supervisor @@ -44,6 +45,15 @@ services: LOGIN_ORIGIN: ${LOGIN_ORIGIN:-http://localhost:8030} API_ORIGIN: ${API_ORIGIN:-http://localhost:8030} ELECTRIC_ORIGIN: http://electric:3000 + # Realtime streams v2, backed by the bundled s2-lite service. This powers + # AI-agent token streaming and run streams. Point the endpoint at a hosted + # S2 (https://s2.dev) instead by overriding these and setting an access token. + # To fall back to the Redis-backed v1 streams, set the version to `v1`. + REALTIME_STREAMS_DEFAULT_VERSION: ${REALTIME_STREAMS_DEFAULT_VERSION:-v2} + REALTIME_STREAMS_S2_BASIN: ${REALTIME_STREAMS_S2_BASIN:-trigger-realtime} + REALTIME_STREAMS_S2_ENDPOINT: ${REALTIME_STREAMS_S2_ENDPOINT:-http://s2/v1} + REALTIME_STREAMS_S2_SKIP_ACCESS_TOKENS: ${REALTIME_STREAMS_S2_SKIP_ACCESS_TOKENS:-true} + REALTIME_STREAMS_S2_ACCESS_TOKEN: ${REALTIME_STREAMS_S2_ACCESS_TOKEN:-} DATABASE_URL: ${DATABASE_URL:-postgresql://postgres:postgres@postgres:5432/main?schema=public&sslmode=disable} DIRECT_URL: ${DIRECT_URL:-postgresql://postgres:postgres@postgres:5432/main?schema=public&sslmode=disable} SESSION_SECRET: ${SESSION_SECRET} @@ -225,12 +235,62 @@ services: retries: 5 start_period: 10s + # Generates the s2-lite basin spec from REALTIME_STREAMS_S2_BASIN so the bundled + # basin always matches the basin the webapp targets. s2-lite is distroless (no + # shell), so a tiny init container writes the spec into a shared volume first. + s2-init: + image: busybox:${BUSYBOX_IMAGE_TAG:-1.37} + restart: "no" + logging: *logging-config + environment: + REALTIME_STREAMS_S2_BASIN: ${REALTIME_STREAMS_S2_BASIN:-trigger-realtime} + command: + - sh + - -c + - | + cat > /config/s2-spec.json <.b.aws.s2.dev/v1", set skipAccessTokens: false above, and + # supply an access token either inline or via an existing secret. + external: + endpoint: "" + accessToken: "" + existingSecret: "" + existingSecretAccessTokenKey: "access-token" + # ClickHouse configuration # Subchart: https://github.com/bitnami/charts/tree/main/bitnami/clickhouse clickhouse: