fix/core-api-request-timeout
626 Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 James Ritchie
|
9f29eb2090 |
fix(webapp): update task filter icons and the scheduled run icon (#4065)
## Summary Fixes the Tasks icon on the Runs, Logs, Errors, and metrics pages (and the task-derived queue indicators) to use the correct "T" tasks icon, and shows a clock icon for scheduled runs in the run inspector header. ## Before <img width="1440" height="704" alt="CleanShot 2026-06-28 at 21 39 57@2x" src="https://github.com/user-attachments/assets/d00d5cbf-4726-4695-825b-59f5bdd84529" /> ## After <img width="1370" height="604" alt="CleanShot 2026-06-28 at 21 38 59@2x" src="https://github.com/user-attachments/assets/fb709073-0b9f-409d-a852-72979bf14da6" /> |
||
|
Eric Allam
|
f1bd11a7ef |
feat(webapp): gracefully shut down the v3 engine behind a flag (#4017)
## Summary Adds a single env flag, `DEPRECATE_V3_ENABLED` (default off), that gracefully winds down the v3 engine (`RunEngineVersion.V1`). While it's off nothing changes, so self-hosted instances still on v3 keep working. When it's on: - Triggers that resolve to v3 are rejected with a clear, actionable error pointing at the [v4 migration guide](https://trigger.dev/docs/migrating-from-v3), instead of silently creating runs that never execute. This covers single triggers, batches, scheduled fires, replays, and `triggerAndWait`, which all funnel through one place. - The legacy `trigger dev` websocket used by v3 CLIs is closed with an upgrade message (v4 CLIs use a different dev transport). - The v3 shared-queue consumer refuses to start, so no deployed v3 runs are dequeued. - The v3 run-lifecycle background jobs (heartbeat timeout, TTL expiry, retry, resume batch/dependency, delayed-run enqueue, and scheduled fires) become no-ops, so abandoned v3 runs stop generating database load. This builds on the existing deploy deprecation flag, which already rejects v3 CLI deploys. ## Design Enforcement is read through one helper, `isV3Disabled()`. Every gate combines it with a per-run or per-project engine check (`isV3Disabled() && engine === "V1"`), so a v4 run that happens to reach a shared service behaves exactly as before. v4 (V2) is never affected. The flag is a hard switch, not a drain: when it's on, in-flight v3 runs are abandoned in place rather than failed or expired, which is the intended behaviour for the final shutdown. |
||
|
Katia Bulatova
|
b1987dc090 |
feat(webapp): billing limits — pause, reject, recovery, and settings UI (#3996)
## Summary Adds Billing Limits to the webapp. Customers can set a monthly spend cap. When usage crosses the limit, billable environments enter a grace period. If the limit is not resolved before grace expires, new triggers are rejected until the organization increases or removes the limit. |
||
|
Chris Arderne
|
4fde283e76 |
chore: format and lint webapp also (#4056)
#3977 added formatting and linting everywhere else. This extends it to the webapp. |
||
|
Eric Allam
|
9ef5cf0055 |
fix(webapp): stop showing the in-dashboard agent to admins by default (#4050)
## Summary The in-dashboard agent button was rendered for all admins and impersonators regardless of the `hasDashboardAgentAccess` flag, so it appeared even where the agent is disabled (for example, floating over the run inspector controls). It is now gated by the flag for everyone, so it stays hidden until the flag is turned on. ## Rollout Both levers default off, so nothing changes for users until deliberately enabled: - **Per-org:** set `hasDashboardAgentAccess` on an org's feature flags to enable the agent for just that org. - **All admins:** set `DASHBOARD_AGENT_ADMIN_PREVIEW=1` to give admins and impersonators an everywhere-preview, independent of the per-org flag. Previously admins bypassed the flag unconditionally, which is why the button showed up before the agent was ready to ship. |
||
|
Chris Arderne
|
df78ef96d9 |
feat: multi dev branches (#4023)
Closes this feature request: [https://triggerdev.featurebase.app/p/isolated-dev-sessions-for-multiple-local-trigger-dev-instances](https://triggerdev.featurebase.app/p/isolated-dev-sessions-for-multiple-local-trigger-dev-instances) ### Feature notes: - CLI `trigger dev` works as before - `trigger dev --branch my-branch` to create a new branch and run against it. - `trigger dev archive --branch my-branch` to archive (or in webapp). - New webapp page to manage and archive dev branches, currently feature flagged. ### Implementation details: - No changes to data model, no backfill. `isBranchableEnvironment` column is ignored for dev branches, we use `parentEnvironmentId IS NULL` instead. - `x-trigger-branch` overloaded for preview and dev branches - New `TRIGGER_DEV_BRANCH` env var available locally. `TRIGGER_PREVIEW_BRANCH` overloaded for child runs. - Lots of new glue code to sanitise the branch checks. ### Rollout - Deploy webapp/API changes (all backwards compatible) - Manual tests on some orgs - Deploy docs, release CLI, flip feature flag for webapp feature ### NB - `api.v1.projects.$projectRef.environments.ts` will return `isBranchableEnvironment: true` for all dev environments. ### Prerequisites - [x] Typecheck will not pass until we make a new release of `@trigger.dev/platform` and bump it here |
||
|
nicktrn
|
bc605eedaf |
fix(webapp): verify deployment image exists before finalizing (#4049)
A deployment could be marked deployed and promoted to current without its image ever landing in the registry. Finalize trusted the CLI: the v1 path never pushed or checked, and the v2/v3 path skips its own push when the CLI sends `skipPushToRegistry` - which the local-build path always does. In the happy path the CLI pushes the image itself, so this stayed latent. But any deviation - `--no-push`/`--load`, a push that lands in a different registry, or an old CLI - promoted a version whose image can't be pulled, so every run failed at pull time while the deploy itself reported success. This adds a registry existence check after push and before finalize. If the image isn't there, the deploy fails loudly instead of promoting a version that can't start. The check is ECR-only (a no-op for other registries, so self-hosted setups are unaffected) and uses `BatchGetImage`, which the deploy role already allows. It fails open on an ambiguous registry error so the check can't itself turn into a deploy outage. The image reference is the platform-generated value and the lookup is bound to the configured registry host; the CLI-supplied digest is validated before use. Can be turned off with `DEPLOY_IMAGE_VERIFICATION_ENABLED=0` for setups that push images out of band (e.g. an air-gapped registry the platform can't reach). refs TRI-11243 |
||
|
Eric Allam
|
2fa84ea124 |
feat(webapp): gate worker dequeues by worker queue via env var (#4030)
## Summary Adds a `RUN_ENGINE_DEQUEUE_DISABLED_WORKER_QUEUES` setting that refuses worker dequeue requests for the listed worker queues (or base regions), so their runs stay queued instead of being handed to workers that can't run them. Blocked dequeues are counted via a `run_engine.dequeue.blocked` OTel counter (labeled by `worker_queue` and `region`). |
||
|
Eric Allam
|
8890d7a258 |
feat(run-engine,webapp): always report worker queue length metrics (#4029)
## Summary The `runqueue.workerQueue.length` gauge only reported a worker queue's depth while runs were being dequeued from it. When dequeues stop, the metric goes stale or missing, so a queue that has backed up because nothing is draining it can't be alerted on. This adds a small observer that refreshes the observed set of worker queues from the `WorkerInstanceGroup` records on an interval, so every active worker queue (and its scheduled split variant) keeps reporting its length regardless of dequeue activity. The observer is off by default and enabled per service via `RUN_ENGINE_WORKER_QUEUE_OBSERVER_ENABLED`, reads from the read replica, and skips a configurable set of cloud providers (`RUN_ENGINE_WORKER_QUEUE_OBSERVER_EXCLUDED_CLOUD_PROVIDERS`, default `digitalocean`). When enabled it is the source of truth for the observed set, so the per-dequeue registration is skipped on that instance, and it groups by worker queue so the per-instance duplicates collapse to the true depth. Also removes the unused `GET`/`POST /api/v1/workers` endpoints. Their only consumer was a CLI command group that is no longer registered. ## Verification Verified end to end against a local stack: the gauge reports each worker queue's length with no dequeues happening, excludes the configured providers, includes hidden groups, and the removed endpoints return as if they never existed. Added a run-engine test (`workerQueueObservation.test.ts`). |
||
|
Eric Allam
|
c06005b353 |
feat(webapp,sdk): in-dashboard AI agent (#4018)
## Summary Adds an in-dashboard AI agent: a chat panel, reachable from any environment page, that answers questions about your runs, errors, tasks, and analytics, diagnoses why a run failed, charts your data, reads your connected repo's source, and answers product and how-to questions. It is gated behind the `hasDashboardAgentAccess` feature flag (global or per-org, default off), so this PR ships disabled: the launcher is hidden unless the flag is enabled. ## Design The agent runs as a standalone `chat.agent` Trigger task in its own internal package, with no access to the webapp database, Prisma, or ClickHouse. It reads the user's data over the public API, acting as the user via a short-lived delegated user-actor token minted server-side each turn (never in the browser), building on [#3997](https://github.com/triggerdotdev/trigger.dev/pull/3997). The error and analytics tools use [#4005](https://github.com/triggerdotdev/trigger.dev/pull/4005) and the TRQL query API. The first turn of a new chat streams from a warm webapp route (Head Start) while the durable agent boots in parallel. Structured answers (a run-failure diagnosis card, a live chart) render through a small typed view catalog rather than arbitrary markup. A knowledge lane forwards product and how-to questions to the support assistant. Conversation history lives in a separate Drizzle-backed store on its own Postgres schema, kept as a display read-model so it can never corrupt the agent's model context. The SDK changes add an `apiClient` option to `chat.createStartSessionAction` and `chat.headStart`, and keep the Head Start tool-approval tail intact across a custom `prepareMessages` hook so prompt caching and Head Start compose. |
||
|
nicktrn
|
bf4c6e92bd |
feat(webapp): trace deployment outcomes and compute template creation (#4015)
Deploy success/failure wasn't easily observable: compute template creation only logged, and the terminal deployment statuses (deployed / failed / timed-out) weren't traced — so deploy health couldn't be seen without querying the database. This adds two spans: - `compute.template.create` around template creation at finalize, tagging the resolved mode and per-preset outcome. `resolveMode` now returns its decision (mode + reason) so the span can record why a mode was chosen. - `deployment.outcome` via a small shared helper (`recordDeploymentOutcome`) emitted at every terminal-status write — finalize (deployed), fail / index-failed / background-worker (failed), and timeout (timed out) — so deploy success/failure is queryable by status and reason. The helper is best-effort (org/project/env enrichment where cheaply available) and never throws, so telemetry can't break a deploy. |
||
|
Daniel Sutton
|
65c545da4e |
refactor(run-store,webapp,run-engine): route Postgres TaskRun reads through the run store (#3990)
## Summary Adds read methods to `RunStore` (`findRun`, `findRunOrThrow`, `findRuns`) and routes every Postgres read of `TaskRun` through them, mirroring how writes already go through the store. Behavior-preserving: each relocated read keeps its exact query, field selection, and database client (writer, replica, or transaction). This lets `TaskRun` reads be retargeted to a different backing store later without touching call sites. Stacked on #3981 (the write adapter); that PR is the base of this one. ## Scope In scope: the run engine, webapp services, presenters, and route loaders. Three reads that pulled `TaskRun` in through a parent model's relation `include` (alert delivery, batch results, attempt-dependency cancellation) are decomposed to fetch the run(s) through the store and stitch them back, since a relation include would not follow `TaskRun` to a new table. Left reading the existing table (out of scope): the legacy MarQS paths, the legacy trigger idempotency read, and one raw-SQL recovery script (commented for revisiting at cutover). ## Notes Reads default to the read replica; callers pass the writer or a transaction client wherever the original read did, so writer-vs-replica behavior is unchanged. |
||
|
Eric Allam
|
5052d895b3 |
feat(webapp,core): add a public HTTP API for errors (#4005)
## Summary
Adds an environment-scoped HTTP API over the Errors feature, mirroring
the runs API. Task-run failures are grouped by a fingerprint into "error
groups," and this exposes everything you can do with them in the
dashboard:
- `GET /api/v1/errors` lists error groups, with
`filter[taskIdentifier]`, `filter[version]`, `filter[status]`
(`unresolved`/`resolved`/`ignored`), `filter[search]`, a time range, and
cursor pagination.
- `GET /api/v1/errors/{errorId}` retrieves a single group (summary,
lifecycle state, affected versions).
- `POST /api/v1/errors/{errorId}/{resolve,ignore,unresolve}` changes its
state.
- `GET /api/v1/runs?filter[error]={errorId}` lists the runs behind a
group.
Request and response schemas are exported from `@trigger.dev/core/v3` so
the SDK can reuse them, and all endpoints are documented in the API
reference (OpenAPI). `errorId` is the `error_<fingerprint>` friendly id.
## Attribution
State changes record who made them. A plain environment API key has no
user, so `resolvedBy`/`ignoredByUserId` stay null. When the caller uses
an environment JWT obtained by exchanging a personal access token or a
delegated user token at `POST /api/v1/projects/:ref/:env/jwt`, that
exchange now stamps an `act` delegation claim, and the write endpoints
read `act.sub` to attribute the change to the acting user. This is the
first endpoint to consume the `act` claim, so two small pieces of
plumbing ride along: the exchange stamps `act` for personal-access-token
subjects too (it was delegated-token-only), and the public-JWT
bearer-auth path surfaces `act.sub` to the handler.
Built on the delegated-token work in #3997.
|
||
|
Daniel Sutton
|
315baf2e54 | refactor(run-engine,webapp): route TaskRun writes through a new RunStore adapter (#3981) | ||
|
Oskar Otwinowski
|
e98a547e6c | feat(sso): SAML/OIDC single sign-on (#3911) | ||
|
Iss
|
c97d246197 |
feat(webapp): sync new orgs + users to Attio CRM on signup (#3896)
Pushes new organizations and users into the Attio CRM at signup time, for Customer Success (TRI-10431). - Orgs → Attio `workspaces`, users → Attio `users`, keyed on Attio's built-in unique `workspace_id` / `user_id` so writes are idempotent upserts. - Runs on the common Redis worker (not inline), so a slow or unavailable Attio never blocks the signup path; failures retry (3 attempts). - Hooks: user-created (alongside the existing Loops call) and org-created (`createOrganization`). - Gated behind `ATTIO_API_KEY`, no key means the sync is skipped entirely, so OSS / self-hosted installs are unaffected. Only creation is covered here (the record "shell"); spend, runs, plan changes, churn, and role/relationship linking are populated by the scheduled full sync, tracked separately. **Deploy note:** requires an Attio API key set as `ATTIO_API_KEY` in the webapp env, with scopes **Records (read-write)** + **Object Configuration (read)**, the assert/upsert endpoint reads object config to resolve the matching attribute. Without the key the sync no-ops. --------- Co-authored-by: Matt Aitken <matt@mattaitken.com> Co-authored-by: devin-ai-integration[bot] <158243242+devin-ai-integration[bot]@users.noreply.github.com> |
||
|
nicktrn
|
ae08c9cb60 |
fix(webapp): admin feature flag number inputs and scrolling (#3979)
The global feature flags admin page had a few rough edges. The percentage flags are numeric (`z.coerce.number()`) but rendered as free-text inputs, so you could type non-numeric values that only failed validation after submitting - and the error surfaced behind the confirm dialog. The control-type detection now recognises numbers and renders a proper number input, with the min/max range as the placeholder so the type is clear even when the field is unset. The save error also shows inside the confirm dialog now, not just behind it. The action buttons were unreachable without zooming out. The admin layout wrapped each page in a plain block, so `h-full` page content overran the viewport by the height of the tab bar and got clipped by the `overflow-hidden` body. Making the layout a flex column bounds each page to the space below the tabs, so the existing per-page scroll works and the feature flags page scrolls like the Users/Orgs tabs. Also capped the confirm dialog's diff list so its footer stays on screen when there are many changes. |
||
|
nicktrn
|
7aa871f37b |
feat(webapp): plan-aware compute migration (#3957)
Adds an opt-in mechanism to route a configurable percentage of organizations onto the compute (MicroVM) backing of their region at trigger time, without changing their stored region settings. Routing is gated by three global feature flags - `computeMigrationEnabled`, `computeMigrationFreePercentage`, `computeMigrationPaidPercentage` - plus a per-org `computeMigrationEnabled` override that wins in both directions. A region's compute backing is resolved from a new `WorkerInstanceGroup.region` column: a container group and its MicroVM group share one geo `region`, so the migration swaps the resolved worker queue to the backing group's queue. Orgs are bucketed deterministically by id, so ramping a percentage down keeps a strict subset rather than reshuffling, and a region with no compute backing is never touched. Everything is off by default - behaviour is unchanged unless the flags are set. The flags and the worker-region groups are read on the trigger hot path from in-memory snapshots rather than the database: a small `createReloadingRegistry` helper loads each at startup and refreshes them on an interval, so no per-trigger query is added and a percentage or kill-switch change propagates within the reload interval. A cold replica whose snapshot hasn't loaded yet reads as not-migrated (the container path) and self-corrects on the next load - the same cold-start contract as the datastore / LLM-pricing registries, with a `reloading_registry_loaded` metric so a never-loaded registry is alertable. The same migration decision is consulted at deploy-time template creation so a migrated org gets a compute template built ahead of its first run. This runs in shadow mode (best-effort, never fails the deploy) by default, or - when the `computeMigrationRequireTemplate` flag is on - in required mode, built synchronously at deploy so the first run never builds on-demand and template errors surface at deploy time. So operators keep "which runs ran where" while customers only see geography: the run's actual worker queue is stored raw, and the geo region is stamped separately on `TaskRun.region` (and a new ClickHouse `region` column) at trigger time. Read surfaces - the dashboard, the API, and the Query/Logs page - show the geo region, falling back to the worker queue for runs written before the column existed. Minor follow-ups left out of scope: the percentage flags render as text inputs on the admin flags page (the catalog UI has no numeric control type yet), and `createReloadingRegistry` could later gain pub/sub for sub-second cross-replica propagation if the reload interval proves too slow. |
||
|
Eric Allam
|
07a0e4ade9 |
feat(webapp): split Models into Your models and Model library tabs (#3958)
## Summary The Models page is now split into two tabs. **Your models** shows the models your project has actually used in the selected time range, with usage charts (cost over time, tokens over time, calls by model), a per-model table of calls / cost / avg TTFC / avg tokens-per-sec, and calls/tokens trend sparklines. **Model library** is the full catalog, reordered from alphabetical to a relevance-based provider order (Anthropic, OpenAI, Google, then the rest), newest models first within each provider, with a "New" badge on models released in the last 7 days. One time-range selector drives the whole Your models tab, so the charts, the table, and the sparklines all share the same window. Opening a model shows its own metrics with an independent range picker and a "View in AI metrics" link that opens the AI metrics dashboard filtered to that model. The active tab is kept in the URL so it survives a refresh and is shareable. ## Prompt caching & cost accuracy Both the Your models tab and the AI metrics dashboard now surface prompt-cache usage: a cache-savings column plus per-model cached-tokens and cache-hit-rate views, and a caching section on the dashboard (hit rate, cached tokens, estimated savings, and hit rate by model). Building this surfaced a cost bug. `input_tokens` is the total prompt count and already includes cache-read and cache-creation tokens, but the cost pipeline charged the full input at the input price and then added a separate cache line, so cached tokens were billed twice (and on Anthropic, cache reads were never discounted because their price is keyed differently). The input price now applies only to the non-cached remainder, with cache prices resolved across the provider-specific keys, so LLM cost and the cache hit-rate metric are accurate. Hit rate is computed as cached reads over total input. ## Notes Also fixes React "invalid DOM property" console warnings from the provider icons (the Llama and DeepSeek SVGs used raw `fill-rule` / `clip-rule` / `clip-path` attributes), which this page surfaces by rendering more provider icons. ## Screenshots **Your models tab:** usage charts and a per-model table with calls/tokens trend sparklines. <img width="2560" height="1267" alt="1-your-models-tab" src="https://github.com/user-attachments/assets/859bd24f-9047-4828-8bbb-83e5882846d6" /> **Model library:** provider-relevance ordering with a "New" badge on models released in the last 7 days. <img width="2560" height="1267" alt="2-model-library-tab" src="https://github.com/user-attachments/assets/46dd54b9-80f9-4922-ade9-5935b08dfebc" /> **Model detail, Metrics tab:** per-model range picker and a "View in AI metrics" link. <img width="2560" height="1267" alt="3-model-detail-metrics" src="https://github.com/user-attachments/assets/0f65d9d0-6142-4918-93f0-110bb277101a" /> **View in AI metrics:** the dashboard deep-linked and filtered to the selected model. <img width="2560" height="1267" alt="4-ai-metrics-filtered" src="https://github.com/user-attachments/assets/821f256c-e305-493c-98c7-eafaf2f57f83" /> |
||
|
Oskar Otwinowski
|
cf4aa7e918 |
fix(webapp): Vercel env var sync rejecting batches containing only reserved keys (#3966)
Fix Vercel onboarding wizard to properly filter out reserved TRIGGER_ env vars |
||
|
James Ritchie
|
af526dea18 |
feat(webapp): chat AI UI improvements, new task landing pages and side menu (#3941)
Major dashboard restructure plus the new task landing pages and self-serve schedules add-on integration. ## Side menu - Full restructure: standalone Tasks / Runs / Sessions block at the top; new collapsible sections for AI, Observability, Deployments, Manage - Persisted collapse state per section in `dashboardPreferences` - New / updated icons across the menu - Dashboards section: built-in Run metrics + AI metrics + custom dashboards, with drag-to-reorder via ReactGridLayout (`DashboardList.tsx`) - DevPresence connection indicator in the env selector (DEV + V2) ## Tasks (`_index` — unified Tasks page) - Replaces the separated Agents / Standard / Schedules listing pages with one table - New `UnifiedTaskListPresenter` composes `TaskListPresenter` + `AgentListPresenter` (shared `currentWorker` lookup) - Columns: Type (with kind badge), ID, File, Running (numeric for tasks; running + suspended pills for agents), Activity (24h stacked-by-status), sticky menu - Search + "Task type" multi-select filter (URL-synced) - Client-side pagination at 25/page - Right-hand "useful links" panel (cookie-persisted state) - Live-reload SSE: page revalidates on `WORKER_CREATED` so onboarding `trigger dev` flips the blank state automatically ## Agent landing page (`/agents/$agentParam`) - New per-agent detail page - Top tabs (Sessions / Runs) toggle both the chart panel and the table - Three dashboard-style chart cards: Sessions/Runs activity, LLM spend, Tokens - `AgentDetailPresenter` queries ClickHouse for run activity, session activity (with FINAL on `sessions_v1`), and LLM cost/token activity from `llm_metrics_v1` - TimeFilter at the top drives all three charts - Sticky table header, resizable horizontal handle, sidebar with Test agent button + properties - Docs link → `ai-chat/overview` ## Standard Task landing page (`/tasks/standard/$taskParam`) - New per-task detail page mirroring the Agent layout - `TaskDetailPresenter` for activity + properties - Chart panel wrapped in a Card with "Runs by status" header - Top bar with title, TimeFilter, pagination - Right sidebar: Test task + identifier, queue, machine, retry, TTL, payload schema, etc. ## Scheduled Task landing page (`/tasks/scheduled/$taskParam`) - New per-task detail page mirroring the Agent / Standard layout - Top-bar actions (right → left): pagination, Bulk replay…, View all runs, TimeFilter, Create schedule - Connected schedules mini-table in the sidebar - **Self-serve schedules add-on integration** (reincarnated from the now-removed `/schedules` listing page during the `origin/main` merge): - Bottom usage bar pinned via `grid-rows-[auto_1fr_auto]` — progress ring + "X/Y of your schedules" + Purchase / Upgrade / Request CTA - At-limit "Create schedule" intercept dialog - `PurchaseSchedulesModal` extracted as a shared component (`apps/webapp/app/components/schedules/PurchaseSchedulesModal.tsx`) handling increase / decrease / above-quota / need-to-delete states - New resource action route at `/resources/orgs/$organizationSlug/schedules-addon` ## Sessions - Index page: list, filters, blank state, help tooltip rework - Detail page: combined input/output chronological view (replaces split tabs) - Improved raw-message view layout (full-height) - AI payload UI: `data-*` parts grouped under "AI SDK data parts:" label - `toSafeUrl` helper guards rendered URLs from streamed content - Fix: duplicate assistant content on inspector tab switch ## Playground (Test agent) - Restructured top menu; back button + agent-selector popover - Improved blank state - Recent agent chat history moved into the tabbed menu - Better message-scroll container (full height) ## Dashboards - New Dashboards landing page (`/dashboards`) — Run metrics, AI metrics, Create your own CTAs - `BuiltInDashboards` updated; new `TasksDashboardPresenter` for the tasks overview - Custom dashboards section gains drag-to-reorder; cosmetic fix for active-row drag-handle blending ## PageHeader / shared primitives - `PageTitle` gains an `accessory` prop supporting string (auto-wrapped in tooltip) and ReactNode - Help tooltips on Tasks, Runs, Sessions PageTitles explaining the concept and sub-categories - `Card` primitive used for dashboard-style chart panels throughout ## Code review fixes (last batch on this branch) - ClickHouse activity queries hardened: `FINAL` + `_is_deleted = 0` on `task_runs_v2` (ReplacingMergeTree); `organization_id` + `project_id` filters for sort-key prefix; `inserted_at` partition filter on `llm_metrics_v1` - `UnifiedTaskListPresenter`: shared `currentWorker` lookup; slug-collision guard in `mergeRunningStates`; off-by-one fixed in 24h bucket alignment - `ScheduleListPresenter`: halved platform RPCs by deriving limit from `currentPlan` instead of calling `getLimit` - Sessions detail: stopped IntersectionObserver / scroll listener re-attach on every chunk; `requestAnimationFrame` deferral on auto-scroll to avoid virtualizer race - URL hardening: `?types=` validated against known kinds; new `parseFiniteInt` helper applied to `from`/`to`/`page` params - AgentView: HITL resolution buffer now cleared once parts reach a terminal state (was an unbounded Map on long sessions); subscription effect deps documented with eslint suppression - `PurchaseSchedulesModal`: bundle state resets on each open instead of persisting stale drafts ## Manual testing Manual smoke-test plan is tracked under [TRI-10883](https://linear.app/triggerdotdev/issue/TRI-10883), broken into 20 sub-issues covering onboarding, self-serve schedules, side menu, the four landing pages, sessions, runs, dashboards, regressions and performance. 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com> |
||
|
Daniel Sutton
|
a7312b1b86 |
fix(webapp): stop logging expected auth/restore conditions as errors (#3931)
Two expected, non-failure conditions were being logged at `error` level, which surfaces them as exceptions in error tracking and adds noise without signal. This downgrades both to `warn`. The first is the checkpoint-restore path when a `RESTORE` event already exists for a checkpoint — a benign idempotency skip on a duplicate or retried event. The second is the `/api/v1/token` endpoint when the authorization code is invalid or expired, which is the expected steady state while the CLI polls the endpoint during login; genuinely unexpected failures there still log at `error`. No behavior or response changes — the token endpoint still returns 400 in the same cases. |
||
|
Iss
|
002c441f50 |
feat(webapp): self serve schedules add-on (#3811)
Adds the purchase UI for extra schedules, mirroring preview branches ## Changes - `setSchedulesAddOn` platform client + `SetSchedulesAddOnService` (purchase + quota-increase via Plain). - `ScheduleListPresenter` surfaces add-on / quota / pricing; `checkSchedule` counts purchased schedules toward the limit (`base + purchased`). - `PurchaseSchedulesModal` on the Schedules page — bought in **bundles of 1,000 ($10/mo each)**; bundle increments enforced client-side and in the action's zod schema. |
||
|
Eric Allam
|
954ee5c572 |
fix(webapp): deliver realtime changes with current content when the read replica lags (#3910)
## Summary When the realtime runs feed (the backend behind the `realtimeBackend` feature flag) hydrates a change from a Postgres read replica, the read can race the replica's apply of the very write that triggered it. The delivered row then carries the previous change's content, and an isolated final change (for example a last `metadata.set` before a run goes quiet) is not corrected until the roughly 20 second backstop poll. Measured against a replica with deliberate apply delay, every delivery trailed exactly one change behind and a final change stranded for the full backstop interval. ## Fix Publishers stamp each change record with the committed row's `updatedAt`, taken from writes they already perform, so the stamp costs no extra queries. The router delays its wake hydrate until the replica's measured lag has passed, anchored to that timestamp: a record that has already spent longer than the lag in transit is hydrated immediately, so only the racing leading edge ever waits. After hydrating, a tripwire compares each row against its record's watermark. Still-stale rows are withheld and retried briefly, and each detection feeds the lag estimate. If retries run out, the rows are delivered anyway (liveness over freshness) and follow-up re-hydrates emit the fresh version through the normal working-set diff once the replica catches up, with the backstop as the terminal net. Replica lag is sampled reader-side only, and only while feeds are active. Aurora reports live lag via `aurora_replica_status()`; vanilla Postgres can only report "caught up or not" (mid-apply lag is not honestly measurable from a replica), so tripwire observations floor the estimate there. Deployments without a replica resolve to zero lag and skip the gate entirely. Tunables live under `REALTIME_BACKEND_NATIVE_REPLICA_LAG_*`, and `realtime_native.stale_hydrates` plus `realtime_native.replica_lag_estimate_ms` make replica health observable. Two adjacent fixes: a metadata update that writes nothing no longer publishes a change record, and buffered parent and root metadata operations now publish when the flusher writes them, so those changes wake live feeds instead of waiting for the backstop. For local testing, `docker-compose` gains an opt-in `database-replica` service (compose profile `replica`) with a configurable `recovery_min_apply_delay`, which reproduces replica-lag behavior deterministically. With the gate disabled this rig reproduces the one-change-behind delivery exactly; with it enabled, deliveries arrive with current content at roughly the true replica lag, across write rates faster and slower than the lag itself. |
||
|
Eric Allam
|
8dc77c0ccd |
fix(webapp): only load env var values for displayed environments (#3903)
## Summary The environment variables page loaded every variable value in the project, unfiltered by environment. Archiving a preview branch does not delete its environment variable value rows, so projects that churn preview branches accumulate values forever, and every page view loaded all of them. On large projects this made the page loader take many seconds and stalled the server while deserializing the oversized result. ## Fix The presenter now loads the displayed environments first and filters the `values` relation to those environment IDs. That matches the display semantics exactly (per-user dev environments and active branch environments included), and the lookup is covered by the existing unique index on `(variableId, environmentId)`. Values in archived branch environments are no longer fetched at all. Covered by a new testcontainers test asserting that values from active environments (including branch environments) are returned while archived branch environments are excluded. |
||
|
Eric Allam
|
187c0476c3 |
perf(webapp): shrink run trace loader payload and add trace span cap controls (#3906)
## Summary The run trace page loader serialized every span's raw OTel events (with full properties) into the response, even though the tree UI only renders the derived `timelineEvents` and the span detail panel refetches what it needs. On event-heavy traces that inflated both the loader payload and the server-side heap copies built per request. This PR keeps raw span events server-side and pairs that with a few related trace-view improvements: - A new optional `TRACE_VIEW_EMERGENCY_SPAN_CAP` env var (unset by default) clamps the trace summary and detailed trace summary span limits on both event store paths, including the public run trace endpoint, so operators can bound trace query sizes in one place without retuning the per-store limits. - The TreeView virtualizer resolved every rendered row with a linear scan over the whole tree (and `getNodeProps` did the same via `findIndex`); rows now resolve through memoized id lookup maps, which matters once traces reach tens of thousands of spans. - The run stream SSE lookup now applies the same organization membership scoping as the rest of the run page presenters, for consistency. Behavior is unchanged by default: the trace tree renders from the same `timelineEvents` it always has, and the new cap only takes effect when set. |
||
|
Eric Allam
|
f9d57d3bd5 |
feat(webapp): add a new backend for the realtime runs feed (#3864)
## Summary Adds a second backend for the realtime runs feed (`useRealtimeRun`, `subscribeToRunsWithTag`, `subscribeToBatch`), built to stay healthy when a single busy environment has many subscribers watching many runs at once. It is gated behind a feature flag with the existing backend as the default, so nothing changes for users until it is enabled per environment. ## Design A run change is published once, as a small self-describing record, to a single per-environment channel. Every feed is then a predicate over that one stream rather than owning a channel: - A per-instance router indexes the currently-held feeds by run, tag, and batch. When a run changes it hydrates the affected rows once and serializes them once, then fans the result to every matching feed. One hot shared tag watched by many subscribers costs a single database query and serialize, not one per subscriber. - Feeds that don't match a change are never woken, wake delivery per environment is coalesced on a leading edge (250ms default) so a burst of changes costs one wake, and cold reads coalesce onto a single short-TTL-cached resolve. - An admission gate bounds how many cold ClickHouse resolves run concurrently, so a mass reconnect across many distinct filters queues instead of stampeding the database. - Changes that land while a client is between long-polls are delivered on its next poll instead of waiting for the periodic backstop: each environment buffers its recent change records, subscriptions linger briefly after the last feed closes, and a newly-armed poll replays exactly the connection's gap. - The per-connection replay cursors behind that are shared across instances via Redis (a single timestamp each), so a poll landing on a different instance behind the load balancer still reads the connection's true gap instead of falling back to a cold resolve. Cursor reads have a bounded deadline and degrade to the cold-read path on any Redis trouble. - Tag subscriptions with multiple tags match runs carrying all of the tags, mirroring the existing backend's filter semantics, and live long-polls hold for about 20 seconds to match its cadence. - The per-environment channel supports Redis Cluster sharded pub/sub, so the wake path scales horizontally across shards by environment. - The backend reports its health through OpenTelemetry metrics (delivery lag, poll resolution paths, backstop outcomes, replay and cursor-store activity), with a provisioned Grafana dashboard for local development. Everything is behind the feature flag and tunable via env vars; the existing backend remains the default. |
||
|
Daniel Sutton
|
6afc9bfa4c |
fix(run-engine): retry getSnapshotsSince on the replica then primary when the read replica lags (#3889)
## Summary
When `RUN_ENGINE_READ_REPLICA_SNAPSHOTS_SINCE_ENABLED` is on,
`RunEngine.getSnapshotsSince` reads from the read replica. During write
spikes the replica can briefly lag, so the snapshot id a runner just
learned from the writer isn't visible there yet: the lookup threw, the
worker route returned a 500, and the runner waited for its next poll —
turning sub-second snapshot notifications into poll-interval latency
exactly when things are busiest. This PR makes the flag safe to enable:
a replica miss of the since snapshot gets one jittered retry on the
replica (most lag windows are shorter than the ~50–200ms wait, so the
writer is never touched), then falls back to the primary, observed via a
new `run_engine.snapshots_since.replica_miss` counter with an `outcome`
attribute (`replica_retry` vs `primary`). Only genuine misses — absent
on the primary too — remain errors.
## Design
- `getExecutionSnapshotsSince` now throws a typed
`ExecutionSnapshotNotFoundError` so the engine can distinguish the
expected lag miss from real failures. The message string is unchanged
and the error never leaves the engine.
- The recovery path only engages when the flag is on, a distinct replica
client is configured, and no transaction client was passed. With the
flag off, the path is behaviorally identical to before.
- Retry delay bounds are configurable
(`RUN_ENGINE_SNAPSHOTS_SINCE_REPLICA_RETRY_MIN_MS`/`MAX_MS`, default
50/200; `MAX_MS=0` skips the replica retry and goes straight to the
primary).
- The warn log fires only when the primary serves the read (the writer
spill is the operationally interesting event); replica-retry recoveries
are counted but quiet. A permanently-missing snapshot id stays an
error-level failure with a `failedDuring` field, so lag metrics aren't
polluted by bogus ids.
- Stale-tail lag (replica has the since snapshot but not newer rows)
deliberately still returns the replica's view; the next poll catches up.
- The since-snapshot anchor lookup is now scoped to the polled run
(`where: { id, runId }`), so a snapshot id from a different run raises
not-found instead of silently anchoring a too-wide window of the run's
snapshots.
## Test plan
All vitest + testcontainers, no mocks. A new `schemaOnlyPrisma` fixture
(migrated-but-empty clone database) simulates a replica that hasn't
caught up, and a real in-memory OTel meter pins the counter semantics
per outcome.
- [x] Replica catches up during the jittered retry window → served by
the replica, `outcome=replica_retry` = 1, primary never consulted
- [x] Replica permanently missing the since snapshot → served by the
primary, `outcome=primary` = 1
- [x] Snapshot missing on both replica and primary → null, counter = 0
- [x] Replica has the since snapshot but lags by one → the replica's
view is served, no fallback (verified discriminating power: the test
fails if reads secretly hit the primary)
- [x] Flag off with a replica configured → primary serves the read
- [x] Transaction client provided → bypasses the replica entirely
- [x] Since snapshot belonging to a different run → null
- [x] Existing getSnapshotsSince + waitpoints suites green; run-engine,
testcontainers, and webapp typechecks pass
🤖 Generated with [Claude Code](https://claude.com/claude-code)
---------
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
|
||
|
Matt Aitken
|
f4a96bdf84 |
Fail dev and deploy on duplicate task ids (#3865)
## What `dev` and `deploy` now fail with a clear error when two tasks are defined with the same id — including across task types (e.g. a scheduled task and a regular task sharing an id). ## Why Tasks are registered into the resource catalog keyed by id, so a second definition with the same id silently overwrote the first. One of the tasks would just vanish from the worker with no warning — easy to miss, hard to debug. (Any earlier duplicate-id check ran against the post-registration task list, which is already de-duplicated, so it never actually fired.) ## How - **Detect at registration** (`@trigger.dev/core`): `StandardResourceCatalog` records a collision when a task id is registered more than once, capturing the files involved — the only point where both definitions are visible before the id-keyed map collapses them. Exposed via `listTaskIdCollisions()`. - **Fail indexing** (`trigger.dev` CLI): both index workers report collisions via a new `TASKS_FAILED_TO_INDEX` message; `indexWorkerManifest` rejects with a new `DuplicateTaskIdsError`. `dev` renders a dedicated error (offending ids + files + docs link); `deploy` fails with the same message. Runtime worker boot is unaffected — it never reads the collisions. - **Server-side backstop** (webapp): background-worker registration also rejects duplicate ids with a clear `ServiceValidationError`, so duplicates are caught even from an older CLI. ## Testing - Unit tests for collision collection in the catalog and for the error-message formatting (standard, same-file, and 3+-definition cases). - Verified end to end against a local webapp: a project with a regular task and a scheduled task sharing an id now fails `dev` with the dedicated error; a project with distinct ids still starts normally. ## Changeset Patch for `@trigger.dev/core` and `trigger.dev`. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com> |
||
|
Eric Allam
|
0ee1461d86 |
fix(webapp): show scheduled runs under their correct region (#3873)
## Summary Runs routed onto a dedicated scheduled worker queue were showing under a phantom region in the dashboard, run details, and the API, and slipped through region filters. They now resolve to their real region everywhere. ## Fix A worker queue can carry a `:scheduled` suffix that routes scheduled-lineage runs onto their own list. That suffix is an internal routing detail, but it was leaking anywhere the worker queue is read as a region. A `baseWorkerQueue` helper strips any `:<class>` suffix back to the base region (region names never contain a colon, so it's everything before the first colon) and is applied at every region read site: the runs list, run detail, the public API, and replay's region override. The runs-replication writer also stores the base region in ClickHouse so the region filter matches. |
||
|
Daniel Sutton
|
d7028e278e |
feat(webapp): label mollifier decisions by enrolled org (#3869)
## Summary
The `mollifier.decisions` metric only carried an `outcome` label, so for
an org that has the mollifier enabled there was no way to see how often
its triggers pass through the gate instead of being diverted — making it
hard to tell why the trip isn't firing for an opted-in org.
This adds two bounded labels: `enrolled` (`"true"`/`"false"`, the
per-org flag) and `org` (the org id, attached **only** when `enrolled`
is true). For an enrolled org you can now compare directly:
`mollifier.decisions{outcome="pass_through", enrolled="true",
org="<id>"}` vs `{outcome="mollify", ...}`.
## Design
`recordDecision` now takes an options object (`{ reason?, enrolled,
orgId? }`). The `org` label is restricted to the enrolled cohort to keep
cardinality bounded — the guard lives in a pure `decisionLabels` helper,
so a non-enrolled org id can never be attached even if one is passed.
The enrolled set is small and capped operationally.
The per-org flag is resolved once at the top of `evaluateGate`
(in-memory, no DB round-trip on the trigger hot path) so every decision
— including the debounce / one-time-use-token / triggerAndWait bypasses
— is labelled consistently.
## Test plan
- [x] `mollifierGate.test.ts` cascade asserts `enrolled`/`org` on every
gate branch
- [x] `mollifierDecisionLabels.test.ts` (new) proves `org` is dropped
for non-enrolled even when an id is passed (cardinality guard)
- [x] `vitest run mollifierGate mollifierDecisionLabels` — 34/34 pass
- [x] `pnpm run typecheck --filter webapp` clean
Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
|
||
|
Matt Aitken
|
ef04cc39ef |
fix(webapp): use composite keyset cursor for run pagination (#3852)
## Problem
`ClickHouseRunsRepository.listRunIds` / `listRuns` order results by the
composite key `(created_at, run_id)`, but the cursor predicate cut on
`run_id` **alone**:
```ts
.where("run_id < {runId: String}", { runId: cursor })
.orderBy("created_at DESC, run_id DESC")
```
This is only sound when `run_id` lexicographic order matches
`created_at` order. `run_id`s are cuids — only coarsely time-sortable —
so when a burst of runs is created within a sub-second window, the two
orders can diverge. When they do, the next-page predicate (`run_id <
cursor`, where `cursor` is the *last* page element = the smallest
`created_at`, not necessarily the smallest `run_id`):
- **re-includes** rows already returned on a previous page (duplicates),
and
- **skips** rows it should have returned (silent data loss).
For bulk **replay** this caused runs to be replayed more than once
(replay has no idempotency guard). For the dashboard and the `runs.list`
API it could silently repeat or skip runs at page boundaries.
## Fix
Make the cursor predicate match the composite ordering:
- Cursors now encode the full `(created_at, run_id)` key as an **opaque
URL-safe base64 token**
(`base64url({"c":<createdAtMs>,"r":"<runId>"})`), and the query cuts on
the matching tuple — `(created_at, run_id) < (…)` forward / `> (…)`
backward.
- The `ORDER BY` is unchanged, so the query stays aligned with the
table's primary key — no performance regression (the tuple range
predicate is actually more index-friendly than `run_id <` alone).
- Cursors are **server-issued opaque tokens** (the SDK only echoes
`pagination.next` / `pagination.previous` back), so this needs **no
client/SDK update**. Legacy cursors were the bare internal `run_id`;
they're detected by decode failure (a cuid isn't a valid base64-wrapped
JSON payload) and fall back to the old `run_id`-only predicate, so
in-flight cursors keep working and drain naturally. New cursors also no
longer expose a bare internal run id.
- `listRunIds` is now the single cursor-aware list primitive: it returns
`{ runIds, pagination: { nextCursor, previousCursor } }`, and `listRuns`
builds on it (one place constructs cursors). Bulk actions consume the
same method and advance by `pagination.nextCursor`, finishing when it's
`null`.
- `getTaskRunsQueryBuilder` now also selects
`toUnixTimestamp64Milli(created_at) AS created_at_ms`, using a dedicated
`TaskRunListQueryResult` schema. The shared `TaskRunV2QueryResult` stays
`run_id`-only so the run-engine pending-version lookup
(`getPendingVersionIdsQueryBuilder`, which selects only `run_id`)
doesn't fail validation on a column it doesn't query.
## Tests
New `runsRepositoryCursor.test.ts` (testcontainer-backed, real
Postgres→ClickHouse replication):
- **forward** pagination returns every run exactly once when `run_id`
order is the reverse of `created_at` order (reproduces the
duplicate/skip bug — fails on `main`; this
walk-until-`nextCursor`-null-and-assert-complete is exactly the bulk
action's iteration),
- **backward** pagination round-trips to the previous page across a
boundary,
- **legacy** bare-`run_id` cursor still uses the old predicate
(backwards compatibility).
The existing `runsRepository` suites (part1–4) still pass; `part4`'s
`count new runs with listRunIds` test was updated for the new `{ runIds,
pagination }` return shape, and the `clickhouse` `taskRuns`
query-builder snapshots were regenerated for the added `created_at_ms`
column.
## Notes
- Separate, pre-existing issue (out of scope, not introduced here):
`listRuns`' backward display-slicing (`rows.slice(1, size+1)` when
`hasMore`) has an off-by-one that can return a straddled page. Tracked
separately.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
|
||
|
Eric Allam
|
97036fb741 |
feat(webapp,clickhouse): export run traces as log, markdown, or jsonl (#3851)
## Summary Adds a trace export to the run page. From the new **Export trace** menu you can copy a run's full trace to the clipboard as Markdown (for pasting into an AI assistant) or download it as a flat Log, a Markdown table, or JSON Lines. Internal engine-debug events are filtered out by default, and errors are surfaced inline with their message. ## Design The export streams events from the store to the gzipped response one at a time and never materialises the span tree, so a trace of any size exports with bounded memory and without stalling the server. Output is flat and chronological: each line carries its own `spanId ← parentSpanId`, so the hierarchy is reconstructable without nesting. Formats share a single streaming pipeline and are pluggable via `?format=log|jsonl|markdown`, so adding a format is an isolated change. ## Screenshots **Export menu** <img width="370" height="252" alt="trace-export-menu" src="https://github.com/user-attachments/assets/3d10304a-8c49-4606-b15d-2859b137419f" /> **In context** <img width="2400" height="1802" alt="trace-export-run-page" src="https://github.com/user-attachments/assets/46c80b30-303b-47c6-9ace-a2fb06f6cb61" /> |
||
|
Eric Allam
|
85886b96da |
feat(webapp,supervisor): isolate scheduled runs on a dedicated worker queue (#3839)
## Summary Scheduled runs and their descendants can now be routed to a dedicated per-region worker queue, processed by a separate worker fleet, so a burst of scheduled crons no longer competes with standard and agent runs for the same queue and inflates their startup latency. It is off by default and enabled per organization via a feature flag (with a global default), so nothing changes until it is turned on. ## Design At trigger time, any run whose lineage originates from a schedule (`rootTriggerSource === "schedule"`, which already propagates from a scheduled run down to all of its children) gets its worker queue suffixed with `:scheduled`. The worker queue name is an opaque string persisted on the run and used verbatim by enqueue and dequeue, so this needs no Lua, message-envelope, or concurrency changes. Concurrency stays keyed by environment and queue, not by worker queue. On the consumer side, the dequeue endpoint gains an optional `queueClass` selector. A supervisor sends `queueClass: "scheduled"` and the server derives the actual queue from the worker's own group, so a token can only ever reach its own region's queues. A fleet picks its class with the `TRIGGER_WORKER_QUEUE_CLASS` env var (`default` or `scheduled`), so a dedicated scheduled fleet can run alongside the standard one. Verified end to end against a local managed-worker setup: scheduled runs route to the dedicated queue, are drained only by the scheduled fleet, and standard runs are left untouched. |
||
|
Katia Bulatova
|
db4074df54 |
fix(webapp): validate packet storage paths (#3830)
## Summary This PR adds packet path validation before key construction and presigning. Invalid paths are rejected before reaching either object-store client implementation, ensuring consistent behavior regardless of the underlying storage configuration. |
||
|
Eric Allam
|
8d5cf313bb |
fix(webapp): fix AI agent dashboard rendering and snapshot loads (#3834)
## Summary Three fixes to the AI agent surface in the dashboard, all surfaced while verifying AI SDK 7 support (the SDK side is #3833): - **AI SDK 7 telemetry rendering.** The generation-span inspector and run metrics now read both the v6 (`ai.*`) and v7 (`gen_ai.*`) telemetry attribute shapes. On v7 the Messages, Provider, and Tools views showed empty/unknown before; now they render correctly. - **HITL approvals in the conversation view.** The agent conversation view renders human-in-the-loop tool approvals and denials (awaiting approval, approved, denied with reason) instead of leaving the tool part blank. This gap predated v7. - **Chat snapshot store mismatch.** Chat session snapshots now resolve through a single storage-key helper shared by the SDK write and the dashboard read. Previously the write applied the default object-store protocol to the key while the read fell back to a different store, so the dashboard 404'd on the snapshot and showed only a partial conversation. |
||
|
Eric Allam
|
64151d6ac8 |
chore(webapp): reduce telemetry ingestion log volume (#3832)
## Summary On a busy webapp the trace/log/metric ingestion path emits several `info` logs per insert batch, which makes up the bulk of the service's log output. This moves that per-batch chatter to `debug` and adds an opt-in to drop successful HTTP access logs, cutting log volume with no loss of error signal. ## Details The per-batch ClickHouse insert logs, the flush scheduler's concurrency adjustments, and the event-loop utilization sample (already exported as a metric, so the log line was redundant) now log at `debug`. Error and warning logs are untouched. New `HTTP_ACCESS_LOG_DISABLED=1` env var: when set, the HTTP access logger skips successful (2xx) requests while still logging non-2xx responses. Defaults off, so existing deployments are unchanged. |
||
|
Katia Bulatova
|
cae3dcb7dd |
Env vars page performance fix (#3829)
## Summary This PR improves performance across the Environment Variables page. ## Changes ### Targeted value loading - load only the non-secret (environmentId, key) pairs required by the page. Secret values continue to be redacted in the UI. ### SSR windowing + virtualization - SSR-render only the first 50 rows - hydrate those rows - virtualize the remaining dataset client-side - search is now URL-driven during SSR, ensuring deep links such as `?search=DATABASE_URL` ### Lightweight 'Create' flow - 'Create' page no longer loads the full Environment Variables dataset. ## Results Large projects no longer render thousands of rows during SSR. Example (~11k rendered rows): Metric | Before | After -- | -- | -- Document size | ~150 MB | ~5 MB SSR rows | ~11k | 50 Browser DOM rows | Thousands | ~26–38 |
||
|
Daniel Sutton
|
4ea3ef138f |
chore(webapp,redis-worker): make mollifier constants configurable (#3822)
## Summary The mollifier had ~21 behavioural constants baked in as hardcoded values — the buffer's ack-grace TTL and Redis retry/reconnect tuning, the drainer's poll interval and backoff envelope, the pre-gate idempotency claim TTL/wait/poll, the buffered-run mutate-with-fallback wait loop, the metadata CAS retry budget and backoff, the stale-sweep scan bounds, and the draining-gauge interval. None could be adjusted without a code change, which makes tuning the system under production load impossible. This exposes all of them as `TRIGGER_MOLLIFIER_*` environment variables, each defaulting to its previous hardcoded value. Behaviour is identical unless an operator sets a var, so it's a safe no-op deploy. ## Design The package-level classes (`MollifierBuffer`, `MollifierDrainer` in `@trigger.dev/redis-worker`) gain optional constructor options defaulting to the old constants — backward compatible, hence a patch changeset. The webapp factories and worker bootstraps read the env and pass them through. The route- and concern-level pure helpers (mutate-with-fallback, metadata mutation, idempotency claim, stale-sweep state) keep their existing `?? DEFAULT` option fallbacks and are fed env values at their call sites, so they stay unit-testable without importing `env.server`. ## Test plan - [x] `@trigger.dev/redis-worker` builds - [x] webapp typecheck passes - [x] mollifier buffer + drainer testcontainer suites pass (modulo a couple of pre-existing flaky timing tests) - [x] Reviewer: confirm the `TRIGGER_MOLLIFIER_*` env var names match ops conventions --------- Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Co-authored-by: coderabbitai[bot] <136622811+coderabbitai[bot]@users.noreply.github.com> |
||
|
Matt Aitken
|
6961004a73 |
fix(webapp): restore Postgres fallback for non-ClickHouse OTLP spans (#3803)
## Problem On environments where runs carry a Postgres-backed `taskEventStore` value (`taskEvent` or `taskEventPartitioned`), OTLP ingest endpoints (`POST /otel/v1/traces` and `/otel/v1/logs`) were returning HTTP 500. **Root cause:** The org-scoped ClickHouse factory introduced in a recent PR routes all OTLP spans through `getEventRepositoryForOrganizationSync` → `buildEventRepository`. That function only handles `"clickhouse"` and `"clickhouse_v2"` store values and throws `Unknown ClickHouse event repository store: <value>` for anything else. The throw occurred inside the grouping loop of `#exportEvents`, unwinding the entire method and returning 500 for the whole batch. The OpenTelemetry collector's `otlphttp` exporter treats HTTP 500 as non-retryable and drops the batch — causing real span loss. **Fix:** Guard the `getEventRepositoryForOrganizationSync` call in `#exportEvents` so it is only invoked for `clickhouse` / `clickhouse_v2` store values. All other values are routed directly to the Postgres `eventRepository`, matching the guard pattern already present in `resolveEventRepositoryForStore` and `getEventRepositoryForStore` in `eventRepository/index.server.ts`. The ClickHouse factory call is also wrapped in a try/catch that falls back to Postgres so any unexpected store value in a future OTLP batch degrades gracefully instead of failing the whole request. ## Changes - `apps/webapp/app/v3/otlpExporter.server.ts` — add Postgres routing guard and try/catch fallback in `#exportEvents` ## Testing The `eventRepository/index.server.ts` module already has the same guard pattern thoroughly covered. The fix brings `#exportEvents` into alignment with that existing, tested pattern. Manual verification: confirm OTLP batches containing Postgres-store spans return 200 and route to the correct repository. |
||
|
Daniel Sutton
|
139eede41c |
feat(redis-worker): batched pop in MollifierDrainer for fast single-env drains (#3797)
## Summary Adds `drainBatchSize` to `MollifierDrainer` (default `1` — preserves existing behaviour) and wires `TRIGGER_MOLLIFIER_DRAIN_BATCH_SIZE` through the webapp (default `50`). Each tick the drainer now pops up to `drainBatchSize` from each chosen env, then dispatches every popped entry through the shared `concurrency`-bounded `pLimit`. Per-org/per-env fairness is unchanged — only the in-env pop count grows. Pre-existing behaviour was one pop per env per tick. For a single-env burst that single-flighted the drain at the per-tick floor of `pop + engine.trigger ≈ 50–60 ms`. With buffer entries piling up under a real-world tenant burst that's tens of minutes of tail latency to fully materialise — even though PG itself could comfortably sustain the writes. ## Why this matters — heavy-tail illustration Scenario: 100 customers in one window — 94 fire 20 triggers each, 5 fire 100, 1 fires 1000. Gate at `THRESHOLD=10/s`, `HOLD_MS=500`. First 10 of each burst hit PG directly; the rest buffer. | Customers | Triggers each | PG direct | Buffered each | Total buffered | |---|---|---|---|---| | 94 small | 20 | 10 | 10 | 940 | | 5 medium | 100 | 10 | 90 | 450 | | 1 heavy | 1000 | 10 | 990 | 990 | **With `DRAIN_BATCH_SIZE=50`, `DRAIN_CONCURRENCY=50`, ~50 ms `engine.trigger`:** | Tick | Pops | Dispatch waves | Wall-clock | |---|---|---|---| | 1 | 94×10 + 5×50 + 1×50 = 1 240 | 25 × 50 ms | ~1 300 ms (94 smalls done) | | 2 | 5×40 + 1×50 = 250 | 5 × 50 ms | ~300 ms (5 mediums done) | | 3–20 | heavy alone, 50/tick | 1 × 50 ms | ~100 ms each | | Customer class | Buffered fully drained | |---|---| | 94 small | **~1.3 s** | | 5 medium | **~1.6 s** | | 1 heavy | **~3.4 s** | **Without batching (one pop per env per tick — current behaviour):** | Customer class | Buffered fully drained | |---|---| | 94 small | ~500 ms | | 5 medium | ~4.5 s | | 1 heavy | **~49 s** | So the heavy single-tenant tail drops from ~49 s to ~3.4 s (~14× faster) without changing PG load characteristics. Smalls go up slightly in this scenario (500 ms → 1.3 s) because all 100 envs share one tick's dispatch queue — that's the trade we accept for the heavy tail; the worst-case small wait is still inside one tick. PG load is identical either way (50 concurrent inserts at a time, capped by `DRAIN_CONCURRENCY`). ## What changed **`packages/redis-worker`** - New `drainBatchSize` option (default 1 — full backward compat). - `runOnce()` refactored to pop per-env batches in parallel, then dispatch all popped entries through the existing global `pLimit`. Mid-batch pop failure aborts only that env's batch and counts as one failure (same semantic as the old per-env path). - Removed the now-unused `processOneFromEnv` helper. **`apps/webapp`** - `TRIGGER_MOLLIFIER_DRAIN_BATCH_SIZE` env var (default 50, matching `DRAIN_CONCURRENCY`). - Wired into `mollifierDrainer.server.ts`. **Test cloud config** (separate cloud PR): `TRIGGER_MOLLIFIER_DRAIN_BATCH_SIZE="50"` on the worker service. Production rollout deferred until we've watched it on test cloud. ## Test plan - [x] All 25 stub-based drainer tests pass (18 pre-existing + 7 new). 7 new tests under `MollifierDrainer.drainBatchSize`: - pops up to `drainBatchSize` across ticks - global `concurrency` cap still holds when batch > concurrency - mid-batch pop failure isolation - multi-env batch fan-out in one tick - **hierarchical org fairness preserved at `drainBatchSize > 1`** (load-bearing — guards against future regressions to per-env-instead-of-per-org rotation) - mixed success/failure accounting in a batched tick - bounded pops on empty queue (no Lua spam past `drainBatchSize`) - [x] All pre-existing tests still pass unchanged at default `drainBatchSize=1` → backward-compat locked. - [x] `pnpm run build --filter @trigger.dev/redis-worker` clean. - [x] `pnpm run typecheck --filter webapp` clean. - [x] `redisTest` block (real Redis via testcontainers) — couldn't run locally on this branch due to testcontainers runtime discovery; will validate in CI. - [ ] Test-cloud smoke after cloud PR lands: fire `burst 50` against a flagged env and confirm the 50th entry's drain time drops from ~2.5 s to <200 ms. ## Notes - Per-tick memory bound: `maxOrgsPerTick × drainBatchSize` entries can sit in the JS pLimit queue between pop and dispatch. At defaults that's `500 × 50 = 25 000` × ~5 KB snapshot ≈ ~125 MB worst case per worker — well within headroom. - The pre-batch model's strict per-env throughput cap of `1/tick` is documented as the fairness baseline elsewhere. Org-level fairness is what callers actually rely on; this change does not weaken that. 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com> |
||
|
Dan
|
cd252801eb |
feat: dashboard agent - package upgrades (#3793)
1. in webapp folder update ai-sdk to 6.x.x 2. update vitest to 4.xx |
||
|
Daniel Sutton
|
e21b68cc5f |
feat(webapp): dashboard parity for mollifier-buffered runs (#3757)
## Summary Dashboard surfaces handle buffered runs by falling back to the mollifier snapshot: - Run detail, span detail, streams view (`_app.../runs.\$runParam`, `resources.../spans.\$spanParam`, `resources.../streams.\$streamKey`). - Redirect routes (`@.runs.\$runParam`, `runs.\$runParam`, `projects.v3.\$projectRef.runs.\$runParam`). - Action routes — cancel / replay / idempotency-reset / debug — under `resources.taskruns/...` and `resources.../idempotencyKey.reset`. - Logs download. - Realtime subscription route + per-run resource (`realtime.v1.runs.\$runId`, `resources.../realtime.v1.*`). - `CancelRunDialog` gains an `onCancelSubmitted` callback so submit isn't raced by the Radix `DialogClose` wrapper. Stacked on the mutations PR. ## Test plan - [x] \`pnpm run typecheck --filter webapp\` passes - [x] \`pnpm run test --filter webapp test/mollifierRealtimeRunResource.test.ts\` passes - [x] \`pnpm run test --filter webapp test/mollifierRealtimeRunResourceBuffer.test.ts\` passes - [x] \`pnpm run test --filter webapp test/mollifierRealtimeSubscription.test.ts\` passes - [x] Manual smoke: trigger a buffered run, open it in the dashboard, replay/cancel from the UI --- ## Ship-gate follow-up fixes - **Auto-redirect to root span on direct nav** — loader sets `?span=` from root span (PG) or buffered snapshot spanId before 302'ing, so bookmark/share-link/direct-nav doesn't leave the panel collapsed. - **RunPresenter switches from `findFirstOrThrow` to `findFirst` + typed `RunNotInPgError`** — kills the per-poll `PrismaClient error` log spam for buffered runs without changing the route-loader's fallback flow. - **Span detail panel renders for buffered runs** — `SpanPresenter.call` now falls back to `findRunByIdWithMollifierFallback` + `buildSyntheticSpanRun` instead of returning undefined and triggering the "Event not found" toast loop. - **Logs download for buffered runs returns a gzipped placeholder line** — replaces the 404 with a content-encoded line explaining the run is queued. Same org-membership gate as the PG path. - **Admin Debug-Run button hidden for buffered runs + SpanRun circular type alias broken** (squashed) — buttons gate on a new `isBuffered` flag on the synthetic SpanRun. Required grounding SpanRun in `SpanPresenter.getRun` to break a circular type alias TS no longer tolerates once `isBuffered` is a literal field on the shape. - **Replay action requires user auth + org-membership** (🚩 Devin finding) — `action` was unauthenticated and the PG `findFirst` had no org filter, so any caller with a valid `runParam` could replay any run. Buffered fallback inherited the same gap. Fixed to mirror the cancel route. --------- Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com> |
||
|
Daniel Sutton
|
e1950778e2 |
feat(webapp): mollifier API mutations on buffered runs (#3756)
## Summary Cancel, replay, reschedule, metadata, tags, and idempotency-key-reset now succeed against a run that's still in the mollifier buffer. Mutations are applied to the buffered snapshot via Lua CAS; the drainer carries the mutation forward when it replays. Primitives added: - `mutateWithFallback` — PG-first / buffer-fallback resolver with bounded-wait safety net for entries that transition mid-mutation. - `applyMetadataMutation` — buffered metadata PUT mirroring the PG-side retry loop with CAS atomicity. - `resolveRunForMutation` — discriminated-union resolver used by route `findResource` so the route builder's pre-action 404 check sees buffered runs. Routes wired (whole files, no GET/POST splits): - `api.v2.runs.\$runParam.cancel.ts` - `api.v1.runs.\$runParam.replay.ts` - `api.v1.runs.\$runParam.reschedule.ts` - `api.v1.runs.\$runId.metadata.ts` - `api.v1.runs.\$runId.tags.ts` - `resetIdempotencyKey.server.ts` Stacked on the reads PR. ## Test plan - [x] \`pnpm run typecheck --filter webapp\` passes - [x] \`pnpm run test --filter webapp test/mollifierMutateWithFallback.test.ts\` passes - [x] \`pnpm run test --filter webapp test/mollifierApplyMetadataMutation.test.ts\` passes - [x] \`pnpm run test --filter webapp test/mollifierResolveRunForMutation.test.ts\` passes --------- Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com> |
||
|
Daniel Sutton
|
a1dc3c5ae1 |
feat(webapp): mollifier API GET read-fallback — synthetic primitives + route wiring (#3755)
## Summary Synthesise QUEUED/FAILED responses from the mollifier buffer when a TaskRun row hasn't landed in Postgres yet. Wires the synthesis into: - `ApiRetrieveRunPresenter` - v1 trace GET route - v1 spans GET route - attempts route gains a GET loader (fixes pre-existing Remix "no loader" 400) The `readFallback` infra itself lives on the trigger PR (consumed by `IdempotencyKeyConcern`); this PR adds the route-level synthetic-rendering primitives. Stacked on the replay PR. ## Test plan - [x] \`pnpm run typecheck --filter webapp\` passes - [x] \`pnpm run test --filter webapp test/mollifierSyntheticRedirectInfo.test.ts\` passes - [x] \`pnpm run test --filter webapp test/mollifierSyntheticSpanRun.test.ts\` passes --------- Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com> |
||
|
Daniel Sutton
|
4745754a7a |
feat(webapp,run-engine): mollifier drainer replay + stale sweep + cancelled-run engine API (#3754)
## Summary The replay side of the mollifier: - `DrainerHandler`: reads buffered snapshots and replays them through `engine.trigger` to materialise PG rows. - `RunEngine.createCancelledRun`: new public method the handler uses to write CANCELED rows directly from snapshots (bypass queue + waitpoint, emit `runCancelled`). Tolerates the cjson empty-table tags edge case found during validation. - Drainer fairness: org → env rotation so a heavy env doesn't starve light ones in the same org. - Stale-entry sweep + telemetry + alertable gauge so a stuck/offline drainer surfaces in alerts. Both the drainer and sweep default-off; nothing fires unless flagged on (`TRIGGER_MOLLIFIER_DRAINER_ENABLED`, `TRIGGER_MOLLIFIER_STALE_SWEEP_ENABLED`). Stacked on the trigger-time decisions PR. ## Test plan - [x] \`pnpm run typecheck --filter webapp\` passes - [x] \`pnpm run test --filter webapp test/mollifierDrainerHandler.test.ts\` passes - [x] \`pnpm run test --filter webapp test/mollifierStaleSweep.test.ts\` passes - [x] \`pnpm run test --filter @internal/run-engine src/engine/tests/createCancelledRun.test.ts\` passes - [x] \`pnpm run test --filter @trigger.dev/redis-worker packages/redis-worker/src/mollifier/drainer.test.ts\` passes --- ## Ship-gate follow-up fix **Drainer writes SYSTEM_FAILURE on max-attempts exhaustion.** Adds an `onTerminalFailure` callback on `MollifierDrainerOptions` so the customer's run lands a SYSTEM_FAILURE PG row even when the drainer exhausts `MAX_ATTEMPTS` on a retryable PG error (previously `buffer.fail()` was called with no row written → silent data loss). The callback runs before `buffer.fail()` on every terminal path (non-retryable AND max-attempts-exhausted), and re-throwing a retryable error from the callback causes the drainer to requeue rather than fail. Bumps `@trigger.dev/redis-worker` to a **minor** changeset (additive option + new exported types). Includes 5 unit tests covering both terminal causes plus the requeue-on-retryable-callback-failure path and no-callback back-compat. --------- Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com> |
||
|
Daniel Sutton
|
577f35eebe |
feat(webapp): mollifier trigger-time decisions — mollify, claim, read fallback (#3753)
## Summary The trigger hot path's mollifier integration: - `mollifyTrigger`: when the gate trips, write the engine.trigger snapshot to the buffer and return a synthesised QUEUED response. Postgres write is deferred to drainer-replay (next PR in the stack). - Pre-gate idempotency-key claim: same-key triggers serialise through Redis so a burst lands in PG / buffer exactly once. - Read-fallback extensions: `findRunByIdWithMollifierFallback` for the trigger-time idempotency lookup that must see buffered runs. - Gate bypasses: `debounce`, `oneTimeUseToken`, `parentTaskRunId`/`triggerAndWait` skip the mollify path entirely. - `triggerTask` + `IdempotencyKeyConcern` wired to the above. All behaviour gated by the master `TRIGGER_MOLLIFIER_ENABLED` switch; off-state hot path is unchanged (the gate is not even consulted). Stacked on the buffer extensions PR. ## Test plan - [x] \`pnpm run typecheck --filter webapp\` passes - [x] \`pnpm run test --filter webapp test/mollifierMollify.test.ts\` passes - [x] \`pnpm run test --filter webapp test/mollifierIdempotencyClaim.test.ts\` passes - [x] \`pnpm run test --filter webapp test/mollifierReadFallback.test.ts\` passes - [x] \`pnpm run test --filter webapp test/mollifierGate.test.ts\` passes - [x] \`pnpm run test --filter webapp test/engine/triggerTask.test.ts\` passes --- ## Ship-gate follow-up fixes - **Batch items bypass the mollifier gate** — fixes `BatchTaskRunItem_taskRunId_fkey` FK violation on batch triggers when the gate trips. End-state is a drainer-side `BatchTaskRunItem` create-on-materialise; batch traffic passes through the gate until that lands. - **IdempotencyKeyConcern honours buffered-run TTL on expiry** — buffered path now clears expired idempotency claims (read-side) and resets the buffer's `mollifier:idempotency:*` SETNX binding (write-side) so a re-trigger past the customer's TTL lands as a fresh run instead of echoing the stale buffered runId. --------- Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com> |
||
|
Daniel Sutton
|
3924aa4adb |
feat(redis-worker,webapp): mollifier buffer extensions + snapshot type (#3752)
## Summary Buffer-side data layer used by the rest of the mollifier phase-3 stack. - `buffer.ts` gains entry inspection (`getEntry`), idempotency lookup (`lookupIdempotency`), in-place snapshot mutation (`mutateSnapshot`), and dwell tracking. All atomic via Lua. - `mollifierSnapshot.server.ts`: shared `MollifierSnapshot` type plus (de)serialise helpers. - Drops the entry-TTL config and its env var. The drainer is the recovery mechanism; an entry that survives the drainer should surface as a stale-sweep alert, not silently TTL away. Adds methods to the buffer interface; nothing consumes them yet. Subsequent PRs in the stack wire trigger-time mollify, read-fallback, and mutation paths against this surface. ## Test plan - [x] \`pnpm run typecheck --filter webapp\` passes - [x] \`pnpm run test --filter @trigger.dev/redis-worker packages/redis-worker/src/mollifier/buffer.test.ts\` passes --------- Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com> |
||
|
Saadi Myftija
|
9cb6fd17c3 |
fix(webapp): idempotent DeploymentBackgroundWorker creation (#3772)
Make `POST /api/v1/deployments/:deploymentId/background-workers` idempotent so client-side retries no longer collide on the `BackgroundWorker` `(project, env, version)` unique index. Helps make deployments more resilient against the class of indexing failures that surfaces in the dashboard as "Indexing timed out", e.g. during transient database issues. |
||
|
Matt Aitken
|
90bbbd1c44 |
fix(webapp): recover from ClickHouse JSON parse failures on out-of-range integers (#3759)
## Summary Second class of poisoned-row failure in the runs replication path. PR #3708 plugged lone UTF-16 surrogates; this one handles bare JSON integer literals outside ClickHouse's `Int64`..`UInt64` range. Recovery stays purely reactive — the existing `sanitizeRows` walker just gains an extra branch, so the hot replication path pays nothing on healthy rows. Fixes the still-firing customer-facing symptom from [TRI-9755](https://linear.app/triggerdotdev/issue/TRI-9755): `scan-social-profiles` runs continued to be stranded in `EXECUTING` on the Tasks page after #3708 deployed. CloudWatch showed `Dropped batch — ClickHouse JSON parse error but sanitizer found nothing to fix` firing **8/8 times** since the previous deploy (zero successful sanitizations). Root cause: upstream JS Number precision loss on a 21-digit Google Plus ID (`117039831458782873093` → `117039831458782870000`) — the precision-lossy value still serialises as a bare integer that exceeds `UInt64.MAX`, which ClickHouse rejects with `INCORRECT_DATA`. ## How the bug ships The customer task emits an output containing a Poshmark profile's `spec_format`: ```json {"key":"gp_id","proper_key":"Gp Id","value":117039831458782870000,"type":"int"} ``` That value is `1.17e20` — comfortably above `UInt64.MAX` (`1.84e19`) but comfortably below `1e21`. `Number.prototype.toString` only switches to exponential form at `|value| >= 1e21`, so `JSON.stringify` emits the bare token `117039831458782870000` and the ClickHouse `JSON(max_dynamic_paths)` column fails with: ``` Code: 117. DB::Exception: Cannot parse JSON object here: {…}: (while reading the value of key output): (at row 1) : While executing ParallelParsingBlockInputFormat. (INCORRECT_DATA) (version 25.12.x) ``` Same error verbatim as prod. The same number quoted (`"117039831458782870000"`) inserts fine — ClickHouse's dynamic JSON column accepts a `String` subtype on the same path. ## What changed `apps/webapp/app/v3/eventRepository/sanitizeRowsOnParseError.server.ts`: - New private `isUnsafeJsonInteger(value)` helper — true iff `value` is a finite integer-valued JS Number where `|value| < 1e21` (so `JSON.stringify` emits integer form, not exponent) **and** `value` falls outside `[Int64.MIN, UInt64.MAX]`. - `sanitizeUnknownInPlace` gains a number-branch: when the predicate holds, replace the Number with `String(value)`. The downstream JSON column dynamic-types the path as String for that row — fine, since the value was already precision-lossy upstream (no JS Number above 2^53 is numerically meaningful anyway). - Float-valued numbers, large floats (>= 1e21), NaN and Infinity are left alone — `JSON.stringify` emits them with exponents or as `null`, both of which ClickHouse accepts. `apps/webapp/test/sanitizeRowsOnParseError.test.ts`: four new unit tests + an extension to `sanitizeRows` covering surrogate + integer fixes counted together across rows. The unit suite now covers: - Positive value above `UInt64.MAX` (`117039831458782870000` — the actual prod value) - Negative value below `Int64.MIN` - Boundary values pass through (`42`, `Number.MAX_SAFE_INTEGER`, `2^63`) - Non-integer numbers untouched (floats, `1e25`, NaN, Infinity) - The actual `scan-social-profiles` nested shape — finds the offending `gp_id` deep inside `output.data.profiles[].spec_format[].platform_variables[].value` `.server-changes/runs-replication-bigint-recovery.md` — release notes entry. ## Why reactive, not pre-flight `#prepareJson` runs millions of times per day on the replication hot path. Walking every JSON tree to look for oversized integers would add bounded-but-real CPU on every healthy row. `sanitizeRows` only fires after a ClickHouse parse-error rejection, which is a few times a day platform-wide. Extending it costs effectively zero on healthy traffic and gains us recovery on the rare poisoned row. ## Verification - Reproduced 1:1 in a throwaway Docker `clickhouse/clickhouse-server:25.12.11.4` (closest available to the prod `25.12.1.1579` build). Pre-sanitize JSON fails with the exact prod error; post-sanitize JSON inserts cleanly and the row is readable with `gp_id` stored as a String subtype. - `pnpm --filter webapp exec vitest run test/sanitizeRowsOnParseError.test.ts` — 22/22 passing (18 existing + 4 new). - `pnpm run typecheck --filter webapp` — clean. ## Test plan - [x] `pnpm run typecheck --filter webapp` - [x] Unit tests pass against new + existing cases - [x] End-to-end Docker ClickHouse repro confirms recovery - [ ] Post-deploy: confirm `Sanitizing batch after ClickHouse JSON parse error` warns fire instead of `Dropped batch …` errors when `scan-social-profiles` outputs trip CH again - [ ] Post-deploy: confirm `permanentlyDroppedBatches` counter stops climbing in `/stp/trigger-app-prod/ecs/replication/service-container/process-logs` ## What this does NOT do - Doesn't backfill the ~120k+ existing stranded `EXECUTING` rows in production. Same as #3708 — that needs a reconciliation/backfill sweep (separate ticket — TRI-9755 fix #3). - Doesn't address the upstream root cause (the customer task emitting a JS-Number-precision-lossy big int). That's a customer-task concern; our replication path needs to be robust to whatever shape arrives. 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 4.7 <noreply@anthropic.com> |