ts7
44 Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 Daniel Sutton
|
ae96b6c175 |
fix: read-your-writes + global-scope idempotency correctness under the run-ops split (#4284)
## What & why
Two related correctness fixes for the run-ops DB split. Under the split,
run-store reads can route to a **lagging read replica**; a just-written
run/waitpoint/batch can then be missed, causing a wrong decision.
**1. Read-your-writes → owning primary.** Surfaced first as an
intermittent `wait.until({ idempotencyKey })` re-wait on retry. Auditing
the run-store read surface found the same class at sibling sites (some
gating mutations or returning spurious 404s, others
tolerable/self-healing). Reads that must observe their own writes now
route to the owning **primary**
(`findRun`/`findWaitpoint`/`findBatchTaskRunByFriendlyId` →
`*OnPrimary`, a primary re-read on a miss, or a retryable 404 where the
SDK polls). Read-view reads stay on the replica. All additive — the
happy path is unchanged.
**2. Global-scope idempotency across the split.** A `global`-scope key
carries no per-run salt, so the same `(env, task, key)` triggered
concurrently from parents resident on **different** run-ops DBs could
dedup-miss on each DB and create a duplicate (the per-DB unique index
can't enforce cross-DB uniqueness). Such triggers (global scope, or
scope-absent, while split is active) are serialized through the existing
Redis idempotency claim, the loser resolves the winner by id across both
DBs, and the claim is reacquired on the expired/failed
clear-and-recreate path. `run`/`attempt` scope embed the run id and
never contend.
## Stacked for review
This is the **base** of a 2-PR stack, split so review is easier:
- **This PR** — production code only (34 files).
- **Stacked tests PR →
https://github.com/triggerdotdev/trigger.dev/pull/4285** — the
caller-driven guards (55 test files) on top of this branch.
## Validation
Local run-ops split, **both 2-DB and 3-DB**, fresh boot on this branch:
SDK canary 64/71 (only the known concurrency/input-streams/s3 failures),
quarantine sweep **0 unexpected** (340 pass / 16 known / 4 local) in
each topology, dashboard e2e 0 failed. No product regressions.
|
||
|
Daniel Sutton
|
84f3e1b39c |
feat(run-ops): webapp write path — trigger/batch minting, idempotency routing, run lifecycle (#4118)
## What Routes the webapp write path through the run-ops split seam: trigger/batch minting, idempotency-key resolution, and the run-lifecycle services now determine residency and dispatch writes to the correct store. - **Trigger & batch** (`runEngine/services/triggerTask.server.ts`, `batchTrigger.server.ts`, `createBatch.server.ts`, `streamBatchItems.server.ts`, `v3/services/batchTriggerV3.server.ts`): mint ids with the run-ops-aware minting and route creation/streaming through the store; batch children inherit the parent's residency. - **Idempotency** (`runEngine/concerns/idempotencyKeys.server.ts` + new `idempotencyResidency.server.ts`): idempotency-key lookup/dedup is residency-aware so a keyed retrigger resolves against the store that owns the original run. - **Run lifecycle services** (`createCheckpoint`, `createTaskRunAttempt`, `enqueueDelayedRun`, `expireEnqueuedRun`, `finalizeTaskRun`, `resumeBatchRun`, `cancelDevSessionRuns`, `executeTasksWaitingForDeploy`, `triggerFailedTask`): resolve their target run through the store rather than a fixed client. - **Reads that fan out from writes** (`runsRepository` + `clickhouseRunsRepository`, `BulkActionV2` + batch read-through, realtime `sessions`/`runReader`, alerts `deliverAlert`/`performTaskRunAlerts`): route through the read-through resolver. - `9535ae63d` — resolves the parent run through an injectable run store in `TriggerFailedTaskService`. - `bf8f7c881` — drops the "known-migrated" concept from write-path and read repos; residency is id-shape only. - `515b897ea` — self-defaults `resolveWaitpointThroughReadThrough` to the safe run-ops clients. ## Why PR6 of the run-ops split stack. This is the write-path counterpart to the read foundation in the previous PRs: with it in place, both reads and writes route through the seam. Additive when the split is disabled (id-shape resolution collapses to the control-plane client); behavior-changing on the minting, idempotency, and lifecycle paths when enabled. ## Tests Large new/expanded vitest suite under `apps/webapp/test/` and colocated service tests: trigger-task and batch-trigger store routing, residency inheritance, idempotency dedup residency + legacy-authority, bulk-action read routing, cancel-dev-session routing, alerts store routing, runs-repository read-through, realtime session/run-reader read-through and stream-registration routing, and the waitpoint read-through default. Testcontainers-backed; no mocks. ## Notes Draft, **stacked on #4117** (`runops/pr05-webapp-foundation`). Review that first; this diff is against it. Server-change / changeset note to be added at stack-assembly time. 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com> |
||
|
Daniel Sutton
|
8465ac5ac3 |
feat(run-ops): webapp db topology, flags, and split-mode resolver wiring (#4117)
## What Wires the run-ops split into the webapp: database topology, environment flags, split-mode gating, and the control-plane resolver/cache layer that the run-store and run-engine seams from the previous PR plug into. - **DB topology & env** (`apps/webapp/app/db.server.ts`, `env.server.ts`, `entry.server.tsx`): adds the run-ops database clients/topology and the environment variables that configure and gate the split. - **runOpsMigration module** (new `apps/webapp/app/v3/runOpsMigration/`): the webapp-side machinery — `splitMode.server.ts`, `controlPlaneResolver.server.ts` + `controlPlaneCache.server.ts`, `readThrough.server.ts`, `crossSeamGuard.server.ts`, `distinctDbSentinel.server.ts`, id-minting helpers (`mintBatchFriendlyId`, `runOpsMintKind`, `resolveInheritedMintKind`), `runOpsCascadeCleanup.server.ts`, the split read gate, and route/unblock catalogs. - **Store/engine wiring** (`app/v3/runStore.server.ts`, `runEngine.server.ts`, `runEngineHandlers.server.ts` + new `runEngineHandlersShared.server.ts`): points the webapp's store/engine construction at the resolver, and factors shared handler logic out so both seams use one path. - **Read-path touch-ups**: `runtimeEnvironment.server.ts`, `eventRepository/index.server.ts`, `taskRunHeartbeatFailed.server.ts`, `engineVersion.server.ts` route their run/environment lookups read-through the resolver. - `413a94511` — interlocks split mode against the native realtime backend so the two aren't enabled in an incompatible combination (see `.server-changes/run-ops-split-realtime-interlock.md`). - `dc74c57fd` — drops the earlier "known-migrated" read layer; residency is determined by id-shape only. ## Why PR5 of the run-ops split stack. This is the webapp foundation layer: it stands up the DB topology, flags, and resolver/cache the rest of the stack depends on, and repoints webapp read paths through the resolver. Additive when the split is not enabled (existing single-DB behavior preserved behind flags); behavior-changing on the read-through paths and the realtime interlock. ## Tests New vitest coverage across `apps/webapp/test/` and colocated `*.server.test.ts` files: db topology, split mode, split read gate, cross-seam guard, mint cutover / flip latency, control-plane cache, control-plane resolver, distinct-db sentinel, read-through loaders (route loaders, run-detail loaders, `findEnvironmentFromRun`), and the run-engine handlers. Testcontainers-backed; no mocks. `pnpm-lock.yaml` synced for the two new webapp deps. ## Notes Draft, **stacked on #4116** (`runops/pr04-store-engine`). Review that first; this diff is against it. Server-change / changeset note to be added at stack-assembly time. 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com> |
||
|
Chris Arderne
|
4fde283e76 |
chore: format and lint webapp also (#4056)
#3977 added formatting and linting everywhere else. This extends it to the webapp. |
||
|
Chris Arderne
|
df78ef96d9 |
feat: multi dev branches (#4023)
Closes this feature request: [https://triggerdev.featurebase.app/p/isolated-dev-sessions-for-multiple-local-trigger-dev-instances](https://triggerdev.featurebase.app/p/isolated-dev-sessions-for-multiple-local-trigger-dev-instances) ### Feature notes: - CLI `trigger dev` works as before - `trigger dev --branch my-branch` to create a new branch and run against it. - `trigger dev archive --branch my-branch` to archive (or in webapp). - New webapp page to manage and archive dev branches, currently feature flagged. ### Implementation details: - No changes to data model, no backfill. `isBranchableEnvironment` column is ignored for dev branches, we use `parentEnvironmentId IS NULL` instead. - `x-trigger-branch` overloaded for preview and dev branches - New `TRIGGER_DEV_BRANCH` env var available locally. `TRIGGER_PREVIEW_BRANCH` overloaded for child runs. - Lots of new glue code to sanitise the branch checks. ### Rollout - Deploy webapp/API changes (all backwards compatible) - Manual tests on some orgs - Deploy docs, release CLI, flip feature flag for webapp feature ### NB - `api.v1.projects.$projectRef.environments.ts` will return `isBranchableEnvironment: true` for all dev environments. ### Prerequisites - [x] Typecheck will not pass until we make a new release of `@trigger.dev/platform` and bump it here |
||
|
Daniel Sutton
|
65c545da4e |
refactor(run-store,webapp,run-engine): route Postgres TaskRun reads through the run store (#3990)
## Summary Adds read methods to `RunStore` (`findRun`, `findRunOrThrow`, `findRuns`) and routes every Postgres read of `TaskRun` through them, mirroring how writes already go through the store. Behavior-preserving: each relocated read keeps its exact query, field selection, and database client (writer, replica, or transaction). This lets `TaskRun` reads be retargeted to a different backing store later without touching call sites. Stacked on #3981 (the write adapter); that PR is the base of this one. ## Scope In scope: the run engine, webapp services, presenters, and route loaders. Three reads that pulled `TaskRun` in through a parent model's relation `include` (alert delivery, batch results, attempt-dependency cancellation) are decomposed to fetch the run(s) through the store and stitch them back, since a relation include would not follow `TaskRun` to a new table. Left reading the existing table (out of scope): the legacy MarQS paths, the legacy trigger idempotency read, and one raw-SQL recovery script (commented for revisiting at cutover). ## Notes Reads default to the read replica; callers pass the writer or a transaction client wherever the original read did, so writer-vs-replica behavior is unchanged. |
||
|
Eric Allam
|
f9d57d3bd5 |
feat(webapp): add a new backend for the realtime runs feed (#3864)
## Summary Adds a second backend for the realtime runs feed (`useRealtimeRun`, `subscribeToRunsWithTag`, `subscribeToBatch`), built to stay healthy when a single busy environment has many subscribers watching many runs at once. It is gated behind a feature flag with the existing backend as the default, so nothing changes for users until it is enabled per environment. ## Design A run change is published once, as a small self-describing record, to a single per-environment channel. Every feed is then a predicate over that one stream rather than owning a channel: - A per-instance router indexes the currently-held feeds by run, tag, and batch. When a run changes it hydrates the affected rows once and serializes them once, then fans the result to every matching feed. One hot shared tag watched by many subscribers costs a single database query and serialize, not one per subscriber. - Feeds that don't match a change are never woken, wake delivery per environment is coalesced on a leading edge (250ms default) so a burst of changes costs one wake, and cold reads coalesce onto a single short-TTL-cached resolve. - An admission gate bounds how many cold ClickHouse resolves run concurrently, so a mass reconnect across many distinct filters queues instead of stampeding the database. - Changes that land while a client is between long-polls are delivered on its next poll instead of waiting for the periodic backstop: each environment buffers its recent change records, subscriptions linger briefly after the last feed closes, and a newly-armed poll replays exactly the connection's gap. - The per-connection replay cursors behind that are shared across instances via Redis (a single timestamp each), so a poll landing on a different instance behind the load balancer still reads the connection's true gap instead of falling back to a cold resolve. Cursor reads have a bounded deadline and degrade to the cold-read path on any Redis trouble. - Tag subscriptions with multiple tags match runs carrying all of the tags, mirroring the existing backend's filter semantics, and live long-polls hold for about 20 seconds to match its cadence. - The per-environment channel supports Redis Cluster sharded pub/sub, so the wake path scales horizontally across shards by environment. - The backend reports its health through OpenTelemetry metrics (delivery lag, poll resolution paths, backstop outcomes, replay and cursor-store activity), with a provisioned Grafana dashboard for local development. Everything is behind the feature flag and tunable via env vars; the existing backend remains the default. |
||
|
Matt Aitken
|
e4981d1b11 |
feat(webapp): consolidate auth path + add comprehensive auth tests (#3499)
## Summary
Consolidates the webapp's authentication and authorization into a small
set of route helpers, replacing the ad-hoc `requireUser` /
`requireUserId` / `authenticatedEnvironmentForAuthentication` calls
scattered across routes. Same security model, but the per-request flow
(authenticate → authorize → load) now lives in one place per route
family.
Introduces a plugin seam (`@trigger.dev/plugins`) that lets the cloud
build install a richer RBAC implementation without touching webapp code.
The OSS fallback keeps the pre-RBAC permissive behaviour intact, so
self-hosted deployments work unchanged.
Adds a comprehensive end-to-end auth test suite that didn't exist before
— 193 `it()` blocks (vitest reports ~199 after `it.each` expansion)
covering API key, PAT and JWT auth across the public API surface, plus
dashboard session auth for admin pages.
## Changes
### Plugin contract — `@trigger.dev/plugins`
`RoleBaseAccessController` interface authoritative for both OSS
(fallback) and cloud (enterprise plugin):
- `authenticateBearer(request, { allowJWT? })` — API-key / public-JWT
auth, returns env + ability
- `authenticateSession(request, { userId, organizationId?, projectId?
})` — dashboard auth, caller resolves `userId` from the session cookie
and passes it in (no `helpers.getSessionUserId` callback — decouples the
plugin host from session-cookie code)
- `authenticatePat(request, { organizationId?, projectId? })` — PAT
auth, returns identity + `lastAccessedAt` so the host can throttle the
per-request update
- `authenticateAuthorize*` variants for the auth-and-check-in-one-call
cases
- `isUsingPlugin(): Promise<boolean>` — capability flag for UI /
branching where plugin-present-ness matters; replaces the
sentinel-string coupling that had `personalAccessToken.server` matching
`"RBAC plugin not installed"` literally
### Dashboard auth (started, partial rollout)
Admin and settings pages migrated to a unified `dashboardLoader` /
`dashboardAction` helper that authenticates the session, runs an
authorization check, and exposes the result to the route. Other
dashboard routes still on the old pattern; remaining migration tracked
in TRI-8730.
Migrated routes:
- `admin.*` (14 admin / back-office / feature-flags / LLM-models /
notifications / orgs / concurrency pages)
- `_app.orgs.$organizationSlug.settings.team`
- `_app.orgs.$organizationSlug.settings.roles`
### API / realtime / engine auth (complete for the migrated families)
71 routes migrated to a unified `apiBuilder` that centralizes Bearer /
PAT / Public-JWT authentication and applies the per-route authorization
check before the handler runs. Includes:
- `api.v1.*` and `api.v2.*` and `api.v3.*` — tasks, runs, batches,
queues, prompts, deployments, query, sessions, waitpoints, packets,
workers, idempotency keys
- `realtime.v1.*` — runs, batches, sessions, streams
- `engine.v1.*` — dev / worker-action protocols
29 routes still on the legacy `authenticateApiRequest*` helpers —
tracked as a post-deploy follow-up in TRI-9228.
Multi-resource auth direction is now explicit at the call site via
`anyResource(...)` (OR) and `everyResource(...)` (AND). Bare arrays no
longer typecheck — fixes a class of bug where a JWT scoped to one
resource could implicitly access others under OR semantics.
PAT auth path consolidated: was three DB queries per request (legacy
`authenticateApiRequestWithPersonalAccessToken` findFirst +
`rbac.authenticatePat` join + `lastAccessedAt` update). Now one query in
the steady state — plugin returns `lastAccessedAt`, host smart-skips the
update via JS-side throttle when fresh.
Side effect: action aliases preserved historic JWT scope semantics where
the new model is stricter (e.g. a `write:tasks` JWT now also satisfies
`trigger` / `batchTrigger` / `update` actions on the same resource —
matched at the auth boundary, not in the route handler).
### Backwards-compat fixes
The strict-match model regressed several real-world JWT shapes. Each
preserved via explicit `anyResource(...)` entries in the route's authz
block:
- **Batch retrieve routes** (`api.v1.batches.$batchId`, `api.v2.*`,
`realtime.v1.batches.*`) accept `read:runs` JWTs again (pre-RBAC
literal-match superScope behaviour)
- **Runs list routes** (`api.v1.runs`, `realtime.v1.runs`) accept
type-level `read:tasks` / `read:tags` on unfiltered queries (matched the
legacy `Object.keys` iteration semantic)
- **PAT/OAT auth shape** normalized through `toAuthenticated` so all
auth methods return the same slim `AuthenticatedEnvironment` (was:
API-key returned the slim shape but PAT/OAT returned raw Prisma
`Decimal` / no `orgMember`)
- **Scope `:` preservation** in resource ids — `read:tags:env:staging`
now correctly identifies the tag id as `env:staging`, not `env`
### Slim `AuthenticatedEnvironment`
Extracted to `@trigger.dev/core/v3/auth/environment` — a structural
shape independent of `@trigger.dev/database`. The plugin contract
returns this; webapp consumers import from there; the cloud plugin
(Drizzle) returns the same shape without Prisma's `Decimal` class
leaking into the public surface. Lets internal-packages (run-engine,
etc.) refer to `AuthenticatedEnvironment` without pulling Prisma in.
### Auth test suite (new — `*.e2e.full.test.ts`)
193 e2e tests run against a real spawned webapp + Postgres (no mocks).
Coverage matrix:
- **API key auth** — read / write / trigger / batchTrigger / deploy
actions across runs, batches, deployments, prompts, queues, query,
sessions, input-streams, waitpoints, tasks, idempotency keys; multi-key
resources (a run carries batch / tag / task identifiers — auth must
accept any matching scope)
- **Personal Access Token auth** — comprehensive matrix: scope match,
scope mismatch, missing scope, expired token, malformed token
- **Public JWT auth** — sub-vs-URL environment resolution, expired JWTs,
signature verification, scope checking, otu (one-time-use) token
semantics, branch-environment signing-key fallback
- **Dashboard session auth** — admin-only pages reject non-admins;
per-action gating
- **Cross-cutting edge cases** — revoked API key grace window, JWT
cross-environment isolation, MissingResource branch behaviour
### Hygiene cleanups
- Deleted dead `app/services/authorization.server.ts` (legacy
`checkAuthorization` + types — no live consumers post-migration) and its
orphaned test
- Dropped the never-populated `scopes` field from
`ApiAuthenticationResultSuccess`
- `scheduleEmail` moved out of `email.server.ts` into its own module —
breaks a `commonWorker → marqs/V1` import chain that was poisoning the
auth test graph
- OSS Roles page shows a deployment-aware empty state ("Roles aren't
available in this self-hosted deployment" vs the plan-upsell copy) via
`rbac.isUsingPlugin()`
- Team action handler: explicit per-intent ability gates
(`manage:billing` for purchase-seats, `manage:members` for set-role +
remove-member with self-leave carve-out)
### Cross-repo coordination
All public-package contract changes paired in `triggerdotdev/cloud#763`
(rbac-packages branch) — the enterprise plugin implements the same
`RoleBaseAccessController` interface against Drizzle.
## Test plan
- [x] `pnpm run typecheck --filter webapp` clean
- [x] `pnpm --filter webapp exec vitest run --config
vitest.e2e.full.config.ts` — 193/193 pass (requires Docker for
testcontainers)
- [x] Spot-check an authed API endpoint with a valid + invalid API key
against a local stack
- [x] Spot-check the migrated admin pages render and gate non-admins
---------
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
|
||
|
Eric Allam
|
99dfee3a57 |
fix(webapp): honor RevokedApiKey grace window for public access tokens (#3464)
## Summary Follow-up to #3420. PATs (public access tokens) minted before an API key rotation 401'd immediately on the realtime stream endpoints, even though the rotation flow advertises a 24h overlap. This fixes the gap. ## Root cause PATs are JWTs signed with the env's `apiKey` at mint time. When that secret is rotated, `validatePublicJwtKey` (`apps/webapp/app/services/realtime/jwtAuth.server.ts`) only verifies the signature against `environment.parentEnvironment?.apiKey ?? environment.apiKey` — i.e. the env's *current* canonical key. Any PAT in the wild signed with the previous key fails signature verification → 401, even within the grace window. #3420 wired up the grace-window fallback in two places — `findEnvironmentByApiKey` (raw secret-key auth) and `api.v1.auth.jwt.ts` (signs new JWTs with the canonical key when minting from an old one) — but the *verify* path for already-issued PATs was never updated. In a typical app, `POST /api/v1/tasks/.../trigger` (Bearer secret) keeps working through rotation because that path has the fallback, but `GET /realtime/v1/streams/run_*/...` and `POST /realtime/v1/streams/run_*/input/...` 401 for runs that were already in flight when the rotation happened. ## Fix After the primary `validateJWT` against the env's current `apiKey`, fall back to non-expired `RevokedApiKey` rows for the signing env (parent env when the request is against a child) — but **only on the failure path**, so the hot success path is unchanged. Uses `$replica` to match the rest of the auth path. Symmetrical to the `findEnvironmentByApiKey` two-step from #3420. ## Changes - `apps/webapp/app/services/realtime/jwtAuth.server.ts` — `validateAgainstRevokedApiKeys` helper invoked only on `!result.ok` - `apps/webapp/app/models/runtimeEnvironment.server.ts` — `findEnvironmentById` also selects `parentEnvironment.id` so we can scope the revoked-keys lookup to the correct env ## Test plan E2E verified locally via curl against `GET /realtime/v1/runs/{runId}` (PAT-authenticated): - [x] Pre-rotation, PAT signed with K1 → **200** with run body - [x] Simulate rotation (insert `RevokedApiKey` row + flip env `apiKey` to K2 in a single transaction, mirroring `regenerateApiKey`) - [x] Same PAT (K1) within grace window → **200** with run body — fallback hits - [x] Fresh PAT signed with K2 → **200** — current key still works - [x] Set `RevokedApiKey.expiresAt` to past → **401** — fallback finds no live row - [x] Bogus signature (no rotation) → **401** - [x] Cleanup verified: env `apiKey` restored, `RevokedApiKey` row deleted - [x] `pnpm run typecheck --filter webapp` passes |
||
|
Eric Allam
|
03e4d5fe31 |
feat(webapp,database): API key rotation grace period (#3420)
## Summary
Regenerating a RuntimeEnvironment API key no longer immediately
invalidates the previous one. Rotation is now overlap-based: the old key
keeps working for 24 hours so customers can roll it out in their env
vars without downtime, then stops working.
## Design
- **New `RevokedApiKey` table** (one row per revocation). Holds the
archived `apiKey`, a FK to the env, an `expiresAt`, and a `createdAt`.
Indexed on `apiKey` (high-cardinality equality — single-row hits) and on
`runtimeEnvironmentId`.
- **`regenerateApiKey` wraps both writes in a single `$transaction`:**
insert a `RevokedApiKey` with `expiresAt = now + 24h`, update the env
with the new `apiKey`/`pkApiKey`.
- **`findEnvironmentByApiKey` does a two-step lookup:** primary
unique-index hit on `RuntimeEnvironment.apiKey` first; on miss,
`RevokedApiKey.findFirst({ apiKey, expiresAt: { gt: now } })` with an
`include: { runtimeEnvironment }`. Two-step (not `OR`-join) keeps the
hot path identical to today and puts the fallback cost only on invalid
keys. Both lookups use `$replica`.
- **Admin endpoint** `POST /admin/api/v1/revoked-api-keys/:id` accepts
`{ expiresAt }` and updates the row. Setting to `now` ends the grace
window immediately; setting to the future extends it.
- **Modal copy** on the regenerate dialog updated — previously warned of
downtime, now explains the 24h overlap.
## Why a separate table instead of columns on `RuntimeEnvironment`
- Keeps the hot auth path's primary lookup unchanged — no
OR/nullable-apiKey semantics to reason about.
- Naturally supports multiple in-flight grace windows (regenerate twice
in a day → two old keys valid until their independent expiries).
- FK + cascade cleans up correctly when an env is deleted; nothing to
backfill.
## Test plan
Verified locally against hello-world with dev and prod env keys:
- [x] baseline — current key authenticates (`GET /api/v1/runs`) → `200`
- [x] regenerate via UI — DB shows old key in `RevokedApiKey` with
`expiresAt ≈ now+24h`, env has new key
- [x] grace window — both old and new keys → `200`; bogus key → `401`
- [x] admin endpoint: `expiresAt = now` → old key `401`
- [x] admin endpoint: `expiresAt = +1h` (after early-expire) → old key
`200` again
- [x] admin endpoint: `expiresAt = past` → old key `401`
- [x] admin 400 (invalid body), 404 (unknown id), 401 (missing/non-admin
PAT)
- [x] same flow exercised end-to-end on a PROD-typed env — behavior
identical
- [x] `pnpm run typecheck --filter webapp` passes
|
||
|
Matt Aitken
|
bc0d1ff59a |
Metrics dashboards (#3019)
Summary - Implemented metrics dashboards with a built-in dashboard and custom dashboards - Added a "Big number” display type What changed - New data format for metric layouts and saving/editing layouts (editing, saving, cancel revert) - QueryWidget usable on Query page and Metrics dashboards - Time filtering, auto-reloading and timeBucket() auto-bin support - Filters added to metrics; widget popover/improved history and blank states - Side menu: - Metrics/Insights section with icons, colors, padding, collapsible behavior and reordering of custom dashboards - Move action logic into service for reuse and API querying; refactor reordering for reuse <!-- devin-review-badge-begin --> --- <a href="https://app.devin.ai/review/triggerdotdev/trigger.dev/pull/3019" target="_blank"> <picture> <source media="(prefers-color-scheme: dark)" srcset="https://static.devin.ai/assets/gh-open-in-devin-review-dark.svg?v=1"> <img src="https://static.devin.ai/assets/gh-open-in-devin-review-light.svg?v=1" alt="Open with Devin"> </picture> </a> <!-- devin-review-badge-end --> --------- Co-authored-by: James Ritchie <james@trigger.dev> |
||
|
Matt Aitken
|
cf1c311dea |
Use the replica for API auth queries (#2932)
When we auth API keys we get the environment, project and org. This is a very hot path so even though these queries are fast they contribute a significant percentage of total load. This moves them to use the read replica instead. |
||
|
Matt Aitken
|
4264fc066e |
chore(webapp): More false errors turned to logs (#2422)
* Remove variables from dequeue log message * Continue snapshot throw json * Waiting for deploy error removed * Realtime ECONNRESET is expected * Redis worker logErrors changes, removed ids * Preview branch without a branch shouldn't log an error, it's a user provided issue * "Task run is not in a cancellable state" isn't an error, it's expected * "CreateCheckpointService: Child run already resumed" is expected * "CreateCheckpointService: Batch already resumed" is expected * "Failed to insert events, will attempt bisection" changed to info, we have errors for complete failures * Ignore "PrismaClient error" * Don't log Redis worker DLQ errors if we're ignoring * "Failed to parse machine config" is fine, sometimes a config is null or undefined * "Failed to parse machine config" for v3 * MetadataTooLargeError shouldn't log an error |
||
|
Eric Allam
|
b38405cb88 |
Realtime and task run performance improvements (#2158)
* Add createdAt filter to realtime subscribing with tags * Filter realtime colums and expose ability to skip some columns * Add sharding support for electric * Use unkey cache for the created at filter caching * Remove 2 unused indexes on TaskRun * Run list now filters by a single runtime environment * Remove project ID indexes * Use clickhouse in task list aggregation queries instead of pg (keep pg for self-hosters) * WIP clickhouse powered runs list stuff * Improve the query to get the latest tasks for the task list presenter * Update the usage task list to use clickhouse * Implement next runs list powered by clickhouse * Add new index for TaskRun for the runs list, by environment ID * Add runTags gin index * Handle possibly malicious inputs * Ignore claude settings * Better handling not finding an environment on the schedule page * Use ms since epoch in test, not seconds * Remove unused function * Fix test * Use an env var for the realtime maximum createdAt filter duration (defaults to 1 day) * Fixed the query builder to correct the group by / order by order * Make sure runs.list still works * Create small-birds-arrive.md |
||
|
Matt Aitken
|
068c024477 |
Preview branches (#2086)
* Initial preview migrations * Modified the staging endpoint to create preview environments * Added isBranchableEnvironment to RuntimeEnvironment * Staging = yellow Preview = orange * Changed the env sort order * Set isBranchableEnvironment correctly. Create preview for new projects * Very basic branch menu * Creating branches from the dashboard * Fix for string icons on project delete page * Don’t show branch API keys * WIP on the manage branches page * RuntimeEnvironment added projectId index * Only create the parentEnvironmentId column if it doesn’t exist already * Improved the limit wording * Add search to the branch list * contains in both places * Many style improvements * Branch dropdown and v4 badge * Arching/unarchive branches working in the dashboard * Tidied imports * Change preview slug from `prev` to `preview` * Use correct color for side menu preview branch icon * Upsert the branch and use the shortcode as a unique constraint * Upserting working with nice messages in the dashboard * Better errors when upserting branches * Button shortcut, don’t allow event to propagate * Better duplicate error message * Filter out archived branches from the env selector * Archiving/creating tweaked some more * Add an archived banner to the app, fixes for archived branches and upsells * Fixed pagination * Disable editing schedules, pausing queues, testing tasks * Don’t allow replaying if the env is archived * When deploying detect the correct environment * Get the projectClient when there’s a branch * createGitMeta function, most code from the vercel CLI repo * Deploy, getting the correct environment client * Added git column to WorkerDeployment * Add GitMeta to core schemas * Create branch when deploying * WIP on branch support in the API * Delete old createTaskRunAttempt fn * apiAuth remove export from internal functions * Rename env var to “TRIGGER_PREVIEW_BRANCH” * Add TRIGGER_PREVIEW_BRANCH to resolved env vars for runs * First preview deploy and run working * Set the preview branch in the main SDK * Added git links to the preview branches table * Better errors when replaying/testing archived branches * Don’t dequeue archived environments * Env var resolution with parent environment * Hello world default machine small-2x to save my memory * Fix for more env var functions * Only return non-archived envs * Switch to controlled state for the checkboxes * Uncheck everything when PREVIEW is checked * WIP on branch UI * Show the preview branch label on the env vars list * Fix for overriding env vars * Adding preview branch env vars working * Progress on new env vars * Only allow selecting a single branch * Layout fix when there are errors * Set the defaultValue so there are some fields * Conform fix for team invite page * Archived environments don’t run scheduled tasks * Added Git data to deployments * Added git data to the deployment inspector * Don’t allow upserting schedules when archived * Deduplicate and blacklist some env vars * Fix for wrong conform function being used * Show a better error if all vars were blacklisted * Added environment variable search (by key and value) * Improved preview branch icon * Replay now supports branches * Schedule page render branches properly * Show the env icon in bottom-left of the test page * When editing older schedules (that have multi-env) show preview branches correctly * Fix for incorrect disallowed branch name character * Extract and improve the directory verification code * WIP for CLI preview archive command * Improved the preview branch action buttons * Redirect to the project if we don’t find a matching env * Archiving branch via the CLI working * Fix for archiving branches * Public access token test task * JWTs working are with preview branches * Add branch and git data to the Run ctx * Updated GitMeta functions to work in CI * Added pullRequestState * Archive when deploying if the PR is closed/merged * Fix for the changesets guide * Fix for CLI dev bug introduced * CLI promote now supports preview branches * Add PR title. Reordered them and added tooltips * syncEnvVars working with branches * Added preview branch support to syncVercelEnvVars() * Detect the branch from Vercel env var (set during build) * Allow passing a branch in * Use process.env.VERCEL_TOKEN as well… this used in Vercel CI * Temp delete * Improved regenerate api key modal * Added Accordion component (with styles) * Redesigned the API keys page * Revert "Temp delete" This reverts commit 177b92cd935a6161456bde65d01294e23ecfd47f. * Changeset * Fixed docs link * The new branch panel closes when a branch is created * Update apps/webapp/app/services/upsertBranch.server.ts Co-authored-by: coderabbitai[bot] <136622811+coderabbitai[bot]@users.noreply.github.com> * Removed findUniques from WorkerGroupTokenService * Made the parentEnvironmentId migrations safe * Latest lockfile * Update packages/cli-v3/src/commands/workers/build.ts Co-authored-by: coderabbitai[bot] <136622811+coderabbitai[bot]@users.noreply.github.com> * Move isValidGitBranchName to a separate file * Move the sanitize fn too * removeBlacklistedVariables moved to a separate file * Moved deduplicateVariableArray to a separate file… * Fix broken sanitizeBranchName import * Another import fix… * Improved blacklisted error message * SImplified migration to use `ADD COLUMN IF NOT EXISTS "parentEnvironmentId" TEXT` --------- Co-authored-by: coderabbitai[bot] <136622811+coderabbitai[bot]@users.noreply.github.com> |
||
|
Eric Allam
|
7a58439728 | WIP queue indexing | ||
|
Matt Aitken
|
0173406999 | Pausing the environment is working | ||
|
Matt Aitken
|
7b862b9438 |
Major dashboard improvements (environment centric) (#1796)
* Delete the proxy app (was v2) * Delete RunPresenterElectric * Select the best proj/org/env * Storing current proj/env in DB. Initial selection logic working with tasks page * 2sm needed to be in the Tailwind merge list * Move the task stream route (although we don’t actually use the env for now) * Alerts moved from /v3 * API keys page moved from /v3 * Concurrency page moved from /v3 * WIP on side menu sections * Improved the accordion animation * Moved schedules from /v3 * More pages moved * Move pages working * Run page working * Schedules working * Moved deployments * Alert pages moved * Delete electric hooks, not used * Started setting up blank states * Test page working * Removed “Select task” from the test page * Some work on deployment page * Style tweaks * Redirect from project root to approriate env * Improved env selector styling * Fix for jsx errors * Better min width on env selector * Improved the env switching logic * Added deployments to env routing * Redirect deployments to the correct env * Redirect run from proj to env * JSX icon fix * Only allow single env schedules from now on * Remove env var count from the API keys page * Move improvements and redirects * Project settings moved * Fix for scroll area on test page * Tweaked the test design * Made recent payloads column narrower * Improved the test layout some more * Added org icon, new project selector menu * WIP on org switching menu * Org switching is working * New menu working well, removed old side menu items * Buttons can now have a component name or an actual component for their icons * Removed the Projects page, instead redirect appropriately * Fix for broken blank states * Minor run table improvements * Removed unused switcher log and logic * Concurrency page fix for invalid html, improved layout * Minor improvements * Moved the side menu to the project level * Improved account styling * Moved org settings pages (with redirects) * Add current plan to billing side menu link * Upgrade to get staging from env dropdown * New env badge on concurrency limits page * Show Run Engine version in span presenter * New promote icon * Concurrency limits page is the sum of engine v1 + v2 queues * Fix for missing batch import * Added currentConcurrencyOfEnvQueue function * Basic avatar setting working * Avatar setting is working * You can change the color of your icon * Avatar improvements * Bugfix for mising prop * Removed some old env badges * Fixed replaying * Removed EnvironmentLabel * Old env badge deleted, changed everywhere to the new one * Fix for Slack integration paths * Fix for waitpoint completion form moving * Bulk replay/cancel env fix * Fix for alert webhook path * Redirect projects/v3/* to project/* * Fixes for CLI redirect routes * Remove welcome email (unused) * Change how we count schedules towards your limits * Use new schedules limits when checking a schedule * Added projectId back in to task queries (indexes) * WIP dev presence * CLI modal * Moved things around and use Context * Fix for p inside p * Dev connected status on run page * Correct dev env (not a teammates) * Show disconnected message at the end * Minor tweak on project dropdown icon padding * Fix for inconsistent date format for presence * Added a message when pushing to the billing page * Center the team page * Project settings page centered * Improvements to the dev presence |
||
|
nicktrn
|
e97704d904 |
Run Engine 2.0 (WIP) (#1575)
* bump worker version * Suggested glossary for the RunEngine, TBC * Removed BatchTaskRun changes from this branch, they were done in main * Set the BatchTaskRun status to completed when all runs are completed * When dequeuing respect passed in maxResources * Ported over the new run props: idempotencyKeyExpiresAt, versions, oneTimeUseToken, maxDurationInSeconds * Didn’t hit save… the new props when triggering tasks passed through * Idempotency expiration + waitpoint edge case * WIP on creating checkpoint, parking for now * fix worker routes * upgrade webapp node types to support generic event emitter * separate event bus handler singleton and run failure alerts * duration waits * fix execution snapshot debug spans * task waits * fix event bus types * temporary fix for react hook run handle type * disable run notifications for now * convert any typecasts to expect errors to more easily fix later * fix webapp types after node types upgrade * updateEnvConcurrencyLimits across marqs and the runqueue * Pass proper values into the run engine * RunQueue settings and removed unused rebalancing workers * Remove rebalancing prop * Tidied more things up * Update/remove queue limits for MARQS and RunQueue * taskQueue/concurrencyLimit changes ported back into the RunEngine * Reworked completing waitpoints to improve performance and reduce race conditions * Improved test robustness * Down to a single run lock only when a run is totally unblocked and ready to continue * warm starts, worker notifications, wait fixes * Fix for Run Engine poll interval env var * Expect the waitpoint to be completed quickly * If a run is locked then it’s too late to expire it * Added VALKEY_ env vars and plugged them into the run engine * Extracted and updated the guard queue function so it can be used when batching * Added logging and universal concurrency changes to trigger task v1 * Added notes back in * Bump @trigger.dev/worker to 3.3.7 * reportInvocationUsage for the runAttemptStarted event * improve execution snapshot span debug span start times * Unfriendly IDs * update lockfile * Created a shared determineEngineVersion function * disable unfinished commands * save new cli config to different location, misc fixes * add basic engine version check via current deploy * new run engine will default to node 22 runtime * block some actions for projects on previous run engine * fix worker group tests * fix triggerAndWait test * one typescript version to rule them all * redlock type patch * fix type issues caused by ts-reset * improve cleanup scripts * add missing socket.io dep * fix run notification handler type * fix worker group test again * generate prisma client for e2e tests * remove worker group tests for now * prevent image pull rate limits during unit tests * increase timeout for queue concurrency limit test * generate prisma client for preview release * same node types everywhere * Updated engine readme, removed legacy system notes * use default machine preset from platform package * worker instances plural in schema * disable pnpm update notifications * return worker group details from connect call * add workers admin route * fix heartbeat route return type * move deployment labels to core apps * refactor run controller env schema * Add firstAttemptStartedAt to TaskRun * RunEngine 2.0 batch trigger support (#1581) * Make it clear when BatchTriggerV2Service is used * Copy of BatchTriggerV2Service * WIP batch triggering * Allow blocking a run with multiple waitpoints at once. Made it atomic * Removed unused param * New batch service * Pass through the parentRunId and resumeParentOnCompletion * Use the new batch service, and correct trigger task version * Force V1 engine if using BatchTriggerV2Service, we’ve already done the check at this point * Removed the $transaction and early exit if nothing changed * Adedd a simple batch task to the hello world reference catalog * Fix for batch waits not working * Added parentRunId in a couple more places * Removed waitForBatch log * Added another parentRunId * Expanded the example to include all the different triggers * More changes to blocking to support continuing after idempotent completed runs * Fix for the wrong type when blocking a run * remove @map * optimise worker auth query * add engine version header to core api client requests * remove unique constraint for default group id * consolidate migrations * the first managed worker becomes the global default * Debug events off by default, added an admin toggle to show them * worker group name can't be an empty string * add exec helper to core * move machine resources to core * add pre-dequeue callback to determine max resources * optionally skip dequeue * bump worker package * move worker to core * fix ReadableStream type error * fix another type issue * update a few more tsconfigs * add metadata changes introduced in #1563 * Run Engine 2.0 trigger idempotency (#1613) * Return isCached from the trigger API endpoint * Fix for the wrong type when blocking a run * Render the idempotent run in the inspector * Event repository for idempotency * Debug events off by default, added an admin toggle to show them * triggerAndWait idempotency span * Some improvements to the reference idempotency task * Removed the cached tracing from the SDK * Server-side creating cached span * Improved idempotency test task * Create cached task spans in a better way * Idempotency span support inc batch trigger * Simplified how the spans are done, using more of the existing code * Improved the idempotency test task * Added Waitpoint Batch type, add to TaskRunWaitpoint with order * Pass batch ids through to the run engine when triggering * Added batchIndex * Better batch support in the run engine * Added settings to batch trigger service, before major overhaul * Allow the longer run/batch ids in the filters * Changed how batching works, includes breaking changes in CLI * Removed batch idempotency because it gets put on the runs instead * Added `runs` to the batch.retrieve call/API * Set firstAttemptStartedAt when creating the first attempt * Do nothing when receiving a BATCH waitpoint * Some fixes in the new batch trigger service… mostly just passing missing optional params through * Tweaked the idempotency test task for more situations * Only block with a batch if it’s a batchTriggerAndWait… 🤦♂️ * Added another case to the idempotency test task: multiple of the same idempotencyKey in a single batch * Support for the same run multiple times in the same batch * Small tweaks * Make sure to complete batches, even if they’re not andWait ones * Export RunDuplicateIdempotencyKeyError from the run engine * Latest lockfile * Trigger with a machine (old run engine) * RE2, allow setting machine when triggering * Fix for new glob patterns * add max run count to dequeue from version route * add worker instance name env var and header * queue consumer pre skip callback * poll for more runs after final execution errors * fix dequeue search param schema * add shortcut to debug switch * expose run engine timeouts as env vars * make warm start durations configurable * add optional status to json reply helper * fix preSkip hook, add debug logs * BLOCKED_BY_WAITPOINTS -> SUSPENDED * exit controller when run suspended * check if already replied before http reply * run controller will wait for next run after the current one is suspended * cancel run button shortcut * minimal event repository environment type * fix update metadata call * run suspension and misc fixes wip * change debug shortcut to shift + D * Started work on the Dev supervisor * Formatting * Fix for bad imports * Before rebuilding SSE * Presence updating from the CLI working via SSE * add worker notification debug logs * send run:stop when exiting run phase * skip current snapshot poll on worker notification * add more logs and route to submit run debug logs * add worker and runner ids to snapshots * improve run notification debug logs * add workload debug log route * misc run controller fixes and refactor * prevent parallel execution of critical functions * update bun to 1.2.1 * WIP with dev dequeuing * Method to convert friendlyIds to non-friendly, do nothing with actual ids * Set the engine on BackgroundWorker, lazily upgrade projects to engine V2 * Runs with ttls were getting immediately expired… oops. * Pass the Waiting for deploy reason through, so we have it on the execution snapshots * Fixed the logic for getting the right background worker for a run * Use the correct ID when dequeuing… * determineEngineVersion is now fully functional * Rate limiter ignores the dev endpoints * Retrieving a batch gives you the runIds * Set a unique version for the RE2 BatchTaskRun * add provisional changeset * The start of dev run execution is working * First dev run working * Moved the dev run controller closer to what Nick did with the managed one * export exec output type * Heartbeat fix: don’t heartbeat if _isHeartbeating == false * Dev runs get notifications, some dev bug fixes * Improved logging or dequeuing * We need to dequeue runs from the latest version too, for triggerAndWait * Ported Eric’s validateWorkerManifest with nicer errors * When flattening an idempotency key if part is undefined, return undefined * Dev logging fixes * Remove sigterm listener * Deprecating workers. Don’t specify a BackgroundWorker when dequeuing an environment * Deleted some old files. Renamed “managed” to “deploy” * When a build finishes, always copy the build dir (otherwise the first one gets trampled on by the 2nd) * Dev master queues should work differently * Deleting old workers * Added debounce function to core * Improvement to canceling * WIP on debounce canceling on socket disconnection * Added environment data to execution snapshots * Dev runs that have stalled get “Canceled” with a reason explaining why * Show CLI messaged when a connection to the platform is lost/restored * Fix TriggerTask after merge * Add trigger task v2 max attempts, replace some findUniques * Port the new queue logic to the run engine * More fixes post-merge * We weren’t setting a `retryConfig` up for the tests… it’s now required * Start the Redis worker inside the Run Engine… 🤦♂️ * Trying to make the testcontainers more reliable * Added keyPrefix: "engine:” * Badly placed bracket in trigger task * Better Redis namespacing * Fix for expired run not getting removed from the queue * Don’t create a redis client in the testcontainers, return the redisOptions instead * Cleanup redis client in the run lock tests * Fix for the RunQueue not supporting keyPrefix * Updated more of the RunQueue scripts rebalancing * Trying to make Redis more robust in the tests… * Improved test resiliciency more * Fix for delays (checkpoint check) * Increase the timeout slightly to fix ttl test * Added priority support when triggering * More wip trying to make test containers more reliable * batchTriggerAndWait test is still failing… some wip to try fix it * Fixed redis tests now we’re not providing a client * Separate Redis clients for the run engine worker/queue/runlock * Made the wait for duration test more resilient * Added idempotencyKeyExpiresAt to Waitpoints * Waitpoint timeouts and idempotency expiry * Use finishWaitpoint, removed extra worker job * Added waitpoint idempotency tests * Creating resume tokens is working * Some improvements to the resume tokens * Moved resumeTokens to just be wait functions 🥳 * Delete old RuntimeManagers * Wait for token is working * Better test for the wait tokens * Improved the test task some more * Hide the accessories in the span inspector * WIP on waitpoint inspector * WIP on complete waitpoint form * Span overview panel can be changed based on the entity type * Improved the waitpoint display * WIP on completing waitpoint form * Use the existing CodeBlock for the tip * Style improvements * Complete waitpoint * All waitpoint sidebar variants * Waits now use a pause icon * Durations waits use the API to create/block with a waitpoint, not the runtime * Fix for engine.blockRunWithWaitpoint required org id * Removed old wait code from the run controllers/task run process * Form action for skipping a datetime waitpoint * Move testDockerCheckpoint to a separate core package export (it can’t be bundled on the client) * Fix for glitchy hourglass animation * Completed waitpoints display better * Increase Redis maxRetriesPerRequest to 20 (default) * Completing and skipping waitpoints is working * Remove the database prisma dev command, since we need to use create only now. Updated docs * Added skip timeout, reworked the UI * Tweaked spacing * Added payload limit to waitpoint token completion from dashboard * Test idempotency works on wait.for and wait.until * Moved the worker-actions to /engine/ from /api/ * Moved dev engine endpoints to /engine/ from /api/ * Separate /engine/ rate limiter * Added parallel wait prevention, it’s working for duration waits but not well for triggerAndWait yet * WIP post-merge conflicts * Set taskEventStore column in the new engine * Remove duplicate keys * Post-merge fixes * Fix for span merge layout * Use executedAt instead of firstAttemptStartedAt --------- Co-authored-by: Matt Aitken <matt@mattaitken.com> |
||
|
Eric Allam
|
5750f35add |
Shared queue consumer telemetry improvements and perf changes (#1619)
* Shared queue consumer telemetry improvements * findUnique -> findFirst to stop the prisma dataloader issue on api key auth * Added action stats, audited all actions and improved reasons * Better debug logging filtering |
||
|
Eric Allam
|
f9ec66c562 |
v3: new build system (#1265)
* upgrade @opentelemetry packages to the latest versions * remove v2 only packages, will be moved to a dedicated repo * remove more v2 code and run pnpm install * use the npm yalt package in the webapp * convert @trigger.dev/core to tshy * Switch from jest to vitest in @trigger.dev/core * Fixed core test * move core-backend code into core subpath export * convert @trigger.dev/sdk to tshy * Removed hono * move core-apps to core/v3/apps, remove core-apps, start converting cli-v3 * Fix up some of the commands * cli now building and loadable * using package-json-from-dist to get package version now in core and cli * dev command WIP * cleaned up some repetition and structure of the entry point stuff * bringing back the background worker stuff * Indexing of the v3 catalog * getting closer to executing dev runs... * centralize dev logging using event emitter * Move indexing to it’s own entry point, simplify code * dev runs working * Get instrumentation to work with openai * debugging achieved internally * provide worker files as part of the worker creation on the server * support for cjs and esm javascript * Fixed timeout * worker manifest now has the config path * auto-upgrade config to non-deprecated alternatives * Adding package preview release * deployment WIP * improve the syncEnvVars output and adapt resolveEnvVars * WIP bun runtime * WIP bun support * seed tasks with the machine preset if listed in the config * deploy run executions WIP, extracted TaskRunProcess into 1 place * deployed tasks running and executing 🎉 * support for waits and better flushing & process cleanup * Fixed the heartbeating * Better warning messages * Improve and unify the indexing between dev and deploy * Support for external deps that need node-gyp to build * build extensions can now install custom packages and run instructions in the image. Also prisma extension now works and also works with multiple schema files * Add back in the main/types/module to sdk * dev no longer is Ink/React, grace period for disconnections in dev * Fix the changeset config * More changeset fixes * Remove config packages * More changeset fixes * Fixed typescript issues (needed to revert back to zod 3.22.3 * Fix pr_checks workflow * Remove the prepare script * Fixed tests and package versions * Remove cli test script * Remove packages from tailwind watch paths * Add repo to public packages * Just commit the generated files and do the building at dev time * Try and get pkg.pr.new working * Try again * Fix emitDecoratorMetadata importing named export from typescript * config file backwards compat with export const config * Fixed issue where import errors weren’t coming through * p-retry is a prod dep * typescript needs to be a prod dependency for emitDecoratorMetadata * Add better debug logging to help track down import-in-the-middle bug * An external is only considered resolvable if it resolves to the same path as the collected external * Fix runtime checks to allow >=18.20 * Move extensions to a new build package * Fixed building packages in dockerfile * Remove the e2e test from publish workflow for now * Don’t treat pkg.pr.new versions has needing upgrading * making sure config handleError works, and discovered path aliases don’t work in config files * Strip empty string env vars so they accidentally override real values * Couple of things * Update version to use preview instead of beta * Hopefully fix re-attempts with >30s delay * Match socket emit messages to current latest in main * Initial guide * Go back to beta * Go back to the preview, and update guide to use pr preview tags * Go back to beta --------- Co-authored-by: Matt Aitken <matt@mattaitken.com> |
||
|
Matt Aitken
|
da6ce3c8d5 |
Concurrency page and more accurate tracking (#1252)
* Initial TaskRunConcurrencyTracker implementation * MARQS calls a subscriber to events * When enqueuing add the extra required metadata * Track concurrency per environment for tasks too * Admin page for global concurrency * Use the new concurrency tracker on the tasks page * Useful performance test task * getAllTaskIdentifiers() * New page for concurrency * BackgroundWorkerTask index for quick lookup of task identifiers * Added a way to get concurrency for environments * Added upgrade/request more concurrency button * Queued task column working * Use defer and suspense * Added queue column to the concurrency environments table * Some comments added for clarity * Fixed bad log message * Sidemenu: move lower and rename to “Concurrency limits” * Only show the environments, not tasks. Renamed to “Concurrency limits” |
||
|
Eric Allam
|
6ca66b76f4 | Remove unnecessary transaction from the runtime environment session handling | ||
|
Eric Allam
|
ba71f959e2 |
v3: Runs List Management SDK and SDK improvements (#1133)
* Improved the existing runs API
* WIP next runs API
* Improve the returned ApiPromise to add ability to return response
* More WIP
* WI{
* Added offset/limit pagination stuff like the cursor one, and converted all API methods to use ApiPromise
* More run API stuff
- Adding schedule output from the retrieveRun endpoint
- Ability to filter by schedule and isTest
* Remove env from retrieve run in openAPI
* prefer duplication over merge
* WIP docs
* Use spread to DRY up some run API schemas
* Finish the overview docs
* Adding changeset
* Fixed typecheck errors
* Typo fix
* Re-export zodfetch from core so the v3 CLI can use it
* Fixed type errors
---------
Co-authored-by: Matt Aitken <matt@mattaitken.com>
|
||
|
Matt Aitken
|
7b3b2e0d8e |
Deal with Dev environments where the team member has been removed (#1141)
* Test page: don’t show deleted dev environments * Function to filter out environments that are dev and have no orgMember or orgMemberId * Don’t show tasks from deleted members (or the associated environment labels) * Show “Dev: Deleted” if the user has been removed. * Only show your environment in the env vars table * Renamed function * Use the displayableEnvironment function |
||
|
Matt Aitken
|
fae93ac747 |
New tasks page (with activity) and multi-user fixes (#1063)
* Fix for “overridden” spelling mistake * Make the task function names grey instead of yellow * The task page now shows each task once * Blankline * WIP on activity bars * Correct size of the graphs * Add data for all days, even if there are no runs * Activity graph is working and has a tooltip * No activity label * Got the bar bg working * Activity chart now has a bg * Added a compound id to make the activity graphs appear faster * Improvements to the activity graph * Tried to disable the bg from animating * Added read replica support to the app, using Nick’s code from his batching PR * Added the read replica to BasePresenter * Added the queued and running columns to the Tasks list * Added avg duration column * Included completed but failed runs in the average duration * Get rid of the old useDevEnvironment hook and remove API keys from the projects query * Unified getting environment info to using a function and doing it in presenters * Removed the path from the tasks table * Onboarding for dev for 2nd+ user. Improved environment sorting. * Removed log * Improved the spacing * Enable staging for new v3 projects |
||
|
Matt Aitken
|
4986bfda2e |
Scheduled tasks (#1036)
* Database schema and migrations for schedules * Added schedules to the side menu * The pagination can optionally hide the page numbers for a compactive mode * Filters for the schedule page * Added triggerSource (“STANDARD”, “SCHEDULED”) to BackgroundWorkerTask * Added the ability to disabled a LinkButton * Started work on the schedule page * Environment buttons * The new schedule form styles * Added a cxouple of extra fields * Allow a checkbox to have a rich label * Added cronstrue package to the webapp * WIP creating tasks using the form * Improved the form styling * Creating schedules is working in the UI * Minor improvements * Basic schedule table is displaying * Creating tasks with triggerSource = scheduled. Refactored how task metadata is stored and accessed to be cleaner * Resource route * Fixes in the form * Added a gap between the environments * WIP on OpenAI generating CRON expressions * AI generated CRON expressions is working * Fix for the CRON field being uneditable after an AI generation * Improvements * Table padding * useThrottle now behaves correctly * Added filtering to the schedules list * Improved the layout and fixed CRON search * Fixed pagination for the schedules list. Just use a regular Prisma query * Page size of 20 * Added links to the schedule rows * Get rid of the Last run column for now * Implement triggered scheduled tasks Also implemented superjson payloads and dev runtime environment “presence” with RuntimeEnvironmentSession * Move CronPattern and CreateSchedule into a common client-accessible file * Latest UI changes * Fix for creating a task schedule with a blank dedup key * Refactor the human to cron stuff into a separate file and use json_object OpenAI response format * Fix for trying to use a hook on the server-side… * A couple of fixes to the new schedule form * WIP on viewing a scheduled run * Make the filters all optional * Use the RunListPresenter from the schedule presenter * Display a table of runs… the wrong runs but still * Runs from the schedule * Deleting schedules from the UI * Tidied imports and fixed name of options object * Disabling a schedule * Editing schedules * Tidied imports * Added icons to the task list, needs some design love * Added a tooltip for CRON pattersn * Show the last run in the schedules table * Some tweaks * Added a placeholder to the CRON AI field * Improved the trigger source icon * Dim out disabled schedules * Scheduled tasks have the correct icon in a run * Added the task source icon to the test task list * Added the date field component to storybook * Style improvements to the date field * Implement Task Schedule API * added a medium sized variant to the date field * Fixed replay run for superjson payload types Also now linking from the replaying run to the original run (using span links) Also added a project metrics prometheus endpoint to detect the state of the queues * WIP on allowing different forms for testing * If you pass a string to prettyPrintPacket which is json or superjson, safeParse it first * Test page, deal json and superjson. Started splitting UI for schedules * WIP on schedule form * Removed the Label from the DateField * WIP on test schedule form * Fix for the runs page showing the wrong message when there are no runs from time filtering * Removed labels from the DateField * Fixes for the form * Test runs are working for schedules * Fix for nextScheduledTimestamps in triggerScheduledTask * Implement idempotency key support and fix issue with cancelled runs causing concurrency usage * Added API documentation for the schedule API * Remove log * Removed console log from runs page * Transform the recent runs test data on the server * Fix for hydration mismatch * Current date as the default for the test form * Recent payloads working * Delete schedule modal * Deal with empty strings from the form * Set the initial value for the scheduled test form * Add option to print console logs in the dev CLI locally (issue #1014) * Export queue from the SDK * Fix for schedules list when you have no schedule tasks * Blank states improved * Make task schedules more generic, to support additional schedule generators in the future * Removed log from maqrs * Removed “v3/schedules” export from the SDK --------- Co-authored-by: Eric Allam <eallam@icloud.com> Co-authored-by: James Ritchie <james@jamesritchie.co.uk> |
||
|
Matt Aitken
|
479445f741 |
V3 Test feature (#915)
* Added isTest to TaskRun * TriggerTaskService now accepts isTest and adds it to TaskRun if set * Set the cliVersion and sdkVersion when indexing * Added a test page * Radio buttons are now circles and accept a ReactNode * The background of the code editor is now darker * WIP on the test task page * Improvements to the test page * Testing is working * Fix typo * Show the real isTest value in the runs table * Added a key to the outlet so it correctly resets all the state on the sub-route |
||
|
Eric Allam
|
07f7c65054 |
v3 queue system (#903)
* Introducing Modular Asynchronous Reliable Queueing System (MarQS). Works in dev * Convert MarQS to using lua and dealing with concurrency * Simplified the timeout queue and current concurrency is now a set instead of a flat value (to support idempotency) * Implement task heartbeating and reconnect the background workers CLI when the websocket connection reconnects * Start adding internal telemetry support for the server * Get env vars to work in dev and implement prisma tracing in webapp * Cleanup telemetry and implement it in the consumer * Implement dequeuing a message from a parent shared queue * Implement a custom logger exporter instead of using console log exporter * Use node instead of shell for generating protocol buffer code * Propogate trace context into debug logs, and allow turning off logger exporter through env vars * Switch to using baselime for internal otel data * Make orgMember optional to fix type issues * Provide the CLI dev env vars through the CLI, don’t build dotenv into facade * Removed the logger import * Address Matt’s comments * Addressing more of Matt’s comments * Handle sending an execution after a websocket connection closes * Remove auth from the env attributes to prevent obfuscation |
||
|
Matt Aitken
|
29edcd3df9 |
Rename and delete projects (#880)
* Side menu: selector title changed to project. Changed section headers and made it Slack + Discord * Added project settings page. Renaming is working * Added project.deletedAt column * Deleting projects WIP * Allow clearing the project from the session * Don’t load projects that are deleted * Improved the project selection logic * Clear the session project id when deleting a project * Deleting the last project in an org now works. The new project page doesn’t have the sidebar anymore. * Removed false code comment * Added the JobRun index back in, using Prisma. Otherwise it tries to remove it on each migration * Only return environments where the project isn’t deleted. This will block API calls * Do the project deletedAt check in code, not SQL * Made it clearer what the code does with a comment and applied the same logic to the public api key |
||
|
Matt Aitken
|
9ff4c0dbd5 |
Project-wide Prettier setup (#237)
* Setup project-wide prettier * Remove old workspace file * Remove old debugging directives * New top-level .prettierignore * Updated Prettier config settings * Contrubuting guide: Fix for some bad code blocks * Added more ignores * Improved the format script command * printWidth set to 100 * Formatted entire repo (pnpm run format) |
||
|
Matt Aitken
|
3a6c25d42d | API authentication updated to work correctly with public and private keys | ||
|
Matt Aitken
|
4ca841a1d5 | Query to get environment from either the apiKey or the pkApiKey | ||
|
Eric Allam
|
9f49b7f4c0 | Move prisma to an internal database package | ||
|
Eric Allam
|
1bba822e20 | Added projects, moved envs to projects, and renamed executions to runs | ||
|
Eric Allam
|
9ab7530070 | Current environment is now workflow specific | ||
|
Eric Allam
|
f39bc44eec |
Projects ✨
- Deploy a new VM when a push event comes through - Live updating project overview page |
||
|
Eric Allam
|
d553701b74 | Display schedule workflow descriptions | ||
|
Matt Aitken
|
9de6ec1d54 | Testing is working | ||
|
Matt Aitken
|
d1d8439dbf | Environment dropdowns behaving now we’ve switched to “development” and “live” | ||
|
Matt Aitken
|
e44e29b682 | RuntimeEnvironments now have a title, used in the dropdown | ||
|
Matt Aitken
|
793fa2df87 | Environments using a cookie | ||
|
Matt Aitken
|
4113c1a638 | Get the current runtime environment from the cookie | ||
|
Eric Allam
|
57a8c0a234 |
Doing real authorization of API keys now so we can get back the organization ID
I’ve also started to add pulsar integration with the webapp, to get messages from coordinators |
