Commit Graph

235 Commits

Author SHA1 Message Date
James Ritchie fe3fe01fe8 feat(queues): Override queue concurrency limits from the dashboard or API (#2609)
* feat(queues): add ability to override concurrency limit via API and dashboard

* Updates the modal layout and tweaks copy

* Improves the dropdown menu item

* Popover supports both Button and LinkButton

* Right align the columns and fix the dropdown menu item styles

* Organize imports,

* Fix spinner icon in dropdown menu

* Remove unused props

* Adds a tooltip to the Concurrency override badge

* Fixes console error with popover menu

* typo

* Fixes incorrect className

* Minimal buttons to view runs

---------

Co-authored-by: Eric Allam <eallam@icloud.com>
2025-10-17 12:58:25 +01:00
Matt Aitken 885d2d3560 Tags listing now uses ClickHouse (#2576)
* WIP using ClickHouse for the tags filter list

* WIP on tags listing

* Webapp: exclude test files when typechecking

* Tags filtering working with CH

* Remove unused import

* The AI filter should only look at the last past 30d of tags

* Do the text query in ClickHouse

* Deal with encoded characters better

* More encoding fixes

* Fix for wrong items being checked

* Put applied tags back

* Add the env.id to the dependencies array
2025-10-15 12:56:08 +01:00
Eric Allam f8977a7b70 chore(db): remove unnecessary FK constraints on TaskRunExecutionSnapshot (#2533) 2025-10-09 14:07:45 +01:00
Eric Allam 679b41dc7e chore(electric): upgrade server to 1.1.14 (#2590) 2025-10-08 14:33:10 +01:00
Eric Allam 200b7354d0 fix(otel): clickhouse logs/span metrics now exclude partials and debug events (#2581) 2025-10-02 13:51:50 -07:00
Eric Allam 128bc437f6 feat(otel): Add support for storing run spans and log data in Clickhouse (#2567) 2025-10-01 12:41:18 -07:00
Saadi Myftija 3ceea774a8 fix(run-engine): waitpoint update misleading error logs (#2566) 2025-09-26 21:16:12 +02:00
Eric Allam 05b6a26c4f fix(run-engine): pass through engine fair dequeue selection strategy options instead of using defaults (#2565) 2025-09-26 17:12:36 +01:00
Eric Allam 558fb11b89 feat(run-engine): ability to repair runs in QUEUED, SUSPENDED, and FINISHED execution status (#2564)
* feat(server): add two admin endpoints for queue and environment concurrency debugging and repairing
feat(run-engine): ability to repair runs in QUEUED, SUSPENDED, and FINISHED execution status

* Handle FINISHED snapshot in the repair
2025-09-26 15:16:10 +01:00
nicktrn 9aedda23a4 fix(run-engine): carryover batchId after PENDING_EXECUTING stalls (#2563) 2025-09-26 14:14:28 +01:00
Eric Allam 743b8dbe0c chore(run-engine): add additional logging around dequeueing and worker queues (#2562) 2025-09-26 11:50:21 +01:00
Eric Allam eb0263e942 feat(server): add two admin endpoints for queue and environment concurrency debugging and repairing (#2559) 2025-09-25 19:33:10 +01:00
Eric Allam a3bdd3c64b chore(run-engine): improve concurrency sweeper logging to get better visibility (#2557) 2025-09-25 16:43:43 +01:00
Eric Allam e22c321dd1 fix(engine) truncate errors before storing them on a run and waitpoint output (#2552) 2025-09-25 13:39:51 +01:00
Eric Allam 59df4af1eb chore(engine): add additional logging when we fail to get snapshots since (#2551) 2025-09-25 12:59:02 +02:00
Eric Allam 8863ff05c9 fix(engine): limit the number of snapshots returned when getting latest snapshots since (#2550) 2025-09-25 11:53:35 +01:00
Saadi Myftija 700a6ea598 feat: enable canceling deployments (#2545)
* Add canceledAt to the deployment db schema

* Expose an api endpoint to cancel deployments

* Show the canceled status description in the dashboard

* Enable canceling deployments from the dashboard

* Show cancelation reason in the deployment details

* Make verifyProjectMembership a function for consistency

* Apply some good 🐰 suggestions
2025-09-24 11:14:29 +02:00
Saadi Myftija cc94d121f2 feat: installing status for deployments (#2544)
* Add installing status to the deployment db schema

* Replace the deployments /start endpoint with /progress

* Show the installing status in the dashboard

* Add installing status to the api schema and cli

* Add changeset
2025-09-24 10:27:43 +02:00
Eric Allam 87b3603b23 feat(webapp): completing spans server-side no longer write-after-read, improving efficiency and perf (#2530)
* Cancel run events which then propogate cancellation status to span ancestors

* WIP

* convert closing cached run spans to new system

* converted expired complete span event to new method

* move v3 over to new methods

* Convert getDetailedTraceSummary to use the new ancestor override stuff

* remove debug logs

* Don't return UNSPECIFIED task events in getRunEvents

* fix the call site for cancelling run event in v3

* Add changeset

* remove methods
2025-09-19 13:39:48 +01:00
Saadi Myftija a3ef6ea236 feat: separate deployment initialize and start steps (#2522)
* Enable setting the initial status on deployment creation

* Expose endpoint to start deployments

* Extend build timeout on deployment start

* Use separate timeout value for queued deployments

* Add startedAt to the deployment schema

* Show the new startedAt instead of createdAt in the dashboard

* Show github user tag also in the deployment details page

* Show `pending` deployment status as `queued` in the dashboard

* Apply some good 🐰 suggestions

* Add missing return
2025-09-18 16:12:17 +02:00
James Ritchie c8858edf0a New jump to parent or root run buttons (#2067)
* Change the color to indigo

* Pro tier pricing information now matches the marketing site

* Update the button styles to secondary

* WIP adding separate links to Parent and Root runs

* TextLink now supports optional shortcuts

* Adds shortcut keys to the root and parent links + the shortcut help panel

* Adds new icons for root and parent

* root friendlyId works

* Updates icons for jump to root and parent

* Copy tweak

* Improve how the Free tier shows no preview branches

* Improve the wording in the tooltip

* Align the x icon better

* Show price for additional preview branches

* Change the shortcut key

* Fixes button alignment

* Adds nested dependencies task hello-world

* Fixes typo “Cancelled”

* Removes taskIdentifier, not needed

* Removes unused taskIdentifier
2025-09-18 14:29:47 +01:00
Eric Allam 9c087646bf fix(engine): prevent race condition that prevents triggerAndWait runs from resuming by atomically creating associated waitpoint records (#2519) 2025-09-17 13:58:17 +01:00
Saadi Myftija 501a383bcd feat: expose project build settings (#2507)
This PR enables setting project build settings in the settings page:
root directory, install command and trigger config file path.

For most cases there should be no need to set these explicitly.
2025-09-16 15:02:09 +02:00
Eric Allam 83bd6f5f9e fix(engine): carry over completed waitpoints on PENDING_EXECUTING stalls (#2503) 2025-09-15 12:19:46 +01:00
Eric Allam e36d78e4fc fix: don't carry over the checkpoint ID when nack and requeuing (#2502) 2025-09-15 12:06:52 +01:00
Eric Allam 3188dc9b28 perf(webapp): Add BatchTaskRun index to speed up the batch list dashboard page (#2499) 2025-09-12 17:23:52 +01:00
Eric Allam 10e7985fbc Add index for waitpoint tokens dashboard query (#2498) 2025-09-12 16:48:42 +01:00
Eric Allam 2eddda1233 fix(webapp): worker actions now catch service validation errors and respond properly (#2481)
This also stops all the unnecessary error logging when throwing 
ServiceValidationErrors
2025-09-12 14:56:54 +01:00
Eric Allam f077d49291 feat(engine): Improve execution stalls troubleshooting, align dev and prod behavior, adding heartbeats.yield utility (#2489)
* feat(engine): Improve execution stalls troubleshooting, align dev and prod behavior, adding heartbeats.yield utility

* A few improvements via the 🐇 review

* Allow treating EXECUTION stalls as OOM errors, improve the error message, add more information to the docs, improve resource monitor and add it to the docs

* Add changeset
2025-09-12 14:54:39 +01:00
Eric Allam 8e66913e59 fix(run-engine): Preserve snapshot checkpoint ID when a PENDING_EXECUTING snapshot stalls (#2493) 2025-09-10 14:55:57 +01:00
Saadi Myftija 5567f49846 feat(webapp): expose project git settings (#2464)
* Fix settigns page delete project width issue

* Apply a couple of touch-ups to the project settings page

* Add UI flow to connect gh repos

* Enabling adding another gh account in the ui

* Enable connecting a repo to a project

* Enable updating git settings

* Enable disconnecting gh repos from a project

* Remove prisma migration drifts

* Hide git settings when github app is disabled

* Fix migration order

* Avoid using `location` to avoid SSR issues

* Make branch tracking optional

* Disable save buttons when there are no field changes

* Disable delete project button unless the input matches the project slug

* Show connected repo connectedAt date

* Check that tracking branch exists when updating git settings

* Show tracking branch hint in the deployments page

* Fix positioning issue of the pagination pane in the deployments page

* Use mono font for branch names

* Add link to git settings

* Show tracking branch hint for the preview env too

* Add a confirmation prompt on repo disconnect

* Add link to configure repo access in gh

* Add rel prop to github links

* Automatically open repo connection modal after app installation

* Apply some fixes suggested by mr rabbit

* Fix flash cookie issue

* Extract project settings actions into a service

* Extract project settings loader into a presenter service

* Introduce neverthrow for error handling

* Try out neverthrow for error handling in the project setting flows

* Move env gh branch resolution to the presenter service
2025-09-09 13:03:43 +02:00
nicktrn 59c17e04e9 feat(run-engine): worker queue resolver (#2476) 2025-09-04 09:27:06 +01:00
Saadi Myftija 436d951b65 feat(webapp): github app installation flow (#2463)
* Add schemas for gh app installations

* Implement gh app installation flow

* Make the gh app configs optional

* Add additional org check on gh app installation callback

* Save account handle and repo default branch on install

* Do repo hard deletes in favor of simplicity

* Disable github app by default

* Fix gh env schema union issue

* Use octokit's iterator for paginating repos

* Parse gh app install callback with a discriminated union

* Remove duplicate env vars

* Use bigint for github integer IDs

* Sanitize redirect paths in the gh installation and auth flow

* Regenerate migration after rebase on main to fix ordering

* Handle gh install updates separately from new installs
2025-09-02 16:35:33 +02:00
Eric Allam 9b1877bef2 fix(run-engine): retry non-zero exit code errors (#2467)
We’re also now saving the retryConfig from the BackgroundWorkerTask on 
TaskRun.lockedRetryConfig when the run is first locked to the version
2025-09-02 14:12:14 +01:00
nicktrn 73e7378459 feat(webapp): add ecs otel resource attributes (#2454)
* add attributes to runlock dequeue span

* feat(webapp): add ecs otel resource attributes
2025-08-29 11:05:47 +01:00
Eric Allam b0b0df69be fix(run-engine): prevent stalled SUSPENDED snapshots from endlessly retrying (#2448) 2025-08-28 14:25:50 +01:00
Eric Allam 82401ec4fe fix(run-engine): dequeue performance improvements and improved telemetry (#2446)
* more eager dequeuing, queue cooloff periods, return workerQueueLength when dequeueing

* Cache worker group authentication and remove old self-hosted worker code (only managed is currently supported)

* add additional spans during dequeue

* Add env vars and additional spans
2025-08-27 21:22:10 +01:00
Matt Aitken 24a915133e Prisma 6.14.0 upgrade (#2444)
* Initial work on upgrading to 6.14.0

Set the output to node_modules still to make it easier

* Use ./generated Prisma folder, update types to fix issues

* Docker compose restart Clickhouse

* Prisma instrumentation update

* Docker

* Removed database dockerignore file, add generated prisma client to the top-level one

* Delete v3-catalog package.json

* Resolved pnpm lock file

* Log errors for very slow queries
2025-08-27 16:52:58 +01:00
Saadi Myftija 1cc62230ab feat: introduce organization access tokens (#2391)
* Create schema and migration for organization access tokens

* Add helpers for creating and authenticating OATs

* Adapt the auth service to also accept OATs

* Accept OATs in the whoami v2 endpoint

* Enable deployments with the CLI using OATs

* Avoid reading env variables directly in the token utils

* Remove duplicate cli token utils

* Validate ENCRYPTION_KEY length when parsing env vars

* Make token utils a server-only module

* Disallow revoking already revoked OATs

* Simplify generics in authenticateRequest

* Use 32 bytes mock encryption key in the test setup

* Update dummy encryption key values in tests and templates

* Add a column in the OATs table to differentiate between user and system generated

* Simplify args for v3ProjectPath

Co-authored-by: Matt Aitken <matt@mattaitken.com>

* Add index on org id and createdAt

* Avoid storing the encrypted oat token and its obfuscated version in the DB at all

It is a safer approach. Also we do not need to ever read the decrypted token value after creation.

* Fix prisma update condition

* Add token type to the OAT table index

* Accept OATs in the mcp auth flow

* Simplify env auth flow around the /projects endpoints

---------

Co-authored-by: Matt Aitken <matt@mattaitken.com>
2025-08-27 13:23:45 +02:00
Eric Allam bf2120d5a7 fix(runengine): remove dev worker queues from metrics (#2434) 2025-08-21 14:11:22 +01:00
Matt Aitken 4264fc066e chore(webapp): More false errors turned to logs (#2422)
* Remove variables from dequeue log message

* Continue snapshot throw json

* Waiting for deploy error removed

* Realtime ECONNRESET is expected

* Redis worker logErrors changes, removed ids

* Preview branch without a branch shouldn't log an error, it's a user provided issue

* "Task run is not in a cancellable state" isn't an error, it's expected

* "CreateCheckpointService: Child run already resumed" is expected

* "CreateCheckpointService: Batch already resumed" is expected

* "Failed to insert events, will attempt bisection" changed to info, we have errors for complete failures

* Ignore "PrismaClient error"

* Don't log Redis worker DLQ errors if we're ignoring

* "Failed to parse machine config" is fine, sometimes a config is null or undefined

* "Failed to parse machine config" for v3

* MetadataTooLargeError shouldn't log an error
2025-08-20 11:44:26 +01:00
Matt Aitken 0ee4234117 Reduce false error logging (round 1) (#2415)
* Don't log an error when the snapshot shouldn't be created, it's normal for this to happen

* Slack alerts, skip `account_inactive` errors

* v3 finalize run with no locked isn't an error

* Another false error

* Finalize run CRASHED runs were logging errors

* All slack alert errors are warnings except invalid blocks
2025-08-19 09:58:21 +01:00
nicktrn 26d4d08bdc chore: move all placement tag helpers to core (#2403)
* chore: move all placement tag helpers to core

* move placement tag utils into server-only
2025-08-15 17:19:39 +01:00
nicktrn fa7f4b1fed feat(k8s): add placement tags for flexible node selection (#2390)
* add tier scheduling support to supervisor

* add billing info to dequeued message w/o cache

* add cache with best effort invalidation

* fix invalidate circular dep

* add changeset

* use new plan type on runs as fallback during dequeue

* tidy up

* be more explicit with plan type fallback

* remove additional billing check from hot path

* switch to placement tags

* update changeset

* update platform package

* start using new entitlement response

* ensure skipChecks optimization validates at batch level

* add optional items to add to queue manager limits

* make the bool env helper only accept boolean defaults

* remove redundant private field

* update placement tag helper to prevent unsupported tags
2025-08-15 09:52:14 +01:00
Matt Aitken cd6a0b3a3e fix: heartbeat race condition (acking) (#2380)
* The logger now supports metadata

* Added metadata to ServiceValidationError in some critical places

* Don't ack the heartbeat if there's a mismatch, it might prevent a brand new one

* Don't ack the heartbeat inside stalled. By returning it will be acked IF the deduplication key matches
2025-08-12 10:35:21 +02:00
Matt Aitken ae19d0f491 Only start an attempt if not finished. Send message to worker if pending executing (#2377)
* Only start an attempt if not finished. Send message to worker if pending executing

* Fix the exit process reason tet

* Fixed cancelling test since bug fix

The old behaviour was wrong for pending executing in the test
2025-08-11 11:46:10 +01:00
nicktrn 363b9b15b4 fix(engine): set correct run status after requeue and snapshot status after dequeue (#2367)
* set correct run status on snapshot after dequeue

* set run status back to PENDING when we requeue

* remove retrying after failure status from v4 and fix tests

* fix one last test
2025-08-07 16:46:27 +01:00
Matt Aitken af14621683 Specify a region when triggering (#2366)
* Map new allowedMasterQueues → allowedWorkerQueues

* ClickHouse worker_queue on task runs

* Added the Region to the run inspector

* Pass a region in when triggering

* Added a changeset

* Added triggering regions docs

* Added region to the ctx

* Fix for backfiller masterQueue/workerQueue
2025-08-07 12:41:39 +01:00
Matt Aitken a782813c6c Regions – dashboard page, switching default, allowedMasterQueues (#2354)
* Initial Regions page

* Fix for bad attribute name

* Switching regions is working. Some style improvements

* Use a dialog for confirmation, not great yet

* Added allowedMasterQueues

* Improves flag icons

* Improves the region switch modal with more info

* Adds “suggest a region” table row

* New icons for the buttons

* Improved the tooltip information

* New “small” badge style

* Make the default badge live in its column

* Better DO icon size

* Remove unused export of regions options

* Show upgrade message for free users to get static IPs

* Admins can view all regions and switch at will

---------

Co-authored-by: James Ritchie <james@trigger.dev>
2025-08-06 15:56:01 +01:00
Eric Allam d950a969bd Update zod package to version 3.25.76 across all modules (#2352)
* Update zod package to version 3.25.76 across all modules

Update the zod library from version 3.23.8 to 3.25.76 in multiple package files to ensure compatibility and take advantage of new features or bug fixes introduced in recent releases. Keeping all modules synchronized with the latest version of zod helps maintain consistency across the project and reduces potential compatibility issues.

- Modified zod version in apps/supervisor, webapp, and various internal packages.
- Updated zod references in pnpm-lock.yaml to reflect the new version.
- Ensure dependencies that rely on zod are using the updated version to avoid mismatches.

* Add changeset
2025-08-06 14:48:43 +01:00