Commit Graph

237 Commits

Author SHA1 Message Date
Eric Allam 892bed8c4c Upgrade to electricsql 1.2.4 (#2668) 2025-11-13 15:19:59 +00:00
Eric Allam 536d9fa217 feat(realtime): Realtime streams v2 (#2632) 2025-11-11 14:54:00 +00:00
James Ritchie fe3fe01fe8 feat(queues): Override queue concurrency limits from the dashboard or API (#2609)
* feat(queues): add ability to override concurrency limit via API and dashboard

* Updates the modal layout and tweaks copy

* Improves the dropdown menu item

* Popover supports both Button and LinkButton

* Right align the columns and fix the dropdown menu item styles

* Organize imports,

* Fix spinner icon in dropdown menu

* Remove unused props

* Adds a tooltip to the Concurrency override badge

* Fixes console error with popover menu

* typo

* Fixes incorrect className

* Minimal buttons to view runs

---------

Co-authored-by: Eric Allam <eallam@icloud.com>
2025-10-17 12:58:25 +01:00
Matt Aitken 885d2d3560 Tags listing now uses ClickHouse (#2576)
* WIP using ClickHouse for the tags filter list

* WIP on tags listing

* Webapp: exclude test files when typechecking

* Tags filtering working with CH

* Remove unused import

* The AI filter should only look at the last past 30d of tags

* Do the text query in ClickHouse

* Deal with encoded characters better

* More encoding fixes

* Fix for wrong items being checked

* Put applied tags back

* Add the env.id to the dependencies array
2025-10-15 12:56:08 +01:00
Eric Allam f8977a7b70 chore(db): remove unnecessary FK constraints on TaskRunExecutionSnapshot (#2533) 2025-10-09 14:07:45 +01:00
Eric Allam 679b41dc7e chore(electric): upgrade server to 1.1.14 (#2590) 2025-10-08 14:33:10 +01:00
Eric Allam 200b7354d0 fix(otel): clickhouse logs/span metrics now exclude partials and debug events (#2581) 2025-10-02 13:51:50 -07:00
Eric Allam 128bc437f6 feat(otel): Add support for storing run spans and log data in Clickhouse (#2567) 2025-10-01 12:41:18 -07:00
Saadi Myftija 3ceea774a8 fix(run-engine): waitpoint update misleading error logs (#2566) 2025-09-26 21:16:12 +02:00
Eric Allam 05b6a26c4f fix(run-engine): pass through engine fair dequeue selection strategy options instead of using defaults (#2565) 2025-09-26 17:12:36 +01:00
Eric Allam 558fb11b89 feat(run-engine): ability to repair runs in QUEUED, SUSPENDED, and FINISHED execution status (#2564)
* feat(server): add two admin endpoints for queue and environment concurrency debugging and repairing
feat(run-engine): ability to repair runs in QUEUED, SUSPENDED, and FINISHED execution status

* Handle FINISHED snapshot in the repair
2025-09-26 15:16:10 +01:00
nicktrn 9aedda23a4 fix(run-engine): carryover batchId after PENDING_EXECUTING stalls (#2563) 2025-09-26 14:14:28 +01:00
Eric Allam 743b8dbe0c chore(run-engine): add additional logging around dequeueing and worker queues (#2562) 2025-09-26 11:50:21 +01:00
Eric Allam eb0263e942 feat(server): add two admin endpoints for queue and environment concurrency debugging and repairing (#2559) 2025-09-25 19:33:10 +01:00
Eric Allam a3bdd3c64b chore(run-engine): improve concurrency sweeper logging to get better visibility (#2557) 2025-09-25 16:43:43 +01:00
Eric Allam e22c321dd1 fix(engine) truncate errors before storing them on a run and waitpoint output (#2552) 2025-09-25 13:39:51 +01:00
Eric Allam 59df4af1eb chore(engine): add additional logging when we fail to get snapshots since (#2551) 2025-09-25 12:59:02 +02:00
Eric Allam 8863ff05c9 fix(engine): limit the number of snapshots returned when getting latest snapshots since (#2550) 2025-09-25 11:53:35 +01:00
Saadi Myftija 700a6ea598 feat: enable canceling deployments (#2545)
* Add canceledAt to the deployment db schema

* Expose an api endpoint to cancel deployments

* Show the canceled status description in the dashboard

* Enable canceling deployments from the dashboard

* Show cancelation reason in the deployment details

* Make verifyProjectMembership a function for consistency

* Apply some good 🐰 suggestions
2025-09-24 11:14:29 +02:00
Saadi Myftija cc94d121f2 feat: installing status for deployments (#2544)
* Add installing status to the deployment db schema

* Replace the deployments /start endpoint with /progress

* Show the installing status in the dashboard

* Add installing status to the api schema and cli

* Add changeset
2025-09-24 10:27:43 +02:00
Eric Allam 87b3603b23 feat(webapp): completing spans server-side no longer write-after-read, improving efficiency and perf (#2530)
* Cancel run events which then propogate cancellation status to span ancestors

* WIP

* convert closing cached run spans to new system

* converted expired complete span event to new method

* move v3 over to new methods

* Convert getDetailedTraceSummary to use the new ancestor override stuff

* remove debug logs

* Don't return UNSPECIFIED task events in getRunEvents

* fix the call site for cancelling run event in v3

* Add changeset

* remove methods
2025-09-19 13:39:48 +01:00
Saadi Myftija a3ef6ea236 feat: separate deployment initialize and start steps (#2522)
* Enable setting the initial status on deployment creation

* Expose endpoint to start deployments

* Extend build timeout on deployment start

* Use separate timeout value for queued deployments

* Add startedAt to the deployment schema

* Show the new startedAt instead of createdAt in the dashboard

* Show github user tag also in the deployment details page

* Show `pending` deployment status as `queued` in the dashboard

* Apply some good 🐰 suggestions

* Add missing return
2025-09-18 16:12:17 +02:00
James Ritchie c8858edf0a New jump to parent or root run buttons (#2067)
* Change the color to indigo

* Pro tier pricing information now matches the marketing site

* Update the button styles to secondary

* WIP adding separate links to Parent and Root runs

* TextLink now supports optional shortcuts

* Adds shortcut keys to the root and parent links + the shortcut help panel

* Adds new icons for root and parent

* root friendlyId works

* Updates icons for jump to root and parent

* Copy tweak

* Improve how the Free tier shows no preview branches

* Improve the wording in the tooltip

* Align the x icon better

* Show price for additional preview branches

* Change the shortcut key

* Fixes button alignment

* Adds nested dependencies task hello-world

* Fixes typo “Cancelled”

* Removes taskIdentifier, not needed

* Removes unused taskIdentifier
2025-09-18 14:29:47 +01:00
Eric Allam 9c087646bf fix(engine): prevent race condition that prevents triggerAndWait runs from resuming by atomically creating associated waitpoint records (#2519) 2025-09-17 13:58:17 +01:00
Saadi Myftija 501a383bcd feat: expose project build settings (#2507)
This PR enables setting project build settings in the settings page:
root directory, install command and trigger config file path.

For most cases there should be no need to set these explicitly.
2025-09-16 15:02:09 +02:00
Eric Allam 83bd6f5f9e fix(engine): carry over completed waitpoints on PENDING_EXECUTING stalls (#2503) 2025-09-15 12:19:46 +01:00
Eric Allam e36d78e4fc fix: don't carry over the checkpoint ID when nack and requeuing (#2502) 2025-09-15 12:06:52 +01:00
Eric Allam 3188dc9b28 perf(webapp): Add BatchTaskRun index to speed up the batch list dashboard page (#2499) 2025-09-12 17:23:52 +01:00
Eric Allam 10e7985fbc Add index for waitpoint tokens dashboard query (#2498) 2025-09-12 16:48:42 +01:00
Eric Allam 2eddda1233 fix(webapp): worker actions now catch service validation errors and respond properly (#2481)
This also stops all the unnecessary error logging when throwing 
ServiceValidationErrors
2025-09-12 14:56:54 +01:00
Eric Allam f077d49291 feat(engine): Improve execution stalls troubleshooting, align dev and prod behavior, adding heartbeats.yield utility (#2489)
* feat(engine): Improve execution stalls troubleshooting, align dev and prod behavior, adding heartbeats.yield utility

* A few improvements via the 🐇 review

* Allow treating EXECUTION stalls as OOM errors, improve the error message, add more information to the docs, improve resource monitor and add it to the docs

* Add changeset
2025-09-12 14:54:39 +01:00
Eric Allam 8e66913e59 fix(run-engine): Preserve snapshot checkpoint ID when a PENDING_EXECUTING snapshot stalls (#2493) 2025-09-10 14:55:57 +01:00
Saadi Myftija 5567f49846 feat(webapp): expose project git settings (#2464)
* Fix settigns page delete project width issue

* Apply a couple of touch-ups to the project settings page

* Add UI flow to connect gh repos

* Enabling adding another gh account in the ui

* Enable connecting a repo to a project

* Enable updating git settings

* Enable disconnecting gh repos from a project

* Remove prisma migration drifts

* Hide git settings when github app is disabled

* Fix migration order

* Avoid using `location` to avoid SSR issues

* Make branch tracking optional

* Disable save buttons when there are no field changes

* Disable delete project button unless the input matches the project slug

* Show connected repo connectedAt date

* Check that tracking branch exists when updating git settings

* Show tracking branch hint in the deployments page

* Fix positioning issue of the pagination pane in the deployments page

* Use mono font for branch names

* Add link to git settings

* Show tracking branch hint for the preview env too

* Add a confirmation prompt on repo disconnect

* Add link to configure repo access in gh

* Add rel prop to github links

* Automatically open repo connection modal after app installation

* Apply some fixes suggested by mr rabbit

* Fix flash cookie issue

* Extract project settings actions into a service

* Extract project settings loader into a presenter service

* Introduce neverthrow for error handling

* Try out neverthrow for error handling in the project setting flows

* Move env gh branch resolution to the presenter service
2025-09-09 13:03:43 +02:00
nicktrn 59c17e04e9 feat(run-engine): worker queue resolver (#2476) 2025-09-04 09:27:06 +01:00
Saadi Myftija 436d951b65 feat(webapp): github app installation flow (#2463)
* Add schemas for gh app installations

* Implement gh app installation flow

* Make the gh app configs optional

* Add additional org check on gh app installation callback

* Save account handle and repo default branch on install

* Do repo hard deletes in favor of simplicity

* Disable github app by default

* Fix gh env schema union issue

* Use octokit's iterator for paginating repos

* Parse gh app install callback with a discriminated union

* Remove duplicate env vars

* Use bigint for github integer IDs

* Sanitize redirect paths in the gh installation and auth flow

* Regenerate migration after rebase on main to fix ordering

* Handle gh install updates separately from new installs
2025-09-02 16:35:33 +02:00
Eric Allam 9b1877bef2 fix(run-engine): retry non-zero exit code errors (#2467)
We’re also now saving the retryConfig from the BackgroundWorkerTask on 
TaskRun.lockedRetryConfig when the run is first locked to the version
2025-09-02 14:12:14 +01:00
nicktrn 73e7378459 feat(webapp): add ecs otel resource attributes (#2454)
* add attributes to runlock dequeue span

* feat(webapp): add ecs otel resource attributes
2025-08-29 11:05:47 +01:00
Eric Allam b0b0df69be fix(run-engine): prevent stalled SUSPENDED snapshots from endlessly retrying (#2448) 2025-08-28 14:25:50 +01:00
Eric Allam 82401ec4fe fix(run-engine): dequeue performance improvements and improved telemetry (#2446)
* more eager dequeuing, queue cooloff periods, return workerQueueLength when dequeueing

* Cache worker group authentication and remove old self-hosted worker code (only managed is currently supported)

* add additional spans during dequeue

* Add env vars and additional spans
2025-08-27 21:22:10 +01:00
Matt Aitken 24a915133e Prisma 6.14.0 upgrade (#2444)
* Initial work on upgrading to 6.14.0

Set the output to node_modules still to make it easier

* Use ./generated Prisma folder, update types to fix issues

* Docker compose restart Clickhouse

* Prisma instrumentation update

* Docker

* Removed database dockerignore file, add generated prisma client to the top-level one

* Delete v3-catalog package.json

* Resolved pnpm lock file

* Log errors for very slow queries
2025-08-27 16:52:58 +01:00
Saadi Myftija 1cc62230ab feat: introduce organization access tokens (#2391)
* Create schema and migration for organization access tokens

* Add helpers for creating and authenticating OATs

* Adapt the auth service to also accept OATs

* Accept OATs in the whoami v2 endpoint

* Enable deployments with the CLI using OATs

* Avoid reading env variables directly in the token utils

* Remove duplicate cli token utils

* Validate ENCRYPTION_KEY length when parsing env vars

* Make token utils a server-only module

* Disallow revoking already revoked OATs

* Simplify generics in authenticateRequest

* Use 32 bytes mock encryption key in the test setup

* Update dummy encryption key values in tests and templates

* Add a column in the OATs table to differentiate between user and system generated

* Simplify args for v3ProjectPath

Co-authored-by: Matt Aitken <matt@mattaitken.com>

* Add index on org id and createdAt

* Avoid storing the encrypted oat token and its obfuscated version in the DB at all

It is a safer approach. Also we do not need to ever read the decrypted token value after creation.

* Fix prisma update condition

* Add token type to the OAT table index

* Accept OATs in the mcp auth flow

* Simplify env auth flow around the /projects endpoints

---------

Co-authored-by: Matt Aitken <matt@mattaitken.com>
2025-08-27 13:23:45 +02:00
Eric Allam bf2120d5a7 fix(runengine): remove dev worker queues from metrics (#2434) 2025-08-21 14:11:22 +01:00
Matt Aitken 4264fc066e chore(webapp): More false errors turned to logs (#2422)
* Remove variables from dequeue log message

* Continue snapshot throw json

* Waiting for deploy error removed

* Realtime ECONNRESET is expected

* Redis worker logErrors changes, removed ids

* Preview branch without a branch shouldn't log an error, it's a user provided issue

* "Task run is not in a cancellable state" isn't an error, it's expected

* "CreateCheckpointService: Child run already resumed" is expected

* "CreateCheckpointService: Batch already resumed" is expected

* "Failed to insert events, will attempt bisection" changed to info, we have errors for complete failures

* Ignore "PrismaClient error"

* Don't log Redis worker DLQ errors if we're ignoring

* "Failed to parse machine config" is fine, sometimes a config is null or undefined

* "Failed to parse machine config" for v3

* MetadataTooLargeError shouldn't log an error
2025-08-20 11:44:26 +01:00
Matt Aitken 0ee4234117 Reduce false error logging (round 1) (#2415)
* Don't log an error when the snapshot shouldn't be created, it's normal for this to happen

* Slack alerts, skip `account_inactive` errors

* v3 finalize run with no locked isn't an error

* Another false error

* Finalize run CRASHED runs were logging errors

* All slack alert errors are warnings except invalid blocks
2025-08-19 09:58:21 +01:00
nicktrn 26d4d08bdc chore: move all placement tag helpers to core (#2403)
* chore: move all placement tag helpers to core

* move placement tag utils into server-only
2025-08-15 17:19:39 +01:00
nicktrn fa7f4b1fed feat(k8s): add placement tags for flexible node selection (#2390)
* add tier scheduling support to supervisor

* add billing info to dequeued message w/o cache

* add cache with best effort invalidation

* fix invalidate circular dep

* add changeset

* use new plan type on runs as fallback during dequeue

* tidy up

* be more explicit with plan type fallback

* remove additional billing check from hot path

* switch to placement tags

* update changeset

* update platform package

* start using new entitlement response

* ensure skipChecks optimization validates at batch level

* add optional items to add to queue manager limits

* make the bool env helper only accept boolean defaults

* remove redundant private field

* update placement tag helper to prevent unsupported tags
2025-08-15 09:52:14 +01:00
Matt Aitken cd6a0b3a3e fix: heartbeat race condition (acking) (#2380)
* The logger now supports metadata

* Added metadata to ServiceValidationError in some critical places

* Don't ack the heartbeat if there's a mismatch, it might prevent a brand new one

* Don't ack the heartbeat inside stalled. By returning it will be acked IF the deduplication key matches
2025-08-12 10:35:21 +02:00
Matt Aitken ae19d0f491 Only start an attempt if not finished. Send message to worker if pending executing (#2377)
* Only start an attempt if not finished. Send message to worker if pending executing

* Fix the exit process reason tet

* Fixed cancelling test since bug fix

The old behaviour was wrong for pending executing in the test
2025-08-11 11:46:10 +01:00
nicktrn 363b9b15b4 fix(engine): set correct run status after requeue and snapshot status after dequeue (#2367)
* set correct run status on snapshot after dequeue

* set run status back to PENDING when we requeue

* remove retrying after failure status from v4 and fix tests

* fix one last test
2025-08-07 16:46:27 +01:00
Matt Aitken af14621683 Specify a region when triggering (#2366)
* Map new allowedMasterQueues → allowedWorkerQueues

* ClickHouse worker_queue on task runs

* Added the Region to the run inspector

* Pass a region in when triggering

* Added a changeset

* Added triggering regions docs

* Added region to the ctx

* Fix for backfiller masterQueue/workerQueue
2025-08-07 12:41:39 +01:00