* Cancel run events which then propogate cancellation status to span ancestors
* WIP
* convert closing cached run spans to new system
* converted expired complete span event to new method
* move v3 over to new methods
* Convert getDetailedTraceSummary to use the new ancestor override stuff
* remove debug logs
* Don't return UNSPECIFIED task events in getRunEvents
* fix the call site for cancelling run event in v3
* Add changeset
* remove methods
* fix: external traces now respect parent sampling, and prevent broken traces when there is no external trace context
* Add changeset
* improve trace flag handling and better internal host checking
* the traceFlags are now being properly passed through as a number
* add tier scheduling support to supervisor
* add billing info to dequeued message w/o cache
* add cache with best effort invalidation
* fix invalidate circular dep
* add changeset
* use new plan type on runs as fallback during dequeue
* tidy up
* be more explicit with plan type fallback
* remove additional billing check from hot path
* switch to placement tags
* update changeset
* update platform package
* start using new entitlement response
* ensure skipChecks optimization validates at batch level
* add optional items to add to queue manager limits
* make the bool env helper only accept boolean defaults
* remove redundant private field
* update placement tag helper to prevent unsupported tags
* Map new allowedMasterQueues → allowedWorkerQueues
* ClickHouse worker_queue on task runs
* Added the Region to the run inspector
* Pass a region in when triggering
* Added a changeset
* Added triggering regions docs
* Added region to the ctx
* Fix for backfiller masterQueue/workerQueue
* WIP
* Make release concurrency system extremely simple, everything just releases all the time
* update the deadlock detection to use the new lockedQueueReleaseConcurrencyOnWaitpoint column
* WIP new release concurrency system
* Remove releaseConcurrency and releaseConcurrencyOnWaitpoint
Also removed deadlock detection, and added environment burst concurrency
* Added new DEQUEUED status
Cleaned up the API run statuses, including now detecting new clients and not breaking older clients by adding an API version header to all requests
* Introduce the new "current dequeued concurrency set"
* Remove QUEUED_EXECUTING because we no longer "eagerly" release before checkpointing
* Remove waitpoint test for QUEUED_EXECUTING
* Add isWaiting
* Add changeset
* Use createdAt for ordering realtime runs instead of number
* Clarify the envCurrentDequeuedKey usage
* mock the db.server file to fix the tests
* Updated changset "EXECUTED" -> "EXECUTING"
---------
Co-authored-by: Matt Aitken <matt@mattaitken.com>
* Add ID of the replayedFrom run to the TaskRun schema
* Propagate the replayedFrom run ID in the replay flow
* Link the replayed run in the run details pane
* useSearchParams has
* useSearchParams has
* useSearchParams has
* Consistent way to get the run filters
* Consistent way to get the run filters
* Consistent way to get the run filters
* Initial work on the new bulk actions
* Initial work on the new bulk actions
* Initial work on the new bulk actions
* WIP actions and filtering
* WIP actions and filtering
* WIP actions and filtering
* Empty filter arrays are set to undefined
* Empty filter arrays are set to undefined
* Empty filter arrays are set to undefined
* WIP prisma schema
Removed extra runtimeEnvironmentId
* WIP prisma schema
Removed extra runtimeEnvironmentId
* WIP prisma schema
Removed extra runtimeEnvironmentId
* Migrations
* Migrations
* Migrations
* BulkActionGroup changed some columns around
* BulkActionGroup changed some columns around
* BulkActionGroup changed some columns around
* New badge variant, removed unused ones
* New badge variant, removed unused ones
* New badge variant, removed unused ones
* Bulk action button
* Bulk action button
* Bulk action button
* Make the next runs page the default now
* Make the next runs page the default now
* Make the next runs page the default now
* Improved the RadioButton style
* Improved the RadioButton style
* Improved the RadioButton style
* Remove the old bulk action bar
* Remove the old bulk action bar
* Remove the old bulk action bar
* More UI progress
* More UI progress
* More UI progress
* Lots of UI changes to the Runs page
* Lots of UI changes to the Runs page
* Lots of UI changes to the Runs page
* Fixed period filter resetting everything
* Fixed period filter resetting everything
* Fixed period filter resetting everything
* Improved the Switch secondary style
* Improved the Switch secondary style
* Improved the Switch secondary style
* Buggy filter fixes
* Buggy filter fixes
* Buggy filter fixes
* Improved the filter display and fixed a bug with search param from object
* Improved the filter display and fixed a bug with search param from object
* Improved the filter display and fixed a bug with search param from object
* Clear button is minimal
* Clear button is minimal
* Clear button is minimal
* Using a presenter now
* Using a presenter now
* Using a presenter now
* Bulk actions are created, but not actually processed (yet)
* Bulk actions are created, but not actually processed (yet)
* Bulk actions are created, but not actually processed (yet)
* Bulk replay/cancel is working
* Bulk replay/cancel is working
* Bulk replay/cancel is working
* Multiple fixes, added bulk column to PG
* Multiple fixes, added bulk column to PG
* Multiple fixes, added bulk column to PG
* Bulk action run filtering working using CH
* Bulk action run filtering working using CH
* Bulk action run filtering working using CH
* Replay setting the bulk id on the runs
* Replay setting the bulk id on the runs
* Replay setting the bulk id on the runs
* Properly cap the time when doing a bulk action
* Properly cap the time when doing a bulk action
* Properly cap the time when doing a bulk action
* If the bulk action isn't recent, add it to the dropdown anyway
* If the bulk action isn't recent, add it to the dropdown anyway
* If the bulk action isn't recent, add it to the dropdown anyway
* Blank version of the bulk actions page
* Blank version of the bulk actions page
* Blank version of the bulk actions page
* Individually selected runs working
* Individually selected runs working
* Individually selected runs working
* Use selected mode if runs are checked
* Use selected mode if runs are checked
* Use selected mode if runs are checked
* Added the modal
* Added the modal
* Added the modal
* Marked the old bulk actions stuff as deprecated
* Marked the old bulk actions stuff as deprecated
* Marked the old bulk actions stuff as deprecated
* Renamed bulk action file
* Renamed bulk action file
* Renamed bulk action file
* Bulk run filter with the name and a default
* Bulk run filter with the name and a default
* Bulk run filter with the name and a default
* WIP on bulk actions page
* WIP on bulk actions page
* WIP on bulk actions page
* Updated panel, added new truncated id component
* Updated panel, added new truncated id component
* Updated panel, added new truncated id component
* Style improvements to the radio buttons
* Style improvements to the radio buttons
* Style improvements to the radio buttons
* Added an option action completion email
* Added an option action completion email
* Added an option action completion email
* Adds a blank state for the bulk actions page
* Adds a blank state for the bulk actions page
* Adds a blank state for the bulk actions page
* Nicer completed email
* Nicer completed email
* Nicer completed email
* Don't open the bulk action panel if there are no runs
* Don't open the bulk action panel if there are no runs
* Don't open the bulk action panel if there are no runs
* Runs blank state and bulk action accordion
* Runs blank state and bulk action accordion
* Runs blank state and bulk action accordion
* Updates secondary/small switch style
* Updates secondary/small switch style
* Updates secondary/small switch style
* Pagination buttons no longer split in twain (WIP)
* Pagination buttons no longer split in twain (WIP)
* Pagination buttons no longer split in twain (WIP)
* Aborting working
* Aborting working
* Aborting working
* Bulk action live reloading
* Bulk action live reloading
* Bulk action live reloading
* ListPagination works correctly in all states
* ListPagination works correctly in all states
* ListPagination works correctly in all states
* Run page, show friendlyId instead of number
* Run page, show friendlyId instead of number
* Run page, show friendlyId instead of number
* Bulk action help open by default if you have none
* Bulk action help open by default if you have none
* Bulk action help open by default if you have none
* Extra status filtering step because of replication delay
* Extra status filtering step because of replication delay
* Extra status filtering step because of replication delay
* Wider bulk action onboarding
* Wider bulk action onboarding
* Wider bulk action onboarding
* More sensible widths on the bulk action side panel
* More sensible widths on the bulk action side panel
* More sensible widths on the bulk action side panel
* Border color tweak to the RadioButton
* Border color tweak to the RadioButton
* Border color tweak to the RadioButton
* Improved the accordion component hover states
* Improved the accordion component hover states
* Improved the accordion component hover states
* Updates the bulk action blank state images to the latest UI
* Updates the bulk action blank state images to the latest UI
* Updates the bulk action blank state images to the latest UI
* Added R and C shortcuts back in
* Added R and C shortcuts back in
* Added R and C shortcuts back in
* Fix for selecting a single run
* Fix for selecting a single run
* Fix for selecting a single run
* Improved exit icon, added shortcut to modal
* Improved exit icon, added shortcut to modal
* Improved exit icon, added shortcut to modal
* Tidy imports
* Tidy imports
* Tidy imports
* Tidy imports
* Tidy imports
* Tidy imports
* Tidy imports
* Tidy imports
* Tidy imports
* Tidy imports
* Fix for grid layout when 1 page of bulk actions visible
* Fix for grid layout when 1 page of bulk actions visible
* Fix for grid layout when 1 page of bulk actions visible
* Removed the ... on the abort button
* Removed the ... on the abort button
* Removed the ... on the abort button
* Removed the ... on the abort button
* Animate the progress bar
* Set TZ="UTC" in the env example
* Filter summary in the bulk inspector
* Improves the pagination styling
* Improves the pagination styling
* Delete old bulk action routes
* Removed old Postgres RunListPresenter
* Retry any replication error where the message contains "timeout"
* Increase wait to make test less flaky
* The test was using run id instead of friendly id
* Safer array access
* Remove error log if there's a bad status
* Nicer frontend type safety with the bulk action and mode
* Switched a log to a debug log
* Retry replication unless the error is a known non-retry error
Flip the strategy to retry by default
* Make ClickHouse required
* Backfill run replication admin API endpoint
* Set a CLICKHOUSE_URL for unit tests
---------
Co-authored-by: James Ritchie <james@trigger.dev>
* Introduce request idempotency to prevent duplicate triggers
* Implement request idempotency on trigger
* Use x-trigger-request-idempotency-key header instead
Plus cleanup shared logic in the routes
* Add changeset
* Oops, lets not hardcode a 408
* A couple of improvements
* Fix incorrect logs on new schedule engine triggered taskss
Also added the ability to recover schedules in the schedule engine via an Admin API endpoint in the new schedule engine
* Fixed schedule recovery failing test
* WIP
* Run queue now works with the worker queue / master queue split
* Acking should also cause the master queue to be processed
* Convert run engine tests and run engine to use runQueue changes
* Include the util files in the test tsconfig
* coordinator target should be es2020 as well
* providers target 2020
* Fix the triggerTask tests in the webapp
* v4 now working with the new worker queues, and added the legacy master queue migration stuff
* report worker queue lengths via opentelemetry metrics
* Adding lock metrics
* Release concurrency bucket metrics
* • Updated RunQueue.removeEnvironmentQueuesFromMasterQueue() method signature to take runtimeEnvironmentId instead of masterQueue parameter
• Added automatic master queue shard calculation using this.keys.masterQueueKeyForEnvironment(runtimeEnvironmentId, this.shardCount)
• Updated RunEngine wrapper method to use new runtimeEnvironmentId parameter
• Updated DeleteProjectService to call the method once per environment instead of once per master queue
• Simplified API by encapsulating master queue sharding logic within RunQueue class
* metrics now working, configure the run queue settings, additional metrics for run engine and redis-worker
* Fix CodeRabbit suggestions
* return undefined from dequeueFromWorkerQueue, not null
* Remove message from worker queue in certain circumstances when acking
* Update log
* Ensure master queue consumers cannot stop from a processing error, and make the consumer interval configurable via an env var
* Change how the run queue master queue consumers are disabled internally
* Fixed tests
* process the queue on nack
* Fix more tests
* Fix priority tests
* Fixed dequeueing test
* WIP clickhouse package with test containers setup
* More clickhouse client setup now with otel and real tests, and the v1 of raw run events
* Add some additional columns to raw_run_events_v1
* WIP runs dashboard service
* Create a new run engine event bus event for the runs dashboard to hook into
* Track run events in the run engine
* make sure engine v1 runs get synced to CH
* Update the attemptNumber of v3 task runs
* Restructure the run events to be more sparse
* emit more stuff
* Setup replication package
* scaffold the replication package
* replication wip
* resolve conflicts
* more replication stuff
* Add ability to drop the replication slot completely on teardown
* Use the new single replacingmergetree task events table for replication
* get it working
* insert payloads into their own table only on insert and then join
* prepare for using clickhouse cloud and now running ch migrations during boot in the entrypoint.sh
* Handover WIP and tests
* Testing the replication service
* Remove the runs dashboard stuff that we aren't using anymore
* Added a test for large payloads
* hacky typecheck fix
* Fix new internal package typecheck issues and start adding telemetry to the replication service
* tracing over spans, some other improvements
* Improvements to the runs replication service, now ready for testing
* Some fixes and cleanups
* Don't need this code anymore
* move transaction types into the runs replication service
* only send spans where there are transaction events
* A couple of suggested tweaks
* Locked task runs will now require queues and tasks to be in the locked version
* Client errors caught in a run function now will skip retrying
* Extracted out the trigger queues logic
* extract validation, idempotency keys, payloads to concerns
* Extracted out a bunch of more stuff and getting trigger tests to work
* Add queue and locked version tests
* Deadlock detection WIP
* more deadlock detection
* Only detect deadlocks when the parent run is waiting on the child run
* Improve the error experience around deadlocks
* A couple tweaks to make CodeRabbit happy and fixing the tests in CI
* Fixed failing test
* Changeset
* wip
* Make sure to scope queries to the runtime env
* Completed batch waitpoints when we completed the BatchTaskRun
* Try complete the batch faster now it’s being used operationally
* Fix for tests that were using the old engine.unblockRunForCreatedBatch() function
* Improve the new run engine Trigger/Batch trigger service class names
* centralize queue timestamp logic in EnqueueSystem, adding queueTimestamp support and propagation
* Fixing the create tags in the new run engine trigger task service
Eric Allam