* WIP
* Make release concurrency system extremely simple, everything just releases all the time
* update the deadlock detection to use the new lockedQueueReleaseConcurrencyOnWaitpoint column
* WIP new release concurrency system
* Remove releaseConcurrency and releaseConcurrencyOnWaitpoint
Also removed deadlock detection, and added environment burst concurrency
* Added new DEQUEUED status
Cleaned up the API run statuses, including now detecting new clients and not breaking older clients by adding an API version header to all requests
* Introduce the new "current dequeued concurrency set"
* Remove QUEUED_EXECUTING because we no longer "eagerly" release before checkpointing
* Remove waitpoint test for QUEUED_EXECUTING
* Add isWaiting
* Add changeset
* Use createdAt for ordering realtime runs instead of number
* Clarify the envCurrentDequeuedKey usage
* mock the db.server file to fix the tests
* Updated changset "EXECUTED" -> "EXECUTING"
---------
Co-authored-by: Matt Aitken <matt@mattaitken.com>
* Adds a new route for logging in with mfa
* New path for security page
* Adds “Security” link to account side menu
* Update the Switch component to allow label positions left and right
* Optionally hide the Close button in the Dialog title bar
* Installs `qrcode` react package for generating QR codes.
* CopyButton component now takes children
* New Security route for setting up MFA
* Adds new OTP package for the chadcn InputOTP component
* Adds new InputOTP chadcn component
* Adds InputOTP chadcn component to the MFA login screen
* InputOTP component supports variant styles
* Improvements to form handling
* Show a confirmation modal before you can disable MFA
* Revert redirect back to the dashboard for now
* Implement MFA enabling and disabling
* Refactor and cleanup mfa management code
* More cleanup
* Handle errors in the management action
* Implement mfa login flow
* recovery code input should be password
* Implement rate limiting on the mfa validation endpoint
* Better error ux
* Implement mfa emails and apply James' updates
* Use latest @better-auth/utils
* Improvements via CodeRabbit review
---------
Co-authored-by: James Ritchie <james@trigger.dev>
* Add createdAt filter to realtime subscribing with tags
* Filter realtime colums and expose ability to skip some columns
* Add sharding support for electric
* Use unkey cache for the created at filter caching
* Remove 2 unused indexes on TaskRun
* Run list now filters by a single runtime environment
* Remove project ID indexes
* Use clickhouse in task list aggregation queries instead of pg (keep pg for self-hosters)
* WIP clickhouse powered runs list
stuff
* Improve the query to get the latest tasks for the task list presenter
* Update the usage task list to use clickhouse
* Implement next runs list powered by clickhouse
* Add new index for TaskRun for the runs list, by environment ID
* Add runTags gin index
* Handle possibly malicious inputs
* Ignore claude settings
* Better handling not finding an environment on the schedule page
* Use ms since epoch in test, not seconds
* Remove unused function
* Fix test
* Use an env var for the realtime maximum createdAt filter duration (defaults to 1 day)
* Fixed the query builder to correct the group by / order by order
* Make sure runs.list still works
* Create small-birds-arrive.md
* Initial preview migrations
* Modified the staging endpoint to create preview environments
* Added isBranchableEnvironment to RuntimeEnvironment
* Staging = yellow Preview = orange
* Changed the env sort order
* Set isBranchableEnvironment correctly. Create preview for new projects
* Very basic branch menu
* Creating branches from the dashboard
* Fix for string icons on project delete page
* Don’t show branch API keys
* WIP on the manage branches page
* RuntimeEnvironment added projectId index
* Only create the parentEnvironmentId column if it doesn’t exist already
* Improved the limit wording
* Add search to the branch list
* contains in both places
* Many style improvements
* Branch dropdown and v4 badge
* Arching/unarchive branches working in the dashboard
* Tidied imports
* Change preview slug from `prev` to `preview`
* Use correct color for side menu preview branch icon
* Upsert the branch and use the shortcode as a unique constraint
* Upserting working with nice messages in the dashboard
* Better errors when upserting branches
* Button shortcut, don’t allow event to propagate
* Better duplicate error message
* Filter out archived branches from the env selector
* Archiving/creating tweaked some more
* Add an archived banner to the app, fixes for archived branches and upsells
* Fixed pagination
* Disable editing schedules, pausing queues, testing tasks
* Don’t allow replaying if the env is archived
* When deploying detect the correct environment
* Get the projectClient when there’s a branch
* createGitMeta function, most code from the vercel CLI repo
* Deploy, getting the correct environment client
* Added git column to WorkerDeployment
* Add GitMeta to core schemas
* Create branch when deploying
* WIP on branch support in the API
* Delete old createTaskRunAttempt fn
* apiAuth remove export from internal functions
* Rename env var to “TRIGGER_PREVIEW_BRANCH”
* Add TRIGGER_PREVIEW_BRANCH to resolved env vars for runs
* First preview deploy and run working
* Set the preview branch in the main SDK
* Added git links to the preview branches table
* Better errors when replaying/testing archived branches
* Don’t dequeue archived environments
* Env var resolution with parent environment
* Hello world default machine small-2x to save my memory
* Fix for more env var functions
* Only return non-archived envs
* Switch to controlled state for the checkboxes
* Uncheck everything when PREVIEW is checked
* WIP on branch UI
* Show the preview branch label on the env vars list
* Fix for overriding env vars
* Adding preview branch env vars working
* Progress on new env vars
* Only allow selecting a single branch
* Layout fix when there are errors
* Set the defaultValue so there are some fields
* Conform fix for team invite page
* Archived environments don’t run scheduled tasks
* Added Git data to deployments
* Added git data to the deployment inspector
* Don’t allow upserting schedules when archived
* Deduplicate and blacklist some env vars
* Fix for wrong conform function being used
* Show a better error if all vars were blacklisted
* Added environment variable search (by key and value)
* Improved preview branch icon
* Replay now supports branches
* Schedule page render branches properly
* Show the env icon in bottom-left of the test page
* When editing older schedules (that have multi-env) show preview branches correctly
* Fix for incorrect disallowed branch name character
* Extract and improve the directory verification code
* WIP for CLI preview archive command
* Improved the preview branch action buttons
* Redirect to the project if we don’t find a matching env
* Archiving branch via the CLI working
* Fix for archiving branches
* Public access token test task
* JWTs working are with preview branches
* Add branch and git data to the Run ctx
* Updated GitMeta functions to work in CI
* Added pullRequestState
* Archive when deploying if the PR is closed/merged
* Fix for the changesets guide
* Fix for CLI dev bug introduced
* CLI promote now supports preview branches
* Add PR title. Reordered them and added tooltips
* syncEnvVars working with branches
* Added preview branch support to syncVercelEnvVars()
* Detect the branch from Vercel env var (set during build)
* Allow passing a branch in
* Use process.env.VERCEL_TOKEN as well… this used in Vercel CI
* Temp delete
* Improved regenerate api key modal
* Added Accordion component (with styles)
* Redesigned the API keys page
* Revert "Temp delete"
This reverts commit 177b92cd935a6161456bde65d01294e23ecfd47f.
* Changeset
* Fixed docs link
* The new branch panel closes when a branch is created
* Update apps/webapp/app/services/upsertBranch.server.ts
Co-authored-by: coderabbitai[bot] <136622811+coderabbitai[bot]@users.noreply.github.com>
* Removed findUniques from WorkerGroupTokenService
* Made the parentEnvironmentId migrations safe
* Latest lockfile
* Update packages/cli-v3/src/commands/workers/build.ts
Co-authored-by: coderabbitai[bot] <136622811+coderabbitai[bot]@users.noreply.github.com>
* Move isValidGitBranchName to a separate file
* Move the sanitize fn too
* removeBlacklistedVariables moved to a separate file
* Moved deduplicateVariableArray to a separate file…
* Fix broken sanitizeBranchName import
* Another import fix…
* Improved blacklisted error message
* SImplified migration to use `ADD COLUMN IF NOT EXISTS "parentEnvironmentId" TEXT`
---------
Co-authored-by: coderabbitai[bot] <136622811+coderabbitai[bot]@users.noreply.github.com>
* Added waitpoints/tokens to the sidebar
* Added indexes to the Waitpoint time for filtering
* Begun work on `WaitpointTokenListPresenter`, the pag is a copy of the Queues page for now
* MVP of waitpoint token page
* Added status
* Expiry of timeout/ttl
* Improvements to the waitpoint table
* Improved columns and icon
* Changes from the RunTag copy on hover branch
* Fix for nested button error
* Added waitpoint tags to the DB/table
* Applied Eric’s task run tag fix (it’s live on prod in the legacy run engine branch)
* Added tags to waitpoints
* Removed todos that have been done
* Added token support for releaseConcurrency. Also added a ton of JSDocs
* Added releaseConcurrency to the API token endpoint…
* WIP on waitpoint page filters
* Fix for tags filtering
* Waitpoint filters working
* Fix for badly named function
* WaitpointPresenter used from SpanPresenter
* Waitpoint detail panel WIP
* Fix for server client hydration issue with CodeBlock
* Selected waitpoint panel
* Added a blank state
* Added waitpoint docs link
* Fix for animated number going past the target
* Fix for the queue list pagination and upgrade status
* Engine version error for waitpoint token list
* RunTag component doesn’t get squished and hover behaviour is nicer
* Associating runs with waitpoints
* Added triggered icon
* Link directly to the waitpoint
* Fix for TS error on waitpoint retrieve
* Added CopyableText component, used for waitpoint id in the table
* Removed the confetti 🎊
* Deleted some old images
* Moved some schemas/types to core. Use `id` instead of `friendlyId`
* Added wait.listTokens() function. Made some changes to the types to make it nicer
* WIP wait.retrieveToken()
* wait.retrieveToken working
* Added data to retrieve token
* Separate ApiWaitpointPresenter completely
* Added completed time to the waitpoint detail panel
* Fix for the Avatar component having SSR issues. Specify the size in rems and removed the useLayoutEffect
* Fix for applied idempotency key filter dropdown showing the id field
* Use parentheses to make sure the token list query respects idempotency key correctly
* Use the proper logger, and have a decent message with info to track the bug down
* Pass the org title into the Avatar
* Better error when failing to creating a manual waitpoint after X attempts
* Improve the new run engine Trigger/Batch trigger service class names
* centralize queue timestamp logic in EnqueueSystem, adding queueTimestamp support and propagation
* Fixing the create tags in the new run engine trigger task service
* Delete the proxy app (was v2)
* Delete RunPresenterElectric
* Select the best proj/org/env
* Storing current proj/env in DB. Initial selection logic working with tasks page
* 2sm needed to be in the Tailwind merge list
* Move the task stream route (although we don’t actually use the env for now)
* Alerts moved from /v3
* API keys page moved from /v3
* Concurrency page moved from /v3
* WIP on side menu sections
* Improved the accordion animation
* Moved schedules from /v3
* More pages moved
* Move pages working
* Run page working
* Schedules working
* Moved deployments
* Alert pages moved
* Delete electric hooks, not used
* Started setting up blank states
* Test page working
* Removed “Select task” from the test page
* Some work on deployment page
* Style tweaks
* Redirect from project root to approriate env
* Improved env selector styling
* Fix for jsx errors
* Better min width on env selector
* Improved the env switching logic
* Added deployments to env routing
* Redirect deployments to the correct env
* Redirect run from proj to env
* JSX icon fix
* Only allow single env schedules from now on
* Remove env var count from the API keys page
* Move improvements and redirects
* Project settings moved
* Fix for scroll area on test page
* Tweaked the test design
* Made recent payloads column narrower
* Improved the test layout some more
* Added org icon, new project selector menu
* WIP on org switching menu
* Org switching is working
* New menu working well, removed old side menu items
* Buttons can now have a component name or an actual component for their icons
* Removed the Projects page, instead redirect appropriately
* Fix for broken blank states
* Minor run table improvements
* Removed unused switcher log and logic
* Concurrency page fix for invalid html, improved layout
* Minor improvements
* Moved the side menu to the project level
* Improved account styling
* Moved org settings pages (with redirects)
* Add current plan to billing side menu link
* Upgrade to get staging from env dropdown
* New env badge on concurrency limits page
* Show Run Engine version in span presenter
* New promote icon
* Concurrency limits page is the sum of engine v1 + v2 queues
* Fix for missing batch import
* Added currentConcurrencyOfEnvQueue function
* Basic avatar setting working
* Avatar setting is working
* You can change the color of your icon
* Avatar improvements
* Bugfix for mising prop
* Removed some old env badges
* Fixed replaying
* Removed EnvironmentLabel
* Old env badge deleted, changed everywhere to the new one
* Fix for Slack integration paths
* Fix for waitpoint completion form moving
* Bulk replay/cancel env fix
* Fix for alert webhook path
* Redirect projects/v3/* to project/*
* Fixes for CLI redirect routes
* Remove welcome email (unused)
* Change how we count schedules towards your limits
* Use new schedules limits when checking a schedule
* Added projectId back in to task queries (indexes)
* WIP dev presence
* CLI modal
* Moved things around and use Context
* Fix for p inside p
* Dev connected status on run page
* Correct dev env (not a teammates)
* Show disconnected message at the end
* Minor tweak on project dropdown icon padding
* Fix for inconsistent date format for presence
* Added a message when pushing to the billing page
* Center the team page
* Project settings page centered
* Improvements to the dev presence
* Delete v2 Stripe routes
* Delete v2 billing/usage pages
* Delete v2 integration pages
* Delete v2 project pages
* Deleted a load of components and services
* Deleted a load more components, presenters and services
* Deleted another 100 files or so…
* Removed old v2 paths
* Removed named icons from form titles
* Removed more string icons
* Delete NamedIcon
* Fixed some type errors
* Delete endpointApi
* Removed v2 from core/sdk
* Post merge fixes
* added explicit return types
* using the new sdk export without v3
* Delete old v2 file
* Added explicit return types because TS was complaining…
* Don’t export RuntimeEnvironmentType from two core files. Was causing TS issue
* Fix for removal of NamedIcon in new route
* Removed strange eslintrc rule
* Use the new redis client
---------
Co-authored-by: James Ritchie <james@trigger.dev>
* bump worker version
* Suggested glossary for the RunEngine, TBC
* Removed BatchTaskRun changes from this branch, they were done in main
* Set the BatchTaskRun status to completed when all runs are completed
* When dequeuing respect passed in maxResources
* Ported over the new run props: idempotencyKeyExpiresAt, versions, oneTimeUseToken, maxDurationInSeconds
* Didn’t hit save… the new props when triggering tasks passed through
* Idempotency expiration + waitpoint edge case
* WIP on creating checkpoint, parking for now
* fix worker routes
* upgrade webapp node types to support generic event emitter
* separate event bus handler singleton and run failure alerts
* duration waits
* fix execution snapshot debug spans
* task waits
* fix event bus types
* temporary fix for react hook run handle type
* disable run notifications for now
* convert any typecasts to expect errors to more easily fix later
* fix webapp types after node types upgrade
* updateEnvConcurrencyLimits across marqs and the runqueue
* Pass proper values into the run engine
* RunQueue settings and removed unused rebalancing workers
* Remove rebalancing prop
* Tidied more things up
* Update/remove queue limits for MARQS and RunQueue
* taskQueue/concurrencyLimit changes ported back into the RunEngine
* Reworked completing waitpoints to improve performance and reduce race conditions
* Improved test robustness
* Down to a single run lock only when a run is totally unblocked and ready to continue
* warm starts, worker notifications, wait fixes
* Fix for Run Engine poll interval env var
* Expect the waitpoint to be completed quickly
* If a run is locked then it’s too late to expire it
* Added VALKEY_ env vars and plugged them into the run engine
* Extracted and updated the guard queue function so it can be used when batching
* Added logging and universal concurrency changes to trigger task v1
* Added notes back in
* Bump @trigger.dev/worker to 3.3.7
* reportInvocationUsage for the runAttemptStarted event
* improve execution snapshot span debug span start times
* Unfriendly IDs
* update lockfile
* Created a shared determineEngineVersion function
* disable unfinished commands
* save new cli config to different location, misc fixes
* add basic engine version check via current deploy
* new run engine will default to node 22 runtime
* block some actions for projects on previous run engine
* fix worker group tests
* fix triggerAndWait test
* one typescript version to rule them all
* redlock type patch
* fix type issues caused by ts-reset
* improve cleanup scripts
* add missing socket.io dep
* fix run notification handler type
* fix worker group test again
* generate prisma client for e2e tests
* remove worker group tests for now
* prevent image pull rate limits during unit tests
* increase timeout for queue concurrency limit test
* generate prisma client for preview release
* same node types everywhere
* Updated engine readme, removed legacy system notes
* use default machine preset from platform package
* worker instances plural in schema
* disable pnpm update notifications
* return worker group details from connect call
* add workers admin route
* fix heartbeat route return type
* move deployment labels to core apps
* refactor run controller env schema
* Add firstAttemptStartedAt to TaskRun
* RunEngine 2.0 batch trigger support (#1581)
* Make it clear when BatchTriggerV2Service is used
* Copy of BatchTriggerV2Service
* WIP batch triggering
* Allow blocking a run with multiple waitpoints at once. Made it atomic
* Removed unused param
* New batch service
* Pass through the parentRunId and resumeParentOnCompletion
* Use the new batch service, and correct trigger task version
* Force V1 engine if using BatchTriggerV2Service, we’ve already done the check at this point
* Removed the $transaction and early exit if nothing changed
* Adedd a simple batch task to the hello world reference catalog
* Fix for batch waits not working
* Added parentRunId in a couple more places
* Removed waitForBatch log
* Added another parentRunId
* Expanded the example to include all the different triggers
* More changes to blocking to support continuing after idempotent completed runs
* Fix for the wrong type when blocking a run
* remove @map
* optimise worker auth query
* add engine version header to core api client requests
* remove unique constraint for default group id
* consolidate migrations
* the first managed worker becomes the global default
* Debug events off by default, added an admin toggle to show them
* worker group name can't be an empty string
* add exec helper to core
* move machine resources to core
* add pre-dequeue callback to determine max resources
* optionally skip dequeue
* bump worker package
* move worker to core
* fix ReadableStream type error
* fix another type issue
* update a few more tsconfigs
* add metadata changes introduced in #1563
* Run Engine 2.0 trigger idempotency (#1613)
* Return isCached from the trigger API endpoint
* Fix for the wrong type when blocking a run
* Render the idempotent run in the inspector
* Event repository for idempotency
* Debug events off by default, added an admin toggle to show them
* triggerAndWait idempotency span
* Some improvements to the reference idempotency task
* Removed the cached tracing from the SDK
* Server-side creating cached span
* Improved idempotency test task
* Create cached task spans in a better way
* Idempotency span support inc batch trigger
* Simplified how the spans are done, using more of the existing code
* Improved the idempotency test task
* Added Waitpoint Batch type, add to TaskRunWaitpoint with order
* Pass batch ids through to the run engine when triggering
* Added batchIndex
* Better batch support in the run engine
* Added settings to batch trigger service, before major overhaul
* Allow the longer run/batch ids in the filters
* Changed how batching works, includes breaking changes in CLI
* Removed batch idempotency because it gets put on the runs instead
* Added `runs` to the batch.retrieve call/API
* Set firstAttemptStartedAt when creating the first attempt
* Do nothing when receiving a BATCH waitpoint
* Some fixes in the new batch trigger service… mostly just passing missing optional params through
* Tweaked the idempotency test task for more situations
* Only block with a batch if it’s a batchTriggerAndWait… 🤦♂️
* Added another case to the idempotency test task: multiple of the same idempotencyKey in a single batch
* Support for the same run multiple times in the same batch
* Small tweaks
* Make sure to complete batches, even if they’re not andWait ones
* Export RunDuplicateIdempotencyKeyError from the run engine
* Latest lockfile
* Trigger with a machine (old run engine)
* RE2, allow setting machine when triggering
* Fix for new glob patterns
* add max run count to dequeue from version route
* add worker instance name env var and header
* queue consumer pre skip callback
* poll for more runs after final execution errors
* fix dequeue search param schema
* add shortcut to debug switch
* expose run engine timeouts as env vars
* make warm start durations configurable
* add optional status to json reply helper
* fix preSkip hook, add debug logs
* BLOCKED_BY_WAITPOINTS -> SUSPENDED
* exit controller when run suspended
* check if already replied before http reply
* run controller will wait for next run after the current one is suspended
* cancel run button shortcut
* minimal event repository environment type
* fix update metadata call
* run suspension and misc fixes wip
* change debug shortcut to shift + D
* Started work on the Dev supervisor
* Formatting
* Fix for bad imports
* Before rebuilding SSE
* Presence updating from the CLI working via SSE
* add worker notification debug logs
* send run:stop when exiting run phase
* skip current snapshot poll on worker notification
* add more logs and route to submit run debug logs
* add worker and runner ids to snapshots
* improve run notification debug logs
* add workload debug log route
* misc run controller fixes and refactor
* prevent parallel execution of critical functions
* update bun to 1.2.1
* WIP with dev dequeuing
* Method to convert friendlyIds to non-friendly, do nothing with actual ids
* Set the engine on BackgroundWorker, lazily upgrade projects to engine V2
* Runs with ttls were getting immediately expired… oops.
* Pass the Waiting for deploy reason through, so we have it on the execution snapshots
* Fixed the logic for getting the right background worker for a run
* Use the correct ID when dequeuing…
* determineEngineVersion is now fully functional
* Rate limiter ignores the dev endpoints
* Retrieving a batch gives you the runIds
* Set a unique version for the RE2 BatchTaskRun
* add provisional changeset
* The start of dev run execution is working
* First dev run working
* Moved the dev run controller closer to what Nick did with the managed one
* export exec output type
* Heartbeat fix: don’t heartbeat if _isHeartbeating == false
* Dev runs get notifications, some dev bug fixes
* Improved logging or dequeuing
* We need to dequeue runs from the latest version too, for triggerAndWait
* Ported Eric’s validateWorkerManifest with nicer errors
* When flattening an idempotency key if part is undefined, return undefined
* Dev logging fixes
* Remove sigterm listener
* Deprecating workers. Don’t specify a BackgroundWorker when dequeuing an environment
* Deleted some old files. Renamed “managed” to “deploy”
* When a build finishes, always copy the build dir (otherwise the first one gets trampled on by the 2nd)
* Dev master queues should work differently
* Deleting old workers
* Added debounce function to core
* Improvement to canceling
* WIP on debounce canceling on socket disconnection
* Added environment data to execution snapshots
* Dev runs that have stalled get “Canceled” with a reason explaining why
* Show CLI messaged when a connection to the platform is lost/restored
* Fix TriggerTask after merge
* Add trigger task v2 max attempts, replace some findUniques
* Port the new queue logic to the run engine
* More fixes post-merge
* We weren’t setting a `retryConfig` up for the tests… it’s now required
* Start the Redis worker inside the Run Engine… 🤦♂️
* Trying to make the testcontainers more reliable
* Added keyPrefix: "engine:”
* Badly placed bracket in trigger task
* Better Redis namespacing
* Fix for expired run not getting removed from the queue
* Don’t create a redis client in the testcontainers, return the redisOptions instead
* Cleanup redis client in the run lock tests
* Fix for the RunQueue not supporting keyPrefix
* Updated more of the RunQueue scripts rebalancing
* Trying to make Redis more robust in the tests…
* Improved test resiliciency more
* Fix for delays (checkpoint check)
* Increase the timeout slightly to fix ttl test
* Added priority support when triggering
* More wip trying to make test containers more reliable
* batchTriggerAndWait test is still failing… some wip to try fix it
* Fixed redis tests now we’re not providing a client
* Separate Redis clients for the run engine worker/queue/runlock
* Made the wait for duration test more resilient
* Added idempotencyKeyExpiresAt to Waitpoints
* Waitpoint timeouts and idempotency expiry
* Use finishWaitpoint, removed extra worker job
* Added waitpoint idempotency tests
* Creating resume tokens is working
* Some improvements to the resume tokens
* Moved resumeTokens to just be wait functions 🥳
* Delete old RuntimeManagers
* Wait for token is working
* Better test for the wait tokens
* Improved the test task some more
* Hide the accessories in the span inspector
* WIP on waitpoint inspector
* WIP on complete waitpoint form
* Span overview panel can be changed based on the entity type
* Improved the waitpoint display
* WIP on completing waitpoint form
* Use the existing CodeBlock for the tip
* Style improvements
* Complete waitpoint
* All waitpoint sidebar variants
* Waits now use a pause icon
* Durations waits use the API to create/block with a waitpoint, not the runtime
* Fix for engine.blockRunWithWaitpoint required org id
* Removed old wait code from the run controllers/task run process
* Form action for skipping a datetime waitpoint
* Move testDockerCheckpoint to a separate core package export (it can’t be bundled on the client)
* Fix for glitchy hourglass animation
* Completed waitpoints display better
* Increase Redis maxRetriesPerRequest to 20 (default)
* Completing and skipping waitpoints is working
* Remove the database prisma dev command, since we need to use create only now. Updated docs
* Added skip timeout, reworked the UI
* Tweaked spacing
* Added payload limit to waitpoint token completion from dashboard
* Test idempotency works on wait.for and wait.until
* Moved the worker-actions to /engine/ from /api/
* Moved dev engine endpoints to /engine/ from /api/
* Separate /engine/ rate limiter
* Added parallel wait prevention, it’s working for duration waits but not well for triggerAndWait yet
* WIP post-merge conflicts
* Set taskEventStore column in the new engine
* Remove duplicate keys
* Post-merge fixes
* Fix for span merge layout
* Use executedAt instead of firstAttemptStartedAt
---------
Co-authored-by: Matt Aitken <matt@mattaitken.com>
* WIP with webhook SDK function and types
* JSDocs added to the schema
* Webhooks are working
* Expanded the alert docs
* Remove duplicate export of waitUntil.js
* Use uncrypto
* Don’t rate limit webhooks
* Create slow-olives-fix.md
* Efficiency improvements to SharedQueueConsumer and MarQS for run engine v1
* Reliabily resolve task queue for a run and ack runs where we can't find the queue
* engine v1: stop nacking after reaching the max nack count, ack to remove the message
* Handle division by 0 possibility in distributeQueues
* WIP batch trigger v2
* Fix for the DateField being one month out… getUTCMonth() is zero indexed 🤦♂️
* Added a custom date range filter
* Deal with closing the custom date range
* Child runs filter
* Fix for the clear button untoggling the child runs
* WIP batchTriggerV2
* Finished removing rate limit from the webapp
* Added an index TaskRun to make useRealtimeBatch performant
* Renamed the period filter labels to be “Last X mins”
* Denormalize background worker columns into TaskRun
* Use the runTags column on TaskRun
* Add TaskRun ("projectId", "id" DESC) index
* Improved the v2 batch trigger endpoint to process items in parallel and also added a threshold, below which the processing of items is async
* Added a runId filter, and WIP for batchId filter
* WIP triggerAll
* Add new batch methods for triggering multiple different tasks in a single batch
* Disabled switch styling
* Batch filtering, force child runs to show if filtering by batch/run
* Added schedule ID filtering
* Force child runs to show when filtering by scheduleId, for consistency
* realtime: allow setting enabled: false on useApiClient
* Batches page
* Always complete batches, not only batchTriggerAndWait in deployed tasks
* Add batch.retrieve and allow filtering by batch in runs.list
* Renamed pending to “In progress”
* Tidied up the table a bit
* Deal with old batches: “Legacy batch”
* Added the Batch to the run inspector
* Fixed the migration that created the new idempotency key index on BatchTaskRun
* Fixed the name of the idempotencyKeyExpiresAt option and now default idempotency key TTL is 30 days, not 24 hours
* Timezone fix: wrong month in Usage page dropdown
* The DateField now defaults to local time, but can be overriden to use utc with an option
* Don’t allow the task icon to get squished
* BatchFilters removed unused imports
* In the batch filtering, use `id` instead of `batchId` in the URL
* BatchFilters: we don’t need a child tasks hidden input field
* Creates some common filter components/functions
* Fix for batchVersion check when filtering by batch status
* Add additional logging around telemetry and more attributes for trigger spans
* Show clear button for specific id filters
* Batch list: only allow environments that are part of this project
* Unnecessary optional chain
Co-authored-by: coderabbitai[bot] <136622811+coderabbitai[bot]@users.noreply.github.com>
* Add JSDocs
---------
Co-authored-by: Matt Aitken <matt@mattaitken.com>
Co-authored-by: coderabbitai[bot] <136622811+coderabbitai[bot]@users.noreply.github.com>
* magic links on span event errors
* prevent task monitor from processing errors handled elsewhere
* exclusively use internal error code enum for completion data
* add complete attempt service opts
* reattempts need to go via the queue for task controllers that may have exited
* only infer retry config if completed via crash or system failure
* enhance error before deciding if retriable
* retry on SIGTERM
* enable retry config helper for latest sdk
* don't retry heartbeat timeouts for now
* enable task monitor to update fatal errors
* add missing service
* update retry config since package version
* don't alter completion time when updating existing error
* Denormalize run tags, increase character limit to 128
* WIP realtime subscribing to runs
* extracted the stream stuff into core, made it more reusable
* WIP tags
* Remove tags for now because it’s not support in electric
* Support async iterables, readable stream, and callback style subscription styles
* Remove tags streaming endpoint
* Add realtime rate limits and scope them to the /realtime path
* WIP rate limt per org
* Introduce per org rate limits
* WIP JWT auth
* Move migrations into new internal db package
* Resolve pnpm lock file
* Authenticating to the realtime API with JWTs are working
* realtime in the client
* Created react-hooks package and starting to move stuff in there
* Improve types for hooks
* schema tasks
* Added useBatch hook
* build uploadthing/fal demo and change how run metadata is synced to the server
* tweaks
* WIL realtime concurrency tracking
* Implement test for realtime client using testcontainers
also updated electric to latest version
* Allow customizing the expiration time of the automatic JWT created after triggering a task
* Add support for subscribing to run tags
* Improve auth types and API
* finalize the realtime API
* Fixed some example stuff
* Allow up to 10 run tags
* Remove core from docker-provider tsconfig paths to prevent it from being typechecked
* do the same for the kubernetes provider
* Fixing some typecheck errors
* Fix webapp type errors
* Update @trigger.dev/platform to 1.0.13
* Fix attw error
* Remove from/to in subscribeToRuns query params
* Add tests for the rate limit middleware and add custom JWT rate limits
* turn off webapp test parallelism
* Finish renaming jwt -> publicAccessToken and automatically give the JWT read access to the tags when using trigger
* Add changeset
* Attempt to fix unit tests in CI
* Skip running the auth rate limit middleware tests for now
* Try a beefier machine
* Try and run webapp tests separately
* Setup env vars
* Make sliding window test more reliabile
* WIP
* Get max duration working on deployed runs
* Actually set the timed out runs to status = TIMED_OUT
* The client status for TIMED_OUT is now MAX_DURATION_EXCEEDED
* New TimedOutIcon
* Added new timedout icon
* Add ability to opt-out of maxDuration with timeout.None
* MAX_DURATION_EXCEEDED -> TIMED_OUT
* changeset
* Improved styling for the status tooltip content
---------
Co-authored-by: James Ritchie <james@trigger.dev>
* upgrade @opentelemetry packages to the latest versions
* remove v2 only packages, will be moved to a dedicated repo
* remove more v2 code and run pnpm install
* use the npm yalt package in the webapp
* convert @trigger.dev/core to tshy
* Switch from jest to vitest in @trigger.dev/core
* Fixed core test
* move core-backend code into core subpath export
* convert @trigger.dev/sdk to tshy
* Removed hono
* move core-apps to core/v3/apps, remove core-apps, start converting cli-v3
* Fix up some of the commands
* cli now building and loadable
* using package-json-from-dist to get package version now in core and cli
* dev command WIP
* cleaned up some repetition and structure of the entry point stuff
* bringing back the background worker stuff
* Indexing of the v3 catalog
* getting closer to executing dev runs...
* centralize dev logging using event emitter
* Move indexing to it’s own entry point, simplify code
* dev runs working
* Get instrumentation to work with openai
* debugging achieved internally
* provide worker files as part of the worker creation on the server
* support for cjs and esm javascript
* Fixed timeout
* worker manifest now has the config path
* auto-upgrade config to non-deprecated alternatives
* Adding package preview release
* deployment WIP
* improve the syncEnvVars output and adapt resolveEnvVars
* WIP bun runtime
* WIP bun support
* seed tasks with the machine preset if listed in the config
* deploy run executions WIP, extracted TaskRunProcess into 1 place
* deployed tasks running and executing 🎉
* support for waits and better flushing & process cleanup
* Fixed the heartbeating
* Better warning messages
* Improve and unify the indexing between dev and deploy
* Support for external deps that need node-gyp to build
* build extensions can now install custom packages and run instructions in the image. Also prisma extension now works and also works with multiple schema files
* Add back in the main/types/module to sdk
* dev no longer is Ink/React, grace period for disconnections in dev
* Fix the changeset config
* More changeset fixes
* Remove config packages
* More changeset fixes
* Fixed typescript issues (needed to revert back to zod 3.22.3
* Fix pr_checks workflow
* Remove the prepare script
* Fixed tests and package versions
* Remove cli test script
* Remove packages from tailwind watch paths
* Add repo to public packages
* Just commit the generated files and do the building at dev time
* Try and get pkg.pr.new working
* Try again
* Fix emitDecoratorMetadata importing named export from typescript
* config file backwards compat with export const config
* Fixed issue where import errors weren’t coming through
* p-retry is a prod dep
* typescript needs to be a prod dependency for emitDecoratorMetadata
* Add better debug logging to help track down import-in-the-middle bug
* An external is only considered resolvable if it resolves to the same path as the collected external
* Fix runtime checks to allow >=18.20
* Move extensions to a new build package
* Fixed building packages in dockerfile
* Remove the e2e test from publish workflow for now
* Don’t treat pkg.pr.new versions has needing upgrading
* making sure config handleError works, and discovered path aliases don’t work in config files
* Strip empty string env vars so they accidentally override real values
* Couple of things
* Update version to use preview instead of beta
* Hopefully fix re-attempts with >30s delay
* Match socket emit messages to current latest in main
* Initial guide
* Go back to beta
* Go back to the preview, and update guide to use pr preview tags
* Go back to beta
---------
Co-authored-by: Matt Aitken <matt@mattaitken.com>
* Initial TaskRunConcurrencyTracker implementation
* MARQS calls a subscriber to events
* When enqueuing add the extra required metadata
* Track concurrency per environment for tasks too
* Admin page for global concurrency
* Use the new concurrency tracker on the tasks page
* Useful performance test task
* getAllTaskIdentifiers()
* New page for concurrency
* BackgroundWorkerTask index for quick lookup of task identifiers
* Added a way to get concurrency for environments
* Added upgrade/request more concurrency button
* Queued task column working
* Use defer and suspense
* Added queue column to the concurrency environments table
* Some comments added for clarity
* Fixed bad log message
* Sidemenu: move lower and rename to “Concurrency limits”
* Only show the environments, not tasks. Renamed to “Concurrency limits”
* PropertyTable component changed to use sub components
* Separate run span component
* Early WIP on tabs that use search query
* Shortcut key tabs for the span panel
* The span timeline is working
* When runs get expired, update the OTEL event with an error
* Improved the expired error message
* Reveal env vars when editing
* Tightened things up a bit
* Added detail tab properties
* Progress dashed line
* Move the env label next to the Run number title
* Top level cancel/replay buttons
* Replay with a different payload and environment
* Fix for non json payloads
* Hide the clear/copy buttons
* UI improvements with large payloads
* Close the panels when you replay/cancel
* Added the new timeline to spans
* Use the u-turn left icon for replay
* Added an index for spanId on TaskRun
* Remove replay/cancel buttons the span view
* Replay shortcut works inside the code editor
* Split the log/span inspector between Overview and Detail as well
* More improvements to the inspector
* Context and output improvements
* Focus on run working
* Added version to run.ctx
* Added some padding to the detail view
* Added context tab with shortcut
* Only load the replay data when the dialog is open
* Replaying uses the tags from the original run
* Links are now text links
* Removed version links for now because we don’t have dropdown filters for them yet
* Tabs are now outside of the scrollview
* The inspector is now 30% of the width by default
* Allow replaying and editing SuperJSON payloads
* Deleted unused CodeGroup file
* Increase the tags limit to 5, do the limiting on the server
* The admin tooltip now always shows basic org, project and user info
* Fix for schedule inspector disabled state layout
* Remove new unused span metadata and context
* Removed unused import
* TaskRunTag migration. Removed unused TaskTag
* Show tags for a run in the run list
* API endpoint for searching tags
* Let’s not expose an API endpoint for tags at the moment
* Filter by tags working from the URL
* Tag async filter working
* We don’t need a “None” option because it’s multi-select
* When triggering a run you can add tags
* SDK runs.retrieve and runs.list with tag support
* Tidied imports
* Changed the run list query so we show all the tags even if a run only matches one of them
* Fix for dealing with weird characters in tags
* Run tags changeset
* Creating a project has a proper loading state (and blocks multiple)
* Improved the error message for tag length
* Added a tags icon for display on the run screen
* Convenient functions for creating and getting run tags
* tags.set() from inside the run function
* Replay a run passes tags through
* Less ridiculous tags for the catalog example
* Allow passing just a string for the tags
* Order by id because there’s an index on the primary key already
* Trim the tags earlier so we don’t accidentally error if a blank string is passed
* Renamed some thing from setTags to addTags
* Use findFirst for the tags project lookup
* Added an index for "TaskRunTag"("name", "id")
It massively improves the performance of run filtering based on tags
* Use `array_agg` for the run list tags so pagination works and we get a single result for each run
* Tidied imports
* More comprehensive test of tags with all triggering functions
* Support tags with tasks.trigger, tasks.batchTrigger and tasks poll variants
* Added tasks.batchTrigger tags
* Sort the tags in the UI so they’re always in the same order
* Added tooltips in the run table for delay, ttl and tags
* Added costInCents, baseCostInCents and durationMs to runs.retrieve and runs.list
* Added support for displaying a split tag if you use key_value or key:value format
* Fix code comment
* Tweaked the JSDoc to make the prefixing clearer
* Added tasks.triggerAndWait to the tags test task
* v3 subscription endpoints
* Use pnpm linked billing package during development
* Moved v2 billing components into a subfolder
* Select plan using the real data
* Improved v3 plan display
* Use new api response that doesn’t require a Stripe call
* Free flow is working
* Added GitHub modal and verified badge
* Deleted old request v3 access component/route
* Allow setting classes on the Tooltip button
* Paid plans working
* Redirect from select plan if you’ve got v3 enabled
* Loading state improvements
* New admin API endpoint to set concurrency across multiple environments
* When projects are created, conditionally create staging based on the plan
* New billing page working with side menu and stripe portal
* Layout, formatting and some plan state improvements/fixes
* Don’t show the period if you’re on the free plan
* Temporary upgrade callout
* Refactored the platform code so it’s easier to call and doesn’t require a isManagedCloud check
* Send taskIdentifier to OpenMeter
* Side menu
* Upgrade prompts
* More improvements to the app-wide usage indicators
* Early work on usage graphs
* Added the usage bar for v3
* Moved code to presenter and now using defer
* Added the tasks table to usage
* Improved the v3 usage bar if theres’ no usage on a paid plan
* If no run data, still render a graph
* Usage page errors when defered loading fails
* Don’t show the public API key for v3, they’re not used and probably never will be
* Improved the upgrade callout and API keys page layout
* Only show the “reveal all” toggle if you have environment variables in the table
* Replaced Upgrade callout with a more generic InfoPanel component
* better panel width
* Show conditional upgrade prompts based on plan and number of schedules used
* Removed duplicate class
* Wider blank state panels for the scheduled page
* Wider info panel for the env var page
* Blank state now using the info panel
* Platform alerts prompt now using the InfoPanel
* Deploy blank state uses InfoPanel
* Github verified badge padding adjustment
* Improved the layout of the page, some style tweaks, organized imports
* Better default tooltip style
* could be undefined fix
* text fix + style updates
* Changed the billing icon in the side menu
* Billing page layout and style improvements
* plan tooltips don’t use dark variant
* Don’t highlight the plan on the billing page
* Tooltip underlines stand out more
* Fixed padding in the PageTitle
* Tooltips use the correct cursor
* Improved the plan banner on the billing page
* Fixed Header1 inconsistent font weight
* Fixed issue where input field focus states were being clipped
* Fixed large button not having large text size
* Added a link to the Get in touch copy and improved the connect to GitHub modal
* Select plan page uses the MainCenteredContainer
* Better logging from the Loops endpoint because this error finally got hit
* Move the ingestion of compute to the platform
* Reporting usage of invocations moved to the platform
* Get the entitlement before triggering a non-dev task
* Contact us enterprise plan button opens the feedback form
* Removed Github discussions link from the Feedback panel
* Swapped billing icon for credit card
* Show a Unlock staging panel on the env var page
* Updated staging environment colour
* Show a prompt to upgrade to get staging in the new env var modal
* Improved the edit env var modal
* Implement ability to disable org concurrency
* Use common logic for the plans
* Use the billing server to get the schedule limits
* Some schedules page fixes
* More convenient way of getting a limit
* Use the new schedule limit
* Team member limiting
* Made the limit visible on the team page
* Limit alerts
* Added an index for TaskRun.scheduleId
* Remove console.log on schedules page
* Added durations to the run table
* Tabular numbers
* Improved the usage page formatting
* Only admins see the compute column on the run table
* Include the base cost on the usage stats
* Moved the status to the sidebar
* Optional table header tooltip
* Allow InfoIconTooltips to have customizable content styles
* Added a tooltip to the duration header, changed no test to a dash
* Removed all references to signing up to v3 from the docs
* Switched @trigger.dev/billing to @trigger.dev/platform
* Passing up the variant for the InfoIconTooltip
* table tooltip max-width fixed
* Switch to the published @trigger.dev/platform 1.0.11
* v2 usage page title changed to include “v2"
* code theme has a transparent background so it works on any background
* duration columns now grouped together nicely at wide screen size
* Last duration column fills the width properly
* Fix for the per run price being in cents not dollars
* Show the total cost with 8 decimal places
* Show 8 decimal places in the usage graph tooltip
* Moved the UpgradePrompt to the v3 folder
* Prepare to use Shadcns chart helpers
* Much nicer chart
* Small tweaks to the graph
* Fix run table col spans for empty/loading messages
* We don’t need isManagedCloud in createProject
* Hide v3 usage/billing pages if there aren’t v3 projects in your org
* Removed unused tooltipStyle
* Usage bar now says “Included usage” instead of “Tier limit” if you’re paying
* Get the plan/usage data in parallel
* The usage page now has a month dropdown and all data is for that calendar month
* Ensure the passed date is the 1st of the month
* Use the machine presets from the platform package
---------
Co-authored-by: James Ritchie <james@jamesritchie.co.uk>
Co-authored-by: Eric Allam <eallam@icloud.com>
* v3: Trigger delayed runs and reschedule them
* Create a `@trigger.dev/core/v3/schemas` export
* fixed the `@trigger.dev/core/v3/schemas` export
* Small docs tweak
* Add ttl option when triggering tasks, expire runs after ttl
Dev runs expire in 10m by default
* prevent downgrades during update check
* detect bun and use npm instead
* detect missing tsconfig during init and print helpful error
* add changeset
* add links to dev worker started message
* allow users to add custom pkg manager args during init
* update changeset
* fix links in unsupported terminals
* deprecate terminalLink
* update changeset
* add amin email regex env var
* fix displayed init command for self-hosted setups
* shared env var to disable telemetry in cli and webapp
* pin sdk version during init
* if specified, add api url to dev command shown after init
* improve checkpoint support detection
* control forced checkpoint simulation via env var
* add public init to providers
* better checkpoint support check for coordinator
* add docker to coordinator image
* update docker provider containerfile
* bump remaining containers to node 20
* add infra image build to default publish workflow
* lockfile
* remove concurrency group from infra workflow
* add docker provider to build matrix
* fix var subst
* checkpoint test is docker specific
* enable v3 projects by default on self-hosted instances
* fix v3 setup command again
* add default posthog key
* self-hosting docs
* add latest tags to versioned infra and webapp builds
* some checkpoint errors should skip retrying
* add changeset
* shorten paragraph
* some docs updates
* update tunnelling section
* add registry setup section
* use correct cli push flag
* add checkout to v3 branch
* update the worker machine setup steps
* fix infra build
* small docs update
* remove unused feature function
* Revert "remove unused feature function"
This reverts commit cfe07887a12b6893dca8ce499964481a9b3dc9db.
* fix self-hosted v3 feature gate
* add note about missing arm support
* simplify helper script syntax
* Improved the existing runs API
* WIP next runs API
* Improve the returned ApiPromise to add ability to return response
* More WIP
* WI{
* Added offset/limit pagination stuff like the cursor one, and converted all API methods to use ApiPromise
* More run API stuff
- Adding schedule output from the retrieveRun endpoint
- Ability to filter by schedule and isTest
* Remove env from retrieve run in openAPI
* prefer duplication over merge
* WIP docs
* Use spread to DRY up some run API schemas
* Finish the overview docs
* Adding changeset
* Fixed typecheck errors
* Typo fix
* Re-export zodfetch from core so the v3 CLI can use it
* Fixed type errors
---------
Co-authored-by: Matt Aitken <matt@mattaitken.com>
* Test page: don’t show deleted dev environments
* Function to filter out environments that are dev and have no orgMember or orgMemberId
* Don’t show tasks from deleted members (or the associated environment labels)
* Show “Dev: Deleted” if the user has been removed.
* Only show your environment in the env vars table
* Renamed function
* Use the displayableEnvironment function
* Added v2Enabled and hasRequestedV3 columns to Organization
* Don’t create a project when you create an org
* Form for requesting v3 access
* Reworked the new project form with the different version states. Refined copy on early access
* If the project isn’t in the org then redirect to the new project page
* Better message for existing users
* Tidy imports
* If it’s not the managed cloud then allow them to create v2 projects
* Fix issues with posting alerts to public slack channels
* Use the actual values in the new environmentTypes column to display the environment type labels in the alerts list
* Implement environment alert options
* Alerts v1
* Encrypt alert webhook secrets and allow them to be generated by the server
* Alert v1 UI
* Remove unnecessary emails
* Move to using `@react-email/components`
* WIP slack alerts
* More slack alerts WIP
* Update pnpm lock after rebase
* Finish implementing Slack alerts
* Use a more error like emoji
* New secondary variant for the segmented control
* Added a simple checkbox style variant to storybook
* Style tweak to the segmented control
* UI improvements to the alert modal
* Use searchable Select for alerts. Changed default variant for SegmentedControl
* Secondary button now using secondary colour
* segmented control style tweak
* Improved the channel column in the alerts table
* Updated logo-mono.png
* Updated email styles
* Email templates updated to new styles
* Don’t log the decrypted secret
* await enqueing the deployment alert when an index fails
* await enqueing the timeout alert
---------
Co-authored-by: James Ritchie <james@jamesritchie.co.uk>
Co-authored-by: Matt Aitken <matt@mattaitken.com>
* Fix for “overridden” spelling mistake
* Make the task function names grey instead of yellow
* The task page now shows each task once
* Blankline
* WIP on activity bars
* Correct size of the graphs
* Add data for all days, even if there are no runs
* Activity graph is working and has a tooltip
* No activity label
* Got the bar bg working
* Activity chart now has a bg
* Added a compound id to make the activity graphs appear faster
* Improvements to the activity graph
* Tried to disable the bg from animating
* Added read replica support to the app, using Nick’s code from his batching PR
* Added the read replica to BasePresenter
* Added the queued and running columns to the Tasks list
* Added avg duration column
* Included completed but failed runs in the average duration
* Get rid of the old useDevEnvironment hook and remove API keys from the projects query
* Unified getting environment info to using a function and doing it in presenters
* Removed the path from the tasks table
* Onboarding for dev for 2nd+ user. Improved environment sorting.
* Removed log
* Improved the spacing
* Enable staging for new v3 projects
* Fixes various issues with triggerAndWait and batchTriggerAndWait
When using idempotency keys, triggerAndWait and batchTriggerAndWait will still work even if the existing runs have already been completed (or even partially completed, in the case of batchTriggerAndWait)
- TaskRunExecutionResult.id is now the run friendlyId, not the attempt friendlyId
- A single TaskRun can now have many batchItems, in the case of batchTriggerAndWait while using idempotency keys
- A run’s idempotencyKey is now added to the ctx as well as the TaskEvent and displayed in the span view
- When resolving batchTriggerAndWait, the runtimes no longer reject promises, leading to an error in the parent task
* Remove the default queue concurrency limit as we now have env and org concurrency limits
* Use the run friendlyId in the completion result id
* Added some error logging
Eric Allam