* WIP on replaying a task from the run page
* Don’t pass the existing runs idempotency key, it will cause the replay to always return the original run
* Replay from the run list
* Don’t use fetchers in the replay/cancel dialogs
* API endpoint for replaying a run
* REST API docs (mostly coming soon) but added replay run
* replayRun function added to the SDK
* Cancel run added to the SDK
* v3-catalog file to test canceling and replaying
* Changed the SDK to be runs.replay and runs.cancel
* Removed comment
* Latest lockfile
* WIP fixing pnpm nested install strat
* Add changeset
* Pass (and log) cwd during background worker initialization
* Log out the env
* manually add the .pnpm/node_modules to NODE_PATH if in a pnpm repo
* Remove supports color dep
* Upgrades to MarQS
- It’s “queue choosing” algorithm is now MUCH better (pretty much didn’t work before) and should be performant even when we have a bunch of prod queues
- There are now concurrency limits at the environment and organization level, as well as the task/queue level. So if any of them are at capacity the message won’t be dequeued. This means we can have an org wide concurrency limit, as well as different limits for dev/prod/staging
- I’ve added an admin API that can be used to update the org/env concurrency limits
* More MarQS upgrades
- Extract the queue priortity choosing strategy into an interface
- Implement a much better weighted average strategy
- “Slide the window” of queue candidates if the parent queue sends all at-capacity queues (so we won’t get stuck attempting to choose the same 12 full queues)
- Added some unit tests for the priority stuff
- Added some ideas for expanding the priority choosing strategy with more dynamic features
* Fixes an issue with the shared queue consumer relying on the queue name to get the env id
* v3: implement configurable log levels via config file and TRIGGER_LOG_LEVEL
Also, test runs automatically set the TRIGGER_LOG_LEVEL to debug
* Fix type error and changeset
* Added a Node.js runtime check for the CLI dev command
* v3: Dynamically import superjson and fix some bundling issues
* Added changeset
* Better error handling in the registry proxy and catch uncaught exceptions and unhandled promise rejections instead of crashing the server
* Await the prettyPrintPackage
* superjson output support
* Better support for superjson in the task events, limiting output attributes
* Offload large outputs to object store (r2)
* Finishing up the offloading of large outputs/payloads to an object store
* Adding a changeset for v3
* Add a version field to @trigger.dev/core-apps package.json
* Build trigger.dev when doing a prerelease
* bundle @trigger.dev/core-apps with trigger.dev cli
* Fix the init command config template
* Don’t use * for the @trigger.dev/core dep version specifier
* strip workspace: from the package version before installing it
* Added dependenciesToBundle config option to bundle ESM only packages
* Added logging around resolving dependency paths
* Try again
* Resolve dependencies based on the project dir first
* flip the bundled default
* Adding some logs around dev task completion notifications
* Adding some additional logs
* Add more logs
* Write out the log using process.stdout
* Store pending completion notifications and resume them when awaited (fixes race condition)
* Cleanup some of the logs
* Copy over the postinstall step from the projects package.json
* Add support for including additional files when deploying (e.g. prisma schema)
* Don’t run scripts when resolving deps
* copy all the files just in case anything is needed in postinstall
* Remove duplicate option
* Use the tag when outputting the dev command
* Remove the postinstall script
* add the trigger dir to the config if the default is not chosen
* Remove the “hud” display in the dev command
* trigger file names with dashes now work
* Better file watching in dev
* Much better duplicate ID experience now
* Much better “Project not found” error
* Export the handleError function types from sdk
* Add support for configuring instrumentation
* Upgrade and unify @opentelemetry/* packages (and remove storybook from the webapp)
* Fix typescript error in react package
* Upgrade react types in webapp
* Allow span icons to be determined based on the span name (e.g. prisma:)
* Ignore built-in env vars when checking for env vars, and allow continuing the deployment even if missing env vars were detected
* Improve the retry.fetch default behavior and option structure
* Fixed typescript errors with packages/email react types
* Update the retry.fetch docs
* add checkpoint restore events
* fix retries.enabledInDev
* fix tsconfig paths
* fix provider build and dev
* update kubernetes provider and fix builds again
* upgrade prod base to node 20
* update infra publish workflow
* rethrow k8s errors after logging
* shorten index container names
* fix error type assertion
* rename type assertion
* remove resource limits for now
* add missing run id on create
* add push to deploy command for self-hosting
* checkpointing fixes
* update coordinator image
* ensure valid registry login
* delete checkpoint archive after successful push
* log options on error
* structured logs for socket connections
* fix structured log merge
* exit process after checkpointing
* update restore pull secret name
* append shortcode to restore names
* log handler payload
* disable post start lifecycle hook
* pass in coordinator host via volume
* replace dapi with taskinfo
* add missing restore label
* don't restart restored containers
* remove init container from create
* atomic post-completion checkpoints
* switch to run id for container names
* improve wait accuracy
* measure basic checkpoint perf
* always log disconnect reason
* use system clock to end wait spans
* checkpoint readiness and cancel signals
* restore from checkpoint events and fix statuses
* remove attempt id env var
* restore dependencies from events
* reconnect wip
* lifecycle hooks are back
* fix hooks and improve reconnect
* make docker send postStart hook
* only checkpoint for retry if large delay
* fix a few more resume issues
* lifecycle hook fixes
* skip connection handler when waiting for post start hook
---------
Co-authored-by: Eric Allam <eric@trigger.dev>
* A bunch of changes to the config file:
- No more default export, the config is exported via the `config` named export
- `handleError` is moved to a separate named export
- Added the ability to include additional packages (e.g. wrangler) that aren’t included by default (think CLI usage)
- More reliably install packages in dev
- The config is now built first instead of directly imported, meaning we can now support typescript config files (`trigger.config.ts`)
* Update to trigger.config.ts in `trigger.dev init`
* Store startTime in nanoseconds since epoch, ensure logs using nanosecond precision for their timestamp (so they are not out of order)
* export task events immediately in dev
* fix basic resume mechanics
* fix dependency resume
* fix deploy page sort
* remove task list console log
* checkpoint restore between attempts
* add checkpoint requirement checks to docker provider
* switch to raw execa for long statements
* catch and report pre-connection index errors
* graceful worker shutdown after completion
* fix final completion
* use same task process for each attempt
* update retry catalog entry
* add immediate return task
---------
Co-authored-by: Eric Allam <eric@trigger.dev>
* Adding telemetry to the deploy command
* Add ability to opt-out of telemetry in the CLI
* Remove log
* Unifying some telemetry stuff and adding it to the login command
* init command
* Remove extra trigger dirs
* Couple tweaks to the init CLI
* Don’t use npm init -y to create the package.json file
* Add the ability to handle errors at the task and project level
* Cancel in progress runs when disconnecting the dev CLI (in between attempts)
* unify the task executor across dev & prod
* WIP proxy/deploy
* WIP, registry proxy in express working
* A couple of notes, preparing for indexing
* Move the changes to prod-worker into the new file
* Deploy command working with indexing and runs (docker provider only for now)
* Removed ts-expect-error directive
* Fixed build command
* add zod socket
* start using zod socket
* use zod socket for shared queue
* use shared provider shell
* update k8s provider task ops
* set otlp endpoint
* update docker actions and fix builds
* remove unused types
* update injected secret key env var name
* complete socket.io types to schemas migration
* update env example to new v3 key var
* fix coordinator build
* set task run label
* ensure attempt id is always set
* increase prod worker timeout
* set otlp endpoint for on runs
* pass otlp env var to prod worker
* auto-remove completed index jobs
* Added the new colours to the Tailwind config
* Fixed Tsailwind compile errors
* WIP Geist working in the app
* WIP Storybook
* Storybook is working again… ouch
* Use Geist mono for CodeMirror
* Some find/replace class changes
* Resizable colours improved
* WIP on primary button
* Minor style improvements
* Improve the side menu header styling
* Improved v2 breadcrumbs
* Account side menu tweaks
* The page now works with the NavBar and UpgradePrompt
* Removed unused import
* Refactored the buttons to be dry-er
* Removed the breadcrumbs
* WIP on nav bar
* WIP updating the button styles
* Fix for the old test page
* Fix for type errors
* Removed console.log
* Colour change in the resizable handle
* New loading bar divider
* Removed the design plugin
* Creating our own storybook pages
* Added some “storybook” pages
* Added DetailCell storybook
* WIP new animation for the loading page
* Added the new loading divider
* Fix for the time frame filter having a green border
* Style improvements to the new test page
* Small style improvements
* Improvements to the list pagination components
* Nav bar back button improved
* Improvements to the run page
* Lots of v2 style improvements
* Help looks better
* Improvements to the regenerate API key modal
* More general improvements
* Added section titles to our storybook
* Added form stories
* Fix for missing key prop
* Loading bar is working
* Fixed button submitting state
* Fix for the test button icon
* Code bg improved
* Improved oauth buttons
* Integration page header fix
* Various improvements, mainly changing button variants
* Segmented control style updates
* Added an outline of 3px for the segment control
* Fixes for the billing pages
* More pricing improvements
* Added dialog to storybook
* Typography now in storybook
* Free usage bar added to storybook
* Added inline clode to storybook
* Separated menu types into a sub section in storybook
* NamedIcon now in storybook
* Added page header to storybook
* Added pricing callout to storybook
* Added radio groups to storybook
* Added segmented controllers to storybook
* Added shortcuts to storybook
* Added switches to storybook
* Added toast to storybook
* Added tooltips to storybook
* Added tree view to storybook
* Added usage charts to storybook
* Tarted up the environment variables page
* Fix for the FreePlanUsage panel bg colours
* Added new badge variants
* Added the <TaskPath /> to the tasks table
* Span header improvements
* Started work on the timeline
* Started work on the timeline
* Fix for the run page header
* Timeline tick marks are working
* Timeline working
* Got the durations showing on hover
* Button improvements
* Experimenting with how the duration should be shown
* Lots of refinements to the timeline
* Much nicer API for the Timeline
* The switch is now purple to match the design, with inverted thumb when on
* Removed old stuff
* Export the props
* Added the span with duration back in
* Simplified more and fixed the span durations
* Put in some decent dummy data
* Rendering defaults
* Wip with the mouse cursor
* Improvements to the story
* Added Slider primitive
* Integrated the icons into the slider
* Lots of layout improvements
* Cleaned up imports
* “Root” indicator
* Animated background on partials
* Pull the styles through to the spans and log dots
* Fixed some bugs in the TreeView when nodes get created or destroyed
* Fix for incorrect parentTask reference
* Added run streaming endpoint for live refreshing
* Added the tabs to Storybook
* Fix for status font size mismatch
* Fix for the misaligned run filter status
* Set a minimum duration on the timeline
* The TreeView now re-calculates if the number of nodes has changed
* Partials now render to the end
* Improved the durations gutter
* The duration is calculated using the current time if there are partial events
* Added the environment label to storybook
* Improved the environment label style
* Removed bottom margin from badge
* Button style improvements
* Callouts style fits the new design better
* Got the scrollbars to appear on the tree/timeline
* Added scroll support
* Synced scrolling working
* Close button and exit icon for the detail panel
* secure field shows start of env
* Clipboard improvements
* Latest lockfile
* Fix for the durations wrapping
---------
Co-authored-by: James Ritchie <james@jamesritchie.co.uk>
* WIP no bundling
* Convert dev CLI to use an unbundled build process to support otel instrumentation
* Fix pnpm lock file
* A couple of fixes to get typechecking to work
* Show runs as cancelled when killing the dev CLI
* Fixed error when waiting for a task
* Deprecated workers will no longer be able to run unlocked task runs
* Fixed selecting the latest background worker version in test list
* Don’t propogate trace context when triggering an async task
Also add batch id to the context and the task events
* WIP
* retry.fetch w/intercepting to test
* Fixed the incorrect renaming of limit to maxAttempts in v2
* Fixed source map support in dev
* Cleanup background worker files once they close
* Support for batch triggers
* Various trace fixes and improvements
* Moved trace into the logger, fixed span cardinality issues with waits and triggers
* Add support for attaching a debugger in dev with —debugger flag
* Added ProjectVersion to projects, defaults to V2
* Moved v3Enabled to the existing feature flags
* Update to using the v3Enabled feature flag
* Allow people to choose v3 when creating a new project
* Added version to the ProjectPresenter
* v2 project and v3 project redirecting
* .env.example for the v3-catalog
* Added sdkVersion and cliVersion columns to the BackgroundWorker table
* Added additional classes for environments
* First draft of the Tasks table
* Link to the task page
* V3 side menu project items
* Moved the ListPagination component into the components folder
* Moved the TaskListPresenter to a v3 folder
* Bare bones task page with tabs
* Task page, we’re going to delete this though
* Bare bones runs table working
* Rejigged the columns
* The run table status
* Environment and status filtering working
* Added time filters
* Cursor and direction is working… I think it’s tricky to know for sure
* Added TaskRun numbers
* Selecting a task now links to the runs page with filters turned on
* Added support for the enqueued status
* Reworked the task table
* Link to the task and environment runs
* The run page is rendering a tree of the events
* Change the task table column to “Created at"
* WIP on making the run tree view look good
* Improvements to the run tree
* Removed the janky scroll bar flash
* Change the title of the task logs to just the task ID
* Live timer when a span is running
* Added “Show parent items” link
* Fixed the bug jumping to parent items
* Style improvements to the run
* Added the resizable handle to the run page
* Moved formatDuration to core/v3
* Wait for now has nice log messages
* The run detail panel is working with just a title for now
* Navigating to a span and persisting between page reloads is working
* The run detail timeline
* Wrap log dates in a paragraph
* Latest span detail view
* Scrolling in the right-hand panel
* Tweak default resizable layout for the run page
* Attempted improvements to stop a recursive issue on TreeView
* TreeView useReducer WIP
* Changed the change callback
* Add changes to the state
* Use the new onSelectedIdChanged in the app
* Don’t update the state from the outside
* Filtering fix, although navigation doesn’t work nicely
* Filtering working when changing the content and clearing it
* Don’t allow the same selectedId callback to be called twice…
* useDebounce hook
* Switch from using defer to just typedjson for now
* Latest attempt at span navigation
* Removed log
* Turn off filtering for now, it’s causing the Links to break somehow
* Fix for page height issue
* Added (Developer Preview) to the v3 project select box
* Revert "Filtering working when changing the content and clearing it"
This reverts commit 20d9fbe36ded619e9d3eb0b23d4ed0568a88a615.
# Conflicts:
# apps/webapp/app/components/primitives/TreeView/TreeView.tsx
* Switch to using the old filtering
* Added useThrottle hook
* Instantly close the panel when deselecting a node, use debounce when navigating to a span
* Better spacing in the right hand panel
* Comment out the v3 side menu pages that don’t exist yet
* Fix for typecheck fail
* Clear the statuses too
* Introducing Modular Asynchronous Reliable Queueing System (MarQS). Works in dev
* Convert MarQS to using lua and dealing with concurrency
* Simplified the timeout queue and current concurrency is now a set instead of a flat value (to support idempotency)
* Implement task heartbeating and reconnect the background workers CLI when the websocket connection reconnects
* Start adding internal telemetry support for the server
* Get env vars to work in dev and implement prisma tracing in webapp
* Cleanup telemetry and implement it in the consumer
* Implement dequeuing a message from a parent shared queue
* Implement a custom logger exporter instead of using console log exporter
* Use node instead of shell for generating protocol buffer code
* Propogate trace context into debug logs, and allow turning off logger exporter through env vars
* Switch to using baselime for internal otel data
* Make orgMember optional to fix type issues
* Provide the CLI dev env vars through the CLI, don’t build dotenv into facade
* Removed the logger import
* Address Matt’s comments
* Addressing more of Matt’s comments
* Handle sending an execution after a websocket connection closes
* Remove auth from the env attributes to prevent obfuscation
* Added a subtask for testing
* Make it easier to run the CLI from the nextjs reference project
* Copies of the run and statuses endpoints, but without simplifying the run statuses
* Use the new v2 endpoints that give the full run statuses
* Removed unused import
* v2 events endpoint with the full run status info
* Changeset
* Initial CLI commit
* Hooked up the CLI so it actually shows stuff…
* Totally reworked the v3 CLI to be based on our existing CLI (Commander)
* WIP on PersonalAccessTokens and AuthorizationCodes
* WIP creating a Personal Access Tokens page. Created a new sidenav for account pages
* Creating tokens is working but the form is broken
* Tokens are created in the UI
* Creating and revoking access tokens from the UI is working
* Improved the create form and copy
* Tokens are a bit shorter and only lowercase
* API endpoint for creating AuthorizationCodes and the web page users hit to create PATs from them
* V3_ENABLED env var and hook that can be used in the UI to show/hide things
* API route to get a PAT (within 10 mins of creating an auth code). Moved some code to core
* Start to build the login command
* Nicer banner when starting the CLI
* Nicer update checking
* Update command style improved
* Removed the template step
* Login command options are now working
* The new CLI is logging in using the Personal Access TOken. But I need to save it still
* Logging in and saving the token is working
* Deal with already being logged in
* Who Am I working with PAT
* Deleted old account side menu header
* Improved the copy on the Auth code page
* Added docs for the login and whoami commands
* Added readme instructions for the update command
* Removed some unused things
* Remove fetchUseProxy for now
Matt Aitken