Commit Graph

1066 Commits

Author SHA1 Message Date
Eric Allam 891278ab1f Enable loading package.json inside of core 2024-03-19 15:24:01 +00:00
Eric Allam 41e1ac9e0b v3: nanosecond precision task events and immediate mode for dev events (#956)
* Store startTime in nanoseconds since epoch, ensure logs using nanosecond precision for their timestamp (so they are not out of order)

* export task events immediately in dev
2024-03-19 15:06:47 +00:00
Eric Allam 86c24afb44 Support json traces and logs in the otel endpoints (#954) 2024-03-19 10:59:50 +00:00
nicktrn b585d1f2c5 v3: checkpoint restore events (#951)
* add checkpoint restore events

* fix retries.enabledInDev

---------

Co-authored-by: Eric Allam <eric@trigger.dev>
2024-03-19 09:02:56 +00:00
Eric Allam e55ca0279b Adding profile support to the v3 CLI (#953) 2024-03-18 17:35:29 +00:00
Matt Aitken 9304c447be Test, output and canceling improvements (#952)
* The vertical lines in the run tree view are now brighter

* Blank state when you click a task with no runs

* Canceling from the runs table

* Fix for styles coming through incorrectly to properties

* Fix for the OTEL output column having a $output prefix if it’s an object
2024-03-18 17:27:52 +00:00
Eric Allam 67cac97b26 Add DEPLOY_REGISTRY_HOST to allow configuring the deploy registry via the server 2024-03-18 15:56:28 +00:00
Eric Allam 0f32dee0eb Prevent depot command from displaying update message 2024-03-18 15:32:26 +00:00
Eric Allam 49e9dd8cb2 Disable fetch experimental warning and detect process resources (#950) 2024-03-18 13:46:24 +00:00
nicktrn 1f190fe680 v3: fix dependency resume (#946)
* fix basic resume mechanics

* fix dependency resume

* fix deploy page sort

* remove task list console log

* checkpoint restore between attempts

* add checkpoint requirement checks to docker provider

* switch to raw execa for long statements

* catch and report pre-connection index errors

* graceful worker shutdown after completion

* fix final completion

* use same task process for each attempt

* update retry catalog entry

* add immediate return task

---------

Co-authored-by: Eric Allam <eric@trigger.dev>
2024-03-18 12:21:40 +00:00
Eric Allam 52c364d691 v3: cli init command + CLI telemetry (#948)
* Adding telemetry to the deploy command

* Add ability to opt-out of telemetry in the CLI

* Remove log

* Unifying some telemetry stuff and adding it to the login command

* init command

* Remove extra trigger dirs

* Couple tweaks to the init CLI

* Don’t use npm init -y to create the package.json file
2024-03-18 10:12:11 +00:00
Matt Aitken a5d83971aa Cancel a run from the span view (#947)
* Tasks get the SHOW_ACTIONS property

* Override properties with the passed in ones in eventRepository

* Link from deployments to environment variables page, no need to link to docs

* “Show parent items” keeps the selected item selected

* LinkButton can accept onMouseDown, onMouseEnter and onMouseLeave

* “Show parent items” keeps any items focused and has a keyboard shortcut

* Action bar with focus button

* Canceling is working

* Nicer layout and button loading and disabled state

* US english spelling “Cancelling” -> “Canceling”
2024-03-17 11:42:30 +00:00
Eric Allam f424c81086 No staging by default, and project external refs now are prefixed with proj_ 2024-03-14 15:11:31 +00:00
Eric Allam 0a33bf7206 v3: handle errors and customizing retrying (#943)
* Add the ability to handle errors at the task and project level

* Cancel in progress runs when disconnecting the dev CLI (in between attempts)

* unify the task executor across dev & prod
2024-03-14 14:38:13 +00:00
Eric Allam 478ce006cb v3: task run status and canceling runs (#941)
* WIP task run status, revamped resuming task dependencies

* Don’t select a span when toggling collapsing

* Cancel runs and attempts, in prod and dev
2024-03-13 15:51:43 +00:00
Eric Allam fe9435f63f Allow alternate config file names and skipping using a config file by passing —project-ref (#940) 2024-03-11 21:31:15 +00:00
Eric Allam d7cc2c9d6c v3: task payload refactor and configurable default retries (#939)
* Make the payload parameter the first argument to the task functions

* Allow configuring default retry behavior, and disabling in dev
2024-03-11 20:30:29 +00:00
nicktrn 20aa1cbe9b v3: checkpoint and restore with sim for local dev (#933)
* zod ipc connection with acks

* pass sender through to ipc handlers

* bits and bobs

* add host networking back in

* disable verbose logs

* restore after tasks and batches.. almost

* restore and resume all the things

* fix for systems without checkpoint support

* Get deployment indexing errors to work with the new ZodIpc stuff

---------

Co-authored-by: Eric Allam <eallam@icloud.com>
2024-03-11 15:20:33 +00:00
Eric Allam 883feab257 v3: various deploy improvements (#936)
* Cache the dependency resolution step of deploys

* Better deploy output and fixed an issue with deleting env vars
2024-03-11 14:31:55 +00:00
Eric Allam 83304f86f8 Capture uncaugh exceptions during indexing and fail deploys, with error output (#934) 2024-03-10 20:57:46 +00:00
Eric Allam e87a0cba1d v3: cli deploy improvements (#932)
* Add support for deploying to staging and prod

* Abort deployment if environment variables are missing

* Actually do typechecking

* Remove the build command and add a —self-hosted option to the deploy command
2024-03-08 14:52:46 +00:00
Eric Allam 47c18d9146 Fixed trigger.dev dev command 2024-03-08 11:16:56 +00:00
Eric Allam c385db63be v3: cli deploy command (#931)
* WIP proxy/deploy

* WIP, registry proxy in express working

* A couple of notes, preparing for indexing

* Move the changes to prod-worker into the new file

* Deploy command working with indexing and runs (docker provider only for now)

* Removed ts-expect-error directive

* Fixed build command
2024-03-08 10:49:42 +00:00
nicktrn 9b6f8f9238 v3: provider and coordinator updates (#927)
* add zod socket

* start using zod socket

* use zod socket for shared queue

* use shared provider shell

* update k8s provider task ops

* set otlp endpoint

* update docker actions and fix builds

* remove unused types

* update injected secret key env var name

* complete socket.io types to schemas migration

* update env example to new v3 key var

* fix coordinator build

* set task run label

* ensure attempt id is always set

* increase prod worker timeout

* set otlp endpoint for on runs

* pass otlp env var to prod worker

* auto-remove completed index jobs
2024-03-06 16:01:59 +00:00
Matt Aitken f50cee66d9 Upgraded to the latest Resend package 2024-03-06 15:18:56 +00:00
Matt Aitken 44d443a048 v3 live run reloading and minor style improvements (#925)
* Initial work creating the Redis publishing and subscribing

* Live view is working

* Show the live reloading status on the page

* Fixed some style issues on the run page with James

* Fix for invalid JSX attributes in the ExitIcon svg

* Some sexy shit

* The end line is now correctly coloured

* millisecondsToNanoseconds exported from core/v3 and imported correctly

* Fix for e2e tests, we changed the h1 to an h2

* Tidied imports

* Removed unused hook
2024-03-06 09:48:00 +00:00
Matt Aitken 2a1273b683 Major v3 web app update (#924)
* Added the new colours to the Tailwind config

* Fixed Tsailwind compile errors

* WIP Geist working in the app

* WIP Storybook

* Storybook is working again… ouch

* Use Geist mono for CodeMirror

* Some find/replace class changes

* Resizable colours improved

* WIP on primary button

* Minor style improvements

* Improve the side menu header styling

* Improved v2 breadcrumbs

* Account side menu tweaks

* The page now works with the NavBar and UpgradePrompt

* Removed unused import

* Refactored the buttons to be dry-er

* Removed the breadcrumbs

* WIP on nav bar

* WIP updating the button styles

* Fix for the old test page

* Fix for type errors

* Removed console.log

* Colour change in the resizable handle

* New loading bar divider

* Removed the design plugin

* Creating our own storybook pages

* Added some “storybook” pages

* Added DetailCell storybook

* WIP new animation for the loading page

* Added the new loading divider

* Fix for the time frame filter having a green border

* Style improvements to the new test page

* Small style improvements

* Improvements to the list pagination components

* Nav bar back button improved

* Improvements to the run page

* Lots of v2 style improvements

* Help looks better

* Improvements to the regenerate API key modal

* More general improvements

* Added section titles to our storybook

* Added form stories

* Fix for missing key prop

* Loading bar is working

* Fixed button submitting state

* Fix for the test button icon

* Code bg improved

* Improved oauth buttons

* Integration page header fix

* Various improvements, mainly changing button variants

* Segmented control style updates

* Added an outline of 3px for the segment control

* Fixes for the billing pages

* More pricing improvements

* Added dialog to storybook

* Typography now in storybook

* Free usage bar added to storybook

* Added inline clode to storybook

* Separated menu types into a sub section in storybook

* NamedIcon now in storybook

* Added page header to storybook

* Added pricing callout to storybook

* Added radio groups to storybook

* Added segmented controllers to storybook

* Added shortcuts to storybook

* Added switches to storybook

* Added toast to storybook

* Added tooltips to storybook

* Added tree view to storybook

* Added usage charts to storybook

* Tarted up the environment variables page

* Fix for the FreePlanUsage panel bg colours

* Added new badge variants

* Added the <TaskPath /> to the tasks table

* Span header improvements

* Started work on the timeline

* Started work on the timeline

* Fix for the run page header

* Timeline tick marks are working

* Timeline working

* Got the durations showing on hover

* Button improvements

* Experimenting with how the duration should be shown

* Lots of refinements to the timeline

* Much nicer API for the Timeline

* The switch is now purple to match the design, with inverted thumb when on

* Removed old stuff

* Export the props

* Added the span with duration back in

* Simplified more and fixed the span durations

* Put in some decent dummy data

* Rendering defaults

* Wip with the mouse cursor

* Improvements to the story

* Added Slider primitive

* Integrated the icons into the slider

* Lots of layout improvements

* Cleaned up imports

* “Root” indicator

* Animated background on partials

* Pull the styles through to the spans and log dots

* Fixed some bugs in the TreeView when nodes get created or destroyed

* Fix for incorrect parentTask reference

* Added run streaming endpoint for live refreshing

* Added the tabs to Storybook

* Fix for status font size mismatch

* Fix for the misaligned run filter status

* Set a minimum duration on the timeline

* The TreeView now re-calculates if the number of nodes has changed

* Partials now render to the end

* Improved the durations gutter

* The duration is calculated using the current time if there are partial events

* Added the environment label to storybook

* Improved the environment label style

* Removed bottom margin from badge

* Button style improvements

* Callouts style fits the new design better

* Got the scrollbars to appear on the tree/timeline

* Added scroll support

* Synced scrolling working

* Close button and exit icon for the detail panel

* secure field shows start of env

* Clipboard improvements

* Latest lockfile

* Fix for the durations wrapping

---------

Co-authored-by: James Ritchie <james@jamesritchie.co.uk>
2024-03-04 17:33:53 +00:00
Eric Allam 64401ee8ec TRIGGER_API_KEY -> TRIGGER_SECRET_KEY in v3 2024-03-04 15:47:58 +00:00
Eric Allam b7845685eb v3: no bundle builds for the dev CLI (#923)
* WIP no bundling

* Convert dev CLI to use an unbundled build process to support otel instrumentation

* Fix pnpm lock file

* A couple of fixes to get typechecking to work
2024-03-04 15:36:09 +00:00
nicktrn 52c9d485f8 v3: cli build command, prod runs, checkpoints (#919)
* fix trailing slash in api url

* add maybe platform down error

* shorten config path

* extend protobuf compiler install instructions

* build command and image model

* new image triggers task indexing

* index support for prod builds

* shorten example provider secret

* multi-stage task prod build

* lock prod tasks to node 18 image

* extract shared build and dev command libs

* pull out more shared deps

* add coordinator and providers

* fix core-apps build

* add dev builds for new apps

* fix cwd

* enable corepack

* some build fixes

* enable buildkit for old docker versions

* coordinator image fixes

* update provider containerfile

* build dev images in parallel

* upgrade pgadmin

* fix prod facade build

* prod runs

* fix merge

* don't knock out platform on invalid attempt id

* fix prod facade

* rename to build.ts

* fix prod builds

* set to executing after fetching payload

* make prod worker listen on random port if in use

* remove experimental warnings in dev

* prod resume

* prevent execution after completion

* exit prod worker after completion

* always restart otel collector

* docker checkpoints and prod runtime messaging

* don't retry indexing without chance of success

* make platform checkpoint aware

* deploy with existing hash sets latest worker

* log restore requests

* only try to checkpoint long waits

* tidying up

* lockfile

* fix build

* prod worker merge fixes

* fix prod complete and cancel

* fix lua nil checks

* socket namespace abstraction

* cleanup

* make all build args optional

* add build script

* don't require env vars for dev

* fix schema

* prod merge

* small fix

* bind correct logger

* fix v3 ref catalog entry

* resume prod batch

* pass socket to error and disconnect handlers

* fix non-batch resume

* fix batch resume

* send connection env vars when not in dev

* create worker via socket

* move api client back into v3 cli

* fix lockfile

* fix resume with failures

* marqs replace message

* typecheck prior to build

* don't define api url in prod builds

* support prod retries after resume

* skip typecheck option
2024-03-04 13:10:07 +00:00
Eric Allam ca47e6bdcd v3: cli cancel improvements and add back in deprecating background workers (#918)
* Show runs as cancelled when killing the dev CLI

* Fixed error when waiting for a task

* Deprecated workers will no longer be able to run unlocked task runs

* Fixed selecting the latest background worker version in test list

* Don’t propogate trace context when triggering an async task

Also add batch id to the context and the task events
2024-02-29 15:34:00 +00:00
Eric Allam 933cf0f119 v3: run.isTest now propagates through child tasks and the trace (#917)
* run.isTest now propagates through child tasks and the trace

* Try to fix the pr checks docker hub rate limit error
2024-02-28 17:01:31 +00:00
Eric Allam b6517af522 v3: Cancel dev runs when the dev CLI exits (#916)
* Cancel dev runs when the dev CLI exits

* Clear out in progress attempts after cancelling them
2024-02-28 16:00:39 +00:00
Matt Aitken 479445f741 V3 Test feature (#915)
* Added isTest to TaskRun

* TriggerTaskService now accepts isTest and adds it to TaskRun if set

* Set the cliVersion and sdkVersion when indexing

* Added a test page

* Radio buttons are now circles and accept a ReactNode

* The background of the code editor is now darker

* WIP on the test task page

* Improvements to the test page

* Testing is working

* Fix typo

* Show the real isTest value in the runs table

* Added a key to the outlet so it correctly resets all the state on the sub-route
2024-02-28 15:52:24 +00:00
Eric Allam 1fabcf68fa Override local env vars with env vars in the server (#914) 2024-02-27 15:44:55 +00:00
Matt Aitken d7c79ee651 V3 API keys page and Environment Variables (#907)
* New API keys page working

* Added EnvironmentVariable and EnvironmentVariableValue tables

* Environment variables presenter and repository started

* Icon now accepts a ReactNode as well

* The create form is working

* Improvements to validation

* Getting all the environment variables is working

* Editing an env var working

* Fix for deleting a value from an env var

* Separate route for adding new env vars. Works with “Save and add another”

* Editing is working properly, and revealing all values

* Deleting env vars

* Get environment variables with the environment id

* Added th env vars count to the Environment variables page

* Removed the commented out old Env & API keys side menu item

* Fix imports in ClipboardField

* Added a storybook page for all the variants of input fields

* Upgraded Tailwind for the :has selector

* Fix for the edit button not filling the width

* Reworked the input fields so they work with a variable width icon on the left

* Improvements to the new env var modal

* Improve the style of the edit environment variable panel

* Fix for when Input is type=“text”

* Icon now works with FunctionComponents and ComponentTypes properly

* Removed a console log

* Removed unused imports

* Use a const schema for SecretValue

* Added friendlyId to EnvironmentVariable
2024-02-27 14:35:43 +00:00
Eric Allam 8d6e710dc2 V3: retry utilities (#913)
* WIP

* retry.fetch w/intercepting to test

* Fixed the incorrect renaming of limit to maxAttempts in v2

* Fixed source map support in dev

* Cleanup background worker files once they close
2024-02-27 13:54:51 +00:00
Eric Allam 90a302f9f7 v3: batch trigger (#908)
* Support for batch triggers

* Various trace fixes and improvements

* Moved trace into the logger, fixed span cardinality issues with waits and triggers

* Add support for attaching a debugger in dev with —debugger flag
2024-02-26 11:21:06 +00:00
Matt Aitken 07efef405d v3 span details (#906)
* WIP on saving the resized state in a cookie

* Progress on remembering the panel width

* The resizable panel now remembers even when closed

* Improvements to the span panel

* Hide integrations in the v3 side menu

* Added “UTC” to the times

* WIP changing how styles work

* Reference unflattenAttributes from the package

* Unflatten now correctly unflattens arrays

* Unflatten typecheck fix

* Improvements to the spans and UI for them

* Added the “events” to a span, including special styling for errors

* Unflatten doesn’t work on an array of Attributes
2024-02-23 09:54:01 +00:00
Eric Allam dd63fe6e9f v3: Implemented in-process task retries in dev w/cache support (#905)
* Implemented in-process task retries in dev w/cache support

* Improve the stack trace correction
2024-02-22 10:12:23 +00:00
Eric Allam 24600d2c01 Fixed unFlattenAttributes import 2024-02-21 20:15:06 +00:00
Matt Aitken 584936fa29 Early v3 project UI (#904)
* Added ProjectVersion to projects, defaults to V2

* Moved v3Enabled to the existing feature flags

* Update to using the v3Enabled feature flag

* Allow people to choose v3 when creating a new project

* Added version to the ProjectPresenter

* v2 project and v3 project redirecting

* .env.example for the v3-catalog

* Added sdkVersion and cliVersion columns to the BackgroundWorker table

* Added additional classes for environments

* First draft of the Tasks table

* Link to the task page

* V3 side menu project items

* Moved the ListPagination component into the components folder

* Moved the TaskListPresenter to a v3 folder

* Bare bones task page with tabs

* Task page, we’re going to delete this though

* Bare bones runs table working

* Rejigged the columns

* The run table status

* Environment and status filtering working

* Added time filters

* Cursor and direction is working… I think it’s tricky to know for sure

* Added TaskRun numbers

* Selecting a task now links to the runs page with filters turned on

* Added support for the enqueued status

* Reworked the task table

* Link to the task and environment runs

* The run page is rendering a tree of the events

* Change the task table column to “Created at"

* WIP on making the run tree view look good

* Improvements to the run tree

* Removed the janky scroll bar flash

* Change the title of the task logs to just the task ID

* Live timer when a span is running

* Added “Show parent items” link

* Fixed the bug jumping to parent items

* Style improvements to the run

* Added the resizable handle to the run page

* Moved formatDuration to core/v3

* Wait for now has nice log messages

* The run detail panel is working with just a title for now

* Navigating to a span and persisting between page reloads is working

* The run detail timeline

* Wrap log dates in a paragraph

* Latest span detail view

* Scrolling in the right-hand panel

* Tweak default resizable layout for the run page

* Attempted improvements to stop a recursive issue on TreeView

* TreeView useReducer WIP

* Changed the change callback

* Add changes to the state

* Use the new onSelectedIdChanged in the app

* Don’t update the state from the outside

* Filtering fix, although navigation doesn’t work nicely

* Filtering working when changing the content and clearing it

* Don’t allow the same selectedId callback to be called twice…

* useDebounce hook

* Switch from using defer to just typedjson for now

* Latest attempt at span navigation

* Removed log

* Turn off filtering for now, it’s causing the Links to break somehow

* Fix for page height issue

* Added (Developer Preview) to the v3 project select box

* Revert "Filtering working when changing the content and clearing it"

This reverts commit 20d9fbe36ded619e9d3eb0b23d4ed0568a88a615.

# Conflicts:
#	apps/webapp/app/components/primitives/TreeView/TreeView.tsx

* Switch to using the old filtering

* Added useThrottle hook

* Instantly close the panel when deselecting a node, use debounce when navigating to a span

* Better spacing in the right hand panel

* Comment out the v3 side menu pages that don’t exist yet

* Fix for typecheck fail

* Clear the statuses too
2024-02-21 18:42:12 +00:00
Eric Allam a54bdb23bf Fixed the Docker build (error in generate-protos.mjs) 2024-02-20 13:31:16 +00:00
Eric Allam 07f7c65054 v3 queue system (#903)
* Introducing Modular Asynchronous Reliable Queueing System (MarQS). Works in dev

* Convert MarQS to using lua and dealing with concurrency

* Simplified the timeout queue and current concurrency is now a set instead of a flat value (to support idempotency)

* Implement task heartbeating and reconnect the background workers CLI when the websocket connection reconnects

* Start adding internal telemetry support for the server

* Get env vars to work in dev and implement prisma tracing in webapp

* Cleanup telemetry and implement it in the consumer

* Implement dequeuing a message from a parent shared queue

* Implement a custom logger exporter instead of using console log exporter

* Use node instead of shell for generating protocol buffer code

* Propogate trace context into debug logs, and allow turning off logger exporter through env vars

* Switch to using baselime for internal otel data

* Make orgMember optional to fix type issues

* Provide the CLI dev env vars through the CLI, don’t build dotenv into facade

* Removed the logger import

* Address Matt’s comments

* Addressing more of Matt’s comments

* Handle sending an execution after a websocket connection closes

* Remove auth from the env attributes to prevent obfuscation
2024-02-20 12:16:41 +00:00
Eric Allam 6be1099a12 Don’t git submodule in docker 2024-02-12 14:53:03 +00:00
Eric Allam baf3a84cda v3 dev cli command + more (#894) 2024-02-12 14:24:32 +00:00
github-actions[bot] a1860dbaae chore: Update version for release (#899)
Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>
2024-02-12 13:00:14 +00:00
Matt Aitken 336029b842 Organization renaming, deleting and one-click email unsubscribe (#885)
🚀 Publish Trigger.dev Docker / typecheck (push) Failing after 0s
🚀 Publish Trigger.dev Docker / e2e (push) Failing after 4s
🚀 Publish Trigger.dev Docker / units (push) Failing after 10s
🚀 Publish Trigger.dev Docker / publish (push) Has been skipped
* Added the org settings page to the sidebar

* Added loading states when renaming/deleting projects

* Don’t show deleted orgs in the app

* The actual db migration file

* The Org settings page with the actions working

* Don’t remove org members, just leave them

* Don’t show invites from orgs that are deleted

* Allow disabling IntegrationConnections

* Don’t refresh IntegrationConnections that are disabled

* Set all the integrations as disabled

* Updated the unsubscribe checkbox text

* Unsubscribe route

* Use the magic link secret, not the encryption key. Also make the error message more vague

* Only members of the org or project can rename them

* Don’t throw an error if the connection can’t be refreshed, return undefined instead
2024-02-07 10:52:50 +00:00
Matt Aitken 29edcd3df9 Rename and delete projects (#880)
🚀 Publish Trigger.dev Docker / units (push) Failing after 0s
🚀 Publish Trigger.dev Docker / e2e (push) Failing after 0s
🚀 Publish Trigger.dev Docker / typecheck (push) Failing after 4s
🚀 Publish Trigger.dev Docker / publish (push) Has been skipped
* Side menu: selector title changed to project. Changed section headers and made it Slack + Discord

* Added project settings page. Renaming is working

* Added project.deletedAt column

* Deleting projects WIP

* Allow clearing the project from the session

* Don’t load projects that are deleted

* Improved the project selection logic

* Clear the session project id when deleting a project

* Deleting the last project in an org now works. The new project page doesn’t have the sidebar anymore.

* Removed false code comment

* Added the JobRun index back in, using Prisma. Otherwise it tries to remove it on each migration

* Only return environments where the project isn’t deleted. This will block API calls

* Do the project deletedAt check in code, not SQL

* Made it clearer what the code does with a comment and applied the same logic to the public api key
2024-02-05 13:14:33 +00:00
Matt Aitken 28052daad9 Faster jobs page (#879)
* Optional next/previous with the list pagination

* Defer the loading of the runs table

* Only return the latest run in a separate query, then do a join in code

* Added a composite index to speed up getting the latest run from a job id
2024-01-30 17:02:04 +00:00