Commit Graph

3357 Commits

Author SHA1 Message Date
nicktrn 8cae1d0875 v3: fix triggering tasks with custom queues (#1242)
* add missing dotenv requires in catalog scripts

* pass task queue options to all task trigger functions

* remove unnecessary include

* fallback to background worker task queue options

* add changeset
2024-08-01 11:26:34 +01:00
Matt Aitken 09413a62a4 Improved run page and replaying (#1240)
* PropertyTable component changed to use sub components

* Separate run span component

* Early WIP on tabs that use search query

* Shortcut key tabs for the span panel

* The span timeline is working

* When runs get expired, update the OTEL event with an error

* Improved the expired error message

* Reveal env vars when editing

* Tightened things up a bit

* Added detail tab properties

* Progress dashed line

* Move the env label next to the Run number title

* Top level cancel/replay buttons

* Replay with a different payload and environment

* Fix for non json payloads

* Hide the clear/copy buttons

* UI improvements with large payloads

* Close the panels when you replay/cancel

* Added the new timeline to spans

* Use the u-turn left icon for replay

* Added an index for spanId on TaskRun

* Remove replay/cancel buttons the span view

* Replay shortcut works inside the code editor

* Split the log/span inspector between Overview and Detail as well

* More improvements to the inspector

* Context and output improvements

* Focus on run working

* Added version to run.ctx

* Added some padding to the detail view

* Added context tab with shortcut

* Only load the replay data when the dialog is open

* Replaying uses the tags from the original run

* Links are now text links

* Removed version links for now because we don’t have dropdown filters for them yet

* Tabs are now outside of the scrollview

* The inspector is now 30% of the width by default

* Allow replaying and editing SuperJSON payloads

* Deleted unused CodeGroup file

* Increase the tags limit to 5, do the limiting on the server

* The admin tooltip now always shows basic org, project and user info

* Fix for schedule inspector disabled state layout

* Remove new unused span metadata and context

* Removed unused import
2024-07-31 15:11:50 +01:00
nicktrn 9882d66f87 v3: pre-pull deployments for faster startups from the first run (#1236)
* pre-pull deployed images on all workers

* automatically clean up pre-pull resources

* rename pre-pull image to deployment

* add changeset
2024-07-26 13:12:58 +01:00
nicktrn e27d5cd861 Prevent registry proxy from trying to send responses twice 2024-07-26 08:44:03 +01:00
nicktrn e64cd5ee86 Fix usage page for previous months 2024-07-25 18:34:14 +01:00
James Ritchie ac088c1a73 Fixed safari bug not respecting recharts height 2024-07-24 12:11:01 +01:00
James Ritchie 12d2bd6cdd Added links to featurebase in the docs and app feedback panel (#1213)
* Added links to the new roadmap and feature request pages. Also redirected the old feature-matrix page to docs home

* Added a new request feature link to the help and feedback panel

* Added Book a call with the founders option in the feedback panel
2024-07-24 11:59:48 +01:00
Matt Aitken 8614ea2107 Tags docs and a couple of minor fixes (#1235)
* runs.list and runs.retrieve OpenAPI docs updated

* Added docs for how to use runs.retrieve and runs.list with usage data

* SDK reference for triggering with tags

* Added Tags docs page to the sidenav, it’s a copy of usage right now

* Fix for type “users” -> “uses”

* Docs for the tags feature

* Fix for bad import

* Added docs how propagating tags to child runs

* Better error messages when Zod parsing fails for triggering and batch triggering

* Tags are now a string or array of strings, this makes it a lot easier to use
2024-07-24 11:22:03 +01:00
Matt Aitken 8dfd47eb41 Fix for the tags making the run table rows taller 2024-07-23 19:17:02 +01:00
Matt Aitken 55264657d7 Run tags (#1232)
* TaskRunTag migration. Removed unused TaskTag

* Show tags for a run in the run list

* API endpoint for searching tags

* Let’s not expose an API endpoint for tags at the moment

* Filter by tags working from the URL

* Tag async filter working

* We don’t need a “None” option because it’s multi-select

* When triggering a run you can add tags

* SDK runs.retrieve and runs.list with tag support

* Tidied imports

* Changed the run list query so we show all the tags even if a run only matches one of them

* Fix for dealing with weird characters in tags

* Run tags changeset

* Creating a project has a proper loading state (and blocks multiple)

* Improved the error message for tag length

* Added a tags icon for display on the run screen

* Convenient functions for creating and getting run tags

* tags.set() from inside the run function

* Replay a run passes tags through

* Less ridiculous tags for the catalog example

* Allow passing just a string for the tags

* Order by id because there’s an index on the primary key already

* Trim the tags earlier so we don’t accidentally error if a blank string is passed

* Renamed some thing from setTags to addTags

* Use findFirst for the tags project lookup

* Added an index for "TaskRunTag"("name", "id")

It massively improves the performance of run filtering based on tags

* Use `array_agg` for the run list tags so pagination works and we get a single result for each run

* Tidied imports

* More comprehensive test of tags with all triggering functions

* Support tags with tasks.trigger, tasks.batchTrigger and tasks poll variants

* Added tasks.batchTrigger tags

* Sort the tags in the UI so they’re always in the same order

* Added tooltips in the run table for delay, ttl and tags

* Added costInCents, baseCostInCents and durationMs to runs.retrieve and runs.list

* Added support for displaying a split tag if you use key_value or key:value format

* Fix code comment

* Tweaked the JSDoc to make the prefixing clearer

* Added tasks.triggerAndWait to the tags test task
2024-07-23 16:48:09 +01:00
Matt Aitken 6293290fb9 Better prediction of usage (uses Linear regression) 2024-07-20 10:50:46 +01:00
James Ritchie da592d867a Removed logo 2024-07-19 15:20:40 +01:00
Eric Allam d7089e12a9 Replace findUnique with findFirst when doing a relation aggregate 2024-07-19 10:18:56 +01:00
Eric Allam 162d864517 Fixed prisma dataLoader slow query when getting an attempt payload 2024-07-19 09:42:48 +01:00
Matt Aitken 8ba9987944 Declarative schedules (#1226)
* Added type (STATIC or DYNAMIC) to TaskSchedule. Defaults to dynamic

* WIP with dev indexing of static schedules

* Added a code comment

* First stab at deleting unused static schedules

* Dashboard changes for the static schedules

* Generate the description. Upsert the instances when editing. Fix for the friendlyId

* Don’t allow deleting of static schedules

* Don’t allow enabling/disabling of static schedules

* Added filtering for schedule types

* Syncing of schedule for deployed tasks

* Static schedules are now created for each environment

* Added a second static schedule for testing

* Add the type to the schedule task run payload and the object you get back from the SDK

* Changed static/dynamic to declarative/imperative

* Timezone example

* Changeset

* Updated scheduled docs to include declarative

* When you test a schedule it set the type to “IMPERATIVE”

* Improved the tooltip

* Fix for queue time continuing to rise when a run is canceled/expired etc

* Update the info panel on a selected declarative schedule

* Check if there are no instances. This should never happen but log an error if it does

* Throw errors and push them through to the CLI dev command

* Fail deployments if creating the background tasks or schedules fails

* Format the deployment error so it gets displayed

* Changed the maxed out schedules error message to remove bit about support
2024-07-18 20:24:54 +01:00
nicktrn e0cdebbe09 Disable excessive websocket heartbeat logs 2024-07-17 13:28:40 +01:00
nicktrn dbda820a71 v3: fix websocket timeouts (#1220)
* prevent uncaught websocket exceptions

* dev cli sends regular pings to keep connection alive

* add changeset

* fix for prod message forwarding

* add server-side websocket ping

* Revert "dev cli sends regular pings to keep connection alive"

This reverts commit 9a6a7a21433b9f0665f72be015d0ba5a8d83eba4.

* add dev cli websocket debug logs, always print errors

* actually start the ping service

* update changeset

* catch remaining dev cli message handler errors
2024-07-17 09:05:15 +01:00
Matt Aitken f91a118958 Fix for dev env vars getting deleted when another user edits theirs 2024-07-16 10:57:35 +01:00
Matt Aitken 0847bee5b8 Fix for stripping out TRIGGER_ env vars 2024-07-15 17:11:15 +01:00
Matt Aitken 6152b2f3dc Don’t error when setting TRIGGER_SECRET_KEY or TRIGGER_API_URL env vars, just filter them out 2024-07-15 16:39:13 +01:00
Matt Aitken ec8381e2d0 Fix for staging environment variable API endpoints
Ugly but effective fix for the fact we can pass the slug in as “staging” when we use “stg” internally
2024-07-15 16:33:00 +01:00
Matt Aitken 76a15fd278 The run filter for tasks query was very slow if you had a lot of synced tasks 2024-07-15 15:59:11 +01:00
nicktrn 824a6205f7 Fix deployments stuck in deploying state 2024-07-15 12:00:16 +01:00
Matt Aitken a0984d5292 Allow alerts from external and unlinked Slack channels 2024-07-12 11:14:57 +01:00
Matt Aitken c61d8ed890 Removed badge from the Tasks in the side menu
It was showing the v2 job count. Mostly this didn't actually show anything but one user reported they saw a number here and it's definitely a bug.
2024-07-12 10:51:54 +01:00
Eric Allam 76a0f61dee v3: improve schedule reliability by ensuring next tick is always scheduled 2024-07-11 15:35:34 +01:00
Jacob Paris d934feb02a feature: add no-trigger-core-import (#1175)
* feature: add no-trigger-core-import

* fix: handle aliases and limit to exported modules

* fix: use longest export map name

* fix: support export type and interface

* fix: simplify message when export map is needed

* feature: fine grained modules for core

* fix: add prerequisite lint plugins

* fix: why do I have to fix these components again???

* fix: tests

* Add eslint config and vscode settings

* Create yellow-roses-arrive.md

* Remove the vscode linting

This is temporary so no one fixes these until we merge the remix branch

---------

Co-authored-by: Eric Allam <eric@trigger.dev>
Co-authored-by: Eric Allam <eallam@icloud.com>
Co-authored-by: Matt Aitken <matt@mattaitken.com>
2024-07-11 15:14:55 +01:00
Eric Allam 7fdedf3137 Fix excessive logs from the MarQS rebalancer 2024-07-11 14:20:59 +01:00
Matt Aitken 3d6cfe5ead Admin API endpoint that adds v3 staging environments, and updates their concurrency 2024-07-11 10:22:21 +01:00
nicktrn 946efd1fb4 v3: deployment index retry (#1205)
* add built at date to worker deployments

* stay on correct deployment page when clicking row

* expose built at via presenters

* more specific deployment statuses

* add deployment indexing retry

* improve deploy timeout logs and add dequeue on completion

* unify deployment indexing retryable status

* only allow index retries of the latest deployment

* improve jsxfu
2024-07-10 17:54:36 +01:00
Matt Aitken 3d645e2b2c Fix for usage of zero displaying weirdly 2024-07-10 17:31:40 +01:00
Matt Aitken 45cb75b8a3 Only show dev runs in the tasks usage table 2024-07-10 17:22:35 +01:00
James Ritchie fa5b3fe11e Only show the github verified badge on the Select Plan onboarding route 2024-07-10 16:50:52 +01:00
James Ritchie 90506322b5 Hide the tooltip content on the usage graph 2024-07-10 16:46:10 +01:00
Matt Aitken a685f5a4ec Removed the enable/disable v3 from the admin area 2024-07-09 19:00:19 +01:00
Matt Aitken 9ba5dbaf91 When manipulating dates for periods force UTC 2024-07-09 09:25:08 +01:00
Matt Aitken 681bb729e9 Added GitHub verification failed route 2024-07-08 18:48:39 +01:00
Matt Aitken 07fb3fd90d When replaying and they’re out of credits return undefined so we can deal with it 2024-07-08 18:48:36 +01:00
Matt Aitken 3bfebc06cf When running a test and they’ve run out of credits, show an error message 2024-07-08 18:48:12 +01:00
Matt Aitken 30f7866e17 Send different error if they’ve run out of credits 2024-07-08 18:47:52 +01:00
Eric Allam e04d44866f v3: better handle null unicode characters when creating task events and completing a failed attempt 2024-07-08 15:59:19 +01:00
Matt Aitken 72f3a4b128 Usage and billing (v3) (#1201)
* v3 subscription endpoints

* Use pnpm linked billing package during development

* Moved v2 billing components into a subfolder

* Select plan using the real data

* Improved v3 plan display

* Use new api response that doesn’t require a Stripe call

* Free flow is working

* Added GitHub modal and verified badge

* Deleted old request v3 access component/route

* Allow setting classes on the Tooltip button

* Paid plans working

* Redirect from select plan if you’ve got v3 enabled

* Loading state improvements

* New admin API endpoint to set concurrency across multiple environments

* When projects are created, conditionally create staging based on the plan

* New billing page working with side menu and stripe portal

* Layout, formatting and some plan state improvements/fixes

* Don’t show the period if you’re on the free plan

* Temporary upgrade callout

* Refactored the platform code so it’s easier to call and doesn’t require a isManagedCloud check

* Send taskIdentifier to OpenMeter

* Side menu

* Upgrade prompts

* More improvements to the app-wide usage indicators

* Early work on usage graphs

* Added the usage bar for v3

* Moved code to presenter and now using defer

* Added the tasks table to usage

* Improved the v3 usage bar if theres’ no usage on a paid plan

* If no run data, still render a graph

* Usage page errors when defered loading fails

* Don’t show the public API key for v3, they’re not used and probably never will be

* Improved the upgrade callout and API keys page layout

* Only show the “reveal all” toggle if you have environment variables in the table

* Replaced Upgrade callout with a more generic InfoPanel component

* better panel width

* Show conditional upgrade prompts based on plan and number of schedules used

* Removed duplicate class

* Wider blank state panels for the scheduled page

* Wider info panel for the env var page

* Blank state now using the info panel

* Platform alerts prompt now using the InfoPanel

* Deploy blank state uses InfoPanel

* Github verified badge padding adjustment

* Improved the layout of the page, some style tweaks, organized imports

* Better default tooltip style

* could be undefined fix

* text fix + style updates

* Changed the billing icon in the side menu

* Billing page layout and style improvements

* plan tooltips don’t use dark variant

* Don’t highlight the plan on the billing page

* Tooltip underlines stand out more

* Fixed padding in the PageTitle

* Tooltips use the correct cursor

* Improved the plan banner on the billing page

* Fixed Header1 inconsistent font weight

* Fixed issue where input field focus states were being clipped

* Fixed large button not having large text size

* Added a link to the Get in touch copy and improved the connect to GitHub modal

* Select plan page uses the MainCenteredContainer

* Better logging from the Loops endpoint because this error finally got hit

* Move the ingestion of compute to the platform

* Reporting usage of invocations moved to the platform

* Get the entitlement before triggering a non-dev task

* Contact us enterprise plan button opens the feedback form

* Removed Github discussions link from the Feedback panel

* Swapped billing icon for credit card

* Show a Unlock staging panel on the env var page

* Updated staging environment colour

* Show a prompt to upgrade to get staging in the new env var modal

* Improved the edit env var modal

* Implement ability to disable org concurrency

* Use common logic for the plans

* Use the billing server to get the schedule limits

* Some schedules page fixes

* More convenient way of getting a limit

* Use the new schedule limit

* Team member limiting

* Made the limit visible on the team page

* Limit alerts

* Added an index for TaskRun.scheduleId

* Remove console.log on schedules page

* Added durations to the run table

* Tabular numbers

* Improved the usage page formatting

* Only admins see the compute column on the run table

* Include the base cost on the usage stats

* Moved the status to the sidebar

* Optional table header tooltip

* Allow InfoIconTooltips to have customizable content styles

* Added a tooltip to the duration header, changed no test to a dash

* Removed all references to signing up to v3 from the docs

* Switched @trigger.dev/billing to @trigger.dev/platform

* Passing up the variant for the InfoIconTooltip

* table tooltip max-width fixed

* Switch to the published @trigger.dev/platform 1.0.11

* v2 usage page title changed to include “v2"

* code theme has a transparent background so it works on any background

* duration columns now grouped together nicely at wide screen size

* Last duration column fills the width properly

* Fix for the per run price being in cents not dollars

* Show the total cost with 8 decimal places

* Show 8 decimal places in the usage graph tooltip

* Moved the UpgradePrompt to the v3 folder

* Prepare to use Shadcns chart helpers

* Much nicer chart

* Small tweaks to the graph

* Fix run table col spans for empty/loading messages

* We don’t need isManagedCloud in createProject

* Hide v3 usage/billing pages if there aren’t v3 projects in your org

* Removed unused tooltipStyle

* Usage bar now says “Included usage” instead of “Tier limit” if you’re paying

* Get the plan/usage data in parallel

* The usage page now has a month dropdown and all data is for that calendar month

* Ensure the passed date is the 1st of the month

* Use the machine presets from the platform package

---------

Co-authored-by: James Ritchie <james@jamesritchie.co.uk>
Co-authored-by: Eric Allam <eallam@icloud.com>
2024-07-08 14:42:19 +01:00
James Ritchie c9ac6ef93f Made the bulk action bar taller, bigger buttons and more colour. (#1202) 2024-07-05 19:19:29 +01:00
Dan a440101af4 Updated the codeblock colours (#1195) 2024-07-05 10:34:56 +01:00
Eric Allam 4f95c9de4e v3: Cancel awaited subtasks and reliable rate-limit recovery (#1200)
* v3: cancel subtasks when parent task runs are cancelled

* v3: recover from server rate limiting errors in a more reliable way

- Changing from sliding window to token bucket in the API rate limiter, to help smooth out traffic
- Adding spans to the API Client core & SDK functions
- Added waiting spans when retrying in the API Client
- Retrying in the API Client now respects the x-ratelimit-reset
- Retrying ApiError’s in tasks now respects the x-ratelimit-reset
- Added AbortTaskRunError that when thrown will stop retries
- Added idempotency keys SDK functions and automatically injecting the run ID when inside a task
- Added the ability to configure ApiRequestOptions (retries only for now) globally and on specific calls
- Implement the maxAttempts TaskRunOption (it wasn’t doing anything before)

* Adding some docs about the request options

* Fix type error

* Remove context propagation through graphile jobs

* Remove logger

* only select a subset of task run columns

* limit columns selected in batchTrigger as well

* added idempotency doc

* allow scoped idempotency keys, and fixed an issue with the unique index on BatchTaskRun and TaskRun

* Removed old cancel task run children code
2024-07-05 10:30:52 +01:00
nicktrn 8f43aecacc Handle redis errors more gracefully 2024-07-04 11:43:11 +01:00
nicktrn eb3b7b6f9e Fix logger import 2024-07-03 17:15:16 +01:00
nicktrn d5772e57bd Remove rogue zod error 2024-07-03 16:57:13 +01:00
nicktrn 14c2bdf89b v3: checkpoint and reliability improvements (#1198)
* only checkpoint retries with delays greater than threshold

* rename checkpoint threshold env var

* log task monitor ignores

* crash runs with unbounded attempts

* fix retry check in shared queue consumer

* add missing stop for env var sync spinner

* prod entry point refactor

* missing awaits

* more verbose prod flush and exit logs

* reduce checkpoint support logs

* heartbeat while checkpointing between retries

* dynamic coordinator config

* measure lazy attempt creation time in prod

* simplify delay threshold

* heartbeat clarifications

* crash run if it doesn't reach checkpointable state

* require dynamic config threshold

* fix retry prep, await previous worker kill

* unify wait mechanics

* fix prod worker without tasks error

* ensure worker is ready to be checkpointed for dependency waits

* improve worker attempt creation logging

* prevent crashes caused by failed socket schema parsing

* fix dynamic imports in v3 catalog

* clarify attempt retry mechanics

* move backoff helper to core-apps

* remove core-apps barrel file

* add backoff execute with callback

* deprecate non-lazy attempt messages

* update socket.io-client to v4.7.5

* fix socket.io types for emits with timeout

* retry all the things

* remove todo

* fix retry restores

* improve index failure logs

* retry incomplete dependency waits

* fix checkpoint in-progress detection

* prevent losing messages during reconnect

* checkpoint when greater or equal to threshold

* improve handling of duration wait edge cases

* add ready for lazy attempt replay

* retry attempt completion

* allow failing runs with unfriendly run id

* fix min max jitter

* cancel checkpoints on run failure

* improve attempt creation errors

* prevent crashing run on failed cleanup

* handle at-least-once execute lazy attempt delivery

* log exit code on prepare for retry

* fix timeout promise

* mark some things

* chaos monkey superpowers

* refactor checkpointer

* set chaos monkey defaults

* less chaos

* fix backoff

* handle uncaught entry point exceptions

* only replay rpcs on true reconnects

* allow resume unless final run status

* add changeset

* small fixes
2024-07-03 16:54:09 +01:00
Matt Aitken 7976d924fb Marqs rebalancing is based on the env var again 2024-07-03 09:54:50 +01:00