Commit Graph

179 Commits

Author SHA1 Message Date
Eric Allam 215b60c742 v3: superjson output support (#971)
* superjson output support

* Better support for superjson in the task events, limiting output attributes

* Offload large outputs to object store (r2)

* Finishing up the offloading of large outputs/payloads to an object store
2024-03-27 11:57:22 +00:00
Eric Allam 395abe1b92 v3: Various fixes to support papermark use-case (#970)
* Adding a changeset for v3

* Add a version field to @trigger.dev/core-apps package.json

* Build trigger.dev when doing a prerelease

* bundle @trigger.dev/core-apps with trigger.dev cli

* Fix the init command config template

* Don’t use * for the @trigger.dev/core dep version specifier

* strip workspace: from the package version before installing it

* Added dependenciesToBundle config option to bundle ESM only packages

* Added logging around resolving dependency paths

* Try again

* Resolve dependencies based on the project dir first

* flip the bundled default

* Adding some logs around dev task completion notifications

* Adding some additional logs

* Add more logs

* Write out the log using process.stdout

* Store pending completion notifications and resume them when awaited (fixes race condition)

* Cleanup some of the logs

* Copy over the postinstall step from the projects package.json

* Add support for including additional files when deploying (e.g. prisma schema)

* Don’t run scripts when resolving deps

* copy all the files just in case anything is needed in postinstall

* Remove duplicate option

* Use the tag when outputting the dev command

* Remove the postinstall script

* add the trigger dir to the config if the default is not chosen

* Remove the “hud” display in the dev command

* trigger file names with dashes now work

* Better file watching in dev

* Much better duplicate ID experience now

* Much better “Project not found” error

* Export the handleError function types from sdk

* Add support for configuring instrumentation

* Upgrade and unify @opentelemetry/* packages (and remove storybook from the webapp)

* Fix typescript error in react package

* Upgrade react types in webapp

* Allow span icons to be determined based on the span name (e.g. prisma:)

* Ignore built-in env vars when checking for env vars, and allow continuing the deployment even if missing env vars were detected

* Improve the retry.fetch default behavior and option structure

* Fixed typescript errors with packages/email react types

* Update the retry.fetch docs
2024-03-25 17:02:49 +00:00
nicktrn 9ecf07731a v3: small features and fixes for prod infra (#968)
* add checkpoint restore events

* fix retries.enabledInDev

* fix tsconfig paths

* fix provider build and dev

* update kubernetes provider and fix builds again

* upgrade prod base to node 20

* update infra publish workflow

* rethrow k8s errors after logging

* shorten index container names

* fix error type assertion

* rename type assertion

* remove resource limits for now

* add missing run id on create

* add push to deploy command for self-hosting

* checkpointing fixes

* update coordinator image

* ensure valid registry login

* delete checkpoint archive after successful push

* log options on error

* structured logs for socket connections

* fix structured log merge

* exit process after checkpointing

* update restore pull secret name

* append shortcode to restore names

* log handler payload

* disable post start lifecycle hook

* pass in coordinator host via volume

* replace dapi with taskinfo

* add missing restore label

* don't restart restored containers

* remove init container from create

* atomic post-completion checkpoints

* switch to run id for container names

* improve wait accuracy

* measure basic checkpoint perf

* always log disconnect reason

* use system clock to end wait spans

* checkpoint readiness and cancel signals

* restore from checkpoint events and fix statuses

* remove attempt id env var

* restore dependencies from events

* reconnect wip

* lifecycle hooks are back

* fix hooks and improve reconnect

* make docker send postStart hook

* only checkpoint for retry if large delay

* fix a few more resume issues

* lifecycle hook fixes

* skip connection handler when waiting for post start hook

---------

Co-authored-by: Eric Allam <eric@trigger.dev>
2024-03-25 16:28:56 +00:00
Eric Allam 7b3ffe040b v3: config file is now ts by default and uses named exports (#962)
* A bunch of changes to the config file:

- No more default export, the config is exported via the `config` named export
- `handleError` is moved to a separate named export
- Added the ability to include additional packages (e.g. wrangler) that aren’t included by default (think CLI usage)
- More reliably install packages in dev
- The config is now built first instead of directly imported, meaning we can now support typescript config files (`trigger.config.ts`)

* Update to trigger.config.ts in `trigger.dev init`
2024-03-21 10:40:44 +00:00
Eric Allam 41e1ac9e0b v3: nanosecond precision task events and immediate mode for dev events (#956)
* Store startTime in nanoseconds since epoch, ensure logs using nanosecond precision for their timestamp (so they are not out of order)

* export task events immediately in dev
2024-03-19 15:06:47 +00:00
nicktrn 1f190fe680 v3: fix dependency resume (#946)
* fix basic resume mechanics

* fix dependency resume

* fix deploy page sort

* remove task list console log

* checkpoint restore between attempts

* add checkpoint requirement checks to docker provider

* switch to raw execa for long statements

* catch and report pre-connection index errors

* graceful worker shutdown after completion

* fix final completion

* use same task process for each attempt

* update retry catalog entry

* add immediate return task

---------

Co-authored-by: Eric Allam <eric@trigger.dev>
2024-03-18 12:21:40 +00:00
Eric Allam 52c364d691 v3: cli init command + CLI telemetry (#948)
* Adding telemetry to the deploy command

* Add ability to opt-out of telemetry in the CLI

* Remove log

* Unifying some telemetry stuff and adding it to the login command

* init command

* Remove extra trigger dirs

* Couple tweaks to the init CLI

* Don’t use npm init -y to create the package.json file
2024-03-18 10:12:11 +00:00
Eric Allam 0a33bf7206 v3: handle errors and customizing retrying (#943)
* Add the ability to handle errors at the task and project level

* Cancel in progress runs when disconnecting the dev CLI (in between attempts)

* unify the task executor across dev & prod
2024-03-14 14:38:13 +00:00
Eric Allam 478ce006cb v3: task run status and canceling runs (#941)
* WIP task run status, revamped resuming task dependencies

* Don’t select a span when toggling collapsing

* Cancel runs and attempts, in prod and dev
2024-03-13 15:51:43 +00:00
Eric Allam d7cc2c9d6c v3: task payload refactor and configurable default retries (#939)
* Make the payload parameter the first argument to the task functions

* Allow configuring default retry behavior, and disabling in dev
2024-03-11 20:30:29 +00:00
Eric Allam c385db63be v3: cli deploy command (#931)
* WIP proxy/deploy

* WIP, registry proxy in express working

* A couple of notes, preparing for indexing

* Move the changes to prod-worker into the new file

* Deploy command working with indexing and runs (docker provider only for now)

* Removed ts-expect-error directive

* Fixed build command
2024-03-08 10:49:42 +00:00
nicktrn 9b6f8f9238 v3: provider and coordinator updates (#927)
* add zod socket

* start using zod socket

* use zod socket for shared queue

* use shared provider shell

* update k8s provider task ops

* set otlp endpoint

* update docker actions and fix builds

* remove unused types

* update injected secret key env var name

* complete socket.io types to schemas migration

* update env example to new v3 key var

* fix coordinator build

* set task run label

* ensure attempt id is always set

* increase prod worker timeout

* set otlp endpoint for on runs

* pass otlp env var to prod worker

* auto-remove completed index jobs
2024-03-06 16:01:59 +00:00
Matt Aitken 2a1273b683 Major v3 web app update (#924)
* Added the new colours to the Tailwind config

* Fixed Tsailwind compile errors

* WIP Geist working in the app

* WIP Storybook

* Storybook is working again… ouch

* Use Geist mono for CodeMirror

* Some find/replace class changes

* Resizable colours improved

* WIP on primary button

* Minor style improvements

* Improve the side menu header styling

* Improved v2 breadcrumbs

* Account side menu tweaks

* The page now works with the NavBar and UpgradePrompt

* Removed unused import

* Refactored the buttons to be dry-er

* Removed the breadcrumbs

* WIP on nav bar

* WIP updating the button styles

* Fix for the old test page

* Fix for type errors

* Removed console.log

* Colour change in the resizable handle

* New loading bar divider

* Removed the design plugin

* Creating our own storybook pages

* Added some “storybook” pages

* Added DetailCell storybook

* WIP new animation for the loading page

* Added the new loading divider

* Fix for the time frame filter having a green border

* Style improvements to the new test page

* Small style improvements

* Improvements to the list pagination components

* Nav bar back button improved

* Improvements to the run page

* Lots of v2 style improvements

* Help looks better

* Improvements to the regenerate API key modal

* More general improvements

* Added section titles to our storybook

* Added form stories

* Fix for missing key prop

* Loading bar is working

* Fixed button submitting state

* Fix for the test button icon

* Code bg improved

* Improved oauth buttons

* Integration page header fix

* Various improvements, mainly changing button variants

* Segmented control style updates

* Added an outline of 3px for the segment control

* Fixes for the billing pages

* More pricing improvements

* Added dialog to storybook

* Typography now in storybook

* Free usage bar added to storybook

* Added inline clode to storybook

* Separated menu types into a sub section in storybook

* NamedIcon now in storybook

* Added page header to storybook

* Added pricing callout to storybook

* Added radio groups to storybook

* Added segmented controllers to storybook

* Added shortcuts to storybook

* Added switches to storybook

* Added toast to storybook

* Added tooltips to storybook

* Added tree view to storybook

* Added usage charts to storybook

* Tarted up the environment variables page

* Fix for the FreePlanUsage panel bg colours

* Added new badge variants

* Added the <TaskPath /> to the tasks table

* Span header improvements

* Started work on the timeline

* Started work on the timeline

* Fix for the run page header

* Timeline tick marks are working

* Timeline working

* Got the durations showing on hover

* Button improvements

* Experimenting with how the duration should be shown

* Lots of refinements to the timeline

* Much nicer API for the Timeline

* The switch is now purple to match the design, with inverted thumb when on

* Removed old stuff

* Export the props

* Added the span with duration back in

* Simplified more and fixed the span durations

* Put in some decent dummy data

* Rendering defaults

* Wip with the mouse cursor

* Improvements to the story

* Added Slider primitive

* Integrated the icons into the slider

* Lots of layout improvements

* Cleaned up imports

* “Root” indicator

* Animated background on partials

* Pull the styles through to the spans and log dots

* Fixed some bugs in the TreeView when nodes get created or destroyed

* Fix for incorrect parentTask reference

* Added run streaming endpoint for live refreshing

* Added the tabs to Storybook

* Fix for status font size mismatch

* Fix for the misaligned run filter status

* Set a minimum duration on the timeline

* The TreeView now re-calculates if the number of nodes has changed

* Partials now render to the end

* Improved the durations gutter

* The duration is calculated using the current time if there are partial events

* Added the environment label to storybook

* Improved the environment label style

* Removed bottom margin from badge

* Button style improvements

* Callouts style fits the new design better

* Got the scrollbars to appear on the tree/timeline

* Added scroll support

* Synced scrolling working

* Close button and exit icon for the detail panel

* secure field shows start of env

* Clipboard improvements

* Latest lockfile

* Fix for the durations wrapping

---------

Co-authored-by: James Ritchie <james@jamesritchie.co.uk>
2024-03-04 17:33:53 +00:00
Eric Allam b7845685eb v3: no bundle builds for the dev CLI (#923)
* WIP no bundling

* Convert dev CLI to use an unbundled build process to support otel instrumentation

* Fix pnpm lock file

* A couple of fixes to get typechecking to work
2024-03-04 15:36:09 +00:00
nicktrn 52c9d485f8 v3: cli build command, prod runs, checkpoints (#919)
* fix trailing slash in api url

* add maybe platform down error

* shorten config path

* extend protobuf compiler install instructions

* build command and image model

* new image triggers task indexing

* index support for prod builds

* shorten example provider secret

* multi-stage task prod build

* lock prod tasks to node 18 image

* extract shared build and dev command libs

* pull out more shared deps

* add coordinator and providers

* fix core-apps build

* add dev builds for new apps

* fix cwd

* enable corepack

* some build fixes

* enable buildkit for old docker versions

* coordinator image fixes

* update provider containerfile

* build dev images in parallel

* upgrade pgadmin

* fix prod facade build

* prod runs

* fix merge

* don't knock out platform on invalid attempt id

* fix prod facade

* rename to build.ts

* fix prod builds

* set to executing after fetching payload

* make prod worker listen on random port if in use

* remove experimental warnings in dev

* prod resume

* prevent execution after completion

* exit prod worker after completion

* always restart otel collector

* docker checkpoints and prod runtime messaging

* don't retry indexing without chance of success

* make platform checkpoint aware

* deploy with existing hash sets latest worker

* log restore requests

* only try to checkpoint long waits

* tidying up

* lockfile

* fix build

* prod worker merge fixes

* fix prod complete and cancel

* fix lua nil checks

* socket namespace abstraction

* cleanup

* make all build args optional

* add build script

* don't require env vars for dev

* fix schema

* prod merge

* small fix

* bind correct logger

* fix v3 ref catalog entry

* resume prod batch

* pass socket to error and disconnect handlers

* fix non-batch resume

* fix batch resume

* send connection env vars when not in dev

* create worker via socket

* move api client back into v3 cli

* fix lockfile

* fix resume with failures

* marqs replace message

* typecheck prior to build

* don't define api url in prod builds

* support prod retries after resume

* skip typecheck option
2024-03-04 13:10:07 +00:00
Eric Allam ca47e6bdcd v3: cli cancel improvements and add back in deprecating background workers (#918)
* Show runs as cancelled when killing the dev CLI

* Fixed error when waiting for a task

* Deprecated workers will no longer be able to run unlocked task runs

* Fixed selecting the latest background worker version in test list

* Don’t propogate trace context when triggering an async task

Also add batch id to the context and the task events
2024-02-29 15:34:00 +00:00
Eric Allam 1fabcf68fa Override local env vars with env vars in the server (#914) 2024-02-27 15:44:55 +00:00
Eric Allam 8d6e710dc2 V3: retry utilities (#913)
* WIP

* retry.fetch w/intercepting to test

* Fixed the incorrect renaming of limit to maxAttempts in v2

* Fixed source map support in dev

* Cleanup background worker files once they close
2024-02-27 13:54:51 +00:00
Eric Allam 90a302f9f7 v3: batch trigger (#908)
* Support for batch triggers

* Various trace fixes and improvements

* Moved trace into the logger, fixed span cardinality issues with waits and triggers

* Add support for attaching a debugger in dev with —debugger flag
2024-02-26 11:21:06 +00:00
Eric Allam dd63fe6e9f v3: Implemented in-process task retries in dev w/cache support (#905)
* Implemented in-process task retries in dev w/cache support

* Improve the stack trace correction
2024-02-22 10:12:23 +00:00
Matt Aitken 584936fa29 Early v3 project UI (#904)
* Added ProjectVersion to projects, defaults to V2

* Moved v3Enabled to the existing feature flags

* Update to using the v3Enabled feature flag

* Allow people to choose v3 when creating a new project

* Added version to the ProjectPresenter

* v2 project and v3 project redirecting

* .env.example for the v3-catalog

* Added sdkVersion and cliVersion columns to the BackgroundWorker table

* Added additional classes for environments

* First draft of the Tasks table

* Link to the task page

* V3 side menu project items

* Moved the ListPagination component into the components folder

* Moved the TaskListPresenter to a v3 folder

* Bare bones task page with tabs

* Task page, we’re going to delete this though

* Bare bones runs table working

* Rejigged the columns

* The run table status

* Environment and status filtering working

* Added time filters

* Cursor and direction is working… I think it’s tricky to know for sure

* Added TaskRun numbers

* Selecting a task now links to the runs page with filters turned on

* Added support for the enqueued status

* Reworked the task table

* Link to the task and environment runs

* The run page is rendering a tree of the events

* Change the task table column to “Created at"

* WIP on making the run tree view look good

* Improvements to the run tree

* Removed the janky scroll bar flash

* Change the title of the task logs to just the task ID

* Live timer when a span is running

* Added “Show parent items” link

* Fixed the bug jumping to parent items

* Style improvements to the run

* Added the resizable handle to the run page

* Moved formatDuration to core/v3

* Wait for now has nice log messages

* The run detail panel is working with just a title for now

* Navigating to a span and persisting between page reloads is working

* The run detail timeline

* Wrap log dates in a paragraph

* Latest span detail view

* Scrolling in the right-hand panel

* Tweak default resizable layout for the run page

* Attempted improvements to stop a recursive issue on TreeView

* TreeView useReducer WIP

* Changed the change callback

* Add changes to the state

* Use the new onSelectedIdChanged in the app

* Don’t update the state from the outside

* Filtering fix, although navigation doesn’t work nicely

* Filtering working when changing the content and clearing it

* Don’t allow the same selectedId callback to be called twice…

* useDebounce hook

* Switch from using defer to just typedjson for now

* Latest attempt at span navigation

* Removed log

* Turn off filtering for now, it’s causing the Links to break somehow

* Fix for page height issue

* Added (Developer Preview) to the v3 project select box

* Revert "Filtering working when changing the content and clearing it"

This reverts commit 20d9fbe36ded619e9d3eb0b23d4ed0568a88a615.

# Conflicts:
#	apps/webapp/app/components/primitives/TreeView/TreeView.tsx

* Switch to using the old filtering

* Added useThrottle hook

* Instantly close the panel when deselecting a node, use debounce when navigating to a span

* Better spacing in the right hand panel

* Comment out the v3 side menu pages that don’t exist yet

* Fix for typecheck fail

* Clear the statuses too
2024-02-21 18:42:12 +00:00
Eric Allam 07f7c65054 v3 queue system (#903)
* Introducing Modular Asynchronous Reliable Queueing System (MarQS). Works in dev

* Convert MarQS to using lua and dealing with concurrency

* Simplified the timeout queue and current concurrency is now a set instead of a flat value (to support idempotency)

* Implement task heartbeating and reconnect the background workers CLI when the websocket connection reconnects

* Start adding internal telemetry support for the server

* Get env vars to work in dev and implement prisma tracing in webapp

* Cleanup telemetry and implement it in the consumer

* Implement dequeuing a message from a parent shared queue

* Implement a custom logger exporter instead of using console log exporter

* Use node instead of shell for generating protocol buffer code

* Propogate trace context into debug logs, and allow turning off logger exporter through env vars

* Switch to using baselime for internal otel data

* Make orgMember optional to fix type issues

* Provide the CLI dev env vars through the CLI, don’t build dotenv into facade

* Removed the logger import

* Address Matt’s comments

* Addressing more of Matt’s comments

* Handle sending an execution after a websocket connection closes

* Remove auth from the env attributes to prevent obfuscation
2024-02-20 12:16:41 +00:00
Eric Allam 2aabb994cf Install the protobuf compiler in the docker builder stage 2024-02-12 16:58:57 +00:00
Eric Allam baf3a84cda v3 dev cli command + more (#894) 2024-02-12 14:24:32 +00:00
Matt Aitken dd879c8e4a Fix api run statuses (#874)
* Added a subtask for testing

* Make it easier to run the CLI from the nextjs reference project

* Copies of the run and statuses endpoints, but without simplifying the run statuses

* Use the new v2 endpoints that give the full run statuses

* Removed unused import

* v2 events endpoint with the full run status info

* Changeset
2024-01-29 11:42:19 +00:00
Eric Allam 6c4047cf21 Fix an issue where runs were stuck executing when a child task failed and the parent task retried 2024-01-17 14:39:06 +00:00
Kritik Jiyaviya 4dd6cf18dc feat: Add support for the new Resend audience API endpoints (#850)
* feat: Add support for the new Resend audience API endpoints

* update pnpm-lock.yaml

* update pnpm-lock.yaml

* Merge branch 'main' into feat/support-for-resend-audience-api

* latest pnpm-lock.yaml

* Create lovely-items-end.md

* Update pnpm-lock.yaml

---------

Co-authored-by: Matt Aitken <matt@mattaitken.com>
2024-01-17 14:06:45 +00:00
Matt Aitken 1bbd7e6dc3 V3 CLI started, with Personal Access Tokens and Who Am I (#841)
* Initial CLI commit

* Hooked up the CLI so it actually shows stuff…

* Totally reworked the v3 CLI to be based on our existing CLI (Commander)

* WIP on PersonalAccessTokens and AuthorizationCodes

* WIP creating a Personal Access Tokens page. Created a new sidenav for account pages

* Creating tokens is working but the form is broken

* Tokens are created in the UI

* Creating and revoking access tokens from the UI is working

* Improved the create form and copy

* Tokens are a bit shorter and only lowercase

* API endpoint for creating AuthorizationCodes and the web page users hit to create PATs from them

* V3_ENABLED env var and hook that can be used in the UI to show/hide things

* API route to get a PAT (within 10 mins of creating an auth code). Moved some code to core

* Start to build the login command

* Nicer banner when starting the CLI

* Nicer update checking

* Update command style improved

* Removed the template step

* Login command options are now working

* The new CLI is logging in using the Personal Access TOken. But I need to save it still

* Logging in and saving the token is working

* Deal with already being logged in

* Who Am I working with PAT

* Deleted old account side menu header

* Improved the copy on the Auth code page

* Added docs for the login and whoami commands

* Added readme instructions for the update command

* Removed some unused things

* Remove fetchUseProxy for now
2024-01-15 14:45:41 +00:00
Matt Aitken adf497bc3e Example in the job-catalog of canceling all runs for a Job 2024-01-03 10:31:56 +00:00
Eric Allam 6a3c563f14 Fixed Job.attachToClient 2023-12-08 10:15:06 +00:00
Eric Allam 17f6f29d05 Feature: Support multiple runtimes other than Node.js (#774)
🚀 Publish Trigger.dev Docker / e2e (push) Failing after 0s
🚀 Publish Trigger.dev Docker / units (push) Failing after 5s
🚀 Publish Trigger.dev Docker / typecheck (push) Failing after 5s
🚀 Publish Trigger.dev Docker / publish (push) Has been skipped
2023-12-06 10:23:37 +00:00
Matt Aitken 68cbfd8d23 Log out the run status 2023-11-30 10:29:48 +00:00
Eric Allam ca05d5f603 @trigger.dev/openai: Adding additional assistant tasks 2023-11-28 22:48:31 +00:00
nicktrn 1dcd87a2aa URI encode keys (#758) 2023-11-28 18:05:15 +00:00
nicktrn 096151c014 Fix: Shopify subpath imports, enhance docs (#755)
🚀 Publish Trigger.dev Docker / typecheck (push) Failing after 0s
🚀 Publish Trigger.dev Docker / units (push) Failing after 0s
🚀 Publish Trigger.dev Docker / e2e (push) Failing after 0s
🚀 Publish Trigger.dev Docker / publish (push) Has been skipped
* Fix invalid subpath imports

* Remove @trigger.dev/tsup dep

* Update lockfile

* Small github docs fix

* Pull runtime adaptor import out of integration

* Fix io.store docs examples

* Add sendEvents to task library

* Add KV to task library and small fixes

* Suppress duplicate http endpoint warnings

* Changeset

* Add task returns
2023-11-27 16:03:03 +00:00
nicktrn 067e19fec9 Feature: New Webhook Triggers, KV Store, and Shopify integration (#745)
* Attach webhook to http endpoint

* Create internal webhook handler job

* Fix delivery

* Add webhook trigger

* Index webhooks

* Fix webhook connect

* Webhook activation

* Rename to TriggerWebhook

* Save webhooks to index

* Extend register payload

* Revert attachSource

* Add update webhook helper

* Use crud interface for registration

* Remove filter from params

* Destructure options

* Missing ohash dep

* Chainable trigger filter

* Optional request verification

* Handler call with context

* Update config on register success

* Rename to webhookData

* Add basic key-value store

* Improve kv store

* Webhook trigger UI

* Webhook delivery UI

* Some fixes

* Airtable experiments

* Fix tabs transitions

* Only run register if config differs

* Use subtasks for delivery

* Missing dep

* Error handing and fixes

* Namespace cursor

* Fix hmac verification

* Gut Airtable

* Lockfile

* Template optional api key

* Template event sources and icons

* Template remove filter from params

* Fix models template

* Template fixme

* Fix webhook trigger header

* Actually send params for registration..

* Add back oauthed client

* Verify signature header encoding param

* Fix delivery context

* Webhook create retry with prior delete

* Webhook source type fixes

* Webhook config merge

* Update webhook icon

* Verify webhook task options

* Rename handler to generateEvents

* Send event icons

* Shopify with webhooks

* Shopify job-catalog entry

* Lockfile

* Add webhook migration

* Better icons

* Type fixes

* Make param type optional

* Scope type

* More examples

* More schemas

* Trigger catalog

* Bump versions

* Just a few more events they said

* Simplify session and config

* Typed serializer

* Fix payload type

* Remove unused getters

* Reduce logging output

* Rest resource tasks

* Lockfile

* Bump version

* Remaining triggers

* Fix event types

* Fix import

* Link from webhook trigger to http endpoint

* Use rest resource tasks for crud

* Fix rest save type

* Improve KV types

* Improve attach webhook logging

* Make header schema more lenient

* Fix save type again

* Final error callback

* Shorten keys

* Remove ohash diff

* Disable Airtable webhooks

* Webhook environments

* Fix shopify error import

* Link to integration and http endpoint

* Integration catalog entry

* Disable oauth

* Require api key, simplify client secret

* Move trigger types

* Update catalog example

* Small KV endpoint refactor

* Fix custom migration

* Webhook environment migration

* Improve crud types

* Remove unused schema

* Fix crud context type

* KV limits and HTTP verbs

* KV improvements

* Jobless webhook delivery

* Airtable delivery fixes

* Make deliveries sexy again

* Some docs

* Fix payload types.. again

* Final payload pass

* Schema cleanup

* Refactor handler service

* Remove more stale schemas

* registerJobNamespace import

* Expose KV on TriggerClient

* Remove dummy click handler

* Remove unused components

* Fix delivery pagination

* Enable registration rerun button

* Comment out registration route action handler

* Rename KV as not tied to IO anymore

* Swap api with trigger client

* Disabled fields param

* KV docs

* Changeset

* Shopify docs fixes
2023-11-24 10:22:12 +00:00
Eric Allam 131d0dbea4 Adding an example of using OpenAI GPT 4 Vision 2023-11-21 10:23:11 +00:00
Eric Allam 0ab51d62bc Don’t send notifications for internal runs and add isTest to the notification
🚀 Publish Trigger.dev Docker / publish (push) Has been skipped
🚀 Publish Trigger.dev Docker / e2e (push) Failing after 19s
🚀 Publish Trigger.dev Docker / units (push) Failing after 3s
🚀 Publish Trigger.dev Docker / typecheck (push) Failing after 6s
2023-11-16 14:45:39 +00:00
Eric Allam 756024da78 Add support for Run Notifications #401 (#738)
* Add support for Run Notifications #401

* Fixed typo and added changeset
2023-11-16 14:19:42 +00:00
Eric Allam 14429b8b66 Fix “transaction already closed” error when running tasks (#735) 2023-11-14 23:22:34 +00:00
Eric Allam 75550f535c Fix undefined task outputs
🚀 Publish Trigger.dev Docker / typecheck (push) Failing after 5s
🚀 Publish Trigger.dev Docker / e2e (push) Failing after 0s
🚀 Publish Trigger.dev Docker / units (push) Failing after 19s
🚀 Publish Trigger.dev Docker / publish (push) Has been skipped
And also fix the SSE connection getting reset on a infinite loop when runs are complete
2023-11-13 16:18:41 +00:00
Eric Allam cb1825bfaf Add OpenAI support for 4.16.0 (#726)
* Add OpenAI support for 4.16.0

* Add support for background polling and use that in OpenAI integration to power assistants

* Much improved OpenAI docs

* Added backgroundPoll docs

* Implements waitForEvent and added docs for more built in tasks

* Add sendEvent API referenc

* Write the task libray

* Add changeset and warning for waitForEvent
2023-11-09 16:59:58 +00:00
nicktrn d02173442c Feature: io.sendEvents() (#728)
* Add io.sendEvents

* Add examples to built-ins catalog entry

* Add docs

* Add changeset

---------

Co-authored-by: Matt Aitken <matt@mattaitken.com>
2023-11-09 16:42:23 +00:00
Eric Allam a74716a1b9 Added waitForRequest built-in tasks (#725)
* Added waitForRequest built-in tasks

* Create grumpy-buttons-thank.md
2023-11-06 18:30:49 +00:00
Eric Allam 42185bc36b Upgrade resend integration to use resend-node 2.0.0 (#724)
* Upgrade resend integration to use resend-node 2.0.0, fixes issue with Next.js 14

* Create mighty-dodos-study.md
2023-11-06 18:30:37 +00:00
Eric Allam 329944841f Fix the cancel event example in job ref 2023-11-06 17:38:18 +00:00
Matt Aitken 9c6838e5c8 HTTP endpoints, HTTP triggers and new sidebar (#720)
* Setup basic storybook page

* storybook preview pages don’t have padding

* Added a storybook story for the side menu

* WIP on a new side menu component

* New variant for the side menu button style

* Side menu items can now recieve a badge ‘count’

* Storybook side menu update

* Added a new font size

* you can now set the spacing between an icon and the text

* Moving the error and count badge to the right side of the button

* Added a menu sub item button style

* added sub items to the menu and all the rest of the items

* Proper background class used

* Added a side menu header and main org header

* Removed the no-mobile blocker overlay so you can use the site on a phone

* Better section name

* Created a cusom popover trigger that can take anything as children. Also the popover titles can now take a variant

* Added a popover for the user avatar menu item

* Removed the tool tip

* Arrow Popover can no also go full width

* neater conditiona;

* Removed unnessary div. And error icon can display an error message

* menu item buttons can take a variant

* Added placeholder items for the org dropdown

* add a max height to the org menu popover and handle scrollbars

* Added a border if the menu scrolls when you have lots of project items

* warning badges display tooltips if you give them some error text

* Removed the sticky header so the scrollbar looks better

* Fixed a layout issue from removing the sticky header

* Better spacing between lists

* Popover can take any react component icon and an icon classname

* Side menu headers now have a dropdown by clicking the ellipsis icon

* Side menu items can take an icon colour

* Updated styling for the dropdown Popover menus

* Installed Tabler icons

* Updated some icons to use Tabler

* Started defining the HttpTrigger class

* RequestFilterSchema is part of core

* Use the RequestFilterSchema from core

* EventSpecificationExample now derived from schema

* Need to define the class like EventTrigger

* Catalog reference that uses the httpTrigger

* The logic for the httpTrigger is in the separate file, just call it from the client using defineHttpTrigger

* Basic HttpTrigger

* Import fixes

* Transforming httpTrigger to the format we want

* Initial httpTrigger types working

* Added HTTP generic type

* Added some notes

* Rejigged some things to support verifying payloads

* Use new security function for verifying

* Added truncating text support if side menu names get long

* Renamed some stuff

* Removed test copy

* HttpEndpoin that isn’t a trigger

* Renamed hostname to source

* Created HttpEndpoint as a top-level concept, it produces triggers using onRequest

* Started working on registration of the httpEndpoints

* Removed the new TriggerMetadata for now, http will use static

* Added PayloadType to EventRecord. Default is JSON.

* The payload from an HTTPEndpoint is now a Request

* Initial TriggerHttpEndpoint migrations

* Initial server side work

* Migration to add shortcode to RuntimeEnvironment

* Added TriggerHttpEndpointEnvironment table

* Made the httptrigger catalog example runnable

* Progress creating/updating httpEndpoints

* HTTPEndpointEnvironment updating now works

* Added a note about EventRecord HttpEndpoint relationship

* When creating environments, create a shortcode

* Fix for secret not being retrieved correctly

* Request filtering

* Add DOM types to Core

* Early work dealing with a webhook arriving

* Added some todo notes to discuss tomorrow

* Added relations to the endpoint so we can call the user’s server

* Basic handling of immediate responses

* The immediate responses are now going back to the webhook provider

* Handling is now generating events and responding where relevant. Not yet matching filters and doing verification

* Added source to TriggerHttpEndpointEnvironment

* The event filter is now optional

* The source is passed through from the httpEndpointEnv

* Added optional payloadType when ingesting an event

* Added tabler

* Updated @remix-run/testing usage for remix 2

* Fixed org menu header width and added org and project types

* Added custom icons for Runs and Endpoints

* Updated the menu icons

* storybook side menu shows static project and org data

* Updated side menu

* Plumbed in paths

* Updated eventFilter to deal with undefined/null payloads

* Started work on filtering, but it’s not working

* Strict null

* async filtering it working

* Added environment shortcodes to the seedCloud script

* Don’t pass the secret back, added a verifyPayload function that gets called before runs start

* Fix for skipping triggering runs

* The verify callback is now passed through to the trigger so it can be called

* If verifying fails, have a reason which we can show to the user

* Don’t verify if it’s a test run

* Fixed the log in the example code

* Added loader and action to storybook preview

* Fix for svg attribute name issues in EndpointIcon

* Started work on the side menu

* This logic has moved to a separate file, in a different branch

* Don’t get the environments when getting orgs

* Lots of fixes for the new SideMenu

* Refactored some org/project things to fit the new structure better

* Some progress restructuring the app for the new sidebar

* More changes to data loading

* Deleted some old pages. The redirecting is working if you’re on the root but logged in

* Basic styling for the account page. We need a back button

* Fixes for the integrations pages

* Breadcrumb implemented

* Add a border to the bottom of the breadcrumb

* Feedback is working, not in Storybook

* Removed expandSidebar from the Handle

* Org level breadcrumbs

* Getting the project in the org route now works properly

* orgslug route before using the presenter

* More UI improvements

* Removed commented out line

* Use presenters for org and root

* Removed forceActive

* Split the project selector and user menu out

* The side menu items have correct active states

* Fixed the scrolling

* Put the org name in

* Improved the admin features

* Created the HTTP Endpoints list page

* Work on the HTTP endpoint page

* WIP on the HTTP Endpoint page

* Fix for response filter not showing

* Change page URLs to “http-endpoints” (from httpendpoints)

* Slugify the HTTP endpoint id

* Added todos for EventRecord associations

* Removed the Job/Trigger tab for now

* First draft of the help panels

* Rejigged the HTTP endpoint page so it splits out the deployed and not deployed environments

* Removed irrelevant columns

* Added a link to the deployment guide

* Removed the background image from the frameworks pages

Also…
- reordered the frameworks so supported ones come first in the list
- added logos to 3 framework pages that were missing them
- removed lots of un-used imports

* Fixed an issue where the breadcrumb could be hidden on small screens so broken the pages

The breadcrumb is always visible now

* Renamed ‘Onboarding’ to ‘Framework setup’

* Added a new http-endpoint icon to the side menu

* Side menu now has a nice looking scrollbar

* Scrollbar on the usage and billing page updated to the nice style

* Icon and colour updates to the side bar menu items

* Small copy improvement

* Fixed bug with the height of the org menu dropdown

* Page breadcrumbs now don’t include duplicate path links

* renamed uiBorder to ui-border

* Remove the padding from the small menu item if it’s used as a button

* Made the paragraph text a bit smaller

* EventRecord has relationships with TriggerHttpEndpoint/TriggerHttpEndpointEnvironment

* Improved the presenter used by the side menu. Also added the HTTP Endpoint badge count to the sidebar

* The dropdown org menu has the same badge as the side menu

* Improved the Projects page

* Fixed button style miss-match in the jobs table dropdown

* Made the popover dialogs support Headers and improved the regenerate API key popover

* Made the active status of a job more subtle

* Allow the Job to have a link on its Trigger

* Simplified loader because we ditched the Jobs menu. Also showing no runs help panel now

* Fixed type errors

* Pages that are org level can scroll if the content gets too long

* Fixed a redirect bug so you now redirect to the org page

* Updated the shortcode migration, to add additional letters to the end

* Removed the nav bar from the welcome onboarding page

* Only show the countBadge if the number is > 0

* Jobs icon now has a default colour the same as the nav menu

* Capital I for integration

* Clone the request before calling verify

* Separate the checks out so they can be easily logged

* Make the failure reason optional

* Made the verification error look better, if it’s provided

* Working WhatsApp example

* Can set the immediateResponseFilter to null in an update

* Deal with a null immediateResponseFilter

* Cal.com webhook working

* Header height reduced slightly

* Fixed button padding

* Removed un-used imports

* http endpoint table displays a message if you have no entries in the table

* Removed some duplicate text and swapped primary buttons for secondary style

* All ‘HTTP Endpoints’ now ‘HTTP endpoints’

* You have to use a RequestFilter when using HTTP Trigger

* Synchronous eventDispatcher filtering again. Better content-type check

* Stripe example

* Side menu “Usage & Billing” is just “Usage” if self-hosting

* Fix for filtering not working

* Removed filter log

* Fix for jobs being optional and null

* Removed old nav bar from the invite page

* Deleted old nav bar component

* import cleanup

* NoTask message only displays if the run has status “SUCCESS”

* Added a global page loading indicator

* Added documentation buttons to the most important pages

* Added GitHub

* Use “verifyRequestSignature” helpr

* Removed todo

* Fixed the layout of the Rerun Job popover

* Jobs table has margin bottom so it sits off the bottom of the page

* Small improvement to the search bar layout on the integration page

* HTTP endpoint and Trigger concept docs

* Reference docs and JSDocs for HTTP endpoint

* SDK reference docs for HttpEndpoint

* Added an HttpTrigger SDK reference, which mainly just point you elsewhere

* Improved the copy in the help panels

* Removed unused code

* Improved HTTP endpoint comment about setting the secret

* Delete unused asyncFilter

* Fixed the delete modal not having a title

* Always try get the arrayBuffer for an HTTP request, catch and use undefined as the default

* Add extra method to invokeTrigger

* Use a transaction for the secret store

* verifyRequestSignature docs page and JSDocs

* Added verifyRequestSignature docs page to nav

* Warn if they use the same HttpEndpoint id twice

* Added another job using the same HttpEndpoint

* Renamed the api route to http-endpoints from httpendpoints

* Debug bar for VSCode is now in the commandCenter

---------

Co-authored-by: James Ritchie <james@jamesritchie.co.uk>
2023-11-06 15:32:40 +00:00
nicktrn cf8f994604 Feature: io.random() (#716)
* Add io.random

* Add changeset

---------

Co-authored-by: Eric Allam <eric@trigger.dev>
2023-11-03 11:07:12 +00:00
Eric Allam 620b83832b Implement manually invokable jobs through the invokeTrigger (#700)
* Implement manually invokable jobs through the invokeTrigger

Also implemented a job run notification system, that will POST details of a run on completion. This combines with the task callbackUrl system to implement the invokeAndWait

* Document the invoke trigger

* batch invoke and wait

* background fetch timeouts

* Use @whatwg-node/fetch instead of the polyfilled fetch

* Fix some outdated dependencies in webapp

* Improved subtask error propogation messages

* Document the OpenAI changes and the batch invoke stuff

* Fix dequeuing jobs

* Don’t retry the OpenAI completion background task

* Added OpenAI changesets

* Use the new ResumeTaskService in ProcessCallbackTimeout as well
2023-11-03 11:05:00 +00:00
nicktrn 9c5a7cf073 Fix: Allow errors and special values on Airtable field set (#697)
* Allow errors and special values on field set

* Add changeset
2023-10-31 10:17:05 +00:00