Commit Graph

117 Commits

Author SHA1 Message Date
Matt Aitken 2a1273b683 Major v3 web app update (#924)
* Added the new colours to the Tailwind config

* Fixed Tsailwind compile errors

* WIP Geist working in the app

* WIP Storybook

* Storybook is working again… ouch

* Use Geist mono for CodeMirror

* Some find/replace class changes

* Resizable colours improved

* WIP on primary button

* Minor style improvements

* Improve the side menu header styling

* Improved v2 breadcrumbs

* Account side menu tweaks

* The page now works with the NavBar and UpgradePrompt

* Removed unused import

* Refactored the buttons to be dry-er

* Removed the breadcrumbs

* WIP on nav bar

* WIP updating the button styles

* Fix for the old test page

* Fix for type errors

* Removed console.log

* Colour change in the resizable handle

* New loading bar divider

* Removed the design plugin

* Creating our own storybook pages

* Added some “storybook” pages

* Added DetailCell storybook

* WIP new animation for the loading page

* Added the new loading divider

* Fix for the time frame filter having a green border

* Style improvements to the new test page

* Small style improvements

* Improvements to the list pagination components

* Nav bar back button improved

* Improvements to the run page

* Lots of v2 style improvements

* Help looks better

* Improvements to the regenerate API key modal

* More general improvements

* Added section titles to our storybook

* Added form stories

* Fix for missing key prop

* Loading bar is working

* Fixed button submitting state

* Fix for the test button icon

* Code bg improved

* Improved oauth buttons

* Integration page header fix

* Various improvements, mainly changing button variants

* Segmented control style updates

* Added an outline of 3px for the segment control

* Fixes for the billing pages

* More pricing improvements

* Added dialog to storybook

* Typography now in storybook

* Free usage bar added to storybook

* Added inline clode to storybook

* Separated menu types into a sub section in storybook

* NamedIcon now in storybook

* Added page header to storybook

* Added pricing callout to storybook

* Added radio groups to storybook

* Added segmented controllers to storybook

* Added shortcuts to storybook

* Added switches to storybook

* Added toast to storybook

* Added tooltips to storybook

* Added tree view to storybook

* Added usage charts to storybook

* Tarted up the environment variables page

* Fix for the FreePlanUsage panel bg colours

* Added new badge variants

* Added the <TaskPath /> to the tasks table

* Span header improvements

* Started work on the timeline

* Started work on the timeline

* Fix for the run page header

* Timeline tick marks are working

* Timeline working

* Got the durations showing on hover

* Button improvements

* Experimenting with how the duration should be shown

* Lots of refinements to the timeline

* Much nicer API for the Timeline

* The switch is now purple to match the design, with inverted thumb when on

* Removed old stuff

* Export the props

* Added the span with duration back in

* Simplified more and fixed the span durations

* Put in some decent dummy data

* Rendering defaults

* Wip with the mouse cursor

* Improvements to the story

* Added Slider primitive

* Integrated the icons into the slider

* Lots of layout improvements

* Cleaned up imports

* “Root” indicator

* Animated background on partials

* Pull the styles through to the spans and log dots

* Fixed some bugs in the TreeView when nodes get created or destroyed

* Fix for incorrect parentTask reference

* Added run streaming endpoint for live refreshing

* Added the tabs to Storybook

* Fix for status font size mismatch

* Fix for the misaligned run filter status

* Set a minimum duration on the timeline

* The TreeView now re-calculates if the number of nodes has changed

* Partials now render to the end

* Improved the durations gutter

* The duration is calculated using the current time if there are partial events

* Added the environment label to storybook

* Improved the environment label style

* Removed bottom margin from badge

* Button style improvements

* Callouts style fits the new design better

* Got the scrollbars to appear on the tree/timeline

* Added scroll support

* Synced scrolling working

* Close button and exit icon for the detail panel

* secure field shows start of env

* Clipboard improvements

* Latest lockfile

* Fix for the durations wrapping

---------

Co-authored-by: James Ritchie <james@jamesritchie.co.uk>
2024-03-04 17:33:53 +00:00
Eric Allam b7845685eb v3: no bundle builds for the dev CLI (#923)
* WIP no bundling

* Convert dev CLI to use an unbundled build process to support otel instrumentation

* Fix pnpm lock file

* A couple of fixes to get typechecking to work
2024-03-04 15:36:09 +00:00
nicktrn 52c9d485f8 v3: cli build command, prod runs, checkpoints (#919)
* fix trailing slash in api url

* add maybe platform down error

* shorten config path

* extend protobuf compiler install instructions

* build command and image model

* new image triggers task indexing

* index support for prod builds

* shorten example provider secret

* multi-stage task prod build

* lock prod tasks to node 18 image

* extract shared build and dev command libs

* pull out more shared deps

* add coordinator and providers

* fix core-apps build

* add dev builds for new apps

* fix cwd

* enable corepack

* some build fixes

* enable buildkit for old docker versions

* coordinator image fixes

* update provider containerfile

* build dev images in parallel

* upgrade pgadmin

* fix prod facade build

* prod runs

* fix merge

* don't knock out platform on invalid attempt id

* fix prod facade

* rename to build.ts

* fix prod builds

* set to executing after fetching payload

* make prod worker listen on random port if in use

* remove experimental warnings in dev

* prod resume

* prevent execution after completion

* exit prod worker after completion

* always restart otel collector

* docker checkpoints and prod runtime messaging

* don't retry indexing without chance of success

* make platform checkpoint aware

* deploy with existing hash sets latest worker

* log restore requests

* only try to checkpoint long waits

* tidying up

* lockfile

* fix build

* prod worker merge fixes

* fix prod complete and cancel

* fix lua nil checks

* socket namespace abstraction

* cleanup

* make all build args optional

* add build script

* don't require env vars for dev

* fix schema

* prod merge

* small fix

* bind correct logger

* fix v3 ref catalog entry

* resume prod batch

* pass socket to error and disconnect handlers

* fix non-batch resume

* fix batch resume

* send connection env vars when not in dev

* create worker via socket

* move api client back into v3 cli

* fix lockfile

* fix resume with failures

* marqs replace message

* typecheck prior to build

* don't define api url in prod builds

* support prod retries after resume

* skip typecheck option
2024-03-04 13:10:07 +00:00
Eric Allam ca47e6bdcd v3: cli cancel improvements and add back in deprecating background workers (#918)
* Show runs as cancelled when killing the dev CLI

* Fixed error when waiting for a task

* Deprecated workers will no longer be able to run unlocked task runs

* Fixed selecting the latest background worker version in test list

* Don’t propogate trace context when triggering an async task

Also add batch id to the context and the task events
2024-02-29 15:34:00 +00:00
Eric Allam 1fabcf68fa Override local env vars with env vars in the server (#914) 2024-02-27 15:44:55 +00:00
Eric Allam 8d6e710dc2 V3: retry utilities (#913)
* WIP

* retry.fetch w/intercepting to test

* Fixed the incorrect renaming of limit to maxAttempts in v2

* Fixed source map support in dev

* Cleanup background worker files once they close
2024-02-27 13:54:51 +00:00
Eric Allam 90a302f9f7 v3: batch trigger (#908)
* Support for batch triggers

* Various trace fixes and improvements

* Moved trace into the logger, fixed span cardinality issues with waits and triggers

* Add support for attaching a debugger in dev with —debugger flag
2024-02-26 11:21:06 +00:00
Eric Allam dd63fe6e9f v3: Implemented in-process task retries in dev w/cache support (#905)
* Implemented in-process task retries in dev w/cache support

* Improve the stack trace correction
2024-02-22 10:12:23 +00:00
Matt Aitken 584936fa29 Early v3 project UI (#904)
* Added ProjectVersion to projects, defaults to V2

* Moved v3Enabled to the existing feature flags

* Update to using the v3Enabled feature flag

* Allow people to choose v3 when creating a new project

* Added version to the ProjectPresenter

* v2 project and v3 project redirecting

* .env.example for the v3-catalog

* Added sdkVersion and cliVersion columns to the BackgroundWorker table

* Added additional classes for environments

* First draft of the Tasks table

* Link to the task page

* V3 side menu project items

* Moved the ListPagination component into the components folder

* Moved the TaskListPresenter to a v3 folder

* Bare bones task page with tabs

* Task page, we’re going to delete this though

* Bare bones runs table working

* Rejigged the columns

* The run table status

* Environment and status filtering working

* Added time filters

* Cursor and direction is working… I think it’s tricky to know for sure

* Added TaskRun numbers

* Selecting a task now links to the runs page with filters turned on

* Added support for the enqueued status

* Reworked the task table

* Link to the task and environment runs

* The run page is rendering a tree of the events

* Change the task table column to “Created at"

* WIP on making the run tree view look good

* Improvements to the run tree

* Removed the janky scroll bar flash

* Change the title of the task logs to just the task ID

* Live timer when a span is running

* Added “Show parent items” link

* Fixed the bug jumping to parent items

* Style improvements to the run

* Added the resizable handle to the run page

* Moved formatDuration to core/v3

* Wait for now has nice log messages

* The run detail panel is working with just a title for now

* Navigating to a span and persisting between page reloads is working

* The run detail timeline

* Wrap log dates in a paragraph

* Latest span detail view

* Scrolling in the right-hand panel

* Tweak default resizable layout for the run page

* Attempted improvements to stop a recursive issue on TreeView

* TreeView useReducer WIP

* Changed the change callback

* Add changes to the state

* Use the new onSelectedIdChanged in the app

* Don’t update the state from the outside

* Filtering fix, although navigation doesn’t work nicely

* Filtering working when changing the content and clearing it

* Don’t allow the same selectedId callback to be called twice…

* useDebounce hook

* Switch from using defer to just typedjson for now

* Latest attempt at span navigation

* Removed log

* Turn off filtering for now, it’s causing the Links to break somehow

* Fix for page height issue

* Added (Developer Preview) to the v3 project select box

* Revert "Filtering working when changing the content and clearing it"

This reverts commit 20d9fbe36ded619e9d3eb0b23d4ed0568a88a615.

# Conflicts:
#	apps/webapp/app/components/primitives/TreeView/TreeView.tsx

* Switch to using the old filtering

* Added useThrottle hook

* Instantly close the panel when deselecting a node, use debounce when navigating to a span

* Better spacing in the right hand panel

* Comment out the v3 side menu pages that don’t exist yet

* Fix for typecheck fail

* Clear the statuses too
2024-02-21 18:42:12 +00:00
Eric Allam 07f7c65054 v3 queue system (#903)
* Introducing Modular Asynchronous Reliable Queueing System (MarQS). Works in dev

* Convert MarQS to using lua and dealing with concurrency

* Simplified the timeout queue and current concurrency is now a set instead of a flat value (to support idempotency)

* Implement task heartbeating and reconnect the background workers CLI when the websocket connection reconnects

* Start adding internal telemetry support for the server

* Get env vars to work in dev and implement prisma tracing in webapp

* Cleanup telemetry and implement it in the consumer

* Implement dequeuing a message from a parent shared queue

* Implement a custom logger exporter instead of using console log exporter

* Use node instead of shell for generating protocol buffer code

* Propogate trace context into debug logs, and allow turning off logger exporter through env vars

* Switch to using baselime for internal otel data

* Make orgMember optional to fix type issues

* Provide the CLI dev env vars through the CLI, don’t build dotenv into facade

* Removed the logger import

* Address Matt’s comments

* Addressing more of Matt’s comments

* Handle sending an execution after a websocket connection closes

* Remove auth from the env attributes to prevent obfuscation
2024-02-20 12:16:41 +00:00
Eric Allam 2aabb994cf Install the protobuf compiler in the docker builder stage 2024-02-12 16:58:57 +00:00
Eric Allam baf3a84cda v3 dev cli command + more (#894) 2024-02-12 14:24:32 +00:00
Matt Aitken dd879c8e4a Fix api run statuses (#874)
* Added a subtask for testing

* Make it easier to run the CLI from the nextjs reference project

* Copies of the run and statuses endpoints, but without simplifying the run statuses

* Use the new v2 endpoints that give the full run statuses

* Removed unused import

* v2 events endpoint with the full run status info

* Changeset
2024-01-29 11:42:19 +00:00
Eric Allam 6c4047cf21 Fix an issue where runs were stuck executing when a child task failed and the parent task retried 2024-01-17 14:39:06 +00:00
Kritik Jiyaviya 4dd6cf18dc feat: Add support for the new Resend audience API endpoints (#850)
* feat: Add support for the new Resend audience API endpoints

* update pnpm-lock.yaml

* update pnpm-lock.yaml

* Merge branch 'main' into feat/support-for-resend-audience-api

* latest pnpm-lock.yaml

* Create lovely-items-end.md

* Update pnpm-lock.yaml

---------

Co-authored-by: Matt Aitken <matt@mattaitken.com>
2024-01-17 14:06:45 +00:00
Matt Aitken 1bbd7e6dc3 V3 CLI started, with Personal Access Tokens and Who Am I (#841)
* Initial CLI commit

* Hooked up the CLI so it actually shows stuff…

* Totally reworked the v3 CLI to be based on our existing CLI (Commander)

* WIP on PersonalAccessTokens and AuthorizationCodes

* WIP creating a Personal Access Tokens page. Created a new sidenav for account pages

* Creating tokens is working but the form is broken

* Tokens are created in the UI

* Creating and revoking access tokens from the UI is working

* Improved the create form and copy

* Tokens are a bit shorter and only lowercase

* API endpoint for creating AuthorizationCodes and the web page users hit to create PATs from them

* V3_ENABLED env var and hook that can be used in the UI to show/hide things

* API route to get a PAT (within 10 mins of creating an auth code). Moved some code to core

* Start to build the login command

* Nicer banner when starting the CLI

* Nicer update checking

* Update command style improved

* Removed the template step

* Login command options are now working

* The new CLI is logging in using the Personal Access TOken. But I need to save it still

* Logging in and saving the token is working

* Deal with already being logged in

* Who Am I working with PAT

* Deleted old account side menu header

* Improved the copy on the Auth code page

* Added docs for the login and whoami commands

* Added readme instructions for the update command

* Removed some unused things

* Remove fetchUseProxy for now
2024-01-15 14:45:41 +00:00
Matt Aitken adf497bc3e Example in the job-catalog of canceling all runs for a Job 2024-01-03 10:31:56 +00:00
Eric Allam 6a3c563f14 Fixed Job.attachToClient 2023-12-08 10:15:06 +00:00
Eric Allam 17f6f29d05 Feature: Support multiple runtimes other than Node.js (#774)
🚀 Publish Trigger.dev Docker / e2e (push) Failing after 0s
🚀 Publish Trigger.dev Docker / units (push) Failing after 5s
🚀 Publish Trigger.dev Docker / typecheck (push) Failing after 5s
🚀 Publish Trigger.dev Docker / publish (push) Has been skipped
2023-12-06 10:23:37 +00:00
Matt Aitken 68cbfd8d23 Log out the run status 2023-11-30 10:29:48 +00:00
Eric Allam ca05d5f603 @trigger.dev/openai: Adding additional assistant tasks 2023-11-28 22:48:31 +00:00
nicktrn 1dcd87a2aa URI encode keys (#758) 2023-11-28 18:05:15 +00:00
nicktrn 096151c014 Fix: Shopify subpath imports, enhance docs (#755)
🚀 Publish Trigger.dev Docker / typecheck (push) Failing after 0s
🚀 Publish Trigger.dev Docker / units (push) Failing after 0s
🚀 Publish Trigger.dev Docker / e2e (push) Failing after 0s
🚀 Publish Trigger.dev Docker / publish (push) Has been skipped
* Fix invalid subpath imports

* Remove @trigger.dev/tsup dep

* Update lockfile

* Small github docs fix

* Pull runtime adaptor import out of integration

* Fix io.store docs examples

* Add sendEvents to task library

* Add KV to task library and small fixes

* Suppress duplicate http endpoint warnings

* Changeset

* Add task returns
2023-11-27 16:03:03 +00:00
nicktrn 067e19fec9 Feature: New Webhook Triggers, KV Store, and Shopify integration (#745)
* Attach webhook to http endpoint

* Create internal webhook handler job

* Fix delivery

* Add webhook trigger

* Index webhooks

* Fix webhook connect

* Webhook activation

* Rename to TriggerWebhook

* Save webhooks to index

* Extend register payload

* Revert attachSource

* Add update webhook helper

* Use crud interface for registration

* Remove filter from params

* Destructure options

* Missing ohash dep

* Chainable trigger filter

* Optional request verification

* Handler call with context

* Update config on register success

* Rename to webhookData

* Add basic key-value store

* Improve kv store

* Webhook trigger UI

* Webhook delivery UI

* Some fixes

* Airtable experiments

* Fix tabs transitions

* Only run register if config differs

* Use subtasks for delivery

* Missing dep

* Error handing and fixes

* Namespace cursor

* Fix hmac verification

* Gut Airtable

* Lockfile

* Template optional api key

* Template event sources and icons

* Template remove filter from params

* Fix models template

* Template fixme

* Fix webhook trigger header

* Actually send params for registration..

* Add back oauthed client

* Verify signature header encoding param

* Fix delivery context

* Webhook create retry with prior delete

* Webhook source type fixes

* Webhook config merge

* Update webhook icon

* Verify webhook task options

* Rename handler to generateEvents

* Send event icons

* Shopify with webhooks

* Shopify job-catalog entry

* Lockfile

* Add webhook migration

* Better icons

* Type fixes

* Make param type optional

* Scope type

* More examples

* More schemas

* Trigger catalog

* Bump versions

* Just a few more events they said

* Simplify session and config

* Typed serializer

* Fix payload type

* Remove unused getters

* Reduce logging output

* Rest resource tasks

* Lockfile

* Bump version

* Remaining triggers

* Fix event types

* Fix import

* Link from webhook trigger to http endpoint

* Use rest resource tasks for crud

* Fix rest save type

* Improve KV types

* Improve attach webhook logging

* Make header schema more lenient

* Fix save type again

* Final error callback

* Shorten keys

* Remove ohash diff

* Disable Airtable webhooks

* Webhook environments

* Fix shopify error import

* Link to integration and http endpoint

* Integration catalog entry

* Disable oauth

* Require api key, simplify client secret

* Move trigger types

* Update catalog example

* Small KV endpoint refactor

* Fix custom migration

* Webhook environment migration

* Improve crud types

* Remove unused schema

* Fix crud context type

* KV limits and HTTP verbs

* KV improvements

* Jobless webhook delivery

* Airtable delivery fixes

* Make deliveries sexy again

* Some docs

* Fix payload types.. again

* Final payload pass

* Schema cleanup

* Refactor handler service

* Remove more stale schemas

* registerJobNamespace import

* Expose KV on TriggerClient

* Remove dummy click handler

* Remove unused components

* Fix delivery pagination

* Enable registration rerun button

* Comment out registration route action handler

* Rename KV as not tied to IO anymore

* Swap api with trigger client

* Disabled fields param

* KV docs

* Changeset

* Shopify docs fixes
2023-11-24 10:22:12 +00:00
Eric Allam 131d0dbea4 Adding an example of using OpenAI GPT 4 Vision 2023-11-21 10:23:11 +00:00
Eric Allam 0ab51d62bc Don’t send notifications for internal runs and add isTest to the notification
🚀 Publish Trigger.dev Docker / publish (push) Has been skipped
🚀 Publish Trigger.dev Docker / e2e (push) Failing after 19s
🚀 Publish Trigger.dev Docker / units (push) Failing after 3s
🚀 Publish Trigger.dev Docker / typecheck (push) Failing after 6s
2023-11-16 14:45:39 +00:00
Eric Allam 756024da78 Add support for Run Notifications #401 (#738)
* Add support for Run Notifications #401

* Fixed typo and added changeset
2023-11-16 14:19:42 +00:00
Eric Allam 14429b8b66 Fix “transaction already closed” error when running tasks (#735) 2023-11-14 23:22:34 +00:00
Eric Allam 75550f535c Fix undefined task outputs
🚀 Publish Trigger.dev Docker / typecheck (push) Failing after 5s
🚀 Publish Trigger.dev Docker / e2e (push) Failing after 0s
🚀 Publish Trigger.dev Docker / units (push) Failing after 19s
🚀 Publish Trigger.dev Docker / publish (push) Has been skipped
And also fix the SSE connection getting reset on a infinite loop when runs are complete
2023-11-13 16:18:41 +00:00
Eric Allam cb1825bfaf Add OpenAI support for 4.16.0 (#726)
* Add OpenAI support for 4.16.0

* Add support for background polling and use that in OpenAI integration to power assistants

* Much improved OpenAI docs

* Added backgroundPoll docs

* Implements waitForEvent and added docs for more built in tasks

* Add sendEvent API referenc

* Write the task libray

* Add changeset and warning for waitForEvent
2023-11-09 16:59:58 +00:00
nicktrn d02173442c Feature: io.sendEvents() (#728)
* Add io.sendEvents

* Add examples to built-ins catalog entry

* Add docs

* Add changeset

---------

Co-authored-by: Matt Aitken <matt@mattaitken.com>
2023-11-09 16:42:23 +00:00
Eric Allam a74716a1b9 Added waitForRequest built-in tasks (#725)
* Added waitForRequest built-in tasks

* Create grumpy-buttons-thank.md
2023-11-06 18:30:49 +00:00
Eric Allam 42185bc36b Upgrade resend integration to use resend-node 2.0.0 (#724)
* Upgrade resend integration to use resend-node 2.0.0, fixes issue with Next.js 14

* Create mighty-dodos-study.md
2023-11-06 18:30:37 +00:00
Eric Allam 329944841f Fix the cancel event example in job ref 2023-11-06 17:38:18 +00:00
Matt Aitken 9c6838e5c8 HTTP endpoints, HTTP triggers and new sidebar (#720)
* Setup basic storybook page

* storybook preview pages don’t have padding

* Added a storybook story for the side menu

* WIP on a new side menu component

* New variant for the side menu button style

* Side menu items can now recieve a badge ‘count’

* Storybook side menu update

* Added a new font size

* you can now set the spacing between an icon and the text

* Moving the error and count badge to the right side of the button

* Added a menu sub item button style

* added sub items to the menu and all the rest of the items

* Proper background class used

* Added a side menu header and main org header

* Removed the no-mobile blocker overlay so you can use the site on a phone

* Better section name

* Created a cusom popover trigger that can take anything as children. Also the popover titles can now take a variant

* Added a popover for the user avatar menu item

* Removed the tool tip

* Arrow Popover can no also go full width

* neater conditiona;

* Removed unnessary div. And error icon can display an error message

* menu item buttons can take a variant

* Added placeholder items for the org dropdown

* add a max height to the org menu popover and handle scrollbars

* Added a border if the menu scrolls when you have lots of project items

* warning badges display tooltips if you give them some error text

* Removed the sticky header so the scrollbar looks better

* Fixed a layout issue from removing the sticky header

* Better spacing between lists

* Popover can take any react component icon and an icon classname

* Side menu headers now have a dropdown by clicking the ellipsis icon

* Side menu items can take an icon colour

* Updated styling for the dropdown Popover menus

* Installed Tabler icons

* Updated some icons to use Tabler

* Started defining the HttpTrigger class

* RequestFilterSchema is part of core

* Use the RequestFilterSchema from core

* EventSpecificationExample now derived from schema

* Need to define the class like EventTrigger

* Catalog reference that uses the httpTrigger

* The logic for the httpTrigger is in the separate file, just call it from the client using defineHttpTrigger

* Basic HttpTrigger

* Import fixes

* Transforming httpTrigger to the format we want

* Initial httpTrigger types working

* Added HTTP generic type

* Added some notes

* Rejigged some things to support verifying payloads

* Use new security function for verifying

* Added truncating text support if side menu names get long

* Renamed some stuff

* Removed test copy

* HttpEndpoin that isn’t a trigger

* Renamed hostname to source

* Created HttpEndpoint as a top-level concept, it produces triggers using onRequest

* Started working on registration of the httpEndpoints

* Removed the new TriggerMetadata for now, http will use static

* Added PayloadType to EventRecord. Default is JSON.

* The payload from an HTTPEndpoint is now a Request

* Initial TriggerHttpEndpoint migrations

* Initial server side work

* Migration to add shortcode to RuntimeEnvironment

* Added TriggerHttpEndpointEnvironment table

* Made the httptrigger catalog example runnable

* Progress creating/updating httpEndpoints

* HTTPEndpointEnvironment updating now works

* Added a note about EventRecord HttpEndpoint relationship

* When creating environments, create a shortcode

* Fix for secret not being retrieved correctly

* Request filtering

* Add DOM types to Core

* Early work dealing with a webhook arriving

* Added some todo notes to discuss tomorrow

* Added relations to the endpoint so we can call the user’s server

* Basic handling of immediate responses

* The immediate responses are now going back to the webhook provider

* Handling is now generating events and responding where relevant. Not yet matching filters and doing verification

* Added source to TriggerHttpEndpointEnvironment

* The event filter is now optional

* The source is passed through from the httpEndpointEnv

* Added optional payloadType when ingesting an event

* Added tabler

* Updated @remix-run/testing usage for remix 2

* Fixed org menu header width and added org and project types

* Added custom icons for Runs and Endpoints

* Updated the menu icons

* storybook side menu shows static project and org data

* Updated side menu

* Plumbed in paths

* Updated eventFilter to deal with undefined/null payloads

* Started work on filtering, but it’s not working

* Strict null

* async filtering it working

* Added environment shortcodes to the seedCloud script

* Don’t pass the secret back, added a verifyPayload function that gets called before runs start

* Fix for skipping triggering runs

* The verify callback is now passed through to the trigger so it can be called

* If verifying fails, have a reason which we can show to the user

* Don’t verify if it’s a test run

* Fixed the log in the example code

* Added loader and action to storybook preview

* Fix for svg attribute name issues in EndpointIcon

* Started work on the side menu

* This logic has moved to a separate file, in a different branch

* Don’t get the environments when getting orgs

* Lots of fixes for the new SideMenu

* Refactored some org/project things to fit the new structure better

* Some progress restructuring the app for the new sidebar

* More changes to data loading

* Deleted some old pages. The redirecting is working if you’re on the root but logged in

* Basic styling for the account page. We need a back button

* Fixes for the integrations pages

* Breadcrumb implemented

* Add a border to the bottom of the breadcrumb

* Feedback is working, not in Storybook

* Removed expandSidebar from the Handle

* Org level breadcrumbs

* Getting the project in the org route now works properly

* orgslug route before using the presenter

* More UI improvements

* Removed commented out line

* Use presenters for org and root

* Removed forceActive

* Split the project selector and user menu out

* The side menu items have correct active states

* Fixed the scrolling

* Put the org name in

* Improved the admin features

* Created the HTTP Endpoints list page

* Work on the HTTP endpoint page

* WIP on the HTTP Endpoint page

* Fix for response filter not showing

* Change page URLs to “http-endpoints” (from httpendpoints)

* Slugify the HTTP endpoint id

* Added todos for EventRecord associations

* Removed the Job/Trigger tab for now

* First draft of the help panels

* Rejigged the HTTP endpoint page so it splits out the deployed and not deployed environments

* Removed irrelevant columns

* Added a link to the deployment guide

* Removed the background image from the frameworks pages

Also…
- reordered the frameworks so supported ones come first in the list
- added logos to 3 framework pages that were missing them
- removed lots of un-used imports

* Fixed an issue where the breadcrumb could be hidden on small screens so broken the pages

The breadcrumb is always visible now

* Renamed ‘Onboarding’ to ‘Framework setup’

* Added a new http-endpoint icon to the side menu

* Side menu now has a nice looking scrollbar

* Scrollbar on the usage and billing page updated to the nice style

* Icon and colour updates to the side bar menu items

* Small copy improvement

* Fixed bug with the height of the org menu dropdown

* Page breadcrumbs now don’t include duplicate path links

* renamed uiBorder to ui-border

* Remove the padding from the small menu item if it’s used as a button

* Made the paragraph text a bit smaller

* EventRecord has relationships with TriggerHttpEndpoint/TriggerHttpEndpointEnvironment

* Improved the presenter used by the side menu. Also added the HTTP Endpoint badge count to the sidebar

* The dropdown org menu has the same badge as the side menu

* Improved the Projects page

* Fixed button style miss-match in the jobs table dropdown

* Made the popover dialogs support Headers and improved the regenerate API key popover

* Made the active status of a job more subtle

* Allow the Job to have a link on its Trigger

* Simplified loader because we ditched the Jobs menu. Also showing no runs help panel now

* Fixed type errors

* Pages that are org level can scroll if the content gets too long

* Fixed a redirect bug so you now redirect to the org page

* Updated the shortcode migration, to add additional letters to the end

* Removed the nav bar from the welcome onboarding page

* Only show the countBadge if the number is > 0

* Jobs icon now has a default colour the same as the nav menu

* Capital I for integration

* Clone the request before calling verify

* Separate the checks out so they can be easily logged

* Make the failure reason optional

* Made the verification error look better, if it’s provided

* Working WhatsApp example

* Can set the immediateResponseFilter to null in an update

* Deal with a null immediateResponseFilter

* Cal.com webhook working

* Header height reduced slightly

* Fixed button padding

* Removed un-used imports

* http endpoint table displays a message if you have no entries in the table

* Removed some duplicate text and swapped primary buttons for secondary style

* All ‘HTTP Endpoints’ now ‘HTTP endpoints’

* You have to use a RequestFilter when using HTTP Trigger

* Synchronous eventDispatcher filtering again. Better content-type check

* Stripe example

* Side menu “Usage & Billing” is just “Usage” if self-hosting

* Fix for filtering not working

* Removed filter log

* Fix for jobs being optional and null

* Removed old nav bar from the invite page

* Deleted old nav bar component

* import cleanup

* NoTask message only displays if the run has status “SUCCESS”

* Added a global page loading indicator

* Added documentation buttons to the most important pages

* Added GitHub

* Use “verifyRequestSignature” helpr

* Removed todo

* Fixed the layout of the Rerun Job popover

* Jobs table has margin bottom so it sits off the bottom of the page

* Small improvement to the search bar layout on the integration page

* HTTP endpoint and Trigger concept docs

* Reference docs and JSDocs for HTTP endpoint

* SDK reference docs for HttpEndpoint

* Added an HttpTrigger SDK reference, which mainly just point you elsewhere

* Improved the copy in the help panels

* Removed unused code

* Improved HTTP endpoint comment about setting the secret

* Delete unused asyncFilter

* Fixed the delete modal not having a title

* Always try get the arrayBuffer for an HTTP request, catch and use undefined as the default

* Add extra method to invokeTrigger

* Use a transaction for the secret store

* verifyRequestSignature docs page and JSDocs

* Added verifyRequestSignature docs page to nav

* Warn if they use the same HttpEndpoint id twice

* Added another job using the same HttpEndpoint

* Renamed the api route to http-endpoints from httpendpoints

* Debug bar for VSCode is now in the commandCenter

---------

Co-authored-by: James Ritchie <james@jamesritchie.co.uk>
2023-11-06 15:32:40 +00:00
nicktrn cf8f994604 Feature: io.random() (#716)
* Add io.random

* Add changeset

---------

Co-authored-by: Eric Allam <eric@trigger.dev>
2023-11-03 11:07:12 +00:00
Eric Allam 620b83832b Implement manually invokable jobs through the invokeTrigger (#700)
* Implement manually invokable jobs through the invokeTrigger

Also implemented a job run notification system, that will POST details of a run on completion. This combines with the task callbackUrl system to implement the invokeAndWait

* Document the invoke trigger

* batch invoke and wait

* background fetch timeouts

* Use @whatwg-node/fetch instead of the polyfilled fetch

* Fix some outdated dependencies in webapp

* Improved subtask error propogation messages

* Document the OpenAI changes and the batch invoke stuff

* Fix dequeuing jobs

* Don’t retry the OpenAI completion background task

* Added OpenAI changesets

* Use the new ResumeTaskService in ProcessCallbackTimeout as well
2023-11-03 11:05:00 +00:00
nicktrn 9c5a7cf073 Fix: Allow errors and special values on Airtable field set (#697)
* Allow errors and special values on field set

* Add changeset
2023-10-31 10:17:05 +00:00
James Ritchie 0876b87526 No Tasks in a Run warning message (#688)
* Card hover state is now grey

* Added a callout warning if you don’t have any tasks

* Conditionally show the no-task warning at the top of the run

* New job to test the no-task warning message

* Created a new example Job for the CLI that uses Tasks

* Updated the Next.js cli joke job

* Updated the Remix app cli job example

* Updated the Astro app cli job example

* Made sure the example job works in remix

* Added changeset

---------

Co-authored-by: Eric Allam <eric@trigger.dev>
2023-10-30 16:24:46 +00:00
Eric Allam 21f62e780c Added tests and refactored response to use camelCase 2023-10-26 11:30:49 +01:00
Eric Allam 36a390449a @trigger.dev/openai: Upgrade to openai-node SDK 4.13.0, add support for request options and defaultQuery/defaultHeaders 2023-10-24 17:45:43 +01:00
Eric Allam cc44aff24b Added some docs about using the OpenAI integration as a universal client 2023-10-24 09:35:20 +01:00
Eric Allam 07d28ee4f5 Add custom baseURL and idempotency key support to OpenAI integration (#682)
This commit introduces the ability to set custom base URLs, which allows for the use of different APIs such as Perplexity. It also ensures that any OpenAI-related task now includes an idempotency key when making requests. Works for background tasks as well.
2023-10-24 09:19:38 +01:00
Eric Allam 0c4aea0338 Add execution duration for background fetch 2023-10-23 16:55:00 +01:00
Eric Allam 044d38e390 Improvement: Auto Execution Yielding (#612)
* Auto-yield run execution to help prevent duplicate task executions

* Add auto-yield config to endpoints

* Refactor run execution with buffer and limits

Introduced constants RUN_CHUNK_EXECUTION_BUFFER and MAX_RUN_CHUNK_EXECUTION_LIMIT. Adjusted PerformRunExecutionV2Service to use the new constants to fine-tune execution timings and buffers.

* Add endpoint probing functionality

Added new `RESPONSE_TIMEOUT_STATUS_CODES` in `consts.ts` to manage timeout responses. Additional functions `detectResponseIsTimeout(response: Response)` was added in `endpoint.server.ts` to detect if a response was a timeout based on the status codes from `RESPONSE_TIMEOUT_STATUS_CODES`.

Update actions to use new endpoint probing endpoint service. This allows for the early probing of endpoints to determine if they're up and running.

A new class `ProbeEndpointService` was created in `probeEndpoint.server.ts` which makes HTTP requests to a given endpoint and updates its properties based on the result.

Finally, `detectResponseIsTimeout(response)` is used in `performRunExecutionV2.server.ts` for marking the execution as succeeded when facing a timeout.

* Refactored probe method in EndpointApi class

The probe method of the EndpointApi class has been refactored to remove the error handling part and it now takes a timeout sent from the client directly. The corresponding changes were also made in the ProbeEndpointService and TriggerClient objects to reflect the alterations in the probe method.

The error handling related to the timeout has been removed and the responsibility of handling the timeout has been shifted to the client. Thus, the probe method has been greatly simplified. The `probeEndpoint.server.ts` file was also changed to accommodate the change in behavior of the probe result.

In the `triggerClient.ts` the timeout for probe is now read from the incoming request object. For backward compatibility, if no timeout is provided in the request, the default value of 15 minutes is used.

* Remove performRunExecution v1 enqueue function
* Better document limits and add docs on increasing function timeouts
* Upgrade webapp docker container to use 18.18.2
* force clients to yield when a run is executing in a gracefully shutting down worker
* Renamed task `key` to `cacheKey` and added more task documentation
* Index the `@trigger.dev/sdk` version on Endpoints
2023-10-20 17:44:10 +01:00
Matt Aitken abc9737a4f Tabler icons added to NamedIcon
commit 13ed268f1d6732ef28c6e4a9eeb927c1af822032
Author: Matt Aitken <matt@mattaitken.com>
Date:   Fri Oct 20 11:38:22 2023 +0100

    Added all the tabler-icons to Storybook

commit 62738d71a40f17c45c0b29332359b2b670233a97
Author: Matt Aitken <matt@mattaitken.com>
Date:   Fri Oct 20 11:38:09 2023 +0100

    Use the no-stroke version of the tabler-sprite so we can control the stroke width

commit a1b7edd57f8b2d3f2a5f4d96cbc9eb0a887f3442
Author: Matt Aitken <matt@mattaitken.com>
Date:   Fri Oct 20 11:37:54 2023 +0100

    Set the default width to 1.5, so they’re a bit thinner

commit 69ba38802b849376611678537f7d8e248b60086b
Author: Matt Aitken <matt@mattaitken.com>
Date:   Fri Oct 20 10:55:34 2023 +0100

    Added a tabler icon to an events job catalog job

commit eb3c5f649d06adcc9ce0641f50e082ecdeead7a2
Author: Matt Aitken <matt@mattaitken.com>
Date:   Fri Oct 20 10:53:40 2023 +0100

    Moved the tabler sprite to the app and use an import. This means it’ll be cached and we can move it

commit c24d76e6df9233b001a0242f8806076562ca2e90
Author: Chaturved Degloorkar <48447253+chaturrved@users.noreply.github.com>
Date:   Fri Oct 20 15:22:59 2023 +0530

    feat: Add support for tabler-icons when using the icon for Tasks  (#629)

    * Add support for tabler-icons

    * Changed the core update type from minor to a patch

    ---------

    Co-authored-by: Matt Aitken <matt@mattaitken.com>
2023-10-20 11:41:45 +01:00
Matt Aitken 6769d6b439 Squashed commit of the following:
commit c028db229cece2bf7d7acf04e7b1c08fea61798c
Author: Matt Aitken <matt@mattaitken.com>
Date:   Fri Oct 13 16:04:54 2023 +0100

    Some fixes for getting the endpoint id

commit e31b7ac5f2d451f70e3d42a3fb3e5f718d93485b
Author: Matt Aitken <matt@mattaitken.com>
Date:   Fri Oct 13 15:39:18 2023 +0100

    Added a link in the instructions if Node isn’t detected

commit 1a59cced775fbeac3e5b94e0c151b490c838d8f8
Author: Matt Aitken <matt@mattaitken.com>
Date:   Fri Oct 13 15:39:05 2023 +0100

    Fixed some conflict issues

commit 77263ed26bcf7025071ac0faf15fb867fff245d0
Author: Matt Aitken <matt@mattaitken.com>
Date:   Fri Oct 13 15:38:56 2023 +0100

    Added a changeset

commit 1a7df3f37f0c65d68cb9a9fe7ac50328bbb4d879
Author: GuyBorderless <100129659+guy-borderless@users.noreply.github.com>
Date:   Fri Oct 13 17:31:01 2023 +0300

    feat: enable deno  (#531)

    * adding deno reference project and JSRuntime file

    * accessing runtime specific functions through JsRuntime

    * adding support for both deno.json and deno.jsonc

    * no automatic setup for deno projects currently

    ---------

    Co-authored-by: Guy Kedem <guy@propertiz.net>
    Co-authored-by: Matt Aitken <matt@mattaitken.com>
2023-10-13 16:20:18 +01:00
Eric Allam 29ca09e42d Fixes #611: intervalTriggers of >10 never starting
🚀 Publish Trigger.dev Docker / typecheck (push) Failing after 5s
🚀 Publish Trigger.dev Docker / e2e (push) Failing after 6s
🚀 Publish Trigger.dev Docker / units (push) Failing after 21s
🚀 Publish Trigger.dev Docker / publish (push) Has been skipped
2023-10-13 09:38:25 +01:00
Matt Aitken 50e3d9e43a Indexing errors don't get displayed anywhere (#605)
* Added EndpointIndex status. Default is PENDING, existing rows are SUCCESS

* Made it easier to create a migration SQL file

* Created a job-catalog file for misconfigured Jobs that should error when running the CLI

* EndpointIndex data and state are now optional

* The Environments page now shows the status of the last refresh

* Improved the UI about endpoints

* Added EndpointIndex error column

* WIP making the endpoint indexing more robust

* Indexing errors are now surfaced

* Improved the error show it shows the job id

* Use “performEndpointIndexing” when you create your first endpoint from the UI

* Use “performEndpointIndexing” for the recurring endpoint checker

* Staging is now auto-indexed every 10 mins too

* Use “performEndpointIndexing” for the webhook

* Moved the throttling to a util

* Removed instructional comments

* Created a reusable retry system with exponential backoff

* Use p-retry for retrying with backoff

* The CLI gets indexing results and displays errors

* Improved the indexing error messages and display in the console

* Use a pre so the Indexing error is correctly split over multiple lines

* Use a db transaction for webhook that triggers endpoint indexing

* Tidied up imports

* Support older versions of the server

* Improved the comment on the misconfigured job

* Changeset: When indexing user's jobs errors are now stored and displayed
2023-10-11 17:31:19 +01:00
Eric Allam 59a94c710e Fixes #583 by filtering out properties with missing values (#584)
This also improves the retrying behavior of the resend integration by skipping retrying certain client errors
2023-10-09 16:04:32 +01:00