Commit Graph

34 Commits

Author SHA1 Message Date
Eric Allam d855d55ea0 re2: env based queue selection algo (#1775)
* re2: fix @trigger.dev/core exports

* re2: WIP env based queue selection algo

* more wip

* WIP

* Get run engine tests to pass

* Adding tests for the fair dequeueing strat in the run engine

* Configure the new queue selection strategy in the webapp and get it all building and typechecks passing

* webapp now uses built packages, building redis-worker, run-engine, database, using better tsconfig setups for tests, moving isomorphic code into core/v3/isomorphic

* Fixed webapp typechecks

* dev now depends on build, fixed supervisor typecheck

* Fixed run engine tests

* Fixed e2e tests
2025-03-07 14:30:19 +00:00
Matt Aitken 983bb411cc Redis client factory (error handling and good defaults) (#1761)
* @internal/redis package for creating Redis clients with sensible defaults and error handling

* Added @internal/redis to webapp
2025-03-05 18:32:46 +00:00
nicktrn e97704d904 Run Engine 2.0 (WIP) (#1575)
* bump worker version

* Suggested glossary for the RunEngine, TBC

* Removed BatchTaskRun changes from this branch, they were done in main

* Set the BatchTaskRun status to completed when all runs are completed

* When dequeuing respect passed in maxResources

* Ported over the new run props: idempotencyKeyExpiresAt, versions, oneTimeUseToken, maxDurationInSeconds

* Didn’t hit save… the new props when triggering tasks passed through

* Idempotency expiration + waitpoint edge case

* WIP on creating checkpoint, parking for now

* fix worker routes

* upgrade webapp node types to support generic event emitter

* separate event bus handler singleton and run failure alerts

* duration waits

* fix execution snapshot debug spans

* task waits

* fix event bus types

* temporary fix for react hook run handle type

* disable run notifications for now

* convert any typecasts to expect errors to more easily fix later

* fix webapp types after node types upgrade

* updateEnvConcurrencyLimits across marqs and the runqueue

* Pass proper values into the run engine

* RunQueue settings and removed unused rebalancing workers

* Remove rebalancing prop

* Tidied more things up

* Update/remove queue limits for MARQS and RunQueue

* taskQueue/concurrencyLimit changes ported back into the RunEngine

* Reworked completing waitpoints to improve performance and reduce race conditions

* Improved test robustness

* Down to a single run lock only when a run is totally unblocked and ready to continue

* warm starts, worker notifications, wait fixes

* Fix for Run Engine poll interval env var

* Expect the waitpoint to be completed quickly

* If a run is locked then it’s too late to expire it

* Added VALKEY_ env vars and plugged them into the run engine

* Extracted and updated the guard queue function so it can be used when batching

* Added logging and universal concurrency changes to trigger task v1

* Added notes back in

* Bump @trigger.dev/worker to 3.3.7

* reportInvocationUsage for the runAttemptStarted event

* improve execution snapshot span debug span start times

* Unfriendly IDs

* update lockfile

* Created a shared determineEngineVersion function

* disable unfinished commands

* save new cli config to different location, misc fixes

* add basic engine version check via current deploy

* new run engine will default to node 22 runtime

* block some actions for projects on previous run engine

* fix worker group tests

* fix triggerAndWait test

* one typescript version to rule them all

* redlock type patch

* fix type issues caused by ts-reset

* improve cleanup scripts

* add missing socket.io dep

* fix run notification handler type

* fix worker group test again

* generate prisma client for e2e tests

* remove worker group tests for now

* prevent image pull rate limits during unit tests

* increase timeout for queue concurrency limit test

* generate prisma client for preview release

* same node types everywhere

* Updated engine readme, removed legacy system notes

* use default machine preset from platform package

* worker instances plural in schema

* disable pnpm update notifications

* return worker group details from connect call

* add workers admin route

* fix heartbeat route return type

* move deployment labels to core apps

* refactor run controller env schema

* Add firstAttemptStartedAt to TaskRun

* RunEngine 2.0 batch trigger support (#1581)

* Make it clear when BatchTriggerV2Service is used

* Copy of BatchTriggerV2Service

* WIP batch triggering

* Allow blocking a run with multiple waitpoints at once. Made it atomic

* Removed unused param

* New batch service

* Pass through the parentRunId and resumeParentOnCompletion

* Use the new batch service, and correct trigger task version

* Force V1 engine if using BatchTriggerV2Service, we’ve already done the check at this point

* Removed the $transaction and early exit if nothing changed

* Adedd a simple batch task to the hello world reference catalog

* Fix for batch waits not working

* Added parentRunId in a couple more places

* Removed waitForBatch log

* Added another parentRunId

* Expanded the example to include all the different triggers

* More changes to blocking to support continuing after idempotent completed runs

* Fix for the wrong type when blocking a run

* remove @map

* optimise worker auth query

* add engine version header to core api client requests

* remove unique constraint for default group id

* consolidate migrations

* the first managed worker becomes the global default

* Debug events off by default, added an admin toggle to show them

* worker group name can't be an empty string

* add exec helper to core

* move machine resources to core

* add pre-dequeue callback to determine max resources

* optionally skip dequeue

* bump worker package

* move worker to core

* fix ReadableStream type error

* fix another type issue

* update a few more tsconfigs

* add metadata changes introduced in #1563

* Run Engine 2.0 trigger idempotency (#1613)

* Return isCached from the trigger API endpoint

* Fix for the wrong type when blocking a run

* Render the idempotent run in the inspector

* Event repository for idempotency

* Debug events off by default, added an admin toggle to show them

* triggerAndWait idempotency span

* Some improvements to the reference idempotency task

* Removed the cached tracing from the SDK

* Server-side creating cached span

* Improved idempotency test task

* Create cached task spans in a better way

* Idempotency span support inc batch trigger

* Simplified how the spans are done, using more of the existing code

* Improved the idempotency test task

* Added Waitpoint Batch type, add to TaskRunWaitpoint with order

* Pass batch ids through to the run engine when triggering

* Added batchIndex

* Better batch support in the run engine

* Added settings to batch trigger service, before major overhaul

* Allow the longer run/batch ids in the filters

* Changed how batching works, includes breaking changes in CLI

* Removed batch idempotency because it gets put on the runs instead

* Added `runs` to the batch.retrieve call/API

* Set firstAttemptStartedAt when creating the first attempt

* Do nothing when receiving a BATCH waitpoint

* Some fixes in the new batch trigger service… mostly just passing missing optional params through

* Tweaked the idempotency test task for more situations

* Only block with a batch if it’s a batchTriggerAndWait… 🤦‍♂️

* Added another case to the idempotency test task: multiple of the same idempotencyKey in a single batch

* Support for the same run multiple times in the same batch

* Small tweaks

* Make sure to complete batches, even if they’re not andWait ones

* Export RunDuplicateIdempotencyKeyError from the run engine

* Latest lockfile

* Trigger with a machine (old run engine)

* RE2, allow setting machine when triggering

* Fix for new glob patterns

* add max run count to dequeue from version route

* add worker instance name env var and header

* queue consumer pre skip callback

* poll for more runs after final execution errors

* fix dequeue search param schema

* add shortcut to debug switch

* expose run engine timeouts as env vars

* make warm start durations configurable

* add optional status to json reply helper

* fix preSkip hook, add debug logs

* BLOCKED_BY_WAITPOINTS -> SUSPENDED

* exit controller when run suspended

* check if already replied before http reply

* run controller will wait for next run after the current one is suspended

* cancel run button shortcut

* minimal event repository environment type

* fix update metadata call

* run suspension and misc fixes wip

* change debug shortcut to shift + D

* Started work on the Dev supervisor

* Formatting

* Fix for bad imports

* Before rebuilding SSE

* Presence updating from the CLI working via SSE

* add worker notification debug logs

* send run:stop when exiting run phase

* skip current snapshot poll on worker notification

* add more logs and route to submit run debug logs

* add worker and runner ids to snapshots

* improve run notification debug logs

* add workload debug log route

* misc run controller fixes and refactor

* prevent parallel execution of critical functions

* update bun to 1.2.1

* WIP with dev dequeuing

* Method to convert friendlyIds to non-friendly, do nothing with actual ids

* Set the engine on BackgroundWorker, lazily upgrade projects to engine V2

* Runs with ttls were getting immediately expired… oops.

* Pass the Waiting for deploy reason through, so we have it on the execution snapshots

* Fixed the logic for getting the right background worker for a run

* Use the correct ID when dequeuing…

* determineEngineVersion is now fully functional

* Rate limiter ignores the dev endpoints

* Retrieving a batch gives you the runIds

* Set a unique version for the RE2 BatchTaskRun

* add provisional changeset

* The start of dev run execution is working

* First dev run working

* Moved the dev run controller closer to what Nick did with the managed one

* export exec output type

* Heartbeat fix: don’t heartbeat if _isHeartbeating == false

* Dev runs get notifications, some dev bug fixes

* Improved logging or dequeuing

* We need to dequeue runs from the latest version too, for triggerAndWait

* Ported Eric’s validateWorkerManifest with nicer errors

* When flattening an idempotency key if part is undefined, return undefined

* Dev logging fixes

* Remove sigterm listener

* Deprecating workers. Don’t specify a BackgroundWorker when dequeuing an environment

* Deleted some old files. Renamed “managed” to “deploy”

* When a build finishes, always copy the build dir (otherwise the first one gets trampled on by the 2nd)

* Dev master queues should work differently

* Deleting old workers

* Added debounce function to core

* Improvement to canceling

* WIP on debounce canceling on socket disconnection

* Added environment data to execution snapshots

* Dev runs that have stalled get “Canceled” with a reason explaining why

* Show CLI messaged when a connection to the platform is lost/restored

* Fix TriggerTask after merge

* Add trigger task v2 max attempts, replace some findUniques

* Port the new queue logic to the run engine

* More fixes post-merge

* We weren’t setting a `retryConfig` up for the tests… it’s now required

* Start the Redis worker inside the Run Engine… 🤦‍♂️

* Trying to make the testcontainers more reliable

* Added keyPrefix: "engine:”

* Badly placed bracket in trigger task

* Better Redis namespacing

* Fix for expired run not getting removed from the queue

* Don’t create a redis client in the testcontainers, return the redisOptions instead

* Cleanup redis client in the run lock tests

* Fix for the RunQueue not supporting keyPrefix

* Updated more of the RunQueue scripts rebalancing

* Trying to make Redis more robust in the tests…

* Improved test resiliciency more

* Fix for delays (checkpoint check)

* Increase the timeout slightly to fix ttl test

* Added priority support when triggering

* More wip trying to make test containers more reliable

* batchTriggerAndWait test is still failing… some wip to try fix it

* Fixed redis tests now we’re not providing a client

* Separate Redis clients for the run engine worker/queue/runlock

* Made the wait for duration test more resilient

* Added idempotencyKeyExpiresAt to Waitpoints

* Waitpoint timeouts and idempotency expiry

* Use finishWaitpoint, removed extra worker job

* Added waitpoint idempotency tests

* Creating resume tokens is working

* Some improvements to the resume tokens

* Moved resumeTokens to just be wait functions 🥳

* Delete old RuntimeManagers

* Wait for token is working

* Better test for the wait tokens

* Improved the test task some more

* Hide the accessories in the span inspector

* WIP on waitpoint inspector

* WIP on complete waitpoint form

* Span overview panel can be changed based on the entity type

* Improved the waitpoint display

* WIP on completing waitpoint form

* Use the existing CodeBlock for the tip

* Style improvements

* Complete waitpoint

* All waitpoint sidebar variants

* Waits now use a pause icon

* Durations waits use the API to create/block with a waitpoint, not the runtime

* Fix for engine.blockRunWithWaitpoint required org id

* Removed old wait code from the run controllers/task run process

* Form action for skipping a datetime waitpoint

* Move testDockerCheckpoint to a separate core package export (it can’t be bundled on the client)

* Fix for glitchy hourglass animation

* Completed waitpoints display better

* Increase Redis maxRetriesPerRequest to 20 (default)

* Completing and skipping waitpoints is working

* Remove the database prisma dev command, since we need to use create only now. Updated docs

* Added skip timeout, reworked the UI

* Tweaked spacing

* Added payload limit to waitpoint token completion from dashboard

* Test idempotency works on wait.for and wait.until

* Moved the worker-actions to /engine/ from /api/

* Moved dev engine endpoints to /engine/ from /api/

* Separate /engine/ rate limiter

* Added parallel wait prevention, it’s working for duration waits but not well for triggerAndWait yet

* WIP post-merge conflicts

* Set taskEventStore column in the new engine

* Remove duplicate keys

* Post-merge fixes

* Fix for span merge layout

* Use executedAt instead of firstAttemptStartedAt

---------

Co-authored-by: Matt Aitken <matt@mattaitken.com>
2025-03-05 14:40:14 +00:00
Eric Allam a2c2d920b7 Use redis worker for run heartbeats and alerts (#1669)
* Move the task run heartbeats to RedisWorker

* Move alerts to redis worker, improving redis worker

* Fix typecheck errors

* Use single threaded tests for redis worker

* Enable/disable the redis workers independently

* Remove preview release from PR checks
2025-02-06 10:21:57 +00:00
Eric Allam 2d8a41b18b feat: realtime (#1402)
* Denormalize run tags, increase character limit to 128

* WIP realtime subscribing to runs

* extracted the stream stuff into core, made it more reusable

* WIP tags

* Remove tags for now because it’s not support in electric

* Support async iterables, readable stream, and callback style subscription styles

* Remove tags streaming endpoint

* Add realtime rate limits and scope them to the /realtime path

* WIP rate limt per org

* Introduce per org rate limits

* WIP JWT auth

* Move migrations into new internal db package

* Resolve pnpm lock file

* Authenticating to the realtime API with JWTs are working

* realtime in the client

* Created react-hooks package and starting to move stuff in there

* Improve types for hooks

* schema tasks

* Added useBatch hook

* build uploadthing/fal demo and change how run metadata is synced to the server

* tweaks

* WIL realtime concurrency tracking

* Implement test for realtime client using testcontainers

also updated electric to latest version

* Allow customizing the expiration time of the automatic JWT created after triggering a task

* Add support for subscribing to run tags

* Improve auth types and API

* finalize the realtime API

* Fixed some example stuff

* Allow up to 10 run tags

* Remove core from docker-provider tsconfig paths to prevent it from being typechecked

* do the same for the kubernetes provider

* Fixing some typecheck errors

* Fix webapp type errors

* Update @trigger.dev/platform to 1.0.13

* Fix attw error

* Remove from/to in subscribeToRuns query params

* Add tests for the rate limit middleware and add custom JWT rate limits

* turn off webapp test parallelism

* Finish renaming jwt -> publicAccessToken and automatically give the JWT read access to the tags when using trigger

* Add changeset

* Attempt to fix unit tests in CI

* Skip running the auth rate limit middleware tests for now

* Try a beefier machine

* Try and run webapp tests separately

* Setup env vars

* Make sliding window test more reliabile
2024-10-21 15:07:08 +01:00
Matt Aitken 3d67bb81e9 Removed run-engine from the webapp package.json/tsconfig 2024-10-08 18:24:31 +01:00
Matt Aitken f2babbf637 Internal packages (testcontainers, redis-worker and zod-worker) (#1392)
* Some notes on the new run engine

* lockfile with setup for the run engine

* Documenting where TaskRun is currently mutated, to try figure out the shape of the new system

* Added notes about how triggering currently works

* Details about when triggering happens

* Lots of notes about waitpoints

* Started scaffolding the RunEngine

* Sketch of Prisma waitpoint schema while it’s fresh in my mind

* Got Prisma working with testcontainers

* Use beforeEach/afterEach

* Simple Prisma and Redis test

* Return Redis options instead of a client

* Simplified things

* A very simple FIFO pull-based queue to check the tests working properly

* Use vitest extend

* Separate redis, postgres and combined tests for faster testing

* Some fixes and test improvements

* Pass a logger into the queue

* A queue processor that processes items from the given queue as fast as it can

* Test for retrying an item that wasn’t processed

* First draft of waitpoints in the Prisma schema

* Remove the custom logger from the test

* Added a completedAt to Waitpoint

* Notes on the flow for an execution starting

* Added redlock, moved some files around

* Starting point for the TaskRunExecutionSnapshot table

* Added relationships to TaskRunExecutionSnapshot

* Change some tsconfig

* Moved some things around

* Added some packages

* WIP on the RunQueue

* Fix for some imports

* Key producer with some tests

* Removed the nv type from the keys… it’s not useful to do global queries

* Passing unit tests for all the public key producer functions

* Some basic tests passing for the RunQueue

* Simple enqueue test working

* Enqueue and dequeue for dev is working

* Don’t log everything during the tests

* Enqueuing/dequeuing from the shared queue is working

* Tests for getting a shared queue

* The key producer sharedQueue can now be named, to allow multiple separate queues

* The key producer uses the name of the queue as the input

* Extra info in the Prisma schema

* Dequeuing a message gets the payload and sets the task concurrency all in one Lua script

* Adding more keys so we can read the concurrency from the queue

* Setting the concurrency with dequeue and enquque is working

* Improved the tests and fixed some bugs

* Acking is resetting the concurrencies

* Check the key has been removed after acking

* Nacking is working

* Changed the package to CommonJS + Node10 so it works with Redlock

* Moved the database, otel and emails packages to be in internal-packages

* Moved some Prisma code to the database package

* Started using the RunEngine for triggering

* Progress on run engine triggering, first waitpoint code

* Create a delay waitpoint

* Moved ZodWorker to an internal package so it can be used in the run engine as well as the webapp

* Web app now uses the zod worker package

* Added parseNaturalLanguageDuration to core/apps

* internal-packages/zod-worker in the lockfile

* Pass in the master queue, remove old rebalance workers code

* Add masterQueue to TaskRun

* Fixed the tests

* Moved waitpoint code into the run engine, also the zod worker

* Completing waitpoints

* An experiment to create a new test container with environment

* More changes to triggering

* Started testing triggering

* Test for a run getting triggered and being enqueued

* Removed dequeueMessageInEnv

* Update dev queue tests to use the shared queue function

* Schema changes for TaskRunExecutionSnapshot

* First execution snapshot when the run is created. Dequeue run function added to the engine

* Separate internal package for testcontainers so they can be used elsewhere

* Remove the simple queue and testcontainers from the run-engine. They’re going to be separate

* Fix for the wrong path to the Prisma schem,a

* Added the testcontainers package to the run-engine

* redis-worker package, just a copy of the simple queue for now

* The queue now uses Lua to enqueue dequeue

* The queue now has a catalog and an invisible period after dequeuing

* Added a visibility timeout and acking, with tests

* Added more Redis connection logging, deleted todos

* Visibility timeouts are now defined on the catalog and can be overridden when enqueuing

* Dequeue multiple items at once

* Test for dequeuing multiple items

* Export some types to be used elsewhere

* Partial refactor of the processor

* First stab at a worker with concurrency and NodeWorkers

* Don’t have a default visibility timeout in the queue

* Worker setup and processing items in a simple test

* Process jobs in parallel with retrying

* Get the attempt when dequeuing

* Workers do exponential backoff

* Moved todos

* DLQ functionality

* DLQ tests

* Same cluster for all keys in the same queue

* Added DLQ tests

* Whitespace

* Redis pubsub to redrive from the worker

* Fixed database paths

* Fix for path to zod-worker

* Fixes for typecheck errors, mostly with TS versions and module resolution

* Redlock required a patch

* Moved the new DB migrations to the new database package folder

* Remove the run-engine package

* Remove the RunEngine prisma schema changes

* Delete triggerTaskV2

* Remove zodworker test script (no tests)

* Update test-containers readme

* Generate the client first

* Use a specific version of the prisma package

* Generate the prisma client before running the unit tests
2024-10-08 17:41:22 +01:00
Eric Allam f9ec66c562 v3: new build system (#1265)
* upgrade @opentelemetry packages to the latest versions

* remove v2 only packages, will be moved to a dedicated repo

* remove more v2 code and run pnpm install

* use the npm yalt package in the webapp

* convert @trigger.dev/core to tshy

* Switch from jest to vitest in @trigger.dev/core

* Fixed core test

* move core-backend code into core subpath export

* convert @trigger.dev/sdk to tshy

* Removed hono

* move core-apps to core/v3/apps, remove core-apps, start converting cli-v3

* Fix up some of the commands

* cli now building and loadable

* using package-json-from-dist to get package version now in core and cli

* dev command WIP

* cleaned up some repetition and structure of the entry point stuff

* bringing back the background worker stuff

* Indexing of the v3 catalog

* getting closer to executing dev runs...

* centralize dev logging using event emitter

* Move indexing to it’s own entry point, simplify code

* dev runs working

* Get instrumentation to work with openai

* debugging achieved internally

* provide worker files as part of the worker creation on the server

* support for cjs and esm javascript

* Fixed timeout

* worker manifest now has the config path

* auto-upgrade config to non-deprecated alternatives

* Adding package preview release

* deployment WIP

* improve the syncEnvVars output and adapt resolveEnvVars

* WIP bun runtime

* WIP bun support

* seed tasks with the machine preset if listed in the config

* deploy run executions WIP, extracted TaskRunProcess into 1 place

* deployed tasks running and executing 🎉

* support for waits and better flushing & process cleanup

* Fixed the heartbeating

* Better warning messages

* Improve and unify the indexing between dev and deploy

* Support for external deps that need node-gyp to build

* build extensions can now install custom packages and run instructions in the image. Also prisma extension now works and also works with multiple schema files

* Add back in the main/types/module to sdk

* dev no longer is Ink/React, grace period for disconnections in dev

* Fix the changeset config

* More changeset fixes

* Remove config packages

* More changeset fixes

* Fixed typescript issues (needed to revert back to zod 3.22.3

* Fix pr_checks workflow

* Remove the prepare script

* Fixed tests and package versions

* Remove cli test script

* Remove packages from tailwind watch paths

* Add repo to public packages

* Just commit the generated files and do the building at dev time

* Try and get pkg.pr.new working

* Try again

* Fix emitDecoratorMetadata importing named export from typescript

* config file backwards compat with export const config

* Fixed issue where import errors weren’t coming through

* p-retry is a prod dep

* typescript needs to be a prod dependency for emitDecoratorMetadata

* Add better debug logging to help track down import-in-the-middle bug

* An external is only considered resolvable if it resolves to the same path as the collected external

* Fix runtime checks to allow >=18.20

* Move extensions to a new build package

* Fixed building packages in dockerfile

* Remove the e2e test from publish workflow for now

* Don’t treat pkg.pr.new versions has needing upgrading

* making sure config handleError works, and discovered path aliases don’t work in config files

* Strip empty string env vars so they accidentally override real values

* Couple of things

* Update version to use preview instead of beta

* Hopefully fix re-attempts with >30s delay

* Match socket emit messages to current latest in main

* Initial guide

* Go back to beta

* Go back to the preview, and update guide to use pr preview tags

* Go back to beta

---------

Co-authored-by: Matt Aitken <matt@mattaitken.com>
2024-08-23 13:10:15 +01:00
Matt Aitken c405ae7117 Schedule limits and timezone support (#1165)
* Added maximumScheduleInstancesLimit column to Org, default to 20

* Docs on the schedule limits and improved soft-limit communication

* Added limit info to the schedules list page

* Created a task that creates schedules, useful for testing

* Make deduplicationKey required when creating/updating a schedule using the SDK

* New schedule button shows an alert if you’re over the limit

* Added timezone to the form and db

* WIP on the timezone dropdown for the create/edit schedule form

* Use the new filter search for timezones

* Made the timezone dropdown faster by fixing the virtualization

* The preview table is working and added a nice message about daylight savings

* Created a page where you can view the full list of timezones

The URL is included in the error message if you send an invalid time using the SDK

* Creating tasks with the timezone

* Added timezone support the the scheduler and the schedules list

* Added timezone support to more of the schedules UI

* The timezone comes through to scheduled runs with nice JSDocs

* Allow setting the timezone from the SDK

* Always have a timezone on a schedule

* Updated jsdocs

* Updated catalog example

* Changed the column to be a string, not null. Added the timezone across the SDK

* API endpoint for getting the timezones

* Added an SDK function to get the list of timezones

* Added timezones to the docs

* Changeset: Added timezone support to schedules

* Added support for testing timezone

* Tidied up imports

* Imports

* Imports

* Update limits.mdx

* Fixed a couple type issues and use the already exported zodfetch

---------

Co-authored-by: Eric Allam <eallam@icloud.com>
2024-06-14 13:31:27 +01:00
Eric Allam a946797d95 v3: Upgrades to MarQS (#989)
* Upgrades to MarQS

- It’s “queue choosing” algorithm is now MUCH better (pretty much didn’t work before) and should be performant even when we have a bunch of prod queues
- There are now concurrency limits at the environment and organization level, as well as the task/queue level. So if any of them are at capacity the message won’t be dequeued. This means we can have an org wide concurrency limit, as well as different limits for dev/prod/staging
- I’ve added an admin API that can be used to update the org/env concurrency limits

* More MarQS upgrades

- Extract the queue priortity choosing strategy into an interface
- Implement a much better weighted average strategy
- “Slide the window” of queue candidates if the parent queue sends all at-capacity queues (so we won’t get stuck attempting to choose the same 12 full queues)
- Added some unit tests for the priority stuff
- Added some ideas for expanding the priority choosing strategy with more dynamic features

* Fixes an issue with the shared queue consumer relying on the queue name to get the env id
2024-03-31 15:58:00 +01:00
Eric Allam baf3a84cda v3 dev cli command + more (#894) 2024-02-12 14:24:32 +00:00
Eric Allam 812b5afe6f Add support for the built-in Trigger.dev tunnel for local development (#795)
🚀 Publish Trigger.dev Docker / units (push) Failing after 4s
🚀 Publish Trigger.dev Docker / e2e (push) Failing after 3s
🚀 Publish Trigger.dev Docker / typecheck (push) Failing after 13s
🚀 Publish Trigger.dev Docker / publish (push) Has been skipped
* Add support for the built-in Trigger.dev tunnel for local development

* Remove the test script from yalt

* Add to the README

* Improve the outdated packages warning
2023-12-14 15:33:56 +00:00
Eric Allam c1710ae72d fix: Buffer import breaks @trigger.dev/react (#714) 2023-10-31 15:28:55 +00:00
Eric Allam c0f9f10b07 Upgrade to Remix v2.1.0 2023-10-26 15:54:21 +01:00
Matt Aitken 711019032f Added the SDK to webapp tsconfig 2023-09-11 19:20:30 +01:00
Matt Aitken 92233f2e90 Core package and move to @latest (#235)
* Changeset should ignore the example projects

* Removed note in Contributing instructions about not adding a changeset for internal

* Renamed @trigger.dev/internal to @trigger.dev/core. Set sdk and internal to be ES2020, so we don’t get errors about private identifiers

* Env vars in nextjs-example use square bracket syntax to avoid Turbo Repo errors

* Set the tsconfigs back for core and sdk

* @examples/nextjs compile error with undefined tasks

* Set the example projects to use ES2015 to avoid private modifier complaints

* package-tester example, which will use built packages

* package-tester package.json

* Created a readme for the package-tester

* Added all the packages to package-tester

* Create .env.local.example and readme instructions

* Added name to the package.json

* Upgraded @types/react and @types/react-dom everywhrre, so we can use server actions

* Upgraded @types/react and @types/react-dom everywhrre, so we can use server actions

* Reworked the react package build, so it generates separate files

* The SDK no longer bundles core

* package-tester setup with a server action and react hooks

* Added new SDK methods with logging

* Working tsup settings for client and server

* Explicit react hooks return types

* Added all the hooks for testing

* Added OpenAI step to the job to check types are still ok in integrations

* Get rid of rogue Changeset ignores

* Changeset: @trigger.dev/core is now a separate package

* Exited prerelease mode, added a changeset

* Changed @next to @latest

* Deleted seed.js, this shouldn’t be committed

* Ignore seed.js

* The webapp was importing @trigger.dev/core with a folder path rather package name…

* More webapp imports instead of @trigger.dev/core were a folder path

* Accidentally edited Stripe internal API url in the comment

* Added “sideEffects”: false so @trigger.dev/core is tree shaken by Remix
2023-07-31 18:47:34 +01:00
Eric Allam 9f49b7f4c0 Move prisma to an internal database package 2023-06-15 14:18:34 +01:00
Matt Aitken b087b5cdee More progress on oauth and started on the api UI 2023-04-21 16:00:25 -07:00
Eric Allam 2d5c2394c8 Initial commit of the new serverless execution runtime 2023-03-31 14:32:00 +01:00
Matt Aitken a20a26e748 Updated settings for WhatsApp to match other integration packages 2023-01-31 09:53:24 +00:00
Eric Allam acf8bf038f Remove trigger-providers, trigger-integrations, and internal-integrations and replace with integrations-catalog 2023-01-30 15:58:25 +00:00
Eric Allam 3769d12bd8 Breakout shopify into it’s own package 2023-01-30 14:05:23 +00:00
Eric Allam 3759dfc434 Breakout resend into it’s own package 2023-01-30 13:52:56 +00:00
Eric Allam f9208f45a7 Breakout slack into it’s own package 2023-01-30 13:25:55 +00:00
Eric Allam 2d0e766b1a Break out github into it’s own package 2023-01-30 12:57:37 +00:00
Eric Allam 39853e5ce0 Add uptime workflow that runs every minute 2023-01-25 21:55:04 +00:00
Matt Aitken 92ae9bd894 Added tsconfig paths so we’re not linking to /dist folders anywhere 2023-01-21 13:32:14 -08:00
Eric Allam e77f259fb1 Improvements to pulsar topics and structure
- Centralized the pulsar client creation and topics to internal-pulsar
- Cleaned up the pulsar message catalogs and split triggers and run command responses
- Improved the naming of the task queue
- Figured out how to use the pulsar client with the hosted version of Pulsar
2023-01-06 19:36:38 +00:00
Matt Aitken 91bc0764d9 Magic link and welcome (without delay atm) email now sending with Resend 2023-01-04 17:13:33 +00:00
Eric Allam 67612a8e0c Registering webhooks with existing connections (and github only) 2022-12-21 16:56:32 +00:00
Eric Allam d457085f7b Workflow runs, with a workflow step for output 2022-12-20 11:14:03 +00:00
Eric Allam d8c22b61ef Workflows are now being registered in the webapp, along with their trigger 2022-12-15 16:41:08 +00:00
Eric Allam 271db37844 Fully rename to @trigger.dev 2022-12-08 13:12:07 +00:00
Matt Aitken dc2e4c3a87 Initial commit of the mono repo 2022-12-06 12:28:16 +00:00