trigger.dev@4.0.0-v4-beta.14
247 Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 Matt Aitken
|
32d2992802 | Security update of nanoid from 3.3.4 to 3.3.8 (#1887) | ||
|
Eric Allam
|
51bb3ee464 |
v4 ai improvements (#1863)
* AI SDK sql generator and approval example WIP * Create some nice spans and extract the tools out so we can get the type of the streams * Enable AI SDK telemetry * Adding ai.tool and removing toolTask, 3rd party telemetry spans now wil create partials, better ai SDK telemetry icons * Created a separate d3-chat example, split out from the openai-agents example * Fixed the tool options being passed to metadata in `ai.tool` * Add a link to the run * Slightly improved design * Add a crawler task using crawl4ai * Use a tool to get the userId * Couple of tweaks * Adding markdown rendering to assistant messages and added an e2b based chart rendering task * Backup to anthropic * Add changeset |
||
|
nicktrn
|
49a3f72e13 |
Publish redis-worker and add graceful shutdown manager (#1810)
* add shutdown manager * update ai test instructions * add shutdown timeout to redis-worker * move redis worker to packages * add unregister method * prep for publishing package * fix types * update ai files * fix cursor terminal links * prevent overly friendly ids * use structured logger * use unique shutdown handler names * rework suspend completion * add trycatch util * rework suspend restore * add http server metrics * add missing prom-client to core * add prom metrics to redis worker * bundle redis-worker * fix esm/cjs interop * remove proxy from changeset ignore and add supervisor * add pause to prerelease script for any manual edits * unregister the correct handler and add early detection * small change to http handler return * fix worker tests * fix shutdown manager tests |
||
|
Matt Aitken
|
6cf86d5916 |
Deleting v2 dashboard/api (#1628)
* Delete v2 Stripe routes * Delete v2 billing/usage pages * Delete v2 integration pages * Delete v2 project pages * Deleted a load of components and services * Deleted a load more components, presenters and services * Deleted another 100 files or so… * Removed old v2 paths * Removed named icons from form titles * Removed more string icons * Delete NamedIcon * Fixed some type errors * Delete endpointApi * Removed v2 from core/sdk * Post merge fixes * added explicit return types * using the new sdk export without v3 * Delete old v2 file * Added explicit return types because TS was complaining… * Don’t export RuntimeEnvironmentType from two core files. Was causing TS issue * Fix for removal of NamedIcon in new route * Removed strange eslintrc rule * Use the new redis client --------- Co-authored-by: James Ritchie <james@trigger.dev> |
||
|
Eric Allam
|
d855d55ea0 |
re2: env based queue selection algo (#1775)
* re2: fix @trigger.dev/core exports * re2: WIP env based queue selection algo * more wip * WIP * Get run engine tests to pass * Adding tests for the fair dequeueing strat in the run engine * Configure the new queue selection strategy in the webapp and get it all building and typechecks passing * webapp now uses built packages, building redis-worker, run-engine, database, using better tsconfig setups for tests, moving isomorphic code into core/v3/isomorphic * Fixed webapp typechecks * dev now depends on build, fixed supervisor typecheck * Fixed run engine tests * Fixed e2e tests |
||
|
Matt Aitken
|
983bb411cc |
Redis client factory (error handling and good defaults) (#1761)
* @internal/redis package for creating Redis clients with sensible defaults and error handling * Added @internal/redis to webapp |
||
|
nicktrn
|
e97704d904 |
Run Engine 2.0 (WIP) (#1575)
* bump worker version * Suggested glossary for the RunEngine, TBC * Removed BatchTaskRun changes from this branch, they were done in main * Set the BatchTaskRun status to completed when all runs are completed * When dequeuing respect passed in maxResources * Ported over the new run props: idempotencyKeyExpiresAt, versions, oneTimeUseToken, maxDurationInSeconds * Didn’t hit save… the new props when triggering tasks passed through * Idempotency expiration + waitpoint edge case * WIP on creating checkpoint, parking for now * fix worker routes * upgrade webapp node types to support generic event emitter * separate event bus handler singleton and run failure alerts * duration waits * fix execution snapshot debug spans * task waits * fix event bus types * temporary fix for react hook run handle type * disable run notifications for now * convert any typecasts to expect errors to more easily fix later * fix webapp types after node types upgrade * updateEnvConcurrencyLimits across marqs and the runqueue * Pass proper values into the run engine * RunQueue settings and removed unused rebalancing workers * Remove rebalancing prop * Tidied more things up * Update/remove queue limits for MARQS and RunQueue * taskQueue/concurrencyLimit changes ported back into the RunEngine * Reworked completing waitpoints to improve performance and reduce race conditions * Improved test robustness * Down to a single run lock only when a run is totally unblocked and ready to continue * warm starts, worker notifications, wait fixes * Fix for Run Engine poll interval env var * Expect the waitpoint to be completed quickly * If a run is locked then it’s too late to expire it * Added VALKEY_ env vars and plugged them into the run engine * Extracted and updated the guard queue function so it can be used when batching * Added logging and universal concurrency changes to trigger task v1 * Added notes back in * Bump @trigger.dev/worker to 3.3.7 * reportInvocationUsage for the runAttemptStarted event * improve execution snapshot span debug span start times * Unfriendly IDs * update lockfile * Created a shared determineEngineVersion function * disable unfinished commands * save new cli config to different location, misc fixes * add basic engine version check via current deploy * new run engine will default to node 22 runtime * block some actions for projects on previous run engine * fix worker group tests * fix triggerAndWait test * one typescript version to rule them all * redlock type patch * fix type issues caused by ts-reset * improve cleanup scripts * add missing socket.io dep * fix run notification handler type * fix worker group test again * generate prisma client for e2e tests * remove worker group tests for now * prevent image pull rate limits during unit tests * increase timeout for queue concurrency limit test * generate prisma client for preview release * same node types everywhere * Updated engine readme, removed legacy system notes * use default machine preset from platform package * worker instances plural in schema * disable pnpm update notifications * return worker group details from connect call * add workers admin route * fix heartbeat route return type * move deployment labels to core apps * refactor run controller env schema * Add firstAttemptStartedAt to TaskRun * RunEngine 2.0 batch trigger support (#1581) * Make it clear when BatchTriggerV2Service is used * Copy of BatchTriggerV2Service * WIP batch triggering * Allow blocking a run with multiple waitpoints at once. Made it atomic * Removed unused param * New batch service * Pass through the parentRunId and resumeParentOnCompletion * Use the new batch service, and correct trigger task version * Force V1 engine if using BatchTriggerV2Service, we’ve already done the check at this point * Removed the $transaction and early exit if nothing changed * Adedd a simple batch task to the hello world reference catalog * Fix for batch waits not working * Added parentRunId in a couple more places * Removed waitForBatch log * Added another parentRunId * Expanded the example to include all the different triggers * More changes to blocking to support continuing after idempotent completed runs * Fix for the wrong type when blocking a run * remove @map * optimise worker auth query * add engine version header to core api client requests * remove unique constraint for default group id * consolidate migrations * the first managed worker becomes the global default * Debug events off by default, added an admin toggle to show them * worker group name can't be an empty string * add exec helper to core * move machine resources to core * add pre-dequeue callback to determine max resources * optionally skip dequeue * bump worker package * move worker to core * fix ReadableStream type error * fix another type issue * update a few more tsconfigs * add metadata changes introduced in #1563 * Run Engine 2.0 trigger idempotency (#1613) * Return isCached from the trigger API endpoint * Fix for the wrong type when blocking a run * Render the idempotent run in the inspector * Event repository for idempotency * Debug events off by default, added an admin toggle to show them * triggerAndWait idempotency span * Some improvements to the reference idempotency task * Removed the cached tracing from the SDK * Server-side creating cached span * Improved idempotency test task * Create cached task spans in a better way * Idempotency span support inc batch trigger * Simplified how the spans are done, using more of the existing code * Improved the idempotency test task * Added Waitpoint Batch type, add to TaskRunWaitpoint with order * Pass batch ids through to the run engine when triggering * Added batchIndex * Better batch support in the run engine * Added settings to batch trigger service, before major overhaul * Allow the longer run/batch ids in the filters * Changed how batching works, includes breaking changes in CLI * Removed batch idempotency because it gets put on the runs instead * Added `runs` to the batch.retrieve call/API * Set firstAttemptStartedAt when creating the first attempt * Do nothing when receiving a BATCH waitpoint * Some fixes in the new batch trigger service… mostly just passing missing optional params through * Tweaked the idempotency test task for more situations * Only block with a batch if it’s a batchTriggerAndWait… 🤦♂️ * Added another case to the idempotency test task: multiple of the same idempotencyKey in a single batch * Support for the same run multiple times in the same batch * Small tweaks * Make sure to complete batches, even if they’re not andWait ones * Export RunDuplicateIdempotencyKeyError from the run engine * Latest lockfile * Trigger with a machine (old run engine) * RE2, allow setting machine when triggering * Fix for new glob patterns * add max run count to dequeue from version route * add worker instance name env var and header * queue consumer pre skip callback * poll for more runs after final execution errors * fix dequeue search param schema * add shortcut to debug switch * expose run engine timeouts as env vars * make warm start durations configurable * add optional status to json reply helper * fix preSkip hook, add debug logs * BLOCKED_BY_WAITPOINTS -> SUSPENDED * exit controller when run suspended * check if already replied before http reply * run controller will wait for next run after the current one is suspended * cancel run button shortcut * minimal event repository environment type * fix update metadata call * run suspension and misc fixes wip * change debug shortcut to shift + D * Started work on the Dev supervisor * Formatting * Fix for bad imports * Before rebuilding SSE * Presence updating from the CLI working via SSE * add worker notification debug logs * send run:stop when exiting run phase * skip current snapshot poll on worker notification * add more logs and route to submit run debug logs * add worker and runner ids to snapshots * improve run notification debug logs * add workload debug log route * misc run controller fixes and refactor * prevent parallel execution of critical functions * update bun to 1.2.1 * WIP with dev dequeuing * Method to convert friendlyIds to non-friendly, do nothing with actual ids * Set the engine on BackgroundWorker, lazily upgrade projects to engine V2 * Runs with ttls were getting immediately expired… oops. * Pass the Waiting for deploy reason through, so we have it on the execution snapshots * Fixed the logic for getting the right background worker for a run * Use the correct ID when dequeuing… * determineEngineVersion is now fully functional * Rate limiter ignores the dev endpoints * Retrieving a batch gives you the runIds * Set a unique version for the RE2 BatchTaskRun * add provisional changeset * The start of dev run execution is working * First dev run working * Moved the dev run controller closer to what Nick did with the managed one * export exec output type * Heartbeat fix: don’t heartbeat if _isHeartbeating == false * Dev runs get notifications, some dev bug fixes * Improved logging or dequeuing * We need to dequeue runs from the latest version too, for triggerAndWait * Ported Eric’s validateWorkerManifest with nicer errors * When flattening an idempotency key if part is undefined, return undefined * Dev logging fixes * Remove sigterm listener * Deprecating workers. Don’t specify a BackgroundWorker when dequeuing an environment * Deleted some old files. Renamed “managed” to “deploy” * When a build finishes, always copy the build dir (otherwise the first one gets trampled on by the 2nd) * Dev master queues should work differently * Deleting old workers * Added debounce function to core * Improvement to canceling * WIP on debounce canceling on socket disconnection * Added environment data to execution snapshots * Dev runs that have stalled get “Canceled” with a reason explaining why * Show CLI messaged when a connection to the platform is lost/restored * Fix TriggerTask after merge * Add trigger task v2 max attempts, replace some findUniques * Port the new queue logic to the run engine * More fixes post-merge * We weren’t setting a `retryConfig` up for the tests… it’s now required * Start the Redis worker inside the Run Engine… 🤦♂️ * Trying to make the testcontainers more reliable * Added keyPrefix: "engine:” * Badly placed bracket in trigger task * Better Redis namespacing * Fix for expired run not getting removed from the queue * Don’t create a redis client in the testcontainers, return the redisOptions instead * Cleanup redis client in the run lock tests * Fix for the RunQueue not supporting keyPrefix * Updated more of the RunQueue scripts rebalancing * Trying to make Redis more robust in the tests… * Improved test resiliciency more * Fix for delays (checkpoint check) * Increase the timeout slightly to fix ttl test * Added priority support when triggering * More wip trying to make test containers more reliable * batchTriggerAndWait test is still failing… some wip to try fix it * Fixed redis tests now we’re not providing a client * Separate Redis clients for the run engine worker/queue/runlock * Made the wait for duration test more resilient * Added idempotencyKeyExpiresAt to Waitpoints * Waitpoint timeouts and idempotency expiry * Use finishWaitpoint, removed extra worker job * Added waitpoint idempotency tests * Creating resume tokens is working * Some improvements to the resume tokens * Moved resumeTokens to just be wait functions 🥳 * Delete old RuntimeManagers * Wait for token is working * Better test for the wait tokens * Improved the test task some more * Hide the accessories in the span inspector * WIP on waitpoint inspector * WIP on complete waitpoint form * Span overview panel can be changed based on the entity type * Improved the waitpoint display * WIP on completing waitpoint form * Use the existing CodeBlock for the tip * Style improvements * Complete waitpoint * All waitpoint sidebar variants * Waits now use a pause icon * Durations waits use the API to create/block with a waitpoint, not the runtime * Fix for engine.blockRunWithWaitpoint required org id * Removed old wait code from the run controllers/task run process * Form action for skipping a datetime waitpoint * Move testDockerCheckpoint to a separate core package export (it can’t be bundled on the client) * Fix for glitchy hourglass animation * Completed waitpoints display better * Increase Redis maxRetriesPerRequest to 20 (default) * Completing and skipping waitpoints is working * Remove the database prisma dev command, since we need to use create only now. Updated docs * Added skip timeout, reworked the UI * Tweaked spacing * Added payload limit to waitpoint token completion from dashboard * Test idempotency works on wait.for and wait.until * Moved the worker-actions to /engine/ from /api/ * Moved dev engine endpoints to /engine/ from /api/ * Separate /engine/ rate limiter * Added parallel wait prevention, it’s working for duration waits but not well for triggerAndWait yet * WIP post-merge conflicts * Set taskEventStore column in the new engine * Remove duplicate keys * Post-merge fixes * Fix for span merge layout * Use executedAt instead of firstAttemptStartedAt --------- Co-authored-by: Matt Aitken <matt@mattaitken.com> |
||
|
Eric Allam
|
a2c2d920b7 |
Use redis worker for run heartbeats and alerts (#1669)
* Move the task run heartbeats to RedisWorker * Move alerts to redis worker, improving redis worker * Fix typecheck errors * Use single threaded tests for redis worker * Enable/disable the redis workers independently * Remove preview release from PR checks |
||
|
Eric Allam
|
f6eeaa1001 | Fix broken cloud deploys by using depot ephemeral registry, skip the registry proxy (#1637) | ||
|
Eric Allam
|
eaf46bac58 |
engine v1 improved dequeue selection algorithm (#1632)
* engine v1 fairness improvements (env based queue selection) * Add ability to bias env selection by concurrency limit and capacity * Add some randomization to the list of queues within each evn * Configure biases using env vars * Typo * revert v3 catalog project ref change * Fix v2 marqs |
||
|
Eric Allam
|
34f8bd588e |
Add ability to update parent and root run metadata from children (#1563)
* Increase the number of active streams from 2 to 5 * WIP parent metadata updates * Fix noop metadata manager * Implement run metadata updates from ancestor tasks * Add changeset * Add ability to stream into parent and root task runs * Don't duplicate exporting run metadata types * Add ability to fetch streams through metadata * Couple of fixes from CodeRabbit * Fix metadata tests * Make sure streams are subscribed in the "background" * Move the stream subscription stuff to the API client, expose it through `runs.fetchStream` * Fixed run stream tests |
||
|
Eric Allam
|
7d4708839f |
Upgrade zod to 3.23.8 (#1484)
* Upgrade to zod 3.23.8 * Add changeset |
||
|
James Ritchie
|
1b3a2f2e5b |
Full width tables (#1474)
* Updated styling * Updated the main Table component styles to the new design * Run page table layout style updates * Updated row styling and new scrolling behaviour * Fixed bulk action bar no appearing * Improved table component styles * Pagination arrows now chevrons * Added link to Runs docs page * Small popover tweak * Moved the extra Test button out of Tables and into the RunsTable * WIP refactoring the sticky table cell * Support for the existing way we have menus in the sticky table cells * Created a specific Docs style button * Improved dialog trigger button styles * Improved docs button styles * Updated Docs buttons to the new style in the tooltips * Adding a Test button on hover * Removed the page padding and made table scroll * Fixed table scrolling * cleaned up imports * WIP adding a test button that links to the test page * added a docs button to the header * Much nicer display bar for showing how many schedules you have (and it works with the updated table) * Fixed the button types in the schedules info panel * Fixed button icon colour * Removed the pagination as it’s not being used * Delete modal button position updated * Fixed padding on recent payloads panel * Fixed button icon colours * Better style for active table row * Table scrolling now behaves properly * environment buttons now have a hover state * Bigger Run test bar to match the Run page * Small layout fixes to the schedule side panel * Test page now has link to docs * fixed text alignment with table headers * API keys page now has full width tables * Fix typo and modal layout * Table uses new sticky button type * Tidy imports * Env var table now full width * Form buttons have a border top * Updated button styles * Updated blank state message style to be consistent * Removed unnecessary form button border * Alerts page tables now full width * Allowing the Detail cell to be styled * Updated the page structure to be the same as the scheduled page – with better progress indicator * Concurrency limits page now full width * Fixed function name label extending too far * Improved the button sizes on the replay modal * Added a minimal style to the info panels * Make sure you can always see the info and unlock staging info when the table scrolls * Info panel style updated to match the others * Removed the duplicate header * Usage page tables now full width * Personal access token page using full width tables * Small CSS fixes * Deployments page table now full width * Deploy page scrolls properly and pagination stays on screen if table is scrollable * TableCellMenu using the correct popover * Added missing button variant * Button group is now named so it doesn’t cause conflicts * Added docs link to the Run page * Typo * Added a link to the test page if you have no runs * Make the table sticky cell buttons more obvious * Env var page now using the new popover menu * Improved the menu items * Copy tweak * Made the icons bright on hover * Updated the cancel run dialog * Improved the API keys sticky menu behaviour * Alerts table menu using the new popover * Correct sized buttons in the Env Var page table menu * Fixed weird Dialog text styling * Schedules page blank state using correct docs button style * Copy tweak * Menu buttons styled correctly * Added close buttons to the deploy dialogs * Made the Dialog footer match the Form footer style * Fixed feedback form buttons * Fixed cancel run dialog * Test page uses the taskIdentifier in the path * Test task button working * Added Spline component for 404 logo to remove spline logo * Removed todo comment * Added a docs button to the PAT page --------- Co-authored-by: Matt Aitken <matt@mattaitken.com> |
||
|
nicktrn
|
212f8539c3 |
Automatically reattempt after internal errors (#1424)
* refactor finalize run service * refactor complete attempt service * remove separate graceful exit handling * refactor task status helpers * clearly separate statuses in prisma schema * all non-final statuses should be failable * new import payload error code * store default retry config if none set on task * failed run service now respects retries * fix merged task retry config indexing * some errors should never be retried * finalize run service takes care of acks now * execution payload helper now with single object arg * internal error code enum export * unify failed and crashed run retries * Prevent uncaught socket ack exceptions (#1415) * catch all the remaining socket acks that could possibly throw * wrap the remaining handlers in try catch * New onboarding question (#1404) * Updated “Twitter” to be “X (Twitter)” * added Textarea to storybook * Updated textarea styling to match input field * WIP adding new text field to org creation page * Added description to field * Submit feedback to Plain when an org signs up * Formatting improvement * type improvement * removed userId * Moved submitting to Plain into its own file * Change orgName with name * use sendToPlain function for the help & feedback email form * use name not orgName * import cleanup * Downgrading plan form uses sendToPlain * Get the userId from requireUser only * Added whitespace-pre-wrap to the message property on the run page * use requireUserId * Removed old Plain submit code * Added a new Context page for the docs (#1416) * Added a new context page with task context properties * Removed code comments * Added more crosslinks * Fix updating many environment variables at once (#1413) * Move code example to the side menu * New docs example for creating a HN email summary * doc: add instructions to create new reference project and run it locally (#1417) * doc: add instructions to create new reference project and run it locally * doc: Add instruction for running tunnel * minor language improvement * Fix several restore and resume bugs (#1418) * try to correct resume messages with missing checkpoint * prevent creating checkpoints for outdated task waits * prevent creating checkpoints for outdated batch waits * use heartbeats to check for and clean up any leftover containers * lint * improve exec logging * improve resume attempt logs * fix for resuming parents of canceled child runs * separate SIGTERM from maybe OOM errors * pretty errors can have magic dashboard links * prevent uncancellable checkpoints * simplify task run error code enum export * grab the last, not the first child run * Revert "prevent creating checkpoints for outdated batch waits" This reverts commit |
||
|
Eric Allam
|
2d8a41b18b |
feat: realtime (#1402)
* Denormalize run tags, increase character limit to 128 * WIP realtime subscribing to runs * extracted the stream stuff into core, made it more reusable * WIP tags * Remove tags for now because it’s not support in electric * Support async iterables, readable stream, and callback style subscription styles * Remove tags streaming endpoint * Add realtime rate limits and scope them to the /realtime path * WIP rate limt per org * Introduce per org rate limits * WIP JWT auth * Move migrations into new internal db package * Resolve pnpm lock file * Authenticating to the realtime API with JWTs are working * realtime in the client * Created react-hooks package and starting to move stuff in there * Improve types for hooks * schema tasks * Added useBatch hook * build uploadthing/fal demo and change how run metadata is synced to the server * tweaks * WIL realtime concurrency tracking * Implement test for realtime client using testcontainers also updated electric to latest version * Allow customizing the expiration time of the automatic JWT created after triggering a task * Add support for subscribing to run tags * Improve auth types and API * finalize the realtime API * Fixed some example stuff * Allow up to 10 run tags * Remove core from docker-provider tsconfig paths to prevent it from being typechecked * do the same for the kubernetes provider * Fixing some typecheck errors * Fix webapp type errors * Update @trigger.dev/platform to 1.0.13 * Fix attw error * Remove from/to in subscribeToRuns query params * Add tests for the rate limit middleware and add custom JWT rate limits * turn off webapp test parallelism * Finish renaming jwt -> publicAccessToken and automatically give the JWT read access to the tags when using trigger * Add changeset * Attempt to fix unit tests in CI * Skip running the auth rate limit middleware tests for now * Try a beefier machine * Try and run webapp tests separately * Setup env vars * Make sliding window test more reliabile |
||
|
Matt Aitken
|
3d67bb81e9 | Removed run-engine from the webapp package.json/tsconfig | ||
|
Matt Aitken
|
f2babbf637 |
Internal packages (testcontainers, redis-worker and zod-worker) (#1392)
* Some notes on the new run engine * lockfile with setup for the run engine * Documenting where TaskRun is currently mutated, to try figure out the shape of the new system * Added notes about how triggering currently works * Details about when triggering happens * Lots of notes about waitpoints * Started scaffolding the RunEngine * Sketch of Prisma waitpoint schema while it’s fresh in my mind * Got Prisma working with testcontainers * Use beforeEach/afterEach * Simple Prisma and Redis test * Return Redis options instead of a client * Simplified things * A very simple FIFO pull-based queue to check the tests working properly * Use vitest extend * Separate redis, postgres and combined tests for faster testing * Some fixes and test improvements * Pass a logger into the queue * A queue processor that processes items from the given queue as fast as it can * Test for retrying an item that wasn’t processed * First draft of waitpoints in the Prisma schema * Remove the custom logger from the test * Added a completedAt to Waitpoint * Notes on the flow for an execution starting * Added redlock, moved some files around * Starting point for the TaskRunExecutionSnapshot table * Added relationships to TaskRunExecutionSnapshot * Change some tsconfig * Moved some things around * Added some packages * WIP on the RunQueue * Fix for some imports * Key producer with some tests * Removed the nv type from the keys… it’s not useful to do global queries * Passing unit tests for all the public key producer functions * Some basic tests passing for the RunQueue * Simple enqueue test working * Enqueue and dequeue for dev is working * Don’t log everything during the tests * Enqueuing/dequeuing from the shared queue is working * Tests for getting a shared queue * The key producer sharedQueue can now be named, to allow multiple separate queues * The key producer uses the name of the queue as the input * Extra info in the Prisma schema * Dequeuing a message gets the payload and sets the task concurrency all in one Lua script * Adding more keys so we can read the concurrency from the queue * Setting the concurrency with dequeue and enquque is working * Improved the tests and fixed some bugs * Acking is resetting the concurrencies * Check the key has been removed after acking * Nacking is working * Changed the package to CommonJS + Node10 so it works with Redlock * Moved the database, otel and emails packages to be in internal-packages * Moved some Prisma code to the database package * Started using the RunEngine for triggering * Progress on run engine triggering, first waitpoint code * Create a delay waitpoint * Moved ZodWorker to an internal package so it can be used in the run engine as well as the webapp * Web app now uses the zod worker package * Added parseNaturalLanguageDuration to core/apps * internal-packages/zod-worker in the lockfile * Pass in the master queue, remove old rebalance workers code * Add masterQueue to TaskRun * Fixed the tests * Moved waitpoint code into the run engine, also the zod worker * Completing waitpoints * An experiment to create a new test container with environment * More changes to triggering * Started testing triggering * Test for a run getting triggered and being enqueued * Removed dequeueMessageInEnv * Update dev queue tests to use the shared queue function * Schema changes for TaskRunExecutionSnapshot * First execution snapshot when the run is created. Dequeue run function added to the engine * Separate internal package for testcontainers so they can be used elsewhere * Remove the simple queue and testcontainers from the run-engine. They’re going to be separate * Fix for the wrong path to the Prisma schem,a * Added the testcontainers package to the run-engine * redis-worker package, just a copy of the simple queue for now * The queue now uses Lua to enqueue dequeue * The queue now has a catalog and an invisible period after dequeuing * Added a visibility timeout and acking, with tests * Added more Redis connection logging, deleted todos * Visibility timeouts are now defined on the catalog and can be overridden when enqueuing * Dequeue multiple items at once * Test for dequeuing multiple items * Export some types to be used elsewhere * Partial refactor of the processor * First stab at a worker with concurrency and NodeWorkers * Don’t have a default visibility timeout in the queue * Worker setup and processing items in a simple test * Process jobs in parallel with retrying * Get the attempt when dequeuing * Workers do exponential backoff * Moved todos * DLQ functionality * DLQ tests * Same cluster for all keys in the same queue * Added DLQ tests * Whitespace * Redis pubsub to redrive from the worker * Fixed database paths * Fix for path to zod-worker * Fixes for typecheck errors, mostly with TS versions and module resolution * Redlock required a patch * Moved the new DB migrations to the new database package folder * Remove the run-engine package * Remove the RunEngine prisma schema changes * Delete triggerTaskV2 * Remove zodworker test script (no tests) * Update test-containers readme * Generate the client first * Use a specific version of the prisma package * Generate the prisma client before running the unit tests |
||
|
nicktrn
|
c5488df8ca |
Fix CLI downgrade check (#1380)
* fix downgrade detection * remove unused semver package from webapp * add changeset |
||
|
Eric Allam
|
ba3c5bdf33 |
Adding missing task run hierarchy to TaskRun table (#1332)
* Add task run hierarchical relationships to the database * Add depth and related runs to the retrieve run API response * Remove prisma optimize * restructure the migrations to create the index concurrently * Delete these tsbuildinfo files * Fix type error by adding depth to the run list presenter * Cleanup the task hierarchy, share more code * Remove some fields from the list run response |
||
|
Matt Aitken
|
9a7ad926aa |
ElectricSQL run page (hidden page for now) (#1297)
* WIP on using react-window-splitter
* WIP with new resizable panels and SSR
* Use the cookie package
* Resizable storybook page
* Increase indexing memory limit
* Fixed v2 usage meter displaying when on paid plan (#1255)
* Fixed v2 usage meter displaying when on paid plan
* Show the free usage panel only for v3 projects
* Concurrency page and more accurate tracking (#1252)
* Initial TaskRunConcurrencyTracker implementation
* MARQS calls a subscriber to events
* When enqueuing add the extra required metadata
* Track concurrency per environment for tasks too
* Admin page for global concurrency
* Use the new concurrency tracker on the tasks page
* Useful performance test task
* getAllTaskIdentifiers()
* New page for concurrency
* BackgroundWorkerTask index for quick lookup of task identifiers
* Added a way to get concurrency for environments
* Added upgrade/request more concurrency button
* Queued task column working
* Use defer and suspense
* Added queue column to the concurrency environments table
* Some comments added for clarity
* Fixed bad log message
* Sidemenu: move lower and rename to “Concurrency limits”
* Only show the environments, not tasks. Renamed to “Concurrency limits”
* v3: fix unfreezable state crashes for runs with multiple waits (#1253)
* support named capture groups
* write crash errors to attempt.error
* make restored pod names unique per checkpoint
* use last eight characters of checkpoint id instead
* add more chaos monkey env vars
* Ignore unfreezable states
* prevent excessive queue config parsing errors
* handle dependency resume edge case
* better entry point logging
* ignore checkpoint cancellation timeouts
* add missing idempotency keys to wait for dep replays
* remove checkpoints between attempts
* fix retry container names on kubernetes
* add changeset
* fix types
* bring back internal duration timers
* Added more logging to TaskRunConcurrencyTracker and some more try/catches
* Call subscriber.messageDequeued in dequeueMessageInSharedQueue
* Added messageReplaced to concurrency tracking (when freezing)
* Added depenenciesToBundle guide to bundle all packages
* Include the old message data when replacing, so we get the projectId etc.
* Fix restored container names
* Fix for schedule page not scrolling
* Added a description panel to the Concurrency admin page
* chore: Update version for release (beta) (#1256)
Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>
* Release 3.0.0-beta.53
* Added a note to use batchTrigger() instead of trigger()
* The latest react-window-splitter fixes the ESM issues
* Set sensible defaults for the run page
* Deployments page
* Test page
* Schedules page
* Latest version of react-window-splitter (0.2.5)
* Updated to the latest version: react-window-splitter
* Callout if runs don’t start right away now has some top margin
* Small padding fix
* styled the handle focus state
* Added isStaticAtRest prop to resizable panel
* Updated resizable storybook
* Inline code blocks behave nicer when text wraps
* Added ElectricSQL to docker-compose, available on 3060
* Extracted some logic out of the eventRepository for getting a trace. This will be used on the frontend
* Use the new util
* More restructuring ready to use the trace summary from the frontend
* Using ElectricSQL for the run page data
* Min size for resizable panel on test page
* Don’t load the trace in the RunPresenter anymore
* Fix for the resizable panels on the run page
* Added overflow hidden to the panel group
* min size for the test page left hand panel
* Updated to latest window-splitter version
* Removed unused const
* One fix for client-server mismatch
* Slight improvement in the loading state
* Restructured the page so the loading is better
* Improvement to the loading states
* Improved the loading behaviour with the inspector
* WIP on auth, having problems with it
* Upgrade Remix to 2.9.1 (same as PR #1096)
* Switched structure around again so we only call the useTrace hook from the client
* Added auth to the sync
* Overscan more rows in the tree view
* Fix for TS error
* Remove duplicate import
* Revert "Upgrade Remix to 2.9.1 (same as PR #1096)"
This reverts commit
|
||
|
Eric Allam
|
87e46d497b | projects now have a dedicated build server (#1281) | ||
|
Matt Aitken
|
09e6f58e84 |
Revert "ElectricSQL run page and react-window-splitter (#1277)"
This reverts commit
|
||
|
Matt Aitken
|
5231381717 |
ElectricSQL run page and react-window-splitter (#1277)
* WIP on using react-window-splitter
* WIP with new resizable panels and SSR
* Use the cookie package
* Resizable storybook page
* Increase indexing memory limit
* Fixed v2 usage meter displaying when on paid plan (#1255)
* Fixed v2 usage meter displaying when on paid plan
* Show the free usage panel only for v3 projects
* Concurrency page and more accurate tracking (#1252)
* Initial TaskRunConcurrencyTracker implementation
* MARQS calls a subscriber to events
* When enqueuing add the extra required metadata
* Track concurrency per environment for tasks too
* Admin page for global concurrency
* Use the new concurrency tracker on the tasks page
* Useful performance test task
* getAllTaskIdentifiers()
* New page for concurrency
* BackgroundWorkerTask index for quick lookup of task identifiers
* Added a way to get concurrency for environments
* Added upgrade/request more concurrency button
* Queued task column working
* Use defer and suspense
* Added queue column to the concurrency environments table
* Some comments added for clarity
* Fixed bad log message
* Sidemenu: move lower and rename to “Concurrency limits”
* Only show the environments, not tasks. Renamed to “Concurrency limits”
* v3: fix unfreezable state crashes for runs with multiple waits (#1253)
* support named capture groups
* write crash errors to attempt.error
* make restored pod names unique per checkpoint
* use last eight characters of checkpoint id instead
* add more chaos monkey env vars
* Ignore unfreezable states
* prevent excessive queue config parsing errors
* handle dependency resume edge case
* better entry point logging
* ignore checkpoint cancellation timeouts
* add missing idempotency keys to wait for dep replays
* remove checkpoints between attempts
* fix retry container names on kubernetes
* add changeset
* fix types
* bring back internal duration timers
* Added more logging to TaskRunConcurrencyTracker and some more try/catches
* Call subscriber.messageDequeued in dequeueMessageInSharedQueue
* Added messageReplaced to concurrency tracking (when freezing)
* Added depenenciesToBundle guide to bundle all packages
* Include the old message data when replacing, so we get the projectId etc.
* Fix restored container names
* Fix for schedule page not scrolling
* Added a description panel to the Concurrency admin page
* chore: Update version for release (beta) (#1256)
Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>
* Release 3.0.0-beta.53
* Added a note to use batchTrigger() instead of trigger()
* The latest react-window-splitter fixes the ESM issues
* Set sensible defaults for the run page
* Deployments page
* Test page
* Schedules page
* Latest version of react-window-splitter (0.2.5)
* Updated to the latest version: react-window-splitter
* Callout if runs don’t start right away now has some top margin
* Small padding fix
* styled the handle focus state
* Added isStaticAtRest prop to resizable panel
* Updated resizable storybook
* Inline code blocks behave nicer when text wraps
* Added ElectricSQL to docker-compose, available on 3060
* Extracted some logic out of the eventRepository for getting a trace. This will be used on the frontend
* Use the new util
* More restructuring ready to use the trace summary from the frontend
* Using ElectricSQL for the run page data
* Min size for resizable panel on test page
* Don’t load the trace in the RunPresenter anymore
* Fix for the resizable panels on the run page
* Added overflow hidden to the panel group
* min size for the test page left hand panel
* Updated to latest window-splitter version
* Removed unused const
* One fix for client-server mismatch
* Slight improvement in the loading state
* Restructured the page so the loading is better
* Improvement to the loading states
* Improved the loading behaviour with the inspector
* WIP on auth, having problems with it
* Upgrade Remix to 2.9.1 (same as PR #1096)
* Switched structure around again so we only call the useTrace hook from the client
* Added auth to the sync
* Overscan more rows in the tree view
* Fix for TS error
* Remove duplicate import
* Revert "Upgrade Remix to 2.9.1 (same as PR #1096)"
This reverts commit
|
||
|
Eric Allam
|
f9ec66c562 |
v3: new build system (#1265)
* upgrade @opentelemetry packages to the latest versions * remove v2 only packages, will be moved to a dedicated repo * remove more v2 code and run pnpm install * use the npm yalt package in the webapp * convert @trigger.dev/core to tshy * Switch from jest to vitest in @trigger.dev/core * Fixed core test * move core-backend code into core subpath export * convert @trigger.dev/sdk to tshy * Removed hono * move core-apps to core/v3/apps, remove core-apps, start converting cli-v3 * Fix up some of the commands * cli now building and loadable * using package-json-from-dist to get package version now in core and cli * dev command WIP * cleaned up some repetition and structure of the entry point stuff * bringing back the background worker stuff * Indexing of the v3 catalog * getting closer to executing dev runs... * centralize dev logging using event emitter * Move indexing to it’s own entry point, simplify code * dev runs working * Get instrumentation to work with openai * debugging achieved internally * provide worker files as part of the worker creation on the server * support for cjs and esm javascript * Fixed timeout * worker manifest now has the config path * auto-upgrade config to non-deprecated alternatives * Adding package preview release * deployment WIP * improve the syncEnvVars output and adapt resolveEnvVars * WIP bun runtime * WIP bun support * seed tasks with the machine preset if listed in the config * deploy run executions WIP, extracted TaskRunProcess into 1 place * deployed tasks running and executing 🎉 * support for waits and better flushing & process cleanup * Fixed the heartbeating * Better warning messages * Improve and unify the indexing between dev and deploy * Support for external deps that need node-gyp to build * build extensions can now install custom packages and run instructions in the image. Also prisma extension now works and also works with multiple schema files * Add back in the main/types/module to sdk * dev no longer is Ink/React, grace period for disconnections in dev * Fix the changeset config * More changeset fixes * Remove config packages * More changeset fixes * Fixed typescript issues (needed to revert back to zod 3.22.3 * Fix pr_checks workflow * Remove the prepare script * Fixed tests and package versions * Remove cli test script * Remove packages from tailwind watch paths * Add repo to public packages * Just commit the generated files and do the building at dev time * Try and get pkg.pr.new working * Try again * Fix emitDecoratorMetadata importing named export from typescript * config file backwards compat with export const config * Fixed issue where import errors weren’t coming through * p-retry is a prod dep * typescript needs to be a prod dependency for emitDecoratorMetadata * Add better debug logging to help track down import-in-the-middle bug * An external is only considered resolvable if it resolves to the same path as the collected external * Fix runtime checks to allow >=18.20 * Move extensions to a new build package * Fixed building packages in dockerfile * Remove the e2e test from publish workflow for now * Don’t treat pkg.pr.new versions has needing upgrading * making sure config handleError works, and discovered path aliases don’t work in config files * Strip empty string env vars so they accidentally override real values * Couple of things * Update version to use preview instead of beta * Hopefully fix re-attempts with >30s delay * Match socket emit messages to current latest in main * Initial guide * Go back to beta * Go back to the preview, and update guide to use pr preview tags * Go back to beta --------- Co-authored-by: Matt Aitken <matt@mattaitken.com> |
||
|
Matt Aitken
|
6293290fb9 | Better prediction of usage (uses Linear regression) | ||
|
Jacob Paris
|
d934feb02a |
feature: add no-trigger-core-import (#1175)
* feature: add no-trigger-core-import * fix: handle aliases and limit to exported modules * fix: use longest export map name * fix: support export type and interface * fix: simplify message when export map is needed * feature: fine grained modules for core * fix: add prerequisite lint plugins * fix: why do I have to fix these components again??? * fix: tests * Add eslint config and vscode settings * Create yellow-roses-arrive.md * Remove the vscode linting This is temporary so no one fixes these until we merge the remix branch --------- Co-authored-by: Eric Allam <eric@trigger.dev> Co-authored-by: Eric Allam <eallam@icloud.com> Co-authored-by: Matt Aitken <matt@mattaitken.com> |
||
|
Matt Aitken
|
72f3a4b128 |
Usage and billing (v3) (#1201)
* v3 subscription endpoints * Use pnpm linked billing package during development * Moved v2 billing components into a subfolder * Select plan using the real data * Improved v3 plan display * Use new api response that doesn’t require a Stripe call * Free flow is working * Added GitHub modal and verified badge * Deleted old request v3 access component/route * Allow setting classes on the Tooltip button * Paid plans working * Redirect from select plan if you’ve got v3 enabled * Loading state improvements * New admin API endpoint to set concurrency across multiple environments * When projects are created, conditionally create staging based on the plan * New billing page working with side menu and stripe portal * Layout, formatting and some plan state improvements/fixes * Don’t show the period if you’re on the free plan * Temporary upgrade callout * Refactored the platform code so it’s easier to call and doesn’t require a isManagedCloud check * Send taskIdentifier to OpenMeter * Side menu * Upgrade prompts * More improvements to the app-wide usage indicators * Early work on usage graphs * Added the usage bar for v3 * Moved code to presenter and now using defer * Added the tasks table to usage * Improved the v3 usage bar if theres’ no usage on a paid plan * If no run data, still render a graph * Usage page errors when defered loading fails * Don’t show the public API key for v3, they’re not used and probably never will be * Improved the upgrade callout and API keys page layout * Only show the “reveal all” toggle if you have environment variables in the table * Replaced Upgrade callout with a more generic InfoPanel component * better panel width * Show conditional upgrade prompts based on plan and number of schedules used * Removed duplicate class * Wider blank state panels for the scheduled page * Wider info panel for the env var page * Blank state now using the info panel * Platform alerts prompt now using the InfoPanel * Deploy blank state uses InfoPanel * Github verified badge padding adjustment * Improved the layout of the page, some style tweaks, organized imports * Better default tooltip style * could be undefined fix * text fix + style updates * Changed the billing icon in the side menu * Billing page layout and style improvements * plan tooltips don’t use dark variant * Don’t highlight the plan on the billing page * Tooltip underlines stand out more * Fixed padding in the PageTitle * Tooltips use the correct cursor * Improved the plan banner on the billing page * Fixed Header1 inconsistent font weight * Fixed issue where input field focus states were being clipped * Fixed large button not having large text size * Added a link to the Get in touch copy and improved the connect to GitHub modal * Select plan page uses the MainCenteredContainer * Better logging from the Loops endpoint because this error finally got hit * Move the ingestion of compute to the platform * Reporting usage of invocations moved to the platform * Get the entitlement before triggering a non-dev task * Contact us enterprise plan button opens the feedback form * Removed Github discussions link from the Feedback panel * Swapped billing icon for credit card * Show a Unlock staging panel on the env var page * Updated staging environment colour * Show a prompt to upgrade to get staging in the new env var modal * Improved the edit env var modal * Implement ability to disable org concurrency * Use common logic for the plans * Use the billing server to get the schedule limits * Some schedules page fixes * More convenient way of getting a limit * Use the new schedule limit * Team member limiting * Made the limit visible on the team page * Limit alerts * Added an index for TaskRun.scheduleId * Remove console.log on schedules page * Added durations to the run table * Tabular numbers * Improved the usage page formatting * Only admins see the compute column on the run table * Include the base cost on the usage stats * Moved the status to the sidebar * Optional table header tooltip * Allow InfoIconTooltips to have customizable content styles * Added a tooltip to the duration header, changed no test to a dash * Removed all references to signing up to v3 from the docs * Switched @trigger.dev/billing to @trigger.dev/platform * Passing up the variant for the InfoIconTooltip * table tooltip max-width fixed * Switch to the published @trigger.dev/platform 1.0.11 * v2 usage page title changed to include “v2" * code theme has a transparent background so it works on any background * duration columns now grouped together nicely at wide screen size * Last duration column fills the width properly * Fix for the per run price being in cents not dollars * Show the total cost with 8 decimal places * Show 8 decimal places in the usage graph tooltip * Moved the UpgradePrompt to the v3 folder * Prepare to use Shadcns chart helpers * Much nicer chart * Small tweaks to the graph * Fix run table col spans for empty/loading messages * We don’t need isManagedCloud in createProject * Hide v3 usage/billing pages if there aren’t v3 projects in your org * Removed unused tooltipStyle * Usage bar now says “Included usage” instead of “Tier limit” if you’re paying * Get the plan/usage data in parallel * The usage page now has a month dropdown and all data is for that calendar month * Ensure the passed date is the 1st of the month * Use the machine presets from the platform package --------- Co-authored-by: James Ritchie <james@jamesritchie.co.uk> Co-authored-by: Eric Allam <eallam@icloud.com> |
||
|
Eric Allam
|
85a543d8ec |
v3: usage tracking (#1163)
* Starting to measure wall time and cpu time in the workers, and reporting that via otel and to completed task run attempts * Move usage tracking outside of the executor * WIP prod usage tracking * WIP * WIP custom fetch to openmeter * Create a usage client * WIP * WIP * Implement new machine preset stuff and send usage reports to OpenMeter from webapp * WIP * Expose usage info to the client * Add usage and cost to TaskEvent * Add ability to globally configure the task machine preset * Report start run usage * Change the machine docs to use presets * setExpirationTime to 24h * Removed logs * Update machines.mdx * Removed console.logs * Handle revalidating JWT tokens * Couple tweaks --------- Co-authored-by: Matt Aitken <matt@mattaitken.com> |
||
|
Matt Aitken
|
bc7bbd4576 |
v2 IngestSendEvent rate limit (#1134)
* Easier to create a rate limiter, use it in the ApiRateLimiter. Upgraded the Upstash package * Always prefix any rate limiter in Redis with “ratelimit:” * By default log when the rate limit is hit * Added rate limiting to IngestSendEvent * Log out the EventRecord id * Increase events.deliverScheduled attempts * INGEST_EVENT_RATE_LIMIT_MAX is optional * Removed old API rate limit code * IngestSendEvent rate limiter is optional. Moved outside of the DB transaction * Log a message out when the rate limiter is created * Return undefined if the rate limit has been crossed |
||
|
Jacob Paris
|
7cb20fbead |
fix: upgrade prism react renderer (#1111)
* fix: upgrade prism react renderer * Update CodeBlock.tsx |
||
|
Eric Allam
|
3f8b6d8fce | v2: Better handle recovering from platform communication errors by auto-yielding back to the platform in case of temporary API failures | ||
|
nicktrn
|
1642fd7baf |
perf: migrate to graphile worker v0.16.6 (#1097)
* migrate to graphile worker v0.16.6 * remove stale docs link * fix jobs cleanup query |
||
|
Eric Allam
|
0e919f56f2 |
Alerts v1 (#1065)
* Alerts v1 * Encrypt alert webhook secrets and allow them to be generated by the server * Alert v1 UI * Remove unnecessary emails * Move to using `@react-email/components` * WIP slack alerts * More slack alerts WIP * Update pnpm lock after rebase * Finish implementing Slack alerts * Use a more error like emoji * New secondary variant for the segmented control * Added a simple checkbox style variant to storybook * Style tweak to the segmented control * UI improvements to the alert modal * Use searchable Select for alerts. Changed default variant for SegmentedControl * Secondary button now using secondary colour * segmented control style tweak * Improved the channel column in the alerts table * Updated logo-mono.png * Updated email styles * Email templates updated to new styles * Don’t log the decrypted secret * await enqueing the deployment alert when an index fails * await enqueing the timeout alert --------- Co-authored-by: James Ritchie <james@jamesritchie.co.uk> Co-authored-by: Matt Aitken <matt@mattaitken.com> |
||
|
Matt Aitken
|
7d6430dc46 |
New filtered dropdown and run filters (#1089)
* Extract out useFilterTasks * WIP on Combobox * Revert "WIP on Combobox" This reverts commit f0a9b2e57c67f24d5e0876630e17a54252ceb86b. * Added ariakit * Started adding a Listbox component * Listbox WIP * More WIP on the list box * More work on the Listbox * WIP on filterable Listbox using headless… but it’s going to be a nightmare to make it good. Wrong approach * WIP using Ariakit * Multiple selection working * WIP in reworking it so search is built in and it’s easier to use * Nicer API for listbox * Allow sections to be rendered with filtering * More progress on the styling of the menu * The empty state * Removed focus outline * Allow filtering based on the section title * Render the shortcut correctly * Allow passing static children to Select and render a separator if a section has no title * Static example and added subtle focus * Allow static data with no items array passed in * Allow shortcuts on select items * Only how heading if true * Tooltip for keyboard shortcut * useShortcutKeys fixes - Can be passed an undefined definition (so we render the same number of hooks always). - Fix for enabledOnInputElements not being used… * Shortcut keys working with lists * Don’t render shortcut numbers higher than 0 * More improvements * WIP on a Link row * Nicer API for the children function * Much nicer shortcut key support * Separated SelectTrigger out * Heading separated out * More styled elements created * Removed unused element * Reordered statuses * Added search context * Fixes for default values * Removed tab elements * Combo box * Simplified it more * Started work on Filter menu * Experiments with filter menu * Filter menu proof of concept working * Nice hook for using a search param * The escape key goes back to the previous menu * Added filter page to storybook * Added other variants * Renamed Select file to OldSelect, implemented Feedback form * Added placeholder support * Added an optional dropdown icon (defaults to a chevron down) * Feedback new Select * Feedback just use static items * New project page * Renamed storybook pages * Lots of progress on the new run filters * Added new tasks filters * Show the applied filters * Added period filtering * Fix for escape resetting the period value to the previous value when escape is pressed * Show the Clear button sooner * Removed unused code * Reworked useSearchParams so multiple values can be set at once… * Renamed file: OldSelect -> SImpleSelect * Fix for new Select not working in forms because props weren’t being passed through * New Select on the schedules page * Better layout when there are lots of run filters * Many style improvements to run filters * Status filter allows you to select as well * Rolled out the menus for the other files * Minor improvements to schedule page and page padding * Latest lockfile |
||
|
Matt Aitken
|
0f0a6884e8 |
Environment variables pasting uses dotenv (#1075)
* Add dotenv package to the webapp frontend, required some polyfills * Use dotenv to parse the pasted env vars. Make the panel wider on larger screens |
||
|
Matt Aitken
|
7ff8f0ebab |
Task and run page improvements (#1076)
* TaskListPresenter: if there are no tasks then don’t do stats queries * RunListPresenter, use BasePresenter and the read replica * Added populate script * Simplified the Runs list query, added live timer * Added TaskRun indexes for the RunList * Status can’t be null now we’re using the TaskRun status * Use defer so the page loads and shows a spinner * Improved the loading style * Get rid of latest run info from the tasks table super slow * Fix for the activity graph tooltip getting clipped * Added a code comment crediting the GitHub issue with the portal fix * Add search to the tasks list * Padding * Fix for the schedules columns not being UTC * Remove unused function |
||
|
Matt Aitken
|
fae93ac747 |
New tasks page (with activity) and multi-user fixes (#1063)
* Fix for “overridden” spelling mistake * Make the task function names grey instead of yellow * The task page now shows each task once * Blankline * WIP on activity bars * Correct size of the graphs * Add data for all days, even if there are no runs * Activity graph is working and has a tooltip * No activity label * Got the bar bg working * Activity chart now has a bg * Added a compound id to make the activity graphs appear faster * Improvements to the activity graph * Tried to disable the bg from animating * Added read replica support to the app, using Nick’s code from his batching PR * Added the read replica to BasePresenter * Added the queued and running columns to the Tasks list * Added avg duration column * Included completed but failed runs in the average duration * Get rid of the old useDevEnvironment hook and remove API keys from the projects query * Unified getting environment info to using a function and doing it in presenters * Removed the path from the tasks table * Onboarding for dev for 2nd+ user. Improved environment sorting. * Removed log * Improved the spacing * Enable staging for new v3 projects |
||
|
Eric Allam
|
b68012f81c |
v3: Various fixes for Next.js projects and projects that use v2 and v3 together (#1051)
* Fixes an issue that was treating v2 trigger directories as v3 * Make msw a normal dependency (for now) to fix Module Not Found error in Next.js. * Extract out all the zod* stuff from core so the SDK does not import it * Add a changeset * Fixing typecheck errors in the webapp * Export the Task and TaskOptions types * Extract additional exports from core/v3 that aren’t used in the SDK * Move to our global system from AsyncLocalStorage for the current task context storage * Update the esbuild core bundling plugin for the new core v3 exports * Fix v3 CLI telemetry * Add support for tasks located in subdirectories inside trigger dirs * Remove the env var check during deploy (too many false negatives) |
||
|
Eric Allam
|
26093896d2 |
v3: Fixes for using (batch)triggerAndWait with idempotency keys (#1043)
* Fixes various issues with triggerAndWait and batchTriggerAndWait When using idempotency keys, triggerAndWait and batchTriggerAndWait will still work even if the existing runs have already been completed (or even partially completed, in the case of batchTriggerAndWait) - TaskRunExecutionResult.id is now the run friendlyId, not the attempt friendlyId - A single TaskRun can now have many batchItems, in the case of batchTriggerAndWait while using idempotency keys - A run’s idempotencyKey is now added to the ctx as well as the TaskEvent and displayed in the span view - When resolving batchTriggerAndWait, the runtimes no longer reject promises, leading to an error in the parent task * Remove the default queue concurrency limit as we now have env and org concurrency limits * Use the run friendlyId in the completion result id * Added some error logging |
||
|
Matt Aitken
|
4986bfda2e |
Scheduled tasks (#1036)
* Database schema and migrations for schedules * Added schedules to the side menu * The pagination can optionally hide the page numbers for a compactive mode * Filters for the schedule page * Added triggerSource (“STANDARD”, “SCHEDULED”) to BackgroundWorkerTask * Added the ability to disabled a LinkButton * Started work on the schedule page * Environment buttons * The new schedule form styles * Added a cxouple of extra fields * Allow a checkbox to have a rich label * Added cronstrue package to the webapp * WIP creating tasks using the form * Improved the form styling * Creating schedules is working in the UI * Minor improvements * Basic schedule table is displaying * Creating tasks with triggerSource = scheduled. Refactored how task metadata is stored and accessed to be cleaner * Resource route * Fixes in the form * Added a gap between the environments * WIP on OpenAI generating CRON expressions * AI generated CRON expressions is working * Fix for the CRON field being uneditable after an AI generation * Improvements * Table padding * useThrottle now behaves correctly * Added filtering to the schedules list * Improved the layout and fixed CRON search * Fixed pagination for the schedules list. Just use a regular Prisma query * Page size of 20 * Added links to the schedule rows * Get rid of the Last run column for now * Implement triggered scheduled tasks Also implemented superjson payloads and dev runtime environment “presence” with RuntimeEnvironmentSession * Move CronPattern and CreateSchedule into a common client-accessible file * Latest UI changes * Fix for creating a task schedule with a blank dedup key * Refactor the human to cron stuff into a separate file and use json_object OpenAI response format * Fix for trying to use a hook on the server-side… * A couple of fixes to the new schedule form * WIP on viewing a scheduled run * Make the filters all optional * Use the RunListPresenter from the schedule presenter * Display a table of runs… the wrong runs but still * Runs from the schedule * Deleting schedules from the UI * Tidied imports and fixed name of options object * Disabling a schedule * Editing schedules * Tidied imports * Added icons to the task list, needs some design love * Added a tooltip for CRON pattersn * Show the last run in the schedules table * Some tweaks * Added a placeholder to the CRON AI field * Improved the trigger source icon * Dim out disabled schedules * Scheduled tasks have the correct icon in a run * Added the task source icon to the test task list * Added the date field component to storybook * Style improvements to the date field * Implement Task Schedule API * added a medium sized variant to the date field * Fixed replay run for superjson payload types Also now linking from the replaying run to the original run (using span links) Also added a project metrics prometheus endpoint to detect the state of the queues * WIP on allowing different forms for testing * If you pass a string to prettyPrintPacket which is json or superjson, safeParse it first * Test page, deal json and superjson. Started splitting UI for schedules * WIP on schedule form * Removed the Label from the DateField * WIP on test schedule form * Fix for the runs page showing the wrong message when there are no runs from time filtering * Removed labels from the DateField * Fixes for the form * Test runs are working for schedules * Fix for nextScheduledTimestamps in triggerScheduledTask * Implement idempotency key support and fix issue with cancelled runs causing concurrency usage * Added API documentation for the schedule API * Remove log * Removed console log from runs page * Transform the recent runs test data on the server * Fix for hydration mismatch * Current date as the default for the test form * Recent payloads working * Delete schedule modal * Deal with empty strings from the form * Set the initial value for the scheduled test form * Add option to print console logs in the dev CLI locally (issue #1014) * Export queue from the SDK * Fix for schedules list when you have no schedule tasks * Blank states improved * Make task schedules more generic, to support additional schedule generators in the future * Removed log from maqrs * Removed “v3/schedules” export from the SDK --------- Co-authored-by: Eric Allam <eallam@icloud.com> Co-authored-by: James Ritchie <james@jamesritchie.co.uk> |
||
|
Eric Allam
|
a5aed0d139 |
Fix memory leak by calling punsubscribe instead of unsubscribe
Also adding a /metrics endpoint with a few custom metrics and the built-in prom-client ones |
||
|
Eric Allam
|
a946797d95 |
v3: Upgrades to MarQS (#989)
* Upgrades to MarQS - It’s “queue choosing” algorithm is now MUCH better (pretty much didn’t work before) and should be performant even when we have a bunch of prod queues - There are now concurrency limits at the environment and organization level, as well as the task/queue level. So if any of them are at capacity the message won’t be dequeued. This means we can have an org wide concurrency limit, as well as different limits for dev/prod/staging - I’ve added an admin API that can be used to update the org/env concurrency limits * More MarQS upgrades - Extract the queue priortity choosing strategy into an interface - Implement a much better weighted average strategy - “Slide the window” of queue candidates if the parent queue sends all at-capacity queues (so we won’t get stuck attempting to choose the same 12 full queues) - Added some unit tests for the priority stuff - Added some ideas for expanding the priority choosing strategy with more dynamic features * Fixes an issue with the shared queue consumer relying on the queue name to get the env id |
||
|
Eric Allam
|
055d468b38 | Setup socket.io redis adapter and fix deployment builds for superjson error | ||
|
Eric Allam
|
215b60c742 |
v3: superjson output support (#971)
* superjson output support * Better support for superjson in the task events, limiting output attributes * Offload large outputs to object store (r2) * Finishing up the offloading of large outputs/payloads to an object store |
||
|
Eric Allam
|
395abe1b92 |
v3: Various fixes to support papermark use-case (#970)
* Adding a changeset for v3 * Add a version field to @trigger.dev/core-apps package.json * Build trigger.dev when doing a prerelease * bundle @trigger.dev/core-apps with trigger.dev cli * Fix the init command config template * Don’t use * for the @trigger.dev/core dep version specifier * strip workspace: from the package version before installing it * Added dependenciesToBundle config option to bundle ESM only packages * Added logging around resolving dependency paths * Try again * Resolve dependencies based on the project dir first * flip the bundled default * Adding some logs around dev task completion notifications * Adding some additional logs * Add more logs * Write out the log using process.stdout * Store pending completion notifications and resume them when awaited (fixes race condition) * Cleanup some of the logs * Copy over the postinstall step from the projects package.json * Add support for including additional files when deploying (e.g. prisma schema) * Don’t run scripts when resolving deps * copy all the files just in case anything is needed in postinstall * Remove duplicate option * Use the tag when outputting the dev command * Remove the postinstall script * add the trigger dir to the config if the default is not chosen * Remove the “hud” display in the dev command * trigger file names with dashes now work * Better file watching in dev * Much better duplicate ID experience now * Much better “Project not found” error * Export the handleError function types from sdk * Add support for configuring instrumentation * Upgrade and unify @opentelemetry/* packages (and remove storybook from the webapp) * Fix typescript error in react package * Upgrade react types in webapp * Allow span icons to be determined based on the span name (e.g. prisma:) * Ignore built-in env vars when checking for env vars, and allow continuing the deployment even if missing env vars were detected * Improve the retry.fetch default behavior and option structure * Fixed typescript errors with packages/email react types * Update the retry.fetch docs |
||
|
Matt Aitken
|
b35eebb666 |
Rate limit API requests and changed SQS reading speed (#969)
* Trying to use the @upstash/ratelimit package with ioredis… * WIP using the redis package instead * Revert the action back * Removed redis * Started refactoring * SQS setting for the poll interval. Set the default queue reading to be slower * API rate limiter as Express middleware * Organise imports * Fixed spelling mistake “limitter” * No authorization header response is problem+json |
||
|
Matt Aitken
|
819b663ad7 |
Test page environment selection (#942)
* useLinkStatus hook * Now you can customize the appearance of InlineCode * Created <TaskFunctionName/> * The test page now has an environment selector at the top |
||
|
Eric Allam
|
c385db63be |
v3: cli deploy command (#931)
* WIP proxy/deploy * WIP, registry proxy in express working * A couple of notes, preparing for indexing * Move the changes to prod-worker into the new file * Deploy command working with indexing and runs (docker provider only for now) * Removed ts-expect-error directive * Fixed build command |
||
|
Matt Aitken
|
2a1273b683 |
Major v3 web app update (#924)
* Added the new colours to the Tailwind config * Fixed Tsailwind compile errors * WIP Geist working in the app * WIP Storybook * Storybook is working again… ouch * Use Geist mono for CodeMirror * Some find/replace class changes * Resizable colours improved * WIP on primary button * Minor style improvements * Improve the side menu header styling * Improved v2 breadcrumbs * Account side menu tweaks * The page now works with the NavBar and UpgradePrompt * Removed unused import * Refactored the buttons to be dry-er * Removed the breadcrumbs * WIP on nav bar * WIP updating the button styles * Fix for the old test page * Fix for type errors * Removed console.log * Colour change in the resizable handle * New loading bar divider * Removed the design plugin * Creating our own storybook pages * Added some “storybook” pages * Added DetailCell storybook * WIP new animation for the loading page * Added the new loading divider * Fix for the time frame filter having a green border * Style improvements to the new test page * Small style improvements * Improvements to the list pagination components * Nav bar back button improved * Improvements to the run page * Lots of v2 style improvements * Help looks better * Improvements to the regenerate API key modal * More general improvements * Added section titles to our storybook * Added form stories * Fix for missing key prop * Loading bar is working * Fixed button submitting state * Fix for the test button icon * Code bg improved * Improved oauth buttons * Integration page header fix * Various improvements, mainly changing button variants * Segmented control style updates * Added an outline of 3px for the segment control * Fixes for the billing pages * More pricing improvements * Added dialog to storybook * Typography now in storybook * Free usage bar added to storybook * Added inline clode to storybook * Separated menu types into a sub section in storybook * NamedIcon now in storybook * Added page header to storybook * Added pricing callout to storybook * Added radio groups to storybook * Added segmented controllers to storybook * Added shortcuts to storybook * Added switches to storybook * Added toast to storybook * Added tooltips to storybook * Added tree view to storybook * Added usage charts to storybook * Tarted up the environment variables page * Fix for the FreePlanUsage panel bg colours * Added new badge variants * Added the <TaskPath /> to the tasks table * Span header improvements * Started work on the timeline * Started work on the timeline * Fix for the run page header * Timeline tick marks are working * Timeline working * Got the durations showing on hover * Button improvements * Experimenting with how the duration should be shown * Lots of refinements to the timeline * Much nicer API for the Timeline * The switch is now purple to match the design, with inverted thumb when on * Removed old stuff * Export the props * Added the span with duration back in * Simplified more and fixed the span durations * Put in some decent dummy data * Rendering defaults * Wip with the mouse cursor * Improvements to the story * Added Slider primitive * Integrated the icons into the slider * Lots of layout improvements * Cleaned up imports * “Root” indicator * Animated background on partials * Pull the styles through to the spans and log dots * Fixed some bugs in the TreeView when nodes get created or destroyed * Fix for incorrect parentTask reference * Added run streaming endpoint for live refreshing * Added the tabs to Storybook * Fix for status font size mismatch * Fix for the misaligned run filter status * Set a minimum duration on the timeline * The TreeView now re-calculates if the number of nodes has changed * Partials now render to the end * Improved the durations gutter * The duration is calculated using the current time if there are partial events * Added the environment label to storybook * Improved the environment label style * Removed bottom margin from badge * Button style improvements * Callouts style fits the new design better * Got the scrollbars to appear on the tree/timeline * Added scroll support * Synced scrolling working * Close button and exit icon for the detail panel * secure field shows start of env * Clipboard improvements * Latest lockfile * Fix for the durations wrapping --------- Co-authored-by: James Ritchie <james@jamesritchie.co.uk> |
||
|
nicktrn
|
52c9d485f8 |
v3: cli build command, prod runs, checkpoints (#919)
* fix trailing slash in api url * add maybe platform down error * shorten config path * extend protobuf compiler install instructions * build command and image model * new image triggers task indexing * index support for prod builds * shorten example provider secret * multi-stage task prod build * lock prod tasks to node 18 image * extract shared build and dev command libs * pull out more shared deps * add coordinator and providers * fix core-apps build * add dev builds for new apps * fix cwd * enable corepack * some build fixes * enable buildkit for old docker versions * coordinator image fixes * update provider containerfile * build dev images in parallel * upgrade pgadmin * fix prod facade build * prod runs * fix merge * don't knock out platform on invalid attempt id * fix prod facade * rename to build.ts * fix prod builds * set to executing after fetching payload * make prod worker listen on random port if in use * remove experimental warnings in dev * prod resume * prevent execution after completion * exit prod worker after completion * always restart otel collector * docker checkpoints and prod runtime messaging * don't retry indexing without chance of success * make platform checkpoint aware * deploy with existing hash sets latest worker * log restore requests * only try to checkpoint long waits * tidying up * lockfile * fix build * prod worker merge fixes * fix prod complete and cancel * fix lua nil checks * socket namespace abstraction * cleanup * make all build args optional * add build script * don't require env vars for dev * fix schema * prod merge * small fix * bind correct logger * fix v3 ref catalog entry * resume prod batch * pass socket to error and disconnect handlers * fix non-batch resume * fix batch resume * send connection env vars when not in dev * create worker via socket * move api client back into v3 cli * fix lockfile * fix resume with failures * marqs replace message * typecheck prior to build * don't define api url in prod builds * support prod retries after resume * skip typecheck option |