trigger.dev@4.0.0-v4-beta.23
260 Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 Matt Aitken
|
5e999c8992 | Use p-map to speed up bulk actions (#2272) | ||
|
Eric Allam
|
6ff0b923e1 |
New internal idempotency implementation for trigger and batch trigger (#2256)
* Introduce request idempotency to prevent duplicate triggers * Implement request idempotency on trigger * Use x-trigger-request-idempotency-key header instead Plus cleanup shared logic in the routes * Add changeset * Oops, lets not hardcode a 408 * A couple of improvements |
||
|
Eric Allam
|
b119a52e08 |
Implement MFA (#2244)
* Adds a new route for logging in with mfa * New path for security page * Adds “Security” link to account side menu * Update the Switch component to allow label positions left and right * Optionally hide the Close button in the Dialog title bar * Installs `qrcode` react package for generating QR codes. * CopyButton component now takes children * New Security route for setting up MFA * Adds new OTP package for the chadcn InputOTP component * Adds new InputOTP chadcn component * Adds InputOTP chadcn component to the MFA login screen * InputOTP component supports variant styles * Improvements to form handling * Show a confirmation modal before you can disable MFA * Revert redirect back to the dashboard for now * Implement MFA enabling and disabling * Refactor and cleanup mfa management code * More cleanup * Handle errors in the management action * Implement mfa login flow * recovery code input should be password * Implement rate limiting on the mfa validation endpoint * Better error ux * Implement mfa emails and apply James' updates * Use latest @better-auth/utils * Improvements via CodeRabbit review --------- Co-authored-by: James Ritchie <james@trigger.dev> |
||
|
nicktrn
|
afc9a6ade1 |
Add ECR support for remote builds (#2224)
* create repo if doesn't exist * fresh auth token for each deploy * optional assume role * log when machine overrides enabled * make test repo namespace configurable * assume role fix and env var changes * improve ecr check * improve tag parsing * tag parsing tests * track if repo created and fix test * missing tryCatch for sts call |
||
|
Eric Allam
|
ea6f66bdfc | Improved schedule engine (#2174) | ||
|
James Ritchie
|
20a8e7c6c1 | Updated kapa sdk to 0.1.3 (#2169) | ||
|
James Ritchie
|
e12c82b444 |
Use the Kapa AI SDK instead of the Kapa AI widget (#2113)
* Install the kapa sdk * WIP using the SDK for the Kapa Ask AI widget * Removes old kapa from root * Now rendering everything inside the dialog component * Fixes min-height of dialog content * Remove kapa from root * prevents kapa using reCaptcha * Adds more functionailty with temporary UI placement for now * Reset conversation button * Adds a new sparkle list icon * Adds some example questions as a blank state * Animate in the example questions * use “marked” package to render markdown * Improve some animations * Submit a question from the URL param * adds custom scroll bar styling * fixes modal to correct height after re-opening it * Add button to stop generating answer mid-stream * Adds buttons states to show submitting, generating, submittable * Adds a helpfull sentence in the blank state * Show a message if the chat returns an error * Adds reset chat and feedback buttons to the bottom of an answer * Makes sure you can give feedback in the different states of chat * Adds a suble background to the dialog * Fix a button inside button error * Improve the shortcut esc key on dialog and sheet component * Fix classname error * organize imports * Use our custom focus-visible * Move the Tooltip for the button into the AskAI component * Improved error message * Organize imports * Animated the modal gradient * Small layout improvements * Adds most asked questions from Kapa * border glow tweak * AskAI component is now a hook that can take a question * remove kapa script * Add a delay before the modal opens when usign the URL params * Remove old component * Update to the latest Kapa version * Rephrased error message * Use correct types for conversation * Fixed types for addFeedback * Adds DOMPurify package * removed unused const * Removed unnecessary platform specification * Reset the timeout when the ai panel pops up Co-authored-by: coderabbitai[bot] <136622811+coderabbitai[bot]@users.noreply.github.com> * Fix for coderabbit bad commit * Clean up imports --------- Co-authored-by: coderabbitai[bot] <136622811+coderabbitai[bot]@users.noreply.github.com> |
||
|
Eric Allam
|
2b3ea692fe |
v4: dequeue performance improvements (split concurrency from dequeue) (#2127)
* WIP * Run queue now works with the worker queue / master queue split * Acking should also cause the master queue to be processed * Convert run engine tests and run engine to use runQueue changes * Include the util files in the test tsconfig * coordinator target should be es2020 as well * providers target 2020 * Fix the triggerTask tests in the webapp * v4 now working with the new worker queues, and added the legacy master queue migration stuff * report worker queue lengths via opentelemetry metrics * Adding lock metrics * Release concurrency bucket metrics * • Updated RunQueue.removeEnvironmentQueuesFromMasterQueue() method signature to take runtimeEnvironmentId instead of masterQueue parameter • Added automatic master queue shard calculation using this.keys.masterQueueKeyForEnvironment(runtimeEnvironmentId, this.shardCount) • Updated RunEngine wrapper method to use new runtimeEnvironmentId parameter • Updated DeleteProjectService to call the method once per environment instead of once per master queue • Simplified API by encapsulating master queue sharding logic within RunQueue class * metrics now working, configure the run queue settings, additional metrics for run engine and redis-worker * Fix CodeRabbit suggestions * return undefined from dequeueFromWorkerQueue, not null * Remove message from worker queue in certain circumstances when acking * Update log * Ensure master queue consumers cannot stop from a processing error, and make the consumer interval configurable via an env var * Change how the run queue master queue consumers are disabled internally * Fixed tests * process the queue on nack * Fix more tests * Fix priority tests * Fixed dequeueing test |
||
|
nicktrn
|
f603725393 |
Feat: unified deploys for self-hosted and cloud users incl. multi-platform support (#2138)
* remove registry proxy * remove --self-hosted flag * automatically set network build flag * update syncEnvVars debug log * improve switch command * always display deploy errors if they exist * fix stuck deploy command after finalize error * webapp-driven deploys, multi-platform support, lots of fixes * add worker deployment migration * rename image platform env var * only try to sync parent env vars for preview deployments * add KEEP_TMP_DIRS * supervisor: docker api version lock, auth, multi-platform * set image ref on create, validate digest * use metadata for digest, fix local multi-platform builds * print git meta branch before commit * improve push and load flag handling * make runs after local builds compatible with load and push * small improvement for platform overrides * add image platform to dequeued message * remove deprecated init request body fields * fix fail deployment id param * remove build debug logs * pass report merge with no tests * structured run debug logs * add required env var for tests * should not be an error log * add changeset |
||
|
Matt Aitken
|
068c024477 |
Preview branches (#2086)
* Initial preview migrations * Modified the staging endpoint to create preview environments * Added isBranchableEnvironment to RuntimeEnvironment * Staging = yellow Preview = orange * Changed the env sort order * Set isBranchableEnvironment correctly. Create preview for new projects * Very basic branch menu * Creating branches from the dashboard * Fix for string icons on project delete page * Don’t show branch API keys * WIP on the manage branches page * RuntimeEnvironment added projectId index * Only create the parentEnvironmentId column if it doesn’t exist already * Improved the limit wording * Add search to the branch list * contains in both places * Many style improvements * Branch dropdown and v4 badge * Arching/unarchive branches working in the dashboard * Tidied imports * Change preview slug from `prev` to `preview` * Use correct color for side menu preview branch icon * Upsert the branch and use the shortcode as a unique constraint * Upserting working with nice messages in the dashboard * Better errors when upserting branches * Button shortcut, don’t allow event to propagate * Better duplicate error message * Filter out archived branches from the env selector * Archiving/creating tweaked some more * Add an archived banner to the app, fixes for archived branches and upsells * Fixed pagination * Disable editing schedules, pausing queues, testing tasks * Don’t allow replaying if the env is archived * When deploying detect the correct environment * Get the projectClient when there’s a branch * createGitMeta function, most code from the vercel CLI repo * Deploy, getting the correct environment client * Added git column to WorkerDeployment * Add GitMeta to core schemas * Create branch when deploying * WIP on branch support in the API * Delete old createTaskRunAttempt fn * apiAuth remove export from internal functions * Rename env var to “TRIGGER_PREVIEW_BRANCH” * Add TRIGGER_PREVIEW_BRANCH to resolved env vars for runs * First preview deploy and run working * Set the preview branch in the main SDK * Added git links to the preview branches table * Better errors when replaying/testing archived branches * Don’t dequeue archived environments * Env var resolution with parent environment * Hello world default machine small-2x to save my memory * Fix for more env var functions * Only return non-archived envs * Switch to controlled state for the checkboxes * Uncheck everything when PREVIEW is checked * WIP on branch UI * Show the preview branch label on the env vars list * Fix for overriding env vars * Adding preview branch env vars working * Progress on new env vars * Only allow selecting a single branch * Layout fix when there are errors * Set the defaultValue so there are some fields * Conform fix for team invite page * Archived environments don’t run scheduled tasks * Added Git data to deployments * Added git data to the deployment inspector * Don’t allow upserting schedules when archived * Deduplicate and blacklist some env vars * Fix for wrong conform function being used * Show a better error if all vars were blacklisted * Added environment variable search (by key and value) * Improved preview branch icon * Replay now supports branches * Schedule page render branches properly * Show the env icon in bottom-left of the test page * When editing older schedules (that have multi-env) show preview branches correctly * Fix for incorrect disallowed branch name character * Extract and improve the directory verification code * WIP for CLI preview archive command * Improved the preview branch action buttons * Redirect to the project if we don’t find a matching env * Archiving branch via the CLI working * Fix for archiving branches * Public access token test task * JWTs working are with preview branches * Add branch and git data to the Run ctx * Updated GitMeta functions to work in CI * Added pullRequestState * Archive when deploying if the PR is closed/merged * Fix for the changesets guide * Fix for CLI dev bug introduced * CLI promote now supports preview branches * Add PR title. Reordered them and added tooltips * syncEnvVars working with branches * Added preview branch support to syncVercelEnvVars() * Detect the branch from Vercel env var (set during build) * Allow passing a branch in * Use process.env.VERCEL_TOKEN as well… this used in Vercel CI * Temp delete * Improved regenerate api key modal * Added Accordion component (with styles) * Redesigned the API keys page * Revert "Temp delete" This reverts commit 177b92cd935a6161456bde65d01294e23ecfd47f. * Changeset * Fixed docs link * The new branch panel closes when a branch is created * Update apps/webapp/app/services/upsertBranch.server.ts Co-authored-by: coderabbitai[bot] <136622811+coderabbitai[bot]@users.noreply.github.com> * Removed findUniques from WorkerGroupTokenService * Made the parentEnvironmentId migrations safe * Latest lockfile * Update packages/cli-v3/src/commands/workers/build.ts Co-authored-by: coderabbitai[bot] <136622811+coderabbitai[bot]@users.noreply.github.com> * Move isValidGitBranchName to a separate file * Move the sanitize fn too * removeBlacklistedVariables moved to a separate file * Moved deduplicateVariableArray to a separate file… * Fix broken sanitizeBranchName import * Another import fix… * Improved blacklisted error message * SImplified migration to use `ADD COLUMN IF NOT EXISTS "parentEnvironmentId" TEXT` --------- Co-authored-by: coderabbitai[bot] <136622811+coderabbitai[bot]@users.noreply.github.com> |
||
|
nicktrn
|
7c791dd519 |
Improve unit test workflow performance (#2096)
* shard unit tests * temp enable for all pushes * fix test workflow * update to latest vitest and only add to root package.json * additionally use default reporter * gather reports before uploading * split up slow replication tests * split up unit tests workflow * move workflows to parent dir * use new paths in parent workflow * prevent artifact clashes * we always need to create the reports dir * speed up merge reports * gather reports even when tests fail * fix artifact patterns * increase shards * disable push trigger again * improve dequeue snapshot test reliability |
||
|
Eric Allam
|
8471169b30 |
Fix various dependabot alerts (#2063)
* Upgrade posthog-node to clear axios vulns * Upgrade @slack/web-api to use a secure version of axios * Update parse-duration to fix security vulns * Mitigate against the ws DoS vuln * Upgrade body-parser to 1.20.3 in the webapp * Upgrade react-use to 17.5.1 to remove the fast-loops transitive dep * Remove unused babel dev deps and config file that's no longer used * Upgrade prismjs to 1.30.0 and bundle parse-duration now * upgrade express to 4.20.0 to fix issue with XSS when redirecting * Upgrade @conform/zod to 0.9.2 |
||
|
Eric Allam
|
65da20c225 |
feat: replicate task runs to clickhouse to power dashboard improvements (#2035)
* WIP clickhouse package with test containers setup * More clickhouse client setup now with otel and real tests, and the v1 of raw run events * Add some additional columns to raw_run_events_v1 * WIP runs dashboard service * Create a new run engine event bus event for the runs dashboard to hook into * Track run events in the run engine * make sure engine v1 runs get synced to CH * Update the attemptNumber of v3 task runs * Restructure the run events to be more sparse * emit more stuff * Setup replication package * scaffold the replication package * replication wip * resolve conflicts * more replication stuff * Add ability to drop the replication slot completely on teardown * Use the new single replacingmergetree task events table for replication * get it working * insert payloads into their own table only on insert and then join * prepare for using clickhouse cloud and now running ch migrations during boot in the entrypoint.sh * Handover WIP and tests * Testing the replication service * Remove the runs dashboard stuff that we aren't using anymore * Added a test for large payloads * hacky typecheck fix * Fix new internal package typecheck issues and start adding telemetry to the replication service * tracing over spans, some other improvements * Improvements to the runs replication service, now ready for testing * Some fixes and cleanups * Don't need this code anymore * move transaction types into the runs replication service * only send spans where there are transaction events * A couple of suggested tweaks |
||
|
Matt Aitken
|
32d2992802 | Security update of nanoid from 3.3.4 to 3.3.8 (#1887) | ||
|
Eric Allam
|
51bb3ee464 |
v4 ai improvements (#1863)
* AI SDK sql generator and approval example WIP * Create some nice spans and extract the tools out so we can get the type of the streams * Enable AI SDK telemetry * Adding ai.tool and removing toolTask, 3rd party telemetry spans now wil create partials, better ai SDK telemetry icons * Created a separate d3-chat example, split out from the openai-agents example * Fixed the tool options being passed to metadata in `ai.tool` * Add a link to the run * Slightly improved design * Add a crawler task using crawl4ai * Use a tool to get the userId * Couple of tweaks * Adding markdown rendering to assistant messages and added an e2b based chart rendering task * Backup to anthropic * Add changeset |
||
|
nicktrn
|
49a3f72e13 |
Publish redis-worker and add graceful shutdown manager (#1810)
* add shutdown manager * update ai test instructions * add shutdown timeout to redis-worker * move redis worker to packages * add unregister method * prep for publishing package * fix types * update ai files * fix cursor terminal links * prevent overly friendly ids * use structured logger * use unique shutdown handler names * rework suspend completion * add trycatch util * rework suspend restore * add http server metrics * add missing prom-client to core * add prom metrics to redis worker * bundle redis-worker * fix esm/cjs interop * remove proxy from changeset ignore and add supervisor * add pause to prerelease script for any manual edits * unregister the correct handler and add early detection * small change to http handler return * fix worker tests * fix shutdown manager tests |
||
|
Matt Aitken
|
6cf86d5916 |
Deleting v2 dashboard/api (#1628)
* Delete v2 Stripe routes * Delete v2 billing/usage pages * Delete v2 integration pages * Delete v2 project pages * Deleted a load of components and services * Deleted a load more components, presenters and services * Deleted another 100 files or so… * Removed old v2 paths * Removed named icons from form titles * Removed more string icons * Delete NamedIcon * Fixed some type errors * Delete endpointApi * Removed v2 from core/sdk * Post merge fixes * added explicit return types * using the new sdk export without v3 * Delete old v2 file * Added explicit return types because TS was complaining… * Don’t export RuntimeEnvironmentType from two core files. Was causing TS issue * Fix for removal of NamedIcon in new route * Removed strange eslintrc rule * Use the new redis client --------- Co-authored-by: James Ritchie <james@trigger.dev> |
||
|
Eric Allam
|
d855d55ea0 |
re2: env based queue selection algo (#1775)
* re2: fix @trigger.dev/core exports * re2: WIP env based queue selection algo * more wip * WIP * Get run engine tests to pass * Adding tests for the fair dequeueing strat in the run engine * Configure the new queue selection strategy in the webapp and get it all building and typechecks passing * webapp now uses built packages, building redis-worker, run-engine, database, using better tsconfig setups for tests, moving isomorphic code into core/v3/isomorphic * Fixed webapp typechecks * dev now depends on build, fixed supervisor typecheck * Fixed run engine tests * Fixed e2e tests |
||
|
Matt Aitken
|
983bb411cc |
Redis client factory (error handling and good defaults) (#1761)
* @internal/redis package for creating Redis clients with sensible defaults and error handling * Added @internal/redis to webapp |
||
|
nicktrn
|
e97704d904 |
Run Engine 2.0 (WIP) (#1575)
* bump worker version * Suggested glossary for the RunEngine, TBC * Removed BatchTaskRun changes from this branch, they were done in main * Set the BatchTaskRun status to completed when all runs are completed * When dequeuing respect passed in maxResources * Ported over the new run props: idempotencyKeyExpiresAt, versions, oneTimeUseToken, maxDurationInSeconds * Didn’t hit save… the new props when triggering tasks passed through * Idempotency expiration + waitpoint edge case * WIP on creating checkpoint, parking for now * fix worker routes * upgrade webapp node types to support generic event emitter * separate event bus handler singleton and run failure alerts * duration waits * fix execution snapshot debug spans * task waits * fix event bus types * temporary fix for react hook run handle type * disable run notifications for now * convert any typecasts to expect errors to more easily fix later * fix webapp types after node types upgrade * updateEnvConcurrencyLimits across marqs and the runqueue * Pass proper values into the run engine * RunQueue settings and removed unused rebalancing workers * Remove rebalancing prop * Tidied more things up * Update/remove queue limits for MARQS and RunQueue * taskQueue/concurrencyLimit changes ported back into the RunEngine * Reworked completing waitpoints to improve performance and reduce race conditions * Improved test robustness * Down to a single run lock only when a run is totally unblocked and ready to continue * warm starts, worker notifications, wait fixes * Fix for Run Engine poll interval env var * Expect the waitpoint to be completed quickly * If a run is locked then it’s too late to expire it * Added VALKEY_ env vars and plugged them into the run engine * Extracted and updated the guard queue function so it can be used when batching * Added logging and universal concurrency changes to trigger task v1 * Added notes back in * Bump @trigger.dev/worker to 3.3.7 * reportInvocationUsage for the runAttemptStarted event * improve execution snapshot span debug span start times * Unfriendly IDs * update lockfile * Created a shared determineEngineVersion function * disable unfinished commands * save new cli config to different location, misc fixes * add basic engine version check via current deploy * new run engine will default to node 22 runtime * block some actions for projects on previous run engine * fix worker group tests * fix triggerAndWait test * one typescript version to rule them all * redlock type patch * fix type issues caused by ts-reset * improve cleanup scripts * add missing socket.io dep * fix run notification handler type * fix worker group test again * generate prisma client for e2e tests * remove worker group tests for now * prevent image pull rate limits during unit tests * increase timeout for queue concurrency limit test * generate prisma client for preview release * same node types everywhere * Updated engine readme, removed legacy system notes * use default machine preset from platform package * worker instances plural in schema * disable pnpm update notifications * return worker group details from connect call * add workers admin route * fix heartbeat route return type * move deployment labels to core apps * refactor run controller env schema * Add firstAttemptStartedAt to TaskRun * RunEngine 2.0 batch trigger support (#1581) * Make it clear when BatchTriggerV2Service is used * Copy of BatchTriggerV2Service * WIP batch triggering * Allow blocking a run with multiple waitpoints at once. Made it atomic * Removed unused param * New batch service * Pass through the parentRunId and resumeParentOnCompletion * Use the new batch service, and correct trigger task version * Force V1 engine if using BatchTriggerV2Service, we’ve already done the check at this point * Removed the $transaction and early exit if nothing changed * Adedd a simple batch task to the hello world reference catalog * Fix for batch waits not working * Added parentRunId in a couple more places * Removed waitForBatch log * Added another parentRunId * Expanded the example to include all the different triggers * More changes to blocking to support continuing after idempotent completed runs * Fix for the wrong type when blocking a run * remove @map * optimise worker auth query * add engine version header to core api client requests * remove unique constraint for default group id * consolidate migrations * the first managed worker becomes the global default * Debug events off by default, added an admin toggle to show them * worker group name can't be an empty string * add exec helper to core * move machine resources to core * add pre-dequeue callback to determine max resources * optionally skip dequeue * bump worker package * move worker to core * fix ReadableStream type error * fix another type issue * update a few more tsconfigs * add metadata changes introduced in #1563 * Run Engine 2.0 trigger idempotency (#1613) * Return isCached from the trigger API endpoint * Fix for the wrong type when blocking a run * Render the idempotent run in the inspector * Event repository for idempotency * Debug events off by default, added an admin toggle to show them * triggerAndWait idempotency span * Some improvements to the reference idempotency task * Removed the cached tracing from the SDK * Server-side creating cached span * Improved idempotency test task * Create cached task spans in a better way * Idempotency span support inc batch trigger * Simplified how the spans are done, using more of the existing code * Improved the idempotency test task * Added Waitpoint Batch type, add to TaskRunWaitpoint with order * Pass batch ids through to the run engine when triggering * Added batchIndex * Better batch support in the run engine * Added settings to batch trigger service, before major overhaul * Allow the longer run/batch ids in the filters * Changed how batching works, includes breaking changes in CLI * Removed batch idempotency because it gets put on the runs instead * Added `runs` to the batch.retrieve call/API * Set firstAttemptStartedAt when creating the first attempt * Do nothing when receiving a BATCH waitpoint * Some fixes in the new batch trigger service… mostly just passing missing optional params through * Tweaked the idempotency test task for more situations * Only block with a batch if it’s a batchTriggerAndWait… 🤦♂️ * Added another case to the idempotency test task: multiple of the same idempotencyKey in a single batch * Support for the same run multiple times in the same batch * Small tweaks * Make sure to complete batches, even if they’re not andWait ones * Export RunDuplicateIdempotencyKeyError from the run engine * Latest lockfile * Trigger with a machine (old run engine) * RE2, allow setting machine when triggering * Fix for new glob patterns * add max run count to dequeue from version route * add worker instance name env var and header * queue consumer pre skip callback * poll for more runs after final execution errors * fix dequeue search param schema * add shortcut to debug switch * expose run engine timeouts as env vars * make warm start durations configurable * add optional status to json reply helper * fix preSkip hook, add debug logs * BLOCKED_BY_WAITPOINTS -> SUSPENDED * exit controller when run suspended * check if already replied before http reply * run controller will wait for next run after the current one is suspended * cancel run button shortcut * minimal event repository environment type * fix update metadata call * run suspension and misc fixes wip * change debug shortcut to shift + D * Started work on the Dev supervisor * Formatting * Fix for bad imports * Before rebuilding SSE * Presence updating from the CLI working via SSE * add worker notification debug logs * send run:stop when exiting run phase * skip current snapshot poll on worker notification * add more logs and route to submit run debug logs * add worker and runner ids to snapshots * improve run notification debug logs * add workload debug log route * misc run controller fixes and refactor * prevent parallel execution of critical functions * update bun to 1.2.1 * WIP with dev dequeuing * Method to convert friendlyIds to non-friendly, do nothing with actual ids * Set the engine on BackgroundWorker, lazily upgrade projects to engine V2 * Runs with ttls were getting immediately expired… oops. * Pass the Waiting for deploy reason through, so we have it on the execution snapshots * Fixed the logic for getting the right background worker for a run * Use the correct ID when dequeuing… * determineEngineVersion is now fully functional * Rate limiter ignores the dev endpoints * Retrieving a batch gives you the runIds * Set a unique version for the RE2 BatchTaskRun * add provisional changeset * The start of dev run execution is working * First dev run working * Moved the dev run controller closer to what Nick did with the managed one * export exec output type * Heartbeat fix: don’t heartbeat if _isHeartbeating == false * Dev runs get notifications, some dev bug fixes * Improved logging or dequeuing * We need to dequeue runs from the latest version too, for triggerAndWait * Ported Eric’s validateWorkerManifest with nicer errors * When flattening an idempotency key if part is undefined, return undefined * Dev logging fixes * Remove sigterm listener * Deprecating workers. Don’t specify a BackgroundWorker when dequeuing an environment * Deleted some old files. Renamed “managed” to “deploy” * When a build finishes, always copy the build dir (otherwise the first one gets trampled on by the 2nd) * Dev master queues should work differently * Deleting old workers * Added debounce function to core * Improvement to canceling * WIP on debounce canceling on socket disconnection * Added environment data to execution snapshots * Dev runs that have stalled get “Canceled” with a reason explaining why * Show CLI messaged when a connection to the platform is lost/restored * Fix TriggerTask after merge * Add trigger task v2 max attempts, replace some findUniques * Port the new queue logic to the run engine * More fixes post-merge * We weren’t setting a `retryConfig` up for the tests… it’s now required * Start the Redis worker inside the Run Engine… 🤦♂️ * Trying to make the testcontainers more reliable * Added keyPrefix: "engine:” * Badly placed bracket in trigger task * Better Redis namespacing * Fix for expired run not getting removed from the queue * Don’t create a redis client in the testcontainers, return the redisOptions instead * Cleanup redis client in the run lock tests * Fix for the RunQueue not supporting keyPrefix * Updated more of the RunQueue scripts rebalancing * Trying to make Redis more robust in the tests… * Improved test resiliciency more * Fix for delays (checkpoint check) * Increase the timeout slightly to fix ttl test * Added priority support when triggering * More wip trying to make test containers more reliable * batchTriggerAndWait test is still failing… some wip to try fix it * Fixed redis tests now we’re not providing a client * Separate Redis clients for the run engine worker/queue/runlock * Made the wait for duration test more resilient * Added idempotencyKeyExpiresAt to Waitpoints * Waitpoint timeouts and idempotency expiry * Use finishWaitpoint, removed extra worker job * Added waitpoint idempotency tests * Creating resume tokens is working * Some improvements to the resume tokens * Moved resumeTokens to just be wait functions 🥳 * Delete old RuntimeManagers * Wait for token is working * Better test for the wait tokens * Improved the test task some more * Hide the accessories in the span inspector * WIP on waitpoint inspector * WIP on complete waitpoint form * Span overview panel can be changed based on the entity type * Improved the waitpoint display * WIP on completing waitpoint form * Use the existing CodeBlock for the tip * Style improvements * Complete waitpoint * All waitpoint sidebar variants * Waits now use a pause icon * Durations waits use the API to create/block with a waitpoint, not the runtime * Fix for engine.blockRunWithWaitpoint required org id * Removed old wait code from the run controllers/task run process * Form action for skipping a datetime waitpoint * Move testDockerCheckpoint to a separate core package export (it can’t be bundled on the client) * Fix for glitchy hourglass animation * Completed waitpoints display better * Increase Redis maxRetriesPerRequest to 20 (default) * Completing and skipping waitpoints is working * Remove the database prisma dev command, since we need to use create only now. Updated docs * Added skip timeout, reworked the UI * Tweaked spacing * Added payload limit to waitpoint token completion from dashboard * Test idempotency works on wait.for and wait.until * Moved the worker-actions to /engine/ from /api/ * Moved dev engine endpoints to /engine/ from /api/ * Separate /engine/ rate limiter * Added parallel wait prevention, it’s working for duration waits but not well for triggerAndWait yet * WIP post-merge conflicts * Set taskEventStore column in the new engine * Remove duplicate keys * Post-merge fixes * Fix for span merge layout * Use executedAt instead of firstAttemptStartedAt --------- Co-authored-by: Matt Aitken <matt@mattaitken.com> |
||
|
Eric Allam
|
a2c2d920b7 |
Use redis worker for run heartbeats and alerts (#1669)
* Move the task run heartbeats to RedisWorker * Move alerts to redis worker, improving redis worker * Fix typecheck errors * Use single threaded tests for redis worker * Enable/disable the redis workers independently * Remove preview release from PR checks |
||
|
Eric Allam
|
f6eeaa1001 | Fix broken cloud deploys by using depot ephemeral registry, skip the registry proxy (#1637) | ||
|
Eric Allam
|
eaf46bac58 |
engine v1 improved dequeue selection algorithm (#1632)
* engine v1 fairness improvements (env based queue selection) * Add ability to bias env selection by concurrency limit and capacity * Add some randomization to the list of queues within each evn * Configure biases using env vars * Typo * revert v3 catalog project ref change * Fix v2 marqs |
||
|
Eric Allam
|
34f8bd588e |
Add ability to update parent and root run metadata from children (#1563)
* Increase the number of active streams from 2 to 5 * WIP parent metadata updates * Fix noop metadata manager * Implement run metadata updates from ancestor tasks * Add changeset * Add ability to stream into parent and root task runs * Don't duplicate exporting run metadata types * Add ability to fetch streams through metadata * Couple of fixes from CodeRabbit * Fix metadata tests * Make sure streams are subscribed in the "background" * Move the stream subscription stuff to the API client, expose it through `runs.fetchStream` * Fixed run stream tests |
||
|
Eric Allam
|
7d4708839f |
Upgrade zod to 3.23.8 (#1484)
* Upgrade to zod 3.23.8 * Add changeset |
||
|
James Ritchie
|
1b3a2f2e5b |
Full width tables (#1474)
* Updated styling * Updated the main Table component styles to the new design * Run page table layout style updates * Updated row styling and new scrolling behaviour * Fixed bulk action bar no appearing * Improved table component styles * Pagination arrows now chevrons * Added link to Runs docs page * Small popover tweak * Moved the extra Test button out of Tables and into the RunsTable * WIP refactoring the sticky table cell * Support for the existing way we have menus in the sticky table cells * Created a specific Docs style button * Improved dialog trigger button styles * Improved docs button styles * Updated Docs buttons to the new style in the tooltips * Adding a Test button on hover * Removed the page padding and made table scroll * Fixed table scrolling * cleaned up imports * WIP adding a test button that links to the test page * added a docs button to the header * Much nicer display bar for showing how many schedules you have (and it works with the updated table) * Fixed the button types in the schedules info panel * Fixed button icon colour * Removed the pagination as it’s not being used * Delete modal button position updated * Fixed padding on recent payloads panel * Fixed button icon colours * Better style for active table row * Table scrolling now behaves properly * environment buttons now have a hover state * Bigger Run test bar to match the Run page * Small layout fixes to the schedule side panel * Test page now has link to docs * fixed text alignment with table headers * API keys page now has full width tables * Fix typo and modal layout * Table uses new sticky button type * Tidy imports * Env var table now full width * Form buttons have a border top * Updated button styles * Updated blank state message style to be consistent * Removed unnecessary form button border * Alerts page tables now full width * Allowing the Detail cell to be styled * Updated the page structure to be the same as the scheduled page – with better progress indicator * Concurrency limits page now full width * Fixed function name label extending too far * Improved the button sizes on the replay modal * Added a minimal style to the info panels * Make sure you can always see the info and unlock staging info when the table scrolls * Info panel style updated to match the others * Removed the duplicate header * Usage page tables now full width * Personal access token page using full width tables * Small CSS fixes * Deployments page table now full width * Deploy page scrolls properly and pagination stays on screen if table is scrollable * TableCellMenu using the correct popover * Added missing button variant * Button group is now named so it doesn’t cause conflicts * Added docs link to the Run page * Typo * Added a link to the test page if you have no runs * Make the table sticky cell buttons more obvious * Env var page now using the new popover menu * Improved the menu items * Copy tweak * Made the icons bright on hover * Updated the cancel run dialog * Improved the API keys sticky menu behaviour * Alerts table menu using the new popover * Correct sized buttons in the Env Var page table menu * Fixed weird Dialog text styling * Schedules page blank state using correct docs button style * Copy tweak * Menu buttons styled correctly * Added close buttons to the deploy dialogs * Made the Dialog footer match the Form footer style * Fixed feedback form buttons * Fixed cancel run dialog * Test page uses the taskIdentifier in the path * Test task button working * Added Spline component for 404 logo to remove spline logo * Removed todo comment * Added a docs button to the PAT page --------- Co-authored-by: Matt Aitken <matt@mattaitken.com> |
||
|
nicktrn
|
212f8539c3 |
Automatically reattempt after internal errors (#1424)
* refactor finalize run service * refactor complete attempt service * remove separate graceful exit handling * refactor task status helpers * clearly separate statuses in prisma schema * all non-final statuses should be failable * new import payload error code * store default retry config if none set on task * failed run service now respects retries * fix merged task retry config indexing * some errors should never be retried * finalize run service takes care of acks now * execution payload helper now with single object arg * internal error code enum export * unify failed and crashed run retries * Prevent uncaught socket ack exceptions (#1415) * catch all the remaining socket acks that could possibly throw * wrap the remaining handlers in try catch * New onboarding question (#1404) * Updated “Twitter” to be “X (Twitter)” * added Textarea to storybook * Updated textarea styling to match input field * WIP adding new text field to org creation page * Added description to field * Submit feedback to Plain when an org signs up * Formatting improvement * type improvement * removed userId * Moved submitting to Plain into its own file * Change orgName with name * use sendToPlain function for the help & feedback email form * use name not orgName * import cleanup * Downgrading plan form uses sendToPlain * Get the userId from requireUser only * Added whitespace-pre-wrap to the message property on the run page * use requireUserId * Removed old Plain submit code * Added a new Context page for the docs (#1416) * Added a new context page with task context properties * Removed code comments * Added more crosslinks * Fix updating many environment variables at once (#1413) * Move code example to the side menu * New docs example for creating a HN email summary * doc: add instructions to create new reference project and run it locally (#1417) * doc: add instructions to create new reference project and run it locally * doc: Add instruction for running tunnel * minor language improvement * Fix several restore and resume bugs (#1418) * try to correct resume messages with missing checkpoint * prevent creating checkpoints for outdated task waits * prevent creating checkpoints for outdated batch waits * use heartbeats to check for and clean up any leftover containers * lint * improve exec logging * improve resume attempt logs * fix for resuming parents of canceled child runs * separate SIGTERM from maybe OOM errors * pretty errors can have magic dashboard links * prevent uncancellable checkpoints * simplify task run error code enum export * grab the last, not the first child run * Revert "prevent creating checkpoints for outdated batch waits" This reverts commit |
||
|
Eric Allam
|
2d8a41b18b |
feat: realtime (#1402)
* Denormalize run tags, increase character limit to 128 * WIP realtime subscribing to runs * extracted the stream stuff into core, made it more reusable * WIP tags * Remove tags for now because it’s not support in electric * Support async iterables, readable stream, and callback style subscription styles * Remove tags streaming endpoint * Add realtime rate limits and scope them to the /realtime path * WIP rate limt per org * Introduce per org rate limits * WIP JWT auth * Move migrations into new internal db package * Resolve pnpm lock file * Authenticating to the realtime API with JWTs are working * realtime in the client * Created react-hooks package and starting to move stuff in there * Improve types for hooks * schema tasks * Added useBatch hook * build uploadthing/fal demo and change how run metadata is synced to the server * tweaks * WIL realtime concurrency tracking * Implement test for realtime client using testcontainers also updated electric to latest version * Allow customizing the expiration time of the automatic JWT created after triggering a task * Add support for subscribing to run tags * Improve auth types and API * finalize the realtime API * Fixed some example stuff * Allow up to 10 run tags * Remove core from docker-provider tsconfig paths to prevent it from being typechecked * do the same for the kubernetes provider * Fixing some typecheck errors * Fix webapp type errors * Update @trigger.dev/platform to 1.0.13 * Fix attw error * Remove from/to in subscribeToRuns query params * Add tests for the rate limit middleware and add custom JWT rate limits * turn off webapp test parallelism * Finish renaming jwt -> publicAccessToken and automatically give the JWT read access to the tags when using trigger * Add changeset * Attempt to fix unit tests in CI * Skip running the auth rate limit middleware tests for now * Try a beefier machine * Try and run webapp tests separately * Setup env vars * Make sliding window test more reliabile |
||
|
Matt Aitken
|
3d67bb81e9 | Removed run-engine from the webapp package.json/tsconfig | ||
|
Matt Aitken
|
f2babbf637 |
Internal packages (testcontainers, redis-worker and zod-worker) (#1392)
* Some notes on the new run engine * lockfile with setup for the run engine * Documenting where TaskRun is currently mutated, to try figure out the shape of the new system * Added notes about how triggering currently works * Details about when triggering happens * Lots of notes about waitpoints * Started scaffolding the RunEngine * Sketch of Prisma waitpoint schema while it’s fresh in my mind * Got Prisma working with testcontainers * Use beforeEach/afterEach * Simple Prisma and Redis test * Return Redis options instead of a client * Simplified things * A very simple FIFO pull-based queue to check the tests working properly * Use vitest extend * Separate redis, postgres and combined tests for faster testing * Some fixes and test improvements * Pass a logger into the queue * A queue processor that processes items from the given queue as fast as it can * Test for retrying an item that wasn’t processed * First draft of waitpoints in the Prisma schema * Remove the custom logger from the test * Added a completedAt to Waitpoint * Notes on the flow for an execution starting * Added redlock, moved some files around * Starting point for the TaskRunExecutionSnapshot table * Added relationships to TaskRunExecutionSnapshot * Change some tsconfig * Moved some things around * Added some packages * WIP on the RunQueue * Fix for some imports * Key producer with some tests * Removed the nv type from the keys… it’s not useful to do global queries * Passing unit tests for all the public key producer functions * Some basic tests passing for the RunQueue * Simple enqueue test working * Enqueue and dequeue for dev is working * Don’t log everything during the tests * Enqueuing/dequeuing from the shared queue is working * Tests for getting a shared queue * The key producer sharedQueue can now be named, to allow multiple separate queues * The key producer uses the name of the queue as the input * Extra info in the Prisma schema * Dequeuing a message gets the payload and sets the task concurrency all in one Lua script * Adding more keys so we can read the concurrency from the queue * Setting the concurrency with dequeue and enquque is working * Improved the tests and fixed some bugs * Acking is resetting the concurrencies * Check the key has been removed after acking * Nacking is working * Changed the package to CommonJS + Node10 so it works with Redlock * Moved the database, otel and emails packages to be in internal-packages * Moved some Prisma code to the database package * Started using the RunEngine for triggering * Progress on run engine triggering, first waitpoint code * Create a delay waitpoint * Moved ZodWorker to an internal package so it can be used in the run engine as well as the webapp * Web app now uses the zod worker package * Added parseNaturalLanguageDuration to core/apps * internal-packages/zod-worker in the lockfile * Pass in the master queue, remove old rebalance workers code * Add masterQueue to TaskRun * Fixed the tests * Moved waitpoint code into the run engine, also the zod worker * Completing waitpoints * An experiment to create a new test container with environment * More changes to triggering * Started testing triggering * Test for a run getting triggered and being enqueued * Removed dequeueMessageInEnv * Update dev queue tests to use the shared queue function * Schema changes for TaskRunExecutionSnapshot * First execution snapshot when the run is created. Dequeue run function added to the engine * Separate internal package for testcontainers so they can be used elsewhere * Remove the simple queue and testcontainers from the run-engine. They’re going to be separate * Fix for the wrong path to the Prisma schem,a * Added the testcontainers package to the run-engine * redis-worker package, just a copy of the simple queue for now * The queue now uses Lua to enqueue dequeue * The queue now has a catalog and an invisible period after dequeuing * Added a visibility timeout and acking, with tests * Added more Redis connection logging, deleted todos * Visibility timeouts are now defined on the catalog and can be overridden when enqueuing * Dequeue multiple items at once * Test for dequeuing multiple items * Export some types to be used elsewhere * Partial refactor of the processor * First stab at a worker with concurrency and NodeWorkers * Don’t have a default visibility timeout in the queue * Worker setup and processing items in a simple test * Process jobs in parallel with retrying * Get the attempt when dequeuing * Workers do exponential backoff * Moved todos * DLQ functionality * DLQ tests * Same cluster for all keys in the same queue * Added DLQ tests * Whitespace * Redis pubsub to redrive from the worker * Fixed database paths * Fix for path to zod-worker * Fixes for typecheck errors, mostly with TS versions and module resolution * Redlock required a patch * Moved the new DB migrations to the new database package folder * Remove the run-engine package * Remove the RunEngine prisma schema changes * Delete triggerTaskV2 * Remove zodworker test script (no tests) * Update test-containers readme * Generate the client first * Use a specific version of the prisma package * Generate the prisma client before running the unit tests |
||
|
nicktrn
|
c5488df8ca |
Fix CLI downgrade check (#1380)
* fix downgrade detection * remove unused semver package from webapp * add changeset |
||
|
Eric Allam
|
ba3c5bdf33 |
Adding missing task run hierarchy to TaskRun table (#1332)
* Add task run hierarchical relationships to the database * Add depth and related runs to the retrieve run API response * Remove prisma optimize * restructure the migrations to create the index concurrently * Delete these tsbuildinfo files * Fix type error by adding depth to the run list presenter * Cleanup the task hierarchy, share more code * Remove some fields from the list run response |
||
|
Matt Aitken
|
9a7ad926aa |
ElectricSQL run page (hidden page for now) (#1297)
* WIP on using react-window-splitter
* WIP with new resizable panels and SSR
* Use the cookie package
* Resizable storybook page
* Increase indexing memory limit
* Fixed v2 usage meter displaying when on paid plan (#1255)
* Fixed v2 usage meter displaying when on paid plan
* Show the free usage panel only for v3 projects
* Concurrency page and more accurate tracking (#1252)
* Initial TaskRunConcurrencyTracker implementation
* MARQS calls a subscriber to events
* When enqueuing add the extra required metadata
* Track concurrency per environment for tasks too
* Admin page for global concurrency
* Use the new concurrency tracker on the tasks page
* Useful performance test task
* getAllTaskIdentifiers()
* New page for concurrency
* BackgroundWorkerTask index for quick lookup of task identifiers
* Added a way to get concurrency for environments
* Added upgrade/request more concurrency button
* Queued task column working
* Use defer and suspense
* Added queue column to the concurrency environments table
* Some comments added for clarity
* Fixed bad log message
* Sidemenu: move lower and rename to “Concurrency limits”
* Only show the environments, not tasks. Renamed to “Concurrency limits”
* v3: fix unfreezable state crashes for runs with multiple waits (#1253)
* support named capture groups
* write crash errors to attempt.error
* make restored pod names unique per checkpoint
* use last eight characters of checkpoint id instead
* add more chaos monkey env vars
* Ignore unfreezable states
* prevent excessive queue config parsing errors
* handle dependency resume edge case
* better entry point logging
* ignore checkpoint cancellation timeouts
* add missing idempotency keys to wait for dep replays
* remove checkpoints between attempts
* fix retry container names on kubernetes
* add changeset
* fix types
* bring back internal duration timers
* Added more logging to TaskRunConcurrencyTracker and some more try/catches
* Call subscriber.messageDequeued in dequeueMessageInSharedQueue
* Added messageReplaced to concurrency tracking (when freezing)
* Added depenenciesToBundle guide to bundle all packages
* Include the old message data when replacing, so we get the projectId etc.
* Fix restored container names
* Fix for schedule page not scrolling
* Added a description panel to the Concurrency admin page
* chore: Update version for release (beta) (#1256)
Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>
* Release 3.0.0-beta.53
* Added a note to use batchTrigger() instead of trigger()
* The latest react-window-splitter fixes the ESM issues
* Set sensible defaults for the run page
* Deployments page
* Test page
* Schedules page
* Latest version of react-window-splitter (0.2.5)
* Updated to the latest version: react-window-splitter
* Callout if runs don’t start right away now has some top margin
* Small padding fix
* styled the handle focus state
* Added isStaticAtRest prop to resizable panel
* Updated resizable storybook
* Inline code blocks behave nicer when text wraps
* Added ElectricSQL to docker-compose, available on 3060
* Extracted some logic out of the eventRepository for getting a trace. This will be used on the frontend
* Use the new util
* More restructuring ready to use the trace summary from the frontend
* Using ElectricSQL for the run page data
* Min size for resizable panel on test page
* Don’t load the trace in the RunPresenter anymore
* Fix for the resizable panels on the run page
* Added overflow hidden to the panel group
* min size for the test page left hand panel
* Updated to latest window-splitter version
* Removed unused const
* One fix for client-server mismatch
* Slight improvement in the loading state
* Restructured the page so the loading is better
* Improvement to the loading states
* Improved the loading behaviour with the inspector
* WIP on auth, having problems with it
* Upgrade Remix to 2.9.1 (same as PR #1096)
* Switched structure around again so we only call the useTrace hook from the client
* Added auth to the sync
* Overscan more rows in the tree view
* Fix for TS error
* Remove duplicate import
* Revert "Upgrade Remix to 2.9.1 (same as PR #1096)"
This reverts commit
|
||
|
Eric Allam
|
87e46d497b | projects now have a dedicated build server (#1281) | ||
|
Matt Aitken
|
09e6f58e84 |
Revert "ElectricSQL run page and react-window-splitter (#1277)"
This reverts commit
|
||
|
Matt Aitken
|
5231381717 |
ElectricSQL run page and react-window-splitter (#1277)
* WIP on using react-window-splitter
* WIP with new resizable panels and SSR
* Use the cookie package
* Resizable storybook page
* Increase indexing memory limit
* Fixed v2 usage meter displaying when on paid plan (#1255)
* Fixed v2 usage meter displaying when on paid plan
* Show the free usage panel only for v3 projects
* Concurrency page and more accurate tracking (#1252)
* Initial TaskRunConcurrencyTracker implementation
* MARQS calls a subscriber to events
* When enqueuing add the extra required metadata
* Track concurrency per environment for tasks too
* Admin page for global concurrency
* Use the new concurrency tracker on the tasks page
* Useful performance test task
* getAllTaskIdentifiers()
* New page for concurrency
* BackgroundWorkerTask index for quick lookup of task identifiers
* Added a way to get concurrency for environments
* Added upgrade/request more concurrency button
* Queued task column working
* Use defer and suspense
* Added queue column to the concurrency environments table
* Some comments added for clarity
* Fixed bad log message
* Sidemenu: move lower and rename to “Concurrency limits”
* Only show the environments, not tasks. Renamed to “Concurrency limits”
* v3: fix unfreezable state crashes for runs with multiple waits (#1253)
* support named capture groups
* write crash errors to attempt.error
* make restored pod names unique per checkpoint
* use last eight characters of checkpoint id instead
* add more chaos monkey env vars
* Ignore unfreezable states
* prevent excessive queue config parsing errors
* handle dependency resume edge case
* better entry point logging
* ignore checkpoint cancellation timeouts
* add missing idempotency keys to wait for dep replays
* remove checkpoints between attempts
* fix retry container names on kubernetes
* add changeset
* fix types
* bring back internal duration timers
* Added more logging to TaskRunConcurrencyTracker and some more try/catches
* Call subscriber.messageDequeued in dequeueMessageInSharedQueue
* Added messageReplaced to concurrency tracking (when freezing)
* Added depenenciesToBundle guide to bundle all packages
* Include the old message data when replacing, so we get the projectId etc.
* Fix restored container names
* Fix for schedule page not scrolling
* Added a description panel to the Concurrency admin page
* chore: Update version for release (beta) (#1256)
Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>
* Release 3.0.0-beta.53
* Added a note to use batchTrigger() instead of trigger()
* The latest react-window-splitter fixes the ESM issues
* Set sensible defaults for the run page
* Deployments page
* Test page
* Schedules page
* Latest version of react-window-splitter (0.2.5)
* Updated to the latest version: react-window-splitter
* Callout if runs don’t start right away now has some top margin
* Small padding fix
* styled the handle focus state
* Added isStaticAtRest prop to resizable panel
* Updated resizable storybook
* Inline code blocks behave nicer when text wraps
* Added ElectricSQL to docker-compose, available on 3060
* Extracted some logic out of the eventRepository for getting a trace. This will be used on the frontend
* Use the new util
* More restructuring ready to use the trace summary from the frontend
* Using ElectricSQL for the run page data
* Min size for resizable panel on test page
* Don’t load the trace in the RunPresenter anymore
* Fix for the resizable panels on the run page
* Added overflow hidden to the panel group
* min size for the test page left hand panel
* Updated to latest window-splitter version
* Removed unused const
* One fix for client-server mismatch
* Slight improvement in the loading state
* Restructured the page so the loading is better
* Improvement to the loading states
* Improved the loading behaviour with the inspector
* WIP on auth, having problems with it
* Upgrade Remix to 2.9.1 (same as PR #1096)
* Switched structure around again so we only call the useTrace hook from the client
* Added auth to the sync
* Overscan more rows in the tree view
* Fix for TS error
* Remove duplicate import
* Revert "Upgrade Remix to 2.9.1 (same as PR #1096)"
This reverts commit
|
||
|
Eric Allam
|
f9ec66c562 |
v3: new build system (#1265)
* upgrade @opentelemetry packages to the latest versions * remove v2 only packages, will be moved to a dedicated repo * remove more v2 code and run pnpm install * use the npm yalt package in the webapp * convert @trigger.dev/core to tshy * Switch from jest to vitest in @trigger.dev/core * Fixed core test * move core-backend code into core subpath export * convert @trigger.dev/sdk to tshy * Removed hono * move core-apps to core/v3/apps, remove core-apps, start converting cli-v3 * Fix up some of the commands * cli now building and loadable * using package-json-from-dist to get package version now in core and cli * dev command WIP * cleaned up some repetition and structure of the entry point stuff * bringing back the background worker stuff * Indexing of the v3 catalog * getting closer to executing dev runs... * centralize dev logging using event emitter * Move indexing to it’s own entry point, simplify code * dev runs working * Get instrumentation to work with openai * debugging achieved internally * provide worker files as part of the worker creation on the server * support for cjs and esm javascript * Fixed timeout * worker manifest now has the config path * auto-upgrade config to non-deprecated alternatives * Adding package preview release * deployment WIP * improve the syncEnvVars output and adapt resolveEnvVars * WIP bun runtime * WIP bun support * seed tasks with the machine preset if listed in the config * deploy run executions WIP, extracted TaskRunProcess into 1 place * deployed tasks running and executing 🎉 * support for waits and better flushing & process cleanup * Fixed the heartbeating * Better warning messages * Improve and unify the indexing between dev and deploy * Support for external deps that need node-gyp to build * build extensions can now install custom packages and run instructions in the image. Also prisma extension now works and also works with multiple schema files * Add back in the main/types/module to sdk * dev no longer is Ink/React, grace period for disconnections in dev * Fix the changeset config * More changeset fixes * Remove config packages * More changeset fixes * Fixed typescript issues (needed to revert back to zod 3.22.3 * Fix pr_checks workflow * Remove the prepare script * Fixed tests and package versions * Remove cli test script * Remove packages from tailwind watch paths * Add repo to public packages * Just commit the generated files and do the building at dev time * Try and get pkg.pr.new working * Try again * Fix emitDecoratorMetadata importing named export from typescript * config file backwards compat with export const config * Fixed issue where import errors weren’t coming through * p-retry is a prod dep * typescript needs to be a prod dependency for emitDecoratorMetadata * Add better debug logging to help track down import-in-the-middle bug * An external is only considered resolvable if it resolves to the same path as the collected external * Fix runtime checks to allow >=18.20 * Move extensions to a new build package * Fixed building packages in dockerfile * Remove the e2e test from publish workflow for now * Don’t treat pkg.pr.new versions has needing upgrading * making sure config handleError works, and discovered path aliases don’t work in config files * Strip empty string env vars so they accidentally override real values * Couple of things * Update version to use preview instead of beta * Hopefully fix re-attempts with >30s delay * Match socket emit messages to current latest in main * Initial guide * Go back to beta * Go back to the preview, and update guide to use pr preview tags * Go back to beta --------- Co-authored-by: Matt Aitken <matt@mattaitken.com> |
||
|
Matt Aitken
|
6293290fb9 | Better prediction of usage (uses Linear regression) | ||
|
Jacob Paris
|
d934feb02a |
feature: add no-trigger-core-import (#1175)
* feature: add no-trigger-core-import * fix: handle aliases and limit to exported modules * fix: use longest export map name * fix: support export type and interface * fix: simplify message when export map is needed * feature: fine grained modules for core * fix: add prerequisite lint plugins * fix: why do I have to fix these components again??? * fix: tests * Add eslint config and vscode settings * Create yellow-roses-arrive.md * Remove the vscode linting This is temporary so no one fixes these until we merge the remix branch --------- Co-authored-by: Eric Allam <eric@trigger.dev> Co-authored-by: Eric Allam <eallam@icloud.com> Co-authored-by: Matt Aitken <matt@mattaitken.com> |
||
|
Matt Aitken
|
72f3a4b128 |
Usage and billing (v3) (#1201)
* v3 subscription endpoints * Use pnpm linked billing package during development * Moved v2 billing components into a subfolder * Select plan using the real data * Improved v3 plan display * Use new api response that doesn’t require a Stripe call * Free flow is working * Added GitHub modal and verified badge * Deleted old request v3 access component/route * Allow setting classes on the Tooltip button * Paid plans working * Redirect from select plan if you’ve got v3 enabled * Loading state improvements * New admin API endpoint to set concurrency across multiple environments * When projects are created, conditionally create staging based on the plan * New billing page working with side menu and stripe portal * Layout, formatting and some plan state improvements/fixes * Don’t show the period if you’re on the free plan * Temporary upgrade callout * Refactored the platform code so it’s easier to call and doesn’t require a isManagedCloud check * Send taskIdentifier to OpenMeter * Side menu * Upgrade prompts * More improvements to the app-wide usage indicators * Early work on usage graphs * Added the usage bar for v3 * Moved code to presenter and now using defer * Added the tasks table to usage * Improved the v3 usage bar if theres’ no usage on a paid plan * If no run data, still render a graph * Usage page errors when defered loading fails * Don’t show the public API key for v3, they’re not used and probably never will be * Improved the upgrade callout and API keys page layout * Only show the “reveal all” toggle if you have environment variables in the table * Replaced Upgrade callout with a more generic InfoPanel component * better panel width * Show conditional upgrade prompts based on plan and number of schedules used * Removed duplicate class * Wider blank state panels for the scheduled page * Wider info panel for the env var page * Blank state now using the info panel * Platform alerts prompt now using the InfoPanel * Deploy blank state uses InfoPanel * Github verified badge padding adjustment * Improved the layout of the page, some style tweaks, organized imports * Better default tooltip style * could be undefined fix * text fix + style updates * Changed the billing icon in the side menu * Billing page layout and style improvements * plan tooltips don’t use dark variant * Don’t highlight the plan on the billing page * Tooltip underlines stand out more * Fixed padding in the PageTitle * Tooltips use the correct cursor * Improved the plan banner on the billing page * Fixed Header1 inconsistent font weight * Fixed issue where input field focus states were being clipped * Fixed large button not having large text size * Added a link to the Get in touch copy and improved the connect to GitHub modal * Select plan page uses the MainCenteredContainer * Better logging from the Loops endpoint because this error finally got hit * Move the ingestion of compute to the platform * Reporting usage of invocations moved to the platform * Get the entitlement before triggering a non-dev task * Contact us enterprise plan button opens the feedback form * Removed Github discussions link from the Feedback panel * Swapped billing icon for credit card * Show a Unlock staging panel on the env var page * Updated staging environment colour * Show a prompt to upgrade to get staging in the new env var modal * Improved the edit env var modal * Implement ability to disable org concurrency * Use common logic for the plans * Use the billing server to get the schedule limits * Some schedules page fixes * More convenient way of getting a limit * Use the new schedule limit * Team member limiting * Made the limit visible on the team page * Limit alerts * Added an index for TaskRun.scheduleId * Remove console.log on schedules page * Added durations to the run table * Tabular numbers * Improved the usage page formatting * Only admins see the compute column on the run table * Include the base cost on the usage stats * Moved the status to the sidebar * Optional table header tooltip * Allow InfoIconTooltips to have customizable content styles * Added a tooltip to the duration header, changed no test to a dash * Removed all references to signing up to v3 from the docs * Switched @trigger.dev/billing to @trigger.dev/platform * Passing up the variant for the InfoIconTooltip * table tooltip max-width fixed * Switch to the published @trigger.dev/platform 1.0.11 * v2 usage page title changed to include “v2" * code theme has a transparent background so it works on any background * duration columns now grouped together nicely at wide screen size * Last duration column fills the width properly * Fix for the per run price being in cents not dollars * Show the total cost with 8 decimal places * Show 8 decimal places in the usage graph tooltip * Moved the UpgradePrompt to the v3 folder * Prepare to use Shadcns chart helpers * Much nicer chart * Small tweaks to the graph * Fix run table col spans for empty/loading messages * We don’t need isManagedCloud in createProject * Hide v3 usage/billing pages if there aren’t v3 projects in your org * Removed unused tooltipStyle * Usage bar now says “Included usage” instead of “Tier limit” if you’re paying * Get the plan/usage data in parallel * The usage page now has a month dropdown and all data is for that calendar month * Ensure the passed date is the 1st of the month * Use the machine presets from the platform package --------- Co-authored-by: James Ritchie <james@jamesritchie.co.uk> Co-authored-by: Eric Allam <eallam@icloud.com> |
||
|
Eric Allam
|
85a543d8ec |
v3: usage tracking (#1163)
* Starting to measure wall time and cpu time in the workers, and reporting that via otel and to completed task run attempts * Move usage tracking outside of the executor * WIP prod usage tracking * WIP * WIP custom fetch to openmeter * Create a usage client * WIP * WIP * Implement new machine preset stuff and send usage reports to OpenMeter from webapp * WIP * Expose usage info to the client * Add usage and cost to TaskEvent * Add ability to globally configure the task machine preset * Report start run usage * Change the machine docs to use presets * setExpirationTime to 24h * Removed logs * Update machines.mdx * Removed console.logs * Handle revalidating JWT tokens * Couple tweaks --------- Co-authored-by: Matt Aitken <matt@mattaitken.com> |
||
|
Matt Aitken
|
bc7bbd4576 |
v2 IngestSendEvent rate limit (#1134)
* Easier to create a rate limiter, use it in the ApiRateLimiter. Upgraded the Upstash package * Always prefix any rate limiter in Redis with “ratelimit:” * By default log when the rate limit is hit * Added rate limiting to IngestSendEvent * Log out the EventRecord id * Increase events.deliverScheduled attempts * INGEST_EVENT_RATE_LIMIT_MAX is optional * Removed old API rate limit code * IngestSendEvent rate limiter is optional. Moved outside of the DB transaction * Log a message out when the rate limiter is created * Return undefined if the rate limit has been crossed |
||
|
Jacob Paris
|
7cb20fbead |
fix: upgrade prism react renderer (#1111)
* fix: upgrade prism react renderer * Update CodeBlock.tsx |
||
|
Eric Allam
|
3f8b6d8fce | v2: Better handle recovering from platform communication errors by auto-yielding back to the platform in case of temporary API failures | ||
|
nicktrn
|
1642fd7baf |
perf: migrate to graphile worker v0.16.6 (#1097)
* migrate to graphile worker v0.16.6 * remove stale docs link * fix jobs cleanup query |
||
|
Eric Allam
|
0e919f56f2 |
Alerts v1 (#1065)
* Alerts v1 * Encrypt alert webhook secrets and allow them to be generated by the server * Alert v1 UI * Remove unnecessary emails * Move to using `@react-email/components` * WIP slack alerts * More slack alerts WIP * Update pnpm lock after rebase * Finish implementing Slack alerts * Use a more error like emoji * New secondary variant for the segmented control * Added a simple checkbox style variant to storybook * Style tweak to the segmented control * UI improvements to the alert modal * Use searchable Select for alerts. Changed default variant for SegmentedControl * Secondary button now using secondary colour * segmented control style tweak * Improved the channel column in the alerts table * Updated logo-mono.png * Updated email styles * Email templates updated to new styles * Don’t log the decrypted secret * await enqueing the deployment alert when an index fails * await enqueing the timeout alert --------- Co-authored-by: James Ritchie <james@jamesritchie.co.uk> Co-authored-by: Matt Aitken <matt@mattaitken.com> |
||
|
Matt Aitken
|
7d6430dc46 |
New filtered dropdown and run filters (#1089)
* Extract out useFilterTasks * WIP on Combobox * Revert "WIP on Combobox" This reverts commit f0a9b2e57c67f24d5e0876630e17a54252ceb86b. * Added ariakit * Started adding a Listbox component * Listbox WIP * More WIP on the list box * More work on the Listbox * WIP on filterable Listbox using headless… but it’s going to be a nightmare to make it good. Wrong approach * WIP using Ariakit * Multiple selection working * WIP in reworking it so search is built in and it’s easier to use * Nicer API for listbox * Allow sections to be rendered with filtering * More progress on the styling of the menu * The empty state * Removed focus outline * Allow filtering based on the section title * Render the shortcut correctly * Allow passing static children to Select and render a separator if a section has no title * Static example and added subtle focus * Allow static data with no items array passed in * Allow shortcuts on select items * Only how heading if true * Tooltip for keyboard shortcut * useShortcutKeys fixes - Can be passed an undefined definition (so we render the same number of hooks always). - Fix for enabledOnInputElements not being used… * Shortcut keys working with lists * Don’t render shortcut numbers higher than 0 * More improvements * WIP on a Link row * Nicer API for the children function * Much nicer shortcut key support * Separated SelectTrigger out * Heading separated out * More styled elements created * Removed unused element * Reordered statuses * Added search context * Fixes for default values * Removed tab elements * Combo box * Simplified it more * Started work on Filter menu * Experiments with filter menu * Filter menu proof of concept working * Nice hook for using a search param * The escape key goes back to the previous menu * Added filter page to storybook * Added other variants * Renamed Select file to OldSelect, implemented Feedback form * Added placeholder support * Added an optional dropdown icon (defaults to a chevron down) * Feedback new Select * Feedback just use static items * New project page * Renamed storybook pages * Lots of progress on the new run filters * Added new tasks filters * Show the applied filters * Added period filtering * Fix for escape resetting the period value to the previous value when escape is pressed * Show the Clear button sooner * Removed unused code * Reworked useSearchParams so multiple values can be set at once… * Renamed file: OldSelect -> SImpleSelect * Fix for new Select not working in forms because props weren’t being passed through * New Select on the schedules page * Better layout when there are lots of run filters * Many style improvements to run filters * Status filter allows you to select as well * Rolled out the menus for the other files * Minor improvements to schedule page and page padding * Latest lockfile |
||
|
Matt Aitken
|
0f0a6884e8 |
Environment variables pasting uses dotenv (#1075)
* Add dotenv package to the webapp frontend, required some polyfills * Use dotenv to parse the pasted env vars. Make the panel wider on larger screens |
||
|
Matt Aitken
|
7ff8f0ebab |
Task and run page improvements (#1076)
* TaskListPresenter: if there are no tasks then don’t do stats queries * RunListPresenter, use BasePresenter and the read replica * Added populate script * Simplified the Runs list query, added live timer * Added TaskRun indexes for the RunList * Status can’t be null now we’re using the TaskRun status * Use defer so the page loads and shows a spinner * Improved the loading style * Get rid of latest run info from the tasks table super slow * Fix for the activity graph tooltip getting clipped * Added a code comment crediting the GitHub issue with the portal fix * Add search to the tasks list * Padding * Fix for the schedules columns not being UTC * Remove unused function |
||
|
Matt Aitken
|
fae93ac747 |
New tasks page (with activity) and multi-user fixes (#1063)
* Fix for “overridden” spelling mistake * Make the task function names grey instead of yellow * The task page now shows each task once * Blankline * WIP on activity bars * Correct size of the graphs * Add data for all days, even if there are no runs * Activity graph is working and has a tooltip * No activity label * Got the bar bg working * Activity chart now has a bg * Added a compound id to make the activity graphs appear faster * Improvements to the activity graph * Tried to disable the bg from animating * Added read replica support to the app, using Nick’s code from his batching PR * Added the read replica to BasePresenter * Added the queued and running columns to the Tasks list * Added avg duration column * Included completed but failed runs in the average duration * Get rid of the old useDevEnvironment hook and remove API keys from the projects query * Unified getting environment info to using a function and doing it in presenters * Removed the path from the tasks table * Onboarding for dev for 2nd+ user. Improved environment sorting. * Removed log * Improved the spacing * Enable staging for new v3 projects |