Commit Graph

1694 Commits

Author SHA1 Message Date
Eric Allam 70b4b1257e fix: Logging large objects is now much more performant and uses less memory (#2263)
* Logging large objects performance fixes

* Add changeset

* docs: new otel attribute count defaults

* Scope the switch/case statement with braces

* Add tests for nested map and sets

---------

Co-authored-by: nicktrn <55853254+nicktrn@users.noreply.github.com>
2025-07-14 13:16:04 +01:00
Eric Allam 46dad7dc76 fix: prevent processKeepAlive OOM error process reuse (#2261)
* Improve TaskRunProcess health detection so we don't try and reuse an unhealthy process

This was happening after the process was killed internally, like by an OOM error

* Add changeset
2025-07-11 10:50:05 +01:00
nicktrn 4fdf23b38b fix(deploy): fail in CI with descriptive message and instructions if no access token set (#2258)
* don't try interactive login in ci, link to docs

* add ci note to cli deploy docs

* add ci note to github actions

* add changeset

* make cron test less strict
2025-07-10 17:28:12 +01:00
nicktrn 0c71dc74df feat: add node 22 and bun runtime support with version display (#2254)
* update node-22 image

* update bun image

* disable io_uring

* fix fallback bun path

* prevent duplicate warnings

* add runtime and version to deployments

* runtime icons

* fallback to nodejs

* prevent empty table cell menu

* log if local build on deploy

* pass io_uring env var to child

* denormalize runtime and version, display on run details

* add changesets

* disable pr checks for changeset commits..

* add runtime data to deployed bg workers
2025-07-10 16:43:10 +01:00
Eric Allam 6ff0b923e1 New internal idempotency implementation for trigger and batch trigger (#2256)
* Introduce request idempotency to prevent duplicate triggers

* Implement request idempotency on trigger

* Use x-trigger-request-idempotency-key header instead

Plus cleanup shared logic in the routes

* Add changeset

* Oops, lets not hardcode a 408

* A couple of improvements
2025-07-10 16:42:32 +01:00
Eric Allam ba2e0cc1c6 Prevent circular references from breaking run logs in task outputs (#2249) 2025-07-09 10:28:12 +01:00
github-actions[bot] 5cff5c45e9 chore: Update version for release (v4-beta) (#2159)
Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
2025-07-04 16:13:20 +01:00
Eric Allam 469a8df0cf Prevent redis worker from enqueuing pending jobs (#2235) 2025-07-04 13:53:32 +01:00
nicktrn 74fdfa93c5 Show project details when running the whoami command (#2231)
* add project details to whoami

* findFirst

* skip project check if no orgs found

* add changeset
2025-07-04 13:40:41 +01:00
Eric Allam e90235b317 fix processKeepAlive disabled performance regression (#2228)
* processKeepAlive disabled performance fix

When processKeepAlive was disabled, the eagerly created TaskRunProcess being discarded and a whole new TaskRunProcess created at execution time.

* Don't use a TaskRunProcess if it's being killed, fixes retryImmediately
2025-07-03 14:21:49 +01:00
Eric Allam 21c70ad673 Serialize metadata to prevent invalid data from breaking run completions 2025-07-02 11:30:13 +01:00
Eric Allam 0c2af6d87c Increase otel attribute limits and make them configurable 2025-07-02 11:30:13 +01:00
Eric Allam 0deb2c640d remove change from process-keep-alive 2025-07-02 11:30:13 +01:00
Eric Allam 0793c2b0f4 address some nick comments in the review 2025-07-02 11:30:13 +01:00
Eric Allam aecee4d35f Don't reset task lifecycle hooks between executions 2025-07-02 11:30:13 +01:00
Eric Allam 9ab590bead Better handle snapshot changes with the new persistent task run process,
Also don’t re-import the task file if the task is already in the resourceCatalog
2025-07-02 11:30:13 +01:00
Eric Allam 2cc1453f25 Always set the cold/warm variant on attempt spans, even in dev 2025-07-02 11:30:13 +01:00
Eric Allam 82f3db6825 Only bootstrap on the cold start execution, and record how many executions in the attempt span 2025-07-02 11:30:13 +01:00
Eric Allam f0bc4ad2bd Don't delete env vars with falsy values 2025-07-02 11:30:13 +01:00
Eric Allam 4d15c8f7cf Better reset in the prod usage manager 2025-07-02 11:30:13 +01:00
Eric Allam 7f3b1c1735 Ensure the usage interval is cleared when resetting 2025-07-02 11:30:13 +01:00
Eric Allam 707bc09dac Make sure maxExecutionsPerProcess is a positive integer 2025-07-02 11:30:13 +01:00
Eric Allam a0f2c982f9 Make sure to clear existing metadata flush timer when resetting 2025-07-02 11:30:13 +01:00
Eric Allam bffd18a6b8 Properly reset listener arrays by setting length to 0 2025-07-02 11:30:13 +01:00
Eric Allam 2018b41d22 TRIGGER -> TRIGGERDOTDEV 2025-07-02 11:30:13 +01:00
Eric Allam c41d63497b Handle immediate retries better 2025-07-02 11:30:13 +01:00
Eric Allam 56146afcb6 TaskRunProcessProvider cleanup is async 2025-07-02 11:30:13 +01:00
Eric Allam 020565c009 Handle dev process pool shutdown errors 2025-07-02 11:30:13 +01:00
Eric Allam a414e65efe handle env vars that have been removed between executions 2025-07-02 11:30:13 +01:00
Eric Allam 38fe854559 Make refreshing a task run process work correctly 2025-07-02 11:30:13 +01:00
Eric Allam c24f3a56cc Reset deployed run worker and also get the warm start stuff to work properly 2025-07-02 11:30:13 +01:00
Eric Allam 8798352be3 Super-warm starts in deployed tasks proof 2025-07-02 11:30:13 +01:00
Eric Allam 1a1e70ae3d WIP deployed processKeepAlive 2025-07-02 11:30:13 +01:00
Eric Allam 19bdf3b6ea scope the pool by background version, allow configuring pool params in dev/prod 2025-07-02 11:30:13 +01:00
Eric Allam 427c450391 Fixed metrics, reusing a process counts as a warm start 2025-07-02 11:30:13 +01:00
Eric Allam 9374aea401 Split resource and span attributes 2025-07-02 11:30:13 +01:00
Eric Allam fd3db5f77b Reset execution environment between runs 2025-07-02 11:30:13 +01:00
Eric Allam e2049fcfbf Initial dev process keep alive. moved resource attributes to span attributes because resource attributes are global and immutable 2025-07-02 11:30:13 +01:00
Eric Allam 1be4fcbb82 Fixes the jitter option for the scanConcurrencySets cron job (#2220) 2025-07-02 11:29:54 +01:00
Eric Allam 1b4dabb8ba v4: current concurrency sweeper (#2206)
* v4: current concurrency sweeper

* Fix webapp tests

* Ensure only a single instance performs concurrency sweeping by using redis-worker cron jobs

* Improved the mark phase

* Ensure cron jobs get rescheduled even if the handler throws an error

* Better property names
2025-06-30 16:22:19 +01:00
nicktrn 119bfc0068 Fix update command version detection (#2199)
* fix root package.json resolution

* add changeset
2025-06-27 10:06:36 +01:00
Eric Allam 0f3179f493 Provide realtime skipColumns option via untamperable public access tokens (#2201)
* Provide realtime skipColumns option via untamperable public access tokens

* Fixed realtime client tests

* Create red-rings-marry.md
2025-06-26 15:13:26 +01:00
Eric Allam b87e1c4630 Improve clickhouse client debug logging (#2197)
* Improve clickhouse client debug logging

* Stop hardcoding ScheduleEngineWorker log level

* Add DLQ debug log message
2025-06-26 15:12:54 +01:00
Eric Allam 485f71c8d4 Improve schedule engine performance (#2191)
* Improve schedule engine performance

* Convert an info log to a debug log
2025-06-23 16:55:18 +01:00
Eric Allam fb9abe5289 fix: Incorrect/missing logs on new schedule engine runs (#2185)
* Fix incorrect logs on new schedule engine triggered taskss

Also added the ability to recover schedules in the schedule engine via an Admin API endpoint in the new schedule engine

* Fixed schedule recovery failing test
2025-06-19 15:37:50 +01:00
Eric Allam ea6f66bdfc Improved schedule engine (#2174) 2025-06-17 06:48:05 +01:00
Eric Allam ab432a2d7d Fix realtime re-subscribing stale data issue (#2162)
* Fix realtime re-subscribing stale data issue

Fixes an issue with realtime when re-subscribing to a run, that would temporarily display stale data and the changes. Now when re-subscribing to a run only the latest changes will be vended

* removed logs
2025-06-12 14:58:01 +01:00
Eric Allam e6fb32194e Add environment filter to declarative schedules (#2148) 2025-06-10 12:11:11 +01:00
Eric Allam b38405cb88 Realtime and task run performance improvements (#2158)
* Add createdAt filter to realtime subscribing with tags

* Filter realtime colums and expose ability to skip some columns

* Add sharding support for electric

* Use unkey cache for the created at filter caching

* Remove 2 unused indexes on TaskRun

* Run list now filters by a single runtime environment

* Remove project ID indexes

* Use clickhouse in task list aggregation queries instead of pg (keep pg for self-hosters)

* WIP clickhouse powered runs list
stuff

* Improve the query to get the latest tasks for the task list presenter

* Update the usage task list to use clickhouse

* Implement next runs list powered by clickhouse

* Add new index for TaskRun for the runs list, by environment ID

* Add runTags gin index

* Handle possibly malicious inputs

* Ignore claude settings

* Better handling not finding an environment on the schedule page

* Use ms since epoch in test, not seconds

* Remove unused function

* Fix test

* Use an env var for the realtime maximum createdAt filter duration (defaults to 1 day)

* Fixed the query builder to correct the group by / order by order

* Make sure runs.list still works

* Create small-birds-arrive.md
2025-06-10 12:11:01 +01:00
github-actions[bot] bb1f0a711a Release 4.0.0-v4-beta.21 (#2149)
🚀 Publish Trigger.dev Docker / typecheck (push) Failing after 4s
🚀 Publish Trigger.dev Docker / publish-webapp (push) Has been skipped
🚀 Publish Trigger.dev Docker / publish-worker (push) Has been skipped
🚀 Publish Trigger.dev Docker / publish-worker-v4 (push) Has been skipped
🚀 Publish Trigger.dev Docker / units (push) Failing after 3s
* chore: Update version for release (v4-beta)

* Release 4.0.0-v4-beta.21

---------

Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
Co-authored-by: nicktrn <55853254+nicktrn@users.noreply.github.com>
2025-06-06 10:50:28 +01:00