4 Commits

Author	SHA1	Message	Date
Eric Allam	eeab6bdeac	fix(run-engine): fix queue cache memory leak and replace MemoryStore with LRU cache (#2945) ... - Fix memory leak in RunAttemptSystem queue cache - was keying by runId instead of queue identifier - Replace `@unkey/cache` MemoryStore with new LRUMemoryStore for O(1) operations and better memory bounds ## Problem ### Cache Key Bug The queue cache in `#resolveTaskRunExecutionQueue` was keyed by `runId`, creating one cache entry per run instead of per queue. With 1-2 hour TTLs and 5000 entry soft cap, these accumulated causing memory growth. ### MemoryStore Performance The `@unkey/cache` MemoryStore uses O(n) synchronous iteration for eviction, blocking the event loop at high throughput. ## Solution ### Cache Key Fix Changed cache key from `params.runId` to queue identifier: ```typescript const cacheKey = params.lockedQueueId ?? `${params.runtimeEnvironmentId}:${params.queueName}`; ``` LRU Cache Created LRUMemoryStore adapter using lru-cache package: - O(1) get/set/delete operations - Strict memory bounds (hard max vs soft cap) - No event loop blocking Test Results: | Metric | Before Fix | After Fix | |---|---|---| | Queue cache entries (per 1000 runs) | ~1000 | 1 | | Old space growth | 32.27 MB | 5.45 MB | | Heap growth | 7.21 MB (3.9%) | 4.81 MB (2.6%) | <!-- devin-review-badge-begin --> --- <a href="https://app.devin.ai/review/triggerdotdev/trigger.dev/pull/2945"> <picture> <source media="(prefers-color-scheme: dark)" srcset="https://static.devin.ai/assets/gh-open-in-devin-review-dark.svg?v=1"> <img src="https://static.devin.ai/assets/gh-open-in-devin-review-light.svg?v=1" alt="Open with Devin"> </picture> </a> <!-- devin-review-badge-end -->	2026-01-26 22:23:23 +00:00
Eric Allam	128bc437f6	feat(otel): Add support for storing run spans and log data in Clickhouse (#2567)	2025-10-01 12:41:18 -07:00
nicktrn	fa7f4b1fed	feat(k8s): add placement tags for flexible node selection (#2390) ... * add tier scheduling support to supervisor * add billing info to dequeued message w/o cache * add cache with best effort invalidation * fix invalidate circular dep * add changeset * use new plan type on runs as fallback during dequeue * tidy up * be more explicit with plan type fallback * remove additional billing check from hot path * switch to placement tags * update changeset * update platform package * start using new entitlement response * ensure skipChecks optimization validates at batch level * add optional items to add to queue manager limits * make the bool env helper only accept boolean defaults * remove redundant private field * update placement tag helper to prevent unsupported tags	2025-08-15 09:52:14 +01:00
Eric Allam	14dcc76f93	feat: run.ctx tidying and additions (#2322) ... * Cleanup context and execution creation, cache stuff, add parent and root task run ids * more efficient by using friendly IDs instead of doing joins * metadata.root/parent now reference current run when run has no root/parent * Adding changeset * try to make test less flaky * Clean imports * Another attempt to fix the flaky test * Fix usage by still passing durationMs and costInCents to the execution, just not the run.ctx	2025-07-30 13:47:12 +01:00