Eric Allam
454f0c949a
## Summary The trigger-task hotpath used to early-return without a DB query when a caller passed both a queue override and a per-trigger TTL — the hottest configuration on the trigger API. Adding `triggerSource` to the resolver so the runs-list "Source" filter could distinguish STANDARD / SCHEDULED / AGENT runs removed those early-returns, costing +2 DB queries per trigger on non-locked calls and +1 on locked calls. This change caches `BackgroundWorkerTask` metadata (`ttl`, `triggerSource`, `queueId`, `queueName`) in Redis so the resolver can satisfy every caller configuration with a single `HGET` on the warm path. PG fallback on miss back-fills the cache. Follow-up to #3542. ## Design Two key spaces: - `task-meta:env:{envId}` — the "current worker" view, refreshed at every deploy promotion. 24h safety TTL. - `task-meta:by-worker:{workerId}` — used for `lockToVersion` triggers. Immutable post-create. 30d sliding TTL so historical workers age out. Cache writes use Lua scripts via `defineCommand` so `DEL` + `HSET` + `EXPIRE` land atomically — concurrent readers never see the empty intermediate state of a naive pipeline. Read-path back-fill uses single-field upserts so concurrent back-fills don't wipe each other's siblings. The cache lives behind its own `TASK_META_CACHE_REDIS_*` env-var prefix that falls back to the default `REDIS_*` set, so operators can route the cache to a dedicated Redis instance if they want. The service/instance file split (`taskMetadataCache.server.ts` for the pure class, `taskMetadataCacheInstance.server.ts` for the env-wired singleton) mirrors the existing `runsReplicationService` / `runsReplicationInstance` pattern. ## Test plan - [ ] `pnpm run typecheck --filter webapp` - [ ] `pnpm run test ./test/engine/triggerTask.test.ts --run` — 8 existing tests untouched + 5 new tests covering warm cache, cold miss with back-fill, queue + ttl path, by-worker vs env keyspace, and the promotion cache write - [ ] End-to-end against a dev worker: registering writes both keyspaces with the expected TTLs, and `redis-cli HGETALL "tr:task-meta:env:<envId>"` returns the cached entries ## Benchmark Measured `DefaultQueueManager.resolveQueueProperties` against a real Postgres + Redis (vitest `containerTest`, single-host docker). 500 sequential calls and 2,000 parallel calls (concurrency=50) per scenario, request shaped as `{ taskId, queue: "bench-queue", ttl: "5m" }` — the hot path this PR restores. ``` sequential (one in flight at a time): [noop cache (baseline)] n=500 mean=1.423ms p50=1.394ms p95=1.735ms p99=2.629ms max=11.100ms [redis cache, cold ] n=500 mean=1.346ms p50=1.283ms p95=1.688ms p99=2.463ms max=5.058ms [redis cache, warm ] n=500 mean=0.084ms p50=0.078ms p95=0.105ms p99=0.156ms max=1.129ms speedup (warm vs baseline, sequential): 16.95x parallel (concurrency=50): [noop cache (baseline)] n=2000 mean=10.069ms p50=8.850ms p95=14.718ms p99=31.887ms total=405ms ops/s=4,940 [redis cache, warm ] n=2000 mean=0.614ms p50=0.568ms p95=1.189ms p99=1.432ms total=25ms ops/s=80,389 throughput speedup (warm vs baseline, parallel): 16.27x ``` Read: - **Warm cache cuts resolver latency 17×** at p50 — from ~1.4 ms to ~78 µs per call. - **Cold cache is on par with baseline** — the extra `HGET` miss adds <50 µs against the two Postgres queries that follow, so the worst case is not worse than today. - **Under burst load (50 concurrent triggers)**, the baseline's p99 jumps to ~32 ms as Postgres connections queue up; warm stays at ~1.4 ms. The cache moves the saturation point from ~5k ops/s (PG pool) to ~80k ops/s (single-client Redis pipelining). Caveats: single-host docker, local Postgres + Redis, resolver-only measurement (excludes the rest of the trigger transaction). Prod adds region-local Redis RTT (~0.3–0.8 ms) which shifts warm absolute numbers up but keeps the ratio intact.
39 lines
1.2 KiB
TypeScript
39 lines
1.2 KiB
TypeScript
import { Redis } from "ioredis";
|
|
import { defaultReconnectOnError } from "@internal/redis";
|
|
import { env } from "~/env.server";
|
|
import { singleton } from "~/utils/singleton";
|
|
import {
|
|
NoopTaskMetadataCache,
|
|
RedisTaskMetadataCache,
|
|
type TaskMetadataCache,
|
|
} from "./taskMetadataCache.server";
|
|
|
|
export const taskMetadataCacheInstance: TaskMetadataCache = singleton(
|
|
"taskMetadataCacheInstance",
|
|
initializeTaskMetadataCache
|
|
);
|
|
|
|
function initializeTaskMetadataCache(): TaskMetadataCache {
|
|
if (!env.TASK_META_CACHE_REDIS_HOST) {
|
|
return new NoopTaskMetadataCache();
|
|
}
|
|
|
|
const redis = new Redis({
|
|
connectionName: "taskMetadataCache",
|
|
host: env.TASK_META_CACHE_REDIS_HOST,
|
|
port: env.TASK_META_CACHE_REDIS_PORT,
|
|
username: env.TASK_META_CACHE_REDIS_USERNAME,
|
|
password: env.TASK_META_CACHE_REDIS_PASSWORD,
|
|
keyPrefix: "tr:",
|
|
enableAutoPipelining: true,
|
|
reconnectOnError: defaultReconnectOnError,
|
|
...(env.TASK_META_CACHE_REDIS_TLS_DISABLED === "true" ? {} : { tls: {} }),
|
|
});
|
|
|
|
return new RedisTaskMetadataCache({
|
|
redis,
|
|
currentEnvTtlSeconds: env.TASK_META_CACHE_CURRENT_ENV_TTL_SECONDS,
|
|
byWorkerTtlSeconds: env.TASK_META_CACHE_BY_WORKER_TTL_SECONDS,
|
|
});
|
|
}
|