github-featured/triggerdotdev--trigger.dev
2
0
Fork 0
Code Issues Pull Requests Deployments Models Agent Notes Packages Projects Releases Wiki Activity

perf(run-engine): cut two Redis calls from the ck vtime registration path

Browse Source 
The idle lookup only matters on the call that actually registers a variant. ZADD
NX is a no-op on one that is already registered, and its tag is already correct,
so the ZSCORE preceding it was wasted on every enqueue after the first. Doing the
ZADD first and the ZSCORE only when it reports an insert takes the common path
from two ops to one. The per-registration EXPIRE of ckVtimeIdle went too: the
park sites are the only writers that put anything in that key and they set its
TTL themselves, so refreshing it on a call that may never write there was pure
cost.

Applies to all four registration sites: enqueue, enqueue-with-ttl, nack, and the
gated-variant branch in the vtime dequeue. Six redis.call per registration
becomes four in the common case, five or six on the rarer call that registers.

Measured on a saturated benchmark (generator co-located with Redis, 1M
invocations per arm, 3 interleaved cycles, Redis CPU/wall 0.97 on every arm, two
independent cost measures agreeing to 0.04 usec). Against the flag-off enqueue
script at 8.603 usec, the vtime path was 10.903 usec (+26.7%) and is now 10.177
usec (+18.3%), so this removes about 30% of the virtual-time enqueue overhead.
That +26.7% independently reproduces the 26/23/24% total-CPU overhead the
cardinality benchmark measured by a different method.

A probe isolating the block gives the model behind it: roughly 1.38 usec fixed
per EVALSHA plus 0.33 usec per redis.call, linear in call count for O(1)
commands on small keys.

Behaviour is unchanged. Final ckVtime tags are identical across already
registered, unregistered, idle above floor, idle below floor and missing floor
key, and the starvation suite that asserts exact tags still passes.
This commit is contained in:
Wes Mason
2026-08-19 11:56:46 +01:00
parent 2dfc9b1ff0
commit 7d8a7cd234
2 changed files with 40 additions and 26 deletions
Download Patch File Download Diff File Expand all files Collapse all files
internal-packages/run-engine/src/run-queue/index.ts
+37 -25
View File
@@ -4363,16 +4363,20 @@ end
-- never rewound. The start is max(floor, remembered idle tag): a variant that drained and
-- came back would otherwise be handed full credit at the floor on every re-enqueue, which
-- starves any variant carrying a persistent backlog.
-- The idle lookup only matters when this call is what registers the variant: ZADD NX is
-- a no-op on an already-registered one, and its tag is already correct. Doing the ZADD
-- first and the ZSCORE only on the registering call takes the common path from two ops to
-- one. Measured saturated: 0.33 usec of Redis CPU per redis.call removed, and the pair of
-- reductions here is ~30% of the vtime enqueue overhead.
local vfloor = redis.call('GET', ckVtimeFloorKey) or '0'
local vstart = tonumber(vfloor)
local vidle = redis.call('ZSCORE', ckVtimeIdleKey, queueName)
if vidle and tonumber(vidle) > vstart then
vstart = tonumber(vidle)
if redis.call('ZADD', ckVtimeKey, 'NX', vfloor, queueName) == 1 then
local vidle = redis.call('ZSCORE', ckVtimeIdleKey, queueName)
if vidle and tonumber(vidle) > tonumber(vfloor) then
redis.call('ZADD', ckVtimeKey, 'XX', vidle, queueName)
end
end
redis.call('ZADD', ckVtimeKey, 'NX', tostring(vstart), queueName)
redis.call('EXPIRE', ckVtimeKey, stateTtl)
redis.call('EXPIRE', ckVtimeFloorKey, stateTtl)
redis.call('EXPIRE', ckVtimeIdleKey, stateTtl)
-- Rebalance master queue with ck:* member
local earliestIdx = redis.call('ZRANGE', ckIndexKey, 0, 0, 'WITHSCORES')
@@ -4510,16 +4514,20 @@ end
-- never rewound. The start is max(floor, remembered idle tag): a variant that drained and
-- came back would otherwise be handed full credit at the floor on every re-enqueue, which
-- starves any variant carrying a persistent backlog.
-- The idle lookup only matters when this call is what registers the variant: ZADD NX is
-- a no-op on an already-registered one, and its tag is already correct. Doing the ZADD
-- first and the ZSCORE only on the registering call takes the common path from two ops to
-- one. Measured saturated: 0.33 usec of Redis CPU per redis.call removed, and the pair of
-- reductions here is ~30% of the vtime enqueue overhead.
local vfloor = redis.call('GET', ckVtimeFloorKey) or '0'
local vstart = tonumber(vfloor)
local vidle = redis.call('ZSCORE', ckVtimeIdleKey, queueName)
if vidle and tonumber(vidle) > vstart then
vstart = tonumber(vidle)
if redis.call('ZADD', ckVtimeKey, 'NX', vfloor, queueName) == 1 then
local vidle = redis.call('ZSCORE', ckVtimeIdleKey, queueName)
if vidle and tonumber(vidle) > tonumber(vfloor) then
redis.call('ZADD', ckVtimeKey, 'XX', vidle, queueName)
end
end
redis.call('ZADD', ckVtimeKey, 'NX', tostring(vstart), queueName)
redis.call('EXPIRE', ckVtimeKey, stateTtl)
redis.call('EXPIRE', ckVtimeFloorKey, stateTtl)
redis.call('EXPIRE', ckVtimeIdleKey, stateTtl)
-- Rebalance master queue with ck:* member
local earliestIdx = redis.call('ZRANGE', ckIndexKey, 0, 0, 'WITHSCORES')
@@ -5547,13 +5555,13 @@ local function tryServe(ckQueueName, mayRaiseFloor)
-- The TTL only needs setting when this actually registered something, since that is the
-- path that can recreate a ckVtime key which expired out from under a live ckIndex.
-- NEW: registers at max(floor, remembered idle tag), same rule as the enqueue path, so
-- a variant that drained under the gate does not come back with full credit.
local gateStart = floor
local gateIdle = redis.call('ZSCORE', ckVtimeIdleKey, ckQueueName)
if gateIdle and tonumber(gateIdle) > gateStart then
gateStart = tonumber(gateIdle)
end
if redis.call('ZADD', ckVtimeKey, 'NX', tostring(gateStart), ckQueueName) == 1 then
-- a variant that drained under the gate does not come back with full credit. Ordered
-- the same way too: the ZADD NX decides whether the idle lookup is worth doing at all.
if redis.call('ZADD', ckVtimeKey, 'NX', tostring(floor), ckQueueName) == 1 then
local gateIdle = redis.call('ZSCORE', ckVtimeIdleKey, ckQueueName)
if gateIdle and tonumber(gateIdle) > floor then
redis.call('ZADD', ckVtimeKey, 'XX', gateIdle, ckQueueName)
end
redis.call('EXPIRE', ckVtimeKey, stateTtl)
end
end
@@ -6461,16 +6469,20 @@ end
-- never rewound. The start is max(floor, remembered idle tag): a nack after the variant
-- drained would otherwise hand back full credit at the floor, which is the same starvation
-- the enqueue path guards against.
-- The idle lookup only matters when this call is what registers the variant: ZADD NX is
-- a no-op on an already-registered one, and its tag is already correct. Doing the ZADD
-- first and the ZSCORE only on the registering call takes the common path from two ops to
-- one. Measured saturated: 0.33 usec of Redis CPU per redis.call removed, and the pair of
-- reductions here is ~30% of the vtime enqueue overhead.
local vfloor = redis.call('GET', ckVtimeFloorKey) or '0'
local vstart = tonumber(vfloor)
local vidle = redis.call('ZSCORE', ckVtimeIdleKey, messageQueueName)
if vidle and tonumber(vidle) > vstart then
vstart = tonumber(vidle)
if redis.call('ZADD', ckVtimeKey, 'NX', vfloor, messageQueueName) == 1 then
local vidle = redis.call('ZSCORE', ckVtimeIdleKey, messageQueueName)
if vidle and tonumber(vidle) > tonumber(vfloor) then
redis.call('ZADD', ckVtimeKey, 'XX', vidle, messageQueueName)
end
end
redis.call('ZADD', ckVtimeKey, 'NX', tostring(vstart), messageQueueName)
redis.call('EXPIRE', ckVtimeKey, stateTtl)
redis.call('EXPIRE', ckVtimeFloorKey, stateTtl)
redis.call('EXPIRE', ckVtimeIdleKey, stateTtl)
-- Rebalance master queue with ck:* member
local earliestIdx = redis.call('ZRANGE', ckIndexKey, 0, 0, 'WITHSCORES')
internal-packages/run-engine/src/run-queue/tests/ckVtimeStarvation.test.ts
+3 -1
View File
@@ -61,7 +61,9 @@ function createQueue(
masterQueueConsumersDisabled: true,
workerOptions: { disabled: true },
...(vtimeEnabled
? { ckVirtualTimeScheduling: { enabled: true, ...(idleMaxEntries ? { idleMaxEntries } : {}) } }
? {
ckVirtualTimeScheduling: { enabled: true, ...(idleMaxEntries ? { idleMaxEntries } : {}) },
}
: {}),
queueSelectionStrategy: new FairQueueSelectionStrategy({
redis: { keyPrefix, host: redisContainer.getHost(), port: redisContainer.getPort() },