Files
triggerdotdev--trigger.dev/references
Eric Allam e8ef374fe0 fix(webapp,run-engine): honor per-queue length cap on concurrency-key queues (#3558)
## Summary

Queues that use concurrency keys can no longer bypass the per-queue
length cap, and the "Queued | Running" columns in the dashboard now show
the true total across all CK variants instead of 0.

The cap and the dashboard both relied on `ZCARD` of the base queue key,
but CK-keyed runs live under `<base>:ck:<variant>` keys. Any queue that
used concurrency keys read 0 — letting a single CK variant grow
unbounded past the user's configured cap.

## Fix

Two per-base-queue counters are maintained inside the CK Lua scripts:
`<base>:lengthCounter` and `<base>:runningCounter`. Non-CK
enqueue/dequeue paths are untouched.

Counters are lazy-initialized the first time a CK enqueue (or nack)
lands on a queue: the Lua script sums `ZCARD` across the variants
tracked by `ckIndex`, sets the counter, then `INCR`s. Pre-existing CK
backlog on already-populated queues is captured automatically — no batch
migration required.

`INCR`/`DECR` is gated on `ZADD`/`SADD` returning 1 (a new entry vs an
idempotent no-op), so duplicate enqueues or re-dequeues don't inflate
the counter.

The counter is `SET` with a 24-hour TTL on init. `INCR`/`DECR` do not
extend the TTL, so the counter expires daily and the next CK operation
re-seeds it from `ckIndex`. This bounds any drift that accumulates
during the rolling-deploy overlap window — where old (un-Tracked) and
new (Tracked) webapp instances briefly coexist — to ≤24 hours, with no
admin sweep or background reconciler needed.

Read paths pipeline `ZCARD`/`SCARD` on the base key + `GET` on the
counter and sum. A missing counter is treated as 0, so pure non-CK
queues see the same answer as before.

The counter-aware scripts ship alongside the originals with a `Tracked`
suffix for rolling-deploy safety; a follow-up PR will drop the originals
once this has rolled out.

## Test plan

- [ ] `pnpm run test --filter @internal/run-engine` — 116 tests pass,
including a new `ckCounters.test.ts` covering lazy init from
pre-existing backlog, churn, floor-at-zero, the non-CK regression case,
mixed CK + non-CK on the same base queue, idempotent re-enqueue
(ZADD-already-exists), 24h TTL on the counter, and nack re-seeding after
counter expiry.
- [ ] Verified end-to-end against a live local environment:
- Triggered 24 CK enqueues across 4 variants → `lengthCounter=16`,
`runningCounter=8`, dashboard showed Queued=16 / Running=8 for the CK
queue.
- Set the env queue cap to 16, triggered 12 more enqueues → 8 succeeded,
4 rejected with `QueueSizeLimitExceededError`.
- Deleted the counter on a queue with 31 messages already sitting in CK
variants, triggered one more enqueue → counter materialized to 31 from
the `ckIndex` sum, then INCR'd.
2026-05-12 18:37:19 +01:00
..

Trigger.dev References

Contains code that tests or uses the @trigger.dev/* packages in some way, either by using them to test out a framework adapter, an integration, or parts of the main SDK.

All the dependencies to the @trigger.dev/* packages will be both referenced in the package.json dependencies as workspace:*, as well as using a direct path from the tsconfig.json file like so:

{
  "extends": "@trigger.dev/tsconfig/node18.json",
  "include": ["./src/**/*.ts"],
  "compilerOptions": {
    "baseUrl": ".",
    "lib": ["DOM", "DOM.Iterable"],
    "paths": {
      "@/*": ["./src/*"],
      "@trigger.dev/sdk": ["../../packages/trigger-sdk/src/index"],
      "@trigger.dev/sdk/*": ["../../packages/trigger-sdk/src/*"],
      "@trigger.dev/express": ["../../packages/express/src/index"],
      "@trigger.dev/express/*": ["../../packages/express/src/*"],
      "@trigger.dev/core": ["../../packages/core/src/index"],
      "@trigger.dev/core/*": ["../../packages/core/src/*"],
      "@trigger.dev/integration-kit": ["../../packages/integration-kit/src/index"],
      "@trigger.dev/integration-kit/*": ["../../packages/integration-kit/src/*"],
      "@trigger.dev/github": ["../../integrations/github/src/index"],
      "@trigger.dev/github/*": ["../../integrations/github/src/*"],
      "@trigger.dev/slack": ["../../integrations/slack/src/index"],
      "@trigger.dev/slack/*": ["../../integrations/slack/src/*"],
      "@trigger.dev/openai": ["../../integrations/openai/src/index"],
      "@trigger.dev/openai/*": ["../../integrations/openai/src/*"],
      "@trigger.dev/resend": ["../../integrations/resend/src/index"],
      "@trigger.dev/resend/*": ["../../integrations/resend/src/*"],
      "@trigger.dev/typeform": ["../../integrations/typeform/src/index"],
      "@trigger.dev/typeform/*": ["../../integrations/typeform/src/*"],
      "@trigger.dev/plain": ["../../integrations/plain/src/index"],
      "@trigger.dev/plain/*": ["../../integrations/plain/src/*"],
      "@trigger.dev/supabase": ["../../integrations/supabase/src/index"],
      "@trigger.dev/supabase/*": ["../../integrations/supabase/src/*"],
      "@trigger.dev/stripe": ["../../integrations/stripe/src/index"],
      "@trigger.dev/stripe/*": ["../../integrations/stripe/src/*"],
      "@trigger.dev/sendgrid": ["../../integrations/sendgrid/src/index"],
      "@trigger.dev/sendgrid/*": ["../../integrations/sendgrid/src/*"],
      "@trigger.dev/airtable": ["../../integrations/airtable/src/index"],
      "@trigger.dev/airtable/*": ["../../integrations/airtable/src/*"]
    }
  }
}

Creating a New Reference Project

This guide assumes that you have followed the Contributing.md instructions to set up a local trigger.dev instance. If not, please complete the setup before continuing.

Step-by-Step Instructions

  1. Run an HTTP tunnel: You will need to run an HTTP tunnel to expose your local webapp, it is required for some API calls during building the image to deploy on your local instance. This is optional if you do not plan to test deployment on your local instance.
  • Download the ngrok CLI. This can be done by following the instructions on ngrok's website.
  • Create an account on ngrok to obtain the authtoken and add it to the CLI.
ngrok config add-authtoken <your-auth-token>

Replace the with the token you obtain from ngrok.

  • Run the tunnel.
ngrok http <your-app-port>

Replace the with the webapp port, default is 3030.

  1. Add your tunnel URL to the env: After running the ngrok tunnel, you will see URL in your terminal, it will look something like https://<your-tunnel-address>.ngrok-free.app. Replace the APP_ORIGIN variable with this URL in your .env file in the root of the trigger.dev project.

  2. Run the webapp on localhost:

pnpm run dev --filter webapp --filter coordinator --filter docker-provider
  1. Build the CLI in a new terminal window:
# Build the CLI
pnpm run build --filter trigger.dev

# Make it accessible to `pnpm exec`
pnpm i
  1. Set up a new project in the webapp:
  • Open the webapp running on localhost:3030.
  • Create a new project in the webapp UI.
  • Go to the Project Settings page and copy the project reference id from there.
  1. Copy the hello-world project as a template:
cp -r references/hello-world references/<new-project>

Replace <new-project> with your desired project name.

  1. Update project details:
  • Open <new-project>/package.json and change the name field. (Tip: Use the same name as in the webapp to avoid confusion.)

  • Open <new-project>/trigger.config.ts and update the project field with the project reference you copied from the webapp.

  • Run pnpm i in your <new-project> directory to sync the dependencies.

  1. Authorize the CLI for your project:
pnpm exec trigger login -a http://localhost:3030 --profile local
  1. Run the new project: You can now run your project using the CLI with the following command:
pnpm exec trigger dev --profile local

You can also deploy them against your local instance with the following command:

pnpm exec trigger deploy --self-hosted --load-image --profile local