Files
triggerdotdev--trigger.dev/apps/webapp/test/setup.ts
T
Katia Bulatova 0b750d00dd feat(webapp): dashboard agent — Watch (#4525)
Watch is the agent noticing something later: you ask it to tell you when
a condition holds, and it answers when it does — or when it can't any
more.

A watch is a **durable one-shot promise**. The condition is checked on a
schedule by deterministic code (no LLM in the checks), the answer lands
in the chat once, and then the watch is over. Ten kinds: three on a run,
five on a queue, error recurrence, health recovery.

## Stack

Stacked on **#4529** (UI), which is stacked on **#4418** (chat, reports,
investigate). Merge those first. **#4516** (storybook gallery) sits on
top of this branch.

## How to review


[**GUIDEBOOK.md**](https://github.com/triggerdotdev/trigger.dev/blob/feat/dashboard-agent-flows-watch/internal-packages/dashboard-agent/GUIDEBOOK.md)
on this branch is the behaviour reference — it states the conditions
rather than the code, so you can predict what happens without running
anything. "The ten watch kinds, and what makes each fire" and "Creating
a watch" describe exactly this PR, and the tables there are the spec the
code is written against.

## What's inside

- **Ten watch kinds**, one deterministic check each
(`dashboardAgentWatch*Checks.ts`), with the spec union in
`dashboard-agent-contracts/src/watch.ts`.
- **Scheduling** — each watch schedules its own next check; due watches
of one `(environment, cadence)` group can be checked together in one
batch pass, with a sweep as the backstop for expiry, redelivery and
retention.
- **Delivery** — the in-chat wake and card, an optional email alert (new
`DASHBOARD_AGENT_WATCH` alert channel, so it shows on the project's
Alerts page with one-click unsubscribe), and an optional investigation
when the outcome needs attention.
- **Submission ledger** — `watch_submissions`, keyed `(chat_id,
client_request_id)`, so a retried card submission replays the recorded
outcome instead of creating a second watch.
- **Watch token** — a delayed-execution credential accepted only by the
watch endpoints, re-checked against the user's live access on every
tick.
- **Unread work** — the panel polls for wakes that landed while it was
closed, so a chat can go unread and light the launcher dot.

## Key decisions

**A check result is a 4-way, and only two of them are verdicts.**
`satisfied` / `terminal_unsatisfied` are answers; `pending` and
`unavailable` are not. Any exception inside any check is caught in one
place and becomes `unavailable` with an unverified observation — a check
that failed is never evidence.

**A completed window is an answer, and whether it is good or bad news is
declared per kind, never inferred.** There is a table for that in the
guidebook: `run_failed` completing its window is *good* news ("hasn't
failed"), `backlog_drain` completing it is not. One rule overrides the
table: a window that completed on an unverified observation is neutral
and says only that the watch ended without a confirmed answer. **An
unreadable source is never a negative answer** — and, because
investigations only open on `attention`, it never starts one either.

**Identity is `(chat, project, environment)` plus the condition,**
enforced by a partial unique index over active rows
(`watches_chat_active_identity_key`), not by the read-then-insert check.
Cadence, window, note and `ticks` are deliberately not part of it. Two
different chats may watch the same thing — a watch is a promise to a
chat.

**The server resolves the target's name, whatever the model calls it.**
The model can't tell a task queue (`task/<id>`) from a custom queue, so
both spellings are tried and the stored one wins — and the rewrite
happens **before** identity and before the row is written, so the
identity, the checks, the link and the wording all see one spelling.

**Freshness fences.** Depth falls back from the live counter to the
newest 60 s ClickHouse bucket, which only counts as current within 60 s
of now. A non-current reading at or below the *quiet line* is refused as
`unavailable` rather than believed, so a stale empty bucket is never
read as "drained". The stall streak is the one piece of carried state:
it lives in the previous check's facts and *freezes* on an unreadable
reading rather than breaking.

**Chain reliability.** There is no shared cron — each watch (or batch
group) schedules its own next tick, so the failure mode to review is the
chain dying. A failed batch check is caught, the next tick is scheduled
anyway and the run resolves rather than failing, so the chain survives a
check that couldn't run; the sweep re-arms groups and finalizes anything
still active past its deadline, even when delivery isn't configured.
Wake redelivery is id-deduped rather than conditional, because the sweep
can't know whether the user was already told. Access is re-authorized on
**every** check against the primary — replica lag would extend access
the user has already lost.

**Wording lives in one place.** `watch-wording.ts` is read by the card,
banner, toast, email and the agent's own narration, and the numbers come
from the frozen observation rather than a fresh read, so a retry
produces the same sentence. Replay reproduces the **recorded** decision
instead of deciding again — the transcript is append-once, so a second
decision would contradict it forever.

**Cancellation is the ending without an answer** — no resolution, no
wake. One exception, decided during testing: a watch the *user*
cancelled leaves a single neutral transcript line ("Stopped watching
…"), keyed off the watch id so a retry can't repeat it. The other four
reasons stay silent.

**Email is opt-in and only a fired watch emails.** An expiry is narrated
in the chat and nowhere else. Both gates (agent access, a configured
email transport) are checked at subscribe time *and* again at delivery,
and the subscription outcome is frozen on the ledger row so a retry
replays it. Neither gate is a plan check.

**One watch offer per turn.** The prompt and the renderer guard this
independently — if the turn already proposed a watch card, the action
button is dropped, because the card is the better affordance. Two eval
cases pin the prompt side: exactly one offer with the line last and the
button after it, and zero offers when the rendered card already carries
one — deterministic assertions, over a real-model run.

## Testing

Unit tests (vitest, testcontainers, no mocks) under
`apps/webapp/test/dashboardAgentWatch*.test.ts` and
`internal-packages/dashboard-agent/src/watch-*.test.ts` cover the
invariants above: the 4-way check results and the freshness fences,
identity/dedup and the submission ledger, queue-name resolution, the
batch chain surviving a failed check, sweep boundaries and alert-once,
tenancy and the watch token's scope, and the wording snapshot. The
load-bearing ones were verified by control-breaking the guard first and
checking the test goes red.

Live-tested end to end against a local stack, following the guidebook:
all ten watch kinds firing and expiring, cancellation, the email pair (a
fired watch mails, an expired one does not), and watch recovery from a
health report.
2026-08-12 09:51:40 +02:00

165 lines
7.2 KiB
TypeScript
Raw Blame History

This file contains ambiguous Unicode characters
This file contains Unicode characters that might be confused with other characters. If you think that this is intentional, you can safely ignore this warning. Use the Escape button to reveal them.
// Load apps/webapp/.env into process.env so env.server's top-level
// EnvironmentSchema.parse(process.env) succeeds in vitest workers.
import { config } from "dotenv";
import path from "node:path";
import { vi } from "vitest";
import type * as IORedisModule from "ioredis";
import type * as TaskMetadataCacheModule from "~/services/taskMetadataCache.server";
config({ path: path.resolve(__dirname, "../.env") });
// CI has no .env and no REDIS_HOST/REDIS_PORT, so import-time guards like
// autoIncrementCounter.server.ts throw and their suites fail to collect. Default
// the pair — the ioredis mock below forces lazyConnect, so nothing ever dials.
process.env.REDIS_HOST ??= "localhost";
process.env.REDIS_PORT ??= "6379";
process.env.SESSION_SECRET ??= "test-session-secret";
process.env.PROVIDER_SECRET ??= "test-provider-secret";
process.env.COORDINATOR_SECRET ??= "test-coordinator-secret";
process.env.MANAGED_WORKER_SECRET ??= "test-managed-worker-secret";
// Worker singletons construct a RedisWorker at import time whose ioredis client
// connects eagerly, so any test importing the service graph opens real Redis
// connections on import — which floods and fails in CI (no Redis). Mock them to
// no-op stubs. Only the worker modules are mocked, never the run store
// (~/v3/runStore.server, ~/db.server), which store-routing tests need real.
function createWorkerStub() {
return {
start: vi.fn(),
stop: vi.fn(),
enqueue: vi.fn().mockResolvedValue(undefined),
enqueueOnce: vi.fn().mockResolvedValue(undefined),
reschedule: vi.fn().mockResolvedValue(undefined),
cancel: vi.fn().mockResolvedValue(undefined),
ack: vi.fn().mockResolvedValue(undefined),
};
}
vi.mock("~/v3/commonWorker.server", () => ({ commonWorker: createWorkerStub() }));
vi.mock("~/v3/batchTriggerWorker.server", () => ({ batchTriggerWorker: createWorkerStub() }));
vi.mock("~/v3/alertsWorker.server", () => ({ alertsWorker: createWorkerStub() }));
// RunEngine and the socket.io server are further singletons that open eager
// ioredis connections at import via the same pattern. No test
// uses these app-level singletons directly (store-routing tests build their own
// engine and run store), so stub them to no-op proxies.
// Recursive no-op proxy: property access at any depth returns another callable
// no-op proxy, so real service tests reaching nested singleton methods (e.g.
// engine.runQueue.updateEnvConcurrencyLimits) don't break on an intermediate stub.
type NoopProxyFn = ((...args: unknown[]) => Promise<undefined>) & Record<string, unknown>;
const noopProxy = (): NoopProxyFn => {
const fn = () => Promise.resolve(undefined);
return new Proxy(fn, {
get: (_target, prop) => (prop === "then" ? undefined : noopProxy()),
apply: () => Promise.resolve(undefined),
}) as unknown as NoopProxyFn;
};
// Beyond the modules mocked above, dozens more app modules construct an
// ioredis client at import time pointed at env-configured Redis, and ioredis
// dials on construction — in CI (no Redis service) that floods ECONNREFUSED at
// shard scale. Force `lazyConnect: true` on every client instead: import-time
// singletons construct but never dial, while anything that actually issues a
// command (tests against live testcontainers) connects on first command
// exactly as before.
vi.mock("ioredis", async (importOriginal) => {
const actual = await importOriginal<typeof IORedisModule>();
// Normalize ioredis's overloaded ctor args — (), (port), (path),
// (port, host), (opts), (port, opts), (port, host, opts), (path, opts) —
// so lazyConnect lands in the options object in every form.
function withLazyConnect(args: unknown[]): unknown[] {
if (args.length === 0) {
return [{ lazyConnect: true }];
}
const last = args[args.length - 1];
if (typeof last === "object" && last !== null) {
return [...args.slice(0, -1), { ...last, lazyConnect: true }];
}
return [...args, { lazyConnect: true }];
}
class LazyRedis extends actual.Redis {
constructor(...args: unknown[]) {
// @ts-expect-error forwarding ioredis's overloaded ctor args
super(...withLazyConnect(args));
}
}
class LazyCluster extends actual.Cluster {
constructor(startupNodes: unknown, options?: Record<string, unknown>) {
// @ts-expect-error forwarding ioredis's ctor args
super(startupNodes, { ...options, lazyConnect: true });
}
}
// Keep the `Redis.Cluster` static alias (`new Redis.Cluster(...)`) working.
// The base class exposes `Cluster` as a getter-only static, so define our
// own property rather than assigning through the inherited getter.
Object.defineProperty(LazyRedis, "Cluster", { value: LazyCluster });
return {
...actual,
default: LazyRedis,
Redis: LazyRedis,
Cluster: LazyCluster,
};
});
// alertsRateLimiter.check() is invoked at runtime by deliverAlert; against
// env-configured Redis each check burns ~20 reconnect cycles before its
// caught error, stalling alert-path tests into timeouts. Allow everything.
vi.mock("~/v3/alertsRateLimiter.server", () => ({
alertsRateLimiter: { check: vi.fn().mockResolvedValue({ allowed: true }) },
}));
// tracePubSub.publish() runs inside eventRepository writes; each publish to
// env-configured Redis stalls ~20 reconnect cycles (errors are allSettled-
// swallowed but awaited), timing out any test that records trace events.
vi.mock("~/v3/services/tracePubSub.server", async () => {
const { EventEmitter } = await import("node:events");
return {
tracePubSub: {
publish: vi.fn().mockResolvedValue(undefined),
subscribeToTrace: vi.fn().mockResolvedValue({
unsubscribe: vi.fn().mockResolvedValue(undefined),
eventEmitter: new EventEmitter(),
}),
},
TracePubSub: class {},
};
});
// Same runtime-stall shape for the task metadata cache (queues concern). CI
// leaves TASK_META_CACHE_REDIS_HOST unset and gets the Noop implementation;
// pin the Noop cache here so env-configured local runs behave identically.
vi.mock("~/services/taskMetadataCacheInstance.server", async () => {
const { NoopTaskMetadataCache } = await vi.importActual<typeof TaskMetadataCacheModule>(
"~/services/taskMetadataCache.server"
);
return { taskMetadataCacheInstance: new NoopTaskMetadataCache() };
});
// The org-data-stores registry singleton is constructed at import (transitively via
// the ClickHouse factory instance, which many presenters pull in). Its ctor fires a
// `forever` pRetry(loadFromDatabase) plus a setInterval reload against db.server's
// $replica; in CI (no Postgres) those retry forever, blocking the worker until any
// awaiting test's hook times out. Stub the instance to a no-op — no unit test uses
// this singleton (the registry-behavior tests construct the class directly).
vi.mock("~/services/dataStores/organizationDataStoresRegistryInstance.server", () => ({
organizationDataStoresRegistry: {
isReady: Promise.resolve(),
isLoaded: true,
get: vi.fn().mockReturnValue(null),
reload: vi.fn().mockResolvedValue(undefined),
loadFromDatabase: vi.fn().mockResolvedValue(undefined),
},
}));
vi.mock("~/v3/runEngine.server", () => ({ engine: noopProxy() }));
vi.mock("~/v3/handleSocketIo.server", () => ({
socketIo: noopProxy(),
roomFromFriendlyRunId: (id: string) => `room:${id}`,
}));