0b750d00dd
Watch is the agent noticing something later: you ask it to tell you when a condition holds, and it answers when it does — or when it can't any more. A watch is a **durable one-shot promise**. The condition is checked on a schedule by deterministic code (no LLM in the checks), the answer lands in the chat once, and then the watch is over. Ten kinds: three on a run, five on a queue, error recurrence, health recovery. ## Stack Stacked on **#4529** (UI), which is stacked on **#4418** (chat, reports, investigate). Merge those first. **#4516** (storybook gallery) sits on top of this branch. ## How to review [**GUIDEBOOK.md**](https://github.com/triggerdotdev/trigger.dev/blob/feat/dashboard-agent-flows-watch/internal-packages/dashboard-agent/GUIDEBOOK.md) on this branch is the behaviour reference — it states the conditions rather than the code, so you can predict what happens without running anything. "The ten watch kinds, and what makes each fire" and "Creating a watch" describe exactly this PR, and the tables there are the spec the code is written against. ## What's inside - **Ten watch kinds**, one deterministic check each (`dashboardAgentWatch*Checks.ts`), with the spec union in `dashboard-agent-contracts/src/watch.ts`. - **Scheduling** — each watch schedules its own next check; due watches of one `(environment, cadence)` group can be checked together in one batch pass, with a sweep as the backstop for expiry, redelivery and retention. - **Delivery** — the in-chat wake and card, an optional email alert (new `DASHBOARD_AGENT_WATCH` alert channel, so it shows on the project's Alerts page with one-click unsubscribe), and an optional investigation when the outcome needs attention. - **Submission ledger** — `watch_submissions`, keyed `(chat_id, client_request_id)`, so a retried card submission replays the recorded outcome instead of creating a second watch. - **Watch token** — a delayed-execution credential accepted only by the watch endpoints, re-checked against the user's live access on every tick. - **Unread work** — the panel polls for wakes that landed while it was closed, so a chat can go unread and light the launcher dot. ## Key decisions **A check result is a 4-way, and only two of them are verdicts.** `satisfied` / `terminal_unsatisfied` are answers; `pending` and `unavailable` are not. Any exception inside any check is caught in one place and becomes `unavailable` with an unverified observation — a check that failed is never evidence. **A completed window is an answer, and whether it is good or bad news is declared per kind, never inferred.** There is a table for that in the guidebook: `run_failed` completing its window is *good* news ("hasn't failed"), `backlog_drain` completing it is not. One rule overrides the table: a window that completed on an unverified observation is neutral and says only that the watch ended without a confirmed answer. **An unreadable source is never a negative answer** — and, because investigations only open on `attention`, it never starts one either. **Identity is `(chat, project, environment)` plus the condition,** enforced by a partial unique index over active rows (`watches_chat_active_identity_key`), not by the read-then-insert check. Cadence, window, note and `ticks` are deliberately not part of it. Two different chats may watch the same thing — a watch is a promise to a chat. **The server resolves the target's name, whatever the model calls it.** The model can't tell a task queue (`task/<id>`) from a custom queue, so both spellings are tried and the stored one wins — and the rewrite happens **before** identity and before the row is written, so the identity, the checks, the link and the wording all see one spelling. **Freshness fences.** Depth falls back from the live counter to the newest 60 s ClickHouse bucket, which only counts as current within 60 s of now. A non-current reading at or below the *quiet line* is refused as `unavailable` rather than believed, so a stale empty bucket is never read as "drained". The stall streak is the one piece of carried state: it lives in the previous check's facts and *freezes* on an unreadable reading rather than breaking. **Chain reliability.** There is no shared cron — each watch (or batch group) schedules its own next tick, so the failure mode to review is the chain dying. A failed batch check is caught, the next tick is scheduled anyway and the run resolves rather than failing, so the chain survives a check that couldn't run; the sweep re-arms groups and finalizes anything still active past its deadline, even when delivery isn't configured. Wake redelivery is id-deduped rather than conditional, because the sweep can't know whether the user was already told. Access is re-authorized on **every** check against the primary — replica lag would extend access the user has already lost. **Wording lives in one place.** `watch-wording.ts` is read by the card, banner, toast, email and the agent's own narration, and the numbers come from the frozen observation rather than a fresh read, so a retry produces the same sentence. Replay reproduces the **recorded** decision instead of deciding again — the transcript is append-once, so a second decision would contradict it forever. **Cancellation is the ending without an answer** — no resolution, no wake. One exception, decided during testing: a watch the *user* cancelled leaves a single neutral transcript line ("Stopped watching …"), keyed off the watch id so a retry can't repeat it. The other four reasons stay silent. **Email is opt-in and only a fired watch emails.** An expiry is narrated in the chat and nowhere else. Both gates (agent access, a configured email transport) are checked at subscribe time *and* again at delivery, and the subscription outcome is frozen on the ledger row so a retry replays it. Neither gate is a plan check. **One watch offer per turn.** The prompt and the renderer guard this independently — if the turn already proposed a watch card, the action button is dropped, because the card is the better affordance. Two eval cases pin the prompt side: exactly one offer with the line last and the button after it, and zero offers when the rendered card already carries one — deterministic assertions, over a real-model run. ## Testing Unit tests (vitest, testcontainers, no mocks) under `apps/webapp/test/dashboardAgentWatch*.test.ts` and `internal-packages/dashboard-agent/src/watch-*.test.ts` cover the invariants above: the 4-way check results and the freshness fences, identity/dedup and the submission ledger, queue-name resolution, the batch chain surviving a failed check, sweep boundaries and alert-once, tenancy and the watch token's scope, and the wording snapshot. The load-bearing ones were verified by control-breaking the guard first and checking the test goes red. Live-tested end to end against a local stack, following the guidebook: all ten watch kinds firing and expiring, cancellation, the email pair (a fired watch mails, an expired one does not), and watch recovery from a health report.
165 lines
7.2 KiB
TypeScript
165 lines
7.2 KiB
TypeScript
// Load apps/webapp/.env into process.env so env.server's top-level
|
||
// EnvironmentSchema.parse(process.env) succeeds in vitest workers.
|
||
import { config } from "dotenv";
|
||
import path from "node:path";
|
||
import { vi } from "vitest";
|
||
import type * as IORedisModule from "ioredis";
|
||
import type * as TaskMetadataCacheModule from "~/services/taskMetadataCache.server";
|
||
|
||
config({ path: path.resolve(__dirname, "../.env") });
|
||
|
||
// CI has no .env and no REDIS_HOST/REDIS_PORT, so import-time guards like
|
||
// autoIncrementCounter.server.ts throw and their suites fail to collect. Default
|
||
// the pair — the ioredis mock below forces lazyConnect, so nothing ever dials.
|
||
process.env.REDIS_HOST ??= "localhost";
|
||
process.env.REDIS_PORT ??= "6379";
|
||
process.env.SESSION_SECRET ??= "test-session-secret";
|
||
process.env.PROVIDER_SECRET ??= "test-provider-secret";
|
||
process.env.COORDINATOR_SECRET ??= "test-coordinator-secret";
|
||
process.env.MANAGED_WORKER_SECRET ??= "test-managed-worker-secret";
|
||
|
||
// Worker singletons construct a RedisWorker at import time whose ioredis client
|
||
// connects eagerly, so any test importing the service graph opens real Redis
|
||
// connections on import — which floods and fails in CI (no Redis). Mock them to
|
||
// no-op stubs. Only the worker modules are mocked, never the run store
|
||
// (~/v3/runStore.server, ~/db.server), which store-routing tests need real.
|
||
function createWorkerStub() {
|
||
return {
|
||
start: vi.fn(),
|
||
stop: vi.fn(),
|
||
enqueue: vi.fn().mockResolvedValue(undefined),
|
||
enqueueOnce: vi.fn().mockResolvedValue(undefined),
|
||
reschedule: vi.fn().mockResolvedValue(undefined),
|
||
cancel: vi.fn().mockResolvedValue(undefined),
|
||
ack: vi.fn().mockResolvedValue(undefined),
|
||
};
|
||
}
|
||
|
||
vi.mock("~/v3/commonWorker.server", () => ({ commonWorker: createWorkerStub() }));
|
||
vi.mock("~/v3/batchTriggerWorker.server", () => ({ batchTriggerWorker: createWorkerStub() }));
|
||
vi.mock("~/v3/alertsWorker.server", () => ({ alertsWorker: createWorkerStub() }));
|
||
|
||
// RunEngine and the socket.io server are further singletons that open eager
|
||
// ioredis connections at import via the same pattern. No test
|
||
// uses these app-level singletons directly (store-routing tests build their own
|
||
// engine and run store), so stub them to no-op proxies.
|
||
// Recursive no-op proxy: property access at any depth returns another callable
|
||
// no-op proxy, so real service tests reaching nested singleton methods (e.g.
|
||
// engine.runQueue.updateEnvConcurrencyLimits) don't break on an intermediate stub.
|
||
type NoopProxyFn = ((...args: unknown[]) => Promise<undefined>) & Record<string, unknown>;
|
||
|
||
const noopProxy = (): NoopProxyFn => {
|
||
const fn = () => Promise.resolve(undefined);
|
||
return new Proxy(fn, {
|
||
get: (_target, prop) => (prop === "then" ? undefined : noopProxy()),
|
||
apply: () => Promise.resolve(undefined),
|
||
}) as unknown as NoopProxyFn;
|
||
};
|
||
|
||
// Beyond the modules mocked above, dozens more app modules construct an
|
||
// ioredis client at import time pointed at env-configured Redis, and ioredis
|
||
// dials on construction — in CI (no Redis service) that floods ECONNREFUSED at
|
||
// shard scale. Force `lazyConnect: true` on every client instead: import-time
|
||
// singletons construct but never dial, while anything that actually issues a
|
||
// command (tests against live testcontainers) connects on first command
|
||
// exactly as before.
|
||
vi.mock("ioredis", async (importOriginal) => {
|
||
const actual = await importOriginal<typeof IORedisModule>();
|
||
|
||
// Normalize ioredis's overloaded ctor args — (), (port), (path),
|
||
// (port, host), (opts), (port, opts), (port, host, opts), (path, opts) —
|
||
// so lazyConnect lands in the options object in every form.
|
||
function withLazyConnect(args: unknown[]): unknown[] {
|
||
if (args.length === 0) {
|
||
return [{ lazyConnect: true }];
|
||
}
|
||
const last = args[args.length - 1];
|
||
if (typeof last === "object" && last !== null) {
|
||
return [...args.slice(0, -1), { ...last, lazyConnect: true }];
|
||
}
|
||
return [...args, { lazyConnect: true }];
|
||
}
|
||
|
||
class LazyRedis extends actual.Redis {
|
||
constructor(...args: unknown[]) {
|
||
// @ts-expect-error – forwarding ioredis's overloaded ctor args
|
||
super(...withLazyConnect(args));
|
||
}
|
||
}
|
||
|
||
class LazyCluster extends actual.Cluster {
|
||
constructor(startupNodes: unknown, options?: Record<string, unknown>) {
|
||
// @ts-expect-error – forwarding ioredis's ctor args
|
||
super(startupNodes, { ...options, lazyConnect: true });
|
||
}
|
||
}
|
||
|
||
// Keep the `Redis.Cluster` static alias (`new Redis.Cluster(...)`) working.
|
||
// The base class exposes `Cluster` as a getter-only static, so define our
|
||
// own property rather than assigning through the inherited getter.
|
||
Object.defineProperty(LazyRedis, "Cluster", { value: LazyCluster });
|
||
|
||
return {
|
||
...actual,
|
||
default: LazyRedis,
|
||
Redis: LazyRedis,
|
||
Cluster: LazyCluster,
|
||
};
|
||
});
|
||
|
||
// alertsRateLimiter.check() is invoked at runtime by deliverAlert; against
|
||
// env-configured Redis each check burns ~20 reconnect cycles before its
|
||
// caught error, stalling alert-path tests into timeouts. Allow everything.
|
||
vi.mock("~/v3/alertsRateLimiter.server", () => ({
|
||
alertsRateLimiter: { check: vi.fn().mockResolvedValue({ allowed: true }) },
|
||
}));
|
||
|
||
// tracePubSub.publish() runs inside eventRepository writes; each publish to
|
||
// env-configured Redis stalls ~20 reconnect cycles (errors are allSettled-
|
||
// swallowed but awaited), timing out any test that records trace events.
|
||
vi.mock("~/v3/services/tracePubSub.server", async () => {
|
||
const { EventEmitter } = await import("node:events");
|
||
return {
|
||
tracePubSub: {
|
||
publish: vi.fn().mockResolvedValue(undefined),
|
||
subscribeToTrace: vi.fn().mockResolvedValue({
|
||
unsubscribe: vi.fn().mockResolvedValue(undefined),
|
||
eventEmitter: new EventEmitter(),
|
||
}),
|
||
},
|
||
TracePubSub: class {},
|
||
};
|
||
});
|
||
|
||
// Same runtime-stall shape for the task metadata cache (queues concern). CI
|
||
// leaves TASK_META_CACHE_REDIS_HOST unset and gets the Noop implementation;
|
||
// pin the Noop cache here so env-configured local runs behave identically.
|
||
vi.mock("~/services/taskMetadataCacheInstance.server", async () => {
|
||
const { NoopTaskMetadataCache } = await vi.importActual<typeof TaskMetadataCacheModule>(
|
||
"~/services/taskMetadataCache.server"
|
||
);
|
||
return { taskMetadataCacheInstance: new NoopTaskMetadataCache() };
|
||
});
|
||
|
||
// The org-data-stores registry singleton is constructed at import (transitively via
|
||
// the ClickHouse factory instance, which many presenters pull in). Its ctor fires a
|
||
// `forever` pRetry(loadFromDatabase) plus a setInterval reload against db.server's
|
||
// $replica; in CI (no Postgres) those retry forever, blocking the worker until any
|
||
// awaiting test's hook times out. Stub the instance to a no-op — no unit test uses
|
||
// this singleton (the registry-behavior tests construct the class directly).
|
||
vi.mock("~/services/dataStores/organizationDataStoresRegistryInstance.server", () => ({
|
||
organizationDataStoresRegistry: {
|
||
isReady: Promise.resolve(),
|
||
isLoaded: true,
|
||
get: vi.fn().mockReturnValue(null),
|
||
reload: vi.fn().mockResolvedValue(undefined),
|
||
loadFromDatabase: vi.fn().mockResolvedValue(undefined),
|
||
},
|
||
}));
|
||
|
||
vi.mock("~/v3/runEngine.server", () => ({ engine: noopProxy() }));
|
||
vi.mock("~/v3/handleSocketIo.server", () => ({
|
||
socketIo: noopProxy(),
|
||
roomFromFriendlyRunId: (id: string) => `room:${id}`,
|
||
}));
|