feat(webapp): dashboard agent — chat, reports, investigate (#4418)

## What & why

This is the system behind the Dashboard Agent — an assistant that
answers questions about a project's runs, errors, queues, deploys and
health, and can investigate failures end to end.

The agent runs as a chat.agent task in its own Trigger project. It has
no access to the main database or ClickHouse; all platform data is read
through the public API using a delegated, read-only user token.

Everything here is behind `canAccessDashboardAgent` and inert with the
flag off. The UI that mounts the panel lands in #4529.

## Stack

`#4418` (this, base) ← `#4529` UI ← `#4525` Watch ← `#4516` storybook
gallery. The scenario/contract reference for the whole stack is
`internal-packages/dashboard-agent/GUIDEBOOK.md` (it lands on the Watch
branch): it states, per feature, what makes each thing happen and where
that is decided.

## What's inside

**Agent runtime and tools** — `internal-packages/dashboard-agent`:
prompt, tool set (API reads, TRQL query, docs, navigation,
evidence/investigations, repo source), conversation compaction, a
prompt-prefix token budget pinned by snapshot test, and sampled
LLM-judged turn evals. The package cannot import webapp server code,
which is what makes the "no DB access" claim structural rather than a
convention.

**Contracts** — `internal-packages/dashboard-agent-contracts`:
`trigger://` URIs, intents, and the block envelope every rendered card
travels in.

**Conversation store** — `internal-packages/dashboard-agent-db`: drizzle
over postgres-js in its own `trigger_dashboard_agent` Postgres schema,
plus one additive migration.

**Auth boundary** — the user-actor token gains an optional environment
claim; one guard (`userActorEnvironment.server.ts`) enforces it so
routes don't each re-derive the rule. Token minting, cap ceiling, and
the RBAC fallback path for self-hosted.

**Transport** — webapp resource routes that mint the token and proxy
each turn, and SDK-side mid-turn reconnect.

**Public API the agent reads through** — orgs, projects, environments,
runs, queue metrics, workers, a run's commit metadata, repo snapshot,
reports, and `POST /api/v1/query`.

**Reports** — the health report's layout is declared once and shared by
the card, the markdown surface and the JSON/MCP surface, so the same
report reads the same in the dashboard, the terminal and an editor.

**Block renderers** — the report and investigation cards the flows above
already emit (`app/components/dashboard-agent/`). The panel that hosts
them, and the rest of the chat UI, is #4529.

**Query safety and CSP** — see below.

## Key decisions

- **The agent is a separate Trigger project, not webapp code.** It reads
platform data over the public API with a delegated user-actor token
whose `cap` ceilings it to read scopes. No Prisma, no ClickHouse, no
webapp imports.
- **The PAT-only auth helper now refuses user-actor tokens.** This is an
intentional behavioral change: its callers consume only a bare userId
and do not enforce delegated-token capabilities. Actor-aware routes
continue through the scoped route builders instead.
- **RBAC fallback builds a delegated token's ability from its own cap**,
never the blanket ability a PAT gets (read-only when the token declares
none). Without this, the agent's read-only cap would buy a write JWT on
self-hosted.
- **Org creation checks RBAC only for user-actor tokens, and only after
the env gate**, so an install with `ORG_CREATION_API_ENABLED` off
returns 404 rather than 403, and an ordinary PAT never consults an
ability the route has no org to scope. Both orderings are pinned by
test.
- **The query path is read-only in depth.** TRQL rejects write
statements at the grammar level (they don't parse, rather than being
filtered), ClickHouse runs with `readonly=1`, and the org/project/env
filters are injected server-side from the credential — the request body
cannot widen scope. An unparseable query denies instead of falling
through to the permissive resource.
- **Document-wide img-src CSP.** Remote images are an
outbound-request/exfiltration surface, so the policy permits only
own-origin/data/blob, the required SSO avatar hosts, and the favicon
endpoint. Operators can add exact origins through CSP_IMG_SRC_ALLOWLIST;
wildcard hosts and bare schemes are intentionally not allowed.
- **The chat transport reconnects on a mid-turn EOF**
(`@trigger.dev/sdk`). A body that ends without a turn-complete is
terminal only when the server says `X-Session-Settled: true`; otherwise
the transport resubscribes from `lastEventId` with bounded backoff, and
any record re-earns the budget. Previously a closed long-poll window or
a proxy restart left the reply stuck as if still generating.
- **Conversations live in their own datastore**, schema-scoped and
foreign-key-free (it references `organizationId`/`userId` by id, because
in cloud it is a different database). It is a display read-model for the
History tab and transport resume; `chat.agent`'s object-store snapshot
remains the model's source of truth.
- **Deterministic first.** Reports and health checks contain no LLM —
they are computed from the same data the dashboard shows, and the model
only narrates and links them. That is what makes a number in an answer
auditable.

## Testing

- 63 new test files, run with `pnpm run test --filter webapp` and
per-package vitest. Heaviest coverage on the auth boundary
(`userActorPatOnlyBoundary`, `userActorTokenClaimsAndScopes`,
`contextlessPatRoutes`, `rbacFallbackBranch`), TRQL read-only, the
report layout, and the SDK reconnect.
- The agent package has a separate eval lane (`pnpm run test:evals`,
`vitest.eval.config.ts`) that hits the real model, so it never runs in
`pnpm test`.
- Live-tested against a local stack scenario by scenario; the GUIDEBOOK
lists the condition each behaviour is expected under, which is what
those runs were checked against.

## Changelog

`.server-changes/dashboard-agent.md`, plus changesets for
`@trigger.dev/core` (report schemas), `@trigger.dev/sdk` (chat
reconnect) and the CLI's `mint-token` help text.
This commit is contained in:
Katia Bulatova
2026-08-11 18:56:14 +02:00
committed by GitHub
parent 02de2e693f
commit 4569657923
228 changed files with 27251 additions and 3175 deletions
@@ -0,0 +1,7 @@
---
"@trigger.dev/core": patch
"@trigger.dev/sdk": patch
"trigger.dev": patch
---
Chat in the browser now reconnects when the connection drops mid-turn, instead of leaving the reply stuck as if it were still generating. Reports can be fetched as structured data with the `json` format, and the shortest report period is now one minute (`30m`, `1h`, `7d`). The `mint-token` command's help is clearer too: a token minted without `--cap` is read-only, and `--ttl` shows the correct maximum lifetime of 7 days.
+1 -1
View File
@@ -154,7 +154,7 @@ PASS: one run, `run_<RID>` (status maps to `FAILED`). Proves `filter[error]` ->
### 6. Attribution — `mint-token` -> JWT exchange records the acting user
```bash
TOKEN=$(cli mint-token --profile $PROFILE --client errors-api-e2e 2>/dev/null) # UAT
TOKEN=$(cli mint-token --profile $PROFILE --client errors-api-e2e --cap read:errors,write:errors 2>/dev/null) # UAT
ENVJWT=$(curl -sS -X POST "$B/api/v1/projects/$REF/dev/jwt" -H "Authorization: Bearer $TOKEN" \
-H 'Content-Type: application/json' -d '{"claims":{"scopes":["read:errors","write:errors"]}}' \
| python3 -c "import sys,json;print(json.load(sys.stdin)['token'])")
+5
View File
@@ -0,0 +1,5 @@
# Generated, not hand-written: collapsed in diffs and excluded from language stats.
internal-packages/dashboard-agent-db/drizzle/meta/*.json linguist-generated=true
internal-packages/dashboard-agent-db/drizzle/meta/** linguist-generated=true
**/__snapshots__/*.snap linguist-generated=true
pnpm-lock.yaml linguist-generated=true
+2
View File
@@ -85,3 +85,5 @@ ailogger-output.log
# observability-map CLI output artifact, not committed
observability-map.json
.claude/worktrees/
+8
View File
@@ -0,0 +1,8 @@
---
area: webapp
type: feature
---
Meet the dashboard agent: a chat in every environment that answers questions about your runs, queues, errors and health with real data and links, replacing Ask AI everywhere it used to appear. Investigate a failed run, an error, a backed-up queue or a run that hasn't started to get a worked-through answer — what happened, why, and how to fix it, with every claim linked to the runs, errors and deploys behind it. It reads your data read-only, works on preview and dev branches with that branch's own data, and reads the same everywhere — dashboard, terminal, editor. A very long chat keeps working: the agent summarises the earlier part and carries on.
A sample of conversations is scored automatically so the agent keeps getting better; only the score and a one-line summary are kept, never your messages, data or code, and we can switch it off for your organization on request. The Docs button is gone from page headers — ask the agent instead, or open Documentation from Help & Feedback. Separately, a queue's wait times, peak depth, throughput and throttling can now be read from the API.
+4 -1
View File
@@ -7,6 +7,9 @@ node_modules
/cypress/screenshots
/cypress/videos
# Output of `pnpm run agent-ui:screenshots`
/screenshots
/app/styles/tailwind.css
# Ensure the .env symlink is not removed by accident
@@ -20,4 +23,4 @@ storybook-static
/prisma/seed.js
/prisma/populate.js
.memory-snapshots
.memory-snapshots
+9
View File
@@ -1,3 +1,9 @@
/**
* @deprecated Superseded by the dashboard agent (`components/dashboard-agent`). Nothing mounts
* this any more — every Ask AI entry point now opens Ask Trigger. Kept until the agent has
* shipped, then removed along with `@kapaai/react-sdk` and `KAPA_AI_WEBSITE_ID`.
*/
import {
ArrowPathIcon,
ArrowUpIcon,
@@ -81,6 +87,8 @@ function useAskAIState() {
* it around the popover, not inside, so the dialog and shortcut survive the popover closing.
* `children` receives the open function, or undefined when Ask AI is unavailable (self-hosted, no
* Kapa website id, or SSR).
*
* @deprecated See the note at the top of this file.
*/
export function AskAIRoot({
children,
@@ -137,6 +145,7 @@ function AskAIRootProvider({
);
}
/** @deprecated See the note at the top of this file. */
export function AskAI({ isCollapsed = false }: { isCollapsed?: boolean }) {
const { isManagedCloud } = useFeatures();
const websiteId = useKapaWebsiteId();
@@ -0,0 +1,80 @@
import { createElement } from "react";
import { renderToStaticMarkup } from "react-dom/server";
import { describe, expect, it } from "vitest";
import { restrictModelUrls, StreamdownRenderer } from "./StreamdownRenderer";
// streamdown calls urlTransform(url, key, node) to compute each url attribute; a
// returned undefined removes the attribute, so no request is ever issued.
const img = { tagName: "img" } as any;
const link = { tagName: "a" } as any;
describe("restrictModelUrls (image src)", () => {
it("drops a remote model-authored image (the favicon beacon)", () => {
expect(
restrictModelUrls("https://www.google.com/s2/favicons?domain=evil", "src", img)
).toBeUndefined();
});
it("drops any absolute or protocol-relative remote image", () => {
expect(restrictModelUrls("http://evil.tld/pixel.gif", "src", img)).toBeUndefined();
expect(restrictModelUrls("//evil.tld/pixel.gif", "src", img)).toBeUndefined();
});
it("keeps inline and same-origin images", () => {
expect(restrictModelUrls("data:image/png;base64,AAAA", "src", img)).toBe(
"data:image/png;base64,AAAA"
);
expect(restrictModelUrls("blob:abc", "src", img)).toBe("blob:abc");
expect(restrictModelUrls("/local/pic.png", "src", img)).toBe("/local/pic.png");
});
});
describe("restrictModelUrls (link href)", () => {
it("keeps http(s), mailto and relative links", () => {
expect(restrictModelUrls("https://trigger.dev/docs", "href", link)).toBe(
"https://trigger.dev/docs"
);
expect(restrictModelUrls("http://example.com", "href", link)).toBe("http://example.com");
expect(restrictModelUrls("mailto:hi@trigger.dev", "href", link)).toBe("mailto:hi@trigger.dev");
expect(restrictModelUrls("/runs/123", "href", link)).toBe("/runs/123");
});
it("drops unsafe link schemes", () => {
expect(restrictModelUrls("javascript:alert(1)", "href", link)).toBeUndefined();
expect(restrictModelUrls("data:text/html,<script>", "href", link)).toBeUndefined();
});
});
// Force the lazy component to load, then return its resolved default so we can render it
// synchronously. This proves the policy is actually wired into the JSX, not just exported.
async function resolveStreamdownRenderer() {
const lazy = StreamdownRenderer as unknown as {
_payload: unknown;
_init: (payload: unknown) => (props: { children: string }) => JSX.Element;
};
try {
lazy._init(lazy._payload);
} catch (thenable) {
await thenable;
}
return lazy._init(lazy._payload);
}
describe("StreamdownRenderer (rendered markdown)", () => {
it("never lets a model-authored remote image src reach the DOM", async () => {
const Renderer = await resolveStreamdownRenderer();
const markdown = [
"![x](https://www.google.com/s2/favicons?domain=SECRET.evil.tld)",
"![y](//evil.tld/pixel.gif)",
"![z](/local/pic.png)",
].join("\n\n");
const html = renderToStaticMarkup(createElement(Renderer, null, markdown));
// No remote host is ever fetched: no absolute or protocol-relative image src survives.
expect(html).not.toContain('src="http');
expect(html).not.toContain('src="//');
expect(html).not.toContain("SECRET.evil.tld");
// A same-origin relative image is untouched, so the policy does not over-block.
expect(html).toContain('src="/local/pic.png"');
});
});
@@ -1,5 +1,32 @@
import { lazy } from "react";
import type { CodeHighlighterPlugin } from "streamdown";
import type { CodeHighlighterPlugin, UrlTransform } from "streamdown";
const SAFE_LINK_SCHEMES = new Set(["http:", "https:", "mailto:"]);
/**
* URL policy for model-authored markdown. A remote image is fetched the moment it
* renders — no click — so it is a zero-click data beacon; we drop the src of any
* non-local image. Links stay clickable but only for safe, human-followable schemes.
* streamdown removes an attribute whose transform returns undefined, so no request fires.
*/
export const restrictModelUrls: UrlTransform = (url, key, node) => {
const value = url.trim();
const isImage = node.tagName === "img" || key === "src" || key === "srcset";
if (isImage) {
// Inline images carry their own bytes; a relative path resolves to our own origin.
if (/^data:/i.test(value) || /^blob:/i.test(value)) return url;
// Absolute or protocol-relative means a remote host — strip it so nothing is fetched.
if (/^[a-z][a-z0-9+.-]*:/i.test(value) || value.startsWith("//")) return undefined;
return url;
}
// Links: relative and protocol-relative are fine; otherwise require a safe scheme.
if (value.startsWith("//")) return url;
const schemeMatch = /^([a-z][a-z0-9+.-]*):/i.exec(value);
if (!schemeMatch) return url;
return SAFE_LINK_SCHEMES.has(`${schemeMatch[1].toLowerCase()}:`) ? url : undefined;
};
export const StreamdownRenderer = lazy(() =>
Promise.all([import("streamdown"), import("@streamdown/code"), import("./shikiTheme")]).then(
@@ -23,6 +50,7 @@ export const StreamdownRenderer = lazy(() =>
isAnimating={isAnimating}
plugins={{ code: codePlugin }}
controls={{ code: { copy: false, download: false } }}
urlTransform={restrictModelUrls}
linkSafety={{ enabled: false }}
>
{children}
@@ -68,9 +68,10 @@ describe("tsqlLinter", () => {
expect(error).toContain("line");
});
it("should handle missing FROM clause", () => {
const error = getTSQLError("SELECT * WHERE id = 1");
expect(error).not.toBeNull();
it("should accept a query without a FROM clause", () => {
// FROM is optional in the grammar (ClickHouse allows e.g. `SELECT 1`),
// so a FROM-less SELECT is syntactically valid.
expect(getTSQLError("SELECT * WHERE id = 1")).toBeNull();
});
});
});
@@ -0,0 +1,63 @@
import { describe, expect, it } from "vitest";
import {
checkMessageParts,
declaredBodyBytes,
exceedsMessageBodyBytes,
MAX_MESSAGE_BODY_BYTES,
MAX_MESSAGE_CHARS,
MAX_MESSAGE_PARTS,
} from "./message-limits";
describe("message limits", () => {
it("lets a long real question through", () => {
const text = "why did this fail?\n".repeat(50);
expect(exceedsMessageBodyBytes(Buffer.byteLength(text, "utf8"))).toBe(false);
expect(checkMessageParts([{ type: "text", text }])).toBeNull();
});
it("refuses a pasted dump by bytes", () => {
expect(exceedsMessageBodyBytes(MAX_MESSAGE_BODY_BYTES)).toBe(false);
expect(exceedsMessageBodyBytes(MAX_MESSAGE_BODY_BYTES + 1)).toBe(true);
});
it("counts multi-byte characters as bytes, not characters", () => {
// Under the char cap, over the byte cap: 4 bytes each.
const emoji = "🙂".repeat(MAX_MESSAGE_BODY_BYTES / 4 + 1);
expect(emoji.length).toBeLessThan(MAX_MESSAGE_BODY_BYTES);
expect(exceedsMessageBodyBytes(Buffer.byteLength(emoji, "utf8"))).toBe(true);
});
it("refuses a dump split across parts", () => {
const parts = Array.from({ length: 4 }, () => ({
type: "text",
text: "x".repeat(MAX_MESSAGE_CHARS / 2),
}));
expect(checkMessageParts(parts)).toBe("too_long");
});
it("refuses too many parts", () => {
const parts = Array.from({ length: MAX_MESSAGE_PARTS + 1 }, () => ({
type: "text",
text: "x",
}));
expect(checkMessageParts(parts)).toBe("too_many_parts");
expect(checkMessageParts(parts.slice(0, MAX_MESSAGE_PARTS))).toBeNull();
});
it("leaves a shape that isn't a parts array to the schema", () => {
expect(checkMessageParts(undefined)).toBeNull();
expect(checkMessageParts("nope")).toBeNull();
});
it("reads the declared size, or nothing when it isn't declared", () => {
expect(declaredBodyBytes(new Headers({ "content-length": "1234" }))).toBe(1234);
expect(declaredBodyBytes(new Headers())).toBeNull();
expect(declaredBodyBytes(new Headers({ "content-length": "nope" }))).toBeNull();
// An undeclared size can't be refused here; the body's own length is.
expect(exceedsMessageBodyBytes(null)).toBe(false);
});
});
@@ -0,0 +1,51 @@
/**
* Caps on one message to the agent, shared by the composer and the two server paths a message
* can arrive through. Generous for a real question with a pasted stack trace, stingy for a dump:
* an unbounded paste is a large model bill and a permanently fat transcript.
*/
/** ~2 pages of text, or a long stack trace. */
export const MAX_MESSAGE_CHARS = 8_000;
/** The counter only shows near the limit, so a normal message never sees it. */
export const MESSAGE_CHARS_WARN_AT = Math.floor(MAX_MESSAGE_CHARS * 0.9);
/** A composed message is a handful of parts; dozens means something is wrong. */
export const MAX_MESSAGE_PARTS = 20;
/**
* The whole request body, in bytes: headroom for {@link MAX_MESSAGE_CHARS} of any script plus
* the per-turn metadata, and nothing like a pasted file.
*/
export const MAX_MESSAGE_BODY_BYTES = 64 * 1024;
export const MESSAGE_TOO_LARGE_CODE = "message_too_large";
export const MESSAGE_TOO_LARGE_ERROR = "That message is too long. Shorten it and send again.";
export type MessagePartsProblem = "too_many_parts" | "too_long";
/** Counts the parts and their text. Anything that isn't a parts array is left to the schema. */
export function checkMessageParts(parts: unknown): MessagePartsProblem | null {
if (!Array.isArray(parts)) return null;
if (parts.length > MAX_MESSAGE_PARTS) return "too_many_parts";
let chars = 0;
for (const part of parts) {
const text = (part as { text?: unknown } | null)?.text;
if (typeof text === "string") chars += text.length;
}
return chars > MAX_MESSAGE_CHARS ? "too_long" : null;
}
/** The declared body size, or null when the client didn't declare one. */
export function declaredBodyBytes(headers: Headers): number | null {
const raw = headers.get("content-length");
if (!raw) return null;
const bytes = Number.parseInt(raw, 10);
return Number.isFinite(bytes) ? bytes : null;
}
export function exceedsMessageBodyBytes(bytes: number | null | undefined): boolean {
return typeof bytes === "number" && bytes > MAX_MESSAGE_BODY_BYTES;
}
@@ -0,0 +1,32 @@
import { describe, expect, it } from "vitest";
import { MAX_URIS_PER_RESOLVE_REQUEST, planUriBatches } from "./resolve-uris";
const uri = (index: number) => `trigger://runs/run_${index}`;
describe("planUriBatches", () => {
it("resolves a card's twenty citations in one request", () => {
const batches = planUriBatches(Array.from({ length: 20 }, (_, index) => uri(index)));
expect(batches).toHaveLength(1);
expect(batches[0]).toHaveLength(20);
});
it("asks about each URI once", () => {
const batches = planUriBatches([uri(1), uri(1), uri(2)]);
expect(batches).toEqual([[uri(1), uri(2)]]);
});
it("caps a request and carries the rest over", () => {
const count = MAX_URIS_PER_RESOLVE_REQUEST + 3;
const batches = planUriBatches(Array.from({ length: count }, (_, index) => uri(index)));
expect(batches).toHaveLength(2);
expect(batches[0]).toHaveLength(MAX_URIS_PER_RESOLVE_REQUEST);
expect(batches[1]).toHaveLength(3);
});
it("has nothing to send for nothing", () => {
expect(planUriBatches([])).toEqual([]);
});
});
@@ -0,0 +1,25 @@
/**
* Batching for `trigger://` resolution. An investigation card cites ten to twenty targets, and
* each request re-authorises and re-resolves the environment — so they go in one request.
*/
/** One environment lookup and one repo lookup serve a whole batch. */
export const MAX_URIS_PER_RESOLVE_REQUEST = 25;
/** A transient failure is worth retrying; a third one isn't. */
export const MAX_RESOLVE_ATTEMPTS = 3;
export const RESOLVE_RETRY_DELAY_MS = 1_000;
/** Deduplicates, then splits into requests no bigger than the cap. */
export function planUriBatches(
uris: readonly string[],
cap: number = MAX_URIS_PER_RESOLVE_REQUEST
): string[][] {
const unique = [...new Set(uris)];
const batches: string[][] = [];
for (let index = 0; index < unique.length; index += cap) {
batches.push(unique.slice(index, index + cap));
}
return batches;
}
@@ -42,6 +42,8 @@ export type MiniLineChartProps = {
* throttled magnitude carried by the tooltip.
*/
throttled?: number[];
/** Tooltip wording for the overlay buckets. Null omits the overlay line. */
overlayLabel?: string | null;
/** Epoch ms of the first bucket's start. When omitted, the last bucket is anchored to now. */
bucketStartMs?: number;
/** Width of each bucket in ms. Defaults to one hour. */
@@ -76,6 +78,7 @@ export type MiniLineChartProps = {
export function MiniLineChart({
data,
throttled,
overlayLabel = "throttled",
bucketStartMs,
bucketIntervalMs,
color = "var(--color-tasks)",
@@ -128,7 +131,7 @@ export function MiniLineChart({
<YAxis domain={[0, max || 1]} hide />
<Tooltip
cursor={{ stroke: "rgba(255, 255, 255, 0.2)", strokeWidth: 1 }}
content={<MiniLineChartTooltip unitLabel={unitLabel} />}
content={<MiniLineChartTooltip unitLabel={unitLabel} overlayLabel={overlayLabel} />}
allowEscapeViewBox={{ x: true, y: true }}
wrapperStyle={{ zIndex: 1000 }}
animationDuration={0}
@@ -195,7 +198,8 @@ function MiniLineChartTooltip({
active,
payload,
unitLabel,
}: TooltipProps<number, string> & { unitLabel: UnitLabel }) {
overlayLabel = "throttled",
}: TooltipProps<number, string> & { unitLabel: UnitLabel; overlayLabel?: string | null }) {
if (!active || !payload || payload.length === 0) return null;
const entry = payload[0].payload as MiniLineChartDatum;
const date = entry.date instanceof Date ? entry.date : new Date(entry.date);
@@ -211,9 +215,9 @@ function MiniLineChartTooltip({
{entry.count === 1 ? unitLabel.singular : unitLabel.plural}
</span>
</div>
{throttled > 0 && (
{throttled > 0 && overlayLabel !== null && (
<div className="mt-1 text-xs text-warning">
<span className="tabular-nums">{throttled.toLocaleString()}</span> throttled
<span className="tabular-nums">{throttled.toLocaleString()}</span> {overlayLabel}
</div>
)}
</div>
@@ -242,8 +242,16 @@ export function SideMenuItem({
/** Button styled to match {@link SideMenuItem}, for entries that open a dialog rather than navigate. */
export const SideMenuItemButton = forwardRef<
HTMLButtonElement,
{ icon: RenderIcon; name: string; trailing?: ReactNode } & ButtonHTMLAttributes<HTMLButtonElement>
>(function SideMenuItemButton({ icon, name, trailing, className, type, ...props }, ref) {
{
icon: RenderIcon;
name: string;
trailing?: ReactNode;
iconClassName?: string;
} & ButtonHTMLAttributes<HTMLButtonElement>
>(function SideMenuItemButton(
{ icon, name, trailing, className, iconClassName, type, ...props },
ref
) {
return (
<button
ref={ref}
@@ -256,7 +264,10 @@ export const SideMenuItemButton = forwardRef<
>
<Icon
icon={icon}
className="size-5 shrink-0 text-text-dimmed group-hover/menuitem:text-text-bright"
className={cn(
"size-5 shrink-0",
iconClassName ?? "text-text-dimmed group-hover/menuitem:text-text-bright"
)}
/>
<SideMenuLabel className="min-w-0 flex-1 select-none text-left text-[0.90625rem] font-medium tracking-[-0.01em]">
{name}
@@ -1,4 +1,5 @@
import { type CSSProperties, useEffect, useRef } from "react";
import { useThemeMode } from "~/hooks/useThemeMode";
// Our own 5x5 dot-matrix system, reverse-engineered from dotmatrix
// (github.com/zzzzshawn/matrix) but written from scratch on canvas.
@@ -117,6 +118,8 @@ export type DotMatrixPalette = {
export const DOT_MATRIX_PALETTES = {
mono: { stops: ["#e2e8f0", "#ffffff", "#94a3b8"], glow: "#ffffff" },
/** `mono` mirrored for light surfaces, where the white ramp would vanish. */
monoLight: { stops: ["#2b2c2f", "#1a1b1f", "#585c64"], glow: "#1a1b1f" },
trigger: { stops: ["#41ff54", "#a4ff53", "#e7ff52"], glow: "#86ff53" },
aurora: { stops: ["#ff3cac", "#784ba0", "#2b86c5"], glow: "#9c64bf" },
ocean: { stops: ["#00c6ff", "#0072ff", "#4facfe"], glow: "#2f8fff" },
@@ -631,3 +634,17 @@ export function AgentDotMatrix({
/>
);
}
/** The agent's monochrome logo. Use this rather than `palette="mono"`, which is invisible on light. */
export function AgentMonoLogo(props: Omit<AgentDotMatrixProps, "palette" | "restColor" | "mode">) {
const mode = useThemeMode();
const light = mode === "light";
return (
<AgentDotMatrix
{...props}
mode={mode}
palette={light ? "monoLight" : "mono"}
restColor={light ? "#1a1b1f" : "#d7d9dd"}
/>
);
}
@@ -9,7 +9,7 @@ import React, {
} from "react";
import { type ShortcutDefinition, useShortcutKeys } from "~/hooks/useShortcutKeys";
import { cn } from "~/utils/cn";
import { AgentDotMatrix } from "./AgentDotMatrix";
import { AgentMonoLogo } from "./AgentDotMatrix";
import { ShortcutKey } from "./ShortcutKey";
import { Tooltip, TooltipContent, TooltipProvider, TooltipTrigger } from "./Tooltip";
import { Icon, type RenderIcon } from "./Icon";
@@ -106,23 +106,22 @@ const theme = {
},
docs: {
textColor:
// System themes: monochrome label, the book icon keeps the blue
"text-callout-docs-text/70 system:text-text-bright transition group-disabled/button:text-text-dimmed/80",
"text-callout-docs-text/70 dark:text-text-bright transition group-disabled/button:text-text-dimmed/80",
button:
"bg-secondary border border-border-bright/50 shadow-xs group-hover/button:bg-background-raised group-disabled/button:bg-tertiary group-disabled/button:opacity-60 group-disabled/button:pointer-events-none",
shortcut:
"border-text-dimmed/40 text-text-dimmed group-hover/button:text-text-bright group-hover/button:border-text-dimmed",
icon: "text-blue-500",
},
// Reserved for the AI agent's "Ask AI" affordance: secondary styling with a
// softened trigger-green border.
"ask-ai": {
textColor: "text-text-bright transition group-disabled/button:text-text-dimmed/80",
// The AI agent's "Ask Trigger" affordance.
"ask-trigger": {
textColor:
"text-text-bright transition light:group-hover/button:text-charcoal-800 group-disabled/button:text-text-dimmed/80",
button:
"cursor-pointer bg-secondary border border-[#41FF54]/25 group-hover/button:bg-surface-control group-hover/button:border-[#41FF54]/40 group-disabled/button:bg-secondary group-disabled/button:opacity-60 group-disabled/button:cursor-default group-disabled/button:pointer-events-none",
"cursor-pointer bg-secondary border border-[#41FF54]/25 dark:group-hover/button:bg-background-raised dark:group-hover/button:border-[#41FF54]/40 light:group-hover/button:bg-[#e4ffe8] light:group-hover/button:border-[#41FF54]/60 light:border-success/60 group-disabled/button:bg-secondary group-disabled/button:opacity-60 group-disabled/button:cursor-default group-disabled/button:pointer-events-none",
shortcut:
"border-text-dimmed/40 text-text-dimmed group-hover/button:text-text-bright group-hover/button:border-text-dimmed",
icon: "text-text-bright",
"border-text-dimmed/40 text-text-dimmed dark:group-hover/button:text-text-bright dark:group-hover/button:border-text-dimmed light:group-hover/button:text-charcoal-800 light:group-hover/button:border-charcoal-800/60",
icon: "text-text-bright light:group-hover/button:text-charcoal-800",
},
};
@@ -141,19 +140,14 @@ function createVariant(sizeName: Size, themeName: Theme) {
};
}
// The ask-ai button always leads with the square agent logo, so it supplies its
// own leading icon and its padding is tuned around it: small = 16px logo, 4px
// left / 6px right; medium 32/16 -> 8px; large 40/20 -> 10px. Pass an explicit
// `LeadingIcon` (e.g. an <AgentDotMatrix active />) to animate it.
function createAskAiVariant(sizeName: Size, opticalPadding: string, logoSize: number) {
const base = createVariant(sizeName, "ask-ai");
// ask-trigger supplies its own leading logo. Pass an explicit `LeadingIcon` to animate it.
function createAskTriggerVariant(sizeName: Size, opticalPadding: string, logoSize: number) {
const base = createVariant(sizeName, "ask-trigger");
return {
...base,
button: cn(base.button, opticalPadding),
iconSpacing: "gap-x-1.5",
defaultLeadingIcon: (
<AgentDotMatrix size={logoSize} palette="mono" restColor="#ffffff" decorative />
),
defaultLeadingIcon: <AgentMonoLogo size={logoSize} decorative />,
};
}
@@ -187,9 +181,9 @@ const variant = {
"docs/medium": createVariant("medium", "docs"),
"docs/large": createVariant("large", "docs"),
"docs/extra-large": createVariant("extra-large", "docs"),
"ask-ai/small": createAskAiVariant("small", "px-1 pr-1.5", 16),
"ask-ai/medium": createAskAiVariant("medium", "px-2", 16),
"ask-ai/large": createAskAiVariant("large", "px-2.5", 20),
"ask-trigger/small": createAskTriggerVariant("small", "px-1 pr-1.5", 16),
"ask-trigger/medium": createAskTriggerVariant("medium", "px-2", 16),
"ask-trigger/large": createAskTriggerVariant("large", "px-2.5", 20),
"menu-item": {
textColor: "text-text-bright px-1",
button:
@@ -275,7 +269,7 @@ export function ButtonContent(props: ButtonContentPropsType) {
}, [isLoading]);
const variation = allVariants.variant[props.variant];
// Some variants (ask-ai) always lead with their own glyph unless overridden.
// Some variants (ask-trigger) always lead with their own glyph unless overridden.
const leadingIcon = LeadingIcon ?? variation.defaultLeadingIcon;
const btnClassName = cn(allVariants.$all, variation.button);
@@ -212,10 +212,11 @@ const popoverArrowTriggerVariants = {
icon: "text-text-dimmed group-hover:text-text-bright",
},
primary: {
// White ink, not text-bright, which flips dark on the light theme.
trigger:
"bg-indigo-600 border border-indigo-500 text-text-bright hover:bg-indigo-500 hover:border-indigo-400 disabled:opacity-50 disabled:pointer-events-none",
text: "text-text-bright hover:text-white",
icon: "text-text-bright",
"bg-indigo-600 border border-indigo-500 text-white hover:bg-indigo-500 hover:border-indigo-400 disabled:opacity-50 disabled:pointer-events-none",
text: "text-white",
icon: "text-white",
},
secondary: {
trigger:
@@ -1,3 +1,4 @@
import { AgentMonoLogo } from "~/components/primitives/AgentDotMatrix";
import { cn } from "~/utils/cn";
type CustomColor = {
@@ -76,3 +77,16 @@ export function ButtonSpinner() {
export function SpinnerWhite({ className }: { className?: string }) {
return <Spinner className={className} color="white" />;
}
/** The dashboard agent's spinner. `size` is the logo's pixel size; the matrix does not scale from CSS. */
export function AgentSpinner({ size = 16 }: { size?: number }) {
return (
<AgentMonoLogo
size={size}
active
// Resting on the playlist's first shape avoids a logo-head flash on mount.
restShape="square"
decorative
/>
);
}
@@ -11,6 +11,10 @@ const variations = {
"text-indigo-500 transition hover:text-indigo-400 inline-flex gap-0.5 items-center group focus-visible:focus-custom",
secondary:
"text-text-dimmed transition hover:text-text-bright inline-flex gap-0.5 items-center group focus-visible:focus-custom",
// The theme-remapped link token, for links inside themed surfaces where the
// raw indigo of `primary` is dark-theme only.
token:
"text-text-link transition hover:underline inline-flex gap-0.5 items-center group focus-visible:focus-custom",
} as const;
type TextLinkProps = {
@@ -1,5 +1,7 @@
import { EnvelopeIcon, ExclamationCircleIcon, XMarkIcon } from "@heroicons/react/20/solid";
import { CheckCircleIcon } from "@heroicons/react/24/solid";
import { useThemeMode } from "~/hooks/useThemeMode";
import { AgentMonoLogo } from "./AgentDotMatrix";
import { useSearchParams } from "@remix-run/react";
import { useEffect, useMemo } from "react";
import { useTypedLoaderData } from "remix-typedjson";
@@ -16,6 +18,7 @@ const permanentToastDuration = 60 * 60 * 24 * 1000;
export function Toast() {
const { toastMessage } = useTypedLoaderData<typeof loader>();
const mode = useThemeMode();
useEffect(() => {
if (!toastMessage) {
return;
@@ -40,7 +43,9 @@ export function Toast() {
);
}, [toastMessage]);
return <Toaster />;
// Sonner stamps its own `data-theme` (default "light") on the toast list and the
// app's theme selectors follow it, so an unthemed Toaster forces every toast light.
return <Toaster theme={mode} />;
}
export function useToast() {
@@ -76,20 +81,24 @@ export function ToastUI({
toastWidth = 356, // Default width, matches what sonner provides by default
title,
action,
actionNode,
}: {
variant: "error" | "success";
variant: "error" | "success" | "agent";
message: string;
t: string;
toastWidth?: string | number;
title?: string;
action?: ToastMessageAction;
/** Caller-rendered action for client-side toasts. `action` stays the serializable server shape. */
actionNode?: React.ReactNode;
}) {
return (
<div
className={cn(
"self-end rounded-md border border-grid-bright bg-background-dimmed",
variant === "success" && "border-success",
variant === "error" && "border-error"
variant === "error" && "border-error",
variant === "agent" && "border-[#41FF54]/25 light:border-success/60 dark:bg-secondary"
)}
style={{
width: toastWidth,
@@ -100,6 +109,10 @@ export function ToastUI({
>
{variant === "success" ? (
<CheckCircleIcon className={cn("size-4 min-w-4 text-success", title && "mt-1")} />
) : variant === "agent" ? (
<span className={cn("flex size-4 min-w-4 items-center", title && "mt-1")}>
<AgentMonoLogo size={16} decorative />
</span>
) : (
<ExclamationCircleIcon className={cn("size-4 min-w-4 text-error", title && "mt-1")} />
)}
@@ -112,6 +125,7 @@ export function ToastUI({
{message}
</Paragraph>
<Action action={action} toastId={t} className="my-2" />
{actionNode}
</div>
<button
className={cn(
@@ -0,0 +1,2 @@
/** Head-of-line wait at which a queue reads as stuck. */
export const OLDEST_WAIT_WARNING_MS = 5 * 60_000;
@@ -163,6 +163,8 @@ export function renderPart(part: UIMessage["parts"][number], i: number) {
resultOutput = lastText?.text ?? undefined;
} else if (p.output != null) {
resultOutput = typeof p.output === "string" ? p.output : JSON.stringify(p.output, null, 2);
} else if (p.state === "output-error" && p.errorText) {
resultOutput = p.errorText;
}
// Status label for the tool row. AI SDK 7 HITL adds the
@@ -178,7 +180,9 @@ export function renderPart(part: UIMessage["parts"][number], i: number) {
? "approved"
: `denied${p.approval?.reason ? `: ${p.approval.reason}` : ""}`;
} else if (p.state === "output-error") {
resultSummary = `error: ${p.errorText ?? "unknown"}`;
const errorText = p.errorText ?? "unknown";
resultSummary =
errorText.length > 160 ? `error: ${errorText.slice(0, 160)}` : `error: ${errorText}`;
}
return (
+31
View File
@@ -19,6 +19,7 @@ import { assertRunOpsSplitSentinel, Prisma } from "./db.server";
import { env } from "./env.server";
import { eventLoopMonitor } from "./eventLoopMonitor.server";
import { logger } from "./services/logger.server";
import { buildImgSrcDirective, parseCspImageOrigins, withImgSrc } from "./utils/cspImageOrigins";
import { singleton } from "./utils/singleton";
import { remoteBuildsEnabled } from "./v3/remoteImageBuilder.server";
import {
@@ -47,6 +48,30 @@ import { workerRegionRegistry } from "./v3/workerRegions.server";
const ABORT_DELAY = 30000;
/**
* Where a document may load images from. The markdown renderer that strips images
* ships in the stacked UI PR, so on this branch the policy is the only thing stopping
* a model- or customer-authored image from reaching a remote host.
*
* The hosts we store avatar URLs for, plus whatever `CSP_IMG_SRC_ALLOWLIST` adds
* (e.g. a self-hosted SSO avatar host).
*/
const IMG_SRC_DIRECTIVE = buildImgSrcDirective(
singleton("CspImageOrigins", () => {
const { origins, rejected } = parseCspImageOrigins(env.CSP_IMG_SRC_ALLOWLIST, {
allowHttp: env.NODE_ENV === "development",
});
for (const entry of rejected) {
logger.warn(
`⚠️ CSP_IMG_SRC_ALLOWLIST entry "${entry.value}" was ignored: it ${entry.reason}.`
);
}
return origins;
})
);
export default function handleRequest(
request: Request,
responseStatusCode: number,
@@ -66,6 +91,11 @@ export default function handleRequest(
responseHeaders.set("Content-Security-Policy", "frame-ancestors 'self'");
}
responseHeaders.set(
"Content-Security-Policy",
withImgSrc(responseHeaders.get("Content-Security-Policy"), IMG_SRC_DIRECTIVE)
);
const acceptLanguage = request.headers.get("accept-language");
const locales = parseAcceptLanguage(acceptLanguage, {
validate: Intl.DateTimeFormat.supportedLocalesOf,
@@ -302,6 +332,7 @@ singleton("SentryTenantContextProcessor", () => {
});
export { apiRateLimiter } from "./services/apiRateLimit.server";
export { dashboardAgentBodyCap } from "./services/dashboardAgentBodyCap.server";
export { deploymentRateLimiter } from "./services/deploymentRateLimit.server";
export { engineRateLimiter } from "./services/engineRateLimit.server";
export { otlpRateLimiter } from "./services/otlpRateLimit.server";
+4 -1
View File
@@ -269,6 +269,9 @@ const EnvironmentSchema = z
LOGIN_ORIGIN: z.string().default("http://localhost:3030"),
LOGIN_RATE_LIMITS_ENABLED: BoolEnv.default(true),
APP_ORIGIN: z.string().default("http://localhost:3030"),
// Extra exact origins (comma separated) added to the document `img-src` CSP,
// e.g. an SSO host serving profile images. Wildcards are refused.
CSP_IMG_SRC_ALLOWLIST: z.string().optional(),
API_ORIGIN: z.string().optional(),
// Alternative API origin for deployed runs whose org has the
// internalApiOriginEnabled feature flag on. Unset = flag is a no-op.
@@ -1726,7 +1729,7 @@ const EnvironmentSchema = z
SLACK_BOT_TOKEN: z.string().optional(),
SLACK_SIGNUP_REASON_CHANNEL_ID: z.string().optional(),
// kapa.ai
// kapa.ai — deprecated, see `AskAI.tsx`. Nothing reads it while no surface mounts the widget.
KAPA_AI_WEBSITE_ID: z.string().optional(),
// BetterStack
+24
View File
@@ -0,0 +1,24 @@
import { useEffect, useState } from "react";
export type ThemeMode = "dark" | "light";
/**
* The active theme's mode, for colors that can't come from a CSS variable. Resolved in an
* effect so server and hydration renders agree; `root.tsx` can flip `data-theme` pre-paint.
*/
export function useThemeMode(): ThemeMode {
const [mode, setMode] = useState<ThemeMode>("dark");
useEffect(() => {
const resolve = () => {
setMode(document.documentElement.getAttribute("data-theme") === "light" ? "light" : "dark");
};
resolve();
const observer = new MutationObserver(resolve);
observer.observe(document.documentElement, {
attributes: true,
attributeFilter: ["data-theme"],
});
return () => observer.disconnect();
}, []);
return mode;
}
@@ -1,27 +1,53 @@
/**
* Thin orchestrator: load -> interpret -> return the generic ReportViewModel. No SQL or render
* here — data access lives in each report's `load`, meaning in `interpret`, presentation in the
* renderers, and the catalog of reports in `report-registry.ts`.
*
* `call` takes a resolved AuthenticatedEnvironment and is transport-independent (Seam B, §7):
* any future surface (MCP Resource, etc.) is just another caller of this same method.
*/
import {
createCache,
createLRUMemoryStore,
DefaultStatefulContext,
Namespace,
type UnkeyCache,
} from "@internal/cache";
import { type AuthenticatedEnvironment } from "~/services/apiAuth.server";
import { REPORT_REGISTRY } from "./report-registry";
import { HEALTH_THRESHOLDS } from "./health/health-core";
import { REPORT_REGISTRY, type ReportLoader } from "./report-registry";
import { type ReportViewModel } from "./report-view-model";
const DEFAULT_PERIOD = "1h";
/**
* Single-flight: collapse concurrent identical requests (same report/env/period) into one
* computation. A report fires up to ~9 ClickHouse queries and MCP/CLI clients easily call it
* several times at once — without this, N callers each launch the full query set and pile onto
* the per-project query-concurrency limit. Keyed per (key, env, period); entry drops on settle.
* How long a finished report stays reusable. Capped at the liveness fresh window so a
* cached report can never render "fresh" while its telemetry is already stale.
*/
export const REPORT_CACHE_TTL_MS = HEALTH_THRESHOLDS.liveness.freshMs;
/** How many report, environment and period triples one instance keeps. */
const REPORT_CACHE_MAX_ENTRIES = 500;
export type ReportCache = UnkeyCache<{ report: ReportViewModel }>;
/** In-process only: an entry is a whole report. `stale` equals `fresh`, so nothing stale is served. */
export function createReportCache(ttlMs: number = REPORT_CACHE_TTL_MS): ReportCache {
return createCache({
report: new Namespace<ReportViewModel>(new DefaultStatefulContext(), {
stores: [createLRUMemoryStore(REPORT_CACHE_MAX_ENTRIES)],
fresh: ttlMs,
stale: ttlMs,
}),
});
}
const defaultReportCache = createReportCache();
/**
* Collapses concurrent identical requests into one computation: the cache only helps once a load has
* finished, so all callers of a cold key would otherwise hit the query-concurrency limit at once.
*/
const inFlight = new Map<string, Promise<ReportViewModel | undefined>>();
export class ReportPresenter {
constructor(
private readonly registry: Record<string, ReportLoader<unknown>> = REPORT_REGISTRY,
private readonly cache: ReportCache = defaultReportCache
) {}
async call({
environment,
key,
@@ -31,16 +57,23 @@ export class ReportPresenter {
key: string;
period?: string;
}): Promise<ReportViewModel | undefined> {
const loader = REPORT_REGISTRY[key];
if (!loader) return undefined;
if (!Object.hasOwn(this.registry, key)) return undefined;
const loader = this.registry[key];
// The environment id is part of the key, so one environment can never read another's report.
const flightKey = `${key} ${environment.id} ${period}`;
const cached = await this.cache.report.get(flightKey);
if (cached.val) return cached.val;
const existing = inFlight.get(flightKey);
if (existing) return existing;
const promise = (async () => {
const input = await loader.load(environment, period);
return loader.interpret(input);
const report = loader.interpret(input);
await this.cache.report.set(flightKey, report);
return report;
})().finally(() => inFlight.delete(flightKey));
inFlight.set(flightKey, promise);
@@ -1,8 +1,3 @@
/**
* EXECUTION analyzer: "are the runs that DO start completing OK?" — failures + durations only.
* Its "read:" line answers whether the flow problem is a code problem.
*/
import { isOk, maxSeverity, type Finding, type Metric } from "../report-view-model";
import { HEALTH_THRESHOLDS, metricById, type HealthInput } from "./health-core";
@@ -40,10 +35,7 @@ export function interpretExecution(metrics: Metric[], input: HealthInput): Findi
};
}
/** Flow causes that provably CAN'T be user code, so a healthy execution reads "not a code problem".
* dequeue_stall is platform-side (capacity free but nothing dequeuing). Trigger spike/surge are
* excluded: a code path fanning out task.trigger can BE the cause, so we only state the runs that
* execute are fine — never the global "not a code problem". */
// Trigger spike and surge are excluded: code fanning out task.trigger can be the cause.
const NOT_A_CODE_PROBLEM_CAUSES = new Set(["dequeue_stall"]);
export function buildExecutionRead(execution: Finding, flow: Finding): string {
@@ -1,10 +1,3 @@
/**
* FLOW analyzer: "is work flowing, and if not, WHY?" Diagnosed by a CAUSE TREE — `flow.reason`
* names why work is backing up (env_limit_saturation, dequeue_stall, …), selected from flow's
* evidence, falling back to v1 symptom reasons when no discriminator fires. Evidence quantities
* carry no severity of their own — they only select the cause.
*/
import {
anomalyWindow,
isOk,
@@ -17,8 +10,10 @@ import {
type Severity,
} from "../report-view-model";
import {
bucketCoverage,
HEALTH_THRESHOLDS,
isPendingIncreasing,
isPendingUnknown,
mean,
metricById,
type HealthInput,
@@ -26,7 +21,9 @@ import {
export const FLOW_METRIC_IDS = ["start_latency_p95", "pending", "throughput"];
/** One row of the declarative cause table — everything a cause defines about itself. */
/** Unmeasurable backlog: verdict is unassessable. Distinct from "unknown", the staleness guard. */
export const FLOW_UNMEASURED = "flow_unmeasured";
type CauseSpec = {
reason: string;
metricIds: string[]; // real metric rows, causal order
@@ -44,46 +41,51 @@ export function interpretFlow(metrics: Metric[], input: HealthInput): Finding {
const flowMetrics = FLOW_METRIC_IDS.map((id) => metricById(metrics, id));
const severity = maxSeverity(...flowMetrics.map((m) => m.severity));
// `pending.now` is a placeholder here, so no cause tree may hang off it. What `runs` measured still
// stands: only a flow with nothing measurably wrong is unassessable, the rest reports its symptom.
if (isPendingUnknown(input)) {
return isOk(severity)
? { type: "flow", severity, reason: FLOW_UNMEASURED, metricIds: FLOW_METRIC_IDS }
: fallbackFlow(flowMetrics, severity);
}
if (isOk(severity)) {
return { type: "flow", severity, reason: "healthy", metricIds: FLOW_METRIC_IDS };
}
// Discriminators (evidence only, no own severity).
const pendingIncreasing = isPendingIncreasing(input.pending.series);
const latencyElevated = !isOk(metricById(metrics, "start_latency_p95").severity);
// Concurrency causes need real running-capacity evidence — without it runningShare is a
// meaningless 0 and would falsely select dequeue_stall on the snapshot path (#1).
const hasConcurrencyEvidence = ev.envLimit > 0 && ev.runningSeries.length > 0;
// Without real running-capacity evidence runningShare is a meaningless 0 that selects
// dequeue_stall; without enough arrived buckets a few fresh ones read as pinned all window.
const coverage = bucketCoverage(input);
const hasConcurrencyEvidence =
ev.envLimit > 0 && ev.runningSeries.length > 0 && coverage.sufficient;
const runningShare = hasConcurrencyEvidence ? mean(ev.runningSeries) / ev.envLimit : 1;
// Pinned share is measured against expected buckets, not received rows.
const pinnedShare = hasConcurrencyEvidence
? ev.runningSeries.filter((r) => r >= t.pinnedLevel * ev.envLimit).length /
ev.runningSeries.length
coverage.expectedBuckets
: 0;
const pinned = pinnedShare >= t.pinnedShare;
const hasTriggerBaseline = input.throughput.normalTriggeredPerMin > 0;
const triggeredMult = hasTriggerBaseline
? input.throughput.triggeredPerMin / input.throughput.normalTriggeredPerMin
: 0;
// No baseline: a multiplier can't be computed, so an absolute rate selects "new volume".
// No baseline means no multiplier, so an absolute rate selects "new volume".
const triggerSurge = !hasTriggerBaseline && input.throughput.triggeredPerMin >= t.surgePerMin;
const donePerMin = input.throughput.donePerMin;
const net = donePerMin - input.throughput.triggeredPerMin;
// Exclusions must be PROVEN, not assumed. "not your code" needs healthy execution; "limits
// aren't the bottleneck" needs no env-pin AND no queue throttling; the workers/spike ones
// state a measured fact (rate) rather than a global "everything's fine" claim.
// Work leaves the queue on any terminal status, not just completions.
const finishedPerMin = input.throughput.finishedPerMin;
const net = finishedPerMin - input.throughput.triggeredPerMin;
// "not your config" requires both no env pin and no queue throttling.
const executionHealthy =
isOk(metricById(metrics, "failures").severity) && isOk(metricById(metrics, "dur_p95").severity);
const queueThrottled = ev.throttledShare >= t.throttledShare;
const configHealthy = !pinned && !queueThrottled;
// A trigger spike/surge is only the CAUSE of a backup when work is actually piling up:
// completions falling behind (net < 0) AND the backlog trending up. Without this a spike that
// drains fine would still be blamed while its read says "queue fills faster than it drains".
// A spike is only a cause when work piles up: finishes behind triggers and backlog trending up.
const triggerBacklog = net < 0 && pendingIncreasing;
// First discriminator that fires wins (fixed priority). dequeue_stall is a last-resort
// "it's on our side" cause — so a known config bottleneck (queue throttling) must rule it
// out first, else throttled-but-idle-capacity is misread as a platform stall (#1).
// First discriminator wins. dequeue_stall is last resort: a known config bottleneck rules it out.
let spec: CauseSpec;
if (
hasConcurrencyEvidence &&
@@ -112,10 +114,8 @@ export function interpretFlow(metrics: Metric[], input: HealthInput): Finding {
drivingMetricId: "concurrency",
annotationCode: "pinned_minutes",
exclusions: [],
// States a measured fact (runs ARE completing at {rate}/min) — evidence the workers aren't
// dead. An observation, not an exclusion: it doesn't claim it's the limit, nor "keeps pace".
observations:
donePerMin > 0 ? [{ code: "not_workers_platform", evidence: { donePerMin } }] : [],
finishedPerMin > 0 ? [{ code: "not_workers_platform", evidence: { finishedPerMin } }] : [],
recommendation: { code: "raise_env_limit", link: "concurrency" },
usesAttribution: true,
};
@@ -125,7 +125,6 @@ export function interpretFlow(metrics: Metric[], input: HealthInput): Finding {
metricIds: ["throttled", "pending"],
drivingMetricId: "throttled",
annotationCode: "throttled_minutes",
// Justified: we're in the not-pinned branch, so the env limit isn't the bottleneck.
exclusions: [{ code: "not_env_limit" }],
observations: [],
recommendation: { code: "raise_queue_limit", link: "queue" },
@@ -138,9 +137,6 @@ export function interpretFlow(metrics: Metric[], input: HealthInput): Finding {
drivingMetricId: "triggered",
annotationCode: "spike_mult",
exclusions: [],
// Only the proven fact — the runs that DO start execute fine. An observation, NOT the
// exclusion "not your code": healthy execution doesn't prove the code isn't the one
// triggering the flood (e.g. a deploy that fans out task.trigger in a loop).
observations: executionHealthy ? [{ code: "execution_healthy" }] : [],
recommendation: { code: "review_trigger_source", link: "runs" },
usesAttribution: false,
@@ -152,20 +148,17 @@ export function interpretFlow(metrics: Metric[], input: HealthInput): Finding {
drivingMetricId: "triggered",
annotationCode: "surge_rate",
exclusions: [],
// Same as a spike: only the proven fact, without ruling out the trigger-producing code.
observations: executionHealthy ? [{ code: "execution_healthy" }] : [],
recommendation: { code: "review_trigger_source", link: "runs" },
usesAttribution: false,
};
} else {
// Fallback — v1 symptom reasons by dominant metric.
return fallbackFlow(flowMetrics, severity);
}
return assembleFlowCause(spec, metrics, input, severity);
}
/** Build a flow Finding from a cause spec: annotation, anomaly window, attribution, evidence. */
function assembleFlowCause(
spec: CauseSpec,
metrics: Metric[],
@@ -175,21 +168,22 @@ function assembleFlowCause(
const t = HEALTH_THRESHOLDS;
const driving = metricById(metrics, spec.drivingMetricId);
// Anomaly window from the driving series. env_limit_saturation breaches ABOVE
// (concurrency pinned at the limit); dequeue_stall breaches BELOW (capacity idle).
// NOTE: runningSeries is at native env_metrics resolution (not resampled), so the "(last N
// min)" figure assumes those buckets are uniform and cover the resolved window. env_metrics
// are emitted on a fixed cadence, so that holds; a gappy/partial window could skew the minutes.
// env_limit_saturation breaches above the threshold, dequeue_stall below it. runningSeries is not
// gap-filled, so the duration counts per real bucket cadence and gaps break the contiguous run.
let aw: Finding["anomalyWindow"];
if (spec.reason === "env_limit_saturation" || spec.reason === "dequeue_stall") {
const below = spec.reason === "dequeue_stall";
const threshold = below
? t.flowCause.stallRunningShare * input.flowEvidence.envLimit
: t.flowCause.pinnedLevel * input.flowEvidence.envLimit;
aw = anomalyWindow(input.flowEvidence.runningSeries, threshold, input.windowMinutes, { below });
const coverage = bucketCoverage(input);
aw = anomalyWindow(input.flowEvidence.runningSeries, threshold, input.windowMinutes, {
below,
bucketMinutes: coverage.known ? coverage.bucketMinutes : undefined,
timestampsMs: input.flowEvidence.runningBucketsMs,
});
}
// Annotation on the driving metric (a fact, not an invented number).
if (spec.annotationCode) {
const value =
spec.annotationCode === "pinned_minutes"
@@ -211,16 +205,13 @@ function assembleFlowCause(
driving.annotation = { code: spec.annotationCode, value };
}
// Attribution — only when a queue owns >= minShare of the problem.
let attribution: Finding["attribution"];
const wq = input.flowEvidence.worstQueue;
if (spec.usesAttribution && wq && wq.share >= t.attribution.minShare) {
attribution = { dim: "queue", key: wq.name, share: wq.share, of: "pending" };
}
// Append "nothing dead-lettered" ONLY on a measured zero (dlqDelta === 0); null means
// unmeasured (snapshot path) — no observation without evidence. It's a supporting fact, not a
// ruled-out cause, so it joins observations.
// Only a measured zero supports "nothing dead-lettered". Null means unmeasured.
const observations =
input.flowEvidence.dlqDelta === 0
? [...spec.observations, { code: "nothing_dead_lettered", evidence: { dlq: 0 } }]
@@ -239,7 +230,6 @@ function assembleFlowCause(
};
}
/** v1 symptom fallback when no cause discriminator fires. */
function fallbackFlow(flowMetrics: Metric[], severity: Severity): Finding {
const firstOff = flowMetrics.find((m) => !isOk(m.severity));
const reason =
@@ -267,10 +257,6 @@ function fallbackFlow(flowMetrics: Metric[], severity: Severity): Finding {
};
}
// ---------------------------------------------------------------------------
// Flow severity policy + the causal "read:" line.
// ---------------------------------------------------------------------------
export function applyFlowPolicy(
flow: Finding,
execution: Finding,
@@ -279,7 +265,6 @@ export function applyFlowPolicy(
): Finding {
if (flow.severity !== "crit") return flow;
// Downgrade a drainable crit to warn only when execution is fine and telemetry isn't stale.
// Unknown/lagging freshness must NOT block this (it's not a signal that anything's wrong).
const severity: Severity =
isOk(execution.severity) && !telemetryStale && isDrainable ? "warn" : "crit";
return { ...flow, severity };
@@ -294,10 +279,10 @@ const CAUSE_READS: Record<string, string> = {
};
export function buildFlowRead(flow: Finding, executionOk: boolean, livenessFresh: boolean): string {
if (flow.reason === "unknown") return "data_stale"; // stale-guarded — no causal read
if (flow.reason === "unknown") return "data_stale";
if (flow.reason === FLOW_UNMEASURED) return "flow_unmeasured";
if (isOk(flow.severity)) return "starting_normally";
if (CAUSE_READS[flow.reason]) return CAUSE_READS[flow.reason];
// fallback symptoms (v1 logic)
if (executionOk && livenessFresh) return "lag_while_triggering_normal";
if (!executionOk) return "lag_and_failures";
return "degraded_generic";
@@ -1,9 +1,3 @@
/**
* Health report FOUNDATION shared by the three analyzers (flow / execution / liveness):
* the input shape, tunable thresholds, small helpers, and `buildMetrics` (numbers +
* per-metric severity). No verdict logic here — that lives in the per-analyzer modules.
*/
import { classifySeverity, delta, type Metric, type Severity } from "../report-view-model";
export type HealthInput = {
@@ -11,16 +5,20 @@ export type HealthInput = {
period: string;
baselineLabel: string;
generatedAt: string;
/** live window length in minutes — for anomaly-window / annotation math. */
windowMinutes: number;
/** provenance; drives caveat text, not logic. */
/** Provenance. Drives caveat text, not logic. */
flowSource: "snapshot+runs" | "queue_metrics_v1";
/**
* now = live env-level depth; normal = 7d baseline (omitted on the snapshot path, which has
* no real 7d pending baseline — so we never mislabel a live-window average as "7d normal");
* series measured (v2) or estimated (v1).
* `normal` is the 7d baseline, omitted on the snapshot path. `availability: "unknown"` means the
* depth was not measured and `now` is a placeholder, not a confident 0.
*/
pending: { now: number; normal?: number; series: number[]; estimated: boolean };
pending: {
now: number;
normal?: number;
series: number[];
estimated: boolean;
availability?: "measured" | "unknown";
};
/**
* p95 wait. `availability: "unknown"` = the source had no measurement, so `p95Ms` is a
* placeholder that must not be graded — a 0 would read as a confident green.
@@ -31,31 +29,39 @@ export type HealthInput = {
series: number[];
availability?: "measured" | "unknown";
};
throughput: { donePerMin: number; triggeredPerMin: number; normalTriggeredPerMin: number };
/** `finishedPerMin` is all terminal runs and the drain rate. `completedPerMin` is successes only. */
throughput: {
finishedPerMin: number;
completedPerMin: number;
triggeredPerMin: number;
normalTriggeredPerMin: number;
};
failures: { rate: number; normalRate: number; series: number[] };
duration: { p95Ms: number; normalP95Ms: number };
/** Age of the freshest telemetry (ms). null = no signal to assess -> freshness unknown. */
/** Age of the freshest telemetry in ms. Null means no signal to assess. */
liveness: { telemetryAgeMs: number | null };
/**
* Flow's cause-tree discriminators (no own severity) — from env_metrics rows + one
* queue_metrics GROUP BY. Empty-ish when unavailable (cause tree falls back to v1).
*/
/** Flow's cause-tree discriminators. No severity of their own. */
flowEvidence: {
runningSeries: number[];
/** Epoch ms per `runningSeries` bucket. Absent means contiguity falls back to index adjacency. */
runningBucketsMs?: number[];
/**
* Cadence and expected bucket count of `runningSeries`, which is not gap-filled. Absent means
* the cadence is unknown and received buckets are assumed to spread evenly over the window.
*/
sampling?: { bucketMinutes: number; expectedBuckets: number } | null;
envLimit: number;
throttledShare: number;
worstQueue: { name: string; share: number } | null;
/** runs dead-lettered in the window: 0 = measured none, null = unmeasured (snapshot). */
/** Runs dead-lettered in the window. 0 is a measured none, null is unmeasured. */
dlqDelta: number | null;
};
/** Lazy — loaded only when execution degrades (attribution line). */
/** Loaded only when execution degrades. */
failureBreakdown?: { task: string; share: number; region?: string };
};
/** Tunable defaults — first-guess; tune against prod once wired. */
export const HEALTH_THRESHOLDS = {
// `floor` = absolute warn/crit used when there's no usable baseline (normal 0/undefined),
// so a spike from a zero baseline (e.g. a never-failing env) isn't classified healthy.
// `floor` is the absolute warn/crit used when there's no usable baseline.
startLatency: { warnMult: 3, critMult: 10, floor: { warn: 30_000, crit: 120_000 } },
pending: { warnMult: 2, critMult: 10, floor: { warn: 500, crit: 5_000 } },
failures: {
@@ -73,21 +79,18 @@ export const HEALTH_THRESHOLDS = {
pinnedShare: 0.5, // env_limit_saturation: >= half the window's buckets pinned
throttledShare: 0.25, // queue_limit_throttling: >= a quarter of the window throttled
spikeMult: 3, // trigger_spike: triggered/min >= 3x the 7d-normal rate
// trigger_surge: with NO usable baseline (normal 0), a multiplier is meaningless, so an
// absolute floor picks the "new volume" cause instead of dropping to the v1 fallback.
surgePerMin: 100,
surgePerMin: 100, // trigger_surge: absolute triggered/min floor used when there's no baseline
// Minimum share of the window's expected buckets that must have arrived before a
// concurrency-shaped cause may be named. Below it flow drops to a symptom-level verdict.
minCoverage: 0.5,
},
attribution: { minShare: 0.5 }, // name a queue/task/region only when it owns >= half the problem
};
// ---------------------------------------------------------------------------
// Helpers.
// ---------------------------------------------------------------------------
export const mean = (xs: number[]) =>
xs.length === 0 ? 0 : xs.reduce((a, b) => a + b, 0) / xs.length;
/** Deterministic "trending up": mean of the last third vs the first third (direction only). */
/** Trending up: mean of the last third against the first third. */
export function isPendingIncreasing(series: number[]): boolean {
if (series.length < 2) return false;
const third = Math.max(1, Math.floor(series.length / 3));
@@ -108,19 +111,56 @@ function multiplierSeverity(
return classifySeverity(value / normal, { warn: warnMult, crit: critMult });
}
/** Look up a metric by id; throws if absent (buildMetrics guarantees the standard set exists). */
/** True when the depth was not measured, so `pending.now` is a placeholder. */
export function isPendingUnknown(input: HealthInput): boolean {
return input.pending.availability === "unknown";
}
export type BucketCoverage = {
/** Buckets the window should contain at the source's cadence. */
expectedBuckets: number;
receivedBuckets: number;
bucketMinutes: number;
/** Received over expected. 1 when the cadence is unknown. */
coverage: number;
/** Enough of the window arrived to support a cause and a duration. */
sufficient: boolean;
/** True when the source reported its cadence, so gaps are detectable. */
known: boolean;
};
/** Without the source's cadence the received buckets are assumed to span the window evenly. */
export function bucketCoverage(input: HealthInput): BucketCoverage {
const received = input.flowEvidence.runningSeries.length;
const sampling = input.flowEvidence.sampling;
if (!sampling || sampling.expectedBuckets <= 0) {
return {
expectedBuckets: received,
receivedBuckets: received,
bucketMinutes: received > 0 ? input.windowMinutes / received : 0,
coverage: 1,
sufficient: true,
known: false,
};
}
const coverage = received / sampling.expectedBuckets;
return {
expectedBuckets: sampling.expectedBuckets,
receivedBuckets: received,
bucketMinutes: sampling.bucketMinutes,
coverage,
sufficient: coverage >= HEALTH_THRESHOLDS.flowCause.minCoverage,
known: true,
};
}
export function metricById(metrics: Metric[], id: string): Metric {
const m = metrics.find((x) => x.id === id);
if (!m) throw new Error(`health: missing metric ${id}`);
return m;
}
// ---------------------------------------------------------------------------
// buildMetrics: numbers + per-metric severity. The standard six carry severity; the
// flow-evidence metrics (concurrency, throttled, triggered) are evidence only (severity ok)
// and exist so a cause's metricIds resolve.
// ---------------------------------------------------------------------------
// Only the standard six metrics carry severity; the flow-evidence metrics exist so metricIds resolve.
export function buildMetrics(input: HealthInput): Metric[] {
const t = HEALTH_THRESHOLDS;
const ev = input.flowEvidence;
@@ -151,32 +191,41 @@ export function buildMetrics(input: HealthInput): Metric[] {
),
};
// An unmeasurable depth must not be classified: a placeholder 0 would read as a confident green.
const pendingUnknown = isPendingUnknown(input);
const pending: Metric = {
id: "pending",
value: input.pending.now,
unit: "count",
availability: pendingUnknown ? "unknown" : "measured",
normal: input.pending.normal,
delta: delta(input.pending.now, input.pending.normal),
delta: pendingUnknown ? undefined : delta(input.pending.now, input.pending.normal),
series: {
points: input.pending.series,
kind: input.pending.estimated ? "estimated" : "measured",
},
severity: multiplierSeverity(
input.pending.now,
input.pending.normal,
t.pending.warnMult,
t.pending.critMult,
t.pending.floor
),
severity: pendingUnknown
? "ok"
: multiplierSeverity(
input.pending.now,
input.pending.normal,
t.pending.warnMult,
t.pending.critMult,
t.pending.floor
),
};
const net = input.throughput.donePerMin - input.throughput.triggeredPerMin;
// Net drain uses all terminal runs, not just completions.
const net = input.throughput.finishedPerMin - input.throughput.triggeredPerMin;
const throughput: Metric = {
id: "throughput",
value: net,
unit: "perMin",
aggregation: "rate",
breakdown: { done: input.throughput.donePerMin, triggered: input.throughput.triggeredPerMin },
breakdown: {
done: input.throughput.finishedPerMin,
triggered: input.throughput.triggeredPerMin,
},
severity: net < 0 && isPendingIncreasing(input.pending.series) ? "warn" : "ok",
};
@@ -218,27 +267,24 @@ export function buildMetrics(input: HealthInput): Metric[] {
};
const ageMs = input.liveness.telemetryAgeMs;
// No signal stays neutral. Lagging telemetry is a real warn; unknown is not.
const livenessSeverity: Severity =
ageMs === null // no signal at all (brand-new/quiet env) — genuinely unknown, so NEUTRAL (ok):
? "ok" // a fine-but-idle env must not surface as a yellow verdict, and it never
: // trust-guards. "lagging" (below) IS a real warn; "unknown" is not.
ageMs > t.liveness.staleMs
ageMs === null
? "ok"
: ageMs > t.liveness.staleMs
? "crit"
: ageMs > t.liveness.freshMs
? "warn"
: "ok";
const liveness: Metric = {
id: "liveness",
// No signal -> value 0 is a placeholder, NOT a real "0ms fresh". `availability: "unknown"`
// says so, so a structured consumer never reads the 0 as freshness (the finding reason
// also carries "freshness_unknown"). A finite number keeps the JSON VM valid (no Infinity).
// With no signal, 0 is a placeholder rather than a real "0ms fresh"; `availability` says so.
value: ageMs ?? 0,
availability: ageMs === null ? "unknown" : "measured",
unit: "ms",
severity: livenessSeverity,
};
// Flow-evidence metrics (severity ok — evidence, not a verdict).
const concurrency: Metric = {
id: "concurrency",
value: ev.runningSeries.length > 0 ? ev.runningSeries[ev.runningSeries.length - 1] : 0,
@@ -275,15 +321,15 @@ export function buildMetrics(input: HealthInput): Metric[] {
];
}
// ---------------------------------------------------------------------------
// Drain computation — shared by the flow policy (flow.ts) and the footer (health.ts).
// ---------------------------------------------------------------------------
export function computeDrain(input: HealthInput): { drainMinutes: number; isDrainable: boolean } {
const donePerMin = input.throughput.donePerMin;
const drainMinutes = donePerMin === 0 ? Number.POSITIVE_INFINITY : input.pending.now / donePerMin;
// The drain rate counts runs leaving the queue, not just completions.
const finishedPerMin = input.throughput.finishedPerMin;
const drainMinutes =
finishedPerMin === 0 ? Number.POSITIVE_INFINITY : input.pending.now / finishedPerMin;
return {
drainMinutes,
isDrainable: drainMinutes < HEALTH_THRESHOLDS.flowPolicy.drainCritMinutes,
// An unmeasurable depth can't produce an ETA.
isDrainable:
!isPendingUnknown(input) && drainMinutes < HEALTH_THRESHOLDS.flowPolicy.drainCritMinutes,
};
}
@@ -1,28 +1,15 @@
/**
* The `health` report's DATA layer — the ONLY place SQL / Redis IO lives. Loads a
* `HealthInput` (plain numbers) that the pure `interpret()` turns into a VM.
*
* Flow signal is sourced behind `FlowSource`:
* - QueueMetricsSource (preferred) — MEASURED queue depth + real scheduling delay
* (p95 wait) from `env_metrics`, which is env-level, so there is no per-queue split.
* - SnapshotFlowSource (fallback) — live Redis depth + an ESTIMATED backlog proxy
* and `runs.queued_duration`, used until the queue-metrics pipeline has populated
* `env_metrics` for this env.
*
* `flowSource` records which ran and drives `pending.estimated`, so the "informational
* only" caveat drops automatically on the measured path. Execution, liveness and
* throughput always come from `runs`.
*/
import { calculateTimeBucketInterval, type TimeBucketInterval } from "@internal/tsql";
import { type AuthenticatedEnvironment } from "~/services/apiAuth.server";
import { logger } from "~/services/logger.server";
import { executeQuery, isQueryConcurrencyRejection } from "~/services/queryService.server";
import { envMetricsSchema } from "~/v3/querySchemas";
import { engine } from "~/v3/runEngine.server";
import { HEALTH_THRESHOLDS, type HealthInput } from "./health";
/** §5.B — failures = user-code failures only (Expired/Canceled excluded from both sides). */
/** User-code failures only. Expired and Canceled are excluded from both sides of the rate. */
const FAILURE_STATUSES = "'Failed','Crashed','System failure','Timed out'";
/** All terminal statuses a run that reached any of these has left the queue (not backlog). */
/** All terminal statuses: a run that reached any of these has left the queue. */
const FINISHED_STATUSES =
"'Completed','Canceled','Expired','Failed','Crashed','System failure','Timed out'";
@@ -49,7 +36,6 @@ function mean(xs: number[]): number {
return xs.length === 0 ? 0 : xs.reduce((a, b) => a + b, 0) / xs.length;
}
/** Downsample a per-bucket series to ~N points so the sparkline width is stable (plan §5.C). */
function resampleSeries(points: number[], target = SPARKLINE_BUCKETS): number[] {
if (points.length <= target) return points;
const out: number[] = [];
@@ -68,13 +54,7 @@ function failureRate(failures: number, completed: number): number {
return denom === 0 ? 0 : failures / denom;
}
/**
* Bug 1 fix — ClickHouse query concurrency cap. The query service rejects the 4th
* concurrent query per project (-> `runQuery` throws -> 500); this loader used to fire
* up to 4 at once. `mapWithConcurrency` runs `fn` over `items` with at most `limit` in
* flight, preserving order. We cap CH calls at 2 to leave headroom under the limit of 3
* for other project traffic. (Redis calls don't count — kept outside this helper.)
*/
/** Order-preserving. The query service rejects the 4th concurrent query per project. */
async function mapWithConcurrency<T, R>(
items: readonly T[],
limit: number,
@@ -94,15 +74,10 @@ async function mapWithConcurrency<T, R>(
return results;
}
/** Max ClickHouse queries this loader keeps in flight (limit of 3, leave headroom). */
/** Max ClickHouse queries in flight. The per-project limit is 3, so this leaves headroom. */
const CH_CONCURRENCY = 2;
/**
* The per-project limit (3) is shared across ALL in-flight requests, not just this
* loader — so concurrent report requests can still transiently exceed it and get a
* rejection. It's retryable (a slot frees when another query finishes), so we back off
* and retry rather than surface a 500. Scheduling only — same query, same result.
*/
// The per-project limit is shared, so concurrent requests can still be rejected. That is retryable.
const CH_REJECTION_RETRIES = 6;
const CH_REJECTION_BACKOFF_MS = 60; // base for exponential "full jitter" backoff
const CH_REJECTION_BACKOFF_CAP_MS = 2000; // ceiling per attempt
@@ -119,39 +94,26 @@ function isConcurrencyRejection(error: unknown): boolean {
return /concurrency|too many|try again/i.test(message);
}
/** rows + the actual (clip-aware) time window the query service resolved for this run. */
/** Rows plus the clip-aware time window the query service resolved for this run. */
type QueryResult = { rows: Row[]; timeRange: { from: Date; to: Date } };
/** Runs one (TRQL) report query. Injectable so tests can drive the loader with canned results. */
export type HealthQueryRunner = (
env: AuthenticatedEnvironment,
query: string,
period: string
) => Promise<QueryResult>;
/**
* The loader's IO boundary (§7 Seam A): ClickHouse via the query service, Redis via the
* engine. Defaults wire the real singletons; overriding lets tests drive the loader's
* orchestration without booting the env-bound query-service client.
*/
/** The loader's IO boundary. */
export type HealthDeps = {
runQuery: HealthQueryRunner;
lengthOfEnvQueue: (env: AuthenticatedEnvironment) => Promise<number | undefined>;
};
/**
* The 7d baseline changes slowly, so cache it briefly (per env + query) to avoid recomputing a
* wide query on every request — the biggest lever on query pressure (#12). Only the default
* runner caches; injected test runners bypass this entirely, so test isolation is preserved.
*/
// The 7d baseline is cached per env and query. Only the default runner caches.
const BASELINE_CACHE_TTL_MS = 5 * 60_000;
const baselineCache = new Map<string, { expiresAt: number; result: QueryResult }>();
/**
* Store a baseline result, first sweeping expired entries so envs that stop requesting reports
* don't linger in memory forever. Writes only happen on a cache miss (~once per env per TTL), so
* a full sweep here is cheap and keeps the map bounded to recently-active environments.
*/
/** Sweeps expired entries first so envs that stop requesting reports don't linger in memory. */
function cacheBaseline(key: string, result: QueryResult, now: number) {
for (const [k, v] of baselineCache) {
if (v.expiresAt <= now) baselineCache.delete(k);
@@ -185,10 +147,7 @@ async function executeReportQuery(
if (cacheKey) cacheBaseline(cacheKey, out, Date.now());
return out;
}
// Retry transient concurrency rejections; rethrow anything else (e.g. a bad query) so
// callers/tryQuery can handle it. Exponential "full jitter" backoff — delay picked uniformly
// from [0, min(cap, base·2^attempt)) — so concurrent report requests don't wake in lockstep
// and re-collide on the same 3 query slots.
// Full-jitter backoff so concurrent report requests don't wake in lockstep and re-collide.
if (attempt < CH_REJECTION_RETRIES && isConcurrencyRejection(result.error)) {
const window = Math.min(CH_REJECTION_BACKOFF_CAP_MS, CH_REJECTION_BACKOFF_MS * 2 ** attempt);
await new Promise((resolve) => setTimeout(resolve, Math.floor(Math.random() * window)));
@@ -198,17 +157,14 @@ async function executeReportQuery(
}
}
// ---------------------------------------------------------------------------
// runs queries (execution + liveness + throughput; also feed the snapshot fallback).
// executeQuery injects tenant isolation + the time window, so we never write WHERE.
// ---------------------------------------------------------------------------
// executeQuery injects tenant isolation and the time window, so these queries never write WHERE.
function runsScalarQuery(): string {
return `SELECT
quantile(0.95)(queued_duration) AS start_latency_p95,
quantile(0.95)(execution_duration) AS dur_p95,
countIf(status IN (${FAILURE_STATUSES})) AS failures,
countIf(status = 'Completed') AS completed,
countIf(status IN (${FINISHED_STATUSES})) AS finished,
count() AS triggered,
max(triggered_at) AS last_activity
FROM runs`;
@@ -227,10 +183,6 @@ GROUP BY t
ORDER BY t`;
}
// ---------------------------------------------------------------------------
// env_metrics queries (measured queue depth + scheduling delay).
// ---------------------------------------------------------------------------
function envSeriesQuery(): string {
return `SELECT
timeBucket() AS t,
@@ -253,9 +205,8 @@ FROM env_metrics`;
}
/**
* Worst queue by share of CURRENT pending. `argMax(max_queued, bucket_start)` = each queue's
* depth in its latest bucket, so the shares sum to a real point-in-time backlog — not a sum of
* per-queue peaks from different moments (which isn't "% of pending" at any instant). Best-effort.
* `argMax(max_queued, bucket_start)` is a point-in-time depth, not a peak. These rows stop at 20, so
* the share's denominator comes from `queueTotalsQuery`.
*/
function queueWorstQuery(): string {
return `SELECT
@@ -268,20 +219,21 @@ LIMIT 20`;
}
/**
* Runs dead-lettered across the window, summed over queues. `dlq_delta` is per-queue
* cumulative-counter state, so it must be merged per queue then summed (never merged
* across queues). Best-effort — absent columns just yield no rows.
* `dlq_delta` must be merged per queue then summed, never merged across queues. `total_queued` must
* be computed here, not by summing the top-20 `queueWorstQuery` rows.
*/
function dlqTotalQuery(): string {
return `SELECT sum(dlq) AS dlq_total
function queueTotalsQuery(): string {
return `SELECT sum(dlq) AS dlq_total, sum(latest_queued) AS total_queued
FROM (
SELECT deltaSumTimestampMerge(dlq_delta) AS dlq
SELECT
deltaSumTimestampMerge(dlq_delta) AS dlq,
argMax(max_queued, bucket_start) AS latest_queued
FROM queue_metrics
GROUP BY queue
)`;
}
/** Top failing task (lazy — only when execution degrades). Best-effort. */
/** Top failing task. Loaded lazily, only when execution degrades. */
function failureBreakdownQuery(): string {
return `SELECT
task_identifier AS task,
@@ -292,40 +244,31 @@ ORDER BY fails DESC
LIMIT 10`;
}
/** Default IO wiring — the real query-service runner + the engine's env-queue length. */
const defaultHealthDeps: HealthDeps = {
runQuery: executeReportQuery,
lengthOfEnvQueue: (env) => engine.lengthOfEnvQueue(env),
};
/** Run a query that may reference not-yet-available columns; never break the report. */
/** Never throws. Callers treat no rows as unmeasured rather than as a measured zero. */
async function tryQuery(
deps: HealthDeps,
env: AuthenticatedEnvironment,
query: string,
period: string
): Promise<Row[]> {
): Promise<QueryResult> {
try {
return (await deps.runQuery(env, query, period)).rows;
return await deps.runQuery(env, query, period);
} catch {
return [];
return { rows: [], timeRange: { from: new Date(0), to: new Date(0) } };
}
}
// ---------------------------------------------------------------------------
// FlowSource seam (§7 Seam A).
// ---------------------------------------------------------------------------
export type FlowData = {
flowSource: HealthInput["flowSource"];
pending: { now: number; normal?: number; series: number[]; estimated: boolean };
pending: HealthInput["pending"];
startLatency: HealthInput["startLatency"];
evidence: HealthInput["flowEvidence"];
/**
* Epoch ms of the freshest telemetry the source saw (latest env_metrics bucket and/or latest
* run) — how the report tells "data current" from "pipeline stale", independent of traffic.
* null when no signal exists at all (brand-new/empty env) -> liveness "unknown", not stale.
*/
/** Epoch ms of the freshest telemetry. Null means no signal, which makes liveness unknown. */
telemetryLastTs: number | null;
};
@@ -337,116 +280,197 @@ const EMPTY_EVIDENCE: HealthInput["flowEvidence"] = {
dlqDelta: null, // snapshot path: dead-letter volume is unmeasured
};
/** The runs results loadHealthInput already fetched, so the snapshot fallback needn't re-query. */
type RunsContext = { liveScalar: Row; liveSeries: Row[]; baselineScalar: Row };
/**
* "unavailable" is a recognized rollout state, so the next source down is a legitimate substitute.
* "failed" is anything else and must make the flow verdict unassessable, never fall through to it.
*/
export type FlowLoadResult =
| { status: "ok"; data: FlowData }
| { status: "unavailable" }
| { status: "failed"; error: unknown };
export interface FlowSource {
loadFlow(
env: AuthenticatedEnvironment,
period: string,
ctx: RunsContext,
deps: HealthDeps
): Promise<FlowData | null>;
): Promise<FlowLoadResult>;
}
/**
* Preferred source: measured queue depth + scheduling-delay p95 from `env_metrics`.
* Returns null when the pipeline hasn't populated the table yet, so the caller can fall
* back to the snapshot.
* The only failures the measured source may treat as a benign fallback: the object itself does not
* exist yet, so waiting is correct. Codes 60 (table) and 81 (database). Matched on error text
* because the client collapses the error into a message.
*
* 47 (UNKNOWN_IDENTIFIER) is deliberately absent: the table exists and a column in our own query
* does not, which is a broken query, not a rollout gap. It must surface as a failed source.
*/
const ROLLOUT_ERROR_PATTERNS = [
/\bUNKNOWN_(?:TABLE|DATABASE)\b/,
/\bCode:\s*(?:60|81)\b/,
/\bTable\b[^.]*\bdoes\s?n?o?t?'?t?\s*exist/i,
/\bUnknown (?:table|database)\b/i,
];
function isRolloutError(error: unknown): boolean {
// Prefer a structured code/type if one ever survives the wrapping.
if (typeof error === "object" && error !== null) {
const record = error as Record<string, unknown>;
const code = String(record.code ?? "");
const type = String(record.type ?? "");
if (code === "60" || code === "81") return true;
if (/^UNKNOWN_(TABLE|DATABASE)$/.test(type)) return true;
}
const message =
error instanceof Error
? error.message
: typeof error === "string"
? error
: String(error ?? "");
return ROLLOUT_ERROR_PATTERNS.some((pattern) => pattern.test(message));
}
/** Measured depth and scheduling-delay p95 from `env_metrics`. Unavailable until it is populated. */
export const QueueMetricsSource: FlowSource = {
async loadFlow(env, period, ctx, deps) {
try {
// Redis depth isn't a CH query, so it runs alongside (doesn't count toward the cap).
// Guard the rejection: if the CH queries throw first we jump to catch without awaiting
// this, and an unhandled Redis rejection would crash the process.
// The rejection must be guarded: if the queries below throw first this is never awaited, and
// an unhandled Redis rejection would crash the process.
const pendingNowPromise = deps.lengthOfEnvQueue(env).catch(() => undefined);
// Bug 1 fix — route all CH queries through the concurrency cap (max 2 in flight).
// Every task returns Row[] (scalars indexed after) so mapWithConcurrency infers a
// single element type — a mixed Row[]/Row union trips its generic inference.
const [seriesRows, liveScalarRows, baselineScalarRows, worstQueueRows, dlqResultRows] =
// Every task must return the same shape; a mixed union trips mapWithConcurrency's inference.
const [seriesResult, liveScalarResult, baselineScalarResult, worstQueueResult, totalsResult] =
await mapWithConcurrency(
[
() => deps.runQuery(env, envSeriesQuery(), period).then((r) => r.rows),
() => deps.runQuery(env, envScalarQuery(), period).then((r) => r.rows),
() => deps.runQuery(env, envScalarQuery(), BASELINE_PERIOD).then((r) => r.rows),
() => deps.runQuery(env, envSeriesQuery(), period),
() => deps.runQuery(env, envScalarQuery(), period),
() => deps.runQuery(env, envScalarQuery(), BASELINE_PERIOD),
() => tryQuery(deps, env, queueWorstQuery(), period),
() => tryQuery(deps, env, dlqTotalQuery(), period),
() => tryQuery(deps, env, queueTotalsQuery(), period),
],
CH_CONCURRENCY,
(task) => task()
);
const liveScalarRow = liveScalarRows[0] ?? {};
const baselineScalarRow = baselineScalarRows[0] ?? {};
const seriesRows = seriesResult.rows;
const liveScalarRow = liveScalarResult.rows[0] ?? {};
const baselineScalarRow = baselineScalarResult.rows[0] ?? {};
const pendingNow = await pendingNowPromise;
if (seriesRows.length === 0) {
return null; // no measured data yet -> snapshot fallback
return { status: "unavailable" }; // no measured data yet -> snapshot fallback
}
// Telemetry freshness = the freshest of the latest env_metrics bucket (a heartbeat
// independent of traffic) and the latest run recorded.
// Freshness is the newer of the latest env_metrics bucket and the latest run.
const telemetryLastTs = freshestTs(liveScalarRow.last_bucket, ctx.liveScalar.last_activity);
return buildQueueMetricsFlow(
seriesRows,
liveScalarRow,
baselineScalarRow,
worstQueueRows,
dlqResultRows,
pendingNow,
telemetryLastTs
);
} catch {
// Bug 2 fix — if `env_metrics` isn't available the queries throw; return null so
// loadHealthInput falls back to the snapshot instead of propagating a 500.
return null;
return {
status: "ok",
data: buildQueueMetricsFlow({
series: seriesRows,
sampling: envSampling(seriesResult.timeRange),
liveScalar: liveScalarRow,
baselineScalar: baselineScalarRow,
worstRows: worstQueueResult.rows,
totalsRows: totalsResult.rows,
pendingNow,
telemetryLastTs,
}),
};
} catch (error) {
// Only a rollout error is a benign fallback. Anything else must surface as failed.
if (isRolloutError(error)) return { status: "unavailable" };
logger.error("report health: measured flow source failed", {
environmentId: env.id,
error: error instanceof Error ? error.message : String(error),
});
return { status: "failed", error };
}
},
};
function buildQueueMetricsFlow(
series: Row[],
liveScalar: Row,
baselineScalar: Row,
worstRows: Row[],
dlqRows: Row[],
pendingNow: number | undefined,
telemetryLastTs: number | null
): FlowData {
// Dead-letter volume (0 = measured none; no rows -> unmeasured -> null).
const dlqDelta = dlqRows.length > 0 ? Math.round(num(dlqRows[0].dlq_total)) : null;
/** Minutes per env_metrics bucket, matching the interval the query printer emits. */
const INTERVAL_UNIT_MINUTES: Record<TimeBucketInterval["unit"], number> = {
SECOND: 1 / 60,
MINUTE: 1,
HOUR: 60,
DAY: 1440,
WEEK: 10_080,
MONTH: 43_200,
};
// Throttled share = fraction of buckets with any queue-level throttling.
const throttledShare =
series.length > 0 ? series.filter((r) => num(r.throttled) > 0).length / series.length : 0;
/** Derived from the query printer's own thresholds, so "expected" matches what the query emits. */
function envSampling(range: {
from: Date;
to: Date;
}): { bucketMinutes: number; expectedBuckets: number } | null {
const windowMinutes = timeRangeMinutes(range);
if (windowMinutes === 0) return null;
const interval = calculateTimeBucketInterval(
range.from,
range.to,
envMetricsSchema.timeBucketThresholds
);
const bucketMinutes = interval.value * INTERVAL_UNIT_MINUTES[interval.unit];
if (!(bucketMinutes > 0)) return null;
return { bucketMinutes, expectedBuckets: Math.max(1, Math.round(windowMinutes / bucketMinutes)) };
}
// Worst queue = top queue's share of current pending (latest-bucket depths, so shares
// sum to a real point-in-time backlog).
function buildQueueMetricsFlow(args: {
series: Row[];
sampling: { bucketMinutes: number; expectedBuckets: number } | null;
liveScalar: Row;
baselineScalar: Row;
worstRows: Row[];
totalsRows: Row[];
pendingNow: number | undefined;
telemetryLastTs: number | null;
}): FlowData {
const { series, sampling, liveScalar, baselineScalar, worstRows, totalsRows } = args;
const totals = totalsRows[0];
// Zero means measured none; no rows means unmeasured.
const dlqDelta = totals !== undefined ? Math.round(num(totals.dlq_total)) : null;
// Measured against expected buckets when the cadence is known, not against received rows.
const throttledBuckets = series.filter((r) => num(r.throttled) > 0).length;
const throttledDenominator = sampling?.expectedBuckets ?? series.length;
const throttledShare = throttledDenominator > 0 ? throttledBuckets / throttledDenominator : 0;
// The denominator is the env-wide total, never the sum of these top-20 rows. No total, no share.
let worstQueue: HealthInput["flowEvidence"]["worstQueue"] = null;
if (worstRows.length > 0) {
const depths = worstRows.map((r) => num(r.latest_queued));
const total = depths.reduce((a, b) => a + b, 0);
if (total > 0) {
worstQueue = { name: String(worstRows[0].name ?? "unknown"), share: depths[0] / total };
const totalQueued = totals !== undefined ? num(totals.total_queued) : 0;
if (worstRows.length > 0 && totalQueued > 0) {
const worstDepth = num(worstRows[0].latest_queued);
if (worstDepth > 0) {
worstQueue = {
name: String(worstRows[0].name ?? "unknown"),
share: Math.min(1, worstDepth / totalQueued),
};
}
}
// Prefer live Redis depth; if it's unavailable fall back to the latest MEASURED queued from
// env_metrics (still a real number) rather than a misleading confident zero (#7).
// Prefer live Redis depth, then the latest measured queued from env_metrics.
const lastMeasuredQueued = num(series[series.length - 1]?.queued);
// Only carried when every bucket parsed: a partial set would make "adjacent" meaningless.
const bucketTimestamps = series.map((r) => parseTimestamp(r.t));
const runningBucketsMs = bucketTimestamps.every((t): t is number => t !== null)
? bucketTimestamps
: undefined;
const waitP95 = optionalNum(liveScalar.wait_p95);
return {
flowSource: "queue_metrics_v1",
pending: {
now: pendingNow ?? lastMeasuredQueued,
now: args.pendingNow ?? lastMeasuredQueued,
normal: Math.round(num(baselineScalar.avg_queued)),
series: resampleSeries(series.map((r) => num(r.queued))),
estimated: false, // measured
availability: "measured",
},
startLatency: {
p95Ms: waitP95 ?? 0,
@@ -455,30 +479,28 @@ function buildQueueMetricsFlow(
availability: waitP95 === undefined ? "unknown" : "measured",
},
evidence: {
// native resolution — cause discriminators read shares off this series.
runningSeries: series.map((r) => num(r.running)),
runningBucketsMs,
sampling,
envLimit: num(liveScalar.env_limit),
throttledShare,
worstQueue,
dlqDelta,
},
telemetryLastTs,
telemetryLastTs: args.telemetryLastTs,
};
}
/**
* Fallback: live Redis depth (accurate "now") + an estimated backlog proxy from `runs`
* (cumulative triggered - finished) and `runs.queued_duration` for latency. The proxy is a
* shape-only TREND (`estimated: true`): it starts at 0 within the window and can't see
* backlog that predates it.
* Fallback: live Redis depth plus a backlog proxy from `runs` (triggered minus finished). The proxy
* is shape-only: it starts at 0 within the window and can't see backlog that predates it.
*/
export const SnapshotFlowSource: FlowSource = {
async loadFlow(env, _period, ctx, deps) {
// Guard Redis: this is the last-resort source, so a failure must not break the report.
const pendingNow = (await deps.lengthOfEnvQueue(env).catch(() => undefined)) ?? 0;
// Last-resort source, so a Redis failure must not break the report.
const pendingNow = await deps.lengthOfEnvQueue(env).catch(() => undefined);
// Subtract ALL terminal runs, not just Completed — else failed/expired/canceled runs
// linger in the proxy as phantom backlog forever.
// Subtract all terminal runs, or failed and canceled runs linger as phantom backlog.
let backlog = 0;
const proxy = ctx.liveSeries.map((r) => {
backlog = Math.max(0, backlog + num(r.triggered) - num(r.finished));
@@ -487,43 +509,44 @@ export const SnapshotFlowSource: FlowSource = {
const series = resampleSeries(proxy);
const startLatencyP95 = optionalNum(ctx.liveScalar.start_latency_p95);
// Redis is the only depth measurement here, so a failure must not substitute 0. Fall back to the
// last proxy point and mark the depth unknown.
const depthUnavailable = pendingNow === undefined;
const lastProxyPoint = proxy.length > 0 ? proxy[proxy.length - 1] : 0;
return {
flowSource: "snapshot+runs",
pending: {
now: pendingNow,
// No 7d pending baseline on this path — omit `normal` rather than pass off a
// live-window proxy average as "7d normal" (#8). Severity falls back to an absolute floor.
normal: undefined,
series,
estimated: true,
status: "ok",
data: {
flowSource: "snapshot+runs",
pending: {
now: pendingNow ?? lastProxyPoint,
// No 7d pending baseline on this path, so severity falls back to an absolute floor.
normal: undefined,
series,
estimated: true,
availability: depthUnavailable ? "unknown" : "measured",
},
startLatency: {
p95Ms: startLatencyP95 ?? 0,
normalP95Ms: optionalNum(ctx.baselineScalar.start_latency_p95),
series: resampleSeries(ctx.liveSeries.map((r) => num(r.start_latency_p95))),
availability: startLatencyP95 === undefined ? "unknown" : "measured",
},
// No cause-tree evidence; interpret falls back to v1 symptoms.
evidence: EMPTY_EVIDENCE,
// This path has no pipeline heartbeat, and run activity is not one.
telemetryLastTs: null,
},
startLatency: {
p95Ms: startLatencyP95 ?? 0,
normalP95Ms: optionalNum(ctx.baselineScalar.start_latency_p95),
series: resampleSeries(ctx.liveSeries.map((r) => num(r.start_latency_p95))),
availability: startLatencyP95 === undefined ? "unknown" : "measured",
},
// No cause-tree evidence; interpret falls back to v1 symptoms.
evidence: EMPTY_EVIDENCE,
// No env_metrics heartbeat here — the only freshness signal is the latest run recorded
// (null when the env has no runs at all -> liveness "unknown", not stale).
telemetryLastTs: freshestTs(ctx.liveScalar.last_activity),
};
},
};
// ---------------------------------------------------------------------------
// loadHealthInput.
// ---------------------------------------------------------------------------
export async function loadHealthInput(
env: AuthenticatedEnvironment,
period: string,
now: Date = new Date(),
deps: HealthDeps = defaultHealthDeps
): Promise<HealthInput> {
// Bug 1 fix — route the runs-phase CH queries through the concurrency cap (max 2)
// instead of firing all 3 at once, so we never exceed the per-project limit.
const [liveScalarRes, liveSeriesRes, baselineScalarRes] = await mapWithConcurrency(
[
() => deps.runQuery(env, runsScalarQuery(), period),
@@ -539,17 +562,24 @@ export async function loadHealthInput(
baselineScalar: baselineScalarRes.rows[0] ?? {},
};
// Window lengths come from the query service's resolved (clip-aware) range, not a
// re-parse of the period — so `maxQueryPeriod` clipping can't skew per-minute rates
// or annotation math. periodToMinutes is a fallback for a degenerate range.
// Window lengths come from the resolved range, not the period, so clipping can't skew rates.
const windowMinutes = timeRangeMinutes(liveSeriesRes.timeRange) || periodToMinutes(period);
const baselineMinutes =
timeRangeMinutes(baselineScalarRes.timeRange) || periodToMinutes(BASELINE_PERIOD);
// Prefer measured queue metrics; fall back to the runs snapshot when unavailable.
const flow =
(await QueueMetricsSource.loadFlow(env, period, ctx, deps)) ??
(await SnapshotFlowSource.loadFlow(env, period, ctx, deps))!;
// A measured source that failed still falls back for the remaining shape, but its depth is marked
// unknown so a failure is never presented as a measurement.
const measured = await QueueMetricsSource.loadFlow(env, period, ctx, deps);
let flow: FlowData;
if (measured.status === "ok") {
flow = measured.data;
} else {
const snapshot = await SnapshotFlowSource.loadFlow(env, period, ctx, deps);
flow = (snapshot as { status: "ok"; data: FlowData }).data;
if (measured.status === "failed") {
flow = { ...flow, pending: { ...flow.pending, availability: "unknown" } };
}
}
const failuresSeries = resampleSeries(
ctx.liveSeries.map((r) => failureRate(num(r.failures), num(r.completed)))
@@ -557,8 +587,12 @@ export async function loadHealthInput(
const triggered = num(ctx.liveScalar.triggered);
const completed = num(ctx.liveScalar.completed);
const donePerMin = windowMinutes === 0 ? 0 : completed / windowMinutes;
const triggeredPerMin = windowMinutes === 0 ? 0 : triggered / windowMinutes;
// Older rows have no `finished`, so fall back to completions rather than a fabricated 0 drain.
const finished = num(ctx.liveScalar.finished, completed);
const perMin = (total: number) => (windowMinutes === 0 ? 0 : total / windowMinutes);
const finishedPerMin = perMin(finished);
const completedPerMin = perMin(completed);
const triggeredPerMin = perMin(triggered);
const normalTriggeredPerMin =
baselineMinutes === 0 ? 0 : num(ctx.baselineScalar.triggered) / baselineMinutes;
@@ -568,9 +602,7 @@ export async function loadHealthInput(
num(ctx.baselineScalar.completed)
);
// Lazy failure attribution — only when execution is actually degraded.
// A fresh failure pattern from a clean 0% baseline is exactly when attribution matters most,
// so a zero baseline (can't form a ratio) counts as degraded once past the floor.
// A zero baseline can't form a ratio, so it counts as degraded once past the floor.
const failureDegraded =
rate >= HEALTH_THRESHOLDS.failures.floorRate &&
(normalRate === 0 || rate / normalRate >= HEALTH_THRESHOLDS.failures.warnMult);
@@ -578,8 +610,7 @@ export async function loadHealthInput(
? await loadFailureBreakdown(deps, env, period, num(ctx.liveScalar.failures))
: undefined;
// Liveness = telemetry freshness (how recent is the newest data), NOT "recent completions":
// a quiet env with a fresh pipeline is fresh; a dead pipeline is stale. null -> unknown.
// Liveness measures telemetry freshness, not recent completions.
const telemetryAgeMs =
flow.telemetryLastTs === null ? null : Math.max(0, now.getTime() - flow.telemetryLastTs);
@@ -592,7 +623,7 @@ export async function loadHealthInput(
flowSource: flow.flowSource,
pending: flow.pending,
startLatency: flow.startLatency,
throughput: { donePerMin, triggeredPerMin, normalTriggeredPerMin },
throughput: { finishedPerMin, completedPerMin, triggeredPerMin, normalTriggeredPerMin },
failures: { rate, normalRate, series: failuresSeries },
duration: {
p95Ms: num(ctx.liveScalar.dur_p95),
@@ -611,16 +642,12 @@ async function loadFailureBreakdown(
totalFails: number
): Promise<HealthInput["failureBreakdown"]> {
if (totalFails <= 0) return undefined;
const rows = await tryQuery(deps, env, failureBreakdownQuery(), period);
const { rows } = await tryQuery(deps, env, failureBreakdownQuery(), period);
if (rows.length === 0) return undefined;
const top = rows[0];
return { task: String(top.task ?? "unknown"), share: num(top.fails) / totalFails };
}
// ---------------------------------------------------------------------------
// Small local helpers.
// ---------------------------------------------------------------------------
function parseTimestamp(value: unknown): number | null {
if (value === null || value === undefined || value === "") return null;
// ClickHouse returns a "1970-01-01 00:00:00" sentinel for max() over no rows.
@@ -1,13 +1,3 @@
/**
* The `health` report's PROSE — the single place its codes resolve to strings. Registered
* under the "health" title so the generic renderer can look it up by `vm.title` without ever
* importing health vocabulary. A future report (Cost, Regression) ships its own catalog the
* same way; `report-messages.ts` stays report-agnostic infrastructure.
*
* Some strings carry {tokens} (e.g. {age}, {rate}) the renderer fills from the finding's
* metrics / evidence — meaning lives here, numbers stay facts.
*/
import { type ReportMessages } from "../report-messages";
import { type ReasonCode, type Severity } from "../report-view-model";
@@ -24,24 +14,17 @@ const METRIC_LABELS: Record<string, string> = {
triggered: "triggered",
};
/**
* Finding headline — keyed by `${findingType}/${reason}`. Degraded = the cause,
* healthy = reassurance. `@expanded` variants show a healthy finding expanded
* (e.g. execution while flow is degraded).
*/
/** Keyed by `${findingType}/${reason}`, with optional `@expanded` variants. */
const FINDING_REASONS: Record<string, string> = {
// flow — causes
"flow/env_limit_saturation": "at your env concurrency limit",
"flow/dequeue_stall": "capacity is free but nothing is dequeuing",
"flow/queue_limit_throttling": "a queue is throttling at its own limit",
"flow/trigger_spike": "a trigger spike is backing up the queue",
"flow/trigger_surge": "a surge of new triggers is backing up the queue",
// flow — fallback symptoms (v1)
"flow/start_latency": "runs are slow to start",
"flow/backlog": "backlog is growing",
"flow/throughput_lag": "completion is falling behind triggers",
"flow/degraded": "flow is degraded",
// flow — healthy (collapsed)
"flow/healthy": "starting normally",
// execution
"execution/failures_up": "runs are failing more than usual",
@@ -49,9 +32,10 @@ const FINDING_REASONS: Record<string, string> = {
"execution/degraded": "execution is degraded",
"execution/unknown": "execution can't be assessed — the telemetry is stale",
"flow/unknown": "flow can't be assessed — the telemetry is stale",
"flow/flow_unmeasured": "flow can't be assessed — the queue depth couldn't be measured",
"execution/healthy": "completing normally", // collapsed
"execution/healthy@expanded": "the runs that DO start are fine",
// liveness = telemetry freshness ({age} filled by the renderer)
"execution/healthy@expanded": "runs are executing normally",
// liveness is telemetry freshness
"liveness/fresh": "fresh — telemetry current, updated {age} ago",
"liveness/lagging": "lagging — telemetry last updated {age} ago",
"liveness/stale": "stale — no telemetry in {age}",
@@ -66,7 +50,6 @@ const READS: Record<string, string> = {
queue_throttle_chain: "queue at its limit → its runs wait → backlog grows",
spike_chain: "triggers jumped {mult}× → queue fills faster than it drains",
surge_chain: "new triggers arriving with no prior baseline → queue fills faster than it drains",
// fallback symptoms (v1)
starting_normally: "runs are starting on time",
lag_while_triggering_normal: "triggering normally, but starts lag → work is backing up",
lag_and_failures: "runs are lagging AND failing — check the code path",
@@ -75,32 +58,33 @@ const READS: Record<string, string> = {
runs_are_fine: "runs are completing normally",
failures_elevated: "failures are elevated — check the code path",
data_stale: "data is stale — the verdict cannot be trusted",
flow_unmeasured: "the queue depth is unavailable — the backlog cannot be assessed",
};
/** Exclusion (ruled-out cause) — rendered under `read:`. {tokens} filled from evidence. */
/** A ruled-out cause, rendered under `read:`. {tokens} are filled from evidence. */
const EXCLUSIONS: Record<string, string> = {
not_env_limit: "env concurrency limit is not the bottleneck",
not_your_code: "not your code — failures and durations normal",
not_your_config: "not your config — limits aren't the bottleneck",
};
/** Observation (supporting fact, not a ruled-out cause) — rendered under `read:` after exclusions. */
/** A supporting fact rather than a ruled-out cause, rendered after the exclusions. */
const OBSERVATIONS: Record<string, string> = {
not_workers_platform: "runs are completing at ~{rate}/min",
not_workers_platform: "runs are finishing at ~{rate}/min",
execution_healthy: "runs that start are completing normally",
nothing_dead_lettered: "nothing dead-lettered",
};
/** Metric annotation shown on a cause line INSTEAD of "(normal ~x)". {tokens} filled by the renderer. */
/** Metric annotation shown on a cause line in place of the normal baseline. */
const ANNOTATIONS: Record<string, string> = {
pinned_minutes: "pinned {value} of last {window} min",
pinned_minutes: "{value} min at limit",
idle_share: "idle — {value} running of {limit}",
throttled_minutes: "throttled {value} of last {window} min",
spike_mult: "{value}× the normal rate",
surge_rate: "{value}/min, no prior baseline",
};
/** Headline statement keyed by `${findingType}/${severity}`. */
/** Headline statement, keyed by `${findingType}/${severity}`. */
const STATEMENTS: Record<string, string> = {
"flow/ok": "Flow healthy",
"flow/warn": "Flow slowing",
@@ -113,9 +97,8 @@ const STATEMENTS: Record<string, string> = {
"liveness/crit": "data stale",
};
/** Recommendation / footer codes -> calm, jargon-free action text. */
/** Recommendation and footer codes resolved to action text. */
const ACTIONS: Record<string, string> = {
// Review = open concrete data · Check = system state · Raise = a settings change
review_start_latency: "Review start latency",
review_failing_tasks: "Review failing tasks",
review_slow_runs: "Review slow runs",
@@ -125,6 +108,8 @@ const ACTIONS: Record<string, string> = {
check_control_plane: "Check control plane",
check_platform_status: "Check status.trigger.dev — no action needed on yours",
raise_env_limit: "Raise the env concurrency limit",
contact_us_raise_limit: "Contact us to raise the limit",
concurrency_docs: "Read concurrency docs",
raise_queue_limit: "Raise the queue's concurrency limit",
do_nothing_drains: "or do nothing — backlog drains in ~{value} min once triggers ease",
region_failover: "region move? ask your agent — depends on your failover setup",
@@ -144,12 +129,16 @@ function findingReason(
}
function statementMessage(findingType: string, severity: Severity, reason?: ReasonCode): string {
// Stale telemetry makes a CH-derived verdict untrustworthy — say so, don't show a severity.
// Stale telemetry makes the verdict untrustworthy, so it replaces the severity.
if (reason === "unknown") {
const label = findingType.charAt(0).toUpperCase() + findingType.slice(1);
return `${label} unknown — data stale`;
}
// No freshness signal is NOT "data lagging" (a real severity) — it's genuinely unknown.
// A missing depth signal is unknown from a failed measurement, not staleness.
if (reason === "flow_unmeasured") {
return "Flow unknown — queue depth unavailable";
}
// No freshness signal is unknown, not lagging.
if (findingType === "liveness" && reason === "freshness_unknown") {
return "data freshness unknown";
}
@@ -1,15 +1,3 @@
/**
* The `health` report — ORCHESTRATOR + public entry. Two layers:
* `assessHealth()` data -> HealthAssessment (all health reasoning)
* `toReportViewModel()` HealthAssessment -> generic ReportViewModel (pure packaging)
* `interpret()` is the thin composition of the two.
*
* The reasoning is split into three independent analyzers, each returning a Finding:
* flow.ts · execution.ts · liveness.ts (foundation in health-core.ts)
* This module only wires them: build metrics -> run the three -> flow policy -> stale guard
* -> reads -> summary/footer. PURE — no IO/clock/LLM/formatting.
*/
import {
isOk,
maxSeverity,
@@ -20,31 +8,28 @@ import {
type Severity,
type SummaryStatement,
} from "../report-view-model";
import { buildMetrics, computeDrain, HEALTH_THRESHOLDS, type HealthInput } from "./health-core";
import {
buildMetrics,
computeDrain,
HEALTH_THRESHOLDS,
isPendingUnknown,
type HealthInput,
} from "./health-core";
import { buildExecutionRead, interpretExecution } from "./execution";
import { applyFlowPolicy, buildFlowRead, interpretFlow } from "./flow";
import { applyFlowPolicy, buildFlowRead, FLOW_UNMEASURED, interpretFlow } from "./flow";
import { interpretLiveness } from "./liveness";
// Registers the "health" message catalog (side effect) so the renderer resolves this report's
// codes. Kept here — the health report's entry module — so loading it always registers its prose.
// Re-exported so the data layer + tests keep a single import path (`./health`).
export { HEALTH_THRESHOLDS, isPendingIncreasing, type HealthInput } from "./health-core";
// ---------------------------------------------------------------------------
// Stale-telemetry trust guard. When telemetry is genuinely stale, both flow and execution are
// CH-derived and untrustworthy: mark them unknown and strip everything that would advise action
// off stale data (recommendation, attribution, exclusions, observations, hedge, anomaly window).
// ---------------------------------------------------------------------------
// Flow and execution are both ClickHouse-derived, so stale telemetry makes both unknown and strips
// anything that would advise action.
function applyStaleGuard(
flow: Finding,
execution: Finding,
telemetryStale: boolean
): { flow: Finding; execution: Finding; treatedCrit: boolean } {
if (!telemetryStale) return { flow, execution, treatedCrit: false };
// Force crit so severity is consistent across summary / section glyph / JSON, and strip the
// ACTIONABLE causal fields so no surface advises off stale data. Raw metrics/evidence stay in
// the VM for diagnostics, flagged informational-only by `facts.trustworthy: false`.
// Force crit so severity is consistent across summary, glyph and JSON.
const untrust = (f: Finding): Finding => ({
...f,
severity: "crit",
@@ -59,10 +44,6 @@ function applyStaleGuard(
return { flow: untrust(flow), execution: untrust(execution), treatedCrit: true };
}
// ---------------------------------------------------------------------------
// Summary.
// ---------------------------------------------------------------------------
function aggregateSummary(
flow: Finding,
execution: Finding,
@@ -75,7 +56,9 @@ function aggregateSummary(
{
findingType: "flow",
severity: flow.severity,
reason: flow.reason === "unknown" ? "unknown" : undefined,
// Both exceptions mean "we can't say", so the statement renders no severity claim.
reason:
flow.reason === "unknown" || flow.reason === FLOW_UNMEASURED ? flow.reason : undefined,
},
{
findingType: "execution",
@@ -91,10 +74,6 @@ function aggregateSummary(
return { severity, statements };
}
// ---------------------------------------------------------------------------
// Footer (dominant action + do-nothing option) + links.
// ---------------------------------------------------------------------------
const SEV_RANK: Record<Severity, number> = { ok: 0, warn: 1, crit: 2 };
function dominantFinding(findings: Finding[]): Finding | undefined {
@@ -122,12 +101,14 @@ function buildFooter(
const dominant = dominantFinding(findings);
if (!dominant?.recommendation) return [{ code: "nothing_to_do" }];
const footer: FooterEntry[] = [
{ code: dominant.recommendation.code, link: dominant.recommendation.link },
];
const footer: FooterEntry[] =
dominant.recommendation.code === "raise_env_limit"
? [
{ code: "raise_env_limit", link: dominant.recommendation.link },
{ code: "concurrency_docs", link: dominant.recommendation.link },
]
: [{ code: dominant.recommendation.code, link: dominant.recommendation.link }];
// Second entry: do-nothing when the backlog drains, or the region-move hedge for a
// dequeue stall (the one place it stays plausible).
if (dominant.type === "flow") {
if (drain.isDrainable && Number.isFinite(drain.drainMinutes)) {
footer.push({ code: "do_nothing_drains", value: Math.round(drain.drainMinutes * 10) / 10 });
@@ -147,35 +128,25 @@ function collectLinks(findings: Finding[]): ReportViewModel["links"] {
return [...keys].map((key) => ({ key, label: key, url: "" }));
}
// ---------------------------------------------------------------------------
// Domain layer: HealthAssessment — the health verdict (flow / execution / liveness findings,
// their causes + recommendations, and the derived state). All health semantics live here; it
// knows nothing about how a report is presented. `toReportViewModel` maps it into the generic,
// report-agnostic ReportViewModel — so the VM stays reusable for future reports (each has its
// own <domain>Assessment + mapper; the renderer/VM primitives are shared).
// ---------------------------------------------------------------------------
// The health verdict. All health semantics live here and no presentation does.
export type HealthAssessment = {
/** header, carried through from the input. */
scope: string;
period: string;
baselineLabel: string;
generatedAt: string;
windowMinutes: number;
/** finalized findings (post-policy, post-stale-guard, with reads built). */
/** Finalized findings: post-policy, post-stale-guard, with reads built. */
flow: Finding;
execution: Finding;
liveness: Finding;
metrics: Metric[];
/** derived domain state the presentation layer needs (footer / summary / trust). */
drain: { drainMinutes: number; isDrainable: boolean };
telemetryStale: boolean;
executionTreatedCrit: boolean;
/** structured payload for agents (already carries the trust marker). */
/** Structured payload for agents. Carries `trustworthy`. */
facts: Record<string, unknown>;
};
/** Data -> domain verdict. Pure: no IO/clock/LLM/formatting — just health reasoning. */
export function assessHealth(input: HealthInput): HealthAssessment {
const metrics = buildMetrics(input);
const drain = computeDrain(input);
@@ -184,23 +155,28 @@ export function assessHealth(input: HealthInput): HealthAssessment {
const executionRaw = interpretExecution(metrics, input);
const liveness = interpretLiveness(metrics, input);
// Telemetry freshness as an explicit state so "unknown" (no signal) is never conflated with
// "lagging" (a real severity). Only GENUINE staleness trust-guards the CH-derived verdicts.
// "none" is not "lagging", and only genuine staleness trust-guards the verdicts. A signal-less env
// stays neutral for the reader but reports `trustworthy: false`.
const ageMs = input.liveness.telemetryAgeMs;
const telemetryStale = ageMs !== null && ageMs > HEALTH_THRESHOLDS.liveness.staleMs;
const telemetry: "none" | "fresh" | "lagging" | "stale" =
ageMs === null
? "none"
: ageMs > HEALTH_THRESHOLDS.liveness.staleMs
? "stale"
: ageMs > HEALTH_THRESHOLDS.liveness.freshMs
? "lagging"
: "fresh";
const telemetryStale = telemetry === "stale";
const flowUnmeasured = isPendingUnknown(input);
flow = applyFlowPolicy(flow, executionRaw, drain.isDrainable, telemetryStale);
// Stale telemetry: flow AND execution are untrustworthy -> mark both unknown and strip their
// actions/attribution/exclusions so nothing advises off stale data.
const guarded = applyStaleGuard(flow, executionRaw, telemetryStale);
flow = guarded.flow;
const execution = guarded.execution;
// interpretFlow mutates the shared metrics array (e.g. sets concurrency.annotation "pinned 40
// of last 60 min") BEFORE the guard runs. The renderers hide it for an unknown finding, but
// format=json would still leak that stale-derived narrative — so strip metric annotations too
// (the twin of the stripped anomaly window). Raw values stay, flagged by facts.trustworthy.
// interpretFlow annotates the shared metrics array before the guard runs, and format=json would
// leak that stale-derived narrative.
if (telemetryStale) {
for (const m of metrics) m.annotation = undefined;
}
@@ -223,12 +199,16 @@ export function assessHealth(input: HealthInput): HealthAssessment {
telemetryStale,
executionTreatedCrit: guarded.treatedCrit,
facts: {
// Trust marker for structured consumers. The metrics/evidence stay (useful for pipeline
// diagnostics), but when telemetry is stale they're informational-only: an agent must not
// act on them (e.g. raise concurrency off a stale backlog). The human renderer is already
// guarded via the "unknown" finding; this is the same guarantee for JSON.
trustworthy: !telemetryStale,
staleReason: telemetryStale ? "telemetry_stale" : undefined,
// Metrics are informational unless this is true. Stale, absent and unmeasurable all read false.
trustworthy: !telemetryStale && telemetry !== "none" && !flowUnmeasured,
telemetry,
untrustworthyReason: telemetryStale
? "telemetry_stale"
: telemetry === "none"
? "telemetry_absent"
: flowUnmeasured
? "flow_unmeasured"
: undefined,
flowSource: input.flowSource,
pendingEstimated: input.pending.estimated,
throughput: input.throughput,
@@ -237,11 +217,6 @@ export function assessHealth(input: HealthInput): HealthAssessment {
};
}
// ---------------------------------------------------------------------------
// Presentation mapping: HealthAssessment -> generic ReportViewModel. Pure packaging only —
// no health reasoning here (summary/footer/links are derived from the findings).
// ---------------------------------------------------------------------------
function toReportViewModel(a: HealthAssessment): ReportViewModel {
const findings = [a.flow, a.execution, a.liveness];
return {
@@ -256,12 +231,10 @@ function toReportViewModel(a: HealthAssessment): ReportViewModel {
metrics: a.metrics,
facts: a.facts,
links: collectLinks(findings),
// Stale telemetry -> footer points at the control plane, not a CH-derived action.
footer: buildFooter(findings, a.drain, a.telemetryStale),
};
}
/** Public entry: data -> generic report. Thin composition of the domain + presentation layers. */
export function interpret(input: HealthInput): ReportViewModel {
return toReportViewModel(assessHealth(input));
}
@@ -1,7 +1,4 @@
/**
* LIVENESS analyzer: telemetry FRESHNESS (age of the freshest signal), NOT "last completion".
* A null age is genuinely unknown (neutral), never a warning; only real staleness is crit.
*/
/** Liveness analyzer: freshest-telemetry age. A null age is unknown, never a warning. */
import { type Finding, type Metric } from "../report-view-model";
import { metricById, type HealthInput } from "./health-core";
@@ -1,77 +1,62 @@
/**
* GENERIC renderer: ReportViewModel -> monospace markdown. Severity-driven disclosure:
* a degraded finding expands into evidence in causal order; a healthy finding collapses
* to one `✓` line. Owns ALL presentation (formatting, glyphs, sparklines, spacing, {token}
* substitution) and resolves the VM's codes -> strings via the report's registered message
* catalog, looked up by `vm.title` — so it holds NO report vocabulary itself.
* The report's text surfaces: monospace markdown and ANSI.
*
* Knows NOTHING about health — walks summary -> findings -> metrics generically.
* Structure, labels, wording and glyphs come from `report-layout.ts`, which the React card also
* consumes, so the surfaces can't drift. This module owns typography only: column alignment, the
* sparkline column, indentation, and (for ANSI) colour.
*
* Nothing here relies on colour to carry meaning: every verdict, direction and caveat is a glyph,
* because an MCP host renders plain text.
*/
import { reportMessages, type ReportMessages } from "./report-messages";
import {
type Finding,
type FooterEntry,
type Metric,
type ReportViewModel,
type Severity,
type Unit,
} from "./report-view-model";
buildReportLayout,
REPORT_GLYPH,
REPORT_LABELS,
type LayoutFinding,
type LayoutMetricRow,
type LayoutViewModel,
type ReportLayout,
} from "./report-layout";
import { reportMessages } from "./report-messages";
import { type ReportViewModel } from "./report-view-model";
const BARS = "▁▂▃▄▅▆▇█";
const MINUS = ""; // U+2212
const SEVERITY_GLYPH: Record<Severity, string> = { ok: "✓", warn: "⚠", crit: "✕" };
/**
* A NEUTRAL marker for a state that's genuinely unknown, not good/bad — e.g. liveness with no
* telemetry signal. It doesn't affect the aggregate severity (that stays driven by real findings),
* but it must not read as a confident green "✓", so it gets its own glyph.
*/
const NEUTRAL_GLYPH = "○";
/**
* Markdown-only status colour. Chat hosts render neither ANSI nor HTML, so swapping
* the glyphs for traffic-light circles is the one colour cue they get — one emoji per
* marker (neutral -> white). ANSI keeps the crisp ✓/⚠/✕/○, so this applies ONLY on markdown.
* Markdown-only status colour. Chat hosts render neither ANSI nor HTML, so swapping the glyphs for
* traffic-light circles is the one colour cue they get. The glyph and the emoji mean the same
* thing, so meaning never depends on which one a host shows.
*/
const MARKDOWN_STATUS_EMOJI: Record<string, string> = {
"✓": "🟢",
"⚠": "🟡",
"✕": "🔴",
"○": "⚪",
[REPORT_GLYPH.ok]: "🟢",
[REPORT_GLYPH.warn]: "🟡",
[REPORT_GLYPH.crit]: "🔴",
[REPORT_GLYPH.neutral]: "⚪",
[REPORT_GLYPH.untrusted]: "🚩",
};
const STATUS_GLYPHS = new RegExp(`[${Object.keys(MARKDOWN_STATUS_EMOJI).join("")}]`, "g");
function toMarkdownEmoji(text: string): string {
return text.replace(/[✓⚠✕○]/g, (g) => MARKDOWN_STATUS_EMOJI[g] ?? g);
return text.replace(STATUS_GLYPHS, (glyph) => MARKDOWN_STATUS_EMOJI[glyph] ?? glyph);
}
/** Glyph for a finding/statement: neutral for a genuinely-unknown freshness, else severity-driven. */
function statusGlyph(severity: Severity, reason?: string): string {
return reason === "freshness_unknown" ? NEUTRAL_GLYPH : SEVERITY_GLYPH[severity];
}
/** Evidence lines (metric rows + attribution) shown for a degraded section. */
/** Metric rows shown for a finding's evidence block. */
const EVIDENCE_CAP = 4;
/** Column where the header's scope·period·baseline starts (mirrors the mockup). */
/** Column where the header's scope, period and baseline start. */
const HEADER_COL = 22;
/** Gap between aligned columns in an evidence block. */
/** Gap between aligned columns. */
const COL_GAP = 3;
/** Section labels pad to this so their glyph/content aligns vertically. */
/** Section labels pad to this so every finding's text starts at the same column. */
const SECTION_LABEL_WIDTH = 9; // "EXECUTION"
const SECTION_GAP = 3;
/** Section label padded so every section's ✓ / cause text starts at the same column. */
function sectionLabel(type: string): string {
return `${type.toUpperCase().padEnd(SECTION_LABEL_WIDTH)}${" ".repeat(SECTION_GAP)}`;
}
// ---------------------------------------------------------------------------
// Formatters.
// ---------------------------------------------------------------------------
/** `read:` / `why:` labels sit in their own column so their lines hang as one paragraph. */
const NOTE_LABEL_WIDTH = Math.max(REPORT_LABELS.read.length, REPORT_LABELS.why.length);
/** All sparklines render at this fixed width so they align in a column. */
const SPARK_WIDTH = 8;
@@ -105,361 +90,142 @@ export function sparklineFromSeries(points: number[]): string {
return p.map((v) => BARS[Math.round(((v - min) / (max - min)) * (BARS.length - 1))]).join("");
}
function fmtCount(n: number): string {
return Math.round(n).toLocaleString("en-US");
/** A metric's trailing aside. An annotation speaks for itself; the rest is context, so it brackets. */
function trailingNote(row: LayoutMetricRow): string {
if (!row.note) return "";
return row.note.kind === "annotation" ? row.note.text : `(${row.note.text})`;
}
function fmtDuration(ms: number): string {
if (ms < 1000) return `${Math.round(ms)}ms`;
const s = ms / 1000;
if (s < 60) return Number.isInteger(s) ? `${s}s` : `${s.toFixed(1)}s`;
const m = s / 60;
return Number.isInteger(m) ? `${m}m` : `${m.toFixed(1)}m`;
}
function fmtPct(ratio: number): string {
return `${(ratio * 100).toFixed(1)}%`;
}
function fmtRate(n: number): string {
return `${fmtCount(n)}/min`;
}
function fmtSignedRate(net: number): string {
const sign = net < 0 ? MINUS : net > 0 ? "+" : "";
return `${sign}${fmtCount(Math.abs(net))}/min`;
}
function fmtValue(value: number, unit: Unit): string {
switch (unit) {
case "ms":
return fmtDuration(value);
case "count":
return fmtCount(value);
case "ratio":
return fmtPct(value);
case "perMin":
return fmtRate(value);
}
}
function fill(template: string, tokens: Record<string, string | number | undefined>): string {
return template.replace(/\{(\w+)\}/g, (_, k) => {
const v = tokens[k];
return v === undefined ? `{${k}}` : String(v);
});
}
function metricById(metrics: Metric[], id: string): Metric | undefined {
return metrics.find((m) => m.id === id);
}
// ---------------------------------------------------------------------------
// Metric line (expanded evidence).
// ---------------------------------------------------------------------------
function deltaSegment(metric: Metric): string {
if (metric.severity === "ok" || !metric.delta || metric.delta.dir === "flat") return "";
const arrow = metric.delta.dir === "up" ? "↑" : "↓";
// "up" shows the multiplier when we have it ("↑ 16×"). "down" is arrow-only: a
// drop rounds to 0×/1×, meaningless — the arrow already says "below normal".
if (metric.delta.dir === "down") return arrow;
return metric.delta.mult === undefined ? arrow : `${arrow} ${metric.delta.mult}×`;
}
function annotationSegment(metric: Metric, vm: ReportViewModel): string {
if (!metric.annotation) return "";
const value = String(metric.annotation.value ?? "");
return fill(reportMessages(vm.title).annotationMessage(metric.annotation.code), {
value,
window: vm.windowMinutes,
limit: metric.breakdown?.limit ?? "",
});
}
function metricValueText(metric: Metric, msg: ReportMessages): string {
// No measurement -> say so; `value` is a placeholder, not a real reading.
if (metric.availability === "unknown") return "unknown";
// concurrency etc. carry a limit -> "running/limit".
if (metric.unit === "count" && metric.breakdown?.limit !== undefined) {
return `${fmtCount(metric.value)}/${fmtCount(metric.breakdown.limit)}`;
}
// composite throughput -> "done vs triggered -> net".
if (metric.unit === "perMin" && metric.breakdown?.done !== undefined) {
const { done, triggered } = metric.breakdown;
return `${fmtRate(done)} done vs ${fmtRate(triggered)} triggered → net ${fmtSignedRate(metric.value)}`;
}
const showAgg =
metric.aggregation === "p95" && !msg.metricLabel(metric.id).toLowerCase().startsWith("p95");
return showAgg
? `p95 ${fmtValue(metric.value, metric.unit)}`
: fmtValue(metric.value, metric.unit);
}
/** Column widths for a section's evidence block, so value/delta/spark align down the page. */
/** Column widths for an evidence block, so value, delta and sparkline align down the page. */
type Cols = { label: number; value: number; delta: number };
function isComposite(metric: Metric): boolean {
return metric.unit === "perMin" && metric.breakdown?.done !== undefined;
}
function computeColumns(rows: Metric[], vm: ReportViewModel, extraLabels: string[]): Cols {
const msg = reportMessages(vm.title);
const labelLens = [
...rows.map((m) => msg.metricLabel(m.id).length),
...extraLabels.map((l) => l.length),
];
const valueLens = rows.filter((m) => !isComposite(m)).map((m) => metricValueText(m, msg).length);
const deltaLens = rows
.filter((m) => !isComposite(m) && !annotationSegment(m, vm))
.map((m) => deltaSegment(m).length);
function computeColumns(rows: LayoutMetricRow[]): Cols {
const labels = rows.flatMap((row) => [
row.label.length,
...row.subRows.map((sub) => sub.label.length + SUB_ROW_INDENT),
]);
const values = rows.flatMap((row) => [
row.value.length,
...row.subRows.map((s) => s.value.length),
]);
return {
label: Math.max(0, ...labelLens),
value: Math.max(0, ...valueLens),
delta: Math.max(0, ...deltaLens),
label: Math.max(0, ...labels),
value: Math.max(0, ...values),
delta: Math.max(0, ...rows.map((row) => (row.delta?.text ?? "").length)),
};
}
function renderMetricRow(metric: Metric, cols: Cols, vm: ReportViewModel): string {
const msg = reportMessages(vm.title);
/** A composite metric's parts sit under it, still on the shared value column. */
const SUB_ROW_INDENT = 2;
function renderMetricRow(row: LayoutMetricRow, cols: Cols, indent: string): string[] {
const gap = " ".repeat(COL_GAP);
const label = msg.metricLabel(metric.id).padEnd(cols.label);
const value = metricValueText(metric, msg);
const spark = row.series ? sparklineFromSeries(row.series) : "";
const note = trailingNote(row);
// composite throughput: label + value only (its own grammar).
if (isComposite(metric)) return ` ${label}${gap}${value}`;
// The fixed-width spark column keeps every sparkline and every trailing note aligned.
let line = `${indent}${row.label.padEnd(cols.label)}${gap}${row.value.padEnd(cols.value)}`;
if (cols.delta > 0) line += `${gap}${(row.delta?.text ?? "").padEnd(cols.delta)}`;
line += `${gap}${spark.padEnd(SPARK_WIDTH)}`;
if (note) line += `${gap}${note}`;
const spark =
metric.series && metric.series.points.length > 0
? sparklineFromSeries(metric.series.points)
: "";
const annotation = annotationSegment(metric, vm);
const delta = annotation ? "" : deltaSegment(metric); // cause line carries an annotation, not a delta
const trailing = annotation
? annotation
: metric.normal !== undefined
? `(normal ~${fmtValue(metric.normal, metric.unit)})`
: metric.series?.kind === "estimated"
? "(estimated)" // proxy trend (e.g. snapshot backlog) — flag it so it isn't read as measured
: "";
// Fixed columns: label · value · delta · SPARK · trailing. The fixed-width spark
// column keeps every spark and the trailing (normal / annotation) aligned.
let line = ` ${label}${gap}${value.padEnd(cols.value)}`;
if (cols.delta > 0) line += `${gap}${delta.padEnd(cols.delta)}`;
line += `${gap}${(spark || "").padEnd(SPARK_WIDTH)}`;
if (trailing) line += `${gap}${trailing}`;
return line.replace(/\s+$/, "");
}
// ---------------------------------------------------------------------------
// Compact facts (collapsed / semi-expanded healthy sections).
// ---------------------------------------------------------------------------
function compactFact(metric: Metric): string | undefined {
switch (metric.id) {
case "pending":
return `pending ${fmtCount(metric.value)}${metric.normal !== undefined ? ` (normal ~${fmtCount(metric.normal)})` : ""}`;
case "start_latency_p95":
return metric.availability === "unknown"
? "starts p95 unknown"
: `starts p95 ${fmtDuration(metric.value)}`;
case "failures":
return `failures ${fmtPct(metric.value)}${metric.normal !== undefined ? ` (normal ~${fmtPct(metric.normal)})` : ""}`;
case "dur_p95":
return metric.severity === "ok"
? "durations normal"
: `durations p95 ${fmtDuration(metric.value)}`;
default:
return undefined; // throughput / evidence metrics get no collapsed fact
}
}
/** Reassuring facts read consequence-first: depth/failures before latency/duration. */
const COMPACT_ORDER = ["pending", "failures", "start_latency_p95", "dur_p95"];
function compactFacts(finding: Finding, metrics: Metric[]): string {
return finding.metricIds
.map((id) => metricById(metrics, id))
.filter((m): m is Metric => m !== undefined)
.slice()
.sort((a, b) => COMPACT_ORDER.indexOf(a.id) - COMPACT_ORDER.indexOf(b.id))
.map(compactFact)
.filter((s): s is string => s !== undefined)
.join(" · ");
}
// ---------------------------------------------------------------------------
// Read / exclusion / attribution / window.
// ---------------------------------------------------------------------------
function readTokens(_finding: Finding, metrics: Metric[]): Record<string, string | number> {
const triggered = metricById(metrics, "triggered");
return {
mult: triggered?.delta?.mult ?? "",
};
}
function windowSuffix(finding: Finding): string {
const aw = finding.anomalyWindow;
if (!aw) return "";
return aw.touchesEnd ? ` (last ${aw.minutes} min)` : ` (${aw.minutes} min window)`;
}
function attributionLine(finding: Finding, cols: Cols): string | undefined {
const a = finding.attribution;
if (!a) return undefined;
const label = `worst ${a.dim}`.padEnd(cols.label);
return ` ${label}${" ".repeat(COL_GAP)}${a.key}${Math.round(a.share * 100)}% of ${a.of}`;
}
// ---------------------------------------------------------------------------
// Sections.
// ---------------------------------------------------------------------------
function renderDegradedSection(finding: Finding, vm: ReportViewModel): string[] {
const msg = reportMessages(vm.title);
const rows = finding.metricIds
.map((id) => metricById(vm.metrics, id))
.filter((m): m is Metric => m !== undefined);
const extraLabels = finding.attribution ? [`worst ${finding.attribution.dim}`] : [];
const cols = computeColumns(rows, vm, extraLabels);
const evidence: string[] = rows.map((m) => renderMetricRow(m, cols, vm));
const attr = attributionLine(finding, cols);
if (attr) evidence.push(attr);
// One blank line between evidence rows so the block breathes.
const spaced = evidence.slice(0, EVIDENCE_CAP).flatMap((l, i) => (i === 0 ? [l] : ["", l]));
const lines = [
// Lead with the glyph so the cause text lines up with the healthy sections' "✓ …".
`${sectionLabel(finding.type)}${SEVERITY_GLYPH[finding.severity]} ${msg.findingReason(finding.type, finding.reason)}${windowSuffix(finding)}`,
"", // blank line between the header and its evidence (matches the section/footer spacing)
...spaced,
return [
line.replace(/\s+$/, ""),
...row.subRows.map((sub) => {
const label = `${" ".repeat(SUB_ROW_INDENT)}${sub.label}`.padEnd(cols.label);
return `${indent}${label}${gap}${sub.value}`;
}),
];
if (finding.read) {
lines.push(
"",
` read: ${fill(msg.readMessage(finding.read), readTokens(finding, vm.metrics))}`
);
// Exclusions ("not your code") first, then supporting observations ("runs completing at ~X/min").
for (const excl of finding.exclusions ?? []) {
lines.push(
` ${fill(msg.exclusionMessage(excl.code), { rate: fmtCount(excl.evidence?.donePerMin ?? 0) })}`
);
}
for (const obs of finding.observations ?? []) {
lines.push(
` ${fill(msg.observationMessage(obs.code), { rate: fmtCount(obs.evidence?.donePerMin ?? 0) })}`
);
}
}
return lines;
}
function renderHealthyExecutionExpanded(finding: Finding, vm: ReportViewModel): string[] {
const msg = reportMessages(vm.title);
const lines = [
`${sectionLabel(finding.type)}${SEVERITY_GLYPH.ok} ${msg.findingReason(finding.type, finding.reason, { expanded: true })}`,
"", // blank line between the header and its facts (matches the section/footer spacing)
` ${compactFacts(finding, vm.metrics)}`,
];
if (finding.read) lines.push(` read: ${msg.readMessage(finding.read)}`);
return lines;
/** A labelled block whose lines hang under the label's column. */
function renderNoteBlock(label: string, lines: string[], indent: string): string[] {
if (lines.length === 0) return [];
const hang = " ".repeat(indent.length + NOTE_LABEL_WIDTH + 1);
return lines.map((line, i) =>
i === 0 ? `${indent}${label.padEnd(NOTE_LABEL_WIDTH)} ${line}` : `${hang}${line}`
);
}
function renderCollapsedSection(finding: Finding, vm: ReportViewModel): string[] {
const msg = reportMessages(vm.title);
const headline = `${sectionLabel(finding.type)}${SEVERITY_GLYPH.ok} ${msg.findingReason(finding.type, finding.reason)}`;
const facts = compactFacts(finding, vm.metrics);
// Healthy sections stay on one line; the facts are short.
return facts ? [`${headline}${facts}`] : [headline];
}
function renderLivenessLine(finding: Finding, metrics: Metric[], msg: ReportMessages): string {
const metric = metricById(metrics, finding.metricIds[0]);
const ageMs = metric?.value;
const age = ageMs !== undefined && Number.isFinite(ageMs) ? fmtDuration(ageMs) : "unknown";
const reason = msg.findingReason(finding.type, finding.reason).replace("{age}", age);
return `${sectionLabel(finding.type)}${statusGlyph(finding.severity, finding.reason)} ${reason}`;
}
function renderFooter(footer: FooterEntry[], msg: ReportMessages): string[] {
return footer.map((entry, i) => {
const text = fill(msg.actionMessage(entry.code), { value: entry.value, min: entry.value });
return i === 0 ? `${text}` : ` ${text}`;
/** A finding's evidence: its metric rows, then `why:`. */
function renderFindingBody(finding: LayoutFinding, indent: string): string[] {
const rows = finding.metrics.slice(0, EVIDENCE_CAP);
const cols = computeColumns(rows);
const metricLines = rows.flatMap((row, i) => {
const rendered = renderMetricRow(row, cols, indent);
// One blank line between rows so the block breathes.
return i === 0 ? rendered : ["", ...rendered];
});
const why = renderNoteBlock(REPORT_LABELS.why, finding.why, indent);
return why.length > 0 ? [...metricLines, "", ...why] : metricLines;
}
// ---------------------------------------------------------------------------
// Top-level render.
// ---------------------------------------------------------------------------
/** Glyph, then the padded section label, so every finding's text starts on one column. */
function findingLine(finding: LayoutFinding): string {
return `${finding.glyph} ${finding.label.padEnd(SECTION_LABEL_WIDTH)}${" ".repeat(SECTION_GAP)}${finding.text}`;
}
const BODY_INDENT = " ";
/** A non-hero finding's body hangs under its section label rather than the page margin. */
const NESTED_INDENT = " ";
/**
* The plain monochrome layout (✓/⚠/✕) shared by every surface. Colour renderers paint
* THIS via `paintReport`; markdown swaps the glyphs for emoji. Internal so the three
* public renderers can't drift.
* The plain monochrome layout every text surface shares. ANSI paints this via `paintReport` and
* markdown swaps the glyphs for emoji, so the two can't drift from each other or from the card.
*/
function renderReportPlain(vm: ReportViewModel): string {
const msg = reportMessages(vm.title);
function renderReportPlain(vm: LayoutViewModel): string {
const layout: ReportLayout = buildReportLayout(vm, reportMessages(vm.title));
const lines: string[] = [];
// header: "/report <title>" padded to a column, then scope · period · baseline.
const left = `/report ${vm.title}`;
const right = [vm.scope, vm.period, vm.baselineLabel].filter(Boolean).join(" · ");
lines.push(`${left.padEnd(HEADER_COL)}${right}`, "");
// header: the report as its command (plus a trust flag), then scope · period · baseline.
const command = `/report ${layout.header.name}`;
const left = layout.trust
? `${command} ${REPORT_GLYPH.untrusted} ${layout.trust.badge}`
: command;
// Two spaces minimum, so a long left side still reads as two columns.
const gutter = " ".repeat(Math.max(2, HEADER_COL - left.length));
lines.push(`${left}${gutter}${layout.header.meta}`.replace(/\s+$/, ""), "");
// Each statement carries its OWN glyph — one leading glyph would read as if it
// applied only to the first statement (e.g. "✕ Flow healthy"). Per-statement is clear.
const verdict = vm.summary.statements
.map(
(s) =>
`${statusGlyph(s.severity, s.reason)} ${msg.statementMessage(s.findingType, s.severity, s.reason)}`
)
.join(" · ");
lines.push(verdict, "");
const { glyph, phrase, text } = layout.headline;
lines.push(`${glyph} ${phrase}${text ? `${text}` : ""}`, "");
const flowDegraded = vm.findings.some((f) => f.type === "flow" && f.severity !== "ok");
if (layout.trust) lines.push(`${REPORT_GLYPH.untrusted} ${layout.trust.note}`, "");
for (const finding of vm.findings) {
if (finding.type === "liveness") {
lines.push(renderLivenessLine(finding, vm.metrics, msg), "");
} else if (finding.reason === "unknown") {
// stale-data guard: no ✓ or facts computed from a silent feed. The guard forces crit,
// so the glyph reads from severity (consistent with the summary + JSON).
lines.push(
`${sectionLabel(finding.type)}${SEVERITY_GLYPH[finding.severity]} ${msg.findingReason(finding.type, finding.reason)}`,
""
);
} else if (finding.severity !== "ok") {
lines.push(...renderDegradedSection(finding, vm), "");
} else if (finding.type === "execution" && flowDegraded) {
lines.push(...renderHealthyExecutionExpanded(finding, vm), "");
} else {
lines.push(...renderCollapsedSection(finding, vm), "");
}
if (layout.hero) {
const body = renderFindingBody(layout.hero, BODY_INDENT);
if (body.length > 0) lines.push(...body, "");
}
lines.push(...renderFooter(vm.footer, msg));
for (const finding of layout.findings) {
lines.push(findingLine(finding));
if (finding.expanded) {
const body = renderFindingBody(finding, NESTED_INDENT);
if (body.length > 0) lines.push("", ...body);
}
lines.push("");
}
for (const statement of layout.statements) {
lines.push(`${statement.glyph} ${statement.text}`, "");
}
const reads = renderNoteBlock(REPORT_LABELS.read, layout.reads, BODY_INDENT);
if (reads.length > 0) lines.push(...reads, "");
// footer: the first entry leads with an arrow, the rest align under it.
lines.push(
...layout.footer.map((entry, i) => (i === 0 ? `${entry.label}` : ` ${entry.label}`))
);
return lines.join("\n").replace(/\n+$/, "\n");
}
/**
* Markdown surface (agents / chat). The plain layout with severity glyphs swapped for
* status emoji — the only decoration markdown gets.
*/
/** Markdown surface: the plain layout with the status glyphs swapped for status emoji. */
export function renderReportMarkdown(vm: ReportViewModel): string {
return toMarkdownEmoji(renderReportPlain(vm));
}
// ---------------------------------------------------------------------------
// ANSI colour renderer (terminal, e.g. `trigger report`). Colourises the SAME
// plain layout as a post-pass via `paintReport`, so terminal output can't drift.
// ---------------------------------------------------------------------------
// The ANSI renderer colourises the same plain layout as a post-pass, so terminal output can't drift.
const SPARK_LOW = "▁▂▃▄▅";
const SPARK_HIGH = "▆▇█";
@@ -481,31 +247,31 @@ function paintSpark(run: string, p: Paint): string {
.join("");
}
/** Colourise the plain report by role. Detection reads the RAW line; wrapping the escaped line. */
/** Colourise the plain report by role. Detection reads the raw line, wrapping the escaped line. */
function paintReport(text: string, p: Paint): string {
const noteLabels = `(?:${REPORT_LABELS.read}|${REPORT_LABELS.why})`;
return text
.split("\n")
.map((raw, i) => {
const line = p.escape(raw);
// header: grey the right column (scope · period · baseline).
if (i === 0) {
return line.replace(/^(\/report \S+\s{2,})(.+)$/, (_, a, b) => a + p.grey(b));
// whole-line secondary: a read:/why: block and its hanging lines.
if (new RegExp(`^\\s*${noteLabels}`).test(raw) || /^\s{6,}\S/.test(raw)) {
return p.grey(p.escape(raw));
}
// whole-line secondary: read: chain, its exclusion lines, do-nothing footer.
if (/^\s*read:/.test(raw) || /^\s{6,}\S/.test(raw)) return p.grey(line);
if (/^\s{2}(or do nothing|open |Check status)/.test(raw)) return p.grey(line);
let l = line;
// header: grey the right column (scope · period · baseline), then paint its glyphs below.
let l = p.escape(raw);
if (i === 0) l = l.replace(/^(\/report .*\s{2,})(\S.*)$/, (_, a, b) => a + p.grey(b));
l = l.replace(/[▁▂▃▄▅▆▇█]+/g, (m) => paintSpark(m, p)); // two-tone sparkline
l = l
.replace(/✓/g, p.green("✓"))
.replace(/⚠/g, p.amber("⚠"))
.replace(/✕/g, p.red("✕"))
.replace(/○/g, p.grey("")); // neutral (unknown) marker
l = l.replace(/↑ ?\d+×|↑/g, (m) => p.amber(m)).replace(/↓ ?\d+×|↓/g, (m) => p.green(m));
l = l.replace(/\(last \d+ min\)|\(\d+ min window\)/g, (m) => p.amber(m)); // anomaly window
l = l.replace(/\(normal ~[^)]*\)|\(estimated\)/g, (m) => p.grey(m)); // baseline/estimate is context
l = l.replace(/^(\s+worst \w+\s+)(.+?)( — )/, (_, a, k, b) => a + p.green(k) + b); // attribution key (may contain spaces)
.replace(new RegExp(REPORT_GLYPH.ok, "g"), p.green(REPORT_GLYPH.ok))
.replace(new RegExp(REPORT_GLYPH.warn, "g"), p.amber(REPORT_GLYPH.warn))
.replace(new RegExp(REPORT_GLYPH.crit, "g"), p.red(REPORT_GLYPH.crit))
.replace(new RegExp(REPORT_GLYPH.neutral, "g"), p.grey(REPORT_GLYPH.neutral))
.replace(new RegExp(REPORT_GLYPH.untrusted, "g"), p.amber(REPORT_GLYPH.untrusted));
l = l.replace(/↑ ?\d+×/g, (m) => p.amber(m)).replace(/↓ ?\d+×/g, (m) => p.green(m));
l = l.replace(/→ flat/g, (m) => p.grey(m));
l = l.replace(/\(last \d+ min\)|\(\d+ min window\)|for the last \d+ min/g, (m) => p.amber(m));
l = l.replace(/\(normal ~[^)]*\)|\(estimated[^)]*\)/g, (m) => p.grey(m)); // context, not a verdict
return l;
})
.join("\n");
@@ -0,0 +1,101 @@
import { describe, expect, it } from "vitest";
import { buildReportLayout, type LayoutViewModel } from "./report-layout";
import { reportMessages } from "./report-messages";
const livenessMetric = {
id: "liveness",
value: 21 * 60 * 1000,
unit: "ms" as const,
severity: "crit" as const,
};
function vmWith(findings: LayoutViewModel["findings"]): LayoutViewModel {
return {
title: "health",
scope: "prod",
period: "last 60 min",
windowMinutes: 60,
summary: { severity: "crit", statements: [] },
findings,
metrics: [livenessMetric],
footer: [],
};
}
const livenessFinding = {
type: "liveness",
severity: "crit" as const,
reason: "stale",
metricIds: ["liveness"],
};
function vmWithMetric(metric: LayoutViewModel["metrics"][number]): LayoutViewModel {
return {
title: "health",
scope: "prod",
period: "last 60 min",
windowMinutes: 60,
summary: { severity: "warn", statements: [] },
findings: [{ type: "queue", severity: "warn", reason: "backlog", metricIds: [metric.id] }],
metrics: [metric],
footer: [],
};
}
function heroDelta(metric: LayoutViewModel["metrics"][number]) {
const layout = buildReportLayout(vmWithMetric(metric), reportMessages("health"));
return layout.hero?.metrics.find((m) => m.id === metric.id)?.delta;
}
describe("buildReportLayout metric deltas", () => {
it("renders no delta when a metric collapsed to nothing", () => {
expect(
heroDelta({
id: "pending",
value: 0,
unit: "count",
severity: "warn",
normal: 40,
delta: { dir: "down", mult: 0 },
})
).toBeUndefined();
});
it("still renders a multiplier for a genuine fall", () => {
expect(
heroDelta({
id: "pending",
value: 10,
unit: "count",
severity: "warn",
normal: 40,
delta: { dir: "down", mult: 0 },
})
).toEqual({ text: "↓ 4×", dir: "down" });
});
});
describe("buildReportLayout self-evident findings", () => {
it("drops the metric row of a non-hero finding whose only metric is its own line", () => {
const layout = buildReportLayout(
vmWith([
{ type: "execution", severity: "crit", reason: "failures_up", metricIds: [] },
livenessFinding,
]),
reportMessages("health")
);
const liveness = layout.findings.find((f) => f.type === "liveness");
expect(liveness?.text).toContain("no telemetry in 21m");
expect(liveness?.expanded).toBe(false);
expect(liveness?.metrics).toEqual([]);
});
it("keeps the metric row when that finding is the hero", () => {
const layout = buildReportLayout(vmWith([livenessFinding]), reportMessages("health"));
expect(layout.hero?.type).toBe("liveness");
expect(layout.hero?.expanded).toBe(true);
expect(layout.hero?.metrics.map((m) => m.id)).toEqual(["liveness"]);
});
});
@@ -0,0 +1,668 @@
/**
* The report's layout, declared once: section order, section labels, the tone -> glyph vocabulary,
* and every string a renderer places. `renderMarkdown` (markdown + ANSI) and the React card both
* build from `buildReportLayout`, so no surface can drift in order, labels or wording.
*
* A renderer decides typography only: colour, indentation, column alignment, tooltips. Anything
* that answers "what is shown, in what order, called what" belongs here.
*
* Input types are deliberately loose (`aggregation?: string`, not the enum) so both the presenter's
* `ReportViewModel` and the agent contracts' `ReportViewModelPayload` satisfy them.
*/
import { type ReportMessages } from "./report-messages";
import { type Severity, type Unit } from "./report-view-model";
// --- vocabulary -------------------------------------------------------------
/**
* A verdict's tone. `neutral` is a genuinely-unknown state: not good, not bad, and it must never
* read as a confident tick.
*/
export type ReportTone = "ok" | "warn" | "crit" | "neutral";
/**
* The one glyph vocabulary. Text surfaces carry meaning in these because an MCP host shows plain
* text and a terminal may be monochrome; ANSI paints them as well. Every character is BMP and
* single-width, so nothing here depends on emoji fonts or colour to be legible.
*/
export const REPORT_GLYPH = {
ok: "✓",
warn: "⚠",
crit: "✕",
/** Genuinely unknown, neither good nor bad. */
neutral: "○",
/** The data behind the report can't be trusted. */
untrusted: "⚑",
/** Above its baseline. */
up: "↑",
/** Below its baseline. */
down: "↓",
/** Compared against a baseline and unmoved. */
flat: "→",
} as const;
/** Section and block labels. Both renderers read them from here. */
export const REPORT_LABELS = {
/** Evidence supporting a finding. */
why: "why:",
/** The causal chain across findings. */
read: "read:",
/** The footer heading. */
nextSteps: "Next steps",
} as const;
/** The flag beside the report's name, and the caveat under its headline. */
export type LayoutTrust = { badge: string; note: string };
/**
* Why a report's numbers can't be trusted, in its own words. Stale, absent and unmeasured are three
* different states: a snapshot with no telemetry feed is not stale, and a caveat may only discount
* the input it names — the aggregates the report did measure stay measured.
*/
const TRUST_CAVEATS: Record<string, LayoutTrust> = {
telemetry_stale: {
badge: "stale data",
note: "The telemetry behind this report is stale, so the numbers below are informational only.",
},
telemetry_absent: {
badge: "no telemetry",
note: "No telemetry feed reached this report, so how current it is can't be confirmed; the numbers below are still measured over the window.",
},
flow_unmeasured: {
badge: "unmeasured",
note: "The queue depth could not be measured, so the backlog can't be assessed; the other numbers below are measured.",
},
};
const TRUST_CAVEAT_FALLBACK: LayoutTrust = {
badge: "unverified data",
note: "The data behind this report could not be verified, so the numbers below are informational only.",
};
/**
* The report's sections, top to bottom. A renderer walks this order; a new section has to be added
* here first, which is what keeps the surfaces aligned. `trust` spans two places: a flag beside the
* report's name, and its caveat under the headline.
*/
export const REPORT_SECTION_ORDER = [
"header",
"trust",
"headline",
"hero",
"findings",
"statements",
"read",
"footer",
] as const;
export type ReportSectionId = (typeof REPORT_SECTION_ORDER)[number];
/**
* Reasons that mean "we can't say" rather than a verdict, so their finding renders headline-only.
* A measured finding never carries one: an unmeasured input costs its own metric, not the verdict.
*/
const UNASSESSABLE_REASONS = new Set(["unknown", "flow_unmeasured"]);
/**
* Reasons whose state is genuinely unknown but not bad. A stale feed is different: the trust guard
* forces crit.
*/
const NEUTRAL_REASONS = new Set(["freshness_unknown", "flow_unmeasured"]);
/**
* `facts.trustworthy === false` means the numbers behind the verdict are informational only. Absent
* = trustworthy (the common case, and what pre-`facts` snapshots imply).
*/
export function reportIsTrustworthy(vm: { facts?: Record<string, unknown> }): boolean {
return vm.facts?.trustworthy !== false;
}
/** The caveat for an untrustworthy report, chosen by `facts.untrustworthyReason`. */
export function reportTrust(vm: { facts?: Record<string, unknown> }): LayoutTrust | undefined {
if (reportIsTrustworthy(vm)) return undefined;
const reason = vm.facts?.untrustworthyReason;
return (typeof reason === "string" ? TRUST_CAVEATS[reason] : undefined) ?? TRUST_CAVEAT_FALLBACK;
}
export function reportTone(severity: Severity, reason?: string): ReportTone {
return reason !== undefined && NEUTRAL_REASONS.has(reason) ? "neutral" : severity;
}
export function reportGlyph(severity: Severity, reason?: string): string {
return REPORT_GLYPH[reportTone(severity, reason)];
}
// --- footer vocabulary ------------------------------------------------------
/**
* How a footer entry renders, keyed off the code rather than its URL so the same code looks the
* same everywhere. `action` is a primary control, `docs` the docs entry, `reference` a place to
* look, and `note` an option stated rather than offered.
*/
export type ReportFooterStyle = "action" | "docs" | "reference" | "note";
const FOOTER_NOTE_CODES = new Set(["nothing_to_do", "do_nothing_drains", "region_failover"]);
const FOOTER_REFERENCE_CODES = new Set(["check_control_plane", "check_platform_status"]);
/** A doc entry names itself one: `concurrency_docs`, `retries_docs`. */
const FOOTER_DOCS_SUFFIX = "_docs";
export function reportFooterStyle(code: string): ReportFooterStyle {
if (FOOTER_NOTE_CODES.has(code)) return "note";
if (code.endsWith(FOOTER_DOCS_SUFFIX)) return "docs";
if (FOOTER_REFERENCE_CODES.has(code)) return "reference";
return "action";
}
// --- formatting -------------------------------------------------------------
const MINUS = ""; // U+2212
export function fmtCount(n: number): string {
return Math.round(n).toLocaleString("en-US");
}
export function fmtDuration(ms: number): string {
if (ms < 1000) return `${Math.round(ms)}ms`;
const s = ms / 1000;
if (s < 60) return Number.isInteger(s) ? `${s}s` : `${s.toFixed(1)}s`;
const m = s / 60;
return Number.isInteger(m) ? `${m}m` : `${m.toFixed(1)}m`;
}
export function fmtPct(ratio: number): string {
return `${(ratio * 100).toFixed(1)}%`;
}
export function fmtRate(n: number): string {
return `${fmtCount(n)}/min`;
}
/** A net rate carries its sign; a plain rate does not, so it isn't read as a change. */
export function fmtSignedRate(net: number): string {
const sign = net < 0 ? MINUS : net > 0 ? "+" : "";
return `${sign}${fmtCount(Math.abs(net))}/min`;
}
export function fmtValue(value: number, unit: Unit): string {
switch (unit) {
case "ms":
return fmtDuration(value);
case "count":
return fmtCount(value);
case "ratio":
return fmtPct(value);
case "perMin":
return fmtRate(value);
}
}
/** Fill the `{token}` placeholders a message catalog leaves for the renderer. */
export function fillTokens(
template: string,
tokens: Record<string, string | number | undefined>
): string {
return template.replace(/\{(\w+)\}/g, (whole, key: string) => {
const value = tokens[key];
if (value === undefined) return whole;
// Grouped but never rounded: a drain ETA of 26.7 min must stay 26.7.
return typeof value === "number" ? value.toLocaleString("en-US") : value;
});
}
// --- input shapes -----------------------------------------------------------
type DeltaInput = { dir: "up" | "down" | "flat"; mult?: number };
export type LayoutMetricInput = {
id: string;
value: number;
unit: Unit;
aggregation?: string;
normal?: number;
delta?: DeltaInput;
series?: { points: number[]; kind: string };
breakdown?: Record<string, number>;
annotation?: { code: string; value?: number };
availability?: string;
severity: Severity;
};
export type LayoutFindingInput = {
type: string;
severity: Severity;
reason: string;
read?: string;
metricIds: string[];
anomalyWindow?: { minutes: number; touchesEnd: boolean };
attribution?: { dim: string; key: string; share: number; of: string };
exclusions?: { code: string; evidence?: Record<string, number> }[];
observations?: { code: string; evidence?: Record<string, number> }[];
};
export type LayoutViewModel = {
title: string;
scope: string;
period: string;
baselineLabel?: string;
windowMinutes: number;
summary: {
severity: Severity;
statements: { findingType: string; severity: Severity; reason?: string }[];
};
findings: LayoutFindingInput[];
metrics: LayoutMetricInput[];
facts?: Record<string, unknown>;
footer: { code: string; link?: string; value?: number }[];
};
// --- output shapes ----------------------------------------------------------
export type LayoutDelta = { text: string; dir: "up" | "down" | "flat" };
/** A metric's aside. `kind` lets a renderer choose its own frame around shared wording. */
export type LayoutNote = { kind: "annotation" | "baseline" | "estimated"; text: string };
export type LayoutMetricRow = {
id: string;
label: string;
value: string;
/** Kept alongside the formatted value so a chart can format its own points. */
unit: Unit;
severity: Severity;
delta?: LayoutDelta;
note?: LayoutNote;
/** The row that explains the finding: its annotation is spelled out rather than tucked away. */
hero: boolean;
series?: number[];
/** Trailing breach window of the driving metric, when it reaches now. */
anomalyMinutes?: number;
/** A composite metric's parts, shown under it. */
subRows: { label: string; value: string }[];
};
export type LayoutFinding = {
type: string;
/** Column label for the section, e.g. "EXECUTION". */
label: string;
severity: Severity;
tone: ReportTone;
glyph: string;
/** The resolved reason, plus its anomaly window. */
text: string;
/** Whether the evidence block is shown at all. */
expanded: boolean;
metrics: LayoutMetricRow[];
/** Evidence lines under `why:`. */
why: string[];
/** The attributed key, so a renderer can pick it out of the first `why:` line. */
attributionKey?: string;
};
export type LayoutStatement = { tone: ReportTone; glyph: string; severity: Severity; text: string };
export type LayoutFooterEntry = {
code: string;
style: ReportFooterStyle;
label: string;
/** Key into `vm.links`. */
link?: string;
};
export type ReportLayout = {
header: { name: string; meta: string };
/** Present only when the data can't be trusted. */
trust?: LayoutTrust;
headline: { tone: ReportTone; glyph: string; severity: Severity; phrase: string; text?: string };
/** The finding the headline speaks for, always expanded. */
hero?: LayoutFinding;
/** The remaining findings, in view-model order. */
findings: LayoutFinding[];
/** Statements with no finding behind them, which still have to be said. */
statements: LayoutStatement[];
reads: string[];
footer: LayoutFooterEntry[];
};
// --- build ------------------------------------------------------------------
/** The layout of `vm`, with every code already resolved through `messages`. */
export function buildReportLayout(vm: LayoutViewModel, messages: ReportMessages): ReportLayout {
const tokens = reportTokens(vm);
const heroIndex = heroIndexOf(vm);
const heroInput = vm.findings[heroIndex];
const heroStatement = vm.summary.statements.find((s) => s.findingType === heroInput?.type);
const hero = heroInput ? findingLayout(vm, messages, heroInput, tokens, true) : undefined;
const findings = vm.findings
.filter((_, i) => i !== heroIndex)
.map((finding) => findingLayout(vm, messages, finding, tokens, false));
// The headline speaks for the hero finding. When its statement carries its own reason (stale
// telemetry, no freshness signal) that statement is the whole sentence and the finding's reason
// would only repeat it.
const phrase = heroInput
? messages.statementMessage(heroInput.type, heroInput.severity, heroStatement?.reason)
: messages.statementMessage(vm.title, vm.summary.severity);
const text =
heroInput && !heroStatement?.reason
? fillTokens(
messages.findingReason(heroInput.type, heroInput.reason, {
expanded: heroInput.severity === "ok",
}),
tokens
) + headlineWindow(heroInput)
: undefined;
const statements = vm.summary.statements
.filter((statement) => !vm.findings.some((f) => f.type === statement.findingType))
.map((statement) => ({
severity: statement.severity,
tone: reportTone(statement.severity, statement.reason),
glyph: reportGlyph(statement.severity, statement.reason),
text: messages.statementMessage(statement.findingType, statement.severity, statement.reason),
}));
// Hero first, then the rest. An unassessable finding contributes nothing: its read would only
// repeat the trust note.
const reads = (
hero ? [heroInput!, ...vm.findings.filter((_, i) => i !== heroIndex)] : vm.findings
)
.filter((finding) => finding.read !== undefined && !UNASSESSABLE_REASONS.has(finding.reason))
.map((finding) => fillTokens(messages.readMessage(finding.read!), tokens));
const trust = reportTrust(vm);
return {
header: {
name: vm.title,
meta: [vm.scope, vm.period, vm.baselineLabel].filter(Boolean).join(" · "),
},
...(trust === undefined ? {} : { trust }),
headline: {
severity: vm.summary.severity,
tone: reportTone(vm.summary.severity, heroStatement?.reason),
glyph: reportGlyph(vm.summary.severity, heroStatement?.reason),
phrase,
...(text === undefined ? {} : { text }),
},
...(hero === undefined ? {} : { hero }),
findings,
statements,
reads,
footer: footerLayout(vm, messages, tokens),
};
}
/** The finding the headline speaks for: the first one at the report's severity. */
function heroIndexOf(vm: LayoutViewModel): number {
const index = vm.findings.findIndex((finding) => finding.severity === vm.summary.severity);
return index === -1 ? 0 : index;
}
/** Tokens the catalog's strings leave for the renderer, resolved from the view model's metrics. */
function reportTokens(vm: LayoutViewModel): Record<string, string | number> {
const metric = (id: string) => vm.metrics.find((m) => m.id === id);
const triggered = metric("triggered");
const throughput = metric("throughput");
const liveness = metric("liveness");
return {
mult: triggered?.delta?.mult ?? "",
rate: Math.round(throughput?.breakdown?.done ?? 0),
age: liveness === undefined || isUnmeasured(liveness) ? "unknown" : fmtDuration(liveness.value),
};
}
function isUnmeasured(metric: LayoutMetricInput): boolean {
return metric.availability === "unknown" || !Number.isFinite(metric.value);
}
/** The anomaly window as the headline says it. */
function headlineWindow(finding: LayoutFindingInput): string {
const window = finding.anomalyWindow;
if (!window) return "";
return window.touchesEnd
? ` for the last ${window.minutes} min`
: ` (${window.minutes} min window)`;
}
/** The same window on a finding line, where it is an aside rather than the sentence's tail. */
function findingWindow(finding: LayoutFindingInput): string {
const window = finding.anomalyWindow;
if (!window) return "";
return window.touchesEnd ? ` (last ${window.minutes} min)` : ` (${window.minutes} min window)`;
}
function findingLayout(
vm: LayoutViewModel,
messages: ReportMessages,
finding: LayoutFindingInput,
tokens: Record<string, string | number>,
hero: boolean
): LayoutFinding {
// A finding whose reason says "we can't say" shows no evidence: the numbers behind it are
// placeholders. Otherwise the hero is always expanded, and the rest only when degraded.
// A self-evident finding's one metric only repeats its own line ("stale — no telemetry in 21m"
// over "liveness 21m"); as the hero that line is the headline, so the row is its only evidence.
const selfEvident = finding.metricIds.length === 1 && finding.metricIds[0] === finding.type;
const expanded =
!UNASSESSABLE_REASONS.has(finding.reason) &&
(hero || (finding.severity !== "ok" && !selfEvident));
const metrics = expanded
? finding.metricIds
.map((id) => vm.metrics.find((m) => m.id === id))
.filter((m): m is LayoutMetricInput => m !== undefined)
.map((metric, i) =>
metricRow(vm, messages, metric, i === 0, i === 0 ? anomalyMinutes(finding) : undefined)
)
: [];
return {
type: finding.type,
label: finding.type.toUpperCase(),
severity: finding.severity,
tone: reportTone(finding.severity, finding.reason),
glyph: reportGlyph(finding.severity, finding.reason),
text:
fillTokens(
messages.findingReason(finding.type, finding.reason, {
expanded: finding.severity === "ok",
}),
tokens
) + findingWindow(finding),
expanded,
metrics,
why: expanded ? whyLines(messages, finding, tokens) : [],
...(expanded && finding.attribution ? { attributionKey: finding.attribution.key } : {}),
};
}
function anomalyMinutes(finding: LayoutFindingInput): number | undefined {
return finding.anomalyWindow?.touchesEnd ? finding.anomalyWindow.minutes : undefined;
}
/** Attribution first, then ruled-out causes, then supporting observations. */
function whyLines(
messages: ReportMessages,
finding: LayoutFindingInput,
tokens: Record<string, string | number>
): string[] {
const lines: string[] = [];
const attribution = finding.attribution;
if (attribution) {
lines.push(
`${Math.round(attribution.share * 100)}% of ${attribution.of} is ${attribution.key}`
);
}
for (const exclusion of finding.exclusions ?? []) {
lines.push(
fillTokens(
messages.exclusionMessage(exclusion.code),
evidenceTokens(tokens, exclusion.evidence)
)
);
}
for (const observation of finding.observations ?? []) {
lines.push(
fillTokens(
messages.observationMessage(observation.code),
evidenceTokens(tokens, observation.evidence)
)
);
}
return lines;
}
/**
* A code's own evidence beats the report-wide tokens. `finishedPerMin` is the measured rate behind
* `{rate}`, so it fills that token rather than needing one of its own.
*/
function evidenceTokens(
tokens: Record<string, string | number>,
evidence: Record<string, number> | undefined
): Record<string, string | number> {
if (!evidence) return tokens;
return {
...tokens,
...evidence,
...(evidence.finishedPerMin === undefined ? {} : { rate: evidence.finishedPerMin }),
};
}
function isComposite(metric: LayoutMetricInput): boolean {
return metric.unit === "perMin" && metric.breakdown?.done !== undefined;
}
function metricValue(metric: LayoutMetricInput, messages: ReportMessages): string {
// A placeholder is never printed as a number.
if (isUnmeasured(metric)) return "unknown";
// concurrency etc. carry a limit -> "running/limit".
if (metric.unit === "count" && metric.breakdown?.limit !== undefined) {
return `${fmtCount(metric.value)}/${fmtCount(metric.breakdown.limit)}`;
}
// A composite throughput's value is the net, which is signed; its parts are sub-rows.
if (isComposite(metric)) return fmtSignedRate(metric.value);
const showAggregation =
metric.aggregation === "p95" &&
!messages.metricLabel(metric.id).toLowerCase().startsWith("p95");
return showAggregation
? `p95 ${fmtValue(metric.value, metric.unit)}`
: fmtValue(metric.value, metric.unit);
}
/**
* How far a metric fell below its baseline: `undefined` when the fall doesn't round past 1×, and
* `null` when it collapsed to nothing and no multiplier can say it.
*/
function fallMultiplier(metric: LayoutMetricInput): number | null | undefined {
if (metric.normal === undefined || metric.normal <= 0) return undefined;
if (metric.value <= 0) return null;
const fall = Math.round(metric.normal / metric.value);
return fall > 1 ? fall : undefined;
}
/**
* A metric's movement against its baseline. A multiplier only reads as movement once it rounds past
* 1×; below that a metric with a baseline is flat, and one without has nothing to compare against.
*/
function metricDelta(metric: LayoutMetricInput): LayoutDelta | undefined {
const delta = metric.delta;
// A fall's own multiplier rounds to 0 or 1, so measure how far it fell instead.
if (delta?.dir === "down") {
const fall = fallMultiplier(metric);
// An arrow with no multiplier behind it says nothing the sparkline hasn't.
if (fall === null) return undefined;
if (fall !== undefined) return { text: `${REPORT_GLYPH.down} ${fall}×`, dir: "down" };
}
if (delta && delta.mult !== undefined && delta.mult > 1 && delta.dir !== "flat") {
return {
text: `${delta.dir === "up" ? REPORT_GLYPH.up : REPORT_GLYPH.down} ${delta.mult}×`,
dir: delta.dir,
};
}
return metric.normal === undefined
? undefined
: { text: `${REPORT_GLYPH.flat} flat`, dir: "flat" };
}
function metricNote(
vm: LayoutViewModel,
messages: ReportMessages,
metric: LayoutMetricInput
): LayoutNote | undefined {
if (metric.annotation) {
return {
kind: "annotation",
text: fillTokens(messages.annotationMessage(metric.annotation.code), {
value: metric.annotation.value,
window: vm.windowMinutes,
limit: metric.breakdown?.limit,
}),
};
}
if (metric.normal !== undefined) {
return { kind: "baseline", text: `normal ~${fmtValue(metric.normal, metric.unit)}` };
}
// A proxy trend (e.g. a snapshot backlog) is a shape, not a measurement, so say so.
if (metric.series?.kind === "estimated") {
return { kind: "estimated", text: "estimated from a proxy signal" };
}
return undefined;
}
function metricRow(
vm: LayoutViewModel,
messages: ReportMessages,
metric: LayoutMetricInput,
hero: boolean,
anomaly: number | undefined
): LayoutMetricRow {
const delta = metricDelta(metric);
const note = metricNote(vm, messages, metric);
const points = isUnmeasured(metric) ? undefined : metric.series?.points;
return {
id: metric.id,
label: messages.metricLabel(metric.id),
value: metricValue(metric, messages),
unit: metric.unit,
severity: metric.severity,
hero,
...(delta === undefined ? {} : { delta }),
...(note === undefined ? {} : { note }),
...(points && points.length > 0 ? { series: points } : {}),
...(anomaly === undefined ? {} : { anomalyMinutes: anomaly }),
subRows: isComposite(metric)
? [
{ label: "done", value: fmtRate(metric.breakdown!.done!) },
{ label: "triggered", value: fmtRate(metric.breakdown!.triggered ?? 0) },
]
: [],
};
}
/** Offered entries first, stated options last — a note is the fallback, not a next step. */
function footerLayout(
vm: LayoutViewModel,
messages: ReportMessages,
tokens: Record<string, string | number>
): LayoutFooterEntry[] {
const entries = vm.footer.map((entry) => ({
code: entry.code,
style: reportFooterStyle(entry.code),
label: fillTokens(messages.actionMessage(entry.code), {
...tokens,
value: entry.value,
min: entry.value,
}),
...(entry.link === undefined ? {} : { link: entry.link }),
}));
return [
...entries.filter((entry) => entry.style !== "note"),
...entries.filter((entry) => entry.style === "note"),
];
}
@@ -1,8 +1,3 @@
/**
* Catalogs by value, in a module that imports ONLY the per-report `*-messages`
* files — no loaders, no IO. Presentation stays decoupled from the data layer,
* and a value import can't be tree-shaken away.
*/
import { healthMessages } from "./health/health-messages";
import { type ReportMessages } from "./report-messages";
@@ -1,17 +1,7 @@
/**
* Report-agnostic message infrastructure. Prose lives in each report's OWN catalog (e.g.
* `health/health-messages.ts`); this file only defines the resolver surface + a registry so the
* generic renderer can turn a VM's codes into strings by looking the catalog up via `vm.title`.
* No report vocabulary here — that would re-couple the renderer to a specific report.
*/
import { REPORT_MESSAGE_CATALOGS } from "./report-message-catalogs";
import { type ReasonCode, type Severity } from "./report-view-model";
/**
* The resolver surface a report provides. Every renderer resolves a VM's codes through this;
* strings may carry {tokens} (e.g. {age}, {rate}) that the renderer fills from evidence.
*/
/** Strings may carry {tokens} the renderer fills from evidence. */
export type ReportMessages = {
metricLabel(id: string): string;
findingReason(findingType: string, reason: ReasonCode, opts?: { expanded?: boolean }): string;
@@ -23,9 +13,7 @@ export type ReportMessages = {
actionMessage(code: ReasonCode): string;
};
/** Look up a report's catalog by `vm.title`. Catalogs are values, never
* registered at import time — side-effect registration is what the production
* bundle tree-shakes away. */
/** Catalogs are plain values, not import-time registrations: the bundle tree-shakes those away. */
export function reportMessages(title: string): ReportMessages {
const messages = REPORT_MESSAGE_CATALOGS[title];
if (!messages) {
@@ -1,18 +1,14 @@
/**
* The report catalog: which reports exist and how each loads + interprets its data. Keyed by
* report name so cost/regression/errors drop in later as new `{ load, interpret }` entries with
* no changes to the VM, renderers, route, tool, or presenter.
*
* Deliberately separate from `ReportPresenter` — the presenter only orchestrates (look up a
* loader by key, run it, single-flight); knowing WHICH reports exist is a distinct concern.
*/
import { type AuthenticatedEnvironment } from "~/services/apiAuth.server";
import { interpret as interpretHealth } from "./health/health";
import { loadHealthInput } from "./health/health-data";
import { type ReportViewModel } from "./report-view-model";
/** A query table a report may read. Same table names the query API authorizes against. */
export type ReportQueryTable = "runs" | "env_metrics" | "queue_metrics";
export type ReportLoader<TInput> = {
/** Authorization metadata: the route derives its per-table JWT scope check from this. */
tables: readonly ReportQueryTable[];
load: (env: AuthenticatedEnvironment, period: string) => Promise<TInput>;
interpret: (input: TInput) => ReportViewModel;
};
@@ -23,6 +19,7 @@ function defineReport<TInput>(loader: ReportLoader<TInput>): ReportLoader<unknow
export const REPORT_REGISTRY: Record<string, ReportLoader<unknown>> = {
health: defineReport({
tables: ["runs", "env_metrics", "queue_metrics"],
load: (env, period) => loadHealthInput(env, period),
interpret: interpretHealth,
}),
@@ -31,7 +28,19 @@ export const REPORT_REGISTRY: Record<string, ReportLoader<unknown>> = {
export const REPORT_KEYS = Object.keys(REPORT_REGISTRY);
export function isReportKey(key: string): boolean {
// Object.hasOwn, not `in`: `in` matches prototype keys ("toString", "__proto__"),
// which would pass the route guard and then 500 in the loader.
// Not `in`: it matches prototype keys like "toString", which would pass the route guard.
return Object.hasOwn(REPORT_REGISTRY, key);
}
type ReportTablesRegistry = Record<string, Pick<ReportLoader<unknown>, "tables">>;
/**
* Input to the route's JWT scope check. An unknown key declares no tables: `checkAuth` denies an
* empty `everyResource`, so a bad key authorizes nothing rather than everything.
*/
export function reportQueryTables(
key: string,
registry: ReportTablesRegistry = REPORT_REGISTRY
): readonly ReportQueryTable[] {
return Object.hasOwn(registry, key) ? registry[key].tables : [];
}
@@ -1,164 +1,43 @@
/**
* Generic, render-agnostic contract for a Report. Semantic, not a UI tree: numbers +
* what they mean (codes, severities, units, series), never formatted strings or layout.
* Every renderer consumes this and owns presentation. Reasons are codes;
* `report-messages.ts` resolves them -> strings, so phrasing lives in one place.
*
* No React/DOM/IO. Report-agnostic — `health` is just one interpreter that emits it.
*/
// The shapes are zod schemas in `@trigger.dev/core/v3/schemas` because `format=json` serves this view
// model verbatim and the clients parse it. Here they are re-aliased to short local names.
export type Severity = "ok" | "warn" | "crit";
export type Unit = "ms" | "count" | "ratio" | "perMin";
import {
type ReportDelta,
type ReportExclusion,
type ReportFinding,
type ReportFooterEntry,
type ReportLink as CoreReportLink,
type ReportLinkKey,
type ReportMetric,
type ReportMetricSeries,
type ReportObservation,
type ReportReasonCode,
type ReportRecommendation,
type ReportSeverity,
type ReportSummaryStatement,
type ReportUnit,
type ReportViewModel as CoreReportViewModel,
} from "@trigger.dev/core/v3/schemas";
export type Severity = ReportSeverity;
export type Unit = ReportUnit;
/** A code resolved to a human string by `report-messages.ts`. */
export type ReasonCode = string;
export type ReasonCode = ReportReasonCode;
/** A key into `ReportViewModel.links`, so a recommendation can point at a URL. */
export type LinkKey = string;
export type LinkKey = ReportLinkKey;
export type Delta = ReportDelta;
export type MetricSeries = ReportMetricSeries;
export type Metric = ReportMetric;
export type Recommendation = ReportRecommendation;
export type FooterEntry = ReportFooterEntry;
export type Exclusion = ReportExclusion;
export type Observation = ReportObservation;
export type Finding = ReportFinding;
export type SummaryStatement = ReportSummaryStatement;
export type ReportLink = CoreReportLink;
export type ReportViewModel = CoreReportViewModel;
export type Delta = {
dir: "up" | "down" | "flat";
/** rounded value/normal multiplier; renderer decides whether to print "6×". */
mult?: number;
};
export type MetricSeries = {
points: number[];
/** "estimated" = a proxy (e.g. pending backlog), shown informational-only. */
kind: "measured" | "estimated";
};
export type Metric = {
/** CODE, e.g. "start_latency_p95" — messages map -> label "start latency". */
id: string;
value: number;
unit: Unit;
aggregation?: "p95" | "rate" | "ratio" | "count";
/** baseline; renderer formats "(normal ~7s)". */
normal?: number;
delta?: Delta;
series?: MetricSeries;
/** named sub-values for composite metrics (e.g. throughput { done, triggered }). */
breakdown?: Record<string, number>;
/** shown on a cause line INSTEAD of "(normal ~x)", e.g. "pinned 40 of last 60 min". */
annotation?: { code: ReasonCode; value?: number };
/**
* Whether `value` is a real measurement. "unknown" = there was no signal, so `value` is a
* placeholder (e.g. liveness age 0) that a structured consumer must NOT read as a real 0 —
* the finding's reason carries the "unknown" meaning. Absent = measured (the common case).
*/
availability?: "measured" | "unknown";
severity: Severity;
};
export type Recommendation = {
code: ReasonCode;
link?: LinkKey;
};
/** A footer line: an action, or the "do nothing" option (carries value). */
export type FooterEntry = {
code: ReasonCode;
link?: LinkKey;
/** a computed fact (e.g. drainMinutes), never invented. */
value?: number;
};
/** A ruled-out cause + its evidence, e.g. "not your code" (never emitted without evidence). */
export type Exclusion = {
code: ReasonCode;
evidence?: Record<string, number>;
};
/**
* A supporting fact backing the verdict — a measured observation, NOT a ruled-out cause,
* e.g. "runs are completing at ~820/min". Kept separate from `Exclusion` so the two aren't
* conflated (an exclusion answers "what it ISN'T"; an observation states "what IS true").
*/
export type Observation = {
code: ReasonCode;
evidence?: Record<string, number>;
};
export type Finding = {
/** "flow" | "execution" | "liveness" | future "infrastructure" | "billing" */
type: string;
severity: Severity;
/** CODE for the state/cause, e.g. "env_limit_saturation" | "healthy". */
reason: ReasonCode;
/** CODE for the "read:" line. Built last, may span findings. */
read?: ReasonCode;
/** metric ids this finding covers, in causal order when degraded. */
metricIds: string[];
/** ONE primary action. */
recommendation?: Recommendation;
/** optional parenthetical — same shape as recommendation. */
hedge?: Recommendation;
/** contiguous breach window of the driving metric -> "(last 40 min)". */
anomalyWindow?: { minutes: number; touchesEnd: boolean };
/**
* which dimension/key owns the problem, only when share >= threshold. `of` is the
* denominator label the renderer prints (e.g. "pending" for flow, "failures" for execution)
* — so it never mislabels a failures share as "% of pending".
*/
attribution?: { dim: string; key: string; share: number; of: string };
/** ruled-out causes + evidence — rendered under the `read:` line ("not your code …"). */
exclusions?: Exclusion[];
/** supporting facts + evidence — rendered under the `read:` line after the exclusions. */
observations?: Observation[];
};
export type SummaryStatement = {
findingType: string;
severity: Severity;
/**
* Normally the statement renders from (findingType, severity). Exceptions carry a reason:
* stale telemetry marks flow AND execution "unknown" -> "Flow/Execution unknown — data stale";
* liveness with no signal is "freshness_unknown" -> "data freshness unknown".
*/
reason?: ReasonCode;
};
export type ReportLink = {
key: LinkKey;
label: string;
url: string;
};
/** a.k.a. ReportDocument — report-agnostic, render-agnostic. */
export type ReportViewModel = {
/** "health" | "cost" | … */
title: string;
/** "prod" */
scope: string;
/** "last 1h" */
period: string;
/** "vs your 7d normal" */
baselineLabel?: string;
/** ISO string — passed in, never read from the clock inside interpret. */
generatedAt: string;
/** live window length in minutes — lets the renderer say "of last 60 min". */
windowMinutes: number;
summary: {
severity: Severity;
statements: SummaryStatement[];
};
findings: Finding[];
metrics: Metric[];
/** dense structured payload for agents. */
facts: Record<string, unknown>;
links: ReportLink[];
/** dominant finding's action + optional "do nothing" option. Max two entries. */
footer: FooterEntry[];
};
// ---------------------------------------------------------------------------
// Interpret-side helpers (produce VM fields, no prose, no IO).
// ---------------------------------------------------------------------------
/** Direction + rounded multiplier of `value` against a `normal` baseline. */
/** Direction and rounded multiplier of `value` against a `normal` baseline. */
export function delta(value: number, normal: number | undefined): Delta {
if (normal === undefined || !Number.isFinite(normal) || normal === 0) {
return { dir: "flat" };
@@ -189,29 +68,37 @@ export function isOk(severity: Severity): boolean {
}
/**
* Trailing contiguous run of buckets that breach `threshold`, in minutes over
* `windowMinutes`. Default breach is at/over threshold (ABOVE, e.g. concurrency
* pinned at the limit); `below: true` counts at/under (BELOW, e.g. running capacity
* idle under a stall floor). `touchesEnd` = the run reaches the latest bucket
* ("(last 40 min)" vs mid-window "(1416h)"). Undefined when nothing breaches.
* Trailing contiguous run of buckets breaching `threshold`, in minutes. `below: true` counts at or
* under it. `bucketMinutes` and `timestampsMs` make it gap-aware; without them the series is gap-free.
*/
export function anomalyWindow(
series: number[],
threshold: number,
windowMinutes: number,
options?: { below?: boolean }
options?: { below?: boolean; bucketMinutes?: number; timestampsMs?: number[] }
): { minutes: number; touchesEnd: boolean } | undefined {
if (series.length === 0) return undefined;
const perBucket = windowMinutes / series.length;
const bucketMinutes = options?.bucketMinutes;
const perBucket =
bucketMinutes !== undefined && bucketMinutes > 0
? bucketMinutes
: windowMinutes / series.length;
const breaches = options?.below ? (v: number) => v <= threshold : (v: number) => v >= threshold;
// A gap larger than about one cadence is a dropped bucket and must not extend the run.
const timestamps = options?.timestampsMs;
const maxGapMs =
timestamps && timestamps.length === series.length && bucketMinutes
? bucketMinutes * 60_000 * 1.5
: undefined;
const adjacent = (i: number) =>
maxGapMs === undefined || i === 0 || timestamps![i] - timestamps![i - 1] <= maxGapMs;
// longest breaching run + whether any run touches the end.
let longest = 0;
let current = 0;
let touchesEnd = false;
for (let i = 0; i < series.length; i++) {
if (breaches(series[i])) {
current++;
current = adjacent(i) ? current + 1 : 1;
longest = Math.max(longest, current);
if (i === series.length - 1) touchesEnd = true;
} else {
@@ -219,8 +106,7 @@ export function anomalyWindow(
}
}
if (longest === 0) return undefined;
// If the run reaches the latest bucket, use the TRAILING length so "(last X min)" is
// accurate — a longer mid-window run must not inflate it.
// Use the trailing length when the run reaches the latest bucket, so a mid-window run can't inflate it.
const runBuckets = touchesEnd ? current : longest;
return { minutes: Math.round(runBuckets * perBucket), touchesEnd };
}
@@ -0,0 +1,36 @@
// Outside the route module because a Remix route may only export loader, action and headers.
import { json } from "@remix-run/server-runtime";
import { ReportFormatSchema, ReportPeriodSchema } from "@trigger.dev/core/v3/schemas";
import { z } from "zod";
import { renderReportAnsi, renderReportMarkdown } from "~/presenters/v3/reports/renderMarkdown";
import { type ReportViewModel } from "~/presenters/v3/reports/report-view-model";
export const ReportParamsSchema = z.object({
key: z.string(),
});
// `period` and `format` come from core, the same definitions the API clients and CLI use.
export const ReportSearchParamsSchema = z.object({
period: ReportPeriodSchema.optional(),
format: ReportFormatSchema.default("markdown"),
});
export type ReportFormatParam = z.infer<typeof ReportFormatSchema>;
/** Render the view model in the requested encoding, with the matching content type. */
export function reportResponse(vm: ReportViewModel, format: ReportFormatParam): Response {
switch (format) {
case "json":
return json(vm, { status: 200 });
case "ansi":
return new Response(renderReportAnsi(vm), {
status: 200,
headers: { "Content-Type": "text/plain; charset=utf-8" },
});
case "markdown":
return new Response(renderReportMarkdown(vm), {
status: 200,
headers: { "Content-Type": "text/markdown; charset=utf-8" },
});
}
}
@@ -0,0 +1,14 @@
// Apart from `reportsApi.server.ts` so that rendering a report doesn't drag the route
// builder — and `env.server` behind it — into everything that serializes one.
import { isReportKey, reportQueryTables } from "~/presenters/v3/reports/report-registry";
import { everyResource } from "~/services/routeBuilders/apiBuilder.server";
/**
* Per-table, not the permissive `{ type: "query", id: "all" }`: a JWT must be scoped to every table
* the report reads, so a partially scoped token can't reach the others.
*/
export function reportAuthResource(key: string) {
// A key that names no report declares no tables, so there is nothing to authorize against.
if (!isReportKey(key)) return everyResource([]);
return everyResource(reportQueryTables(key).map((id) => ({ type: "query", id })));
}
+3 -7
View File
@@ -75,6 +75,7 @@ export const loader = async ({ request }: LoaderFunctionArgs) => {
const features = featuresForRequest(request);
const timezone = await getTimezonePreference(request);
// Deprecated with `AskAI.tsx`: kept so the widget still has its config if it is ever remounted.
const kapa = {
websiteId: env.KAPA_AI_WEBSITE_ID,
};
@@ -170,13 +171,8 @@ export function ErrorBoundary() {
}
export default function App() {
const {
posthogProjectKey,
posthogUiHost,
kapa: _kapa,
themePreference,
themeContrast,
} = useTypedLoaderData<typeof loader>();
const { posthogProjectKey, posthogUiHost, themePreference, themeContrast } =
useTypedLoaderData<typeof loader>();
usePostHog(posthogProjectKey, posthogUiHost);
useSystemThemeSync(themePreference);
// SSR falls back to dark for `system`; the inline script below corrects it
@@ -189,6 +189,11 @@ export default function Page() {
>
Private connection docs
</LinkButton>
{hasPrivateNetworking && canAdd && (
<LinkButton variant="primary/small" LeadingIcon={PlusIcon} to="new">
Add Connection
</LinkButton>
)}
</PageAccessories>
</NavBar>
<PageBody scrollable={true}>
@@ -20,7 +20,7 @@ import { Header2, Header3 } from "~/components/primitives/Headers";
import { Input } from "~/components/primitives/Input";
import { InputGroup } from "~/components/primitives/InputGroup";
import { Label } from "~/components/primitives/Label";
import { NavBar, PageAccessories, PageTitle } from "~/components/primitives/PageHeader";
import { NavBar, PageTitle } from "~/components/primitives/PageHeader";
import { Paragraph } from "~/components/primitives/Paragraph";
import { Select, SelectItem } from "~/components/primitives/Select";
import { prisma } from "~/db.server";
@@ -38,7 +38,6 @@ import {
} from "~/utils/pathBuilder";
import {
ArrowTopRightOnSquareIcon,
BookOpenIcon,
CommandLineIcon,
DocumentTextIcon,
PencilSquareIcon,
@@ -553,15 +552,6 @@ export default function Page() {
text: "Private Connections",
}}
/>
<PageAccessories>
<LinkButton
variant="docs/small"
LeadingIcon={BookOpenIcon}
to={docsPath("private-networking/overview")}
>
Private connection docs
</LinkButton>
</PageAccessories>
</NavBar>
<PageBody scrollable={true}>
<MainHorizontallyCenteredContainer className="max-w-3xl">
@@ -1,6 +1,6 @@
import { getFormProps, getInputProps, useForm } from "@conform-to/react";
import { parseWithZod } from "@conform-to/zod";
import { BookOpenIcon, ShieldCheckIcon, TrashIcon } from "@heroicons/react/20/solid";
import { ShieldCheckIcon, TrashIcon } from "@heroicons/react/20/solid";
import { ShieldExclamationIcon } from "@heroicons/react/24/solid";
import { DialogClose } from "@radix-ui/react-dialog";
import { Form, useActionData, useFetcher } from "@remix-run/react";
@@ -9,7 +9,7 @@ import { useState } from "react";
import { typedjson, useTypedLoaderData } from "remix-typedjson";
import { z } from "zod";
import { PageBody, PageContainer } from "~/components/layout/AppLayout";
import { Button, LinkButton } from "~/components/primitives/Buttons";
import { Button } from "~/components/primitives/Buttons";
import { Callout } from "~/components/primitives/Callout";
import { ClipboardField } from "~/components/primitives/ClipboardField";
import { DateTime } from "~/components/primitives/DateTime";
@@ -46,7 +46,8 @@ import {
revokePersonalAccessToken,
} from "~/services/personalAccessToken.server";
import { requireUserId } from "~/services/session.server";
import { docsPath, personalAccessTokensPath } from "~/utils/pathBuilder";
import { personalAccessTokensPath } from "~/utils/pathBuilder";
import { pageMeta } from "~/utils/pageTitle";
export const meta = pageMeta("Personal Access Tokens");
@@ -246,13 +247,6 @@ export default function Page() {
<NavBar>
<PageTitle title="Personal Access Tokens" />
<PageAccessories>
<LinkButton
LeadingIcon={BookOpenIcon}
to={docsPath("management/overview#personal-access-token-pat")}
variant="docs/small"
>
Personal Access Token docs
</LinkButton>
<Dialog>
<DialogTrigger asChild>
<Button variant="primary/small">Create new token</Button>
@@ -9,7 +9,7 @@ import { rbac } from "~/services/rbac.server";
// with the PAT. The default is short, but the ceiling allows long-lived tokens
// for callers that need them (e.g. a long-running integration).
const DEFAULT_UAT_TTL_SECONDS = 60 * 60; // 1 hour
const MAX_UAT_TTL_SECONDS = 365 * 24 * 60 * 60; // 365 days
const MAX_UAT_TTL_SECONDS = 7 * 24 * 60 * 60; // 7 days
// Mint a short-lived delegated user-actor token (`tr_uat_`) from a personal
// access token. A UAT is a strict downgrade of the PAT: same user identity,
@@ -68,6 +68,8 @@ export async function action({ request }: ActionFunctionArgs) {
const token = await signUserActorToken(env.SESSION_SECRET, {
userId: patAuth.userId,
client: body.client ?? "personal-access-token",
// Bind the token to its source PAT so revoking the PAT invalidates it.
pat: patAuth.tokenId,
cap: body.cap,
// Absolute exp (seconds since epoch). jose treats a number as absolute.
expirationTime: Math.floor(Date.now() / 1000) + ttlSeconds,
@@ -0,0 +1,40 @@
import { json, type LoaderFunctionArgs } from "@remix-run/server-runtime";
import { z } from "zod";
import { orgAllowsDashboardAgentTurnEvals } from "~/services/dashboardAgentEvalPolicy.server";
import { authenticateUatOrApiRequest } from "~/services/uatRoutePreamble.server";
/**
* The gate the agent checks before it judges a turn. Only its own delegated user-actor
* token is accepted, and the org is scoped to the token's user. Answers `false` rather than
* an error whenever the setting can't be resolved, so the agent's fail-closed path is the
* same for "off" and "unknown".
*/
const QuerySchema = z.object({ organizationId: z.string().min(1) });
// obs-map-disable request-context -- the only call here that can throw catches its own failure
// and logs it with organizationId, in dashboardAgentEvalPolicy.server.ts; the rest are early
// returns, and the one failure that does reach the boundary is auth, where no tenant is known yet.
export async function loader({ request }: LoaderFunctionArgs) {
const authentication = await authenticateUatOrApiRequest(request);
if (!authentication?.userActor) {
return json({ error: "Invalid or missing access token" }, { status: 401 });
}
if (authentication.userActor.client !== "dashboard-agent") {
return json({ error: "Not allowed", code: "forbidden_client" }, { status: 403 });
}
const parsed = QuerySchema.safeParse(
Object.fromEntries(new URL(request.url).searchParams.entries())
);
if (!parsed.success) {
return json({ error: "organizationId is required" }, { status: 400 });
}
const turnEvalsEnabled = await orgAllowsDashboardAgentTurnEvals({
userId: authentication.userActor.userId,
organizationId: parsed.data.organizationId,
});
return json({ turnEvalsEnabled });
}
+42 -24
View File
@@ -11,44 +11,62 @@ import {
import { extractDomain, faviconUrl } from "~/utils/favicon";
// Identity-only: lists the caller's own orgs, so no authorization gate.
export const loader = createLoaderPATApiRoute({}, async ({ authentication }) => {
const orgs = await prisma.organization.findMany({
where: {
deletedAt: null,
members: {
some: {
userId: authentication.userId,
export const loader = createLoaderPATApiRoute(
{ identityOnly: true },
async ({ authentication }) => {
const orgs = await prisma.organization.findMany({
where: {
deletedAt: null,
members: {
some: {
userId: authentication.userId,
},
},
},
},
});
});
if (!orgs) {
return json({ error: "Orgs not found" }, { status: 404 });
if (!orgs) {
return json({ error: "Orgs not found" }, { status: 404 });
}
const result: GetOrgsResponseBody = orgs.map((org) => ({
id: org.id,
title: org.title,
slug: org.slug,
createdAt: org.createdAt,
}));
return json(result);
}
);
const result: GetOrgsResponseBody = orgs.map((org) => ({
id: org.id,
title: org.title,
slug: org.slug,
createdAt: org.createdAt,
}));
return json(result);
});
// No org exists yet, so no authorization gate; any authenticated user can
// create an org and becomes its ADMIN.
// No org exists yet, so there is nothing to scope a route-level gate to; any authenticated user
// can create an org and becomes its ADMIN. A narrowly-capped delegated token (which cannot
// `manage`) is still refused rather than inheriting its user's full reach — but only for
// user-actor tokens, so an ordinary PAT is unaffected.
export const action = createActionPATApiRoute(
{
method: "POST",
body: CreateOrgRequestBody,
},
async ({ body, authentication }) => {
async ({ body, authentication, ability }) => {
if (env.ORG_CREATION_API_ENABLED !== "1") {
return json({ error: "Not found" }, { status: 404 });
}
// After the env gate: an install with the API disabled should 404, not 403.
if (authentication.userActor && !ability.can("manage", { type: "organization" })) {
return json(
{
error: "Unauthorized",
code: "unauthorized",
param: "access_token",
type: "authorization",
},
{ status: 403 }
);
}
// Mirror the dashboard: stash companyUrl/companySize as onboarding data and
// derive the org avatar from the company domain's favicon.
const onboardingData: Record<string, string> = {};
@@ -1,6 +1,14 @@
import { type ActionFunctionArgs, json } from "@remix-run/node";
import { generateJWT as internal_generateJWT } from "@trigger.dev/core/v3";
import { isUserActorToken, verifyUserActorToken } from "@trigger.dev/rbac";
import {
buildJwtAbility,
CAPLESS_USER_ACTOR_SCOPES,
isUserActorToken,
scopesWithinAbility,
verifyUserActorToken,
type UserActorClaims,
} from "@trigger.dev/rbac";
import parseDuration from "parse-duration";
import { z } from "zod";
import {
authenticatedEnvironmentForAuthentication,
@@ -9,6 +17,8 @@ import {
type AuthenticationResult,
} from "~/services/apiAuth.server";
import { env as appEnv } from "~/env.server";
import { assertUserActorEnvironment } from "~/services/userActorEnvironment.server";
import { assertSourcePatActive } from "~/services/personalAccessToken.server";
import { logger } from "~/services/logger.server";
import { authorizePatEnvironmentAccess } from "~/services/environmentVariableApiAccess.server";
@@ -26,6 +36,27 @@ const RequestBodySchema = z.object({
expirationTime: z.union([z.number(), z.string()]).optional(),
});
// A requested `expirationTime` above this (epoch seconds, ~2001) is an absolute
// timestamp; a smaller number is a relative offset in seconds.
const EXPIRY_EPOCH_THRESHOLD_SECONDS = 1_000_000_000;
const DEFAULT_EXPIRY = "1h";
// Resolve the requested expiry to an absolute epoch-second timestamp so it can be
// clamped against a delegated token's own expiry.
function resolveRequestedExpirySeconds(
expirationTime: number | string | undefined,
nowSec: number
): number {
if (typeof expirationTime === "number") {
return expirationTime > EXPIRY_EPOCH_THRESHOLD_SECONDS
? expirationTime
: nowSec + expirationTime;
}
const durationMs = parseDuration(expirationTime ?? DEFAULT_EXPIRY);
const seconds = durationMs != null ? Math.floor(durationMs / 1000) : 60 * 60;
return nowSec + seconds;
}
export async function action({ request, params }: ActionFunctionArgs) {
try {
const bearer = request.headers
@@ -41,14 +72,20 @@ export async function action({ request, params }: ActionFunctionArgs) {
// minted env JWT below.
let uatCap: string[] | undefined;
let userActorId: string | undefined;
let userActor: UserActorClaims | undefined;
let authenticationResult: AuthenticationResult | undefined;
if (isUat) {
const claims = await verifyUserActorToken(appEnv.SESSION_SECRET, bearer!);
if (!claims) {
return json({ error: "Invalid or Missing Access Token" }, { status: 401 });
}
// A token minted from a PAT dies with it — the PAT must still be live.
if (!(await assertSourcePatActive(claims))) {
return json({ error: "Invalid or Missing Access Token" }, { status: 401 });
}
uatCap = claims.cap;
userActorId = claims.userId;
userActor = claims;
// The env lookup keys purely on the user, identical to a PAT.
authenticationResult = {
type: "personalAccessToken",
@@ -82,6 +119,9 @@ export async function action({ request, params }: ActionFunctionArgs) {
triggerBranch
);
// A user-actor token signed for one environment mints only for that one.
assertUserActorEnvironment(userActor, runtimeEnv.id);
// This mints a JWT signed with the environment's secret key. For a PAT
// (a user), gate it on env-tier read:apiKeys so a restricted role can't
// obtain deployed-environment credentials (and therefore can't deploy).
@@ -107,19 +147,25 @@ export async function action({ request, params }: ActionFunctionArgs) {
// The env JWT carries scopes only — downstream auth builds its ability
// from them with no role context. So for a user-actor token we ceiling
// the scopes by the token's own cap here (a read-only agent token can't
// widen its grant through the exchange) and stamp the user via `act` so
// the minted env JWT stays attributable. The cap is a ceiling, not a
// replacement: intersect what the caller asked for with the cap (or use
// the full cap if they asked for nothing). No cap → the request passes
// through, same as a PAT.
// the scopes here (a read-only agent token can't widen its grant through
// the exchange) and stamp the user via `act` so the minted env JWT stays
// attributable. A capless token's ceiling is read-only, never full access.
// The ceiling is applied through the scope grammar, not literal membership:
// `read:all` is a wildcard no literal request string equals, so a literal
// filter against it would wrongly deny every read.
const requestedScopes = parsedBody.data.claims?.scopes;
const scopes =
isUat && uatCap
? requestedScopes && requestedScopes.length > 0
? requestedScopes.filter((scope) => uatCap.includes(scope))
: uatCap
: requestedScopes;
let scopes: string[] | undefined;
if (isUat) {
const ceiling = uatCap && uatCap.length > 0 ? uatCap : CAPLESS_USER_ACTOR_SCOPES;
const ability = buildJwtAbility(ceiling);
const scopeGranted = (scope: string) => scopesWithinAbility([scope], ability).ok;
scopes =
requestedScopes && requestedScopes.length > 0
? requestedScopes.filter(scopeGranted)
: ceiling;
} else {
scopes = requestedScopes;
}
// Attribution: stamp the acting user on the minted env JWT. A UAT carries
// its user as `userActorId`; a PAT exchange resolves the user from the
@@ -136,13 +182,24 @@ export async function action({ request, params }: ActionFunctionArgs) {
sub: runtimeEnv.id,
pub: true,
...(scopes ? { scopes } : {}),
...(actorUserId ? { act: { sub: actorUserId } } : {}),
...(actorUserId
? { act: { sub: actorUserId, client: userActor?.client ?? "personal-access-token" } }
: {}),
};
// A delegated token can't mint a longer-lived JWT than itself: clamp the
// requested expiry to the token's own `exp`. Non-UAT callers are unchanged.
const nowSec = Math.floor(Date.now() / 1000);
const requestedAbsSec = resolveRequestedExpirySeconds(parsedBody.data.expirationTime, nowSec);
const expirationTime =
isUat && userActor?.expiresAt !== undefined
? Math.min(requestedAbsSec, userActor.expiresAt)
: (parsedBody.data.expirationTime ?? DEFAULT_EXPIRY);
const jwt = await internal_generateJWT({
secretKey: runtimeEnv.apiKey,
payload: claims,
expirationTime: parsedBody.data.expirationTime ?? "1h",
expirationTime,
});
return json({ token: jwt });
@@ -1,24 +1,20 @@
import { json, type LoaderFunctionArgs } from "@remix-run/server-runtime";
import { isUserActorToken, verifyUserActorToken } from "@trigger.dev/rbac";
import { z } from "zod";
import { env as $env } from "~/env.server";
import {
type AuthenticationResult,
authenticatedEnvironmentForAuthentication,
authenticateRequest,
} from "~/services/apiAuth.server";
import { authenticatedEnvironmentForAuthentication } from "~/services/apiAuth.server";
import {
resolveDashboardAgentRepoSnapshot,
resolveRunCommit,
} from "~/services/dashboardAgent.server";
import { authorizePatEnvironmentAccess } from "~/services/environmentVariableApiAccess.server";
import { logger } from "~/services/logger.server";
import { authenticateUatOrApiRequest } from "~/services/uatRoutePreamble.server";
// Resolve a signed source-archive pointer for the project's connected repo, used
// by the dashboard agent's code tools. With `?runId=run_...` it pins to the
// commit that run's deployed version came from (run-SHA pinning); without it,
// the tracked branch head. The GitHub token never leaves the server, only the
// short-lived signed URL is returned. Auth mirrors the worker-by-tag route: a
// delegated user-actor token authenticates as its user (identity-only).
// short-lived signed URL is returned. A delegated user-actor token authenticates as its user and
// is gated on that user's env-tier role, like the JWT exchange.
const ParamsSchema = z.object({
projectRef: z.string(),
@@ -27,23 +23,8 @@ const ParamsSchema = z.object({
export async function loader({ request, params }: LoaderFunctionArgs) {
try {
const bearer = request.headers
.get("Authorization")
?.replace(/^Bearer /, "")
.trim();
let authenticationResult: AuthenticationResult | undefined;
if (bearer && isUserActorToken(bearer)) {
const claims = await verifyUserActorToken($env.SESSION_SECRET, bearer);
if (!claims) return json({ error: "Invalid or Missing Access Token" }, { status: 401 });
authenticationResult = { type: "personalAccessToken", result: { userId: claims.userId } };
} else {
authenticationResult = await authenticateRequest(request, {
personalAccessToken: true,
organizationAccessToken: true,
apiKey: false,
});
}
if (!authenticationResult) {
const authentication = await authenticateUatOrApiRequest(request);
if (!authentication) {
return json({ error: "Invalid or Missing Access Token" }, { status: 401 });
}
@@ -53,12 +34,25 @@ export async function loader({ request, params }: LoaderFunctionArgs) {
const triggerBranch = request.headers.get("x-trigger-branch") ?? undefined;
const runtimeEnv = await authenticatedEnvironmentForAuthentication(
authenticationResult,
authentication.authenticationResult,
projectRef,
env,
triggerBranch
);
// The signed URL exposes the project's whole source tree, so gate it like the environment's
// other secrets: env-tier `read:apiKeys`, the same check the JWT exchange applies.
const denied = await authorizePatEnvironmentAccess({
request,
authType: authentication.authenticationResult.type,
organizationId: runtimeEnv.organizationId,
projectId: runtimeEnv.project.id,
envType: runtimeEnv.type,
resource: "apiKeys",
action: "read",
});
if (denied) return denied;
const runId = new URL(request.url).searchParams.get("runId") ?? undefined;
let ref: string | undefined;
@@ -0,0 +1,113 @@
import { json, type LoaderFunctionArgs } from "@remix-run/server-runtime";
import { z } from "zod";
import { $replica } from "~/db.server";
import {
authenticatedEnvironmentForAuthentication,
type AuthenticatedEnvironment,
} from "~/services/apiAuth.server";
import { resolveRunCommit } from "~/services/dashboardAgent.server";
import { authorizePatEnvironmentAccess } from "~/services/environmentVariableApiAccess.server";
import { logger } from "~/services/logger.server";
import { authenticateUatOrApiRequest } from "~/services/uatRoutePreamble.server";
/** The commit a run's deployed version came from, plus that deployment's git metadata. */
const ParamsSchema = z.object({
projectRef: z.string(),
env: z.enum(["dev", "staging", "prod", "preview"]),
runId: z.string(),
});
type GitMetaBlob = {
source?: string;
commitAuthorName?: string;
commitMessage?: string;
commitRef?: string;
remoteUrl?: string;
ghUsername?: string;
pullRequestNumber?: number;
pullRequestTitle?: string;
pullRequestState?: string;
};
export async function loader({ request, params }: LoaderFunctionArgs) {
// Hoisted so a failure below can name the tenant it happened to.
let runtimeEnv: AuthenticatedEnvironment | undefined;
try {
const authentication = await authenticateUatOrApiRequest(request);
if (!authentication) {
return json({ error: "Invalid or Missing Access Token" }, { status: 401 });
}
const parsed = ParamsSchema.safeParse(params);
if (!parsed.success) return json({ error: "Invalid Params" }, { status: 400 });
const { projectRef, env, runId } = parsed.data;
const triggerBranch = request.headers.get("x-trigger-branch") ?? undefined;
runtimeEnv = await authenticatedEnvironmentForAuthentication(
authentication.authenticationResult,
projectRef,
env,
triggerBranch
);
// The answer is a deployment's git metadata, so it's gated like the deployments list.
const denied = await authorizePatEnvironmentAccess({
request,
authType: authentication.authenticationResult.type,
organizationId: runtimeEnv.organizationId,
projectId: runtimeEnv.project.id,
envType: runtimeEnv.type,
resource: "deployments",
action: "read",
});
if (denied) return denied;
const commit = await resolveRunCommit(runtimeEnv.id, runId);
if (!commit) {
return json(
{ error: "That run has no deployed commit (it may be a dev run)." },
{ status: 404 }
);
}
const deployment = await $replica.workerDeployment.findFirst({
where: { environmentId: runtimeEnv.id, version: commit.version },
select: { git: true, shortCode: true, deployedAt: true },
});
const git = (deployment?.git ?? undefined) as GitMetaBlob | undefined;
return json({
runId,
version: commit.version,
sha: commit.sha,
dirty: commit.dirty,
shortCode: deployment?.shortCode,
deployedAt: deployment?.deployedAt ?? undefined,
git: git
? {
source: git.source,
commitMessage: git.commitMessage,
commitAuthorName: git.commitAuthorName,
commitRef: git.commitRef,
remoteUrl: git.remoteUrl,
ghUsername: git.ghUsername,
pullRequestNumber: git.pullRequestNumber,
pullRequestTitle: git.pullRequestTitle,
pullRequestState: git.pullRequestState,
}
: undefined,
});
} catch (error) {
if (error instanceof Response) throw error;
logger.error("Failed to resolve run commit", {
error,
environmentId: runtimeEnv?.id,
projectId: runtimeEnv?.project.id,
organizationId: runtimeEnv?.organizationId,
});
return json({ error: "Internal Server Error" }, { status: 500 });
}
}
@@ -1,16 +1,14 @@
import { json, type LoaderFunctionArgs } from "@remix-run/server-runtime";
import { type GetWorkerByTagResponse } from "@trigger.dev/core/v3/schemas";
import { isUserActorToken, verifyUserActorToken } from "@trigger.dev/rbac";
import { z } from "zod";
import { $replica } from "~/db.server";
import { env as $env } from "~/env.server";
import {
authenticatedEnvironmentForAuthentication,
authenticateRequest,
branchNameFromRequest,
type AuthenticationResult,
} from "~/services/apiAuth.server";
import { logger } from "~/services/logger.server";
import { authenticateUatOrApiRequest } from "~/services/uatRoutePreamble.server";
import { v3RunsPath } from "~/utils/pathBuilder";
import { findCurrentWorkerFromEnvironment } from "~/v3/models/workerDeployment.server";
@@ -24,32 +22,11 @@ type ParamsSchema = z.infer<typeof ParamsSchema>;
export async function loader({ request, params }: LoaderFunctionArgs) {
try {
// A delegated user-actor token authenticates as its user, like a PAT.
// Resolve it here (the shared `authenticateRequest` deliberately doesn't
// accept UATs) so the dashboard agent can list a project's deployed tasks
// on the user's behalf. Identity-only, same as the PAT path below — there's
// no ability check on this route, so the cap isn't enforced here (matches
// PAT behavior).
const bearer = request.headers
.get("Authorization")
?.replace(/^Bearer /, "")
.trim();
let authenticationResult: AuthenticationResult | undefined;
if (bearer && isUserActorToken(bearer)) {
const claims = await verifyUserActorToken($env.SESSION_SECRET, bearer);
if (!claims) {
return json({ error: "Invalid or Missing Access Token" }, { status: 401 });
}
authenticationResult = { type: "personalAccessToken", result: { userId: claims.userId } };
} else {
authenticationResult = await authenticateRequest(request, {
personalAccessToken: true,
organizationAccessToken: true,
apiKey: false,
});
}
// Accepts a user-actor token as well as a PAT. There's no ability check here, so the
// token's cap isn't enforced (matches PAT behavior).
const authentication = await authenticateUatOrApiRequest(request);
if (!authenticationResult) {
if (!authentication) {
return json({ error: "Invalid or Missing Access Token" }, { status: 401 });
}
@@ -63,7 +40,7 @@ export async function loader({ request, params }: LoaderFunctionArgs) {
const triggerBranch = branchNameFromRequest(request);
const runtimeEnv = await authenticatedEnvironmentForAuthentication(
authenticationResult,
authentication.authenticationResult,
projectRef,
env,
triggerBranch
@@ -4,6 +4,7 @@ import { z } from "zod";
import { $replica } from "~/db.server";
import { findProjectByRef } from "~/models/project.server";
import { createLoaderPATApiRoute } from "~/services/routeBuilders/apiBuilder.server";
import { resolveUserActorEnvironmentScope } from "~/services/userActorEnvironment.server";
import { sortEnvironments } from "~/utils/environmentSort";
import { isBranchableEnvironment } from "~/utils/branchableEnvironment";
@@ -35,12 +36,18 @@ export const loader = createLoaderPATApiRoute(
return json({ error: "Project not found" }, { status: 404 });
}
// A delegated token signed for one environment only ever lists that one.
const scope = await resolveUserActorEnvironmentScope(authentication.userActor, {
projectId: project.id,
});
const environments = await $replica.runtimeEnvironment.findMany({
where: {
projectId: project.id,
// Only base/parent environments. Branch children (preview branches)
// are excluded — syncs target the parent and branches override elsewhere.
parentEnvironmentId: null,
// A scoped token lists exactly the environment it was signed for, branch child or not —
// otherwise a token minted on a preview branch would list nothing at all. Unscoped
// callers get base/parent environments only: syncs target the parent.
...(scope.scoped ? { id: scope.environmentId } : { parentEnvironmentId: null }),
archivedAt: null,
OR: [
{ type: { in: ["STAGING", "PRODUCTION", "PREVIEW"] } },
@@ -7,6 +7,7 @@ import {
ApiRunListSearchParams,
} from "~/presenters/v3/ApiRunListPresenter.server";
import { createLoaderPATApiRoute } from "~/services/routeBuilders/apiBuilder.server";
import { resolveUserActorEnvironmentScope } from "~/services/userActorEnvironment.server";
const ParamsSchema = z.object({
projectRef: z.string(),
@@ -39,8 +40,20 @@ export const loader = createLoaderPATApiRoute(
return json({ error: "Project not found" }, { status: 404 });
}
// A delegated token signed for one environment only ever lists that environment's runs, and a
// request filter naming another one is refused rather than overridden.
const scope = await resolveUserActorEnvironmentScope(authentication.userActor, {
projectId: project.id,
requestedEnvironmentSlugs: searchParams["filter[env]"],
});
const presenter = new ApiRunListPresenter();
const result = await presenter.call(project, searchParams, apiVersion);
const result = await presenter.call(
project,
searchParams,
apiVersion,
scope.scoped ? { id: scope.environmentId, organizationId: scope.organizationId } : undefined
);
if (!result) {
return json({ data: [] });
+39 -36
View File
@@ -4,44 +4,47 @@ import { prisma } from "~/db.server";
import { createLoaderPATApiRoute } from "~/services/routeBuilders/apiBuilder.server";
// Identity-only: lists projects across the caller's orgs, so no authorization gate.
export const loader = createLoaderPATApiRoute({}, async ({ authentication }) => {
const projects = await prisma.project.findMany({
where: {
organization: {
deletedAt: null,
members: {
some: {
userId: authentication.userId,
export const loader = createLoaderPATApiRoute(
{ identityOnly: true },
async ({ authentication }) => {
const projects = await prisma.project.findMany({
where: {
organization: {
deletedAt: null,
members: {
some: {
userId: authentication.userId,
},
},
},
version: "V3",
deletedAt: null,
},
version: "V3",
deletedAt: null,
},
include: {
organization: true,
defaultWorkerGroup: { select: { name: true } },
},
});
include: {
organization: true,
defaultWorkerGroup: { select: { name: true } },
},
});
if (!projects) {
return json({ error: "Projects not found" }, { status: 404 });
if (!projects) {
return json({ error: "Projects not found" }, { status: 404 });
}
const result: GetProjectsResponseBody = projects.map((project) => ({
id: project.id,
externalRef: project.externalRef,
name: project.name,
slug: project.slug,
createdAt: project.createdAt,
defaultRegion: project.defaultWorkerGroup?.name ?? null,
organization: {
id: project.organization.id,
title: project.organization.title,
slug: project.organization.slug,
createdAt: project.organization.createdAt,
},
}));
return json(result);
}
const result: GetProjectsResponseBody = projects.map((project) => ({
id: project.id,
externalRef: project.externalRef,
name: project.name,
slug: project.slug,
createdAt: project.createdAt,
defaultRegion: project.defaultWorkerGroup?.name ?? null,
organization: {
id: project.organization.id,
title: project.organization.title,
slug: project.organization.slug,
createdAt: project.organization.createdAt,
},
}));
return json(result);
});
);
+11 -1
View File
@@ -7,6 +7,7 @@ import { logger } from "~/services/logger.server";
import { rowsToCSV } from "~/utils/dataExport";
import { detectQueryTables } from "~/v3/detectQueryTables";
import { querySchemas } from "~/v3/querySchemas";
import { queryScopeCeilingFor, resolveQueryScope } from "~/v3/queryScope";
const BodySchema = z.object({
query: z.string(),
@@ -51,11 +52,20 @@ const { action, loader } = createActionApiRoute(
const { query, scope, period, from, to, format } = body;
const env = authentication.environment;
// The credential's own scope is the ceiling, not the body's. See queryScope.ts.
const resolvedScope = resolveQueryScope({
ceiling: queryScopeCeilingFor(authentication.type),
requested: scope as QueryScope,
});
if (!resolvedScope.ok) {
return json({ error: resolvedScope.error }, { status: 403 });
}
const queryResult = await executeQuery({
name: "api-query",
query,
userAuthoredQuery: true,
scope: scope as QueryScope,
scope: resolvedScope.scope,
organizationId: env.organization.id,
projectId: env.project.id,
environmentId: env.id,
@@ -0,0 +1,144 @@
import { json } from "@remix-run/server-runtime";
import { z } from "zod";
import { clickhouseFactory } from "~/services/clickhouse/clickhouseFactoryInstance.server";
import { logger } from "~/services/logger.server";
import { createLoaderApiRoute } from "~/services/routeBuilders/apiBuilder.server";
import { queueDepthSeries } from "~/v3/queueDepthSeries";
/**
* Per-queue metrics over a window. `queueParam` is the queue name; `?type=task` (the default)
* adds the `task/` prefix. An unknown queue returns zeroed metrics, not a 404.
*/
const UNIT_MS: Record<string, number> = { s: 1e3, m: 6e4, h: 36e5, d: 864e5, w: 6048e5 };
const MAX_PERIOD_MS = 7 * UNIT_MS.d;
const PeriodSchema = z
.string()
.regex(/^[1-9]\d*[smhdw]$/, "period must be a shorthand like '15m', '1h', or '24h'")
.refine(
(p) => Number(p.slice(0, -1)) * UNIT_MS[p.slice(-1)] <= MAX_PERIOD_MS,
"period is too large (max 7d)"
);
const SearchParamsSchema = z.object({
type: z.enum(["task", "custom"]).default("task"),
period: PeriodSchema.default("1h"),
});
const TREND_POINTS = 12;
function periodMs(period: string): number {
return Number(period.slice(0, -1)) * UNIT_MS[period.slice(-1)];
}
function formatClickhouseDateTime(date: Date): string {
return date.toISOString().slice(0, 19).replace("T", " ");
}
function finiteOrNull(value: number | undefined): number | null {
return typeof value === "number" && Number.isFinite(value) ? value : null;
}
export const loader = createLoaderApiRoute(
{
params: z.object({
queueParam: z.string().transform((val) => val.replace(/%2F/g, "/")),
}),
searchParams: SearchParamsSchema,
allowJWT: true,
corsStrategy: "none",
findResource: async () => 1, // dummy — the queue name isn't resolved against Postgres
authorization: {
action: "read",
resource: () => ({ type: "query", id: "queue_metrics" }),
},
},
async ({ params, searchParams, authentication }) => {
// Already decoded by Remix and the schema; decoding again would 500 on a literal "%".
const name = params.queueParam;
const queue = searchParams.type === "task" && !name.startsWith("task/") ? `task/${name}` : name;
const windowMs = periodMs(searchParams.period);
const windowMinutes = windowMs / 60_000;
const bucketSeconds = Math.max(60, Math.round(windowMs / 1000 / TREND_POINTS));
// Snap both bounds to the bucket grid so repeated calls share ClickHouse cache entries.
const bucketIntervalMs = bucketSeconds * 1000;
const endMs = Math.ceil(Date.now() / bucketIntervalMs) * bucketIntervalMs;
const startMs = endMs - windowMs;
// The trend grid covers whole buckets, so a period that isn't a bucket multiple still lines up.
const gridStartMs = Math.floor(startMs / bucketIntervalMs) * bucketIntervalMs;
const numBuckets = Math.round((endMs - gridStartMs) / bucketIntervalMs);
try {
const clickhouse = await clickhouseFactory.getClickhouseForOrganization(
authentication.environment.organizationId,
"query"
);
const ids = {
organizationId: authentication.environment.organizationId,
projectId: authentication.environment.projectId,
environmentId: authentication.environment.id,
queueNames: [queue],
startTime: formatClickhouseDateTime(new Date(startMs)),
endTime: formatClickhouseDateTime(new Date(endMs)),
};
const [summaryResult, trendResult] = await Promise.all([
clickhouse.queueMetrics.listSummary(ids),
clickhouse.queueMetrics.depthSparklines({ ...ids, bucketSeconds }),
]);
const [summaryError, summaryRows] = summaryResult;
const [trendError, trendRows] = trendResult;
if (summaryError || trendError) {
logger.warn("Failed to read queue metrics", {
summaryError: summaryError?.message,
trendError: trendError?.message,
organizationId: ids.organizationId,
projectId: ids.projectId,
environmentId: ids.environmentId,
});
return json({ error: "Queue metrics are unavailable right now." }, { status: 503 });
}
const summary = summaryRows?.[0];
const startedCount = summary?.started_count ?? 0;
return json({
queue,
period: searchParams.period,
from: new Date(startMs).toISOString(),
to: new Date(endMs).toISOString(),
waitMs: {
p50: finiteOrNull(summary?.p50_wait_ms),
p95: finiteOrNull(summary?.p95_wait_ms),
},
peakQueued: summary?.peak_queued ?? 0,
startedCount,
startedPerMin: Number((startedCount / windowMinutes).toFixed(2)),
throttledCount: summary?.throttled_count ?? 0,
bucketIntervalMs,
// Oldest first, one point per bucket: a bucket with no sample carries the previous depth.
depthTrend: queueDepthSeries(trendRows ?? [], {
startMs: gridStartMs,
bucketIntervalMs,
numBuckets,
}).depth,
});
} catch (error) {
// Rethrow Responses: swallowing one would turn it into a 500.
if (error instanceof Response) throw error;
logger.error("Failed to read queue metrics", {
error,
queue,
organizationId: authentication.environment.organizationId,
projectId: authentication.environment.projectId,
environmentId: authentication.environment.id,
});
return json({ error: "Something went wrong, please try again." }, { status: 500 });
}
}
);
+13 -52
View File
@@ -1,53 +1,28 @@
import { json } from "@remix-run/server-runtime";
import { z } from "zod";
import { ReportPresenter } from "~/presenters/v3/reports/ReportPresenter.server";
import { isReportKey, REPORT_KEYS } from "~/presenters/v3/reports/report-registry";
import { renderReportAnsi, renderReportMarkdown } from "~/presenters/v3/reports/renderMarkdown";
import {
ReportParamsSchema,
ReportSearchParamsSchema,
reportResponse,
} from "~/presenters/v3/reports/reportsApi.server";
import { reportAuthResource } from "~/presenters/v3/reports/reportsApiAuth.server";
import { logger } from "~/services/logger.server";
import { createLoaderApiRoute, everyResource } from "~/services/routeBuilders/apiBuilder.server";
const ParamsSchema = z.object({
key: z.string(),
});
/**
* The query tables the reports read. Authorize per-table (like api.v1.query.ts) rather than
* the permissive `{ type: "query", id: "all" }`: a JWT must be scoped to every table a report
* touches, so a token scoped to only some tables can't fetch a report that reads others. This
* is the union across reports; `health` reads all three.
*/
const REPORT_QUERY_TABLES = ["runs", "env_metrics", "queue_metrics"] as const;
/** Canonical shorthand ("1h" / "30m" / "7d") with an upper bound, so the public API rejects
* garbage and absurd ranges (e.g. "999999999d") itself rather than relying on downstream clip. */
const UNIT_MS: Record<string, number> = { s: 1e3, m: 6e4, h: 36e5, d: 864e5, w: 6048e5 };
const MAX_PERIOD_MS = 90 * UNIT_MS.d;
const PeriodSchema = z
.string()
.regex(/^[1-9]\d*[smhdw]$/, "period must be a shorthand like '1h', '30m', or '7d'")
.refine(
(p) => Number(p.slice(0, -1)) * UNIT_MS[p.slice(-1)] <= MAX_PERIOD_MS,
"period is too large (max 90d)"
);
const SearchParamsSchema = z.object({
period: PeriodSchema.optional(),
// markdown (default) for CLI/MCP · json (the raw VM) for web · ansi for a colour terminal.
format: z.enum(["markdown", "json", "ansi"]).default("markdown"),
});
import { createLoaderApiRoute } from "~/services/routeBuilders/apiBuilder.server";
// Only `loader` may be exported here; the vite build flags server code reachable from a
// non-loader export. Schemas and helpers live in reportsApi.server.ts.
export const loader = createLoaderApiRoute(
{
params: ParamsSchema,
searchParams: SearchParamsSchema,
params: ReportParamsSchema,
searchParams: ReportSearchParamsSchema,
// The MCP `get_report` tool calls this with a scoped JWT (read:query), so JWT auth
// must be allowed — same as api.v1.query.ts.
allowJWT: true,
findResource: async () => 1, // dummy — report key validated in the handler
authorization: {
action: "read",
// Per-table, not `id: "all"`: a JWT must be scoped to every query table the report reads.
resource: () => everyResource(REPORT_QUERY_TABLES.map((id) => ({ type: "query", id }))),
resource: (_, params) => reportAuthResource(params.key),
},
},
async ({ params, searchParams, authentication }) => {
@@ -70,21 +45,7 @@ export const loader = createLoaderApiRoute(
return json({ error: `Unknown report "${params.key}".` }, { status: 404 });
}
if (searchParams.format === "json") {
return json(vm, { status: 200 });
}
if (searchParams.format === "ansi") {
return new Response(renderReportAnsi(vm), {
status: 200,
headers: { "Content-Type": "text/plain; charset=utf-8" },
});
}
return new Response(renderReportMarkdown(vm), {
status: 200,
headers: { "Content-Type": "text/markdown; charset=utf-8" },
});
return reportResponse(vm, searchParams.format);
} catch (error) {
logger.error("Failed to render report", { error, key: params.key });
return json({ error: "Something went wrong, please try again." }, { status: 500 });
@@ -38,14 +38,15 @@ export async function loader({ params, request }: LoaderFunctionArgs) {
const query = url.searchParams.get("q");
if (!query) {
return new Response("No query", { status: 404 });
return new Response("No query", { status: 400 });
}
const newUrl = new URL(
v3EnvironmentPath({ slug: project.organization.slug }, { slug: project.slug }, { slug: "dev" }),
env.LOGIN_ORIGIN
);
newUrl.searchParams.set("aiHelp", query);
// The `ask` param is picked up in the environment layout (`useDashboardAgentOpenRequests`).
newUrl.searchParams.set("ask", query);
return redirect(newUrl.toString());
}
@@ -1,29 +1,28 @@
import { type ActionFunctionArgs, json } from "@remix-run/server-runtime";
import { $replica } from "~/db.server";
import {
checkMessageParts,
declaredBodyBytes,
exceedsMessageBodyBytes,
MAX_MESSAGE_BODY_BYTES,
MESSAGE_TOO_LARGE_CODE,
MESSAGE_TOO_LARGE_ERROR,
} from "~/components/dashboard-agent/message-limits";
import { findProjectBySlug } from "~/models/project.server";
import { findEnvironmentBySlug } from "~/models/runtimeEnvironment.server";
import {
dashboardAgentApiOrigin,
mintDashboardAgentUserActorToken,
resolveDashboardAgentRepoSnapshot,
} from "~/services/dashboardAgent.server";
import { dashboardAgentEnvironmentAddress } from "~/services/dashboardAgentEnvironmentAddress.server";
import { logger } from "~/services/logger.server";
import { requireUser } from "~/services/session.server";
import { readBoundedBodyText } from "~/utils/boundedRequestBody.server";
import { EnvironmentParamSchema } from "~/utils/pathBuilder";
import { canAccessDashboardAgent } from "~/v3/canAccessDashboardAgent.server";
// Same-origin proxy for the chat "in"/append request. The transport routes the
// `in` endpoint here (and the `out` SSE stream direct to the Trigger API), so
// every turn passes through the dashboard's own session before reaching the
// agent. We use that hop to mint a fresh read-only delegated token for the
// signed-in user and inject it into the turn's metadata server-side. The token
// reaches the agent without ever touching the browser, and minting stays tied
// to the user's own session (no shared-secret backdoor).
//
// The append body is `{ kind, payload: { metadata, ... } }`; we add the token
// (plus the API origin and the server-vouched project ref + env) to
// `payload.metadata`. Only `kind === "message"` turns carry metadata — stop
// chunks pass through untouched. We forward only the headers the API needs and
// deliberately drop the dashboard session cookie.
// Same-origin proxy for the chat append request. It mints a read-only delegated token scoped
// to the environment in this URL, so the token never reaches the browser.
const FORWARDED_HEADERS = [
"authorization",
@@ -33,16 +32,24 @@ const FORWARDED_HEADERS = [
"x-trigger-branch",
];
// The API's env routes key on the canonical env name (dev/staging/prod/preview),
// not the dashboard URL slug (e.g. staging's slug is "stg"). Map from the env
// type so the agent's tools address the right environment. Preview branches
// aren't threaded yet (they'd need the branch on every tool call) — a follow-up.
const ENV_NAME_BY_TYPE: Record<string, string> = {
DEVELOPMENT: "dev",
STAGING: "staging",
PRODUCTION: "prod",
PREVIEW: "preview",
};
// The only turn metadata a browser may set: everything else the agent reads is injected
// server-side. A whitelist — a new clientData field is server-owned until listed here on purpose.
const CLIENT_METADATA_KEYS = ["currentPage", "pageContext"] as const;
export function pickAgentClientMetadata(
metadata: Record<string, unknown> | undefined
): Record<string, unknown> {
const picked: Record<string, unknown> = {};
if (!metadata) return picked;
for (const key of CLIENT_METADATA_KEYS) {
if (metadata[key] !== undefined) picked[key] = metadata[key];
}
return picked;
}
function tooLarge() {
return json({ error: MESSAGE_TOO_LARGE_ERROR, code: MESSAGE_TOO_LARGE_CODE }, { status: 413 });
}
export async function action({ request, params }: ActionFunctionArgs) {
const user = await requireUser(request);
@@ -59,6 +66,12 @@ export async function action({ request, params }: ActionFunctionArgs) {
return json({ error: "Not found" }, { status: 404 });
}
// The declared size is refused before any lookup. It is advisory, so the read below is
// bounded too: without it a chunked body would be buffered whole before being refused.
if (exceedsMessageBodyBytes(declaredBodyBytes(request.headers))) {
return tooLarge();
}
const project = await findProjectBySlug(organizationSlug, projectParam, user.id);
if (!project) return json({ error: "Project not found" }, { status: 404 });
@@ -71,38 +84,76 @@ export async function action({ request, params }: ActionFunctionArgs) {
const url = new URL(request.url);
const upstreamUrl = `${apiOrigin.replace(/\/$/, "")}/${upstreamPath}${url.search}`;
// Resolve the dashboard env slug to the canonical API env name its tools use.
const runtimeEnv = await $replica.runtimeEnvironment.findFirst({
where: { projectId: project.id, slug: envParam },
select: { type: true },
});
const environmentName = runtimeEnv ? ENV_NAME_BY_TYPE[runtimeEnv.type] : undefined;
// Membership-scoped: `(projectId, slug)` is not unique because every developer has their own
// dev row, and a token must never be minted for someone else's environment — or for none.
const runtimeEnv = await findEnvironmentBySlug(project.id, envParam, user.id);
if (!runtimeEnv) return json({ error: "Environment not found" }, { status: 404 });
const environmentAddress = dashboardAgentEnvironmentAddress(runtimeEnv);
// When the project has a connected GitHub repo, resolve a signed source-archive
// pointer (code mode). Null otherwise -> the agent stays in assistant mode.
// Null without a connected GitHub repo, and the agent stays in assistant mode.
const repoSnapshot = await resolveDashboardAgentRepoSnapshot(project.id);
// Inject the delegated token + context into the turn's metadata.
const raw = await request.text();
const read = await readBoundedBodyText(request, MAX_MESSAGE_BODY_BYTES);
if (!read.ok) return tooLarge();
const raw = read.text;
let body = raw;
try {
const parsed = JSON.parse(raw) as {
kind?: string;
payload?: { metadata?: Record<string, unknown> };
type AgentTurn = {
kind?: string;
payload?: {
trigger?: string;
metadata?: Record<string, unknown>;
message?: { parts?: unknown };
};
};
let parsed: AgentTurn | undefined;
// Only the parse is tolerated: non-JSON is forwarded unchanged rather than break the turn.
// Everything after it must fail loudly — a swallowed mint would forward with no credential.
try {
parsed = JSON.parse(raw) as AgentTurn;
} catch {
parsed = undefined;
}
if (parsed) {
// Actions are placed by the server only, and this proxy is the one path a browser
// can reach `.in` through.
if (parsed.payload?.trigger === "action") {
return json({ error: "Not allowed" }, { status: 403 });
}
if (parsed.kind === "message" && parsed.payload) {
// A body under the byte cap can still be one huge part or hundreds of small ones.
if (checkMessageParts(parsed.payload.message?.parts) !== null) {
return tooLarge();
}
let userActorToken: string;
try {
userActorToken = await mintDashboardAgentUserActorToken(user.id, {
environmentId: runtimeEnv.id,
});
} catch (error) {
logger.error("Dashboard agent in-proxy could not mint a token", { error, upstreamPath });
return json({ error: "The dashboard agent couldn't send that message." }, { status: 500 });
}
parsed.payload.metadata = {
...(parsed.payload.metadata ?? {}),
userActorToken: await mintDashboardAgentUserActorToken(user.id),
...pickAgentClientMetadata(parsed.payload.metadata),
userActorToken,
apiOrigin,
projectRef: project.externalRef,
environmentName,
// Server-owned: the eval opt-out and every tenancy check key on these.
organizationId: project.organizationId,
userId: user.id,
projectId: project.id,
// `(projectId, slug)` isn't unique (dev is per-member), so anything addressing
// one environment row uses this id. The address is for name-addressed tools.
environmentId: runtimeEnv.id,
...environmentAddress,
...(repoSnapshot ? { repoSnapshot } : {}),
};
body = JSON.stringify(parsed);
}
} catch {
// Non-JSON or unexpected shape — forward unchanged rather than break the turn.
}
const headers = new Headers();
@@ -1,9 +1,11 @@
import { json, type ActionFunctionArgs, type LoaderFunctionArgs } from "@remix-run/server-runtime";
import {
chatExists,
countUserMessages,
createChat,
getChatMessages,
getSession,
listChatIdsWithOpenInvestigations,
listChats,
renameChat,
setChatPinned,
@@ -12,9 +14,18 @@ import {
import { generateFriendlyId } from "@trigger.dev/core/v3/isomorphic";
import type { UIMessage } from "ai";
import { z } from "zod";
import {
checkMessageParts,
declaredBodyBytes,
exceedsMessageBodyBytes,
MESSAGE_TOO_LARGE_CODE,
MESSAGE_TOO_LARGE_ERROR,
} from "~/components/dashboard-agent/message-limits";
import { MAX_URIS_PER_RESOLVE_REQUEST } from "~/components/dashboard-agent/resolve-uris";
import { $replica } from "~/db.server";
import { env } from "~/env.server";
import { findProjectBySlug } from "~/models/project.server";
import { findEnvironmentBySlug } from "~/models/runtimeEnvironment.server";
import {
dashboardAgentApiOrigin,
isDashboardAgentConfigured,
@@ -23,23 +34,29 @@ import {
resolveDashboardAgentRepoSnapshot,
startDashboardAgentSession,
} from "~/services/dashboardAgent.server";
import { dashboardAgentEnvironmentAddress } from "~/services/dashboardAgentEnvironmentAddress.server";
import { startDashboardAgentHeadStart } from "~/services/dashboardAgentHeadStart.server";
import { dashboardAgentDb } from "~/services/dashboardAgentDb.server";
import { logger } from "~/services/logger.server";
import { resolveTriggerUri } from "~/services/resolveTriggerUri.server";
import { requireUser } from "~/services/session.server";
import { EnvironmentParamSchema } from "~/utils/pathBuilder";
import { canAccessDashboardAgent } from "~/v3/canAccessDashboardAgent.server";
// The agent's tools address the canonical env name, not the dashboard URL slug.
const ENV_NAME_BY_TYPE: Record<string, string> = {
DEVELOPMENT: "dev",
STAGING: "staging",
PRODUCTION: "prod",
PREVIEW: "preview",
};
// The client-metadata whitelist lives with the `in` proxy, the other mint site, so the two cannot
// drift apart.
import { pickAgentClientMetadata } from "./resources.orgs.$organizationSlug.projects.$projectParam.env.$envParam.dashboard-agent.in.$";
const ActionBody = z.object({
intent: z.enum(["start", "create", "token", "rename", "pin", "delete"]),
intent: z.enum([
"start",
"create",
"token",
"rename",
"pin",
"delete",
"resolve",
"resolve-many",
]),
// Omitted for `create` (the server generates it); required for the rest.
chatId: z.string().min(1).optional(),
// The first user message (JSON UIMessage), for `create`.
@@ -47,9 +64,14 @@ const ActionBody = z.object({
clientData: z.string().optional(),
title: z.string().optional(),
pinned: z.enum(["true", "false"]).optional(),
// A `trigger://` URI, for `resolve`.
uri: z.string().optional(),
// A JSON array of `trigger://` URIs, for `resolve-many`.
uris: z.string().optional(),
});
// History list, or — with ?chatId= — the stored transcript + session for resume.
// History list by default. `?chatId=` returns the stored transcript plus session,
// `?quota=1` the message count.
export const loader = async ({ request, params }: LoaderFunctionArgs) => {
const user = await requireUser(request);
const userId = user.id;
@@ -66,14 +88,27 @@ export const loader = async ({ request, params }: LoaderFunctionArgs) => {
return json({ error: "Not found" }, { status: 404 });
}
const searchParams = new URL(request.url).searchParams;
const project = await findProjectBySlug(organizationSlug, projectParam, userId);
if (!project) return json({ error: "Project not found" }, { status: 404 });
const chatId = new URL(request.url).searchParams.get("chatId");
// The open chat is excluded and counted from the live transcript instead, so an
// unpersisted turn still counts against the cap.
if (searchParams.get("quota") === "1") {
const used = await countUserMessages(dashboardAgentDb, {
organizationId: project.organizationId,
userId,
excludeChatId: searchParams.get("chatId") ?? undefined,
});
return json({ used });
}
const chatId = searchParams.get("chatId");
if (chatId) {
const [messages, session] = await Promise.all([
getChatMessages(dashboardAgentDb, { chatId, userId }),
getSession(dashboardAgentDb, { chatId, userId }),
getChatMessages(dashboardAgentDb, { chatId, userId, organizationId: project.organizationId }),
getSession(dashboardAgentDb, { chatId, userId, organizationId: project.organizationId }),
]);
return json({ messages: messages ?? [], session });
}
@@ -82,9 +117,39 @@ export const loader = async ({ request, params }: LoaderFunctionArgs) => {
organizationId: project.organizationId,
userId,
});
return json({ chats });
// One query for all the listed chats, never one per row.
const investigatingChatIds = await listChatIdsWithOpenInvestigations(dashboardAgentDb, {
organizationId: project.organizationId,
userId,
});
return json({
chats: chats.map((chat) => ({
...chat,
hasOpenInvestigation: investigatingChatIds.has(chat.id),
})),
});
};
/** Only a source URI needs the connected repository, so a batch without one skips the read. */
async function findRepositoryForSourceUris(projectId: string, uris: string[]) {
if (!uris.some((uri) => uri.includes("/source/"))) return null;
const connected = await $replica.connectedGithubRepository.findFirst({
where: {
projectId,
repository: { installation: { deletedAt: null, suspendedAt: null } },
},
select: { repository: { select: { fullName: true } } },
});
return connected?.repository ?? null;
}
function messageTooLarge() {
return json({ error: MESSAGE_TOO_LARGE_ERROR, code: MESSAGE_TOO_LARGE_CODE }, { status: 413 });
}
export const action = async ({ request, params }: ActionFunctionArgs) => {
const user = await requireUser(request);
const userId = user.id;
@@ -101,16 +166,19 @@ export const action = async ({ request, params }: ActionFunctionArgs) => {
return json({ error: "Not found" }, { status: 404 });
}
// A declared oversize is refused here, before any lookup. Without a content-length the
// ingress cap has already ended the request mid-stream, so this never sees it.
if (exceedsMessageBodyBytes(declaredBodyBytes(request.headers))) {
return messageTooLarge();
}
const project = await findProjectBySlug(organizationSlug, projectParam, userId);
if (!project) return json({ error: "Project not found" }, { status: 404 });
const parsed = ActionBody.safeParse(Object.fromEntries(await request.formData()));
if (!parsed.success) return json({ error: "Invalid request" }, { status: 400 });
// Create a new chat: the SERVER generates the id and owns the chat record, so
// a client can never name another user's chat. Kicks off the first turn (head
// start when configured, else a cold session) and returns the id + token. The
// client mounts with that id and resumes the stream.
// The server generates the chat id, so a client can never name another user's chat.
if (parsed.data.intent === "create") {
if (!isDashboardAgentConfigured()) {
return json({ error: "The dashboard agent is not configured." }, { status: 501 });
@@ -126,6 +194,14 @@ export const action = async ({ request, params }: ActionFunctionArgs) => {
}
if (!firstMessage) return json({ error: "message is required" }, { status: 400 });
// A body under the byte cap can still be one huge part or hundreds of small ones.
if (
exceedsMessageBodyBytes(Buffer.byteLength(parsed.data.message ?? "", "utf8")) ||
checkMessageParts(firstMessage.parts) !== null
) {
return messageTooLarge();
}
let clientData: Record<string, unknown> | undefined;
try {
clientData = parsed.data.clientData
@@ -134,50 +210,108 @@ export const action = async ({ request, params }: ActionFunctionArgs) => {
} catch {
/* invalid JSON — create without context metadata */
}
// Only the whitelisted page context survives; the rest is injected below.
const clientContext = pickAgentClientMetadata(clientData);
// Membership-scoped: dev rows are per-developer, so a token must never be minted for
// someone else's environment — or, when nothing resolves, for no environment at all.
const runtimeEnv = await findEnvironmentBySlug(project.id, envParam, userId);
if (!runtimeEnv) return json({ error: "Environment not found" }, { status: 404 });
const environmentAddress = dashboardAgentEnvironmentAddress(runtimeEnv);
const chatId = generateFriendlyId("chat");
try {
const repoSnapshot = await resolveDashboardAgentRepoSnapshot(project.id);
const headStarted = Boolean(env.ANTHROPIC_API_KEY);
// The lookups and the mint all run before the chat row exists, so a failure here can't
// leave an empty chat behind in the user's history.
const headStartMetadata = headStarted
? {
// The agent validates run metadata against its clientDataSchema, so the
// per-turn client context must accompany the injected auth and context fields.
...clientContext,
userActorToken: await mintDashboardAgentUserActorToken(userId, {
environmentId: runtimeEnv.id,
}),
apiOrigin: dashboardAgentApiOrigin(),
projectRef: project.externalRef,
// Server-owned, like the `in` proxy: the eval opt-out and every tenancy check
// key on these, so the client can't set them at all.
organizationId: project.organizationId,
userId,
projectId: project.id,
// Same environment identity the `in` proxy injects.
environmentId: runtimeEnv.id,
...environmentAddress,
...(repoSnapshot ? { repoSnapshot } : {}),
}
: undefined;
await createChat(dashboardAgentDb, {
id: chatId,
organizationId: project.organizationId,
userId,
...(clientData ? { metadata: { context: clientData } } : {}),
...(clientData ? { metadata: { context: clientContext } } : {}),
});
const runtimeEnv = await $replica.runtimeEnvironment.findFirst({
where: { projectId: project.id, slug: envParam },
select: { type: true },
});
const environmentName = runtimeEnv ? ENV_NAME_BY_TYPE[runtimeEnv.type] : undefined;
const repoSnapshot = await resolveDashboardAgentRepoSnapshot(project.id);
const headStarted = Boolean(env.ANTHROPIC_API_KEY);
if (headStarted) {
// Head start runs the warm step-1 with this first message and injects the
// delegated token + context into the run's payload server-side.
await startDashboardAgentHeadStart({
try {
if (headStartMetadata) {
// Injects the delegated token and context into the run's payload server-side.
await startDashboardAgentHeadStart({
chatId,
messages: [firstMessage],
mode: repoSnapshot ? "code" : "assistant",
metadata: headStartMetadata,
});
} else {
// Cold start: the client sends the first message through the `in` proxy, which
// injects the token.
// Same server-owned identity the head-start path injects; the `in` proxy adds the
// delegated token on the first turn.
await startDashboardAgentSession({
chatId,
clientData: {
...clientContext,
organizationId: project.organizationId,
userId,
projectId: project.id,
environmentId: runtimeEnv.id,
...environmentAddress,
},
});
}
} catch (error) {
// Both starts are one create-session-and-trigger round trip, so a rejection means no
// handover was dispatched and no message was sent: a session the call did create in
// spite of the error idles out having done nothing. The empty row is all there is to undo.
// Swallowed so the start's own error is what surfaces and gets logged.
await softDeleteChat(dashboardAgentDb, {
chatId,
messages: [firstMessage],
mode: repoSnapshot ? "code" : "assistant",
metadata: {
// The agent validates the run metadata against its clientDataSchema
// (userId, organizationId, …), so the per-turn clientData has to be
// present alongside the injected auth/context fields.
...(clientData ?? {}),
userActorToken: await mintDashboardAgentUserActorToken(userId),
apiOrigin: dashboardAgentApiOrigin(),
projectRef: project.externalRef,
environmentName,
...(repoSnapshot ? { repoSnapshot } : {}),
},
userId,
organizationId: project.organizationId,
}).catch((cleanupError) => {
logger.error("Failed to remove a dashboard agent chat whose start failed", {
chatId,
error: cleanupError,
});
});
} else {
// Cold start: create the session (preload); the client sends the first
// message through the transport, where the `in` proxy injects the token.
await startDashboardAgentSession({ chatId, clientData });
throw error;
}
const publicAccessToken = await mintDashboardAgentToken(chatId);
let publicAccessToken: string;
try {
publicAccessToken = await mintDashboardAgentToken(chatId);
} catch (error) {
// The start resolved, so the session is live and a head start is already streaming into
// it. Deleting the chat here would hide a running agent; the client can ask for a token
// again through the `token` intent.
logger.error("Dashboard agent chat started but its token mint failed", { chatId, error });
return json(
{ error: "The dashboard agent started but couldn't be opened. Try opening it again." },
{ status: 500 }
);
}
return json({ chatId, publicAccessToken, headStarted });
} catch (error) {
logger.error("Failed to create dashboard agent chat", { chatId, error });
@@ -188,6 +322,64 @@ export const action = async ({ request, params }: ActionFunctionArgs) => {
}
}
// Scoped by the environment in the URL: the resolver refuses a URI naming a
// different project or environment.
if (parsed.data.intent === "resolve") {
const uri = parsed.data.uri;
if (!uri) return json({ error: "uri is required" }, { status: 400 });
const environment = await findEnvironmentBySlug(project.id, envParam, userId);
if (!environment) return json({ error: "Environment not found" }, { status: 404 });
const repository = await findRepositoryForSourceUris(project.id, [uri]);
const resolved = resolveTriggerUri({ ...environment, repository }, uri);
if (!resolved) return json({ error: "Nothing to open for that link" }, { status: 404 });
return json({
path: resolved.url,
label: resolved.label,
external: resolved.external ?? false,
});
}
// The card's citations in one request: one environment lookup and one repo lookup for the
// whole batch, same environment scope as `resolve`.
if (parsed.data.intent === "resolve-many") {
let uris: string[];
try {
const list = JSON.parse(parsed.data.uris ?? "") as unknown;
if (!Array.isArray(list) || list.some((uri) => typeof uri !== "string")) {
return json({ error: "uris is required" }, { status: 400 });
}
uris = [...new Set(list as string[])];
} catch {
return json({ error: "uris is required" }, { status: 400 });
}
if (uris.length === 0) return json({ error: "uris is required" }, { status: 400 });
if (uris.length > MAX_URIS_PER_RESOLVE_REQUEST) {
return json({ error: "Too many links in one request" }, { status: 400 });
}
const environment = await findEnvironmentBySlug(project.id, envParam, userId);
if (!environment) return json({ error: "Environment not found" }, { status: 404 });
const repository = await findRepositoryForSourceUris(project.id, uris);
const scope = { ...environment, repository };
// A null entry is the definitive "nothing to open": the client caches it.
const resolved: Record<string, { path: string; label: string; external: boolean } | null> = {};
for (const uri of uris) {
const hit = resolveTriggerUri(scope, uri);
resolved[uri] = hit
? { path: hit.url, label: hit.label, external: hit.external ?? false }
: null;
}
return json({ resolved });
}
const { intent, chatId } = parsed.data;
if (!chatId) return json({ error: "chatId is required" }, { status: 400 });
@@ -196,10 +388,7 @@ export const action = async ({ request, params }: ActionFunctionArgs) => {
if (!isDashboardAgentConfigured()) {
return json({ error: "The dashboard agent is not configured." }, { status: 501 });
}
// Resume-only: new chats are created via the `create` intent (server-owned
// id). The transport falls back here to re-establish a session for an
// existing chat (e.g. after its token expired), so verify ownership before
// issuing one — a client-supplied chatId must belong to the caller.
// Resume only, so a client-supplied chatId is checked against the caller first.
if (
!(await chatExists(dashboardAgentDb, {
chatId,
@@ -233,8 +422,7 @@ export const action = async ({ request, params }: ActionFunctionArgs) => {
if (!isDashboardAgentConfigured()) {
return json({ error: "The dashboard agent is not configured." }, { status: 501 });
}
// Only mint a session token for a chat the caller owns, so a client-supplied
// chatId can't be used to get a token for someone else's session.
// Only mint a token for a chat the caller owns.
if (
!(await chatExists(dashboardAgentDb, {
chatId,
@@ -249,7 +437,12 @@ export const action = async ({ request, params }: ActionFunctionArgs) => {
case "rename": {
if (!parsed.data.title) return json({ error: "title is required" }, { status: 400 });
await renameChat(dashboardAgentDb, { chatId, userId, title: parsed.data.title });
await renameChat(dashboardAgentDb, {
chatId,
userId,
organizationId: project.organizationId,
title: parsed.data.title,
});
return json({ ok: true });
}
@@ -257,13 +450,29 @@ export const action = async ({ request, params }: ActionFunctionArgs) => {
await setChatPinned(dashboardAgentDb, {
chatId,
userId,
organizationId: project.organizationId,
pinned: parsed.data.pinned === "true",
});
return json({ ok: true });
}
case "delete": {
await softDeleteChat(dashboardAgentDb, { chatId, userId });
// Existence check gives a 404 for a chat this caller can't see; the delete itself
// is org- and owner-scoped too.
if (
!(await chatExists(dashboardAgentDb, {
chatId,
userId,
organizationId: project.organizationId,
}))
) {
return json({ error: "Chat not found" }, { status: 404 });
}
await softDeleteChat(dashboardAgentDb, {
chatId,
userId,
organizationId: project.organizationId,
});
return json({ ok: true });
}
}
@@ -89,9 +89,9 @@ function DotMatrixTab() {
<div className="flex flex-wrap items-center gap-6 rounded-md border border-grid-bright bg-background-bright px-6 py-5">
{(
[
["ask-ai/small", 16, "small"],
["ask-ai/medium", 16, "medium"],
["ask-ai/large", 20, "large"],
["ask-trigger/small", 16, "small"],
["ask-trigger/medium", 16, "medium"],
["ask-trigger/large", 20, "large"],
] as [ButtonVariant, number, string][]
).map(([variant, matrixSize, label]) => (
<div key={variant} className="flex flex-col items-center gap-2">
@@ -106,7 +106,7 @@ function DotMatrixTab() {
<div className="flex flex-wrap items-center gap-6 rounded-md border border-grid-bright bg-background-bright px-6 py-5">
{[14, 15, 16].map((s) => (
<div key={s} className="flex flex-col items-center gap-2">
<AskAiButton variant="ask-ai/small" matrixSize={s} />
<AskAiButton variant="ask-trigger/small" matrixSize={s} />
<div className="text-[10px] uppercase tracking-wide text-text-dimmed">{s}px icon</div>
</div>
))}
@@ -231,7 +231,7 @@ function FaceButton({ name }: { name: DotShapeName }) {
return (
<Button
variant="ask-ai/small"
variant="ask-trigger/small"
onClick={trigger}
LeadingIcon={
<AgentDotMatrix
@@ -399,18 +399,18 @@ export default function Story() {
</Button>
</div>
</div>
<Header1 className="mb-2 mt-8">Ask AI button</Header1>
<Header1 className="mb-2 mt-8">Ask Trigger button</Header1>
<div className="grid grid-cols-4 gap-8 border-b border-grid-bright pb-8">
<div className="flex flex-col items-start gap-2">
<Header3 className="mb-1 uppercase">Sizes</Header3>
<Button variant="ask-ai/small">Ask AI</Button>
<Button variant="ask-ai/medium">Ask AI</Button>
<Button variant="ask-ai/large">Ask AI</Button>
<Button variant="ask-trigger/small">Ask Trigger</Button>
<Button variant="ask-trigger/medium">Ask Trigger</Button>
<Button variant="ask-trigger/large">Ask Trigger</Button>
</div>
<div className="flex flex-col items-start gap-2">
<Header3 className="mb-1 uppercase">Disabled</Header3>
<Button variant="ask-ai/small" disabled>
Ask AI
<Button variant="ask-trigger/small" disabled>
Ask Trigger
</Button>
</div>
<div className="col-span-2 self-end text-sm text-text-dimmed">
+48 -2
View File
@@ -13,7 +13,8 @@ import {
findEnvironmentByPublicApiKey,
toAuthenticated,
} from "~/models/runtimeEnvironment.server";
import type { RbacAbility, RbacResource } from "@trigger.dev/rbac";
import type { RbacAbility, RbacResource, UserActorClaims } from "@trigger.dev/rbac";
import { assertUserActorEnvironment } from "./userActorEnvironment.server";
import { type RuntimeEnvironmentForEnvRepo } from "~/v3/environmentVariables/environmentVariablesRepository.server";
import { logger } from "./logger.server";
import { safeEnvironmentLogFields } from "./safeEnvironmentLog";
@@ -44,6 +45,13 @@ const ClaimsSchema = z.object({
skipColumns: z.array(z.string()).optional(),
})
.optional(),
// Identity only. Authorization comes from `sub` and `scopes`, never from `act`.
act: z
.object({
sub: z.string(),
client: z.string().optional(),
})
.optional(),
});
// Re-export the slim shape defined in @trigger.dev/core. Single source of
@@ -74,6 +82,7 @@ export type ApiAuthenticationResultSuccess = {
// API keys (no user) and JWTs minted without delegation.
actor?: {
sub: string;
client?: string;
};
};
@@ -187,6 +196,7 @@ export async function authenticateApiKey(
environment: validationResults.environment,
oneTimeUse: parsedClaims.success ? parsedClaims.data.otu : false,
realtime: parsedClaims.success ? parsedClaims.data.realtime : undefined,
actor: parsedClaims.success ? parsedClaims.data.act : undefined,
};
}
}
@@ -279,6 +289,7 @@ async function authenticateApiKeyWithFailure(
environment: validationResults.environment,
oneTimeUse: parsedClaims.success ? parsedClaims.data.otu : false,
realtime: parsedClaims.success ? parsedClaims.data.realtime : undefined,
actor: parsedClaims.success ? parsedClaims.data.act : undefined,
};
}
}
@@ -394,10 +405,22 @@ function getApiKeyResult(apiKey: string): {
return { apiKey, type };
}
/**
* The authenticated user-actor. A user-actor token authenticates as its user, so it is the same
* shape a PAT authenticates to — that shape now carries the token's verified claims itself, so
* any layer holding the actor holds its environment scope.
*/
export type UserActorAuthenticatedActor = PersonalAccessTokenAuthenticationResult;
export type AuthenticationResult =
| {
type: "personalAccessToken";
result: PersonalAccessTokenAuthenticationResult;
result: UserActorAuthenticatedActor;
/**
* Claims of the delegated user-actor token the caller presented, if any. A UAT authenticates
* as its user, so it rides on this variant; its environment scope is enforced on resolution.
*/
userActor?: UserActorClaims;
}
| {
type: "organizationAccessToken";
@@ -522,11 +545,34 @@ export async function authenticateRequest<
return;
}
/**
* Resolve the environment a request targets, and enforce the caller's environment scope.
*
* Every route that turns an authentication result into an environment goes through here, so the
* user-actor token's `environmentId` claim is checked once, at the seam — a new endpoint can't
* forget it.
*/
export async function authenticatedEnvironmentForAuthentication(
auth: AuthenticationResult,
projectRef: string,
slug: string,
branch?: string
): Promise<AuthenticatedEnvironment> {
const environment = await resolveEnvironmentForAuthentication(auth, projectRef, slug, branch);
if (auth.type === "personalAccessToken") {
// Either place the claims ride: on the actor (the shape every layer keeps) or beside it.
assertUserActorEnvironment(auth.result.userActor ?? auth.userActor, environment.id);
}
return environment;
}
async function resolveEnvironmentForAuthentication(
auth: AuthenticationResult,
projectRef: string,
slug: string,
branch?: string
): Promise<AuthenticatedEnvironment> {
if (slug === "staging") {
slug = "stg";
+68 -41
View File
@@ -9,6 +9,73 @@ import type { Duration } from "./rateLimiter.server";
const BATCH_STREAM_ITEMS_PATH = /^\/api\/v3\/batches\/([^/]+)\/items$/;
// Rate-limit key for a delegated (agent/PAT-minted) JWT. Its token value rotates every
// turn, so keying on the token would hand each turn a fresh bucket. Key on env+acting-user
// so the agent's traffic shares one bucket across turns. The `jwt-actor:` prefix keeps it
// off PRIVATE-key buckets, which key on the bare environment id.
export function jwtActorRateLimitIdentifier(environmentId: string, actorSub: string): string {
return `jwt-actor:${environmentId}:${actorSub}`;
}
// The per-request bucket decision for the API limiter. Exported so the branch below
// (a delegated JWT keys on env+acting-user, everything else keeps its prior key) is
// testable without standing up the middleware and its Redis.
export async function resolveApiRateLimitOverride(
authorizationValue: string
): Promise<{ config?: unknown; identifier?: string } | undefined> {
const rawApiKey = authorizationValue.replace(/^Bearer /, "");
if (rawApiKey.startsWith("tr_")) {
const scope = await resolvePrivateApiKeyRateLimitScope(rawApiKey);
if (!scope) {
return;
}
return {
config: scope.apiRateLimiterConfig,
identifier: scope.environmentId,
};
}
const authenticatedEnv = await authenticateAuthorizationHeader(authorizationValue, {
allowPublicKey: true,
allowJWT: true,
});
if (!authenticatedEnv || !authenticatedEnv.ok) {
return;
}
if (authenticatedEnv.type === "PUBLIC_JWT") {
const config = {
type: "fixedWindow",
window: env.API_RATE_LIMIT_JWT_WINDOW,
tokens: env.API_RATE_LIMIT_JWT_TOKENS,
} as const;
// A delegated JWT (agent/PAT-minted) shares one bucket per env+acting-user across turns.
// A browser realtime JWT carries no `act`, so it keeps the hashed-token fallback.
if (authenticatedEnv.actor?.sub) {
return {
config,
identifier: jwtActorRateLimitIdentifier(
authenticatedEnv.environment.id,
authenticatedEnv.actor.sub
),
};
}
return { config };
}
return {
config: authenticatedEnv.environment.organization.apiRateLimiterConfig,
// Public keys are browser-distributed, so keep them on per-key buckets.
identifier: authenticatedEnv.type === "PRIVATE" ? authenticatedEnv.environment.id : undefined,
};
}
export const apiRateLimiter = authorizationRateLimitMiddleware({
redis: {
port: env.RATE_LIMIT_REDIS_PORT,
@@ -30,47 +97,7 @@ export const apiRateLimiter = authorizationRateLimitMiddleware({
stale: 60_000 * 20, // Date is stale after 20 minutes
maxItems: 1000,
},
limiterConfigOverride: async (authorizationValue) => {
const rawApiKey = authorizationValue.replace(/^Bearer /, "");
if (rawApiKey.startsWith("tr_")) {
const scope = await resolvePrivateApiKeyRateLimitScope(rawApiKey);
if (!scope) {
return;
}
return {
config: scope.apiRateLimiterConfig,
identifier: scope.environmentId,
};
}
const authenticatedEnv = await authenticateAuthorizationHeader(authorizationValue, {
allowPublicKey: true,
allowJWT: true,
});
if (!authenticatedEnv || !authenticatedEnv.ok) {
return;
}
if (authenticatedEnv.type === "PUBLIC_JWT") {
return {
config: {
type: "fixedWindow",
window: env.API_RATE_LIMIT_JWT_WINDOW,
tokens: env.API_RATE_LIMIT_JWT_TOKENS,
},
};
}
return {
config: authenticatedEnv.environment.organization.apiRateLimiterConfig,
// Public keys are browser-distributed, so keep them on per-key buckets.
identifier: authenticatedEnv.type === "PRIVATE" ? authenticatedEnv.environment.id : undefined,
};
},
limiterConfigOverride: resolveApiRateLimitOverride,
pathMatchers: [/^\/api/],
// Allow /api/v1/tasks/:id/callback/:secret
pathWhiteList: [
@@ -37,10 +37,17 @@ export function dashboardAgentApiOrigin(): string {
// service from the dashboard session (never a PAT), so a user can only ever
// mint a token for themselves. The `in` proxy injects this into the turn's
// metadata so the token reaches the agent without ever touching the browser.
export function mintDashboardAgentUserActorToken(userId: string): Promise<string> {
//
// Endpoints that bind something to one environment read `environmentId` off the token,
// so the agent can't name a different one in a request body.
export function mintDashboardAgentUserActorToken(
userId: string,
opts: { environmentId: string }
): Promise<string> {
return signUserActorToken(env.SESSION_SECRET, {
userId,
client: "dashboard-agent",
environmentId: opts.environmentId,
cap: DASHBOARD_AGENT_UAT_CAP,
expirationTime: Math.floor(Date.now() / 1000) + DASHBOARD_AGENT_UAT_TTL_SECONDS,
});
@@ -0,0 +1,73 @@
import type { NextFunction, Request, Response } from "express";
import {
MAX_MESSAGE_BODY_BYTES,
MESSAGE_TOO_LARGE_CODE,
MESSAGE_TOO_LARGE_ERROR,
} from "~/components/dashboard-agent/message-limits";
/**
* The ingress cap for the agent's chat paths. A route can only refuse a body after it has read
* it, and `content-length` is optional, so a chunked upload would be buffered whole before the
* route ever saw its size. This counts the bytes as they arrive and refuses mid-stream.
*/
/** Headroom over the message cap for multipart framing and the per-turn metadata. */
const INGRESS_SLACK_BYTES = 8 * 1024;
export const DASHBOARD_AGENT_MAX_INGRESS_BYTES = MAX_MESSAGE_BODY_BYTES + INGRESS_SLACK_BYTES;
// The agent's own routes only: the `/api/v1/dashboard-agent/…` endpoints and the
// `/…/env/<env>/dashboard-agent…` chat resources. Anchored so a task named
// `dashboard-agent` (`/api/v1/tasks/dashboard-agent/trigger`) is not capped.
const AGENT_PATH = /^\/api\/v1\/dashboard-agent(\/|$)|\/env\/[^/]+\/dashboard-agent(\/|$)/;
/** Methods that can carry one. GET and HEAD cannot, and streaming them would be wasted work. */
const BODY_METHODS = new Set(["POST", "PUT", "PATCH", "DELETE"]);
function refuse(res: Response): void {
if (res.headersSent) return;
res.status(413).json({ error: MESSAGE_TOO_LARGE_ERROR, code: MESSAGE_TOO_LARGE_CODE });
}
/**
* Attaches a counting listener and pauses the stream again immediately, so the route's own
* reader still receives every chunk while nothing flows until it asks for it. Crossing the
* limit ends the request: pausing alone wouldn't stop the route resuming the stream itself.
*/
export function capRequestBody(req: Request, res: Response, limit: number): void {
const declared = Number.parseInt(req.headers["content-length"] ?? "", 10);
if (Number.isFinite(declared) && declared > limit) {
refuse(res);
return;
}
let received = 0;
const onData = (chunk: Buffer | string) => {
received += Buffer.byteLength(chunk);
if (received <= limit) return;
req.off("data", onData);
req.pause();
refuse(res);
// Torn down only once the refusal is on the wire, or the client never reads it.
res.once("finish", () => req.destroy());
};
req.on("data", onData);
req.pause();
req.once("end", () => req.off("data", onData));
}
/**
* Only the agent's own paths: every other route keeps the body handling it had. Matched
* case-insensitively because Remix routes are, and on every method — a DELETE reads a body too.
*/
export function dashboardAgentBodyCap(req: Request, res: Response, next: NextFunction): void {
if (!BODY_METHODS.has(req.method) || !AGENT_PATH.test(req.path.toLowerCase())) {
next();
return;
}
capRequestBody(req, res, DASHBOARD_AGENT_MAX_INGRESS_BYTES);
if (res.headersSent) return;
next();
}
@@ -0,0 +1,74 @@
/**
* Retention for soft-deleted chats. A deleted chat is kept for a grace window and then
* hard-deleted with all its child rows; one bounded statement per run, oldest first.
* Also the eventual purge behind organization deletion, which soft-deletes the org's
* chats so this same sweep removes them.
*/
import {
hardDeleteChatsSoftDeletedBefore,
softDeleteChatsForOrganization,
} from "@internal/dashboard-agent-db";
import { dashboardAgentDb } from "~/services/dashboardAgentDb.server";
import { logger } from "~/services/logger.server";
/**
* How long a soft-deleted chat is kept before it and its children are hard-deleted.
* Long enough that an accidental delete can still be investigated; org deletion soft-
* deletes the org's chats, so those are removed the same way once the window passes.
*/
export const CHAT_SOFT_DELETE_RETENTION_MS = 30 * 24 * 60 * 60 * 1000;
/** Per-run cap. Retention is one bounded statement, not a row-at-a-time loop. */
const RETENTION_BATCH_LIMIT = 500;
export type ChatRetentionResult = {
/** Soft-deleted chats past the retention window dropped this run. */
purged: number;
failed: number;
};
export type ChatRetentionDeps = {
now?: () => Date;
limit?: number;
/** Hard-delete chats soft-deleted before `before`. Returns how many went. */
purge?: (params: { before: Date; limit: number }) => Promise<number>;
};
export async function sweepDashboardAgentSoftDeletedChats(
deps: ChatRetentionDeps = {}
): Promise<ChatRetentionResult> {
const now = deps.now?.() ?? new Date();
const limit = deps.limit ?? RETENTION_BATCH_LIMIT;
const purge =
deps.purge ?? ((params) => hardDeleteChatsSoftDeletedBefore(dashboardAgentDb, params));
const result: ChatRetentionResult = { purged: 0, failed: 0 };
try {
result.purged = await purge({
before: new Date(now.getTime() - CHAT_SOFT_DELETE_RETENTION_MS),
limit,
});
} catch (error) {
result.failed++;
logger.error("Dashboard agent chat retention failed", { error });
}
if (result.failed > 0) {
throw new Error("The dashboard agent chat retention pass failed");
}
return result;
}
/**
* Soft-delete every chat belonging to a deleted organization. The retention sweep above
* hard-deletes them once the window passes, so the org-deletion request never runs a
* cross-database hard delete.
*/
export async function purgeDashboardAgentChatsForOrganization(params: {
organizationId: string;
}): Promise<number> {
return softDeleteChatsForOrganization(dashboardAgentDb, params);
}
@@ -0,0 +1,37 @@
/**
* How the dashboard agent addresses one environment on the name-addressed API routes.
*
* The name is derived from the environment's type, and every branch shares its parent's type — so
* the name alone does not identify an environment, it identifies a family. The branch is the rest
* of the address, and the API's resolver needs both to land on the row the dashboard selected.
*
* Returned as a pair so no caller can take the name without it. Handing the JWT exchange a bare
* "preview" resolves the parent, and the delegated token's `environmentId` claim then correctly
* refuses it — the guard is the detector, not the defect.
*
* Kept free of heavy imports so both mint sites and their tests can use the real thing.
*/
// The API's env routes key on the canonical env name, not the dashboard URL slug
// (staging's slug is "stg").
const ENV_NAME_BY_TYPE: Record<string, string> = {
DEVELOPMENT: "dev",
STAGING: "staging",
PRODUCTION: "prod",
PREVIEW: "preview",
};
export type DashboardAgentEnvironmentAddress = {
environmentName?: string;
environmentBranch?: string;
};
export function dashboardAgentEnvironmentAddress(
environment: { type: string; branchName?: string | null } | undefined
): DashboardAgentEnvironmentAddress {
if (!environment) return {};
return {
environmentName: ENV_NAME_BY_TYPE[environment.type],
...(environment.branchName ? { environmentBranch: environment.branchName } : {}),
};
}
@@ -0,0 +1,54 @@
/**
* Whether an org's agent turns may be judged. The agent has no main-database access, so it
* asks the API for this and treats anything but an explicit yes as no.
*/
import { prisma } from "~/db.server";
import { logger } from "~/services/logger.server";
import { FEATURE_FLAG, hasUnreadableTurnEvalsOverride } from "~/v3/featureFlags";
import { makeFlag } from "~/v3/featureFlags.server";
/** Judging is on unless an org turns it off. */
const DEFAULT_TURN_EVALS_ENABLED = true;
/**
* Resolves `dashboardAgentTurnEvalsEnabled` for one org, with a per-org override winning in
* both directions. Membership-scoped: a token can name any org, so the caller's membership
* is the tenant floor. Returns false when the org (or its setting) can't be read — a judged
* turn goes to a third-party model, so an unknown answer must not read as consent.
*/
export async function orgAllowsDashboardAgentTurnEvals(params: {
userId: string;
organizationId: string;
}): Promise<boolean> {
try {
const org = await prisma.organization.findFirst({
where: {
id: params.organizationId,
members: { some: { userId: params.userId } },
},
select: { featureFlags: true },
});
if (!org) return false;
const overrides = (org.featureFlags as Record<string, unknown>) ?? {};
// `flag()` ignores an override the schema rejects and falls through to the global default,
// which is on — so an org that tried to turn judging off would keep being judged.
if (hasUnreadableTurnEvalsOverride(overrides)) return false;
const flag = makeFlag();
return Boolean(
await flag({
key: FEATURE_FLAG.dashboardAgentTurnEvalsEnabled,
defaultValue: DEFAULT_TURN_EVALS_ENABLED,
overrides,
})
);
} catch (error) {
logger.error("Couldn't read the org's dashboard agent turn-eval setting", {
organizationId: params.organizationId,
error,
});
return false;
}
}
@@ -0,0 +1,58 @@
/**
* Retention for the agent's judged-turn rows. The table is append-only quality data with
* no reader, so it can't be left to grow forever; one bounded statement per run, oldest
* first. Runs whether or not the agent is configured — rows outlive the agent project.
*/
import { deleteTurnEvalsOlderThan } from "@internal/dashboard-agent-db";
import { dashboardAgentDb } from "~/services/dashboardAgentDb.server";
import { logger } from "~/services/logger.server";
/**
* How long a judged turn is kept. Nothing reads the table today, and the rows carry the
* user's question next to the agent's answer, so the period is the shortest one that still
* lets a month of product review (capability and docs gaps) be aggregated.
*/
export const TURN_EVAL_RETENTION_MS = 30 * 24 * 60 * 60 * 1000;
/** Per-run cap. Retention is one statement, not a row-at-a-time loop. */
const RETENTION_BATCH_LIMIT = 500;
export type TurnEvalRetentionResult = {
/** Rows past the retention period dropped this run. */
purged: number;
failed: number;
};
export type TurnEvalRetentionDeps = {
now?: () => Date;
limit?: number;
/** Drop rows created before `before`. Returns how many went. */
purge?: (params: { before: Date; limit: number }) => Promise<number>;
};
export async function sweepDashboardAgentTurnEvals(
deps: TurnEvalRetentionDeps = {}
): Promise<TurnEvalRetentionResult> {
const now = deps.now?.() ?? new Date();
const limit = deps.limit ?? RETENTION_BATCH_LIMIT;
const purge = deps.purge ?? ((params) => deleteTurnEvalsOlderThan(dashboardAgentDb, params));
const result: TurnEvalRetentionResult = { purged: 0, failed: 0 };
try {
result.purged = await purge({
before: new Date(now.getTime() - TURN_EVAL_RETENTION_MS),
limit,
});
} catch (error) {
result.failed++;
logger.error("Dashboard agent turn-eval retention failed", { error });
}
if (result.failed > 0) {
throw new Error("The dashboard agent turn-eval retention pass failed");
}
return result;
}
@@ -6,8 +6,14 @@ import {
dashboardAgentCodeToolSchemas,
dashboardAgentToolSchemas,
} from "@internal/dashboard-agent/tool-schemas";
import {
describePromptPrefix,
PROMPT_CACHE_CONTROL,
promptCacheAttributes,
} from "@internal/dashboard-agent/prompt-prefix";
import { ApiClient, SessionStreamInstance, writeTurnCompleteRecord } from "@trigger.dev/core/v3";
import { chat as chatServer } from "@trigger.dev/sdk/chat-server";
import { streamText, type UIMessage } from "ai";
import { streamText, type UIMessage, type UIMessageChunk } from "ai";
import { env } from "~/env.server";
import {
dashboardAgentApiOrigin,
@@ -19,17 +25,69 @@ const TASK_ID = "dashboard-agent";
const anthropic = createAnthropic({ apiKey: env.ANTHROPIC_API_KEY });
/** Shown when the warm first turn produced nothing. The provider error is only logged. */
export const HEAD_START_FAILURE_ERROR_TEXT =
"The assistant couldn't start this response. Please send your message again.";
/** A seam so the failure path is testable without S2 credentials or a live session. */
export type DashboardAgentSessionOutWriter = {
writeChunk(chunk: UIMessageChunk): Promise<void>;
/** The `turn-complete` control record that closes the client's stream. */
writeTurnComplete(): Promise<void>;
};
/**
* Server-owned head start. The webapp generates the chatId and owns the chat
* record, then kicks off step 1 here via `chat.startHeadStart` (the detached
* flow): it creates the session (externalId = chatId), triggers the
* handover-prepare run, and streams step 1 into `session.out` in the background.
* The browser resumes that stream rather than streaming step 1 inline. Step 1
* runs the agent's SCHEMA-ONLY tools + the shared model/prompt for the mode the
* agent run will be in; the agent run picks up tool execution and step 2+.
*
* `metadata` (the delegated UAT + context) is merged into the run's wire payload
* server-side, so it reaches the agent without touching the browser.
* Surface a failed warm step 1 as a visible error turn. `turn-complete` is written even if
* the error chunk fails, so a resumed stream always terminates.
*/
export async function writeHeadStartFailureToSessionOut(
writer: DashboardAgentSessionOutWriter
): Promise<void> {
try {
await writer.writeChunk({
type: "error",
errorText: HEAD_START_FAILURE_ERROR_TEXT,
} as UIMessageChunk);
} finally {
await writer.writeTurnComplete();
}
}
function singleChunkStream(chunk: UIMessageChunk): ReadableStream<UIMessageChunk> {
return new ReadableStream<UIMessageChunk>({
start(controller) {
controller.enqueue(chunk);
controller.close();
},
});
}
// Writes as the agent's own environment: `.out` appends are private-only.
function createSessionOutWriter(
chatId: string,
accessToken: string
): DashboardAgentSessionOutWriter {
const apiClient = new ApiClient(dashboardAgentApiOrigin(), accessToken);
return {
async writeChunk(chunk) {
const instance = new SessionStreamInstance<UIMessageChunk>({
apiClient,
baseUrl: apiClient.baseUrl,
sessionId: chatId, // Sessions are addressable by externalId (chatId).
io: "out",
source: singleChunkStream(chunk),
});
await instance.wait();
},
async writeTurnComplete() {
await writeTurnCompleteRecord(apiClient, chatId);
},
};
}
/**
* Server-owned head start: creates the session, triggers the handover-prepare run, and
* streams step 1 into `session.out`. `metadata` is merged into the run's payload server-side.
*/
export async function startDashboardAgentHeadStart(params: {
chatId: string;
@@ -47,8 +105,7 @@ export async function startDashboardAgentHeadStart(params: {
messages: params.messages,
metadata: params.metadata,
triggerConfig: dashboardAgentTriggerConfig(),
// Scope session creation + the agent trigger to the agent's project/env. The
// Anthropic key here only powers the warm step-1 call.
// Scopes session creation and the agent trigger to the agent's own environment.
apiClient: {
baseURL: dashboardAgentApiOrigin(),
accessToken: env.DASHBOARD_AGENT_SECRET_KEY,
@@ -57,16 +114,44 @@ export async function startDashboardAgentHeadStart(params: {
streamText({
...helper.toStreamTextOptions({ tools }),
model: anthropic(DASHBOARD_AGENT_MODEL),
system,
// A structured system message, not a bare string: without provider options
// Anthropic neither writes nor reads the cache, so this call paid full price
// for the prefix and the agent's step 2 then paid for a fresh write. The tool
// key order is frozen (see `tool-schemas.ts`) so both prefixes are identical
// — the logged fingerprint is how a drift becomes visible.
system: {
role: "system",
content: system,
providerOptions: { anthropic: { cacheControl: PROMPT_CACHE_CONTROL } },
},
onStepFinish: (step) => {
logger.info(
"Dashboard agent prompt cache",
promptCacheAttributes({
source: "head-start",
usage: step.usage,
prefix: describePromptPrefix({ system, tools }),
})
);
},
}),
});
// The webapp is long-lived, so step 1's drain + the handover dispatch run in
// the background after this resolves (createSession + trigger have completed).
// Log a warm-step failure for observability: startHeadStart has already fired
// handover-skip so the agent run exits cleanly, but the client (mounted as
// streaming) then resumes an empty session.out, so the turn looks lost.
completion.catch((error) => {
// Step 1's drain and the handover dispatch continue in the background. On failure
// `startHeadStart` already fired handover-skip, so nothing else writes to session.out.
completion.catch(async (error) => {
logger.error("Dashboard agent head start failed", { chatId: params.chatId, error });
const accessToken = env.DASHBOARD_AGENT_SECRET_KEY;
if (!accessToken) return;
try {
await writeHeadStartFailureToSessionOut(createSessionOutWriter(params.chatId, accessToken));
} catch (writeError) {
logger.error("Failed to write dashboard agent head start error to session.out", {
chatId: params.chatId,
error: writeError,
});
}
});
}
@@ -0,0 +1,112 @@
/**
* The investigation backstop, for cards left `in_progress`. They settle as `inconclusive`,
* conditional on the row still being `in_progress`, so a concluding turn wins the race.
*/
import {
listStaleOpenInvestigations,
settleInvestigationAndCloseCard,
type Investigation,
type SettledInvestigationCard,
} from "@internal/dashboard-agent-db";
import { UNSETTLED_INVESTIGATION_NOTE } from "@internal/dashboard-agent-contracts";
import { dashboardAgentDb } from "~/services/dashboardAgentDb.server";
import { logger } from "~/services/logger.server";
/**
* How long a card may sit `in_progress` before the sweep settles it. Must outlast the
* slowest live turn, which bumps `updated_at` on every revision.
*/
export const INVESTIGATION_STALE_MS = 30 * 60 * 1000;
/** Per-run cap. Oldest first, so the rest land next run. */
const SWEEP_BATCH_LIMIT = 100;
export type InvestigationSweepResult = {
/** Stale `in_progress` rows seen. */
stale: number;
settled: number;
/** Settled rows whose closing card reached the chat. */
closed: number;
/** A turn (or another sweep) settled it first. */
alreadySettled: number;
failed: number;
};
export type InvestigationSweepDeps = {
now?: () => Date;
limit?: number;
listStale?: (params: { olderThan: Date; limit: number }) => Promise<Investigation[]>;
/**
* Settle one row and deliver its closing card as a single operation. Null when the
* row was no longer `in_progress`.
*/
settleAndClose?: (params: {
id: string;
chatId: string;
note: string;
}) => Promise<SettledInvestigationCard | null>;
};
/**
* Settle every card `in_progress` past the grace window. Each row is handled on its own,
* and the run throws at the end if any failed so the job is retried.
*/
export async function sweepDashboardAgentInvestigations(
deps: InvestigationSweepDeps = {}
): Promise<InvestigationSweepResult> {
const now = deps.now?.() ?? new Date();
const limit = deps.limit ?? SWEEP_BATCH_LIMIT;
const listStale =
deps.listStale ?? ((params) => listStaleOpenInvestigations(dashboardAgentDb, params));
const settleAndClose =
deps.settleAndClose ?? ((params) => settleInvestigationAndCloseCard(dashboardAgentDb, params));
const result: InvestigationSweepResult = {
stale: 0,
settled: 0,
closed: 0,
alreadySettled: 0,
failed: 0,
};
const stale = await listStale({
olderThan: new Date(now.getTime() - INVESTIGATION_STALE_MS),
limit,
});
result.stale = stale.length;
for (const investigation of stale) {
try {
// Settling the row fixes nothing on its own: the chat renders the winning card
// from its own transcript, so an unappended settle is still a stuck spinner —
// which is why both writes are one operation that rolls back together.
const outcome = await settleAndClose({
id: investigation.id,
chatId: investigation.chatId,
note: UNSETTLED_INVESTIGATION_NOTE,
});
if (!outcome) {
result.alreadySettled++;
continue;
}
result.settled++;
if (outcome.closed) result.closed++;
} catch (error) {
result.failed++;
logger.error("Dashboard agent investigation sweep: failed to settle an investigation", {
investigationId: investigation.id,
chatId: investigation.chatId,
error,
});
}
}
if (result.failed > 0) {
throw new Error(
`The dashboard agent investigation sweep failed on ${result.failed} investigations`
);
}
return result;
}
@@ -5,6 +5,8 @@ import { featuresForRequest } from "~/features.server";
import { DeleteProjectService } from "./deleteProject.server";
import { getCurrentPlan } from "./platform.v3.server";
import { controlPlaneResolver } from "~/v3/runOpsMigration/controlPlaneResolver.server";
import { commonWorker } from "~/v3/commonWorker.server";
import { logger } from "./logger.server";
export class DeleteOrganizationService {
#prismaClient: PrismaClient;
@@ -86,5 +88,21 @@ export class DeleteOrganizationService {
// runsEnabled + the org's projects (project.deletedAt) changed; drop all cached env rows.
controlPlaneResolver.invalidateOrganization(organization.id);
// Soft-delete the org's dashboard agent chats; retention purges them later. Enqueued,
// not inline: the agent store is a separate database in cloud.
// Best-effort: a failed enqueue must not fail org deletion (the org is already deleted).
try {
await commonWorker.enqueue({
id: `dashboardAgent.purgeOrganization:${organization.id}`,
job: "dashboardAgent.purgeOrganization",
payload: { organizationId: organization.id },
});
} catch (error) {
logger.warn("Failed to enqueue dashboard agent purge for deleted organization", {
organizationId: organization.id,
error,
});
}
}
}
@@ -9,7 +9,7 @@ import {
} from "~/services/apiAuth.server";
import { rbac } from "~/services/rbac.server";
type EnvironmentScopedResource = "envvars" | "apiKeys";
type EnvironmentScopedResource = "envvars" | "apiKeys" | "deployments";
type EnvironmentScopedAuthentication =
| { ok: true; authentication: AuthenticationResult }
@@ -78,6 +78,7 @@ export function authenticateEnvVarApiRequest(
const RESOURCE_LABELS: Record<EnvironmentScopedResource, string> = {
envvars: "environment variables",
apiKeys: "API keys",
deployments: "deployments",
};
/**
@@ -6,7 +6,7 @@ import { logger } from "./logger.server";
import { rbac } from "./rbac.server";
import { decryptToken, encryptToken, hashToken } from "~/utils/tokens.server";
import { env } from "~/env.server";
import { isUserActorToken } from "@trigger.dev/rbac";
import { isUserActorToken, verifyUserActorToken, type UserActorClaims } from "@trigger.dev/rbac";
const tokenValueLength = 40;
//lowercase only, removed 0 and l to avoid confusion
@@ -115,6 +115,11 @@ export async function revokePersonalAccessToken(tokenId: string, userId: string)
export type PersonalAccessTokenAuthenticationResult = {
userId: string;
/**
* Verified claims when the caller presented a delegated user-actor token. They ride on the
* result so no caller can hold the actor without its environment scope.
*/
userActor?: UserActorClaims;
};
/**
@@ -167,11 +172,12 @@ export async function authenticateApiRequestWithPersonalAccessToken(
return;
}
// A user-actor token authenticates as the user wherever a PAT does.
// The plugin verifies it (identity path → no org context to floor against).
// PAT-only: this helper checks no scopes and no capability context, and its callers include
// actions and the admin gate. A delegated token is refused at the entrance and reaches the
// API through the actor-aware routes instead, which do enforce its claims.
if (isUserActorToken(token)) {
const result = await rbac.authenticateUserActor(request, {});
return result.ok ? { userId: result.userId } : undefined;
logger.warn("Rejected a user-actor token at the PAT-only authentication helper");
return;
}
return authenticatePersonalAccessToken(token);
@@ -284,6 +290,41 @@ export function isPersonalAccessToken(token: string) {
return token.startsWith(tokenPrefix);
}
/**
* A user-actor token minted from a PAT carries the source PAT's id (`claims.pat`).
* The token is stateless, so revoking the PAT can't invalidate it by itself — hosts
* recheck the source PAT is still live here. A token with no `pat` (e.g. the dashboard
* agent's) has no source to check and is left alone.
*/
export async function assertSourcePatActive(claims: UserActorClaims): Promise<boolean> {
if (!claims.pat) return true;
const found = await prisma.personalAccessToken.findFirst({
where: { id: claims.pat, revokedAt: null },
select: { id: true },
});
return Boolean(found);
}
/**
* Resolve + source-PAT-recheck a user-actor token for a verify site where `claims` may be
* supplied by the RBAC plugin (the apiBuilder path). We re-verify the bearer locally so the
* recheck reads the token's OWN `pat`, not the plugin's: a plugin image predating the `pat`
* claim would deliver pat-less claims and silently no-op revocation here. Returns the claims
* to act on, or undefined to deny. The direct verify sites (jwt exchange, the UAT preamble)
* don't go through a plugin and call `assertSourcePatActive` on their own verified claims.
*/
export async function resolveAndRecheckUserActorClaims(
claims: UserActorClaims | undefined,
bearer: string
): Promise<UserActorClaims | undefined> {
const verified = await verifyUserActorToken(env.SESSION_SECRET, bearer);
const resolved = claims ?? verified;
if (!resolved) return undefined;
// Recheck against the locally-verified claims when available, so `pat` is authoritative.
return (await assertSourcePatActive(verified ?? resolved)) ? resolved : undefined;
}
/**
* Read-only check that an authorization code is still mintable: it exists, is
* unconsumed (`personalAccessTokenId: null`), and within the TTL. Lets the
@@ -70,7 +70,10 @@ export type ExecuteQueryOptions<TOut extends z.ZodSchema> = Omit<
organizationId: string;
projectId: string;
environmentId: string;
/** The scope of the query - determines tenant isolation */
/**
* The scope of the query - determines tenant isolation. Callers that take it from
* a request body must cap it against the credential first; see `v3/queryScope.ts`.
*/
scope: QueryScope;
period?: string | null;
from?: string | null;
@@ -0,0 +1,162 @@
/**
* `trigger://` URI to dashboard link. Pure: a URI's `{env}` is a RuntimeEnvironment id but
* dashboard URLs need slugs, so the caller supplies the already-resolved scope.
*/
import {
safeParseTriggerUri,
type ParsedTriggerUri,
type TriggerUri,
} from "@internal/dashboard-agent-contracts";
import {
v3DeploymentVersionPath,
v3ErrorPath,
v3QueuesPath,
v3RunPath,
v3RunSpanPath,
v3RunsPath,
} from "~/utils/pathBuilder";
/** Structurally satisfied by `AuthenticatedEnvironment`. */
export type TriggerUriScope = {
/** RuntimeEnvironment id. Must match the URI's `{env}` segment. */
id: string;
slug: string;
project: { slug: string; externalRef: string };
organization: { slug: string };
/** Only a `source` URI needs this. Omitted, a source URI resolves to nothing. */
repository?: { fullName?: string | null; remoteUrl?: string | null } | null;
};
export type ResolvedTriggerUri = {
label: string;
/** Dashboard path, relative to the app origin, unless `external` is set. */
url: string;
/** True when `url` is absolute and off the dashboard, so a host must not route it. */
external?: boolean;
};
/**
* Resolve one URI against one environment, returning `null` rather than guessing. A stored
* transcript can hold foreign URIs, which must never resolve into this project's URL space.
*/
export function resolveTriggerUri(
scope: TriggerUriScope,
uri: TriggerUri | string
): ResolvedTriggerUri | null {
const parsed = safeParseTriggerUri(uri);
if (!parsed.success) return null;
if (!isInScope(scope, parsed.data)) return null;
return resolveInScope(scope, parsed.data);
}
const GITHUB_ORIGIN = "https://github.com";
/** `owner/repo`, GitHub's own character set and nothing that could add a path. */
const FULL_NAME = /^[A-Za-z0-9._-]+\/[A-Za-z0-9._-]+$/;
/**
* The repository's canonical `https://github.com/{owner}/{repo}` base, or null. `remoteUrl` is
* normalized as the deployments UI does; anything but github.com is rejected, not guessed at.
*/
function githubRepoBaseUrl(repository: TriggerUriScope["repository"]): string | null {
const fullName = repository?.fullName?.trim();
if (fullName && FULL_NAME.test(fullName)) return `${GITHUB_ORIGIN}/${fullName}`;
const remoteUrl = repository?.remoteUrl?.trim();
if (!remoteUrl) return null;
const normalized = remoteUrl
.replace(/^git@github\.com:/, `${GITHUB_ORIGIN}/`)
.replace(/^ssh:\/\/git@github\.com\//, `${GITHUB_ORIGIN}/`)
.replace(/\.git$/, "");
let url: URL;
try {
url = new URL(normalized);
} catch {
return null;
}
if (url.hostname !== "github.com") return null;
const path = url.pathname.replace(/^\/+|\/+$/g, "");
if (!FULL_NAME.test(path)) return null;
return `${GITHUB_ORIGIN}/${path}`;
}
/** True when the URI names this exact project and environment. */
function isInScope(scope: TriggerUriScope, parsed: ParsedTriggerUri): boolean {
return parsed.projectRef === scope.project.externalRef && parsed.environmentId === scope.id;
}
function resolveInScope(
scope: TriggerUriScope,
parsed: ParsedTriggerUri
): ResolvedTriggerUri | null {
const { organization, project } = scope;
const environment = { slug: scope.slug };
switch (parsed.kind) {
case "runs":
// The navigate intent's `filters` become URL params at the host.
return {
label: "Runs",
url: v3RunsPath(organization, project, environment),
};
case "run":
return {
label: parsed.runId,
url: v3RunPath(organization, project, environment, { friendlyId: parsed.runId }),
};
case "span":
return {
label: `${parsed.runId} (${parsed.spanId})`,
url: v3RunSpanPath(
organization,
project,
environment,
{ friendlyId: parsed.runId },
{ spanId: parsed.spanId }
),
};
case "error":
return {
label: parsed.fingerprint,
url: v3ErrorPath(organization, project, environment, { fingerprint: parsed.fingerprint }),
};
case "queue":
// The queue detail route is keyed by friendlyId, which a URI doesn't carry, so this
// resolves to the queues list filtered to the name.
return {
label: parsed.name,
url: `${v3QueuesPath(organization, project, environment)}?query=${encodeURIComponent(
parsed.name
)}`,
};
case "deployment":
return {
label: parsed.version,
url: v3DeploymentVersionPath(organization, project, environment, parsed.version),
};
case "source": {
// The URI pins the commit and repo-relative path; the connected repo says where that
// lives. Without a connection there is nothing to open.
const base = githubRepoBaseUrl(scope.repository);
const label = parsed.line === undefined ? parsed.path : `${parsed.path}:${parsed.line}`;
if (!base) return null;
const path = parsed.path.split("/").map(encodeURIComponent).join("/");
const fragment = parsed.line === undefined ? "" : `#L${parsed.line}`;
return {
label,
url: `${base}/blob/${encodeURIComponent(parsed.sha)}/${path}${fragment}`,
external: true,
};
}
// No dashboard page exists for these yet, so the caller renders a label with no link.
case "report":
case "investigation":
return null;
default: {
const unreachable: never = parsed;
throw new Error(`Unhandled trigger:// kind: ${JSON.stringify(unreachable)}`);
}
}
}
@@ -1,5 +1,8 @@
import type { z } from "zod";
import type { ApiAuthenticationResultSuccess } from "../apiAuth.server";
import type {
ApiAuthenticationResultSuccess,
UserActorAuthenticatedActor,
} from "../apiAuth.server";
import type { ActionFunctionArgs, LoaderFunctionArgs } from "@remix-run/server-runtime";
import { json } from "@remix-run/server-runtime";
import { fromZodError } from "zod-validation-error";
@@ -9,8 +12,11 @@ import { rbac } from "../rbac.server";
import { authenticateBearerWithTelemetry } from "~/services/authTelemetry.server";
import type { RbacAbility, RbacResource } from "@trigger.dev/rbac";
import { isUserActorToken } from "@trigger.dev/rbac";
import type { PersonalAccessTokenAuthenticationResult } from "../personalAccessToken.server";
import { updateLastAccessedAtIfStale } from "../personalAccessToken.server";
import {
resolveAndRecheckUserActorClaims,
updateLastAccessedAtIfStale,
} from "../personalAccessToken.server";
import { assertUserActorScope } from "../userActorEnvironment.server";
import { safeJsonParse } from "~/utils/json";
import type { AuthenticatedWorkerInstance } from "~/v3/services/worker/workerGroupTokenService.server";
import { WorkerGroupTokenService } from "~/v3/services/worker/workerGroupTokenService.server";
@@ -422,7 +428,10 @@ export function createLoaderApiRoute<
const apiVersion = getApiVersion(request);
const result = await tenantContext.run(
tenantContextFromAuthEnvironment(authenticationResult.environment),
tenantContextFromAuthEnvironment(
authenticationResult.environment,
authenticationResult.actor
),
() =>
handler({
params: parsedParams,
@@ -458,6 +467,10 @@ export function createLoaderApiRoute<
};
}
// `environmentId` is checked against a user-actor token's environment claim, so an env-scoped
// route enforces the scope by declaring it here.
type PATRouteContext = { organizationId?: string; projectId?: string; environmentId?: string };
type PATRouteBuilderOptions<
TParamsSchema extends AnyZodSchema | undefined = undefined,
TSearchParamsSchema extends AnyZodSchema | undefined = undefined,
@@ -479,9 +492,7 @@ type PATRouteBuilderOptions<
? z.infer<TParamsSchema>
: undefined,
request: Request
) =>
| { organizationId?: string; projectId?: string }
| Promise<{ organizationId?: string; projectId?: string }>;
) => PATRouteContext | Promise<PATRouteContext>;
authorization?: {
action: string;
resource: (
@@ -500,6 +511,18 @@ type PATRouteBuilderOptions<
};
};
type PATLoaderRouteBuilderOptions<
TParamsSchema extends AnyZodSchema | undefined = undefined,
TSearchParamsSchema extends AnyZodSchema | undefined = undefined,
THeadersSchema extends AnyZodSchema | undefined = undefined,
> = PATRouteBuilderOptions<TParamsSchema, TSearchParamsSchema, THeadersSchema> & {
// Opts a contextless route into being reachable by an environment-scoped user-actor token.
// Only for routes whose answer is the caller's own identity (their orgs, their projects) and
// which mutate nothing — otherwise such a token is refused for want of anything to check.
// Loaders only: an action mutates by definition, so the action options forbid it.
identityOnly?: true;
};
type PATHandlerFunction<
TParamsSchema extends AnyZodSchema | undefined,
TSearchParamsSchema extends AnyZodSchema | undefined,
@@ -516,7 +539,7 @@ type PATHandlerFunction<
headers: THeadersSchema extends z.ZodFirstPartySchemaTypes | z.ZodDiscriminatedUnion<any, any>
? z.infer<THeadersSchema>
: undefined;
authentication: PersonalAccessTokenAuthenticationResult;
authentication: UserActorAuthenticatedActor;
ability: RbacAbility;
request: Request;
apiVersion: API_VERSIONS;
@@ -527,7 +550,7 @@ export function createLoaderPATApiRoute<
TSearchParamsSchema extends AnyZodSchema | undefined = undefined,
THeadersSchema extends AnyZodSchema | undefined = undefined,
>(
options: PATRouteBuilderOptions<TParamsSchema, TSearchParamsSchema, THeadersSchema>,
options: PATLoaderRouteBuilderOptions<TParamsSchema, TSearchParamsSchema, THeadersSchema>,
handler: PATHandlerFunction<TParamsSchema, TSearchParamsSchema, THeadersSchema>
) {
return async function loader({ request, params }: LoaderFunctionArgs) {
@@ -537,6 +560,7 @@ export function createLoaderPATApiRoute<
headers: headersSchema,
corsStrategy = "none",
context: contextFn,
identityOnly,
authorization,
} = options;
@@ -614,7 +638,7 @@ export function createLoaderPATApiRoute<
// cached timestamp is fresher than the throttle window).
const ctx = contextFn ? await contextFn(parsedParams, request) : {};
let authenticationResult: PersonalAccessTokenAuthenticationResult;
let authenticationResult: UserActorAuthenticatedActor;
let ability: RbacAbility;
const bearer = request.headers
@@ -632,7 +656,16 @@ export function createLoaderPATApiRoute<
corsStrategy !== "none"
);
}
authenticationResult = { userId: uatAuth.userId };
const claims = await resolveAndRecheckUserActorClaims(uatAuth.claims, bearer);
if (!claims) {
return await wrapResponse(
request,
json({ error: "Invalid user-actor token" }, { status: 401 }),
corsStrategy !== "none"
);
}
await assertUserActorScope(claims, ctx, { identityOnly });
authenticationResult = { userId: uatAuth.userId, userActor: claims };
ability = uatAuth.ability;
} else {
// PAT: validate + compute the cap-and-floor ability in one query.
@@ -721,6 +754,9 @@ type PATActionRouteBuilderOptions<
// A single verb, or a list for multi-method routes (e.g. ["PATCH", "DELETE"]).
method?: PATActionMethod | PATActionMethod[];
body?: TBodySchema;
// `identityOnly` waives the contextless refusal for reads that mutate nothing. An action
// never qualifies, so it cannot be declared here.
identityOnly?: never;
};
type PATActionHandlerFunction<
@@ -743,7 +779,7 @@ type PATActionHandlerFunction<
body: TBodySchema extends z.ZodFirstPartySchemaTypes | z.ZodDiscriminatedUnion<any, any>
? z.infer<TBodySchema>
: undefined;
authentication: PersonalAccessTokenAuthenticationResult;
authentication: UserActorAuthenticatedActor;
ability: RbacAbility;
request: Request;
apiVersion: API_VERSIONS;
@@ -879,7 +915,7 @@ export function createActionPATApiRoute<
// caller's role floor for the cap intersection (see the loader builder).
const ctx = contextFn ? await contextFn(parsedParams, request) : {};
let authenticationResult: PersonalAccessTokenAuthenticationResult;
let authenticationResult: UserActorAuthenticatedActor;
let ability: RbacAbility;
const bearer = request.headers
@@ -895,7 +931,16 @@ export function createActionPATApiRoute<
corsStrategy !== "none"
);
}
authenticationResult = { userId: uatAuth.userId };
const claims = await resolveAndRecheckUserActorClaims(uatAuth.claims, bearer);
if (!claims) {
return await wrapResponse(
request,
json({ error: "Invalid user-actor token" }, { status: 401 }),
corsStrategy !== "none"
);
}
await assertUserActorScope(claims, ctx);
authenticationResult = { userId: uatAuth.userId, userActor: claims };
ability = uatAuth.ability;
} else {
const patAuth = await rbac.authenticatePat(request, ctx);
@@ -1278,7 +1323,10 @@ export function createActionApiRoute<
}
const result = await tenantContext.run(
tenantContextFromAuthEnvironment(authenticationResult.environment),
tenantContextFromAuthEnvironment(
authenticationResult.environment,
authenticationResult.actor
),
() =>
handler({
params: parsedParams,
@@ -1543,7 +1591,10 @@ export function createMultiMethodApiRoute<
// Dispatch to method handler
const result = await tenantContext.run(
tenantContextFromAuthEnvironment(authenticationResult.environment),
tenantContextFromAuthEnvironment(
authenticationResult.environment,
authenticationResult.actor
),
() =>
methodConfig.handler({
params: parsedParams,
@@ -1,12 +1,7 @@
import { AsyncLocalStorage } from "node:async_hooks";
import type { AuthenticatedEnvironment } from "./apiAuth.server";
// All fields are optional. The middleware establishes an empty scope per
// request; entry points fill what they know:
// - URL-matching paths get the slug trio from the Express middleware (zero IO).
// - The `_app` layout adds `userId` for any authenticated request.
// - The env layout adds tenant IDs / env type after its own existing DB query.
// - API routes get the full set up-front from `authenticationResult.environment`.
// Every field is optional: each entry point fills only what it already knows.
export type TenantContext = {
userId?: string;
orgSlug?: string;
@@ -35,9 +30,13 @@ export const tenantContext = {
},
};
export function tenantContextFromAuthEnvironment(env: AuthenticatedEnvironment): TenantContext {
// `actor` wins over `orgMember`, which only exists on dev environments.
export function tenantContextFromAuthEnvironment(
env: AuthenticatedEnvironment,
actor?: { sub: string }
): TenantContext {
return {
userId: env.orgMember?.userId,
userId: actor?.sub ?? env.orgMember?.userId,
orgSlug: env.organization.slug,
projectSlug: env.project.slug,
envSlug: env.slug,
@@ -0,0 +1,49 @@
import { isUserActorToken, verifyUserActorToken, type UserActorClaims } from "@trigger.dev/rbac";
import { env } from "~/env.server";
import { authenticateRequest, type AuthenticationResult } from "~/services/apiAuth.server";
import { assertSourcePatActive } from "~/services/personalAccessToken.server";
/**
* Auth preamble for `api.v1` routes that opt into delegated user-actor tokens alongside a PAT
* or org token. A UAT authenticates as its user, so the result is the `personalAccessToken` shape.
*/
export type UatAuthentication = {
authenticationResult: AuthenticationResult;
/** Present only when the caller presented a user-actor token. */
userActor?: UserActorClaims;
};
export async function authenticateUatOrApiRequest(
request: Request
): Promise<UatAuthentication | undefined> {
const bearer = request.headers
.get("Authorization")
?.replace(/^Bearer /, "")
.trim();
if (bearer && isUserActorToken(bearer)) {
const claims = await verifyUserActorToken(env.SESSION_SECRET, bearer);
if (!claims) return undefined;
// A token minted from a PAT dies with it — the PAT must still be live.
if (!(await assertSourcePatActive(claims))) return undefined;
return {
// The claims ride on the authentication result too: resolving an environment from it
// enforces the token's environment scope, so no route has to remember to.
authenticationResult: {
type: "personalAccessToken",
result: { userId: claims.userId },
userActor: claims,
},
userActor: claims,
};
}
const authenticationResult = await authenticateRequest(request, {
personalAccessToken: true,
organizationAccessToken: true,
apiKey: false,
});
if (!authenticationResult) return undefined;
return { authenticationResult };
}
@@ -0,0 +1,126 @@
/**
* The Dashboard Agent uses an environment-scoped form of the existing user-actor credential.
* MCP and the CLI may use their existing ones. Both normalize into the same authorized
* capability context, so every route calls in here rather than deriving the rule itself.
*
* The rule: a token signed for one environment may only act inside it. A route that names nothing
* to check the claim against is refused too, unless it declares itself identity-only. Anything
* with no claim is environment-agnostic and unaffected — except a dashboard-agent token, which
* always carries one, so its absence is a failed mint rather than a flow. Mismatches throw 403.
*/
import { json } from "@remix-run/server-runtime";
import { type UserActorClaims } from "@trigger.dev/rbac";
import { $replica } from "~/db.server";
export const FORBIDDEN_ENVIRONMENT_CODE = "forbidden_environment";
const DASHBOARD_AGENT_CLIENT = "dashboard-agent";
export function assertUserActorEnvironment(
userActor: UserActorClaims | undefined,
environmentId: string
): void {
if (!userActor) return;
if (!userActor.environmentId) {
assertClaimIsOptional(userActor);
return;
}
if (userActor.environmentId === environmentId) return;
throw forbiddenEnvironment("This token isn't scoped to that environment.");
}
/** The same check for a route that names an org/project rather than one environment. */
export async function assertUserActorScope(
userActor: UserActorClaims | undefined,
scope: { organizationId?: string; projectId?: string; environmentId?: string },
route?: { identityOnly?: boolean }
): Promise<void> {
if (!userActor) return;
if (!userActor.environmentId) {
assertClaimIsOptional(userActor);
return;
}
if (scope.environmentId) {
assertUserActorEnvironment(userActor, scope.environmentId);
return;
}
// A route that names nothing offers no way to honour the claim, so it isn't reachable unless it
// has declared itself identity-only.
if (!scope.organizationId && !scope.projectId) {
if (route?.identityOnly) return;
throw forbiddenEnvironment("This token is scoped to an environment this route doesn't name.");
}
const environment = await $replica.runtimeEnvironment.findFirst({
where: { id: userActor.environmentId },
select: { organizationId: true, projectId: true },
});
// A claim naming an environment that no longer exists cannot be checked, so it isn't honoured.
if (!environment) {
throw forbiddenEnvironment("This token isn't scoped to an environment.");
}
if (scope.projectId && environment.projectId !== scope.projectId) {
throw forbiddenEnvironment("This token isn't scoped to that project.");
}
if (scope.organizationId && environment.organizationId !== scope.organizationId) {
throw forbiddenEnvironment("This token isn't scoped to that organization.");
}
}
/** `scoped: false` keeps the project-wide answer every claimless caller already gets. */
export type UserActorEnvironmentScope =
| { scoped: false }
| { scoped: true; environmentId: string; slug: string; organizationId: string };
/**
* The claim as a mandatory filter for a route that lists across a project. A conflicting request
* filter is refused rather than overridden, so a caller never gets another environment's answer.
*/
export async function resolveUserActorEnvironmentScope(
userActor: UserActorClaims | undefined,
target: { projectId: string; requestedEnvironmentSlugs?: string[] }
): Promise<UserActorEnvironmentScope> {
if (!userActor) return { scoped: false };
if (!userActor.environmentId) {
assertClaimIsOptional(userActor);
return { scoped: false };
}
const environment = await $replica.runtimeEnvironment.findFirst({
where: { id: userActor.environmentId, projectId: target.projectId },
select: { id: true, slug: true, organizationId: true },
});
// A claim naming an environment that can't be found in this project isn't honoured.
if (!environment) {
throw forbiddenEnvironment("This token isn't scoped to that project.");
}
const requested = target.requestedEnvironmentSlugs;
if (requested && (requested.length !== 1 || requested[0] !== environment.slug)) {
throw forbiddenEnvironment(`This token is scoped to the "${environment.slug}" environment.`);
}
return {
scoped: true,
environmentId: environment.id,
slug: environment.slug,
organizationId: environment.organizationId,
};
}
function assertClaimIsOptional(userActor: UserActorClaims): void {
if (userActor.client !== DASHBOARD_AGENT_CLIENT) return;
throw forbiddenEnvironment("This token isn't scoped to an environment.");
}
function forbiddenEnvironment(error: string) {
return json({ error, code: FORBIDDEN_ENVIRONMENT_CODE }, { status: 403 });
}
+10 -21
View File
@@ -658,12 +658,9 @@
--sidebar: var(--color-background-dimmed);
/* Code block styling */
& [data-code-block-container] {
& [data-streamdown="code-block"] {
@apply rounded-sm my-2 border-charcoal-700;
}
& [data-code-block] {
border-top: none;
}
& p {
@apply my-1;
@@ -701,9 +698,10 @@
& li {
@apply my-0.5;
}
/* Inline code (not in pre blocks) */
/* Theme-mapped, not raw charcoal: fenced blocks stay dark via shiki. */
& code:not(pre code) {
@apply bg-charcoal-700 px-1 py-0.5 rounded-sm text-text-bright font-mono;
@apply px-1 py-0.5 rounded-sm text-text-bright font-mono;
background-color: var(--muted);
}
& blockquote {
@apply border-l-2 border-charcoal-600 pl-3 my-2 italic;
@@ -725,25 +723,16 @@
}
& th,
& td {
@apply border border-charcoal-600 px-2 py-1 text-left;
@apply border border-grid-bright px-2 py-1 text-left;
}
& th {
@apply bg-charcoal-700 font-semibold;
@apply font-semibold;
background-color: var(--muted);
}
& [data-code-block-header] {
@apply bg-charcoal-800 text-text-dimmed border-b border-charcoal-700;
}
/* Override the bg-muted/40 class to let inline styles work */
& [data-code-block] pre {
background-color: inherit !important;
@apply scrollbar-thin scrollbar-track-transparent scrollbar-thumb-charcoal-600;
}
& [data-code-block] pre code {
@apply bg-transparent;
}
& [data-code-block] .line {
@apply leading-relaxed;
/* The body scrolls and sizes the code; streamdown ships it at text-sm. */
& [data-streamdown="code-block-body"] {
@apply text-xs leading-relaxed scrollbar-thin scrollbar-track-transparent scrollbar-thumb-charcoal-600;
}
}
@@ -0,0 +1,45 @@
import { describe, expect, it } from "vitest";
import { readBoundedBodyText } from "./boundedRequestBody.server";
/** A body with no `content-length`, delivered in chunks, counting what was pulled. */
function streamed(chunkCount: number, chunkBytes: number) {
let pulled = 0;
const body = new ReadableStream<Uint8Array>({
pull(controller) {
if (pulled >= chunkCount) {
controller.close();
return;
}
pulled += 1;
controller.enqueue(new Uint8Array(chunkBytes).fill(97));
},
});
const request = new Request("http://localhost/in", {
method: "POST",
body,
// @ts-expect-error — required for a streamed request body.
duplex: "half",
});
return { request, pulled: () => pulled };
}
describe("readBoundedBodyText", () => {
it("returns a body under the limit", async () => {
const { request } = streamed(2, 8);
expect(await readBoundedBodyText(request, 1024)).toEqual({ ok: true, text: "a".repeat(16) });
});
it("stops reading as soon as the limit is crossed", async () => {
const { request, pulled } = streamed(100, 64);
expect(await readBoundedBodyText(request, 128)).toEqual({ ok: false, reason: "too_large" });
// Three chunks: two fit, the third crossed it. Nothing beyond was ever pulled.
expect(pulled()).toBe(3);
});
it("treats a missing body as empty", async () => {
const request = new Request("http://localhost/in", { method: "POST" });
expect(await readBoundedBodyText(request, 8)).toEqual({ ok: true, text: "" });
});
});
@@ -0,0 +1,35 @@
/**
* Reading a request body with a ceiling. `request.text()` buffers the whole body before the
* caller can look at its size, so a route that only checks afterwards has already paid for it.
*/
export type BoundedBody = { ok: true; text: string } | { ok: false; reason: "too_large" };
/** Stops at the first chunk that crosses `maxBytes` and cancels the stream. */
export async function readBoundedBodyText(
request: Request,
maxBytes: number
): Promise<BoundedBody> {
if (!request.body) return { ok: true, text: "" };
const reader = request.body.getReader();
const chunks: Uint8Array[] = [];
let received = 0;
try {
for (;;) {
const { done, value } = await reader.read();
if (done) break;
if (!value) continue;
received += value.byteLength;
if (received > maxBytes) {
await reader.cancel();
return { ok: false, reason: "too_large" };
}
chunks.push(value);
}
} finally {
reader.releaseLock();
}
return { ok: true, text: Buffer.concat(chunks).toString("utf8") };
}
@@ -0,0 +1,164 @@
import { describe, expect, it } from "vitest";
import { faviconUrl } from "./favicon";
import {
BASE_IMG_SRC_SOURCES,
buildImgSrcDirective,
parseCspImageOrigins,
withImgSrc,
} from "./cspImageOrigins";
/** True if a source expression in the directive would match the given image URL. */
function directivePermits(directive: string, imageUrl: string): boolean {
const url = new URL(imageUrl);
return directive
.split(" ")
.slice(1)
.some((source) => {
if (!source.startsWith("http")) return false;
const parsed = new URL(source);
if (parsed.protocol !== url.protocol || parsed.host !== url.host) return false;
return parsed.pathname === "/" || parsed.pathname === url.pathname;
});
}
describe("parseCspImageOrigins", () => {
it("accepts exact https origins, with or without a port", () => {
const { origins, rejected } = parseCspImageOrigins(
"https://sso.example.com, https://images.example.com:8443"
);
expect(origins).toEqual(["https://sso.example.com", "https://images.example.com:8443"]);
expect(rejected).toEqual([]);
});
it("returns nothing when unset or empty", () => {
expect(parseCspImageOrigins(undefined).origins).toEqual([]);
expect(parseCspImageOrigins(" , ,").origins).toEqual([]);
});
it("deduplicates repeated origins", () => {
const { origins } = parseCspImageOrigins(
"https://sso.example.com,https://sso.example.com/,https://sso.example.com"
);
expect(origins).toEqual(["https://sso.example.com"]);
});
it.each([
["*", "wildcards are not allowed, list each origin exactly"],
["https://*.example.com", "wildcards are not allowed, list each origin exactly"],
["https://example.com/avatars", "must be an origin only, with no path, query or hash"],
["https://example.com?x=1", "must be an origin only, with no path, query or hash"],
["https://example.com#frag", "must be an origin only, with no path, query or hash"],
["example.com", "is not a valid absolute URL"],
["https://user:pw@example.com", "must not contain credentials"],
["https://a.com;script-src", "must not contain ';' or ',' — these delimit CSP directives"],
])("rejects %s and says why", (value, reason) => {
const { origins, rejected } = parseCspImageOrigins(value);
expect(origins).toEqual([]);
expect(rejected).toEqual([{ value, reason }]);
});
it("does not let a ';' smuggle a second directive into img-src", () => {
const { origins } = parseCspImageOrigins("https://a.com;script-src");
expect(origins).toEqual([]);
expect(buildImgSrcDirective(origins)).not.toContain("script-src");
});
it("splits on ',' so a comma can never ride inside a single origin", () => {
// "https://a.com" is valid; the "x" fragment after the comma is rejected on its own.
const { origins } = parseCspImageOrigins("https://a.com,x");
expect(origins).toEqual(["https://a.com"]);
expect(origins.some((origin) => origin.includes(","))).toBe(false);
});
it("keeps the valid entries when a sibling entry is rejected", () => {
const { origins, rejected } = parseCspImageOrigins(
"https://*.evil.com,https://sso.example.com"
);
expect(origins).toEqual(["https://sso.example.com"]);
expect(rejected.map((entry) => entry.value)).toEqual(["https://*.evil.com"]);
});
it("rejects http outside development", () => {
const { origins, rejected } = parseCspImageOrigins("http://sso.example.com");
expect(origins).toEqual([]);
expect(rejected).toEqual([
{ value: "http://sso.example.com", reason: 'scheme "http:" is not https:' },
]);
});
it("allows http only when allowHttp is set", () => {
const { origins, rejected } = parseCspImageOrigins("http://localhost:4000", {
allowHttp: true,
});
expect(origins).toEqual(["http://localhost:4000"]);
expect(rejected).toEqual([]);
});
it("rejects a non-http scheme even when allowHttp is set", () => {
const { origins, rejected } = parseCspImageOrigins("ftp://example.com", { allowHttp: true });
expect(origins).toEqual([]);
expect(rejected).toEqual([
{ value: "ftp://example.com", reason: 'scheme "ftp:" is not http: or https:' },
]);
});
});
describe("buildImgSrcDirective", () => {
it("is self, data, blob, the SSO avatar hosts and the favicon endpoint by default", () => {
expect(buildImgSrcDirective()).toBe(
"img-src 'self' data: blob: https://avatars.githubusercontent.com https://lh3.googleusercontent.com https://www.google.com/s2/favicons"
);
});
it("permits the org avatar URL the app actually stores", () => {
expect(directivePermits(buildImgSrcDirective(), faviconUrl("example.com"))).toBe(true);
});
it("permits nothing else on the favicon host", () => {
expect(directivePermits(buildImgSrcDirective(), "https://www.google.com/beacon.png")).toBe(
false
);
});
it("permits both OAuth avatar hosts", () => {
const directive = buildImgSrcDirective();
expect(directivePermits(directive, "https://avatars.githubusercontent.com/u/1?v=4")).toBe(true);
expect(directivePermits(directive, "https://lh3.googleusercontent.com/a/abc=s96-c")).toBe(true);
});
it("has no wildcard host and no bare scheme host", () => {
const directive = buildImgSrcDirective(parseCspImageOrigins("https://sso.example.com").origins);
expect(directive).not.toContain("*");
// The avatar hosts are exact origins; a wildcard over them would not be.
expect(directive).not.toContain("*.googleusercontent.com");
expect(directive).not.toMatch(/(^|\s)https?:(\s|$)/);
});
it("appends configured origins after the base sources", () => {
expect(buildImgSrcDirective(["https://sso.example.com"])).toBe(
`img-src ${BASE_IMG_SRC_SOURCES.join(" ")} https://sso.example.com`
);
});
});
describe("withImgSrc", () => {
const directive = buildImgSrcDirective();
it("is the whole policy when a route set nothing", () => {
expect(withImgSrc(null, directive)).toBe(directive);
});
it("appends to a policy that has other directives", () => {
expect(withImgSrc("frame-ancestors 'self';", directive)).toBe(
`frame-ancestors 'self'; ${directive}`
);
});
it("leaves a route's own img-src untouched", () => {
const routePolicy = "img-src 'none'";
expect(withImgSrc(routePolicy, directive)).toBe(routePolicy);
expect(withImgSrc("default-src 'self'; img-src 'none'", directive)).toBe(
"default-src 'self'; img-src 'none'"
);
});
});
+120
View File
@@ -0,0 +1,120 @@
/**
* The document `img-src` allowlist. Remote images are a beacon channel: rendering
* one is the outbound request, no click needed. So no wildcard host and no bare
* scheme. Operator-supplied entries are exact origins; a base source may pin a path
* to narrow the host further.
*/
/**
* Always allowed: own origin, inline data, object URLs, the SSO avatar hosts, and the
* favicon endpoint org avatars are stored as (see `utils/favicon.ts`). The path pins
* that one endpoint — CSP matches the path and ignores the query string.
*/
export const BASE_IMG_SRC_SOURCES = [
"'self'",
"data:",
"blob:",
"https://avatars.githubusercontent.com",
"https://lh3.googleusercontent.com",
"https://www.google.com/s2/favicons",
] as const;
export type RejectedOrigin = { value: string; reason: string };
export type ParsedImageOrigins = {
/** Accepted, canonicalised (`scheme://host[:port]`) and deduplicated. */
origins: string[];
rejected: RejectedOrigin[];
};
export type ParseImageOriginsOptions = {
/** Only a local development deployment may serve images over plain http. */
allowHttp?: boolean;
};
/**
* Parses a comma-separated `CSP_IMG_SRC_ALLOWLIST`. Never throws: bad entries are
* reported in `rejected` so the caller can warn and boot with the valid ones.
*/
export function parseCspImageOrigins(
raw: string | undefined | null,
options: ParseImageOriginsOptions = {}
): ParsedImageOrigins {
const allowHttp = options.allowHttp ?? false;
const origins: string[] = [];
const seen = new Set<string>();
const rejected: RejectedOrigin[] = [];
for (const entry of (raw ?? "").split(",")) {
const value = entry.trim();
if (value.length === 0) continue;
const reason = rejectionReason(value, allowHttp);
if (reason) {
rejected.push({ value, reason });
continue;
}
const url = new URL(value);
const origin = `${url.protocol}//${url.host}`;
if (seen.has(origin)) continue;
seen.add(origin);
origins.push(origin);
}
return { origins, rejected };
}
/** Returns why the entry is not an acceptable origin, or undefined if it is one. */
function rejectionReason(value: string, allowHttp: boolean): string | undefined {
if (value.includes("*")) {
return "wildcards are not allowed, list each origin exactly";
}
if (/\s/.test(value)) {
return "contains whitespace";
}
// `;` and `,` delimit CSP directives / source lists; an entry containing one would
// land verbatim in the space-joined img-src and inject or truncate a directive.
if (/[;,]/.test(value)) {
return "must not contain ';' or ',' — these delimit CSP directives";
}
let url: URL;
try {
url = new URL(value);
} catch {
return "is not a valid absolute URL";
}
const allowedProtocols = allowHttp ? ["https:", "http:"] : ["https:"];
if (!allowedProtocols.includes(url.protocol)) {
return allowHttp
? `scheme "${url.protocol}" is not http: or https:`
: `scheme "${url.protocol}" is not https:`;
}
if (url.host.length === 0) {
return "has no host";
}
if (url.username.length > 0 || url.password.length > 0) {
return "must not contain credentials";
}
if (url.pathname !== "/" || url.search.length > 0 || url.hash.length > 0) {
return "must be an origin only, with no path, query or hash";
}
return undefined;
}
/** The full directive: the base sources plus any configured extra origins. */
export function buildImgSrcDirective(extraOrigins: readonly string[] = []): string {
return ["img-src", ...BASE_IMG_SRC_SOURCES, ...extraOrigins].join(" ");
}
/**
* Appends the directive to whatever a route already set, rather than replacing it.
* A route that set its own `img-src` keeps it.
*/
export function withImgSrc(existing: string | null | undefined, directive: string): string {
if (!existing) return directive;
if (/(^|;)\s*img-src\s/.test(existing)) return existing;
return `${existing.replace(/;\s*$/, "")}; ${directive}`;
}
@@ -4,20 +4,16 @@ import { FEATURE_FLAG } from "~/v3/featureFlags";
import { makeFlag } from "~/v3/featureFlags.server";
/**
* Whether the in-dashboard AI agent is available to this user in this org.
* Gated by the global / per-org `hasDashboardAgentAccess` flag, with
* `DASHBOARD_AGENT_ENABLED` as the global default (a per-org override wins).
* Admins/impersonators bypass it only when `DASHBOARD_AGENT_ADMIN_PREVIEW` is on
* (default off). Enforced server-side so a non-flagged user can't start sessions.
* Whether the in-dashboard AI agent is available to this user in this org, per the
* `hasDashboardAgentAccess` flag with a per-org override winning. Must stay server-side.
* Both env defaults are off, so an unflagged install has no agent and can start no session.
*/
export async function canAccessDashboardAgent(options: {
userId: string;
isAdmin: boolean;
isImpersonating: boolean;
organizationSlug: string;
// When the caller already has the org's `featureFlags` loaded (e.g. a layout
// loader that queried the org with a membership check), pass them to skip the
// extra org lookup. Omit it and we query the org ourselves.
// The org's already-loaded `featureFlags`. Omitted means we query the org ourselves.
orgFeatureFlags?: Record<string, unknown> | null;
}): Promise<boolean> {
const { userId, isAdmin, isImpersonating, organizationSlug, orgFeatureFlags } = options;
+77 -1
View File
@@ -1,5 +1,5 @@
import { Logger } from "@trigger.dev/core/logger";
import { Worker as RedisWorker } from "@trigger.dev/redis-worker";
import { CronSchema, Worker as RedisWorker } from "@trigger.dev/redis-worker";
import { DeliverEmailSchema } from "emails";
import { z } from "zod";
import { env } from "~/env.server";
@@ -11,6 +11,12 @@ import {
runAttioUserSync,
runAttioWorkspaceSync,
} from "~/services/attio.server";
import {
purgeDashboardAgentChatsForOrganization,
sweepDashboardAgentSoftDeletedChats,
} from "~/services/dashboardAgentChatRetention.server";
import { sweepDashboardAgentTurnEvals } from "~/services/dashboardAgentEvalRetention.server";
import { sweepDashboardAgentInvestigations } from "~/services/dashboardAgentInvestigationSweep.server";
import { logger } from "~/services/logger.server";
import {
MembershipDevEnvironmentsSchema,
@@ -37,6 +43,11 @@ function initializeWorker() {
logger.debug(`👨‍🏭 Initializing common worker at host ${env.COMMON_WORKER_REDIS_HOST}`);
// Only schedule the agent maintenance cron where the agent is actually set up. Otherwise
// its sweeps hit a missing schema and drip a dead-letter entry every run.
const dashboardAgentConfigured =
env.DASHBOARD_AGENT_ENABLED === "1" || Boolean(env.DASHBOARD_AGENT_DATABASE_URL);
const worker = new RedisWorker({
name: "common-worker",
redisOptions,
@@ -146,6 +157,25 @@ function initializeWorker() {
maxAttempts: 5,
},
},
// Stuck investigation cards and turn-eval retention.
"dashboardAgent.maintenance": {
schema: CronSchema,
visibilityTimeoutMs: 60_000 * 5,
...(dashboardAgentConfigured ? { cron: "*/5 * * * *", jitterInMs: 30_000 } : {}),
retry: {
maxAttempts: 1,
},
},
// Soft-deletes a deleted organization's chats; the maintenance sweep purges them.
"dashboardAgent.purgeOrganization": {
schema: z.object({
organizationId: z.string(),
}),
visibilityTimeoutMs: 60_000,
retry: {
maxAttempts: 5,
},
},
},
concurrency: {
workers: env.COMMON_WORKER_CONCURRENCY_WORKERS,
@@ -204,6 +234,52 @@ function initializeWorker() {
const service = new BulkActionService();
await service.process(payload.bulkActionId);
},
"dashboardAgent.maintenance": async () => {
// Each backstop runs independently; the first failure is rethrown at the end.
let failure: unknown;
try {
const investigations = await sweepDashboardAgentInvestigations();
if (investigations.stale > 0) {
logger.debug("Dashboard agent investigation sweep", investigations);
}
} catch (error) {
failure ??= error;
}
// Retention on the judged-turn rows. Independent of the agent being configured.
try {
const evals = await sweepDashboardAgentTurnEvals();
if (evals.purged > 0) {
logger.debug("Dashboard agent turn-eval retention", evals);
}
} catch (error) {
failure ??= error;
}
// Hard-delete chats soft-deleted past the retention window, with their children.
try {
const chats = await sweepDashboardAgentSoftDeletedChats();
if (chats.purged > 0) {
logger.debug("Dashboard agent chat retention", chats);
}
} catch (error) {
failure ??= error;
}
if (failure) throw failure;
},
"dashboardAgent.purgeOrganization": async ({ payload }) => {
const soft = await purgeDashboardAgentChatsForOrganization({
organizationId: payload.organizationId,
});
if (soft > 0) {
logger.debug("Dashboard agent organization purge", {
organizationId: payload.organizationId,
softDeleted: soft,
});
}
},
},
});
+3
View File
@@ -103,6 +103,9 @@ export function detectQueryTables(query: string, allowedTableNames: Set<string>)
visitSelectSet(tableExpr as SelectSetQuery);
}
}
// The `ON expr` can embed a SELECT that reads a real table, e.g.
// `JOIN x ON id IN (SELECT … FROM runs)`.
scanForSubqueries(node.constraint);
if (node.next_join) visitJoin(node.next_join);
}
+14 -10
View File
@@ -25,22 +25,26 @@ export function makeFlag(_prisma: PrismaClientOrTransaction = prisma) {
async function flag<T extends FeatureFlagKey>(
opts: FlagsOptions<T>
): Promise<z.infer<(typeof FeatureFlagCatalog)[T]> | undefined> {
const flagSchema = FeatureFlagCatalog[opts.key];
const override = opts.overrides?.[opts.key];
if (override !== undefined) {
const parsed = flagSchema.safeParse(override);
if (parsed.success) {
return parsed.data;
}
// an override that fails the schema is ignored: the global value still wins
}
const value = await _prisma.featureFlag.findFirst({
where: {
key: opts.key,
},
});
const flagSchema = FeatureFlagCatalog[opts.key];
if (opts.overrides?.[opts.key] !== undefined) {
const parsed = flagSchema.safeParse(opts.overrides[opts.key]);
if (parsed.success) {
return parsed.data;
}
}
if (value !== null) {
const parsed = flagSchema.safeParse(value.value);
+31
View File
@@ -7,6 +7,8 @@ export const FEATURE_FLAG = {
hasLogsPageAccess: "hasLogsPageAccess",
hasAiAccess: "hasAiAccess",
hasDashboardAgentAccess: "hasDashboardAgentAccess",
dashboardAgentTurnEvalsEnabled: "dashboardAgentTurnEvalsEnabled",
promotedDashboardAgentPrompt: "promotedDashboardAgentPrompt",
hasComputeAccess: "hasComputeAccess",
hasPrivateConnections: "hasPrivateConnections",
hasSso: "hasSso",
@@ -43,6 +45,15 @@ export const FeatureFlagCatalog = {
// Gates the in-dashboard AI agent panel. Controllable globally and per-org
// (org wins). Defaults off via DASHBOARD_AGENT_ENABLED.
[FEATURE_FLAG.hasDashboardAgentAccess]: z.coerce.boolean(),
// Whether this org's agent turns may be sampled for the quality judge. A data-handling
// switch, not an entitlement: an org that turns it off has its turns judged never, and a
// setting that can't be read is treated as off. Per-org override wins; on by default.
// Strict z.boolean(): coercion reads the string "false" as true, which would keep judging
// an org that asked us to stop.
[FEATURE_FLAG.dashboardAgentTurnEvalsEnabled]: z.boolean(),
// A JSON string because this catalog is scalar-only. Validated where it's read, in
// `suggested-prompts/promotedPrompt.server.ts`.
[FEATURE_FLAG.promotedDashboardAgentPrompt]: z.string(),
[FEATURE_FLAG.hasComputeAccess]: z.coerce.boolean(),
[FEATURE_FLAG.hasPrivateConnections]: z.coerce.boolean(),
[FEATURE_FLAG.hasSso]: z.coerce.boolean(),
@@ -162,6 +173,26 @@ export function resolveInternalApiOriginEnabled({
return globalDefault;
}
/**
* Whether the org set `dashboardAgentTurnEvalsEnabled` to something the schema rejects.
* That flag is a consent switch, not an entitlement, so an unreadable override must not fall
* through to the global default the way `resolveInternalApiOriginEnabled` does: the org that
* wrote it was trying to say something, and the only safe reading of an unknown answer is no.
*/
export function hasUnreadableTurnEvalsOverride(orgFeatureFlags: unknown): boolean {
if (!orgFeatureFlags || typeof orgFeatureFlags !== "object" || Array.isArray(orgFeatureFlags)) {
return false;
}
const override = (orgFeatureFlags as Record<string, unknown>)[
FEATURE_FLAG.dashboardAgentTurnEvalsEnabled
];
if (override === undefined) return false;
return !FeatureFlagCatalog[FEATURE_FLAG.dashboardAgentTurnEvalsEnabled].safeParse(override)
.success;
}
export type FlagControlType =
| { type: "boolean" }
| { type: "enum"; options: string[] }
+42
View File
@@ -0,0 +1,42 @@
import type { QueryScope } from "~/v3/querySchemas";
/**
* The widest scope a credential may query at.
*
* `executeQuery` always isolates by organization and widens or narrows from there
* on the caller's `scope`. That makes the request body, not the credential, the
* ceiling — which is wrong for a public access token: it is minted for one
* environment and handed to a browser, so anyone holding it could read the whole
* organization's analytics by changing one field.
*
* A secret key is deliberately NOT capped here. It is a server-side credential the
* organization's own owner installs, and capping it would change the public API's
* behaviour for callers who query at organization scope today. A session or PAT
* caller (the Query page) never goes through this: it picks its scope in the UI,
* authorized by organization membership.
*/
export type QueryScopeCeiling = "environment" | "unbounded";
export type QueryScopeDecision = { ok: true; scope: QueryScope } | { ok: false; error: string };
/**
* Rejected rather than narrowed. Silently answering about one environment when the
* caller asked about the organization gives them a number that means something else,
* with nothing in the response to say so.
*/
export function resolveQueryScope(args: {
ceiling: QueryScopeCeiling;
requested: QueryScope;
}): QueryScopeDecision {
if (args.ceiling === "unbounded") return { ok: true, scope: args.requested };
if (args.requested === "environment") return { ok: true, scope: "environment" };
return {
ok: false,
error: `This token is scoped to one environment, so it can't run a ${args.requested}-scoped query. Use scope "environment", or a secret key.`,
};
}
/** A public access token is environment-bound; every other bearer credential isn't. */
export function queryScopeCeilingFor(authenticationType: string): QueryScopeCeiling {
return authenticationType === "PUBLIC_JWT" ? "environment" : "unbounded";
}
+49
View File
@@ -0,0 +1,49 @@
/**
* `getQueueDepthSparklines` emits a row only for buckets that reported, so a caller has to place
* every row on the bucket grid itself. Depth is carry-forward filled: no emission means unchanged,
* not zero. Throttled is not filled — only real per-bucket counts tint a bar.
*/
export type QueueDepthBucketRow = { bucket: string; depth: number; throttled: number };
export type QueueDepthGrid = { startMs: number; bucketIntervalMs: number; numBuckets: number };
/** Rows placed on the grid by bucket index. Rows outside the window are dropped. */
export function indexQueueDepthRows(
rows: QueueDepthBucketRow[],
grid: QueueDepthGrid
): Map<number, { depth: number; throttled: number }> {
const byIndex = new Map<number, { depth: number; throttled: number }>();
for (const row of rows) {
const bucketMs = Date.parse(row.bucket.replace(" ", "T") + "Z");
if (Number.isNaN(bucketMs)) continue;
const index = Math.round((bucketMs - grid.startMs) / grid.bucketIntervalMs);
if (index < 0 || index >= grid.numBuckets) continue;
byIndex.set(index, { depth: row.depth, throttled: row.throttled });
}
return byIndex;
}
/** A fixed-width series per grid bucket, so a gap can never shift later points in time. */
export function fillQueueDepthSeries(
byIndex: Map<number, { depth: number; throttled: number }>,
numBuckets: number
): { depth: number[]; throttled: number[] } {
const depth: number[] = new Array(numBuckets);
const throttled: number[] = new Array(numBuckets);
let last = 0;
for (let i = 0; i < numBuckets; i++) {
const bucket = byIndex.get(i);
if (bucket !== undefined) last = bucket.depth;
depth[i] = last;
throttled[i] = bucket?.throttled ?? 0;
}
return { depth, throttled };
}
export function queueDepthSeries(
rows: QueueDepthBucketRow[],
grid: QueueDepthGrid
): { depth: number[]; throttled: number[] } {
return fillQueueDepthSeries(indexQueueDepthRows(rows, grid), grid.numBuckets);
}
@@ -392,6 +392,7 @@ export class DeliverAlertService extends BaseService {
break;
}
case "ERROR_GROUP": {
// Payload-carried alert types create no ProjectAlert row, so never seen here.
break;
}
default: {
@@ -747,6 +748,7 @@ export class DeliverAlertService extends BaseService {
break;
}
case "ERROR_GROUP": {
// Payload-carried alert types create no ProjectAlert row, so never seen here.
break;
}
default: {
@@ -1023,6 +1025,7 @@ export class DeliverAlertService extends BaseService {
}
}
case "ERROR_GROUP": {
// Payload-carried alert types create no ProjectAlert row, so never seen here.
break;
}
default: {
+2 -1
View File
@@ -5,7 +5,7 @@
"sideEffects": false,
"scripts": {
"build": "run-s build:** && pnpm run upload:sourcemaps",
"build:remix": "remix vite:build",
"build:remix": "cross-env NODE_OPTIONS=\"--max-old-space-size=8192\" remix vite:build",
"build:server": "esbuild --platform=node --format=cjs ./server.ts --outdir=build --sourcemap",
"build:otlpworker": "esbuild --platform=node --format=cjs --bundle ./app/v3/otlpTransformWorker.ts --outfile=build/otlpTransformWorker.cjs --sourcemap",
"build:sentry": "esbuild --platform=node --format=cjs --outbase=. ./sentry.server.ts ./app/utils/sentryTraceContext.server.ts --outdir=build --sourcemap",
@@ -54,6 +54,7 @@
"@internal/cache": "workspace:*",
"@internal/compute": "workspace:*",
"@internal/dashboard-agent": "workspace:*",
"@internal/dashboard-agent-contracts": "workspace:*",
"@internal/dashboard-agent-db": "workspace:*",
"@internal/llm-model-catalog": "workspace:*",
"@internal/metrics-pipeline": "workspace:*",
+32 -88
View File
@@ -1,14 +1,18 @@
import { prisma } from "./app/db.server";
import { boundedIn } from "@trigger.dev/database";
import { createOrganization } from "./app/models/organization.server";
import { createProject } from "./app/models/project.server";
import { ClickHouse } from "@internal/clickhouse";
import type { QueueMetricsRawV1Input } from "@internal/clickhouse";
import { generateFriendlyId } from "./app/v3/friendlyIdentifiers";
// App modules compile to CommonJS under tsx, so import them as default bindings.
import dbServer from "./app/db.server";
import organizationServer from "./app/models/organization.server";
import projectServer from "./app/models/project.server";
import friendlyIdentifiers from "./app/v3/friendlyIdentifiers";
// Queue metrics simulator: writes realistic raw rows into a synthetic tenant's
// queue_metrics_raw_v1 and lets the MV build queue_metrics_v1 (the same path the real
// consumer uses), so the dashboard can be built without the run engine. See TRI-10407.
const { prisma } = dbServer;
const { createOrganization } = organizationServer;
const { createProject } = projectServer;
const { generateFriendlyId } = friendlyIdentifiers;
// Writes raw rows into queue_metrics_raw_v1 and lets the MVs build the rollups. See TRI-10407.
const ORG_TITLE = "Queue Metrics Dev";
const PROJECT_NAME = "queue-metrics-demo";
@@ -17,10 +21,9 @@ type Rng = () => number;
type QueueProfile = {
name: string;
limit: (bucket: number) => number;
arrivals: (bucket: number, rng: Rng) => number; // expected new runs enqueued this bucket
arrivals: (bucket: number, rng: Rng) => number;
waitBaseMs: number;
sparse?: boolean; // emit no rows when the queue is fully idle (tests carry-forward gaps)
// Concurrency-key queue: adds CK-health gauge fields + live ckIndex staging (--usage)
sparse?: boolean;
ck?: {
backlogged: (bucket: number, rng: Rng) => number;
maxWaitMs: (bucket: number, rng: Rng) => number;
@@ -32,10 +35,6 @@ type Scenario = {
queues: QueueProfile[];
};
// ---------------------------------------------------------------------------
// CLI args
// ---------------------------------------------------------------------------
function parseArgs(argv: string[]) {
const flags: Record<string, string> = {};
for (let i = 0; i < argv.length; i++) {
@@ -60,10 +59,6 @@ function parseDuration(s: string): number {
return n * { s: 1, m: 60, h: 3600, d: 86400 }[unit]!;
}
// ---------------------------------------------------------------------------
// Deterministic RNG + distributions
// ---------------------------------------------------------------------------
function mulberry32(seed: number): Rng {
let a = seed >>> 0;
return () => {
@@ -104,17 +99,12 @@ function formatChDateTime(date: Date): string {
return date.toISOString().slice(0, 19).replace("T", " ");
}
// ---------------------------------------------------------------------------
// Scenarios
// ---------------------------------------------------------------------------
const steady = (): QueueProfile[] => [
{ name: "emails", limit: () => 20, arrivals: (_b, r) => poisson(12, r), waitBaseMs: 40 },
{ name: "webhooks", limit: () => 15, arrivals: (_b, r) => poisson(9, r), waitBaseMs: 40 },
{ name: "reports", limit: () => 10, arrivals: (_b, r) => poisson(5, r), waitBaseMs: 60 },
];
// periodic bursts every ~30 buckets
const bursty = (name: string, limit: number, base: number): QueueProfile => ({
name,
limit: () => limit,
@@ -135,7 +125,6 @@ const scenarios: Record<string, (totalBuckets: number, bucketSec: number) => Sce
queues: [bursty("ingest", 20, 6), bursty("transform", 20, 7)],
}),
// Tela case: sum of per-queue limits far exceeds the env limit, so queues compete.
"over-allocated-env": () => ({
description: "Sum(queue limits)=120 >> env limit=40; env saturates, queues env-limited",
envLimit: () => 40,
@@ -191,8 +180,6 @@ const scenarios: Record<string, (totalBuckets: number, bucketSec: number) => Sce
],
}),
// Pagination + relevance-ranking design surface: one runaway queue, a busy-but-healthy
// head, a bursty middle, and a long sparse tail across 61 queues (the list pages at 25).
"many-queues": () => ({
description:
"61 queues: one runaway, busy head, bursty middle, long sparse tail (pagination + ranking)",
@@ -224,9 +211,6 @@ const scenarios: Record<string, (totalBuckets: number, bucketSec: number) => Sce
],
}),
// Per-tenant concurrency keys: a hog tenant periodically floods the queue and starves
// the others, so the CK charts (keys with backlog, most-starved wait) and the live
// per-key table on the queue detail page have something to show. Use with --usage.
"tenant-hotspot": () => ({
description:
"CK queue where a hog tenant starves others: CK charts + live key table (use --usage)",
@@ -249,10 +233,9 @@ const scenarios: Record<string, (totalBuckets: number, bucketSec: number) => Sce
],
}),
// Default: one env with a variety of queue behaviours + occasional env saturation.
mixed: (totalBuckets) => ({
description: "variety of queue profiles in one env, with occasional env saturation",
envLimit: (b) => (b % 40 < 12 ? 45 : 70), // dips low periodically to flip env saturation
envLimit: (b) => (b % 40 < 12 ? 45 : 70),
queues: [
{ name: "emails", limit: () => 20, arrivals: (_b, r) => poisson(12, r), waitBaseMs: 40 },
bursty("webhooks", 20, 6),
@@ -274,18 +257,13 @@ const scenarios: Record<string, (totalBuckets: number, bucketSec: number) => Sce
}),
};
// ---------------------------------------------------------------------------
// Simulation
// ---------------------------------------------------------------------------
type Ids = { organization_id: string; project_id: string; environment_id: string };
const WAIT_SIGMA = 0.6;
const NACK_RATE = 0.02;
const DLQ_RATE = 0.004;
type CounterOp = "enqueue" | "started" | "ack" | "nack" | "dlq";
// Per-(queue, op) odometers, mirroring the production emitter: cumulative readings with a
// cum=0 baseline on the first one, so deltaSumTimestamp captures the 0->1 delta.
// Cumulative odometers: the first reading must be cum=0 so deltaSumTimestamp sees the 0->1 delta.
type CounterState = Record<CounterOp, number>[];
function counterRows(
@@ -326,8 +304,7 @@ function newCounterState(n: number): CounterState {
return Array.from({ length: n }, () => ({ enqueue: 0, started: 0, ack: 0, nack: 0, dlq: 0 }));
}
// Per-key simulation for CK profiles: 12 tenants (tenant-01 is the hog, matching
// stageRedisUsage), per-tenant backlog drained round-robin, per-tenant odometers.
// tenant-01 is the hog here and in stageRedisUsage; keep the two in step.
const CK_TENANT_COUNT = 12;
type CkSimState = { backlog: number[]; counters: Map<number, Record<CounterOp, number>> };
const ckSim = new Map<number, CkSimState>();
@@ -370,9 +347,7 @@ function ckCounterRows(
return rows;
}
// Advance one bucket of the simulation for every queue, returning the raw rows to insert.
// `backlog` and `counters` are mutated in place so state carries across buckets (and into
// live mode).
// `backlog` and `counters` are mutated in place: state carries across buckets and into live mode.
function simulateBucket(
scenario: Scenario,
bucket: number,
@@ -392,12 +367,11 @@ function simulateBucket(
for (let q = 0; q < n; q++) {
limit[q] = scenario.queues[q].limit(bucket);
const arrivals = Math.min(500, scenario.queues[q].arrivals(bucket, rng));
const prior = backlog[q]; // backlog carried from earlier buckets, before this bucket's arrivals
backlog[q] += arrivals; // arrivals join the backlog; recorded as enqueues below
const prior = backlog[q];
backlog[q] += arrivals;
(desired as any)[q] = { arrivals, prior, want: Math.min(limit[q], backlog[q]) };
}
// Env cap: if the queues collectively want more concurrency than the env allows, scale down.
const sumWant = desired.reduce((s: number, d: any) => s + d.want, 0);
const scale = sumWant > envLimit && sumWant > 0 ? envLimit / sumWant : 1;
@@ -413,8 +387,7 @@ function simulateBucket(
envQueued += queued[q];
}
// Order keys are time-based (like the production stream ids) so appended runs and live
// mode stay monotonic; the per-bucket sequence keeps them unique within a bucket.
// Order keys must be monotonic across processes, so they are time-based plus a per-bucket seq.
let bucketSeq = 0;
const orderKey = () => bucketEpochSec * 1_000_000 + bucketSeq++;
@@ -423,14 +396,13 @@ function simulateBucket(
const profile = scenario.queues[q];
const started = running[q];
const arrivals = (desired[q] as any).arrivals as number;
const prior = (desired[q] as any).prior as number; // depth a starting run actually queued behind
backlog[q] = queued[q]; // carry the unserved remainder forward
const prior = (desired[q] as any).prior as number;
backlog[q] = queued[q];
if (profile.sparse && arrivals === 0 && started === 0 && prior === 0) {
continue; // fully idle: leave a gap so carry-forward is exercised
continue;
}
// CK-health fields stay coherent with the depth: no queued runs means no backlogged keys.
const ckBacklogged = profile.ck
? queued[q] > 0
? Math.max(1, Math.min(profile.ck.backlogged(bucket, rng), queued[q]))
@@ -461,8 +433,6 @@ function simulateBucket(
rows.push(...counterRows(counters, q, ids, profile.name, eventTime, orderKey, "enqueue"));
}
// Per-key rows for CK profiles: assign arrivals hog-weighted, drain round-robin
// (fair share), then emit per-tenant odometers + a per-key gauge per active tenant.
if (profile.ck) {
let ckq = ckSim.get(q);
if (!ckq) {
@@ -544,10 +514,6 @@ function simulateBucket(
return rows;
}
// ---------------------------------------------------------------------------
// ClickHouse
// ---------------------------------------------------------------------------
function clickhouse(): ClickHouse {
const clickhouseUrl = process.env.CLICKHOUSE_URL ?? process.env.EVENTS_CLICKHOUSE_URL;
if (!clickhouseUrl) {
@@ -555,7 +521,7 @@ function clickhouse(): ClickHouse {
process.exit(1);
}
const url = new URL(clickhouseUrl);
// Allowlist local hosts only (this script TRUNCATEs), and never echo the URL (it carries creds).
// Local hosts only (this script deletes rows); never echo the URL, it carries credentials.
const localHosts = new Set(["localhost", "127.0.0.1", "::1", "0.0.0.0"]);
if (!localHosts.has(url.hostname)) {
console.error(`Refusing to run against a non-local ClickHouse host: ${url.hostname}`);
@@ -597,7 +563,6 @@ async function resetEnv(ch: ClickHouse, environmentId: string) {
console.log(`Reset queue metrics for environment ${environmentId}`);
}
// Fake running counts in the run-queue Redis (Running column + allocation usage bars).
// Reconciled every run: staged with --usage, cleared otherwise.
async function stageRedisUsage(scenario: Scenario, ids: Ids, seed: number, clear: boolean) {
const host = process.env.RUN_ENGINE_RUN_QUEUE_REDIS_HOST ?? process.env.REDIS_HOST ?? "localhost";
@@ -617,16 +582,11 @@ async function stageRedisUsage(scenario: Scenario, ids: Ids, seed: number, clear
const logicalBase = `{org:${ids.organization_id}}:proj:${ids.project_id}:env:${ids.environment_id}:queue:`;
const base = `${prefix}${logicalBase}`;
// Env-level structures the Queues list "Queued"/"Running" blocks read:
// lengthOfEnvQueue -> ZCARD(envQueueKey) (ZSET, no proj section)
// concurrencyOfEnvQueue -> SCARD(envCurrentDequeuedKey) (SET)
// We accumulate the per-queue staged counts below and stage these so the blocks
// equal the table's per-queue sums instead of showing 0/0.
// envQueue is a ZSET with no proj section; envCurrentDequeued is a SET with one.
const envQueueKey = `${prefix}{org:${ids.organization_id}}:env:${ids.environment_id}`;
const envCurrentDequeuedKey = `${prefix}{org:${ids.organization_id}}:proj:${ids.project_id}:env:${ids.environment_id}:currentDequeued`;
await redis.del(envQueueKey, envCurrentDequeuedKey);
// Table Queued = ZCARD(base) + lengthCounter (base zset unstaged -> only CK queues
// contribute their lengthCounter). Table Running = SCARD(currentDequeued) per queue.
// Per-queue Queued = ZCARD(base) + lengthCounter; Running = SCARD(currentDequeued).
let envQueuedTotal = 0;
let envRunningTotal = 0;
@@ -634,8 +594,7 @@ async function stageRedisUsage(scenario: Scenario, ids: Ids, seed: number, clear
const key = `${base}${profile.name}:currentDequeued`;
await redis.del(key);
// CK staging (ckIndex + per-key subqueues) feeds the live per-key table on the queue
// detail page. Members are stored unprefixed, exactly like the run-queue Lua does.
// ckIndex members are stored unprefixed, exactly like the run-queue Lua does.
const ckIndexKey = `${base}${profile.name}:ckIndex`;
const lengthCounterKey = `${base}${profile.name}:lengthCounter`;
const staleMembers = await redis.zrange(ckIndexKey, 0, -1);
@@ -646,7 +605,6 @@ async function stageRedisUsage(scenario: Scenario, ids: Ids, seed: number, clear
if (clear) continue;
const limit = profile.limit(0);
// First queue rides at/over its limit, the rest at 30-90%, sparse mostly idle.
const count = profile.sparse
? rng() < 0.3
? 1
@@ -684,15 +642,12 @@ async function stageRedisUsage(scenario: Scenario, ids: Ids, seed: number, clear
await redis.zadd(ckIndexKey, now - oldestAgeMs, member);
totalCkQueued += queuedCount;
}
// The aggregate "Queued now" reads ZCARD(base) + this counter; keep them coherent.
await redis.set(lengthCounterKey, totalCkQueued, "EX", 24 * 3600);
envQueuedTotal += totalCkQueued;
}
}
// Stage the env-level structures so the list-page blocks match the table sums.
// Members are unique across queues (each per-queue set uses its own key, but the
// env set/zset needs distinct members to reach the summed cardinality).
// The env set/zset needs members distinct across queues to reach the summed cardinality.
if (!clear) {
if (envRunningTotal > 0) {
await redis.sadd(
@@ -719,13 +674,8 @@ async function stageRedisUsage(scenario: Scenario, ids: Ids, seed: number, clear
}
}
// ---------------------------------------------------------------------------
// Main
// ---------------------------------------------------------------------------
// Make the synthetic project a V2 engine project with a current dev worker + a Postgres
// TaskQueue per simulated queue, so the /queues list renders the V2 table (it pages from
// Postgres and gates on engine version; ClickHouse only holds the metrics).
// The /queues list pages from Postgres and gates on engine version, so the project needs
// engine V2, a worker, and a TaskQueue row per simulated queue.
async function ensureTaskQueues(
scenario: Scenario,
projectId: string,
@@ -769,9 +719,7 @@ async function ensureTaskQueues(
projectId,
type: "NAMED",
},
// Reset any dashboard override left from manual testing: re-seeding overwrites the
// materialized concurrencyLimit, so a surviving override percent/base would contradict it
// (e.g. "10 (77%)" with an env limit of 25).
// Re-seeding overwrites concurrencyLimit, so a surviving override would contradict it.
update: {
concurrencyLimit,
concurrencyLimitBase: null,
@@ -782,8 +730,6 @@ async function ensureTaskQueues(
});
}
// Drop queues left over from a previously seeded scenario so switching scenarios
// does not leave metric-less rows in the list.
const { count: pruned } = await prisma.taskQueue.deleteMany({
where: {
runtimeEnvironmentId,
@@ -917,7 +863,6 @@ async function main() {
`Backfilling ${totalBuckets} x ${bucketSec}s buckets (${flags.window ?? "2h"}) for ${scenario.queues.length} queues...`
);
// Backfill: buckets from (now - window) up to now, aligned to the bucket grid.
const nowBucket = Math.floor(Date.now() / 1000 / bucketSec) * bucketSec;
const startBucket = nowBucket - totalBuckets * bucketSec;
const counters = newCounterState(scenario.queues.length);
@@ -942,8 +887,7 @@ async function main() {
await insertBatched(ch, rows, nonce);
console.log(`Inserted ${rows.length} raw rows.`);
// Merge the AggregatingMergeTree partials so argMax "current value" widgets read cleanly.
// The real pipeline relies on background merges; the simulator forces it for a tidy demo.
// The rollups are AggregatingMergeTrees; a read straight after the insert can't wait for merges.
const raw = (
ch.writer as unknown as { client: { command: (a: { query: string }) => Promise<unknown> } }
).client;

Some files were not shown because too many files have changed in this diff Show More