feat(webapp): dashboard agent — chat, reports, investigate (#4418)
## What & why This is the system behind the Dashboard Agent — an assistant that answers questions about a project's runs, errors, queues, deploys and health, and can investigate failures end to end. The agent runs as a chat.agent task in its own Trigger project. It has no access to the main database or ClickHouse; all platform data is read through the public API using a delegated, read-only user token. Everything here is behind `canAccessDashboardAgent` and inert with the flag off. The UI that mounts the panel lands in #4529. ## Stack `#4418` (this, base) ← `#4529` UI ← `#4525` Watch ← `#4516` storybook gallery. The scenario/contract reference for the whole stack is `internal-packages/dashboard-agent/GUIDEBOOK.md` (it lands on the Watch branch): it states, per feature, what makes each thing happen and where that is decided. ## What's inside **Agent runtime and tools** — `internal-packages/dashboard-agent`: prompt, tool set (API reads, TRQL query, docs, navigation, evidence/investigations, repo source), conversation compaction, a prompt-prefix token budget pinned by snapshot test, and sampled LLM-judged turn evals. The package cannot import webapp server code, which is what makes the "no DB access" claim structural rather than a convention. **Contracts** — `internal-packages/dashboard-agent-contracts`: `trigger://` URIs, intents, and the block envelope every rendered card travels in. **Conversation store** — `internal-packages/dashboard-agent-db`: drizzle over postgres-js in its own `trigger_dashboard_agent` Postgres schema, plus one additive migration. **Auth boundary** — the user-actor token gains an optional environment claim; one guard (`userActorEnvironment.server.ts`) enforces it so routes don't each re-derive the rule. Token minting, cap ceiling, and the RBAC fallback path for self-hosted. **Transport** — webapp resource routes that mint the token and proxy each turn, and SDK-side mid-turn reconnect. **Public API the agent reads through** — orgs, projects, environments, runs, queue metrics, workers, a run's commit metadata, repo snapshot, reports, and `POST /api/v1/query`. **Reports** — the health report's layout is declared once and shared by the card, the markdown surface and the JSON/MCP surface, so the same report reads the same in the dashboard, the terminal and an editor. **Block renderers** — the report and investigation cards the flows above already emit (`app/components/dashboard-agent/`). The panel that hosts them, and the rest of the chat UI, is #4529. **Query safety and CSP** — see below. ## Key decisions - **The agent is a separate Trigger project, not webapp code.** It reads platform data over the public API with a delegated user-actor token whose `cap` ceilings it to read scopes. No Prisma, no ClickHouse, no webapp imports. - **The PAT-only auth helper now refuses user-actor tokens.** This is an intentional behavioral change: its callers consume only a bare userId and do not enforce delegated-token capabilities. Actor-aware routes continue through the scoped route builders instead. - **RBAC fallback builds a delegated token's ability from its own cap**, never the blanket ability a PAT gets (read-only when the token declares none). Without this, the agent's read-only cap would buy a write JWT on self-hosted. - **Org creation checks RBAC only for user-actor tokens, and only after the env gate**, so an install with `ORG_CREATION_API_ENABLED` off returns 404 rather than 403, and an ordinary PAT never consults an ability the route has no org to scope. Both orderings are pinned by test. - **The query path is read-only in depth.** TRQL rejects write statements at the grammar level (they don't parse, rather than being filtered), ClickHouse runs with `readonly=1`, and the org/project/env filters are injected server-side from the credential — the request body cannot widen scope. An unparseable query denies instead of falling through to the permissive resource. - **Document-wide img-src CSP.** Remote images are an outbound-request/exfiltration surface, so the policy permits only own-origin/data/blob, the required SSO avatar hosts, and the favicon endpoint. Operators can add exact origins through CSP_IMG_SRC_ALLOWLIST; wildcard hosts and bare schemes are intentionally not allowed. - **The chat transport reconnects on a mid-turn EOF** (`@trigger.dev/sdk`). A body that ends without a turn-complete is terminal only when the server says `X-Session-Settled: true`; otherwise the transport resubscribes from `lastEventId` with bounded backoff, and any record re-earns the budget. Previously a closed long-poll window or a proxy restart left the reply stuck as if still generating. - **Conversations live in their own datastore**, schema-scoped and foreign-key-free (it references `organizationId`/`userId` by id, because in cloud it is a different database). It is a display read-model for the History tab and transport resume; `chat.agent`'s object-store snapshot remains the model's source of truth. - **Deterministic first.** Reports and health checks contain no LLM — they are computed from the same data the dashboard shows, and the model only narrates and links them. That is what makes a number in an answer auditable. ## Testing - 63 new test files, run with `pnpm run test --filter webapp` and per-package vitest. Heaviest coverage on the auth boundary (`userActorPatOnlyBoundary`, `userActorTokenClaimsAndScopes`, `contextlessPatRoutes`, `rbacFallbackBranch`), TRQL read-only, the report layout, and the SDK reconnect. - The agent package has a separate eval lane (`pnpm run test:evals`, `vitest.eval.config.ts`) that hits the real model, so it never runs in `pnpm test`. - Live-tested against a local stack scenario by scenario; the GUIDEBOOK lists the condition each behaviour is expected under, which is what those runs were checked against. ## Changelog `.server-changes/dashboard-agent.md`, plus changesets for `@trigger.dev/core` (report schemas), `@trigger.dev/sdk` (chat reconnect) and the CLI's `mint-token` help text.
This commit is contained in:
@@ -0,0 +1,7 @@
|
||||
---
|
||||
"@trigger.dev/core": patch
|
||||
"@trigger.dev/sdk": patch
|
||||
"trigger.dev": patch
|
||||
---
|
||||
|
||||
Chat in the browser now reconnects when the connection drops mid-turn, instead of leaving the reply stuck as if it were still generating. Reports can be fetched as structured data with the `json` format, and the shortest report period is now one minute (`30m`, `1h`, `7d`). The `mint-token` command's help is clearer too: a token minted without `--cap` is read-only, and `--ttl` shows the correct maximum lifetime of 7 days.
|
||||
@@ -154,7 +154,7 @@ PASS: one run, `run_<RID>` (status maps to `FAILED`). Proves `filter[error]` ->
|
||||
### 6. Attribution — `mint-token` -> JWT exchange records the acting user
|
||||
|
||||
```bash
|
||||
TOKEN=$(cli mint-token --profile $PROFILE --client errors-api-e2e 2>/dev/null) # UAT
|
||||
TOKEN=$(cli mint-token --profile $PROFILE --client errors-api-e2e --cap read:errors,write:errors 2>/dev/null) # UAT
|
||||
ENVJWT=$(curl -sS -X POST "$B/api/v1/projects/$REF/dev/jwt" -H "Authorization: Bearer $TOKEN" \
|
||||
-H 'Content-Type: application/json' -d '{"claims":{"scopes":["read:errors","write:errors"]}}' \
|
||||
| python3 -c "import sys,json;print(json.load(sys.stdin)['token'])")
|
||||
|
||||
@@ -0,0 +1,5 @@
|
||||
# Generated, not hand-written: collapsed in diffs and excluded from language stats.
|
||||
internal-packages/dashboard-agent-db/drizzle/meta/*.json linguist-generated=true
|
||||
internal-packages/dashboard-agent-db/drizzle/meta/** linguist-generated=true
|
||||
**/__snapshots__/*.snap linguist-generated=true
|
||||
pnpm-lock.yaml linguist-generated=true
|
||||
@@ -85,3 +85,5 @@ ailogger-output.log
|
||||
|
||||
# observability-map CLI output artifact, not committed
|
||||
observability-map.json
|
||||
|
||||
.claude/worktrees/
|
||||
|
||||
@@ -0,0 +1,8 @@
|
||||
---
|
||||
area: webapp
|
||||
type: feature
|
||||
---
|
||||
|
||||
Meet the dashboard agent: a chat in every environment that answers questions about your runs, queues, errors and health with real data and links, replacing Ask AI everywhere it used to appear. Investigate a failed run, an error, a backed-up queue or a run that hasn't started to get a worked-through answer — what happened, why, and how to fix it, with every claim linked to the runs, errors and deploys behind it. It reads your data read-only, works on preview and dev branches with that branch's own data, and reads the same everywhere — dashboard, terminal, editor. A very long chat keeps working: the agent summarises the earlier part and carries on.
|
||||
|
||||
A sample of conversations is scored automatically so the agent keeps getting better; only the score and a one-line summary are kept, never your messages, data or code, and we can switch it off for your organization on request. The Docs button is gone from page headers — ask the agent instead, or open Documentation from Help & Feedback. Separately, a queue's wait times, peak depth, throughput and throttling can now be read from the API.
|
||||
@@ -7,6 +7,9 @@ node_modules
|
||||
/cypress/screenshots
|
||||
/cypress/videos
|
||||
|
||||
# Output of `pnpm run agent-ui:screenshots`
|
||||
/screenshots
|
||||
|
||||
/app/styles/tailwind.css
|
||||
|
||||
# Ensure the .env symlink is not removed by accident
|
||||
|
||||
@@ -1,3 +1,9 @@
|
||||
/**
|
||||
* @deprecated Superseded by the dashboard agent (`components/dashboard-agent`). Nothing mounts
|
||||
* this any more — every Ask AI entry point now opens Ask Trigger. Kept until the agent has
|
||||
* shipped, then removed along with `@kapaai/react-sdk` and `KAPA_AI_WEBSITE_ID`.
|
||||
*/
|
||||
|
||||
import {
|
||||
ArrowPathIcon,
|
||||
ArrowUpIcon,
|
||||
@@ -81,6 +87,8 @@ function useAskAIState() {
|
||||
* it around the popover, not inside, so the dialog and shortcut survive the popover closing.
|
||||
* `children` receives the open function, or undefined when Ask AI is unavailable (self-hosted, no
|
||||
* Kapa website id, or SSR).
|
||||
*
|
||||
* @deprecated See the note at the top of this file.
|
||||
*/
|
||||
export function AskAIRoot({
|
||||
children,
|
||||
@@ -137,6 +145,7 @@ function AskAIRootProvider({
|
||||
);
|
||||
}
|
||||
|
||||
/** @deprecated See the note at the top of this file. */
|
||||
export function AskAI({ isCollapsed = false }: { isCollapsed?: boolean }) {
|
||||
const { isManagedCloud } = useFeatures();
|
||||
const websiteId = useKapaWebsiteId();
|
||||
|
||||
@@ -0,0 +1,80 @@
|
||||
import { createElement } from "react";
|
||||
import { renderToStaticMarkup } from "react-dom/server";
|
||||
import { describe, expect, it } from "vitest";
|
||||
import { restrictModelUrls, StreamdownRenderer } from "./StreamdownRenderer";
|
||||
|
||||
// streamdown calls urlTransform(url, key, node) to compute each url attribute; a
|
||||
// returned undefined removes the attribute, so no request is ever issued.
|
||||
const img = { tagName: "img" } as any;
|
||||
const link = { tagName: "a" } as any;
|
||||
|
||||
describe("restrictModelUrls (image src)", () => {
|
||||
it("drops a remote model-authored image (the favicon beacon)", () => {
|
||||
expect(
|
||||
restrictModelUrls("https://www.google.com/s2/favicons?domain=evil", "src", img)
|
||||
).toBeUndefined();
|
||||
});
|
||||
|
||||
it("drops any absolute or protocol-relative remote image", () => {
|
||||
expect(restrictModelUrls("http://evil.tld/pixel.gif", "src", img)).toBeUndefined();
|
||||
expect(restrictModelUrls("//evil.tld/pixel.gif", "src", img)).toBeUndefined();
|
||||
});
|
||||
|
||||
it("keeps inline and same-origin images", () => {
|
||||
expect(restrictModelUrls("data:image/png;base64,AAAA", "src", img)).toBe(
|
||||
"data:image/png;base64,AAAA"
|
||||
);
|
||||
expect(restrictModelUrls("blob:abc", "src", img)).toBe("blob:abc");
|
||||
expect(restrictModelUrls("/local/pic.png", "src", img)).toBe("/local/pic.png");
|
||||
});
|
||||
});
|
||||
|
||||
describe("restrictModelUrls (link href)", () => {
|
||||
it("keeps http(s), mailto and relative links", () => {
|
||||
expect(restrictModelUrls("https://trigger.dev/docs", "href", link)).toBe(
|
||||
"https://trigger.dev/docs"
|
||||
);
|
||||
expect(restrictModelUrls("http://example.com", "href", link)).toBe("http://example.com");
|
||||
expect(restrictModelUrls("mailto:hi@trigger.dev", "href", link)).toBe("mailto:hi@trigger.dev");
|
||||
expect(restrictModelUrls("/runs/123", "href", link)).toBe("/runs/123");
|
||||
});
|
||||
|
||||
it("drops unsafe link schemes", () => {
|
||||
expect(restrictModelUrls("javascript:alert(1)", "href", link)).toBeUndefined();
|
||||
expect(restrictModelUrls("data:text/html,<script>", "href", link)).toBeUndefined();
|
||||
});
|
||||
});
|
||||
|
||||
// Force the lazy component to load, then return its resolved default so we can render it
|
||||
// synchronously. This proves the policy is actually wired into the JSX, not just exported.
|
||||
async function resolveStreamdownRenderer() {
|
||||
const lazy = StreamdownRenderer as unknown as {
|
||||
_payload: unknown;
|
||||
_init: (payload: unknown) => (props: { children: string }) => JSX.Element;
|
||||
};
|
||||
try {
|
||||
lazy._init(lazy._payload);
|
||||
} catch (thenable) {
|
||||
await thenable;
|
||||
}
|
||||
return lazy._init(lazy._payload);
|
||||
}
|
||||
|
||||
describe("StreamdownRenderer (rendered markdown)", () => {
|
||||
it("never lets a model-authored remote image src reach the DOM", async () => {
|
||||
const Renderer = await resolveStreamdownRenderer();
|
||||
const markdown = [
|
||||
"",
|
||||
"",
|
||||
"",
|
||||
].join("\n\n");
|
||||
const html = renderToStaticMarkup(createElement(Renderer, null, markdown));
|
||||
|
||||
// No remote host is ever fetched: no absolute or protocol-relative image src survives.
|
||||
expect(html).not.toContain('src="http');
|
||||
expect(html).not.toContain('src="//');
|
||||
expect(html).not.toContain("SECRET.evil.tld");
|
||||
// A same-origin relative image is untouched, so the policy does not over-block.
|
||||
expect(html).toContain('src="/local/pic.png"');
|
||||
});
|
||||
});
|
||||
@@ -1,5 +1,32 @@
|
||||
import { lazy } from "react";
|
||||
import type { CodeHighlighterPlugin } from "streamdown";
|
||||
import type { CodeHighlighterPlugin, UrlTransform } from "streamdown";
|
||||
|
||||
const SAFE_LINK_SCHEMES = new Set(["http:", "https:", "mailto:"]);
|
||||
|
||||
/**
|
||||
* URL policy for model-authored markdown. A remote image is fetched the moment it
|
||||
* renders — no click — so it is a zero-click data beacon; we drop the src of any
|
||||
* non-local image. Links stay clickable but only for safe, human-followable schemes.
|
||||
* streamdown removes an attribute whose transform returns undefined, so no request fires.
|
||||
*/
|
||||
export const restrictModelUrls: UrlTransform = (url, key, node) => {
|
||||
const value = url.trim();
|
||||
const isImage = node.tagName === "img" || key === "src" || key === "srcset";
|
||||
|
||||
if (isImage) {
|
||||
// Inline images carry their own bytes; a relative path resolves to our own origin.
|
||||
if (/^data:/i.test(value) || /^blob:/i.test(value)) return url;
|
||||
// Absolute or protocol-relative means a remote host — strip it so nothing is fetched.
|
||||
if (/^[a-z][a-z0-9+.-]*:/i.test(value) || value.startsWith("//")) return undefined;
|
||||
return url;
|
||||
}
|
||||
|
||||
// Links: relative and protocol-relative are fine; otherwise require a safe scheme.
|
||||
if (value.startsWith("//")) return url;
|
||||
const schemeMatch = /^([a-z][a-z0-9+.-]*):/i.exec(value);
|
||||
if (!schemeMatch) return url;
|
||||
return SAFE_LINK_SCHEMES.has(`${schemeMatch[1].toLowerCase()}:`) ? url : undefined;
|
||||
};
|
||||
|
||||
export const StreamdownRenderer = lazy(() =>
|
||||
Promise.all([import("streamdown"), import("@streamdown/code"), import("./shikiTheme")]).then(
|
||||
@@ -23,6 +50,7 @@ export const StreamdownRenderer = lazy(() =>
|
||||
isAnimating={isAnimating}
|
||||
plugins={{ code: codePlugin }}
|
||||
controls={{ code: { copy: false, download: false } }}
|
||||
urlTransform={restrictModelUrls}
|
||||
linkSafety={{ enabled: false }}
|
||||
>
|
||||
{children}
|
||||
|
||||
@@ -68,9 +68,10 @@ describe("tsqlLinter", () => {
|
||||
expect(error).toContain("line");
|
||||
});
|
||||
|
||||
it("should handle missing FROM clause", () => {
|
||||
const error = getTSQLError("SELECT * WHERE id = 1");
|
||||
expect(error).not.toBeNull();
|
||||
it("should accept a query without a FROM clause", () => {
|
||||
// FROM is optional in the grammar (ClickHouse allows e.g. `SELECT 1`),
|
||||
// so a FROM-less SELECT is syntactically valid.
|
||||
expect(getTSQLError("SELECT * WHERE id = 1")).toBeNull();
|
||||
});
|
||||
});
|
||||
});
|
||||
|
||||
@@ -0,0 +1,63 @@
|
||||
import { describe, expect, it } from "vitest";
|
||||
import {
|
||||
checkMessageParts,
|
||||
declaredBodyBytes,
|
||||
exceedsMessageBodyBytes,
|
||||
MAX_MESSAGE_BODY_BYTES,
|
||||
MAX_MESSAGE_CHARS,
|
||||
MAX_MESSAGE_PARTS,
|
||||
} from "./message-limits";
|
||||
|
||||
describe("message limits", () => {
|
||||
it("lets a long real question through", () => {
|
||||
const text = "why did this fail?\n".repeat(50);
|
||||
|
||||
expect(exceedsMessageBodyBytes(Buffer.byteLength(text, "utf8"))).toBe(false);
|
||||
expect(checkMessageParts([{ type: "text", text }])).toBeNull();
|
||||
});
|
||||
|
||||
it("refuses a pasted dump by bytes", () => {
|
||||
expect(exceedsMessageBodyBytes(MAX_MESSAGE_BODY_BYTES)).toBe(false);
|
||||
expect(exceedsMessageBodyBytes(MAX_MESSAGE_BODY_BYTES + 1)).toBe(true);
|
||||
});
|
||||
|
||||
it("counts multi-byte characters as bytes, not characters", () => {
|
||||
// Under the char cap, over the byte cap: 4 bytes each.
|
||||
const emoji = "🙂".repeat(MAX_MESSAGE_BODY_BYTES / 4 + 1);
|
||||
|
||||
expect(emoji.length).toBeLessThan(MAX_MESSAGE_BODY_BYTES);
|
||||
expect(exceedsMessageBodyBytes(Buffer.byteLength(emoji, "utf8"))).toBe(true);
|
||||
});
|
||||
|
||||
it("refuses a dump split across parts", () => {
|
||||
const parts = Array.from({ length: 4 }, () => ({
|
||||
type: "text",
|
||||
text: "x".repeat(MAX_MESSAGE_CHARS / 2),
|
||||
}));
|
||||
|
||||
expect(checkMessageParts(parts)).toBe("too_long");
|
||||
});
|
||||
|
||||
it("refuses too many parts", () => {
|
||||
const parts = Array.from({ length: MAX_MESSAGE_PARTS + 1 }, () => ({
|
||||
type: "text",
|
||||
text: "x",
|
||||
}));
|
||||
|
||||
expect(checkMessageParts(parts)).toBe("too_many_parts");
|
||||
expect(checkMessageParts(parts.slice(0, MAX_MESSAGE_PARTS))).toBeNull();
|
||||
});
|
||||
|
||||
it("leaves a shape that isn't a parts array to the schema", () => {
|
||||
expect(checkMessageParts(undefined)).toBeNull();
|
||||
expect(checkMessageParts("nope")).toBeNull();
|
||||
});
|
||||
|
||||
it("reads the declared size, or nothing when it isn't declared", () => {
|
||||
expect(declaredBodyBytes(new Headers({ "content-length": "1234" }))).toBe(1234);
|
||||
expect(declaredBodyBytes(new Headers())).toBeNull();
|
||||
expect(declaredBodyBytes(new Headers({ "content-length": "nope" }))).toBeNull();
|
||||
// An undeclared size can't be refused here; the body's own length is.
|
||||
expect(exceedsMessageBodyBytes(null)).toBe(false);
|
||||
});
|
||||
});
|
||||
@@ -0,0 +1,51 @@
|
||||
/**
|
||||
* Caps on one message to the agent, shared by the composer and the two server paths a message
|
||||
* can arrive through. Generous for a real question with a pasted stack trace, stingy for a dump:
|
||||
* an unbounded paste is a large model bill and a permanently fat transcript.
|
||||
*/
|
||||
|
||||
/** ~2 pages of text, or a long stack trace. */
|
||||
export const MAX_MESSAGE_CHARS = 8_000;
|
||||
|
||||
/** The counter only shows near the limit, so a normal message never sees it. */
|
||||
export const MESSAGE_CHARS_WARN_AT = Math.floor(MAX_MESSAGE_CHARS * 0.9);
|
||||
|
||||
/** A composed message is a handful of parts; dozens means something is wrong. */
|
||||
export const MAX_MESSAGE_PARTS = 20;
|
||||
|
||||
/**
|
||||
* The whole request body, in bytes: headroom for {@link MAX_MESSAGE_CHARS} of any script plus
|
||||
* the per-turn metadata, and nothing like a pasted file.
|
||||
*/
|
||||
export const MAX_MESSAGE_BODY_BYTES = 64 * 1024;
|
||||
|
||||
export const MESSAGE_TOO_LARGE_CODE = "message_too_large";
|
||||
|
||||
export const MESSAGE_TOO_LARGE_ERROR = "That message is too long. Shorten it and send again.";
|
||||
|
||||
export type MessagePartsProblem = "too_many_parts" | "too_long";
|
||||
|
||||
/** Counts the parts and their text. Anything that isn't a parts array is left to the schema. */
|
||||
export function checkMessageParts(parts: unknown): MessagePartsProblem | null {
|
||||
if (!Array.isArray(parts)) return null;
|
||||
if (parts.length > MAX_MESSAGE_PARTS) return "too_many_parts";
|
||||
|
||||
let chars = 0;
|
||||
for (const part of parts) {
|
||||
const text = (part as { text?: unknown } | null)?.text;
|
||||
if (typeof text === "string") chars += text.length;
|
||||
}
|
||||
return chars > MAX_MESSAGE_CHARS ? "too_long" : null;
|
||||
}
|
||||
|
||||
/** The declared body size, or null when the client didn't declare one. */
|
||||
export function declaredBodyBytes(headers: Headers): number | null {
|
||||
const raw = headers.get("content-length");
|
||||
if (!raw) return null;
|
||||
const bytes = Number.parseInt(raw, 10);
|
||||
return Number.isFinite(bytes) ? bytes : null;
|
||||
}
|
||||
|
||||
export function exceedsMessageBodyBytes(bytes: number | null | undefined): boolean {
|
||||
return typeof bytes === "number" && bytes > MAX_MESSAGE_BODY_BYTES;
|
||||
}
|
||||
@@ -0,0 +1,32 @@
|
||||
import { describe, expect, it } from "vitest";
|
||||
import { MAX_URIS_PER_RESOLVE_REQUEST, planUriBatches } from "./resolve-uris";
|
||||
|
||||
const uri = (index: number) => `trigger://runs/run_${index}`;
|
||||
|
||||
describe("planUriBatches", () => {
|
||||
it("resolves a card's twenty citations in one request", () => {
|
||||
const batches = planUriBatches(Array.from({ length: 20 }, (_, index) => uri(index)));
|
||||
|
||||
expect(batches).toHaveLength(1);
|
||||
expect(batches[0]).toHaveLength(20);
|
||||
});
|
||||
|
||||
it("asks about each URI once", () => {
|
||||
const batches = planUriBatches([uri(1), uri(1), uri(2)]);
|
||||
|
||||
expect(batches).toEqual([[uri(1), uri(2)]]);
|
||||
});
|
||||
|
||||
it("caps a request and carries the rest over", () => {
|
||||
const count = MAX_URIS_PER_RESOLVE_REQUEST + 3;
|
||||
const batches = planUriBatches(Array.from({ length: count }, (_, index) => uri(index)));
|
||||
|
||||
expect(batches).toHaveLength(2);
|
||||
expect(batches[0]).toHaveLength(MAX_URIS_PER_RESOLVE_REQUEST);
|
||||
expect(batches[1]).toHaveLength(3);
|
||||
});
|
||||
|
||||
it("has nothing to send for nothing", () => {
|
||||
expect(planUriBatches([])).toEqual([]);
|
||||
});
|
||||
});
|
||||
@@ -0,0 +1,25 @@
|
||||
/**
|
||||
* Batching for `trigger://` resolution. An investigation card cites ten to twenty targets, and
|
||||
* each request re-authorises and re-resolves the environment — so they go in one request.
|
||||
*/
|
||||
|
||||
/** One environment lookup and one repo lookup serve a whole batch. */
|
||||
export const MAX_URIS_PER_RESOLVE_REQUEST = 25;
|
||||
|
||||
/** A transient failure is worth retrying; a third one isn't. */
|
||||
export const MAX_RESOLVE_ATTEMPTS = 3;
|
||||
|
||||
export const RESOLVE_RETRY_DELAY_MS = 1_000;
|
||||
|
||||
/** Deduplicates, then splits into requests no bigger than the cap. */
|
||||
export function planUriBatches(
|
||||
uris: readonly string[],
|
||||
cap: number = MAX_URIS_PER_RESOLVE_REQUEST
|
||||
): string[][] {
|
||||
const unique = [...new Set(uris)];
|
||||
const batches: string[][] = [];
|
||||
for (let index = 0; index < unique.length; index += cap) {
|
||||
batches.push(unique.slice(index, index + cap));
|
||||
}
|
||||
return batches;
|
||||
}
|
||||
@@ -42,6 +42,8 @@ export type MiniLineChartProps = {
|
||||
* throttled magnitude carried by the tooltip.
|
||||
*/
|
||||
throttled?: number[];
|
||||
/** Tooltip wording for the overlay buckets. Null omits the overlay line. */
|
||||
overlayLabel?: string | null;
|
||||
/** Epoch ms of the first bucket's start. When omitted, the last bucket is anchored to now. */
|
||||
bucketStartMs?: number;
|
||||
/** Width of each bucket in ms. Defaults to one hour. */
|
||||
@@ -76,6 +78,7 @@ export type MiniLineChartProps = {
|
||||
export function MiniLineChart({
|
||||
data,
|
||||
throttled,
|
||||
overlayLabel = "throttled",
|
||||
bucketStartMs,
|
||||
bucketIntervalMs,
|
||||
color = "var(--color-tasks)",
|
||||
@@ -128,7 +131,7 @@ export function MiniLineChart({
|
||||
<YAxis domain={[0, max || 1]} hide />
|
||||
<Tooltip
|
||||
cursor={{ stroke: "rgba(255, 255, 255, 0.2)", strokeWidth: 1 }}
|
||||
content={<MiniLineChartTooltip unitLabel={unitLabel} />}
|
||||
content={<MiniLineChartTooltip unitLabel={unitLabel} overlayLabel={overlayLabel} />}
|
||||
allowEscapeViewBox={{ x: true, y: true }}
|
||||
wrapperStyle={{ zIndex: 1000 }}
|
||||
animationDuration={0}
|
||||
@@ -195,7 +198,8 @@ function MiniLineChartTooltip({
|
||||
active,
|
||||
payload,
|
||||
unitLabel,
|
||||
}: TooltipProps<number, string> & { unitLabel: UnitLabel }) {
|
||||
overlayLabel = "throttled",
|
||||
}: TooltipProps<number, string> & { unitLabel: UnitLabel; overlayLabel?: string | null }) {
|
||||
if (!active || !payload || payload.length === 0) return null;
|
||||
const entry = payload[0].payload as MiniLineChartDatum;
|
||||
const date = entry.date instanceof Date ? entry.date : new Date(entry.date);
|
||||
@@ -211,9 +215,9 @@ function MiniLineChartTooltip({
|
||||
{entry.count === 1 ? unitLabel.singular : unitLabel.plural}
|
||||
</span>
|
||||
</div>
|
||||
{throttled > 0 && (
|
||||
{throttled > 0 && overlayLabel !== null && (
|
||||
<div className="mt-1 text-xs text-warning">
|
||||
<span className="tabular-nums">{throttled.toLocaleString()}</span> throttled
|
||||
<span className="tabular-nums">{throttled.toLocaleString()}</span> {overlayLabel}
|
||||
</div>
|
||||
)}
|
||||
</div>
|
||||
|
||||
@@ -242,8 +242,16 @@ export function SideMenuItem({
|
||||
/** Button styled to match {@link SideMenuItem}, for entries that open a dialog rather than navigate. */
|
||||
export const SideMenuItemButton = forwardRef<
|
||||
HTMLButtonElement,
|
||||
{ icon: RenderIcon; name: string; trailing?: ReactNode } & ButtonHTMLAttributes<HTMLButtonElement>
|
||||
>(function SideMenuItemButton({ icon, name, trailing, className, type, ...props }, ref) {
|
||||
{
|
||||
icon: RenderIcon;
|
||||
name: string;
|
||||
trailing?: ReactNode;
|
||||
iconClassName?: string;
|
||||
} & ButtonHTMLAttributes<HTMLButtonElement>
|
||||
>(function SideMenuItemButton(
|
||||
{ icon, name, trailing, className, iconClassName, type, ...props },
|
||||
ref
|
||||
) {
|
||||
return (
|
||||
<button
|
||||
ref={ref}
|
||||
@@ -256,7 +264,10 @@ export const SideMenuItemButton = forwardRef<
|
||||
>
|
||||
<Icon
|
||||
icon={icon}
|
||||
className="size-5 shrink-0 text-text-dimmed group-hover/menuitem:text-text-bright"
|
||||
className={cn(
|
||||
"size-5 shrink-0",
|
||||
iconClassName ?? "text-text-dimmed group-hover/menuitem:text-text-bright"
|
||||
)}
|
||||
/>
|
||||
<SideMenuLabel className="min-w-0 flex-1 select-none text-left text-[0.90625rem] font-medium tracking-[-0.01em]">
|
||||
{name}
|
||||
|
||||
@@ -1,4 +1,5 @@
|
||||
import { type CSSProperties, useEffect, useRef } from "react";
|
||||
import { useThemeMode } from "~/hooks/useThemeMode";
|
||||
|
||||
// Our own 5x5 dot-matrix system, reverse-engineered from dotmatrix
|
||||
// (github.com/zzzzshawn/matrix) but written from scratch on canvas.
|
||||
@@ -117,6 +118,8 @@ export type DotMatrixPalette = {
|
||||
|
||||
export const DOT_MATRIX_PALETTES = {
|
||||
mono: { stops: ["#e2e8f0", "#ffffff", "#94a3b8"], glow: "#ffffff" },
|
||||
/** `mono` mirrored for light surfaces, where the white ramp would vanish. */
|
||||
monoLight: { stops: ["#2b2c2f", "#1a1b1f", "#585c64"], glow: "#1a1b1f" },
|
||||
trigger: { stops: ["#41ff54", "#a4ff53", "#e7ff52"], glow: "#86ff53" },
|
||||
aurora: { stops: ["#ff3cac", "#784ba0", "#2b86c5"], glow: "#9c64bf" },
|
||||
ocean: { stops: ["#00c6ff", "#0072ff", "#4facfe"], glow: "#2f8fff" },
|
||||
@@ -631,3 +634,17 @@ export function AgentDotMatrix({
|
||||
/>
|
||||
);
|
||||
}
|
||||
|
||||
/** The agent's monochrome logo. Use this rather than `palette="mono"`, which is invisible on light. */
|
||||
export function AgentMonoLogo(props: Omit<AgentDotMatrixProps, "palette" | "restColor" | "mode">) {
|
||||
const mode = useThemeMode();
|
||||
const light = mode === "light";
|
||||
return (
|
||||
<AgentDotMatrix
|
||||
{...props}
|
||||
mode={mode}
|
||||
palette={light ? "monoLight" : "mono"}
|
||||
restColor={light ? "#1a1b1f" : "#d7d9dd"}
|
||||
/>
|
||||
);
|
||||
}
|
||||
|
||||
@@ -9,7 +9,7 @@ import React, {
|
||||
} from "react";
|
||||
import { type ShortcutDefinition, useShortcutKeys } from "~/hooks/useShortcutKeys";
|
||||
import { cn } from "~/utils/cn";
|
||||
import { AgentDotMatrix } from "./AgentDotMatrix";
|
||||
import { AgentMonoLogo } from "./AgentDotMatrix";
|
||||
import { ShortcutKey } from "./ShortcutKey";
|
||||
import { Tooltip, TooltipContent, TooltipProvider, TooltipTrigger } from "./Tooltip";
|
||||
import { Icon, type RenderIcon } from "./Icon";
|
||||
@@ -106,23 +106,22 @@ const theme = {
|
||||
},
|
||||
docs: {
|
||||
textColor:
|
||||
// System themes: monochrome label, the book icon keeps the blue
|
||||
"text-callout-docs-text/70 system:text-text-bright transition group-disabled/button:text-text-dimmed/80",
|
||||
"text-callout-docs-text/70 dark:text-text-bright transition group-disabled/button:text-text-dimmed/80",
|
||||
button:
|
||||
"bg-secondary border border-border-bright/50 shadow-xs group-hover/button:bg-background-raised group-disabled/button:bg-tertiary group-disabled/button:opacity-60 group-disabled/button:pointer-events-none",
|
||||
shortcut:
|
||||
"border-text-dimmed/40 text-text-dimmed group-hover/button:text-text-bright group-hover/button:border-text-dimmed",
|
||||
icon: "text-blue-500",
|
||||
},
|
||||
// Reserved for the AI agent's "Ask AI" affordance: secondary styling with a
|
||||
// softened trigger-green border.
|
||||
"ask-ai": {
|
||||
textColor: "text-text-bright transition group-disabled/button:text-text-dimmed/80",
|
||||
// The AI agent's "Ask Trigger" affordance.
|
||||
"ask-trigger": {
|
||||
textColor:
|
||||
"text-text-bright transition light:group-hover/button:text-charcoal-800 group-disabled/button:text-text-dimmed/80",
|
||||
button:
|
||||
"cursor-pointer bg-secondary border border-[#41FF54]/25 group-hover/button:bg-surface-control group-hover/button:border-[#41FF54]/40 group-disabled/button:bg-secondary group-disabled/button:opacity-60 group-disabled/button:cursor-default group-disabled/button:pointer-events-none",
|
||||
"cursor-pointer bg-secondary border border-[#41FF54]/25 dark:group-hover/button:bg-background-raised dark:group-hover/button:border-[#41FF54]/40 light:group-hover/button:bg-[#e4ffe8] light:group-hover/button:border-[#41FF54]/60 light:border-success/60 group-disabled/button:bg-secondary group-disabled/button:opacity-60 group-disabled/button:cursor-default group-disabled/button:pointer-events-none",
|
||||
shortcut:
|
||||
"border-text-dimmed/40 text-text-dimmed group-hover/button:text-text-bright group-hover/button:border-text-dimmed",
|
||||
icon: "text-text-bright",
|
||||
"border-text-dimmed/40 text-text-dimmed dark:group-hover/button:text-text-bright dark:group-hover/button:border-text-dimmed light:group-hover/button:text-charcoal-800 light:group-hover/button:border-charcoal-800/60",
|
||||
icon: "text-text-bright light:group-hover/button:text-charcoal-800",
|
||||
},
|
||||
};
|
||||
|
||||
@@ -141,19 +140,14 @@ function createVariant(sizeName: Size, themeName: Theme) {
|
||||
};
|
||||
}
|
||||
|
||||
// The ask-ai button always leads with the square agent logo, so it supplies its
|
||||
// own leading icon and its padding is tuned around it: small = 16px logo, 4px
|
||||
// left / 6px right; medium 32/16 -> 8px; large 40/20 -> 10px. Pass an explicit
|
||||
// `LeadingIcon` (e.g. an <AgentDotMatrix active />) to animate it.
|
||||
function createAskAiVariant(sizeName: Size, opticalPadding: string, logoSize: number) {
|
||||
const base = createVariant(sizeName, "ask-ai");
|
||||
// ask-trigger supplies its own leading logo. Pass an explicit `LeadingIcon` to animate it.
|
||||
function createAskTriggerVariant(sizeName: Size, opticalPadding: string, logoSize: number) {
|
||||
const base = createVariant(sizeName, "ask-trigger");
|
||||
return {
|
||||
...base,
|
||||
button: cn(base.button, opticalPadding),
|
||||
iconSpacing: "gap-x-1.5",
|
||||
defaultLeadingIcon: (
|
||||
<AgentDotMatrix size={logoSize} palette="mono" restColor="#ffffff" decorative />
|
||||
),
|
||||
defaultLeadingIcon: <AgentMonoLogo size={logoSize} decorative />,
|
||||
};
|
||||
}
|
||||
|
||||
@@ -187,9 +181,9 @@ const variant = {
|
||||
"docs/medium": createVariant("medium", "docs"),
|
||||
"docs/large": createVariant("large", "docs"),
|
||||
"docs/extra-large": createVariant("extra-large", "docs"),
|
||||
"ask-ai/small": createAskAiVariant("small", "px-1 pr-1.5", 16),
|
||||
"ask-ai/medium": createAskAiVariant("medium", "px-2", 16),
|
||||
"ask-ai/large": createAskAiVariant("large", "px-2.5", 20),
|
||||
"ask-trigger/small": createAskTriggerVariant("small", "px-1 pr-1.5", 16),
|
||||
"ask-trigger/medium": createAskTriggerVariant("medium", "px-2", 16),
|
||||
"ask-trigger/large": createAskTriggerVariant("large", "px-2.5", 20),
|
||||
"menu-item": {
|
||||
textColor: "text-text-bright px-1",
|
||||
button:
|
||||
@@ -275,7 +269,7 @@ export function ButtonContent(props: ButtonContentPropsType) {
|
||||
}, [isLoading]);
|
||||
|
||||
const variation = allVariants.variant[props.variant];
|
||||
// Some variants (ask-ai) always lead with their own glyph unless overridden.
|
||||
// Some variants (ask-trigger) always lead with their own glyph unless overridden.
|
||||
const leadingIcon = LeadingIcon ?? variation.defaultLeadingIcon;
|
||||
|
||||
const btnClassName = cn(allVariants.$all, variation.button);
|
||||
|
||||
@@ -212,10 +212,11 @@ const popoverArrowTriggerVariants = {
|
||||
icon: "text-text-dimmed group-hover:text-text-bright",
|
||||
},
|
||||
primary: {
|
||||
// White ink, not text-bright, which flips dark on the light theme.
|
||||
trigger:
|
||||
"bg-indigo-600 border border-indigo-500 text-text-bright hover:bg-indigo-500 hover:border-indigo-400 disabled:opacity-50 disabled:pointer-events-none",
|
||||
text: "text-text-bright hover:text-white",
|
||||
icon: "text-text-bright",
|
||||
"bg-indigo-600 border border-indigo-500 text-white hover:bg-indigo-500 hover:border-indigo-400 disabled:opacity-50 disabled:pointer-events-none",
|
||||
text: "text-white",
|
||||
icon: "text-white",
|
||||
},
|
||||
secondary: {
|
||||
trigger:
|
||||
|
||||
@@ -1,3 +1,4 @@
|
||||
import { AgentMonoLogo } from "~/components/primitives/AgentDotMatrix";
|
||||
import { cn } from "~/utils/cn";
|
||||
|
||||
type CustomColor = {
|
||||
@@ -76,3 +77,16 @@ export function ButtonSpinner() {
|
||||
export function SpinnerWhite({ className }: { className?: string }) {
|
||||
return <Spinner className={className} color="white" />;
|
||||
}
|
||||
|
||||
/** The dashboard agent's spinner. `size` is the logo's pixel size; the matrix does not scale from CSS. */
|
||||
export function AgentSpinner({ size = 16 }: { size?: number }) {
|
||||
return (
|
||||
<AgentMonoLogo
|
||||
size={size}
|
||||
active
|
||||
// Resting on the playlist's first shape avoids a logo-head flash on mount.
|
||||
restShape="square"
|
||||
decorative
|
||||
/>
|
||||
);
|
||||
}
|
||||
|
||||
@@ -11,6 +11,10 @@ const variations = {
|
||||
"text-indigo-500 transition hover:text-indigo-400 inline-flex gap-0.5 items-center group focus-visible:focus-custom",
|
||||
secondary:
|
||||
"text-text-dimmed transition hover:text-text-bright inline-flex gap-0.5 items-center group focus-visible:focus-custom",
|
||||
// The theme-remapped link token, for links inside themed surfaces where the
|
||||
// raw indigo of `primary` is dark-theme only.
|
||||
token:
|
||||
"text-text-link transition hover:underline inline-flex gap-0.5 items-center group focus-visible:focus-custom",
|
||||
} as const;
|
||||
|
||||
type TextLinkProps = {
|
||||
|
||||
@@ -1,5 +1,7 @@
|
||||
import { EnvelopeIcon, ExclamationCircleIcon, XMarkIcon } from "@heroicons/react/20/solid";
|
||||
import { CheckCircleIcon } from "@heroicons/react/24/solid";
|
||||
import { useThemeMode } from "~/hooks/useThemeMode";
|
||||
import { AgentMonoLogo } from "./AgentDotMatrix";
|
||||
import { useSearchParams } from "@remix-run/react";
|
||||
import { useEffect, useMemo } from "react";
|
||||
import { useTypedLoaderData } from "remix-typedjson";
|
||||
@@ -16,6 +18,7 @@ const permanentToastDuration = 60 * 60 * 24 * 1000;
|
||||
|
||||
export function Toast() {
|
||||
const { toastMessage } = useTypedLoaderData<typeof loader>();
|
||||
const mode = useThemeMode();
|
||||
useEffect(() => {
|
||||
if (!toastMessage) {
|
||||
return;
|
||||
@@ -40,7 +43,9 @@ export function Toast() {
|
||||
);
|
||||
}, [toastMessage]);
|
||||
|
||||
return <Toaster />;
|
||||
// Sonner stamps its own `data-theme` (default "light") on the toast list and the
|
||||
// app's theme selectors follow it, so an unthemed Toaster forces every toast light.
|
||||
return <Toaster theme={mode} />;
|
||||
}
|
||||
|
||||
export function useToast() {
|
||||
@@ -76,20 +81,24 @@ export function ToastUI({
|
||||
toastWidth = 356, // Default width, matches what sonner provides by default
|
||||
title,
|
||||
action,
|
||||
actionNode,
|
||||
}: {
|
||||
variant: "error" | "success";
|
||||
variant: "error" | "success" | "agent";
|
||||
message: string;
|
||||
t: string;
|
||||
toastWidth?: string | number;
|
||||
title?: string;
|
||||
action?: ToastMessageAction;
|
||||
/** Caller-rendered action for client-side toasts. `action` stays the serializable server shape. */
|
||||
actionNode?: React.ReactNode;
|
||||
}) {
|
||||
return (
|
||||
<div
|
||||
className={cn(
|
||||
"self-end rounded-md border border-grid-bright bg-background-dimmed",
|
||||
variant === "success" && "border-success",
|
||||
variant === "error" && "border-error"
|
||||
variant === "error" && "border-error",
|
||||
variant === "agent" && "border-[#41FF54]/25 light:border-success/60 dark:bg-secondary"
|
||||
)}
|
||||
style={{
|
||||
width: toastWidth,
|
||||
@@ -100,6 +109,10 @@ export function ToastUI({
|
||||
>
|
||||
{variant === "success" ? (
|
||||
<CheckCircleIcon className={cn("size-4 min-w-4 text-success", title && "mt-1")} />
|
||||
) : variant === "agent" ? (
|
||||
<span className={cn("flex size-4 min-w-4 items-center", title && "mt-1")}>
|
||||
<AgentMonoLogo size={16} decorative />
|
||||
</span>
|
||||
) : (
|
||||
<ExclamationCircleIcon className={cn("size-4 min-w-4 text-error", title && "mt-1")} />
|
||||
)}
|
||||
@@ -112,6 +125,7 @@ export function ToastUI({
|
||||
{message}
|
||||
</Paragraph>
|
||||
<Action action={action} toastId={t} className="my-2" />
|
||||
{actionNode}
|
||||
</div>
|
||||
<button
|
||||
className={cn(
|
||||
|
||||
@@ -0,0 +1,2 @@
|
||||
/** Head-of-line wait at which a queue reads as stuck. */
|
||||
export const OLDEST_WAIT_WARNING_MS = 5 * 60_000;
|
||||
@@ -163,6 +163,8 @@ export function renderPart(part: UIMessage["parts"][number], i: number) {
|
||||
resultOutput = lastText?.text ?? undefined;
|
||||
} else if (p.output != null) {
|
||||
resultOutput = typeof p.output === "string" ? p.output : JSON.stringify(p.output, null, 2);
|
||||
} else if (p.state === "output-error" && p.errorText) {
|
||||
resultOutput = p.errorText;
|
||||
}
|
||||
|
||||
// Status label for the tool row. AI SDK 7 HITL adds the
|
||||
@@ -178,7 +180,9 @@ export function renderPart(part: UIMessage["parts"][number], i: number) {
|
||||
? "approved"
|
||||
: `denied${p.approval?.reason ? `: ${p.approval.reason}` : ""}`;
|
||||
} else if (p.state === "output-error") {
|
||||
resultSummary = `error: ${p.errorText ?? "unknown"}`;
|
||||
const errorText = p.errorText ?? "unknown";
|
||||
resultSummary =
|
||||
errorText.length > 160 ? `error: ${errorText.slice(0, 160)}…` : `error: ${errorText}`;
|
||||
}
|
||||
|
||||
return (
|
||||
|
||||
@@ -19,6 +19,7 @@ import { assertRunOpsSplitSentinel, Prisma } from "./db.server";
|
||||
import { env } from "./env.server";
|
||||
import { eventLoopMonitor } from "./eventLoopMonitor.server";
|
||||
import { logger } from "./services/logger.server";
|
||||
import { buildImgSrcDirective, parseCspImageOrigins, withImgSrc } from "./utils/cspImageOrigins";
|
||||
import { singleton } from "./utils/singleton";
|
||||
import { remoteBuildsEnabled } from "./v3/remoteImageBuilder.server";
|
||||
import {
|
||||
@@ -47,6 +48,30 @@ import { workerRegionRegistry } from "./v3/workerRegions.server";
|
||||
|
||||
const ABORT_DELAY = 30000;
|
||||
|
||||
/**
|
||||
* Where a document may load images from. The markdown renderer that strips images
|
||||
* ships in the stacked UI PR, so on this branch the policy is the only thing stopping
|
||||
* a model- or customer-authored image from reaching a remote host.
|
||||
*
|
||||
* The hosts we store avatar URLs for, plus whatever `CSP_IMG_SRC_ALLOWLIST` adds
|
||||
* (e.g. a self-hosted SSO avatar host).
|
||||
*/
|
||||
const IMG_SRC_DIRECTIVE = buildImgSrcDirective(
|
||||
singleton("CspImageOrigins", () => {
|
||||
const { origins, rejected } = parseCspImageOrigins(env.CSP_IMG_SRC_ALLOWLIST, {
|
||||
allowHttp: env.NODE_ENV === "development",
|
||||
});
|
||||
|
||||
for (const entry of rejected) {
|
||||
logger.warn(
|
||||
`⚠️ CSP_IMG_SRC_ALLOWLIST entry "${entry.value}" was ignored: it ${entry.reason}.`
|
||||
);
|
||||
}
|
||||
|
||||
return origins;
|
||||
})
|
||||
);
|
||||
|
||||
export default function handleRequest(
|
||||
request: Request,
|
||||
responseStatusCode: number,
|
||||
@@ -66,6 +91,11 @@ export default function handleRequest(
|
||||
responseHeaders.set("Content-Security-Policy", "frame-ancestors 'self'");
|
||||
}
|
||||
|
||||
responseHeaders.set(
|
||||
"Content-Security-Policy",
|
||||
withImgSrc(responseHeaders.get("Content-Security-Policy"), IMG_SRC_DIRECTIVE)
|
||||
);
|
||||
|
||||
const acceptLanguage = request.headers.get("accept-language");
|
||||
const locales = parseAcceptLanguage(acceptLanguage, {
|
||||
validate: Intl.DateTimeFormat.supportedLocalesOf,
|
||||
@@ -302,6 +332,7 @@ singleton("SentryTenantContextProcessor", () => {
|
||||
});
|
||||
|
||||
export { apiRateLimiter } from "./services/apiRateLimit.server";
|
||||
export { dashboardAgentBodyCap } from "./services/dashboardAgentBodyCap.server";
|
||||
export { deploymentRateLimiter } from "./services/deploymentRateLimit.server";
|
||||
export { engineRateLimiter } from "./services/engineRateLimit.server";
|
||||
export { otlpRateLimiter } from "./services/otlpRateLimit.server";
|
||||
|
||||
@@ -269,6 +269,9 @@ const EnvironmentSchema = z
|
||||
LOGIN_ORIGIN: z.string().default("http://localhost:3030"),
|
||||
LOGIN_RATE_LIMITS_ENABLED: BoolEnv.default(true),
|
||||
APP_ORIGIN: z.string().default("http://localhost:3030"),
|
||||
// Extra exact origins (comma separated) added to the document `img-src` CSP,
|
||||
// e.g. an SSO host serving profile images. Wildcards are refused.
|
||||
CSP_IMG_SRC_ALLOWLIST: z.string().optional(),
|
||||
API_ORIGIN: z.string().optional(),
|
||||
// Alternative API origin for deployed runs whose org has the
|
||||
// internalApiOriginEnabled feature flag on. Unset = flag is a no-op.
|
||||
@@ -1726,7 +1729,7 @@ const EnvironmentSchema = z
|
||||
SLACK_BOT_TOKEN: z.string().optional(),
|
||||
SLACK_SIGNUP_REASON_CHANNEL_ID: z.string().optional(),
|
||||
|
||||
// kapa.ai
|
||||
// kapa.ai — deprecated, see `AskAI.tsx`. Nothing reads it while no surface mounts the widget.
|
||||
KAPA_AI_WEBSITE_ID: z.string().optional(),
|
||||
|
||||
// BetterStack
|
||||
|
||||
@@ -0,0 +1,24 @@
|
||||
import { useEffect, useState } from "react";
|
||||
|
||||
export type ThemeMode = "dark" | "light";
|
||||
|
||||
/**
|
||||
* The active theme's mode, for colors that can't come from a CSS variable. Resolved in an
|
||||
* effect so server and hydration renders agree; `root.tsx` can flip `data-theme` pre-paint.
|
||||
*/
|
||||
export function useThemeMode(): ThemeMode {
|
||||
const [mode, setMode] = useState<ThemeMode>("dark");
|
||||
useEffect(() => {
|
||||
const resolve = () => {
|
||||
setMode(document.documentElement.getAttribute("data-theme") === "light" ? "light" : "dark");
|
||||
};
|
||||
resolve();
|
||||
const observer = new MutationObserver(resolve);
|
||||
observer.observe(document.documentElement, {
|
||||
attributes: true,
|
||||
attributeFilter: ["data-theme"],
|
||||
});
|
||||
return () => observer.disconnect();
|
||||
}, []);
|
||||
return mode;
|
||||
}
|
||||
@@ -1,27 +1,53 @@
|
||||
/**
|
||||
* Thin orchestrator: load -> interpret -> return the generic ReportViewModel. No SQL or render
|
||||
* here — data access lives in each report's `load`, meaning in `interpret`, presentation in the
|
||||
* renderers, and the catalog of reports in `report-registry.ts`.
|
||||
*
|
||||
* `call` takes a resolved AuthenticatedEnvironment and is transport-independent (Seam B, §7):
|
||||
* any future surface (MCP Resource, etc.) is just another caller of this same method.
|
||||
*/
|
||||
|
||||
import {
|
||||
createCache,
|
||||
createLRUMemoryStore,
|
||||
DefaultStatefulContext,
|
||||
Namespace,
|
||||
type UnkeyCache,
|
||||
} from "@internal/cache";
|
||||
import { type AuthenticatedEnvironment } from "~/services/apiAuth.server";
|
||||
import { REPORT_REGISTRY } from "./report-registry";
|
||||
import { HEALTH_THRESHOLDS } from "./health/health-core";
|
||||
import { REPORT_REGISTRY, type ReportLoader } from "./report-registry";
|
||||
import { type ReportViewModel } from "./report-view-model";
|
||||
|
||||
const DEFAULT_PERIOD = "1h";
|
||||
|
||||
/**
|
||||
* Single-flight: collapse concurrent identical requests (same report/env/period) into one
|
||||
* computation. A report fires up to ~9 ClickHouse queries and MCP/CLI clients easily call it
|
||||
* several times at once — without this, N callers each launch the full query set and pile onto
|
||||
* the per-project query-concurrency limit. Keyed per (key, env, period); entry drops on settle.
|
||||
* How long a finished report stays reusable. Capped at the liveness fresh window so a
|
||||
* cached report can never render "fresh" while its telemetry is already stale.
|
||||
*/
|
||||
export const REPORT_CACHE_TTL_MS = HEALTH_THRESHOLDS.liveness.freshMs;
|
||||
|
||||
/** How many report, environment and period triples one instance keeps. */
|
||||
const REPORT_CACHE_MAX_ENTRIES = 500;
|
||||
|
||||
export type ReportCache = UnkeyCache<{ report: ReportViewModel }>;
|
||||
|
||||
/** In-process only: an entry is a whole report. `stale` equals `fresh`, so nothing stale is served. */
|
||||
export function createReportCache(ttlMs: number = REPORT_CACHE_TTL_MS): ReportCache {
|
||||
return createCache({
|
||||
report: new Namespace<ReportViewModel>(new DefaultStatefulContext(), {
|
||||
stores: [createLRUMemoryStore(REPORT_CACHE_MAX_ENTRIES)],
|
||||
fresh: ttlMs,
|
||||
stale: ttlMs,
|
||||
}),
|
||||
});
|
||||
}
|
||||
|
||||
const defaultReportCache = createReportCache();
|
||||
|
||||
/**
|
||||
* Collapses concurrent identical requests into one computation: the cache only helps once a load has
|
||||
* finished, so all callers of a cold key would otherwise hit the query-concurrency limit at once.
|
||||
*/
|
||||
const inFlight = new Map<string, Promise<ReportViewModel | undefined>>();
|
||||
|
||||
export class ReportPresenter {
|
||||
constructor(
|
||||
private readonly registry: Record<string, ReportLoader<unknown>> = REPORT_REGISTRY,
|
||||
private readonly cache: ReportCache = defaultReportCache
|
||||
) {}
|
||||
|
||||
async call({
|
||||
environment,
|
||||
key,
|
||||
@@ -31,16 +57,23 @@ export class ReportPresenter {
|
||||
key: string;
|
||||
period?: string;
|
||||
}): Promise<ReportViewModel | undefined> {
|
||||
const loader = REPORT_REGISTRY[key];
|
||||
if (!loader) return undefined;
|
||||
if (!Object.hasOwn(this.registry, key)) return undefined;
|
||||
const loader = this.registry[key];
|
||||
|
||||
// The environment id is part of the key, so one environment can never read another's report.
|
||||
const flightKey = `${key} ${environment.id} ${period}`;
|
||||
|
||||
const cached = await this.cache.report.get(flightKey);
|
||||
if (cached.val) return cached.val;
|
||||
|
||||
const existing = inFlight.get(flightKey);
|
||||
if (existing) return existing;
|
||||
|
||||
const promise = (async () => {
|
||||
const input = await loader.load(environment, period);
|
||||
return loader.interpret(input);
|
||||
const report = loader.interpret(input);
|
||||
await this.cache.report.set(flightKey, report);
|
||||
return report;
|
||||
})().finally(() => inFlight.delete(flightKey));
|
||||
|
||||
inFlight.set(flightKey, promise);
|
||||
|
||||
@@ -1,8 +1,3 @@
|
||||
/**
|
||||
* EXECUTION analyzer: "are the runs that DO start completing OK?" — failures + durations only.
|
||||
* Its "read:" line answers whether the flow problem is a code problem.
|
||||
*/
|
||||
|
||||
import { isOk, maxSeverity, type Finding, type Metric } from "../report-view-model";
|
||||
import { HEALTH_THRESHOLDS, metricById, type HealthInput } from "./health-core";
|
||||
|
||||
@@ -40,10 +35,7 @@ export function interpretExecution(metrics: Metric[], input: HealthInput): Findi
|
||||
};
|
||||
}
|
||||
|
||||
/** Flow causes that provably CAN'T be user code, so a healthy execution reads "not a code problem".
|
||||
* dequeue_stall is platform-side (capacity free but nothing dequeuing). Trigger spike/surge are
|
||||
* excluded: a code path fanning out task.trigger can BE the cause, so we only state the runs that
|
||||
* execute are fine — never the global "not a code problem". */
|
||||
// Trigger spike and surge are excluded: code fanning out task.trigger can be the cause.
|
||||
const NOT_A_CODE_PROBLEM_CAUSES = new Set(["dequeue_stall"]);
|
||||
|
||||
export function buildExecutionRead(execution: Finding, flow: Finding): string {
|
||||
|
||||
@@ -1,10 +1,3 @@
|
||||
/**
|
||||
* FLOW analyzer: "is work flowing, and if not, WHY?" Diagnosed by a CAUSE TREE — `flow.reason`
|
||||
* names why work is backing up (env_limit_saturation, dequeue_stall, …), selected from flow's
|
||||
* evidence, falling back to v1 symptom reasons when no discriminator fires. Evidence quantities
|
||||
* carry no severity of their own — they only select the cause.
|
||||
*/
|
||||
|
||||
import {
|
||||
anomalyWindow,
|
||||
isOk,
|
||||
@@ -17,8 +10,10 @@ import {
|
||||
type Severity,
|
||||
} from "../report-view-model";
|
||||
import {
|
||||
bucketCoverage,
|
||||
HEALTH_THRESHOLDS,
|
||||
isPendingIncreasing,
|
||||
isPendingUnknown,
|
||||
mean,
|
||||
metricById,
|
||||
type HealthInput,
|
||||
@@ -26,7 +21,9 @@ import {
|
||||
|
||||
export const FLOW_METRIC_IDS = ["start_latency_p95", "pending", "throughput"];
|
||||
|
||||
/** One row of the declarative cause table — everything a cause defines about itself. */
|
||||
/** Unmeasurable backlog: verdict is unassessable. Distinct from "unknown", the staleness guard. */
|
||||
export const FLOW_UNMEASURED = "flow_unmeasured";
|
||||
|
||||
type CauseSpec = {
|
||||
reason: string;
|
||||
metricIds: string[]; // real metric rows, causal order
|
||||
@@ -44,46 +41,51 @@ export function interpretFlow(metrics: Metric[], input: HealthInput): Finding {
|
||||
const flowMetrics = FLOW_METRIC_IDS.map((id) => metricById(metrics, id));
|
||||
const severity = maxSeverity(...flowMetrics.map((m) => m.severity));
|
||||
|
||||
// `pending.now` is a placeholder here, so no cause tree may hang off it. What `runs` measured still
|
||||
// stands: only a flow with nothing measurably wrong is unassessable, the rest reports its symptom.
|
||||
if (isPendingUnknown(input)) {
|
||||
return isOk(severity)
|
||||
? { type: "flow", severity, reason: FLOW_UNMEASURED, metricIds: FLOW_METRIC_IDS }
|
||||
: fallbackFlow(flowMetrics, severity);
|
||||
}
|
||||
|
||||
if (isOk(severity)) {
|
||||
return { type: "flow", severity, reason: "healthy", metricIds: FLOW_METRIC_IDS };
|
||||
}
|
||||
|
||||
// Discriminators (evidence only, no own severity).
|
||||
const pendingIncreasing = isPendingIncreasing(input.pending.series);
|
||||
const latencyElevated = !isOk(metricById(metrics, "start_latency_p95").severity);
|
||||
// Concurrency causes need real running-capacity evidence — without it runningShare is a
|
||||
// meaningless 0 and would falsely select dequeue_stall on the snapshot path (#1).
|
||||
const hasConcurrencyEvidence = ev.envLimit > 0 && ev.runningSeries.length > 0;
|
||||
// Without real running-capacity evidence runningShare is a meaningless 0 that selects
|
||||
// dequeue_stall; without enough arrived buckets a few fresh ones read as pinned all window.
|
||||
const coverage = bucketCoverage(input);
|
||||
const hasConcurrencyEvidence =
|
||||
ev.envLimit > 0 && ev.runningSeries.length > 0 && coverage.sufficient;
|
||||
const runningShare = hasConcurrencyEvidence ? mean(ev.runningSeries) / ev.envLimit : 1;
|
||||
// Pinned share is measured against expected buckets, not received rows.
|
||||
const pinnedShare = hasConcurrencyEvidence
|
||||
? ev.runningSeries.filter((r) => r >= t.pinnedLevel * ev.envLimit).length /
|
||||
ev.runningSeries.length
|
||||
coverage.expectedBuckets
|
||||
: 0;
|
||||
const pinned = pinnedShare >= t.pinnedShare;
|
||||
const hasTriggerBaseline = input.throughput.normalTriggeredPerMin > 0;
|
||||
const triggeredMult = hasTriggerBaseline
|
||||
? input.throughput.triggeredPerMin / input.throughput.normalTriggeredPerMin
|
||||
: 0;
|
||||
// No baseline: a multiplier can't be computed, so an absolute rate selects "new volume".
|
||||
// No baseline means no multiplier, so an absolute rate selects "new volume".
|
||||
const triggerSurge = !hasTriggerBaseline && input.throughput.triggeredPerMin >= t.surgePerMin;
|
||||
|
||||
const donePerMin = input.throughput.donePerMin;
|
||||
const net = donePerMin - input.throughput.triggeredPerMin;
|
||||
// Exclusions must be PROVEN, not assumed. "not your code" needs healthy execution; "limits
|
||||
// aren't the bottleneck" needs no env-pin AND no queue throttling; the workers/spike ones
|
||||
// state a measured fact (rate) rather than a global "everything's fine" claim.
|
||||
// Work leaves the queue on any terminal status, not just completions.
|
||||
const finishedPerMin = input.throughput.finishedPerMin;
|
||||
const net = finishedPerMin - input.throughput.triggeredPerMin;
|
||||
// "not your config" requires both no env pin and no queue throttling.
|
||||
const executionHealthy =
|
||||
isOk(metricById(metrics, "failures").severity) && isOk(metricById(metrics, "dur_p95").severity);
|
||||
const queueThrottled = ev.throttledShare >= t.throttledShare;
|
||||
const configHealthy = !pinned && !queueThrottled;
|
||||
// A trigger spike/surge is only the CAUSE of a backup when work is actually piling up:
|
||||
// completions falling behind (net < 0) AND the backlog trending up. Without this a spike that
|
||||
// drains fine would still be blamed while its read says "queue fills faster than it drains".
|
||||
// A spike is only a cause when work piles up: finishes behind triggers and backlog trending up.
|
||||
const triggerBacklog = net < 0 && pendingIncreasing;
|
||||
|
||||
// First discriminator that fires wins (fixed priority). dequeue_stall is a last-resort
|
||||
// "it's on our side" cause — so a known config bottleneck (queue throttling) must rule it
|
||||
// out first, else throttled-but-idle-capacity is misread as a platform stall (#1).
|
||||
// First discriminator wins. dequeue_stall is last resort: a known config bottleneck rules it out.
|
||||
let spec: CauseSpec;
|
||||
if (
|
||||
hasConcurrencyEvidence &&
|
||||
@@ -112,10 +114,8 @@ export function interpretFlow(metrics: Metric[], input: HealthInput): Finding {
|
||||
drivingMetricId: "concurrency",
|
||||
annotationCode: "pinned_minutes",
|
||||
exclusions: [],
|
||||
// States a measured fact (runs ARE completing at {rate}/min) — evidence the workers aren't
|
||||
// dead. An observation, not an exclusion: it doesn't claim it's the limit, nor "keeps pace".
|
||||
observations:
|
||||
donePerMin > 0 ? [{ code: "not_workers_platform", evidence: { donePerMin } }] : [],
|
||||
finishedPerMin > 0 ? [{ code: "not_workers_platform", evidence: { finishedPerMin } }] : [],
|
||||
recommendation: { code: "raise_env_limit", link: "concurrency" },
|
||||
usesAttribution: true,
|
||||
};
|
||||
@@ -125,7 +125,6 @@ export function interpretFlow(metrics: Metric[], input: HealthInput): Finding {
|
||||
metricIds: ["throttled", "pending"],
|
||||
drivingMetricId: "throttled",
|
||||
annotationCode: "throttled_minutes",
|
||||
// Justified: we're in the not-pinned branch, so the env limit isn't the bottleneck.
|
||||
exclusions: [{ code: "not_env_limit" }],
|
||||
observations: [],
|
||||
recommendation: { code: "raise_queue_limit", link: "queue" },
|
||||
@@ -138,9 +137,6 @@ export function interpretFlow(metrics: Metric[], input: HealthInput): Finding {
|
||||
drivingMetricId: "triggered",
|
||||
annotationCode: "spike_mult",
|
||||
exclusions: [],
|
||||
// Only the proven fact — the runs that DO start execute fine. An observation, NOT the
|
||||
// exclusion "not your code": healthy execution doesn't prove the code isn't the one
|
||||
// triggering the flood (e.g. a deploy that fans out task.trigger in a loop).
|
||||
observations: executionHealthy ? [{ code: "execution_healthy" }] : [],
|
||||
recommendation: { code: "review_trigger_source", link: "runs" },
|
||||
usesAttribution: false,
|
||||
@@ -152,20 +148,17 @@ export function interpretFlow(metrics: Metric[], input: HealthInput): Finding {
|
||||
drivingMetricId: "triggered",
|
||||
annotationCode: "surge_rate",
|
||||
exclusions: [],
|
||||
// Same as a spike: only the proven fact, without ruling out the trigger-producing code.
|
||||
observations: executionHealthy ? [{ code: "execution_healthy" }] : [],
|
||||
recommendation: { code: "review_trigger_source", link: "runs" },
|
||||
usesAttribution: false,
|
||||
};
|
||||
} else {
|
||||
// Fallback — v1 symptom reasons by dominant metric.
|
||||
return fallbackFlow(flowMetrics, severity);
|
||||
}
|
||||
|
||||
return assembleFlowCause(spec, metrics, input, severity);
|
||||
}
|
||||
|
||||
/** Build a flow Finding from a cause spec: annotation, anomaly window, attribution, evidence. */
|
||||
function assembleFlowCause(
|
||||
spec: CauseSpec,
|
||||
metrics: Metric[],
|
||||
@@ -175,21 +168,22 @@ function assembleFlowCause(
|
||||
const t = HEALTH_THRESHOLDS;
|
||||
const driving = metricById(metrics, spec.drivingMetricId);
|
||||
|
||||
// Anomaly window from the driving series. env_limit_saturation breaches ABOVE
|
||||
// (concurrency pinned at the limit); dequeue_stall breaches BELOW (capacity idle).
|
||||
// NOTE: runningSeries is at native env_metrics resolution (not resampled), so the "(last N
|
||||
// min)" figure assumes those buckets are uniform and cover the resolved window. env_metrics
|
||||
// are emitted on a fixed cadence, so that holds; a gappy/partial window could skew the minutes.
|
||||
// env_limit_saturation breaches above the threshold, dequeue_stall below it. runningSeries is not
|
||||
// gap-filled, so the duration counts per real bucket cadence and gaps break the contiguous run.
|
||||
let aw: Finding["anomalyWindow"];
|
||||
if (spec.reason === "env_limit_saturation" || spec.reason === "dequeue_stall") {
|
||||
const below = spec.reason === "dequeue_stall";
|
||||
const threshold = below
|
||||
? t.flowCause.stallRunningShare * input.flowEvidence.envLimit
|
||||
: t.flowCause.pinnedLevel * input.flowEvidence.envLimit;
|
||||
aw = anomalyWindow(input.flowEvidence.runningSeries, threshold, input.windowMinutes, { below });
|
||||
const coverage = bucketCoverage(input);
|
||||
aw = anomalyWindow(input.flowEvidence.runningSeries, threshold, input.windowMinutes, {
|
||||
below,
|
||||
bucketMinutes: coverage.known ? coverage.bucketMinutes : undefined,
|
||||
timestampsMs: input.flowEvidence.runningBucketsMs,
|
||||
});
|
||||
}
|
||||
|
||||
// Annotation on the driving metric (a fact, not an invented number).
|
||||
if (spec.annotationCode) {
|
||||
const value =
|
||||
spec.annotationCode === "pinned_minutes"
|
||||
@@ -211,16 +205,13 @@ function assembleFlowCause(
|
||||
driving.annotation = { code: spec.annotationCode, value };
|
||||
}
|
||||
|
||||
// Attribution — only when a queue owns >= minShare of the problem.
|
||||
let attribution: Finding["attribution"];
|
||||
const wq = input.flowEvidence.worstQueue;
|
||||
if (spec.usesAttribution && wq && wq.share >= t.attribution.minShare) {
|
||||
attribution = { dim: "queue", key: wq.name, share: wq.share, of: "pending" };
|
||||
}
|
||||
|
||||
// Append "nothing dead-lettered" ONLY on a measured zero (dlqDelta === 0); null means
|
||||
// unmeasured (snapshot path) — no observation without evidence. It's a supporting fact, not a
|
||||
// ruled-out cause, so it joins observations.
|
||||
// Only a measured zero supports "nothing dead-lettered". Null means unmeasured.
|
||||
const observations =
|
||||
input.flowEvidence.dlqDelta === 0
|
||||
? [...spec.observations, { code: "nothing_dead_lettered", evidence: { dlq: 0 } }]
|
||||
@@ -239,7 +230,6 @@ function assembleFlowCause(
|
||||
};
|
||||
}
|
||||
|
||||
/** v1 symptom fallback when no cause discriminator fires. */
|
||||
function fallbackFlow(flowMetrics: Metric[], severity: Severity): Finding {
|
||||
const firstOff = flowMetrics.find((m) => !isOk(m.severity));
|
||||
const reason =
|
||||
@@ -267,10 +257,6 @@ function fallbackFlow(flowMetrics: Metric[], severity: Severity): Finding {
|
||||
};
|
||||
}
|
||||
|
||||
// ---------------------------------------------------------------------------
|
||||
// Flow severity policy + the causal "read:" line.
|
||||
// ---------------------------------------------------------------------------
|
||||
|
||||
export function applyFlowPolicy(
|
||||
flow: Finding,
|
||||
execution: Finding,
|
||||
@@ -279,7 +265,6 @@ export function applyFlowPolicy(
|
||||
): Finding {
|
||||
if (flow.severity !== "crit") return flow;
|
||||
// Downgrade a drainable crit to warn only when execution is fine and telemetry isn't stale.
|
||||
// Unknown/lagging freshness must NOT block this (it's not a signal that anything's wrong).
|
||||
const severity: Severity =
|
||||
isOk(execution.severity) && !telemetryStale && isDrainable ? "warn" : "crit";
|
||||
return { ...flow, severity };
|
||||
@@ -294,10 +279,10 @@ const CAUSE_READS: Record<string, string> = {
|
||||
};
|
||||
|
||||
export function buildFlowRead(flow: Finding, executionOk: boolean, livenessFresh: boolean): string {
|
||||
if (flow.reason === "unknown") return "data_stale"; // stale-guarded — no causal read
|
||||
if (flow.reason === "unknown") return "data_stale";
|
||||
if (flow.reason === FLOW_UNMEASURED) return "flow_unmeasured";
|
||||
if (isOk(flow.severity)) return "starting_normally";
|
||||
if (CAUSE_READS[flow.reason]) return CAUSE_READS[flow.reason];
|
||||
// fallback symptoms (v1 logic)
|
||||
if (executionOk && livenessFresh) return "lag_while_triggering_normal";
|
||||
if (!executionOk) return "lag_and_failures";
|
||||
return "degraded_generic";
|
||||
|
||||
@@ -1,9 +1,3 @@
|
||||
/**
|
||||
* Health report FOUNDATION shared by the three analyzers (flow / execution / liveness):
|
||||
* the input shape, tunable thresholds, small helpers, and `buildMetrics` (numbers +
|
||||
* per-metric severity). No verdict logic here — that lives in the per-analyzer modules.
|
||||
*/
|
||||
|
||||
import { classifySeverity, delta, type Metric, type Severity } from "../report-view-model";
|
||||
|
||||
export type HealthInput = {
|
||||
@@ -11,16 +5,20 @@ export type HealthInput = {
|
||||
period: string;
|
||||
baselineLabel: string;
|
||||
generatedAt: string;
|
||||
/** live window length in minutes — for anomaly-window / annotation math. */
|
||||
windowMinutes: number;
|
||||
/** provenance; drives caveat text, not logic. */
|
||||
/** Provenance. Drives caveat text, not logic. */
|
||||
flowSource: "snapshot+runs" | "queue_metrics_v1";
|
||||
/**
|
||||
* now = live env-level depth; normal = 7d baseline (omitted on the snapshot path, which has
|
||||
* no real 7d pending baseline — so we never mislabel a live-window average as "7d normal");
|
||||
* series measured (v2) or estimated (v1).
|
||||
* `normal` is the 7d baseline, omitted on the snapshot path. `availability: "unknown"` means the
|
||||
* depth was not measured and `now` is a placeholder, not a confident 0.
|
||||
*/
|
||||
pending: { now: number; normal?: number; series: number[]; estimated: boolean };
|
||||
pending: {
|
||||
now: number;
|
||||
normal?: number;
|
||||
series: number[];
|
||||
estimated: boolean;
|
||||
availability?: "measured" | "unknown";
|
||||
};
|
||||
/**
|
||||
* p95 wait. `availability: "unknown"` = the source had no measurement, so `p95Ms` is a
|
||||
* placeholder that must not be graded — a 0 would read as a confident green.
|
||||
@@ -31,31 +29,39 @@ export type HealthInput = {
|
||||
series: number[];
|
||||
availability?: "measured" | "unknown";
|
||||
};
|
||||
throughput: { donePerMin: number; triggeredPerMin: number; normalTriggeredPerMin: number };
|
||||
/** `finishedPerMin` is all terminal runs and the drain rate. `completedPerMin` is successes only. */
|
||||
throughput: {
|
||||
finishedPerMin: number;
|
||||
completedPerMin: number;
|
||||
triggeredPerMin: number;
|
||||
normalTriggeredPerMin: number;
|
||||
};
|
||||
failures: { rate: number; normalRate: number; series: number[] };
|
||||
duration: { p95Ms: number; normalP95Ms: number };
|
||||
/** Age of the freshest telemetry (ms). null = no signal to assess -> freshness unknown. */
|
||||
/** Age of the freshest telemetry in ms. Null means no signal to assess. */
|
||||
liveness: { telemetryAgeMs: number | null };
|
||||
/**
|
||||
* Flow's cause-tree discriminators (no own severity) — from env_metrics rows + one
|
||||
* queue_metrics GROUP BY. Empty-ish when unavailable (cause tree falls back to v1).
|
||||
*/
|
||||
/** Flow's cause-tree discriminators. No severity of their own. */
|
||||
flowEvidence: {
|
||||
runningSeries: number[];
|
||||
/** Epoch ms per `runningSeries` bucket. Absent means contiguity falls back to index adjacency. */
|
||||
runningBucketsMs?: number[];
|
||||
/**
|
||||
* Cadence and expected bucket count of `runningSeries`, which is not gap-filled. Absent means
|
||||
* the cadence is unknown and received buckets are assumed to spread evenly over the window.
|
||||
*/
|
||||
sampling?: { bucketMinutes: number; expectedBuckets: number } | null;
|
||||
envLimit: number;
|
||||
throttledShare: number;
|
||||
worstQueue: { name: string; share: number } | null;
|
||||
/** runs dead-lettered in the window: 0 = measured none, null = unmeasured (snapshot). */
|
||||
/** Runs dead-lettered in the window. 0 is a measured none, null is unmeasured. */
|
||||
dlqDelta: number | null;
|
||||
};
|
||||
/** Lazy — loaded only when execution degrades (attribution line). */
|
||||
/** Loaded only when execution degrades. */
|
||||
failureBreakdown?: { task: string; share: number; region?: string };
|
||||
};
|
||||
|
||||
/** Tunable defaults — first-guess; tune against prod once wired. */
|
||||
export const HEALTH_THRESHOLDS = {
|
||||
// `floor` = absolute warn/crit used when there's no usable baseline (normal 0/undefined),
|
||||
// so a spike from a zero baseline (e.g. a never-failing env) isn't classified healthy.
|
||||
// `floor` is the absolute warn/crit used when there's no usable baseline.
|
||||
startLatency: { warnMult: 3, critMult: 10, floor: { warn: 30_000, crit: 120_000 } },
|
||||
pending: { warnMult: 2, critMult: 10, floor: { warn: 500, crit: 5_000 } },
|
||||
failures: {
|
||||
@@ -73,21 +79,18 @@ export const HEALTH_THRESHOLDS = {
|
||||
pinnedShare: 0.5, // env_limit_saturation: >= half the window's buckets pinned
|
||||
throttledShare: 0.25, // queue_limit_throttling: >= a quarter of the window throttled
|
||||
spikeMult: 3, // trigger_spike: triggered/min >= 3x the 7d-normal rate
|
||||
// trigger_surge: with NO usable baseline (normal 0), a multiplier is meaningless, so an
|
||||
// absolute floor picks the "new volume" cause instead of dropping to the v1 fallback.
|
||||
surgePerMin: 100,
|
||||
surgePerMin: 100, // trigger_surge: absolute triggered/min floor used when there's no baseline
|
||||
// Minimum share of the window's expected buckets that must have arrived before a
|
||||
// concurrency-shaped cause may be named. Below it flow drops to a symptom-level verdict.
|
||||
minCoverage: 0.5,
|
||||
},
|
||||
attribution: { minShare: 0.5 }, // name a queue/task/region only when it owns >= half the problem
|
||||
};
|
||||
|
||||
// ---------------------------------------------------------------------------
|
||||
// Helpers.
|
||||
// ---------------------------------------------------------------------------
|
||||
|
||||
export const mean = (xs: number[]) =>
|
||||
xs.length === 0 ? 0 : xs.reduce((a, b) => a + b, 0) / xs.length;
|
||||
|
||||
/** Deterministic "trending up": mean of the last third vs the first third (direction only). */
|
||||
/** Trending up: mean of the last third against the first third. */
|
||||
export function isPendingIncreasing(series: number[]): boolean {
|
||||
if (series.length < 2) return false;
|
||||
const third = Math.max(1, Math.floor(series.length / 3));
|
||||
@@ -108,19 +111,56 @@ function multiplierSeverity(
|
||||
return classifySeverity(value / normal, { warn: warnMult, crit: critMult });
|
||||
}
|
||||
|
||||
/** Look up a metric by id; throws if absent (buildMetrics guarantees the standard set exists). */
|
||||
/** True when the depth was not measured, so `pending.now` is a placeholder. */
|
||||
export function isPendingUnknown(input: HealthInput): boolean {
|
||||
return input.pending.availability === "unknown";
|
||||
}
|
||||
|
||||
export type BucketCoverage = {
|
||||
/** Buckets the window should contain at the source's cadence. */
|
||||
expectedBuckets: number;
|
||||
receivedBuckets: number;
|
||||
bucketMinutes: number;
|
||||
/** Received over expected. 1 when the cadence is unknown. */
|
||||
coverage: number;
|
||||
/** Enough of the window arrived to support a cause and a duration. */
|
||||
sufficient: boolean;
|
||||
/** True when the source reported its cadence, so gaps are detectable. */
|
||||
known: boolean;
|
||||
};
|
||||
|
||||
/** Without the source's cadence the received buckets are assumed to span the window evenly. */
|
||||
export function bucketCoverage(input: HealthInput): BucketCoverage {
|
||||
const received = input.flowEvidence.runningSeries.length;
|
||||
const sampling = input.flowEvidence.sampling;
|
||||
if (!sampling || sampling.expectedBuckets <= 0) {
|
||||
return {
|
||||
expectedBuckets: received,
|
||||
receivedBuckets: received,
|
||||
bucketMinutes: received > 0 ? input.windowMinutes / received : 0,
|
||||
coverage: 1,
|
||||
sufficient: true,
|
||||
known: false,
|
||||
};
|
||||
}
|
||||
const coverage = received / sampling.expectedBuckets;
|
||||
return {
|
||||
expectedBuckets: sampling.expectedBuckets,
|
||||
receivedBuckets: received,
|
||||
bucketMinutes: sampling.bucketMinutes,
|
||||
coverage,
|
||||
sufficient: coverage >= HEALTH_THRESHOLDS.flowCause.minCoverage,
|
||||
known: true,
|
||||
};
|
||||
}
|
||||
|
||||
export function metricById(metrics: Metric[], id: string): Metric {
|
||||
const m = metrics.find((x) => x.id === id);
|
||||
if (!m) throw new Error(`health: missing metric ${id}`);
|
||||
return m;
|
||||
}
|
||||
|
||||
// ---------------------------------------------------------------------------
|
||||
// buildMetrics: numbers + per-metric severity. The standard six carry severity; the
|
||||
// flow-evidence metrics (concurrency, throttled, triggered) are evidence only (severity ok)
|
||||
// and exist so a cause's metricIds resolve.
|
||||
// ---------------------------------------------------------------------------
|
||||
|
||||
// Only the standard six metrics carry severity; the flow-evidence metrics exist so metricIds resolve.
|
||||
export function buildMetrics(input: HealthInput): Metric[] {
|
||||
const t = HEALTH_THRESHOLDS;
|
||||
const ev = input.flowEvidence;
|
||||
@@ -151,17 +191,22 @@ export function buildMetrics(input: HealthInput): Metric[] {
|
||||
),
|
||||
};
|
||||
|
||||
// An unmeasurable depth must not be classified: a placeholder 0 would read as a confident green.
|
||||
const pendingUnknown = isPendingUnknown(input);
|
||||
const pending: Metric = {
|
||||
id: "pending",
|
||||
value: input.pending.now,
|
||||
unit: "count",
|
||||
availability: pendingUnknown ? "unknown" : "measured",
|
||||
normal: input.pending.normal,
|
||||
delta: delta(input.pending.now, input.pending.normal),
|
||||
delta: pendingUnknown ? undefined : delta(input.pending.now, input.pending.normal),
|
||||
series: {
|
||||
points: input.pending.series,
|
||||
kind: input.pending.estimated ? "estimated" : "measured",
|
||||
},
|
||||
severity: multiplierSeverity(
|
||||
severity: pendingUnknown
|
||||
? "ok"
|
||||
: multiplierSeverity(
|
||||
input.pending.now,
|
||||
input.pending.normal,
|
||||
t.pending.warnMult,
|
||||
@@ -170,13 +215,17 @@ export function buildMetrics(input: HealthInput): Metric[] {
|
||||
),
|
||||
};
|
||||
|
||||
const net = input.throughput.donePerMin - input.throughput.triggeredPerMin;
|
||||
// Net drain uses all terminal runs, not just completions.
|
||||
const net = input.throughput.finishedPerMin - input.throughput.triggeredPerMin;
|
||||
const throughput: Metric = {
|
||||
id: "throughput",
|
||||
value: net,
|
||||
unit: "perMin",
|
||||
aggregation: "rate",
|
||||
breakdown: { done: input.throughput.donePerMin, triggered: input.throughput.triggeredPerMin },
|
||||
breakdown: {
|
||||
done: input.throughput.finishedPerMin,
|
||||
triggered: input.throughput.triggeredPerMin,
|
||||
},
|
||||
severity: net < 0 && isPendingIncreasing(input.pending.series) ? "warn" : "ok",
|
||||
};
|
||||
|
||||
@@ -218,27 +267,24 @@ export function buildMetrics(input: HealthInput): Metric[] {
|
||||
};
|
||||
|
||||
const ageMs = input.liveness.telemetryAgeMs;
|
||||
// No signal stays neutral. Lagging telemetry is a real warn; unknown is not.
|
||||
const livenessSeverity: Severity =
|
||||
ageMs === null // no signal at all (brand-new/quiet env) — genuinely unknown, so NEUTRAL (ok):
|
||||
? "ok" // a fine-but-idle env must not surface as a yellow verdict, and it never
|
||||
: // trust-guards. "lagging" (below) IS a real warn; "unknown" is not.
|
||||
ageMs > t.liveness.staleMs
|
||||
ageMs === null
|
||||
? "ok"
|
||||
: ageMs > t.liveness.staleMs
|
||||
? "crit"
|
||||
: ageMs > t.liveness.freshMs
|
||||
? "warn"
|
||||
: "ok";
|
||||
const liveness: Metric = {
|
||||
id: "liveness",
|
||||
// No signal -> value 0 is a placeholder, NOT a real "0ms fresh". `availability: "unknown"`
|
||||
// says so, so a structured consumer never reads the 0 as freshness (the finding reason
|
||||
// also carries "freshness_unknown"). A finite number keeps the JSON VM valid (no Infinity).
|
||||
// With no signal, 0 is a placeholder rather than a real "0ms fresh"; `availability` says so.
|
||||
value: ageMs ?? 0,
|
||||
availability: ageMs === null ? "unknown" : "measured",
|
||||
unit: "ms",
|
||||
severity: livenessSeverity,
|
||||
};
|
||||
|
||||
// Flow-evidence metrics (severity ok — evidence, not a verdict).
|
||||
const concurrency: Metric = {
|
||||
id: "concurrency",
|
||||
value: ev.runningSeries.length > 0 ? ev.runningSeries[ev.runningSeries.length - 1] : 0,
|
||||
@@ -275,15 +321,15 @@ export function buildMetrics(input: HealthInput): Metric[] {
|
||||
];
|
||||
}
|
||||
|
||||
// ---------------------------------------------------------------------------
|
||||
// Drain computation — shared by the flow policy (flow.ts) and the footer (health.ts).
|
||||
// ---------------------------------------------------------------------------
|
||||
|
||||
export function computeDrain(input: HealthInput): { drainMinutes: number; isDrainable: boolean } {
|
||||
const donePerMin = input.throughput.donePerMin;
|
||||
const drainMinutes = donePerMin === 0 ? Number.POSITIVE_INFINITY : input.pending.now / donePerMin;
|
||||
// The drain rate counts runs leaving the queue, not just completions.
|
||||
const finishedPerMin = input.throughput.finishedPerMin;
|
||||
const drainMinutes =
|
||||
finishedPerMin === 0 ? Number.POSITIVE_INFINITY : input.pending.now / finishedPerMin;
|
||||
return {
|
||||
drainMinutes,
|
||||
isDrainable: drainMinutes < HEALTH_THRESHOLDS.flowPolicy.drainCritMinutes,
|
||||
// An unmeasurable depth can't produce an ETA.
|
||||
isDrainable:
|
||||
!isPendingUnknown(input) && drainMinutes < HEALTH_THRESHOLDS.flowPolicy.drainCritMinutes,
|
||||
};
|
||||
}
|
||||
|
||||
@@ -1,28 +1,15 @@
|
||||
/**
|
||||
* The `health` report's DATA layer — the ONLY place SQL / Redis IO lives. Loads a
|
||||
* `HealthInput` (plain numbers) that the pure `interpret()` turns into a VM.
|
||||
*
|
||||
* Flow signal is sourced behind `FlowSource`:
|
||||
* - QueueMetricsSource (preferred) — MEASURED queue depth + real scheduling delay
|
||||
* (p95 wait) from `env_metrics`, which is env-level, so there is no per-queue split.
|
||||
* - SnapshotFlowSource (fallback) — live Redis depth + an ESTIMATED backlog proxy
|
||||
* and `runs.queued_duration`, used until the queue-metrics pipeline has populated
|
||||
* `env_metrics` for this env.
|
||||
*
|
||||
* `flowSource` records which ran and drives `pending.estimated`, so the "informational
|
||||
* only" caveat drops automatically on the measured path. Execution, liveness and
|
||||
* throughput always come from `runs`.
|
||||
*/
|
||||
|
||||
import { calculateTimeBucketInterval, type TimeBucketInterval } from "@internal/tsql";
|
||||
import { type AuthenticatedEnvironment } from "~/services/apiAuth.server";
|
||||
import { logger } from "~/services/logger.server";
|
||||
import { executeQuery, isQueryConcurrencyRejection } from "~/services/queryService.server";
|
||||
import { envMetricsSchema } from "~/v3/querySchemas";
|
||||
import { engine } from "~/v3/runEngine.server";
|
||||
import { HEALTH_THRESHOLDS, type HealthInput } from "./health";
|
||||
|
||||
/** §5.B — failures = user-code failures only (Expired/Canceled excluded from both sides). */
|
||||
/** User-code failures only. Expired and Canceled are excluded from both sides of the rate. */
|
||||
const FAILURE_STATUSES = "'Failed','Crashed','System failure','Timed out'";
|
||||
|
||||
/** All terminal statuses — a run that reached any of these has left the queue (not backlog). */
|
||||
/** All terminal statuses: a run that reached any of these has left the queue. */
|
||||
const FINISHED_STATUSES =
|
||||
"'Completed','Canceled','Expired','Failed','Crashed','System failure','Timed out'";
|
||||
|
||||
@@ -49,7 +36,6 @@ function mean(xs: number[]): number {
|
||||
return xs.length === 0 ? 0 : xs.reduce((a, b) => a + b, 0) / xs.length;
|
||||
}
|
||||
|
||||
/** Downsample a per-bucket series to ~N points so the sparkline width is stable (plan §5.C). */
|
||||
function resampleSeries(points: number[], target = SPARKLINE_BUCKETS): number[] {
|
||||
if (points.length <= target) return points;
|
||||
const out: number[] = [];
|
||||
@@ -68,13 +54,7 @@ function failureRate(failures: number, completed: number): number {
|
||||
return denom === 0 ? 0 : failures / denom;
|
||||
}
|
||||
|
||||
/**
|
||||
* Bug 1 fix — ClickHouse query concurrency cap. The query service rejects the 4th
|
||||
* concurrent query per project (-> `runQuery` throws -> 500); this loader used to fire
|
||||
* up to 4 at once. `mapWithConcurrency` runs `fn` over `items` with at most `limit` in
|
||||
* flight, preserving order. We cap CH calls at 2 to leave headroom under the limit of 3
|
||||
* for other project traffic. (Redis calls don't count — kept outside this helper.)
|
||||
*/
|
||||
/** Order-preserving. The query service rejects the 4th concurrent query per project. */
|
||||
async function mapWithConcurrency<T, R>(
|
||||
items: readonly T[],
|
||||
limit: number,
|
||||
@@ -94,15 +74,10 @@ async function mapWithConcurrency<T, R>(
|
||||
return results;
|
||||
}
|
||||
|
||||
/** Max ClickHouse queries this loader keeps in flight (limit of 3, leave headroom). */
|
||||
/** Max ClickHouse queries in flight. The per-project limit is 3, so this leaves headroom. */
|
||||
const CH_CONCURRENCY = 2;
|
||||
|
||||
/**
|
||||
* The per-project limit (3) is shared across ALL in-flight requests, not just this
|
||||
* loader — so concurrent report requests can still transiently exceed it and get a
|
||||
* rejection. It's retryable (a slot frees when another query finishes), so we back off
|
||||
* and retry rather than surface a 500. Scheduling only — same query, same result.
|
||||
*/
|
||||
// The per-project limit is shared, so concurrent requests can still be rejected. That is retryable.
|
||||
const CH_REJECTION_RETRIES = 6;
|
||||
const CH_REJECTION_BACKOFF_MS = 60; // base for exponential "full jitter" backoff
|
||||
const CH_REJECTION_BACKOFF_CAP_MS = 2000; // ceiling per attempt
|
||||
@@ -119,39 +94,26 @@ function isConcurrencyRejection(error: unknown): boolean {
|
||||
return /concurrency|too many|try again/i.test(message);
|
||||
}
|
||||
|
||||
/** rows + the actual (clip-aware) time window the query service resolved for this run. */
|
||||
/** Rows plus the clip-aware time window the query service resolved for this run. */
|
||||
type QueryResult = { rows: Row[]; timeRange: { from: Date; to: Date } };
|
||||
|
||||
/** Runs one (TRQL) report query. Injectable so tests can drive the loader with canned results. */
|
||||
export type HealthQueryRunner = (
|
||||
env: AuthenticatedEnvironment,
|
||||
query: string,
|
||||
period: string
|
||||
) => Promise<QueryResult>;
|
||||
|
||||
/**
|
||||
* The loader's IO boundary (§7 Seam A): ClickHouse via the query service, Redis via the
|
||||
* engine. Defaults wire the real singletons; overriding lets tests drive the loader's
|
||||
* orchestration without booting the env-bound query-service client.
|
||||
*/
|
||||
/** The loader's IO boundary. */
|
||||
export type HealthDeps = {
|
||||
runQuery: HealthQueryRunner;
|
||||
lengthOfEnvQueue: (env: AuthenticatedEnvironment) => Promise<number | undefined>;
|
||||
};
|
||||
|
||||
/**
|
||||
* The 7d baseline changes slowly, so cache it briefly (per env + query) to avoid recomputing a
|
||||
* wide query on every request — the biggest lever on query pressure (#12). Only the default
|
||||
* runner caches; injected test runners bypass this entirely, so test isolation is preserved.
|
||||
*/
|
||||
// The 7d baseline is cached per env and query. Only the default runner caches.
|
||||
const BASELINE_CACHE_TTL_MS = 5 * 60_000;
|
||||
const baselineCache = new Map<string, { expiresAt: number; result: QueryResult }>();
|
||||
|
||||
/**
|
||||
* Store a baseline result, first sweeping expired entries so envs that stop requesting reports
|
||||
* don't linger in memory forever. Writes only happen on a cache miss (~once per env per TTL), so
|
||||
* a full sweep here is cheap and keeps the map bounded to recently-active environments.
|
||||
*/
|
||||
/** Sweeps expired entries first so envs that stop requesting reports don't linger in memory. */
|
||||
function cacheBaseline(key: string, result: QueryResult, now: number) {
|
||||
for (const [k, v] of baselineCache) {
|
||||
if (v.expiresAt <= now) baselineCache.delete(k);
|
||||
@@ -185,10 +147,7 @@ async function executeReportQuery(
|
||||
if (cacheKey) cacheBaseline(cacheKey, out, Date.now());
|
||||
return out;
|
||||
}
|
||||
// Retry transient concurrency rejections; rethrow anything else (e.g. a bad query) so
|
||||
// callers/tryQuery can handle it. Exponential "full jitter" backoff — delay picked uniformly
|
||||
// from [0, min(cap, base·2^attempt)) — so concurrent report requests don't wake in lockstep
|
||||
// and re-collide on the same 3 query slots.
|
||||
// Full-jitter backoff so concurrent report requests don't wake in lockstep and re-collide.
|
||||
if (attempt < CH_REJECTION_RETRIES && isConcurrencyRejection(result.error)) {
|
||||
const window = Math.min(CH_REJECTION_BACKOFF_CAP_MS, CH_REJECTION_BACKOFF_MS * 2 ** attempt);
|
||||
await new Promise((resolve) => setTimeout(resolve, Math.floor(Math.random() * window)));
|
||||
@@ -198,17 +157,14 @@ async function executeReportQuery(
|
||||
}
|
||||
}
|
||||
|
||||
// ---------------------------------------------------------------------------
|
||||
// runs queries (execution + liveness + throughput; also feed the snapshot fallback).
|
||||
// executeQuery injects tenant isolation + the time window, so we never write WHERE.
|
||||
// ---------------------------------------------------------------------------
|
||||
|
||||
// executeQuery injects tenant isolation and the time window, so these queries never write WHERE.
|
||||
function runsScalarQuery(): string {
|
||||
return `SELECT
|
||||
quantile(0.95)(queued_duration) AS start_latency_p95,
|
||||
quantile(0.95)(execution_duration) AS dur_p95,
|
||||
countIf(status IN (${FAILURE_STATUSES})) AS failures,
|
||||
countIf(status = 'Completed') AS completed,
|
||||
countIf(status IN (${FINISHED_STATUSES})) AS finished,
|
||||
count() AS triggered,
|
||||
max(triggered_at) AS last_activity
|
||||
FROM runs`;
|
||||
@@ -227,10 +183,6 @@ GROUP BY t
|
||||
ORDER BY t`;
|
||||
}
|
||||
|
||||
// ---------------------------------------------------------------------------
|
||||
// env_metrics queries (measured queue depth + scheduling delay).
|
||||
// ---------------------------------------------------------------------------
|
||||
|
||||
function envSeriesQuery(): string {
|
||||
return `SELECT
|
||||
timeBucket() AS t,
|
||||
@@ -253,9 +205,8 @@ FROM env_metrics`;
|
||||
}
|
||||
|
||||
/**
|
||||
* Worst queue by share of CURRENT pending. `argMax(max_queued, bucket_start)` = each queue's
|
||||
* depth in its latest bucket, so the shares sum to a real point-in-time backlog — not a sum of
|
||||
* per-queue peaks from different moments (which isn't "% of pending" at any instant). Best-effort.
|
||||
* `argMax(max_queued, bucket_start)` is a point-in-time depth, not a peak. These rows stop at 20, so
|
||||
* the share's denominator comes from `queueTotalsQuery`.
|
||||
*/
|
||||
function queueWorstQuery(): string {
|
||||
return `SELECT
|
||||
@@ -268,20 +219,21 @@ LIMIT 20`;
|
||||
}
|
||||
|
||||
/**
|
||||
* Runs dead-lettered across the window, summed over queues. `dlq_delta` is per-queue
|
||||
* cumulative-counter state, so it must be merged per queue then summed (never merged
|
||||
* across queues). Best-effort — absent columns just yield no rows.
|
||||
* `dlq_delta` must be merged per queue then summed, never merged across queues. `total_queued` must
|
||||
* be computed here, not by summing the top-20 `queueWorstQuery` rows.
|
||||
*/
|
||||
function dlqTotalQuery(): string {
|
||||
return `SELECT sum(dlq) AS dlq_total
|
||||
function queueTotalsQuery(): string {
|
||||
return `SELECT sum(dlq) AS dlq_total, sum(latest_queued) AS total_queued
|
||||
FROM (
|
||||
SELECT deltaSumTimestampMerge(dlq_delta) AS dlq
|
||||
SELECT
|
||||
deltaSumTimestampMerge(dlq_delta) AS dlq,
|
||||
argMax(max_queued, bucket_start) AS latest_queued
|
||||
FROM queue_metrics
|
||||
GROUP BY queue
|
||||
)`;
|
||||
}
|
||||
|
||||
/** Top failing task (lazy — only when execution degrades). Best-effort. */
|
||||
/** Top failing task. Loaded lazily, only when execution degrades. */
|
||||
function failureBreakdownQuery(): string {
|
||||
return `SELECT
|
||||
task_identifier AS task,
|
||||
@@ -292,40 +244,31 @@ ORDER BY fails DESC
|
||||
LIMIT 10`;
|
||||
}
|
||||
|
||||
/** Default IO wiring — the real query-service runner + the engine's env-queue length. */
|
||||
const defaultHealthDeps: HealthDeps = {
|
||||
runQuery: executeReportQuery,
|
||||
lengthOfEnvQueue: (env) => engine.lengthOfEnvQueue(env),
|
||||
};
|
||||
|
||||
/** Run a query that may reference not-yet-available columns; never break the report. */
|
||||
/** Never throws. Callers treat no rows as unmeasured rather than as a measured zero. */
|
||||
async function tryQuery(
|
||||
deps: HealthDeps,
|
||||
env: AuthenticatedEnvironment,
|
||||
query: string,
|
||||
period: string
|
||||
): Promise<Row[]> {
|
||||
): Promise<QueryResult> {
|
||||
try {
|
||||
return (await deps.runQuery(env, query, period)).rows;
|
||||
return await deps.runQuery(env, query, period);
|
||||
} catch {
|
||||
return [];
|
||||
return { rows: [], timeRange: { from: new Date(0), to: new Date(0) } };
|
||||
}
|
||||
}
|
||||
|
||||
// ---------------------------------------------------------------------------
|
||||
// FlowSource seam (§7 Seam A).
|
||||
// ---------------------------------------------------------------------------
|
||||
|
||||
export type FlowData = {
|
||||
flowSource: HealthInput["flowSource"];
|
||||
pending: { now: number; normal?: number; series: number[]; estimated: boolean };
|
||||
pending: HealthInput["pending"];
|
||||
startLatency: HealthInput["startLatency"];
|
||||
evidence: HealthInput["flowEvidence"];
|
||||
/**
|
||||
* Epoch ms of the freshest telemetry the source saw (latest env_metrics bucket and/or latest
|
||||
* run) — how the report tells "data current" from "pipeline stale", independent of traffic.
|
||||
* null when no signal exists at all (brand-new/empty env) -> liveness "unknown", not stale.
|
||||
*/
|
||||
/** Epoch ms of the freshest telemetry. Null means no signal, which makes liveness unknown. */
|
||||
telemetryLastTs: number | null;
|
||||
};
|
||||
|
||||
@@ -337,116 +280,197 @@ const EMPTY_EVIDENCE: HealthInput["flowEvidence"] = {
|
||||
dlqDelta: null, // snapshot path: dead-letter volume is unmeasured
|
||||
};
|
||||
|
||||
/** The runs results loadHealthInput already fetched, so the snapshot fallback needn't re-query. */
|
||||
type RunsContext = { liveScalar: Row; liveSeries: Row[]; baselineScalar: Row };
|
||||
|
||||
/**
|
||||
* "unavailable" is a recognized rollout state, so the next source down is a legitimate substitute.
|
||||
* "failed" is anything else and must make the flow verdict unassessable, never fall through to it.
|
||||
*/
|
||||
export type FlowLoadResult =
|
||||
| { status: "ok"; data: FlowData }
|
||||
| { status: "unavailable" }
|
||||
| { status: "failed"; error: unknown };
|
||||
|
||||
export interface FlowSource {
|
||||
loadFlow(
|
||||
env: AuthenticatedEnvironment,
|
||||
period: string,
|
||||
ctx: RunsContext,
|
||||
deps: HealthDeps
|
||||
): Promise<FlowData | null>;
|
||||
): Promise<FlowLoadResult>;
|
||||
}
|
||||
|
||||
/**
|
||||
* Preferred source: measured queue depth + scheduling-delay p95 from `env_metrics`.
|
||||
* Returns null when the pipeline hasn't populated the table yet, so the caller can fall
|
||||
* back to the snapshot.
|
||||
* The only failures the measured source may treat as a benign fallback: the object itself does not
|
||||
* exist yet, so waiting is correct. Codes 60 (table) and 81 (database). Matched on error text
|
||||
* because the client collapses the error into a message.
|
||||
*
|
||||
* 47 (UNKNOWN_IDENTIFIER) is deliberately absent: the table exists and a column in our own query
|
||||
* does not, which is a broken query, not a rollout gap. It must surface as a failed source.
|
||||
*/
|
||||
const ROLLOUT_ERROR_PATTERNS = [
|
||||
/\bUNKNOWN_(?:TABLE|DATABASE)\b/,
|
||||
/\bCode:\s*(?:60|81)\b/,
|
||||
/\bTable\b[^.]*\bdoes\s?n?o?t?'?t?\s*exist/i,
|
||||
/\bUnknown (?:table|database)\b/i,
|
||||
];
|
||||
|
||||
function isRolloutError(error: unknown): boolean {
|
||||
// Prefer a structured code/type if one ever survives the wrapping.
|
||||
if (typeof error === "object" && error !== null) {
|
||||
const record = error as Record<string, unknown>;
|
||||
const code = String(record.code ?? "");
|
||||
const type = String(record.type ?? "");
|
||||
if (code === "60" || code === "81") return true;
|
||||
if (/^UNKNOWN_(TABLE|DATABASE)$/.test(type)) return true;
|
||||
}
|
||||
const message =
|
||||
error instanceof Error
|
||||
? error.message
|
||||
: typeof error === "string"
|
||||
? error
|
||||
: String(error ?? "");
|
||||
return ROLLOUT_ERROR_PATTERNS.some((pattern) => pattern.test(message));
|
||||
}
|
||||
|
||||
/** Measured depth and scheduling-delay p95 from `env_metrics`. Unavailable until it is populated. */
|
||||
export const QueueMetricsSource: FlowSource = {
|
||||
async loadFlow(env, period, ctx, deps) {
|
||||
try {
|
||||
// Redis depth isn't a CH query, so it runs alongside (doesn't count toward the cap).
|
||||
// Guard the rejection: if the CH queries throw first we jump to catch without awaiting
|
||||
// this, and an unhandled Redis rejection would crash the process.
|
||||
// The rejection must be guarded: if the queries below throw first this is never awaited, and
|
||||
// an unhandled Redis rejection would crash the process.
|
||||
const pendingNowPromise = deps.lengthOfEnvQueue(env).catch(() => undefined);
|
||||
|
||||
// Bug 1 fix — route all CH queries through the concurrency cap (max 2 in flight).
|
||||
// Every task returns Row[] (scalars indexed after) so mapWithConcurrency infers a
|
||||
// single element type — a mixed Row[]/Row union trips its generic inference.
|
||||
const [seriesRows, liveScalarRows, baselineScalarRows, worstQueueRows, dlqResultRows] =
|
||||
// Every task must return the same shape; a mixed union trips mapWithConcurrency's inference.
|
||||
const [seriesResult, liveScalarResult, baselineScalarResult, worstQueueResult, totalsResult] =
|
||||
await mapWithConcurrency(
|
||||
[
|
||||
() => deps.runQuery(env, envSeriesQuery(), period).then((r) => r.rows),
|
||||
() => deps.runQuery(env, envScalarQuery(), period).then((r) => r.rows),
|
||||
() => deps.runQuery(env, envScalarQuery(), BASELINE_PERIOD).then((r) => r.rows),
|
||||
() => deps.runQuery(env, envSeriesQuery(), period),
|
||||
() => deps.runQuery(env, envScalarQuery(), period),
|
||||
() => deps.runQuery(env, envScalarQuery(), BASELINE_PERIOD),
|
||||
() => tryQuery(deps, env, queueWorstQuery(), period),
|
||||
() => tryQuery(deps, env, dlqTotalQuery(), period),
|
||||
() => tryQuery(deps, env, queueTotalsQuery(), period),
|
||||
],
|
||||
CH_CONCURRENCY,
|
||||
(task) => task()
|
||||
);
|
||||
const liveScalarRow = liveScalarRows[0] ?? {};
|
||||
const baselineScalarRow = baselineScalarRows[0] ?? {};
|
||||
const seriesRows = seriesResult.rows;
|
||||
const liveScalarRow = liveScalarResult.rows[0] ?? {};
|
||||
const baselineScalarRow = baselineScalarResult.rows[0] ?? {};
|
||||
|
||||
const pendingNow = await pendingNowPromise;
|
||||
|
||||
if (seriesRows.length === 0) {
|
||||
return null; // no measured data yet -> snapshot fallback
|
||||
return { status: "unavailable" }; // no measured data yet -> snapshot fallback
|
||||
}
|
||||
|
||||
// Telemetry freshness = the freshest of the latest env_metrics bucket (a heartbeat
|
||||
// independent of traffic) and the latest run recorded.
|
||||
// Freshness is the newer of the latest env_metrics bucket and the latest run.
|
||||
const telemetryLastTs = freshestTs(liveScalarRow.last_bucket, ctx.liveScalar.last_activity);
|
||||
|
||||
return buildQueueMetricsFlow(
|
||||
seriesRows,
|
||||
liveScalarRow,
|
||||
baselineScalarRow,
|
||||
worstQueueRows,
|
||||
dlqResultRows,
|
||||
return {
|
||||
status: "ok",
|
||||
data: buildQueueMetricsFlow({
|
||||
series: seriesRows,
|
||||
sampling: envSampling(seriesResult.timeRange),
|
||||
liveScalar: liveScalarRow,
|
||||
baselineScalar: baselineScalarRow,
|
||||
worstRows: worstQueueResult.rows,
|
||||
totalsRows: totalsResult.rows,
|
||||
pendingNow,
|
||||
telemetryLastTs
|
||||
);
|
||||
} catch {
|
||||
// Bug 2 fix — if `env_metrics` isn't available the queries throw; return null so
|
||||
// loadHealthInput falls back to the snapshot instead of propagating a 500.
|
||||
return null;
|
||||
telemetryLastTs,
|
||||
}),
|
||||
};
|
||||
} catch (error) {
|
||||
// Only a rollout error is a benign fallback. Anything else must surface as failed.
|
||||
if (isRolloutError(error)) return { status: "unavailable" };
|
||||
logger.error("report health: measured flow source failed", {
|
||||
environmentId: env.id,
|
||||
error: error instanceof Error ? error.message : String(error),
|
||||
});
|
||||
return { status: "failed", error };
|
||||
}
|
||||
},
|
||||
};
|
||||
|
||||
function buildQueueMetricsFlow(
|
||||
series: Row[],
|
||||
liveScalar: Row,
|
||||
baselineScalar: Row,
|
||||
worstRows: Row[],
|
||||
dlqRows: Row[],
|
||||
pendingNow: number | undefined,
|
||||
telemetryLastTs: number | null
|
||||
): FlowData {
|
||||
// Dead-letter volume (0 = measured none; no rows -> unmeasured -> null).
|
||||
const dlqDelta = dlqRows.length > 0 ? Math.round(num(dlqRows[0].dlq_total)) : null;
|
||||
/** Minutes per env_metrics bucket, matching the interval the query printer emits. */
|
||||
const INTERVAL_UNIT_MINUTES: Record<TimeBucketInterval["unit"], number> = {
|
||||
SECOND: 1 / 60,
|
||||
MINUTE: 1,
|
||||
HOUR: 60,
|
||||
DAY: 1440,
|
||||
WEEK: 10_080,
|
||||
MONTH: 43_200,
|
||||
};
|
||||
|
||||
// Throttled share = fraction of buckets with any queue-level throttling.
|
||||
const throttledShare =
|
||||
series.length > 0 ? series.filter((r) => num(r.throttled) > 0).length / series.length : 0;
|
||||
/** Derived from the query printer's own thresholds, so "expected" matches what the query emits. */
|
||||
function envSampling(range: {
|
||||
from: Date;
|
||||
to: Date;
|
||||
}): { bucketMinutes: number; expectedBuckets: number } | null {
|
||||
const windowMinutes = timeRangeMinutes(range);
|
||||
if (windowMinutes === 0) return null;
|
||||
const interval = calculateTimeBucketInterval(
|
||||
range.from,
|
||||
range.to,
|
||||
envMetricsSchema.timeBucketThresholds
|
||||
);
|
||||
const bucketMinutes = interval.value * INTERVAL_UNIT_MINUTES[interval.unit];
|
||||
if (!(bucketMinutes > 0)) return null;
|
||||
return { bucketMinutes, expectedBuckets: Math.max(1, Math.round(windowMinutes / bucketMinutes)) };
|
||||
}
|
||||
|
||||
// Worst queue = top queue's share of current pending (latest-bucket depths, so shares
|
||||
// sum to a real point-in-time backlog).
|
||||
function buildQueueMetricsFlow(args: {
|
||||
series: Row[];
|
||||
sampling: { bucketMinutes: number; expectedBuckets: number } | null;
|
||||
liveScalar: Row;
|
||||
baselineScalar: Row;
|
||||
worstRows: Row[];
|
||||
totalsRows: Row[];
|
||||
pendingNow: number | undefined;
|
||||
telemetryLastTs: number | null;
|
||||
}): FlowData {
|
||||
const { series, sampling, liveScalar, baselineScalar, worstRows, totalsRows } = args;
|
||||
const totals = totalsRows[0];
|
||||
// Zero means measured none; no rows means unmeasured.
|
||||
const dlqDelta = totals !== undefined ? Math.round(num(totals.dlq_total)) : null;
|
||||
|
||||
// Measured against expected buckets when the cadence is known, not against received rows.
|
||||
const throttledBuckets = series.filter((r) => num(r.throttled) > 0).length;
|
||||
const throttledDenominator = sampling?.expectedBuckets ?? series.length;
|
||||
const throttledShare = throttledDenominator > 0 ? throttledBuckets / throttledDenominator : 0;
|
||||
|
||||
// The denominator is the env-wide total, never the sum of these top-20 rows. No total, no share.
|
||||
let worstQueue: HealthInput["flowEvidence"]["worstQueue"] = null;
|
||||
if (worstRows.length > 0) {
|
||||
const depths = worstRows.map((r) => num(r.latest_queued));
|
||||
const total = depths.reduce((a, b) => a + b, 0);
|
||||
if (total > 0) {
|
||||
worstQueue = { name: String(worstRows[0].name ?? "unknown"), share: depths[0] / total };
|
||||
const totalQueued = totals !== undefined ? num(totals.total_queued) : 0;
|
||||
if (worstRows.length > 0 && totalQueued > 0) {
|
||||
const worstDepth = num(worstRows[0].latest_queued);
|
||||
if (worstDepth > 0) {
|
||||
worstQueue = {
|
||||
name: String(worstRows[0].name ?? "unknown"),
|
||||
share: Math.min(1, worstDepth / totalQueued),
|
||||
};
|
||||
}
|
||||
}
|
||||
|
||||
// Prefer live Redis depth; if it's unavailable fall back to the latest MEASURED queued from
|
||||
// env_metrics (still a real number) rather than a misleading confident zero (#7).
|
||||
// Prefer live Redis depth, then the latest measured queued from env_metrics.
|
||||
const lastMeasuredQueued = num(series[series.length - 1]?.queued);
|
||||
|
||||
// Only carried when every bucket parsed: a partial set would make "adjacent" meaningless.
|
||||
const bucketTimestamps = series.map((r) => parseTimestamp(r.t));
|
||||
const runningBucketsMs = bucketTimestamps.every((t): t is number => t !== null)
|
||||
? bucketTimestamps
|
||||
: undefined;
|
||||
|
||||
const waitP95 = optionalNum(liveScalar.wait_p95);
|
||||
|
||||
return {
|
||||
flowSource: "queue_metrics_v1",
|
||||
pending: {
|
||||
now: pendingNow ?? lastMeasuredQueued,
|
||||
now: args.pendingNow ?? lastMeasuredQueued,
|
||||
normal: Math.round(num(baselineScalar.avg_queued)),
|
||||
series: resampleSeries(series.map((r) => num(r.queued))),
|
||||
estimated: false, // measured
|
||||
availability: "measured",
|
||||
},
|
||||
startLatency: {
|
||||
p95Ms: waitP95 ?? 0,
|
||||
@@ -455,30 +479,28 @@ function buildQueueMetricsFlow(
|
||||
availability: waitP95 === undefined ? "unknown" : "measured",
|
||||
},
|
||||
evidence: {
|
||||
// native resolution — cause discriminators read shares off this series.
|
||||
runningSeries: series.map((r) => num(r.running)),
|
||||
runningBucketsMs,
|
||||
sampling,
|
||||
envLimit: num(liveScalar.env_limit),
|
||||
throttledShare,
|
||||
worstQueue,
|
||||
dlqDelta,
|
||||
},
|
||||
telemetryLastTs,
|
||||
telemetryLastTs: args.telemetryLastTs,
|
||||
};
|
||||
}
|
||||
|
||||
/**
|
||||
* Fallback: live Redis depth (accurate "now") + an estimated backlog proxy from `runs`
|
||||
* (cumulative triggered - finished) and `runs.queued_duration` for latency. The proxy is a
|
||||
* shape-only TREND (`estimated: true`): it starts at 0 within the window and can't see
|
||||
* backlog that predates it.
|
||||
* Fallback: live Redis depth plus a backlog proxy from `runs` (triggered minus finished). The proxy
|
||||
* is shape-only: it starts at 0 within the window and can't see backlog that predates it.
|
||||
*/
|
||||
export const SnapshotFlowSource: FlowSource = {
|
||||
async loadFlow(env, _period, ctx, deps) {
|
||||
// Guard Redis: this is the last-resort source, so a failure must not break the report.
|
||||
const pendingNow = (await deps.lengthOfEnvQueue(env).catch(() => undefined)) ?? 0;
|
||||
// Last-resort source, so a Redis failure must not break the report.
|
||||
const pendingNow = await deps.lengthOfEnvQueue(env).catch(() => undefined);
|
||||
|
||||
// Subtract ALL terminal runs, not just Completed — else failed/expired/canceled runs
|
||||
// linger in the proxy as phantom backlog forever.
|
||||
// Subtract all terminal runs, or failed and canceled runs linger as phantom backlog.
|
||||
let backlog = 0;
|
||||
const proxy = ctx.liveSeries.map((r) => {
|
||||
backlog = Math.max(0, backlog + num(r.triggered) - num(r.finished));
|
||||
@@ -487,15 +509,22 @@ export const SnapshotFlowSource: FlowSource = {
|
||||
const series = resampleSeries(proxy);
|
||||
const startLatencyP95 = optionalNum(ctx.liveScalar.start_latency_p95);
|
||||
|
||||
// Redis is the only depth measurement here, so a failure must not substitute 0. Fall back to the
|
||||
// last proxy point and mark the depth unknown.
|
||||
const depthUnavailable = pendingNow === undefined;
|
||||
const lastProxyPoint = proxy.length > 0 ? proxy[proxy.length - 1] : 0;
|
||||
|
||||
return {
|
||||
status: "ok",
|
||||
data: {
|
||||
flowSource: "snapshot+runs",
|
||||
pending: {
|
||||
now: pendingNow,
|
||||
// No 7d pending baseline on this path — omit `normal` rather than pass off a
|
||||
// live-window proxy average as "7d normal" (#8). Severity falls back to an absolute floor.
|
||||
now: pendingNow ?? lastProxyPoint,
|
||||
// No 7d pending baseline on this path, so severity falls back to an absolute floor.
|
||||
normal: undefined,
|
||||
series,
|
||||
estimated: true,
|
||||
availability: depthUnavailable ? "unknown" : "measured",
|
||||
},
|
||||
startLatency: {
|
||||
p95Ms: startLatencyP95 ?? 0,
|
||||
@@ -505,25 +534,19 @@ export const SnapshotFlowSource: FlowSource = {
|
||||
},
|
||||
// No cause-tree evidence; interpret falls back to v1 symptoms.
|
||||
evidence: EMPTY_EVIDENCE,
|
||||
// No env_metrics heartbeat here — the only freshness signal is the latest run recorded
|
||||
// (null when the env has no runs at all -> liveness "unknown", not stale).
|
||||
telemetryLastTs: freshestTs(ctx.liveScalar.last_activity),
|
||||
// This path has no pipeline heartbeat, and run activity is not one.
|
||||
telemetryLastTs: null,
|
||||
},
|
||||
};
|
||||
},
|
||||
};
|
||||
|
||||
// ---------------------------------------------------------------------------
|
||||
// loadHealthInput.
|
||||
// ---------------------------------------------------------------------------
|
||||
|
||||
export async function loadHealthInput(
|
||||
env: AuthenticatedEnvironment,
|
||||
period: string,
|
||||
now: Date = new Date(),
|
||||
deps: HealthDeps = defaultHealthDeps
|
||||
): Promise<HealthInput> {
|
||||
// Bug 1 fix — route the runs-phase CH queries through the concurrency cap (max 2)
|
||||
// instead of firing all 3 at once, so we never exceed the per-project limit.
|
||||
const [liveScalarRes, liveSeriesRes, baselineScalarRes] = await mapWithConcurrency(
|
||||
[
|
||||
() => deps.runQuery(env, runsScalarQuery(), period),
|
||||
@@ -539,17 +562,24 @@ export async function loadHealthInput(
|
||||
baselineScalar: baselineScalarRes.rows[0] ?? {},
|
||||
};
|
||||
|
||||
// Window lengths come from the query service's resolved (clip-aware) range, not a
|
||||
// re-parse of the period — so `maxQueryPeriod` clipping can't skew per-minute rates
|
||||
// or annotation math. periodToMinutes is a fallback for a degenerate range.
|
||||
// Window lengths come from the resolved range, not the period, so clipping can't skew rates.
|
||||
const windowMinutes = timeRangeMinutes(liveSeriesRes.timeRange) || periodToMinutes(period);
|
||||
const baselineMinutes =
|
||||
timeRangeMinutes(baselineScalarRes.timeRange) || periodToMinutes(BASELINE_PERIOD);
|
||||
|
||||
// Prefer measured queue metrics; fall back to the runs snapshot when unavailable.
|
||||
const flow =
|
||||
(await QueueMetricsSource.loadFlow(env, period, ctx, deps)) ??
|
||||
(await SnapshotFlowSource.loadFlow(env, period, ctx, deps))!;
|
||||
// A measured source that failed still falls back for the remaining shape, but its depth is marked
|
||||
// unknown so a failure is never presented as a measurement.
|
||||
const measured = await QueueMetricsSource.loadFlow(env, period, ctx, deps);
|
||||
let flow: FlowData;
|
||||
if (measured.status === "ok") {
|
||||
flow = measured.data;
|
||||
} else {
|
||||
const snapshot = await SnapshotFlowSource.loadFlow(env, period, ctx, deps);
|
||||
flow = (snapshot as { status: "ok"; data: FlowData }).data;
|
||||
if (measured.status === "failed") {
|
||||
flow = { ...flow, pending: { ...flow.pending, availability: "unknown" } };
|
||||
}
|
||||
}
|
||||
|
||||
const failuresSeries = resampleSeries(
|
||||
ctx.liveSeries.map((r) => failureRate(num(r.failures), num(r.completed)))
|
||||
@@ -557,8 +587,12 @@ export async function loadHealthInput(
|
||||
|
||||
const triggered = num(ctx.liveScalar.triggered);
|
||||
const completed = num(ctx.liveScalar.completed);
|
||||
const donePerMin = windowMinutes === 0 ? 0 : completed / windowMinutes;
|
||||
const triggeredPerMin = windowMinutes === 0 ? 0 : triggered / windowMinutes;
|
||||
// Older rows have no `finished`, so fall back to completions rather than a fabricated 0 drain.
|
||||
const finished = num(ctx.liveScalar.finished, completed);
|
||||
const perMin = (total: number) => (windowMinutes === 0 ? 0 : total / windowMinutes);
|
||||
const finishedPerMin = perMin(finished);
|
||||
const completedPerMin = perMin(completed);
|
||||
const triggeredPerMin = perMin(triggered);
|
||||
const normalTriggeredPerMin =
|
||||
baselineMinutes === 0 ? 0 : num(ctx.baselineScalar.triggered) / baselineMinutes;
|
||||
|
||||
@@ -568,9 +602,7 @@ export async function loadHealthInput(
|
||||
num(ctx.baselineScalar.completed)
|
||||
);
|
||||
|
||||
// Lazy failure attribution — only when execution is actually degraded.
|
||||
// A fresh failure pattern from a clean 0% baseline is exactly when attribution matters most,
|
||||
// so a zero baseline (can't form a ratio) counts as degraded once past the floor.
|
||||
// A zero baseline can't form a ratio, so it counts as degraded once past the floor.
|
||||
const failureDegraded =
|
||||
rate >= HEALTH_THRESHOLDS.failures.floorRate &&
|
||||
(normalRate === 0 || rate / normalRate >= HEALTH_THRESHOLDS.failures.warnMult);
|
||||
@@ -578,8 +610,7 @@ export async function loadHealthInput(
|
||||
? await loadFailureBreakdown(deps, env, period, num(ctx.liveScalar.failures))
|
||||
: undefined;
|
||||
|
||||
// Liveness = telemetry freshness (how recent is the newest data), NOT "recent completions":
|
||||
// a quiet env with a fresh pipeline is fresh; a dead pipeline is stale. null -> unknown.
|
||||
// Liveness measures telemetry freshness, not recent completions.
|
||||
const telemetryAgeMs =
|
||||
flow.telemetryLastTs === null ? null : Math.max(0, now.getTime() - flow.telemetryLastTs);
|
||||
|
||||
@@ -592,7 +623,7 @@ export async function loadHealthInput(
|
||||
flowSource: flow.flowSource,
|
||||
pending: flow.pending,
|
||||
startLatency: flow.startLatency,
|
||||
throughput: { donePerMin, triggeredPerMin, normalTriggeredPerMin },
|
||||
throughput: { finishedPerMin, completedPerMin, triggeredPerMin, normalTriggeredPerMin },
|
||||
failures: { rate, normalRate, series: failuresSeries },
|
||||
duration: {
|
||||
p95Ms: num(ctx.liveScalar.dur_p95),
|
||||
@@ -611,16 +642,12 @@ async function loadFailureBreakdown(
|
||||
totalFails: number
|
||||
): Promise<HealthInput["failureBreakdown"]> {
|
||||
if (totalFails <= 0) return undefined;
|
||||
const rows = await tryQuery(deps, env, failureBreakdownQuery(), period);
|
||||
const { rows } = await tryQuery(deps, env, failureBreakdownQuery(), period);
|
||||
if (rows.length === 0) return undefined;
|
||||
const top = rows[0];
|
||||
return { task: String(top.task ?? "unknown"), share: num(top.fails) / totalFails };
|
||||
}
|
||||
|
||||
// ---------------------------------------------------------------------------
|
||||
// Small local helpers.
|
||||
// ---------------------------------------------------------------------------
|
||||
|
||||
function parseTimestamp(value: unknown): number | null {
|
||||
if (value === null || value === undefined || value === "") return null;
|
||||
// ClickHouse returns a "1970-01-01 00:00:00" sentinel for max() over no rows.
|
||||
|
||||
@@ -1,13 +1,3 @@
|
||||
/**
|
||||
* The `health` report's PROSE — the single place its codes resolve to strings. Registered
|
||||
* under the "health" title so the generic renderer can look it up by `vm.title` without ever
|
||||
* importing health vocabulary. A future report (Cost, Regression) ships its own catalog the
|
||||
* same way; `report-messages.ts` stays report-agnostic infrastructure.
|
||||
*
|
||||
* Some strings carry {tokens} (e.g. {age}, {rate}) the renderer fills from the finding's
|
||||
* metrics / evidence — meaning lives here, numbers stay facts.
|
||||
*/
|
||||
|
||||
import { type ReportMessages } from "../report-messages";
|
||||
import { type ReasonCode, type Severity } from "../report-view-model";
|
||||
|
||||
@@ -24,24 +14,17 @@ const METRIC_LABELS: Record<string, string> = {
|
||||
triggered: "triggered",
|
||||
};
|
||||
|
||||
/**
|
||||
* Finding headline — keyed by `${findingType}/${reason}`. Degraded = the cause,
|
||||
* healthy = reassurance. `@expanded` variants show a healthy finding expanded
|
||||
* (e.g. execution while flow is degraded).
|
||||
*/
|
||||
/** Keyed by `${findingType}/${reason}`, with optional `@expanded` variants. */
|
||||
const FINDING_REASONS: Record<string, string> = {
|
||||
// flow — causes
|
||||
"flow/env_limit_saturation": "at your env concurrency limit",
|
||||
"flow/dequeue_stall": "capacity is free but nothing is dequeuing",
|
||||
"flow/queue_limit_throttling": "a queue is throttling at its own limit",
|
||||
"flow/trigger_spike": "a trigger spike is backing up the queue",
|
||||
"flow/trigger_surge": "a surge of new triggers is backing up the queue",
|
||||
// flow — fallback symptoms (v1)
|
||||
"flow/start_latency": "runs are slow to start",
|
||||
"flow/backlog": "backlog is growing",
|
||||
"flow/throughput_lag": "completion is falling behind triggers",
|
||||
"flow/degraded": "flow is degraded",
|
||||
// flow — healthy (collapsed)
|
||||
"flow/healthy": "starting normally",
|
||||
// execution
|
||||
"execution/failures_up": "runs are failing more than usual",
|
||||
@@ -49,9 +32,10 @@ const FINDING_REASONS: Record<string, string> = {
|
||||
"execution/degraded": "execution is degraded",
|
||||
"execution/unknown": "execution can't be assessed — the telemetry is stale",
|
||||
"flow/unknown": "flow can't be assessed — the telemetry is stale",
|
||||
"flow/flow_unmeasured": "flow can't be assessed — the queue depth couldn't be measured",
|
||||
"execution/healthy": "completing normally", // collapsed
|
||||
"execution/healthy@expanded": "the runs that DO start are fine",
|
||||
// liveness = telemetry freshness ({age} filled by the renderer)
|
||||
"execution/healthy@expanded": "runs are executing normally",
|
||||
// liveness is telemetry freshness
|
||||
"liveness/fresh": "fresh — telemetry current, updated {age} ago",
|
||||
"liveness/lagging": "lagging — telemetry last updated {age} ago",
|
||||
"liveness/stale": "stale — no telemetry in {age}",
|
||||
@@ -66,7 +50,6 @@ const READS: Record<string, string> = {
|
||||
queue_throttle_chain: "queue at its limit → its runs wait → backlog grows",
|
||||
spike_chain: "triggers jumped {mult}× → queue fills faster than it drains",
|
||||
surge_chain: "new triggers arriving with no prior baseline → queue fills faster than it drains",
|
||||
// fallback symptoms (v1)
|
||||
starting_normally: "runs are starting on time",
|
||||
lag_while_triggering_normal: "triggering normally, but starts lag → work is backing up",
|
||||
lag_and_failures: "runs are lagging AND failing — check the code path",
|
||||
@@ -75,32 +58,33 @@ const READS: Record<string, string> = {
|
||||
runs_are_fine: "runs are completing normally",
|
||||
failures_elevated: "failures are elevated — check the code path",
|
||||
data_stale: "data is stale — the verdict cannot be trusted",
|
||||
flow_unmeasured: "the queue depth is unavailable — the backlog cannot be assessed",
|
||||
};
|
||||
|
||||
/** Exclusion (ruled-out cause) — rendered under `read:`. {tokens} filled from evidence. */
|
||||
/** A ruled-out cause, rendered under `read:`. {tokens} are filled from evidence. */
|
||||
const EXCLUSIONS: Record<string, string> = {
|
||||
not_env_limit: "env concurrency limit is not the bottleneck",
|
||||
not_your_code: "not your code — failures and durations normal",
|
||||
not_your_config: "not your config — limits aren't the bottleneck",
|
||||
};
|
||||
|
||||
/** Observation (supporting fact, not a ruled-out cause) — rendered under `read:` after exclusions. */
|
||||
/** A supporting fact rather than a ruled-out cause, rendered after the exclusions. */
|
||||
const OBSERVATIONS: Record<string, string> = {
|
||||
not_workers_platform: "runs are completing at ~{rate}/min",
|
||||
not_workers_platform: "runs are finishing at ~{rate}/min",
|
||||
execution_healthy: "runs that start are completing normally",
|
||||
nothing_dead_lettered: "nothing dead-lettered",
|
||||
};
|
||||
|
||||
/** Metric annotation shown on a cause line INSTEAD of "(normal ~x)". {tokens} filled by the renderer. */
|
||||
/** Metric annotation shown on a cause line in place of the normal baseline. */
|
||||
const ANNOTATIONS: Record<string, string> = {
|
||||
pinned_minutes: "pinned {value} of last {window} min",
|
||||
pinned_minutes: "{value} min at limit",
|
||||
idle_share: "idle — {value} running of {limit}",
|
||||
throttled_minutes: "throttled {value} of last {window} min",
|
||||
spike_mult: "{value}× the normal rate",
|
||||
surge_rate: "{value}/min, no prior baseline",
|
||||
};
|
||||
|
||||
/** Headline statement — keyed by `${findingType}/${severity}`. */
|
||||
/** Headline statement, keyed by `${findingType}/${severity}`. */
|
||||
const STATEMENTS: Record<string, string> = {
|
||||
"flow/ok": "Flow healthy",
|
||||
"flow/warn": "Flow slowing",
|
||||
@@ -113,9 +97,8 @@ const STATEMENTS: Record<string, string> = {
|
||||
"liveness/crit": "data stale",
|
||||
};
|
||||
|
||||
/** Recommendation / footer codes -> calm, jargon-free action text. */
|
||||
/** Recommendation and footer codes resolved to action text. */
|
||||
const ACTIONS: Record<string, string> = {
|
||||
// Review = open concrete data · Check = system state · Raise = a settings change
|
||||
review_start_latency: "Review start latency",
|
||||
review_failing_tasks: "Review failing tasks",
|
||||
review_slow_runs: "Review slow runs",
|
||||
@@ -125,6 +108,8 @@ const ACTIONS: Record<string, string> = {
|
||||
check_control_plane: "Check control plane",
|
||||
check_platform_status: "Check status.trigger.dev — no action needed on yours",
|
||||
raise_env_limit: "Raise the env concurrency limit",
|
||||
contact_us_raise_limit: "Contact us to raise the limit",
|
||||
concurrency_docs: "Read concurrency docs",
|
||||
raise_queue_limit: "Raise the queue's concurrency limit",
|
||||
do_nothing_drains: "or do nothing — backlog drains in ~{value} min once triggers ease",
|
||||
region_failover: "region move? ask your agent — depends on your failover setup",
|
||||
@@ -144,12 +129,16 @@ function findingReason(
|
||||
}
|
||||
|
||||
function statementMessage(findingType: string, severity: Severity, reason?: ReasonCode): string {
|
||||
// Stale telemetry makes a CH-derived verdict untrustworthy — say so, don't show a severity.
|
||||
// Stale telemetry makes the verdict untrustworthy, so it replaces the severity.
|
||||
if (reason === "unknown") {
|
||||
const label = findingType.charAt(0).toUpperCase() + findingType.slice(1);
|
||||
return `${label} unknown — data stale`;
|
||||
}
|
||||
// No freshness signal is NOT "data lagging" (a real severity) — it's genuinely unknown.
|
||||
// A missing depth signal is unknown from a failed measurement, not staleness.
|
||||
if (reason === "flow_unmeasured") {
|
||||
return "Flow unknown — queue depth unavailable";
|
||||
}
|
||||
// No freshness signal is unknown, not lagging.
|
||||
if (findingType === "liveness" && reason === "freshness_unknown") {
|
||||
return "data freshness unknown";
|
||||
}
|
||||
|
||||
@@ -1,15 +1,3 @@
|
||||
/**
|
||||
* The `health` report — ORCHESTRATOR + public entry. Two layers:
|
||||
* `assessHealth()` data -> HealthAssessment (all health reasoning)
|
||||
* `toReportViewModel()` HealthAssessment -> generic ReportViewModel (pure packaging)
|
||||
* `interpret()` is the thin composition of the two.
|
||||
*
|
||||
* The reasoning is split into three independent analyzers, each returning a Finding:
|
||||
* flow.ts · execution.ts · liveness.ts (foundation in health-core.ts)
|
||||
* This module only wires them: build metrics -> run the three -> flow policy -> stale guard
|
||||
* -> reads -> summary/footer. PURE — no IO/clock/LLM/formatting.
|
||||
*/
|
||||
|
||||
import {
|
||||
isOk,
|
||||
maxSeverity,
|
||||
@@ -20,31 +8,28 @@ import {
|
||||
type Severity,
|
||||
type SummaryStatement,
|
||||
} from "../report-view-model";
|
||||
import { buildMetrics, computeDrain, HEALTH_THRESHOLDS, type HealthInput } from "./health-core";
|
||||
import {
|
||||
buildMetrics,
|
||||
computeDrain,
|
||||
HEALTH_THRESHOLDS,
|
||||
isPendingUnknown,
|
||||
type HealthInput,
|
||||
} from "./health-core";
|
||||
import { buildExecutionRead, interpretExecution } from "./execution";
|
||||
import { applyFlowPolicy, buildFlowRead, interpretFlow } from "./flow";
|
||||
import { applyFlowPolicy, buildFlowRead, FLOW_UNMEASURED, interpretFlow } from "./flow";
|
||||
import { interpretLiveness } from "./liveness";
|
||||
// Registers the "health" message catalog (side effect) so the renderer resolves this report's
|
||||
// codes. Kept here — the health report's entry module — so loading it always registers its prose.
|
||||
|
||||
// Re-exported so the data layer + tests keep a single import path (`./health`).
|
||||
export { HEALTH_THRESHOLDS, isPendingIncreasing, type HealthInput } from "./health-core";
|
||||
|
||||
// ---------------------------------------------------------------------------
|
||||
// Stale-telemetry trust guard. When telemetry is genuinely stale, both flow and execution are
|
||||
// CH-derived and untrustworthy: mark them unknown and strip everything that would advise action
|
||||
// off stale data (recommendation, attribution, exclusions, observations, hedge, anomaly window).
|
||||
// ---------------------------------------------------------------------------
|
||||
|
||||
// Flow and execution are both ClickHouse-derived, so stale telemetry makes both unknown and strips
|
||||
// anything that would advise action.
|
||||
function applyStaleGuard(
|
||||
flow: Finding,
|
||||
execution: Finding,
|
||||
telemetryStale: boolean
|
||||
): { flow: Finding; execution: Finding; treatedCrit: boolean } {
|
||||
if (!telemetryStale) return { flow, execution, treatedCrit: false };
|
||||
// Force crit so severity is consistent across summary / section glyph / JSON, and strip the
|
||||
// ACTIONABLE causal fields so no surface advises off stale data. Raw metrics/evidence stay in
|
||||
// the VM for diagnostics, flagged informational-only by `facts.trustworthy: false`.
|
||||
// Force crit so severity is consistent across summary, glyph and JSON.
|
||||
const untrust = (f: Finding): Finding => ({
|
||||
...f,
|
||||
severity: "crit",
|
||||
@@ -59,10 +44,6 @@ function applyStaleGuard(
|
||||
return { flow: untrust(flow), execution: untrust(execution), treatedCrit: true };
|
||||
}
|
||||
|
||||
// ---------------------------------------------------------------------------
|
||||
// Summary.
|
||||
// ---------------------------------------------------------------------------
|
||||
|
||||
function aggregateSummary(
|
||||
flow: Finding,
|
||||
execution: Finding,
|
||||
@@ -75,7 +56,9 @@ function aggregateSummary(
|
||||
{
|
||||
findingType: "flow",
|
||||
severity: flow.severity,
|
||||
reason: flow.reason === "unknown" ? "unknown" : undefined,
|
||||
// Both exceptions mean "we can't say", so the statement renders no severity claim.
|
||||
reason:
|
||||
flow.reason === "unknown" || flow.reason === FLOW_UNMEASURED ? flow.reason : undefined,
|
||||
},
|
||||
{
|
||||
findingType: "execution",
|
||||
@@ -91,10 +74,6 @@ function aggregateSummary(
|
||||
return { severity, statements };
|
||||
}
|
||||
|
||||
// ---------------------------------------------------------------------------
|
||||
// Footer (dominant action + do-nothing option) + links.
|
||||
// ---------------------------------------------------------------------------
|
||||
|
||||
const SEV_RANK: Record<Severity, number> = { ok: 0, warn: 1, crit: 2 };
|
||||
|
||||
function dominantFinding(findings: Finding[]): Finding | undefined {
|
||||
@@ -122,12 +101,14 @@ function buildFooter(
|
||||
const dominant = dominantFinding(findings);
|
||||
if (!dominant?.recommendation) return [{ code: "nothing_to_do" }];
|
||||
|
||||
const footer: FooterEntry[] = [
|
||||
{ code: dominant.recommendation.code, link: dominant.recommendation.link },
|
||||
];
|
||||
const footer: FooterEntry[] =
|
||||
dominant.recommendation.code === "raise_env_limit"
|
||||
? [
|
||||
{ code: "raise_env_limit", link: dominant.recommendation.link },
|
||||
{ code: "concurrency_docs", link: dominant.recommendation.link },
|
||||
]
|
||||
: [{ code: dominant.recommendation.code, link: dominant.recommendation.link }];
|
||||
|
||||
// Second entry: do-nothing when the backlog drains, or the region-move hedge for a
|
||||
// dequeue stall (the one place it stays plausible).
|
||||
if (dominant.type === "flow") {
|
||||
if (drain.isDrainable && Number.isFinite(drain.drainMinutes)) {
|
||||
footer.push({ code: "do_nothing_drains", value: Math.round(drain.drainMinutes * 10) / 10 });
|
||||
@@ -147,35 +128,25 @@ function collectLinks(findings: Finding[]): ReportViewModel["links"] {
|
||||
return [...keys].map((key) => ({ key, label: key, url: "" }));
|
||||
}
|
||||
|
||||
// ---------------------------------------------------------------------------
|
||||
// Domain layer: HealthAssessment — the health verdict (flow / execution / liveness findings,
|
||||
// their causes + recommendations, and the derived state). All health semantics live here; it
|
||||
// knows nothing about how a report is presented. `toReportViewModel` maps it into the generic,
|
||||
// report-agnostic ReportViewModel — so the VM stays reusable for future reports (each has its
|
||||
// own <domain>Assessment + mapper; the renderer/VM primitives are shared).
|
||||
// ---------------------------------------------------------------------------
|
||||
|
||||
// The health verdict. All health semantics live here and no presentation does.
|
||||
export type HealthAssessment = {
|
||||
/** header, carried through from the input. */
|
||||
scope: string;
|
||||
period: string;
|
||||
baselineLabel: string;
|
||||
generatedAt: string;
|
||||
windowMinutes: number;
|
||||
/** finalized findings (post-policy, post-stale-guard, with reads built). */
|
||||
/** Finalized findings: post-policy, post-stale-guard, with reads built. */
|
||||
flow: Finding;
|
||||
execution: Finding;
|
||||
liveness: Finding;
|
||||
metrics: Metric[];
|
||||
/** derived domain state the presentation layer needs (footer / summary / trust). */
|
||||
drain: { drainMinutes: number; isDrainable: boolean };
|
||||
telemetryStale: boolean;
|
||||
executionTreatedCrit: boolean;
|
||||
/** structured payload for agents (already carries the trust marker). */
|
||||
/** Structured payload for agents. Carries `trustworthy`. */
|
||||
facts: Record<string, unknown>;
|
||||
};
|
||||
|
||||
/** Data -> domain verdict. Pure: no IO/clock/LLM/formatting — just health reasoning. */
|
||||
export function assessHealth(input: HealthInput): HealthAssessment {
|
||||
const metrics = buildMetrics(input);
|
||||
const drain = computeDrain(input);
|
||||
@@ -184,23 +155,28 @@ export function assessHealth(input: HealthInput): HealthAssessment {
|
||||
const executionRaw = interpretExecution(metrics, input);
|
||||
const liveness = interpretLiveness(metrics, input);
|
||||
|
||||
// Telemetry freshness as an explicit state so "unknown" (no signal) is never conflated with
|
||||
// "lagging" (a real severity). Only GENUINE staleness trust-guards the CH-derived verdicts.
|
||||
// "none" is not "lagging", and only genuine staleness trust-guards the verdicts. A signal-less env
|
||||
// stays neutral for the reader but reports `trustworthy: false`.
|
||||
const ageMs = input.liveness.telemetryAgeMs;
|
||||
const telemetryStale = ageMs !== null && ageMs > HEALTH_THRESHOLDS.liveness.staleMs;
|
||||
const telemetry: "none" | "fresh" | "lagging" | "stale" =
|
||||
ageMs === null
|
||||
? "none"
|
||||
: ageMs > HEALTH_THRESHOLDS.liveness.staleMs
|
||||
? "stale"
|
||||
: ageMs > HEALTH_THRESHOLDS.liveness.freshMs
|
||||
? "lagging"
|
||||
: "fresh";
|
||||
const telemetryStale = telemetry === "stale";
|
||||
const flowUnmeasured = isPendingUnknown(input);
|
||||
|
||||
flow = applyFlowPolicy(flow, executionRaw, drain.isDrainable, telemetryStale);
|
||||
|
||||
// Stale telemetry: flow AND execution are untrustworthy -> mark both unknown and strip their
|
||||
// actions/attribution/exclusions so nothing advises off stale data.
|
||||
const guarded = applyStaleGuard(flow, executionRaw, telemetryStale);
|
||||
flow = guarded.flow;
|
||||
const execution = guarded.execution;
|
||||
|
||||
// interpretFlow mutates the shared metrics array (e.g. sets concurrency.annotation "pinned 40
|
||||
// of last 60 min") BEFORE the guard runs. The renderers hide it for an unknown finding, but
|
||||
// format=json would still leak that stale-derived narrative — so strip metric annotations too
|
||||
// (the twin of the stripped anomaly window). Raw values stay, flagged by facts.trustworthy.
|
||||
// interpretFlow annotates the shared metrics array before the guard runs, and format=json would
|
||||
// leak that stale-derived narrative.
|
||||
if (telemetryStale) {
|
||||
for (const m of metrics) m.annotation = undefined;
|
||||
}
|
||||
@@ -223,12 +199,16 @@ export function assessHealth(input: HealthInput): HealthAssessment {
|
||||
telemetryStale,
|
||||
executionTreatedCrit: guarded.treatedCrit,
|
||||
facts: {
|
||||
// Trust marker for structured consumers. The metrics/evidence stay (useful for pipeline
|
||||
// diagnostics), but when telemetry is stale they're informational-only: an agent must not
|
||||
// act on them (e.g. raise concurrency off a stale backlog). The human renderer is already
|
||||
// guarded via the "unknown" finding; this is the same guarantee for JSON.
|
||||
trustworthy: !telemetryStale,
|
||||
staleReason: telemetryStale ? "telemetry_stale" : undefined,
|
||||
// Metrics are informational unless this is true. Stale, absent and unmeasurable all read false.
|
||||
trustworthy: !telemetryStale && telemetry !== "none" && !flowUnmeasured,
|
||||
telemetry,
|
||||
untrustworthyReason: telemetryStale
|
||||
? "telemetry_stale"
|
||||
: telemetry === "none"
|
||||
? "telemetry_absent"
|
||||
: flowUnmeasured
|
||||
? "flow_unmeasured"
|
||||
: undefined,
|
||||
flowSource: input.flowSource,
|
||||
pendingEstimated: input.pending.estimated,
|
||||
throughput: input.throughput,
|
||||
@@ -237,11 +217,6 @@ export function assessHealth(input: HealthInput): HealthAssessment {
|
||||
};
|
||||
}
|
||||
|
||||
// ---------------------------------------------------------------------------
|
||||
// Presentation mapping: HealthAssessment -> generic ReportViewModel. Pure packaging only —
|
||||
// no health reasoning here (summary/footer/links are derived from the findings).
|
||||
// ---------------------------------------------------------------------------
|
||||
|
||||
function toReportViewModel(a: HealthAssessment): ReportViewModel {
|
||||
const findings = [a.flow, a.execution, a.liveness];
|
||||
return {
|
||||
@@ -256,12 +231,10 @@ function toReportViewModel(a: HealthAssessment): ReportViewModel {
|
||||
metrics: a.metrics,
|
||||
facts: a.facts,
|
||||
links: collectLinks(findings),
|
||||
// Stale telemetry -> footer points at the control plane, not a CH-derived action.
|
||||
footer: buildFooter(findings, a.drain, a.telemetryStale),
|
||||
};
|
||||
}
|
||||
|
||||
/** Public entry: data -> generic report. Thin composition of the domain + presentation layers. */
|
||||
export function interpret(input: HealthInput): ReportViewModel {
|
||||
return toReportViewModel(assessHealth(input));
|
||||
}
|
||||
|
||||
@@ -1,7 +1,4 @@
|
||||
/**
|
||||
* LIVENESS analyzer: telemetry FRESHNESS (age of the freshest signal), NOT "last completion".
|
||||
* A null age is genuinely unknown (neutral), never a warning; only real staleness is crit.
|
||||
*/
|
||||
/** Liveness analyzer: freshest-telemetry age. A null age is unknown, never a warning. */
|
||||
|
||||
import { type Finding, type Metric } from "../report-view-model";
|
||||
import { metricById, type HealthInput } from "./health-core";
|
||||
|
||||
@@ -1,77 +1,62 @@
|
||||
/**
|
||||
* GENERIC renderer: ReportViewModel -> monospace markdown. Severity-driven disclosure:
|
||||
* a degraded finding expands into evidence in causal order; a healthy finding collapses
|
||||
* to one `✓` line. Owns ALL presentation (formatting, glyphs, sparklines, spacing, {token}
|
||||
* substitution) and resolves the VM's codes -> strings via the report's registered message
|
||||
* catalog, looked up by `vm.title` — so it holds NO report vocabulary itself.
|
||||
* The report's text surfaces: monospace markdown and ANSI.
|
||||
*
|
||||
* Knows NOTHING about health — walks summary -> findings -> metrics generically.
|
||||
* Structure, labels, wording and glyphs come from `report-layout.ts`, which the React card also
|
||||
* consumes, so the surfaces can't drift. This module owns typography only: column alignment, the
|
||||
* sparkline column, indentation, and (for ANSI) colour.
|
||||
*
|
||||
* Nothing here relies on colour to carry meaning: every verdict, direction and caveat is a glyph,
|
||||
* because an MCP host renders plain text.
|
||||
*/
|
||||
|
||||
import { reportMessages, type ReportMessages } from "./report-messages";
|
||||
import {
|
||||
type Finding,
|
||||
type FooterEntry,
|
||||
type Metric,
|
||||
type ReportViewModel,
|
||||
type Severity,
|
||||
type Unit,
|
||||
} from "./report-view-model";
|
||||
buildReportLayout,
|
||||
REPORT_GLYPH,
|
||||
REPORT_LABELS,
|
||||
type LayoutFinding,
|
||||
type LayoutMetricRow,
|
||||
type LayoutViewModel,
|
||||
type ReportLayout,
|
||||
} from "./report-layout";
|
||||
import { reportMessages } from "./report-messages";
|
||||
import { type ReportViewModel } from "./report-view-model";
|
||||
|
||||
const BARS = "▁▂▃▄▅▆▇█";
|
||||
const MINUS = "−"; // U+2212
|
||||
|
||||
const SEVERITY_GLYPH: Record<Severity, string> = { ok: "✓", warn: "⚠", crit: "✕" };
|
||||
|
||||
/**
|
||||
* A NEUTRAL marker for a state that's genuinely unknown, not good/bad — e.g. liveness with no
|
||||
* telemetry signal. It doesn't affect the aggregate severity (that stays driven by real findings),
|
||||
* but it must not read as a confident green "✓", so it gets its own glyph.
|
||||
*/
|
||||
const NEUTRAL_GLYPH = "○";
|
||||
|
||||
/**
|
||||
* Markdown-only status colour. Chat hosts render neither ANSI nor HTML, so swapping
|
||||
* the glyphs for traffic-light circles is the one colour cue they get — one emoji per
|
||||
* marker (neutral -> white). ANSI keeps the crisp ✓/⚠/✕/○, so this applies ONLY on markdown.
|
||||
* Markdown-only status colour. Chat hosts render neither ANSI nor HTML, so swapping the glyphs for
|
||||
* traffic-light circles is the one colour cue they get. The glyph and the emoji mean the same
|
||||
* thing, so meaning never depends on which one a host shows.
|
||||
*/
|
||||
const MARKDOWN_STATUS_EMOJI: Record<string, string> = {
|
||||
"✓": "🟢",
|
||||
"⚠": "🟡",
|
||||
"✕": "🔴",
|
||||
"○": "⚪",
|
||||
[REPORT_GLYPH.ok]: "🟢",
|
||||
[REPORT_GLYPH.warn]: "🟡",
|
||||
[REPORT_GLYPH.crit]: "🔴",
|
||||
[REPORT_GLYPH.neutral]: "⚪",
|
||||
[REPORT_GLYPH.untrusted]: "🚩",
|
||||
};
|
||||
|
||||
const STATUS_GLYPHS = new RegExp(`[${Object.keys(MARKDOWN_STATUS_EMOJI).join("")}]`, "g");
|
||||
|
||||
function toMarkdownEmoji(text: string): string {
|
||||
return text.replace(/[✓⚠✕○]/g, (g) => MARKDOWN_STATUS_EMOJI[g] ?? g);
|
||||
return text.replace(STATUS_GLYPHS, (glyph) => MARKDOWN_STATUS_EMOJI[glyph] ?? glyph);
|
||||
}
|
||||
|
||||
/** Glyph for a finding/statement: neutral for a genuinely-unknown freshness, else severity-driven. */
|
||||
function statusGlyph(severity: Severity, reason?: string): string {
|
||||
return reason === "freshness_unknown" ? NEUTRAL_GLYPH : SEVERITY_GLYPH[severity];
|
||||
}
|
||||
|
||||
/** Evidence lines (metric rows + attribution) shown for a degraded section. */
|
||||
/** Metric rows shown for a finding's evidence block. */
|
||||
const EVIDENCE_CAP = 4;
|
||||
|
||||
/** Column where the header's scope·period·baseline starts (mirrors the mockup). */
|
||||
/** Column where the header's scope, period and baseline start. */
|
||||
const HEADER_COL = 22;
|
||||
|
||||
/** Gap between aligned columns in an evidence block. */
|
||||
/** Gap between aligned columns. */
|
||||
const COL_GAP = 3;
|
||||
|
||||
/** Section labels pad to this so their glyph/content aligns vertically. */
|
||||
/** Section labels pad to this so every finding's text starts at the same column. */
|
||||
const SECTION_LABEL_WIDTH = 9; // "EXECUTION"
|
||||
const SECTION_GAP = 3;
|
||||
|
||||
/** Section label padded so every section's ✓ / cause text starts at the same column. */
|
||||
function sectionLabel(type: string): string {
|
||||
return `${type.toUpperCase().padEnd(SECTION_LABEL_WIDTH)}${" ".repeat(SECTION_GAP)}`;
|
||||
}
|
||||
|
||||
// ---------------------------------------------------------------------------
|
||||
// Formatters.
|
||||
// ---------------------------------------------------------------------------
|
||||
/** `read:` / `why:` labels sit in their own column so their lines hang as one paragraph. */
|
||||
const NOTE_LABEL_WIDTH = Math.max(REPORT_LABELS.read.length, REPORT_LABELS.why.length);
|
||||
|
||||
/** All sparklines render at this fixed width so they align in a column. */
|
||||
const SPARK_WIDTH = 8;
|
||||
@@ -105,361 +90,142 @@ export function sparklineFromSeries(points: number[]): string {
|
||||
return p.map((v) => BARS[Math.round(((v - min) / (max - min)) * (BARS.length - 1))]).join("");
|
||||
}
|
||||
|
||||
function fmtCount(n: number): string {
|
||||
return Math.round(n).toLocaleString("en-US");
|
||||
/** A metric's trailing aside. An annotation speaks for itself; the rest is context, so it brackets. */
|
||||
function trailingNote(row: LayoutMetricRow): string {
|
||||
if (!row.note) return "";
|
||||
return row.note.kind === "annotation" ? row.note.text : `(${row.note.text})`;
|
||||
}
|
||||
|
||||
function fmtDuration(ms: number): string {
|
||||
if (ms < 1000) return `${Math.round(ms)}ms`;
|
||||
const s = ms / 1000;
|
||||
if (s < 60) return Number.isInteger(s) ? `${s}s` : `${s.toFixed(1)}s`;
|
||||
const m = s / 60;
|
||||
return Number.isInteger(m) ? `${m}m` : `${m.toFixed(1)}m`;
|
||||
}
|
||||
|
||||
function fmtPct(ratio: number): string {
|
||||
return `${(ratio * 100).toFixed(1)}%`;
|
||||
}
|
||||
|
||||
function fmtRate(n: number): string {
|
||||
return `${fmtCount(n)}/min`;
|
||||
}
|
||||
|
||||
function fmtSignedRate(net: number): string {
|
||||
const sign = net < 0 ? MINUS : net > 0 ? "+" : "";
|
||||
return `${sign}${fmtCount(Math.abs(net))}/min`;
|
||||
}
|
||||
|
||||
function fmtValue(value: number, unit: Unit): string {
|
||||
switch (unit) {
|
||||
case "ms":
|
||||
return fmtDuration(value);
|
||||
case "count":
|
||||
return fmtCount(value);
|
||||
case "ratio":
|
||||
return fmtPct(value);
|
||||
case "perMin":
|
||||
return fmtRate(value);
|
||||
}
|
||||
}
|
||||
|
||||
function fill(template: string, tokens: Record<string, string | number | undefined>): string {
|
||||
return template.replace(/\{(\w+)\}/g, (_, k) => {
|
||||
const v = tokens[k];
|
||||
return v === undefined ? `{${k}}` : String(v);
|
||||
});
|
||||
}
|
||||
|
||||
function metricById(metrics: Metric[], id: string): Metric | undefined {
|
||||
return metrics.find((m) => m.id === id);
|
||||
}
|
||||
|
||||
// ---------------------------------------------------------------------------
|
||||
// Metric line (expanded evidence).
|
||||
// ---------------------------------------------------------------------------
|
||||
|
||||
function deltaSegment(metric: Metric): string {
|
||||
if (metric.severity === "ok" || !metric.delta || metric.delta.dir === "flat") return "";
|
||||
const arrow = metric.delta.dir === "up" ? "↑" : "↓";
|
||||
// "up" shows the multiplier when we have it ("↑ 16×"). "down" is arrow-only: a
|
||||
// drop rounds to 0×/1×, meaningless — the arrow already says "below normal".
|
||||
if (metric.delta.dir === "down") return arrow;
|
||||
return metric.delta.mult === undefined ? arrow : `${arrow} ${metric.delta.mult}×`;
|
||||
}
|
||||
|
||||
function annotationSegment(metric: Metric, vm: ReportViewModel): string {
|
||||
if (!metric.annotation) return "";
|
||||
const value = String(metric.annotation.value ?? "");
|
||||
return fill(reportMessages(vm.title).annotationMessage(metric.annotation.code), {
|
||||
value,
|
||||
window: vm.windowMinutes,
|
||||
limit: metric.breakdown?.limit ?? "",
|
||||
});
|
||||
}
|
||||
|
||||
function metricValueText(metric: Metric, msg: ReportMessages): string {
|
||||
// No measurement -> say so; `value` is a placeholder, not a real reading.
|
||||
if (metric.availability === "unknown") return "unknown";
|
||||
// concurrency etc. carry a limit -> "running/limit".
|
||||
if (metric.unit === "count" && metric.breakdown?.limit !== undefined) {
|
||||
return `${fmtCount(metric.value)}/${fmtCount(metric.breakdown.limit)}`;
|
||||
}
|
||||
// composite throughput -> "done vs triggered -> net".
|
||||
if (metric.unit === "perMin" && metric.breakdown?.done !== undefined) {
|
||||
const { done, triggered } = metric.breakdown;
|
||||
return `${fmtRate(done)} done vs ${fmtRate(triggered)} triggered → net ${fmtSignedRate(metric.value)}`;
|
||||
}
|
||||
const showAgg =
|
||||
metric.aggregation === "p95" && !msg.metricLabel(metric.id).toLowerCase().startsWith("p95");
|
||||
return showAgg
|
||||
? `p95 ${fmtValue(metric.value, metric.unit)}`
|
||||
: fmtValue(metric.value, metric.unit);
|
||||
}
|
||||
|
||||
/** Column widths for a section's evidence block, so value/delta/spark align down the page. */
|
||||
/** Column widths for an evidence block, so value, delta and sparkline align down the page. */
|
||||
type Cols = { label: number; value: number; delta: number };
|
||||
|
||||
function isComposite(metric: Metric): boolean {
|
||||
return metric.unit === "perMin" && metric.breakdown?.done !== undefined;
|
||||
}
|
||||
|
||||
function computeColumns(rows: Metric[], vm: ReportViewModel, extraLabels: string[]): Cols {
|
||||
const msg = reportMessages(vm.title);
|
||||
const labelLens = [
|
||||
...rows.map((m) => msg.metricLabel(m.id).length),
|
||||
...extraLabels.map((l) => l.length),
|
||||
];
|
||||
const valueLens = rows.filter((m) => !isComposite(m)).map((m) => metricValueText(m, msg).length);
|
||||
const deltaLens = rows
|
||||
.filter((m) => !isComposite(m) && !annotationSegment(m, vm))
|
||||
.map((m) => deltaSegment(m).length);
|
||||
function computeColumns(rows: LayoutMetricRow[]): Cols {
|
||||
const labels = rows.flatMap((row) => [
|
||||
row.label.length,
|
||||
...row.subRows.map((sub) => sub.label.length + SUB_ROW_INDENT),
|
||||
]);
|
||||
const values = rows.flatMap((row) => [
|
||||
row.value.length,
|
||||
...row.subRows.map((s) => s.value.length),
|
||||
]);
|
||||
return {
|
||||
label: Math.max(0, ...labelLens),
|
||||
value: Math.max(0, ...valueLens),
|
||||
delta: Math.max(0, ...deltaLens),
|
||||
label: Math.max(0, ...labels),
|
||||
value: Math.max(0, ...values),
|
||||
delta: Math.max(0, ...rows.map((row) => (row.delta?.text ?? "").length)),
|
||||
};
|
||||
}
|
||||
|
||||
function renderMetricRow(metric: Metric, cols: Cols, vm: ReportViewModel): string {
|
||||
const msg = reportMessages(vm.title);
|
||||
/** A composite metric's parts sit under it, still on the shared value column. */
|
||||
const SUB_ROW_INDENT = 2;
|
||||
|
||||
function renderMetricRow(row: LayoutMetricRow, cols: Cols, indent: string): string[] {
|
||||
const gap = " ".repeat(COL_GAP);
|
||||
const label = msg.metricLabel(metric.id).padEnd(cols.label);
|
||||
const value = metricValueText(metric, msg);
|
||||
const spark = row.series ? sparklineFromSeries(row.series) : "";
|
||||
const note = trailingNote(row);
|
||||
|
||||
// composite throughput: label + value only (its own grammar).
|
||||
if (isComposite(metric)) return ` ${label}${gap}${value}`;
|
||||
// The fixed-width spark column keeps every sparkline and every trailing note aligned.
|
||||
let line = `${indent}${row.label.padEnd(cols.label)}${gap}${row.value.padEnd(cols.value)}`;
|
||||
if (cols.delta > 0) line += `${gap}${(row.delta?.text ?? "").padEnd(cols.delta)}`;
|
||||
line += `${gap}${spark.padEnd(SPARK_WIDTH)}`;
|
||||
if (note) line += `${gap}${note}`;
|
||||
|
||||
const spark =
|
||||
metric.series && metric.series.points.length > 0
|
||||
? sparklineFromSeries(metric.series.points)
|
||||
: "";
|
||||
const annotation = annotationSegment(metric, vm);
|
||||
const delta = annotation ? "" : deltaSegment(metric); // cause line carries an annotation, not a delta
|
||||
const trailing = annotation
|
||||
? annotation
|
||||
: metric.normal !== undefined
|
||||
? `(normal ~${fmtValue(metric.normal, metric.unit)})`
|
||||
: metric.series?.kind === "estimated"
|
||||
? "(estimated)" // proxy trend (e.g. snapshot backlog) — flag it so it isn't read as measured
|
||||
: "";
|
||||
|
||||
// Fixed columns: label · value · delta · SPARK · trailing. The fixed-width spark
|
||||
// column keeps every spark and the trailing (normal / annotation) aligned.
|
||||
let line = ` ${label}${gap}${value.padEnd(cols.value)}`;
|
||||
if (cols.delta > 0) line += `${gap}${delta.padEnd(cols.delta)}`;
|
||||
line += `${gap}${(spark || "").padEnd(SPARK_WIDTH)}`;
|
||||
if (trailing) line += `${gap}${trailing}`;
|
||||
return line.replace(/\s+$/, "");
|
||||
}
|
||||
|
||||
// ---------------------------------------------------------------------------
|
||||
// Compact facts (collapsed / semi-expanded healthy sections).
|
||||
// ---------------------------------------------------------------------------
|
||||
|
||||
function compactFact(metric: Metric): string | undefined {
|
||||
switch (metric.id) {
|
||||
case "pending":
|
||||
return `pending ${fmtCount(metric.value)}${metric.normal !== undefined ? ` (normal ~${fmtCount(metric.normal)})` : ""}`;
|
||||
case "start_latency_p95":
|
||||
return metric.availability === "unknown"
|
||||
? "starts p95 unknown"
|
||||
: `starts p95 ${fmtDuration(metric.value)}`;
|
||||
case "failures":
|
||||
return `failures ${fmtPct(metric.value)}${metric.normal !== undefined ? ` (normal ~${fmtPct(metric.normal)})` : ""}`;
|
||||
case "dur_p95":
|
||||
return metric.severity === "ok"
|
||||
? "durations normal"
|
||||
: `durations p95 ${fmtDuration(metric.value)}`;
|
||||
default:
|
||||
return undefined; // throughput / evidence metrics get no collapsed fact
|
||||
}
|
||||
}
|
||||
|
||||
/** Reassuring facts read consequence-first: depth/failures before latency/duration. */
|
||||
const COMPACT_ORDER = ["pending", "failures", "start_latency_p95", "dur_p95"];
|
||||
|
||||
function compactFacts(finding: Finding, metrics: Metric[]): string {
|
||||
return finding.metricIds
|
||||
.map((id) => metricById(metrics, id))
|
||||
.filter((m): m is Metric => m !== undefined)
|
||||
.slice()
|
||||
.sort((a, b) => COMPACT_ORDER.indexOf(a.id) - COMPACT_ORDER.indexOf(b.id))
|
||||
.map(compactFact)
|
||||
.filter((s): s is string => s !== undefined)
|
||||
.join(" · ");
|
||||
}
|
||||
|
||||
// ---------------------------------------------------------------------------
|
||||
// Read / exclusion / attribution / window.
|
||||
// ---------------------------------------------------------------------------
|
||||
|
||||
function readTokens(_finding: Finding, metrics: Metric[]): Record<string, string | number> {
|
||||
const triggered = metricById(metrics, "triggered");
|
||||
return {
|
||||
mult: triggered?.delta?.mult ?? "",
|
||||
};
|
||||
}
|
||||
|
||||
function windowSuffix(finding: Finding): string {
|
||||
const aw = finding.anomalyWindow;
|
||||
if (!aw) return "";
|
||||
return aw.touchesEnd ? ` (last ${aw.minutes} min)` : ` (${aw.minutes} min window)`;
|
||||
}
|
||||
|
||||
function attributionLine(finding: Finding, cols: Cols): string | undefined {
|
||||
const a = finding.attribution;
|
||||
if (!a) return undefined;
|
||||
const label = `worst ${a.dim}`.padEnd(cols.label);
|
||||
return ` ${label}${" ".repeat(COL_GAP)}${a.key} — ${Math.round(a.share * 100)}% of ${a.of}`;
|
||||
}
|
||||
|
||||
// ---------------------------------------------------------------------------
|
||||
// Sections.
|
||||
// ---------------------------------------------------------------------------
|
||||
|
||||
function renderDegradedSection(finding: Finding, vm: ReportViewModel): string[] {
|
||||
const msg = reportMessages(vm.title);
|
||||
const rows = finding.metricIds
|
||||
.map((id) => metricById(vm.metrics, id))
|
||||
.filter((m): m is Metric => m !== undefined);
|
||||
|
||||
const extraLabels = finding.attribution ? [`worst ${finding.attribution.dim}`] : [];
|
||||
const cols = computeColumns(rows, vm, extraLabels);
|
||||
|
||||
const evidence: string[] = rows.map((m) => renderMetricRow(m, cols, vm));
|
||||
const attr = attributionLine(finding, cols);
|
||||
if (attr) evidence.push(attr);
|
||||
|
||||
// One blank line between evidence rows so the block breathes.
|
||||
const spaced = evidence.slice(0, EVIDENCE_CAP).flatMap((l, i) => (i === 0 ? [l] : ["", l]));
|
||||
|
||||
const lines = [
|
||||
// Lead with the glyph so the cause text lines up with the healthy sections' "✓ …".
|
||||
`${sectionLabel(finding.type)}${SEVERITY_GLYPH[finding.severity]} ${msg.findingReason(finding.type, finding.reason)}${windowSuffix(finding)}`,
|
||||
"", // blank line between the header and its evidence (matches the section/footer spacing)
|
||||
...spaced,
|
||||
return [
|
||||
line.replace(/\s+$/, ""),
|
||||
...row.subRows.map((sub) => {
|
||||
const label = `${" ".repeat(SUB_ROW_INDENT)}${sub.label}`.padEnd(cols.label);
|
||||
return `${indent}${label}${gap}${sub.value}`;
|
||||
}),
|
||||
];
|
||||
}
|
||||
|
||||
if (finding.read) {
|
||||
lines.push(
|
||||
"",
|
||||
` read: ${fill(msg.readMessage(finding.read), readTokens(finding, vm.metrics))}`
|
||||
);
|
||||
// Exclusions ("not your code") first, then supporting observations ("runs completing at ~X/min").
|
||||
for (const excl of finding.exclusions ?? []) {
|
||||
lines.push(
|
||||
` ${fill(msg.exclusionMessage(excl.code), { rate: fmtCount(excl.evidence?.donePerMin ?? 0) })}`
|
||||
/** A labelled block whose lines hang under the label's column. */
|
||||
function renderNoteBlock(label: string, lines: string[], indent: string): string[] {
|
||||
if (lines.length === 0) return [];
|
||||
const hang = " ".repeat(indent.length + NOTE_LABEL_WIDTH + 1);
|
||||
return lines.map((line, i) =>
|
||||
i === 0 ? `${indent}${label.padEnd(NOTE_LABEL_WIDTH)} ${line}` : `${hang}${line}`
|
||||
);
|
||||
}
|
||||
for (const obs of finding.observations ?? []) {
|
||||
lines.push(
|
||||
` ${fill(msg.observationMessage(obs.code), { rate: fmtCount(obs.evidence?.donePerMin ?? 0) })}`
|
||||
);
|
||||
}
|
||||
}
|
||||
return lines;
|
||||
}
|
||||
|
||||
function renderHealthyExecutionExpanded(finding: Finding, vm: ReportViewModel): string[] {
|
||||
const msg = reportMessages(vm.title);
|
||||
const lines = [
|
||||
`${sectionLabel(finding.type)}${SEVERITY_GLYPH.ok} ${msg.findingReason(finding.type, finding.reason, { expanded: true })}`,
|
||||
"", // blank line between the header and its facts (matches the section/footer spacing)
|
||||
` ${compactFacts(finding, vm.metrics)}`,
|
||||
];
|
||||
if (finding.read) lines.push(` read: ${msg.readMessage(finding.read)}`);
|
||||
return lines;
|
||||
}
|
||||
|
||||
function renderCollapsedSection(finding: Finding, vm: ReportViewModel): string[] {
|
||||
const msg = reportMessages(vm.title);
|
||||
const headline = `${sectionLabel(finding.type)}${SEVERITY_GLYPH.ok} ${msg.findingReason(finding.type, finding.reason)}`;
|
||||
const facts = compactFacts(finding, vm.metrics);
|
||||
// Healthy sections stay on one line; the facts are short.
|
||||
return facts ? [`${headline} — ${facts}`] : [headline];
|
||||
}
|
||||
|
||||
function renderLivenessLine(finding: Finding, metrics: Metric[], msg: ReportMessages): string {
|
||||
const metric = metricById(metrics, finding.metricIds[0]);
|
||||
const ageMs = metric?.value;
|
||||
const age = ageMs !== undefined && Number.isFinite(ageMs) ? fmtDuration(ageMs) : "unknown";
|
||||
const reason = msg.findingReason(finding.type, finding.reason).replace("{age}", age);
|
||||
return `${sectionLabel(finding.type)}${statusGlyph(finding.severity, finding.reason)} ${reason}`;
|
||||
}
|
||||
|
||||
function renderFooter(footer: FooterEntry[], msg: ReportMessages): string[] {
|
||||
return footer.map((entry, i) => {
|
||||
const text = fill(msg.actionMessage(entry.code), { value: entry.value, min: entry.value });
|
||||
return i === 0 ? `→ ${text}` : ` ${text}`;
|
||||
/** A finding's evidence: its metric rows, then `why:`. */
|
||||
function renderFindingBody(finding: LayoutFinding, indent: string): string[] {
|
||||
const rows = finding.metrics.slice(0, EVIDENCE_CAP);
|
||||
const cols = computeColumns(rows);
|
||||
const metricLines = rows.flatMap((row, i) => {
|
||||
const rendered = renderMetricRow(row, cols, indent);
|
||||
// One blank line between rows so the block breathes.
|
||||
return i === 0 ? rendered : ["", ...rendered];
|
||||
});
|
||||
const why = renderNoteBlock(REPORT_LABELS.why, finding.why, indent);
|
||||
return why.length > 0 ? [...metricLines, "", ...why] : metricLines;
|
||||
}
|
||||
|
||||
// ---------------------------------------------------------------------------
|
||||
// Top-level render.
|
||||
// ---------------------------------------------------------------------------
|
||||
/** Glyph, then the padded section label, so every finding's text starts on one column. */
|
||||
function findingLine(finding: LayoutFinding): string {
|
||||
return `${finding.glyph} ${finding.label.padEnd(SECTION_LABEL_WIDTH)}${" ".repeat(SECTION_GAP)}${finding.text}`;
|
||||
}
|
||||
|
||||
const BODY_INDENT = " ";
|
||||
/** A non-hero finding's body hangs under its section label rather than the page margin. */
|
||||
const NESTED_INDENT = " ";
|
||||
|
||||
/**
|
||||
* The plain monochrome layout (✓/⚠/✕) shared by every surface. Colour renderers paint
|
||||
* THIS via `paintReport`; markdown swaps the glyphs for emoji. Internal so the three
|
||||
* public renderers can't drift.
|
||||
* The plain monochrome layout every text surface shares. ANSI paints this via `paintReport` and
|
||||
* markdown swaps the glyphs for emoji, so the two can't drift from each other or from the card.
|
||||
*/
|
||||
function renderReportPlain(vm: ReportViewModel): string {
|
||||
const msg = reportMessages(vm.title);
|
||||
function renderReportPlain(vm: LayoutViewModel): string {
|
||||
const layout: ReportLayout = buildReportLayout(vm, reportMessages(vm.title));
|
||||
const lines: string[] = [];
|
||||
|
||||
// header: "/report <title>" padded to a column, then scope · period · baseline.
|
||||
const left = `/report ${vm.title}`;
|
||||
const right = [vm.scope, vm.period, vm.baselineLabel].filter(Boolean).join(" · ");
|
||||
lines.push(`${left.padEnd(HEADER_COL)}${right}`, "");
|
||||
// header: the report as its command (plus a trust flag), then scope · period · baseline.
|
||||
const command = `/report ${layout.header.name}`;
|
||||
const left = layout.trust
|
||||
? `${command} ${REPORT_GLYPH.untrusted} ${layout.trust.badge}`
|
||||
: command;
|
||||
// Two spaces minimum, so a long left side still reads as two columns.
|
||||
const gutter = " ".repeat(Math.max(2, HEADER_COL - left.length));
|
||||
lines.push(`${left}${gutter}${layout.header.meta}`.replace(/\s+$/, ""), "");
|
||||
|
||||
// Each statement carries its OWN glyph — one leading glyph would read as if it
|
||||
// applied only to the first statement (e.g. "✕ Flow healthy"). Per-statement is clear.
|
||||
const verdict = vm.summary.statements
|
||||
.map(
|
||||
(s) =>
|
||||
`${statusGlyph(s.severity, s.reason)} ${msg.statementMessage(s.findingType, s.severity, s.reason)}`
|
||||
)
|
||||
.join(" · ");
|
||||
lines.push(verdict, "");
|
||||
const { glyph, phrase, text } = layout.headline;
|
||||
lines.push(`${glyph} ${phrase}${text ? ` — ${text}` : ""}`, "");
|
||||
|
||||
const flowDegraded = vm.findings.some((f) => f.type === "flow" && f.severity !== "ok");
|
||||
if (layout.trust) lines.push(`${REPORT_GLYPH.untrusted} ${layout.trust.note}`, "");
|
||||
|
||||
for (const finding of vm.findings) {
|
||||
if (finding.type === "liveness") {
|
||||
lines.push(renderLivenessLine(finding, vm.metrics, msg), "");
|
||||
} else if (finding.reason === "unknown") {
|
||||
// stale-data guard: no ✓ or facts computed from a silent feed. The guard forces crit,
|
||||
// so the glyph reads from severity (consistent with the summary + JSON).
|
||||
if (layout.hero) {
|
||||
const body = renderFindingBody(layout.hero, BODY_INDENT);
|
||||
if (body.length > 0) lines.push(...body, "");
|
||||
}
|
||||
|
||||
for (const finding of layout.findings) {
|
||||
lines.push(findingLine(finding));
|
||||
if (finding.expanded) {
|
||||
const body = renderFindingBody(finding, NESTED_INDENT);
|
||||
if (body.length > 0) lines.push("", ...body);
|
||||
}
|
||||
lines.push("");
|
||||
}
|
||||
|
||||
for (const statement of layout.statements) {
|
||||
lines.push(`${statement.glyph} ${statement.text}`, "");
|
||||
}
|
||||
|
||||
const reads = renderNoteBlock(REPORT_LABELS.read, layout.reads, BODY_INDENT);
|
||||
if (reads.length > 0) lines.push(...reads, "");
|
||||
|
||||
// footer: the first entry leads with an arrow, the rest align under it.
|
||||
lines.push(
|
||||
`${sectionLabel(finding.type)}${SEVERITY_GLYPH[finding.severity]} ${msg.findingReason(finding.type, finding.reason)}`,
|
||||
""
|
||||
...layout.footer.map((entry, i) => (i === 0 ? `→ ${entry.label}` : ` ${entry.label}`))
|
||||
);
|
||||
} else if (finding.severity !== "ok") {
|
||||
lines.push(...renderDegradedSection(finding, vm), "");
|
||||
} else if (finding.type === "execution" && flowDegraded) {
|
||||
lines.push(...renderHealthyExecutionExpanded(finding, vm), "");
|
||||
} else {
|
||||
lines.push(...renderCollapsedSection(finding, vm), "");
|
||||
}
|
||||
}
|
||||
|
||||
lines.push(...renderFooter(vm.footer, msg));
|
||||
|
||||
return lines.join("\n").replace(/\n+$/, "\n");
|
||||
}
|
||||
|
||||
/**
|
||||
* Markdown surface (agents / chat). The plain layout with severity glyphs swapped for
|
||||
* status emoji — the only decoration markdown gets.
|
||||
*/
|
||||
/** Markdown surface: the plain layout with the status glyphs swapped for status emoji. */
|
||||
export function renderReportMarkdown(vm: ReportViewModel): string {
|
||||
return toMarkdownEmoji(renderReportPlain(vm));
|
||||
}
|
||||
|
||||
// ---------------------------------------------------------------------------
|
||||
// ANSI colour renderer (terminal, e.g. `trigger report`). Colourises the SAME
|
||||
// plain layout as a post-pass via `paintReport`, so terminal output can't drift.
|
||||
// ---------------------------------------------------------------------------
|
||||
// The ANSI renderer colourises the same plain layout as a post-pass, so terminal output can't drift.
|
||||
|
||||
const SPARK_LOW = "▁▂▃▄▅";
|
||||
const SPARK_HIGH = "▆▇█";
|
||||
@@ -481,31 +247,31 @@ function paintSpark(run: string, p: Paint): string {
|
||||
.join("");
|
||||
}
|
||||
|
||||
/** Colourise the plain report by role. Detection reads the RAW line; wrapping the escaped line. */
|
||||
/** Colourise the plain report by role. Detection reads the raw line, wrapping the escaped line. */
|
||||
function paintReport(text: string, p: Paint): string {
|
||||
const noteLabels = `(?:${REPORT_LABELS.read}|${REPORT_LABELS.why})`;
|
||||
return text
|
||||
.split("\n")
|
||||
.map((raw, i) => {
|
||||
const line = p.escape(raw);
|
||||
// header: grey the right column (scope · period · baseline).
|
||||
if (i === 0) {
|
||||
return line.replace(/^(\/report \S+\s{2,})(.+)$/, (_, a, b) => a + p.grey(b));
|
||||
// whole-line secondary: a read:/why: block and its hanging lines.
|
||||
if (new RegExp(`^\\s*${noteLabels}`).test(raw) || /^\s{6,}\S/.test(raw)) {
|
||||
return p.grey(p.escape(raw));
|
||||
}
|
||||
// whole-line secondary: read: chain, its exclusion lines, do-nothing footer.
|
||||
if (/^\s*read:/.test(raw) || /^\s{6,}\S/.test(raw)) return p.grey(line);
|
||||
if (/^\s{2}(or do nothing|open |Check status)/.test(raw)) return p.grey(line);
|
||||
|
||||
let l = line;
|
||||
// header: grey the right column (scope · period · baseline), then paint its glyphs below.
|
||||
let l = p.escape(raw);
|
||||
if (i === 0) l = l.replace(/^(\/report .*\s{2,})(\S.*)$/, (_, a, b) => a + p.grey(b));
|
||||
l = l.replace(/[▁▂▃▄▅▆▇█]+/g, (m) => paintSpark(m, p)); // two-tone sparkline
|
||||
l = l
|
||||
.replace(/✓/g, p.green("✓"))
|
||||
.replace(/⚠/g, p.amber("⚠"))
|
||||
.replace(/✕/g, p.red("✕"))
|
||||
.replace(/○/g, p.grey("○")); // neutral (unknown) marker
|
||||
l = l.replace(/↑ ?\d+×|↑/g, (m) => p.amber(m)).replace(/↓ ?\d+×|↓/g, (m) => p.green(m));
|
||||
l = l.replace(/\(last \d+ min\)|\(\d+ min window\)/g, (m) => p.amber(m)); // anomaly window
|
||||
l = l.replace(/\(normal ~[^)]*\)|\(estimated\)/g, (m) => p.grey(m)); // baseline/estimate is context
|
||||
l = l.replace(/^(\s+worst \w+\s+)(.+?)( — )/, (_, a, k, b) => a + p.green(k) + b); // attribution key (may contain spaces)
|
||||
.replace(new RegExp(REPORT_GLYPH.ok, "g"), p.green(REPORT_GLYPH.ok))
|
||||
.replace(new RegExp(REPORT_GLYPH.warn, "g"), p.amber(REPORT_GLYPH.warn))
|
||||
.replace(new RegExp(REPORT_GLYPH.crit, "g"), p.red(REPORT_GLYPH.crit))
|
||||
.replace(new RegExp(REPORT_GLYPH.neutral, "g"), p.grey(REPORT_GLYPH.neutral))
|
||||
.replace(new RegExp(REPORT_GLYPH.untrusted, "g"), p.amber(REPORT_GLYPH.untrusted));
|
||||
l = l.replace(/↑ ?\d+×/g, (m) => p.amber(m)).replace(/↓ ?\d+×/g, (m) => p.green(m));
|
||||
l = l.replace(/→ flat/g, (m) => p.grey(m));
|
||||
l = l.replace(/\(last \d+ min\)|\(\d+ min window\)|for the last \d+ min/g, (m) => p.amber(m));
|
||||
l = l.replace(/\(normal ~[^)]*\)|\(estimated[^)]*\)/g, (m) => p.grey(m)); // context, not a verdict
|
||||
return l;
|
||||
})
|
||||
.join("\n");
|
||||
|
||||
@@ -0,0 +1,101 @@
|
||||
import { describe, expect, it } from "vitest";
|
||||
import { buildReportLayout, type LayoutViewModel } from "./report-layout";
|
||||
import { reportMessages } from "./report-messages";
|
||||
|
||||
const livenessMetric = {
|
||||
id: "liveness",
|
||||
value: 21 * 60 * 1000,
|
||||
unit: "ms" as const,
|
||||
severity: "crit" as const,
|
||||
};
|
||||
|
||||
function vmWith(findings: LayoutViewModel["findings"]): LayoutViewModel {
|
||||
return {
|
||||
title: "health",
|
||||
scope: "prod",
|
||||
period: "last 60 min",
|
||||
windowMinutes: 60,
|
||||
summary: { severity: "crit", statements: [] },
|
||||
findings,
|
||||
metrics: [livenessMetric],
|
||||
footer: [],
|
||||
};
|
||||
}
|
||||
|
||||
const livenessFinding = {
|
||||
type: "liveness",
|
||||
severity: "crit" as const,
|
||||
reason: "stale",
|
||||
metricIds: ["liveness"],
|
||||
};
|
||||
|
||||
function vmWithMetric(metric: LayoutViewModel["metrics"][number]): LayoutViewModel {
|
||||
return {
|
||||
title: "health",
|
||||
scope: "prod",
|
||||
period: "last 60 min",
|
||||
windowMinutes: 60,
|
||||
summary: { severity: "warn", statements: [] },
|
||||
findings: [{ type: "queue", severity: "warn", reason: "backlog", metricIds: [metric.id] }],
|
||||
metrics: [metric],
|
||||
footer: [],
|
||||
};
|
||||
}
|
||||
|
||||
function heroDelta(metric: LayoutViewModel["metrics"][number]) {
|
||||
const layout = buildReportLayout(vmWithMetric(metric), reportMessages("health"));
|
||||
return layout.hero?.metrics.find((m) => m.id === metric.id)?.delta;
|
||||
}
|
||||
|
||||
describe("buildReportLayout metric deltas", () => {
|
||||
it("renders no delta when a metric collapsed to nothing", () => {
|
||||
expect(
|
||||
heroDelta({
|
||||
id: "pending",
|
||||
value: 0,
|
||||
unit: "count",
|
||||
severity: "warn",
|
||||
normal: 40,
|
||||
delta: { dir: "down", mult: 0 },
|
||||
})
|
||||
).toBeUndefined();
|
||||
});
|
||||
|
||||
it("still renders a multiplier for a genuine fall", () => {
|
||||
expect(
|
||||
heroDelta({
|
||||
id: "pending",
|
||||
value: 10,
|
||||
unit: "count",
|
||||
severity: "warn",
|
||||
normal: 40,
|
||||
delta: { dir: "down", mult: 0 },
|
||||
})
|
||||
).toEqual({ text: "↓ 4×", dir: "down" });
|
||||
});
|
||||
});
|
||||
|
||||
describe("buildReportLayout self-evident findings", () => {
|
||||
it("drops the metric row of a non-hero finding whose only metric is its own line", () => {
|
||||
const layout = buildReportLayout(
|
||||
vmWith([
|
||||
{ type: "execution", severity: "crit", reason: "failures_up", metricIds: [] },
|
||||
livenessFinding,
|
||||
]),
|
||||
reportMessages("health")
|
||||
);
|
||||
|
||||
const liveness = layout.findings.find((f) => f.type === "liveness");
|
||||
expect(liveness?.text).toContain("no telemetry in 21m");
|
||||
expect(liveness?.expanded).toBe(false);
|
||||
expect(liveness?.metrics).toEqual([]);
|
||||
});
|
||||
|
||||
it("keeps the metric row when that finding is the hero", () => {
|
||||
const layout = buildReportLayout(vmWith([livenessFinding]), reportMessages("health"));
|
||||
|
||||
expect(layout.hero?.type).toBe("liveness");
|
||||
expect(layout.hero?.expanded).toBe(true);
|
||||
expect(layout.hero?.metrics.map((m) => m.id)).toEqual(["liveness"]);
|
||||
});
|
||||
});
|
||||
@@ -0,0 +1,668 @@
|
||||
/**
|
||||
* The report's layout, declared once: section order, section labels, the tone -> glyph vocabulary,
|
||||
* and every string a renderer places. `renderMarkdown` (markdown + ANSI) and the React card both
|
||||
* build from `buildReportLayout`, so no surface can drift in order, labels or wording.
|
||||
*
|
||||
* A renderer decides typography only: colour, indentation, column alignment, tooltips. Anything
|
||||
* that answers "what is shown, in what order, called what" belongs here.
|
||||
*
|
||||
* Input types are deliberately loose (`aggregation?: string`, not the enum) so both the presenter's
|
||||
* `ReportViewModel` and the agent contracts' `ReportViewModelPayload` satisfy them.
|
||||
*/
|
||||
|
||||
import { type ReportMessages } from "./report-messages";
|
||||
import { type Severity, type Unit } from "./report-view-model";
|
||||
|
||||
// --- vocabulary -------------------------------------------------------------
|
||||
|
||||
/**
|
||||
* A verdict's tone. `neutral` is a genuinely-unknown state: not good, not bad, and it must never
|
||||
* read as a confident tick.
|
||||
*/
|
||||
export type ReportTone = "ok" | "warn" | "crit" | "neutral";
|
||||
|
||||
/**
|
||||
* The one glyph vocabulary. Text surfaces carry meaning in these because an MCP host shows plain
|
||||
* text and a terminal may be monochrome; ANSI paints them as well. Every character is BMP and
|
||||
* single-width, so nothing here depends on emoji fonts or colour to be legible.
|
||||
*/
|
||||
export const REPORT_GLYPH = {
|
||||
ok: "✓",
|
||||
warn: "⚠",
|
||||
crit: "✕",
|
||||
/** Genuinely unknown, neither good nor bad. */
|
||||
neutral: "○",
|
||||
/** The data behind the report can't be trusted. */
|
||||
untrusted: "⚑",
|
||||
/** Above its baseline. */
|
||||
up: "↑",
|
||||
/** Below its baseline. */
|
||||
down: "↓",
|
||||
/** Compared against a baseline and unmoved. */
|
||||
flat: "→",
|
||||
} as const;
|
||||
|
||||
/** Section and block labels. Both renderers read them from here. */
|
||||
export const REPORT_LABELS = {
|
||||
/** Evidence supporting a finding. */
|
||||
why: "why:",
|
||||
/** The causal chain across findings. */
|
||||
read: "read:",
|
||||
/** The footer heading. */
|
||||
nextSteps: "Next steps",
|
||||
} as const;
|
||||
|
||||
/** The flag beside the report's name, and the caveat under its headline. */
|
||||
export type LayoutTrust = { badge: string; note: string };
|
||||
|
||||
/**
|
||||
* Why a report's numbers can't be trusted, in its own words. Stale, absent and unmeasured are three
|
||||
* different states: a snapshot with no telemetry feed is not stale, and a caveat may only discount
|
||||
* the input it names — the aggregates the report did measure stay measured.
|
||||
*/
|
||||
const TRUST_CAVEATS: Record<string, LayoutTrust> = {
|
||||
telemetry_stale: {
|
||||
badge: "stale data",
|
||||
note: "The telemetry behind this report is stale, so the numbers below are informational only.",
|
||||
},
|
||||
telemetry_absent: {
|
||||
badge: "no telemetry",
|
||||
note: "No telemetry feed reached this report, so how current it is can't be confirmed; the numbers below are still measured over the window.",
|
||||
},
|
||||
flow_unmeasured: {
|
||||
badge: "unmeasured",
|
||||
note: "The queue depth could not be measured, so the backlog can't be assessed; the other numbers below are measured.",
|
||||
},
|
||||
};
|
||||
|
||||
const TRUST_CAVEAT_FALLBACK: LayoutTrust = {
|
||||
badge: "unverified data",
|
||||
note: "The data behind this report could not be verified, so the numbers below are informational only.",
|
||||
};
|
||||
|
||||
/**
|
||||
* The report's sections, top to bottom. A renderer walks this order; a new section has to be added
|
||||
* here first, which is what keeps the surfaces aligned. `trust` spans two places: a flag beside the
|
||||
* report's name, and its caveat under the headline.
|
||||
*/
|
||||
export const REPORT_SECTION_ORDER = [
|
||||
"header",
|
||||
"trust",
|
||||
"headline",
|
||||
"hero",
|
||||
"findings",
|
||||
"statements",
|
||||
"read",
|
||||
"footer",
|
||||
] as const;
|
||||
|
||||
export type ReportSectionId = (typeof REPORT_SECTION_ORDER)[number];
|
||||
|
||||
/**
|
||||
* Reasons that mean "we can't say" rather than a verdict, so their finding renders headline-only.
|
||||
* A measured finding never carries one: an unmeasured input costs its own metric, not the verdict.
|
||||
*/
|
||||
const UNASSESSABLE_REASONS = new Set(["unknown", "flow_unmeasured"]);
|
||||
|
||||
/**
|
||||
* Reasons whose state is genuinely unknown but not bad. A stale feed is different: the trust guard
|
||||
* forces crit.
|
||||
*/
|
||||
const NEUTRAL_REASONS = new Set(["freshness_unknown", "flow_unmeasured"]);
|
||||
|
||||
/**
|
||||
* `facts.trustworthy === false` means the numbers behind the verdict are informational only. Absent
|
||||
* = trustworthy (the common case, and what pre-`facts` snapshots imply).
|
||||
*/
|
||||
export function reportIsTrustworthy(vm: { facts?: Record<string, unknown> }): boolean {
|
||||
return vm.facts?.trustworthy !== false;
|
||||
}
|
||||
|
||||
/** The caveat for an untrustworthy report, chosen by `facts.untrustworthyReason`. */
|
||||
export function reportTrust(vm: { facts?: Record<string, unknown> }): LayoutTrust | undefined {
|
||||
if (reportIsTrustworthy(vm)) return undefined;
|
||||
const reason = vm.facts?.untrustworthyReason;
|
||||
return (typeof reason === "string" ? TRUST_CAVEATS[reason] : undefined) ?? TRUST_CAVEAT_FALLBACK;
|
||||
}
|
||||
|
||||
export function reportTone(severity: Severity, reason?: string): ReportTone {
|
||||
return reason !== undefined && NEUTRAL_REASONS.has(reason) ? "neutral" : severity;
|
||||
}
|
||||
|
||||
export function reportGlyph(severity: Severity, reason?: string): string {
|
||||
return REPORT_GLYPH[reportTone(severity, reason)];
|
||||
}
|
||||
|
||||
// --- footer vocabulary ------------------------------------------------------
|
||||
|
||||
/**
|
||||
* How a footer entry renders, keyed off the code rather than its URL so the same code looks the
|
||||
* same everywhere. `action` is a primary control, `docs` the docs entry, `reference` a place to
|
||||
* look, and `note` an option stated rather than offered.
|
||||
*/
|
||||
export type ReportFooterStyle = "action" | "docs" | "reference" | "note";
|
||||
|
||||
const FOOTER_NOTE_CODES = new Set(["nothing_to_do", "do_nothing_drains", "region_failover"]);
|
||||
|
||||
const FOOTER_REFERENCE_CODES = new Set(["check_control_plane", "check_platform_status"]);
|
||||
|
||||
/** A doc entry names itself one: `concurrency_docs`, `retries_docs`. */
|
||||
const FOOTER_DOCS_SUFFIX = "_docs";
|
||||
|
||||
export function reportFooterStyle(code: string): ReportFooterStyle {
|
||||
if (FOOTER_NOTE_CODES.has(code)) return "note";
|
||||
if (code.endsWith(FOOTER_DOCS_SUFFIX)) return "docs";
|
||||
if (FOOTER_REFERENCE_CODES.has(code)) return "reference";
|
||||
return "action";
|
||||
}
|
||||
|
||||
// --- formatting -------------------------------------------------------------
|
||||
|
||||
const MINUS = "−"; // U+2212
|
||||
|
||||
export function fmtCount(n: number): string {
|
||||
return Math.round(n).toLocaleString("en-US");
|
||||
}
|
||||
|
||||
export function fmtDuration(ms: number): string {
|
||||
if (ms < 1000) return `${Math.round(ms)}ms`;
|
||||
const s = ms / 1000;
|
||||
if (s < 60) return Number.isInteger(s) ? `${s}s` : `${s.toFixed(1)}s`;
|
||||
const m = s / 60;
|
||||
return Number.isInteger(m) ? `${m}m` : `${m.toFixed(1)}m`;
|
||||
}
|
||||
|
||||
export function fmtPct(ratio: number): string {
|
||||
return `${(ratio * 100).toFixed(1)}%`;
|
||||
}
|
||||
|
||||
export function fmtRate(n: number): string {
|
||||
return `${fmtCount(n)}/min`;
|
||||
}
|
||||
|
||||
/** A net rate carries its sign; a plain rate does not, so it isn't read as a change. */
|
||||
export function fmtSignedRate(net: number): string {
|
||||
const sign = net < 0 ? MINUS : net > 0 ? "+" : "";
|
||||
return `${sign}${fmtCount(Math.abs(net))}/min`;
|
||||
}
|
||||
|
||||
export function fmtValue(value: number, unit: Unit): string {
|
||||
switch (unit) {
|
||||
case "ms":
|
||||
return fmtDuration(value);
|
||||
case "count":
|
||||
return fmtCount(value);
|
||||
case "ratio":
|
||||
return fmtPct(value);
|
||||
case "perMin":
|
||||
return fmtRate(value);
|
||||
}
|
||||
}
|
||||
|
||||
/** Fill the `{token}` placeholders a message catalog leaves for the renderer. */
|
||||
export function fillTokens(
|
||||
template: string,
|
||||
tokens: Record<string, string | number | undefined>
|
||||
): string {
|
||||
return template.replace(/\{(\w+)\}/g, (whole, key: string) => {
|
||||
const value = tokens[key];
|
||||
if (value === undefined) return whole;
|
||||
// Grouped but never rounded: a drain ETA of 26.7 min must stay 26.7.
|
||||
return typeof value === "number" ? value.toLocaleString("en-US") : value;
|
||||
});
|
||||
}
|
||||
|
||||
// --- input shapes -----------------------------------------------------------
|
||||
|
||||
type DeltaInput = { dir: "up" | "down" | "flat"; mult?: number };
|
||||
|
||||
export type LayoutMetricInput = {
|
||||
id: string;
|
||||
value: number;
|
||||
unit: Unit;
|
||||
aggregation?: string;
|
||||
normal?: number;
|
||||
delta?: DeltaInput;
|
||||
series?: { points: number[]; kind: string };
|
||||
breakdown?: Record<string, number>;
|
||||
annotation?: { code: string; value?: number };
|
||||
availability?: string;
|
||||
severity: Severity;
|
||||
};
|
||||
|
||||
export type LayoutFindingInput = {
|
||||
type: string;
|
||||
severity: Severity;
|
||||
reason: string;
|
||||
read?: string;
|
||||
metricIds: string[];
|
||||
anomalyWindow?: { minutes: number; touchesEnd: boolean };
|
||||
attribution?: { dim: string; key: string; share: number; of: string };
|
||||
exclusions?: { code: string; evidence?: Record<string, number> }[];
|
||||
observations?: { code: string; evidence?: Record<string, number> }[];
|
||||
};
|
||||
|
||||
export type LayoutViewModel = {
|
||||
title: string;
|
||||
scope: string;
|
||||
period: string;
|
||||
baselineLabel?: string;
|
||||
windowMinutes: number;
|
||||
summary: {
|
||||
severity: Severity;
|
||||
statements: { findingType: string; severity: Severity; reason?: string }[];
|
||||
};
|
||||
findings: LayoutFindingInput[];
|
||||
metrics: LayoutMetricInput[];
|
||||
facts?: Record<string, unknown>;
|
||||
footer: { code: string; link?: string; value?: number }[];
|
||||
};
|
||||
|
||||
// --- output shapes ----------------------------------------------------------
|
||||
|
||||
export type LayoutDelta = { text: string; dir: "up" | "down" | "flat" };
|
||||
|
||||
/** A metric's aside. `kind` lets a renderer choose its own frame around shared wording. */
|
||||
export type LayoutNote = { kind: "annotation" | "baseline" | "estimated"; text: string };
|
||||
|
||||
export type LayoutMetricRow = {
|
||||
id: string;
|
||||
label: string;
|
||||
value: string;
|
||||
/** Kept alongside the formatted value so a chart can format its own points. */
|
||||
unit: Unit;
|
||||
severity: Severity;
|
||||
delta?: LayoutDelta;
|
||||
note?: LayoutNote;
|
||||
/** The row that explains the finding: its annotation is spelled out rather than tucked away. */
|
||||
hero: boolean;
|
||||
series?: number[];
|
||||
/** Trailing breach window of the driving metric, when it reaches now. */
|
||||
anomalyMinutes?: number;
|
||||
/** A composite metric's parts, shown under it. */
|
||||
subRows: { label: string; value: string }[];
|
||||
};
|
||||
|
||||
export type LayoutFinding = {
|
||||
type: string;
|
||||
/** Column label for the section, e.g. "EXECUTION". */
|
||||
label: string;
|
||||
severity: Severity;
|
||||
tone: ReportTone;
|
||||
glyph: string;
|
||||
/** The resolved reason, plus its anomaly window. */
|
||||
text: string;
|
||||
/** Whether the evidence block is shown at all. */
|
||||
expanded: boolean;
|
||||
metrics: LayoutMetricRow[];
|
||||
/** Evidence lines under `why:`. */
|
||||
why: string[];
|
||||
/** The attributed key, so a renderer can pick it out of the first `why:` line. */
|
||||
attributionKey?: string;
|
||||
};
|
||||
|
||||
export type LayoutStatement = { tone: ReportTone; glyph: string; severity: Severity; text: string };
|
||||
|
||||
export type LayoutFooterEntry = {
|
||||
code: string;
|
||||
style: ReportFooterStyle;
|
||||
label: string;
|
||||
/** Key into `vm.links`. */
|
||||
link?: string;
|
||||
};
|
||||
|
||||
export type ReportLayout = {
|
||||
header: { name: string; meta: string };
|
||||
/** Present only when the data can't be trusted. */
|
||||
trust?: LayoutTrust;
|
||||
headline: { tone: ReportTone; glyph: string; severity: Severity; phrase: string; text?: string };
|
||||
/** The finding the headline speaks for, always expanded. */
|
||||
hero?: LayoutFinding;
|
||||
/** The remaining findings, in view-model order. */
|
||||
findings: LayoutFinding[];
|
||||
/** Statements with no finding behind them, which still have to be said. */
|
||||
statements: LayoutStatement[];
|
||||
reads: string[];
|
||||
footer: LayoutFooterEntry[];
|
||||
};
|
||||
|
||||
// --- build ------------------------------------------------------------------
|
||||
|
||||
/** The layout of `vm`, with every code already resolved through `messages`. */
|
||||
export function buildReportLayout(vm: LayoutViewModel, messages: ReportMessages): ReportLayout {
|
||||
const tokens = reportTokens(vm);
|
||||
|
||||
const heroIndex = heroIndexOf(vm);
|
||||
const heroInput = vm.findings[heroIndex];
|
||||
const heroStatement = vm.summary.statements.find((s) => s.findingType === heroInput?.type);
|
||||
|
||||
const hero = heroInput ? findingLayout(vm, messages, heroInput, tokens, true) : undefined;
|
||||
const findings = vm.findings
|
||||
.filter((_, i) => i !== heroIndex)
|
||||
.map((finding) => findingLayout(vm, messages, finding, tokens, false));
|
||||
|
||||
// The headline speaks for the hero finding. When its statement carries its own reason (stale
|
||||
// telemetry, no freshness signal) that statement is the whole sentence and the finding's reason
|
||||
// would only repeat it.
|
||||
const phrase = heroInput
|
||||
? messages.statementMessage(heroInput.type, heroInput.severity, heroStatement?.reason)
|
||||
: messages.statementMessage(vm.title, vm.summary.severity);
|
||||
const text =
|
||||
heroInput && !heroStatement?.reason
|
||||
? fillTokens(
|
||||
messages.findingReason(heroInput.type, heroInput.reason, {
|
||||
expanded: heroInput.severity === "ok",
|
||||
}),
|
||||
tokens
|
||||
) + headlineWindow(heroInput)
|
||||
: undefined;
|
||||
|
||||
const statements = vm.summary.statements
|
||||
.filter((statement) => !vm.findings.some((f) => f.type === statement.findingType))
|
||||
.map((statement) => ({
|
||||
severity: statement.severity,
|
||||
tone: reportTone(statement.severity, statement.reason),
|
||||
glyph: reportGlyph(statement.severity, statement.reason),
|
||||
text: messages.statementMessage(statement.findingType, statement.severity, statement.reason),
|
||||
}));
|
||||
|
||||
// Hero first, then the rest. An unassessable finding contributes nothing: its read would only
|
||||
// repeat the trust note.
|
||||
const reads = (
|
||||
hero ? [heroInput!, ...vm.findings.filter((_, i) => i !== heroIndex)] : vm.findings
|
||||
)
|
||||
.filter((finding) => finding.read !== undefined && !UNASSESSABLE_REASONS.has(finding.reason))
|
||||
.map((finding) => fillTokens(messages.readMessage(finding.read!), tokens));
|
||||
|
||||
const trust = reportTrust(vm);
|
||||
|
||||
return {
|
||||
header: {
|
||||
name: vm.title,
|
||||
meta: [vm.scope, vm.period, vm.baselineLabel].filter(Boolean).join(" · "),
|
||||
},
|
||||
...(trust === undefined ? {} : { trust }),
|
||||
headline: {
|
||||
severity: vm.summary.severity,
|
||||
tone: reportTone(vm.summary.severity, heroStatement?.reason),
|
||||
glyph: reportGlyph(vm.summary.severity, heroStatement?.reason),
|
||||
phrase,
|
||||
...(text === undefined ? {} : { text }),
|
||||
},
|
||||
...(hero === undefined ? {} : { hero }),
|
||||
findings,
|
||||
statements,
|
||||
reads,
|
||||
footer: footerLayout(vm, messages, tokens),
|
||||
};
|
||||
}
|
||||
|
||||
/** The finding the headline speaks for: the first one at the report's severity. */
|
||||
function heroIndexOf(vm: LayoutViewModel): number {
|
||||
const index = vm.findings.findIndex((finding) => finding.severity === vm.summary.severity);
|
||||
return index === -1 ? 0 : index;
|
||||
}
|
||||
|
||||
/** Tokens the catalog's strings leave for the renderer, resolved from the view model's metrics. */
|
||||
function reportTokens(vm: LayoutViewModel): Record<string, string | number> {
|
||||
const metric = (id: string) => vm.metrics.find((m) => m.id === id);
|
||||
const triggered = metric("triggered");
|
||||
const throughput = metric("throughput");
|
||||
const liveness = metric("liveness");
|
||||
return {
|
||||
mult: triggered?.delta?.mult ?? "",
|
||||
rate: Math.round(throughput?.breakdown?.done ?? 0),
|
||||
age: liveness === undefined || isUnmeasured(liveness) ? "unknown" : fmtDuration(liveness.value),
|
||||
};
|
||||
}
|
||||
|
||||
function isUnmeasured(metric: LayoutMetricInput): boolean {
|
||||
return metric.availability === "unknown" || !Number.isFinite(metric.value);
|
||||
}
|
||||
|
||||
/** The anomaly window as the headline says it. */
|
||||
function headlineWindow(finding: LayoutFindingInput): string {
|
||||
const window = finding.anomalyWindow;
|
||||
if (!window) return "";
|
||||
return window.touchesEnd
|
||||
? ` for the last ${window.minutes} min`
|
||||
: ` (${window.minutes} min window)`;
|
||||
}
|
||||
|
||||
/** The same window on a finding line, where it is an aside rather than the sentence's tail. */
|
||||
function findingWindow(finding: LayoutFindingInput): string {
|
||||
const window = finding.anomalyWindow;
|
||||
if (!window) return "";
|
||||
return window.touchesEnd ? ` (last ${window.minutes} min)` : ` (${window.minutes} min window)`;
|
||||
}
|
||||
|
||||
function findingLayout(
|
||||
vm: LayoutViewModel,
|
||||
messages: ReportMessages,
|
||||
finding: LayoutFindingInput,
|
||||
tokens: Record<string, string | number>,
|
||||
hero: boolean
|
||||
): LayoutFinding {
|
||||
// A finding whose reason says "we can't say" shows no evidence: the numbers behind it are
|
||||
// placeholders. Otherwise the hero is always expanded, and the rest only when degraded.
|
||||
// A self-evident finding's one metric only repeats its own line ("stale — no telemetry in 21m"
|
||||
// over "liveness 21m"); as the hero that line is the headline, so the row is its only evidence.
|
||||
const selfEvident = finding.metricIds.length === 1 && finding.metricIds[0] === finding.type;
|
||||
const expanded =
|
||||
!UNASSESSABLE_REASONS.has(finding.reason) &&
|
||||
(hero || (finding.severity !== "ok" && !selfEvident));
|
||||
|
||||
const metrics = expanded
|
||||
? finding.metricIds
|
||||
.map((id) => vm.metrics.find((m) => m.id === id))
|
||||
.filter((m): m is LayoutMetricInput => m !== undefined)
|
||||
.map((metric, i) =>
|
||||
metricRow(vm, messages, metric, i === 0, i === 0 ? anomalyMinutes(finding) : undefined)
|
||||
)
|
||||
: [];
|
||||
|
||||
return {
|
||||
type: finding.type,
|
||||
label: finding.type.toUpperCase(),
|
||||
severity: finding.severity,
|
||||
tone: reportTone(finding.severity, finding.reason),
|
||||
glyph: reportGlyph(finding.severity, finding.reason),
|
||||
text:
|
||||
fillTokens(
|
||||
messages.findingReason(finding.type, finding.reason, {
|
||||
expanded: finding.severity === "ok",
|
||||
}),
|
||||
tokens
|
||||
) + findingWindow(finding),
|
||||
expanded,
|
||||
metrics,
|
||||
why: expanded ? whyLines(messages, finding, tokens) : [],
|
||||
...(expanded && finding.attribution ? { attributionKey: finding.attribution.key } : {}),
|
||||
};
|
||||
}
|
||||
|
||||
function anomalyMinutes(finding: LayoutFindingInput): number | undefined {
|
||||
return finding.anomalyWindow?.touchesEnd ? finding.anomalyWindow.minutes : undefined;
|
||||
}
|
||||
|
||||
/** Attribution first, then ruled-out causes, then supporting observations. */
|
||||
function whyLines(
|
||||
messages: ReportMessages,
|
||||
finding: LayoutFindingInput,
|
||||
tokens: Record<string, string | number>
|
||||
): string[] {
|
||||
const lines: string[] = [];
|
||||
const attribution = finding.attribution;
|
||||
if (attribution) {
|
||||
lines.push(
|
||||
`${Math.round(attribution.share * 100)}% of ${attribution.of} is ${attribution.key}`
|
||||
);
|
||||
}
|
||||
for (const exclusion of finding.exclusions ?? []) {
|
||||
lines.push(
|
||||
fillTokens(
|
||||
messages.exclusionMessage(exclusion.code),
|
||||
evidenceTokens(tokens, exclusion.evidence)
|
||||
)
|
||||
);
|
||||
}
|
||||
for (const observation of finding.observations ?? []) {
|
||||
lines.push(
|
||||
fillTokens(
|
||||
messages.observationMessage(observation.code),
|
||||
evidenceTokens(tokens, observation.evidence)
|
||||
)
|
||||
);
|
||||
}
|
||||
return lines;
|
||||
}
|
||||
|
||||
/**
|
||||
* A code's own evidence beats the report-wide tokens. `finishedPerMin` is the measured rate behind
|
||||
* `{rate}`, so it fills that token rather than needing one of its own.
|
||||
*/
|
||||
function evidenceTokens(
|
||||
tokens: Record<string, string | number>,
|
||||
evidence: Record<string, number> | undefined
|
||||
): Record<string, string | number> {
|
||||
if (!evidence) return tokens;
|
||||
return {
|
||||
...tokens,
|
||||
...evidence,
|
||||
...(evidence.finishedPerMin === undefined ? {} : { rate: evidence.finishedPerMin }),
|
||||
};
|
||||
}
|
||||
|
||||
function isComposite(metric: LayoutMetricInput): boolean {
|
||||
return metric.unit === "perMin" && metric.breakdown?.done !== undefined;
|
||||
}
|
||||
|
||||
function metricValue(metric: LayoutMetricInput, messages: ReportMessages): string {
|
||||
// A placeholder is never printed as a number.
|
||||
if (isUnmeasured(metric)) return "unknown";
|
||||
// concurrency etc. carry a limit -> "running/limit".
|
||||
if (metric.unit === "count" && metric.breakdown?.limit !== undefined) {
|
||||
return `${fmtCount(metric.value)}/${fmtCount(metric.breakdown.limit)}`;
|
||||
}
|
||||
// A composite throughput's value is the net, which is signed; its parts are sub-rows.
|
||||
if (isComposite(metric)) return fmtSignedRate(metric.value);
|
||||
const showAggregation =
|
||||
metric.aggregation === "p95" &&
|
||||
!messages.metricLabel(metric.id).toLowerCase().startsWith("p95");
|
||||
return showAggregation
|
||||
? `p95 ${fmtValue(metric.value, metric.unit)}`
|
||||
: fmtValue(metric.value, metric.unit);
|
||||
}
|
||||
|
||||
/**
|
||||
* How far a metric fell below its baseline: `undefined` when the fall doesn't round past 1×, and
|
||||
* `null` when it collapsed to nothing and no multiplier can say it.
|
||||
*/
|
||||
function fallMultiplier(metric: LayoutMetricInput): number | null | undefined {
|
||||
if (metric.normal === undefined || metric.normal <= 0) return undefined;
|
||||
if (metric.value <= 0) return null;
|
||||
const fall = Math.round(metric.normal / metric.value);
|
||||
return fall > 1 ? fall : undefined;
|
||||
}
|
||||
|
||||
/**
|
||||
* A metric's movement against its baseline. A multiplier only reads as movement once it rounds past
|
||||
* 1×; below that a metric with a baseline is flat, and one without has nothing to compare against.
|
||||
*/
|
||||
function metricDelta(metric: LayoutMetricInput): LayoutDelta | undefined {
|
||||
const delta = metric.delta;
|
||||
// A fall's own multiplier rounds to 0 or 1, so measure how far it fell instead.
|
||||
if (delta?.dir === "down") {
|
||||
const fall = fallMultiplier(metric);
|
||||
// An arrow with no multiplier behind it says nothing the sparkline hasn't.
|
||||
if (fall === null) return undefined;
|
||||
if (fall !== undefined) return { text: `${REPORT_GLYPH.down} ${fall}×`, dir: "down" };
|
||||
}
|
||||
if (delta && delta.mult !== undefined && delta.mult > 1 && delta.dir !== "flat") {
|
||||
return {
|
||||
text: `${delta.dir === "up" ? REPORT_GLYPH.up : REPORT_GLYPH.down} ${delta.mult}×`,
|
||||
dir: delta.dir,
|
||||
};
|
||||
}
|
||||
return metric.normal === undefined
|
||||
? undefined
|
||||
: { text: `${REPORT_GLYPH.flat} flat`, dir: "flat" };
|
||||
}
|
||||
|
||||
function metricNote(
|
||||
vm: LayoutViewModel,
|
||||
messages: ReportMessages,
|
||||
metric: LayoutMetricInput
|
||||
): LayoutNote | undefined {
|
||||
if (metric.annotation) {
|
||||
return {
|
||||
kind: "annotation",
|
||||
text: fillTokens(messages.annotationMessage(metric.annotation.code), {
|
||||
value: metric.annotation.value,
|
||||
window: vm.windowMinutes,
|
||||
limit: metric.breakdown?.limit,
|
||||
}),
|
||||
};
|
||||
}
|
||||
if (metric.normal !== undefined) {
|
||||
return { kind: "baseline", text: `normal ~${fmtValue(metric.normal, metric.unit)}` };
|
||||
}
|
||||
// A proxy trend (e.g. a snapshot backlog) is a shape, not a measurement, so say so.
|
||||
if (metric.series?.kind === "estimated") {
|
||||
return { kind: "estimated", text: "estimated from a proxy signal" };
|
||||
}
|
||||
return undefined;
|
||||
}
|
||||
|
||||
function metricRow(
|
||||
vm: LayoutViewModel,
|
||||
messages: ReportMessages,
|
||||
metric: LayoutMetricInput,
|
||||
hero: boolean,
|
||||
anomaly: number | undefined
|
||||
): LayoutMetricRow {
|
||||
const delta = metricDelta(metric);
|
||||
const note = metricNote(vm, messages, metric);
|
||||
const points = isUnmeasured(metric) ? undefined : metric.series?.points;
|
||||
|
||||
return {
|
||||
id: metric.id,
|
||||
label: messages.metricLabel(metric.id),
|
||||
value: metricValue(metric, messages),
|
||||
unit: metric.unit,
|
||||
severity: metric.severity,
|
||||
hero,
|
||||
...(delta === undefined ? {} : { delta }),
|
||||
...(note === undefined ? {} : { note }),
|
||||
...(points && points.length > 0 ? { series: points } : {}),
|
||||
...(anomaly === undefined ? {} : { anomalyMinutes: anomaly }),
|
||||
subRows: isComposite(metric)
|
||||
? [
|
||||
{ label: "done", value: fmtRate(metric.breakdown!.done!) },
|
||||
{ label: "triggered", value: fmtRate(metric.breakdown!.triggered ?? 0) },
|
||||
]
|
||||
: [],
|
||||
};
|
||||
}
|
||||
|
||||
/** Offered entries first, stated options last — a note is the fallback, not a next step. */
|
||||
function footerLayout(
|
||||
vm: LayoutViewModel,
|
||||
messages: ReportMessages,
|
||||
tokens: Record<string, string | number>
|
||||
): LayoutFooterEntry[] {
|
||||
const entries = vm.footer.map((entry) => ({
|
||||
code: entry.code,
|
||||
style: reportFooterStyle(entry.code),
|
||||
label: fillTokens(messages.actionMessage(entry.code), {
|
||||
...tokens,
|
||||
value: entry.value,
|
||||
min: entry.value,
|
||||
}),
|
||||
...(entry.link === undefined ? {} : { link: entry.link }),
|
||||
}));
|
||||
return [
|
||||
...entries.filter((entry) => entry.style !== "note"),
|
||||
...entries.filter((entry) => entry.style === "note"),
|
||||
];
|
||||
}
|
||||
@@ -1,8 +1,3 @@
|
||||
/**
|
||||
* Catalogs by value, in a module that imports ONLY the per-report `*-messages`
|
||||
* files — no loaders, no IO. Presentation stays decoupled from the data layer,
|
||||
* and a value import can't be tree-shaken away.
|
||||
*/
|
||||
import { healthMessages } from "./health/health-messages";
|
||||
import { type ReportMessages } from "./report-messages";
|
||||
|
||||
|
||||
@@ -1,17 +1,7 @@
|
||||
/**
|
||||
* Report-agnostic message infrastructure. Prose lives in each report's OWN catalog (e.g.
|
||||
* `health/health-messages.ts`); this file only defines the resolver surface + a registry so the
|
||||
* generic renderer can turn a VM's codes into strings by looking the catalog up via `vm.title`.
|
||||
* No report vocabulary here — that would re-couple the renderer to a specific report.
|
||||
*/
|
||||
|
||||
import { REPORT_MESSAGE_CATALOGS } from "./report-message-catalogs";
|
||||
import { type ReasonCode, type Severity } from "./report-view-model";
|
||||
|
||||
/**
|
||||
* The resolver surface a report provides. Every renderer resolves a VM's codes through this;
|
||||
* strings may carry {tokens} (e.g. {age}, {rate}) that the renderer fills from evidence.
|
||||
*/
|
||||
/** Strings may carry {tokens} the renderer fills from evidence. */
|
||||
export type ReportMessages = {
|
||||
metricLabel(id: string): string;
|
||||
findingReason(findingType: string, reason: ReasonCode, opts?: { expanded?: boolean }): string;
|
||||
@@ -23,9 +13,7 @@ export type ReportMessages = {
|
||||
actionMessage(code: ReasonCode): string;
|
||||
};
|
||||
|
||||
/** Look up a report's catalog by `vm.title`. Catalogs are values, never
|
||||
* registered at import time — side-effect registration is what the production
|
||||
* bundle tree-shakes away. */
|
||||
/** Catalogs are plain values, not import-time registrations: the bundle tree-shakes those away. */
|
||||
export function reportMessages(title: string): ReportMessages {
|
||||
const messages = REPORT_MESSAGE_CATALOGS[title];
|
||||
if (!messages) {
|
||||
|
||||
@@ -1,18 +1,14 @@
|
||||
/**
|
||||
* The report catalog: which reports exist and how each loads + interprets its data. Keyed by
|
||||
* report name so cost/regression/errors drop in later as new `{ load, interpret }` entries with
|
||||
* no changes to the VM, renderers, route, tool, or presenter.
|
||||
*
|
||||
* Deliberately separate from `ReportPresenter` — the presenter only orchestrates (look up a
|
||||
* loader by key, run it, single-flight); knowing WHICH reports exist is a distinct concern.
|
||||
*/
|
||||
|
||||
import { type AuthenticatedEnvironment } from "~/services/apiAuth.server";
|
||||
import { interpret as interpretHealth } from "./health/health";
|
||||
import { loadHealthInput } from "./health/health-data";
|
||||
import { type ReportViewModel } from "./report-view-model";
|
||||
|
||||
/** A query table a report may read. Same table names the query API authorizes against. */
|
||||
export type ReportQueryTable = "runs" | "env_metrics" | "queue_metrics";
|
||||
|
||||
export type ReportLoader<TInput> = {
|
||||
/** Authorization metadata: the route derives its per-table JWT scope check from this. */
|
||||
tables: readonly ReportQueryTable[];
|
||||
load: (env: AuthenticatedEnvironment, period: string) => Promise<TInput>;
|
||||
interpret: (input: TInput) => ReportViewModel;
|
||||
};
|
||||
@@ -23,6 +19,7 @@ function defineReport<TInput>(loader: ReportLoader<TInput>): ReportLoader<unknow
|
||||
|
||||
export const REPORT_REGISTRY: Record<string, ReportLoader<unknown>> = {
|
||||
health: defineReport({
|
||||
tables: ["runs", "env_metrics", "queue_metrics"],
|
||||
load: (env, period) => loadHealthInput(env, period),
|
||||
interpret: interpretHealth,
|
||||
}),
|
||||
@@ -31,7 +28,19 @@ export const REPORT_REGISTRY: Record<string, ReportLoader<unknown>> = {
|
||||
export const REPORT_KEYS = Object.keys(REPORT_REGISTRY);
|
||||
|
||||
export function isReportKey(key: string): boolean {
|
||||
// Object.hasOwn, not `in`: `in` matches prototype keys ("toString", "__proto__"),
|
||||
// which would pass the route guard and then 500 in the loader.
|
||||
// Not `in`: it matches prototype keys like "toString", which would pass the route guard.
|
||||
return Object.hasOwn(REPORT_REGISTRY, key);
|
||||
}
|
||||
|
||||
type ReportTablesRegistry = Record<string, Pick<ReportLoader<unknown>, "tables">>;
|
||||
|
||||
/**
|
||||
* Input to the route's JWT scope check. An unknown key declares no tables: `checkAuth` denies an
|
||||
* empty `everyResource`, so a bad key authorizes nothing rather than everything.
|
||||
*/
|
||||
export function reportQueryTables(
|
||||
key: string,
|
||||
registry: ReportTablesRegistry = REPORT_REGISTRY
|
||||
): readonly ReportQueryTable[] {
|
||||
return Object.hasOwn(registry, key) ? registry[key].tables : [];
|
||||
}
|
||||
|
||||
@@ -1,164 +1,43 @@
|
||||
/**
|
||||
* Generic, render-agnostic contract for a Report. Semantic, not a UI tree: numbers +
|
||||
* what they mean (codes, severities, units, series), never formatted strings or layout.
|
||||
* Every renderer consumes this and owns presentation. Reasons are codes;
|
||||
* `report-messages.ts` resolves them -> strings, so phrasing lives in one place.
|
||||
*
|
||||
* No React/DOM/IO. Report-agnostic — `health` is just one interpreter that emits it.
|
||||
*/
|
||||
// The shapes are zod schemas in `@trigger.dev/core/v3/schemas` because `format=json` serves this view
|
||||
// model verbatim and the clients parse it. Here they are re-aliased to short local names.
|
||||
|
||||
export type Severity = "ok" | "warn" | "crit";
|
||||
|
||||
export type Unit = "ms" | "count" | "ratio" | "perMin";
|
||||
import {
|
||||
type ReportDelta,
|
||||
type ReportExclusion,
|
||||
type ReportFinding,
|
||||
type ReportFooterEntry,
|
||||
type ReportLink as CoreReportLink,
|
||||
type ReportLinkKey,
|
||||
type ReportMetric,
|
||||
type ReportMetricSeries,
|
||||
type ReportObservation,
|
||||
type ReportReasonCode,
|
||||
type ReportRecommendation,
|
||||
type ReportSeverity,
|
||||
type ReportSummaryStatement,
|
||||
type ReportUnit,
|
||||
type ReportViewModel as CoreReportViewModel,
|
||||
} from "@trigger.dev/core/v3/schemas";
|
||||
|
||||
export type Severity = ReportSeverity;
|
||||
export type Unit = ReportUnit;
|
||||
/** A code resolved to a human string by `report-messages.ts`. */
|
||||
export type ReasonCode = string;
|
||||
|
||||
export type ReasonCode = ReportReasonCode;
|
||||
/** A key into `ReportViewModel.links`, so a recommendation can point at a URL. */
|
||||
export type LinkKey = string;
|
||||
export type LinkKey = ReportLinkKey;
|
||||
export type Delta = ReportDelta;
|
||||
export type MetricSeries = ReportMetricSeries;
|
||||
export type Metric = ReportMetric;
|
||||
export type Recommendation = ReportRecommendation;
|
||||
export type FooterEntry = ReportFooterEntry;
|
||||
export type Exclusion = ReportExclusion;
|
||||
export type Observation = ReportObservation;
|
||||
export type Finding = ReportFinding;
|
||||
export type SummaryStatement = ReportSummaryStatement;
|
||||
export type ReportLink = CoreReportLink;
|
||||
export type ReportViewModel = CoreReportViewModel;
|
||||
|
||||
export type Delta = {
|
||||
dir: "up" | "down" | "flat";
|
||||
/** rounded value/normal multiplier; renderer decides whether to print "6×". */
|
||||
mult?: number;
|
||||
};
|
||||
|
||||
export type MetricSeries = {
|
||||
points: number[];
|
||||
/** "estimated" = a proxy (e.g. pending backlog), shown informational-only. */
|
||||
kind: "measured" | "estimated";
|
||||
};
|
||||
|
||||
export type Metric = {
|
||||
/** CODE, e.g. "start_latency_p95" — messages map -> label "start latency". */
|
||||
id: string;
|
||||
value: number;
|
||||
unit: Unit;
|
||||
aggregation?: "p95" | "rate" | "ratio" | "count";
|
||||
/** baseline; renderer formats "(normal ~7s)". */
|
||||
normal?: number;
|
||||
delta?: Delta;
|
||||
series?: MetricSeries;
|
||||
/** named sub-values for composite metrics (e.g. throughput { done, triggered }). */
|
||||
breakdown?: Record<string, number>;
|
||||
/** shown on a cause line INSTEAD of "(normal ~x)", e.g. "pinned 40 of last 60 min". */
|
||||
annotation?: { code: ReasonCode; value?: number };
|
||||
/**
|
||||
* Whether `value` is a real measurement. "unknown" = there was no signal, so `value` is a
|
||||
* placeholder (e.g. liveness age 0) that a structured consumer must NOT read as a real 0 —
|
||||
* the finding's reason carries the "unknown" meaning. Absent = measured (the common case).
|
||||
*/
|
||||
availability?: "measured" | "unknown";
|
||||
severity: Severity;
|
||||
};
|
||||
|
||||
export type Recommendation = {
|
||||
code: ReasonCode;
|
||||
link?: LinkKey;
|
||||
};
|
||||
|
||||
/** A footer line: an action, or the "do nothing" option (carries value). */
|
||||
export type FooterEntry = {
|
||||
code: ReasonCode;
|
||||
link?: LinkKey;
|
||||
/** a computed fact (e.g. drainMinutes), never invented. */
|
||||
value?: number;
|
||||
};
|
||||
|
||||
/** A ruled-out cause + its evidence, e.g. "not your code" (never emitted without evidence). */
|
||||
export type Exclusion = {
|
||||
code: ReasonCode;
|
||||
evidence?: Record<string, number>;
|
||||
};
|
||||
|
||||
/**
|
||||
* A supporting fact backing the verdict — a measured observation, NOT a ruled-out cause,
|
||||
* e.g. "runs are completing at ~820/min". Kept separate from `Exclusion` so the two aren't
|
||||
* conflated (an exclusion answers "what it ISN'T"; an observation states "what IS true").
|
||||
*/
|
||||
export type Observation = {
|
||||
code: ReasonCode;
|
||||
evidence?: Record<string, number>;
|
||||
};
|
||||
|
||||
export type Finding = {
|
||||
/** "flow" | "execution" | "liveness" | future "infrastructure" | "billing" */
|
||||
type: string;
|
||||
severity: Severity;
|
||||
/** CODE for the state/cause, e.g. "env_limit_saturation" | "healthy". */
|
||||
reason: ReasonCode;
|
||||
/** CODE for the "read:" line. Built last, may span findings. */
|
||||
read?: ReasonCode;
|
||||
/** metric ids this finding covers, in causal order when degraded. */
|
||||
metricIds: string[];
|
||||
/** ONE primary action. */
|
||||
recommendation?: Recommendation;
|
||||
/** optional parenthetical — same shape as recommendation. */
|
||||
hedge?: Recommendation;
|
||||
/** contiguous breach window of the driving metric -> "(last 40 min)". */
|
||||
anomalyWindow?: { minutes: number; touchesEnd: boolean };
|
||||
/**
|
||||
* which dimension/key owns the problem, only when share >= threshold. `of` is the
|
||||
* denominator label the renderer prints (e.g. "pending" for flow, "failures" for execution)
|
||||
* — so it never mislabels a failures share as "% of pending".
|
||||
*/
|
||||
attribution?: { dim: string; key: string; share: number; of: string };
|
||||
/** ruled-out causes + evidence — rendered under the `read:` line ("not your code …"). */
|
||||
exclusions?: Exclusion[];
|
||||
/** supporting facts + evidence — rendered under the `read:` line after the exclusions. */
|
||||
observations?: Observation[];
|
||||
};
|
||||
|
||||
export type SummaryStatement = {
|
||||
findingType: string;
|
||||
severity: Severity;
|
||||
/**
|
||||
* Normally the statement renders from (findingType, severity). Exceptions carry a reason:
|
||||
* stale telemetry marks flow AND execution "unknown" -> "Flow/Execution unknown — data stale";
|
||||
* liveness with no signal is "freshness_unknown" -> "data freshness unknown".
|
||||
*/
|
||||
reason?: ReasonCode;
|
||||
};
|
||||
|
||||
export type ReportLink = {
|
||||
key: LinkKey;
|
||||
label: string;
|
||||
url: string;
|
||||
};
|
||||
|
||||
/** a.k.a. ReportDocument — report-agnostic, render-agnostic. */
|
||||
export type ReportViewModel = {
|
||||
/** "health" | "cost" | … */
|
||||
title: string;
|
||||
/** "prod" */
|
||||
scope: string;
|
||||
/** "last 1h" */
|
||||
period: string;
|
||||
/** "vs your 7d normal" */
|
||||
baselineLabel?: string;
|
||||
/** ISO string — passed in, never read from the clock inside interpret. */
|
||||
generatedAt: string;
|
||||
/** live window length in minutes — lets the renderer say "of last 60 min". */
|
||||
windowMinutes: number;
|
||||
|
||||
summary: {
|
||||
severity: Severity;
|
||||
statements: SummaryStatement[];
|
||||
};
|
||||
findings: Finding[];
|
||||
metrics: Metric[];
|
||||
/** dense structured payload for agents. */
|
||||
facts: Record<string, unknown>;
|
||||
links: ReportLink[];
|
||||
/** dominant finding's action + optional "do nothing" option. Max two entries. */
|
||||
footer: FooterEntry[];
|
||||
};
|
||||
|
||||
// ---------------------------------------------------------------------------
|
||||
// Interpret-side helpers (produce VM fields, no prose, no IO).
|
||||
// ---------------------------------------------------------------------------
|
||||
|
||||
/** Direction + rounded multiplier of `value` against a `normal` baseline. */
|
||||
/** Direction and rounded multiplier of `value` against a `normal` baseline. */
|
||||
export function delta(value: number, normal: number | undefined): Delta {
|
||||
if (normal === undefined || !Number.isFinite(normal) || normal === 0) {
|
||||
return { dir: "flat" };
|
||||
@@ -189,29 +68,37 @@ export function isOk(severity: Severity): boolean {
|
||||
}
|
||||
|
||||
/**
|
||||
* Trailing contiguous run of buckets that breach `threshold`, in minutes over
|
||||
* `windowMinutes`. Default breach is at/over threshold (ABOVE, e.g. concurrency
|
||||
* pinned at the limit); `below: true` counts at/under (BELOW, e.g. running capacity
|
||||
* idle under a stall floor). `touchesEnd` = the run reaches the latest bucket
|
||||
* ("(last 40 min)" vs mid-window "(14–16h)"). Undefined when nothing breaches.
|
||||
* Trailing contiguous run of buckets breaching `threshold`, in minutes. `below: true` counts at or
|
||||
* under it. `bucketMinutes` and `timestampsMs` make it gap-aware; without them the series is gap-free.
|
||||
*/
|
||||
export function anomalyWindow(
|
||||
series: number[],
|
||||
threshold: number,
|
||||
windowMinutes: number,
|
||||
options?: { below?: boolean }
|
||||
options?: { below?: boolean; bucketMinutes?: number; timestampsMs?: number[] }
|
||||
): { minutes: number; touchesEnd: boolean } | undefined {
|
||||
if (series.length === 0) return undefined;
|
||||
const perBucket = windowMinutes / series.length;
|
||||
const bucketMinutes = options?.bucketMinutes;
|
||||
const perBucket =
|
||||
bucketMinutes !== undefined && bucketMinutes > 0
|
||||
? bucketMinutes
|
||||
: windowMinutes / series.length;
|
||||
const breaches = options?.below ? (v: number) => v <= threshold : (v: number) => v >= threshold;
|
||||
// A gap larger than about one cadence is a dropped bucket and must not extend the run.
|
||||
const timestamps = options?.timestampsMs;
|
||||
const maxGapMs =
|
||||
timestamps && timestamps.length === series.length && bucketMinutes
|
||||
? bucketMinutes * 60_000 * 1.5
|
||||
: undefined;
|
||||
const adjacent = (i: number) =>
|
||||
maxGapMs === undefined || i === 0 || timestamps![i] - timestamps![i - 1] <= maxGapMs;
|
||||
|
||||
// longest breaching run + whether any run touches the end.
|
||||
let longest = 0;
|
||||
let current = 0;
|
||||
let touchesEnd = false;
|
||||
for (let i = 0; i < series.length; i++) {
|
||||
if (breaches(series[i])) {
|
||||
current++;
|
||||
current = adjacent(i) ? current + 1 : 1;
|
||||
longest = Math.max(longest, current);
|
||||
if (i === series.length - 1) touchesEnd = true;
|
||||
} else {
|
||||
@@ -219,8 +106,7 @@ export function anomalyWindow(
|
||||
}
|
||||
}
|
||||
if (longest === 0) return undefined;
|
||||
// If the run reaches the latest bucket, use the TRAILING length so "(last X min)" is
|
||||
// accurate — a longer mid-window run must not inflate it.
|
||||
// Use the trailing length when the run reaches the latest bucket, so a mid-window run can't inflate it.
|
||||
const runBuckets = touchesEnd ? current : longest;
|
||||
return { minutes: Math.round(runBuckets * perBucket), touchesEnd };
|
||||
}
|
||||
|
||||
@@ -0,0 +1,36 @@
|
||||
// Outside the route module because a Remix route may only export loader, action and headers.
|
||||
import { json } from "@remix-run/server-runtime";
|
||||
import { ReportFormatSchema, ReportPeriodSchema } from "@trigger.dev/core/v3/schemas";
|
||||
import { z } from "zod";
|
||||
import { renderReportAnsi, renderReportMarkdown } from "~/presenters/v3/reports/renderMarkdown";
|
||||
import { type ReportViewModel } from "~/presenters/v3/reports/report-view-model";
|
||||
|
||||
export const ReportParamsSchema = z.object({
|
||||
key: z.string(),
|
||||
});
|
||||
|
||||
// `period` and `format` come from core, the same definitions the API clients and CLI use.
|
||||
export const ReportSearchParamsSchema = z.object({
|
||||
period: ReportPeriodSchema.optional(),
|
||||
format: ReportFormatSchema.default("markdown"),
|
||||
});
|
||||
|
||||
export type ReportFormatParam = z.infer<typeof ReportFormatSchema>;
|
||||
|
||||
/** Render the view model in the requested encoding, with the matching content type. */
|
||||
export function reportResponse(vm: ReportViewModel, format: ReportFormatParam): Response {
|
||||
switch (format) {
|
||||
case "json":
|
||||
return json(vm, { status: 200 });
|
||||
case "ansi":
|
||||
return new Response(renderReportAnsi(vm), {
|
||||
status: 200,
|
||||
headers: { "Content-Type": "text/plain; charset=utf-8" },
|
||||
});
|
||||
case "markdown":
|
||||
return new Response(renderReportMarkdown(vm), {
|
||||
status: 200,
|
||||
headers: { "Content-Type": "text/markdown; charset=utf-8" },
|
||||
});
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,14 @@
|
||||
// Apart from `reportsApi.server.ts` so that rendering a report doesn't drag the route
|
||||
// builder — and `env.server` behind it — into everything that serializes one.
|
||||
import { isReportKey, reportQueryTables } from "~/presenters/v3/reports/report-registry";
|
||||
import { everyResource } from "~/services/routeBuilders/apiBuilder.server";
|
||||
|
||||
/**
|
||||
* Per-table, not the permissive `{ type: "query", id: "all" }`: a JWT must be scoped to every table
|
||||
* the report reads, so a partially scoped token can't reach the others.
|
||||
*/
|
||||
export function reportAuthResource(key: string) {
|
||||
// A key that names no report declares no tables, so there is nothing to authorize against.
|
||||
if (!isReportKey(key)) return everyResource([]);
|
||||
return everyResource(reportQueryTables(key).map((id) => ({ type: "query", id })));
|
||||
}
|
||||
@@ -75,6 +75,7 @@ export const loader = async ({ request }: LoaderFunctionArgs) => {
|
||||
const features = featuresForRequest(request);
|
||||
const timezone = await getTimezonePreference(request);
|
||||
|
||||
// Deprecated with `AskAI.tsx`: kept so the widget still has its config if it is ever remounted.
|
||||
const kapa = {
|
||||
websiteId: env.KAPA_AI_WEBSITE_ID,
|
||||
};
|
||||
@@ -170,13 +171,8 @@ export function ErrorBoundary() {
|
||||
}
|
||||
|
||||
export default function App() {
|
||||
const {
|
||||
posthogProjectKey,
|
||||
posthogUiHost,
|
||||
kapa: _kapa,
|
||||
themePreference,
|
||||
themeContrast,
|
||||
} = useTypedLoaderData<typeof loader>();
|
||||
const { posthogProjectKey, posthogUiHost, themePreference, themeContrast } =
|
||||
useTypedLoaderData<typeof loader>();
|
||||
usePostHog(posthogProjectKey, posthogUiHost);
|
||||
useSystemThemeSync(themePreference);
|
||||
// SSR falls back to dark for `system`; the inline script below corrects it
|
||||
|
||||
+5
@@ -189,6 +189,11 @@ export default function Page() {
|
||||
>
|
||||
Private connection docs
|
||||
</LinkButton>
|
||||
{hasPrivateNetworking && canAdd && (
|
||||
<LinkButton variant="primary/small" LeadingIcon={PlusIcon} to="new">
|
||||
Add Connection
|
||||
</LinkButton>
|
||||
)}
|
||||
</PageAccessories>
|
||||
</NavBar>
|
||||
<PageBody scrollable={true}>
|
||||
|
||||
+1
-11
@@ -20,7 +20,7 @@ import { Header2, Header3 } from "~/components/primitives/Headers";
|
||||
import { Input } from "~/components/primitives/Input";
|
||||
import { InputGroup } from "~/components/primitives/InputGroup";
|
||||
import { Label } from "~/components/primitives/Label";
|
||||
import { NavBar, PageAccessories, PageTitle } from "~/components/primitives/PageHeader";
|
||||
import { NavBar, PageTitle } from "~/components/primitives/PageHeader";
|
||||
import { Paragraph } from "~/components/primitives/Paragraph";
|
||||
import { Select, SelectItem } from "~/components/primitives/Select";
|
||||
import { prisma } from "~/db.server";
|
||||
@@ -38,7 +38,6 @@ import {
|
||||
} from "~/utils/pathBuilder";
|
||||
import {
|
||||
ArrowTopRightOnSquareIcon,
|
||||
BookOpenIcon,
|
||||
CommandLineIcon,
|
||||
DocumentTextIcon,
|
||||
PencilSquareIcon,
|
||||
@@ -553,15 +552,6 @@ export default function Page() {
|
||||
text: "Private Connections",
|
||||
}}
|
||||
/>
|
||||
<PageAccessories>
|
||||
<LinkButton
|
||||
variant="docs/small"
|
||||
LeadingIcon={BookOpenIcon}
|
||||
to={docsPath("private-networking/overview")}
|
||||
>
|
||||
Private connection docs
|
||||
</LinkButton>
|
||||
</PageAccessories>
|
||||
</NavBar>
|
||||
<PageBody scrollable={true}>
|
||||
<MainHorizontallyCenteredContainer className="max-w-3xl">
|
||||
|
||||
@@ -1,6 +1,6 @@
|
||||
import { getFormProps, getInputProps, useForm } from "@conform-to/react";
|
||||
import { parseWithZod } from "@conform-to/zod";
|
||||
import { BookOpenIcon, ShieldCheckIcon, TrashIcon } from "@heroicons/react/20/solid";
|
||||
import { ShieldCheckIcon, TrashIcon } from "@heroicons/react/20/solid";
|
||||
import { ShieldExclamationIcon } from "@heroicons/react/24/solid";
|
||||
import { DialogClose } from "@radix-ui/react-dialog";
|
||||
import { Form, useActionData, useFetcher } from "@remix-run/react";
|
||||
@@ -9,7 +9,7 @@ import { useState } from "react";
|
||||
import { typedjson, useTypedLoaderData } from "remix-typedjson";
|
||||
import { z } from "zod";
|
||||
import { PageBody, PageContainer } from "~/components/layout/AppLayout";
|
||||
import { Button, LinkButton } from "~/components/primitives/Buttons";
|
||||
import { Button } from "~/components/primitives/Buttons";
|
||||
import { Callout } from "~/components/primitives/Callout";
|
||||
import { ClipboardField } from "~/components/primitives/ClipboardField";
|
||||
import { DateTime } from "~/components/primitives/DateTime";
|
||||
@@ -46,7 +46,8 @@ import {
|
||||
revokePersonalAccessToken,
|
||||
} from "~/services/personalAccessToken.server";
|
||||
import { requireUserId } from "~/services/session.server";
|
||||
import { docsPath, personalAccessTokensPath } from "~/utils/pathBuilder";
|
||||
import { personalAccessTokensPath } from "~/utils/pathBuilder";
|
||||
|
||||
import { pageMeta } from "~/utils/pageTitle";
|
||||
|
||||
export const meta = pageMeta("Personal Access Tokens");
|
||||
@@ -246,13 +247,6 @@ export default function Page() {
|
||||
<NavBar>
|
||||
<PageTitle title="Personal Access Tokens" />
|
||||
<PageAccessories>
|
||||
<LinkButton
|
||||
LeadingIcon={BookOpenIcon}
|
||||
to={docsPath("management/overview#personal-access-token-pat")}
|
||||
variant="docs/small"
|
||||
>
|
||||
Personal Access Token docs
|
||||
</LinkButton>
|
||||
<Dialog>
|
||||
<DialogTrigger asChild>
|
||||
<Button variant="primary/small">Create new token…</Button>
|
||||
|
||||
@@ -9,7 +9,7 @@ import { rbac } from "~/services/rbac.server";
|
||||
// with the PAT. The default is short, but the ceiling allows long-lived tokens
|
||||
// for callers that need them (e.g. a long-running integration).
|
||||
const DEFAULT_UAT_TTL_SECONDS = 60 * 60; // 1 hour
|
||||
const MAX_UAT_TTL_SECONDS = 365 * 24 * 60 * 60; // 365 days
|
||||
const MAX_UAT_TTL_SECONDS = 7 * 24 * 60 * 60; // 7 days
|
||||
|
||||
// Mint a short-lived delegated user-actor token (`tr_uat_`) from a personal
|
||||
// access token. A UAT is a strict downgrade of the PAT: same user identity,
|
||||
@@ -68,6 +68,8 @@ export async function action({ request }: ActionFunctionArgs) {
|
||||
const token = await signUserActorToken(env.SESSION_SECRET, {
|
||||
userId: patAuth.userId,
|
||||
client: body.client ?? "personal-access-token",
|
||||
// Bind the token to its source PAT so revoking the PAT invalidates it.
|
||||
pat: patAuth.tokenId,
|
||||
cap: body.cap,
|
||||
// Absolute exp (seconds since epoch). jose treats a number as absolute.
|
||||
expirationTime: Math.floor(Date.now() / 1000) + ttlSeconds,
|
||||
|
||||
@@ -0,0 +1,40 @@
|
||||
import { json, type LoaderFunctionArgs } from "@remix-run/server-runtime";
|
||||
import { z } from "zod";
|
||||
import { orgAllowsDashboardAgentTurnEvals } from "~/services/dashboardAgentEvalPolicy.server";
|
||||
import { authenticateUatOrApiRequest } from "~/services/uatRoutePreamble.server";
|
||||
|
||||
/**
|
||||
* The gate the agent checks before it judges a turn. Only its own delegated user-actor
|
||||
* token is accepted, and the org is scoped to the token's user. Answers `false` rather than
|
||||
* an error whenever the setting can't be resolved, so the agent's fail-closed path is the
|
||||
* same for "off" and "unknown".
|
||||
*/
|
||||
|
||||
const QuerySchema = z.object({ organizationId: z.string().min(1) });
|
||||
|
||||
// obs-map-disable request-context -- the only call here that can throw catches its own failure
|
||||
// and logs it with organizationId, in dashboardAgentEvalPolicy.server.ts; the rest are early
|
||||
// returns, and the one failure that does reach the boundary is auth, where no tenant is known yet.
|
||||
export async function loader({ request }: LoaderFunctionArgs) {
|
||||
const authentication = await authenticateUatOrApiRequest(request);
|
||||
if (!authentication?.userActor) {
|
||||
return json({ error: "Invalid or missing access token" }, { status: 401 });
|
||||
}
|
||||
if (authentication.userActor.client !== "dashboard-agent") {
|
||||
return json({ error: "Not allowed", code: "forbidden_client" }, { status: 403 });
|
||||
}
|
||||
|
||||
const parsed = QuerySchema.safeParse(
|
||||
Object.fromEntries(new URL(request.url).searchParams.entries())
|
||||
);
|
||||
if (!parsed.success) {
|
||||
return json({ error: "organizationId is required" }, { status: 400 });
|
||||
}
|
||||
|
||||
const turnEvalsEnabled = await orgAllowsDashboardAgentTurnEvals({
|
||||
userId: authentication.userActor.userId,
|
||||
organizationId: parsed.data.organizationId,
|
||||
});
|
||||
|
||||
return json({ turnEvalsEnabled });
|
||||
}
|
||||
@@ -11,7 +11,9 @@ import {
|
||||
import { extractDomain, faviconUrl } from "~/utils/favicon";
|
||||
|
||||
// Identity-only: lists the caller's own orgs, so no authorization gate.
|
||||
export const loader = createLoaderPATApiRoute({}, async ({ authentication }) => {
|
||||
export const loader = createLoaderPATApiRoute(
|
||||
{ identityOnly: true },
|
||||
async ({ authentication }) => {
|
||||
const orgs = await prisma.organization.findMany({
|
||||
where: {
|
||||
deletedAt: null,
|
||||
@@ -35,20 +37,36 @@ export const loader = createLoaderPATApiRoute({}, async ({ authentication }) =>
|
||||
}));
|
||||
|
||||
return json(result);
|
||||
});
|
||||
}
|
||||
);
|
||||
|
||||
// No org exists yet, so no authorization gate; any authenticated user can
|
||||
// create an org and becomes its ADMIN.
|
||||
// No org exists yet, so there is nothing to scope a route-level gate to; any authenticated user
|
||||
// can create an org and becomes its ADMIN. A narrowly-capped delegated token (which cannot
|
||||
// `manage`) is still refused rather than inheriting its user's full reach — but only for
|
||||
// user-actor tokens, so an ordinary PAT is unaffected.
|
||||
export const action = createActionPATApiRoute(
|
||||
{
|
||||
method: "POST",
|
||||
body: CreateOrgRequestBody,
|
||||
},
|
||||
async ({ body, authentication }) => {
|
||||
async ({ body, authentication, ability }) => {
|
||||
if (env.ORG_CREATION_API_ENABLED !== "1") {
|
||||
return json({ error: "Not found" }, { status: 404 });
|
||||
}
|
||||
|
||||
// After the env gate: an install with the API disabled should 404, not 403.
|
||||
if (authentication.userActor && !ability.can("manage", { type: "organization" })) {
|
||||
return json(
|
||||
{
|
||||
error: "Unauthorized",
|
||||
code: "unauthorized",
|
||||
param: "access_token",
|
||||
type: "authorization",
|
||||
},
|
||||
{ status: 403 }
|
||||
);
|
||||
}
|
||||
|
||||
// Mirror the dashboard: stash companyUrl/companySize as onboarding data and
|
||||
// derive the org avatar from the company domain's favicon.
|
||||
const onboardingData: Record<string, string> = {};
|
||||
|
||||
@@ -1,6 +1,14 @@
|
||||
import { type ActionFunctionArgs, json } from "@remix-run/node";
|
||||
import { generateJWT as internal_generateJWT } from "@trigger.dev/core/v3";
|
||||
import { isUserActorToken, verifyUserActorToken } from "@trigger.dev/rbac";
|
||||
import {
|
||||
buildJwtAbility,
|
||||
CAPLESS_USER_ACTOR_SCOPES,
|
||||
isUserActorToken,
|
||||
scopesWithinAbility,
|
||||
verifyUserActorToken,
|
||||
type UserActorClaims,
|
||||
} from "@trigger.dev/rbac";
|
||||
import parseDuration from "parse-duration";
|
||||
import { z } from "zod";
|
||||
import {
|
||||
authenticatedEnvironmentForAuthentication,
|
||||
@@ -9,6 +17,8 @@ import {
|
||||
type AuthenticationResult,
|
||||
} from "~/services/apiAuth.server";
|
||||
import { env as appEnv } from "~/env.server";
|
||||
import { assertUserActorEnvironment } from "~/services/userActorEnvironment.server";
|
||||
import { assertSourcePatActive } from "~/services/personalAccessToken.server";
|
||||
import { logger } from "~/services/logger.server";
|
||||
import { authorizePatEnvironmentAccess } from "~/services/environmentVariableApiAccess.server";
|
||||
|
||||
@@ -26,6 +36,27 @@ const RequestBodySchema = z.object({
|
||||
expirationTime: z.union([z.number(), z.string()]).optional(),
|
||||
});
|
||||
|
||||
// A requested `expirationTime` above this (epoch seconds, ~2001) is an absolute
|
||||
// timestamp; a smaller number is a relative offset in seconds.
|
||||
const EXPIRY_EPOCH_THRESHOLD_SECONDS = 1_000_000_000;
|
||||
const DEFAULT_EXPIRY = "1h";
|
||||
|
||||
// Resolve the requested expiry to an absolute epoch-second timestamp so it can be
|
||||
// clamped against a delegated token's own expiry.
|
||||
function resolveRequestedExpirySeconds(
|
||||
expirationTime: number | string | undefined,
|
||||
nowSec: number
|
||||
): number {
|
||||
if (typeof expirationTime === "number") {
|
||||
return expirationTime > EXPIRY_EPOCH_THRESHOLD_SECONDS
|
||||
? expirationTime
|
||||
: nowSec + expirationTime;
|
||||
}
|
||||
const durationMs = parseDuration(expirationTime ?? DEFAULT_EXPIRY);
|
||||
const seconds = durationMs != null ? Math.floor(durationMs / 1000) : 60 * 60;
|
||||
return nowSec + seconds;
|
||||
}
|
||||
|
||||
export async function action({ request, params }: ActionFunctionArgs) {
|
||||
try {
|
||||
const bearer = request.headers
|
||||
@@ -41,14 +72,20 @@ export async function action({ request, params }: ActionFunctionArgs) {
|
||||
// minted env JWT below.
|
||||
let uatCap: string[] | undefined;
|
||||
let userActorId: string | undefined;
|
||||
let userActor: UserActorClaims | undefined;
|
||||
let authenticationResult: AuthenticationResult | undefined;
|
||||
if (isUat) {
|
||||
const claims = await verifyUserActorToken(appEnv.SESSION_SECRET, bearer!);
|
||||
if (!claims) {
|
||||
return json({ error: "Invalid or Missing Access Token" }, { status: 401 });
|
||||
}
|
||||
// A token minted from a PAT dies with it — the PAT must still be live.
|
||||
if (!(await assertSourcePatActive(claims))) {
|
||||
return json({ error: "Invalid or Missing Access Token" }, { status: 401 });
|
||||
}
|
||||
uatCap = claims.cap;
|
||||
userActorId = claims.userId;
|
||||
userActor = claims;
|
||||
// The env lookup keys purely on the user, identical to a PAT.
|
||||
authenticationResult = {
|
||||
type: "personalAccessToken",
|
||||
@@ -82,6 +119,9 @@ export async function action({ request, params }: ActionFunctionArgs) {
|
||||
triggerBranch
|
||||
);
|
||||
|
||||
// A user-actor token signed for one environment mints only for that one.
|
||||
assertUserActorEnvironment(userActor, runtimeEnv.id);
|
||||
|
||||
// This mints a JWT signed with the environment's secret key. For a PAT
|
||||
// (a user), gate it on env-tier read:apiKeys so a restricted role can't
|
||||
// obtain deployed-environment credentials (and therefore can't deploy).
|
||||
@@ -107,19 +147,25 @@ export async function action({ request, params }: ActionFunctionArgs) {
|
||||
|
||||
// The env JWT carries scopes only — downstream auth builds its ability
|
||||
// from them with no role context. So for a user-actor token we ceiling
|
||||
// the scopes by the token's own cap here (a read-only agent token can't
|
||||
// widen its grant through the exchange) and stamp the user via `act` so
|
||||
// the minted env JWT stays attributable. The cap is a ceiling, not a
|
||||
// replacement: intersect what the caller asked for with the cap (or use
|
||||
// the full cap if they asked for nothing). No cap → the request passes
|
||||
// through, same as a PAT.
|
||||
// the scopes here (a read-only agent token can't widen its grant through
|
||||
// the exchange) and stamp the user via `act` so the minted env JWT stays
|
||||
// attributable. A capless token's ceiling is read-only, never full access.
|
||||
// The ceiling is applied through the scope grammar, not literal membership:
|
||||
// `read:all` is a wildcard no literal request string equals, so a literal
|
||||
// filter against it would wrongly deny every read.
|
||||
const requestedScopes = parsedBody.data.claims?.scopes;
|
||||
const scopes =
|
||||
isUat && uatCap
|
||||
? requestedScopes && requestedScopes.length > 0
|
||||
? requestedScopes.filter((scope) => uatCap.includes(scope))
|
||||
: uatCap
|
||||
: requestedScopes;
|
||||
let scopes: string[] | undefined;
|
||||
if (isUat) {
|
||||
const ceiling = uatCap && uatCap.length > 0 ? uatCap : CAPLESS_USER_ACTOR_SCOPES;
|
||||
const ability = buildJwtAbility(ceiling);
|
||||
const scopeGranted = (scope: string) => scopesWithinAbility([scope], ability).ok;
|
||||
scopes =
|
||||
requestedScopes && requestedScopes.length > 0
|
||||
? requestedScopes.filter(scopeGranted)
|
||||
: ceiling;
|
||||
} else {
|
||||
scopes = requestedScopes;
|
||||
}
|
||||
|
||||
// Attribution: stamp the acting user on the minted env JWT. A UAT carries
|
||||
// its user as `userActorId`; a PAT exchange resolves the user from the
|
||||
@@ -136,13 +182,24 @@ export async function action({ request, params }: ActionFunctionArgs) {
|
||||
sub: runtimeEnv.id,
|
||||
pub: true,
|
||||
...(scopes ? { scopes } : {}),
|
||||
...(actorUserId ? { act: { sub: actorUserId } } : {}),
|
||||
...(actorUserId
|
||||
? { act: { sub: actorUserId, client: userActor?.client ?? "personal-access-token" } }
|
||||
: {}),
|
||||
};
|
||||
|
||||
// A delegated token can't mint a longer-lived JWT than itself: clamp the
|
||||
// requested expiry to the token's own `exp`. Non-UAT callers are unchanged.
|
||||
const nowSec = Math.floor(Date.now() / 1000);
|
||||
const requestedAbsSec = resolveRequestedExpirySeconds(parsedBody.data.expirationTime, nowSec);
|
||||
const expirationTime =
|
||||
isUat && userActor?.expiresAt !== undefined
|
||||
? Math.min(requestedAbsSec, userActor.expiresAt)
|
||||
: (parsedBody.data.expirationTime ?? DEFAULT_EXPIRY);
|
||||
|
||||
const jwt = await internal_generateJWT({
|
||||
secretKey: runtimeEnv.apiKey,
|
||||
payload: claims,
|
||||
expirationTime: parsedBody.data.expirationTime ?? "1h",
|
||||
expirationTime,
|
||||
});
|
||||
|
||||
return json({ token: jwt });
|
||||
|
||||
@@ -1,24 +1,20 @@
|
||||
import { json, type LoaderFunctionArgs } from "@remix-run/server-runtime";
|
||||
import { isUserActorToken, verifyUserActorToken } from "@trigger.dev/rbac";
|
||||
import { z } from "zod";
|
||||
import { env as $env } from "~/env.server";
|
||||
import {
|
||||
type AuthenticationResult,
|
||||
authenticatedEnvironmentForAuthentication,
|
||||
authenticateRequest,
|
||||
} from "~/services/apiAuth.server";
|
||||
import { authenticatedEnvironmentForAuthentication } from "~/services/apiAuth.server";
|
||||
import {
|
||||
resolveDashboardAgentRepoSnapshot,
|
||||
resolveRunCommit,
|
||||
} from "~/services/dashboardAgent.server";
|
||||
import { authorizePatEnvironmentAccess } from "~/services/environmentVariableApiAccess.server";
|
||||
import { logger } from "~/services/logger.server";
|
||||
import { authenticateUatOrApiRequest } from "~/services/uatRoutePreamble.server";
|
||||
|
||||
// Resolve a signed source-archive pointer for the project's connected repo, used
|
||||
// by the dashboard agent's code tools. With `?runId=run_...` it pins to the
|
||||
// commit that run's deployed version came from (run-SHA pinning); without it,
|
||||
// the tracked branch head. The GitHub token never leaves the server, only the
|
||||
// short-lived signed URL is returned. Auth mirrors the worker-by-tag route: a
|
||||
// delegated user-actor token authenticates as its user (identity-only).
|
||||
// short-lived signed URL is returned. A delegated user-actor token authenticates as its user and
|
||||
// is gated on that user's env-tier role, like the JWT exchange.
|
||||
|
||||
const ParamsSchema = z.object({
|
||||
projectRef: z.string(),
|
||||
@@ -27,23 +23,8 @@ const ParamsSchema = z.object({
|
||||
|
||||
export async function loader({ request, params }: LoaderFunctionArgs) {
|
||||
try {
|
||||
const bearer = request.headers
|
||||
.get("Authorization")
|
||||
?.replace(/^Bearer /, "")
|
||||
.trim();
|
||||
let authenticationResult: AuthenticationResult | undefined;
|
||||
if (bearer && isUserActorToken(bearer)) {
|
||||
const claims = await verifyUserActorToken($env.SESSION_SECRET, bearer);
|
||||
if (!claims) return json({ error: "Invalid or Missing Access Token" }, { status: 401 });
|
||||
authenticationResult = { type: "personalAccessToken", result: { userId: claims.userId } };
|
||||
} else {
|
||||
authenticationResult = await authenticateRequest(request, {
|
||||
personalAccessToken: true,
|
||||
organizationAccessToken: true,
|
||||
apiKey: false,
|
||||
});
|
||||
}
|
||||
if (!authenticationResult) {
|
||||
const authentication = await authenticateUatOrApiRequest(request);
|
||||
if (!authentication) {
|
||||
return json({ error: "Invalid or Missing Access Token" }, { status: 401 });
|
||||
}
|
||||
|
||||
@@ -53,12 +34,25 @@ export async function loader({ request, params }: LoaderFunctionArgs) {
|
||||
|
||||
const triggerBranch = request.headers.get("x-trigger-branch") ?? undefined;
|
||||
const runtimeEnv = await authenticatedEnvironmentForAuthentication(
|
||||
authenticationResult,
|
||||
authentication.authenticationResult,
|
||||
projectRef,
|
||||
env,
|
||||
triggerBranch
|
||||
);
|
||||
|
||||
// The signed URL exposes the project's whole source tree, so gate it like the environment's
|
||||
// other secrets: env-tier `read:apiKeys`, the same check the JWT exchange applies.
|
||||
const denied = await authorizePatEnvironmentAccess({
|
||||
request,
|
||||
authType: authentication.authenticationResult.type,
|
||||
organizationId: runtimeEnv.organizationId,
|
||||
projectId: runtimeEnv.project.id,
|
||||
envType: runtimeEnv.type,
|
||||
resource: "apiKeys",
|
||||
action: "read",
|
||||
});
|
||||
if (denied) return denied;
|
||||
|
||||
const runId = new URL(request.url).searchParams.get("runId") ?? undefined;
|
||||
|
||||
let ref: string | undefined;
|
||||
|
||||
@@ -0,0 +1,113 @@
|
||||
import { json, type LoaderFunctionArgs } from "@remix-run/server-runtime";
|
||||
import { z } from "zod";
|
||||
import { $replica } from "~/db.server";
|
||||
import {
|
||||
authenticatedEnvironmentForAuthentication,
|
||||
type AuthenticatedEnvironment,
|
||||
} from "~/services/apiAuth.server";
|
||||
import { resolveRunCommit } from "~/services/dashboardAgent.server";
|
||||
import { authorizePatEnvironmentAccess } from "~/services/environmentVariableApiAccess.server";
|
||||
import { logger } from "~/services/logger.server";
|
||||
import { authenticateUatOrApiRequest } from "~/services/uatRoutePreamble.server";
|
||||
|
||||
/** The commit a run's deployed version came from, plus that deployment's git metadata. */
|
||||
|
||||
const ParamsSchema = z.object({
|
||||
projectRef: z.string(),
|
||||
env: z.enum(["dev", "staging", "prod", "preview"]),
|
||||
runId: z.string(),
|
||||
});
|
||||
|
||||
type GitMetaBlob = {
|
||||
source?: string;
|
||||
commitAuthorName?: string;
|
||||
commitMessage?: string;
|
||||
commitRef?: string;
|
||||
remoteUrl?: string;
|
||||
ghUsername?: string;
|
||||
pullRequestNumber?: number;
|
||||
pullRequestTitle?: string;
|
||||
pullRequestState?: string;
|
||||
};
|
||||
|
||||
export async function loader({ request, params }: LoaderFunctionArgs) {
|
||||
// Hoisted so a failure below can name the tenant it happened to.
|
||||
let runtimeEnv: AuthenticatedEnvironment | undefined;
|
||||
|
||||
try {
|
||||
const authentication = await authenticateUatOrApiRequest(request);
|
||||
if (!authentication) {
|
||||
return json({ error: "Invalid or Missing Access Token" }, { status: 401 });
|
||||
}
|
||||
|
||||
const parsed = ParamsSchema.safeParse(params);
|
||||
if (!parsed.success) return json({ error: "Invalid Params" }, { status: 400 });
|
||||
const { projectRef, env, runId } = parsed.data;
|
||||
|
||||
const triggerBranch = request.headers.get("x-trigger-branch") ?? undefined;
|
||||
runtimeEnv = await authenticatedEnvironmentForAuthentication(
|
||||
authentication.authenticationResult,
|
||||
projectRef,
|
||||
env,
|
||||
triggerBranch
|
||||
);
|
||||
|
||||
// The answer is a deployment's git metadata, so it's gated like the deployments list.
|
||||
const denied = await authorizePatEnvironmentAccess({
|
||||
request,
|
||||
authType: authentication.authenticationResult.type,
|
||||
organizationId: runtimeEnv.organizationId,
|
||||
projectId: runtimeEnv.project.id,
|
||||
envType: runtimeEnv.type,
|
||||
resource: "deployments",
|
||||
action: "read",
|
||||
});
|
||||
if (denied) return denied;
|
||||
|
||||
const commit = await resolveRunCommit(runtimeEnv.id, runId);
|
||||
if (!commit) {
|
||||
return json(
|
||||
{ error: "That run has no deployed commit (it may be a dev run)." },
|
||||
{ status: 404 }
|
||||
);
|
||||
}
|
||||
|
||||
const deployment = await $replica.workerDeployment.findFirst({
|
||||
where: { environmentId: runtimeEnv.id, version: commit.version },
|
||||
select: { git: true, shortCode: true, deployedAt: true },
|
||||
});
|
||||
|
||||
const git = (deployment?.git ?? undefined) as GitMetaBlob | undefined;
|
||||
|
||||
return json({
|
||||
runId,
|
||||
version: commit.version,
|
||||
sha: commit.sha,
|
||||
dirty: commit.dirty,
|
||||
shortCode: deployment?.shortCode,
|
||||
deployedAt: deployment?.deployedAt ?? undefined,
|
||||
git: git
|
||||
? {
|
||||
source: git.source,
|
||||
commitMessage: git.commitMessage,
|
||||
commitAuthorName: git.commitAuthorName,
|
||||
commitRef: git.commitRef,
|
||||
remoteUrl: git.remoteUrl,
|
||||
ghUsername: git.ghUsername,
|
||||
pullRequestNumber: git.pullRequestNumber,
|
||||
pullRequestTitle: git.pullRequestTitle,
|
||||
pullRequestState: git.pullRequestState,
|
||||
}
|
||||
: undefined,
|
||||
});
|
||||
} catch (error) {
|
||||
if (error instanceof Response) throw error;
|
||||
logger.error("Failed to resolve run commit", {
|
||||
error,
|
||||
environmentId: runtimeEnv?.id,
|
||||
projectId: runtimeEnv?.project.id,
|
||||
organizationId: runtimeEnv?.organizationId,
|
||||
});
|
||||
return json({ error: "Internal Server Error" }, { status: 500 });
|
||||
}
|
||||
}
|
||||
@@ -1,16 +1,14 @@
|
||||
import { json, type LoaderFunctionArgs } from "@remix-run/server-runtime";
|
||||
import { type GetWorkerByTagResponse } from "@trigger.dev/core/v3/schemas";
|
||||
import { isUserActorToken, verifyUserActorToken } from "@trigger.dev/rbac";
|
||||
import { z } from "zod";
|
||||
import { $replica } from "~/db.server";
|
||||
import { env as $env } from "~/env.server";
|
||||
import {
|
||||
authenticatedEnvironmentForAuthentication,
|
||||
authenticateRequest,
|
||||
branchNameFromRequest,
|
||||
type AuthenticationResult,
|
||||
} from "~/services/apiAuth.server";
|
||||
import { logger } from "~/services/logger.server";
|
||||
import { authenticateUatOrApiRequest } from "~/services/uatRoutePreamble.server";
|
||||
import { v3RunsPath } from "~/utils/pathBuilder";
|
||||
import { findCurrentWorkerFromEnvironment } from "~/v3/models/workerDeployment.server";
|
||||
|
||||
@@ -24,32 +22,11 @@ type ParamsSchema = z.infer<typeof ParamsSchema>;
|
||||
|
||||
export async function loader({ request, params }: LoaderFunctionArgs) {
|
||||
try {
|
||||
// A delegated user-actor token authenticates as its user, like a PAT.
|
||||
// Resolve it here (the shared `authenticateRequest` deliberately doesn't
|
||||
// accept UATs) so the dashboard agent can list a project's deployed tasks
|
||||
// on the user's behalf. Identity-only, same as the PAT path below — there's
|
||||
// no ability check on this route, so the cap isn't enforced here (matches
|
||||
// PAT behavior).
|
||||
const bearer = request.headers
|
||||
.get("Authorization")
|
||||
?.replace(/^Bearer /, "")
|
||||
.trim();
|
||||
let authenticationResult: AuthenticationResult | undefined;
|
||||
if (bearer && isUserActorToken(bearer)) {
|
||||
const claims = await verifyUserActorToken($env.SESSION_SECRET, bearer);
|
||||
if (!claims) {
|
||||
return json({ error: "Invalid or Missing Access Token" }, { status: 401 });
|
||||
}
|
||||
authenticationResult = { type: "personalAccessToken", result: { userId: claims.userId } };
|
||||
} else {
|
||||
authenticationResult = await authenticateRequest(request, {
|
||||
personalAccessToken: true,
|
||||
organizationAccessToken: true,
|
||||
apiKey: false,
|
||||
});
|
||||
}
|
||||
// Accepts a user-actor token as well as a PAT. There's no ability check here, so the
|
||||
// token's cap isn't enforced (matches PAT behavior).
|
||||
const authentication = await authenticateUatOrApiRequest(request);
|
||||
|
||||
if (!authenticationResult) {
|
||||
if (!authentication) {
|
||||
return json({ error: "Invalid or Missing Access Token" }, { status: 401 });
|
||||
}
|
||||
|
||||
@@ -63,7 +40,7 @@ export async function loader({ request, params }: LoaderFunctionArgs) {
|
||||
const triggerBranch = branchNameFromRequest(request);
|
||||
|
||||
const runtimeEnv = await authenticatedEnvironmentForAuthentication(
|
||||
authenticationResult,
|
||||
authentication.authenticationResult,
|
||||
projectRef,
|
||||
env,
|
||||
triggerBranch
|
||||
|
||||
@@ -4,6 +4,7 @@ import { z } from "zod";
|
||||
import { $replica } from "~/db.server";
|
||||
import { findProjectByRef } from "~/models/project.server";
|
||||
import { createLoaderPATApiRoute } from "~/services/routeBuilders/apiBuilder.server";
|
||||
import { resolveUserActorEnvironmentScope } from "~/services/userActorEnvironment.server";
|
||||
import { sortEnvironments } from "~/utils/environmentSort";
|
||||
import { isBranchableEnvironment } from "~/utils/branchableEnvironment";
|
||||
|
||||
@@ -35,12 +36,18 @@ export const loader = createLoaderPATApiRoute(
|
||||
return json({ error: "Project not found" }, { status: 404 });
|
||||
}
|
||||
|
||||
// A delegated token signed for one environment only ever lists that one.
|
||||
const scope = await resolveUserActorEnvironmentScope(authentication.userActor, {
|
||||
projectId: project.id,
|
||||
});
|
||||
|
||||
const environments = await $replica.runtimeEnvironment.findMany({
|
||||
where: {
|
||||
projectId: project.id,
|
||||
// Only base/parent environments. Branch children (preview branches)
|
||||
// are excluded — syncs target the parent and branches override elsewhere.
|
||||
parentEnvironmentId: null,
|
||||
// A scoped token lists exactly the environment it was signed for, branch child or not —
|
||||
// otherwise a token minted on a preview branch would list nothing at all. Unscoped
|
||||
// callers get base/parent environments only: syncs target the parent.
|
||||
...(scope.scoped ? { id: scope.environmentId } : { parentEnvironmentId: null }),
|
||||
archivedAt: null,
|
||||
OR: [
|
||||
{ type: { in: ["STAGING", "PRODUCTION", "PREVIEW"] } },
|
||||
|
||||
@@ -7,6 +7,7 @@ import {
|
||||
ApiRunListSearchParams,
|
||||
} from "~/presenters/v3/ApiRunListPresenter.server";
|
||||
import { createLoaderPATApiRoute } from "~/services/routeBuilders/apiBuilder.server";
|
||||
import { resolveUserActorEnvironmentScope } from "~/services/userActorEnvironment.server";
|
||||
|
||||
const ParamsSchema = z.object({
|
||||
projectRef: z.string(),
|
||||
@@ -39,8 +40,20 @@ export const loader = createLoaderPATApiRoute(
|
||||
return json({ error: "Project not found" }, { status: 404 });
|
||||
}
|
||||
|
||||
// A delegated token signed for one environment only ever lists that environment's runs, and a
|
||||
// request filter naming another one is refused rather than overridden.
|
||||
const scope = await resolveUserActorEnvironmentScope(authentication.userActor, {
|
||||
projectId: project.id,
|
||||
requestedEnvironmentSlugs: searchParams["filter[env]"],
|
||||
});
|
||||
|
||||
const presenter = new ApiRunListPresenter();
|
||||
const result = await presenter.call(project, searchParams, apiVersion);
|
||||
const result = await presenter.call(
|
||||
project,
|
||||
searchParams,
|
||||
apiVersion,
|
||||
scope.scoped ? { id: scope.environmentId, organizationId: scope.organizationId } : undefined
|
||||
);
|
||||
|
||||
if (!result) {
|
||||
return json({ data: [] });
|
||||
|
||||
@@ -4,7 +4,9 @@ import { prisma } from "~/db.server";
|
||||
import { createLoaderPATApiRoute } from "~/services/routeBuilders/apiBuilder.server";
|
||||
|
||||
// Identity-only: lists projects across the caller's orgs, so no authorization gate.
|
||||
export const loader = createLoaderPATApiRoute({}, async ({ authentication }) => {
|
||||
export const loader = createLoaderPATApiRoute(
|
||||
{ identityOnly: true },
|
||||
async ({ authentication }) => {
|
||||
const projects = await prisma.project.findMany({
|
||||
where: {
|
||||
organization: {
|
||||
@@ -44,4 +46,5 @@ export const loader = createLoaderPATApiRoute({}, async ({ authentication }) =>
|
||||
}));
|
||||
|
||||
return json(result);
|
||||
});
|
||||
}
|
||||
);
|
||||
|
||||
@@ -7,6 +7,7 @@ import { logger } from "~/services/logger.server";
|
||||
import { rowsToCSV } from "~/utils/dataExport";
|
||||
import { detectQueryTables } from "~/v3/detectQueryTables";
|
||||
import { querySchemas } from "~/v3/querySchemas";
|
||||
import { queryScopeCeilingFor, resolveQueryScope } from "~/v3/queryScope";
|
||||
|
||||
const BodySchema = z.object({
|
||||
query: z.string(),
|
||||
@@ -51,11 +52,20 @@ const { action, loader } = createActionApiRoute(
|
||||
const { query, scope, period, from, to, format } = body;
|
||||
const env = authentication.environment;
|
||||
|
||||
// The credential's own scope is the ceiling, not the body's. See queryScope.ts.
|
||||
const resolvedScope = resolveQueryScope({
|
||||
ceiling: queryScopeCeilingFor(authentication.type),
|
||||
requested: scope as QueryScope,
|
||||
});
|
||||
if (!resolvedScope.ok) {
|
||||
return json({ error: resolvedScope.error }, { status: 403 });
|
||||
}
|
||||
|
||||
const queryResult = await executeQuery({
|
||||
name: "api-query",
|
||||
query,
|
||||
userAuthoredQuery: true,
|
||||
scope: scope as QueryScope,
|
||||
scope: resolvedScope.scope,
|
||||
organizationId: env.organization.id,
|
||||
projectId: env.project.id,
|
||||
environmentId: env.id,
|
||||
|
||||
@@ -0,0 +1,144 @@
|
||||
import { json } from "@remix-run/server-runtime";
|
||||
import { z } from "zod";
|
||||
import { clickhouseFactory } from "~/services/clickhouse/clickhouseFactoryInstance.server";
|
||||
import { logger } from "~/services/logger.server";
|
||||
import { createLoaderApiRoute } from "~/services/routeBuilders/apiBuilder.server";
|
||||
import { queueDepthSeries } from "~/v3/queueDepthSeries";
|
||||
|
||||
/**
|
||||
* Per-queue metrics over a window. `queueParam` is the queue name; `?type=task` (the default)
|
||||
* adds the `task/` prefix. An unknown queue returns zeroed metrics, not a 404.
|
||||
*/
|
||||
|
||||
const UNIT_MS: Record<string, number> = { s: 1e3, m: 6e4, h: 36e5, d: 864e5, w: 6048e5 };
|
||||
const MAX_PERIOD_MS = 7 * UNIT_MS.d;
|
||||
|
||||
const PeriodSchema = z
|
||||
.string()
|
||||
.regex(/^[1-9]\d*[smhdw]$/, "period must be a shorthand like '15m', '1h', or '24h'")
|
||||
.refine(
|
||||
(p) => Number(p.slice(0, -1)) * UNIT_MS[p.slice(-1)] <= MAX_PERIOD_MS,
|
||||
"period is too large (max 7d)"
|
||||
);
|
||||
|
||||
const SearchParamsSchema = z.object({
|
||||
type: z.enum(["task", "custom"]).default("task"),
|
||||
period: PeriodSchema.default("1h"),
|
||||
});
|
||||
|
||||
const TREND_POINTS = 12;
|
||||
|
||||
function periodMs(period: string): number {
|
||||
return Number(period.slice(0, -1)) * UNIT_MS[period.slice(-1)];
|
||||
}
|
||||
|
||||
function formatClickhouseDateTime(date: Date): string {
|
||||
return date.toISOString().slice(0, 19).replace("T", " ");
|
||||
}
|
||||
|
||||
function finiteOrNull(value: number | undefined): number | null {
|
||||
return typeof value === "number" && Number.isFinite(value) ? value : null;
|
||||
}
|
||||
|
||||
export const loader = createLoaderApiRoute(
|
||||
{
|
||||
params: z.object({
|
||||
queueParam: z.string().transform((val) => val.replace(/%2F/g, "/")),
|
||||
}),
|
||||
searchParams: SearchParamsSchema,
|
||||
allowJWT: true,
|
||||
corsStrategy: "none",
|
||||
findResource: async () => 1, // dummy — the queue name isn't resolved against Postgres
|
||||
authorization: {
|
||||
action: "read",
|
||||
resource: () => ({ type: "query", id: "queue_metrics" }),
|
||||
},
|
||||
},
|
||||
async ({ params, searchParams, authentication }) => {
|
||||
// Already decoded by Remix and the schema; decoding again would 500 on a literal "%".
|
||||
const name = params.queueParam;
|
||||
const queue = searchParams.type === "task" && !name.startsWith("task/") ? `task/${name}` : name;
|
||||
|
||||
const windowMs = periodMs(searchParams.period);
|
||||
const windowMinutes = windowMs / 60_000;
|
||||
const bucketSeconds = Math.max(60, Math.round(windowMs / 1000 / TREND_POINTS));
|
||||
// Snap both bounds to the bucket grid so repeated calls share ClickHouse cache entries.
|
||||
const bucketIntervalMs = bucketSeconds * 1000;
|
||||
const endMs = Math.ceil(Date.now() / bucketIntervalMs) * bucketIntervalMs;
|
||||
const startMs = endMs - windowMs;
|
||||
// The trend grid covers whole buckets, so a period that isn't a bucket multiple still lines up.
|
||||
const gridStartMs = Math.floor(startMs / bucketIntervalMs) * bucketIntervalMs;
|
||||
const numBuckets = Math.round((endMs - gridStartMs) / bucketIntervalMs);
|
||||
|
||||
try {
|
||||
const clickhouse = await clickhouseFactory.getClickhouseForOrganization(
|
||||
authentication.environment.organizationId,
|
||||
"query"
|
||||
);
|
||||
|
||||
const ids = {
|
||||
organizationId: authentication.environment.organizationId,
|
||||
projectId: authentication.environment.projectId,
|
||||
environmentId: authentication.environment.id,
|
||||
queueNames: [queue],
|
||||
startTime: formatClickhouseDateTime(new Date(startMs)),
|
||||
endTime: formatClickhouseDateTime(new Date(endMs)),
|
||||
};
|
||||
|
||||
const [summaryResult, trendResult] = await Promise.all([
|
||||
clickhouse.queueMetrics.listSummary(ids),
|
||||
clickhouse.queueMetrics.depthSparklines({ ...ids, bucketSeconds }),
|
||||
]);
|
||||
|
||||
const [summaryError, summaryRows] = summaryResult;
|
||||
const [trendError, trendRows] = trendResult;
|
||||
|
||||
if (summaryError || trendError) {
|
||||
logger.warn("Failed to read queue metrics", {
|
||||
summaryError: summaryError?.message,
|
||||
trendError: trendError?.message,
|
||||
organizationId: ids.organizationId,
|
||||
projectId: ids.projectId,
|
||||
environmentId: ids.environmentId,
|
||||
});
|
||||
return json({ error: "Queue metrics are unavailable right now." }, { status: 503 });
|
||||
}
|
||||
|
||||
const summary = summaryRows?.[0];
|
||||
const startedCount = summary?.started_count ?? 0;
|
||||
|
||||
return json({
|
||||
queue,
|
||||
period: searchParams.period,
|
||||
from: new Date(startMs).toISOString(),
|
||||
to: new Date(endMs).toISOString(),
|
||||
waitMs: {
|
||||
p50: finiteOrNull(summary?.p50_wait_ms),
|
||||
p95: finiteOrNull(summary?.p95_wait_ms),
|
||||
},
|
||||
peakQueued: summary?.peak_queued ?? 0,
|
||||
startedCount,
|
||||
startedPerMin: Number((startedCount / windowMinutes).toFixed(2)),
|
||||
throttledCount: summary?.throttled_count ?? 0,
|
||||
bucketIntervalMs,
|
||||
// Oldest first, one point per bucket: a bucket with no sample carries the previous depth.
|
||||
depthTrend: queueDepthSeries(trendRows ?? [], {
|
||||
startMs: gridStartMs,
|
||||
bucketIntervalMs,
|
||||
numBuckets,
|
||||
}).depth,
|
||||
});
|
||||
} catch (error) {
|
||||
// Rethrow Responses: swallowing one would turn it into a 500.
|
||||
if (error instanceof Response) throw error;
|
||||
logger.error("Failed to read queue metrics", {
|
||||
error,
|
||||
queue,
|
||||
organizationId: authentication.environment.organizationId,
|
||||
projectId: authentication.environment.projectId,
|
||||
environmentId: authentication.environment.id,
|
||||
});
|
||||
return json({ error: "Something went wrong, please try again." }, { status: 500 });
|
||||
}
|
||||
}
|
||||
);
|
||||
@@ -1,53 +1,28 @@
|
||||
import { json } from "@remix-run/server-runtime";
|
||||
import { z } from "zod";
|
||||
import { ReportPresenter } from "~/presenters/v3/reports/ReportPresenter.server";
|
||||
import { isReportKey, REPORT_KEYS } from "~/presenters/v3/reports/report-registry";
|
||||
import { renderReportAnsi, renderReportMarkdown } from "~/presenters/v3/reports/renderMarkdown";
|
||||
import {
|
||||
ReportParamsSchema,
|
||||
ReportSearchParamsSchema,
|
||||
reportResponse,
|
||||
} from "~/presenters/v3/reports/reportsApi.server";
|
||||
import { reportAuthResource } from "~/presenters/v3/reports/reportsApiAuth.server";
|
||||
import { logger } from "~/services/logger.server";
|
||||
import { createLoaderApiRoute, everyResource } from "~/services/routeBuilders/apiBuilder.server";
|
||||
|
||||
const ParamsSchema = z.object({
|
||||
key: z.string(),
|
||||
});
|
||||
|
||||
/**
|
||||
* The query tables the reports read. Authorize per-table (like api.v1.query.ts) rather than
|
||||
* the permissive `{ type: "query", id: "all" }`: a JWT must be scoped to every table a report
|
||||
* touches, so a token scoped to only some tables can't fetch a report that reads others. This
|
||||
* is the union across reports; `health` reads all three.
|
||||
*/
|
||||
const REPORT_QUERY_TABLES = ["runs", "env_metrics", "queue_metrics"] as const;
|
||||
|
||||
/** Canonical shorthand ("1h" / "30m" / "7d") with an upper bound, so the public API rejects
|
||||
* garbage and absurd ranges (e.g. "999999999d") itself rather than relying on downstream clip. */
|
||||
const UNIT_MS: Record<string, number> = { s: 1e3, m: 6e4, h: 36e5, d: 864e5, w: 6048e5 };
|
||||
const MAX_PERIOD_MS = 90 * UNIT_MS.d;
|
||||
const PeriodSchema = z
|
||||
.string()
|
||||
.regex(/^[1-9]\d*[smhdw]$/, "period must be a shorthand like '1h', '30m', or '7d'")
|
||||
.refine(
|
||||
(p) => Number(p.slice(0, -1)) * UNIT_MS[p.slice(-1)] <= MAX_PERIOD_MS,
|
||||
"period is too large (max 90d)"
|
||||
);
|
||||
|
||||
const SearchParamsSchema = z.object({
|
||||
period: PeriodSchema.optional(),
|
||||
// markdown (default) for CLI/MCP · json (the raw VM) for web · ansi for a colour terminal.
|
||||
format: z.enum(["markdown", "json", "ansi"]).default("markdown"),
|
||||
});
|
||||
import { createLoaderApiRoute } from "~/services/routeBuilders/apiBuilder.server";
|
||||
|
||||
// Only `loader` may be exported here; the vite build flags server code reachable from a
|
||||
// non-loader export. Schemas and helpers live in reportsApi.server.ts.
|
||||
export const loader = createLoaderApiRoute(
|
||||
{
|
||||
params: ParamsSchema,
|
||||
searchParams: SearchParamsSchema,
|
||||
params: ReportParamsSchema,
|
||||
searchParams: ReportSearchParamsSchema,
|
||||
// The MCP `get_report` tool calls this with a scoped JWT (read:query), so JWT auth
|
||||
// must be allowed — same as api.v1.query.ts.
|
||||
allowJWT: true,
|
||||
findResource: async () => 1, // dummy — report key validated in the handler
|
||||
authorization: {
|
||||
action: "read",
|
||||
// Per-table, not `id: "all"`: a JWT must be scoped to every query table the report reads.
|
||||
resource: () => everyResource(REPORT_QUERY_TABLES.map((id) => ({ type: "query", id }))),
|
||||
resource: (_, params) => reportAuthResource(params.key),
|
||||
},
|
||||
},
|
||||
async ({ params, searchParams, authentication }) => {
|
||||
@@ -70,21 +45,7 @@ export const loader = createLoaderApiRoute(
|
||||
return json({ error: `Unknown report "${params.key}".` }, { status: 404 });
|
||||
}
|
||||
|
||||
if (searchParams.format === "json") {
|
||||
return json(vm, { status: 200 });
|
||||
}
|
||||
|
||||
if (searchParams.format === "ansi") {
|
||||
return new Response(renderReportAnsi(vm), {
|
||||
status: 200,
|
||||
headers: { "Content-Type": "text/plain; charset=utf-8" },
|
||||
});
|
||||
}
|
||||
|
||||
return new Response(renderReportMarkdown(vm), {
|
||||
status: 200,
|
||||
headers: { "Content-Type": "text/markdown; charset=utf-8" },
|
||||
});
|
||||
return reportResponse(vm, searchParams.format);
|
||||
} catch (error) {
|
||||
logger.error("Failed to render report", { error, key: params.key });
|
||||
return json({ error: "Something went wrong, please try again." }, { status: 500 });
|
||||
|
||||
@@ -38,14 +38,15 @@ export async function loader({ params, request }: LoaderFunctionArgs) {
|
||||
const query = url.searchParams.get("q");
|
||||
|
||||
if (!query) {
|
||||
return new Response("No query", { status: 404 });
|
||||
return new Response("No query", { status: 400 });
|
||||
}
|
||||
|
||||
const newUrl = new URL(
|
||||
v3EnvironmentPath({ slug: project.organization.slug }, { slug: project.slug }, { slug: "dev" }),
|
||||
env.LOGIN_ORIGIN
|
||||
);
|
||||
newUrl.searchParams.set("aiHelp", query);
|
||||
// The `ask` param is picked up in the environment layout (`useDashboardAgentOpenRequests`).
|
||||
newUrl.searchParams.set("ask", query);
|
||||
|
||||
return redirect(newUrl.toString());
|
||||
}
|
||||
|
||||
+93
-42
@@ -1,29 +1,28 @@
|
||||
import { type ActionFunctionArgs, json } from "@remix-run/server-runtime";
|
||||
import { $replica } from "~/db.server";
|
||||
import {
|
||||
checkMessageParts,
|
||||
declaredBodyBytes,
|
||||
exceedsMessageBodyBytes,
|
||||
MAX_MESSAGE_BODY_BYTES,
|
||||
MESSAGE_TOO_LARGE_CODE,
|
||||
MESSAGE_TOO_LARGE_ERROR,
|
||||
} from "~/components/dashboard-agent/message-limits";
|
||||
import { findProjectBySlug } from "~/models/project.server";
|
||||
import { findEnvironmentBySlug } from "~/models/runtimeEnvironment.server";
|
||||
import {
|
||||
dashboardAgentApiOrigin,
|
||||
mintDashboardAgentUserActorToken,
|
||||
resolveDashboardAgentRepoSnapshot,
|
||||
} from "~/services/dashboardAgent.server";
|
||||
import { dashboardAgentEnvironmentAddress } from "~/services/dashboardAgentEnvironmentAddress.server";
|
||||
import { logger } from "~/services/logger.server";
|
||||
import { requireUser } from "~/services/session.server";
|
||||
import { readBoundedBodyText } from "~/utils/boundedRequestBody.server";
|
||||
import { EnvironmentParamSchema } from "~/utils/pathBuilder";
|
||||
import { canAccessDashboardAgent } from "~/v3/canAccessDashboardAgent.server";
|
||||
|
||||
// Same-origin proxy for the chat "in"/append request. The transport routes the
|
||||
// `in` endpoint here (and the `out` SSE stream direct to the Trigger API), so
|
||||
// every turn passes through the dashboard's own session before reaching the
|
||||
// agent. We use that hop to mint a fresh read-only delegated token for the
|
||||
// signed-in user and inject it into the turn's metadata server-side. The token
|
||||
// reaches the agent without ever touching the browser, and minting stays tied
|
||||
// to the user's own session (no shared-secret backdoor).
|
||||
//
|
||||
// The append body is `{ kind, payload: { metadata, ... } }`; we add the token
|
||||
// (plus the API origin and the server-vouched project ref + env) to
|
||||
// `payload.metadata`. Only `kind === "message"` turns carry metadata — stop
|
||||
// chunks pass through untouched. We forward only the headers the API needs and
|
||||
// deliberately drop the dashboard session cookie.
|
||||
// Same-origin proxy for the chat append request. It mints a read-only delegated token scoped
|
||||
// to the environment in this URL, so the token never reaches the browser.
|
||||
|
||||
const FORWARDED_HEADERS = [
|
||||
"authorization",
|
||||
@@ -33,16 +32,24 @@ const FORWARDED_HEADERS = [
|
||||
"x-trigger-branch",
|
||||
];
|
||||
|
||||
// The API's env routes key on the canonical env name (dev/staging/prod/preview),
|
||||
// not the dashboard URL slug (e.g. staging's slug is "stg"). Map from the env
|
||||
// type so the agent's tools address the right environment. Preview branches
|
||||
// aren't threaded yet (they'd need the branch on every tool call) — a follow-up.
|
||||
const ENV_NAME_BY_TYPE: Record<string, string> = {
|
||||
DEVELOPMENT: "dev",
|
||||
STAGING: "staging",
|
||||
PRODUCTION: "prod",
|
||||
PREVIEW: "preview",
|
||||
};
|
||||
// The only turn metadata a browser may set: everything else the agent reads is injected
|
||||
// server-side. A whitelist — a new clientData field is server-owned until listed here on purpose.
|
||||
const CLIENT_METADATA_KEYS = ["currentPage", "pageContext"] as const;
|
||||
|
||||
export function pickAgentClientMetadata(
|
||||
metadata: Record<string, unknown> | undefined
|
||||
): Record<string, unknown> {
|
||||
const picked: Record<string, unknown> = {};
|
||||
if (!metadata) return picked;
|
||||
for (const key of CLIENT_METADATA_KEYS) {
|
||||
if (metadata[key] !== undefined) picked[key] = metadata[key];
|
||||
}
|
||||
return picked;
|
||||
}
|
||||
|
||||
function tooLarge() {
|
||||
return json({ error: MESSAGE_TOO_LARGE_ERROR, code: MESSAGE_TOO_LARGE_CODE }, { status: 413 });
|
||||
}
|
||||
|
||||
export async function action({ request, params }: ActionFunctionArgs) {
|
||||
const user = await requireUser(request);
|
||||
@@ -59,6 +66,12 @@ export async function action({ request, params }: ActionFunctionArgs) {
|
||||
return json({ error: "Not found" }, { status: 404 });
|
||||
}
|
||||
|
||||
// The declared size is refused before any lookup. It is advisory, so the read below is
|
||||
// bounded too: without it a chunked body would be buffered whole before being refused.
|
||||
if (exceedsMessageBodyBytes(declaredBodyBytes(request.headers))) {
|
||||
return tooLarge();
|
||||
}
|
||||
|
||||
const project = await findProjectBySlug(organizationSlug, projectParam, user.id);
|
||||
if (!project) return json({ error: "Project not found" }, { status: 404 });
|
||||
|
||||
@@ -71,38 +84,76 @@ export async function action({ request, params }: ActionFunctionArgs) {
|
||||
const url = new URL(request.url);
|
||||
const upstreamUrl = `${apiOrigin.replace(/\/$/, "")}/${upstreamPath}${url.search}`;
|
||||
|
||||
// Resolve the dashboard env slug to the canonical API env name its tools use.
|
||||
const runtimeEnv = await $replica.runtimeEnvironment.findFirst({
|
||||
where: { projectId: project.id, slug: envParam },
|
||||
select: { type: true },
|
||||
});
|
||||
const environmentName = runtimeEnv ? ENV_NAME_BY_TYPE[runtimeEnv.type] : undefined;
|
||||
// Membership-scoped: `(projectId, slug)` is not unique because every developer has their own
|
||||
// dev row, and a token must never be minted for someone else's environment — or for none.
|
||||
const runtimeEnv = await findEnvironmentBySlug(project.id, envParam, user.id);
|
||||
if (!runtimeEnv) return json({ error: "Environment not found" }, { status: 404 });
|
||||
const environmentAddress = dashboardAgentEnvironmentAddress(runtimeEnv);
|
||||
|
||||
// When the project has a connected GitHub repo, resolve a signed source-archive
|
||||
// pointer (code mode). Null otherwise -> the agent stays in assistant mode.
|
||||
// Null without a connected GitHub repo, and the agent stays in assistant mode.
|
||||
const repoSnapshot = await resolveDashboardAgentRepoSnapshot(project.id);
|
||||
|
||||
// Inject the delegated token + context into the turn's metadata.
|
||||
const raw = await request.text();
|
||||
const read = await readBoundedBodyText(request, MAX_MESSAGE_BODY_BYTES);
|
||||
if (!read.ok) return tooLarge();
|
||||
|
||||
const raw = read.text;
|
||||
let body = raw;
|
||||
try {
|
||||
const parsed = JSON.parse(raw) as {
|
||||
type AgentTurn = {
|
||||
kind?: string;
|
||||
payload?: { metadata?: Record<string, unknown> };
|
||||
payload?: {
|
||||
trigger?: string;
|
||||
metadata?: Record<string, unknown>;
|
||||
message?: { parts?: unknown };
|
||||
};
|
||||
};
|
||||
let parsed: AgentTurn | undefined;
|
||||
// Only the parse is tolerated: non-JSON is forwarded unchanged rather than break the turn.
|
||||
// Everything after it must fail loudly — a swallowed mint would forward with no credential.
|
||||
try {
|
||||
parsed = JSON.parse(raw) as AgentTurn;
|
||||
} catch {
|
||||
parsed = undefined;
|
||||
}
|
||||
|
||||
if (parsed) {
|
||||
// Actions are placed by the server only, and this proxy is the one path a browser
|
||||
// can reach `.in` through.
|
||||
if (parsed.payload?.trigger === "action") {
|
||||
return json({ error: "Not allowed" }, { status: 403 });
|
||||
}
|
||||
if (parsed.kind === "message" && parsed.payload) {
|
||||
// A body under the byte cap can still be one huge part or hundreds of small ones.
|
||||
if (checkMessageParts(parsed.payload.message?.parts) !== null) {
|
||||
return tooLarge();
|
||||
}
|
||||
|
||||
let userActorToken: string;
|
||||
try {
|
||||
userActorToken = await mintDashboardAgentUserActorToken(user.id, {
|
||||
environmentId: runtimeEnv.id,
|
||||
});
|
||||
} catch (error) {
|
||||
logger.error("Dashboard agent in-proxy could not mint a token", { error, upstreamPath });
|
||||
return json({ error: "The dashboard agent couldn't send that message." }, { status: 500 });
|
||||
}
|
||||
|
||||
parsed.payload.metadata = {
|
||||
...(parsed.payload.metadata ?? {}),
|
||||
userActorToken: await mintDashboardAgentUserActorToken(user.id),
|
||||
...pickAgentClientMetadata(parsed.payload.metadata),
|
||||
userActorToken,
|
||||
apiOrigin,
|
||||
projectRef: project.externalRef,
|
||||
environmentName,
|
||||
// Server-owned: the eval opt-out and every tenancy check key on these.
|
||||
organizationId: project.organizationId,
|
||||
userId: user.id,
|
||||
projectId: project.id,
|
||||
// `(projectId, slug)` isn't unique (dev is per-member), so anything addressing
|
||||
// one environment row uses this id. The address is for name-addressed tools.
|
||||
environmentId: runtimeEnv.id,
|
||||
...environmentAddress,
|
||||
...(repoSnapshot ? { repoSnapshot } : {}),
|
||||
};
|
||||
body = JSON.stringify(parsed);
|
||||
}
|
||||
} catch {
|
||||
// Non-JSON or unexpected shape — forward unchanged rather than break the turn.
|
||||
}
|
||||
|
||||
const headers = new Headers();
|
||||
|
||||
+262
-53
@@ -1,9 +1,11 @@
|
||||
import { json, type ActionFunctionArgs, type LoaderFunctionArgs } from "@remix-run/server-runtime";
|
||||
import {
|
||||
chatExists,
|
||||
countUserMessages,
|
||||
createChat,
|
||||
getChatMessages,
|
||||
getSession,
|
||||
listChatIdsWithOpenInvestigations,
|
||||
listChats,
|
||||
renameChat,
|
||||
setChatPinned,
|
||||
@@ -12,9 +14,18 @@ import {
|
||||
import { generateFriendlyId } from "@trigger.dev/core/v3/isomorphic";
|
||||
import type { UIMessage } from "ai";
|
||||
import { z } from "zod";
|
||||
import {
|
||||
checkMessageParts,
|
||||
declaredBodyBytes,
|
||||
exceedsMessageBodyBytes,
|
||||
MESSAGE_TOO_LARGE_CODE,
|
||||
MESSAGE_TOO_LARGE_ERROR,
|
||||
} from "~/components/dashboard-agent/message-limits";
|
||||
import { MAX_URIS_PER_RESOLVE_REQUEST } from "~/components/dashboard-agent/resolve-uris";
|
||||
import { $replica } from "~/db.server";
|
||||
import { env } from "~/env.server";
|
||||
import { findProjectBySlug } from "~/models/project.server";
|
||||
import { findEnvironmentBySlug } from "~/models/runtimeEnvironment.server";
|
||||
import {
|
||||
dashboardAgentApiOrigin,
|
||||
isDashboardAgentConfigured,
|
||||
@@ -23,23 +34,29 @@ import {
|
||||
resolveDashboardAgentRepoSnapshot,
|
||||
startDashboardAgentSession,
|
||||
} from "~/services/dashboardAgent.server";
|
||||
import { dashboardAgentEnvironmentAddress } from "~/services/dashboardAgentEnvironmentAddress.server";
|
||||
import { startDashboardAgentHeadStart } from "~/services/dashboardAgentHeadStart.server";
|
||||
import { dashboardAgentDb } from "~/services/dashboardAgentDb.server";
|
||||
import { logger } from "~/services/logger.server";
|
||||
import { resolveTriggerUri } from "~/services/resolveTriggerUri.server";
|
||||
import { requireUser } from "~/services/session.server";
|
||||
import { EnvironmentParamSchema } from "~/utils/pathBuilder";
|
||||
import { canAccessDashboardAgent } from "~/v3/canAccessDashboardAgent.server";
|
||||
|
||||
// The agent's tools address the canonical env name, not the dashboard URL slug.
|
||||
const ENV_NAME_BY_TYPE: Record<string, string> = {
|
||||
DEVELOPMENT: "dev",
|
||||
STAGING: "staging",
|
||||
PRODUCTION: "prod",
|
||||
PREVIEW: "preview",
|
||||
};
|
||||
// The client-metadata whitelist lives with the `in` proxy, the other mint site, so the two cannot
|
||||
// drift apart.
|
||||
import { pickAgentClientMetadata } from "./resources.orgs.$organizationSlug.projects.$projectParam.env.$envParam.dashboard-agent.in.$";
|
||||
|
||||
const ActionBody = z.object({
|
||||
intent: z.enum(["start", "create", "token", "rename", "pin", "delete"]),
|
||||
intent: z.enum([
|
||||
"start",
|
||||
"create",
|
||||
"token",
|
||||
"rename",
|
||||
"pin",
|
||||
"delete",
|
||||
"resolve",
|
||||
"resolve-many",
|
||||
]),
|
||||
// Omitted for `create` (the server generates it); required for the rest.
|
||||
chatId: z.string().min(1).optional(),
|
||||
// The first user message (JSON UIMessage), for `create`.
|
||||
@@ -47,9 +64,14 @@ const ActionBody = z.object({
|
||||
clientData: z.string().optional(),
|
||||
title: z.string().optional(),
|
||||
pinned: z.enum(["true", "false"]).optional(),
|
||||
// A `trigger://` URI, for `resolve`.
|
||||
uri: z.string().optional(),
|
||||
// A JSON array of `trigger://` URIs, for `resolve-many`.
|
||||
uris: z.string().optional(),
|
||||
});
|
||||
|
||||
// History list, or — with ?chatId= — the stored transcript + session for resume.
|
||||
// History list by default. `?chatId=` returns the stored transcript plus session,
|
||||
// `?quota=1` the message count.
|
||||
export const loader = async ({ request, params }: LoaderFunctionArgs) => {
|
||||
const user = await requireUser(request);
|
||||
const userId = user.id;
|
||||
@@ -66,14 +88,27 @@ export const loader = async ({ request, params }: LoaderFunctionArgs) => {
|
||||
return json({ error: "Not found" }, { status: 404 });
|
||||
}
|
||||
|
||||
const searchParams = new URL(request.url).searchParams;
|
||||
|
||||
const project = await findProjectBySlug(organizationSlug, projectParam, userId);
|
||||
if (!project) return json({ error: "Project not found" }, { status: 404 });
|
||||
|
||||
const chatId = new URL(request.url).searchParams.get("chatId");
|
||||
// The open chat is excluded and counted from the live transcript instead, so an
|
||||
// unpersisted turn still counts against the cap.
|
||||
if (searchParams.get("quota") === "1") {
|
||||
const used = await countUserMessages(dashboardAgentDb, {
|
||||
organizationId: project.organizationId,
|
||||
userId,
|
||||
excludeChatId: searchParams.get("chatId") ?? undefined,
|
||||
});
|
||||
return json({ used });
|
||||
}
|
||||
|
||||
const chatId = searchParams.get("chatId");
|
||||
if (chatId) {
|
||||
const [messages, session] = await Promise.all([
|
||||
getChatMessages(dashboardAgentDb, { chatId, userId }),
|
||||
getSession(dashboardAgentDb, { chatId, userId }),
|
||||
getChatMessages(dashboardAgentDb, { chatId, userId, organizationId: project.organizationId }),
|
||||
getSession(dashboardAgentDb, { chatId, userId, organizationId: project.organizationId }),
|
||||
]);
|
||||
return json({ messages: messages ?? [], session });
|
||||
}
|
||||
@@ -82,9 +117,39 @@ export const loader = async ({ request, params }: LoaderFunctionArgs) => {
|
||||
organizationId: project.organizationId,
|
||||
userId,
|
||||
});
|
||||
return json({ chats });
|
||||
|
||||
// One query for all the listed chats, never one per row.
|
||||
const investigatingChatIds = await listChatIdsWithOpenInvestigations(dashboardAgentDb, {
|
||||
organizationId: project.organizationId,
|
||||
userId,
|
||||
});
|
||||
|
||||
return json({
|
||||
chats: chats.map((chat) => ({
|
||||
...chat,
|
||||
hasOpenInvestigation: investigatingChatIds.has(chat.id),
|
||||
})),
|
||||
});
|
||||
};
|
||||
|
||||
/** Only a source URI needs the connected repository, so a batch without one skips the read. */
|
||||
async function findRepositoryForSourceUris(projectId: string, uris: string[]) {
|
||||
if (!uris.some((uri) => uri.includes("/source/"))) return null;
|
||||
|
||||
const connected = await $replica.connectedGithubRepository.findFirst({
|
||||
where: {
|
||||
projectId,
|
||||
repository: { installation: { deletedAt: null, suspendedAt: null } },
|
||||
},
|
||||
select: { repository: { select: { fullName: true } } },
|
||||
});
|
||||
return connected?.repository ?? null;
|
||||
}
|
||||
|
||||
function messageTooLarge() {
|
||||
return json({ error: MESSAGE_TOO_LARGE_ERROR, code: MESSAGE_TOO_LARGE_CODE }, { status: 413 });
|
||||
}
|
||||
|
||||
export const action = async ({ request, params }: ActionFunctionArgs) => {
|
||||
const user = await requireUser(request);
|
||||
const userId = user.id;
|
||||
@@ -101,16 +166,19 @@ export const action = async ({ request, params }: ActionFunctionArgs) => {
|
||||
return json({ error: "Not found" }, { status: 404 });
|
||||
}
|
||||
|
||||
// A declared oversize is refused here, before any lookup. Without a content-length the
|
||||
// ingress cap has already ended the request mid-stream, so this never sees it.
|
||||
if (exceedsMessageBodyBytes(declaredBodyBytes(request.headers))) {
|
||||
return messageTooLarge();
|
||||
}
|
||||
|
||||
const project = await findProjectBySlug(organizationSlug, projectParam, userId);
|
||||
if (!project) return json({ error: "Project not found" }, { status: 404 });
|
||||
|
||||
const parsed = ActionBody.safeParse(Object.fromEntries(await request.formData()));
|
||||
if (!parsed.success) return json({ error: "Invalid request" }, { status: 400 });
|
||||
|
||||
// Create a new chat: the SERVER generates the id and owns the chat record, so
|
||||
// a client can never name another user's chat. Kicks off the first turn (head
|
||||
// start when configured, else a cold session) and returns the id + token. The
|
||||
// client mounts with that id and resumes the stream.
|
||||
// The server generates the chat id, so a client can never name another user's chat.
|
||||
if (parsed.data.intent === "create") {
|
||||
if (!isDashboardAgentConfigured()) {
|
||||
return json({ error: "The dashboard agent is not configured." }, { status: 501 });
|
||||
@@ -126,6 +194,14 @@ export const action = async ({ request, params }: ActionFunctionArgs) => {
|
||||
}
|
||||
if (!firstMessage) return json({ error: "message is required" }, { status: 400 });
|
||||
|
||||
// A body under the byte cap can still be one huge part or hundreds of small ones.
|
||||
if (
|
||||
exceedsMessageBodyBytes(Buffer.byteLength(parsed.data.message ?? "", "utf8")) ||
|
||||
checkMessageParts(firstMessage.parts) !== null
|
||||
) {
|
||||
return messageTooLarge();
|
||||
}
|
||||
|
||||
let clientData: Record<string, unknown> | undefined;
|
||||
try {
|
||||
clientData = parsed.data.clientData
|
||||
@@ -134,50 +210,108 @@ export const action = async ({ request, params }: ActionFunctionArgs) => {
|
||||
} catch {
|
||||
/* invalid JSON — create without context metadata */
|
||||
}
|
||||
// Only the whitelisted page context survives; the rest is injected below.
|
||||
const clientContext = pickAgentClientMetadata(clientData);
|
||||
|
||||
// Membership-scoped: dev rows are per-developer, so a token must never be minted for
|
||||
// someone else's environment — or, when nothing resolves, for no environment at all.
|
||||
const runtimeEnv = await findEnvironmentBySlug(project.id, envParam, userId);
|
||||
if (!runtimeEnv) return json({ error: "Environment not found" }, { status: 404 });
|
||||
const environmentAddress = dashboardAgentEnvironmentAddress(runtimeEnv);
|
||||
|
||||
const chatId = generateFriendlyId("chat");
|
||||
try {
|
||||
const repoSnapshot = await resolveDashboardAgentRepoSnapshot(project.id);
|
||||
const headStarted = Boolean(env.ANTHROPIC_API_KEY);
|
||||
|
||||
// The lookups and the mint all run before the chat row exists, so a failure here can't
|
||||
// leave an empty chat behind in the user's history.
|
||||
const headStartMetadata = headStarted
|
||||
? {
|
||||
// The agent validates run metadata against its clientDataSchema, so the
|
||||
// per-turn client context must accompany the injected auth and context fields.
|
||||
...clientContext,
|
||||
userActorToken: await mintDashboardAgentUserActorToken(userId, {
|
||||
environmentId: runtimeEnv.id,
|
||||
}),
|
||||
apiOrigin: dashboardAgentApiOrigin(),
|
||||
projectRef: project.externalRef,
|
||||
// Server-owned, like the `in` proxy: the eval opt-out and every tenancy check
|
||||
// key on these, so the client can't set them at all.
|
||||
organizationId: project.organizationId,
|
||||
userId,
|
||||
projectId: project.id,
|
||||
// Same environment identity the `in` proxy injects.
|
||||
environmentId: runtimeEnv.id,
|
||||
...environmentAddress,
|
||||
...(repoSnapshot ? { repoSnapshot } : {}),
|
||||
}
|
||||
: undefined;
|
||||
|
||||
await createChat(dashboardAgentDb, {
|
||||
id: chatId,
|
||||
organizationId: project.organizationId,
|
||||
userId,
|
||||
...(clientData ? { metadata: { context: clientData } } : {}),
|
||||
...(clientData ? { metadata: { context: clientContext } } : {}),
|
||||
});
|
||||
|
||||
const runtimeEnv = await $replica.runtimeEnvironment.findFirst({
|
||||
where: { projectId: project.id, slug: envParam },
|
||||
select: { type: true },
|
||||
});
|
||||
const environmentName = runtimeEnv ? ENV_NAME_BY_TYPE[runtimeEnv.type] : undefined;
|
||||
const repoSnapshot = await resolveDashboardAgentRepoSnapshot(project.id);
|
||||
|
||||
const headStarted = Boolean(env.ANTHROPIC_API_KEY);
|
||||
if (headStarted) {
|
||||
// Head start runs the warm step-1 with this first message and injects the
|
||||
// delegated token + context into the run's payload server-side.
|
||||
try {
|
||||
if (headStartMetadata) {
|
||||
// Injects the delegated token and context into the run's payload server-side.
|
||||
await startDashboardAgentHeadStart({
|
||||
chatId,
|
||||
messages: [firstMessage],
|
||||
mode: repoSnapshot ? "code" : "assistant",
|
||||
metadata: {
|
||||
// The agent validates the run metadata against its clientDataSchema
|
||||
// (userId, organizationId, …), so the per-turn clientData has to be
|
||||
// present alongside the injected auth/context fields.
|
||||
...(clientData ?? {}),
|
||||
userActorToken: await mintDashboardAgentUserActorToken(userId),
|
||||
apiOrigin: dashboardAgentApiOrigin(),
|
||||
projectRef: project.externalRef,
|
||||
environmentName,
|
||||
...(repoSnapshot ? { repoSnapshot } : {}),
|
||||
},
|
||||
metadata: headStartMetadata,
|
||||
});
|
||||
} else {
|
||||
// Cold start: create the session (preload); the client sends the first
|
||||
// message through the transport, where the `in` proxy injects the token.
|
||||
await startDashboardAgentSession({ chatId, clientData });
|
||||
// Cold start: the client sends the first message through the `in` proxy, which
|
||||
// injects the token.
|
||||
// Same server-owned identity the head-start path injects; the `in` proxy adds the
|
||||
// delegated token on the first turn.
|
||||
await startDashboardAgentSession({
|
||||
chatId,
|
||||
clientData: {
|
||||
...clientContext,
|
||||
organizationId: project.organizationId,
|
||||
userId,
|
||||
projectId: project.id,
|
||||
environmentId: runtimeEnv.id,
|
||||
...environmentAddress,
|
||||
},
|
||||
});
|
||||
}
|
||||
} catch (error) {
|
||||
// Both starts are one create-session-and-trigger round trip, so a rejection means no
|
||||
// handover was dispatched and no message was sent: a session the call did create in
|
||||
// spite of the error idles out having done nothing. The empty row is all there is to undo.
|
||||
// Swallowed so the start's own error is what surfaces and gets logged.
|
||||
await softDeleteChat(dashboardAgentDb, {
|
||||
chatId,
|
||||
userId,
|
||||
organizationId: project.organizationId,
|
||||
}).catch((cleanupError) => {
|
||||
logger.error("Failed to remove a dashboard agent chat whose start failed", {
|
||||
chatId,
|
||||
error: cleanupError,
|
||||
});
|
||||
});
|
||||
throw error;
|
||||
}
|
||||
|
||||
const publicAccessToken = await mintDashboardAgentToken(chatId);
|
||||
let publicAccessToken: string;
|
||||
try {
|
||||
publicAccessToken = await mintDashboardAgentToken(chatId);
|
||||
} catch (error) {
|
||||
// The start resolved, so the session is live and a head start is already streaming into
|
||||
// it. Deleting the chat here would hide a running agent; the client can ask for a token
|
||||
// again through the `token` intent.
|
||||
logger.error("Dashboard agent chat started but its token mint failed", { chatId, error });
|
||||
return json(
|
||||
{ error: "The dashboard agent started but couldn't be opened. Try opening it again." },
|
||||
{ status: 500 }
|
||||
);
|
||||
}
|
||||
return json({ chatId, publicAccessToken, headStarted });
|
||||
} catch (error) {
|
||||
logger.error("Failed to create dashboard agent chat", { chatId, error });
|
||||
@@ -188,6 +322,64 @@ export const action = async ({ request, params }: ActionFunctionArgs) => {
|
||||
}
|
||||
}
|
||||
|
||||
// Scoped by the environment in the URL: the resolver refuses a URI naming a
|
||||
// different project or environment.
|
||||
if (parsed.data.intent === "resolve") {
|
||||
const uri = parsed.data.uri;
|
||||
if (!uri) return json({ error: "uri is required" }, { status: 400 });
|
||||
|
||||
const environment = await findEnvironmentBySlug(project.id, envParam, userId);
|
||||
if (!environment) return json({ error: "Environment not found" }, { status: 404 });
|
||||
|
||||
const repository = await findRepositoryForSourceUris(project.id, [uri]);
|
||||
|
||||
const resolved = resolveTriggerUri({ ...environment, repository }, uri);
|
||||
if (!resolved) return json({ error: "Nothing to open for that link" }, { status: 404 });
|
||||
|
||||
return json({
|
||||
path: resolved.url,
|
||||
label: resolved.label,
|
||||
external: resolved.external ?? false,
|
||||
});
|
||||
}
|
||||
|
||||
// The card's citations in one request: one environment lookup and one repo lookup for the
|
||||
// whole batch, same environment scope as `resolve`.
|
||||
if (parsed.data.intent === "resolve-many") {
|
||||
let uris: string[];
|
||||
try {
|
||||
const list = JSON.parse(parsed.data.uris ?? "") as unknown;
|
||||
if (!Array.isArray(list) || list.some((uri) => typeof uri !== "string")) {
|
||||
return json({ error: "uris is required" }, { status: 400 });
|
||||
}
|
||||
uris = [...new Set(list as string[])];
|
||||
} catch {
|
||||
return json({ error: "uris is required" }, { status: 400 });
|
||||
}
|
||||
|
||||
if (uris.length === 0) return json({ error: "uris is required" }, { status: 400 });
|
||||
if (uris.length > MAX_URIS_PER_RESOLVE_REQUEST) {
|
||||
return json({ error: "Too many links in one request" }, { status: 400 });
|
||||
}
|
||||
|
||||
const environment = await findEnvironmentBySlug(project.id, envParam, userId);
|
||||
if (!environment) return json({ error: "Environment not found" }, { status: 404 });
|
||||
|
||||
const repository = await findRepositoryForSourceUris(project.id, uris);
|
||||
const scope = { ...environment, repository };
|
||||
|
||||
// A null entry is the definitive "nothing to open": the client caches it.
|
||||
const resolved: Record<string, { path: string; label: string; external: boolean } | null> = {};
|
||||
for (const uri of uris) {
|
||||
const hit = resolveTriggerUri(scope, uri);
|
||||
resolved[uri] = hit
|
||||
? { path: hit.url, label: hit.label, external: hit.external ?? false }
|
||||
: null;
|
||||
}
|
||||
|
||||
return json({ resolved });
|
||||
}
|
||||
|
||||
const { intent, chatId } = parsed.data;
|
||||
if (!chatId) return json({ error: "chatId is required" }, { status: 400 });
|
||||
|
||||
@@ -196,10 +388,7 @@ export const action = async ({ request, params }: ActionFunctionArgs) => {
|
||||
if (!isDashboardAgentConfigured()) {
|
||||
return json({ error: "The dashboard agent is not configured." }, { status: 501 });
|
||||
}
|
||||
// Resume-only: new chats are created via the `create` intent (server-owned
|
||||
// id). The transport falls back here to re-establish a session for an
|
||||
// existing chat (e.g. after its token expired), so verify ownership before
|
||||
// issuing one — a client-supplied chatId must belong to the caller.
|
||||
// Resume only, so a client-supplied chatId is checked against the caller first.
|
||||
if (
|
||||
!(await chatExists(dashboardAgentDb, {
|
||||
chatId,
|
||||
@@ -233,8 +422,7 @@ export const action = async ({ request, params }: ActionFunctionArgs) => {
|
||||
if (!isDashboardAgentConfigured()) {
|
||||
return json({ error: "The dashboard agent is not configured." }, { status: 501 });
|
||||
}
|
||||
// Only mint a session token for a chat the caller owns, so a client-supplied
|
||||
// chatId can't be used to get a token for someone else's session.
|
||||
// Only mint a token for a chat the caller owns.
|
||||
if (
|
||||
!(await chatExists(dashboardAgentDb, {
|
||||
chatId,
|
||||
@@ -249,7 +437,12 @@ export const action = async ({ request, params }: ActionFunctionArgs) => {
|
||||
|
||||
case "rename": {
|
||||
if (!parsed.data.title) return json({ error: "title is required" }, { status: 400 });
|
||||
await renameChat(dashboardAgentDb, { chatId, userId, title: parsed.data.title });
|
||||
await renameChat(dashboardAgentDb, {
|
||||
chatId,
|
||||
userId,
|
||||
organizationId: project.organizationId,
|
||||
title: parsed.data.title,
|
||||
});
|
||||
return json({ ok: true });
|
||||
}
|
||||
|
||||
@@ -257,13 +450,29 @@ export const action = async ({ request, params }: ActionFunctionArgs) => {
|
||||
await setChatPinned(dashboardAgentDb, {
|
||||
chatId,
|
||||
userId,
|
||||
organizationId: project.organizationId,
|
||||
pinned: parsed.data.pinned === "true",
|
||||
});
|
||||
return json({ ok: true });
|
||||
}
|
||||
|
||||
case "delete": {
|
||||
await softDeleteChat(dashboardAgentDb, { chatId, userId });
|
||||
// Existence check gives a 404 for a chat this caller can't see; the delete itself
|
||||
// is org- and owner-scoped too.
|
||||
if (
|
||||
!(await chatExists(dashboardAgentDb, {
|
||||
chatId,
|
||||
userId,
|
||||
organizationId: project.organizationId,
|
||||
}))
|
||||
) {
|
||||
return json({ error: "Chat not found" }, { status: 404 });
|
||||
}
|
||||
await softDeleteChat(dashboardAgentDb, {
|
||||
chatId,
|
||||
userId,
|
||||
organizationId: project.organizationId,
|
||||
});
|
||||
return json({ ok: true });
|
||||
}
|
||||
}
|
||||
|
||||
@@ -89,9 +89,9 @@ function DotMatrixTab() {
|
||||
<div className="flex flex-wrap items-center gap-6 rounded-md border border-grid-bright bg-background-bright px-6 py-5">
|
||||
{(
|
||||
[
|
||||
["ask-ai/small", 16, "small"],
|
||||
["ask-ai/medium", 16, "medium"],
|
||||
["ask-ai/large", 20, "large"],
|
||||
["ask-trigger/small", 16, "small"],
|
||||
["ask-trigger/medium", 16, "medium"],
|
||||
["ask-trigger/large", 20, "large"],
|
||||
] as [ButtonVariant, number, string][]
|
||||
).map(([variant, matrixSize, label]) => (
|
||||
<div key={variant} className="flex flex-col items-center gap-2">
|
||||
@@ -106,7 +106,7 @@ function DotMatrixTab() {
|
||||
<div className="flex flex-wrap items-center gap-6 rounded-md border border-grid-bright bg-background-bright px-6 py-5">
|
||||
{[14, 15, 16].map((s) => (
|
||||
<div key={s} className="flex flex-col items-center gap-2">
|
||||
<AskAiButton variant="ask-ai/small" matrixSize={s} />
|
||||
<AskAiButton variant="ask-trigger/small" matrixSize={s} />
|
||||
<div className="text-[10px] uppercase tracking-wide text-text-dimmed">{s}px icon</div>
|
||||
</div>
|
||||
))}
|
||||
@@ -231,7 +231,7 @@ function FaceButton({ name }: { name: DotShapeName }) {
|
||||
|
||||
return (
|
||||
<Button
|
||||
variant="ask-ai/small"
|
||||
variant="ask-trigger/small"
|
||||
onClick={trigger}
|
||||
LeadingIcon={
|
||||
<AgentDotMatrix
|
||||
|
||||
@@ -399,18 +399,18 @@ export default function Story() {
|
||||
</Button>
|
||||
</div>
|
||||
</div>
|
||||
<Header1 className="mb-2 mt-8">Ask AI button</Header1>
|
||||
<Header1 className="mb-2 mt-8">Ask Trigger button</Header1>
|
||||
<div className="grid grid-cols-4 gap-8 border-b border-grid-bright pb-8">
|
||||
<div className="flex flex-col items-start gap-2">
|
||||
<Header3 className="mb-1 uppercase">Sizes</Header3>
|
||||
<Button variant="ask-ai/small">Ask AI</Button>
|
||||
<Button variant="ask-ai/medium">Ask AI</Button>
|
||||
<Button variant="ask-ai/large">Ask AI</Button>
|
||||
<Button variant="ask-trigger/small">Ask Trigger</Button>
|
||||
<Button variant="ask-trigger/medium">Ask Trigger</Button>
|
||||
<Button variant="ask-trigger/large">Ask Trigger</Button>
|
||||
</div>
|
||||
<div className="flex flex-col items-start gap-2">
|
||||
<Header3 className="mb-1 uppercase">Disabled</Header3>
|
||||
<Button variant="ask-ai/small" disabled>
|
||||
Ask AI
|
||||
<Button variant="ask-trigger/small" disabled>
|
||||
Ask Trigger
|
||||
</Button>
|
||||
</div>
|
||||
<div className="col-span-2 self-end text-sm text-text-dimmed">
|
||||
|
||||
@@ -13,7 +13,8 @@ import {
|
||||
findEnvironmentByPublicApiKey,
|
||||
toAuthenticated,
|
||||
} from "~/models/runtimeEnvironment.server";
|
||||
import type { RbacAbility, RbacResource } from "@trigger.dev/rbac";
|
||||
import type { RbacAbility, RbacResource, UserActorClaims } from "@trigger.dev/rbac";
|
||||
import { assertUserActorEnvironment } from "./userActorEnvironment.server";
|
||||
import { type RuntimeEnvironmentForEnvRepo } from "~/v3/environmentVariables/environmentVariablesRepository.server";
|
||||
import { logger } from "./logger.server";
|
||||
import { safeEnvironmentLogFields } from "./safeEnvironmentLog";
|
||||
@@ -44,6 +45,13 @@ const ClaimsSchema = z.object({
|
||||
skipColumns: z.array(z.string()).optional(),
|
||||
})
|
||||
.optional(),
|
||||
// Identity only. Authorization comes from `sub` and `scopes`, never from `act`.
|
||||
act: z
|
||||
.object({
|
||||
sub: z.string(),
|
||||
client: z.string().optional(),
|
||||
})
|
||||
.optional(),
|
||||
});
|
||||
|
||||
// Re-export the slim shape defined in @trigger.dev/core. Single source of
|
||||
@@ -74,6 +82,7 @@ export type ApiAuthenticationResultSuccess = {
|
||||
// API keys (no user) and JWTs minted without delegation.
|
||||
actor?: {
|
||||
sub: string;
|
||||
client?: string;
|
||||
};
|
||||
};
|
||||
|
||||
@@ -187,6 +196,7 @@ export async function authenticateApiKey(
|
||||
environment: validationResults.environment,
|
||||
oneTimeUse: parsedClaims.success ? parsedClaims.data.otu : false,
|
||||
realtime: parsedClaims.success ? parsedClaims.data.realtime : undefined,
|
||||
actor: parsedClaims.success ? parsedClaims.data.act : undefined,
|
||||
};
|
||||
}
|
||||
}
|
||||
@@ -279,6 +289,7 @@ async function authenticateApiKeyWithFailure(
|
||||
environment: validationResults.environment,
|
||||
oneTimeUse: parsedClaims.success ? parsedClaims.data.otu : false,
|
||||
realtime: parsedClaims.success ? parsedClaims.data.realtime : undefined,
|
||||
actor: parsedClaims.success ? parsedClaims.data.act : undefined,
|
||||
};
|
||||
}
|
||||
}
|
||||
@@ -394,10 +405,22 @@ function getApiKeyResult(apiKey: string): {
|
||||
return { apiKey, type };
|
||||
}
|
||||
|
||||
/**
|
||||
* The authenticated user-actor. A user-actor token authenticates as its user, so it is the same
|
||||
* shape a PAT authenticates to — that shape now carries the token's verified claims itself, so
|
||||
* any layer holding the actor holds its environment scope.
|
||||
*/
|
||||
export type UserActorAuthenticatedActor = PersonalAccessTokenAuthenticationResult;
|
||||
|
||||
export type AuthenticationResult =
|
||||
| {
|
||||
type: "personalAccessToken";
|
||||
result: PersonalAccessTokenAuthenticationResult;
|
||||
result: UserActorAuthenticatedActor;
|
||||
/**
|
||||
* Claims of the delegated user-actor token the caller presented, if any. A UAT authenticates
|
||||
* as its user, so it rides on this variant; its environment scope is enforced on resolution.
|
||||
*/
|
||||
userActor?: UserActorClaims;
|
||||
}
|
||||
| {
|
||||
type: "organizationAccessToken";
|
||||
@@ -522,11 +545,34 @@ export async function authenticateRequest<
|
||||
return;
|
||||
}
|
||||
|
||||
/**
|
||||
* Resolve the environment a request targets, and enforce the caller's environment scope.
|
||||
*
|
||||
* Every route that turns an authentication result into an environment goes through here, so the
|
||||
* user-actor token's `environmentId` claim is checked once, at the seam — a new endpoint can't
|
||||
* forget it.
|
||||
*/
|
||||
export async function authenticatedEnvironmentForAuthentication(
|
||||
auth: AuthenticationResult,
|
||||
projectRef: string,
|
||||
slug: string,
|
||||
branch?: string
|
||||
): Promise<AuthenticatedEnvironment> {
|
||||
const environment = await resolveEnvironmentForAuthentication(auth, projectRef, slug, branch);
|
||||
|
||||
if (auth.type === "personalAccessToken") {
|
||||
// Either place the claims ride: on the actor (the shape every layer keeps) or beside it.
|
||||
assertUserActorEnvironment(auth.result.userActor ?? auth.userActor, environment.id);
|
||||
}
|
||||
|
||||
return environment;
|
||||
}
|
||||
|
||||
async function resolveEnvironmentForAuthentication(
|
||||
auth: AuthenticationResult,
|
||||
projectRef: string,
|
||||
slug: string,
|
||||
branch?: string
|
||||
): Promise<AuthenticatedEnvironment> {
|
||||
if (slug === "staging") {
|
||||
slug = "stg";
|
||||
|
||||
@@ -9,28 +9,20 @@ import type { Duration } from "./rateLimiter.server";
|
||||
|
||||
const BATCH_STREAM_ITEMS_PATH = /^\/api\/v3\/batches\/([^/]+)\/items$/;
|
||||
|
||||
export const apiRateLimiter = authorizationRateLimitMiddleware({
|
||||
redis: {
|
||||
port: env.RATE_LIMIT_REDIS_PORT,
|
||||
host: env.RATE_LIMIT_REDIS_HOST,
|
||||
username: env.RATE_LIMIT_REDIS_USERNAME,
|
||||
password: env.RATE_LIMIT_REDIS_PASSWORD,
|
||||
tlsDisabled: env.RATE_LIMIT_REDIS_TLS_DISABLED === "true",
|
||||
clusterMode: env.RATE_LIMIT_REDIS_CLUSTER_MODE_ENABLED === "1",
|
||||
},
|
||||
keyPrefix: "api",
|
||||
defaultLimiter: {
|
||||
type: "tokenBucket",
|
||||
refillRate: env.API_RATE_LIMIT_REFILL_RATE,
|
||||
interval: env.API_RATE_LIMIT_REFILL_INTERVAL as Duration,
|
||||
maxTokens: env.API_RATE_LIMIT_MAX,
|
||||
},
|
||||
limiterCache: {
|
||||
fresh: 60_000 * 10, // Data is fresh for 10 minutes
|
||||
stale: 60_000 * 20, // Date is stale after 20 minutes
|
||||
maxItems: 1000,
|
||||
},
|
||||
limiterConfigOverride: async (authorizationValue) => {
|
||||
// Rate-limit key for a delegated (agent/PAT-minted) JWT. Its token value rotates every
|
||||
// turn, so keying on the token would hand each turn a fresh bucket. Key on env+acting-user
|
||||
// so the agent's traffic shares one bucket across turns. The `jwt-actor:` prefix keeps it
|
||||
// off PRIVATE-key buckets, which key on the bare environment id.
|
||||
export function jwtActorRateLimitIdentifier(environmentId: string, actorSub: string): string {
|
||||
return `jwt-actor:${environmentId}:${actorSub}`;
|
||||
}
|
||||
|
||||
// The per-request bucket decision for the API limiter. Exported so the branch below
|
||||
// (a delegated JWT keys on env+acting-user, everything else keeps its prior key) is
|
||||
// testable without standing up the middleware and its Redis.
|
||||
export async function resolveApiRateLimitOverride(
|
||||
authorizationValue: string
|
||||
): Promise<{ config?: unknown; identifier?: string } | undefined> {
|
||||
const rawApiKey = authorizationValue.replace(/^Bearer /, "");
|
||||
|
||||
if (rawApiKey.startsWith("tr_")) {
|
||||
@@ -56,21 +48,56 @@ export const apiRateLimiter = authorizationRateLimitMiddleware({
|
||||
}
|
||||
|
||||
if (authenticatedEnv.type === "PUBLIC_JWT") {
|
||||
return {
|
||||
config: {
|
||||
const config = {
|
||||
type: "fixedWindow",
|
||||
window: env.API_RATE_LIMIT_JWT_WINDOW,
|
||||
tokens: env.API_RATE_LIMIT_JWT_TOKENS,
|
||||
},
|
||||
} as const;
|
||||
|
||||
// A delegated JWT (agent/PAT-minted) shares one bucket per env+acting-user across turns.
|
||||
// A browser realtime JWT carries no `act`, so it keeps the hashed-token fallback.
|
||||
if (authenticatedEnv.actor?.sub) {
|
||||
return {
|
||||
config,
|
||||
identifier: jwtActorRateLimitIdentifier(
|
||||
authenticatedEnv.environment.id,
|
||||
authenticatedEnv.actor.sub
|
||||
),
|
||||
};
|
||||
}
|
||||
|
||||
return { config };
|
||||
}
|
||||
|
||||
return {
|
||||
config: authenticatedEnv.environment.organization.apiRateLimiterConfig,
|
||||
// Public keys are browser-distributed, so keep them on per-key buckets.
|
||||
identifier: authenticatedEnv.type === "PRIVATE" ? authenticatedEnv.environment.id : undefined,
|
||||
};
|
||||
}
|
||||
|
||||
export const apiRateLimiter = authorizationRateLimitMiddleware({
|
||||
redis: {
|
||||
port: env.RATE_LIMIT_REDIS_PORT,
|
||||
host: env.RATE_LIMIT_REDIS_HOST,
|
||||
username: env.RATE_LIMIT_REDIS_USERNAME,
|
||||
password: env.RATE_LIMIT_REDIS_PASSWORD,
|
||||
tlsDisabled: env.RATE_LIMIT_REDIS_TLS_DISABLED === "true",
|
||||
clusterMode: env.RATE_LIMIT_REDIS_CLUSTER_MODE_ENABLED === "1",
|
||||
},
|
||||
keyPrefix: "api",
|
||||
defaultLimiter: {
|
||||
type: "tokenBucket",
|
||||
refillRate: env.API_RATE_LIMIT_REFILL_RATE,
|
||||
interval: env.API_RATE_LIMIT_REFILL_INTERVAL as Duration,
|
||||
maxTokens: env.API_RATE_LIMIT_MAX,
|
||||
},
|
||||
limiterCache: {
|
||||
fresh: 60_000 * 10, // Data is fresh for 10 minutes
|
||||
stale: 60_000 * 20, // Date is stale after 20 minutes
|
||||
maxItems: 1000,
|
||||
},
|
||||
limiterConfigOverride: resolveApiRateLimitOverride,
|
||||
pathMatchers: [/^\/api/],
|
||||
// Allow /api/v1/tasks/:id/callback/:secret
|
||||
pathWhiteList: [
|
||||
|
||||
@@ -37,10 +37,17 @@ export function dashboardAgentApiOrigin(): string {
|
||||
// service from the dashboard session (never a PAT), so a user can only ever
|
||||
// mint a token for themselves. The `in` proxy injects this into the turn's
|
||||
// metadata so the token reaches the agent without ever touching the browser.
|
||||
export function mintDashboardAgentUserActorToken(userId: string): Promise<string> {
|
||||
//
|
||||
// Endpoints that bind something to one environment read `environmentId` off the token,
|
||||
// so the agent can't name a different one in a request body.
|
||||
export function mintDashboardAgentUserActorToken(
|
||||
userId: string,
|
||||
opts: { environmentId: string }
|
||||
): Promise<string> {
|
||||
return signUserActorToken(env.SESSION_SECRET, {
|
||||
userId,
|
||||
client: "dashboard-agent",
|
||||
environmentId: opts.environmentId,
|
||||
cap: DASHBOARD_AGENT_UAT_CAP,
|
||||
expirationTime: Math.floor(Date.now() / 1000) + DASHBOARD_AGENT_UAT_TTL_SECONDS,
|
||||
});
|
||||
|
||||
@@ -0,0 +1,73 @@
|
||||
import type { NextFunction, Request, Response } from "express";
|
||||
import {
|
||||
MAX_MESSAGE_BODY_BYTES,
|
||||
MESSAGE_TOO_LARGE_CODE,
|
||||
MESSAGE_TOO_LARGE_ERROR,
|
||||
} from "~/components/dashboard-agent/message-limits";
|
||||
|
||||
/**
|
||||
* The ingress cap for the agent's chat paths. A route can only refuse a body after it has read
|
||||
* it, and `content-length` is optional, so a chunked upload would be buffered whole before the
|
||||
* route ever saw its size. This counts the bytes as they arrive and refuses mid-stream.
|
||||
*/
|
||||
|
||||
/** Headroom over the message cap for multipart framing and the per-turn metadata. */
|
||||
const INGRESS_SLACK_BYTES = 8 * 1024;
|
||||
|
||||
export const DASHBOARD_AGENT_MAX_INGRESS_BYTES = MAX_MESSAGE_BODY_BYTES + INGRESS_SLACK_BYTES;
|
||||
|
||||
// The agent's own routes only: the `/api/v1/dashboard-agent/…` endpoints and the
|
||||
// `/…/env/<env>/dashboard-agent…` chat resources. Anchored so a task named
|
||||
// `dashboard-agent` (`/api/v1/tasks/dashboard-agent/trigger`) is not capped.
|
||||
const AGENT_PATH = /^\/api\/v1\/dashboard-agent(\/|$)|\/env\/[^/]+\/dashboard-agent(\/|$)/;
|
||||
|
||||
/** Methods that can carry one. GET and HEAD cannot, and streaming them would be wasted work. */
|
||||
const BODY_METHODS = new Set(["POST", "PUT", "PATCH", "DELETE"]);
|
||||
|
||||
function refuse(res: Response): void {
|
||||
if (res.headersSent) return;
|
||||
res.status(413).json({ error: MESSAGE_TOO_LARGE_ERROR, code: MESSAGE_TOO_LARGE_CODE });
|
||||
}
|
||||
|
||||
/**
|
||||
* Attaches a counting listener and pauses the stream again immediately, so the route's own
|
||||
* reader still receives every chunk while nothing flows until it asks for it. Crossing the
|
||||
* limit ends the request: pausing alone wouldn't stop the route resuming the stream itself.
|
||||
*/
|
||||
export function capRequestBody(req: Request, res: Response, limit: number): void {
|
||||
const declared = Number.parseInt(req.headers["content-length"] ?? "", 10);
|
||||
if (Number.isFinite(declared) && declared > limit) {
|
||||
refuse(res);
|
||||
return;
|
||||
}
|
||||
|
||||
let received = 0;
|
||||
const onData = (chunk: Buffer | string) => {
|
||||
received += Buffer.byteLength(chunk);
|
||||
if (received <= limit) return;
|
||||
req.off("data", onData);
|
||||
req.pause();
|
||||
refuse(res);
|
||||
// Torn down only once the refusal is on the wire, or the client never reads it.
|
||||
res.once("finish", () => req.destroy());
|
||||
};
|
||||
|
||||
req.on("data", onData);
|
||||
req.pause();
|
||||
req.once("end", () => req.off("data", onData));
|
||||
}
|
||||
|
||||
/**
|
||||
* Only the agent's own paths: every other route keeps the body handling it had. Matched
|
||||
* case-insensitively because Remix routes are, and on every method — a DELETE reads a body too.
|
||||
*/
|
||||
export function dashboardAgentBodyCap(req: Request, res: Response, next: NextFunction): void {
|
||||
if (!BODY_METHODS.has(req.method) || !AGENT_PATH.test(req.path.toLowerCase())) {
|
||||
next();
|
||||
return;
|
||||
}
|
||||
|
||||
capRequestBody(req, res, DASHBOARD_AGENT_MAX_INGRESS_BYTES);
|
||||
if (res.headersSent) return;
|
||||
next();
|
||||
}
|
||||
@@ -0,0 +1,74 @@
|
||||
/**
|
||||
* Retention for soft-deleted chats. A deleted chat is kept for a grace window and then
|
||||
* hard-deleted with all its child rows; one bounded statement per run, oldest first.
|
||||
* Also the eventual purge behind organization deletion, which soft-deletes the org's
|
||||
* chats so this same sweep removes them.
|
||||
*/
|
||||
|
||||
import {
|
||||
hardDeleteChatsSoftDeletedBefore,
|
||||
softDeleteChatsForOrganization,
|
||||
} from "@internal/dashboard-agent-db";
|
||||
import { dashboardAgentDb } from "~/services/dashboardAgentDb.server";
|
||||
import { logger } from "~/services/logger.server";
|
||||
|
||||
/**
|
||||
* How long a soft-deleted chat is kept before it and its children are hard-deleted.
|
||||
* Long enough that an accidental delete can still be investigated; org deletion soft-
|
||||
* deletes the org's chats, so those are removed the same way once the window passes.
|
||||
*/
|
||||
export const CHAT_SOFT_DELETE_RETENTION_MS = 30 * 24 * 60 * 60 * 1000;
|
||||
|
||||
/** Per-run cap. Retention is one bounded statement, not a row-at-a-time loop. */
|
||||
const RETENTION_BATCH_LIMIT = 500;
|
||||
|
||||
export type ChatRetentionResult = {
|
||||
/** Soft-deleted chats past the retention window dropped this run. */
|
||||
purged: number;
|
||||
failed: number;
|
||||
};
|
||||
|
||||
export type ChatRetentionDeps = {
|
||||
now?: () => Date;
|
||||
limit?: number;
|
||||
/** Hard-delete chats soft-deleted before `before`. Returns how many went. */
|
||||
purge?: (params: { before: Date; limit: number }) => Promise<number>;
|
||||
};
|
||||
|
||||
export async function sweepDashboardAgentSoftDeletedChats(
|
||||
deps: ChatRetentionDeps = {}
|
||||
): Promise<ChatRetentionResult> {
|
||||
const now = deps.now?.() ?? new Date();
|
||||
const limit = deps.limit ?? RETENTION_BATCH_LIMIT;
|
||||
const purge =
|
||||
deps.purge ?? ((params) => hardDeleteChatsSoftDeletedBefore(dashboardAgentDb, params));
|
||||
|
||||
const result: ChatRetentionResult = { purged: 0, failed: 0 };
|
||||
|
||||
try {
|
||||
result.purged = await purge({
|
||||
before: new Date(now.getTime() - CHAT_SOFT_DELETE_RETENTION_MS),
|
||||
limit,
|
||||
});
|
||||
} catch (error) {
|
||||
result.failed++;
|
||||
logger.error("Dashboard agent chat retention failed", { error });
|
||||
}
|
||||
|
||||
if (result.failed > 0) {
|
||||
throw new Error("The dashboard agent chat retention pass failed");
|
||||
}
|
||||
|
||||
return result;
|
||||
}
|
||||
|
||||
/**
|
||||
* Soft-delete every chat belonging to a deleted organization. The retention sweep above
|
||||
* hard-deletes them once the window passes, so the org-deletion request never runs a
|
||||
* cross-database hard delete.
|
||||
*/
|
||||
export async function purgeDashboardAgentChatsForOrganization(params: {
|
||||
organizationId: string;
|
||||
}): Promise<number> {
|
||||
return softDeleteChatsForOrganization(dashboardAgentDb, params);
|
||||
}
|
||||
@@ -0,0 +1,37 @@
|
||||
/**
|
||||
* How the dashboard agent addresses one environment on the name-addressed API routes.
|
||||
*
|
||||
* The name is derived from the environment's type, and every branch shares its parent's type — so
|
||||
* the name alone does not identify an environment, it identifies a family. The branch is the rest
|
||||
* of the address, and the API's resolver needs both to land on the row the dashboard selected.
|
||||
*
|
||||
* Returned as a pair so no caller can take the name without it. Handing the JWT exchange a bare
|
||||
* "preview" resolves the parent, and the delegated token's `environmentId` claim then correctly
|
||||
* refuses it — the guard is the detector, not the defect.
|
||||
*
|
||||
* Kept free of heavy imports so both mint sites and their tests can use the real thing.
|
||||
*/
|
||||
|
||||
// The API's env routes key on the canonical env name, not the dashboard URL slug
|
||||
// (staging's slug is "stg").
|
||||
const ENV_NAME_BY_TYPE: Record<string, string> = {
|
||||
DEVELOPMENT: "dev",
|
||||
STAGING: "staging",
|
||||
PRODUCTION: "prod",
|
||||
PREVIEW: "preview",
|
||||
};
|
||||
|
||||
export type DashboardAgentEnvironmentAddress = {
|
||||
environmentName?: string;
|
||||
environmentBranch?: string;
|
||||
};
|
||||
|
||||
export function dashboardAgentEnvironmentAddress(
|
||||
environment: { type: string; branchName?: string | null } | undefined
|
||||
): DashboardAgentEnvironmentAddress {
|
||||
if (!environment) return {};
|
||||
return {
|
||||
environmentName: ENV_NAME_BY_TYPE[environment.type],
|
||||
...(environment.branchName ? { environmentBranch: environment.branchName } : {}),
|
||||
};
|
||||
}
|
||||
@@ -0,0 +1,54 @@
|
||||
/**
|
||||
* Whether an org's agent turns may be judged. The agent has no main-database access, so it
|
||||
* asks the API for this and treats anything but an explicit yes as no.
|
||||
*/
|
||||
|
||||
import { prisma } from "~/db.server";
|
||||
import { logger } from "~/services/logger.server";
|
||||
import { FEATURE_FLAG, hasUnreadableTurnEvalsOverride } from "~/v3/featureFlags";
|
||||
import { makeFlag } from "~/v3/featureFlags.server";
|
||||
|
||||
/** Judging is on unless an org turns it off. */
|
||||
const DEFAULT_TURN_EVALS_ENABLED = true;
|
||||
|
||||
/**
|
||||
* Resolves `dashboardAgentTurnEvalsEnabled` for one org, with a per-org override winning in
|
||||
* both directions. Membership-scoped: a token can name any org, so the caller's membership
|
||||
* is the tenant floor. Returns false when the org (or its setting) can't be read — a judged
|
||||
* turn goes to a third-party model, so an unknown answer must not read as consent.
|
||||
*/
|
||||
export async function orgAllowsDashboardAgentTurnEvals(params: {
|
||||
userId: string;
|
||||
organizationId: string;
|
||||
}): Promise<boolean> {
|
||||
try {
|
||||
const org = await prisma.organization.findFirst({
|
||||
where: {
|
||||
id: params.organizationId,
|
||||
members: { some: { userId: params.userId } },
|
||||
},
|
||||
select: { featureFlags: true },
|
||||
});
|
||||
if (!org) return false;
|
||||
|
||||
const overrides = (org.featureFlags as Record<string, unknown>) ?? {};
|
||||
// `flag()` ignores an override the schema rejects and falls through to the global default,
|
||||
// which is on — so an org that tried to turn judging off would keep being judged.
|
||||
if (hasUnreadableTurnEvalsOverride(overrides)) return false;
|
||||
|
||||
const flag = makeFlag();
|
||||
return Boolean(
|
||||
await flag({
|
||||
key: FEATURE_FLAG.dashboardAgentTurnEvalsEnabled,
|
||||
defaultValue: DEFAULT_TURN_EVALS_ENABLED,
|
||||
overrides,
|
||||
})
|
||||
);
|
||||
} catch (error) {
|
||||
logger.error("Couldn't read the org's dashboard agent turn-eval setting", {
|
||||
organizationId: params.organizationId,
|
||||
error,
|
||||
});
|
||||
return false;
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,58 @@
|
||||
/**
|
||||
* Retention for the agent's judged-turn rows. The table is append-only quality data with
|
||||
* no reader, so it can't be left to grow forever; one bounded statement per run, oldest
|
||||
* first. Runs whether or not the agent is configured — rows outlive the agent project.
|
||||
*/
|
||||
|
||||
import { deleteTurnEvalsOlderThan } from "@internal/dashboard-agent-db";
|
||||
import { dashboardAgentDb } from "~/services/dashboardAgentDb.server";
|
||||
import { logger } from "~/services/logger.server";
|
||||
|
||||
/**
|
||||
* How long a judged turn is kept. Nothing reads the table today, and the rows carry the
|
||||
* user's question next to the agent's answer, so the period is the shortest one that still
|
||||
* lets a month of product review (capability and docs gaps) be aggregated.
|
||||
*/
|
||||
export const TURN_EVAL_RETENTION_MS = 30 * 24 * 60 * 60 * 1000;
|
||||
|
||||
/** Per-run cap. Retention is one statement, not a row-at-a-time loop. */
|
||||
const RETENTION_BATCH_LIMIT = 500;
|
||||
|
||||
export type TurnEvalRetentionResult = {
|
||||
/** Rows past the retention period dropped this run. */
|
||||
purged: number;
|
||||
failed: number;
|
||||
};
|
||||
|
||||
export type TurnEvalRetentionDeps = {
|
||||
now?: () => Date;
|
||||
limit?: number;
|
||||
/** Drop rows created before `before`. Returns how many went. */
|
||||
purge?: (params: { before: Date; limit: number }) => Promise<number>;
|
||||
};
|
||||
|
||||
export async function sweepDashboardAgentTurnEvals(
|
||||
deps: TurnEvalRetentionDeps = {}
|
||||
): Promise<TurnEvalRetentionResult> {
|
||||
const now = deps.now?.() ?? new Date();
|
||||
const limit = deps.limit ?? RETENTION_BATCH_LIMIT;
|
||||
const purge = deps.purge ?? ((params) => deleteTurnEvalsOlderThan(dashboardAgentDb, params));
|
||||
|
||||
const result: TurnEvalRetentionResult = { purged: 0, failed: 0 };
|
||||
|
||||
try {
|
||||
result.purged = await purge({
|
||||
before: new Date(now.getTime() - TURN_EVAL_RETENTION_MS),
|
||||
limit,
|
||||
});
|
||||
} catch (error) {
|
||||
result.failed++;
|
||||
logger.error("Dashboard agent turn-eval retention failed", { error });
|
||||
}
|
||||
|
||||
if (result.failed > 0) {
|
||||
throw new Error("The dashboard agent turn-eval retention pass failed");
|
||||
}
|
||||
|
||||
return result;
|
||||
}
|
||||
@@ -6,8 +6,14 @@ import {
|
||||
dashboardAgentCodeToolSchemas,
|
||||
dashboardAgentToolSchemas,
|
||||
} from "@internal/dashboard-agent/tool-schemas";
|
||||
import {
|
||||
describePromptPrefix,
|
||||
PROMPT_CACHE_CONTROL,
|
||||
promptCacheAttributes,
|
||||
} from "@internal/dashboard-agent/prompt-prefix";
|
||||
import { ApiClient, SessionStreamInstance, writeTurnCompleteRecord } from "@trigger.dev/core/v3";
|
||||
import { chat as chatServer } from "@trigger.dev/sdk/chat-server";
|
||||
import { streamText, type UIMessage } from "ai";
|
||||
import { streamText, type UIMessage, type UIMessageChunk } from "ai";
|
||||
import { env } from "~/env.server";
|
||||
import {
|
||||
dashboardAgentApiOrigin,
|
||||
@@ -19,17 +25,69 @@ const TASK_ID = "dashboard-agent";
|
||||
|
||||
const anthropic = createAnthropic({ apiKey: env.ANTHROPIC_API_KEY });
|
||||
|
||||
/** Shown when the warm first turn produced nothing. The provider error is only logged. */
|
||||
export const HEAD_START_FAILURE_ERROR_TEXT =
|
||||
"The assistant couldn't start this response. Please send your message again.";
|
||||
|
||||
/** A seam so the failure path is testable without S2 credentials or a live session. */
|
||||
export type DashboardAgentSessionOutWriter = {
|
||||
writeChunk(chunk: UIMessageChunk): Promise<void>;
|
||||
/** The `turn-complete` control record that closes the client's stream. */
|
||||
writeTurnComplete(): Promise<void>;
|
||||
};
|
||||
|
||||
/**
|
||||
* Server-owned head start. The webapp generates the chatId and owns the chat
|
||||
* record, then kicks off step 1 here via `chat.startHeadStart` (the detached
|
||||
* flow): it creates the session (externalId = chatId), triggers the
|
||||
* handover-prepare run, and streams step 1 into `session.out` in the background.
|
||||
* The browser resumes that stream rather than streaming step 1 inline. Step 1
|
||||
* runs the agent's SCHEMA-ONLY tools + the shared model/prompt for the mode the
|
||||
* agent run will be in; the agent run picks up tool execution and step 2+.
|
||||
*
|
||||
* `metadata` (the delegated UAT + context) is merged into the run's wire payload
|
||||
* server-side, so it reaches the agent without touching the browser.
|
||||
* Surface a failed warm step 1 as a visible error turn. `turn-complete` is written even if
|
||||
* the error chunk fails, so a resumed stream always terminates.
|
||||
*/
|
||||
export async function writeHeadStartFailureToSessionOut(
|
||||
writer: DashboardAgentSessionOutWriter
|
||||
): Promise<void> {
|
||||
try {
|
||||
await writer.writeChunk({
|
||||
type: "error",
|
||||
errorText: HEAD_START_FAILURE_ERROR_TEXT,
|
||||
} as UIMessageChunk);
|
||||
} finally {
|
||||
await writer.writeTurnComplete();
|
||||
}
|
||||
}
|
||||
|
||||
function singleChunkStream(chunk: UIMessageChunk): ReadableStream<UIMessageChunk> {
|
||||
return new ReadableStream<UIMessageChunk>({
|
||||
start(controller) {
|
||||
controller.enqueue(chunk);
|
||||
controller.close();
|
||||
},
|
||||
});
|
||||
}
|
||||
|
||||
// Writes as the agent's own environment: `.out` appends are private-only.
|
||||
function createSessionOutWriter(
|
||||
chatId: string,
|
||||
accessToken: string
|
||||
): DashboardAgentSessionOutWriter {
|
||||
const apiClient = new ApiClient(dashboardAgentApiOrigin(), accessToken);
|
||||
return {
|
||||
async writeChunk(chunk) {
|
||||
const instance = new SessionStreamInstance<UIMessageChunk>({
|
||||
apiClient,
|
||||
baseUrl: apiClient.baseUrl,
|
||||
sessionId: chatId, // Sessions are addressable by externalId (chatId).
|
||||
io: "out",
|
||||
source: singleChunkStream(chunk),
|
||||
});
|
||||
await instance.wait();
|
||||
},
|
||||
async writeTurnComplete() {
|
||||
await writeTurnCompleteRecord(apiClient, chatId);
|
||||
},
|
||||
};
|
||||
}
|
||||
|
||||
/**
|
||||
* Server-owned head start: creates the session, triggers the handover-prepare run, and
|
||||
* streams step 1 into `session.out`. `metadata` is merged into the run's payload server-side.
|
||||
*/
|
||||
export async function startDashboardAgentHeadStart(params: {
|
||||
chatId: string;
|
||||
@@ -47,8 +105,7 @@ export async function startDashboardAgentHeadStart(params: {
|
||||
messages: params.messages,
|
||||
metadata: params.metadata,
|
||||
triggerConfig: dashboardAgentTriggerConfig(),
|
||||
// Scope session creation + the agent trigger to the agent's project/env. The
|
||||
// Anthropic key here only powers the warm step-1 call.
|
||||
// Scopes session creation and the agent trigger to the agent's own environment.
|
||||
apiClient: {
|
||||
baseURL: dashboardAgentApiOrigin(),
|
||||
accessToken: env.DASHBOARD_AGENT_SECRET_KEY,
|
||||
@@ -57,16 +114,44 @@ export async function startDashboardAgentHeadStart(params: {
|
||||
streamText({
|
||||
...helper.toStreamTextOptions({ tools }),
|
||||
model: anthropic(DASHBOARD_AGENT_MODEL),
|
||||
system,
|
||||
// A structured system message, not a bare string: without provider options
|
||||
// Anthropic neither writes nor reads the cache, so this call paid full price
|
||||
// for the prefix and the agent's step 2 then paid for a fresh write. The tool
|
||||
// key order is frozen (see `tool-schemas.ts`) so both prefixes are identical
|
||||
// — the logged fingerprint is how a drift becomes visible.
|
||||
system: {
|
||||
role: "system",
|
||||
content: system,
|
||||
providerOptions: { anthropic: { cacheControl: PROMPT_CACHE_CONTROL } },
|
||||
},
|
||||
onStepFinish: (step) => {
|
||||
logger.info(
|
||||
"Dashboard agent prompt cache",
|
||||
promptCacheAttributes({
|
||||
source: "head-start",
|
||||
usage: step.usage,
|
||||
prefix: describePromptPrefix({ system, tools }),
|
||||
})
|
||||
);
|
||||
},
|
||||
}),
|
||||
});
|
||||
|
||||
// The webapp is long-lived, so step 1's drain + the handover dispatch run in
|
||||
// the background after this resolves (createSession + trigger have completed).
|
||||
// Log a warm-step failure for observability: startHeadStart has already fired
|
||||
// handover-skip so the agent run exits cleanly, but the client (mounted as
|
||||
// streaming) then resumes an empty session.out, so the turn looks lost.
|
||||
completion.catch((error) => {
|
||||
// Step 1's drain and the handover dispatch continue in the background. On failure
|
||||
// `startHeadStart` already fired handover-skip, so nothing else writes to session.out.
|
||||
completion.catch(async (error) => {
|
||||
logger.error("Dashboard agent head start failed", { chatId: params.chatId, error });
|
||||
|
||||
const accessToken = env.DASHBOARD_AGENT_SECRET_KEY;
|
||||
if (!accessToken) return;
|
||||
|
||||
try {
|
||||
await writeHeadStartFailureToSessionOut(createSessionOutWriter(params.chatId, accessToken));
|
||||
} catch (writeError) {
|
||||
logger.error("Failed to write dashboard agent head start error to session.out", {
|
||||
chatId: params.chatId,
|
||||
error: writeError,
|
||||
});
|
||||
}
|
||||
});
|
||||
}
|
||||
|
||||
@@ -0,0 +1,112 @@
|
||||
/**
|
||||
* The investigation backstop, for cards left `in_progress`. They settle as `inconclusive`,
|
||||
* conditional on the row still being `in_progress`, so a concluding turn wins the race.
|
||||
*/
|
||||
|
||||
import {
|
||||
listStaleOpenInvestigations,
|
||||
settleInvestigationAndCloseCard,
|
||||
type Investigation,
|
||||
type SettledInvestigationCard,
|
||||
} from "@internal/dashboard-agent-db";
|
||||
import { UNSETTLED_INVESTIGATION_NOTE } from "@internal/dashboard-agent-contracts";
|
||||
import { dashboardAgentDb } from "~/services/dashboardAgentDb.server";
|
||||
import { logger } from "~/services/logger.server";
|
||||
|
||||
/**
|
||||
* How long a card may sit `in_progress` before the sweep settles it. Must outlast the
|
||||
* slowest live turn, which bumps `updated_at` on every revision.
|
||||
*/
|
||||
export const INVESTIGATION_STALE_MS = 30 * 60 * 1000;
|
||||
|
||||
/** Per-run cap. Oldest first, so the rest land next run. */
|
||||
const SWEEP_BATCH_LIMIT = 100;
|
||||
|
||||
export type InvestigationSweepResult = {
|
||||
/** Stale `in_progress` rows seen. */
|
||||
stale: number;
|
||||
settled: number;
|
||||
/** Settled rows whose closing card reached the chat. */
|
||||
closed: number;
|
||||
/** A turn (or another sweep) settled it first. */
|
||||
alreadySettled: number;
|
||||
failed: number;
|
||||
};
|
||||
|
||||
export type InvestigationSweepDeps = {
|
||||
now?: () => Date;
|
||||
limit?: number;
|
||||
listStale?: (params: { olderThan: Date; limit: number }) => Promise<Investigation[]>;
|
||||
/**
|
||||
* Settle one row and deliver its closing card as a single operation. Null when the
|
||||
* row was no longer `in_progress`.
|
||||
*/
|
||||
settleAndClose?: (params: {
|
||||
id: string;
|
||||
chatId: string;
|
||||
note: string;
|
||||
}) => Promise<SettledInvestigationCard | null>;
|
||||
};
|
||||
|
||||
/**
|
||||
* Settle every card `in_progress` past the grace window. Each row is handled on its own,
|
||||
* and the run throws at the end if any failed so the job is retried.
|
||||
*/
|
||||
export async function sweepDashboardAgentInvestigations(
|
||||
deps: InvestigationSweepDeps = {}
|
||||
): Promise<InvestigationSweepResult> {
|
||||
const now = deps.now?.() ?? new Date();
|
||||
const limit = deps.limit ?? SWEEP_BATCH_LIMIT;
|
||||
const listStale =
|
||||
deps.listStale ?? ((params) => listStaleOpenInvestigations(dashboardAgentDb, params));
|
||||
const settleAndClose =
|
||||
deps.settleAndClose ?? ((params) => settleInvestigationAndCloseCard(dashboardAgentDb, params));
|
||||
|
||||
const result: InvestigationSweepResult = {
|
||||
stale: 0,
|
||||
settled: 0,
|
||||
closed: 0,
|
||||
alreadySettled: 0,
|
||||
failed: 0,
|
||||
};
|
||||
|
||||
const stale = await listStale({
|
||||
olderThan: new Date(now.getTime() - INVESTIGATION_STALE_MS),
|
||||
limit,
|
||||
});
|
||||
result.stale = stale.length;
|
||||
|
||||
for (const investigation of stale) {
|
||||
try {
|
||||
// Settling the row fixes nothing on its own: the chat renders the winning card
|
||||
// from its own transcript, so an unappended settle is still a stuck spinner —
|
||||
// which is why both writes are one operation that rolls back together.
|
||||
const outcome = await settleAndClose({
|
||||
id: investigation.id,
|
||||
chatId: investigation.chatId,
|
||||
note: UNSETTLED_INVESTIGATION_NOTE,
|
||||
});
|
||||
if (!outcome) {
|
||||
result.alreadySettled++;
|
||||
continue;
|
||||
}
|
||||
result.settled++;
|
||||
if (outcome.closed) result.closed++;
|
||||
} catch (error) {
|
||||
result.failed++;
|
||||
logger.error("Dashboard agent investigation sweep: failed to settle an investigation", {
|
||||
investigationId: investigation.id,
|
||||
chatId: investigation.chatId,
|
||||
error,
|
||||
});
|
||||
}
|
||||
}
|
||||
|
||||
if (result.failed > 0) {
|
||||
throw new Error(
|
||||
`The dashboard agent investigation sweep failed on ${result.failed} investigations`
|
||||
);
|
||||
}
|
||||
|
||||
return result;
|
||||
}
|
||||
@@ -5,6 +5,8 @@ import { featuresForRequest } from "~/features.server";
|
||||
import { DeleteProjectService } from "./deleteProject.server";
|
||||
import { getCurrentPlan } from "./platform.v3.server";
|
||||
import { controlPlaneResolver } from "~/v3/runOpsMigration/controlPlaneResolver.server";
|
||||
import { commonWorker } from "~/v3/commonWorker.server";
|
||||
import { logger } from "./logger.server";
|
||||
|
||||
export class DeleteOrganizationService {
|
||||
#prismaClient: PrismaClient;
|
||||
@@ -86,5 +88,21 @@ export class DeleteOrganizationService {
|
||||
|
||||
// runsEnabled + the org's projects (project.deletedAt) changed; drop all cached env rows.
|
||||
controlPlaneResolver.invalidateOrganization(organization.id);
|
||||
|
||||
// Soft-delete the org's dashboard agent chats; retention purges them later. Enqueued,
|
||||
// not inline: the agent store is a separate database in cloud.
|
||||
// Best-effort: a failed enqueue must not fail org deletion (the org is already deleted).
|
||||
try {
|
||||
await commonWorker.enqueue({
|
||||
id: `dashboardAgent.purgeOrganization:${organization.id}`,
|
||||
job: "dashboardAgent.purgeOrganization",
|
||||
payload: { organizationId: organization.id },
|
||||
});
|
||||
} catch (error) {
|
||||
logger.warn("Failed to enqueue dashboard agent purge for deleted organization", {
|
||||
organizationId: organization.id,
|
||||
error,
|
||||
});
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
@@ -9,7 +9,7 @@ import {
|
||||
} from "~/services/apiAuth.server";
|
||||
import { rbac } from "~/services/rbac.server";
|
||||
|
||||
type EnvironmentScopedResource = "envvars" | "apiKeys";
|
||||
type EnvironmentScopedResource = "envvars" | "apiKeys" | "deployments";
|
||||
|
||||
type EnvironmentScopedAuthentication =
|
||||
| { ok: true; authentication: AuthenticationResult }
|
||||
@@ -78,6 +78,7 @@ export function authenticateEnvVarApiRequest(
|
||||
const RESOURCE_LABELS: Record<EnvironmentScopedResource, string> = {
|
||||
envvars: "environment variables",
|
||||
apiKeys: "API keys",
|
||||
deployments: "deployments",
|
||||
};
|
||||
|
||||
/**
|
||||
|
||||
@@ -6,7 +6,7 @@ import { logger } from "./logger.server";
|
||||
import { rbac } from "./rbac.server";
|
||||
import { decryptToken, encryptToken, hashToken } from "~/utils/tokens.server";
|
||||
import { env } from "~/env.server";
|
||||
import { isUserActorToken } from "@trigger.dev/rbac";
|
||||
import { isUserActorToken, verifyUserActorToken, type UserActorClaims } from "@trigger.dev/rbac";
|
||||
|
||||
const tokenValueLength = 40;
|
||||
//lowercase only, removed 0 and l to avoid confusion
|
||||
@@ -115,6 +115,11 @@ export async function revokePersonalAccessToken(tokenId: string, userId: string)
|
||||
|
||||
export type PersonalAccessTokenAuthenticationResult = {
|
||||
userId: string;
|
||||
/**
|
||||
* Verified claims when the caller presented a delegated user-actor token. They ride on the
|
||||
* result so no caller can hold the actor without its environment scope.
|
||||
*/
|
||||
userActor?: UserActorClaims;
|
||||
};
|
||||
|
||||
/**
|
||||
@@ -167,11 +172,12 @@ export async function authenticateApiRequestWithPersonalAccessToken(
|
||||
return;
|
||||
}
|
||||
|
||||
// A user-actor token authenticates as the user wherever a PAT does.
|
||||
// The plugin verifies it (identity path → no org context to floor against).
|
||||
// PAT-only: this helper checks no scopes and no capability context, and its callers include
|
||||
// actions and the admin gate. A delegated token is refused at the entrance and reaches the
|
||||
// API through the actor-aware routes instead, which do enforce its claims.
|
||||
if (isUserActorToken(token)) {
|
||||
const result = await rbac.authenticateUserActor(request, {});
|
||||
return result.ok ? { userId: result.userId } : undefined;
|
||||
logger.warn("Rejected a user-actor token at the PAT-only authentication helper");
|
||||
return;
|
||||
}
|
||||
|
||||
return authenticatePersonalAccessToken(token);
|
||||
@@ -284,6 +290,41 @@ export function isPersonalAccessToken(token: string) {
|
||||
return token.startsWith(tokenPrefix);
|
||||
}
|
||||
|
||||
/**
|
||||
* A user-actor token minted from a PAT carries the source PAT's id (`claims.pat`).
|
||||
* The token is stateless, so revoking the PAT can't invalidate it by itself — hosts
|
||||
* recheck the source PAT is still live here. A token with no `pat` (e.g. the dashboard
|
||||
* agent's) has no source to check and is left alone.
|
||||
*/
|
||||
export async function assertSourcePatActive(claims: UserActorClaims): Promise<boolean> {
|
||||
if (!claims.pat) return true;
|
||||
|
||||
const found = await prisma.personalAccessToken.findFirst({
|
||||
where: { id: claims.pat, revokedAt: null },
|
||||
select: { id: true },
|
||||
});
|
||||
return Boolean(found);
|
||||
}
|
||||
|
||||
/**
|
||||
* Resolve + source-PAT-recheck a user-actor token for a verify site where `claims` may be
|
||||
* supplied by the RBAC plugin (the apiBuilder path). We re-verify the bearer locally so the
|
||||
* recheck reads the token's OWN `pat`, not the plugin's: a plugin image predating the `pat`
|
||||
* claim would deliver pat-less claims and silently no-op revocation here. Returns the claims
|
||||
* to act on, or undefined to deny. The direct verify sites (jwt exchange, the UAT preamble)
|
||||
* don't go through a plugin and call `assertSourcePatActive` on their own verified claims.
|
||||
*/
|
||||
export async function resolveAndRecheckUserActorClaims(
|
||||
claims: UserActorClaims | undefined,
|
||||
bearer: string
|
||||
): Promise<UserActorClaims | undefined> {
|
||||
const verified = await verifyUserActorToken(env.SESSION_SECRET, bearer);
|
||||
const resolved = claims ?? verified;
|
||||
if (!resolved) return undefined;
|
||||
// Recheck against the locally-verified claims when available, so `pat` is authoritative.
|
||||
return (await assertSourcePatActive(verified ?? resolved)) ? resolved : undefined;
|
||||
}
|
||||
|
||||
/**
|
||||
* Read-only check that an authorization code is still mintable: it exists, is
|
||||
* unconsumed (`personalAccessTokenId: null`), and within the TTL. Lets the
|
||||
|
||||
@@ -70,7 +70,10 @@ export type ExecuteQueryOptions<TOut extends z.ZodSchema> = Omit<
|
||||
organizationId: string;
|
||||
projectId: string;
|
||||
environmentId: string;
|
||||
/** The scope of the query - determines tenant isolation */
|
||||
/**
|
||||
* The scope of the query - determines tenant isolation. Callers that take it from
|
||||
* a request body must cap it against the credential first; see `v3/queryScope.ts`.
|
||||
*/
|
||||
scope: QueryScope;
|
||||
period?: string | null;
|
||||
from?: string | null;
|
||||
|
||||
@@ -0,0 +1,162 @@
|
||||
/**
|
||||
* `trigger://` URI to dashboard link. Pure: a URI's `{env}` is a RuntimeEnvironment id but
|
||||
* dashboard URLs need slugs, so the caller supplies the already-resolved scope.
|
||||
*/
|
||||
import {
|
||||
safeParseTriggerUri,
|
||||
type ParsedTriggerUri,
|
||||
type TriggerUri,
|
||||
} from "@internal/dashboard-agent-contracts";
|
||||
import {
|
||||
v3DeploymentVersionPath,
|
||||
v3ErrorPath,
|
||||
v3QueuesPath,
|
||||
v3RunPath,
|
||||
v3RunSpanPath,
|
||||
v3RunsPath,
|
||||
} from "~/utils/pathBuilder";
|
||||
|
||||
/** Structurally satisfied by `AuthenticatedEnvironment`. */
|
||||
export type TriggerUriScope = {
|
||||
/** RuntimeEnvironment id. Must match the URI's `{env}` segment. */
|
||||
id: string;
|
||||
slug: string;
|
||||
project: { slug: string; externalRef: string };
|
||||
organization: { slug: string };
|
||||
/** Only a `source` URI needs this. Omitted, a source URI resolves to nothing. */
|
||||
repository?: { fullName?: string | null; remoteUrl?: string | null } | null;
|
||||
};
|
||||
|
||||
export type ResolvedTriggerUri = {
|
||||
label: string;
|
||||
/** Dashboard path, relative to the app origin, unless `external` is set. */
|
||||
url: string;
|
||||
/** True when `url` is absolute and off the dashboard, so a host must not route it. */
|
||||
external?: boolean;
|
||||
};
|
||||
|
||||
/**
|
||||
* Resolve one URI against one environment, returning `null` rather than guessing. A stored
|
||||
* transcript can hold foreign URIs, which must never resolve into this project's URL space.
|
||||
*/
|
||||
export function resolveTriggerUri(
|
||||
scope: TriggerUriScope,
|
||||
uri: TriggerUri | string
|
||||
): ResolvedTriggerUri | null {
|
||||
const parsed = safeParseTriggerUri(uri);
|
||||
if (!parsed.success) return null;
|
||||
if (!isInScope(scope, parsed.data)) return null;
|
||||
return resolveInScope(scope, parsed.data);
|
||||
}
|
||||
|
||||
const GITHUB_ORIGIN = "https://github.com";
|
||||
/** `owner/repo`, GitHub's own character set and nothing that could add a path. */
|
||||
const FULL_NAME = /^[A-Za-z0-9._-]+\/[A-Za-z0-9._-]+$/;
|
||||
|
||||
/**
|
||||
* The repository's canonical `https://github.com/{owner}/{repo}` base, or null. `remoteUrl` is
|
||||
* normalized as the deployments UI does; anything but github.com is rejected, not guessed at.
|
||||
*/
|
||||
function githubRepoBaseUrl(repository: TriggerUriScope["repository"]): string | null {
|
||||
const fullName = repository?.fullName?.trim();
|
||||
if (fullName && FULL_NAME.test(fullName)) return `${GITHUB_ORIGIN}/${fullName}`;
|
||||
|
||||
const remoteUrl = repository?.remoteUrl?.trim();
|
||||
if (!remoteUrl) return null;
|
||||
|
||||
const normalized = remoteUrl
|
||||
.replace(/^git@github\.com:/, `${GITHUB_ORIGIN}/`)
|
||||
.replace(/^ssh:\/\/git@github\.com\//, `${GITHUB_ORIGIN}/`)
|
||||
.replace(/\.git$/, "");
|
||||
|
||||
let url: URL;
|
||||
try {
|
||||
url = new URL(normalized);
|
||||
} catch {
|
||||
return null;
|
||||
}
|
||||
if (url.hostname !== "github.com") return null;
|
||||
|
||||
const path = url.pathname.replace(/^\/+|\/+$/g, "");
|
||||
if (!FULL_NAME.test(path)) return null;
|
||||
return `${GITHUB_ORIGIN}/${path}`;
|
||||
}
|
||||
|
||||
/** True when the URI names this exact project and environment. */
|
||||
function isInScope(scope: TriggerUriScope, parsed: ParsedTriggerUri): boolean {
|
||||
return parsed.projectRef === scope.project.externalRef && parsed.environmentId === scope.id;
|
||||
}
|
||||
|
||||
function resolveInScope(
|
||||
scope: TriggerUriScope,
|
||||
parsed: ParsedTriggerUri
|
||||
): ResolvedTriggerUri | null {
|
||||
const { organization, project } = scope;
|
||||
const environment = { slug: scope.slug };
|
||||
|
||||
switch (parsed.kind) {
|
||||
case "runs":
|
||||
// The navigate intent's `filters` become URL params at the host.
|
||||
return {
|
||||
label: "Runs",
|
||||
url: v3RunsPath(organization, project, environment),
|
||||
};
|
||||
case "run":
|
||||
return {
|
||||
label: parsed.runId,
|
||||
url: v3RunPath(organization, project, environment, { friendlyId: parsed.runId }),
|
||||
};
|
||||
case "span":
|
||||
return {
|
||||
label: `${parsed.runId} (${parsed.spanId})`,
|
||||
url: v3RunSpanPath(
|
||||
organization,
|
||||
project,
|
||||
environment,
|
||||
{ friendlyId: parsed.runId },
|
||||
{ spanId: parsed.spanId }
|
||||
),
|
||||
};
|
||||
case "error":
|
||||
return {
|
||||
label: parsed.fingerprint,
|
||||
url: v3ErrorPath(organization, project, environment, { fingerprint: parsed.fingerprint }),
|
||||
};
|
||||
case "queue":
|
||||
// The queue detail route is keyed by friendlyId, which a URI doesn't carry, so this
|
||||
// resolves to the queues list filtered to the name.
|
||||
return {
|
||||
label: parsed.name,
|
||||
url: `${v3QueuesPath(organization, project, environment)}?query=${encodeURIComponent(
|
||||
parsed.name
|
||||
)}`,
|
||||
};
|
||||
case "deployment":
|
||||
return {
|
||||
label: parsed.version,
|
||||
url: v3DeploymentVersionPath(organization, project, environment, parsed.version),
|
||||
};
|
||||
case "source": {
|
||||
// The URI pins the commit and repo-relative path; the connected repo says where that
|
||||
// lives. Without a connection there is nothing to open.
|
||||
const base = githubRepoBaseUrl(scope.repository);
|
||||
const label = parsed.line === undefined ? parsed.path : `${parsed.path}:${parsed.line}`;
|
||||
if (!base) return null;
|
||||
const path = parsed.path.split("/").map(encodeURIComponent).join("/");
|
||||
const fragment = parsed.line === undefined ? "" : `#L${parsed.line}`;
|
||||
return {
|
||||
label,
|
||||
url: `${base}/blob/${encodeURIComponent(parsed.sha)}/${path}${fragment}`,
|
||||
external: true,
|
||||
};
|
||||
}
|
||||
// No dashboard page exists for these yet, so the caller renders a label with no link.
|
||||
case "report":
|
||||
case "investigation":
|
||||
return null;
|
||||
default: {
|
||||
const unreachable: never = parsed;
|
||||
throw new Error(`Unhandled trigger:// kind: ${JSON.stringify(unreachable)}`);
|
||||
}
|
||||
}
|
||||
}
|
||||
@@ -1,5 +1,8 @@
|
||||
import type { z } from "zod";
|
||||
import type { ApiAuthenticationResultSuccess } from "../apiAuth.server";
|
||||
import type {
|
||||
ApiAuthenticationResultSuccess,
|
||||
UserActorAuthenticatedActor,
|
||||
} from "../apiAuth.server";
|
||||
import type { ActionFunctionArgs, LoaderFunctionArgs } from "@remix-run/server-runtime";
|
||||
import { json } from "@remix-run/server-runtime";
|
||||
import { fromZodError } from "zod-validation-error";
|
||||
@@ -9,8 +12,11 @@ import { rbac } from "../rbac.server";
|
||||
import { authenticateBearerWithTelemetry } from "~/services/authTelemetry.server";
|
||||
import type { RbacAbility, RbacResource } from "@trigger.dev/rbac";
|
||||
import { isUserActorToken } from "@trigger.dev/rbac";
|
||||
import type { PersonalAccessTokenAuthenticationResult } from "../personalAccessToken.server";
|
||||
import { updateLastAccessedAtIfStale } from "../personalAccessToken.server";
|
||||
import {
|
||||
resolveAndRecheckUserActorClaims,
|
||||
updateLastAccessedAtIfStale,
|
||||
} from "../personalAccessToken.server";
|
||||
import { assertUserActorScope } from "../userActorEnvironment.server";
|
||||
import { safeJsonParse } from "~/utils/json";
|
||||
import type { AuthenticatedWorkerInstance } from "~/v3/services/worker/workerGroupTokenService.server";
|
||||
import { WorkerGroupTokenService } from "~/v3/services/worker/workerGroupTokenService.server";
|
||||
@@ -422,7 +428,10 @@ export function createLoaderApiRoute<
|
||||
const apiVersion = getApiVersion(request);
|
||||
|
||||
const result = await tenantContext.run(
|
||||
tenantContextFromAuthEnvironment(authenticationResult.environment),
|
||||
tenantContextFromAuthEnvironment(
|
||||
authenticationResult.environment,
|
||||
authenticationResult.actor
|
||||
),
|
||||
() =>
|
||||
handler({
|
||||
params: parsedParams,
|
||||
@@ -458,6 +467,10 @@ export function createLoaderApiRoute<
|
||||
};
|
||||
}
|
||||
|
||||
// `environmentId` is checked against a user-actor token's environment claim, so an env-scoped
|
||||
// route enforces the scope by declaring it here.
|
||||
type PATRouteContext = { organizationId?: string; projectId?: string; environmentId?: string };
|
||||
|
||||
type PATRouteBuilderOptions<
|
||||
TParamsSchema extends AnyZodSchema | undefined = undefined,
|
||||
TSearchParamsSchema extends AnyZodSchema | undefined = undefined,
|
||||
@@ -479,9 +492,7 @@ type PATRouteBuilderOptions<
|
||||
? z.infer<TParamsSchema>
|
||||
: undefined,
|
||||
request: Request
|
||||
) =>
|
||||
| { organizationId?: string; projectId?: string }
|
||||
| Promise<{ organizationId?: string; projectId?: string }>;
|
||||
) => PATRouteContext | Promise<PATRouteContext>;
|
||||
authorization?: {
|
||||
action: string;
|
||||
resource: (
|
||||
@@ -500,6 +511,18 @@ type PATRouteBuilderOptions<
|
||||
};
|
||||
};
|
||||
|
||||
type PATLoaderRouteBuilderOptions<
|
||||
TParamsSchema extends AnyZodSchema | undefined = undefined,
|
||||
TSearchParamsSchema extends AnyZodSchema | undefined = undefined,
|
||||
THeadersSchema extends AnyZodSchema | undefined = undefined,
|
||||
> = PATRouteBuilderOptions<TParamsSchema, TSearchParamsSchema, THeadersSchema> & {
|
||||
// Opts a contextless route into being reachable by an environment-scoped user-actor token.
|
||||
// Only for routes whose answer is the caller's own identity (their orgs, their projects) and
|
||||
// which mutate nothing — otherwise such a token is refused for want of anything to check.
|
||||
// Loaders only: an action mutates by definition, so the action options forbid it.
|
||||
identityOnly?: true;
|
||||
};
|
||||
|
||||
type PATHandlerFunction<
|
||||
TParamsSchema extends AnyZodSchema | undefined,
|
||||
TSearchParamsSchema extends AnyZodSchema | undefined,
|
||||
@@ -516,7 +539,7 @@ type PATHandlerFunction<
|
||||
headers: THeadersSchema extends z.ZodFirstPartySchemaTypes | z.ZodDiscriminatedUnion<any, any>
|
||||
? z.infer<THeadersSchema>
|
||||
: undefined;
|
||||
authentication: PersonalAccessTokenAuthenticationResult;
|
||||
authentication: UserActorAuthenticatedActor;
|
||||
ability: RbacAbility;
|
||||
request: Request;
|
||||
apiVersion: API_VERSIONS;
|
||||
@@ -527,7 +550,7 @@ export function createLoaderPATApiRoute<
|
||||
TSearchParamsSchema extends AnyZodSchema | undefined = undefined,
|
||||
THeadersSchema extends AnyZodSchema | undefined = undefined,
|
||||
>(
|
||||
options: PATRouteBuilderOptions<TParamsSchema, TSearchParamsSchema, THeadersSchema>,
|
||||
options: PATLoaderRouteBuilderOptions<TParamsSchema, TSearchParamsSchema, THeadersSchema>,
|
||||
handler: PATHandlerFunction<TParamsSchema, TSearchParamsSchema, THeadersSchema>
|
||||
) {
|
||||
return async function loader({ request, params }: LoaderFunctionArgs) {
|
||||
@@ -537,6 +560,7 @@ export function createLoaderPATApiRoute<
|
||||
headers: headersSchema,
|
||||
corsStrategy = "none",
|
||||
context: contextFn,
|
||||
identityOnly,
|
||||
authorization,
|
||||
} = options;
|
||||
|
||||
@@ -614,7 +638,7 @@ export function createLoaderPATApiRoute<
|
||||
// cached timestamp is fresher than the throttle window).
|
||||
const ctx = contextFn ? await contextFn(parsedParams, request) : {};
|
||||
|
||||
let authenticationResult: PersonalAccessTokenAuthenticationResult;
|
||||
let authenticationResult: UserActorAuthenticatedActor;
|
||||
let ability: RbacAbility;
|
||||
|
||||
const bearer = request.headers
|
||||
@@ -632,7 +656,16 @@ export function createLoaderPATApiRoute<
|
||||
corsStrategy !== "none"
|
||||
);
|
||||
}
|
||||
authenticationResult = { userId: uatAuth.userId };
|
||||
const claims = await resolveAndRecheckUserActorClaims(uatAuth.claims, bearer);
|
||||
if (!claims) {
|
||||
return await wrapResponse(
|
||||
request,
|
||||
json({ error: "Invalid user-actor token" }, { status: 401 }),
|
||||
corsStrategy !== "none"
|
||||
);
|
||||
}
|
||||
await assertUserActorScope(claims, ctx, { identityOnly });
|
||||
authenticationResult = { userId: uatAuth.userId, userActor: claims };
|
||||
ability = uatAuth.ability;
|
||||
} else {
|
||||
// PAT: validate + compute the cap-and-floor ability in one query.
|
||||
@@ -721,6 +754,9 @@ type PATActionRouteBuilderOptions<
|
||||
// A single verb, or a list for multi-method routes (e.g. ["PATCH", "DELETE"]).
|
||||
method?: PATActionMethod | PATActionMethod[];
|
||||
body?: TBodySchema;
|
||||
// `identityOnly` waives the contextless refusal for reads that mutate nothing. An action
|
||||
// never qualifies, so it cannot be declared here.
|
||||
identityOnly?: never;
|
||||
};
|
||||
|
||||
type PATActionHandlerFunction<
|
||||
@@ -743,7 +779,7 @@ type PATActionHandlerFunction<
|
||||
body: TBodySchema extends z.ZodFirstPartySchemaTypes | z.ZodDiscriminatedUnion<any, any>
|
||||
? z.infer<TBodySchema>
|
||||
: undefined;
|
||||
authentication: PersonalAccessTokenAuthenticationResult;
|
||||
authentication: UserActorAuthenticatedActor;
|
||||
ability: RbacAbility;
|
||||
request: Request;
|
||||
apiVersion: API_VERSIONS;
|
||||
@@ -879,7 +915,7 @@ export function createActionPATApiRoute<
|
||||
// caller's role floor for the cap intersection (see the loader builder).
|
||||
const ctx = contextFn ? await contextFn(parsedParams, request) : {};
|
||||
|
||||
let authenticationResult: PersonalAccessTokenAuthenticationResult;
|
||||
let authenticationResult: UserActorAuthenticatedActor;
|
||||
let ability: RbacAbility;
|
||||
|
||||
const bearer = request.headers
|
||||
@@ -895,7 +931,16 @@ export function createActionPATApiRoute<
|
||||
corsStrategy !== "none"
|
||||
);
|
||||
}
|
||||
authenticationResult = { userId: uatAuth.userId };
|
||||
const claims = await resolveAndRecheckUserActorClaims(uatAuth.claims, bearer);
|
||||
if (!claims) {
|
||||
return await wrapResponse(
|
||||
request,
|
||||
json({ error: "Invalid user-actor token" }, { status: 401 }),
|
||||
corsStrategy !== "none"
|
||||
);
|
||||
}
|
||||
await assertUserActorScope(claims, ctx);
|
||||
authenticationResult = { userId: uatAuth.userId, userActor: claims };
|
||||
ability = uatAuth.ability;
|
||||
} else {
|
||||
const patAuth = await rbac.authenticatePat(request, ctx);
|
||||
@@ -1278,7 +1323,10 @@ export function createActionApiRoute<
|
||||
}
|
||||
|
||||
const result = await tenantContext.run(
|
||||
tenantContextFromAuthEnvironment(authenticationResult.environment),
|
||||
tenantContextFromAuthEnvironment(
|
||||
authenticationResult.environment,
|
||||
authenticationResult.actor
|
||||
),
|
||||
() =>
|
||||
handler({
|
||||
params: parsedParams,
|
||||
@@ -1543,7 +1591,10 @@ export function createMultiMethodApiRoute<
|
||||
|
||||
// Dispatch to method handler
|
||||
const result = await tenantContext.run(
|
||||
tenantContextFromAuthEnvironment(authenticationResult.environment),
|
||||
tenantContextFromAuthEnvironment(
|
||||
authenticationResult.environment,
|
||||
authenticationResult.actor
|
||||
),
|
||||
() =>
|
||||
methodConfig.handler({
|
||||
params: parsedParams,
|
||||
|
||||
@@ -1,12 +1,7 @@
|
||||
import { AsyncLocalStorage } from "node:async_hooks";
|
||||
import type { AuthenticatedEnvironment } from "./apiAuth.server";
|
||||
|
||||
// All fields are optional. The middleware establishes an empty scope per
|
||||
// request; entry points fill what they know:
|
||||
// - URL-matching paths get the slug trio from the Express middleware (zero IO).
|
||||
// - The `_app` layout adds `userId` for any authenticated request.
|
||||
// - The env layout adds tenant IDs / env type after its own existing DB query.
|
||||
// - API routes get the full set up-front from `authenticationResult.environment`.
|
||||
// Every field is optional: each entry point fills only what it already knows.
|
||||
export type TenantContext = {
|
||||
userId?: string;
|
||||
orgSlug?: string;
|
||||
@@ -35,9 +30,13 @@ export const tenantContext = {
|
||||
},
|
||||
};
|
||||
|
||||
export function tenantContextFromAuthEnvironment(env: AuthenticatedEnvironment): TenantContext {
|
||||
// `actor` wins over `orgMember`, which only exists on dev environments.
|
||||
export function tenantContextFromAuthEnvironment(
|
||||
env: AuthenticatedEnvironment,
|
||||
actor?: { sub: string }
|
||||
): TenantContext {
|
||||
return {
|
||||
userId: env.orgMember?.userId,
|
||||
userId: actor?.sub ?? env.orgMember?.userId,
|
||||
orgSlug: env.organization.slug,
|
||||
projectSlug: env.project.slug,
|
||||
envSlug: env.slug,
|
||||
|
||||
@@ -0,0 +1,49 @@
|
||||
import { isUserActorToken, verifyUserActorToken, type UserActorClaims } from "@trigger.dev/rbac";
|
||||
import { env } from "~/env.server";
|
||||
import { authenticateRequest, type AuthenticationResult } from "~/services/apiAuth.server";
|
||||
import { assertSourcePatActive } from "~/services/personalAccessToken.server";
|
||||
|
||||
/**
|
||||
* Auth preamble for `api.v1` routes that opt into delegated user-actor tokens alongside a PAT
|
||||
* or org token. A UAT authenticates as its user, so the result is the `personalAccessToken` shape.
|
||||
*/
|
||||
export type UatAuthentication = {
|
||||
authenticationResult: AuthenticationResult;
|
||||
/** Present only when the caller presented a user-actor token. */
|
||||
userActor?: UserActorClaims;
|
||||
};
|
||||
|
||||
export async function authenticateUatOrApiRequest(
|
||||
request: Request
|
||||
): Promise<UatAuthentication | undefined> {
|
||||
const bearer = request.headers
|
||||
.get("Authorization")
|
||||
?.replace(/^Bearer /, "")
|
||||
.trim();
|
||||
|
||||
if (bearer && isUserActorToken(bearer)) {
|
||||
const claims = await verifyUserActorToken(env.SESSION_SECRET, bearer);
|
||||
if (!claims) return undefined;
|
||||
// A token minted from a PAT dies with it — the PAT must still be live.
|
||||
if (!(await assertSourcePatActive(claims))) return undefined;
|
||||
return {
|
||||
// The claims ride on the authentication result too: resolving an environment from it
|
||||
// enforces the token's environment scope, so no route has to remember to.
|
||||
authenticationResult: {
|
||||
type: "personalAccessToken",
|
||||
result: { userId: claims.userId },
|
||||
userActor: claims,
|
||||
},
|
||||
userActor: claims,
|
||||
};
|
||||
}
|
||||
|
||||
const authenticationResult = await authenticateRequest(request, {
|
||||
personalAccessToken: true,
|
||||
organizationAccessToken: true,
|
||||
apiKey: false,
|
||||
});
|
||||
if (!authenticationResult) return undefined;
|
||||
|
||||
return { authenticationResult };
|
||||
}
|
||||
@@ -0,0 +1,126 @@
|
||||
/**
|
||||
* The Dashboard Agent uses an environment-scoped form of the existing user-actor credential.
|
||||
* MCP and the CLI may use their existing ones. Both normalize into the same authorized
|
||||
* capability context, so every route calls in here rather than deriving the rule itself.
|
||||
*
|
||||
* The rule: a token signed for one environment may only act inside it. A route that names nothing
|
||||
* to check the claim against is refused too, unless it declares itself identity-only. Anything
|
||||
* with no claim is environment-agnostic and unaffected — except a dashboard-agent token, which
|
||||
* always carries one, so its absence is a failed mint rather than a flow. Mismatches throw 403.
|
||||
*/
|
||||
|
||||
import { json } from "@remix-run/server-runtime";
|
||||
import { type UserActorClaims } from "@trigger.dev/rbac";
|
||||
import { $replica } from "~/db.server";
|
||||
|
||||
export const FORBIDDEN_ENVIRONMENT_CODE = "forbidden_environment";
|
||||
|
||||
const DASHBOARD_AGENT_CLIENT = "dashboard-agent";
|
||||
|
||||
export function assertUserActorEnvironment(
|
||||
userActor: UserActorClaims | undefined,
|
||||
environmentId: string
|
||||
): void {
|
||||
if (!userActor) return;
|
||||
if (!userActor.environmentId) {
|
||||
assertClaimIsOptional(userActor);
|
||||
return;
|
||||
}
|
||||
if (userActor.environmentId === environmentId) return;
|
||||
|
||||
throw forbiddenEnvironment("This token isn't scoped to that environment.");
|
||||
}
|
||||
|
||||
/** The same check for a route that names an org/project rather than one environment. */
|
||||
export async function assertUserActorScope(
|
||||
userActor: UserActorClaims | undefined,
|
||||
scope: { organizationId?: string; projectId?: string; environmentId?: string },
|
||||
route?: { identityOnly?: boolean }
|
||||
): Promise<void> {
|
||||
if (!userActor) return;
|
||||
|
||||
if (!userActor.environmentId) {
|
||||
assertClaimIsOptional(userActor);
|
||||
return;
|
||||
}
|
||||
|
||||
if (scope.environmentId) {
|
||||
assertUserActorEnvironment(userActor, scope.environmentId);
|
||||
return;
|
||||
}
|
||||
|
||||
// A route that names nothing offers no way to honour the claim, so it isn't reachable unless it
|
||||
// has declared itself identity-only.
|
||||
if (!scope.organizationId && !scope.projectId) {
|
||||
if (route?.identityOnly) return;
|
||||
throw forbiddenEnvironment("This token is scoped to an environment this route doesn't name.");
|
||||
}
|
||||
|
||||
const environment = await $replica.runtimeEnvironment.findFirst({
|
||||
where: { id: userActor.environmentId },
|
||||
select: { organizationId: true, projectId: true },
|
||||
});
|
||||
|
||||
// A claim naming an environment that no longer exists cannot be checked, so it isn't honoured.
|
||||
if (!environment) {
|
||||
throw forbiddenEnvironment("This token isn't scoped to an environment.");
|
||||
}
|
||||
if (scope.projectId && environment.projectId !== scope.projectId) {
|
||||
throw forbiddenEnvironment("This token isn't scoped to that project.");
|
||||
}
|
||||
if (scope.organizationId && environment.organizationId !== scope.organizationId) {
|
||||
throw forbiddenEnvironment("This token isn't scoped to that organization.");
|
||||
}
|
||||
}
|
||||
|
||||
/** `scoped: false` keeps the project-wide answer every claimless caller already gets. */
|
||||
export type UserActorEnvironmentScope =
|
||||
| { scoped: false }
|
||||
| { scoped: true; environmentId: string; slug: string; organizationId: string };
|
||||
|
||||
/**
|
||||
* The claim as a mandatory filter for a route that lists across a project. A conflicting request
|
||||
* filter is refused rather than overridden, so a caller never gets another environment's answer.
|
||||
*/
|
||||
export async function resolveUserActorEnvironmentScope(
|
||||
userActor: UserActorClaims | undefined,
|
||||
target: { projectId: string; requestedEnvironmentSlugs?: string[] }
|
||||
): Promise<UserActorEnvironmentScope> {
|
||||
if (!userActor) return { scoped: false };
|
||||
|
||||
if (!userActor.environmentId) {
|
||||
assertClaimIsOptional(userActor);
|
||||
return { scoped: false };
|
||||
}
|
||||
|
||||
const environment = await $replica.runtimeEnvironment.findFirst({
|
||||
where: { id: userActor.environmentId, projectId: target.projectId },
|
||||
select: { id: true, slug: true, organizationId: true },
|
||||
});
|
||||
|
||||
// A claim naming an environment that can't be found in this project isn't honoured.
|
||||
if (!environment) {
|
||||
throw forbiddenEnvironment("This token isn't scoped to that project.");
|
||||
}
|
||||
|
||||
const requested = target.requestedEnvironmentSlugs;
|
||||
if (requested && (requested.length !== 1 || requested[0] !== environment.slug)) {
|
||||
throw forbiddenEnvironment(`This token is scoped to the "${environment.slug}" environment.`);
|
||||
}
|
||||
|
||||
return {
|
||||
scoped: true,
|
||||
environmentId: environment.id,
|
||||
slug: environment.slug,
|
||||
organizationId: environment.organizationId,
|
||||
};
|
||||
}
|
||||
|
||||
function assertClaimIsOptional(userActor: UserActorClaims): void {
|
||||
if (userActor.client !== DASHBOARD_AGENT_CLIENT) return;
|
||||
throw forbiddenEnvironment("This token isn't scoped to an environment.");
|
||||
}
|
||||
|
||||
function forbiddenEnvironment(error: string) {
|
||||
return json({ error, code: FORBIDDEN_ENVIRONMENT_CODE }, { status: 403 });
|
||||
}
|
||||
@@ -658,12 +658,9 @@
|
||||
--sidebar: var(--color-background-dimmed);
|
||||
|
||||
/* Code block styling */
|
||||
& [data-code-block-container] {
|
||||
& [data-streamdown="code-block"] {
|
||||
@apply rounded-sm my-2 border-charcoal-700;
|
||||
}
|
||||
& [data-code-block] {
|
||||
border-top: none;
|
||||
}
|
||||
|
||||
& p {
|
||||
@apply my-1;
|
||||
@@ -701,9 +698,10 @@
|
||||
& li {
|
||||
@apply my-0.5;
|
||||
}
|
||||
/* Inline code (not in pre blocks) */
|
||||
/* Theme-mapped, not raw charcoal: fenced blocks stay dark via shiki. */
|
||||
& code:not(pre code) {
|
||||
@apply bg-charcoal-700 px-1 py-0.5 rounded-sm text-text-bright font-mono;
|
||||
@apply px-1 py-0.5 rounded-sm text-text-bright font-mono;
|
||||
background-color: var(--muted);
|
||||
}
|
||||
& blockquote {
|
||||
@apply border-l-2 border-charcoal-600 pl-3 my-2 italic;
|
||||
@@ -725,25 +723,16 @@
|
||||
}
|
||||
& th,
|
||||
& td {
|
||||
@apply border border-charcoal-600 px-2 py-1 text-left;
|
||||
@apply border border-grid-bright px-2 py-1 text-left;
|
||||
}
|
||||
& th {
|
||||
@apply bg-charcoal-700 font-semibold;
|
||||
@apply font-semibold;
|
||||
background-color: var(--muted);
|
||||
}
|
||||
|
||||
& [data-code-block-header] {
|
||||
@apply bg-charcoal-800 text-text-dimmed border-b border-charcoal-700;
|
||||
}
|
||||
/* Override the bg-muted/40 class to let inline styles work */
|
||||
& [data-code-block] pre {
|
||||
background-color: inherit !important;
|
||||
@apply scrollbar-thin scrollbar-track-transparent scrollbar-thumb-charcoal-600;
|
||||
}
|
||||
& [data-code-block] pre code {
|
||||
@apply bg-transparent;
|
||||
}
|
||||
& [data-code-block] .line {
|
||||
@apply leading-relaxed;
|
||||
/* The body scrolls and sizes the code; streamdown ships it at text-sm. */
|
||||
& [data-streamdown="code-block-body"] {
|
||||
@apply text-xs leading-relaxed scrollbar-thin scrollbar-track-transparent scrollbar-thumb-charcoal-600;
|
||||
}
|
||||
}
|
||||
|
||||
|
||||
@@ -0,0 +1,45 @@
|
||||
import { describe, expect, it } from "vitest";
|
||||
import { readBoundedBodyText } from "./boundedRequestBody.server";
|
||||
|
||||
/** A body with no `content-length`, delivered in chunks, counting what was pulled. */
|
||||
function streamed(chunkCount: number, chunkBytes: number) {
|
||||
let pulled = 0;
|
||||
const body = new ReadableStream<Uint8Array>({
|
||||
pull(controller) {
|
||||
if (pulled >= chunkCount) {
|
||||
controller.close();
|
||||
return;
|
||||
}
|
||||
pulled += 1;
|
||||
controller.enqueue(new Uint8Array(chunkBytes).fill(97));
|
||||
},
|
||||
});
|
||||
|
||||
const request = new Request("http://localhost/in", {
|
||||
method: "POST",
|
||||
body,
|
||||
// @ts-expect-error — required for a streamed request body.
|
||||
duplex: "half",
|
||||
});
|
||||
return { request, pulled: () => pulled };
|
||||
}
|
||||
|
||||
describe("readBoundedBodyText", () => {
|
||||
it("returns a body under the limit", async () => {
|
||||
const { request } = streamed(2, 8);
|
||||
expect(await readBoundedBodyText(request, 1024)).toEqual({ ok: true, text: "a".repeat(16) });
|
||||
});
|
||||
|
||||
it("stops reading as soon as the limit is crossed", async () => {
|
||||
const { request, pulled } = streamed(100, 64);
|
||||
|
||||
expect(await readBoundedBodyText(request, 128)).toEqual({ ok: false, reason: "too_large" });
|
||||
// Three chunks: two fit, the third crossed it. Nothing beyond was ever pulled.
|
||||
expect(pulled()).toBe(3);
|
||||
});
|
||||
|
||||
it("treats a missing body as empty", async () => {
|
||||
const request = new Request("http://localhost/in", { method: "POST" });
|
||||
expect(await readBoundedBodyText(request, 8)).toEqual({ ok: true, text: "" });
|
||||
});
|
||||
});
|
||||
@@ -0,0 +1,35 @@
|
||||
/**
|
||||
* Reading a request body with a ceiling. `request.text()` buffers the whole body before the
|
||||
* caller can look at its size, so a route that only checks afterwards has already paid for it.
|
||||
*/
|
||||
export type BoundedBody = { ok: true; text: string } | { ok: false; reason: "too_large" };
|
||||
|
||||
/** Stops at the first chunk that crosses `maxBytes` and cancels the stream. */
|
||||
export async function readBoundedBodyText(
|
||||
request: Request,
|
||||
maxBytes: number
|
||||
): Promise<BoundedBody> {
|
||||
if (!request.body) return { ok: true, text: "" };
|
||||
|
||||
const reader = request.body.getReader();
|
||||
const chunks: Uint8Array[] = [];
|
||||
let received = 0;
|
||||
|
||||
try {
|
||||
for (;;) {
|
||||
const { done, value } = await reader.read();
|
||||
if (done) break;
|
||||
if (!value) continue;
|
||||
received += value.byteLength;
|
||||
if (received > maxBytes) {
|
||||
await reader.cancel();
|
||||
return { ok: false, reason: "too_large" };
|
||||
}
|
||||
chunks.push(value);
|
||||
}
|
||||
} finally {
|
||||
reader.releaseLock();
|
||||
}
|
||||
|
||||
return { ok: true, text: Buffer.concat(chunks).toString("utf8") };
|
||||
}
|
||||
@@ -0,0 +1,164 @@
|
||||
import { describe, expect, it } from "vitest";
|
||||
import { faviconUrl } from "./favicon";
|
||||
import {
|
||||
BASE_IMG_SRC_SOURCES,
|
||||
buildImgSrcDirective,
|
||||
parseCspImageOrigins,
|
||||
withImgSrc,
|
||||
} from "./cspImageOrigins";
|
||||
|
||||
/** True if a source expression in the directive would match the given image URL. */
|
||||
function directivePermits(directive: string, imageUrl: string): boolean {
|
||||
const url = new URL(imageUrl);
|
||||
return directive
|
||||
.split(" ")
|
||||
.slice(1)
|
||||
.some((source) => {
|
||||
if (!source.startsWith("http")) return false;
|
||||
const parsed = new URL(source);
|
||||
if (parsed.protocol !== url.protocol || parsed.host !== url.host) return false;
|
||||
return parsed.pathname === "/" || parsed.pathname === url.pathname;
|
||||
});
|
||||
}
|
||||
|
||||
describe("parseCspImageOrigins", () => {
|
||||
it("accepts exact https origins, with or without a port", () => {
|
||||
const { origins, rejected } = parseCspImageOrigins(
|
||||
"https://sso.example.com, https://images.example.com:8443"
|
||||
);
|
||||
expect(origins).toEqual(["https://sso.example.com", "https://images.example.com:8443"]);
|
||||
expect(rejected).toEqual([]);
|
||||
});
|
||||
|
||||
it("returns nothing when unset or empty", () => {
|
||||
expect(parseCspImageOrigins(undefined).origins).toEqual([]);
|
||||
expect(parseCspImageOrigins(" , ,").origins).toEqual([]);
|
||||
});
|
||||
|
||||
it("deduplicates repeated origins", () => {
|
||||
const { origins } = parseCspImageOrigins(
|
||||
"https://sso.example.com,https://sso.example.com/,https://sso.example.com"
|
||||
);
|
||||
expect(origins).toEqual(["https://sso.example.com"]);
|
||||
});
|
||||
|
||||
it.each([
|
||||
["*", "wildcards are not allowed, list each origin exactly"],
|
||||
["https://*.example.com", "wildcards are not allowed, list each origin exactly"],
|
||||
["https://example.com/avatars", "must be an origin only, with no path, query or hash"],
|
||||
["https://example.com?x=1", "must be an origin only, with no path, query or hash"],
|
||||
["https://example.com#frag", "must be an origin only, with no path, query or hash"],
|
||||
["example.com", "is not a valid absolute URL"],
|
||||
["https://user:pw@example.com", "must not contain credentials"],
|
||||
["https://a.com;script-src", "must not contain ';' or ',' — these delimit CSP directives"],
|
||||
])("rejects %s and says why", (value, reason) => {
|
||||
const { origins, rejected } = parseCspImageOrigins(value);
|
||||
expect(origins).toEqual([]);
|
||||
expect(rejected).toEqual([{ value, reason }]);
|
||||
});
|
||||
|
||||
it("does not let a ';' smuggle a second directive into img-src", () => {
|
||||
const { origins } = parseCspImageOrigins("https://a.com;script-src");
|
||||
expect(origins).toEqual([]);
|
||||
expect(buildImgSrcDirective(origins)).not.toContain("script-src");
|
||||
});
|
||||
|
||||
it("splits on ',' so a comma can never ride inside a single origin", () => {
|
||||
// "https://a.com" is valid; the "x" fragment after the comma is rejected on its own.
|
||||
const { origins } = parseCspImageOrigins("https://a.com,x");
|
||||
expect(origins).toEqual(["https://a.com"]);
|
||||
expect(origins.some((origin) => origin.includes(","))).toBe(false);
|
||||
});
|
||||
|
||||
it("keeps the valid entries when a sibling entry is rejected", () => {
|
||||
const { origins, rejected } = parseCspImageOrigins(
|
||||
"https://*.evil.com,https://sso.example.com"
|
||||
);
|
||||
expect(origins).toEqual(["https://sso.example.com"]);
|
||||
expect(rejected.map((entry) => entry.value)).toEqual(["https://*.evil.com"]);
|
||||
});
|
||||
|
||||
it("rejects http outside development", () => {
|
||||
const { origins, rejected } = parseCspImageOrigins("http://sso.example.com");
|
||||
expect(origins).toEqual([]);
|
||||
expect(rejected).toEqual([
|
||||
{ value: "http://sso.example.com", reason: 'scheme "http:" is not https:' },
|
||||
]);
|
||||
});
|
||||
|
||||
it("allows http only when allowHttp is set", () => {
|
||||
const { origins, rejected } = parseCspImageOrigins("http://localhost:4000", {
|
||||
allowHttp: true,
|
||||
});
|
||||
expect(origins).toEqual(["http://localhost:4000"]);
|
||||
expect(rejected).toEqual([]);
|
||||
});
|
||||
|
||||
it("rejects a non-http scheme even when allowHttp is set", () => {
|
||||
const { origins, rejected } = parseCspImageOrigins("ftp://example.com", { allowHttp: true });
|
||||
expect(origins).toEqual([]);
|
||||
expect(rejected).toEqual([
|
||||
{ value: "ftp://example.com", reason: 'scheme "ftp:" is not http: or https:' },
|
||||
]);
|
||||
});
|
||||
});
|
||||
|
||||
describe("buildImgSrcDirective", () => {
|
||||
it("is self, data, blob, the SSO avatar hosts and the favicon endpoint by default", () => {
|
||||
expect(buildImgSrcDirective()).toBe(
|
||||
"img-src 'self' data: blob: https://avatars.githubusercontent.com https://lh3.googleusercontent.com https://www.google.com/s2/favicons"
|
||||
);
|
||||
});
|
||||
|
||||
it("permits the org avatar URL the app actually stores", () => {
|
||||
expect(directivePermits(buildImgSrcDirective(), faviconUrl("example.com"))).toBe(true);
|
||||
});
|
||||
|
||||
it("permits nothing else on the favicon host", () => {
|
||||
expect(directivePermits(buildImgSrcDirective(), "https://www.google.com/beacon.png")).toBe(
|
||||
false
|
||||
);
|
||||
});
|
||||
|
||||
it("permits both OAuth avatar hosts", () => {
|
||||
const directive = buildImgSrcDirective();
|
||||
expect(directivePermits(directive, "https://avatars.githubusercontent.com/u/1?v=4")).toBe(true);
|
||||
expect(directivePermits(directive, "https://lh3.googleusercontent.com/a/abc=s96-c")).toBe(true);
|
||||
});
|
||||
|
||||
it("has no wildcard host and no bare scheme host", () => {
|
||||
const directive = buildImgSrcDirective(parseCspImageOrigins("https://sso.example.com").origins);
|
||||
expect(directive).not.toContain("*");
|
||||
// The avatar hosts are exact origins; a wildcard over them would not be.
|
||||
expect(directive).not.toContain("*.googleusercontent.com");
|
||||
expect(directive).not.toMatch(/(^|\s)https?:(\s|$)/);
|
||||
});
|
||||
|
||||
it("appends configured origins after the base sources", () => {
|
||||
expect(buildImgSrcDirective(["https://sso.example.com"])).toBe(
|
||||
`img-src ${BASE_IMG_SRC_SOURCES.join(" ")} https://sso.example.com`
|
||||
);
|
||||
});
|
||||
});
|
||||
|
||||
describe("withImgSrc", () => {
|
||||
const directive = buildImgSrcDirective();
|
||||
|
||||
it("is the whole policy when a route set nothing", () => {
|
||||
expect(withImgSrc(null, directive)).toBe(directive);
|
||||
});
|
||||
|
||||
it("appends to a policy that has other directives", () => {
|
||||
expect(withImgSrc("frame-ancestors 'self';", directive)).toBe(
|
||||
`frame-ancestors 'self'; ${directive}`
|
||||
);
|
||||
});
|
||||
|
||||
it("leaves a route's own img-src untouched", () => {
|
||||
const routePolicy = "img-src 'none'";
|
||||
expect(withImgSrc(routePolicy, directive)).toBe(routePolicy);
|
||||
expect(withImgSrc("default-src 'self'; img-src 'none'", directive)).toBe(
|
||||
"default-src 'self'; img-src 'none'"
|
||||
);
|
||||
});
|
||||
});
|
||||
@@ -0,0 +1,120 @@
|
||||
/**
|
||||
* The document `img-src` allowlist. Remote images are a beacon channel: rendering
|
||||
* one is the outbound request, no click needed. So no wildcard host and no bare
|
||||
* scheme. Operator-supplied entries are exact origins; a base source may pin a path
|
||||
* to narrow the host further.
|
||||
*/
|
||||
|
||||
/**
|
||||
* Always allowed: own origin, inline data, object URLs, the SSO avatar hosts, and the
|
||||
* favicon endpoint org avatars are stored as (see `utils/favicon.ts`). The path pins
|
||||
* that one endpoint — CSP matches the path and ignores the query string.
|
||||
*/
|
||||
export const BASE_IMG_SRC_SOURCES = [
|
||||
"'self'",
|
||||
"data:",
|
||||
"blob:",
|
||||
"https://avatars.githubusercontent.com",
|
||||
"https://lh3.googleusercontent.com",
|
||||
"https://www.google.com/s2/favicons",
|
||||
] as const;
|
||||
|
||||
export type RejectedOrigin = { value: string; reason: string };
|
||||
|
||||
export type ParsedImageOrigins = {
|
||||
/** Accepted, canonicalised (`scheme://host[:port]`) and deduplicated. */
|
||||
origins: string[];
|
||||
rejected: RejectedOrigin[];
|
||||
};
|
||||
|
||||
export type ParseImageOriginsOptions = {
|
||||
/** Only a local development deployment may serve images over plain http. */
|
||||
allowHttp?: boolean;
|
||||
};
|
||||
|
||||
/**
|
||||
* Parses a comma-separated `CSP_IMG_SRC_ALLOWLIST`. Never throws: bad entries are
|
||||
* reported in `rejected` so the caller can warn and boot with the valid ones.
|
||||
*/
|
||||
export function parseCspImageOrigins(
|
||||
raw: string | undefined | null,
|
||||
options: ParseImageOriginsOptions = {}
|
||||
): ParsedImageOrigins {
|
||||
const allowHttp = options.allowHttp ?? false;
|
||||
const origins: string[] = [];
|
||||
const seen = new Set<string>();
|
||||
const rejected: RejectedOrigin[] = [];
|
||||
|
||||
for (const entry of (raw ?? "").split(",")) {
|
||||
const value = entry.trim();
|
||||
if (value.length === 0) continue;
|
||||
|
||||
const reason = rejectionReason(value, allowHttp);
|
||||
if (reason) {
|
||||
rejected.push({ value, reason });
|
||||
continue;
|
||||
}
|
||||
|
||||
const url = new URL(value);
|
||||
const origin = `${url.protocol}//${url.host}`;
|
||||
if (seen.has(origin)) continue;
|
||||
seen.add(origin);
|
||||
origins.push(origin);
|
||||
}
|
||||
|
||||
return { origins, rejected };
|
||||
}
|
||||
|
||||
/** Returns why the entry is not an acceptable origin, or undefined if it is one. */
|
||||
function rejectionReason(value: string, allowHttp: boolean): string | undefined {
|
||||
if (value.includes("*")) {
|
||||
return "wildcards are not allowed, list each origin exactly";
|
||||
}
|
||||
if (/\s/.test(value)) {
|
||||
return "contains whitespace";
|
||||
}
|
||||
// `;` and `,` delimit CSP directives / source lists; an entry containing one would
|
||||
// land verbatim in the space-joined img-src and inject or truncate a directive.
|
||||
if (/[;,]/.test(value)) {
|
||||
return "must not contain ';' or ',' — these delimit CSP directives";
|
||||
}
|
||||
|
||||
let url: URL;
|
||||
try {
|
||||
url = new URL(value);
|
||||
} catch {
|
||||
return "is not a valid absolute URL";
|
||||
}
|
||||
|
||||
const allowedProtocols = allowHttp ? ["https:", "http:"] : ["https:"];
|
||||
if (!allowedProtocols.includes(url.protocol)) {
|
||||
return allowHttp
|
||||
? `scheme "${url.protocol}" is not http: or https:`
|
||||
: `scheme "${url.protocol}" is not https:`;
|
||||
}
|
||||
if (url.host.length === 0) {
|
||||
return "has no host";
|
||||
}
|
||||
if (url.username.length > 0 || url.password.length > 0) {
|
||||
return "must not contain credentials";
|
||||
}
|
||||
if (url.pathname !== "/" || url.search.length > 0 || url.hash.length > 0) {
|
||||
return "must be an origin only, with no path, query or hash";
|
||||
}
|
||||
return undefined;
|
||||
}
|
||||
|
||||
/** The full directive: the base sources plus any configured extra origins. */
|
||||
export function buildImgSrcDirective(extraOrigins: readonly string[] = []): string {
|
||||
return ["img-src", ...BASE_IMG_SRC_SOURCES, ...extraOrigins].join(" ");
|
||||
}
|
||||
|
||||
/**
|
||||
* Appends the directive to whatever a route already set, rather than replacing it.
|
||||
* A route that set its own `img-src` keeps it.
|
||||
*/
|
||||
export function withImgSrc(existing: string | null | undefined, directive: string): string {
|
||||
if (!existing) return directive;
|
||||
if (/(^|;)\s*img-src\s/.test(existing)) return existing;
|
||||
return `${existing.replace(/;\s*$/, "")}; ${directive}`;
|
||||
}
|
||||
@@ -4,20 +4,16 @@ import { FEATURE_FLAG } from "~/v3/featureFlags";
|
||||
import { makeFlag } from "~/v3/featureFlags.server";
|
||||
|
||||
/**
|
||||
* Whether the in-dashboard AI agent is available to this user in this org.
|
||||
* Gated by the global / per-org `hasDashboardAgentAccess` flag, with
|
||||
* `DASHBOARD_AGENT_ENABLED` as the global default (a per-org override wins).
|
||||
* Admins/impersonators bypass it only when `DASHBOARD_AGENT_ADMIN_PREVIEW` is on
|
||||
* (default off). Enforced server-side so a non-flagged user can't start sessions.
|
||||
* Whether the in-dashboard AI agent is available to this user in this org, per the
|
||||
* `hasDashboardAgentAccess` flag with a per-org override winning. Must stay server-side.
|
||||
* Both env defaults are off, so an unflagged install has no agent and can start no session.
|
||||
*/
|
||||
export async function canAccessDashboardAgent(options: {
|
||||
userId: string;
|
||||
isAdmin: boolean;
|
||||
isImpersonating: boolean;
|
||||
organizationSlug: string;
|
||||
// When the caller already has the org's `featureFlags` loaded (e.g. a layout
|
||||
// loader that queried the org with a membership check), pass them to skip the
|
||||
// extra org lookup. Omit it and we query the org ourselves.
|
||||
// The org's already-loaded `featureFlags`. Omitted means we query the org ourselves.
|
||||
orgFeatureFlags?: Record<string, unknown> | null;
|
||||
}): Promise<boolean> {
|
||||
const { userId, isAdmin, isImpersonating, organizationSlug, orgFeatureFlags } = options;
|
||||
|
||||
@@ -1,5 +1,5 @@
|
||||
import { Logger } from "@trigger.dev/core/logger";
|
||||
import { Worker as RedisWorker } from "@trigger.dev/redis-worker";
|
||||
import { CronSchema, Worker as RedisWorker } from "@trigger.dev/redis-worker";
|
||||
import { DeliverEmailSchema } from "emails";
|
||||
import { z } from "zod";
|
||||
import { env } from "~/env.server";
|
||||
@@ -11,6 +11,12 @@ import {
|
||||
runAttioUserSync,
|
||||
runAttioWorkspaceSync,
|
||||
} from "~/services/attio.server";
|
||||
import {
|
||||
purgeDashboardAgentChatsForOrganization,
|
||||
sweepDashboardAgentSoftDeletedChats,
|
||||
} from "~/services/dashboardAgentChatRetention.server";
|
||||
import { sweepDashboardAgentTurnEvals } from "~/services/dashboardAgentEvalRetention.server";
|
||||
import { sweepDashboardAgentInvestigations } from "~/services/dashboardAgentInvestigationSweep.server";
|
||||
import { logger } from "~/services/logger.server";
|
||||
import {
|
||||
MembershipDevEnvironmentsSchema,
|
||||
@@ -37,6 +43,11 @@ function initializeWorker() {
|
||||
|
||||
logger.debug(`👨🏭 Initializing common worker at host ${env.COMMON_WORKER_REDIS_HOST}`);
|
||||
|
||||
// Only schedule the agent maintenance cron where the agent is actually set up. Otherwise
|
||||
// its sweeps hit a missing schema and drip a dead-letter entry every run.
|
||||
const dashboardAgentConfigured =
|
||||
env.DASHBOARD_AGENT_ENABLED === "1" || Boolean(env.DASHBOARD_AGENT_DATABASE_URL);
|
||||
|
||||
const worker = new RedisWorker({
|
||||
name: "common-worker",
|
||||
redisOptions,
|
||||
@@ -146,6 +157,25 @@ function initializeWorker() {
|
||||
maxAttempts: 5,
|
||||
},
|
||||
},
|
||||
// Stuck investigation cards and turn-eval retention.
|
||||
"dashboardAgent.maintenance": {
|
||||
schema: CronSchema,
|
||||
visibilityTimeoutMs: 60_000 * 5,
|
||||
...(dashboardAgentConfigured ? { cron: "*/5 * * * *", jitterInMs: 30_000 } : {}),
|
||||
retry: {
|
||||
maxAttempts: 1,
|
||||
},
|
||||
},
|
||||
// Soft-deletes a deleted organization's chats; the maintenance sweep purges them.
|
||||
"dashboardAgent.purgeOrganization": {
|
||||
schema: z.object({
|
||||
organizationId: z.string(),
|
||||
}),
|
||||
visibilityTimeoutMs: 60_000,
|
||||
retry: {
|
||||
maxAttempts: 5,
|
||||
},
|
||||
},
|
||||
},
|
||||
concurrency: {
|
||||
workers: env.COMMON_WORKER_CONCURRENCY_WORKERS,
|
||||
@@ -204,6 +234,52 @@ function initializeWorker() {
|
||||
const service = new BulkActionService();
|
||||
await service.process(payload.bulkActionId);
|
||||
},
|
||||
"dashboardAgent.maintenance": async () => {
|
||||
// Each backstop runs independently; the first failure is rethrown at the end.
|
||||
let failure: unknown;
|
||||
|
||||
try {
|
||||
const investigations = await sweepDashboardAgentInvestigations();
|
||||
if (investigations.stale > 0) {
|
||||
logger.debug("Dashboard agent investigation sweep", investigations);
|
||||
}
|
||||
} catch (error) {
|
||||
failure ??= error;
|
||||
}
|
||||
|
||||
// Retention on the judged-turn rows. Independent of the agent being configured.
|
||||
try {
|
||||
const evals = await sweepDashboardAgentTurnEvals();
|
||||
if (evals.purged > 0) {
|
||||
logger.debug("Dashboard agent turn-eval retention", evals);
|
||||
}
|
||||
} catch (error) {
|
||||
failure ??= error;
|
||||
}
|
||||
|
||||
// Hard-delete chats soft-deleted past the retention window, with their children.
|
||||
try {
|
||||
const chats = await sweepDashboardAgentSoftDeletedChats();
|
||||
if (chats.purged > 0) {
|
||||
logger.debug("Dashboard agent chat retention", chats);
|
||||
}
|
||||
} catch (error) {
|
||||
failure ??= error;
|
||||
}
|
||||
|
||||
if (failure) throw failure;
|
||||
},
|
||||
"dashboardAgent.purgeOrganization": async ({ payload }) => {
|
||||
const soft = await purgeDashboardAgentChatsForOrganization({
|
||||
organizationId: payload.organizationId,
|
||||
});
|
||||
if (soft > 0) {
|
||||
logger.debug("Dashboard agent organization purge", {
|
||||
organizationId: payload.organizationId,
|
||||
softDeleted: soft,
|
||||
});
|
||||
}
|
||||
},
|
||||
},
|
||||
});
|
||||
|
||||
|
||||
@@ -103,6 +103,9 @@ export function detectQueryTables(query: string, allowedTableNames: Set<string>)
|
||||
visitSelectSet(tableExpr as SelectSetQuery);
|
||||
}
|
||||
}
|
||||
// The `ON expr` can embed a SELECT that reads a real table, e.g.
|
||||
// `JOIN x ON id IN (SELECT … FROM runs)`.
|
||||
scanForSubqueries(node.constraint);
|
||||
if (node.next_join) visitJoin(node.next_join);
|
||||
}
|
||||
|
||||
|
||||
@@ -25,22 +25,26 @@ export function makeFlag(_prisma: PrismaClientOrTransaction = prisma) {
|
||||
async function flag<T extends FeatureFlagKey>(
|
||||
opts: FlagsOptions<T>
|
||||
): Promise<z.infer<(typeof FeatureFlagCatalog)[T]> | undefined> {
|
||||
const flagSchema = FeatureFlagCatalog[opts.key];
|
||||
|
||||
const override = opts.overrides?.[opts.key];
|
||||
|
||||
if (override !== undefined) {
|
||||
const parsed = flagSchema.safeParse(override);
|
||||
|
||||
if (parsed.success) {
|
||||
return parsed.data;
|
||||
}
|
||||
|
||||
// an override that fails the schema is ignored: the global value still wins
|
||||
}
|
||||
|
||||
const value = await _prisma.featureFlag.findFirst({
|
||||
where: {
|
||||
key: opts.key,
|
||||
},
|
||||
});
|
||||
|
||||
const flagSchema = FeatureFlagCatalog[opts.key];
|
||||
|
||||
if (opts.overrides?.[opts.key] !== undefined) {
|
||||
const parsed = flagSchema.safeParse(opts.overrides[opts.key]);
|
||||
|
||||
if (parsed.success) {
|
||||
return parsed.data;
|
||||
}
|
||||
}
|
||||
|
||||
if (value !== null) {
|
||||
const parsed = flagSchema.safeParse(value.value);
|
||||
|
||||
|
||||
@@ -7,6 +7,8 @@ export const FEATURE_FLAG = {
|
||||
hasLogsPageAccess: "hasLogsPageAccess",
|
||||
hasAiAccess: "hasAiAccess",
|
||||
hasDashboardAgentAccess: "hasDashboardAgentAccess",
|
||||
dashboardAgentTurnEvalsEnabled: "dashboardAgentTurnEvalsEnabled",
|
||||
promotedDashboardAgentPrompt: "promotedDashboardAgentPrompt",
|
||||
hasComputeAccess: "hasComputeAccess",
|
||||
hasPrivateConnections: "hasPrivateConnections",
|
||||
hasSso: "hasSso",
|
||||
@@ -43,6 +45,15 @@ export const FeatureFlagCatalog = {
|
||||
// Gates the in-dashboard AI agent panel. Controllable globally and per-org
|
||||
// (org wins). Defaults off via DASHBOARD_AGENT_ENABLED.
|
||||
[FEATURE_FLAG.hasDashboardAgentAccess]: z.coerce.boolean(),
|
||||
// Whether this org's agent turns may be sampled for the quality judge. A data-handling
|
||||
// switch, not an entitlement: an org that turns it off has its turns judged never, and a
|
||||
// setting that can't be read is treated as off. Per-org override wins; on by default.
|
||||
// Strict z.boolean(): coercion reads the string "false" as true, which would keep judging
|
||||
// an org that asked us to stop.
|
||||
[FEATURE_FLAG.dashboardAgentTurnEvalsEnabled]: z.boolean(),
|
||||
// A JSON string because this catalog is scalar-only. Validated where it's read, in
|
||||
// `suggested-prompts/promotedPrompt.server.ts`.
|
||||
[FEATURE_FLAG.promotedDashboardAgentPrompt]: z.string(),
|
||||
[FEATURE_FLAG.hasComputeAccess]: z.coerce.boolean(),
|
||||
[FEATURE_FLAG.hasPrivateConnections]: z.coerce.boolean(),
|
||||
[FEATURE_FLAG.hasSso]: z.coerce.boolean(),
|
||||
@@ -162,6 +173,26 @@ export function resolveInternalApiOriginEnabled({
|
||||
return globalDefault;
|
||||
}
|
||||
|
||||
/**
|
||||
* Whether the org set `dashboardAgentTurnEvalsEnabled` to something the schema rejects.
|
||||
* That flag is a consent switch, not an entitlement, so an unreadable override must not fall
|
||||
* through to the global default the way `resolveInternalApiOriginEnabled` does: the org that
|
||||
* wrote it was trying to say something, and the only safe reading of an unknown answer is no.
|
||||
*/
|
||||
export function hasUnreadableTurnEvalsOverride(orgFeatureFlags: unknown): boolean {
|
||||
if (!orgFeatureFlags || typeof orgFeatureFlags !== "object" || Array.isArray(orgFeatureFlags)) {
|
||||
return false;
|
||||
}
|
||||
|
||||
const override = (orgFeatureFlags as Record<string, unknown>)[
|
||||
FEATURE_FLAG.dashboardAgentTurnEvalsEnabled
|
||||
];
|
||||
if (override === undefined) return false;
|
||||
|
||||
return !FeatureFlagCatalog[FEATURE_FLAG.dashboardAgentTurnEvalsEnabled].safeParse(override)
|
||||
.success;
|
||||
}
|
||||
|
||||
export type FlagControlType =
|
||||
| { type: "boolean" }
|
||||
| { type: "enum"; options: string[] }
|
||||
|
||||
@@ -0,0 +1,42 @@
|
||||
import type { QueryScope } from "~/v3/querySchemas";
|
||||
|
||||
/**
|
||||
* The widest scope a credential may query at.
|
||||
*
|
||||
* `executeQuery` always isolates by organization and widens or narrows from there
|
||||
* on the caller's `scope`. That makes the request body, not the credential, the
|
||||
* ceiling — which is wrong for a public access token: it is minted for one
|
||||
* environment and handed to a browser, so anyone holding it could read the whole
|
||||
* organization's analytics by changing one field.
|
||||
*
|
||||
* A secret key is deliberately NOT capped here. It is a server-side credential the
|
||||
* organization's own owner installs, and capping it would change the public API's
|
||||
* behaviour for callers who query at organization scope today. A session or PAT
|
||||
* caller (the Query page) never goes through this: it picks its scope in the UI,
|
||||
* authorized by organization membership.
|
||||
*/
|
||||
export type QueryScopeCeiling = "environment" | "unbounded";
|
||||
|
||||
export type QueryScopeDecision = { ok: true; scope: QueryScope } | { ok: false; error: string };
|
||||
|
||||
/**
|
||||
* Rejected rather than narrowed. Silently answering about one environment when the
|
||||
* caller asked about the organization gives them a number that means something else,
|
||||
* with nothing in the response to say so.
|
||||
*/
|
||||
export function resolveQueryScope(args: {
|
||||
ceiling: QueryScopeCeiling;
|
||||
requested: QueryScope;
|
||||
}): QueryScopeDecision {
|
||||
if (args.ceiling === "unbounded") return { ok: true, scope: args.requested };
|
||||
if (args.requested === "environment") return { ok: true, scope: "environment" };
|
||||
return {
|
||||
ok: false,
|
||||
error: `This token is scoped to one environment, so it can't run a ${args.requested}-scoped query. Use scope "environment", or a secret key.`,
|
||||
};
|
||||
}
|
||||
|
||||
/** A public access token is environment-bound; every other bearer credential isn't. */
|
||||
export function queryScopeCeilingFor(authenticationType: string): QueryScopeCeiling {
|
||||
return authenticationType === "PUBLIC_JWT" ? "environment" : "unbounded";
|
||||
}
|
||||
@@ -0,0 +1,49 @@
|
||||
/**
|
||||
* `getQueueDepthSparklines` emits a row only for buckets that reported, so a caller has to place
|
||||
* every row on the bucket grid itself. Depth is carry-forward filled: no emission means unchanged,
|
||||
* not zero. Throttled is not filled — only real per-bucket counts tint a bar.
|
||||
*/
|
||||
|
||||
export type QueueDepthBucketRow = { bucket: string; depth: number; throttled: number };
|
||||
|
||||
export type QueueDepthGrid = { startMs: number; bucketIntervalMs: number; numBuckets: number };
|
||||
|
||||
/** Rows placed on the grid by bucket index. Rows outside the window are dropped. */
|
||||
export function indexQueueDepthRows(
|
||||
rows: QueueDepthBucketRow[],
|
||||
grid: QueueDepthGrid
|
||||
): Map<number, { depth: number; throttled: number }> {
|
||||
const byIndex = new Map<number, { depth: number; throttled: number }>();
|
||||
for (const row of rows) {
|
||||
const bucketMs = Date.parse(row.bucket.replace(" ", "T") + "Z");
|
||||
if (Number.isNaN(bucketMs)) continue;
|
||||
const index = Math.round((bucketMs - grid.startMs) / grid.bucketIntervalMs);
|
||||
if (index < 0 || index >= grid.numBuckets) continue;
|
||||
byIndex.set(index, { depth: row.depth, throttled: row.throttled });
|
||||
}
|
||||
return byIndex;
|
||||
}
|
||||
|
||||
/** A fixed-width series per grid bucket, so a gap can never shift later points in time. */
|
||||
export function fillQueueDepthSeries(
|
||||
byIndex: Map<number, { depth: number; throttled: number }>,
|
||||
numBuckets: number
|
||||
): { depth: number[]; throttled: number[] } {
|
||||
const depth: number[] = new Array(numBuckets);
|
||||
const throttled: number[] = new Array(numBuckets);
|
||||
let last = 0;
|
||||
for (let i = 0; i < numBuckets; i++) {
|
||||
const bucket = byIndex.get(i);
|
||||
if (bucket !== undefined) last = bucket.depth;
|
||||
depth[i] = last;
|
||||
throttled[i] = bucket?.throttled ?? 0;
|
||||
}
|
||||
return { depth, throttled };
|
||||
}
|
||||
|
||||
export function queueDepthSeries(
|
||||
rows: QueueDepthBucketRow[],
|
||||
grid: QueueDepthGrid
|
||||
): { depth: number[]; throttled: number[] } {
|
||||
return fillQueueDepthSeries(indexQueueDepthRows(rows, grid), grid.numBuckets);
|
||||
}
|
||||
@@ -392,6 +392,7 @@ export class DeliverAlertService extends BaseService {
|
||||
break;
|
||||
}
|
||||
case "ERROR_GROUP": {
|
||||
// Payload-carried alert types create no ProjectAlert row, so never seen here.
|
||||
break;
|
||||
}
|
||||
default: {
|
||||
@@ -747,6 +748,7 @@ export class DeliverAlertService extends BaseService {
|
||||
break;
|
||||
}
|
||||
case "ERROR_GROUP": {
|
||||
// Payload-carried alert types create no ProjectAlert row, so never seen here.
|
||||
break;
|
||||
}
|
||||
default: {
|
||||
@@ -1023,6 +1025,7 @@ export class DeliverAlertService extends BaseService {
|
||||
}
|
||||
}
|
||||
case "ERROR_GROUP": {
|
||||
// Payload-carried alert types create no ProjectAlert row, so never seen here.
|
||||
break;
|
||||
}
|
||||
default: {
|
||||
|
||||
@@ -5,7 +5,7 @@
|
||||
"sideEffects": false,
|
||||
"scripts": {
|
||||
"build": "run-s build:** && pnpm run upload:sourcemaps",
|
||||
"build:remix": "remix vite:build",
|
||||
"build:remix": "cross-env NODE_OPTIONS=\"--max-old-space-size=8192\" remix vite:build",
|
||||
"build:server": "esbuild --platform=node --format=cjs ./server.ts --outdir=build --sourcemap",
|
||||
"build:otlpworker": "esbuild --platform=node --format=cjs --bundle ./app/v3/otlpTransformWorker.ts --outfile=build/otlpTransformWorker.cjs --sourcemap",
|
||||
"build:sentry": "esbuild --platform=node --format=cjs --outbase=. ./sentry.server.ts ./app/utils/sentryTraceContext.server.ts --outdir=build --sourcemap",
|
||||
@@ -54,6 +54,7 @@
|
||||
"@internal/cache": "workspace:*",
|
||||
"@internal/compute": "workspace:*",
|
||||
"@internal/dashboard-agent": "workspace:*",
|
||||
"@internal/dashboard-agent-contracts": "workspace:*",
|
||||
"@internal/dashboard-agent-db": "workspace:*",
|
||||
"@internal/llm-model-catalog": "workspace:*",
|
||||
"@internal/metrics-pipeline": "workspace:*",
|
||||
|
||||
@@ -1,14 +1,18 @@
|
||||
import { prisma } from "./app/db.server";
|
||||
import { boundedIn } from "@trigger.dev/database";
|
||||
import { createOrganization } from "./app/models/organization.server";
|
||||
import { createProject } from "./app/models/project.server";
|
||||
import { ClickHouse } from "@internal/clickhouse";
|
||||
import type { QueueMetricsRawV1Input } from "@internal/clickhouse";
|
||||
import { generateFriendlyId } from "./app/v3/friendlyIdentifiers";
|
||||
// App modules compile to CommonJS under tsx, so import them as default bindings.
|
||||
import dbServer from "./app/db.server";
|
||||
import organizationServer from "./app/models/organization.server";
|
||||
import projectServer from "./app/models/project.server";
|
||||
import friendlyIdentifiers from "./app/v3/friendlyIdentifiers";
|
||||
|
||||
// Queue metrics simulator: writes realistic raw rows into a synthetic tenant's
|
||||
// queue_metrics_raw_v1 and lets the MV build queue_metrics_v1 (the same path the real
|
||||
// consumer uses), so the dashboard can be built without the run engine. See TRI-10407.
|
||||
const { prisma } = dbServer;
|
||||
const { createOrganization } = organizationServer;
|
||||
const { createProject } = projectServer;
|
||||
const { generateFriendlyId } = friendlyIdentifiers;
|
||||
|
||||
// Writes raw rows into queue_metrics_raw_v1 and lets the MVs build the rollups. See TRI-10407.
|
||||
|
||||
const ORG_TITLE = "Queue Metrics Dev";
|
||||
const PROJECT_NAME = "queue-metrics-demo";
|
||||
@@ -17,10 +21,9 @@ type Rng = () => number;
|
||||
type QueueProfile = {
|
||||
name: string;
|
||||
limit: (bucket: number) => number;
|
||||
arrivals: (bucket: number, rng: Rng) => number; // expected new runs enqueued this bucket
|
||||
arrivals: (bucket: number, rng: Rng) => number;
|
||||
waitBaseMs: number;
|
||||
sparse?: boolean; // emit no rows when the queue is fully idle (tests carry-forward gaps)
|
||||
// Concurrency-key queue: adds CK-health gauge fields + live ckIndex staging (--usage)
|
||||
sparse?: boolean;
|
||||
ck?: {
|
||||
backlogged: (bucket: number, rng: Rng) => number;
|
||||
maxWaitMs: (bucket: number, rng: Rng) => number;
|
||||
@@ -32,10 +35,6 @@ type Scenario = {
|
||||
queues: QueueProfile[];
|
||||
};
|
||||
|
||||
// ---------------------------------------------------------------------------
|
||||
// CLI args
|
||||
// ---------------------------------------------------------------------------
|
||||
|
||||
function parseArgs(argv: string[]) {
|
||||
const flags: Record<string, string> = {};
|
||||
for (let i = 0; i < argv.length; i++) {
|
||||
@@ -60,10 +59,6 @@ function parseDuration(s: string): number {
|
||||
return n * { s: 1, m: 60, h: 3600, d: 86400 }[unit]!;
|
||||
}
|
||||
|
||||
// ---------------------------------------------------------------------------
|
||||
// Deterministic RNG + distributions
|
||||
// ---------------------------------------------------------------------------
|
||||
|
||||
function mulberry32(seed: number): Rng {
|
||||
let a = seed >>> 0;
|
||||
return () => {
|
||||
@@ -104,17 +99,12 @@ function formatChDateTime(date: Date): string {
|
||||
return date.toISOString().slice(0, 19).replace("T", " ");
|
||||
}
|
||||
|
||||
// ---------------------------------------------------------------------------
|
||||
// Scenarios
|
||||
// ---------------------------------------------------------------------------
|
||||
|
||||
const steady = (): QueueProfile[] => [
|
||||
{ name: "emails", limit: () => 20, arrivals: (_b, r) => poisson(12, r), waitBaseMs: 40 },
|
||||
{ name: "webhooks", limit: () => 15, arrivals: (_b, r) => poisson(9, r), waitBaseMs: 40 },
|
||||
{ name: "reports", limit: () => 10, arrivals: (_b, r) => poisson(5, r), waitBaseMs: 60 },
|
||||
];
|
||||
|
||||
// periodic bursts every ~30 buckets
|
||||
const bursty = (name: string, limit: number, base: number): QueueProfile => ({
|
||||
name,
|
||||
limit: () => limit,
|
||||
@@ -135,7 +125,6 @@ const scenarios: Record<string, (totalBuckets: number, bucketSec: number) => Sce
|
||||
queues: [bursty("ingest", 20, 6), bursty("transform", 20, 7)],
|
||||
}),
|
||||
|
||||
// Tela case: sum of per-queue limits far exceeds the env limit, so queues compete.
|
||||
"over-allocated-env": () => ({
|
||||
description: "Sum(queue limits)=120 >> env limit=40; env saturates, queues env-limited",
|
||||
envLimit: () => 40,
|
||||
@@ -191,8 +180,6 @@ const scenarios: Record<string, (totalBuckets: number, bucketSec: number) => Sce
|
||||
],
|
||||
}),
|
||||
|
||||
// Pagination + relevance-ranking design surface: one runaway queue, a busy-but-healthy
|
||||
// head, a bursty middle, and a long sparse tail across 61 queues (the list pages at 25).
|
||||
"many-queues": () => ({
|
||||
description:
|
||||
"61 queues: one runaway, busy head, bursty middle, long sparse tail (pagination + ranking)",
|
||||
@@ -224,9 +211,6 @@ const scenarios: Record<string, (totalBuckets: number, bucketSec: number) => Sce
|
||||
],
|
||||
}),
|
||||
|
||||
// Per-tenant concurrency keys: a hog tenant periodically floods the queue and starves
|
||||
// the others, so the CK charts (keys with backlog, most-starved wait) and the live
|
||||
// per-key table on the queue detail page have something to show. Use with --usage.
|
||||
"tenant-hotspot": () => ({
|
||||
description:
|
||||
"CK queue where a hog tenant starves others: CK charts + live key table (use --usage)",
|
||||
@@ -249,10 +233,9 @@ const scenarios: Record<string, (totalBuckets: number, bucketSec: number) => Sce
|
||||
],
|
||||
}),
|
||||
|
||||
// Default: one env with a variety of queue behaviours + occasional env saturation.
|
||||
mixed: (totalBuckets) => ({
|
||||
description: "variety of queue profiles in one env, with occasional env saturation",
|
||||
envLimit: (b) => (b % 40 < 12 ? 45 : 70), // dips low periodically to flip env saturation
|
||||
envLimit: (b) => (b % 40 < 12 ? 45 : 70),
|
||||
queues: [
|
||||
{ name: "emails", limit: () => 20, arrivals: (_b, r) => poisson(12, r), waitBaseMs: 40 },
|
||||
bursty("webhooks", 20, 6),
|
||||
@@ -274,18 +257,13 @@ const scenarios: Record<string, (totalBuckets: number, bucketSec: number) => Sce
|
||||
}),
|
||||
};
|
||||
|
||||
// ---------------------------------------------------------------------------
|
||||
// Simulation
|
||||
// ---------------------------------------------------------------------------
|
||||
|
||||
type Ids = { organization_id: string; project_id: string; environment_id: string };
|
||||
const WAIT_SIGMA = 0.6;
|
||||
const NACK_RATE = 0.02;
|
||||
const DLQ_RATE = 0.004;
|
||||
|
||||
type CounterOp = "enqueue" | "started" | "ack" | "nack" | "dlq";
|
||||
// Per-(queue, op) odometers, mirroring the production emitter: cumulative readings with a
|
||||
// cum=0 baseline on the first one, so deltaSumTimestamp captures the 0->1 delta.
|
||||
// Cumulative odometers: the first reading must be cum=0 so deltaSumTimestamp sees the 0->1 delta.
|
||||
type CounterState = Record<CounterOp, number>[];
|
||||
|
||||
function counterRows(
|
||||
@@ -326,8 +304,7 @@ function newCounterState(n: number): CounterState {
|
||||
return Array.from({ length: n }, () => ({ enqueue: 0, started: 0, ack: 0, nack: 0, dlq: 0 }));
|
||||
}
|
||||
|
||||
// Per-key simulation for CK profiles: 12 tenants (tenant-01 is the hog, matching
|
||||
// stageRedisUsage), per-tenant backlog drained round-robin, per-tenant odometers.
|
||||
// tenant-01 is the hog here and in stageRedisUsage; keep the two in step.
|
||||
const CK_TENANT_COUNT = 12;
|
||||
type CkSimState = { backlog: number[]; counters: Map<number, Record<CounterOp, number>> };
|
||||
const ckSim = new Map<number, CkSimState>();
|
||||
@@ -370,9 +347,7 @@ function ckCounterRows(
|
||||
return rows;
|
||||
}
|
||||
|
||||
// Advance one bucket of the simulation for every queue, returning the raw rows to insert.
|
||||
// `backlog` and `counters` are mutated in place so state carries across buckets (and into
|
||||
// live mode).
|
||||
// `backlog` and `counters` are mutated in place: state carries across buckets and into live mode.
|
||||
function simulateBucket(
|
||||
scenario: Scenario,
|
||||
bucket: number,
|
||||
@@ -392,12 +367,11 @@ function simulateBucket(
|
||||
for (let q = 0; q < n; q++) {
|
||||
limit[q] = scenario.queues[q].limit(bucket);
|
||||
const arrivals = Math.min(500, scenario.queues[q].arrivals(bucket, rng));
|
||||
const prior = backlog[q]; // backlog carried from earlier buckets, before this bucket's arrivals
|
||||
backlog[q] += arrivals; // arrivals join the backlog; recorded as enqueues below
|
||||
const prior = backlog[q];
|
||||
backlog[q] += arrivals;
|
||||
(desired as any)[q] = { arrivals, prior, want: Math.min(limit[q], backlog[q]) };
|
||||
}
|
||||
|
||||
// Env cap: if the queues collectively want more concurrency than the env allows, scale down.
|
||||
const sumWant = desired.reduce((s: number, d: any) => s + d.want, 0);
|
||||
const scale = sumWant > envLimit && sumWant > 0 ? envLimit / sumWant : 1;
|
||||
|
||||
@@ -413,8 +387,7 @@ function simulateBucket(
|
||||
envQueued += queued[q];
|
||||
}
|
||||
|
||||
// Order keys are time-based (like the production stream ids) so appended runs and live
|
||||
// mode stay monotonic; the per-bucket sequence keeps them unique within a bucket.
|
||||
// Order keys must be monotonic across processes, so they are time-based plus a per-bucket seq.
|
||||
let bucketSeq = 0;
|
||||
const orderKey = () => bucketEpochSec * 1_000_000 + bucketSeq++;
|
||||
|
||||
@@ -423,14 +396,13 @@ function simulateBucket(
|
||||
const profile = scenario.queues[q];
|
||||
const started = running[q];
|
||||
const arrivals = (desired[q] as any).arrivals as number;
|
||||
const prior = (desired[q] as any).prior as number; // depth a starting run actually queued behind
|
||||
backlog[q] = queued[q]; // carry the unserved remainder forward
|
||||
const prior = (desired[q] as any).prior as number;
|
||||
backlog[q] = queued[q];
|
||||
|
||||
if (profile.sparse && arrivals === 0 && started === 0 && prior === 0) {
|
||||
continue; // fully idle: leave a gap so carry-forward is exercised
|
||||
continue;
|
||||
}
|
||||
|
||||
// CK-health fields stay coherent with the depth: no queued runs means no backlogged keys.
|
||||
const ckBacklogged = profile.ck
|
||||
? queued[q] > 0
|
||||
? Math.max(1, Math.min(profile.ck.backlogged(bucket, rng), queued[q]))
|
||||
@@ -461,8 +433,6 @@ function simulateBucket(
|
||||
rows.push(...counterRows(counters, q, ids, profile.name, eventTime, orderKey, "enqueue"));
|
||||
}
|
||||
|
||||
// Per-key rows for CK profiles: assign arrivals hog-weighted, drain round-robin
|
||||
// (fair share), then emit per-tenant odometers + a per-key gauge per active tenant.
|
||||
if (profile.ck) {
|
||||
let ckq = ckSim.get(q);
|
||||
if (!ckq) {
|
||||
@@ -544,10 +514,6 @@ function simulateBucket(
|
||||
return rows;
|
||||
}
|
||||
|
||||
// ---------------------------------------------------------------------------
|
||||
// ClickHouse
|
||||
// ---------------------------------------------------------------------------
|
||||
|
||||
function clickhouse(): ClickHouse {
|
||||
const clickhouseUrl = process.env.CLICKHOUSE_URL ?? process.env.EVENTS_CLICKHOUSE_URL;
|
||||
if (!clickhouseUrl) {
|
||||
@@ -555,7 +521,7 @@ function clickhouse(): ClickHouse {
|
||||
process.exit(1);
|
||||
}
|
||||
const url = new URL(clickhouseUrl);
|
||||
// Allowlist local hosts only (this script TRUNCATEs), and never echo the URL (it carries creds).
|
||||
// Local hosts only (this script deletes rows); never echo the URL, it carries credentials.
|
||||
const localHosts = new Set(["localhost", "127.0.0.1", "::1", "0.0.0.0"]);
|
||||
if (!localHosts.has(url.hostname)) {
|
||||
console.error(`Refusing to run against a non-local ClickHouse host: ${url.hostname}`);
|
||||
@@ -597,7 +563,6 @@ async function resetEnv(ch: ClickHouse, environmentId: string) {
|
||||
console.log(`Reset queue metrics for environment ${environmentId}`);
|
||||
}
|
||||
|
||||
// Fake running counts in the run-queue Redis (Running column + allocation usage bars).
|
||||
// Reconciled every run: staged with --usage, cleared otherwise.
|
||||
async function stageRedisUsage(scenario: Scenario, ids: Ids, seed: number, clear: boolean) {
|
||||
const host = process.env.RUN_ENGINE_RUN_QUEUE_REDIS_HOST ?? process.env.REDIS_HOST ?? "localhost";
|
||||
@@ -617,16 +582,11 @@ async function stageRedisUsage(scenario: Scenario, ids: Ids, seed: number, clear
|
||||
const logicalBase = `{org:${ids.organization_id}}:proj:${ids.project_id}:env:${ids.environment_id}:queue:`;
|
||||
const base = `${prefix}${logicalBase}`;
|
||||
|
||||
// Env-level structures the Queues list "Queued"/"Running" blocks read:
|
||||
// lengthOfEnvQueue -> ZCARD(envQueueKey) (ZSET, no proj section)
|
||||
// concurrencyOfEnvQueue -> SCARD(envCurrentDequeuedKey) (SET)
|
||||
// We accumulate the per-queue staged counts below and stage these so the blocks
|
||||
// equal the table's per-queue sums instead of showing 0/0.
|
||||
// envQueue is a ZSET with no proj section; envCurrentDequeued is a SET with one.
|
||||
const envQueueKey = `${prefix}{org:${ids.organization_id}}:env:${ids.environment_id}`;
|
||||
const envCurrentDequeuedKey = `${prefix}{org:${ids.organization_id}}:proj:${ids.project_id}:env:${ids.environment_id}:currentDequeued`;
|
||||
await redis.del(envQueueKey, envCurrentDequeuedKey);
|
||||
// Table Queued = ZCARD(base) + lengthCounter (base zset unstaged -> only CK queues
|
||||
// contribute their lengthCounter). Table Running = SCARD(currentDequeued) per queue.
|
||||
// Per-queue Queued = ZCARD(base) + lengthCounter; Running = SCARD(currentDequeued).
|
||||
let envQueuedTotal = 0;
|
||||
let envRunningTotal = 0;
|
||||
|
||||
@@ -634,8 +594,7 @@ async function stageRedisUsage(scenario: Scenario, ids: Ids, seed: number, clear
|
||||
const key = `${base}${profile.name}:currentDequeued`;
|
||||
await redis.del(key);
|
||||
|
||||
// CK staging (ckIndex + per-key subqueues) feeds the live per-key table on the queue
|
||||
// detail page. Members are stored unprefixed, exactly like the run-queue Lua does.
|
||||
// ckIndex members are stored unprefixed, exactly like the run-queue Lua does.
|
||||
const ckIndexKey = `${base}${profile.name}:ckIndex`;
|
||||
const lengthCounterKey = `${base}${profile.name}:lengthCounter`;
|
||||
const staleMembers = await redis.zrange(ckIndexKey, 0, -1);
|
||||
@@ -646,7 +605,6 @@ async function stageRedisUsage(scenario: Scenario, ids: Ids, seed: number, clear
|
||||
|
||||
if (clear) continue;
|
||||
const limit = profile.limit(0);
|
||||
// First queue rides at/over its limit, the rest at 30-90%, sparse mostly idle.
|
||||
const count = profile.sparse
|
||||
? rng() < 0.3
|
||||
? 1
|
||||
@@ -684,15 +642,12 @@ async function stageRedisUsage(scenario: Scenario, ids: Ids, seed: number, clear
|
||||
await redis.zadd(ckIndexKey, now - oldestAgeMs, member);
|
||||
totalCkQueued += queuedCount;
|
||||
}
|
||||
// The aggregate "Queued now" reads ZCARD(base) + this counter; keep them coherent.
|
||||
await redis.set(lengthCounterKey, totalCkQueued, "EX", 24 * 3600);
|
||||
envQueuedTotal += totalCkQueued;
|
||||
}
|
||||
}
|
||||
|
||||
// Stage the env-level structures so the list-page blocks match the table sums.
|
||||
// Members are unique across queues (each per-queue set uses its own key, but the
|
||||
// env set/zset needs distinct members to reach the summed cardinality).
|
||||
// The env set/zset needs members distinct across queues to reach the summed cardinality.
|
||||
if (!clear) {
|
||||
if (envRunningTotal > 0) {
|
||||
await redis.sadd(
|
||||
@@ -719,13 +674,8 @@ async function stageRedisUsage(scenario: Scenario, ids: Ids, seed: number, clear
|
||||
}
|
||||
}
|
||||
|
||||
// ---------------------------------------------------------------------------
|
||||
// Main
|
||||
// ---------------------------------------------------------------------------
|
||||
|
||||
// Make the synthetic project a V2 engine project with a current dev worker + a Postgres
|
||||
// TaskQueue per simulated queue, so the /queues list renders the V2 table (it pages from
|
||||
// Postgres and gates on engine version; ClickHouse only holds the metrics).
|
||||
// The /queues list pages from Postgres and gates on engine version, so the project needs
|
||||
// engine V2, a worker, and a TaskQueue row per simulated queue.
|
||||
async function ensureTaskQueues(
|
||||
scenario: Scenario,
|
||||
projectId: string,
|
||||
@@ -769,9 +719,7 @@ async function ensureTaskQueues(
|
||||
projectId,
|
||||
type: "NAMED",
|
||||
},
|
||||
// Reset any dashboard override left from manual testing: re-seeding overwrites the
|
||||
// materialized concurrencyLimit, so a surviving override percent/base would contradict it
|
||||
// (e.g. "10 (77%)" with an env limit of 25).
|
||||
// Re-seeding overwrites concurrencyLimit, so a surviving override would contradict it.
|
||||
update: {
|
||||
concurrencyLimit,
|
||||
concurrencyLimitBase: null,
|
||||
@@ -782,8 +730,6 @@ async function ensureTaskQueues(
|
||||
});
|
||||
}
|
||||
|
||||
// Drop queues left over from a previously seeded scenario so switching scenarios
|
||||
// does not leave metric-less rows in the list.
|
||||
const { count: pruned } = await prisma.taskQueue.deleteMany({
|
||||
where: {
|
||||
runtimeEnvironmentId,
|
||||
@@ -917,7 +863,6 @@ async function main() {
|
||||
`Backfilling ${totalBuckets} x ${bucketSec}s buckets (${flags.window ?? "2h"}) for ${scenario.queues.length} queues...`
|
||||
);
|
||||
|
||||
// Backfill: buckets from (now - window) up to now, aligned to the bucket grid.
|
||||
const nowBucket = Math.floor(Date.now() / 1000 / bucketSec) * bucketSec;
|
||||
const startBucket = nowBucket - totalBuckets * bucketSec;
|
||||
const counters = newCounterState(scenario.queues.length);
|
||||
@@ -942,8 +887,7 @@ async function main() {
|
||||
await insertBatched(ch, rows, nonce);
|
||||
console.log(`Inserted ${rows.length} raw rows.`);
|
||||
|
||||
// Merge the AggregatingMergeTree partials so argMax "current value" widgets read cleanly.
|
||||
// The real pipeline relies on background merges; the simulator forces it for a tidy demo.
|
||||
// The rollups are AggregatingMergeTrees; a read straight after the insert can't wait for merges.
|
||||
const raw = (
|
||||
ch.writer as unknown as { client: { command: (a: { query: string }) => Promise<unknown> } }
|
||||
).client;
|
||||
|
||||
Some files were not shown because too many files have changed in this diff Show More
Reference in New Issue
Block a user