feat(webapp): dashboard parity for mollifier-buffered runs (#3757)

## Summary

Dashboard surfaces handle buffered runs by falling back to the mollifier
snapshot:

- Run detail, span detail, streams view (`_app.../runs.\$runParam`,
`resources.../spans.\$spanParam`, `resources.../streams.\$streamKey`).
- Redirect routes (`@.runs.\$runParam`, `runs.\$runParam`,
`projects.v3.\$projectRef.runs.\$runParam`).
- Action routes — cancel / replay / idempotency-reset / debug — under
`resources.taskruns/...` and `resources.../idempotencyKey.reset`.
- Logs download.
- Realtime subscription route + per-run resource
(`realtime.v1.runs.\$runId`, `resources.../realtime.v1.*`).
- `CancelRunDialog` gains an `onCancelSubmitted` callback so submit
isn't raced by the Radix `DialogClose` wrapper.

Stacked on the mutations PR.

## Test plan

- [x] \`pnpm run typecheck --filter webapp\` passes
- [x] \`pnpm run test --filter webapp
test/mollifierRealtimeRunResource.test.ts\` passes
- [x] \`pnpm run test --filter webapp
test/mollifierRealtimeRunResourceBuffer.test.ts\` passes
- [x] \`pnpm run test --filter webapp
test/mollifierRealtimeSubscription.test.ts\` passes
- [x] Manual smoke: trigger a buffered run, open it in the dashboard,
replay/cancel from the UI

---

## Ship-gate follow-up fixes

- **Auto-redirect to root span on direct nav** — loader sets `?span=`
from root span (PG) or buffered snapshot spanId before 302'ing, so
bookmark/share-link/direct-nav doesn't leave the panel collapsed.
- **RunPresenter switches from `findFirstOrThrow` to `findFirst` + typed
`RunNotInPgError`** — kills the per-poll `PrismaClient error` log spam
for buffered runs without changing the route-loader's fallback flow.
- **Span detail panel renders for buffered runs** — `SpanPresenter.call`
now falls back to `findRunByIdWithMollifierFallback` +
`buildSyntheticSpanRun` instead of returning undefined and triggering
the "Event not found" toast loop.
- **Logs download for buffered runs returns a gzipped placeholder line**
— replaces the 404 with a content-encoded line explaining the run is
queued. Same org-membership gate as the PG path.
- **Admin Debug-Run button hidden for buffered runs + SpanRun circular
type alias broken** (squashed) — buttons gate on a new `isBuffered` flag
on the synthetic SpanRun. Required grounding SpanRun in
`SpanPresenter.getRun` to break a circular type alias TS no longer
tolerates once `isBuffered` is a literal field on the shape.
- **Replay action requires user auth + org-membership** (🚩 Devin
finding) — `action` was unauthenticated and the PG `findFirst` had no
org filter, so any caller with a valid `runParam` could replay any run.
Buffered fallback inherited the same gap. Fixed to mirror the cancel
route.

---------

Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
This commit is contained in:
Daniel Sutton
2026-06-01 16:50:31 +01:00
committed by GitHub
parent e1950778e2
commit e21b68cc5f
19 changed files with 1219 additions and 39 deletions
+6
View File
@@ -0,0 +1,6 @@
---
area: webapp
type: feature
---
Mollifier dashboard surface: run-detail page renders buffered runs via synthetic trace, header, and span shapes; admin-only "Buffered" indicator and drainer LOG event in the trace tree.
@@ -10,9 +10,18 @@ import { SpinnerWhite } from "~/components/primitives/Spinner";
type CancelRunDialogProps = {
runFriendlyId: string;
redirectPath: string;
// Fired on submit so the parent can close the Radix Dialog without
// wrapping the submit button in `DialogClose` — that wrapper races
// submit (close fires first, unmounts the form, and the cancel POST
// never lands). Optional so existing call sites still type-check.
onCancelSubmitted?: () => void;
};
export function CancelRunDialog({ runFriendlyId, redirectPath }: CancelRunDialogProps) {
export function CancelRunDialog({
runFriendlyId,
redirectPath,
onCancelSubmitted,
}: CancelRunDialogProps) {
const navigation = useNavigation();
const formAction = `/resources/taskruns/${runFriendlyId}/cancel`;
@@ -27,7 +36,11 @@ export function CancelRunDialog({ runFriendlyId, redirectPath }: CancelRunDialog
</Paragraph>
<FormButtons
confirmButton={
<Form action={`/resources/taskruns/${runFriendlyId}/cancel`} method="post">
<Form
action={`/resources/taskruns/${runFriendlyId}/cancel`}
method="post"
onSubmit={() => onCancelSubmitted?.()}
>
<Button
type="submit"
name="redirectUrl"
@@ -20,6 +20,20 @@ export class RunEnvironmentMismatchError extends Error {
}
}
// Thrown by `call()` when the run isn't in PG. The route loader catches
// this and falls back to the mollifier buffer via `tryMollifiedRunFallback`.
// Using a typed error (rather than Prisma's `findFirstOrThrow` exception)
// keeps the buffered case off the PrismaClient error path — that path
// emits a `PrismaClient error` log every time it fires, which on the
// run-detail page polls becomes per-tick log spam and Sentry noise for
// any run that legitimately lives in the buffer.
export class RunNotInPgError extends Error {
constructor(public readonly runFriendlyId: string) {
super(`Run ${runFriendlyId} not in PG`);
this.name = "RunNotInPgError";
}
}
export class RunPresenter {
#prismaClient: PrismaClient;
@@ -42,7 +56,13 @@ export class RunPresenter {
showDeletedLogs: boolean;
showDebug: boolean;
}) {
const run = await this.#prismaClient.taskRun.findFirstOrThrow({
// `findFirst` + explicit null check (not `findFirstOrThrow`) because
// a missing PG row is the *expected* path for buffered runs — the
// route catches `RunNotInPgError` and falls back to the synthesised
// buffer view. `findFirstOrThrow` would log a `PrismaClient error`
// every tick of the page poll, masking real DB issues with synthetic
// not-found noise.
const run = await this.#prismaClient.taskRun.findFirst({
select: {
id: true,
createdAt: true,
@@ -106,6 +126,10 @@ export class RunPresenter {
},
});
if (!run) {
throw new RunNotInPgError(runFriendlyId);
}
if (environmentSlug !== run.runtimeEnvironment.slug) {
throw new RunEnvironmentMismatchError(
`Run ${runFriendlyId} is not in environment ${environmentSlug}`
@@ -3,6 +3,8 @@ import { logger } from "~/services/logger.server";
import { singleton } from "~/utils/singleton";
import { ABORT_REASON_SEND_ERROR, createSSELoader, SendFunction } from "~/utils/sse";
import { throttle } from "~/utils/throttle";
import { getMollifierBuffer } from "~/v3/mollifier/mollifierBuffer.server";
import { deserialiseMollifierSnapshot } from "~/v3/mollifier/mollifierSnapshot.server";
import { tracePubSub } from "~/v3/services/tracePubSub.server";
const PING_INTERVAL = 5_000;
@@ -37,17 +39,48 @@ export class RunStreamPresenter {
},
});
if (!run) {
// Fall back to the mollifier buffer when the run isn't in PG yet.
// The buffered run has no execution events to stream, but we still
// attach a trace-pubsub subscription using the snapshot's traceId
// so that the moment the drainer materialises the row and execution
// begins, those events flow to this open SSE connection. Closing
// with 404 would force the dashboard to keep retrying.
let traceId: string | null = run?.traceId ?? null;
if (!traceId) {
const buffer = getMollifierBuffer();
if (buffer) {
try {
const entry = await buffer.getEntry(runFriendlyId);
if (entry) {
// Go through the webapp wrapper so this read-side module
// shares a single deserialisation path with readFallback —
// see the contract comment in syntheticRedirectInfo.server.ts.
const snapshot = deserialiseMollifierSnapshot(entry.payload);
if (typeof snapshot.traceId === "string") {
traceId = snapshot.traceId;
}
}
} catch (err) {
logger.warn("RunStreamPresenter buffer fallback failed", {
runFriendlyId,
err: err instanceof Error ? err.message : String(err),
});
}
}
}
if (!traceId) {
throw new Response("Not found", { status: 404 });
}
const resolvedRun = { traceId };
logger.info("RunStreamPresenter.start", {
runFriendlyId,
traceId: run.traceId,
traceId: resolvedRun.traceId,
});
// Subscribe to trace updates
const { unsubscribe, eventEmitter } = await tracePubSub.subscribeToTrace(run.traceId);
const { unsubscribe, eventEmitter } = await tracePubSub.subscribeToTrace(resolvedRun.traceId);
// Only send max every 1 second
const throttledSend = throttle(
@@ -105,7 +138,7 @@ export class RunStreamPresenter {
cleanup: () => {
logger.info("RunStreamPresenter.cleanup", {
runFriendlyId,
traceId: run.traceId,
traceId: resolvedRun.traceId,
});
// Remove message listener
@@ -119,13 +152,13 @@ export class RunStreamPresenter {
.then(() => {
logger.info("RunStreamPresenter.cleanup.unsubscribe succeeded", {
runFriendlyId,
traceId: run.traceId,
traceId: resolvedRun.traceId,
});
})
.catch((error) => {
logger.error("RunStreamPresenter.cleanup.unsubscribe failed", {
runFriendlyId,
traceId: run.traceId,
traceId: resolvedRun.traceId,
error: {
name: error.name,
message: error.message,
@@ -32,6 +32,8 @@ import {
extractAIEmbedData,
} from "~/components/runs/v3/ai";
import { getEventRepositoryForStore } from "~/v3/eventRepository/index.server";
import { findRunByIdWithMollifierFallback } from "~/v3/mollifier/readFallback.server";
import { buildSyntheticSpanRun } from "~/v3/mollifier/syntheticSpanRun.server";
export type PromptSpanData = {
slug: string;
@@ -72,9 +74,21 @@ function extractPromptSpanData(properties: Record<string, unknown>): PromptSpanD
};
}
// SpanRun is grounded in the PG-path `getRun` method rather than
// inferred from `call`'s return type. The buffered branch of `call`
// routes through `buildSyntheticSpanRun`, and that helper is annotated
// `Promise<SpanRun>` — if SpanRun were derived from `call` it would
// close a loop TS no longer tolerates ("Type alias 'Result' circularly
// references itself"). `getRun` is the canonical source for the shape
// (the synthetic helper just rebuilds the same shape from a buffer
// snapshot), and it doesn't recurse, so grounding here breaks the
// cycle while keeping Span available off `call` (Span's path through
// `#getSpan` has no synthetic indirection).
export type SpanRun = NonNullable<
Awaited<ReturnType<InstanceType<typeof SpanPresenter>["getRun"]>>
>;
type Result = Awaited<ReturnType<SpanPresenter["call"]>>;
export type Span = NonNullable<NonNullable<Result>["span"]>;
export type SpanRun = NonNullable<NonNullable<Result>["run"]>;
type FindRunResult = NonNullable<
Awaited<ReturnType<InstanceType<typeof SpanPresenter>["findRun"]>>
>;
@@ -84,12 +98,18 @@ export class SpanPresenter extends BasePresenter {
public async call({
userId,
projectSlug,
envSlug,
spanId,
runFriendlyId,
linkedRunId,
}: {
userId: string;
projectSlug: string;
// Optional for backwards compatibility, required for the mollifier
// buffer fallback when the parent run isn't yet in PG — we need to
// resolve the env id to satisfy `findRunByIdWithMollifierFallback`'s
// auth check.
envSlug?: string;
spanId: string;
runFriendlyId: string;
linkedRunId?: string;
@@ -127,7 +147,32 @@ export class SpanPresenter extends BasePresenter {
});
if (!parentRun) {
return;
// PG miss → fall back to the mollifier buffer. Without this the
// right-side span detail panel on the run-detail page never
// resolves for buffered runs: `call()` returns undefined, the
// resource route redirects with an "Event not found" toast, the
// run-detail page reloads, the toast fires again — a perpetual
// spin until the drainer materialises the row. Synthesise a
// SpanRun straight from the buffer snapshot, reusing
// `buildSyntheticSpanRun` (the same helper the run-detail
// loader's header fallback already uses).
if (!envSlug) return;
const envRow = await this._replica.runtimeEnvironment.findFirst({
where: { project: { id: project.id }, slug: envSlug },
select: { id: true, slug: true, type: true, organizationId: true },
});
if (!envRow) return;
const buffered = await findRunByIdWithMollifierFallback({
runId: runFriendlyId,
environmentId: envRow.id,
organizationId: envRow.organizationId,
});
if (!buffered) return;
const synth = await buildSyntheticSpanRun({
run: buffered,
environment: { id: envRow.id, slug: envRow.slug, type: envRow.type },
});
return { type: "run" as const, run: synth };
}
const { traceId } = parentRun;
@@ -373,6 +418,7 @@ export class SpanPresenter extends BasePresenter {
traceId: run.traceId,
spanId: run.spanId,
isCached: !!linkedRunId,
isBuffered: false,
machinePreset: machine?.name,
taskEventStore: run.taskEventStore,
externalTraceId,
+34 -3
View File
@@ -3,7 +3,8 @@ import { z } from "zod";
import { prisma } from "~/db.server";
import { redirectWithErrorMessage } from "~/models/message.server";
import { requireUser } from "~/services/session.server";
import { impersonate, rootPath, v3RunPath } from "~/utils/pathBuilder";
import { impersonate, rootPath, v3RunPath, v3RunSpanPath } from "~/utils/pathBuilder";
import { findBufferedRunRedirectInfo } from "~/v3/mollifier/syntheticRedirectInfo.server";
const ParamsSchema = z.object({
runParam: z.string(),
@@ -32,6 +33,7 @@ export async function loader({ params, request }: LoaderFunctionArgs) {
friendlyId: runParam,
},
select: {
spanId: true,
runtimeEnvironment: {
select: {
slug: true,
@@ -51,16 +53,45 @@ export async function loader({ params, request }: LoaderFunctionArgs) {
});
if (!run) {
// Admin impersonation route — bypass org membership so admins can
// open any buffered run by friendlyId, mirroring the existing PG
// behaviour above (no membership filter on the find).
const buffered = await findBufferedRunRedirectInfo({
runFriendlyId: runParam,
userId: user.id,
skipOrgMembershipCheck: true,
});
if (buffered) {
// Preselect the root span so the run-detail trace tree opens with
// the buffered run's span highlighted, matching the sibling
// redirect routes (runs.$runParam.ts, projects.v3.$projectRef…).
const path = buffered.spanId
? v3RunSpanPath(
{ slug: buffered.organizationSlug },
{ slug: buffered.projectSlug },
{ slug: buffered.environmentSlug },
{ friendlyId: runParam },
{ spanId: buffered.spanId }
)
: v3RunPath(
{ slug: buffered.organizationSlug },
{ slug: buffered.projectSlug },
{ slug: buffered.environmentSlug },
{ friendlyId: runParam }
);
return redirect(impersonate(path));
}
return redirectWithErrorMessage(rootPath(), request, "Run doesn't exist", {
ephemeral: false,
});
}
const path = v3RunPath(
const path = v3RunSpanPath(
{ slug: run.project.organization.slug },
{ slug: run.project.slug },
{ slug: run.runtimeEnvironment.slug },
{ friendlyId: runParam }
{ friendlyId: runParam },
{ spanId: run.spanId }
);
return redirect(impersonate(path));
@@ -88,10 +88,18 @@ import { useReplaceSearchParams } from "~/hooks/useReplaceSearchParams";
import { useSearchParams } from "~/hooks/useSearchParam";
import { type Shortcut, useShortcutKeys } from "~/hooks/useShortcutKeys";
import { useHasAdminAccess } from "~/hooks/useUser";
import { env } from "~/env.server";
import { findProjectBySlug } from "~/models/project.server";
import { findEnvironmentBySlug } from "~/models/runtimeEnvironment.server";
import { NextRunListPresenter } from "~/presenters/v3/NextRunListPresenter.server";
import { RunEnvironmentMismatchError, RunPresenter } from "~/presenters/v3/RunPresenter.server";
import {
RunEnvironmentMismatchError,
RunNotInPgError,
RunPresenter,
} from "~/presenters/v3/RunPresenter.server";
import { findRunByIdWithMollifierFallback } from "~/v3/mollifier/readFallback.server";
import { buildSyntheticRunHeader } from "~/v3/mollifier/syntheticRunHeader.server";
import { buildSyntheticTraceForBufferedRun } from "~/v3/mollifier/syntheticTrace.server";
import { clickhouseFactory } from "~/services/clickhouse/clickhouseFactoryInstance.server";
import { getImpersonationId } from "~/services/impersonation.server";
import { logger } from "~/services/logger.server";
@@ -277,9 +285,78 @@ export const loader = async ({ request, params }: LoaderFunctionArgs) => {
);
}
// Only fall back to the mollifier buffer on a genuine PG miss. Any
// other error (DB timeout during trace queries, event-repository
// failure, etc.) means the run WAS in PG but a downstream lookup
// failed — falling back to the buffer here would either return a
// stale synth entry if one happens to exist in the brief drainer-
// materialisation race window, or quietly mask the real failure.
// `RunNotInPgError` is the typed signal RunPresenter throws for the
// route loader's specific case (`RunPresenter.server.ts:130`).
if (!(error instanceof RunNotInPgError)) {
throw error;
}
// PG miss → try the mollifier buffer. When the gate diverts a trigger
// the run sits in Redis until the drainer materialises it; without
// this fallback the run-detail page 404s for the brief buffered window
// even though the API has accepted the trigger and returned an id.
const buffered = await tryMollifiedRunFallback({
runFriendlyId: runParam,
organizationSlug,
projectSlug: projectParam,
envSlug: envParam,
userId,
});
if (buffered) {
// Preselect the root span on the initial page load when the URL
// doesn't already carry `?span=`. The sibling redirect routes
// (runs.$runParam.ts, @.runs.$runParam.ts,
// projects.v3.$projectRef.runs.$runParam.ts) all do this, but
// direct navigation to the canonical project-scoped URL never
// hit those redirects — leaving the right detail panel collapsed.
// Skip on `_data` requests (Remix data fetches): they're
// client-driven follow-ups and the client URL is what matters,
// not the loader's view of it.
if (
!url.searchParams.has("span") &&
!url.searchParams.has("_data") &&
buffered.run.spanId
) {
url.searchParams.set("span", buffered.run.spanId);
throw redirect(url.pathname + "?" + url.searchParams.toString());
}
const parent = await getResizableSnapshot(request, resizableSettings.parent.autosaveId);
const tree = await getResizableSnapshot(request, resizableSettings.tree.autosaveId);
return json({
run: buffered.run,
trace: buffered.trace,
maximumLiveReloadingSetting: env.MAXIMUM_LIVE_RELOADING_EVENTS,
resizable: { parent, tree },
runsList: null,
});
}
throw error;
}
// Preselect the root span on the initial page load when the URL
// doesn't already carry `?span=`. See the comment on the equivalent
// block in the buffered fallback above — the sibling redirect routes
// do this, but direct navigation to the canonical project-scoped URL
// never hits them, leaving the right detail panel collapsed.
if (
!url.searchParams.has("span") &&
!url.searchParams.has("_data") &&
result.run.spanId
) {
url.searchParams.set("span", result.run.spanId);
throw redirect(url.pathname + "?" + url.searchParams.toString());
}
//resizable settings
const parent = await getResizableSnapshot(request, resizableSettings.parent.autosaveId);
const tree = await getResizableSnapshot(request, resizableSettings.tree.autosaveId);
@@ -305,6 +382,39 @@ export const loader = async ({ request, params }: LoaderFunctionArgs) => {
});
};
async function tryMollifiedRunFallback(args: {
runFriendlyId: string;
organizationSlug: string;
projectSlug: string;
envSlug: string;
userId: string;
}) {
const project = await findProjectBySlug(args.organizationSlug, args.projectSlug, args.userId);
if (!project) return null;
const environment = await findEnvironmentBySlug(project.id, args.envSlug, args.userId);
if (!environment) return null;
const buffered = await findRunByIdWithMollifierFallback({
runId: args.runFriendlyId,
environmentId: environment.id,
organizationId: project.organizationId,
});
if (!buffered) return null;
return {
run: buildSyntheticRunHeader({
run: buffered,
environment: {
id: environment.id,
organizationId: project.organizationId,
type: environment.type,
slug: environment.slug,
},
}),
trace: buildSyntheticTraceForBufferedRun(buffered),
};
}
type LoaderData = SerializeFrom<typeof loader>;
export default function Page() {
@@ -407,23 +517,17 @@ export default function Page() {
/>
</Dialog>
{run.isFinished ? null : (
<Dialog key={`cancel-${run.friendlyId}`}>
<DialogTrigger asChild>
<Button variant="danger/small" LeadingIcon={StopCircleIcon} shortcut={{ key: "C" }}>
Cancel run
</Button>
</DialogTrigger>
<CancelRunDialog
runFriendlyId={run.friendlyId}
redirectPath={v3RunSpanPath(
organization,
project,
environment,
{ friendlyId: run.friendlyId },
{ spanId: run.spanId }
)}
/>
</Dialog>
<ControlledCancelRunDialog
key={`cancel-${run.friendlyId}`}
runFriendlyId={run.friendlyId}
redirectPath={v3RunSpanPath(
organization,
project,
environment,
{ friendlyId: run.friendlyId },
{ spanId: run.spanId }
)}
/>
)}
</PageAccessories>
</NavBar>
@@ -587,6 +691,35 @@ function TraceView({
);
}
// Controlled wrapper around the cancel dialog. Owns the Radix open state
// so the dialog closes itself once the cancel action transitions through
// submission. We can't `<DialogClose asChild>`-wrap the submit button
// because Radix's onClick handler swallows the button's name=value pair
// that the form action depends on for `redirectUrl`.
function ControlledCancelRunDialog({
runFriendlyId,
redirectPath,
}: {
runFriendlyId: string;
redirectPath: string;
}) {
const [open, setOpen] = useState(false);
return (
<Dialog open={open} onOpenChange={setOpen}>
<DialogTrigger asChild>
<Button variant="danger/small" LeadingIcon={StopCircleIcon} shortcut={{ key: "C" }}>
Cancel run
</Button>
</DialogTrigger>
<CancelRunDialog
runFriendlyId={runFriendlyId}
redirectPath={redirectPath}
onCancelSubmitted={() => setOpen(false)}
/>
</Dialog>
);
}
function NoLogsView({ run, resizable }: Pick<LoaderData, "run" | "resizable">) {
const plan = useCurrentPlan();
const organization = useOrganization();
@@ -616,6 +749,11 @@ function NoLogsView({ run, resizable }: Pick<LoaderData, "run" | "resizable">) {
>
<div className="grid h-full place-items-center">
{daysSinceCompleted === undefined ? (
// NoLogsView only renders when the loader returns no trace.
// Buffered runs always carry a synthetic trace (see
// buildSyntheticTraceForBufferedRun) so they never reach
// this branch — the message here is the pre-mollifier
// copy for runs with no completedAt and no logs.
<InfoPanel variant="info" icon={InformationCircleIcon} title="We delete old logs">
<Paragraph variant="small">
We tidy up older logs to keep things running smoothly.
@@ -120,6 +120,7 @@ export const loader = async ({ request, params }: LoaderFunctionArgs) => {
try {
const result = await presenter.call({
projectSlug: projectParam,
envSlug: envParam,
spanId: spanParam,
runFriendlyId: runParam,
userId,
@@ -1021,6 +1022,10 @@ function RunBody({
<Paragraph spacing variant="small" className="text-yellow-500">
Admin only
</Paragraph>
<Property.Item>
<Property.Label>Buffered</Property.Label>
<Property.Value>{run.isBuffered ? "Yes" : "No"}</Property.Value>
</Property.Item>
<Property.Item>
<Property.Label>Worker queue</Property.Label>
<Property.Value>{run.workerQueue}</Property.Value>
@@ -1096,7 +1101,7 @@ function RunBody({
{run.isCached ? "Jump to original run" : "Focus on run"}
</LinkButton>
)}
<AdminDebugRun friendlyId={run.friendlyId} />
{!run.isBuffered && <AdminDebugRun friendlyId={run.friendlyId} />}
</div>
<div className="flex items-center">
{run.logsDeletedAt === null ? (
@@ -9,6 +9,8 @@ import { formatDurationMilliseconds } from "@trigger.dev/core/v3/utils/durations
import { getTaskEventStoreTableForRun } from "~/v3/taskEventStore.server";
import { TaskEventKind } from "@trigger.dev/database";
import { getEventRepositoryForStore } from "~/v3/eventRepository/index.server";
import { getMollifierBuffer } from "~/v3/mollifier/mollifierBuffer.server";
import { deserialiseMollifierSnapshot } from "~/v3/mollifier/mollifierSnapshot.server";
export async function loader({ params, request }: LoaderFunctionArgs) {
const user = await requireUser(request);
@@ -30,6 +32,67 @@ export async function loader({ params, request }: LoaderFunctionArgs) {
});
if (!run || !run.organizationId) {
// Buffered run? It hasn't executed, so there are no events to
// stream — but a 404 is wrong: the run does exist, the customer's
// "Download logs" button on the run-detail page generates this
// exact URL, and a 404 reads as "your run vanished" rather than
// "no logs yet". Verify the entry exists in the buffer (with the
// user as a member of the entry's org), and if so stream a single
// informational line in the same `<timestamp> <task> <level>
// <message>` shape `formatRunEvent` uses below — so a downstream
// log viewer / grep over the downloaded file produces a
// meaningful explanation, not a 0-byte mystery.
const buffer = getMollifierBuffer();
if (buffer) {
const entry = await buffer.getEntry(parsedParams.runParam);
if (entry) {
const member = await prisma.orgMember.findFirst({
where: { userId: user.id, organizationId: entry.orgId },
select: { id: true },
});
if (member) {
let taskIdentifier: string | undefined;
try {
// Use the shared webapp wrapper rather than raw JSON.parse so
// every read-side module shares a single deserialisation path
// (see contract comment in `mollifierSnapshot.server.ts` and
// `syntheticRedirectInfo.server.ts`). Keeps behaviour
// consistent if the snapshot encoding ever changes.
const snapshot = deserialiseMollifierSnapshot(entry.payload) as {
taskIdentifier?: unknown;
};
if (typeof snapshot.taskIdentifier === "string") {
taskIdentifier = snapshot.taskIdentifier;
}
} catch {
// Fall through — taskIdentifier stays undefined.
}
const placeholderParts = [
entry.createdAt.toISOString(),
...(taskIdentifier ? [taskIdentifier] : []),
"INFO",
"Run is queued, has not started executing yet — no logs to download.",
];
const placeholder = placeholderParts.join(" ") + "\n";
const placeholderReadable = new Readable({
read() {
this.push(placeholder);
this.push(null);
},
});
const gzipStream = createGzip();
const compressed = placeholderReadable.pipe(gzipStream);
return new Response(compressed as any, {
status: 200,
headers: {
"Content-Type": "application/octet-stream",
"Content-Disposition": `attachment; filename="${parsedParams.runParam}.log"`,
"Content-Encoding": "gzip",
},
});
}
}
}
return new Response("Not found", { status: 404 });
}
@@ -6,6 +6,7 @@ import { redirectWithErrorMessage, redirectWithSuccessMessage } from "~/models/m
import { logger } from "~/services/logger.server";
import { requireUserId } from "~/services/session.server";
import { CancelTaskRunService } from "~/v3/services/cancelTaskRun.server";
import { getMollifierBuffer } from "~/v3/mollifier/mollifierBuffer.server";
export const cancelSchema = z.object({
redirectUrl: z.string(),
@@ -42,15 +43,56 @@ export const action: ActionFunction = async ({ request, params }) => {
},
});
if (!taskRun) {
if (taskRun) {
const cancelRunService = new CancelTaskRunService();
await cancelRunService.call(taskRun);
return redirectWithSuccessMessage(submission.value.redirectUrl, request, `Canceled run`);
}
// PG miss — try the mollifier buffer. The customer can hit cancel
// on a buffered run from the dashboard during the burst window.
// Snapshot a `mark_cancelled` patch; the drainer's
// bifurcation routes the run to `engine.createCancelledRun` on
// next pop.
const buffer = getMollifierBuffer();
const entry = buffer ? await buffer.getEntry(runParam) : null;
if (!entry) {
submission.error = { runParam: ["Run not found"] };
return json(submission);
}
const cancelRunService = new CancelTaskRunService();
await cancelRunService.call(taskRun);
// Dashboard auth: verify the requesting user is a member of the
// buffered run's org. The API path scopes by env id from the
// authenticated request; the dashboard route uses org-membership
// because the URL doesn't carry an envId.
const member = await prisma.orgMember.findFirst({
where: { userId, organizationId: entry.orgId },
select: { id: true },
});
if (!member) {
submission.error = { runParam: ["Run not found"] };
return json(submission);
}
return redirectWithSuccessMessage(submission.value.redirectUrl, request, `Canceled run`);
const result = await buffer!.mutateSnapshot(runParam, {
type: "mark_cancelled",
cancelledAt: new Date().toISOString(),
cancelReason: "Canceled by user",
});
if (result === "applied_to_snapshot") {
return redirectWithSuccessMessage(submission.value.redirectUrl, request, `Canceled run`);
}
// "not_found" or "busy" — both indicate the drainer raced us between
// the getEntry check above and mutateSnapshot. On "not_found" the
// entry was just popped and the PG row is in flight; on "busy" the
// drainer is mid-materialisation. Either way the customer should
// retry — by then the PG row exists and the regular cancel path at
// the top of this action takes over.
return redirectWithErrorMessage(
submission.value.redirectUrl,
request,
"Run is materialising — retry in a moment"
);
} catch (error) {
if (error instanceof Error) {
logger.error("Failed to cancel run", {
@@ -11,6 +11,12 @@ import { requireUser } from "~/services/session.server";
import { sortEnvironments } from "~/utils/environmentSort";
import { v3RunSpanPath } from "~/utils/pathBuilder";
import { ReplayTaskRunService } from "~/v3/services/replayTaskRun.server";
import { getMollifierBuffer } from "~/v3/mollifier/mollifierBuffer.server";
import { findRunByIdWithMollifierFallback } from "~/v3/mollifier/readFallback.server";
import {
buildSyntheticReplayTaskRun,
type SyntheticReplayTaskRun,
} from "~/v3/mollifier/syntheticReplayTaskRun.server";
import parseDuration from "parse-duration";
import { findCurrentWorkerDeployment } from "~/v3/models/workerDeployment.server";
import { queueTypeFromType } from "~/presenters/v3/QueueRetrievePresenter.server";
@@ -33,7 +39,7 @@ export async function loader({ request, params }: LoaderFunctionArgs) {
Object.fromEntries(new URL(request.url).searchParams)
);
const run = await $replica.taskRun.findFirst({
let run = await $replica.taskRun.findFirst({
select: {
payload: true,
payloadType: true,
@@ -88,6 +94,83 @@ export async function loader({ request, params }: LoaderFunctionArgs) {
where: { friendlyId: runParam, project: { organization: { members: { some: { userId } } } } },
});
let synthetic:
| (Awaited<ReturnType<typeof findRunByIdWithMollifierFallback>> & { __synth: true })
| undefined;
if (!run) {
// Buffered fallback: read the snapshot and look up the env list via
// the snapshot's organizationId. Without this the Replay dialog
// 404s for runs queued in the mollifier buffer, which dumps the
// user back to the task list.
const buffer = getMollifierBuffer();
const entry = buffer ? await buffer.getEntry(runParam) : null;
if (!entry) throw new Response("Not Found", { status: 404 });
const member = await prisma.orgMember.findFirst({
where: { userId, organizationId: entry.orgId },
select: { id: true },
});
if (!member) throw new Response("Not Found", { status: 404 });
const buffered = await findRunByIdWithMollifierFallback({
runId: runParam,
environmentId: entry.envId,
organizationId: entry.orgId,
});
if (!buffered) throw new Response("Not Found", { status: 404 });
synthetic = Object.assign(buffered, { __synth: true as const });
// Scope the project lookup to the buffer entry's org as well as the
// env id. The prior `orgMember.findFirst` above confirms the user
// belongs to `entry.orgId`; pinning `organizationId` here means a
// malformed entry whose envId resolves to a different org can't leak
// that project's data through this loader. Mirrors the PG path's
// `project.organization.members.some.userId` scoping (lines 42-95)
// — the env filter and select shape are kept identical so the Replay
// dialog renders the same dropdown either way.
const orgProject = await $replica.project.findFirst({
where: {
organizationId: entry.orgId,
environments: { some: { id: entry.envId } },
},
select: {
slug: true,
environments: {
select: {
id: true,
type: true,
slug: true,
branchName: true,
orgMember: { select: { user: true } },
},
where: {
archivedAt: null,
OR: [
{ type: { in: ["PREVIEW", "STAGING", "PRODUCTION"] } },
{ type: "DEVELOPMENT", orgMember: { userId } },
],
},
},
},
});
if (!orgProject) throw new Response("Not Found", { status: 404 });
run = {
payload: buffered.payload,
payloadType: buffered.payloadType ?? "application/json",
seedMetadata: buffered.seedMetadata ?? null,
seedMetadataType: buffered.seedMetadataType ?? null,
runtimeEnvironmentId: entry.envId,
concurrencyKey: buffered.concurrencyKey ?? null,
maxAttempts: buffered.maxAttempts ?? null,
maxDurationInSeconds: buffered.maxDurationInSeconds ?? null,
machinePreset: buffered.machinePreset ?? null,
workerQueue: buffered.workerQueue ?? null,
ttl: buffered.ttl ?? null,
idempotencyKey: buffered.idempotencyKey ?? null,
runTags: buffered.runTags,
queue: buffered.queue ?? "task/",
taskIdentifier: buffered.taskIdentifier ?? "",
project: orgProject,
} as unknown as typeof run;
}
if (!run) {
throw new Response("Not Found", { status: 404 });
}
@@ -164,6 +247,15 @@ export async function loader({ request, params }: LoaderFunctionArgs) {
}
export const action: ActionFunction = async ({ request, params }) => {
// Dashboard auth: identical pattern to resources.taskruns.$runParam.cancel.ts.
// The loader above this action already gates with `requireUser`, but
// Remix's action runs independently — without this call any request
// with a valid runParam could submit a replay. The PG findFirst below
// also adds the org-membership filter so a PAT can't replay another
// org's run, and the buffered fallback verifies org membership via
// orgMember.findFirst against the snapshot's orgId.
const user = await requireUser(request);
const userId = user.id;
const { runParam } = ParamSchema.parse(params);
const formData = await request.formData();
@@ -174,9 +266,18 @@ export const action: ActionFunction = async ({ request, params }) => {
}
try {
const taskRun = await prisma.taskRun.findFirst({
const pgRun = await prisma.taskRun.findFirst({
where: {
friendlyId: runParam,
project: {
organization: {
members: {
some: {
userId,
},
},
},
},
},
include: {
runtimeEnvironment: {
@@ -192,6 +293,50 @@ export const action: ActionFunction = async ({ request, params }) => {
},
});
// Mollifier read-fallback: if the original isn't in PG yet,
// synthesise a TaskRun from the buffered snapshot. The B4-extended
// SyntheticRun carries every field ReplayTaskRunService reads. We
// also need projectSlug + orgSlug + envSlug for the redirect path,
// so look those up via the snapshot's runtimeEnvironmentId.
let taskRun: SyntheticReplayTaskRun | null = pgRun ?? null;
if (!taskRun) {
const buffer = getMollifierBuffer();
const entry = buffer ? await buffer.getEntry(runParam) : null;
if (entry) {
// Same org-membership gate as the PG path above. Without this
// any authenticated user who knows a runId could replay the
// buffered run across orgs.
const member = await prisma.orgMember.findFirst({
where: { userId, organizationId: entry.orgId },
select: { id: true },
});
if (!member) {
return redirectWithErrorMessage(
submission.value.failedRedirect,
request,
"Run not found"
);
}
const synthetic = await findRunByIdWithMollifierFallback({
runId: runParam,
environmentId: entry.envId,
organizationId: entry.orgId,
});
if (synthetic) {
const envRow = await prisma.runtimeEnvironment.findFirst({
where: { id: entry.envId },
select: {
slug: true,
project: { select: { slug: true, organization: { select: { slug: true } } } },
},
});
if (envRow) {
taskRun = buildSyntheticReplayTaskRun({ synthetic, envRow });
}
}
}
}
if (!taskRun) {
return redirectWithErrorMessage(submission.value.failedRedirect, request, "Run not found");
}
@@ -7,6 +7,7 @@ import type {
MollifierDrainerTerminalFailureHandler,
} from "@trigger.dev/redis-worker";
import { logger } from "~/services/logger.server";
import { recordRunDebugLog } from "~/v3/eventRepository/index.server";
import { PerformTaskRunAlertsService } from "~/v3/services/alerts/performTaskRunAlerts.server";
import { startSpan } from "~/v3/tracing.server";
import type { MollifierSnapshot } from "./mollifierSnapshot.server";
@@ -162,8 +163,10 @@ export function createDrainerHandler(deps: {
span.setAttribute("mollifier.run_friendly_id", input.runId);
span.setAttribute("taskRunId", input.runId);
let triggerSucceeded = false;
try {
await deps.engine.trigger(input.payload as any, deps.prisma);
triggerSucceeded = true;
} catch (err) {
// The retryable-PG class re-throws so the drainer's outer
// worker loop can `buffer.requeue` (handled in
@@ -212,6 +215,54 @@ export function createDrainerHandler(deps: {
throw err;
}
}
// Admin-only audit trail emitted once engine.trigger has
// landed a PG row. `recordRunDebugLog` flips this to the
// admin-gated debug kind (TaskEventKind.LOG in the PG store /
// DEBUG_EVENT in the ClickHouse store) which the trace view +
// logs download already strip for non-admins
// (`eventRepository.server.ts:108`,
// `resources.runs.$runParam.logs.download.ts:118`).
//
// Placement: emit as a zero-duration marker AT materialisation
// time, not as a back-dated bar spanning the buffered window.
// `engine.trigger` rewrites the run's root span at
// materialisation (it adopts the synth root via traceId/spanId
// carryover but updates start_time to "now"), so the trace
// renderer treats materialisation time as t=0. A back-dated
// event with startTime = bufferedAt would land before that t=0
// and get clipped from the tree. Same pattern as the
// `[engine] QUEUED` markers. The window itself is preserved
// in metadata so admins can read it off the span detail pane.
//
// Best-effort: `recordRunDebugLog` has its own try/catch and
// returns a result, so it never throws into the materialisation
// path. Failures are logged but not surfaced because the
// customer-visible run has already landed.
if (triggerSucceeded) {
const debugResult = await recordRunDebugLog(
RunId.fromFriendlyId(input.runId),
`Mollifier buffered ${dwellMs}ms before materialising`,
{
attributes: {
runId: input.runId,
metadata: {
"mollifier.bufferedAt": input.createdAt.toISOString(),
"mollifier.materialisedAt": new Date().toISOString(),
"mollifier.dwellMs": dwellMs,
"mollifier.attempts": input.attempts,
},
},
parentId: snapshotSpanId,
}
);
if (!debugResult.success && debugResult.code !== "RUN_NOT_FOUND") {
logger.warn("mollifier drainer: failed to record admin debug log", {
runId: input.runId,
code: debugResult.code,
});
}
}
});
});
};
@@ -0,0 +1,51 @@
import type { TaskRun } from "@trigger.dev/database";
import type { SyntheticRun } from "./readFallback.server";
export type SyntheticReplayTaskRun = TaskRun & {
project: { slug: string; organization: { slug: string } };
runtimeEnvironment: { slug: string };
};
// Adapt a buffered-run snapshot into the TaskRun-shaped input that
// `ReplayTaskRunService.call` expects. ReplayTaskRunService builds the
// new run's traceparent as `00-${existingTaskRun.traceId}-${existingTaskRun.spanId}-01`
// without guarding for undefined, so a synthetic with missing traceId
// or spanId (older snapshots — both fields are documented optional on
// `SyntheticRun`) would produce `00-undefined-undefined-01`, an invalid
// W3C traceparent that OTel silently drops, severing the replay's trace
// link to the original run.
//
// Returns null when those fields are missing — the caller surfaces this
// as "Run not found" so the customer retries once the drainer has
// materialised the PG row, where traceId/spanId are guaranteed present.
export function buildSyntheticReplayTaskRun(args: {
synthetic: SyntheticRun;
envRow: {
slug: string;
project: { slug: string; organization: { slug: string } };
};
}): SyntheticReplayTaskRun | null {
const { synthetic, envRow } = args;
if (!synthetic.traceId || !synthetic.spanId) return null;
return {
// The double `as unknown as TaskRun` cast is load-bearing — a direct
// `synthetic as TaskRun` won't compile. `SyntheticRun` carries the
// subset of fields that `ReplayTaskRunService.call` actually reads
// (the contract is enumerated on the SyntheticRun type comment in
// readFallback.server.ts), but its shape is not structurally
// assignable to the full Prisma `TaskRun` row: optional vs required
// fields diverge, several PG columns (number, batchId variants,
// status enum widening) are deliberately absent or narrower on the
// synthetic. Routing it through `unknown` is the explicit "we know
// this is a subset, we've audited which fields are read" signal,
// and the traceId/spanId guard above prevents the only field
// ReplayTaskRunService consumes that would corrupt downstream
// behaviour (the OTel traceparent) when undefined.
...(synthetic as unknown as TaskRun),
project: {
slug: envRow.project.slug,
organization: { slug: envRow.project.organization.slug },
},
runtimeEnvironment: { slug: envRow.slug },
};
}
@@ -0,0 +1,75 @@
import type { SyntheticRun } from "./readFallback.server";
// Synthesise the run-detail page's `run` header shape (the NavBar +
// status badge + Cancel-button gate) from a buffered run snapshot. The
// shape matches `RunPresenter.getRun`'s `runData` — keep this in sync
// when fields are added there.
//
// CANCELED and FAILED state is reflected back from
// `SyntheticRun.cancelledAt` / `status` so terminal buffered runs show
// the correct status in the NavBar + isFinished:true (which collapses
// the Cancel button on the page header) before the drainer materialises
// the PG row. This mirrors what `buildSyntheticSpanRun` does for the
// right-side details panel — the SyntheticRun.cancelledAt contract
// comment in readFallback.server.ts names this exact UI surface.
//
// FAILED status maps to `SYSTEM_FAILURE` to match the drainer's
// non-retryable terminal path, which is what `buildSyntheticSpanRun`
// uses too. Symmetric across the header + span-detail panel so an
// admin doesn't see "Pending" + "FAILED" simultaneously on the same
// run.
export function buildSyntheticRunHeader(args: {
run: SyntheticRun;
environment: {
id: string;
organizationId: string;
type: "PRODUCTION" | "DEVELOPMENT" | "STAGING" | "PREVIEW";
slug: string;
};
}) {
const { run, environment } = args;
const isCancelled = run.status === "CANCELED";
const isFailed = run.status === "FAILED";
return {
// `id` mirrors RunPresenter.getRun's runData (the PG path), which
// is the internal cuid — not the friendlyId. SyntheticRun.id is
// already the cuid (RunId.fromFriendlyId(entry.runId) in
// readFallback.server.ts) so the admin debug tooltip on the run
// detail page shows the same format for buffered + materialised
// runs.
id: run.id,
number: 1,
friendlyId: run.friendlyId,
traceId: run.traceId ?? "",
spanId: run.spanId ?? "",
status: isCancelled
? ("CANCELED" as const)
: isFailed
? ("SYSTEM_FAILURE" as const)
: ("PENDING" as const),
isFinished: isCancelled || isFailed,
startedAt: null,
// Symmetric with `buildSyntheticSpanRun` and the
// `ApiRetrieveRunPresenter` synth path. The run-detail route
// derives `isCompleted` from `completedAt !== null` and gates SSE
// live-reloading on it (`route.tsx:459`, `:551`); leaving
// `completedAt` null for FAILED would keep a terminal buffered run
// live-reloading forever. PG-resident SYSTEM_FAILURE rows always
// have completedAt set, so fall back to createdAt (the buffer
// entry has no separate failedAt — closest proxy for when the
// terminal state landed).
completedAt: run.cancelledAt ?? (isFailed ? run.createdAt : null),
logsDeletedAt: null,
rootTaskRun: null,
parentTaskRun: null,
environment: {
id: environment.id,
organizationId: environment.organizationId,
type: environment.type,
slug: environment.slug,
userId: undefined,
userName: undefined,
},
};
}
@@ -189,6 +189,7 @@ export async function buildSyntheticSpanRun(args: {
traceId: run.traceId ?? "",
spanId: run.spanId ?? "",
isCached: false,
isBuffered: true,
machinePreset: narrowMachinePreset(run.machinePreset),
taskEventStore: "taskEvent",
externalTraceId: undefined,
@@ -19,6 +19,22 @@ vi.mock("~/v3/services/alerts/performTaskRunAlerts.server", () => ({
},
}));
// The drainer calls `recordRunDebugLog` after a successful engine.trigger
// to emit an admin-only LOG-kind event encoding the buffered window.
// The real implementation imports the configured event repository (prisma
// + clickhouse + env), which has heavy side-effects on first import.
// Stub it to a vi.fn so the unit tests can assert call shape without
// dragging the whole eventRepository graph into webapp test setup.
// `vi.hoisted` is required because `vi.mock` factories are hoisted above
// regular `const`s — referencing a top-level variable from inside the
// factory otherwise fires `Cannot access 'X' before initialization`.
const { recordRunDebugLogMock } = vi.hoisted(() => ({
recordRunDebugLogMock: vi.fn(async () => ({ success: true as const })),
}));
vi.mock("~/v3/eventRepository/index.server", () => ({
recordRunDebugLog: recordRunDebugLogMock,
}));
import {
createDrainerHandler,
isRetryablePgError,
@@ -450,4 +466,109 @@ describe("createDrainerHandler", () => {
).rejects.toThrow("engine rejected the snapshot");
expect(createFailedTaskRun).not.toHaveBeenCalled();
});
it("emits an admin-only LOG-kind event with the buffered window after engine.trigger succeeds", async () => {
// The drainer's audit trail rides the existing TaskEventKind.LOG
// filter pattern (`eventRepository.server.ts:108` + `logs.download.ts:118`)
// — admins see the buffered window in the trace; non-admins don't.
recordRunDebugLogMock.mockClear();
const trigger = vi.fn(async () => ({ friendlyId: "run_z" }));
const handler = createDrainerHandler({
engine: { trigger } as any,
prisma: {} as any,
});
const bufferedAt = new Date(Date.now() - 4_000);
await handler({
runId: "run_z",
envId: "env_a",
orgId: "org_1",
payload: { taskIdentifier: "t", spanId: "snapspan", traceId: "snaptrace" },
attempts: 2,
createdAt: bufferedAt,
} as any);
expect(recordRunDebugLogMock).toHaveBeenCalledOnce();
const [callRunId, message, options] = recordRunDebugLogMock.mock.calls[0] as [
string,
string,
any,
];
// Internal cuid derived from the friendlyId, mirroring what
// `findRunForEventCreation` queries on.
expect(callRunId).toBe("z");
expect(message).toMatch(/Mollifier buffered \d+ms before materialising/);
// Emitted as a marker at materialisation time (no `startTime` /
// `duration` overrides) — engine.trigger has just rewritten the
// root span's start_time to "now", so back-dating the event would
// clip it off-screen in the trace renderer. The historical window
// is preserved in metadata so admins can still read it.
expect(options.startTime).toBeUndefined();
expect(options.duration).toBeUndefined();
expect(options.parentId).toBe("snapspan");
expect(options.attributes.metadata["mollifier.bufferedAt"]).toBe(bufferedAt.toISOString());
expect(options.attributes.metadata["mollifier.attempts"]).toBe(2);
expect(options.attributes.metadata["mollifier.dwellMs"]).toBeGreaterThan(0);
});
it("does NOT emit the admin LOG event when engine.trigger fails non-retryably", async () => {
// The audit trail is for runs that actually materialised. On a
// terminal SYSTEM_FAILURE path the customer-visible outcome is the
// failure row; emitting a "buffered for Xms" event next to it would
// imply the buffered window completed normally.
recordRunDebugLogMock.mockClear();
const trigger = vi.fn(async () => {
throw new Error("engine rejected the snapshot");
});
const createFailedTaskRun = vi.fn(async () => ({ id: "internal" }));
const handler = createDrainerHandler({
engine: { trigger, createFailedTaskRun } as any,
prisma: {} as any,
});
await handler({
runId: "run_z",
envId: "env_a",
orgId: "org_1",
payload: { taskIdentifier: "t", environment: envFixture },
attempts: 0,
createdAt: new Date(),
} as any);
expect(recordRunDebugLogMock).not.toHaveBeenCalled();
});
it("does NOT emit the admin LOG event on the cancel-bifurcation path", async () => {
// Cancel-bifurcation writes a CANCELED row directly without calling
// engine.trigger. There's no buffered-then-materialised window to
// describe — the run never ran.
recordRunDebugLogMock.mockClear();
const friendlyId = RunId.generate().friendlyId;
const createCancelledRun = vi.fn(async () => ({
id: "internal",
friendlyId,
status: "CANCELED",
}));
const handler = createDrainerHandler({
engine: { createCancelledRun } as any,
prisma: {} as any,
});
await handler({
runId: friendlyId,
envId: "env_a",
orgId: "org_1",
payload: {
friendlyId,
taskIdentifier: "t",
environment: envFixture,
cancelledAt: new Date().toISOString(),
cancelReason: "Canceled by user",
},
attempts: 0,
createdAt: new Date(),
} as any);
expect(recordRunDebugLogMock).not.toHaveBeenCalled();
});
});
@@ -0,0 +1,99 @@
import { describe, expect, it } from "vitest";
import {
serialiseMollifierSnapshot,
deserialiseMollifierSnapshot,
} from "~/v3/mollifier/mollifierSnapshot.server";
import { prettyPrintPacket } from "@trigger.dev/core/v3";
// Regression test for the Devin "Buffered replay loader passes
// non-string payload to prettyPrintPacket" finding on PR #3757.
//
// Devin's claim is that the snapshot codec double-unwraps the
// payload: `engine.trigger` carries it pre-serialised, then the
// snapshot serialise/deserialise round-trip would JSON.parse it a
// second time, leaving `buffered.payload` as a *parsed* object —
// which `prettyPrintPacket` then mis-handles, producing malformed
// payload display in the Replay dialog.
//
// This test pins the actual contract: the snapshot codec is a single
// JSON.stringify / JSON.parse layer. The payload field stored on the
// engine trigger input is a string (the SDK-serialised payload from
// `payloadPacket.data`). A string round-trips through
// JSON.stringify/JSON.parse unchanged — it does NOT get a second
// unwrap. Therefore `buffered.payload` reaches the replay loader as
// a string, exactly the shape `prettyPrintPacket` expects.
describe("mollifier replay payload shape", () => {
it("serialise/deserialise preserves the payload as a string", () => {
// Shape mirrors what `triggerTask.server.ts:#buildEngineTriggerInput`
// produces — `payload` is `args.payloadPacket.data`, already a JSON
// string from the SDK's packet serialisation.
const triggerInput = {
friendlyId: "run_x",
taskIdentifier: "hello-world",
payload: JSON.stringify({ hello: "world", n: 42 }),
payloadType: "application/json",
traceId: "trace_x",
spanId: "span_x",
};
const serialised = serialiseMollifierSnapshot(triggerInput);
const roundTripped = deserialiseMollifierSnapshot(serialised);
expect(typeof roundTripped.payload).toBe("string");
expect(roundTripped.payload).toBe(triggerInput.payload);
expect(roundTripped.payloadType).toBe("application/json");
});
it("prettyPrintPacket on the round-tripped payload produces the expected pretty JSON", async () => {
const original = { hello: "world", nested: { count: 3 } };
const triggerInput = {
payload: JSON.stringify(original),
payloadType: "application/json",
};
const roundTripped = deserialiseMollifierSnapshot(
serialiseMollifierSnapshot(triggerInput),
);
// This is exactly the call the replay loader makes:
// prettyPrintPacket(run.payload, run.payloadType)
// If Devin were right, the payload here would be a parsed object
// and prettyPrintPacket would either double-encode or skip
// formatting. In reality it's a string, so we get correct pretty
// JSON.
const pretty = await prettyPrintPacket(
roundTripped.payload,
roundTripped.payloadType as string,
);
expect(pretty).toBe(JSON.stringify(original, null, 2));
});
it("string payload survives the buffer-codec round-trip even with snapshot fields around it", () => {
// Replicate the realistic snapshot shape (the engine.trigger input
// has many sibling fields). Confirms there's no field-shape
// interaction that would mutate payload.
const triggerInput = {
friendlyId: "run_x",
environment: {
id: "env",
type: "DEVELOPMENT",
project: { id: "p" },
organization: { id: "o" },
},
taskIdentifier: "t",
payload: '{"a":1}',
payloadType: "application/json",
context: { run: { id: "x" } },
traceContext: { traceparent: "00-...-..." },
traceId: "abc",
spanId: "def",
tags: ["one", "two"],
depth: 2,
isTest: false,
};
const out = deserialiseMollifierSnapshot(serialiseMollifierSnapshot(triggerInput));
expect(typeof out.payload).toBe("string");
expect(out.payload).toBe('{"a":1}');
});
});
@@ -0,0 +1,106 @@
import { describe, expect, it, vi } from "vitest";
vi.mock("~/db.server", () => ({ prisma: {}, $replica: {} }));
import { buildSyntheticReplayTaskRun } from "~/v3/mollifier/syntheticReplayTaskRun.server";
import type { SyntheticRun } from "~/v3/mollifier/readFallback.server";
const NOW = new Date("2026-05-21T10:00:00Z");
function makeSyntheticRun(overrides: Partial<SyntheticRun> = {}): SyntheticRun {
return {
id: "run_internal_1",
friendlyId: "run_friendly_1",
status: "QUEUED",
cancelledAt: undefined,
cancelReason: undefined,
delayUntil: undefined,
taskIdentifier: "hello-world",
createdAt: NOW,
payload: { message: "hi" },
payloadType: "application/json",
metadata: undefined,
metadataType: undefined,
seedMetadata: undefined,
seedMetadataType: undefined,
idempotencyKey: undefined,
idempotencyKeyOptions: undefined,
isTest: false,
depth: 0,
ttl: "10m",
tags: [],
runTags: [],
lockedToVersion: undefined,
resumeParentOnCompletion: false,
parentTaskRunId: undefined,
traceId: "trace_1",
spanId: "span_1",
parentSpanId: undefined,
runtimeEnvironmentId: "env_a",
engine: "V2",
workerQueue: "worker-queue-1",
queue: "task/hello-world",
concurrencyKey: undefined,
machinePreset: "small-1x",
realtimeStreamsVersion: "v1",
maxAttempts: 3,
maxDurationInSeconds: 3600,
replayedFromTaskRunFriendlyId: undefined,
annotations: undefined,
traceContext: undefined,
scheduleId: undefined,
batchId: undefined,
parentTaskRunFriendlyId: undefined,
rootTaskRunFriendlyId: undefined,
...overrides,
};
}
const ENV_ROW = {
slug: "dev",
project: { slug: "hello-world", organization: { slug: "references" } },
};
describe("buildSyntheticReplayTaskRun", () => {
it("returns the adapted TaskRun shape when traceId and spanId are present", () => {
const taskRun = buildSyntheticReplayTaskRun({
synthetic: makeSyntheticRun(),
envRow: ENV_ROW,
});
expect(taskRun).not.toBeNull();
expect(taskRun!.traceId).toBe("trace_1");
expect(taskRun!.spanId).toBe("span_1");
expect(taskRun!.project.slug).toBe("hello-world");
expect(taskRun!.project.organization.slug).toBe("references");
expect(taskRun!.runtimeEnvironment.slug).toBe("dev");
});
it("returns null when the snapshot has no traceId", () => {
// ReplayTaskRunService builds `00-${traceId}-${spanId}-01` without
// guarding for undefined. Falling through with a missing traceId
// would emit `00-undefined-...-01`, an invalid W3C traceparent that
// OTel silently drops, breaking the replayed run's trace linkage to
// the original. The helper must refuse rather than degrade silently.
const taskRun = buildSyntheticReplayTaskRun({
synthetic: makeSyntheticRun({ traceId: undefined }),
envRow: ENV_ROW,
});
expect(taskRun).toBeNull();
});
it("returns null when the snapshot has no spanId", () => {
const taskRun = buildSyntheticReplayTaskRun({
synthetic: makeSyntheticRun({ spanId: undefined }),
envRow: ENV_ROW,
});
expect(taskRun).toBeNull();
});
it("returns null when both traceId and spanId are missing", () => {
const taskRun = buildSyntheticReplayTaskRun({
synthetic: makeSyntheticRun({ traceId: undefined, spanId: undefined }),
envRow: ENV_ROW,
});
expect(taskRun).toBeNull();
});
});
@@ -0,0 +1,130 @@
import { describe, expect, it, vi } from "vitest";
vi.mock("~/db.server", () => ({ prisma: {}, $replica: {} }));
import { buildSyntheticRunHeader } from "~/v3/mollifier/syntheticRunHeader.server";
import type { SyntheticRun } from "~/v3/mollifier/readFallback.server";
const NOW = new Date("2026-05-21T10:00:00Z");
const CANCELLED_AT = new Date("2026-05-21T10:00:30Z");
function makeSyntheticRun(overrides: Partial<SyntheticRun> = {}): SyntheticRun {
return {
id: "run_internal_1",
friendlyId: "run_friendly_1",
status: "QUEUED",
cancelledAt: undefined,
cancelReason: undefined,
delayUntil: undefined,
taskIdentifier: "hello-world",
createdAt: NOW,
payload: { message: "hi" },
payloadType: "application/json",
metadata: undefined,
metadataType: undefined,
seedMetadata: undefined,
seedMetadataType: undefined,
idempotencyKey: undefined,
idempotencyKeyOptions: undefined,
isTest: false,
depth: 0,
ttl: "10m",
tags: [],
runTags: [],
lockedToVersion: undefined,
resumeParentOnCompletion: false,
parentTaskRunId: undefined,
traceId: "trace_1",
spanId: "span_1",
parentSpanId: undefined,
runtimeEnvironmentId: "env_a",
engine: "V2",
workerQueue: "worker-queue-1",
queue: "task/hello-world",
concurrencyKey: undefined,
machinePreset: "small-1x",
realtimeStreamsVersion: "v1",
maxAttempts: 3,
maxDurationInSeconds: 3600,
replayedFromTaskRunFriendlyId: undefined,
annotations: undefined,
traceContext: undefined,
scheduleId: undefined,
batchId: undefined,
parentTaskRunFriendlyId: undefined,
rootTaskRunFriendlyId: undefined,
...overrides,
};
}
const ENV = {
id: "env_a",
organizationId: "org_a",
type: "DEVELOPMENT" as const,
slug: "dev",
};
describe("buildSyntheticRunHeader", () => {
it("returns PENDING / non-final state for a queued buffered run", () => {
const header = buildSyntheticRunHeader({ run: makeSyntheticRun(), environment: ENV });
expect(header.status).toBe("PENDING");
expect(header.isFinished).toBe(false);
expect(header.completedAt).toBeNull();
});
it("reflects CANCELED state from the snapshot so the NavBar and Cancel-button gate update before the drainer materialises", () => {
const header = buildSyntheticRunHeader({
run: makeSyntheticRun({ status: "CANCELED", cancelledAt: CANCELLED_AT }),
environment: ENV,
});
// The Cancel button in route.tsx is gated on `!run.isFinished` and the
// status badge reads `run.status`. Both must flip on buffered-cancel
// or the user sees a "Pending" badge with a Cancel button on a run
// that's already cancelled in the snapshot.
expect(header.status).toBe("CANCELED");
expect(header.isFinished).toBe(true);
expect(header.completedAt).toEqual(CANCELLED_AT);
});
it("populates completedAt for FAILED runs so the route stops live-reloading and renders as completed", () => {
// The run-detail route derives `isCompleted` from
// `run.completedAt !== null` and gates SSE live-reloading on it
// (`route.tsx:459`, `:551`). Leaving completedAt null for FAILED
// buffered runs would keep a terminal run live-reloading forever
// while the badge already says SYSTEM_FAILURE. Symmetric with
// buildSyntheticSpanRun + ApiRetrieveRunPresenter.
const header = buildSyntheticRunHeader({
run: makeSyntheticRun({ status: "FAILED" }),
environment: ENV,
});
expect(header.status).toBe("SYSTEM_FAILURE");
expect(header.isFinished).toBe(true);
expect(header.completedAt).toEqual(NOW);
});
it("forwards identity and environment fields from the snapshot", () => {
const header = buildSyntheticRunHeader({ run: makeSyntheticRun(), environment: ENV });
expect(header.friendlyId).toBe("run_friendly_1");
// `id` mirrors RunPresenter.getRun (the PG path) which puts the
// internal cuid in this field. SyntheticRun.id is the cuid; the
// header must surface it (not the friendlyId).
expect(header.id).toBe("run_internal_1");
expect(header.traceId).toBe("trace_1");
expect(header.spanId).toBe("span_1");
expect(header.environment).toMatchObject({
id: "env_a",
organizationId: "org_a",
type: "DEVELOPMENT",
slug: "dev",
});
});
it("falls back to empty strings when the snapshot has no trace/span ids", () => {
const header = buildSyntheticRunHeader({
run: makeSyntheticRun({ traceId: undefined, spanId: undefined }),
environment: ENV,
});
expect(header.traceId).toBe("");
expect(header.spanId).toBe("");
});
});