`curateReport` still read `facts.staleReason`, renamed to `untrustworthyReason`
in dc3b50260 and split into telemetry_stale / telemetry_absent / flow_unmeasured.
The read had been undefined since, so the agent got "untrustworthy" with no why,
and the prompt still told it every such case was stale telemetry.
`facts` is `z.record(z.unknown())`, so nothing typechecked the key. The new test
goes presenter -> reports route JSON -> curateReport without naming a facts key
on the way in, and asserts curation carries every key the presenter emits.
The prompt requires render_view and the schema union allows actions, investigation
and report blocks, but ViewBlocks only had cases for diagnosis and chart, so those
three rendered as an empty div. Adds the three renderers, the block-envelope
latest-wins resolution the switch keys on, and a contract test that reads the block
types off viewBlockSchema, so a new union member fails until it has a renderer.
An unmeasurable depth made the whole flow finding unassessable, so a
critical start latency measured off runs showed as crit in format=json
while the rendered report said 'Flow unknown ... nothing to do' and
dropped the row that earned it. The depth still buys no cause, no
attribution and no drain ETA, but the measured symptom is now reported.
A missing telemetry feed said the run aggregates were a point-in-time
snapshot; they are measured over the window either way, and what is
actually unknown is how current they are. An unmeasured queue depth
blamed throughput, which the report does measure, and contradicted its
own headline. Tests pin the claim rather than the wording.
The agent could not read anything on a preview or dev branch. Its environment name
is derived from the environment's type, and every branch shares its parent's type —
so the name identifies a family, not a row. The API's env routes are name-addressed,
so a bare "preview" or "dev" resolved to the parent, and the delegated token's
`environmentId` claim then correctly refused it. The guard was the detector, not the
defect: the exchange was never given enough identity to resolve the environment the
dashboard had selected.
Name and branch are now resolved together into one address, so no caller can take
the name without the branch, and both mint sites share the one type map instead of
keeping a copy each. The address travels to the JWT exchange and to the three
delegated-token reads that resolve by name (list_tasks, correlate_version, the repo
snapshot). The env-JWT reads address the environment by id and are untouched.
Second, and why nobody saw the first: the exchange reported the same "no
environment" for a genuine absence and for any failure, and cached the failure for
the whole turn. Following the queue live-read precedent, only a missing environment
is stated as one; anything else says the read didn't land and carries its status.
The helper checks no scopes and no capability context, so a read-only user-actor token
reached an alert-channel write and every admin route behind requireAdminApiRequest.
Actor-aware routes are unaffected: they authenticate the token through the route
builders, which enforce its claims.
flag() queried the FeatureFlag row before looking at the caller-supplied overrides, so a
per-org hit still paid a round-trip. Check the override first and only fall through to the
query when it fails the schema, which keeps today's resolution order intact.
A start that rejects dispatched no handover and sent no message, so the chat
row is taken back. A failed access-token mint is left alone: the session is
live by then and removing the chat would hide a running agent.
A token signed for a branch child listed nothing: the scope filter and the
base-environments-only filter could never hold together. Unscoped callers
still see base environments only.
The environment lookup, repo lookup and token mint now all run before the chat
row is created, so a 404 or a mint failure can't orphan a chat in the history.
Absent telemetry and an unmeasured flow were both labelled stale data, so every
snapshot-based report claimed staleness it could not have measured. Choose the
badge and caveat from the reason instead.
A fall's multiplier rounds to 0 or 1, so every drop rendered as "flat" — a metric
that collapsed from 100 to 5 read as unchanged. Measure the fall against the
baseline instead, and show a bare arrow when it collapsed to nothing.
The per-queue metrics route mapped ClickHouse rows straight to an array, so a
bucket with no sample shortened the trend and shifted every later point in time.
Fill a fixed-width grid the way the two sibling callers do.
Dropping the column was irreversible and blocked on a production row count; retiring it is not. The avatar host is an exact origin the app already knows, like the GitHub one.
The route served a deployment's git blob — commit message, author, branch,
PR title — with no ability check, while the deployments list serves the same
blob behind read on deployments. Apply that check here too.
The shape descriptor for a redacted object copied its key names verbatim, so
a payload keyed by an email address sent that address to the judge model and
into the chat_turn_evals row. Emit a count instead, in both the shape and the
depth-cap descriptor, and stop allow-listing "keys" so a tool's own field of
that name is redacted like any other.
ask_support defaulted its URL to localhost and gated only on the secret, so
with the secret set and the URL unset the user's question and the bearer
secret went to a local port inside the agent's container.
list_environments, get_run, get_run_trace and get_error interpolated a bare
z.string() straight into the request path, so a model-chosen id could steer
the call at another same-origin route.
assertUserActorScope returned early whenever the passed scope carried no
org, project or environment, and the route builder passes {} for any route
that declares no context — so the guard was a no-op there. api.v1.orgs's
action is such a route and has no authorization block either, letting a
read-only agent token create an organization.
Fail closed instead, with an explicit identityOnly opt-in for the two
contextless loaders that answer with the caller's own identity, and give
org creation the gate its siblings have.
The scope ceiling rewrite landed here and was reverted two PRs up, leaving the
stack asserting both directions. The exchange intersects requested scopes with the
token's cap again, a capless token passes through like a PAT, and the route keeps
only the environment claim check and the acting client.
The dashboard agent asks for a queue's live row — paused, depth, limit — through the environment JWT it exchanges for. The metrics route has accepted that JWT all along; the retrieve route answered 401, so the agent saw no queue at all.
Review of #4418: the cloud path builds the ability from the user's role, so a read-only delegated token could exchange it for a write JWT; and the finalisable set was the whole replayed transcript rather than what the turn produced.
⚒️ Publish Worker (v4) / build (supervisor) (push) Has been cancelled
## Summary
Prisma expands `in` / `notIn` into one bind parameter per element, so
every distinct list
length is a separate prepared statement. Where the length tracks data
volume (a batch size,
a run-graph fan-out, a prior query's id set) one call site can mint
hundreds of them. Each
is used about once, but inserting it evicts an entry that was being
reused, so the cost
lands on unrelated queries sharing the pooler's statement cache. An
unbounded list also
risks the 65535 bind-parameter ceiling.
`boundedIn()` pads a filter list to the next power of two by repeating
its last element.
`IN` and `NOT IN` ignore duplicates, so results are unchanged, and a
call site drops from
one statement per length to at most `log2(cap)`. Applied to all existing
sites.
## Enforcement
Two oxlint rules require the helper: a list filter must be an inline
array literal or a
`boundedIn()` call.
- The first covers filters reached through `where` / `having` /
`cursor`, and deliberately
never descends into `data`, `create`, `update`, `set` or `equals`. A key
named `in` in
those positions is user data, not a predicate, and rewriting it would
corrupt what gets
stored or compared.
- The second covers bare filter objects passed to where-building
helpers, which the first
cannot see. It found five sites in the run-graph batch loaders that were
otherwise
invisible.
Both rules follow filters through the shapes they are actually written
in: conditional
expressions, logical-and objects, spread-conditional properties,
computed keys, and call
arguments. An array literal only counts as fixed-arity when nothing
spreads into it, since
`[...new Set(ids)]` has a runtime length. Twelve sites were hidden
behind those shapes
until the rules handled them.
Scoped to `in` and `notIn`. The scalar-list filters `hasSome` and
`hasEvery` compile to
`&& $1` and `@> $1`, passing the whole array as a single bind parameter,
so their arity never
reaches the statement text and there is nothing to bound.
Both rules are `error`, so new call sites fail CI. That ratchet has
already caught four
sites added by other PRs while this one was in review.
## Notes
`boundedIn` pads by repeating rather than with null: `x NOT IN (a, b,
NULL)` is never true,
so null-padding a `notIn` filter would silently return no rows. Lists
above 32768 are
returned unchanged so padding can never push a query past the parameter
limit.
Route modules reach the helper through `~/db.server` rather than
importing the database
barrel directly, since a value import of that barrel into a module that
also exports a React
component is only safe while dead-code elimination prunes it.
Measured on a local rig: 300 distinct list lengths produce 300 prepared
statements
unpadded, 10 padded. Verified end-to-end against a local stack with the
full task-suite
sweep, which surfaced no regressions.
## What
The internal tracing `ParentBasedSampler` in `tracer.server.ts` left
`remoteParentSampled` at its default of `AlwaysOn`. Any request arriving
with a `traceparent` whose sampled flag was set got recorded in full,
bypassing `INTERNAL_OTEL_TRACE_SAMPLING_RATE` entirely. Because the SDK
propagates its (always-sampled) trace context on calls back to the
platform from inside running tasks, the large majority of API server
spans inherited a sampled parent and ignored the divisor. The sampling
knob was effectively inert on the busiest service.
This registers a custom propagator
(`NonInheritingTraceContextPropagator`) that stops adopting the inbound
trace as the parent:
- `inject` still delegates to the standard W3C trace + baggage
propagators, so outbound propagation is unchanged.
- `extract` drops the parent span (`trace.deleteSpan`) while preserving
baggage, so every incoming request roots its own trace and the ratio
sampler applies uniformly.
`remoteParentSampled` is also set to the ratio sampler as a
belt-and-suspenders fallback, in case an inbound sampled parent ever
reaches the sampler another way.
Two effects: the divisor becomes effective on the API server, and the
API no longer stitches onto (and inflates) the propagated task-run
traces, which is where the very large, un-thinnable trace chains came
from. Rooting each request removes those chains rather than only
diluting them.
Only the internal APM trace pipeline
(`INTERNAL_OTEL_TRACE_EXPORTER_URL`) is affected. The user-facing
run-trace pipeline (`otel.v1.traces` -> ClickHouse) is a separate path
and is untouched. The only consumer of the global propagator's `extract`
is the OTel HTTP/Express auto-instrumentation, so the blast radius is
inbound-request trace shape.
## Evidence (local full-stack red/green, divisor 10)
A local OTLP/JSON sink counting spans; a driver fires N requests at a
real endpoint, each carrying a distinct sampled `traceparent`, then
counts how many spans/traces carry that run's marker.
| run | code | sent | kept traces | kept fraction |
| --- | --- | --- | --- | --- |
| before | unmodified | 500 | 500 | 1.00 |
| after | this PR | 500 | 67 | 0.134 |
| after | this PR | 2000 | 213 | 0.1065 |
Before: 100% of inherited-sampled requests kept, divisor ignored. After:
~10% kept (the divisor), converging on it at larger N. In every
after-run each kept request is a single self-rooted trace (kept spans ==
kept distinct traces), confirming the inherited chains are gone, not
just thinned. `typecheck` passes.
## Rollout / rollback
No flag. Behavior stays governed by the existing
`INTERNAL_OTEL_TRACE_SAMPLING_RATE`. Rollback is a straight revert with
no data migration.
## Notes
Internal dashboards that count raw span or request volume from this
pipeline will read lower once this ships. That is expected: those counts
were inflated by the bypass, not a real drop in traffic.
Latency/percentile monitors retain plenty of samples at the current
divisor.
refs TRI-13031
The gate counted transcript length, and a warm first turn arrives with the model's opening step already in it — so the very first exchange looked like a later one.
A turn stores its messages before the model finishes, so the completed bodies arrived against ids that already existed and were skipped. Reopening a chat then replayed a tool call that never ends.
The system — contracts, storage, auth, the agent package and its webapp routes — lands first; the panel, the page-context marks and the entry points follow in their own PR.
## What
A trigger request carrying a Unicode NUL (`U+0000`) in the **idempotency
key** or **debounce key** reached `prisma.taskRun.create()` and failed
the insert, so the caller got an opaque 500 and the run was never
created.
These two keys are stored in `jsonb` columns (`idempotencyKeyOptions`,
`debounce`), and Postgres rejects a NUL inside a `jsonb` value with
`SQLSTATE 22P05` ("unsupported Unicode escape sequence ... cannot be
converted to text"). This fix strips the NUL from both keys at the
single trigger-input chokepoint (`#buildEngineTriggerInput`), which
every trigger path flows through (single, batch item, mollified, and
drainer replay).
Stripping matches the existing precedent for run errors and task events.
It does not change dedup behaviour: the idempotency **dedup identity**
is the hashed key (a clean 64-char digest), computed independently of
the raw key we clean, so dedup keeps working exactly as before. For
debounce the key is used directly, so the cleaned key also becomes the
grouping key, an acceptable change for input that is already malformed.
## Why not payload / metadata / tags
Those are `text` columns fed by `JSON.stringify`, which escapes a NUL to
a safe escape sequence, so they do not hit this failure on the normal
JSON path. (A raw NUL in a `text` column throws a different code,
`22021`, and is not what triggers this issue.) The observed failures are
the `jsonb` `22P05` variant, which is only reachable via the two key
fields.
## Evidence
Red then green (containerTest, real Postgres): with the fix reverted,
triggering through the real service with a NUL in
`idempotencyKeyOptions.key` / `debounce.key` fails with the exact
`22P05` signature; with the fix, the run is created and the stored key
has the NUL removed.
Full-stack e2e (isolated stack, real HTTP): `POST
/api/v1/tasks/:taskId/trigger` with a NUL inside
`idempotencyKeyOptions.key` (`"acme<NUL>inc"`) and, separately,
`debounce.key` (`"grp<NUL>1"`):
- both returned `HTTP 200` with a created run (previously `500`)
- stored `idempotencyKeyOptions` = `{ "key": "acmeinc", "scope": "run"
}` (7 chars, NUL removed)
- stored `debounce.key` = `"grp1"` (4 chars, NUL removed)
- both runs render in the dashboard
Unit tests cover the helper (strip, no-op fast path, object-reference
reuse, null/undefined pass-through).
## Rollout / rollback
Server-only webapp change, no flag. Zero behaviour change for clean
input; only affects inputs that previously 500'd. Rollback is a straight
revert, no data migration.
## Known limitation
A raw NUL in a plain-string idempotency key (not created via
`idempotencyKeys.create()`) lands in a `text` column and throws `22021`
instead. That variant is not addressed here because stripping it would
change the dedup identity, so it warrants a separate decision. Not
observed in practice.
refs TRI-13030
Katia Bulatova