Eric Allam
6e5f0f0fe7
## Summary
A query sent to the query API with a typo in it, like a column name that
does not exist, was being reported as a server error. That put customer
SQL mistakes into our error alerting, where they made up almost all of
the volume on one of our noisiest alerts, and it drowned out the
failures that are actually ours to fix. This makes the level match who
is at fault, and fixes two related problems found alongside it.
## Invalid queries are the caller's, not ours
The query API route already got this right. It checks for `QueryError`,
logs at warn, and returns a 400, with a comment saying the system
handles it gracefully and no alert is needed.
The layer underneath ignored that. `executeTSQL` logged every exception
out of its catch block at error, including the compile failures the
route was about to turn into a 400, and error-level logs are forwarded
to error reporting.
The TSQL package already draws the line we need:
```ts
export class ExposedTSQLError extends BaseTSQLError {
/** An exception that can be exposed to the user. */
}
export class InternalTSQLError extends BaseTSQLError {
/** An internal exception in the TSQL engine. */
}
```
`SyntaxError` and `QueryError` extend the first. So the catch block now
branches on `ExposedTSQLError` and logs those at warn, keeping error for
`InternalTSQLError` and anything unanticipated, which is a genuine
compiler bug.
## SQL the caller wrote is their mistake, not ours
The same asymmetry showed up one level down. A query that compiles fine
can still be rejected by ClickHouse at execution, and most of those
rejections mean the caller's SQL is wrong rather than that we generated
something bad.
This is where the volume actually is. Checking production, one error
group alone, a missing `GROUP BY` on the public query API
(`NOT_AN_AGGREGATE`), accounts for over a million events across hundreds
of users. It is by far the largest error group in the project, and
classifying only by resource limit would have left every one of those at
error level.
So rejections are split three ways in `ClickhouseClient`, which is the
only place holding the parsed `ClickHouseError` and its symbolic type.
By the time the error reaches `executeTSQL` it has been wrapped and the
type is gone, and the type never appears in the message text, so it
cannot be recovered by string matching.
- **Resource limits** (memory ceiling, timeout, row/byte caps) log at
warn. The query is valid, it just asked for more than it is allowed to
spend.
- **Invalid SQL** (`NOT_AN_AGGREGATE`, `UNKNOWN_IDENTIFIER`,
`SYNTAX_ERROR`, the type and parse families) logs at warn **only when
the caller wrote the SQL**.
- **Everything else** keeps alerting.
That gate matters. The client is shared, so the identical rejection on
TRQL *we* generated is our bug and has to stay at error. Callers opt in
with `userAuthoredQuery`:
| caller | who wrote the SQL | opts in |
| --- | --- | --- |
| public query API | the customer | yes |
| query editor | the customer | yes |
| agent charts | the agent's model | yes |
| built-in dashboard tiles | us, in code | no |
| queue metric cards | us, in code | no |
| health report | us, in code | no |
The agent is the one judgement call. Its TRQL is not typed by a person,
but it is also not something a code fix makes correct, so a query it
gets wrong is not worth waking anyone for. The same endpoint serves
built-in tiles whose TRQL we do write, so the opt-in lives with the
caller rather than the route.
Separately, when one of these queries did fail, the log recorded the
generated ClickHouse SQL but not the query the caller actually wrote,
which made the reports hard to act on. `queryWithStats` takes an
optional `logFields` that `executeTSQL` uses to attach the original
TSQL.
## Events were attributed to the wrong request
Chasing the above turned up something broader: only a tenth of the
events on that alert pointed at the query API. The rest were pinned to
unrelated requests that happened to be in flight at the same time, so
the alert looked like the trigger endpoint was failing.
`Sentry.init` runs with `skipOpenTelemetrySetup: true`, because we
register our own OTel pipeline. That skips `initOpenTelemetry`, and one
of the things it does is:
```js
api.context.setGlobalContextManager(new SentryContextManager());
```
The async-context strategy is still installed, but `withIsolationScope`
only marks the OTel context and delegates the actual fork to that
context manager:
```js
// "We depend on the otelContextManager to handle the context/hub"
return api.context.with(ctx.setValue(SENTRY_FORK_ISOLATION_SCOPE_CONTEXT_KEY, true), ...)
```
`provider.register()` installed a plain
`AsyncLocalStorageContextManager`, which does not know that key. The
lookup found no scopes on the context and fell back to the
process-global default isolation scope, so every request wrote its
request data into the same object and the last writer won.
The tracer now registers `SentryContextManager`, which subclasses
`AsyncLocalStorageContextManager`, so OTel behaviour is unchanged. It is
also registered on the path where tracing is disabled, which previously
never called `register()` at all and so had no context manager of its
own.
Tenant tags were always correct, because those come from our own async
local storage rather than the isolation scope. That is why the
attribution being wrong was not obvious.
This affects every error report the webapp sends, not just the query
API.
## Verification
`internal-packages/clickhouse`: 76 tests pass, including eight covering
each level decision against a real ClickHouse container. Three pairs pin
the gate open and shut at both layers: an invalid query, a compile
failure, and a real limit breach driven with `max_rows_to_read` each log
at warn with `userAuthoredQuery` and at error without it.
The isolation fix has a test that reproduces the leak before asserting
the fix. Two overlapping requests each tag their own isolation scope;
with the plain context manager the slower one reads back the other's
tag, and with `SentryContextManager` each reads back its own.
Measured separately against a faithful reproduction of the server's
wiring (own OTel pipeline, CommonJS entry) at 200 concurrent requests:
per-request attribution goes from 0.5% to 100%, while span nesting,
context propagation across awaits, and distinct trace IDs are identical
before and after.
Build and deploy fully‑managed AI agents and workflows
Website | Docs | Issues | Example projects | Feature requests | Public roadmap | Self-hosting
About Trigger.dev
Trigger.dev is the open-source platform for building AI workflows in TypeScript. Long-running tasks with retries, queues, observability, and elastic scaling.
The platform designed for building AI agents
Build AI agents using all the frameworks, services and LLMs you're used to, deploy them to Trigger.dev and get durable, long-running tasks with retries, queues, observability, and elastic scaling out of the box.
-
Long-running without timeouts: Execute your tasks with absolutely no timeouts, unlike AWS Lambda, Vercel, and other serverless platforms.
-
Durability, retries & queues: Build rock solid agents and AI applications using our durable tasks, retries, queues and idempotency.
-
True runtime freedom: Customize your deployed tasks with system packages – run browsers, Python scripts, FFmpeg and more.
-
Human-in-the-loop: Programmatically pause your tasks until a human can approve, reject or give feedback.
-
Realtime apps & streaming: Move your background jobs to the foreground by subscribing to runs or streaming AI responses to your app.
-
Observability & monitoring: Each run has full tracing and logs. Configure error alerts to catch bugs fast.
Key features:
- JavaScript and TypeScript SDK - Build background tasks using familiar programming models
- Long-running tasks - Handle resource-heavy tasks without timeouts
- Durable cron schedules - Create and attach recurring schedules of up to a year
- Trigger.dev Realtime - Trigger, subscribe to, and get real-time updates for runs, with LLM streaming support
- Build extensions - Hook directly into the build system and customize the build process. Run Python scripts, FFmpeg, browsers, and more.
- React hooks - Interact with the Trigger.dev API on your frontend using our React hooks package
- Batch triggering - Use batchTrigger() to initiate multiple runs of a task with custom payloads and options
- Structured inputs / outputs - Define precise data schemas for your tasks with runtime payload validation
- Waits - Add waits to your tasks to pause execution for a specified duration
- Preview branches - Create isolated environments for testing and development. Integrates with Vercel and git workflows
- Waitpoints - Add human-in-the-loop judgment at critical decision points without disrupting workflow
- Concurrency & queues - Set concurrency rules to manage how multiple tasks execute
- Multiple environments - Support for DEV, PREVIEW, STAGING, and PROD environments
- No infrastructure to manage - Auto-scaling infrastructure that eliminates timeouts and server management
- Automatic retries - If your task encounters an uncaught error, we automatically attempt to run it again
- Checkpointing - Tasks are inherently durable, thanks to our checkpointing feature
- Versioning - Atomic versioning allows you to deploy new versions without affecting running tasks
- Machines - Configure the number of vCPUs and GBs of RAM you want the task to use
- Observability & monitoring - Monitor every aspect of your tasks' performance with comprehensive logging and visualization tools
- Logging & tracing - Comprehensive logging and tracing for all your tasks
- Tags - Attach up to ten tags to each run, allowing you to filter via the dashboard, realtime, and the SDK
- Run metadata - Attach metadata to runs which updates as the run progresses and is available to use in your frontend for live updates
- Bulk actions - Perform actions on multiple runs simultaneously, including replaying and cancelling
- Real-time alerts - Choose your preferred notification method for run failures and deployments
Write tasks in your codebase
Create tasks where they belong: in your codebase. Version control, localhost, test and review like you're already used to.
import { task } from "@trigger.dev/sdk";
//1. You need to export each task
export const helloWorld = task({
//2. Use a unique id for each task
id: "hello-world",
//3. The run function is the main function of the task
run: async (payload: { message: string }) => {
//4. You can write code that runs for a long time here, there are no timeouts
console.log(payload.message);
},
});
Deployment
Use our SDK to write tasks in your codebase. There's no infrastructure to manage, your tasks automatically scale and connect to our cloud. Or you can always self-host.
Environments
We support Development, Staging, Preview, and Production environments, allowing you to test your tasks before deploying them to production.
Full visibility of every job run
View every task in every run so you can tell exactly what happened. We provide a full trace view of every task run so you can see what happened at every step.
Getting started
The quickest way to get started is to create an account and project in our web app, and follow the instructions in the onboarding. Build and deploy your first task in minutes.
Useful links:
- Quick start - get up and running in minutes
- How it works - understand how Trigger.dev works under the hood
- Guides and examples - walk-through guides and code examples for popular frameworks and use cases
Self-hosting
If you prefer to self-host Trigger.dev, you can follow our self-hosting guides:
- Docker self-hosting guide - use Docker Compose to spin up a Trigger.dev instance
- Kubernetes self-hosting guide - use our official Helm chart to deploy Trigger.dev to your Kubernetes cluster
Support and community
We have a large active community in our official Discord server for support, including a dedicated channel for self-hosting.
Development
To setup and develop locally or contribute to the open source project, follow our development guide.