(`OBJECT_STORE_BASE_URL`) and a named protocol provider
(`OBJECT_STORE_DEFAULT_PROTOCOL=s3`), chat.agent session snapshot writes
landed in the named provider but reads fell through to the default — so
the recovery boot couldn't find the snapshot it had just written.
After a mid-stream cancel, the missing snapshot triggered a fallback
replay path that dropped the user's follow-up message, leaving the chat
stuck in `submitted` indefinitely.
Fix:
- New `/api/v1/sessions/:id/snapshot-url` route handles PUT + GET
symmetrically — both prefix unprefixed keys with
`OBJECT_STORE_DEFAULT_PROTOCOL` so they always round-trip through the
same store.
- `Session.chatSnapshotStoragePath` persists the resolved URI on first
write so future protocol changes don't strand existing snapshots.
Reads prefer the stored URI and fall back to the computed default for
pre-column sessions.
- SDK calls `createChatSnapshotUploadUrl` / `getChatSnapshotUrl`; the
generic v1/v2 packets endpoints are unchanged.
## Test plan
- [x] Configure local with two providers (R2 default + MinIO `s3` named)
and `OBJECT_STORE_DEFAULT_PROTOCOL=s3`.
- [x] Reproduce hang: send a message, cancel mid-stream, send another —
without the fix it hangs in `submitted`; with the fix it streams.
- [x] Snapshot lands in the `s3`-protocol bucket and
`Session.chatSnapshotStoragePath` is set after first write.
- [x] SDK unit tests pass; webapp typecheck passes.
Adds Sessions, a durable, run-aware stream primitive that scopes
session.in / session.out records to a session (not a single run).
Records survive run boundaries; reconnect-from-last-event-id is built in.
Server foundation:
- New /realtime/v1/sessions/:session/:io/append + /records routes
- sessionRunManager + sessionsRepository + clickhouseSessionsRepository
- mintRunToken for short-lived per-session tokens
- s2Append retry-with-backoff + undici cause diagnostics
- /api/v[12]/packets/* exempt from customer rate limits
- BackgroundWorker schema gains taskKind enum (TASK, AGENT, SCHEDULED)
- TaskRun.taskKind column + clickhouse 029_add_task_kind_to_task_runs_v2
Core types:
- new sessionStreams, inputStreams, realtimeStreams packages in @trigger.dev/core
- session-streams-api / realtime-streams-api surface
Sessions dashboard UI (the primitive's own viewer):
- /sessions index + detail routes
- SessionsTable, SessionFilters, SessionStatus, CloseSessionDialog
- AGENT/SCHEDULED filter in RunFilters + TaskTriggerSource
Includes the sessions-primitive changeset.
Deployments are affected by general API rate limits, this is just a
quick fix by whitelisting the deployment related endpoints. In a follow
up PR we'll add a separate rate limiter for this group of endpoints.
* Initial commit with a plan for what we’re going to do
* Some initial types and improved plan
* Add Waitpoint resolver
* Add resolver + status index
* Remove type + status index
* Only drop if exists
* Remove type index
* Update waitpoint list presenter to use resolver
* Added resolver to the engine
* Made the existing waitpoint list presenter more flexible
* Initial implentation ofr wait.forHttpCallback()
* Added the callback endpoint (no API rate limit)
* schema version
* Added jsdocs, removed schema version because of errors
* Show callback URL if it’s set
* Dashboard pages and panels
* Remove todos
* Added temporary icon
* Added a blank state
* Some tweaks and added a Replicate example
* Implement unwrap() for httpCallback
* Added unwrap to wait.forToken() as well
* Improved jsdocs
* Added docs
* Added unwrap to the token docs
* Show a dash if there are no tags
Co-authored-by: coderabbitai[bot] <136622811+coderabbitai[bot]@users.noreply.github.com>
* Make the timeout error safer
* Fixed migrations… should use id desc not createdAt desc
* Fixed page title
* Fixed migration so it only adds them if they don’t exist. This allows us to manuall run in cloud first
* Respect the max content length by getting the length of the body
* Added more docs details about the callback format
* Remove code comment
* Improved the error
* Added a hash to the HTTP callback URLs
* Add the apiKey to the API input type to fix TS error
* Return the error responses. They were being caught and not preserved
* The content-length header is required. Deal with an empty body
* Removed unused types
* Added some new span icons
* Reworked http callback to be a create call then just use wait.forToken()
* Added a changeset
* Updated the docs
* Updated the wait overview docs
* Simplify to just a call
* WIP stripping right back to waitpoints just having a URL associated with them…
* More deletions
* Remove missing icon
* Updated the changeset
* Add URL to the token return types
* Remove wait for http callback page
* Updated docs
* More tidying
* Type and import fix
* Remove unused import
* Some type fixes for the retrieve
---------
Co-authored-by: coderabbitai[bot] <136622811+coderabbitai[bot]@users.noreply.github.com>
* Remove unnecessary disabled org check
* Add support for separate redis servers for caching, rate limiter, marqs, and the pub/sub stuff (they all default to the previously used env vars for redis)
* Stop using Response errors when validating API Keys, instead introduce a new "Result" type that has success and failure conditions. Adding in a way to progressively adopt because this touches everything.
* Make sure authenticateApiRequestWithFailure always returns a result
* Denormalize run tags, increase character limit to 128
* WIP realtime subscribing to runs
* extracted the stream stuff into core, made it more reusable
* WIP tags
* Remove tags for now because it’s not support in electric
* Support async iterables, readable stream, and callback style subscription styles
* Remove tags streaming endpoint
* Add realtime rate limits and scope them to the /realtime path
* WIP rate limt per org
* Introduce per org rate limits
* WIP JWT auth
* Move migrations into new internal db package
* Resolve pnpm lock file
* Authenticating to the realtime API with JWTs are working
* realtime in the client
* Created react-hooks package and starting to move stuff in there
* Improve types for hooks
* schema tasks
* Added useBatch hook
* build uploadthing/fal demo and change how run metadata is synced to the server
* tweaks
* WIL realtime concurrency tracking
* Implement test for realtime client using testcontainers
also updated electric to latest version
* Allow customizing the expiration time of the automatic JWT created after triggering a task
* Add support for subscribing to run tags
* Improve auth types and API
* finalize the realtime API
* Fixed some example stuff
* Allow up to 10 run tags
* Remove core from docker-provider tsconfig paths to prevent it from being typechecked
* do the same for the kubernetes provider
* Fixing some typecheck errors
* Fix webapp type errors
* Update @trigger.dev/platform to 1.0.13
* Fix attw error
* Remove from/to in subscribeToRuns query params
* Add tests for the rate limit middleware and add custom JWT rate limits
* turn off webapp test parallelism
* Finish renaming jwt -> publicAccessToken and automatically give the JWT read access to the tags when using trigger
* Add changeset
* Attempt to fix unit tests in CI
* Skip running the auth rate limit middleware tests for now
* Try a beefier machine
* Try and run webapp tests separately
* Setup env vars
* Make sliding window test more reliabile
* v3: cancel subtasks when parent task runs are cancelled
* v3: recover from server rate limiting errors in a more reliable way
- Changing from sliding window to token bucket in the API rate limiter, to help smooth out traffic
- Adding spans to the API Client core & SDK functions
- Added waiting spans when retrying in the API Client
- Retrying in the API Client now respects the x-ratelimit-reset
- Retrying ApiError’s in tasks now respects the x-ratelimit-reset
- Added AbortTaskRunError that when thrown will stop retries
- Added idempotency keys SDK functions and automatically injecting the run ID when inside a task
- Added the ability to configure ApiRequestOptions (retries only for now) globally and on specific calls
- Implement the maxAttempts TaskRunOption (it wasn’t doing anything before)
* Adding some docs about the request options
* Fix type error
* Remove context propagation through graphile jobs
* Remove logger
* only select a subset of task run columns
* limit columns selected in batchTrigger as well
* added idempotency doc
* allow scoped idempotency keys, and fixed an issue with the unique index on BatchTaskRun and TaskRun
* Removed old cancel task run children code
* Starting to measure wall time and cpu time in the workers, and reporting that via otel and to completed task run attempts
* Move usage tracking outside of the executor
* WIP prod usage tracking
* WIP
* WIP custom fetch to openmeter
* Create a usage client
* WIP
* WIP
* Implement new machine preset stuff and send usage reports to OpenMeter from webapp
* WIP
* Expose usage info to the client
* Add usage and cost to TaskEvent
* Add ability to globally configure the task machine preset
* Report start run usage
* Change the machine docs to use presets
* setExpirationTime to 24h
* Removed logs
* Update machines.mdx
* Removed console.logs
* Handle revalidating JWT tokens
* Couple tweaks
---------
Co-authored-by: Matt Aitken <matt@mattaitken.com>
* Added maximumScheduleInstancesLimit column to Org, default to 20
* Docs on the schedule limits and improved soft-limit communication
* Added limit info to the schedules list page
* Created a task that creates schedules, useful for testing
* Make deduplicationKey required when creating/updating a schedule using the SDK
* New schedule button shows an alert if you’re over the limit
* Added timezone to the form and db
* WIP on the timezone dropdown for the create/edit schedule form
* Use the new filter search for timezones
* Made the timezone dropdown faster by fixing the virtualization
* The preview table is working and added a nice message about daylight savings
* Created a page where you can view the full list of timezones
The URL is included in the error message if you send an invalid time using the SDK
* Creating tasks with the timezone
* Added timezone support the the scheduler and the schedules list
* Added timezone support to more of the schedules UI
* The timezone comes through to scheduled runs with nice JSDocs
* Allow setting the timezone from the SDK
* Always have a timezone on a schedule
* Updated jsdocs
* Updated catalog example
* Changed the column to be a string, not null. Added the timezone across the SDK
* API endpoint for getting the timezones
* Added an SDK function to get the list of timezones
* Added timezones to the docs
* Changeset: Added timezone support to schedules
* Added support for testing timezone
* Tidied up imports
* Imports
* Imports
* Update limits.mdx
* Fixed a couple type issues and use the already exported zodfetch
---------
Co-authored-by: Eric Allam <eallam@icloud.com>
* Easier to create a rate limiter, use it in the ApiRateLimiter. Upgraded the Upstash package
* Always prefix any rate limiter in Redis with “ratelimit:”
* By default log when the rate limit is hit
* Added rate limiting to IngestSendEvent
* Log out the EventRecord id
* Increase events.deliverScheduled attempts
* INGEST_EVENT_RATE_LIMIT_MAX is optional
* Removed old API rate limit code
* IngestSendEvent rate limiter is optional. Moved outside of the DB transaction
* Log a message out when the rate limiter is created
* Return undefined if the rate limit has been crossed
* Fix: API rate limit error has the correct seconds until reset
* When a v2 run hits the rate limit, reschedule using the reset timestamp
* Still throw AutoYieldRateLimitErrors
* Reschedule runs from the rate limit
* The stress test timeout should be inside the task
* If the rate limit error is thrown, don’t retry the API request
* Trying to use the @upstash/ratelimit package with ioredis…
* WIP using the redis package instead
* Revert the action back
* Removed redis
* Started refactoring
* SQS setting for the poll interval. Set the default queue reading to be slower
* API rate limiter as Express middleware
* Organise imports
* Fixed spelling mistake “limitter”
* No authorization header response is problem+json
Eric Allam