Moving usage updates into the run engine to prevent inefficient &
additional incremental updates to the TaskRun table. Read/Modify/Write
pattern is safe inside of the run engine because of the run lock. We can
also now cap the usageDurationMs value from overflowing and causing an
error.
## Why?
This is preventing at least one update per TaskRun and instead updating
these values piggybacking on other updates.
## Aurora PostgreSQL Reader Consistency Notes
### TL;DR
Aurora readers share the same storage as the writer, but maintain
separate in-memory page caches. This means:
- **Storage is always consistent** - writes are synchronously committed
to shared storage
- **Page cache can lag** - typically <100ms, but can cause stale reads
if data is cached
### How It Works
1. Writer commits to shared storage (synchronous 4/6 quorum)
2. Writer sends cache invalidation messages to readers (asynchronous)
3. If reader has data in cache → returns cached (potentially stale)
value
4. If reader has cache miss → fetches from shared storage (always
current)
### Monitoring
```sql
SELECT server_id,
CASE WHEN session_id = 'MASTER_SESSION_ID' THEN 'Writer' ELSE 'Reader' END AS role,
replica_lag_in_msec
FROM aurora_replica_status();
```
## Summary
- Store the original user-provided idempotency key and scope alongside
the hash
- Expose `ctx.run.idempotencyKey` as the user-provided key (not the
hash)
- Add `ctx.run.idempotencyKeyScope` to show the scope ("run", "attempt",
or "global")
<img width="539" height="450" alt="CleanShot 2026-01-19 at 11 40 46"
src="https://github.com/user-attachments/assets/b6f42991-697e-4314-a164-aef77b8fd25c"
/>
## Problem
Idempotency keys were hashed (SHA-256) before storage, making debugging
difficult since users couldn't see the value they originally set or
search for runs by idempotency key.
## Solution
Attach metadata to the `String` object returned by
`idempotencyKeys.create()` using a Symbol, extract it in the SDK before
the API call, and store it in the database alongside the hash.
```typescript
const key = await idempotencyKeys.create("my-key", { scope: "global" });
await childTask.triggerAndWait(payload, { idempotencyKey: key });
// In child task:
ctx.run.idempotencyKey // "my-key" (previously showed the hash)
ctx.run.idempotencyKeyScope // "global"
```
Test plan
- Trigger task with idempotencyKeys.create() using different scopes (run, attempt, global)
- Verify ctx.run.idempotencyKey returns user-provided key
- Verify ctx.run.idempotencyKeyScope returns correct scope
- Verify PostgreSQL stores idempotencyKeyOptions JSON
- Verify ClickHouse receives idempotency_key_user and idempotency_key_scope via replication
---------
Co-authored-by: James Ritchie <james@trigger.dev>
Adds support for **debounced task runs** - when triggering a task with a
debounce key, subsequent triggers with the same key will reschedule the
existing delayed run instead of creating new runs. This continues until
no new triggers occur within the delay window.
## Usage
```typescript
await myTask.trigger({ userId: "123" }, {
debounce: {
key: "user-123-update",
delay: "5s",
mode: "leading", // default
}
});
```
- **key**: Scoped to the task identifier
- **delay**: How long to wait before executing (supports duration
strings like `"5s"`, `"1m"`)
- **mode**: Either `"leading"` or `"trailing"`. Leading debounce will
use the payload and options from the first run created with the debounce
key. Trailing will use payload and options from the last run.
### "trailing" mode overrides
When using `mode: "trailing"` with debounce, the following options are
updated from the **last** trigger:
- **`payload`** - The task input data
- **`metadata`** - Run metadata
- **`tags`** - Run tags (replaces existing tags)
- **`maxAttempts`** - Maximum retry attempts
- **`maxDuration`** - Maximum compute time
- **`machine`** - Machine preset (cpu/memory)
## Behavior
- **First run wins**: The first trigger creates the run, subsequent
triggers push its execution time later
- **Idempotency keys take precedence**: If both are specified,
idempotency is checked first
- **Max duration**: Configurable via `DEBOUNCE_MAX_DURATION_MS` env var
(default: 10 minutes)
Works with `triggerAndWait` - parent runs correctly block on the
debounced run.
New batch trigger system with larger payloads, streaming ingestion,
larger batch sizes, and a fair processing system.
This PR introduces a new `FairQueue` abstraction inspired by our own
`RunQueue` that enables multi-tenant fair queueing with concurrency
limits. The new `BatchQueue` is built on top of the `FairQueue`, and
handles processing Batch triggers in a fair manner with per-environment
concurrency limits defined per-org. Additionally, there is a global
concurrency limit to prevent the BatchQueue system from creating too
many runs too quickly, which can cause downstream issues.
For this new BatchQueue system we have a completely new batch trigger
creation and ingestion system. Previously this was a single endpoint
with a single JSON body that defined details about the batch as well as
all the items in the batch.
We're introducing a two-phase batch trigger ingestion system. In the
first phase, the BatchTaskRun record is created (and possibly rate
limited). The second phase is another endpoint that accepts an NDJSON
body with each line being a single item/run with payload and options.
At ingestion time all items are added to a queue, in order, and then
processed by the BatchQueue system.
## New batch trigger rate limits
This PR implements a new batch trigger specific rate limit, configured
on the `Organization.batchRateLimitConfig` column, and defaults using
these environment variables:
- `BATCH_RATE_LIMIT_REFILL_RATE` defaults to 10
- `BATCH_RATE_LIMIT_REFILL_INTERVAL` the duration interval, defaults to
`"10s"`
- `BATCH_RATE_LIMIT_MAX` defaults to 1200
This rate limiter is scoped to the environment ID and controls how many
runs can be submitted via batch triggers per interval. The SDK handles
the retrying side.
## Batch queue concurrency limits
The new column `Organization.batchQueueConcurrencyConfig` now defines an
org specific `processingConcurrency` value, with a backup of the env var
`BATCH_CONCURRENCY_LIMIT_DEFAULT` which defaults to 10. This controls
how many batch queue items are processed concurrently per environment.
There is also a global rate limit for the batch queue set via the
`BATCH_QUEUE_GLOBAL_RATE_LIMIT` which defaults to being disabled. If
set, the entire batch queue system won't process more than
`BATCH_QUEUE_GLOBAL_RATE_LIMIT` items per second. This allows
controlling the maximum number of runs created per second via batch
triggers.
## Batch trigger settings
- `STREAMING_BATCH_MAX_ITEMS` controls the maximum number of items in a
single batch
- `STREAMING_BATCH_ITEM_MAXIMUM_SIZE` controls the maximum size of each
item in a batch
- `BATCH_CONCURRENCY_DEFAULT_CONCURRENCY` controls the default
environment concurrency
- `BATCH_QUEUE_DRR_QUANTUM` how many credits each environment gets each
round for the DRR scheduler
- `BATCH_QUEUE_MAX_DEFICIT` the maximum deficit for the DRR scheduler
- `BATCH_QUEUE_CONSUMER_COUNT` how many queue consumers to run
- `BATCH_QUEUE_CONSUMER_INTERVAL_MS` how frequently they poll for items
in the queue
### Configuration Recommendations by Use Case
**High-throughput priority (fairness acceptable at 0.98+):**
```env
BATCH_QUEUE_DRR_QUANTUM=25
BATCH_QUEUE_MAX_DEFICIT=100
BATCH_QUEUE_CONSUMER_COUNT=10
BATCH_QUEUE_CONSUMER_INTERVAL_MS=50
BATCH_CONCURRENCY_DEFAULT_CONCURRENCY=25
```
**Strict fairness priority (throughput can be lower):**
```env
BATCH_QUEUE_DRR_QUANTUM=5
BATCH_QUEUE_MAX_DEFICIT=25
BATCH_QUEUE_CONSUMER_COUNT=3
BATCH_QUEUE_CONSUMER_INTERVAL_MS=100
BATCH_CONCURRENCY_DEFAULT_CONCURRENCY=5
```
This PR adds support for CLI deployments using the native build server.
**Background**
The deployment command currently does the following:
- bundles the code
- submits the build context to our external build provider and waits for
the build
- triggers deployment state transitions using the platform API
Upstream build provider outages cause issue with deployments,
potentially blocking deployments entirely. We recently introduced the
`--force-local-build` flag as a fallback to enable deployment without a
dependency on the upstream build provider, though it requires users to
have docker in their systems. This PR continues that work by providing a
remote build path which uses our own build server and does not rely on
the external provider.
**Changes in this PR**
Introduced the new `--native-build-server` flag, which does the
following:
- scans all files relevant for the Trigger deployment and evaluates
ignore rules
- packages it up in an archive and uploads it as a deployment artifact
- queues the deployment and triggers the build
- streams logs from the build server
This no longer relies on external build services. Also deployment state
transitions happen on the server-side, giving us more flexibility to
evolve the flow and schemas of related deployment API endpoints. In
general it gives us better control of the whole build and deployment
process. This path will eventually become the default.
The `--detach` flag is also new, allowing to trigger deployments without
waiting for the result.
The deployment artifacts are uploaded via pre-signed URLs to avoid
unnecessary load on the platform. The new `/artifacts` endpoint
generates the pre-signed URLs; size limits are enforced on s3. This
endpoint is deliberately generic, we could extend it in the future to
upload other artifacts client-side in a similar way, e.g., large payload
packets.
* feat(queues): add ability to override concurrency limit via API and dashboard
* Updates the modal layout and tweaks copy
* Improves the dropdown menu item
* Popover supports both Button and LinkButton
* Right align the columns and fix the dropdown menu item styles
* Organize imports,
* Fix spinner icon in dropdown menu
* Remove unused props
* Adds a tooltip to the Concurrency override badge
* Fixes console error with popover menu
* typo
* Fixes incorrect className
* Minimal buttons to view runs
---------
Co-authored-by: Eric Allam <eallam@icloud.com>
* Cancel run events which then propogate cancellation status to span ancestors
* WIP
* convert closing cached run spans to new system
* converted expired complete span event to new method
* move v3 over to new methods
* Convert getDetailedTraceSummary to use the new ancestor override stuff
* remove debug logs
* Don't return UNSPECIFIED task events in getRunEvents
* fix the call site for cancelling run event in v3
* Add changeset
* remove methods
* Change the color to indigo
* Pro tier pricing information now matches the marketing site
* Update the button styles to secondary
* WIP adding separate links to Parent and Root runs
* TextLink now supports optional shortcuts
* Adds shortcut keys to the root and parent links + the shortcut help panel
* Adds new icons for root and parent
* root friendlyId works
* Updates icons for jump to root and parent
* Copy tweak
* Improve how the Free tier shows no preview branches
* Improve the wording in the tooltip
* Align the x icon better
* Show price for additional preview branches
* Change the shortcut key
* Fixes button alignment
* Adds nested dependencies task hello-world
* Fixes typo “Cancelled”
* Removes taskIdentifier, not needed
* Removes unused taskIdentifier
* Mo-Stashed changes
* fix(core): prettyPrintingPacket will now do a structuredClone on non-circular references instead of outputting [Circular]
This also fixes an issue with replaying of runs that include
non-circular references
* feat(engine): Improve execution stalls troubleshooting, align dev and prod behavior, adding heartbeats.yield utility
* A few improvements via the 🐇 review
* Allow treating EXECUTION stalls as OOM errors, improve the error message, add more information to the docs, improve resource monitor and add it to the docs
* Add changeset
* Map new allowedMasterQueues → allowedWorkerQueues
* ClickHouse worker_queue on task runs
* Added the Region to the run inspector
* Pass a region in when triggering
* Added a changeset
* Added triggering regions docs
* Added region to the ctx
* Fix for backfiller masterQueue/workerQueue
* Update zod package to version 3.25.76 across all modules
Update the zod library from version 3.23.8 to 3.25.76 in multiple package files to ensure compatibility and take advantage of new features or bug fixes introduced in recent releases. Keeping all modules synchronized with the latest version of zod helps maintain consistency across the project and reduces potential compatibility issues.
- Modified zod version in apps/supervisor, webapp, and various internal packages.
- Updated zod references in pnpm-lock.yaml to reflect the new version.
- Ensure dependencies that rely on zod are using the updated version to avoid mismatches.
* Add changeset
* Add payload schema handling for task indexing
This change introduces support for handling payload schemas during task indexing. By incorporating the `payloadSchema` attribute into various components, we ensure that each task's payload structure is clearly defined and can be validated before processing.
- Updated the TaskManifest and task metadata structures to include an optional `payloadSchema` attribute. This addition allows for more robust validation and handling of task payloads.
- Enhanced several core modules to export and utilize the new `getSchemaToJsonSchema` function, providing easier conversion of schema types to JSON schemas.
- Modified the database schema to store the `payloadSchema` attribute, ensuring that the payload schema information is persisted.
- The change helps in maintaining consistency in data handling and improves the integrity of task data across the application.
* Refactor: Remove getSchemaToJsonSchema in favor of schemaToJsonSchema
The `getSchemaToJsonSchema` function was removed and replaced with `schemaToJsonSchema` across the codebase. This update introduces a new `@trigger.dev/schema-to-json` package to handle conversions of schema validation libraries to JSON Schema format, centralizing the functionality and improving maintainability.
- Removed `getSchemaToJsonSchema` exports and references.
- Added new schema conversion utility `@trigger.dev/schema-to-json`.
- Updated `trigger-sdk` package to utilize `schemaToJsonSchema` for payloads.
- Extensive testing coverage included to ensure conversion accuracy across various schema libraries including Zod, Yup, ArkType, Effect, and TypeBox.
- The update ensures consistent and reliable schema conversions, facilitating future enhancements and supporting additional schema libraries.
* Add support for Zod 4 in schema-to-json
This change enhances the schema-to-json package by adding support for Zod version 4, which introduces the native `toJsonSchema` method. This method facilitates a direct conversion of Zod schemas to JSON Schema format, improving performance and reducing reliance on the `zod-to-json-schema` library.
- Updated README to reflect Zod 4 support with native method and retained support for Zod 3 via existing library.
- Modified package.json to allow installation of both Zod 3 and 4 versions.
- Implemented handling for Zod 4 schemas in `src/index.ts` using their native method.
- Added a test case to verify the proper conversion of Zod 4 schemas to JSON Schema.
- Included a script for updating the package version based on the root package.json.
- Introduced a specific TypeScript config for source files.
* Revise schema-to-json for bundle safety and tests
The package @trigger.dev/schema-to-json has been revised to ensure bundle safety by removing direct dependencies on schema libraries such as Zod, Yup, and Effect. This change minimizes bundle size and enhances tree-shaking by allowing external conversion libraries to be utilized only at runtime if necessary. As a result, the README was updated to reflect this usage pattern.
- Introduced `initializeSchemaConverters` function to load necessary conversion libraries at runtime, keeping the base package slim.
- Adjusted test suite to initialize converters before tests, ensuring accurate testing of schema conversion capabilities.
- Updated `schemaToJsonSchema` function to dynamically check for availability of conversion libraries, improving flexibility without increasing the package size.
- Added configuration files for Vitest to support the new testing framework, reflecting the transition from previous test setups.
These enhancements ensure that only the schema libraries actively used in an application are bundled, optimizing performance and resource usage.
* Refine JSON Schema typing across packages
The changes introduce stricter typing for JSON Schema-related definitions, specifically replacing vague types with more precise ones, such as using `z.record(z.unknown())` instead of `z.any()` and `Record<string, unknown>` in place of `any`. This is part of an effort to better align with common practices and improve type safety in the packages.
- Updated the `payloadSchema` in several files to use `z.record(z.unknown())`, enhancing the type strictness and consistency with JSON Schema Draft 7 recommendations.
- Added `@types/json-schema` as a dependency, utilizing its definitions for improved type clarity and adherence to best practices in TypeScript.
- Modified various comments to explicitly mention JSON Schema Draft 7, ensuring developers are aware of the JSON Schema version being implemented.
- These adjustments are informed by research into how popular libraries and tools handle JSON Schema typing, aiming to integrate best practices for improved maintainability and interoperability.
* Add JSON Schema examples using various libraries
The change introduces extensive examples of using JSON Schemas in the 'references/hello-world' project within the 'trigger.dev' repository. These examples utilize libraries like Zod, Yup, and TypeBox for JSON Schema conversion and validation. The new examples demonstrate different use cases, including automatic conversion with schemaTask, manual schema provision, and schema conversion at build time. We also updated the dependencies in 'package.json' to include the necessary libraries for schema conversion and validation.
- Included examples of processing tasks with JSON Schema using libraries such as Zod, Yup, TypeBox, and ArkType.
- Showcased schema conversion techniques and type-safe JSON Schema creation.
- Updated 'package.json' to ensure all necessary dependencies for schema operations are available.
- Created illustrative scripts that cover task management from user processing to complex schema implementations.
* Refactor SDK to encapsulate schema-to-json package
The previous implementation required users to directly import and initialize functions from the `@trigger.dev/schema-to-json` package, which was not the intended user experience. This change refactors the SDK so that all necessary functions and types from `@trigger.dev/schema-to-json` are encapsulated within the `@trigger.dev/*` packages.
- The examples in `usage.ts` have been updated to clearly mark `@trigger.dev/schema-to-json` as an internal-only package.
- Re-export JSON Schema types and conversions in the SDK to improve developer experience (DX).
- Removed unnecessary direct dependencies on `@trigger.dev/schema-to-json` from user-facing code, ensuring initialization and conversion logic is handled internally.
- Replaced instances where users were required to manually perform schema conversions with automatic handling within the SDK for simplification and better maintainability.
* Add JSONSchema type for payloadSchema in tasks
The change was necessary to improve type safety by using a proper JSONSchema type definition instead of a generic Record<string, unknown>. This enhances the developer experience and ensures that task payloads conform to the JSON Schema Draft 7 specification. The JSONSchema type is now re-exported from the SDK for user convenience, hiding internal complexity and maintaining a seamless developer experience.
- Added JSONSchema type based on Draft 7 specification
- Updated task metadata and options to use JSONSchema type
- Hid internal schema conversion logic from users by re-exporting types from SDK
- Improved bundle safety and dependency management
* Add JSON schema testing and revert package dependencies
This commit introduces a comprehensive set of JSON schema testing within the monorepo, specifically adding a new test project in `references/json-schema-test`. This includes a variety of schema definitions and tasks utilizing multiple validation libraries to ensure robust type-checking and runtime validation.
Additionally, the dependency versions for `@effect/schema` have been adjusted from `^0.76.5` to `^0.75.5` to maintain compatibility across the project components. This ensures consistent behavior and compatibility with existing code bases without introducing breaking changes or unexpected behavior due to version discrepancies.
Key updates include:
- Added new test project with extensive schema validation tests.
- Ensured type safety across various task implementations.
- Reverted dependency versions to ensure compatibility.
- Created multiple schema tasks using libraries like Zod, Yup, and others for thorough testing.
* Refactor JSON Schema test files for clarity
Whitespace and formatting changes were applied across the `json-schema-test` reference project to enhance code readability and cohesion. This included removing unnecessary trailing spaces and ensuring consistent indentation patterns, which improves maintainability and readability by following the project's code style guidelines.
- Renamed JSONSchema type annotations to adhere to TypeScript conventions, ensuring that all schema definitions properly satisfy the JSONSchema interface.
- Restructured some object declarations for improved clarity, especially within complex schema definitions.
- These adjustments are crucial for better future maintainability, reducing potential developer errors when interacting with these test schemas.
* Fixed some stuff
* WIP
* we now convert schema to jsonSchema on the CLI side via the indexing
* Remove the json-schema-test reference project
* Improve schema-to-json peer deps and fix effect schema
* Explain the casting and match the version numbers
* Fixed a bunch more schema stuff
* Don't clean files that might be written to
* Don't use a custom version of vitest in the new package
* fix attw in schema-to-json
* Cleanup context and execution creation, cache stuff, add parent and root task run ids
* more efficient by using friendly IDs instead of doing joins
* metadata.root/parent now reference current run when run has no root/parent
* Adding changeset
* try to make test less flaky
* Clean imports
* Another attempt to fix the flaky test
* Fix usage by still passing durationMs and costInCents to the execution, just not the run.ctx
* Improve dynamic flush scheduler for otel data
The changes introduce a more flexible and adaptive dynamic flush scheduler to address production issues where the system wasn't flushing data fast enough, causing memory growth and crashes. This issue arises from the existing scheduler handling only a single flush at a time, limiting concurrency and failing to cope with the influx of logs.
- Added configuration options for setting minimum and maximum concurrency levels, maximum batch size, and memory pressure threshold. These parameters ensure that flush operations adjust dynamically based on workload and pressure.
- Implemented `pLimit` to facilitate concurrent flush operations, with adjustments made according to batch queue length and memory pressure.
- Metrics reporting improvements were added to monitor the dynamic behavior of the flush scheduler, aiding in identifying performance issues and optimizing the operation accordingly.
* Implement load shedding for TaskEvent records
This change introduces load shedding mechanisms to manage TaskEvent
records, particularly those of kind LOG, when the system experiences
high volumes and is unable to flush to the database in a timely
manner. The addition aims to prevent overwhelming the system and
ensure critical tasks are prioritized.
- Added configuration options for `loadSheddingThreshold` and
`loadSheddingEnabled` in multiple modules to activate load shedding.
- Introduced `isDroppableEvent` function to allow specific events to
be dropped when load shedding is enabled.
- Ensured metrics are updated to reflect dropped events and load
shedding status, providing visibility into system performance
during high load conditions.
- Updated loggers to inform about load shedding state changes,
ensuring timely awareness of load management activities.
* Fix undefined 'queuePressure' variable in DynamicFlushScheduler
The 'queuePressure' variable was being used without being defined
in the DynamicFlushScheduler class, causing potential runtime
errors. This commit adds the missing definition and ensures that
the variable is correctly calculated based on the 'totalQueuedItems'
and 'memoryPressureThreshold'.
- Addressed code inconsistencies and improved formatting.
- Defined 'queuePressure' in the 'adjustConcurrency' method
to prevent potential undefined errors.
- Enhanced readability by maintaining consistent spacing and
format across the file, contributing to the stability and
maintainability of the code.
- Adjusted batch size logic based on the newly defined 'queuePressure'
variable.
* Refactor concurrency adjustment logic in scheduler
The concurrency adjustment logic in the dynamic flush scheduler has been refactored to improve clarity and maintainability. This change moves the calculation of pressure metrics outside of the conditional blocks to ensure they are always determined prior to decision-making.
- The queue pressure and time since last flush calculations were moved up in the code to be independent of the 'backOff' condition.
- This refactor sets up the groundwork for more reliable concurrency scaling and better performance monitoring capabilities. The overall logic of adjusting concurrency based on system pressure metrics remains unchanged.
This adjustment addresses ongoing issues with the scheduler that were not resolved by previous changes.
* Some tweaks
* Queue in run table and filtering
* Debounce the filter changes
* Remove console log
* Added machine filtering
* Added version filtering
* Filter by version in the db
* Removed duplicate classes
* Version filtering hasFilters consistency
* Added queues and machines to the bulk action summary
* runs.list filtering for queue and machine
* Fix for machine errors
* WIP
* Make release concurrency system extremely simple, everything just releases all the time
* update the deadlock detection to use the new lockedQueueReleaseConcurrencyOnWaitpoint column
* WIP new release concurrency system
* Remove releaseConcurrency and releaseConcurrencyOnWaitpoint
Also removed deadlock detection, and added environment burst concurrency
* Added new DEQUEUED status
Cleaned up the API run statuses, including now detecting new clients and not breaking older clients by adding an API version header to all requests
* Introduce the new "current dequeued concurrency set"
* Remove QUEUED_EXECUTING because we no longer "eagerly" release before checkpointing
* Remove waitpoint test for QUEUED_EXECUTING
* Add isWaiting
* Add changeset
* Use createdAt for ordering realtime runs instead of number
* Clarify the envCurrentDequeuedKey usage
* mock the db.server file to fix the tests
* Updated changset "EXECUTED" -> "EXECUTING"
---------
Co-authored-by: Matt Aitken <matt@mattaitken.com>
* Improve TaskRunProcess health detection so we don't try and reuse an unhealthy process
This was happening after the process was killed internally, like by an OOM error
* Add changeset
* v4: current concurrency sweeper
* Fix webapp tests
* Ensure only a single instance performs concurrency sweeping by using redis-worker cron jobs
* Improved the mark phase
* Ensure cron jobs get rescheduled even if the handler throws an error
* Better property names
* Fix realtime re-subscribing stale data issue
Fixes an issue with realtime when re-subscribing to a run, that would temporarily display stale data and the changes. Now when re-subscribing to a run only the latest changes will be vended
* removed logs
* Add createdAt filter to realtime subscribing with tags
* Filter realtime colums and expose ability to skip some columns
* Add sharding support for electric
* Use unkey cache for the created at filter caching
* Remove 2 unused indexes on TaskRun
* Run list now filters by a single runtime environment
* Remove project ID indexes
* Use clickhouse in task list aggregation queries instead of pg (keep pg for self-hosters)
* WIP clickhouse powered runs list
stuff
* Improve the query to get the latest tasks for the task list presenter
* Update the usage task list to use clickhouse
* Implement next runs list powered by clickhouse
* Add new index for TaskRun for the runs list, by environment ID
* Add runTags gin index
* Handle possibly malicious inputs
* Ignore claude settings
* Better handling not finding an environment on the schedule page
* Use ms since epoch in test, not seconds
* Remove unused function
* Fix test
* Use an env var for the realtime maximum createdAt filter duration (defaults to 1 day)
* Fixed the query builder to correct the group by / order by order
* Make sure runs.list still works
* Create small-birds-arrive.md
* remove registry proxy
* remove --self-hosted flag
* automatically set network build flag
* update syncEnvVars debug log
* improve switch command
* always display deploy errors if they exist
* fix stuck deploy command after finalize error
* webapp-driven deploys, multi-platform support, lots of fixes
* add worker deployment migration
* rename image platform env var
* only try to sync parent env vars for preview deployments
* add KEEP_TMP_DIRS
* supervisor: docker api version lock, auth, multi-platform
* set image ref on create, validate digest
* use metadata for digest, fix local multi-platform builds
* print git meta branch before commit
* improve push and load flag handling
* make runs after local builds compatible with load and push
* small improvement for platform overrides
* add image platform to dequeued message
* remove deprecated init request body fields
* fix fail deployment id param
* remove build debug logs
* pass report merge with no tests
* structured run debug logs
* add required env var for tests
* should not be an error log
* add changeset
* Prefer branch variables, but load preview envs too
* WIP with vercel preview env var syncing
* Return success if either child or parent env vars are synced
* Removed log of env vars
* Remove another log
* Initial preview migrations
* Modified the staging endpoint to create preview environments
* Added isBranchableEnvironment to RuntimeEnvironment
* Staging = yellow Preview = orange
* Changed the env sort order
* Set isBranchableEnvironment correctly. Create preview for new projects
* Very basic branch menu
* Creating branches from the dashboard
* Fix for string icons on project delete page
* Don’t show branch API keys
* WIP on the manage branches page
* RuntimeEnvironment added projectId index
* Only create the parentEnvironmentId column if it doesn’t exist already
* Improved the limit wording
* Add search to the branch list
* contains in both places
* Many style improvements
* Branch dropdown and v4 badge
* Arching/unarchive branches working in the dashboard
* Tidied imports
* Change preview slug from `prev` to `preview`
* Use correct color for side menu preview branch icon
* Upsert the branch and use the shortcode as a unique constraint
* Upserting working with nice messages in the dashboard
* Better errors when upserting branches
* Button shortcut, don’t allow event to propagate
* Better duplicate error message
* Filter out archived branches from the env selector
* Archiving/creating tweaked some more
* Add an archived banner to the app, fixes for archived branches and upsells
* Fixed pagination
* Disable editing schedules, pausing queues, testing tasks
* Don’t allow replaying if the env is archived
* When deploying detect the correct environment
* Get the projectClient when there’s a branch
* createGitMeta function, most code from the vercel CLI repo
* Deploy, getting the correct environment client
* Added git column to WorkerDeployment
* Add GitMeta to core schemas
* Create branch when deploying
* WIP on branch support in the API
* Delete old createTaskRunAttempt fn
* apiAuth remove export from internal functions
* Rename env var to “TRIGGER_PREVIEW_BRANCH”
* Add TRIGGER_PREVIEW_BRANCH to resolved env vars for runs
* First preview deploy and run working
* Set the preview branch in the main SDK
* Added git links to the preview branches table
* Better errors when replaying/testing archived branches
* Don’t dequeue archived environments
* Env var resolution with parent environment
* Hello world default machine small-2x to save my memory
* Fix for more env var functions
* Only return non-archived envs
* Switch to controlled state for the checkboxes
* Uncheck everything when PREVIEW is checked
* WIP on branch UI
* Show the preview branch label on the env vars list
* Fix for overriding env vars
* Adding preview branch env vars working
* Progress on new env vars
* Only allow selecting a single branch
* Layout fix when there are errors
* Set the defaultValue so there are some fields
* Conform fix for team invite page
* Archived environments don’t run scheduled tasks
* Added Git data to deployments
* Added git data to the deployment inspector
* Don’t allow upserting schedules when archived
* Deduplicate and blacklist some env vars
* Fix for wrong conform function being used
* Show a better error if all vars were blacklisted
* Added environment variable search (by key and value)
* Improved preview branch icon
* Replay now supports branches
* Schedule page render branches properly
* Show the env icon in bottom-left of the test page
* When editing older schedules (that have multi-env) show preview branches correctly
* Fix for incorrect disallowed branch name character
* Extract and improve the directory verification code
* WIP for CLI preview archive command
* Improved the preview branch action buttons
* Redirect to the project if we don’t find a matching env
* Archiving branch via the CLI working
* Fix for archiving branches
* Public access token test task
* JWTs working are with preview branches
* Add branch and git data to the Run ctx
* Updated GitMeta functions to work in CI
* Added pullRequestState
* Archive when deploying if the PR is closed/merged
* Fix for the changesets guide
* Fix for CLI dev bug introduced
* CLI promote now supports preview branches
* Add PR title. Reordered them and added tooltips
* syncEnvVars working with branches
* Added preview branch support to syncVercelEnvVars()
* Detect the branch from Vercel env var (set during build)
* Allow passing a branch in
* Use process.env.VERCEL_TOKEN as well… this used in Vercel CI
* Temp delete
* Improved regenerate api key modal
* Added Accordion component (with styles)
* Redesigned the API keys page
* Revert "Temp delete"
This reverts commit 177b92cd935a6161456bde65d01294e23ecfd47f.
* Changeset
* Fixed docs link
* The new branch panel closes when a branch is created
* Update apps/webapp/app/services/upsertBranch.server.ts
Co-authored-by: coderabbitai[bot] <136622811+coderabbitai[bot]@users.noreply.github.com>
* Removed findUniques from WorkerGroupTokenService
* Made the parentEnvironmentId migrations safe
* Latest lockfile
* Update packages/cli-v3/src/commands/workers/build.ts
Co-authored-by: coderabbitai[bot] <136622811+coderabbitai[bot]@users.noreply.github.com>
* Move isValidGitBranchName to a separate file
* Move the sanitize fn too
* removeBlacklistedVariables moved to a separate file
* Moved deduplicateVariableArray to a separate file…
* Fix broken sanitizeBranchName import
* Another import fix…
* Improved blacklisted error message
* SImplified migration to use `ADD COLUMN IF NOT EXISTS "parentEnvironmentId" TEXT`
---------
Co-authored-by: coderabbitai[bot] <136622811+coderabbitai[bot]@users.noreply.github.com>
* WIP clickhouse package with test containers setup
* More clickhouse client setup now with otel and real tests, and the v1 of raw run events
* Add some additional columns to raw_run_events_v1
* WIP runs dashboard service
* Create a new run engine event bus event for the runs dashboard to hook into
* Track run events in the run engine
* make sure engine v1 runs get synced to CH
* Update the attemptNumber of v3 task runs
* Restructure the run events to be more sparse
* emit more stuff
* Setup replication package
* scaffold the replication package
* replication wip
* resolve conflicts
* more replication stuff
* Add ability to drop the replication slot completely on teardown
* Use the new single replacingmergetree task events table for replication
* get it working
* insert payloads into their own table only on insert and then join
* prepare for using clickhouse cloud and now running ch migrations during boot in the entrypoint.sh
* Handover WIP and tests
* Testing the replication service
* Remove the runs dashboard stuff that we aren't using anymore
* Added a test for large payloads
* hacky typecheck fix
* Fix new internal package typecheck issues and start adding telemetry to the replication service
* tracing over spans, some other improvements
* Improvements to the runs replication service, now ready for testing
* Some fixes and cleanups
* Don't need this code anymore
* move transaction types into the runs replication service
* only send spans where there are transaction events
* A couple of suggested tweaks
* Initial commit with a plan for what we’re going to do
* Some initial types and improved plan
* Add Waitpoint resolver
* Add resolver + status index
* Remove type + status index
* Only drop if exists
* Remove type index
* Update waitpoint list presenter to use resolver
* Added resolver to the engine
* Made the existing waitpoint list presenter more flexible
* Initial implentation ofr wait.forHttpCallback()
* Added the callback endpoint (no API rate limit)
* schema version
* Added jsdocs, removed schema version because of errors
* Show callback URL if it’s set
* Dashboard pages and panels
* Remove todos
* Added temporary icon
* Added a blank state
* Some tweaks and added a Replicate example
* Implement unwrap() for httpCallback
* Added unwrap to wait.forToken() as well
* Improved jsdocs
* Added docs
* Added unwrap to the token docs
* Show a dash if there are no tags
Co-authored-by: coderabbitai[bot] <136622811+coderabbitai[bot]@users.noreply.github.com>
* Make the timeout error safer
* Fixed migrations… should use id desc not createdAt desc
* Fixed page title
* Fixed migration so it only adds them if they don’t exist. This allows us to manuall run in cloud first
* Respect the max content length by getting the length of the body
* Added more docs details about the callback format
* Remove code comment
* Improved the error
* Added a hash to the HTTP callback URLs
* Add the apiKey to the API input type to fix TS error
* Return the error responses. They were being caught and not preserved
* The content-length header is required. Deal with an empty body
* Removed unused types
* Added some new span icons
* Reworked http callback to be a create call then just use wait.forToken()
* Added a changeset
* Updated the docs
* Updated the wait overview docs
* Simplify to just a call
* WIP stripping right back to waitpoints just having a URL associated with them…
* More deletions
* Remove missing icon
* Updated the changeset
* Add URL to the token return types
* Remove wait for http callback page
* Updated docs
* More tidying
* Type and import fix
* Remove unused import
* Some type fixes for the retrieve
---------
Co-authored-by: coderabbitai[bot] <136622811+coderabbitai[bot]@users.noreply.github.com>
Eric Allam