Commit Graph

6821 Commits

Author SHA1 Message Date
Matt Aitken cd9dec6506 Added virtual column support 2025-12-22 21:08:41 +00:00
Matt Aitken 426fec5819 Allow querying just using org id, project and env are optional 2025-12-22 21:08:41 +00:00
Matt Aitken 8ca6cecbe9 Allow mapping values to user friendly names 2025-12-22 21:08:41 +00:00
Matt Aitken 66992f33f8 Changed some HogQL reference to TSQL 2025-12-22 21:08:41 +00:00
Matt Aitken a680ca7d13 Unneeded ts-expects 2025-12-22 21:08:41 +00:00
Matt Aitken 957defe32d Lints now based on possible tables, columns and values 2025-12-22 21:08:41 +00:00
Matt Aitken 3594727488 Fix for test 2025-12-22 21:08:41 +00:00
Matt Aitken 1d4a8a64dc Fix for TS compilation errors in the web app 2025-12-22 21:08:41 +00:00
Matt Aitken 24164559e4 Fix for codemirror versions 2025-12-22 21:08:41 +00:00
Matt Aitken 3940b8ac92 Storybook is available only to admins (but now when deployed) 2025-12-22 21:08:41 +00:00
Matt Aitken 05cc682b72 Type fixes 2025-12-22 21:08:41 +00:00
Matt Aitken 06127d5bf6 TRQL editor WIP 2025-12-22 21:08:41 +00:00
Matt Aitken 0312439f46 Formatting 2025-12-22 21:08:41 +00:00
Matt Aitken 34dd68185e Added column/table mapping 2025-12-22 21:08:41 +00:00
Matt Aitken 3985d6e0c1 E2E clickhouse query tests using tsql 2025-12-22 21:08:41 +00:00
Matt Aitken 4ef27fe063 Added some specific security tests 2025-12-22 21:08:41 +00:00
Matt Aitken 3b7d0ae28d Actual query function 2025-12-22 21:08:41 +00:00
Matt Aitken 8809e1be79 Latest lockfile 2025-12-22 21:08:41 +00:00
Matt Aitken aade98d11b Fix types 2025-12-22 21:08:41 +00:00
Matt Aitken 1d99e8a85d Port of PostHog AST to CH 2025-12-22 21:08:41 +00:00
Matt Aitken 06556ede59 Made the tests cleaner 2025-12-22 21:08:41 +00:00
Matt Aitken cee635ac92 Simplified test 2025-12-22 21:08:41 +00:00
Matt Aitken 0295082dca Fixes and test improvements 2025-12-22 21:08:41 +00:00
Matt Aitken ca45a6e359 Casting 2025-12-22 21:08:41 +00:00
Matt Aitken 45b123d25e WIP adding strong return type to visitor… 2025-12-22 21:08:41 +00:00
Matt Aitken 497210830f Added stronger types to parser.ts 2025-12-22 21:08:41 +00:00
Matt Aitken e4ffbf4935 Adding stronger types 2025-12-22 21:08:41 +00:00
Matt Aitken fa7db2c965 WIP fixing parser 2025-12-22 21:08:41 +00:00
Matt Aitken d6a3845f6a Incomplete port of PostHog Python code 2025-12-22 21:08:41 +00:00
Matt Aitken b50c6321da Custom intermediate format 2025-12-22 21:08:41 +00:00
Matt Aitken a9f8dbfb8d Latest lockfile 2025-12-22 21:08:41 +00:00
Matt Aitken d2748ada4a Converting TSQL to CH draft 1 2025-12-22 21:08:41 +00:00
Matt Aitken fbc6a3c519 First set of tests working 2025-12-22 21:08:41 +00:00
Matt Aitken bcfa15361d First generated parser 2025-12-22 21:08:41 +00:00
Matt Aitken f6b2839c28 Fix for improper casing and added gitignore 2025-12-22 21:08:41 +00:00
Matt Aitken b37a969e18 Use antlr4ts 2025-12-22 21:08:41 +00:00
Matt Aitken 459a179ef3 TSQL initial commit 2025-12-22 21:08:41 +00:00
Oskar Otwinowski acc10e847c chore(docs): add more cost saving tips (#2806)
Co-authored-by: Eric Allam <eallam@icloud.com>
2025-12-22 15:36:54 +00:00
Eric Allam f1a83cffc4 chore(docs): debounce options (#2797) 2025-12-22 15:35:51 +00:00
Eric Allam 61fee91830 chore(docs): upgrade for new batch trigger limits and functionality (#2787) 2025-12-22 15:35:43 +00:00
github-actions[bot] caa40ce925 chore: release v4.3.1 (#2788)
🚀 Publish Trigger.dev Docker / typecheck (push) Failing after 0s
🚀 Publish Trigger.dev Docker / publish-webapp (push) Has been skipped
🚀 Publish Trigger.dev Docker / publish-worker (push) Has been skipped
🚀 Publish Trigger.dev Docker / publish-worker-v4 (push) Has been skipped
🚀 Publish Trigger.dev Docker / units (push) Failing after 4s
This PR was opened by the [Changesets
release](https://github.com/changesets/action) GitHub action. When
you're ready to do a release, you can merge this and publish to npm
yourself or [setup this action to publish
automatically](https://github.com/changesets/action#with-publishing). If
you're not ready to do a release yet, that's fine, whenever you add more
changesets to main, this PR will be updated.


# Releases
## @trigger.dev/build@4.3.1

### Patch Changes

-   Updated dependencies:
    -   `@trigger.dev/core@4.3.1`

## trigger.dev@4.3.1

### Patch Changes

-   Updated dependencies:
    -   `@trigger.dev/core@4.3.1`
    -   `@trigger.dev/build@4.3.1`
    -   `@trigger.dev/schema-to-json@4.3.1`

## @trigger.dev/core@4.3.1

### Patch Changes

- Added support for idempotency reset
([#2777](https://github.com/triggerdotdev/trigger.dev/pull/2777))

## @trigger.dev/python@4.3.1

### Patch Changes

-   Updated dependencies:
    -   `@trigger.dev/sdk@4.3.1`
    -   `@trigger.dev/core@4.3.1`
    -   `@trigger.dev/build@4.3.1`

## @trigger.dev/react-hooks@4.3.1

### Patch Changes

-   Updated dependencies:
    -   `@trigger.dev/core@4.3.1`

## @trigger.dev/redis-worker@4.3.1

### Patch Changes

-   Updated dependencies:
    -   `@trigger.dev/core@4.3.1`

## @trigger.dev/rsc@4.3.1

### Patch Changes

-   Updated dependencies:
    -   `@trigger.dev/core@4.3.1`

## @trigger.dev/schema-to-json@4.3.1

### Patch Changes

-   Updated dependencies:
    -   `@trigger.dev/core@4.3.1`

## @trigger.dev/sdk@4.3.1

### Patch Changes

- feat: Support for new batch trigger system
([#2779](https://github.com/triggerdotdev/trigger.dev/pull/2779))
- feat(sdk): Support debouncing runs when triggering with new debounce
options
([#2794](https://github.com/triggerdotdev/trigger.dev/pull/2794))
- Added support for idempotency reset
([#2777](https://github.com/triggerdotdev/trigger.dev/pull/2777))
-   Updated dependencies:
    -   `@trigger.dev/core@4.3.1`

---------

Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>
@trigger.dev/build@4.3.1 @trigger.dev/core@4.3.1 @trigger.dev/python@4.3.1 @trigger.dev/react-hooks@4.3.1 @trigger.dev/redis-worker@4.3.1 @trigger.dev/rsc@4.3.1 @trigger.dev/schema-to-json@4.3.1 @trigger.dev/sdk@4.3.1 trigger.dev@4.3.1 v.docker.4.3.1
2025-12-22 15:29:18 +00:00
Dan ba9b0e17c1 Added Claude Agent SDK guide and examples (#2803) 2025-12-19 17:10:32 +00:00
Oskar Otwinowski 469808cf09 fix(webapp): Make spans fluid again while task is executing (#2801)
Closes #<issue>

##  Checklist

- [x] I have followed every step in the [contributing
guide](https://github.com/triggerdotdev/trigger.dev/blob/main/CONTRIBUTING.md)
- [x] The PR title follows the convention.
- [x] I ran and tested the code works

---

## Testing

Tested on executing tasks and on moving between tasks

---

## Changelog

Small UI fix for spans on the task runs page

---

## Screenshots



https://github.com/user-attachments/assets/72095b03-c74f-4472-afde-9c63e6e7c224



💯
2025-12-19 15:45:54 +01:00
Mihai Popescu 7574c69c2d feat(webapp): Add support for resetting idempotency keys (#2777)
Add support for resetting idempotency keys both from ui and sdk

##  Checklist

- [x] I have followed every step in the [contributing
guide](https://github.com/triggerdotdev/trigger.dev/blob/main/CONTRIBUTING.md)
- [x] The PR title follows the convention.
- [x] I ran and tested the code works

---

## Testing
- Created a new run with a idempotency idempotencyKey.
- Started a new run with the same task and got redirected to the first
run.
- Deleted the key from the UI on the run details
- Started a new run with the same task  and it created a new one
- Did the above steps using the SDK


---

## Changelog

- Add new action route for resetting idempotency keys via UI
- Add reset button in Idempotency section of run detail view
- Added API and SDK for resetting imdepotency
- Updated docs page for this feature


---

## Screenshots

_[Screenshots]_
<img width="438" height="363" alt="Screenshot 2025-12-11 at 11 56 37"
src="https://github.com/user-attachments/assets/30b8ef5e-8aac-4d04-b57a-9bf30d085dcb"
/>
2025-12-19 11:55:04 +00:00
Eric Allam 06cbe6e3ca chore(docs): upgrade sentry guide to use new global init and global hooks (#2799) 2025-12-19 00:00:02 +00:00
Eric Allam 3875bb292a feat(engine): run debounce system (#2794)
Adds support for **debounced task runs** - when triggering a task with a
debounce key, subsequent triggers with the same key will reschedule the
existing delayed run instead of creating new runs. This continues until
no new triggers occur within the delay window.

## Usage

```typescript
await myTask.trigger({ userId: "123" }, {
  debounce: {
    key: "user-123-update",
    delay: "5s",
    mode: "leading", // default
  }
});
```

- **key**: Scoped to the task identifier
- **delay**: How long to wait before executing (supports duration
strings like `"5s"`, `"1m"`)
- **mode**: Either `"leading"` or `"trailing"`. Leading debounce will
use the payload and options from the first run created with the debounce
key. Trailing will use payload and options from the last run.

### "trailing" mode overrides

When using `mode: "trailing"` with debounce, the following options are
updated from the **last** trigger:

- **`payload`** - The task input data
- **`metadata`** - Run metadata
- **`tags`** - Run tags (replaces existing tags)
- **`maxAttempts`** - Maximum retry attempts
- **`maxDuration`** - Maximum compute time
- **`machine`**  - Machine preset (cpu/memory)

## Behavior

- **First run wins**: The first trigger creates the run, subsequent
triggers push its execution time later
- **Idempotency keys take precedence**: If both are specified,
idempotency is checked first
- **Max duration**: Configurable via `DEBOUNCE_MAX_DURATION_MS` env var
(default: 10 minutes)

Works with `triggerAndWait` - parent runs correctly block on the
debounced run.
2025-12-18 16:04:43 +00:00
Saadi Myftija ff80742ab7 fix(ci): use workflow_dispatch instead of repository_dispatch for publish.yml (#2791)
Mixed up the trigger in #2790
2025-12-16 16:53:22 +01:00
Saadi Myftija 11366e658c chore(ci): add manual trigger to the image publishing workflow (#2790)
Useful to retry failures manually.
2025-12-16 16:34:57 +01:00
Eric Allam e751f8832e fix(app): incorrectly duplicated env vars for new batch trigger system (#2789) 2025-12-16 15:27:13 +00:00
Eric Allam a999d9ea3f feat(engine): Batch trigger reloaded (#2779)
New batch trigger system with larger payloads, streaming ingestion,
larger batch sizes, and a fair processing system.

This PR introduces a new `FairQueue` abstraction inspired by our own
`RunQueue` that enables multi-tenant fair queueing with concurrency
limits. The new `BatchQueue` is built on top of the `FairQueue`, and
handles processing Batch triggers in a fair manner with per-environment
concurrency limits defined per-org. Additionally, there is a global
concurrency limit to prevent the BatchQueue system from creating too
many runs too quickly, which can cause downstream issues.

For this new BatchQueue system we have a completely new batch trigger
creation and ingestion system. Previously this was a single endpoint
with a single JSON body that defined details about the batch as well as
all the items in the batch.

We're introducing a two-phase batch trigger ingestion system. In the
first phase, the BatchTaskRun record is created (and possibly rate
limited). The second phase is another endpoint that accepts an NDJSON
body with each line being a single item/run with payload and options.

At ingestion time all items are added to a queue, in order, and then
processed by the BatchQueue system.

## New batch trigger rate limits

This PR implements a new batch trigger specific rate limit, configured
on the `Organization.batchRateLimitConfig` column, and defaults using
these environment variables:

- `BATCH_RATE_LIMIT_REFILL_RATE` defaults to 10
- `BATCH_RATE_LIMIT_REFILL_INTERVAL` the duration interval, defaults to
`"10s"`
- `BATCH_RATE_LIMIT_MAX` defaults to 1200

This rate limiter is scoped to the environment ID and controls how many
runs can be submitted via batch triggers per interval. The SDK handles
the retrying side.

## Batch queue concurrency limits

The new column `Organization.batchQueueConcurrencyConfig` now defines an
org specific `processingConcurrency` value, with a backup of the env var
`BATCH_CONCURRENCY_LIMIT_DEFAULT` which defaults to 10. This controls
how many batch queue items are processed concurrently per environment.

There is also a global rate limit for the batch queue set via the
`BATCH_QUEUE_GLOBAL_RATE_LIMIT` which defaults to being disabled. If
set, the entire batch queue system won't process more than
`BATCH_QUEUE_GLOBAL_RATE_LIMIT` items per second. This allows
controlling the maximum number of runs created per second via batch
triggers.

## Batch trigger settings

- `STREAMING_BATCH_MAX_ITEMS` controls the maximum number of items in a
single batch
- `STREAMING_BATCH_ITEM_MAXIMUM_SIZE` controls the maximum size of each
item in a batch
- `BATCH_CONCURRENCY_DEFAULT_CONCURRENCY` controls the default
environment concurrency
- `BATCH_QUEUE_DRR_QUANTUM` how many credits each environment gets each
round for the DRR scheduler
- `BATCH_QUEUE_MAX_DEFICIT` the maximum deficit for the DRR scheduler
- `BATCH_QUEUE_CONSUMER_COUNT` how many queue consumers to run
- `BATCH_QUEUE_CONSUMER_INTERVAL_MS` how frequently they poll for items
in the queue

### Configuration Recommendations by Use Case

**High-throughput priority (fairness acceptable at 0.98+):**

```env
BATCH_QUEUE_DRR_QUANTUM=25
BATCH_QUEUE_MAX_DEFICIT=100
BATCH_QUEUE_CONSUMER_COUNT=10
BATCH_QUEUE_CONSUMER_INTERVAL_MS=50
BATCH_CONCURRENCY_DEFAULT_CONCURRENCY=25
```

**Strict fairness priority (throughput can be lower):**

```env
BATCH_QUEUE_DRR_QUANTUM=5
BATCH_QUEUE_MAX_DEFICIT=25
BATCH_QUEUE_CONSUMER_COUNT=3
BATCH_QUEUE_CONSUMER_INTERVAL_MS=100
BATCH_CONCURRENCY_DEFAULT_CONCURRENCY=5
```
2025-12-16 14:32:49 +00:00