Release 3.0.0-beta.55

* tasks no longer inside a group in the side menu (and added “cron”)

* Delay using a timezone

* Added React Not Defined error to the troubleshooting page

* Improved the React common problem

* Link to v2 docs

* New Development section and entry in Common Problems

* Concurrently running the terminal

* Fixed the .env weirdness

* Added section on creating PATs for Github actions

* Improved the Machine spec and limits page

* Quick start steps now have nice images

* Added a diagram for the lifecycle functions

* Added note about onFailure

* CRON -> cron/Cron

* WIP adding more steps to the next.js guide

* WIP next.js

* WIP adding tabbed steps for pages/app router

* References to Infisical links to their homepage so it’s clearer

* WIP updating the nextjs guide

* WIP nextjs guide

* More nextjs guide steps

* More copy

* Added rate limit trouble shooting

* Removed old prisma error title

* Added secret key step

* Added a note for logging in using a specified domain if self hosting

* typo

* App router docs copy

* Deploy copy update

* Added a favicon.png to fix a docs build error

* Removed unused snippet

* Server actions now inside a tab

* Server actions + restructured the triggering section

* Added troubleshooting snippet for react event handlers

* Added a new troubleshooting snippet for ESM

* Updated old replaying image to reflect the new UI

* Removed references to reattempting

* Updated replaying from the run page

* Added a bulk replay section

* Updated the writing tasks intro page

* Removed edge runtime code for now

* Added edge runtime – it seems to just work!

* import type

.changeset/angry-trees-drop.md

@@ -0,0 +1,5 @@
+---
+"trigger.dev": patch
+---
+
+Increase span attribute value length limit to 2048

.changeset/brown-boats-bathe.md

@@ -0,0 +1,6 @@
+---
+"@trigger.dev/core-apps": patch
+"@trigger.dev/core": patch
+---
+
+Pre-pull deployment images for faster startups

.changeset/curly-monkeys-tell.md

@@ -0,0 +1,7 @@
+---
+"trigger.dev": patch
+"@trigger.dev/core": patch
+---
+
+- Prevent uncaught exceptions when handling WebSocket messages
+- Improve CLI dev command WebSocket debug and error logging

.changeset/early-impalas-itch.md

@@ -0,0 +1,5 @@
+---
+"@trigger.dev/sdk": patch
+---
+
+Fixes for continuing after waits

.changeset/fast-colts-relax.md

@@ -0,0 +1,5 @@
+---
+"trigger.dev": patch
+---
+
+Increase dev worker timeout

.changeset/fast-melons-listen.md

@@ -0,0 +1,5 @@
+---
+"@trigger.dev/core": patch
+---
+
+Added version to ctx.run

.changeset/few-poems-vanish.md

@@ -0,0 +1,5 @@
+---
+"@trigger.dev/sdk": patch
+---
+
+Fix trigger functions for custom queues

.changeset/fifty-lions-think.md

@@ -0,0 +1,5 @@
+---
+"trigger.dev": patch
+---
+
+Improve prisma errors for missing postinstall

.changeset/flat-onions-punch.md

@@ -0,0 +1,5 @@
+---
+"@trigger.dev/sdk": patch
+---
+
+Fix return type of runs.retrieve, and allow passing the type of the task to runs.retrieve

.changeset/friendly-walls-repair.md

@@ -0,0 +1,5 @@
+---
+"trigger.dev": patch
+---
+
+Add sox and audiowaveform binaries to worker images

.changeset/gorgeous-cycles-guess.md

@@ -0,0 +1,5 @@
+---
+"trigger.dev": patch
+---
+
+try/catch opening the login URL

.changeset/green-pens-battle.md

@@ -0,0 +1,5 @@
+---
+"trigger.dev": patch
+---
+
+Fix automatic opening of login URL on linux-server systems with missing xdg-open

.changeset/hungry-sloths-promise.md

@@ -0,0 +1,12 @@
+---
+"@trigger.dev/core-apps": patch
+"trigger.dev": patch
+"@trigger.dev/core": patch
+---
+
+Fix issues that could result in unreezable state run crashes. Details:
+- Never checkpoint between attempts
+- Some messages and socket data now include attempt numbers
+- Remove attempt completion replays
+- Additional prod entry point logging
+- Fail runs that receive deprecated (pre-lazy attempt) execute messages

.changeset/large-seahorses-cheat.md

@@ -0,0 +1,5 @@
+---
+"@trigger.dev/sdk": patch
+---
+
+Extract common trigger code into internal functions and add a tasks.batchTriggerAndWait function

.changeset/lemon-sloths-hide.md

@@ -0,0 +1,6 @@
+---
+"@trigger.dev/sdk": patch
+"@trigger.dev/core": patch
+---
+
+v3: recover from server rate limiting errors in a more reliable way

.changeset/long-feet-invent.md

@@ -0,0 +1,5 @@
+---
+"@trigger.dev/sdk": patch
+---
+
+Rollback to try and fix some dependent attempt issues

.changeset/long-hounds-wave.md

@@ -0,0 +1,5 @@
+---
+"trigger.dev": patch
+---
+
+v3: update @depot/cli to latest 0.0.1-cli.2.71.0

.changeset/lucky-items-film.md

@@ -0,0 +1,5 @@
+---
+"trigger.dev": patch
+---
+
+Automatically bundle internal packages that use file, link or workspace protocl

.changeset/mighty-eggs-grab.md

@@ -0,0 +1,30 @@
+---
+"@trigger.dev/core-apps": patch
+"trigger.dev": patch
+"@trigger.dev/core": patch
+---
+
+Tasks should now be much more robust and resilient to reconnects during crucial operations and other failure scenarios.
+
+Task runs now have to signal checkpointable state prior to ALL checkpoints. This ensures flushing always happens.
+
+All important socket.io RPCs will now be retried with backoff. Actions relying on checkpoints will be replayed if we haven't been checkpointed and restored as expected, e.g. after reconnect.
+
+Other changes:
+
+- Fix retry check in shared queue
+- Fix env var sync spinner
+- Heartbeat between retries
+- Fix retry prep
+- Fix prod worker no tasks detection
+- Fail runs above `MAX_TASK_RUN_ATTEMPTS`
+- Additional debug logs in all places
+- Prevent crashes due to failed socket schema parsing
+- Remove core-apps barrel
+- Upgrade socket.io-client to fix an ACK memleak
+- Additional index failure logs
+- Prevent message loss during reconnect
+- Prevent burst of heartbeats on reconnect
+- Prevent crash on failed cleanup
+- Handle at-least-once lazy execute message delivery
+- Handle uncaught entry point exceptions

.changeset/modern-stingrays-end.md

@@ -0,0 +1,6 @@
+---
+"@trigger.dev/sdk": patch
+"@trigger.dev/core": patch
+---
+
+v3: Trigger delayed runs and reschedule them

.changeset/nervous-baboons-sin.md

@@ -0,0 +1,6 @@
+---
+"trigger.dev": patch
+"@trigger.dev/core": patch
+---
+
+Added config option extraCACerts to ProjectConfig type. This copies the ca file along with additionalFiles and sets NODE_EXTRA_CA_CERTS environment variable in built image as well as running the task.

.changeset/nervous-planets-sparkle.md

@@ -0,0 +1,9 @@
+---
+"trigger.dev": patch
+"@trigger.dev/core": patch
+---
+
+- Improve non-zero exit code error messages
+- Detect OOM conditions within worker child processes
+- Internal errors can have optional stack traces
+- Docker provider can be set to enforce machine presets

.changeset/nervous-seas-shave.md

@@ -0,0 +1,5 @@
+---
+"@trigger.dev/sdk": patch
+---
+
+v3: Export AbortTaskRunError from @trigger.dev/sdk/v3

.changeset/odd-beds-wonder.md

@@ -0,0 +1,6 @@
+---
+"@trigger.dev/sdk": patch
+"@trigger.dev/core": patch
+---
+
+You can now add tags to runs and list runs using them

.changeset/polite-pots-walk.md

@@ -0,0 +1,7 @@
+---
+"trigger.dev": patch
+---
+
+Add e2e fixtures corresponding to past issues
+Implement e2e suite parallelism
+Enhance log level for specific e2e suite messages

.changeset/pre.json

@@ -46,11 +46,13 @@
   "changesets": [
     "afraid-sheep-joke",
     "angry-eagles-trade",
+    "angry-trees-drop",
     "beige-pears-explode",
     "beige-pens-dance",
     "big-tomatoes-deliver",
     "blue-pumas-whisper",
     "breezy-gorillas-mate",
+    "brown-boats-bathe",
     "brown-spies-burn",
     "chilled-hornets-move",
     "clean-pianos-listen",
@@ -60,49 +62,73 @@
     "cool-comics-burn",
     "cool-glasses-bake",
     "cuddly-feet-approve",
+    "curly-monkeys-tell",
     "dry-walls-check",
     "dull-mangos-press",
+    "early-impalas-itch",
     "eight-pumas-float",
     "eleven-paws-join",
     "famous-boats-tease",
+    "fast-colts-relax",
+    "fast-melons-listen",
+    "few-poems-vanish",
     "few-students-share",
+    "fifty-lions-think",
     "five-toes-destroy",
+    "flat-onions-punch",
+    "friendly-walls-repair",
     "funny-swans-destroy",
+    "gorgeous-cycles-guess",
     "gorgeous-gorillas-compete",
     "green-bags-wink",
+    "green-pens-battle",
     "hot-buckets-behave",
     "hot-fishes-retire",
     "hot-wasps-sin",
+    "hungry-sloths-promise",
     "itchy-chairs-itch",
     "khaki-apricots-design",
     "khaki-poems-lay",
+    "large-seahorses-cheat",
     "late-icons-lie",
     "late-steaks-behave",
     "lazy-files-lay",
     "lemon-jobs-repair",
+    "lemon-sloths-hide",
     "light-bulldogs-press",
     "light-dragons-complain",
     "little-crabs-cross",
+    "long-feet-invent",
     "long-fireants-search",
+    "long-hounds-wave",
     "loud-actors-remember",
     "lovely-drinks-flash",
+    "lucky-items-film",
     "many-ligers-pump",
     "mighty-camels-joke",
+    "mighty-eggs-grab",
     "mighty-flowers-train",
     "mighty-parrots-sin",
+    "modern-stingrays-end",
     "nasty-jars-pump",
+    "nervous-baboons-sin",
+    "nervous-planets-sparkle",
+    "nervous-seas-shave",
     "new-pants-beg",
     "new-rivers-tell",
     "nice-bulldogs-turn",
     "ninety-pets-travel",
+    "odd-beds-wonder",
     "odd-poets-own",
     "pink-pumas-rhyme",
     "plenty-ducks-beam",
     "polite-ducks-switch",
     "polite-pears-grow",
+    "polite-pots-walk",
     "polite-rockets-matter",
     "poor-flowers-cross",
     "purple-garlics-shop",
+    "purple-spiders-care",
     "rare-lamps-promise",
     "rare-roses-float",
     "real-planets-stare",
@@ -113,10 +139,12 @@
     "selfish-ducks-sort",
     "serious-hats-rest",
     "shaggy-spoons-taste",
+    "shaggy-weeks-live",
     "sharp-emus-compare",
     "sharp-zebras-serve",
     "shiny-coats-cry",
     "silly-buses-obey",
+    "silly-forks-kiss",
     "silly-suits-switch",
     "silver-doors-juggle",
     "six-ligers-exist",
@@ -128,6 +156,7 @@
     "smart-needles-move",
     "smart-olives-eat",
     "sour-pugs-teach",
+    "spicy-frogs-remain",
     "spicy-lamps-smoke",
     "spicy-terms-bow",
     "strange-ghosts-matter",
@@ -141,12 +170,15 @@
     "sweet-lizards-press",
     "swift-dragons-peel",
     "tall-bees-wave",
+    "tall-masks-repeat",
+    "tame-apricots-clap",
     "tame-guests-know",
     "tender-moose-tell",
     "tender-oranges-rhyme",
     "tender-turkeys-compete",
     "thick-carrots-sneeze",
     "thin-parents-heal",
+    "thirty-hotels-raise",
     "thirty-islands-kiss",
     "tidy-balloons-suffer",
     "tidy-dryers-sleep",
@@ -161,6 +193,8 @@
     "violet-clocks-notice",
     "warm-olives-provide",
     "warm-planes-taste",
+    "yellow-roses-arrive",
+    "young-jars-wait",
     "young-snails-sell"
   ]
 }

.changeset/purple-spiders-care.md

@@ -0,0 +1,5 @@
+---
+"trigger.dev": patch
+---
+
+Await file watcher cleanup in dev

.changeset/shaggy-weeks-live.md

@@ -0,0 +1,5 @@
+---
+"@trigger.dev/core": patch
+---
+
+v3: sanitize errors with null unicode characters in some places

.changeset/silly-forks-kiss.md

@@ -0,0 +1,5 @@
+---
+"trigger.dev": patch
+---
+
+v3: Copy over more of the project's package.json keys into the deployed package.json (support for custom config like zenstack)

.changeset/spicy-frogs-remain.md

@@ -0,0 +1,10 @@
+---
+"trigger.dev": patch
+---
+
+- Prevent downgrades during update check and advise to upgrade CLI
+- Detect bun and use npm instead
+- During init, fail early and advise if not a TypeScript project
+- During init, allow specifying custom package manager args
+- Add links to dev worker started message
+- Fix links in unsupported terminals

.changeset/tall-masks-repeat.md

@@ -0,0 +1,9 @@
+---
+"@trigger.dev/core-apps": patch
+"trigger.dev": patch
+"@trigger.dev/core": patch
+---
+
+- Fix artifact detection logs
+- Fix OOM detection and error messages
+- Add test link to cli deployment completion

.changeset/tame-apricots-clap.md

@@ -0,0 +1,5 @@
+---
+"trigger.dev": patch
+---
+
+v3: postInstall config option now replaces the postinstall script found in package.json

.changeset/thirty-hotels-raise.md

@@ -0,0 +1,6 @@
+---
+"@trigger.dev/sdk": patch
+"@trigger.dev/core": patch
+---
+
+Added declarative cron schedules

.changeset/yellow-roses-arrive.md

@@ -0,0 +1,5 @@
+---
+"@trigger.dev/core": patch
+---
+
+Add more package exports that can be used from the web app

.changeset/young-jars-wait.md

@@ -0,0 +1,5 @@
+---
+"trigger.dev": patch
+---
+
+Fix various e2e issues for 'resolve-legacy-peer-deps' fixture, installation of fixture deps and lockfile-based test skipping'

.env.example

@@ -11,7 +11,6 @@ REMIX_APP_PORT=3030
 APP_ENV=development
 APP_ORIGIN=http://localhost:3030
 NODE_ENV=development
-CLOUD_ENV=development
 V3_ENABLED=true
 
 # Redis is used for the v3 queuing and v2 concurrency control
@@ -71,7 +70,7 @@ COORDINATOR_SECRET=coordinator-secret # generate the actual secret with `openssl
 # OBJECT_STORE_BASE_URL="https://{bucket}.{accountId}.r2.cloudflarestorage.com"
 # OBJECT_STORE_ACCESS_KEY_ID=
 # OBJECT_STORE_SECRET_ACCESS_KEY=
-# RUNTIME_WAIT_THRESHOLD_IN_MS=10000
+# CHECKPOINT_THRESHOLD_IN_MS=10000
 
 # These control the server-side internal telemetry
 # INTERNAL_OTEL_TRACE_EXPORTER_URL=<URL to send traces to>
@@ -79,4 +78,4 @@ COORDINATOR_SECRET=coordinator-secret # generate the actual secret with `openssl
 # INTERNAL_OTEL_TRACE_EXPORTER_AUTH_HEADER_VALUE=<Auth token value>
 # INTERNAL_OTEL_TRACE_LOGGING_ENABLED=1
 # INTERNAL_OTEL_TRACE_SAMPING_RATE=20 # this means 1/20 traces or 5% of traces will be sampled (sampled = recorded)
-# INTERNAL_OTEL_TRACE_INSTRUMENT_PRISMA_ENABLED=0,
+# INTERNAL_OTEL_TRACE_INSTRUMENT_PRISMA_ENABLED=0,

.github/workflows/e2e.yml

@@ -23,7 +23,7 @@ jobs:
           fetch-depth: 0
 
       - name: ⎔ Setup pnpm
-        uses: pnpm/action-setup@v2.2.4
+        uses: pnpm/action-setup@v4
         with:
           version: 8.15.5
 
@@ -42,6 +42,9 @@ jobs:
       - name: 🔧 Build worker template files
         run: pnpm --filter trigger.dev run build:workers
 
+      - name: Enable corepack
+        run: corepack enable
+
       - name: Run E2E Tests
         run: |
           PM=${{ matrix.package-manager }} pnpm --filter trigger.dev run test:e2e
@@ -63,7 +66,7 @@ jobs:
           submodules: recursive
 
       - name: ⎔ Setup pnpm
-        uses: pnpm/action-setup@v2.2.4
+        uses: pnpm/action-setup@v4
         with:
           version: 8.15.5
 

.github/workflows/publish-docker.yml

@@ -51,9 +51,9 @@ jobs:
           ref_without_tag=ghcr.io/triggerdotdev/trigger.dev
           image_tags=$ref_without_tag:${{ steps.get_version.outputs.version }}
 
-          # if it's a versioned tag, also tag it as latest
+          # if it's a versioned tag, also tag it as v3
           if [[ "${{ github.ref_name }}" == v.docker.* ]]; then
-            image_tags=$image_tags,$ref_without_tag:latest
+            image_tags=$image_tags,$ref_without_tag:v3
           fi
           
           echo "IMAGE_TAGS=${image_tags}" >> "$GITHUB_OUTPUT"

.github/workflows/publish-infra.yml

@@ -111,11 +111,11 @@ jobs:
           REPOSITORY: ${{ steps.prep.outputs.REPOSITORY }}
           IMAGE_TAG: ${{ steps.prep.outputs.IMAGE_TAG }}
 
-      - name: 🐙 Push 'latest' to GitHub Container Registry
+      - name: 🐙 Push 'v3' tag to GitHub Container Registry
         if: startsWith(github.ref_name, 'v.docker.')
         run: |
-          docker tag infra_image $REGISTRY/$REPOSITORY:latest
-          docker push $REGISTRY/$REPOSITORY:latest
+          docker tag infra_image $REGISTRY/$REPOSITORY:v3
+          docker push $REGISTRY/$REPOSITORY:v3
         env:
           REGISTRY: ghcr.io/triggerdotdev
           REPOSITORY: ${{ steps.prep.outputs.REPOSITORY }}

.github/workflows/release.yml

@@ -28,7 +28,7 @@ jobs:
           fetch-depth: 0
 
       - name: ⎔ Setup pnpm
-        uses: pnpm/action-setup@v2.2.4
+        uses: pnpm/action-setup@v4
         with:
           version: 8.15.5
 

.github/workflows/typecheck.yml

@@ -12,7 +12,7 @@ jobs:
           fetch-depth: 0
 
       - name: ⎔ Setup pnpm
-        uses: pnpm/action-setup@v2.2.4
+        uses: pnpm/action-setup@v4
         with:
           version: 8.15.5
 

.github/workflows/unit-tests.yml

@@ -12,7 +12,7 @@ jobs:
           fetch-depth: 0
 
       - name: ⎔ Setup pnpm
-        uses: pnpm/action-setup@v2.2.4
+        uses: pnpm/action-setup@v4
         with:
           version: 8.15.5
 

README.md

@@ -5,64 +5,71 @@
   <img alt="Trigger.dev logo" src="https://imagedelivery.net/3TbraffuDZ4aEf8KWOmI_w/a45d1fa2-0ae8-4a39-4409-f4f934bfae00/public">
 </picture>
   
-### The open source background jobs platform
+### Open source background jobs with no timeouts
 
 [Discord](https://trigger.dev/discord) | [Website](https://trigger.dev) | [Issues](https://github.com/triggerdotdev/trigger.dev/issues) | [Docs](https://trigger.dev/docs)
 
 [![Twitter](https://img.shields.io/twitter/url/https/twitter.com/triggerdotdev.svg?style=social&label=Follow%20%40trigger.dev)](https://twitter.com/triggerdotdev)
-[![GitHub Repo stars](https://img.shields.io/github/stars/triggerdotdev/trigger.dev?style=social)](https://github.com/triggerdotdev/trigger.dev)
 
 </div>
 
-> The Trigger.dev v3 developer preview is now open. For more information and to get early access, check out our [developer preview launch post](https://trigger.dev/blog/v3-developer-preview-launch/).
-
 ## About Trigger.dev
 
-Create long-running jobs directly in your codebase with features like API integrations, webhooks, scheduling and delays.
+Trigger.dev is an open source platform and SDK which allows you to create long-running background jobs with no timeouts. Write normal async code, deploy, and never hit a timeout.
 
-## Long running Jobs on serverless
+#### Features:
 
-Reliably run jobs and don’t worry about function timeouts, we handle those for you.
-
-- Auto-resume after a function timeout
-- Auto-resume after a server outage
-- Add delays of up to a year
+- JavaScript and TypeScript SDK
+- Write reliable code by default
+- No infrastructure to manage
+- Works with your existing tech stack
 
 ## In your codebase
 
-Create Jobs where they belong: in your codebase. Version control, localhost, test, review, and deploy like you're already used to.
+Create tasks where they belong: in your codebase. Version control, localhost, test and review like you're already used to.
 
-## Secure by design
+```ts
+import { task } from "@trigger.dev/sdk/v3";
 
-We only receive Triggers and the data you choose to send to us. You can even completely self-host the entire platform.
+//1. You need to export each task
+export const helloWorld = task({
+  //2. Use a unique id for each task
+  id: "hello-world",
+  //3. The run function is the main function of the task
+  run: async (payload: { message: string }) => {
+    //4. You can write code that runs for a long time here, there are no timeouts
+    console.log(payload.message);
+  },
+});
+```
 
-## Don't worry about deployment
+## Deployment
 
-Just use our SDK to write Jobs in your codebase. There's nothing extra to deploy and no CI to configure, your Jobs just connect to our cloud. Or you can always self-host.
+Use our SDK to write tasks in your codebase. There's no infrastructure to manage, your tasks automatically scale and connect to our cloud. Or you can always [self-host](https://trigger.dev/docs/v3/open-source-self-hosting#overview).
+
+## Environments
+
+We support `Development`, `Staging`, and `Production` environments, allowing you to test your tasks before deploying them to production.
 
 ## Full visibility of every job run
 
-View every Task in every Run so you can tell exactly what happened.
+View every task in every run so you can tell exactly what happened. We provide a full trace view of every task run so you can see what happened at every step.
 
-![image](https://www.trigger.dev/build/_assets/web-app-2QFKXFLW.png)
-
-## Built-in integrations
-
-Easily integrate with hundreds of third-party APIs – including your own. Use API keys (which never leave your server) or let us handle OAuth for you. Install our integration packages and easily subscribe to webhooks and perform common tasks, or you can easily use your existing favorite Node.JS SDKs and get resumability and idempotency through our `runTask` function.
+![Trace view image](https://imagedelivery.net/3TbraffuDZ4aEf8KWOmI_w/7c1b347f-004c-4482-38a7-3f6fa9c00d00/public)
 
 # Getting started
 
-Visit our docs [here](https://trigger.dev/docs).
+Visit our docs [here](https://trigger.dev/docs/v3/introduction) for a full guide on how to get started with Trigger.dev.
 
 ## Self-host
 
-We provide an official trigger.dev docker image you can use to easily self-host the platform. We're working on more extensive guides but we currently provide a [Fly.io example repository](https://github.com/triggerdotdev/fly.io) with instructions in the README for deploying and using a self-hosted instance of Trigger.dev on Fly.io.
+If you prefer to self-host, you can follow our [self-hosting guide](https://trigger.dev/docs/v3/open-source-self-hosting#overview).
 
 ## Development
 
 To setup and develop locally or contribute to the open source project, follow our [development guide](./CONTRIBUTING.md).
 
-## Meet the Amazing People Behind This Project 🚀
+## Meet the Amazing People Behind This Project:
 
 <a href="https://github.com/triggerdotdev/trigger.dev/graphs/contributors">
   <img src="https://contrib.rocks/image?repo=triggerdotdev/trigger.dev" />

apps/coordinator/package.json

@@ -21,8 +21,7 @@
     "execa": "^8.0.1",
     "nanoid": "^5.0.6",
     "prom-client": "^15.1.0",
-    "socket.io": "4.7.4",
-    "socket.io-client": "4.7.4"
+    "socket.io": "4.7.4"
   },
   "devDependencies": {
     "@types/node": "^18",

apps/coordinator/src/chaosMonkey.ts

@@ -0,0 +1,99 @@
+import type { Execa$ } from "execa";
+import { setTimeout as timeout } from "node:timers/promises";
+
+class ChaosMonkeyError extends Error {
+  constructor(message: string) {
+    super(message);
+    this.name = "ChaosMonkeyError";
+  }
+}
+
+export class ChaosMonkey {
+  private chaosEventRate = 0.2;
+  private delayInSeconds = 45;
+
+  constructor(
+    private enabled = false,
+    private disableErrors = false,
+    private disableDelays = false
+  ) {
+    if (this.enabled) {
+      console.log("🍌 Chaos monkey enabled");
+    }
+  }
+
+  static Error = ChaosMonkeyError;
+
+  enable() {
+    this.enabled = true;
+    console.log("🍌 Chaos monkey enabled");
+  }
+
+  disable() {
+    this.enabled = false;
+    console.log("🍌 Chaos monkey disabled");
+  }
+
+  async call({
+    $,
+    throwErrors = !this.disableErrors,
+    addDelays = !this.disableDelays,
+  }: {
+    $?: Execa$<string>;
+    throwErrors?: boolean;
+    addDelays?: boolean;
+  } = {}) {
+    if (!this.enabled) {
+      return;
+    }
+
+    const random = Math.random();
+
+    if (random > this.chaosEventRate) {
+      // Don't interfere with normal operation
+      return;
+    }
+
+    const chaosEvents: Array<() => Promise<any>> = [];
+
+    if (addDelays) {
+      chaosEvents.push(async () => {
+        console.log("🍌 Chaos monkey: Add delay");
+
+        if ($) {
+          await $`sleep ${this.delayInSeconds}`;
+        } else {
+          await timeout(this.delayInSeconds * 1000);
+        }
+      });
+    }
+
+    if (throwErrors) {
+      chaosEvents.push(async () => {
+        console.log("🍌 Chaos monkey: Throw error");
+
+        if ($) {
+          await $`false`;
+        } else {
+          throw new ChaosMonkey.Error("🍌 Chaos monkey: Throw error");
+        }
+      });
+    }
+
+    if (chaosEvents.length === 0) {
+      console.error("🍌 Chaos monkey: No events selected");
+      return;
+    }
+
+    const randomIndex = Math.floor(Math.random() * chaosEvents.length);
+
+    const chaosEvent = chaosEvents[randomIndex];
+
+    if (!chaosEvent) {
+      console.error("🍌 Chaos monkey: No event found");
+      return;
+    }
+
+    await chaosEvent();
+  }
+}

apps/coordinator/src/checkpointer.ts

@@ -0,0 +1,593 @@
+import { ExponentialBackoff } from "@trigger.dev/core-apps/backoff";
+import { isExecaChildProcess, testDockerCheckpoint } from "@trigger.dev/core-apps/checkpoints";
+import { SimpleLogger } from "@trigger.dev/core-apps/logger";
+import { $ } from "execa";
+import { nanoid } from "nanoid";
+import fs from "node:fs/promises";
+import { ChaosMonkey } from "./chaosMonkey";
+
+type CheckpointerInitializeReturn = {
+  canCheckpoint: boolean;
+  willSimulate: boolean;
+};
+
+type CheckpointAndPushOptions = {
+  runId: string;
+  leaveRunning?: boolean;
+  projectRef: string;
+  deploymentVersion: string;
+  shouldHeartbeat?: boolean;
+  attemptNumber?: number;
+};
+
+type CheckpointAndPushResult =
+  | { success: true; checkpoint: CheckpointData }
+  | {
+      success: false;
+      reason?: "CANCELED" | "DISABLED" | "ERROR" | "IN_PROGRESS" | "NO_SUPPORT" | "SKIP_RETRYING";
+    };
+
+type CheckpointData = {
+  location: string;
+  docker: boolean;
+};
+
+type CheckpointerOptions = {
+  dockerMode: boolean;
+  forceSimulate: boolean;
+  heartbeat: (runId: string) => void;
+  registryHost?: string;
+  registryNamespace?: string;
+  registryTlsVerify?: boolean;
+  disableCheckpointSupport?: boolean;
+  checkpointPath?: string;
+  simulateCheckpointFailure?: boolean;
+  simulateCheckpointFailureSeconds?: number;
+  simulatePushFailure?: boolean;
+  simulatePushFailureSeconds?: number;
+  chaosMonkey?: ChaosMonkey;
+};
+
+async function getFileSize(filePath: string): Promise<number> {
+  try {
+    const stats = await fs.stat(filePath);
+    return stats.size;
+  } catch (error) {
+    console.error("Error getting file size:", error);
+    return -1;
+  }
+}
+
+async function getParsedFileSize(filePath: string) {
+  const sizeInBytes = await getFileSize(filePath);
+
+  let message = `Size in bytes: ${sizeInBytes}`;
+
+  if (sizeInBytes > 1024 * 1024) {
+    const sizeInMB = (sizeInBytes / 1024 / 1024).toFixed(2);
+    message = `Size in MB (rounded): ${sizeInMB}`;
+  } else if (sizeInBytes > 1024) {
+    const sizeInKB = (sizeInBytes / 1024).toFixed(2);
+    message = `Size in KB (rounded): ${sizeInKB}`;
+  }
+
+  return {
+    path: filePath,
+    sizeInBytes,
+    message,
+  };
+}
+
+export class Checkpointer {
+  #initialized = false;
+  #canCheckpoint = false;
+  #dockerMode: boolean;
+
+  #logger = new SimpleLogger("[checkptr]");
+  #abortControllers = new Map<string, AbortController>();
+  #failedCheckpoints = new Map<string, unknown>();
+  #waitingForRetry = new Set<string>();
+
+  private registryHost: string;
+  private registryNamespace: string;
+  private registryTlsVerify: boolean;
+
+  private disableCheckpointSupport: boolean;
+  private checkpointPath: string;
+
+  private simulateCheckpointFailure: boolean;
+  private simulateCheckpointFailureSeconds: number;
+  private simulatePushFailure: boolean;
+  private simulatePushFailureSeconds: number;
+
+  private chaosMonkey: ChaosMonkey;
+
+  constructor(private opts: CheckpointerOptions) {
+    this.#dockerMode = opts.dockerMode;
+
+    this.registryHost = opts.registryHost ?? "localhost:5000";
+    this.registryNamespace = opts.registryNamespace ?? "trigger";
+    this.registryTlsVerify = opts.registryTlsVerify ?? true;
+
+    this.disableCheckpointSupport = opts.disableCheckpointSupport ?? false;
+    this.checkpointPath = opts.checkpointPath ?? "/checkpoints";
+
+    this.simulateCheckpointFailure = opts.simulateCheckpointFailure ?? false;
+    this.simulateCheckpointFailureSeconds = opts.simulateCheckpointFailureSeconds ?? 300;
+    this.simulatePushFailure = opts.simulatePushFailure ?? false;
+    this.simulatePushFailureSeconds = opts.simulatePushFailureSeconds ?? 300;
+
+    this.chaosMonkey = opts.chaosMonkey ?? new ChaosMonkey(!!process.env.CHAOS_MONKEY_ENABLED);
+  }
+
+  async init(): Promise<CheckpointerInitializeReturn> {
+    if (this.#initialized) {
+      return this.#getInitReturn(this.#canCheckpoint);
+    }
+
+    this.#logger.log(`${this.#dockerMode ? "Docker" : "Kubernetes"} mode`);
+
+    if (this.#dockerMode) {
+      const testCheckpoint = await testDockerCheckpoint();
+
+      if (testCheckpoint.ok) {
+        return this.#getInitReturn(true);
+      }
+
+      this.#logger.error(testCheckpoint.message, testCheckpoint.error ?? "");
+      return this.#getInitReturn(false);
+    } else {
+      try {
+        await $`buildah login --get-login ${this.registryHost}`;
+      } catch (error) {
+        this.#logger.error(`No checkpoint support: Not logged in to registry ${this.registryHost}`);
+        return this.#getInitReturn(false);
+      }
+    }
+
+    return this.#getInitReturn(true);
+  }
+
+  #getInitReturn(canCheckpoint: boolean): CheckpointerInitializeReturn {
+    this.#canCheckpoint = canCheckpoint;
+
+    if (canCheckpoint) {
+      if (!this.#initialized) {
+        this.#logger.log("Full checkpoint support!");
+      }
+    }
+
+    this.#initialized = true;
+
+    const willSimulate = this.#dockerMode && (!this.#canCheckpoint || this.opts.forceSimulate);
+
+    if (willSimulate) {
+      this.#logger.log("Simulation mode enabled. Containers will be paused, not checkpointed.", {
+        forceSimulate: this.opts.forceSimulate,
+      });
+    }
+
+    return {
+      canCheckpoint,
+      willSimulate,
+    };
+  }
+
+  #getImageRef(projectRef: string, deploymentVersion: string, shortCode: string) {
+    return `${this.registryHost}/${this.registryNamespace}/${projectRef}:${deploymentVersion}.prod-${shortCode}`;
+  }
+
+  #getExportLocation(projectRef: string, deploymentVersion: string, shortCode: string) {
+    const basename = `${projectRef}-${deploymentVersion}-${shortCode}`;
+
+    if (this.#dockerMode) {
+      return basename;
+    } else {
+      return `${this.checkpointPath}/${basename}.tar`;
+    }
+  }
+
+  async checkpointAndPush(opts: CheckpointAndPushOptions): Promise<CheckpointData | undefined> {
+    const start = performance.now();
+    this.#logger.log(`checkpointAndPush() start`, { start, opts });
+
+    let interval: NodeJS.Timer | undefined;
+
+    if (opts.shouldHeartbeat) {
+      interval = setInterval(() => {
+        this.#logger.log("Sending heartbeat", { runId: opts.runId });
+        this.opts.heartbeat(opts.runId);
+      }, 20_000);
+    }
+
+    try {
+      const result = await this.#checkpointAndPushWithBackoff(opts);
+
+      const end = performance.now();
+      this.#logger.log(`checkpointAndPush() end`, {
+        start,
+        end,
+        diff: end - start,
+        opts,
+        success: result.success,
+      });
+
+      if (!result.success) {
+        return;
+      }
+
+      return result.checkpoint;
+    } finally {
+      if (opts.shouldHeartbeat) {
+        clearInterval(interval);
+      }
+    }
+  }
+
+  isCheckpointing(runId: string) {
+    return this.#abortControllers.has(runId) || this.#waitingForRetry.has(runId);
+  }
+
+  cancelCheckpoint(runId: string): boolean {
+    // If the last checkpoint failed, pretend we canceled it
+    // This ensures tasks don't wait for external resume messages to continue
+    if (this.#hasFailedCheckpoint(runId)) {
+      this.#clearFailedCheckpoint(runId);
+      return true;
+    }
+
+    if (this.#waitingForRetry.has(runId)) {
+      this.#waitingForRetry.delete(runId);
+      return true;
+    }
+
+    const controller = this.#abortControllers.get(runId);
+
+    if (!controller) {
+      this.#logger.debug("Nothing to cancel", { runId });
+      return false;
+    }
+
+    controller.abort("cancelCheckpointing()");
+    this.#abortControllers.delete(runId);
+
+    return true;
+  }
+
+  async #checkpointAndPushWithBackoff({
+    runId,
+    leaveRunning = true, // This mirrors kubernetes behaviour more accurately
+    projectRef,
+    deploymentVersion,
+    attemptNumber,
+  }: CheckpointAndPushOptions): Promise<CheckpointAndPushResult> {
+    this.#logger.log("Checkpointing with backoff", {
+      runId,
+      leaveRunning,
+      projectRef,
+      deploymentVersion,
+    });
+
+    const backoff = new ExponentialBackoff()
+      .type("EqualJitter")
+      .base(3)
+      .max(3 * 3600)
+      .maxElapsed(48 * 3600);
+
+    for await (const { delay, retry } of backoff) {
+      try {
+        if (retry > 0) {
+          this.#logger.error("Retrying checkpoint", {
+            runId,
+            retry,
+            delay,
+          });
+
+          this.#waitingForRetry.add(runId);
+          await new Promise((resolve) => setTimeout(resolve, delay.milliseconds));
+
+          if (!this.#waitingForRetry.has(runId)) {
+            this.#logger.log("Checkpoint canceled while waiting for retry", { runId });
+            return { success: false, reason: "CANCELED" };
+          } else {
+            this.#waitingForRetry.delete(runId);
+          }
+        }
+
+        const result = await this.#checkpointAndPush({
+          runId,
+          leaveRunning,
+          projectRef,
+          deploymentVersion,
+          attemptNumber,
+        });
+
+        if (result.success) {
+          return result;
+        }
+
+        if (result.reason === "CANCELED") {
+          this.#logger.log("Checkpoint canceled, won't retry", { runId });
+          // Don't fail the checkpoint, as it was canceled
+          return result;
+        }
+
+        if (result.reason === "IN_PROGRESS") {
+          this.#logger.log("Checkpoint already in progress, won't retry", { runId });
+          this.#failCheckpoint(runId, result.reason);
+          return result;
+        }
+
+        if (result.reason === "NO_SUPPORT") {
+          this.#logger.log("No checkpoint support, won't retry", { runId });
+          this.#failCheckpoint(runId, result.reason);
+          return result;
+        }
+
+        if (result.reason === "DISABLED") {
+          this.#logger.log("Checkpoint support disabled, won't retry", { runId });
+          this.#failCheckpoint(runId, result.reason);
+          return result;
+        }
+
+        if (result.reason === "SKIP_RETRYING") {
+          this.#logger.log("Skipping retrying", { runId });
+          return result;
+        }
+
+        continue;
+      } catch (error) {
+        this.#logger.error("Checkpoint error", {
+          retry,
+          runId,
+          delay,
+          error: error instanceof Error ? error.message : error,
+        });
+      }
+    }
+
+    this.#logger.error(`Checkpoint failed after exponential backoff`, {
+      runId,
+      leaveRunning,
+      projectRef,
+      deploymentVersion,
+    });
+    this.#failCheckpoint(runId, "ERROR");
+
+    return { success: false, reason: "ERROR" };
+  }
+
+  async #checkpointAndPush({
+    runId,
+    leaveRunning = true, // This mirrors kubernetes behaviour more accurately
+    projectRef,
+    deploymentVersion,
+    attemptNumber,
+  }: CheckpointAndPushOptions): Promise<CheckpointAndPushResult> {
+    await this.init();
+
+    const options = {
+      runId,
+      leaveRunning,
+      projectRef,
+      deploymentVersion,
+      attemptNumber,
+    };
+
+    if (!this.#dockerMode && !this.#canCheckpoint) {
+      this.#logger.error("No checkpoint support. Simulation requires docker.");
+      return { success: false, reason: "NO_SUPPORT" };
+    }
+
+    if (this.isCheckpointing(runId)) {
+      this.#logger.error("Checkpoint procedure already in progress", { options });
+      return { success: false, reason: "IN_PROGRESS" };
+    }
+
+    // This is a new checkpoint, clear any last failure for this run
+    this.#clearFailedCheckpoint(runId);
+
+    if (this.disableCheckpointSupport) {
+      this.#logger.error("Checkpoint support disabled", { options });
+      return { success: false, reason: "DISABLED" };
+    }
+
+    const controller = new AbortController();
+    this.#abortControllers.set(runId, controller);
+
+    const $$ = $({ signal: controller.signal });
+
+    const shortCode = nanoid(8);
+    const imageRef = this.#getImageRef(projectRef, deploymentVersion, shortCode);
+    const exportLocation = this.#getExportLocation(projectRef, deploymentVersion, shortCode);
+
+    const cleanup = async () => {
+      if (this.#dockerMode) {
+        return;
+      }
+
+      try {
+        await $`rm ${exportLocation}`;
+        this.#logger.log("Deleted checkpoint archive", { exportLocation });
+
+        await $`buildah rmi ${imageRef}`;
+        this.#logger.log("Deleted checkpoint image", { imageRef });
+      } catch (error) {
+        this.#logger.error("Failure during checkpoint cleanup", { exportLocation, error });
+      }
+    };
+
+    try {
+      await this.chaosMonkey.call({ $: $$ });
+
+      this.#logger.log("Checkpointing:", { options });
+
+      const containterName = this.#getRunContainerName(runId);
+      const containterNameWithAttempt = this.#getRunContainerName(runId, attemptNumber);
+
+      // Create checkpoint (docker)
+      if (this.#dockerMode) {
+        try {
+          if (this.opts.forceSimulate || !this.#canCheckpoint) {
+            this.#logger.log("Simulating checkpoint");
+            this.#logger.debug(await $$`docker pause ${containterNameWithAttempt}`);
+          } else {
+            if (this.simulateCheckpointFailure) {
+              if (performance.now() < this.simulateCheckpointFailureSeconds * 1000) {
+                this.#logger.error("Simulating checkpoint failure", { options });
+                throw new Error("SIMULATE_CHECKPOINT_FAILURE");
+              }
+            }
+
+            if (leaveRunning) {
+              this.#logger.debug(
+                await $$`docker checkpoint create --leave-running ${containterNameWithAttempt} ${exportLocation}`
+              );
+            } else {
+              this.#logger.debug(
+                await $$`docker checkpoint create ${containterNameWithAttempt} ${exportLocation}`
+              );
+            }
+          }
+        } catch (error) {
+          this.#logger.error("Failed while creating docker checkpoint", { exportLocation });
+          throw error;
+        }
+
+        this.#logger.log("checkpoint created:", {
+          runId,
+          location: exportLocation,
+        });
+
+        return {
+          success: true,
+          checkpoint: {
+            location: exportLocation,
+            docker: true,
+          },
+        };
+      }
+
+      // Create checkpoint (CRI)
+      if (!this.#canCheckpoint) {
+        this.#logger.error("No checkpoint support in kubernetes mode.");
+        return { success: false, reason: "SKIP_RETRYING" };
+      }
+
+      const containerId = this.#logger.debug(
+        // @ts-expect-error
+        await $$`crictl ps`
+          .pipeStdout($$({ stdin: "pipe" })`grep ${containterName}`)
+          .pipeStdout($$({ stdin: "pipe" })`cut -f1 ${"-d "}`)
+      );
+
+      if (!containerId.stdout) {
+        this.#logger.error("could not find container id", { options, containterName });
+        return { success: false, reason: "SKIP_RETRYING" };
+      }
+
+      const start = performance.now();
+
+      if (this.simulateCheckpointFailure) {
+        if (performance.now() < this.simulateCheckpointFailureSeconds * 1000) {
+          this.#logger.error("Simulating checkpoint failure", { options });
+          throw new Error("SIMULATE_CHECKPOINT_FAILURE");
+        }
+      }
+
+      // Create checkpoint
+      this.#logger.debug(await $$`crictl checkpoint --export=${exportLocation} ${containerId}`);
+      const postCheckpoint = performance.now();
+
+      // Print checkpoint size
+      const size = await getParsedFileSize(exportLocation);
+      this.#logger.log("checkpoint archive created", { size, options });
+
+      // Create image from checkpoint
+      const container = this.#logger.debug(await $$`buildah from scratch`);
+      const postFrom = performance.now();
+
+      this.#logger.debug(await $$`buildah add ${container} ${exportLocation} /`);
+      const postAdd = performance.now();
+
+      this.#logger.debug(
+        await $$`buildah config --annotation=io.kubernetes.cri-o.annotations.checkpoint.name=counter ${container}`
+      );
+      const postConfig = performance.now();
+
+      this.#logger.debug(await $$`buildah commit ${container} ${imageRef}`);
+      const postCommit = performance.now();
+
+      this.#logger.debug(await $$`buildah rm ${container}`);
+      const postRm = performance.now();
+
+      if (this.simulatePushFailure) {
+        if (performance.now() < this.simulatePushFailureSeconds * 1000) {
+          this.#logger.error("Simulating push failure", { options });
+          throw new Error("SIMULATE_PUSH_FAILURE");
+        }
+      }
+
+      // Push checkpoint image
+      this.#logger.debug(
+        await $$`buildah push --tls-verify=${String(this.registryTlsVerify)} ${imageRef}`
+      );
+      const postPush = performance.now();
+
+      const perf = {
+        "crictl checkpoint": postCheckpoint - start,
+        "buildah from": postFrom - postCheckpoint,
+        "buildah add": postAdd - postFrom,
+        "buildah config": postConfig - postAdd,
+        "buildah commit": postCommit - postConfig,
+        "buildah rm": postRm - postCommit,
+        "buildah push": postPush - postRm,
+      };
+
+      this.#logger.log("Checkpointed and pushed image to:", { location: imageRef, perf });
+
+      return {
+        success: true,
+        checkpoint: {
+          location: imageRef,
+          docker: false,
+        },
+      };
+    } catch (error) {
+      if (isExecaChildProcess(error)) {
+        if (error.isCanceled) {
+          this.#logger.error("Checkpoint canceled", { options, error });
+
+          return { success: false, reason: "CANCELED" };
+        }
+
+        this.#logger.error("Checkpoint command error", { options, error });
+
+        return { success: false, reason: "ERROR" };
+      }
+
+      this.#logger.error("Unhandled checkpoint error", { options, error });
+
+      return { success: false, reason: "ERROR" };
+    } finally {
+      this.#abortControllers.delete(runId);
+      await cleanup();
+    }
+  }
+
+  #failCheckpoint(runId: string, error: unknown) {
+    this.#failedCheckpoints.set(runId, error);
+  }
+
+  #clearFailedCheckpoint(runId: string) {
+    this.#failedCheckpoints.delete(runId);
+  }
+
+  #hasFailedCheckpoint(runId: string) {
+    return this.#failedCheckpoints.has(runId);
+  }
+
+  #getRunContainerName(suffix: string, attemptNumber?: number) {
+    return `task-run-${suffix}${attemptNumber && attemptNumber > 1 ? `-att${attemptNumber}` : ""}`;
+  }
+}

apps/docker-provider/package.json

@@ -18,8 +18,7 @@
   "dependencies": {
     "@trigger.dev/core": "workspace:*",
     "@trigger.dev/core-apps": "workspace:*",
-    "execa": "^8.0.1",
-    "socket.io-client": "^4.7.4"
+    "execa": "^8.0.1"
   },
   "devDependencies": {
     "@types/node": "^18.19.8",

apps/docker-provider/src/index.ts

@@ -1,14 +1,13 @@
 import { $, type ExecaChildProcess, execa } from "execa";
 import {
-  SimpleLogger,
-  TaskOperations,
   ProviderShell,
-  TaskOperationsRestoreOptions,
+  TaskOperations,
   TaskOperationsCreateOptions,
   TaskOperationsIndexOptions,
-  isExecaChildProcess,
-  testDockerCheckpoint,
-} from "@trigger.dev/core-apps";
+  TaskOperationsRestoreOptions,
+} from "@trigger.dev/core-apps/provider";
+import { SimpleLogger } from "@trigger.dev/core-apps/logger";
+import { isExecaChildProcess, testDockerCheckpoint } from "@trigger.dev/core-apps/checkpoints";
 import { setTimeout } from "node:timers/promises";
 import { PostStartCauses, PreStopCauses } from "@trigger.dev/core/v3";
 
@@ -54,13 +53,16 @@ class DockerTaskOperations implements TaskOperations {
   }
 
   #getInitReturn(canCheckpoint: boolean): TaskOperationsInitReturn {
-    this.#initialized = true;
     this.#canCheckpoint = canCheckpoint;
 
     if (canCheckpoint) {
-      logger.log("Full checkpoint support!");
+      if (!this.#initialized) {
+        logger.log("Full checkpoint support!");
+      }
     }
 
+    this.#initialized = true;
+
     const willSimulate = !canCheckpoint || this.opts.forceSimulate;
 
     if (willSimulate) {
@@ -107,24 +109,29 @@ class DockerTaskOperations implements TaskOperations {
   async create(opts: TaskOperationsCreateOptions) {
     await this.init();
 
-    const containerName = this.#getRunContainerName(opts.runId);
+    const containerName = this.#getRunContainerName(opts.runId, opts.nextAttemptNumber);
+
+    const runArgs = [
+      "run",
+      "--network=host",
+      "--detach",
+      `--env=TRIGGER_ENV_ID=${opts.envId}`,
+      `--env=TRIGGER_RUN_ID=${opts.runId}`,
+      `--env=OTEL_EXPORTER_OTLP_ENDPOINT=${OTEL_EXPORTER_OTLP_ENDPOINT}`,
+      `--env=POD_NAME=${containerName}`,
+      `--env=COORDINATOR_HOST=${COORDINATOR_HOST}`,
+      `--env=COORDINATOR_PORT=${COORDINATOR_PORT}`,
+      `--name=${containerName}`,
+    ];
+
+    if (process.env.ENFORCE_MACHINE_PRESETS) {
+      runArgs.push(`--cpus=${opts.machine.cpu}`, `--memory=${opts.machine.memory}G`);
+    }
+
+    runArgs.push(`${opts.image}`);
 
     try {
-      logger.debug(
-        await execa("docker", [
-          "run",
-          "--network=host",
-          "--detach",
-          `--env=TRIGGER_ENV_ID=${opts.envId}`,
-          `--env=TRIGGER_RUN_ID=${opts.runId}`,
-          `--env=OTEL_EXPORTER_OTLP_ENDPOINT=${OTEL_EXPORTER_OTLP_ENDPOINT}`,
-          `--env=POD_NAME=${containerName}`,
-          `--env=COORDINATOR_HOST=${COORDINATOR_HOST}`,
-          `--env=COORDINATOR_PORT=${COORDINATOR_PORT}`,
-          `--name=${containerName}`,
-          `${opts.image}`,
-        ])
-      );
+      logger.debug(await execa("docker", runArgs));
     } catch (error) {
       if (!isExecaChildProcess(error)) {
         throw error;
@@ -143,7 +150,7 @@ class DockerTaskOperations implements TaskOperations {
   async restore(opts: TaskOperationsRestoreOptions) {
     await this.init();
 
-    const containerName = this.#getRunContainerName(opts.runId);
+    const containerName = this.#getRunContainerName(opts.runId, opts.attemptNumber);
 
     if (!this.#canCheckpoint || this.opts.forceSimulate) {
       logger.log("Simulating restore");
@@ -188,8 +195,8 @@ class DockerTaskOperations implements TaskOperations {
     return `task-index-${suffix}`;
   }
 
-  #getRunContainerName(suffix: string) {
-    return `task-run-${suffix}`;
+  #getRunContainerName(suffix: string, attemptNumber?: number) {
+    return `task-run-${suffix}${attemptNumber && attemptNumber > 1 ? `-att${attemptNumber}` : ""}`;
   }
 
   async #sendPostStart(containerName: string): Promise<void> {

apps/docker-provider/tsconfig.json

@@ -1,6 +1,6 @@
 {
   "compilerOptions": {
-    "target": "es2016",
+    "target": "es2018",
     "module": "commonjs",
     "esModuleInterop": true,
     "forceConsistentCasingInFileNames": true,

apps/kubernetes-provider/package.json

@@ -19,8 +19,7 @@
     "@kubernetes/client-node": "^0.20.0",
     "@trigger.dev/core": "workspace:*",
     "@trigger.dev/core-apps": "workspace:*",
-    "p-queue": "^8.0.1",
-    "socket.io-client": "^4.7.4"
+    "p-queue": "^8.0.1"
   },
   "devDependencies": {
     "dotenv": "^16.4.2",

apps/kubernetes-provider/src/index.ts

@@ -1,12 +1,13 @@
 import * as k8s from "@kubernetes/client-node";
 import {
   ProviderShell,
-  SimpleLogger,
   TaskOperations,
   TaskOperationsCreateOptions,
   TaskOperationsIndexOptions,
+  TaskOperationsPrePullDeploymentOptions,
   TaskOperationsRestoreOptions,
-} from "@trigger.dev/core-apps";
+} from "@trigger.dev/core-apps/provider";
+import { SimpleLogger } from "@trigger.dev/core-apps/logger";
 import {
   MachinePreset,
   PostStartCauses,
@@ -16,13 +17,21 @@ import {
 import { randomUUID } from "crypto";
 import { TaskMonitor } from "./taskMonitor";
 import { PodCleaner } from "./podCleaner";
+import { UptimeHeartbeat } from "./uptimeHeartbeat";
 
 const RUNTIME_ENV = process.env.KUBERNETES_PORT ? "kubernetes" : "local";
 const NODE_NAME = process.env.NODE_NAME || "local";
 const OTEL_EXPORTER_OTLP_ENDPOINT =
   process.env.OTEL_EXPORTER_OTLP_ENDPOINT ?? "http://0.0.0.0:4318";
+
 const POD_CLEANER_INTERVAL_SECONDS = Number(process.env.POD_CLEANER_INTERVAL_SECONDS || "300");
 
+const UPTIME_HEARTBEAT_URL = process.env.UPTIME_HEARTBEAT_URL;
+const UPTIME_INTERVAL_SECONDS = Number(process.env.UPTIME_INTERVAL_SECONDS || "60");
+const UPTIME_MAX_PENDING_RUNS = Number(process.env.UPTIME_MAX_PENDING_RUNS || "25");
+const UPTIME_MAX_PENDING_INDECES = Number(process.env.UPTIME_MAX_PENDING_INDECES || "10");
+const UPTIME_MAX_PENDING_ERRORS = Number(process.env.UPTIME_MAX_PENDING_ERRORS || "10");
+
 const logger = new SimpleLogger(`[${NODE_NAME}]`);
 logger.log(`running in ${RUNTIME_ENV} mode`);
 
@@ -41,6 +50,7 @@ class KubernetesTaskOperations implements TaskOperations {
   #k8sApi: {
     core: k8s.CoreV1Api;
     batch: k8s.BatchV1Api;
+    apps: k8s.AppsV1Api;
   };
 
   constructor(namespace = "default") {
@@ -92,7 +102,7 @@ class KubernetesTaskOperations implements TaskOperations {
                   resources: {
                     limits: {
                       cpu: "1",
-                      memory: "1G",
+                      memory: "2G",
                       "ephemeral-storage": "2Gi",
                     },
                   },
@@ -129,10 +139,12 @@ class KubernetesTaskOperations implements TaskOperations {
   }
 
   async create(opts: TaskOperationsCreateOptions) {
+    const containerName = this.#getRunContainerName(opts.runId, opts.nextAttemptNumber);
+
     await this.#createPod(
       {
         metadata: {
-          name: this.#getRunContainerName(opts.runId),
+          name: containerName,
           namespace: this.#namespace.metadata.name,
           labels: {
             ...this.#getSharedLabels(opts),
@@ -147,7 +159,7 @@ class KubernetesTaskOperations implements TaskOperations {
           terminationGracePeriodSeconds: 60 * 60,
           containers: [
             {
-              name: this.#getRunContainerName(opts.runId),
+              name: containerName,
               image: opts.image,
               ports: [
                 {
@@ -201,7 +213,7 @@ class KubernetesTaskOperations implements TaskOperations {
     await this.#createPod(
       {
         metadata: {
-          name: `${this.#getRunContainerName(opts.runId)}-${randomUUID().slice(0, 8)}`,
+          name: `${this.#getRunContainerName(opts.runId)}-${opts.checkpointId.slice(-8)}`,
           namespace: this.#namespace.metadata.name,
           labels: {
             ...this.#getSharedLabels(opts),
@@ -305,6 +317,72 @@ class KubernetesTaskOperations implements TaskOperations {
     await this.#getPod(opts.runId, this.#namespace);
   }
 
+  async prePullDeployment(opts: TaskOperationsPrePullDeploymentOptions) {
+    const metaName = this.#getPrePullContainerName(opts.shortCode);
+
+    const metaLabels = {
+      ...this.#getSharedLabels(opts),
+      app: "task-prepull",
+      "app.kubernetes.io/part-of": "trigger-worker",
+      "app.kubernetes.io/component": "prepull",
+      deployment: opts.deploymentId,
+      name: metaName,
+    } satisfies k8s.V1ObjectMeta["labels"];
+
+    await this.#createDaemonSet(
+      {
+        metadata: {
+          name: metaName,
+          namespace: this.#namespace.metadata.name,
+          labels: metaLabels,
+        },
+        spec: {
+          selector: {
+            matchLabels: {
+              name: metaName,
+            },
+          },
+          template: {
+            metadata: {
+              labels: metaLabels,
+            },
+            spec: {
+              ...this.#defaultPodSpec,
+              restartPolicy: "Always",
+              initContainers: [
+                {
+                  name: "prepull",
+                  image: opts.imageRef,
+                  command: ["/usr/bin/true"],
+                  resources: {
+                    limits: {
+                      cpu: "0.25",
+                      memory: "100Mi",
+                      "ephemeral-storage": "1Gi",
+                    },
+                  },
+                },
+              ],
+              containers: [
+                {
+                  name: "pause",
+                  image: "registry.k8s.io/pause:3.9",
+                  resources: {
+                    limits: {
+                      cpu: "1m",
+                      memory: "12Mi",
+                    },
+                  },
+                },
+              ],
+            },
+          },
+        },
+      },
+      this.#namespace
+    );
+  }
+
   #envTypeToLabelValue(type: EnvironmentType) {
     switch (type) {
       case "PRODUCTION":
@@ -394,7 +472,11 @@ class KubernetesTaskOperations implements TaskOperations {
   }
 
   #getSharedLabels(
-    opts: TaskOperationsIndexOptions | TaskOperationsCreateOptions | TaskOperationsRestoreOptions
+    opts:
+      | TaskOperationsIndexOptions
+      | TaskOperationsCreateOptions
+      | TaskOperationsRestoreOptions
+      | TaskOperationsPrePullDeploymentOptions
   ): Record<string, string> {
     return {
       env: opts.envId,
@@ -434,8 +516,12 @@ class KubernetesTaskOperations implements TaskOperations {
     return `task-index-${suffix}`;
   }
 
-  #getRunContainerName(suffix: string) {
-    return `task-run-${suffix}`;
+  #getRunContainerName(suffix: string, attemptNumber?: number) {
+    return `task-run-${suffix}${attemptNumber && attemptNumber > 1 ? `-att${attemptNumber}` : ""}`;
+  }
+
+  #getPrePullContainerName(suffix: string) {
+    return `task-prepull-${suffix}`;
   }
 
   #createK8sApi() {
@@ -452,6 +538,7 @@ class KubernetesTaskOperations implements TaskOperations {
     return {
       core: kubeConfig.makeApiClient(k8s.CoreV1Api),
       batch: kubeConfig.makeApiClient(k8s.BatchV1Api),
+      apps: kubeConfig.makeApiClient(k8s.AppsV1Api),
     };
   }
 
@@ -495,6 +582,18 @@ class KubernetesTaskOperations implements TaskOperations {
     }
   }
 
+  async #createDaemonSet(daemonSet: k8s.V1DaemonSet, namespace: Namespace) {
+    try {
+      const res = await this.#k8sApi.apps.createNamespacedDaemonSet(
+        namespace.metadata.name,
+        daemonSet
+      );
+      logger.debug(res.body);
+    } catch (err: unknown) {
+      this.#handleK8sError(err);
+    }
+  }
+
   #throwUnlessRecord(candidate: unknown): asserts candidate is Record<string, unknown> {
     if (typeof candidate !== "object" || candidate === null) {
       throw candidate;
@@ -529,27 +628,28 @@ provider.listen();
 
 const taskMonitor = new TaskMonitor({
   runtimeEnv: RUNTIME_ENV,
-  onIndexFailure: async (deploymentId, failureInfo) => {
-    logger.log("Indexing failed", { deploymentId, failureInfo });
+  onIndexFailure: async (deploymentId, details) => {
+    logger.log("Indexing failed", { deploymentId, details });
 
     try {
       provider.platformSocket.send("INDEXING_FAILED", {
         deploymentId,
         error: {
-          name: `Crashed with exit code ${failureInfo.exitCode}`,
-          message: failureInfo.reason,
-          stack: failureInfo.logs,
+          name: `Crashed with exit code ${details.exitCode}`,
+          message: details.reason,
+          stack: details.logs,
         },
+        overrideCompletion: details.overrideCompletion,
       });
     } catch (error) {
       logger.error(error);
     }
   },
-  onRunFailure: async (runId, failureInfo) => {
-    logger.log("Run failed:", { runId, failureInfo });
+  onRunFailure: async (runId, details) => {
+    logger.log("Run failed:", { runId, details });
 
     try {
-      provider.platformSocket.send("WORKER_CRASHED", { runId, ...failureInfo });
+      provider.platformSocket.send("WORKER_CRASHED", { runId, ...details });
     } catch (error) {
       logger.error(error);
     }
@@ -565,3 +665,19 @@ const podCleaner = new PodCleaner({
 });
 
 podCleaner.start();
+
+if (UPTIME_HEARTBEAT_URL) {
+  const uptimeHeartbeat = new UptimeHeartbeat({
+    runtimeEnv: RUNTIME_ENV,
+    namespace: "default",
+    intervalInSeconds: UPTIME_INTERVAL_SECONDS,
+    pingUrl: UPTIME_HEARTBEAT_URL,
+    maxPendingRuns: UPTIME_MAX_PENDING_RUNS,
+    maxPendingIndeces: UPTIME_MAX_PENDING_INDECES,
+    maxPendingErrors: UPTIME_MAX_PENDING_ERRORS,
+  });
+
+  uptimeHeartbeat.start();
+} else {
+  logger.log("Uptime heartbeat is disabled, set UPTIME_HEARTBEAT_URL to enable.");
+}

apps/kubernetes-provider/src/podCleaner.ts

@@ -1,5 +1,5 @@
 import * as k8s from "@kubernetes/client-node";
-import { SimpleLogger } from "@trigger.dev/core-apps";
+import { SimpleLogger } from "@trigger.dev/core-apps/logger";
 
 type PodCleanerOptions = {
   runtimeEnv: "local" | "kubernetes";
@@ -15,6 +15,7 @@ export class PodCleaner {
   private logger = new SimpleLogger("[PodCleaner]");
   private k8sClient: {
     core: k8s.CoreV1Api;
+    apps: k8s.AppsV1Api;
     kubeConfig: k8s.KubeConfig;
   };
 
@@ -43,6 +44,7 @@ export class PodCleaner {
 
     return {
       core: kubeConfig.makeApiClient(k8s.CoreV1Api),
+      apps: kubeConfig.makeApiClient(k8s.AppsV1Api),
       kubeConfig: kubeConfig,
     };
   }
@@ -98,6 +100,25 @@ export class PodCleaner {
       .catch(this.#handleK8sError.bind(this));
   }
 
+  async #deleteDaemonSets(opts: {
+    namespace: string;
+    dryRun?: boolean;
+    fieldSelector?: string;
+    labelSelector?: string;
+  }) {
+    return await this.k8sClient.apps
+      .deleteCollectionNamespacedDaemonSet(
+        opts.namespace,
+        undefined, // pretty
+        undefined, // continue
+        opts.dryRun ? "All" : undefined,
+        opts.fieldSelector,
+        undefined, // gracePeriodSeconds
+        opts.labelSelector
+      )
+      .catch(this.#handleK8sError.bind(this));
+  }
+
   async #deleteCompletedRuns() {
     this.logger.log("Deleting completed runs");
 
@@ -152,6 +173,28 @@ export class PodCleaner {
     });
   }
 
+  async #deleteCompletedPrePulls() {
+    this.logger.log("Deleting completed pre-pulls");
+
+    const start = Date.now();
+
+    const result = await this.#deleteDaemonSets({
+      namespace: this.namespace,
+      labelSelector: "app=task-prepull",
+    });
+
+    const elapsedMs = Date.now() - start;
+
+    if (!result) {
+      this.logger.log("Deleting completed pre-pulls: No delete result", { elapsedMs });
+      return;
+    }
+
+    const total = (result.response as any)?.body?.items?.length ?? 0;
+
+    this.logger.log("Deleting completed pre-pulls: Done", { total, elapsedMs });
+  }
+
   async start() {
     this.enabled = true;
     this.logger.log("Starting");
@@ -186,6 +229,22 @@ export class PodCleaner {
       2 * this.intervalInSeconds * 1000
     );
 
+    const completedPrePullInterval = setInterval(
+      async () => {
+        if (!this.enabled) {
+          clearInterval(completedPrePullInterval);
+          return;
+        }
+
+        try {
+          await this.#deleteCompletedPrePulls();
+        } catch (error) {
+          this.logger.error("Error deleting completed pre-pulls", error);
+        }
+      },
+      2 * this.intervalInSeconds * 1000
+    );
+
     // this.#launchTests();
   }
 

apps/kubernetes-provider/src/taskMonitor.ts

@@ -1,25 +1,20 @@
 import * as k8s from "@kubernetes/client-node";
-import { SimpleLogger } from "@trigger.dev/core-apps";
+import { SimpleLogger } from "@trigger.dev/core-apps/logger";
+import { EXIT_CODE_ALREADY_HANDLED, EXIT_CODE_CHILD_NONZERO } from "@trigger.dev/core-apps/process";
 import { setTimeout } from "timers/promises";
 import PQueue from "p-queue";
+import type { Prettify } from "@trigger.dev/core/v3";
 
-type IndexFailureHandler = (
-  deploymentId: string,
-  failureInfo: {
-    exitCode: number;
-    reason: string;
-    logs: string;
-  }
-) => Promise<any>;
+type FailureDetails = Prettify<{
+  exitCode: number;
+  reason: string;
+  logs: string;
+  overrideCompletion: boolean;
+}>;
 
-type RunFailureHandler = (
-  runId: string,
-  failureInfo: {
-    exitCode: number;
-    reason: string;
-    logs: string;
-  }
-) => Promise<any>;
+type IndexFailureHandler = (deploymentId: string, details: FailureDetails) => Promise<any>;
+
+type RunFailureHandler = (runId: string, details: FailureDetails) => Promise<any>;
 
 type TaskMonitorOptions = {
   runtimeEnv: "local" | "kubernetes";
@@ -144,8 +139,10 @@ export class TaskMonitor {
     const containerState = this.#getContainerStateSummary(containerStatus.state);
     const exitCode = containerState.exitCode ?? -1;
 
-    // We use this special exit code to signal any errors were already handled elsewhere
-    if (exitCode === 111) {
+    if (exitCode === EXIT_CODE_ALREADY_HANDLED) {
+      this.#logger.debug("Ignoring pod failure, already handled by worker", {
+        podName,
+      });
       return;
     }
 
@@ -162,6 +159,7 @@ export class TaskMonitor {
 
     let reason = rawReason || "Unknown error";
     let logs = rawLogs || "";
+    let overrideCompletion = false;
 
     switch (rawReason) {
       case "Error":
@@ -181,7 +179,10 @@ export class TaskMonitor {
         }
         break;
       case "OOMKilled":
-        reason = "Out of memory! Try increasing the memory on this task.";
+        overrideCompletion = true;
+        reason = `${
+          exitCode === EXIT_CODE_CHILD_NONZERO ? "Child process" : "Parent process"
+        } ran out of memory! Try choosing a machine preset with more memory for this task.`;
         break;
       default:
         break;
@@ -191,7 +192,8 @@ export class TaskMonitor {
       exitCode,
       reason,
       logs,
-    };
+      overrideCompletion,
+    } satisfies FailureDetails;
 
     const app = pod.metadata?.labels?.app;
 

apps/kubernetes-provider/src/uptimeHeartbeat.ts

@@ -0,0 +1,272 @@
+import * as k8s from "@kubernetes/client-node";
+import { SimpleLogger } from "@trigger.dev/core-apps/logger";
+
+type UptimeHeartbeatOptions = {
+  runtimeEnv: "local" | "kubernetes";
+  pingUrl: string;
+  namespace?: string;
+  intervalInSeconds?: number;
+  maxPendingRuns?: number;
+  maxPendingIndeces?: number;
+  maxPendingErrors?: number;
+  leadingEdge?: boolean;
+};
+
+export class UptimeHeartbeat {
+  private enabled = false;
+  private namespace: string;
+
+  private intervalInSeconds: number;
+  private maxPendingRuns: number;
+  private maxPendingIndeces: number;
+  private maxPendingErrors: number;
+
+  private leadingEdge = true;
+
+  private logger = new SimpleLogger("[UptimeHeartbeat]");
+  private k8sClient: {
+    core: k8s.CoreV1Api;
+    kubeConfig: k8s.KubeConfig;
+  };
+
+  constructor(private opts: UptimeHeartbeatOptions) {
+    this.namespace = opts.namespace ?? "default";
+
+    this.intervalInSeconds = opts.intervalInSeconds ?? 60;
+    this.maxPendingRuns = opts.maxPendingRuns ?? 25;
+    this.maxPendingIndeces = opts.maxPendingIndeces ?? 10;
+    this.maxPendingErrors = opts.maxPendingErrors ?? 10;
+
+    this.k8sClient = this.#createK8sClient();
+  }
+
+  #createK8sClient() {
+    const kubeConfig = new k8s.KubeConfig();
+
+    if (this.opts.runtimeEnv === "local") {
+      kubeConfig.loadFromDefault();
+    } else if (this.opts.runtimeEnv === "kubernetes") {
+      kubeConfig.loadFromCluster();
+    } else {
+      throw new Error(`Unsupported runtime environment: ${this.opts.runtimeEnv}`);
+    }
+
+    return {
+      core: kubeConfig.makeApiClient(k8s.CoreV1Api),
+      kubeConfig: kubeConfig,
+    };
+  }
+
+  #isRecord(candidate: unknown): candidate is Record<string, unknown> {
+    if (typeof candidate !== "object" || candidate === null) {
+      return false;
+    } else {
+      return true;
+    }
+  }
+
+  #logK8sError(err: unknown, debugOnly = false) {
+    if (debugOnly) {
+      this.logger.debug("K8s API Error", err);
+    } else {
+      this.logger.error("K8s API Error", err);
+    }
+  }
+
+  #handleK8sError(err: unknown) {
+    if (!this.#isRecord(err) || !this.#isRecord(err.body)) {
+      this.#logK8sError(err);
+      return;
+    }
+
+    this.#logK8sError(err, true);
+
+    if (typeof err.body.message === "string") {
+      this.#logK8sError({ message: err.body.message });
+      return;
+    }
+
+    this.#logK8sError({ body: err.body });
+  }
+
+  async #getPods(opts: {
+    namespace: string;
+    fieldSelector?: string;
+    labelSelector?: string;
+  }): Promise<Array<k8s.V1Pod> | undefined> {
+    const listReturn = await this.k8sClient.core
+      .listNamespacedPod(
+        opts.namespace,
+        undefined, // pretty
+        undefined, // allowWatchBookmarks
+        undefined, // _continue
+        opts.fieldSelector,
+        opts.labelSelector,
+        this.maxPendingRuns * 2, // limit
+        undefined, // resourceVersion
+        undefined, // resourceVersionMatch
+        undefined, // sendInitialEvents
+        this.intervalInSeconds, // timeoutSeconds,
+        undefined // watch
+      )
+      .catch(this.#handleK8sError.bind(this));
+
+    return listReturn?.body.items;
+  }
+
+  async #getPendingIndeces(): Promise<Array<k8s.V1Pod> | undefined> {
+    return await this.#getPods({
+      namespace: this.namespace,
+      fieldSelector: "status.phase=Pending",
+      labelSelector: "app=task-index",
+    });
+  }
+
+  async #getPendingTasks(): Promise<Array<k8s.V1Pod> | undefined> {
+    return await this.#getPods({
+      namespace: this.namespace,
+      fieldSelector: "status.phase=Pending",
+      labelSelector: "app=task-run",
+    });
+  }
+
+  #countPods(pods: Array<k8s.V1Pod>): number {
+    return pods.length;
+  }
+
+  #filterPendingPods(
+    pods: Array<k8s.V1Pod>,
+    waitingReason: "CreateContainerError" | "RunContainerError"
+  ): Array<k8s.V1Pod> {
+    return pods.filter((pod) => {
+      const containerStatus = pod.status?.containerStatuses?.[0];
+      return containerStatus?.state?.waiting?.reason === waitingReason;
+    });
+  }
+
+  async #sendPing() {
+    this.logger.log("Sending ping");
+
+    const start = Date.now();
+    const controller = new AbortController();
+
+    const timeoutMs = (this.intervalInSeconds * 1000) / 2;
+
+    const fetchTimeout = setTimeout(() => {
+      controller.abort();
+    }, timeoutMs);
+
+    try {
+      const response = await fetch(this.opts.pingUrl, {
+        signal: controller.signal,
+      });
+
+      if (!response.ok) {
+        this.logger.error("Failed to send ping, response not OK", {
+          status: response.status,
+        });
+        return;
+      }
+
+      const elapsedMs = Date.now() - start;
+      this.logger.log("Ping sent", { elapsedMs });
+    } catch (error) {
+      if (error instanceof DOMException && error.name === "AbortError") {
+        this.logger.log("Ping timeout", { timeoutSeconds: timeoutMs });
+        return;
+      }
+
+      this.logger.error("Failed to send ping", error);
+    } finally {
+      clearTimeout(fetchTimeout);
+    }
+  }
+
+  async #heartbeat() {
+    this.logger.log("Performing heartbeat");
+
+    const start = Date.now();
+
+    const pendingTasks = await this.#getPendingTasks();
+
+    if (!pendingTasks) {
+      this.logger.error("Failed to get pending tasks");
+      return;
+    }
+
+    const totalPendingTasks = this.#countPods(pendingTasks);
+
+    const pendingIndeces = await this.#getPendingIndeces();
+
+    if (!pendingIndeces) {
+      this.logger.error("Failed to get pending indeces");
+      return;
+    }
+
+    const totalPendingIndeces = this.#countPods(pendingIndeces);
+
+    const elapsedMs = Date.now() - start;
+
+    this.logger.log("Finished heartbeat checks", { elapsedMs });
+
+    if (totalPendingTasks > this.maxPendingRuns) {
+      this.logger.log("Too many pending tasks, skipping heartbeat", { totalPendingTasks });
+      return;
+    }
+
+    if (totalPendingIndeces > this.maxPendingIndeces) {
+      this.logger.log("Too many pending indeces, skipping heartbeat", { totalPendingIndeces });
+      return;
+    }
+
+    const totalCreateContainerErrors = this.#countPods(
+      this.#filterPendingPods(pendingTasks, "CreateContainerError")
+    );
+    const totalRunContainerErrors = this.#countPods(
+      this.#filterPendingPods(pendingTasks, "RunContainerError")
+    );
+
+    if (totalCreateContainerErrors + totalRunContainerErrors > this.maxPendingErrors) {
+      this.logger.log("Too many pending tasks with errors, skipping heartbeat", {
+        totalRunContainerErrors,
+        totalCreateContainerErrors,
+      });
+      return;
+    }
+
+    await this.#sendPing();
+
+    this.logger.log("Heartbeat done", { totalPendingTasks, elapsedMs });
+  }
+
+  async start() {
+    this.enabled = true;
+    this.logger.log("Starting");
+
+    if (this.leadingEdge) {
+      await this.#heartbeat();
+    }
+
+    const heartbeat = setInterval(async () => {
+      if (!this.enabled) {
+        clearInterval(heartbeat);
+        return;
+      }
+
+      try {
+        await this.#heartbeat();
+      } catch (error) {
+        this.logger.error("Error while heartbeating", error);
+      }
+    }, this.intervalInSeconds * 1000);
+  }
+
+  async stop() {
+    if (!this.enabled) {
+      return;
+    }
+
+    this.enabled = false;
+    this.logger.log("Shutting down..");
+  }
+}

apps/webapp/.eslintrc

@@ -1,13 +1,36 @@
 {
-  "extends": ["@remix-run/eslint-config", "@remix-run/eslint-config/node", "prettier"],
-  "rules": {
-    "@typescript-eslint/strict-boolean-expressions": [
-      "error",
-      {
-        "allowNullableBoolean": true,
-        "allowNullableString": true,
-        "allowNullableNumber": true
+  "plugins": [
+    "@trigger.dev/eslint-plugin",
+    "react-hooks",
+    "@typescript-eslint/eslint-plugin",
+    "import"
+  ],
+  "parser": "@typescript-eslint/parser",
+  "overrides": [
+    {
+      "files": ["*.ts", "*.tsx"],
+      "rules": {
+        // Autofixes imports from "@trigger.dev/core" to fine grained modules
+        "@trigger.dev/no-trigger-core-import": "error",
+        // Normalize `import type {}` and `import { type }`
+        "@typescript-eslint/consistent-type-imports": [
+          "warn",
+          {
+            // the "type" annotation can get tangled and cause syntax errors
+            // during some autofixes, so easier to just turn it off
+            "prefer": "type-imports",
+            "disallowTypeAnnotations": true,
+            "fixStyle": "inline-type-imports"
+          }
+        ],
+        // no-trigger-core-import splits imports into multiple lines
+        // this one merges them back into a single line
+        // if they still import from the same module
+        "import/no-duplicates": ["warn", { "prefer-inline": true }],
+        // lots of undeclared vars, enable this rule if you want to clean them up
+        "turbo/no-undeclared-env-vars": "off"
       }
-    ]
-  }
+    }
+  ],
+  "ignorePatterns": ["seed.js", "seedCloud.ts", "populate.js"]
 }

apps/webapp/app/components/DefinitionTooltip.tsx

@@ -15,11 +15,11 @@ export function DefinitionTip({
     <TooltipProvider>
       <Tooltip disableHoverableContent>
         <TooltipTrigger>
-          <span className="underline decoration-charcoal-600 decoration-dashed underline-offset-4 transition hover:decoration-charcoal-500">
+          <span className="cursor-default underline decoration-charcoal-500 decoration-dashed underline-offset-4 transition hover:decoration-charcoal-400">
             {children}
           </span>
         </TooltipTrigger>
-        <TooltipContent align="end" side="right" variant="dark" className="w-[16rem] min-w-[16rem]">
+        <TooltipContent align="end" side="right" className="w-[16rem] min-w-[16rem]">
           <Header3 className="mb-1">{title}</Header3>
           {typeof content === "string" ? (
             <Paragraph variant="small">{content}</Paragraph>

apps/webapp/app/components/Feedback.tsx

@@ -1,19 +1,25 @@
 import { conform, useForm } from "@conform-to/react";
 import { parse } from "@conform-to/zod";
 import { BookOpenIcon } from "@heroicons/react/20/solid";
-import { ChevronRightIcon } from "@heroicons/react/24/solid";
+import {
+  CalendarDaysIcon,
+  ChevronRightIcon,
+  EnvelopeIcon,
+  LifebuoyIcon,
+  LightBulbIcon,
+} from "@heroicons/react/24/solid";
 import { Form, useActionData, useLocation, useNavigation } from "@remix-run/react";
-import { DiscordIcon, GitHubLightIcon } from "@trigger.dev/companyicons";
+import { DiscordIcon } from "@trigger.dev/companyicons";
 import { ActivityIcon } from "lucide-react";
-import { ReactNode, useState } from "react";
-import { FeedbackType, feedbackTypeLabel, schema } from "~/routes/resources.feedback";
+import { type ReactNode, useState } from "react";
+import { type FeedbackType, feedbackTypeLabel, schema } from "~/routes/resources.feedback";
 import { cn } from "~/utils/cn";
-import { docsPath } from "~/utils/pathBuilder";
+import { docsTroubleshootingPath } from "~/utils/pathBuilder";
 import { Button, LinkButton } from "./primitives/Buttons";
 import { Fieldset } from "./primitives/Fieldset";
 import { FormButtons } from "./primitives/FormButtons";
 import { FormError } from "./primitives/FormError";
-import { Header1, Header2 } from "./primitives/Headers";
+import { Header1 } from "./primitives/Headers";
 import { InputGroup } from "./primitives/InputGroup";
 import { Label } from "./primitives/Label";
 import { Paragraph } from "./primitives/Paragraph";
@@ -55,123 +61,137 @@ export function Feedback({ button, defaultValue = "bug" }: FeedbackProps) {
       <SheetTrigger asChild={true}>{button}</SheetTrigger>
       <SheetContent className="@container">
         <SheetBody className="flex h-full flex-col justify-between">
-          <Header2 className="mb-2.5 text-xl">Get help from the community</Header2>
-          <Paragraph className="mb-4">
-            The quickest way to get help and feedback or to provide advice to others is to join our
-            Discord.
-          </Paragraph>
-          <div className="flex flex-col gap-x-4 @[30rem]:flex-row">
-            <DiscordBanner />
-            <GitHubDiscussionsBanner />
-          </div>
-          <hr className="mb-4" />
-          <Header2 className="mb-2.5 text-xl">Send us an email</Header2>
-          <Paragraph className="mb-4">We read every message and respond quickly.</Paragraph>
-          <Form method="post" action="/resources/feedback" {...form.props}>
-            <Fieldset className="max-w-full gap-y-3">
-              <input value={location.pathname} {...conform.input(path, { type: "hidden" })} />
-              <InputGroup className="max-w-full">
-                <Select
-                  {...conform.select(feedbackType)}
-                  variant="tertiary/medium"
-                  defaultValue={defaultValue}
-                  placeholder="Select type"
-                  text={(value) => feedbackTypeLabel[value]}
-                  dropdownIcon
-                >
-                  {Object.entries(feedbackTypeLabel).map(([name, title]) => (
-                    <SelectItem key={name} value={name}>
-                      {title}
-                    </SelectItem>
-                  ))}
-                </Select>
-                <FormError id={feedbackType.errorId}>{feedbackType.error}</FormError>
-              </InputGroup>
-              <InputGroup className="max-w-full">
-                <Label>Message</Label>
-                <TextArea {...conform.textarea(message)} />
-                <FormError id={message.errorId}>{message.error}</FormError>
-              </InputGroup>
-              <FormError>{form.error}</FormError>
-              <div className="flex w-full justify-end">
-                <FormButtons
-                  className="m-0 w-max"
-                  confirmButton={
-                    <Button type="submit" variant="tertiary/medium">
-                      Send message
-                    </Button>
-                  }
-                />
-              </div>
-            </Fieldset>
-          </Form>
-          <hr className="my-4" />
-          <Header2 className="mb-2.5 text-xl">Troubleshooting</Header2>
-          <Paragraph className="mb-4">
-            If you're having trouble, check out our documentation or the Trigger.dev Status page.
-          </Paragraph>
-          <div className="flex flex-wrap gap-2">
-            <LinkButton to={docsPath("")} variant="tertiary/medium" LeadingIcon={BookOpenIcon}>
-              Docs
-            </LinkButton>
-            <LinkButton
-              to={"https://status.trigger.dev/"}
-              variant="tertiary/medium"
-              LeadingIcon={ActivityIcon}
-            >
-              Trigger.dev Status
-            </LinkButton>
-          </div>
+          <LinkBanner
+            title="Join our Discord community"
+            icon={<DiscordIcon className="size-9" />}
+            to="https://trigger.dev/discord"
+            className="hover:border-text-link"
+          >
+            <Paragraph>The quickest way to get answers from the Trigger.dev community.</Paragraph>
+          </LinkBanner>
+          <LinkBanner
+            title="Book a 15 min chat with the founders"
+            icon={<CalendarDaysIcon className="size-9 text-green-500" />}
+            to="https://cal.com/team/triggerdotdev/founders-call"
+            className="hover:border-green-500"
+          >
+            <Paragraph>Have a question or want to chat? Book a time to talk with us.</Paragraph>
+          </LinkBanner>
+          <LinkBanner
+            title="Suggest a feature"
+            icon={<LightBulbIcon className="size-9 text-sun-500" />}
+            to="https://feedback.trigger.dev/"
+            className="hover:border-sun-400"
+          >
+            <Paragraph>Have an idea for a new feature or improvement? Let us know!</Paragraph>
+          </LinkBanner>
+          <LinkBanner
+            title="Troubleshooting"
+            icon={<LifebuoyIcon className="size-9 text-rose-500" />}
+          >
+            <Paragraph>
+              If you're having trouble, check out our troubleshooting guide or the Trigger.dev
+              Status page.
+            </Paragraph>
+            <div className="flex flex-wrap gap-2">
+              <LinkButton
+                to={docsTroubleshootingPath("")}
+                variant="tertiary/medium"
+                LeadingIcon={BookOpenIcon}
+              >
+                Troubleshooting Docs
+              </LinkButton>
+              <LinkButton
+                to={"https://status.trigger.dev/"}
+                variant="tertiary/medium"
+                LeadingIcon={ActivityIcon}
+              >
+                Trigger.dev Status
+              </LinkButton>
+            </div>
+          </LinkBanner>
+          <LinkBanner
+            title="Send us an email"
+            icon={<EnvelopeIcon className="size-9 text-blue-500" />}
+          >
+            <Paragraph>We read every message and respond quickly.</Paragraph>
+            <Form method="post" action="/resources/feedback" {...form.props} className="w-full">
+              <Fieldset className="max-w-full gap-y-3">
+                <input value={location.pathname} {...conform.input(path, { type: "hidden" })} />
+                <InputGroup className="max-w-full">
+                  <Select
+                    {...conform.select(feedbackType)}
+                    variant="tertiary/medium"
+                    defaultValue={defaultValue}
+                    placeholder="Select type"
+                    text={(value) => feedbackTypeLabel[value]}
+                    dropdownIcon
+                  >
+                    {Object.entries(feedbackTypeLabel).map(([name, title]) => (
+                      <SelectItem key={name} value={name}>
+                        {title}
+                      </SelectItem>
+                    ))}
+                  </Select>
+                  <FormError id={feedbackType.errorId}>{feedbackType.error}</FormError>
+                </InputGroup>
+                <InputGroup className="max-w-full">
+                  <Label>Message</Label>
+                  <TextArea {...conform.textarea(message)} />
+                  <FormError id={message.errorId}>{message.error}</FormError>
+                </InputGroup>
+                <FormError>{form.error}</FormError>
+                <div className="flex w-full justify-end">
+                  <FormButtons
+                    className="m-0 w-max"
+                    confirmButton={
+                      <Button type="submit" variant="tertiary/medium">
+                        Send message
+                      </Button>
+                    }
+                  />
+                </div>
+              </Fieldset>
+            </Form>
+          </LinkBanner>
         </SheetBody>
       </SheetContent>
     </Sheet>
   );
 }
 
-function DiscordBanner({ className }: { className?: string }) {
+function LinkBanner({
+  className,
+  icon,
+  title,
+  children,
+  to,
+}: {
+  className?: string;
+  icon?: ReactNode;
+  title?: string;
+  children?: ReactNode;
+  to?: string;
+}) {
   return (
     <a
-      href="https://trigger.dev/discord"
+      href={to}
       target="_blank"
       className={cn(
-        "group mb-4 flex w-full items-center justify-between rounded-md border border-charcoal-600 p-4 transition hover:border-text-link",
+        "group/banner mb-4 flex w-full items-center justify-between rounded-md border border-grid-bright bg-charcoal-750 p-4 transition",
         className
       )}
     >
-      <div className="flex flex-col gap-y-2">
-        <DiscordIcon className="h-8 w-8" />
-        <Header1 className="text-2xl font-semibold text-text-bright transition group-hover:text-white">
-          Join our Discord community
-        </Header1>
-        <Paragraph variant="small" className="mb-4">
-          Get help or answer questions from the Trigger.dev community.
-        </Paragraph>
+      <div className="flex w-full items-start gap-4">
+        <span>{icon}</span>
+        <div className="flex w-full flex-col gap-2">
+          <Header1 className="text-2xl font-semibold text-text-bright">{title}</Header1>
+          {children}
+        </div>
       </div>
-      <ChevronRightIcon className="size-5 text-charcoal-500 transition group-hover:translate-x-1 group-hover:text-text-link" />
-    </a>
-  );
-}
-
-function GitHubDiscussionsBanner({ className }: { className?: string }) {
-  return (
-    <a
-      href="https://github.com/triggerdotdev/trigger.dev/discussions"
-      target="_blank"
-      className={cn(
-        "group mb-4 flex w-full items-center justify-between rounded-md border border-charcoal-600 p-4 transition hover:border-text-dimmed",
-        className
+      {to && (
+        <ChevronRightIcon className="size-5 text-charcoal-500 transition group-hover:translate-x-1 group-hover/banner:text-text-bright" />
       )}
-    >
-      <div className="flex flex-col gap-y-2">
-        <GitHubLightIcon className="mb-1 h-7 w-7" />
-        <Header1 className="text-2xl font-semibold text-text-bright transition group-hover:text-white">
-          View our GitHub Discussions
-        </Header1>
-        <Paragraph variant="small">
-          Post your questions, feedback, and feature requests on GitHub.
-        </Paragraph>
-      </div>
-      <ChevronRightIcon className="size-5 text-charcoal-500 transition group-hover:translate-x-1 group-hover:text-text-bright" />
     </a>
   );
 }

apps/webapp/app/components/LoginPageLayout.tsx

@@ -2,7 +2,6 @@ import { useEffect, useState } from "react";
 import { ATAndTLogo } from "~/assets/logos/ATAndTLogo";
 import { AppsmithLogo } from "~/assets/logos/AppsmithLogo";
 import { CalComLogo } from "~/assets/logos/CalComLogo";
-import { OktaLogo } from "~/assets/logos/OktaLogo";
 import { ShopifyLogo } from "~/assets/logos/ShopifyLogo";
 import { VerizonLogo } from "~/assets/logos/VerizonLogo";
 import { LogoType } from "./LogoType";
@@ -75,7 +74,6 @@ export function LoginPageLayout({ children }: { children: React.ReactNode }) {
           <div className="flex w-full flex-wrap items-center justify-center gap-x-6 gap-y-3 text-charcoal-500 xl:justify-between xl:gap-0">
             <VerizonLogo />
             <ShopifyLogo />
-            <OktaLogo />
             <ATAndTLogo />
             <AppsmithLogo />
             <CalComLogo />

apps/webapp/app/components/admin/debugTooltip.tsx

@@ -1,14 +1,16 @@
 import { ShieldCheckIcon } from "@heroicons/react/20/solid";
+import * as Property from "~/components/primitives/PropertyTable";
 import {
   Tooltip,
   TooltipContent,
   TooltipProvider,
   TooltipTrigger,
 } from "~/components/primitives/Tooltip";
-import { useIsImpersonating } from "~/hooks/useOrganizations";
-import { useHasAdminAccess } from "~/hooks/useUser";
+import { useIsImpersonating, useOptionalOrganization } from "~/hooks/useOrganizations";
+import { useOptionalProject } from "~/hooks/useProject";
+import { useHasAdminAccess, useUser } from "~/hooks/useUser";
 
-export function AdminDebugTooltip({ children }: { children: React.ReactNode }) {
+export function AdminDebugTooltip({ children }: { children?: React.ReactNode }) {
   const hasAdminAccess = useHasAdminAccess();
   const isImpersonating = useIsImpersonating();
 
@@ -22,10 +24,46 @@ export function AdminDebugTooltip({ children }: { children: React.ReactNode }) {
         <TooltipTrigger>
           <ShieldCheckIcon className="size-5" />
         </TooltipTrigger>
-        <TooltipContent className="flex max-h-[90vh] items-center gap-1 overflow-y-auto">
-          {children}
+        <TooltipContent className="max-h-[90vh] overflow-y-auto">
+          <Content>{children}</Content>
         </TooltipContent>
       </Tooltip>
     </TooltipProvider>
   );
 }
+
+function Content({ children }: { children: React.ReactNode }) {
+  const organization = useOptionalOrganization();
+  const project = useOptionalProject();
+  const user = useUser();
+
+  return (
+    <div className="flex flex-col gap-2 divide-y divide-slate-700">
+      <Property.Table>
+        <Property.Item>
+          <Property.Label>User ID</Property.Label>
+          <Property.Value>{user.id}</Property.Value>
+        </Property.Item>
+        {organization && (
+          <Property.Item>
+            <Property.Label>Org ID</Property.Label>
+            <Property.Value>{organization.id}</Property.Value>
+          </Property.Item>
+        )}
+        {project && (
+          <>
+            <Property.Item>
+              <Property.Label>Project ID</Property.Label>
+              <Property.Value>{project.id}</Property.Value>
+            </Property.Item>
+            <Property.Item>
+              <Property.Label>Project ref</Property.Label>
+              <Property.Value>{project.ref}</Property.Value>
+            </Property.Item>
+          </>
+        )}
+      </Property.Table>
+      <div className="pt-2">{children}</div>
+    </div>
+  );
+}

apps/webapp/app/components/billing/UpgradePrompt.tsx

@@ -1,61 +0,0 @@
-import { ExclamationCircleIcon } from "@heroicons/react/20/solid";
-import { ArrowUpCircleIcon } from "@heroicons/react/24/outline";
-import tileBgPath from "~/assets/images/error-banner-tile@2x.png";
-import { MatchedOrganization } from "~/hooks/useOrganizations";
-import { useCurrentPlan } from "~/routes/_app.orgs.$organizationSlug/route";
-import { formatNumberCompact } from "~/utils/numberFormatter";
-import { LinkButton } from "../primitives/Buttons";
-import { Icon } from "../primitives/Icon";
-import { Paragraph } from "../primitives/Paragraph";
-
-type UpgradePromptProps = {
-  runsEnabled: boolean;
-  runCountCap: number;
-  planPath: string;
-};
-
-export function UpgradePrompt({ runsEnabled, runCountCap, planPath }: UpgradePromptProps) {
-  return (
-    <div
-      className="flex h-10 items-center justify-between border border-error bg-repeat py-0 pl-3 pr-2"
-      style={{ backgroundImage: `url(${tileBgPath})`, backgroundSize: "8px 8px" }}
-    >
-      <div className="flex items-center gap-2">
-        <Icon icon={ExclamationCircleIcon} className="h-5 w-5 text-error" />
-        <Paragraph variant="small" className="text-error">
-          {runsEnabled
-            ? `You have exceeded the monthly ${formatNumberCompact(runCountCap)} runs
-          limit`
-            : `No runs are executing because you have exceeded the free limit`}
-        </Paragraph>
-      </div>
-      <LinkButton
-        variant={"primary/small"}
-        LeadingIcon={ArrowUpCircleIcon}
-        leadingIconClassName="px-0"
-        to={planPath}
-      >
-        Upgrade
-      </LinkButton>
-    </div>
-  );
-}
-
-export function useShowUpgradePrompt(organization?: MatchedOrganization) {
-  const currentPlan = useCurrentPlan();
-  const shouldShow =
-    organization !== undefined &&
-    currentPlan !== undefined &&
-    currentPlan.usage.exceededRunCount &&
-    currentPlan.usage.runCountCap !== undefined;
-
-  if (!shouldShow) {
-    return { shouldShow };
-  }
-
-  return {
-    shouldShow,
-    runCountCap: currentPlan.usage.runCountCap!,
-    runsEnabled: organization.runsEnabled,
-  };
-}

apps/webapp/app/components/billing/v2/ConcurrentRunsChart.tsx

@@ -8,7 +8,7 @@ import {
   XAxis,
   YAxis,
 } from "recharts";
-import { Paragraph } from "../primitives/Paragraph";
+import { Paragraph } from "../../primitives/Paragraph";
 
 const tooltipStyle = {
   display: "flex",

apps/webapp/app/components/billing/v2/DailyRunsChat.tsx

@@ -1,5 +1,5 @@
 import { Label, Line, LineChart, ResponsiveContainer, Tooltip, XAxis, YAxis } from "recharts";
-import { Paragraph } from "../primitives/Paragraph";
+import { Paragraph } from "../../primitives/Paragraph";
 
 const tooltipStyle = {
   display: "flex",

apps/webapp/app/components/billing/v2/FreePlanUsage.tsx

@@ -1,6 +1,6 @@
 import { ArrowUpCircleIcon } from "@heroicons/react/24/outline";
 import { motion, useMotionValue, useTransform } from "framer-motion";
-import { Paragraph } from "../primitives/Paragraph";
+import { Paragraph } from "../../primitives/Paragraph";
 import { Link } from "@remix-run/react";
 import { cn } from "~/utils/cn";
 
@@ -28,7 +28,7 @@ export function FreePlanUsage({ to, percentage }: { to: string; percentage: numb
           <Paragraph className="text-2sm text-text-bright">Free Plan</Paragraph>
         </div>
         <Link to={to} className="text-2sm text-text-link">
-          Learn more
+          Upgrade
         </Link>
       </div>
       <div className="relative mt-3 h-1 rounded-full bg-background-dimmed">

apps/webapp/app/components/billing/v2/PricingCalculator.tsx

@@ -1,9 +1,9 @@
 import * as Slider from "@radix-ui/react-slider";
-import { Plans, estimate } from "@trigger.dev/billing";
+import { Plans, estimate } from "@trigger.dev/platform/v2";
 import { useCallback, useState } from "react";
-import { DefinitionTip } from "../DefinitionTooltip";
-import { Header2 } from "../primitives/Headers";
-import { Paragraph } from "../primitives/Paragraph";
+import { DefinitionTip } from "../../DefinitionTooltip";
+import { Header2 } from "../../primitives/Headers";
+import { Paragraph } from "../../primitives/Paragraph";
 import { formatCurrency, formatNumberCompact } from "~/utils/numberFormatter";
 import { cn } from "~/utils/cn";
 

apps/webapp/app/components/billing/v2/PricingTiers.tsx

@@ -2,17 +2,17 @@ import { useForm } from "@conform-to/react";
 import { parse } from "@conform-to/zod";
 import { CheckIcon, XMarkIcon } from "@heroicons/react/24/solid";
 import { Form, useActionData, useNavigation } from "@remix-run/react";
-import { ActiveSubscription, Plan, Plans, SetPlanBodySchema } from "@trigger.dev/billing";
+import { ActiveSubscription, Plan, Plans, SetPlanBodySchema } from "@trigger.dev/platform/v2";
 import { useState } from "react";
 import { useCurrentPlan } from "~/routes/_app.orgs.$organizationSlug/route";
 import { cn } from "~/utils/cn";
 import { formatNumberCompact } from "~/utils/numberFormatter";
-import { DefinitionTip } from "../DefinitionTooltip";
-import { Feedback } from "../Feedback";
-import { Button, LinkButton } from "../primitives/Buttons";
-import SegmentedControl from "../primitives/SegmentedControl";
+import { DefinitionTip } from "../../DefinitionTooltip";
+import { Feedback } from "../../Feedback";
+import { Button, LinkButton } from "../../primitives/Buttons";
+import SegmentedControl from "../../primitives/SegmentedControl";
 import { RunsVolumeDiscountTable } from "./RunsVolumeDiscountTable";
-import { Spinner } from "../primitives/Spinner";
+import { Spinner } from "../../primitives/Spinner";
 
 const pricingDefinitions = {
   concurrentRuns: {

apps/webapp/app/components/billing/v2/RunsVolumeDiscountTable.tsx

@@ -1,6 +1,6 @@
-import { RunPriceBracket } from "@trigger.dev/billing";
-import { Header2 } from "../primitives/Headers";
-import { Paragraph } from "../primitives/Paragraph";
+import { RunPriceBracket } from "@trigger.dev/platform/v2";
+import { Header2 } from "../../primitives/Headers";
+import { Paragraph } from "../../primitives/Paragraph";
 import { formatNumberCompact } from "~/utils/numberFormatter";
 
 export function RunsVolumeDiscountTable({

apps/webapp/app/components/billing/v2/UsageBar.tsx

@@ -1,7 +1,7 @@
 import { cn } from "~/utils/cn";
 import { formatNumberCompact } from "~/utils/numberFormatter";
-import { Paragraph } from "../primitives/Paragraph";
-import { SimpleTooltip } from "../primitives/Tooltip";
+import { Paragraph } from "../../primitives/Paragraph";
+import { SimpleTooltip } from "../../primitives/Tooltip";
 import { motion } from "framer-motion";
 
 type UsageBarProps = {

apps/webapp/app/components/billing/v3/UpgradePrompt.tsx

@@ -0,0 +1,53 @@
+import { ExclamationCircleIcon } from "@heroicons/react/20/solid";
+import tileBgPath from "~/assets/images/error-banner-tile@2x.png";
+import { MatchedOrganization, useOrganization } from "~/hooks/useOrganizations";
+import { useCurrentPlan } from "~/routes/_app.orgs.$organizationSlug/route";
+import { v3BillingPath } from "~/utils/pathBuilder";
+import { LinkButton } from "../../primitives/Buttons";
+import { Icon } from "../../primitives/Icon";
+import { Paragraph } from "../../primitives/Paragraph";
+import { DateTime } from "~/components/primitives/DateTime";
+
+export function UpgradePrompt() {
+  const organization = useOrganization();
+  const plan = useCurrentPlan();
+
+  if (!plan || !plan.v3Usage.hasExceededFreeTier) {
+    return null;
+  }
+
+  const nextMonth = new Date();
+  nextMonth.setUTCMonth(nextMonth.getMonth() + 1);
+  nextMonth.setUTCDate(1);
+  nextMonth.setUTCHours(0, 0, 0, 0);
+
+  return (
+    <div
+      className="flex h-10 items-center justify-between border border-error bg-repeat py-0 pl-3 pr-2"
+      style={{ backgroundImage: `url(${tileBgPath})`, backgroundSize: "8px 8px" }}
+    >
+      <div className="flex items-center gap-2">
+        <Icon icon={ExclamationCircleIcon} className="h-5 w-5 text-error" />
+        <Paragraph variant="small" className="text-error">
+          You have exceeded the monthly $
+          {(plan.v3Subscription?.plan?.limits.includedUsage ?? 500) / 100} free credits. No runs
+          will execute in Prod until{" "}
+          <DateTime date={nextMonth} includeTime={false} timeZone="utc" />, or you upgrade.
+        </Paragraph>
+      </div>
+      <LinkButton
+        variant={"primary/small"}
+        leadingIconClassName="px-0"
+        to={v3BillingPath(organization)}
+      >
+        Upgrade
+      </LinkButton>
+    </div>
+  );
+}
+
+export function useShowUpgradePrompt(organization?: MatchedOrganization) {
+  const currentPlan = useCurrentPlan();
+  const shouldShow = currentPlan?.v3Usage.hasExceededFreeTier === true;
+  return { shouldShow };
+}

apps/webapp/app/components/billing/v3/UsageBar.tsx

@@ -0,0 +1,170 @@
+import { cn } from "~/utils/cn";
+import { formatCurrency } from "~/utils/numberFormatter";
+import { Paragraph } from "../../primitives/Paragraph";
+import { SimpleTooltip } from "../../primitives/Tooltip";
+import { motion } from "framer-motion";
+
+type UsageBarProps = {
+  current: number;
+  billingLimit?: number;
+  tierLimit?: number;
+  projectedUsage?: number;
+  isPaying: boolean;
+};
+
+const startFactor = 4;
+
+export function UsageBar({
+  current,
+  billingLimit,
+  tierLimit,
+  projectedUsage,
+  isPaying,
+}: UsageBarProps) {
+  const getLargestNumber = Math.max(
+    current,
+    tierLimit ?? -Infinity,
+    projectedUsage ?? -Infinity,
+    billingLimit ?? -Infinity,
+    5
+  );
+  //creates a maximum range for the progress bar, add 10% to the largest number so the bar doesn't reach the end
+  const maxRange = Math.round(getLargestNumber * 1.1);
+  const tierRunLimitPercentage = tierLimit ? Math.round((tierLimit / maxRange) * 100) : 0;
+  const projectedRunsPercentage = projectedUsage
+    ? Math.round((projectedUsage / maxRange) * 100)
+    : 0;
+  const billingLimitPercentage =
+    billingLimit !== undefined ? Math.round((billingLimit / maxRange) * 100) : 0;
+  const usagePercentage = Math.round((current / maxRange) * 100);
+
+  //cap the usagePercentage to the freeRunLimitPercentage
+  const usageCappedToLimitPercentage = Math.min(usagePercentage, tierRunLimitPercentage);
+
+  return (
+    <div className="h-fit w-full py-12">
+      <div className="relative h-3 w-full rounded-sm bg-background-bright">
+        {billingLimit !== undefined && (
+          <motion.div
+            initial={{ width: billingLimitPercentage / startFactor + "%" }}
+            animate={{ width: billingLimitPercentage + "%" }}
+            transition={{ duration: 1.5, type: "spring" }}
+            style={{ width: `${billingLimitPercentage}%` }}
+            className="absolute h-3 rounded-l-sm"
+          >
+            <Legend
+              text="Billing limit:"
+              value={formatCurrency(billingLimit, false)}
+              position="bottomRow2"
+              percentage={billingLimitPercentage}
+            />
+          </motion.div>
+        )}
+        {tierLimit !== undefined && (
+          <motion.div
+            initial={{ width: tierRunLimitPercentage / startFactor + "%" }}
+            animate={{ width: tierRunLimitPercentage + "%" }}
+            transition={{ duration: 1.5, type: "spring" }}
+            style={{ width: `${tierRunLimitPercentage}%` }}
+            className="absolute h-3 rounded-l-sm bg-green-900/50"
+          >
+            <Legend
+              text={isPaying ? `Included usage:` : `Tier limit:`}
+              value={formatCurrency(tierLimit, false)}
+              position="bottomRow1"
+              percentage={tierRunLimitPercentage}
+            />
+          </motion.div>
+        )}
+        {projectedUsage !== undefined && projectedUsage !== 0 && (
+          <motion.div
+            initial={{ width: projectedRunsPercentage / startFactor + "%" }}
+            animate={{ width: projectedRunsPercentage + "%" }}
+            transition={{ duration: 1.5, type: "spring" }}
+            style={{ width: `${projectedRunsPercentage}%` }}
+            className="absolute h-3 rounded-l-sm"
+          >
+            <Legend
+              text="Projected:"
+              value={formatCurrency(projectedUsage, false)}
+              position="topRow2"
+              percentage={projectedRunsPercentage}
+            />
+          </motion.div>
+        )}
+        <motion.div
+          initial={{ width: usagePercentage / startFactor + "%" }}
+          animate={{ width: usagePercentage + "%" }}
+          transition={{ duration: 1.5, type: "spring" }}
+          style={{ width: `${usagePercentage}%` }}
+          className={cn(
+            "absolute h-3 rounded-l-sm",
+            tierLimit && current > tierLimit ? "bg-rose-600" : "bg-green-600"
+          )}
+        >
+          <Legend
+            text="Used:"
+            value={formatCurrency(current, false)}
+            position="topRow1"
+            percentage={usagePercentage}
+          />
+        </motion.div>
+        <motion.div
+          initial={{ width: usageCappedToLimitPercentage / startFactor + "%" }}
+          animate={{ width: usageCappedToLimitPercentage + "%" }}
+          transition={{ duration: 1.5, type: "spring" }}
+          style={{ width: `${usageCappedToLimitPercentage}%` }}
+          className="absolute h-3 rounded-l-sm bg-green-600"
+        />
+      </div>
+    </div>
+  );
+}
+
+const positions = {
+  topRow1: "bottom-0 h-9",
+  topRow2: "bottom-0 h-14",
+  bottomRow1: "top-0 h-9 items-end",
+  bottomRow2: "top-0 h-14 items-end",
+};
+
+type LegendProps = {
+  text: string;
+  value: number | string;
+  percentage: number;
+  position: keyof typeof positions;
+  tooltipContent?: string;
+};
+
+function Legend({ text, value, position, percentage, tooltipContent }: LegendProps) {
+  const flipLegendPositionValue = 80;
+  const flipLegendPosition = percentage > flipLegendPositionValue ? true : false;
+  return (
+    <div
+      className={cn(
+        "absolute left-full z-10 flex border-charcoal-500",
+        positions[position],
+        flipLegendPosition === true ? "-translate-x-full border-r" : "border-l"
+      )}
+    >
+      {tooltipContent ? (
+        <SimpleTooltip
+          button={
+            <Paragraph className="mr-px h-fit whitespace-nowrap bg-background-bright px-1.5 text-xs text-text-bright">
+              {text}
+              <span className="ml-1 text-text-dimmed">{value}</span>
+            </Paragraph>
+          }
+          side="top"
+          content={tooltipContent}
+          className="z-50 h-fit"
+        />
+      ) : (
+        <Paragraph className="mr-px h-fit whitespace-nowrap bg-background-bright px-1.5 text-xs text-text-bright">
+          {text}
+          <span className="ml-1 text-text-dimmed">{value}</span>
+        </Paragraph>
+      )}
+    </div>
+  );
+}

apps/webapp/app/components/code/CodeBlock.tsx

@@ -58,103 +58,116 @@ const extraLinesWhenClipping = 0.35;
 
 const defaultTheme: PrismTheme = {
   plain: {
-    color: "#9CDCFE",
+    color: "#9C9AF2",
     backgroundColor: "rgba(0, 0, 0, 0)",
   },
   styles: [
     {
-      types: ["prolog"],
+      types: ["comment", "prolog", "doctype", "cdata"],
       style: {
-        color: "rgb(0, 0, 128)",
+        color: "#5F6570",
       },
     },
     {
-      types: ["comment"],
-      style: {
-        color: "rgb(106, 153, 85)",
-      },
-    },
-    {
-      types: ["builtin", "changed", "keyword", "interpolation-punctuation"],
-      style: {
-        color: "rgb(86, 156, 214)",
-      },
-    },
-    {
-      types: ["number", "inserted"],
-      style: {
-        color: "rgb(181, 206, 168)",
-      },
-    },
-    {
-      types: ["constant"],
-      style: {
-        color: "rgb(100, 102, 149)",
-      },
-    },
-    {
-      types: ["attr-name", "variable"],
-      style: {
-        color: "rgb(156, 220, 254)",
-      },
-    },
-    {
-      types: ["deleted", "string", "attr-value", "template-punctuation"],
-      style: {
-        color: "rgb(206, 145, 120)",
-      },
-    },
-    {
-      types: ["selector"],
-      style: {
-        color: "rgb(215, 186, 125)",
-      },
-    },
-    {
-      // Fix tag color
-      types: ["tag"],
-      style: {
-        color: "rgb(78, 201, 176)",
-      },
-    },
-    {
-      // Fix tag color for HTML
-      types: ["tag"],
-      languages: ["markup"],
-      style: {
-        color: "rgb(86, 156, 214)",
-      },
-    },
-    {
-      types: ["punctuation", "operator"],
-      style: {
-        color: "rgb(212, 212, 212)",
-      },
-    },
-    {
-      // Fix punctuation color for HTML
       types: ["punctuation"],
-      languages: ["markup"],
       style: {
-        color: "#808080",
+        color: "#878C99",
       },
     },
     {
-      types: ["function"],
+      types: ["property", "tag", "boolean", "number", "constant", "symbol", "deleted"],
       style: {
-        color: "rgb(220, 220, 170)",
+        color: "#9B99FF",
       },
     },
     {
-      types: ["class-name"],
+      types: ["selector", "attr-name", "string", "char", "builtin", "inserted"],
       style: {
-        color: "rgb(78, 201, 176)",
+        color: "#AFEC73",
+      },
+    },
+    {
+      types: ["operator", "entity", "url"],
+      style: {
+        color: "#D4D4D4",
+      },
+    },
+    {
+      types: ["variable"],
+      style: {
+        color: "#CCCBFF",
+      },
+    },
+    {
+      types: ["atrule", "attr-value", "keyword"],
+      style: {
+        color: "#E888F8",
+      },
+    },
+    {
+      types: ["function", "class-name"],
+      style: {
+        color: "#D9F07C",
+      },
+    },
+    {
+      types: ["regex"],
+      style: {
+        color: "#d16969",
+      },
+    },
+    {
+      types: ["important", "bold"],
+      style: {
+        fontWeight: "bold",
+      },
+    },
+    {
+      types: ["italic"],
+      style: {
+        fontStyle: "italic",
+      },
+    },
+    {
+      types: ["namespace"],
+      style: {
+        opacity: 0.7,
+      },
+    },
+    {
+      types: ["deleted"],
+      style: {
+        color: "#F85149",
+      },
+    },
+    {
+      types: ["boolean"],
+      style: {
+        color: "#9B99FF",
       },
     },
     {
       types: ["char"],
       style: {
-        color: "rgb(209, 105, 105)",
+        color: "#b5cea8",
+      },
+    },
+    {
+      types: ["tag"],
+      style: {
+        color: "#D7BA7D",
+      },
+    },
+    {
+      types: ["keyword.operator"],
+      style: {
+        color: "#8271ED",
+      },
+    },
+    {
+      types: ["meta.template.expression"],
+      style: {
+        color: "#d4d4d4",
       },
     },
   ],
@@ -228,16 +241,12 @@ export const CodeBlock = forwardRef<HTMLDivElement, CodeBlockProps>(
                 onMouseEnter={() => setMouseOver(true)}
                 onMouseLeave={() => setMouseOver(false)}
                 className={cn(
-                  "absolute  right-3 z-50 transition-colors duration-100 hover:cursor-pointer",
-                  showChrome ? "top-10" : "top-3",
+                  "absolute right-3 z-50 transition-colors duration-100 hover:cursor-pointer",
+                  showChrome ? "top-10" : "top-2.5",
                   copied ? "text-emerald-500" : "text-charcoal-500 hover:text-charcoal-300"
                 )}
               >
-                {copied ? (
-                  <ClipboardCheck className="h-5 w-5" />
-                ) : (
-                  <Clipboard className="h-5 w-5" />
-                )}
+                {copied ? <ClipboardCheck className="size-4" /> : <Clipboard className="size-4" />}
               </TooltipTrigger>
               <TooltipContent side="left" className="text-xs">
                 {copied ? "Copied" : "Copy"}
@@ -378,8 +387,8 @@ function Chrome({ title }: { title?: string }) {
 
 export function TitleRow({ title }: { title: string }) {
   return (
-    <div className="flex items-center justify-between px-4">
-      <Paragraph variant="base/bright" className="w-full border-b border-grid-dimmed py-2.5">
+    <div className="flex items-center justify-between px-3">
+      <Paragraph variant="small/bright" className="w-full border-b border-grid-dimmed py-2">
         {title}
       </Paragraph>
     </div>

apps/webapp/app/components/code/JSONEditor.tsx

@@ -107,39 +107,51 @@ export function JSONEditor(opts: JSONEditorProps) {
     }, 1500);
   }, [view]);
 
+  const showButtons = showClearButton || showCopyButton;
+
   return (
-    <div className={cn(opts.className, "grid grid-rows-[2.5rem_1fr]")}>
-      <div className="mx-3 flex items-center justify-end gap-2 border-b border-grid-dimmed">
-        {showClearButton && (
-          <Button
-            type="button"
-            variant="minimal/small"
-            TrailingIcon={TrashIcon}
-            onClick={(event) => {
-              event.preventDefault();
-              event.stopPropagation();
-              clear();
-            }}
-          >
-            Clear
-          </Button>
-        )}
-        {showCopyButton && (
-          <Button
-            type="button"
-            variant="minimal/small"
-            TrailingIcon={copied ? CheckIcon : ClipboardIcon}
-            trailingIconClassName={copied ? "text-green-500 group-hover:text-green-500" : undefined}
-            onClick={(event) => {
-              event.preventDefault();
-              event.stopPropagation();
-              copy();
-            }}
-          >
-            Copy
-          </Button>
-        )}
-      </div>
+    <div
+      className={cn(
+        opts.className,
+        "grid",
+        showButtons ? "grid-rows-[2.5rem_1fr]" : "grid-rows-[1fr]"
+      )}
+    >
+      {showButtons && (
+        <div className="mx-3 flex items-center justify-end gap-2 border-b border-grid-dimmed">
+          {showClearButton && (
+            <Button
+              type="button"
+              variant="minimal/small"
+              TrailingIcon={TrashIcon}
+              onClick={(event) => {
+                event.preventDefault();
+                event.stopPropagation();
+                clear();
+              }}
+            >
+              Clear
+            </Button>
+          )}
+          {showCopyButton && (
+            <Button
+              type="button"
+              variant="minimal/small"
+              TrailingIcon={copied ? CheckIcon : ClipboardIcon}
+              trailingIconClassName={
+                copied ? "text-green-500 group-hover:text-green-500" : undefined
+              }
+              onClick={(event) => {
+                event.preventDefault();
+                event.stopPropagation();
+                copy();
+              }}
+            >
+              Copy
+            </Button>
+          )}
+        </div>
+      )}
       <div
         className="w-full overflow-auto"
         ref={editor}

apps/webapp/app/components/layout/AppLayout.tsx

@@ -1,6 +1,6 @@
 import { useOptionalOrganization } from "~/hooks/useOrganizations";
 import { cn } from "~/utils/cn";
-import { useShowUpgradePrompt } from "../billing/UpgradePrompt";
+import { useShowUpgradePrompt } from "../billing/v3/UpgradePrompt";
 
 /** This container is used to surround the entire app, it correctly places the nav bar */
 export function AppContainer({ children }: { children: React.ReactNode }) {
@@ -60,7 +60,9 @@ export function MainCenteredContainer({
 }) {
   return (
     <div className="h-full w-full overflow-y-auto scrollbar-thin scrollbar-track-transparent scrollbar-thumb-charcoal-600">
-      <div className={cn("mx-auto mt-[25vh] max-w-xs overflow-y-auto", className)}>{children}</div>
+      <div className={cn("mx-auto mt-6 max-w-xs overflow-y-auto p-1 md:mt-[22vh]", className)}>
+        {children}
+      </div>
     </div>
   );
 }

apps/webapp/app/components/navigation/SideMenu.tsx

@@ -1,16 +1,18 @@
 import {
   AcademicCapIcon,
-  ArrowRightIcon,
   ArrowRightOnRectangleIcon,
   BeakerIcon,
   BellAlertIcon,
   ChartBarIcon,
   ClockIcon,
+  CreditCardIcon,
   CursorArrowRaysIcon,
   IdentificationIcon,
   KeyIcon,
+  RectangleStackIcon,
   ServerStackIcon,
   ShieldCheckIcon,
+  SignalIcon,
 } from "@heroicons/react/20/solid";
 import { UserGroupIcon, UserPlusIcon } from "@heroicons/react/24/solid";
 import { useNavigation } from "@remix-run/react";
@@ -18,9 +20,9 @@ import { DiscordIcon, SlackIcon } from "@trigger.dev/companyicons";
 import { Fragment, useEffect, useRef, useState } from "react";
 import { TaskIcon } from "~/assets/icons/TaskIcon";
 import { useFeatures } from "~/hooks/useFeatures";
-import { MatchedOrganization } from "~/hooks/useOrganizations";
-import { MatchedProject } from "~/hooks/useProject";
-import { User } from "~/models/user.server";
+import { type MatchedOrganization } from "~/hooks/useOrganizations";
+import { type MatchedProject } from "~/hooks/useProject";
+import { type User } from "~/models/user.server";
 import { useCurrentPlan } from "~/routes/_app.orgs.$organizationSlug/route";
 import { cn } from "~/utils/cn";
 import {
@@ -44,6 +46,8 @@ import {
   projectSetupPath,
   projectTriggersPath,
   v3ApiKeysPath,
+  v3BillingPath,
+  v3ConcurrencyPath,
   v3DeploymentsPath,
   v3EnvironmentVariablesPath,
   v3ProjectAlertsPath,
@@ -52,16 +56,16 @@ import {
   v3RunsPath,
   v3SchedulesPath,
   v3TestPath,
+  v3UsagePath,
 } from "~/utils/pathBuilder";
 import { Feedback } from "../Feedback";
 import { ImpersonationBanner } from "../ImpersonationBanner";
 import { LogoIcon } from "../LogoIcon";
 import { StepContentContainer } from "../StepContentContainer";
 import { UserProfilePhoto } from "../UserProfilePhoto";
-import { FreePlanUsage } from "../billing/FreePlanUsage";
+import { FreePlanUsage } from "../billing/v2/FreePlanUsage";
 import { Badge } from "../primitives/Badge";
-import { Button } from "../primitives/Buttons";
-import { Callout } from "../primitives/Callout";
+import { Button, LinkButton } from "../primitives/Buttons";
 import { ClipboardField } from "../primitives/ClipboardField";
 import { Dialog, DialogContent, DialogHeader, DialogTrigger } from "../primitives/Dialog";
 import { Icon } from "../primitives/Icon";
@@ -75,8 +79,9 @@ import {
   PopoverSectionHeader,
 } from "../primitives/Popover";
 import { StepNumber } from "../primitives/StepNumber";
+import { TextLink } from "../primitives/TextLink";
 import { SideMenuHeader } from "./SideMenuHeader";
-import { MenuCount, SideMenuItem } from "./SideMenuItem";
+import { SideMenuItem } from "./SideMenuItem";
 
 type SideMenuUser = Pick<User, "email" | "admin"> & { isImpersonating: boolean };
 type SideMenuProject = Pick<
@@ -100,8 +105,11 @@ type SideMenuProps = {
 export function SideMenu({ user, project, organization, organizations }: SideMenuProps) {
   const borderRef = useRef<HTMLDivElement>(null);
   const [showHeaderDivider, setShowHeaderDivider] = useState(false);
-  const { isManagedCloud } = useFeatures();
   const currentPlan = useCurrentPlan();
+  const { isManagedCloud } = useFeatures();
+
+  const isV3Project = project.version === "V3";
+  const isFreeV3User = currentPlan?.v3Subscription?.isPaying === false;
 
   useEffect(() => {
     const handleScroll = () => {
@@ -173,16 +181,41 @@ export function SideMenu({ user, project, organization, organizations }: SideMen
               name="Team"
               icon={UserGroupIcon}
               to={organizationTeamPath(organization)}
-              iconColor="text-sky-500"
+              iconColor="text-amber-500"
               data-action="team"
             />
-            <SideMenuItem
-              name={isManagedCloud ? "Usage & Billing" : "Usage"}
-              icon={ChartBarIcon}
-              to={organizationBillingPath(organization)}
-              iconColor="text-green-600"
-              data-action="usage & billing"
-            />
+            {organization.projects.some((proj) => proj.version === "V3") && isManagedCloud && (
+              <>
+                <SideMenuItem
+                  name="Usage"
+                  icon={ChartBarIcon}
+                  to={v3UsagePath(organization)}
+                  iconColor="text-green-600"
+                  data-action="usage"
+                />
+                <SideMenuItem
+                  name="Billing"
+                  icon={CreditCardIcon}
+                  to={v3BillingPath(organization)}
+                  iconColor="text-blue-600"
+                  data-action="billing"
+                  badge={
+                    currentPlan?.v3Subscription?.isPaying
+                      ? currentPlan?.v3Subscription?.plan?.title
+                      : undefined
+                  }
+                />
+              </>
+            )}
+            {organization.projects.some((proj) => proj.version === "V2") && (
+              <SideMenuItem
+                name="Usage (v2)"
+                icon={ChartBarIcon}
+                to={organizationBillingPath(organization)}
+                iconColor="text-green-600"
+                data-action="usage & billing"
+              />
+            )}
             <SideMenuItem
               name="Organization settings"
               icon="settings"
@@ -193,22 +226,30 @@ export function SideMenu({ user, project, organization, organizations }: SideMen
           </div>
         </div>
         <div className="m-2">
-          {project.version === "V2" ? (
-            <Callout variant={"info"}>This is a v2 project</Callout>
-          ) : (
-            <Callout variant={"idea"}>This is a v3 project in Developer Preview</Callout>
+          {project.version === "V2" && (
+            <div className="flex flex-col gap-3 rounded border border-success/50 bg-success/10 p-3">
+              <Paragraph variant="small/bright">
+                This is a v2 project. V2 will be deprecated on January 31, 2025.{" "}
+                <TextLink
+                  className="text-text-bright underline decoration-text-dimmed underline-offset-2 transition hover:text-text-bright hover:decoration-text-bright"
+                  to="https://trigger.dev/blog/v2-end-of-life-announcement"
+                >
+                  Learn more
+                </TextLink>
+                .
+              </Paragraph>
+              <LinkButton
+                variant="primary/medium"
+                to="https://trigger.dev/docs/v3/upgrading-from-v2"
+                fullWidth
+              >
+                Upgrade to v3
+              </LinkButton>
+            </div>
           )}
         </div>
         <div className="flex flex-col gap-1 border-t border-grid-bright p-1">
-          {project.version === "V2" && (
-            <SideMenuItem
-              to="https://trigger.dev/v3-early-access"
-              target="_blank"
-              name="Request access to v3"
-              icon={V3Icon}
-            />
-          )}
-          {currentPlan?.subscription?.isPaying === true && (
+          {currentPlan?.v3Subscription?.plan?.limits.support === "slack" && (
             <Dialog>
               <DialogTrigger asChild>
                 <Button
@@ -263,23 +304,13 @@ export function SideMenu({ user, project, organization, organizations }: SideMen
             data-action="join our discord"
             target="_blank"
           />
-          {project.version === "V2" ? (
-            <SideMenuItem
-              name="Documentation"
-              icon="docs"
-              to="https://trigger.dev/docs"
-              data-action="documentation"
-              target="_blank"
-            />
-          ) : (
-            <SideMenuItem
-              name="Documentation (v3)"
-              icon="docs"
-              to="https://trigger.dev/docs/v3"
-              data-action="documentation"
-              target="_blank"
-            />
-          )}
+          <SideMenuItem
+            name="Documentation"
+            icon="docs"
+            to="https://trigger.dev/docs"
+            data-action="documentation"
+            target="_blank"
+          />
           <SideMenuItem
             name="Changelog"
             icon="star"
@@ -287,41 +318,31 @@ export function SideMenu({ user, project, organization, organizations }: SideMen
             data-action="changelog"
             target="_blank"
           />
-          {project.version === "V2" ? (
-            <Feedback
-              button={
-                <Button
-                  variant="small-menu-item"
-                  LeadingIcon="log"
-                  data-action="help & feedback"
-                  fullWidth
-                  textAlignLeft
-                >
-                  Help & Feedback
-                </Button>
-              }
-            />
-          ) : (
-            <Feedback
-              defaultValue="developer preview"
-              button={
-                <Button
-                  variant="small-menu-item"
-                  LeadingIcon="log"
-                  leadingIconClassName="text-primary"
-                  data-action="help & feedback"
-                  fullWidth
-                  textAlignLeft
-                >
-                  <span className="text-primary">Give feedback on v3</span>
-                </Button>
-              }
-            />
-          )}
-          {currentPlan && !currentPlan.subscription?.isPaying && currentPlan.usage.runCountCap && (
+          <SideMenuItem
+            name="Status"
+            icon={SignalIcon}
+            to="https://status.trigger.dev/"
+            data-action="status"
+            target="_blank"
+          />
+          <Feedback
+            button={
+              <Button
+                variant="small-menu-item"
+                LeadingIcon="log"
+                leadingIconClassName="text-primary"
+                data-action="help & feedback"
+                fullWidth
+                textAlignLeft
+              >
+                <span className="text-primary">Help & Feedback</span>
+              </Button>
+            }
+          />
+          {isV3Project && isFreeV3User && (
             <FreePlanUsage
-              to={organizationBillingPath(organization)}
-              percentage={currentPlan.usage.currentRunCount / currentPlan.usage.runCountCap}
+              to={v3BillingPath(organization)}
+              percentage={currentPlan.v3Usage.usagePercentage}
             />
           )}
         </div>
@@ -373,10 +394,10 @@ function ProjectSelector({
                       title={
                         <div className="flex w-full items-center justify-between text-text-bright">
                           <span className="grow truncate text-left">{p.name}</span>
-                          {p.version === "V2" ? (
-                            <MenuCount count={p.jobCount} />
-                          ) : (
-                            <Badge variant="v3">v3</Badge>
+                          {p.version === "V2" && (
+                            <Badge variant="small" className="normal-case">
+                              v2
+                            </Badge>
                           )}
                         </div>
                       }
@@ -395,7 +416,7 @@ function ProjectSelector({
             </div>
           </Fragment>
         ))}
-        <div className="border-t border-charcoal-800 p-1">
+        <div className="border-t border-charcoal-700 p-1">
           <PopoverMenuItem to={newOrganizationPath()} title="New Organization" icon="plus" />
         </div>
       </PopoverContent>
@@ -474,7 +495,7 @@ function V2ProjectSideMenu({
 }) {
   return (
     <>
-      <SideMenuHeader title={"Project"}>
+      <SideMenuHeader title={"Project (v2)"}>
         <PopoverMenuItem
           to={projectSetupPath(organization, project)}
           title="Framework setup"
@@ -485,7 +506,6 @@ function V2ProjectSideMenu({
         name="Jobs"
         icon="job"
         iconColor="text-indigo-500"
-        count={project.jobCount}
         to={projectPath(organization, project)}
         data-action="jobs"
       />
@@ -513,7 +533,6 @@ function V2ProjectSideMenu({
         name="HTTP endpoints"
         icon="http-endpoint"
         iconColor="text-pink-500"
-        count={project.httpEndpointCount}
         to={projectHttpEndpointsPath(organization, project)}
         data-action="httpendpoints"
       />
@@ -546,12 +565,11 @@ function V3ProjectSideMenu({
 
   return (
     <>
-      <SideMenuHeader title={"Project (v3)"} />
+      <SideMenuHeader title={"Project"} />
       <SideMenuItem
         name="Tasks"
         icon={TaskIcon}
         iconColor="text-blue-500"
-        count={project.jobCount}
         to={v3ProjectPath(organization, project)}
         data-action="tasks"
       />
@@ -589,6 +607,7 @@ function V3ProjectSideMenu({
         to={v3EnvironmentVariablesPath(organization, project)}
         data-action="environment variables"
       />
+
       <SideMenuItem
         name="Deployments"
         icon={ServerStackIcon}
@@ -605,6 +624,13 @@ function V3ProjectSideMenu({
           data-action="alerts"
         />
       )}
+      <SideMenuItem
+        name="Concurrency limits"
+        icon={RectangleStackIcon}
+        iconColor="text-indigo-500"
+        to={v3ConcurrencyPath(organization, project)}
+        data-action="concurrency"
+      />
       <SideMenuItem
         name="Project settings"
         icon="settings"
@@ -615,15 +641,3 @@ function V3ProjectSideMenu({
     </>
   );
 }
-
-function V3Icon() {
-  return (
-    <svg width="16" height="16" viewBox="0 0 16 16" fill="none" xmlns="http://www.w3.org/2000/svg">
-      <rect width="16" height="16" rx="8" fill="#A8FF53" />
-      <path
-        d="M7.7488 6.17L5.5818 12H3.6678L1.5008 6.17H3.2058L4.6248 10.339L6.0328 6.17H7.7488ZM11.0527 7.292C11.6357 7.303 12.2297 7.006 12.2297 6.28C12.2297 5.774 11.7787 5.433 11.0527 5.433C10.4147 5.433 9.98567 5.741 9.91967 6.214L8.22567 6.126C8.36867 4.861 9.51267 4.014 11.0857 4.014C12.8457 4.014 13.9567 4.806 13.9567 6.049C13.9567 6.951 13.3847 7.534 12.3067 7.776C13.5387 8.04 14.2207 8.777 14.2207 9.855C14.2207 11.274 13.0107 12.176 11.0857 12.176C9.32567 12.176 8.12667 11.197 8.04967 9.712L9.75467 9.646C9.83167 10.405 10.4917 10.757 11.0967 10.757C11.8007 10.757 12.4937 10.394 12.4937 9.591C12.4937 8.81 11.7897 8.425 11.0527 8.447L10.3817 8.458V7.281L11.0527 7.292Z"
-        fill="#15171A"
-      />
-    </svg>
-  );
-}

apps/webapp/app/components/navigation/SideMenuItem.tsx

@@ -1,8 +1,8 @@
-import { AnchorHTMLAttributes } from "react";
+import { type AnchorHTMLAttributes } from "react";
 import { usePathName } from "~/hooks/usePathName";
 import { cn } from "~/utils/cn";
 import { LinkButton } from "../primitives/Buttons";
-import { IconNames } from "../primitives/NamedIcon";
+import { type IconNames } from "../primitives/NamedIcon";
 import { Tooltip, TooltipContent, TooltipProvider, TooltipTrigger } from "../primitives/Tooltip";
 import { Icon } from "../primitives/Icon";
 import { IconExclamationCircle } from "@tabler/icons-react";
@@ -13,7 +13,7 @@ export function SideMenuItem({
   name,
   to,
   hasWarning,
-  count,
+  badge,
   target,
   subItem = false,
 }: {
@@ -22,7 +22,7 @@ export function SideMenuItem({
   name: string;
   to: string;
   hasWarning?: string | boolean;
-  count?: number;
+  badge?: string;
   target?: AnchorHTMLAttributes<HTMLAnchorElement>["target"];
   subItem?: boolean;
 }) {
@@ -47,20 +47,20 @@ export function SideMenuItem({
       <div className="flex w-full items-center justify-between">
         {name}
         <div className="flex items-center gap-1">
-          {count !== undefined && count > 0 && <MenuCount count={count} />}
+          {badge !== undefined && <MenuCount count={badge} />}
           {typeof hasWarning === "string" ? (
             <TooltipProvider>
               <Tooltip>
                 <TooltipTrigger>
-                  <Icon icon={IconExclamationCircle} className="h-5 w-5 text-rose-500" />
+                  <Icon icon={IconExclamationCircle} className="h-5 w-5 text-error" />
                 </TooltipTrigger>
-                <TooltipContent className="flex items-center gap-1 border border-rose-500 bg-rose-500/20 backdrop-blur-xl">
+                <TooltipContent className="flex items-center gap-1 border border-error bg-error/20 backdrop-blur-xl">
                   {hasWarning}
                 </TooltipContent>
               </Tooltip>
             </TooltipProvider>
           ) : (
-            hasWarning && <Icon icon={IconExclamationCircle} className="h-5 w-5 text-rose-500" />
+            hasWarning && <Icon icon={IconExclamationCircle} className="h-5 w-5 text-error" />
           )}
         </div>
       </div>
@@ -68,8 +68,10 @@ export function SideMenuItem({
   );
 }
 
-export function MenuCount({ count }: { count: number | string }) {
+function MenuCount({ count }: { count: number | string }) {
   return (
-    <div className="rounded-full bg-charcoal-900 px-2 py-1 text-xxs text-text-dimmed">{count}</div>
+    <div className="rounded-full bg-charcoal-900 px-2 py-1 text-xxs uppercase tracking-wider text-text-dimmed">
+      {count}
+    </div>
   );
 }

apps/webapp/app/components/primitives/Alert.tsx

@@ -8,12 +8,8 @@ const Alert = AlertDialogPrimitive.Root;
 
 const AlertTrigger = AlertDialogPrimitive.Trigger;
 
-const AlertPortal = ({
-  className,
-  children,
-  ...props
-}: AlertDialogPrimitive.AlertDialogPortalProps) => (
-  <AlertDialogPrimitive.Portal className={cn(className)} {...props}>
+const AlertPortal = ({ children, ...props }: AlertDialogPrimitive.AlertDialogPortalProps) => (
+  <AlertDialogPrimitive.Portal {...props}>
     <div className="fixed inset-0 z-50 flex items-end justify-center sm:items-center">
       {children}
     </div>

apps/webapp/app/components/primitives/Badge.tsx

@@ -8,7 +8,6 @@ const variants = {
     "grid place-items-center rounded-full px-[0.4rem] h-4 tracking-wider text-xxs bg-background-dimmed text-text-dimmed uppercase whitespace-nowrap",
   outline:
     "grid place-items-center rounded-sm px-1.5 h-5 tracking-wider text-xxs border border-dimmed text-text-dimmed uppercase whitespace-nowrap",
-  v3: "grid place-items-center rounded-full px-[0.4rem] h-5 tracking-wider text-xxs bg-charcoal-750 text-primary whitespace-nowrap",
   "outline-rounded":
     "grid place-items-center rounded-full px-1 h-4 tracking-wider text-xxs border border-blue-500 text-blue-500 uppercase whitespace-nowrap",
 };

apps/webapp/app/components/primitives/Buttons.tsx

@@ -21,7 +21,7 @@ const sizes = {
     shortcut: "-ml-0.5 -mr-1.5 rounded justify-self-center",
   },
   large: {
-    button: "h-10 px-2 text-sm font-medium",
+    button: "h-10 px-2 text-base font-medium",
     icon: "h-5",
     iconSpacing: "gap-x-0.5",
     shortcutVariant: "medium" as const,
@@ -245,12 +245,12 @@ export function ButtonContent(props: ButtonContentPropsType) {
 
 type ButtonPropsType = Pick<
   JSX.IntrinsicElements["button"],
-  "type" | "disabled" | "onClick" | "name" | "value" | "form"
+  "type" | "disabled" | "onClick" | "name" | "value" | "form" | "autoFocus"
 > &
   React.ComponentProps<typeof ButtonContent>;
 
 export const Button = forwardRef<HTMLButtonElement, ButtonPropsType>(
-  ({ type, disabled, onClick, ...props }, ref) => {
+  ({ type, disabled, autoFocus, onClick, ...props }, ref) => {
     const innerRef = useRef<HTMLButtonElement>(null);
     useImperativeHandle(ref, () => innerRef.current as HTMLButtonElement);
 
@@ -276,6 +276,7 @@ export const Button = forwardRef<HTMLButtonElement, ButtonPropsType>(
         value={props.value}
         ref={innerRef}
         form={props.form}
+        autoFocus={autoFocus}
       >
         <ButtonContent {...props} />
       </button>

apps/webapp/app/components/primitives/Chart.tsx

@@ -0,0 +1,326 @@
+"use client";
+
+import * as React from "react";
+import * as RechartsPrimitive from "recharts";
+import { cn } from "~/utils/cn";
+
+// Format: { THEME_NAME: CSS_SELECTOR }
+const THEMES = { light: "", dark: ".dark" } as const;
+
+export type ChartConfig = {
+  [k in string]: {
+    label?: React.ReactNode;
+    icon?: React.ComponentType;
+  } & (
+    | { color?: string; theme?: never }
+    | { color?: never; theme: Record<keyof typeof THEMES, string> }
+  );
+};
+
+type ChartContextProps = {
+  config: ChartConfig;
+};
+
+const ChartContext = React.createContext<ChartContextProps | null>(null);
+
+function useChart() {
+  const context = React.useContext(ChartContext);
+
+  if (!context) {
+    throw new Error("useChart must be used within a <ChartContainer />");
+  }
+
+  return context;
+}
+
+const ChartContainer = React.forwardRef<
+  HTMLDivElement,
+  React.ComponentProps<"div"> & {
+    config: ChartConfig;
+    children: React.ComponentProps<typeof RechartsPrimitive.ResponsiveContainer>["children"];
+  }
+>(({ id, className, children, config, ...props }, ref) => {
+  const uniqueId = React.useId();
+  const chartId = `chart-${id || uniqueId.replace(/:/g, "")}`;
+
+  return (
+    <ChartContext.Provider value={{ config }}>
+      <div
+        data-chart={chartId}
+        ref={ref}
+        className={cn(
+          "flex aspect-video justify-center text-xs [&_.recharts-cartesian-axis-tick_text]:fill-text-dimmed/50 [&_.recharts-cartesian-grid_line]:stroke-grid-bright/50 [&_.recharts-curve.recharts-tooltip-cursor]:stroke-grid-dimmed [&_.recharts-dot[stroke='#fff']]:stroke-transparent [&_.recharts-layer]:outline-none [&_.recharts-polar-grid_[stroke='#ccc']]:stroke-grid-dimmed [&_.recharts-radial-bar-background-sector]:fill-grid-dimmed [&_.recharts-rectangle.recharts-tooltip-cursor]:fill-grid-dimmed [&_.recharts-reference-line-line]:stroke-grid-dimmed [&_.recharts-sector[stroke='#fff']]:stroke-transparent [&_.recharts-sector]:outline-none [&_.recharts-surface]:outline-none",
+          className
+        )}
+        {...props}
+      >
+        <ChartStyle id={chartId} config={config} />
+        <RechartsPrimitive.ResponsiveContainer className="!h-auto">
+          {children}
+        </RechartsPrimitive.ResponsiveContainer>
+      </div>
+    </ChartContext.Provider>
+  );
+});
+ChartContainer.displayName = "Chart";
+
+const ChartStyle = ({ id, config }: { id: string; config: ChartConfig }) => {
+  const colorConfig = Object.entries(config).filter(([_, config]) => config.theme || config.color);
+
+  if (!colorConfig.length) {
+    return null;
+  }
+
+  return (
+    <style
+      dangerouslySetInnerHTML={{
+        __html: Object.entries(THEMES).map(
+          ([theme, prefix]) => `
+${prefix} [data-chart=${id}] {
+${colorConfig
+  .map(([key, itemConfig]) => {
+    const color = itemConfig.theme?.[theme as keyof typeof itemConfig.theme] || itemConfig.color;
+    return color ? `  --color-${key}: ${color};` : null;
+  })
+  .join("\n")}
+}
+`
+        ),
+      }}
+    />
+  );
+};
+
+const ChartTooltip = RechartsPrimitive.Tooltip;
+
+const ChartTooltipContent = React.forwardRef<
+  HTMLDivElement,
+  React.ComponentProps<typeof RechartsPrimitive.Tooltip> &
+    React.ComponentProps<"div"> & {
+      hideLabel?: boolean;
+      hideIndicator?: boolean;
+      indicator?: "line" | "dot" | "dashed";
+      nameKey?: string;
+      labelKey?: string;
+    }
+>(
+  (
+    {
+      active,
+      payload,
+      className,
+      indicator = "dot",
+      hideLabel = false,
+      hideIndicator = false,
+      label,
+      labelFormatter,
+      labelClassName,
+      formatter,
+      color,
+      nameKey,
+      labelKey,
+    },
+    ref
+  ) => {
+    const { config } = useChart();
+
+    const tooltipLabel = React.useMemo(() => {
+      if (hideLabel || !payload?.length) {
+        return null;
+      }
+
+      const [item] = payload;
+      const key = `${labelKey || item.dataKey || item.name || "value"}`;
+      const itemConfig = getPayloadConfigFromPayload(config, item, key);
+      const value =
+        !labelKey && typeof label === "string"
+          ? config[label as keyof typeof config]?.label || label
+          : itemConfig?.label;
+
+      if (labelFormatter) {
+        return (
+          <div className={cn("font-medium", labelClassName)}>{labelFormatter(value, payload)}</div>
+        );
+      }
+
+      if (!value) {
+        return null;
+      }
+
+      return <div className={cn("font-medium", labelClassName)}>{value}</div>;
+    }, [label, labelFormatter, payload, hideLabel, labelClassName, config, labelKey]);
+
+    if (!active || !payload?.length) {
+      return null;
+    }
+
+    const nestLabel = payload.length === 1 && indicator !== "dot";
+
+    return (
+      <div
+        ref={ref}
+        className={cn(
+          "grid min-w-[8rem] items-start gap-1.5 rounded-lg border border-grid-bright/50 bg-background-dimmed px-2.5 py-1.5 text-xs shadow-xl",
+          className
+        )}
+      >
+        {!nestLabel ? tooltipLabel : null}
+        <div className="grid gap-1.5">
+          {payload.map((item, index) => {
+            const key = `${nameKey || item.name || item.dataKey || "value"}`;
+            const itemConfig = getPayloadConfigFromPayload(config, item, key);
+            const indicatorColor = color || item.payload.fill || item.color;
+
+            return (
+              <div
+                key={item.dataKey}
+                className={cn(
+                  "flex w-full items-stretch gap-2 [&>svg]:h-2.5 [&>svg]:w-2.5 [&>svg]:text-text-dimmed",
+                  indicator === "dot" && "items-center"
+                )}
+              >
+                {formatter && item.value && item.name ? (
+                  formatter(item.value, item.name, item, index, item.payload)
+                ) : (
+                  <>
+                    {itemConfig?.icon ? (
+                      <itemConfig.icon />
+                    ) : (
+                      !hideIndicator && (
+                        <div
+                          className={cn(
+                            "shrink-0 rounded-[2px] border-[--color-border] bg-[--color-bg]",
+                            {
+                              "h-2.5 w-2.5": indicator === "dot",
+                              "w-1": indicator === "line",
+                              "w-0 border-[1.5px] border-dashed bg-transparent":
+                                indicator === "dashed",
+                              "my-0.5": nestLabel && indicator === "dashed",
+                            }
+                          )}
+                          style={
+                            {
+                              "--color-bg": indicatorColor,
+                              "--color-border": indicatorColor,
+                            } as React.CSSProperties
+                          }
+                        />
+                      )
+                    )}
+                    <div
+                      className={cn(
+                        "flex flex-1 justify-between gap-1.5 leading-none",
+                        nestLabel ? "items-end" : "items-center"
+                      )}
+                    >
+                      <div className="grid gap-1.5">
+                        {nestLabel ? tooltipLabel : null}
+                        <span className="text-text-dimmed">{itemConfig?.label || item.name}</span>
+                      </div>
+                      {item.value && (
+                        <span className="font-mono font-medium tabular-nums text-text-bright">
+                          {item.value.toLocaleString()}
+                        </span>
+                      )}
+                    </div>
+                  </>
+                )}
+              </div>
+            );
+          })}
+        </div>
+      </div>
+    );
+  }
+);
+ChartTooltipContent.displayName = "ChartTooltip";
+
+const ChartLegend = RechartsPrimitive.Legend;
+
+const ChartLegendContent = React.forwardRef<
+  HTMLDivElement,
+  React.ComponentProps<"div"> &
+    Pick<RechartsPrimitive.LegendProps, "payload" | "verticalAlign"> & {
+      hideIcon?: boolean;
+      nameKey?: string;
+    }
+>(({ className, hideIcon = false, payload, verticalAlign = "bottom", nameKey }, ref) => {
+  const { config } = useChart();
+
+  if (!payload?.length) {
+    return null;
+  }
+
+  return (
+    <div
+      ref={ref}
+      className={cn(
+        "flex items-center justify-center gap-4",
+        verticalAlign === "top" ? "pb-3" : "pt-3",
+        className
+      )}
+    >
+      {payload.map((item) => {
+        const key = `${nameKey || item.dataKey || "value"}`;
+        const itemConfig = getPayloadConfigFromPayload(config, item, key);
+
+        return (
+          <div
+            key={item.value}
+            className={cn(
+              "flex items-center gap-1.5 [&>svg]:h-3 [&>svg]:w-3 [&>svg]:text-text-dimmed"
+            )}
+          >
+            {itemConfig?.icon && !hideIcon ? (
+              <itemConfig.icon />
+            ) : (
+              <div
+                className="h-2 w-2 shrink-0 rounded-[2px]"
+                style={{
+                  backgroundColor: item.color,
+                }}
+              />
+            )}
+            {itemConfig?.label}
+          </div>
+        );
+      })}
+    </div>
+  );
+});
+ChartLegendContent.displayName = "ChartLegend";
+
+// Helper to extract item config from a payload.
+function getPayloadConfigFromPayload(config: ChartConfig, payload: unknown, key: string) {
+  if (typeof payload !== "object" || payload === null) {
+    return undefined;
+  }
+
+  const payloadPayload =
+    "payload" in payload && typeof payload.payload === "object" && payload.payload !== null
+      ? payload.payload
+      : undefined;
+
+  let configLabelKey: string = key;
+
+  if (key in payload && typeof payload[key as keyof typeof payload] === "string") {
+    configLabelKey = payload[key as keyof typeof payload] as string;
+  } else if (
+    payloadPayload &&
+    key in payloadPayload &&
+    typeof payloadPayload[key as keyof typeof payloadPayload] === "string"
+  ) {
+    configLabelKey = payloadPayload[key as keyof typeof payloadPayload] as string;
+  }
+
+  return configLabelKey in config ? config[configLabelKey] : config[key as keyof typeof config];
+}
+
+export {
+  ChartContainer,
+  ChartTooltip,
+  ChartTooltipContent,
+  ChartLegend,
+  ChartLegendContent,
+  ChartStyle,
+};

apps/webapp/app/components/primitives/Dialog.tsx

@@ -10,8 +10,8 @@ const Dialog = DialogPrimitive.Root;
 
 const DialogTrigger = DialogPrimitive.Trigger;
 
-const DialogPortal = ({ className, children, ...props }: DialogPrimitive.DialogPortalProps) => (
-  <DialogPrimitive.Portal className={cn(className)} {...props}>
+const DialogPortal = ({ children, ...props }: DialogPrimitive.DialogPortalProps) => (
+  <DialogPrimitive.Portal {...props}>
     <div className="fixed inset-0 z-50 flex items-start justify-center sm:items-center">
       {children}
     </div>

apps/webapp/app/components/primitives/Headers.tsx

@@ -2,7 +2,7 @@ import { cn } from "~/utils/cn";
 
 const headerVariants = {
   header1: {
-    text: "font-sans text-2xl leading-5 md:leading-6 lg:leading-7 font-semibold",
+    text: "font-sans text-2xl leading-5 md:leading-6 lg:leading-7 font-medium",
     spacing: "mb-2",
   },
   header2: {

apps/webapp/app/components/primitives/InfoPanel.tsx

@@ -0,0 +1,71 @@
+import { LinkButton } from "./Buttons";
+import { Header3 } from "./Headers";
+import { Paragraph } from "./Paragraph";
+import { cn } from "~/utils/cn";
+
+const variants = {
+  info: {
+    panelStyle: "border-grid-bright bg-background-bright",
+  },
+  upgrade: {
+    panelStyle: "border-indigo-400/20 bg-indigo-800/10",
+  },
+};
+
+type InfoPanelVariant = keyof typeof variants;
+
+type Props = {
+  title?: string;
+  children: React.ReactNode;
+  to?: string;
+  buttonLabel?: string;
+  icon: React.ComponentType<any>;
+  iconClassName?: string;
+  variant?: InfoPanelVariant;
+  panelClassName?: string;
+};
+
+export function InfoPanel({
+  title,
+  children,
+  to,
+  buttonLabel,
+  icon,
+  iconClassName,
+  variant = "info",
+  panelClassName = "max-w-sm",
+}: Props) {
+  const Icon = icon;
+  const variantStyle = variants[variant];
+
+  return (
+    <div
+      className={cn(
+        variantStyle.panelStyle,
+        title ? "flex-col" : "",
+        "flex h-fit items-start gap-3 rounded-md border p-4",
+        panelClassName
+      )}
+    >
+      <div className={cn("flex items-center gap-2", to ? "w-full justify-between" : "")}>
+        <Icon className={cn("size-5", iconClassName)} />
+
+        {to && (
+          <LinkButton to={to} variant="secondary/small">
+            {buttonLabel}
+          </LinkButton>
+        )}
+      </div>
+      <div className="flex flex-col gap-1">
+        {title && <Header3 className="text-text-bright">{title}</Header3>}
+        {typeof children === "string" ? (
+          <Paragraph variant={"small"} className="text-text-dimmed">
+            {children}
+          </Paragraph>
+        ) : (
+          children
+        )}
+      </div>
+    </div>
+  );
+}

apps/webapp/app/components/primitives/PageHeader.tsx

@@ -1,10 +1,8 @@
 import { ArrowUpRightIcon } from "@heroicons/react/20/solid";
 import { Link, useNavigation } from "@remix-run/react";
 import { useOptionalOrganization } from "~/hooks/useOrganizations";
-import { useOptionalProject } from "~/hooks/useProject";
 import { cn } from "~/utils/cn";
-import { plansPath } from "~/utils/pathBuilder";
-import { UpgradePrompt, useShowUpgradePrompt } from "../billing/UpgradePrompt";
+import { UpgradePrompt, useShowUpgradePrompt } from "../billing/v3/UpgradePrompt";
 import { BreadcrumbIcon } from "./BreadcrumbIcon";
 import { LinkButton } from "./Buttons";
 import { Header2 } from "./Headers";
@@ -12,6 +10,7 @@ import { LoadingBarDivider } from "./LoadingBarDivider";
 import { NamedIcon } from "./NamedIcon";
 import { Paragraph } from "./Paragraph";
 import { Tabs, TabsProps } from "./Tabs";
+import { ReactNode } from "react";
 
 type WithChildren = {
   children: React.ReactNode;
@@ -23,27 +22,20 @@ export function NavBar({ children }: WithChildren) {
   const showUpgradePrompt = useShowUpgradePrompt(organization);
   const navigation = useNavigation();
   const isLoading = navigation.state === "loading" || navigation.state === "submitting";
-  const project = useOptionalProject();
 
   return (
     <div>
       <div className="grid h-10 w-full grid-rows-[auto_1px] bg-background-bright">
-        <div className="flex w-full items-center justify-between pl-3 pr-1">{children}</div>
+        <div className="flex w-full items-center justify-between pl-3 pr-2">{children}</div>
         <LoadingBarDivider isLoading={isLoading} />
       </div>
-      {showUpgradePrompt.shouldShow && organization && (
-        <UpgradePrompt
-          runsEnabled={showUpgradePrompt.runsEnabled}
-          runCountCap={showUpgradePrompt.runCountCap}
-          planPath={plansPath(organization)}
-        />
-      )}
+      {showUpgradePrompt.shouldShow && organization && <UpgradePrompt />}
     </div>
   );
 }
 
 type PageTitleProps = {
-  title: string;
+  title: ReactNode;
   backButton?: {
     to: string;
     text: string;
@@ -70,7 +62,7 @@ export function PageTitle({ title, backButton }: PageTitleProps) {
 }
 
 export function PageAccessories({ children }: WithChildren) {
-  return <div className="flex items-center gap-3">{children}</div>;
+  return <div className="flex items-center gap-2">{children}</div>;
 }
 
 export function PageInfoRow({ children, className }: WithChildren) {

apps/webapp/app/components/primitives/Popover.tsx

@@ -4,8 +4,8 @@ import { ChevronDownIcon, EllipsisVerticalIcon } from "@heroicons/react/24/solid
 import * as PopoverPrimitive from "@radix-ui/react-popover";
 import * as React from "react";
 import { cn } from "~/utils/cn";
-import { ButtonContentPropsType, LinkButton } from "./Buttons";
-import { Paragraph, ParagraphVariant } from "./Paragraph";
+import { type ButtonContentPropsType, LinkButton } from "./Buttons";
+import { Paragraph, type ParagraphVariant } from "./Paragraph";
 
 const Popover = PopoverPrimitive.Root;
 const PopoverTrigger = PopoverPrimitive.Trigger;
@@ -41,7 +41,7 @@ function PopoverSectionHeader({
   variant?: ParagraphVariant;
 }) {
   return (
-    <Paragraph variant={variant} className="bg-charcoal-900 px-2.5 py-1.5">
+    <Paragraph variant={variant} className="bg-charcoal-750 px-2.5 py-1.5">
       {title}
     </Paragraph>
   );
@@ -71,7 +71,10 @@ function PopoverMenuItem({
       fullWidth
       textAlignLeft
       TrailingIcon={isSelected ? "check" : undefined}
-      className={isSelected ? "bg-charcoal-750 group-hover:bg-charcoal-800" : undefined}
+      className={cn(
+        "group-hover:bg-charcoal-700",
+        isSelected ? "bg-charcoal-750 group-hover:bg-charcoal-600/50" : undefined
+      )}
     >
       {title}
     </LinkButton>

apps/webapp/app/components/primitives/PropertyTable.tsx

@@ -1,40 +1,31 @@
-import { ReactNode } from "react";
+import { type ReactNode } from "react";
 import { Paragraph } from "./Paragraph";
 import { cn } from "~/utils/cn";
 
-export function PropertyTable({
-  children,
-  className,
-}: {
+type ChildrenClassName = {
   children: ReactNode;
   className?: string;
-}) {
-  return (
-    <div className={cn("grid grid-cols-[auto,1fr] items-center gap-x-4 gap-y-2", className)}>
-      {children}
-    </div>
-  );
-}
-
-export type PropertyProps = {
-  label: ReactNode;
-  labelClassName?: string;
-  children: ReactNode;
 };
 
-export function Property({ label, labelClassName, children }: PropertyProps) {
-  return (
-    <>
-      <div className={labelClassName}>
-        {typeof label === "string" ? <Paragraph variant="small">{label}</Paragraph> : label}
-      </div>
-      <div>
-        {typeof children === "string" ? (
-          <Paragraph variant="small/bright">{children}</Paragraph>
-        ) : (
-          children
-        )}
-      </div>
-    </>
-  );
+function PropertyTable({ children, className }: { children: ReactNode; className?: string }) {
+  return <div className={cn("flex flex-col gap-y-3", className)}>{children}</div>;
 }
+
+function PropertyItem({ children, className }: ChildrenClassName) {
+  return <div className={cn("flex flex-col gap-0 text-sm", className)}>{children}</div>;
+}
+
+function PropertyLabel({ children, className }: ChildrenClassName) {
+  return <div className={cn("font-medium text-text-bright", className)}>{children}</div>;
+}
+
+function PropertyValue({ children, className }: ChildrenClassName) {
+  return <div className={cn("text-text-dimmed", className)}>{children}</div>;
+}
+
+export {
+  PropertyTable as Table,
+  PropertyItem as Item,
+  PropertyLabel as Label,
+  PropertyValue as Value,
+};

apps/webapp/app/components/primitives/Sheet.tsx

@@ -28,8 +28,8 @@ interface SheetPortalProps
   extends SheetPrimitive.DialogPortalProps,
     VariantProps<typeof portalVariants> {}
 
-const SheetPortal = ({ position, className, children, ...props }: SheetPortalProps) => (
-  <SheetPrimitive.Portal className={cn(className)} {...props}>
+const SheetPortal = ({ position, children, ...props }: SheetPortalProps) => (
+  <SheetPrimitive.Portal {...props}>
     <div className={portalVariants({ position })}>{children}</div>
   </SheetPrimitive.Portal>
 );

apps/webapp/app/components/primitives/Spinner.tsx

@@ -1,32 +1,37 @@
 import { cn } from "~/utils/cn";
 
+type CustomColor = {
+  background: string;
+  foreground: string;
+};
+
 export function Spinner({
   className,
   color = "blue",
 }: {
   className?: string;
-  color?: "blue" | "white" | "muted" | "dark";
+  color?: "blue" | "white" | "muted" | "dark" | CustomColor;
 }) {
   const colors = {
     blue: {
-      light: "rgba(59, 130, 246, 0.4)",
-      dark: "rgba(59, 130, 246)",
+      background: "rgba(59, 130, 246, 0.4)",
+      foreground: "rgba(59, 130, 246)",
     },
     white: {
-      light: "rgba(255, 255, 255, 0.4)",
-      dark: "rgba(255, 255, 255)",
+      background: "rgba(255, 255, 255, 0.4)",
+      foreground: "rgba(255, 255, 255)",
     },
     muted: {
-      light: "#1C2433",
-      dark: "#3C4B62",
+      background: "#1C2433",
+      foreground: "#3C4B62",
     },
     dark: {
-      light: "#15171A",
-      dark: "#272A2E",
+      background: "#15171A",
+      foreground: "#272A2E",
     },
   };
 
-  const currentColor = colors[color];
+  const currentColor = typeof color === "string" ? colors[color] : color;
 
   return (
     <svg
@@ -37,13 +42,33 @@ export function Spinner({
       xmlns="http://www.w3.org/2000/svg"
       className={cn("animate-spin motion-reduce:hidden", className)}
     >
-      <rect x="2" y="2" width="16" height="16" rx="8" stroke={currentColor.light} strokeWidth="3" />
+      <rect
+        x="2"
+        y="2"
+        width="16"
+        height="16"
+        rx="8"
+        stroke={currentColor.background}
+        strokeWidth="3"
+      />
       <path
         d="M10 18C5.58172 18 2 14.4183 2 10C2 5.58172 5.58172 2 10 2"
-        stroke={currentColor.dark}
+        stroke={currentColor.foreground}
         strokeWidth="3"
         strokeLinecap="round"
       />
     </svg>
   );
 }
+
+export function ButtonSpinner() {
+  return (
+    <Spinner
+      className="size-3"
+      color={{
+        foreground: "rgba(0, 0, 0, 1)",
+        background: "rgba(0, 0, 0, 0.25)",
+      }}
+    />
+  );
+}

apps/webapp/app/components/primitives/Table.tsx

@@ -3,6 +3,7 @@ import { Link } from "@remix-run/react";
 import { ReactNode, forwardRef, useState } from "react";
 import { cn } from "~/utils/cn";
 import { Popover, PopoverContent, PopoverVerticalEllipseTrigger } from "./Popover";
+import { InfoIconTooltip } from "./Tooltip";
 
 type TableProps = {
   containerClassName?: string;
@@ -87,10 +88,11 @@ type TableCellBasicProps = {
 
 type TableHeaderCellProps = TableCellBasicProps & {
   hiddenLabel?: boolean;
+  tooltip?: ReactNode;
 };
 
 export const TableHeaderCell = forwardRef<HTMLTableCellElement, TableHeaderCellProps>(
-  ({ className, alignment = "left", children, colSpan, hiddenLabel = false }, ref) => {
+  ({ className, alignment = "left", children, colSpan, hiddenLabel = false, tooltip }, ref) => {
     let alignmentClassName = "text-left";
     switch (alignment) {
       case "center":
@@ -112,7 +114,16 @@ export const TableHeaderCell = forwardRef<HTMLTableCellElement, TableHeaderCellP
         )}
         colSpan={colSpan}
       >
-        {hiddenLabel ? <span className="sr-only">{children}</span> : children}
+        {hiddenLabel ? (
+          <span className="sr-only">{children}</span>
+        ) : tooltip ? (
+          <div className="flex items-center gap-1">
+            {children}
+            <InfoIconTooltip content={tooltip} contentClassName="normal-case tracking-normal" />
+          </div>
+        ) : (
+          children
+        )}
       </th>
     );
   }

apps/webapp/app/components/primitives/Tabs.tsx

@@ -1,6 +1,11 @@
-import { NavLink, useLocation } from "@remix-run/react";
+import { Link, NavLink, useLocation } from "@remix-run/react";
 import { motion } from "framer-motion";
+import { ReactNode, useRef } from "react";
+import { useOptimisticLocation } from "~/hooks/useOptimisticLocation";
+import { ShortcutDefinition, useShortcutKeys } from "~/hooks/useShortcutKeys";
 import { cn } from "~/utils/cn";
+import { projectPubSub } from "~/v3/services/projectPubSub.server";
+import { ShortcutKey } from "./ShortcutKey";
 
 export type TabsProps = {
   tabs: {
@@ -13,28 +18,106 @@ export type TabsProps = {
 
 export function Tabs({ tabs, className, layoutId }: TabsProps) {
   return (
-    <div className={cn(`flex flex-row gap-x-6 border-b border-grid-bright`, className)}>
+    <TabContainer className={className}>
       {tabs.map((tab, index) => (
-        <NavLink key={index} to={tab.to} className="group flex flex-col items-center pt-1" end>
-          {({ isActive, isPending }) => (
-            <>
-              <span
-                className={cn(
-                  "text-sm transition duration-200",
-                  isActive || isPending ? "text-indigo-500" : "text-charcoal-200"
-                )}
-              >
-                {tab.label}
-              </span>
-              {isActive || isPending ? (
-                <motion.div layoutId={layoutId} className="mt-1 h-0.5 w-full bg-indigo-500" />
-              ) : (
-                <div className="mt-1 h-0.5 w-full bg-charcoal-500 opacity-0 transition duration-200 group-hover:opacity-100" />
-              )}
-            </>
-          )}
-        </NavLink>
+        <TabLink key={index} to={tab.to} layoutId={layoutId}>
+          {tab.label}
+        </TabLink>
       ))}
+    </TabContainer>
+  );
+}
+
+export function TabContainer({ children, className }: { children: ReactNode; className?: string }) {
+  return (
+    <div className={cn(`flex flex-row gap-x-6 border-b border-grid-bright`, className)}>
+      {children}
     </div>
   );
 }
+
+export function TabLink({
+  to,
+  children,
+  layoutId,
+}: {
+  to: string;
+  children: ReactNode;
+  layoutId: string;
+}) {
+  return (
+    <NavLink to={to} className="group flex flex-col items-center pt-1" end>
+      {({ isActive, isPending }) => {
+        return (
+          <>
+            <span
+              className={cn(
+                "text-sm transition duration-200",
+                isActive || isPending ? "text-text-bright" : "text-text-bright"
+              )}
+            >
+              {children}
+            </span>
+            {isActive || isPending ? (
+              <motion.div layoutId={layoutId} className="mt-1 h-0.5 w-full bg-indigo-500" />
+            ) : (
+              <div className="mt-1 h-0.5 w-full bg-charcoal-500 opacity-0 transition duration-200 group-hover:opacity-100" />
+            )}
+          </>
+        );
+      }}
+    </NavLink>
+  );
+}
+
+export function TabButton({
+  isActive,
+  layoutId,
+  shortcut,
+  ...props
+}: {
+  isActive: boolean;
+  shortcut?: ShortcutDefinition;
+  layoutId: string;
+} & React.ButtonHTMLAttributes<HTMLButtonElement>) {
+  const ref = useRef<HTMLButtonElement>(null);
+
+  if (shortcut) {
+    useShortcutKeys({
+      shortcut: shortcut,
+      action: () => {
+        if (ref.current) {
+          ref.current.click();
+        }
+      },
+      disabled: props.disabled,
+    });
+  }
+
+  return (
+    <button
+      className={cn("group flex flex-col items-center pt-1", props.className)}
+      ref={ref}
+      {...props}
+    >
+      <>
+        <div className="flex items-center gap-1">
+          <span
+            className={cn(
+              "text-sm transition duration-200",
+              isActive ? "text-text-bright" : "text-text-bright"
+            )}
+          >
+            {props.children}
+          </span>
+          {shortcut && <ShortcutKey className={cn("")} shortcut={shortcut} variant={"small"} />}
+        </div>
+        {isActive ? (
+          <motion.div layoutId={layoutId} className="mt-1 h-0.5 w-full bg-indigo-500" />
+        ) : (
+          <div className="mt-1 h-0.5 w-full bg-charcoal-500 opacity-0 transition duration-200 group-hover:opacity-100" />
+        )}
+      </>
+    </button>
+  );
+}

apps/webapp/app/components/primitives/Tooltip.tsx

@@ -5,7 +5,7 @@ import { InformationCircleIcon } from "@heroicons/react/20/solid";
 
 const variantClasses = {
   basic:
-    "bg-background-dimmed border border-charcoal-700 rounded-md px-3 py-1.5 text-sm text-text-bright shadow-md fade-in-50",
+    "bg-background-bright border border-grid-bright rounded px-3 py-2 text-sm text-text-bright shadow-md fade-in-50",
   dark: "bg-background-dimmed border border-grid-bright rounded px-3 py-2 text-sm text-text-bright shadow-md fade-in-50",
 };
 
@@ -58,6 +58,7 @@ function SimpleTooltip({
   variant,
   disableHoverableContent = false,
   className,
+  buttonClassName,
 }: {
   button: React.ReactNode;
   content: React.ReactNode;
@@ -66,11 +67,12 @@ function SimpleTooltip({
   variant?: Variant;
   disableHoverableContent?: boolean;
   className?: string;
+  buttonClassName?: string;
 }) {
   return (
     <TooltipProvider disableHoverableContent={disableHoverableContent}>
       <Tooltip>
-        <TooltipTrigger className="h-fit">{button}</TooltipTrigger>
+        <TooltipTrigger className={cn("h-fit", buttonClassName)}>{button}</TooltipTrigger>
         <TooltipContent
           side={side}
           hidden={hidden}
@@ -87,17 +89,22 @@ function SimpleTooltip({
 export function InfoIconTooltip({
   content,
   buttonClassName,
+  contentClassName,
+  variant = "basic",
 }: {
   content: React.ReactNode;
   buttonClassName?: string;
+  contentClassName?: string;
+  variant?: Variant;
 }) {
   return (
     <SimpleTooltip
       button={
-        <InformationCircleIcon className={cn("h-3.5 w-3.5 text-text-dimmed", buttonClassName)} />
+        <InformationCircleIcon className={cn("size-3.5 text-text-dimmed", buttonClassName)} />
       }
       content={content}
-      variant="dark"
+      variant={variant}
+      className={contentClassName}
     />
   );
 }

apps/webapp/app/components/runs/v3/CancelRunDialog.tsx

@@ -23,8 +23,7 @@ export function CancelRunDialog({ runFriendlyId, redirectPath }: CancelRunDialog
     <DialogContent key="cancel">
       <DialogHeader>Cancel this run?</DialogHeader>
       <DialogDescription>
-        Canceling a run will stop execution. If you want to run this later you will have to replay
-        the entire run with the original payload.
+        Canceling a run will stop execution, along with any executing subtasks.
       </DialogDescription>
       <DialogFooter>
         <Form action={`/resources/taskruns/${runFriendlyId}/cancel`} method="post">

apps/webapp/app/components/runs/v3/DeploymentStatus.tsx

@@ -11,22 +11,32 @@ import { cn } from "~/utils/cn";
 
 export function DeploymentStatus({
   status,
+  isBuilt,
   className,
 }: {
   status: WorkerDeploymentStatus;
+  isBuilt: boolean;
   className?: string;
 }) {
   return (
     <span className={cn("flex items-center gap-1", className)}>
       <DeploymentStatusIcon status={status} className="h-4 w-4" />
-      <DeploymentStatusLabel status={status} />
+      <DeploymentStatusLabel status={status} isBuilt={isBuilt} />
     </span>
   );
 }
 
-export function DeploymentStatusLabel({ status }: { status: WorkerDeploymentStatus }) {
+export function DeploymentStatusLabel({
+  status,
+  isBuilt,
+}: {
+  status: WorkerDeploymentStatus;
+  isBuilt: boolean;
+}) {
   return (
-    <span className={deploymentStatusClassNameColor(status)}>{deploymentStatusTitle(status)}</span>
+    <span className={deploymentStatusClassNameColor(status)}>
+      {deploymentStatusTitle(status, isBuilt)}
+    </span>
   );
 }
 
@@ -79,7 +89,7 @@ export function deploymentStatusClassNameColor(status: WorkerDeploymentStatus):
   }
 }
 
-export function deploymentStatusTitle(status: WorkerDeploymentStatus): string {
+export function deploymentStatusTitle(status: WorkerDeploymentStatus, isBuilt: boolean): string {
   switch (status) {
     case "PENDING":
       return "Pending…";
@@ -92,9 +102,17 @@ export function deploymentStatusTitle(status: WorkerDeploymentStatus): string {
     case "CANCELED":
       return "Canceled";
     case "TIMED_OUT":
-      return "Timed out";
+      if (!isBuilt) {
+        return "Build timed out";
+      }
+
+      return "Indexing timed out";
     case "FAILED":
-      return "Failed";
+      if (!isBuilt) {
+        return "Build failed";
+      }
+
+      return "Indexing failed";
     default: {
       assertNever(status);
     }

apps/webapp/app/components/runs/v3/LiveTimer.tsx

@@ -23,11 +23,11 @@ export function LiveTimer({
     }, updateInterval);
 
     return () => clearInterval(interval);
-  }, [startTime]);
+  }, [startTime, endTime]);
 
   return (
     <>
-      {formatDuration(startTime, now, {
+      {formatDuration(startTime, endTime ?? now, {
         style: "short",
         maxDecimalPoints: 0,
         units: ["d", "h", "m", "s"],

apps/webapp/app/components/runs/v3/ReplayRunDialog.tsx

@@ -1,12 +1,16 @@
-import { ArrowPathIcon } from "@heroicons/react/20/solid";
-import { Form, useFetcher, useNavigation } from "@remix-run/react";
+import { Form, useFetcher, useNavigation, useSubmit } from "@remix-run/react";
+import { useCallback, useEffect, useRef } from "react";
+import { UseDataFunctionReturn, useTypedFetcher } from "remix-typedjson";
+import { JSONEditor } from "~/components/code/JSONEditor";
+import { EnvironmentLabel } from "~/components/environments/EnvironmentLabel";
 import { Button } from "~/components/primitives/Buttons";
-import {
-  DialogContent,
-  DialogDescription,
-  DialogFooter,
-  DialogHeader,
-} from "~/components/primitives/Dialog";
+import { DialogContent, DialogDescription, DialogHeader } from "~/components/primitives/Dialog";
+import { Header3 } from "~/components/primitives/Headers";
+import { InputGroup } from "~/components/primitives/InputGroup";
+import { Label } from "~/components/primitives/Label";
+import { Select, SelectItem } from "~/components/primitives/Select";
+import { ButtonSpinner, Spinner } from "~/components/primitives/Spinner";
+import { type loader } from "~/routes/resources.taskruns.$runParam.replay";
 
 type ReplayRunDialogProps = {
   runFriendlyId: string;
@@ -14,31 +18,142 @@ type ReplayRunDialogProps = {
 };
 
 export function ReplayRunDialog({ runFriendlyId, failedRedirect }: ReplayRunDialogProps) {
-  const navigation = useNavigation();
-
-  const formAction = `/resources/taskruns/${runFriendlyId}/replay`;
-  const isLoading = navigation.formAction === formAction;
-
   return (
-    <DialogContent key="replay">
-      <DialogHeader>Replay this run?</DialogHeader>
-      <DialogDescription>
-        Replaying a run will create a new run with the same payload and environment as the original.
-      </DialogDescription>
-      <DialogFooter>
-        <Form action={formAction} method="post">
-          <input type="hidden" name="failedRedirect" value={failedRedirect} />
-          <Button
-            type="submit"
-            variant="primary/small"
-            LeadingIcon={isLoading ? "spinner-white" : ArrowPathIcon}
-            disabled={isLoading}
-            shortcut={{ modifiers: ["meta"], key: "enter" }}
-          >
-            {isLoading ? "Replaying..." : "Replay run"}
-          </Button>
-        </Form>
-      </DialogFooter>
+    <DialogContent key={`replay`} className="md:max-w-3xl">
+      <ReplayContent runFriendlyId={runFriendlyId} failedRedirect={failedRedirect} />
     </DialogContent>
   );
 }
+
+function ReplayContent({ runFriendlyId, failedRedirect }: ReplayRunDialogProps) {
+  const fetcher = useTypedFetcher<typeof loader>();
+  const isLoading = fetcher.state !== "idle";
+
+  useEffect(() => {
+    fetcher.load(`/resources/taskruns/${runFriendlyId}/replay`);
+  }, [runFriendlyId]);
+
+  return (
+    <>
+      <DialogHeader>Replay this run</DialogHeader>
+      {isLoading ? (
+        <div className="grid place-items-center p-6">
+          <Spinner />
+        </div>
+      ) : fetcher.data ? (
+        <ReplayForm
+          {...fetcher.data}
+          failedRedirect={failedRedirect}
+          runFriendlyId={runFriendlyId}
+        />
+      ) : (
+        <>Failed to get run data</>
+      )}
+    </>
+  );
+}
+
+function ReplayForm({
+  payload,
+  payloadType,
+  environment,
+  environments,
+  failedRedirect,
+  runFriendlyId,
+}: UseDataFunctionReturn<typeof loader> & { failedRedirect: string; runFriendlyId: string }) {
+  const navigation = useNavigation();
+  const submit = useSubmit();
+  const currentJson = useRef<string>(payload);
+  const formAction = `/resources/taskruns/${runFriendlyId}/replay`;
+  const isSubmitting = navigation.formAction === formAction;
+
+  const editablePayload =
+    payloadType === "application/json" || payloadType === "application/super+json";
+
+  const submitForm = useCallback(
+    (e: React.FormEvent<HTMLFormElement>) => {
+      const formData = new FormData(e.currentTarget);
+      const data: Record<string, string> = {
+        environment: formData.get("environment") as string,
+        failedRedirect: formData.get("failedRedirect") as string,
+      };
+
+      if (editablePayload) {
+        data.payload = currentJson.current;
+      }
+
+      submit(data, {
+        action: formAction,
+        method: "post",
+      });
+      e.preventDefault();
+    },
+    [currentJson]
+  );
+
+  return (
+    <Form action={formAction} method="post" onSubmit={(e) => submitForm(e)} className="pt-2">
+      {editablePayload ? (
+        <>
+          <Header3 spacing>Payload</Header3>
+          <div className="mb-3 max-h-[70vh] overflow-y-auto rounded-sm border border-grid-dimmed bg-charcoal-900 scrollbar-thin scrollbar-track-transparent scrollbar-thumb-charcoal-600">
+            <JSONEditor
+              autoFocus
+              defaultValue={currentJson.current}
+              readOnly={false}
+              basicSetup
+              onChange={(v) => {
+                currentJson.current = v;
+              }}
+              showClearButton={false}
+              showCopyButton={false}
+              height="100%"
+              min-height="100%"
+              max-height="100%"
+            />
+          </div>
+        </>
+      ) : null}
+      <InputGroup>
+        <Label>Environment</Label>
+        <Select
+          id="environment"
+          name="environment"
+          placeholder="Select an environment"
+          defaultValue={environment.id}
+          items={environments}
+          dropdownIcon
+          variant="tertiary/medium"
+          className="w-fit pl-2"
+          text={(value) => {
+            const env = environments.find((env) => env.id === value)!;
+            return (
+              <div className="flex items-center pr-2">
+                <EnvironmentLabel environment={env} userName={env.userName} />
+              </div>
+            );
+          }}
+        >
+          {(matches) =>
+            matches.map((env) => (
+              <SelectItem key={env.id} value={env.id}>
+                <EnvironmentLabel environment={env} userName={env.userName} />
+              </SelectItem>
+            ))
+          }
+        </Select>
+      </InputGroup>
+      <input type="hidden" name="failedRedirect" value={failedRedirect} />
+      <Button
+        type="submit"
+        variant="primary/medium"
+        LeadingIcon={isSubmitting ? ButtonSpinner : undefined}
+        disabled={isSubmitting}
+        shortcut={{ modifiers: ["meta"], key: "enter", enabledOnInputElements: true }}
+        className="mt-5"
+      >
+        {isSubmitting ? "Replaying..." : "Replay run"}
+      </Button>
+    </Form>
+  );
+}

apps/webapp/app/components/runs/v3/RetryDeploymentIndexingDialog.tsx

@@ -0,0 +1,55 @@
+import { ArrowPathIcon } from "@heroicons/react/20/solid";
+import { Form, useNavigation } from "@remix-run/react";
+import { Button } from "~/components/primitives/Buttons";
+import {
+  DialogContent,
+  DialogDescription,
+  DialogFooter,
+  DialogHeader,
+} from "~/components/primitives/Dialog";
+
+type RetryDeploymentIndexingDialogProps = {
+  projectId: string;
+  deploymentShortCode: string;
+  redirectPath: string;
+};
+
+export function RetryDeploymentIndexingDialog({
+  projectId,
+  deploymentShortCode,
+  redirectPath,
+}: RetryDeploymentIndexingDialogProps) {
+  const navigation = useNavigation();
+
+  const formAction = `/resources/${projectId}/deployments/${deploymentShortCode}/retry-indexing`;
+  const isLoading = navigation.formAction === formAction;
+
+  return (
+    <DialogContent key="retry-indexing">
+      <DialogHeader>Retry indexing this deployment?</DialogHeader>
+      <DialogDescription>
+        Retrying can be useful if indexing failed due to missing environment variables. Make sure
+        you set them before retrying. In most other cases, indexing will keep failing until you fix
+        any errors and re-deploy.
+      </DialogDescription>
+      <DialogFooter>
+        <Form
+          action={`/resources/${projectId}/deployments/${deploymentShortCode}/retry-indexing`}
+          method="post"
+        >
+          <Button
+            type="submit"
+            name="redirectUrl"
+            value={redirectPath}
+            variant="primary/small"
+            LeadingIcon={isLoading ? "spinner-white" : ArrowPathIcon}
+            disabled={isLoading}
+            shortcut={{ modifiers: ["meta"], key: "enter" }}
+          >
+            {isLoading ? "Retrying..." : "Retry indexing"}
+          </Button>
+        </Form>
+      </DialogFooter>
+    </DialogContent>
+  );
+}

apps/webapp/app/components/runs/v3/RunFilters.tsx

@@ -4,9 +4,10 @@ import {
   CalendarIcon,
   CpuChipIcon,
   InboxStackIcon,
+  TagIcon,
   XMarkIcon,
 } from "@heroicons/react/20/solid";
-import { Form } from "@remix-run/react";
+import { Form, useFetcher } from "@remix-run/react";
 import type {
   RuntimeEnvironment,
   TaskTriggerSource,
@@ -15,7 +16,7 @@ import type {
 } from "@trigger.dev/database";
 import { ListFilterIcon } from "lucide-react";
 import type { ReactNode } from "react";
-import { startTransition, useCallback, useMemo, useState } from "react";
+import { startTransition, useCallback, useEffect, useMemo, useState } from "react";
 import { z } from "zod";
 import { TaskIcon } from "~/assets/icons/TaskIcon";
 import { EnvironmentLabel, environmentTitle } from "~/components/environments/EnvironmentLabel";
@@ -51,6 +52,10 @@ import {
 import { TaskTriggerSourceIcon } from "./TaskTriggerSource";
 import { DateTime } from "~/components/primitives/DateTime";
 import { BulkActionStatusCombo } from "./BulkAction";
+import { type loader } from "~/routes/resources.projects.$projectParam.runs.tags";
+import { useProject } from "~/hooks/useProject";
+import { Spinner } from "~/components/primitives/Spinner";
+import { matchSorter } from "match-sorter";
 
 export const TaskAttemptStatus = z.enum(allTaskRunStatuses);
 
@@ -73,6 +78,10 @@ export const TaskRunListSearchFilters = z.object({
     (value) => (typeof value === "string" ? [value] : value),
     TaskAttemptStatus.array().optional()
   ),
+  tags: z.preprocess(
+    (value) => (typeof value === "string" ? [value] : value),
+    z.string().array().optional()
+  ),
   period: z.preprocess((value) => (value === "all" ? undefined : value), z.string().optional()),
   bulkId: z.string().optional(),
   from: z.coerce.number().optional(),
@@ -104,7 +113,8 @@ export function RunsFilters(props: RunFiltersProps) {
     searchParams.has("environments") ||
     searchParams.has("tasks") ||
     searchParams.has("period") ||
-    searchParams.has("bulkId");
+    searchParams.has("bulkId") ||
+    searchParams.has("tags");
 
   return (
     <div className="flex flex-row flex-wrap items-center gap-1">
@@ -133,6 +143,7 @@ const filterTypes = [
   },
   { name: "environments", title: "Environment", icon: <CpuChipIcon className="size-4" /> },
   { name: "tasks", title: "Tasks", icon: <TaskIcon className="size-4" /> },
+  { name: "tags", title: "Tags", icon: <TagIcon className="size-4" /> },
   { name: "created", title: "Created", icon: <CalendarIcon className="size-4" /> },
   { name: "bulk", title: "Bulk action", icon: <InboxStackIcon className="size-4" /> },
 ] as const;
@@ -146,7 +157,6 @@ function FilterMenu(props: RunFiltersProps) {
 
   const filterTrigger = (
     <SelectTrigger
-      autoFocus
       icon={
         <div className="flex size-4 items-center justify-center">
           <ListFilterIcon className="size-3.5" />
@@ -210,6 +220,7 @@ function AppliedFilters({ possibleEnvironments, possibleTasks, bulkActions }: Ru
       <AppliedStatusFilter />
       <AppliedEnvironmentFilter possibleEnvironments={possibleEnvironments} />
       <AppliedTaskFilter possibleTasks={possibleTasks} />
+      <AppliedTagsFilter />
       <AppliedPeriodFilter />
       <AppliedBulkActionsFilter bulkActions={bulkActions} />
     </>
@@ -238,6 +249,8 @@ function Menu(props: MenuProps) {
       return <CreatedDropdown onClose={() => props.setFilterType(undefined)} {...props} />;
     case "bulk":
       return <BulkActionsDropdown onClose={() => props.setFilterType(undefined)} {...props} />;
+    case "tags":
+      return <TagsDropdown onClose={() => props.setFilterType(undefined)} {...props} />;
   }
 }
 
@@ -649,6 +662,124 @@ function AppliedBulkActionsFilter({ bulkActions }: Pick<RunFiltersProps, "bulkAc
   );
 }
 
+function TagsDropdown({
+  trigger,
+  clearSearchValue,
+  searchValue,
+  onClose,
+}: {
+  trigger: ReactNode;
+  clearSearchValue: () => void;
+  searchValue: string;
+  onClose?: () => void;
+}) {
+  const project = useProject();
+  const { values, replace } = useSearchParams();
+
+  const handleChange = (values: string[]) => {
+    clearSearchValue();
+    replace({
+      tags: values,
+      cursor: undefined,
+      direction: undefined,
+    });
+  };
+
+  const fetcher = useFetcher<typeof loader>();
+
+  useEffect(() => {
+    const searchParams = new URLSearchParams();
+    if (searchValue) {
+      searchParams.set("name", encodeURIComponent(searchValue));
+    }
+    fetcher.load(`/resources/projects/${project.slug}/runs/tags?${searchParams}`);
+  }, [searchValue]);
+
+  const filtered = useMemo(() => {
+    let items: string[] = [];
+    if (searchValue === "") {
+      items = values("tags");
+    }
+
+    if (fetcher.data === undefined) {
+      return matchSorter(items, searchValue);
+    }
+
+    items.push(...fetcher.data.tags.map((t) => t.name));
+
+    return matchSorter(Array.from(new Set(items)), searchValue);
+  }, [searchValue, fetcher.data]);
+
+  return (
+    <SelectProvider value={values("tags")} setValue={handleChange} virtualFocus={true}>
+      {trigger}
+      <SelectPopover
+        className="min-w-0 max-w-[min(240px,var(--popover-available-width))]"
+        hideOnEscape={() => {
+          if (onClose) {
+            onClose();
+            return false;
+          }
+
+          return true;
+        }}
+      >
+        <ComboBox
+          value={searchValue}
+          render={(props) => (
+            <div className="flex items-center justify-stretch">
+              <input {...props} placeholder={"Filter by tags..."} />
+              {fetcher.state === "loading" && <Spinner color="muted" />}
+            </div>
+          )}
+        />
+        <SelectList>
+          {filtered.length > 0
+            ? filtered.map((tag, index) => (
+                <SelectItem key={tag} value={tag}>
+                  {tag}
+                </SelectItem>
+              ))
+            : null}
+          {filtered.length === 0 && fetcher.state !== "loading" && (
+            <SelectItem disabled>No tags found</SelectItem>
+          )}
+        </SelectList>
+      </SelectPopover>
+    </SelectProvider>
+  );
+}
+
+function AppliedTagsFilter() {
+  const { values, del } = useSearchParams();
+
+  const tags = values("tags");
+
+  if (tags.length === 0) {
+    return null;
+  }
+
+  return (
+    <FilterMenuProvider>
+      {(search, setSearch) => (
+        <TagsDropdown
+          trigger={
+            <Ariakit.Select render={<div className="group cursor-pointer" />}>
+              <AppliedFilter
+                label="Tags"
+                value={appliedSummary(values("tags"))}
+                onRemove={() => del(["tags", "cursor", "direction"])}
+              />
+            </Ariakit.Select>
+          }
+          searchValue={search}
+          clearSearchValue={() => setSearch("")}
+        />
+      )}
+    </FilterMenuProvider>
+  );
+}
+
 const timePeriods = [
   {
     label: "All periods",

apps/webapp/app/components/runs/v3/RunIcon.tsx

@@ -3,6 +3,7 @@ import {
   HandRaisedIcon,
   InformationCircleIcon,
   Squares2X2Icon,
+  TagIcon,
 } from "@heroicons/react/20/solid";
 import { AttemptIcon } from "~/assets/icons/AttemptIcon";
 import { TaskIcon } from "~/assets/icons/TaskIcon";
@@ -48,6 +49,8 @@ export function RunIcon({ name, className, spanName }: TaskIconProps) {
       return <ClockIcon className={cn(className, "text-teal-500")} />;
     case "trace":
       return <Squares2X2Icon className={cn(className, "text-text-dimmed")} />;
+    case "tag":
+      return <TagIcon className={cn(className, "text-text-dimmed")} />;
     //log levels
     case "debug":
     case "log":

apps/webapp/app/components/runs/v3/RunTag.tsx

@@ -0,0 +1,48 @@
+import { useMemo } from "react";
+import tagLeftPath from "./tag-left.svg";
+
+type Tag = string | { key: string; value: string };
+
+export function RunTag({ tag }: { tag: string }) {
+  const tagResult = useMemo(() => splitTag(tag), [tag]);
+
+  if (typeof tagResult === "string") {
+    return (
+      <span className="flex h-6 items-stretch">
+        <img src={tagLeftPath} alt="" className="block h-full w-[0.5625rem]" />
+        <span className="flex items-center rounded-r-sm border-y border-r border-charcoal-700 bg-charcoal-800 pr-1.5 text-text-dimmed">
+          {tag}
+        </span>
+      </span>
+    );
+  } else {
+    return (
+      <span className="flex h-6 items-stretch">
+        <img src={tagLeftPath} alt="" className="block h-full w-[0.5625rem]" />
+        <span className="flex items-center border-y border-r border-charcoal-700 bg-charcoal-800 pr-1.5 text-text-dimmed">
+          {tagResult.key}
+        </span>
+        <span className="flex items-center rounded-r-sm border-y border-r border-charcoal-700 bg-charcoal-750 px-1.5 text-text-dimmed">
+          {tagResult.value}
+        </span>
+      </span>
+    );
+  }
+}
+
+/** Takes a string and turns it into a tag
+ *
+ * If the string has 12 or fewer alpha characters followed by an underscore or colon then we return an object with a key and value
+ * Otherwise we return the original string
+ */
+function splitTag(tag: string): Tag {
+  if (tag.match(/^[a-zA-Z]{1,12}[_:]/)) {
+    const components = tag.split(/[_:]/);
+    if (components.length !== 2) {
+      return tag;
+    }
+    return { key: components[0], value: components[1] };
+  }
+
+  return tag;
+}

apps/webapp/app/components/runs/v3/ScheduleFilters.tsx

@@ -1,6 +1,6 @@
 import { XMarkIcon } from "@heroicons/react/20/solid";
 import { useNavigate } from "@remix-run/react";
-import { RuntimeEnvironment } from "@trigger.dev/database";
+import { type RuntimeEnvironment } from "@trigger.dev/database";
 import { useCallback } from "react";
 import { z } from "zod";
 import { Input } from "~/components/primitives/Input";
@@ -17,6 +17,7 @@ import {
   SelectTrigger,
   SelectValue,
 } from "../../primitives/SimpleSelect";
+import { ScheduleTypeCombo } from "./ScheduleType";
 
 export const ScheduleListFilters = z.object({
   page: z.coerce.number().default(1),
@@ -28,6 +29,7 @@ export const ScheduleListFilters = z.object({
     .string()
     .optional()
     .transform((value) => (value ? value.split(",") : undefined)),
+  type: z.union([z.literal("declarative"), z.literal("imperative")]).optional(),
   search: z.string().optional(),
 });
 
@@ -48,7 +50,7 @@ export function ScheduleFilters({ possibleEnvironments, possibleTasks }: Schedul
   const navigate = useNavigate();
   const location = useOptimisticLocation();
   const searchParams = new URLSearchParams(location.search);
-  const { environments, tasks, page, search } = ScheduleListFilters.parse(
+  const { environments, tasks, page, search, type } = ScheduleListFilters.parse(
     Object.fromEntries(searchParams.entries())
   );
 
@@ -73,6 +75,10 @@ export function ScheduleFilters({ possibleEnvironments, possibleTasks }: Schedul
     handleFilterChange("environments", value === "ALL" ? undefined : value);
   }, []);
 
+  const handleTypeChange = useCallback((value: string | typeof All) => {
+    handleFilterChange("type", value === "ALL" ? undefined : value);
+  }, []);
+
   const handleSearchChange = useThrottle((value: string) => {
     handleFilterChange("search", value.length === 0 ? undefined : value);
   }, 300);
@@ -97,6 +103,30 @@ export function ScheduleFilters({ possibleEnvironments, possibleTasks }: Schedul
         defaultValue={search}
         onChange={(e) => handleSearchChange(e.target.value)}
       />
+      <SelectGroup>
+        <Select name="type" value={type ?? "ALL"} onValueChange={handleTypeChange}>
+          <SelectTrigger size="minimal" width="full">
+            <SelectValue placeholder={"Select type"} className="ml-2 whitespace-nowrap p-0" />
+          </SelectTrigger>
+          <SelectContent>
+            <SelectItem value={"ALL"}>
+              <Paragraph
+                variant="extra-small"
+                className="whitespace-nowrap pl-0.5 transition group-hover:text-text-bright"
+              >
+                All types
+              </Paragraph>
+            </SelectItem>
+            <SelectItem value={"declarative"}>
+              <ScheduleTypeCombo type="DECLARATIVE" className="text-xs text-text-dimmed" />
+            </SelectItem>
+            <SelectItem value={"imperative"}>
+              <ScheduleTypeCombo type="IMPERATIVE" className="text-xs text-text-dimmed" />
+            </SelectItem>
+          </SelectContent>
+        </Select>
+      </SelectGroup>
+
       <SelectGroup>
         <Select
           name="environment"