feat: realtime (#1402)

* Denormalize run tags, increase character limit to 128

* WIP realtime subscribing to runs

* extracted the stream stuff into core, made it more reusable

* WIP tags

* Remove tags for now because it’s not support in electric

* Support async iterables, readable stream, and callback style subscription styles

* Remove tags streaming endpoint

* Add realtime rate limits and scope them to the /realtime path

* WIP rate limt per org

* Introduce per org rate limits

* WIP JWT auth

* Move migrations into new internal db package

* Resolve pnpm lock file

* Authenticating to the realtime API with JWTs are working

* realtime in the client

* Created react-hooks package and starting to move stuff in there

* Improve types for hooks

* schema tasks

* Added useBatch hook

* build uploadthing/fal demo and change how run metadata is synced to the server

* tweaks

* WIL realtime concurrency tracking

* Implement test for realtime client using testcontainers

also updated electric to latest version

* Allow customizing the expiration time of the automatic JWT created after triggering a task

* Add support for subscribing to run tags

* Improve auth types and API

* finalize the realtime API

* Fixed some example stuff

* Allow up to 10 run tags

* Remove core from docker-provider tsconfig paths to prevent it from being typechecked

* do the same for the kubernetes provider

* Fixing some typecheck errors

* Fix webapp type errors

* Update @trigger.dev/platform to 1.0.13

* Fix attw error

* Remove from/to in subscribeToRuns query params

* Add tests for the rate limit middleware and add custom JWT rate limits

* turn off webapp test parallelism

* Finish renaming jwt -> publicAccessToken and automatically give the JWT read access to the tags when using trigger

* Add changeset

* Attempt to fix unit tests in CI

* Skip running the auth rate limit middleware tests for now

* Try a beefier machine

* Try and run webapp tests separately

* Setup env vars

* Make sliding window test more reliabile

.github/workflows/release.yml

@@ -17,7 +17,7 @@ concurrency:
 jobs:
   release:
     name: 🦋 Changesets Release
-    runs-on: buildjet-8vcpu-ubuntu-2204
+    runs-on: ubuntu-latest
     if: github.repository == 'triggerdotdev/trigger.dev'
     outputs:
       published: ${{ steps.changesets.outputs.published }}

.github/workflows/unit-tests.yml

@@ -6,7 +6,7 @@ on:
 jobs:
   unitTests:
     name: "🧪 Unit Tests"
-    runs-on: buildjet-8vcpu-ubuntu-2204
+    runs-on: buildjet-16vcpu-ubuntu-2204
     steps:
       - name: ⬇️ Checkout repo
         uses: actions/checkout@v4
@@ -30,5 +30,15 @@ jobs:
       - name: 📀 Generate Prisma Client
         run: pnpm run generate
 
-      - name: 🧪 Run Unit Tests
-        run: pnpm run test
+      - name: 🧪 Run Webapp Unit Tests
+        run: pnpm run test --filter webapp
+        env:
+          DATABASE_URL: postgresql://postgres:postgres@localhost:5432/postgres
+          DIRECT_URL: postgresql://postgres:postgres@localhost:5432/postgres
+          SESSION_SECRET: "secret"
+          MAGIC_LINK_SECRET: "secret"
+          ENCRYPTION_KEY: "secret"
+
+      
+      - name: 🧪 Run Internal Unit Tests
+        run: pnpm run test --filter "@internal/*"