Files
triggerdotdev--trigger.dev/references
Eric Allam 65da20c225 feat: replicate task runs to clickhouse to power dashboard improvements (#2035)
* WIP clickhouse package with test containers setup

* More clickhouse client setup now with otel and real tests, and the v1 of raw run events

* Add some additional columns to raw_run_events_v1

* WIP runs dashboard service

* Create a new run engine event bus event for the runs dashboard to hook into

* Track run events in the run engine

* make sure engine v1 runs get synced to CH

* Update the attemptNumber of v3 task runs

* Restructure the run events to be more sparse

* emit more stuff

* Setup replication package

* scaffold the replication package

* replication wip

* resolve conflicts

* more replication stuff

* Add ability to drop the replication slot completely on teardown

* Use the new single replacingmergetree task events table for replication

* get it working

* insert payloads into their own table only on insert and then join

* prepare for using clickhouse cloud and now running ch migrations during boot in the entrypoint.sh

* Handover WIP and tests

* Testing the replication service

* Remove the runs dashboard stuff that we aren't using anymore

* Added a test for large payloads

* hacky typecheck fix

* Fix new internal package typecheck issues and start adding telemetry to the replication service

* tracing over spans, some other improvements

* Improvements to the runs replication service, now ready for testing

* Some fixes and cleanups

* Don't need this code anymore

* move transaction types into the runs replication service

* only send spans where there are transaction events

* A couple of suggested tweaks
2025-05-12 22:12:36 +01:00
..
2025-03-05 14:40:14 +00:00
2025-03-05 14:40:14 +00:00

Trigger.dev References

Contains code that tests or uses the @trigger.dev/* packages in some way, either by using them to test out a framework adapter, an integration, or parts of the main SDK.

All the dependencies to the @trigger.dev/* packages will be both referenced in the package.json dependencies as workspace:*, as well as using a direct path from the tsconfig.json file like so:

{
  "extends": "@trigger.dev/tsconfig/node18.json",
  "include": ["./src/**/*.ts"],
  "compilerOptions": {
    "baseUrl": ".",
    "lib": ["DOM", "DOM.Iterable"],
    "paths": {
      "@/*": ["./src/*"],
      "@trigger.dev/sdk": ["../../packages/trigger-sdk/src/index"],
      "@trigger.dev/sdk/*": ["../../packages/trigger-sdk/src/*"],
      "@trigger.dev/express": ["../../packages/express/src/index"],
      "@trigger.dev/express/*": ["../../packages/express/src/*"],
      "@trigger.dev/core": ["../../packages/core/src/index"],
      "@trigger.dev/core/*": ["../../packages/core/src/*"],
      "@trigger.dev/integration-kit": ["../../packages/integration-kit/src/index"],
      "@trigger.dev/integration-kit/*": ["../../packages/integration-kit/src/*"],
      "@trigger.dev/github": ["../../integrations/github/src/index"],
      "@trigger.dev/github/*": ["../../integrations/github/src/*"],
      "@trigger.dev/slack": ["../../integrations/slack/src/index"],
      "@trigger.dev/slack/*": ["../../integrations/slack/src/*"],
      "@trigger.dev/openai": ["../../integrations/openai/src/index"],
      "@trigger.dev/openai/*": ["../../integrations/openai/src/*"],
      "@trigger.dev/resend": ["../../integrations/resend/src/index"],
      "@trigger.dev/resend/*": ["../../integrations/resend/src/*"],
      "@trigger.dev/typeform": ["../../integrations/typeform/src/index"],
      "@trigger.dev/typeform/*": ["../../integrations/typeform/src/*"],
      "@trigger.dev/plain": ["../../integrations/plain/src/index"],
      "@trigger.dev/plain/*": ["../../integrations/plain/src/*"],
      "@trigger.dev/supabase": ["../../integrations/supabase/src/index"],
      "@trigger.dev/supabase/*": ["../../integrations/supabase/src/*"],
      "@trigger.dev/stripe": ["../../integrations/stripe/src/index"],
      "@trigger.dev/stripe/*": ["../../integrations/stripe/src/*"],
      "@trigger.dev/sendgrid": ["../../integrations/sendgrid/src/index"],
      "@trigger.dev/sendgrid/*": ["../../integrations/sendgrid/src/*"],
      "@trigger.dev/airtable": ["../../integrations/airtable/src/index"],
      "@trigger.dev/airtable/*": ["../../integrations/airtable/src/*"]
    }
  }
}

Creating a New Reference Project

This guide assumes that you have followed the Contributing.md instructions to set up a local trigger.dev instance. If not, please complete the setup before continuing.

Step-by-Step Instructions

  1. Run an HTTP tunnel: You will need to run an HTTP tunnel to expose your local webapp, it is required for some API calls during building the image to deploy on your local instance. This is optional if you do not plan to test deployment on your local instance.
  • Download the ngrok CLI. This can be done by following the instructions on ngrok's website.
  • Create an account on ngrok to obtain the authtoken and add it to the CLI.
ngrok config add-authtoken <your-auth-token>

Replace the with the token you obtain from ngrok.

  • Run the tunnel.
ngrok http <your-app-port>

Replace the with the webapp port, default is 3030.

  1. Add your tunnel URL to the env: After running the ngrok tunnel, you will see URL in your terminal, it will look something like https://<your-tunnel-address>.ngrok-free.app. Replace the APP_ORIGIN variable with this URL in your .env file in the root of the trigger.dev project.

  2. Run the webapp on localhost:

pnpm run dev --filter webapp --filter coordinator --filter docker-provider
  1. Build the CLI in a new terminal window:
# Build the CLI
pnpm run build --filter trigger.dev

# Make it accessible to `pnpm exec`
pnpm i
  1. Set up a new project in the webapp:
  • Open the webapp running on localhost:3030.
  • Create a new project in the webapp UI.
  • Go to the Project Settings page and copy the project reference id from there.
  1. Copy the hello-world project as a template:
cp -r references/hello-world references/<new-project>

Replace <new-project> with your desired project name.

  1. Update project details:
  • Open <new-project>/package.json and change the name field. (Tip: Use the same name as in the webapp to avoid confusion.)

  • Open <new-project>/trigger.config.ts and update the project field with the project reference you copied from the webapp.

  • Run pnpm i in your <new-project> directory to sync the dependencies.

  1. Authorize the CLI for your project:
pnpm exec trigger login -a http://localhost:3030 --profile local
  1. Run the new project: You can now run your project using the CLI with the following command:
pnpm exec trigger dev --profile local

You can also deploy them against your local instance with the following command:

pnpm exec trigger deploy --self-hosted --load-image --profile local