github-featured/triggerdotdev--trigger.dev
2
0
Fork 0
Code Issues Pull Requests Deployments Models Agent Notes Packages Projects Releases Wiki Activity
Files
f2babbf6373822004c3557faa7fa1eda0218f872
triggerdotdev--trigger.dev/apps/webapp/package.json
T
Matt Aitken f2babbf637 Internal packages (testcontainers, redis-worker and zod-worker) (#1392) 
* Some notes on the new run engine

* lockfile with setup for the run engine

* Documenting where TaskRun is currently mutated, to try figure out the shape of the new system

* Added notes about how triggering currently works

* Details about when triggering happens

* Lots of notes about waitpoints

* Started scaffolding the RunEngine

* Sketch of Prisma waitpoint schema while it’s fresh in my mind

* Got Prisma working with testcontainers

* Use beforeEach/afterEach

* Simple Prisma and Redis test

* Return Redis options instead of a client

* Simplified things

* A very simple FIFO pull-based queue to check the tests working properly

* Use vitest extend

* Separate redis, postgres and combined tests for faster testing

* Some fixes and test improvements

* Pass a logger into the queue

* A queue processor that processes items from the given queue as fast as it can

* Test for retrying an item that wasn’t processed

* First draft of waitpoints in the Prisma schema

* Remove the custom logger from the test

* Added a completedAt to Waitpoint

* Notes on the flow for an execution starting

* Added redlock, moved some files around

* Starting point for the TaskRunExecutionSnapshot table

* Added relationships to TaskRunExecutionSnapshot

* Change some tsconfig

* Moved some things around

* Added some packages

* WIP on the RunQueue

* Fix for some imports

* Key producer with some tests

* Removed the nv type from the keys… it’s not useful to do global queries

* Passing unit tests for all the public key producer functions

* Some basic tests passing for the RunQueue

* Simple enqueue test working

* Enqueue and dequeue for dev is working

* Don’t log everything during the tests

* Enqueuing/dequeuing from the shared queue is working

* Tests for getting a shared queue

* The key producer sharedQueue can now be named, to allow multiple separate queues

* The key producer uses the name of the queue as the input

* Extra info in the Prisma schema

* Dequeuing a message gets the payload and sets the task concurrency all in one Lua script

* Adding more keys so we can read the concurrency from the queue

* Setting the concurrency with dequeue and enquque is working

* Improved the tests and fixed some bugs

* Acking is resetting the concurrencies

* Check the key has been removed after acking

* Nacking is working

* Changed the package to CommonJS + Node10 so it works with Redlock

* Moved the database, otel and emails packages to be in internal-packages

* Moved some Prisma code to the database package

* Started using the RunEngine for triggering

* Progress on run engine triggering, first waitpoint code

* Create a delay waitpoint

* Moved ZodWorker to an internal package so it can be used in the run engine as well as the webapp

* Web app now uses the zod worker package

* Added parseNaturalLanguageDuration to core/apps

* internal-packages/zod-worker in the lockfile

* Pass in the master queue, remove old rebalance workers code

* Add masterQueue to TaskRun

* Fixed the tests

* Moved waitpoint code into the run engine, also the zod worker

* Completing waitpoints

* An experiment to create a new test container with environment

* More changes to triggering

* Started testing triggering

* Test for a run getting triggered and being enqueued

* Removed dequeueMessageInEnv

* Update dev queue tests to use the shared queue function

* Schema changes for TaskRunExecutionSnapshot

* First execution snapshot when the run is created. Dequeue run function added to the engine

* Separate internal package for testcontainers so they can be used elsewhere

* Remove the simple queue and testcontainers from the run-engine. They’re going to be separate

* Fix for the wrong path to the Prisma schem,a

* Added the testcontainers package to the run-engine

* redis-worker package, just a copy of the simple queue for now

* The queue now uses Lua to enqueue dequeue

* The queue now has a catalog and an invisible period after dequeuing

* Added a visibility timeout and acking, with tests

* Added more Redis connection logging, deleted todos

* Visibility timeouts are now defined on the catalog and can be overridden when enqueuing

* Dequeue multiple items at once

* Test for dequeuing multiple items

* Export some types to be used elsewhere

* Partial refactor of the processor

* First stab at a worker with concurrency and NodeWorkers

* Don’t have a default visibility timeout in the queue

* Worker setup and processing items in a simple test

* Process jobs in parallel with retrying

* Get the attempt when dequeuing

* Workers do exponential backoff

* Moved todos

* DLQ functionality

* DLQ tests

* Same cluster for all keys in the same queue

* Added DLQ tests

* Whitespace

* Redis pubsub to redrive from the worker

* Fixed database paths

* Fix for path to zod-worker

* Fixes for typecheck errors, mostly with TS versions and module resolution

* Redlock required a patch

* Moved the new DB migrations to the new database package folder

* Remove the run-engine package

* Remove the RunEngine prisma schema changes

* Delete triggerTaskV2

* Remove zodworker test script (no tests)

* Update test-containers readme

* Generate the client first

* Use a specific version of the prisma package

* Generate the prisma client before running the unit tests
2024-10-08 17:41:22 +01:00

250 lines
8.6 KiB
JSON
Raw Blame History

{
"private": true,
"name": "webapp",
"version": "1.0.0",
"sideEffects": false,
"scripts": {
"build": "run-s build:**",
"build:db:seed": "esbuild --platform=node --bundle --minify --format=cjs ./prisma/seed.ts --outdir=prisma",
"build:remix": "remix build",
"build:server": "esbuild --platform=node --format=cjs ./server.ts --outdir=build",
"dev": "cross-env PORT=3030 remix dev -c \"node ./build/server.js\"",
"dev:worker": "cross-env NODE_PATH=../../node_modules/.pnpm/node_modules node ./build/server.js",
"format": "prettier --write .",
"lint": "eslint --cache --cache-location ./node_modules/.cache/eslint .",
"start": "cross-env NODE_ENV=production node --max-old-space-size=8192 ./build/server.js",
"start:local": "cross-env node --max-old-space-size=8192 ./build/server.js",
"typecheck": "tsc -p ./tsconfig.check.json",
"db:seed": "node prisma/seed.js",
"db:seed:local": "ts-node prisma/seed.ts",
"build:db:populate": "esbuild --platform=node --bundle --minify --format=cjs ./prisma/populate.ts --outdir=prisma",
"db:populate": "node prisma/populate.js --",
"generate:sourcemaps": "remix build --sourcemap",
"clean:sourcemaps": "run-s clean:sourcemaps:*",
"clean:sourcemaps:public": "rimraf ./build/**/*.map",
"clean:sourcemaps:build": "rimraf ./public/build/**/*.map",
"test": "vitest"
},
"eslintIgnore": [
"/node_modules",
"/build",
"/public/build"
],
"dependencies": {
"@ariakit/react": "^0.4.6",
"@ariakit/react-core": "^0.4.6",
"@aws-sdk/client-sqs": "^3.445.0",
"@codemirror/autocomplete": "^6.3.1",
"@codemirror/commands": "^6.1.2",
"@codemirror/lang-javascript": "^6.1.1",
"@codemirror/lang-json": "^6.0.1",
"@codemirror/language": "^6.3.1",
"@codemirror/lint": "^6.4.2",
"@codemirror/search": "^6.2.3",
"@codemirror/state": "^6.1.3",
"@codemirror/view": "^6.5.0",
"@conform-to/react": "^0.6.1",
"@conform-to/zod": "^0.6.1",
"@depot/sdk-node": "^1.0.0",
"@electric-sql/react": "^0.3.5",
"@headlessui/react": "^1.7.8",
"@heroicons/react": "^2.0.12",
"@internal/run-engine": "workspace:*",
"@internal/zod-worker": "workspace:*",
"@internationalized/date": "^3.5.1",
"@lezer/highlight": "^1.1.6",
"@opentelemetry/api": "1.9.0",
"@opentelemetry/core": "1.25.1",
"@opentelemetry/exporter-logs-otlp-http": "0.52.1",
"@opentelemetry/exporter-trace-otlp-http": "0.52.1",
"@opentelemetry/instrumentation": "0.52.1",
"@opentelemetry/instrumentation-express": "^0.36.1",
"@opentelemetry/instrumentation-http": "0.52.1",
"@opentelemetry/resources": "1.25.1",
"@opentelemetry/sdk-logs": "0.52.1",
"@opentelemetry/sdk-node": "0.52.1",
"@opentelemetry/sdk-trace-base": "1.25.1",
"@opentelemetry/sdk-trace-node": "1.25.1",
"@opentelemetry/semantic-conventions": "1.25.1",
"@popperjs/core": "^2.11.8",
"@prisma/instrumentation": "^5.11.0",
"@radix-ui/react-alert-dialog": "^1.0.4",
"@radix-ui/react-dialog": "^1.0.3",
"@radix-ui/react-label": "^2.0.1",
"@radix-ui/react-popover": "^1.0.5",
"@radix-ui/react-radio-group": "^1.1.3",
"@radix-ui/react-select": "^1.2.1",
"@radix-ui/react-slider": "^1.1.2",
"@radix-ui/react-switch": "^1.0.3",
"@radix-ui/react-tabs": "^1.0.3",
"@radix-ui/react-tooltip": "^1.0.5",
"@react-aria/datepicker": "^3.9.1",
"@react-stately/datepicker": "^3.9.1",
"@react-types/datepicker": "^3.7.1",
"@remix-run/express": "2.1.0",
"@remix-run/node": "2.1.0",
"@remix-run/react": "2.1.0",
"@remix-run/router": "^1.15.3",
"@remix-run/serve": "2.1.0",
"@remix-run/server-runtime": "2.1.0",
"@remix-run/v1-meta": "^0.1.3",
"@slack/web-api": "^6.8.1",
"@socket.io/redis-adapter": "^8.3.0",
"@tabler/icons-react": "^2.39.0",
"@tailwindcss/container-queries": "^0.1.1",
"@tanstack/react-virtual": "^3.0.4",
"@team-plain/typescript-sdk": "^3.5.0",
"@trigger.dev/companyicons": "^1.5.35",
"@trigger.dev/core": "workspace:*",
"@trigger.dev/database": "workspace:*",
"@trigger.dev/otlp-importer": "workspace:*",
"@trigger.dev/platform": "1.0.12",
"@trigger.dev/sdk": "workspace:*",
"@trigger.dev/yalt": "npm:@trigger.dev/yalt",
"@types/pg": "8.6.6",
"@uiw/react-codemirror": "^4.19.5",
"@upstash/ratelimit": "^1.1.3",
"@whatwg-node/fetch": "^0.9.14",
"assert-never": "^1.2.1",
"aws4fetch": "^1.0.18",
"class-variance-authority": "^0.5.2",
"clsx": "^1.2.1",
"compression": "^1.7.4",
"cookie": "^0.6.0",
"cron-parser": "^4.9.0",
"cronstrue": "^2.21.0",
"cross-env": "^7.0.3",
"cuid": "^2.1.8",
"dotenv": "^16.4.5",
"emails": "workspace:*",
"evt": "^2.4.13",
"express": "^4.18.1",
"framer-motion": "^10.12.11",
"graphile-worker": "0.16.6",
"highlight.run": "^7.3.4",
"humanize-duration": "^3.27.3",
"intl-parse-accept-language": "^1.0.0",
"ioredis": "^5.3.2",
"isbot": "^3.6.5",
"jose": "^5.4.0",
"jsonpointer": "^5.0.1",
"jwt-decode": "^4.0.0",
"lodash.omit": "^4.5.0",
"lucide-react": "^0.229.0",
"marked": "^4.0.18",
"match-sorter": "^6.3.4",
"morgan": "^1.10.0",
"nanoid": "^3.3.4",
"non.geist": "^1.0.2",
"ohash": "^1.1.3",
"openai": "^4.33.1",
"parse-duration": "^1.1.0",
"posthog-js": "^1.93.3",
"posthog-node": "^3.1.3",
"prism-react-renderer": "^2.3.1",
"prismjs": "^1.29.0",
"prom-client": "^15.1.0",
"random-words": "^2.0.0",
"react": "^18.2.0",
"react-aria": "^3.31.1",
"react-collapse": "^5.1.1",
"react-dom": "^18.2.0",
"react-hotkeys-hook": "^4.4.1",
"react-popper": "^2.3.0",
"react-resizable-panels": "^2.0.9",
"react-stately": "^3.29.1",
"react-use": "^17.4.0",
"react-window-splitter": "^0.4.1",
"recharts": "^2.12.6",
"regression": "^2.0.1",
"remix-auth": "^3.6.0",
"remix-auth-email-link": "2.0.2",
"remix-auth-github": "^1.6.0",
"remix-typedjson": "0.3.1",
"remix-utils": "^7.1.0",
"seedrandom": "^3.0.5",
"simple-oauth2": "^5.0.0",
"simplur": "^3.0.1",
"slug": "^6.0.0",
"socket.io": "4.7.4",
"socket.io-adapter": "^2.5.4",
"sonner": "^1.0.3",
"sqs-consumer": "^7.4.0",
"superjson": "^2.2.1",
"tailwind-merge": "^1.12.0",
"tailwind-scrollbar-hide": "^1.1.7",
"tailwindcss-animate": "^1.0.5",
"tailwindcss-textshadow": "^2.1.3",
"tiny-invariant": "^1.2.0",
"ulid": "^2.3.0",
"ulidx": "^2.2.1",
"ws": "^8.11.0",
"zod": "3.22.3",
"zod-error": "1.5.0",
"zod-validation-error": "^1.5.0"
},
"devDependencies": {
"@remix-run/dev": "2.1.0",
"@remix-run/eslint-config": "2.1.0",
"@remix-run/testing": "^2.1.0",
"@swc/core": "^1.3.4",
"@swc/helpers": "^0.4.11",
"@tailwindcss/forms": "^0.5.3",
"@tailwindcss/typography": "^0.5.9",
"@total-typescript/ts-reset": "^0.4.2",
"@types/bcryptjs": "^2.4.2",
"@types/compression": "^1.7.2",
"@types/cookie": "^0.6.0",
"@types/eslint": "^8.4.6",
"@types/express": "^4.17.13",
"@types/humanize-duration": "^3.27.1",
"@types/json-query": "^2.2.3",
"@types/lodash.omit": "^4.5.7",
"@types/marked": "^4.0.3",
"@types/morgan": "^1.9.3",
"@types/node": "^18.11.15",
"@types/node-fetch": "^2.6.2",
"@types/prismjs": "^1.26.0",
"@types/qs": "^6.9.7",
"@types/react": "18.2.69",
"@types/react-collapse": "^5.0.4",
"@types/react-dom": "18.2.7",
"@types/regression": "^2.0.6",
"@types/seedrandom": "^3.0.8",
"@types/simple-oauth2": "^5.0.4",
"@types/slug": "^5.0.3",
"@types/tar": "^6.1.4",
"@types/ws": "^8.5.3",
"@typescript-eslint/eslint-plugin": "^5.59.6",
"@typescript-eslint/parser": "^5.59.6",
"autoprefixer": "^10.4.13",
"babel-loader": "^9.1.3",
"babel-preset-react-app": "^10.0.1",
"css-loader": "^6.10.0",
"datepicker": "link:@types/@react-aria/datepicker",
"engine.io": "^6.5.4",
"esbuild": "^0.15.10",
"eslint": "^8.24.0",
"eslint-config-prettier": "^8.5.0",
"eslint-plugin-import": "^2.29.1",
"eslint-plugin-react-hooks": "^4.6.2",
"eslint-plugin-turbo": "^2.0.4",
"npm-run-all": "^4.1.5",
"postcss-import": "^16.0.1",
"postcss-loader": "^8.1.1",
"prettier": "^2.8.8",
"prettier-plugin-tailwindcss": "^0.3.0",
"prop-types": "^15.8.1",
"rimraf": "^3.0.2",
"style-loader": "^3.3.4",
"tailwind-scrollbar": "^3.0.1",
"tailwindcss": "3.4.1",
"ts-node": "^10.7.0",
"tsconfig-paths": "^3.14.1",
"typescript": "^5.1.6",
"vitest": "^1.4.0"
},
"engines": {
"node": ">=16.0.0"
}
}
Reference in New Issue View Git Blame Copy Permalink