* WIP job run performance improvements
- Added a `perf` tool to better measure job run performance under heavy load
- Removed `runFinished` job (not really needed)
- startQueuedRuns now uses a jobKey with replace
- Fixed an issue with ZodWorker when using jobKey
* Publish improvement docker images
* fixed the improvement docker publishing
* Downgrade back to prisma 4.16.0 because 5.1.x broke docker builds
* Changes to how queued runs work
- Split the worker into two different workers, one dedicated to performRunExecution
- Schedule performRunExecution in a single place, with a queue and using a round robin manually controlled concurrency
- Remove startQueuedRuns
- All runs are queued before they are started
- Setting the worker maxPoolSize to the same as the worker concurrency
- Starting to be able to split the docker image
* Remove queue name from startRun graphile job
* Make the prisma connection pool stuff configurable through env vars
* Hardcode (for now) the max concurrent runs limit
* Rewrite performRunExecution to be more performant
PerformRunExecutionV2:
- Does not create and manage jobRunExecution records
- Does not reimplement retrying, uses graphile worker retrying instead
I’ve kept around PerformRunExecutionV1 so this works when deploying. Definitely needs LOTS of testing
* Fix issues with cached tasks
- Limit the size of the cached tasks sent when executing a run, using the knapsack problem dynamic programming approach
- Actually USE the cached tasks in IO by using the idempotencyKey instead of the task ID
- Remove output from all logs
- Added a stress test job catalog
* Forgot to commit the logger updates
* Never log connectionString
* Login to docker hub to get around rate limits
* Add additional logging to the graphile workers
* Fix the *_ENABLED env vars
* Allow adding and removing jobs to be done from the webapp
* Don’t set the job to failed if it’s being retried
* Deprecated queue options in the job and removed startPosition. Now using the job/env combo as the job queue name
* Dequeung jobs doesn’t check if the runner is initialized
* Fixed issues with retrying a run getting stuck on a cancelled task, and errors from parsing the results of dequeing a job
* Remove queued round robin thing that isn’t used anymore
* Added slack to job catalog
* Better forwards compat
* Added long delay
* Fixed lock file
* chore: make possible to write tests aside with implementations
* test: write getUserPkgManager test TO-DOs
* test: add test to building the path
* test: add tests to manual file checking
* test: add tests to npm_config_user_agent
* Update packages/cli/src/utils/getUserPkgManager.spec.ts
* test: add test to no-duplicated-task-keys rule
* feat: implement no-duplicated-task-keys
* chore: create testing scripts
* refactor: fill in rule meta
* feat: export no-duplicated-task-keys rule
* chore: bump package version to 0.0.1
* chore: create eslint plugin
* feat: create no-duplicated-task-keys rule
* refactor: delete no-duplicated-task-keys from config-custom
* refactor: rename eslint-plugin folder
* feat: cover case from nextjs-example
* feat: cover additional cases from examples/package-tester
* chore: add eslint-plugin on nextjs-example
* feat: add more cases from `nextjs-example`
* revert: rollback changes on eslint-config-custom
* Set version to 2.0.9, inline with other packages
* Create chilly-pianos-try.md
---------
Co-authored-by: Matt Aitken <matt@mattaitken.com>
* FIx: [TRI-1006] New @trigger.dev/cli whoami command.
* revert dev.ts zod schema to original
* Removed telemetry
* remove all telemetry
* Made the clientId optional again
---------
Co-authored-by: Matt Aitken <matt@mattaitken.com>
* chore: run `pnpm install --prefer-frozen-lockfile`
* chore: install `jest` and `ts-jest`
* test: create `jest.config.js`
* chore: install `@types/jest`
* chore: add `jest` on `tsconfig.json` `types` property
* chore: create test script
* test: create example test 🎉🎊🕺🏼
* chore: install `@gmrchk/cli-testing-library`
* chore: override `tsconfig` with `@trigger.dev/tsconfig`
* test: make poc of cli test
* chore: run build before test
* docs: improve comment
* Added a changeset
---------
Co-authored-by: Matt Aitken <matt@mattaitken.com>
* chore: update prisma from version 4.16.0 to 5.1.0
* fixed the ts-errors due to the prisma update
---------
Co-authored-by: Matt Aitken <matt@mattaitken.com>
* fix: support express jsonParser middleware if used in the Express app
* Create sixty-windows-run.md
---------
Co-authored-by: Eric Allam <eallam@icloud.com>
* fix: Handle ngrok config upgrade error in createTunnel function
* Improved the output when upgrading the ngrok configuration
* Create three-flies-sneeze.md
---------
Co-authored-by: Eric Allam <eallam@icloud.com>
* Changed the example job to include a delay
* Improved the example job copy and added ‘in a separate terminal’ to 2.
* Create kind-eggs-pump.md
---------
Co-authored-by: Matt Aitken <matt@mattaitken.com>
* feat: adds a job catalog file when an integration is created via the CLI
* added a changeset
* fix: create-integration CLI throws an error when it tries to install dependencies
* Update getUserPkgManager.ts to main
---------
Co-authored-by: Matt Aitken <matt@mattaitken.com>
* If an endpointId isn’t in the arguments or package, then use the project slug. Don’t ask for it.
* Should await the package artifact detection
* Catch error thrown if there’s no next.config file when detecting the type of Next project
* Create long-carrots-camp.md
* Detect presence of a “next” dependency
* Use a strict undefined check instead
* Changeset: Detect Next.js project by looking at dependencies, not next.config.js
* Read a package json file
* First check for next.config file, otherwise use next dependency
* Update changeset description
* Created the sheet
* Started work on the resource route
* Created ValidateCreateEndpointService and the EndpointValidateApi (name TBC)
* The TriggerClient responds with the id
* Tidied some stuff up
* Alternative: EndpointApi doesn’t take endpointSlug. Instead just Ping() does
* Create cool-snakes-deny.md
---------
Co-authored-by: Eric Allam <eallam@icloud.com>
* update cli init to create index file in examples folder
* add patch changeset
* Create silly-baboons-join.md
* remove comment in router file
add comment in jobs index to instruct them to
* rename examplesIndex to examplesIndexFileName
* import jobs module in routeContent for createTriggerPageRoute just as we did for createTriggerAppRoute
---------
Co-authored-by: Matt Aitken <matt@mattaitken.com>
github-actions[bot]