* Boilerplate server-only use case
* wip: integration suite instrumentation setup
* Working poc testing compileProject
* Add pnpm script to run e2e tests only
* Use vitest globals
* Remove commented line
* Remove useless export
* Add modifier to test only one fixture project
* Handle package manager and log level choice
* Update server-only example
* Setup / teardown + split compile for package manager capabilities
* Ignore yarn files
* Fix issue with corepack, store version in engines field
* Rename test file
* Fix npm updates yarn.lock
* Move typecheking in a dedicated test
* Stop bundling the compile command to allow for more granular testing
* Put config resolving in separate test
* Add no-config test case and add test case expected errors configuration
* Add wantCompilationError option
* Add dependencies handling
* Use packageManager passed as option to resolve required deps
* Remove unused guard clauses
* Add postinstall & hash handling step
* Add worker start test
* Handle yarn.lock copy renaming on sigterm and sigkill
* Update vitest and use concurrent option
* Add a readme file
* Add CI workflow
* Fix handle cli deps
* Run cli v3 e2e tests on publish action
* Increase timeout on deps resolving step
* Add changeset
* Remove .pnp.cjs as we use yarn with nodeLinker node-modules
* Add missing .yarnrc.yml file
* No need to build CLI to run E2E tests
* Remove bun.lockb files
* Update beige-pears-explode.md
---------
Co-authored-by: Eric Allam <eallam@icloud.com>
* add amin email regex env var
* fix displayed init command for self-hosted setups
* shared env var to disable telemetry in cli and webapp
* pin sdk version during init
* if specified, add api url to dev command shown after init
* improve checkpoint support detection
* control forced checkpoint simulation via env var
* add public init to providers
* better checkpoint support check for coordinator
* add docker to coordinator image
* update docker provider containerfile
* bump remaining containers to node 20
* add infra image build to default publish workflow
* lockfile
* remove concurrency group from infra workflow
* add docker provider to build matrix
* fix var subst
* checkpoint test is docker specific
* enable v3 projects by default on self-hosted instances
* fix v3 setup command again
* add default posthog key
* self-hosting docs
* add latest tags to versioned infra and webapp builds
* some checkpoint errors should skip retrying
* add changeset
* shorten paragraph
* some docs updates
* update tunnelling section
* add registry setup section
* use correct cli push flag
* add checkout to v3 branch
* update the worker machine setup steps
* fix infra build
* small docs update
* remove unused feature function
* Revert "remove unused feature function"
This reverts commit cfe07887a12b6893dca8ce499964481a9b3dc9db.
* fix self-hosted v3 feature gate
* add note about missing arm support
* simplify helper script syntax
* add checkpoint restore events
* fix retries.enabledInDev
* fix tsconfig paths
* fix provider build and dev
* update kubernetes provider and fix builds again
* upgrade prod base to node 20
* update infra publish workflow
* rethrow k8s errors after logging
* shorten index container names
* fix error type assertion
* rename type assertion
* remove resource limits for now
* add missing run id on create
* add push to deploy command for self-hosting
* checkpointing fixes
* update coordinator image
* ensure valid registry login
* delete checkpoint archive after successful push
* log options on error
* structured logs for socket connections
* fix structured log merge
* exit process after checkpointing
* update restore pull secret name
* append shortcode to restore names
* log handler payload
* disable post start lifecycle hook
* pass in coordinator host via volume
* replace dapi with taskinfo
* add missing restore label
* don't restart restored containers
* remove init container from create
* atomic post-completion checkpoints
* switch to run id for container names
* improve wait accuracy
* measure basic checkpoint perf
* always log disconnect reason
* use system clock to end wait spans
* checkpoint readiness and cancel signals
* restore from checkpoint events and fix statuses
* remove attempt id env var
* restore dependencies from events
* reconnect wip
* lifecycle hooks are back
* fix hooks and improve reconnect
* make docker send postStart hook
* only checkpoint for retry if large delay
* fix a few more resume issues
* lifecycle hook fixes
* skip connection handler when waiting for post start hook
---------
Co-authored-by: Eric Allam <eric@trigger.dev>
* add zod socket
* start using zod socket
* use zod socket for shared queue
* use shared provider shell
* update k8s provider task ops
* set otlp endpoint
* update docker actions and fix builds
* remove unused types
* update injected secret key env var name
* complete socket.io types to schemas migration
* update env example to new v3 key var
* fix coordinator build
* set task run label
* ensure attempt id is always set
* increase prod worker timeout
* set otlp endpoint for on runs
* pass otlp env var to prod worker
* auto-remove completed index jobs
* Allow PRs that have /publish-docker in the body to cause docker images to be published
* Also ignore mdx file changes
* pr_checks now uses pull_request instead of pull_request_target
* WIP job run performance improvements
- Added a `perf` tool to better measure job run performance under heavy load
- Removed `runFinished` job (not really needed)
- startQueuedRuns now uses a jobKey with replace
- Fixed an issue with ZodWorker when using jobKey
* Publish improvement docker images
* fixed the improvement docker publishing
* Downgrade back to prisma 4.16.0 because 5.1.x broke docker builds
* Changes to how queued runs work
- Split the worker into two different workers, one dedicated to performRunExecution
- Schedule performRunExecution in a single place, with a queue and using a round robin manually controlled concurrency
- Remove startQueuedRuns
- All runs are queued before they are started
- Setting the worker maxPoolSize to the same as the worker concurrency
- Starting to be able to split the docker image
* Remove queue name from startRun graphile job
* Make the prisma connection pool stuff configurable through env vars
* Hardcode (for now) the max concurrent runs limit
* Rewrite performRunExecution to be more performant
PerformRunExecutionV2:
- Does not create and manage jobRunExecution records
- Does not reimplement retrying, uses graphile worker retrying instead
I’ve kept around PerformRunExecutionV1 so this works when deploying. Definitely needs LOTS of testing
* Fix issues with cached tasks
- Limit the size of the cached tasks sent when executing a run, using the knapsack problem dynamic programming approach
- Actually USE the cached tasks in IO by using the idempotencyKey instead of the task ID
- Remove output from all logs
- Added a stress test job catalog
* Forgot to commit the logger updates
* Never log connectionString
* Login to docker hub to get around rate limits
* Add additional logging to the graphile workers
* Fix the *_ENABLED env vars
* Allow adding and removing jobs to be done from the webapp
* Don’t set the job to failed if it’s being retried
* Deprecated queue options in the job and removed startPosition. Now using the job/env combo as the job queue name
* Dequeung jobs doesn’t check if the runner is initialized
* Fixed issues with retrying a run getting stuck on a cancelled task, and errors from parsing the results of dequeing a job
* Remove queued round robin thing that isn’t used anymore
* Added slack to job catalog
* Better forwards compat
* Added long delay
* Fixed lock file
Émile Ré