Eric Allam
b1b9321ad2
* WIP job run performance improvements - Added a `perf` tool to better measure job run performance under heavy load - Removed `runFinished` job (not really needed) - startQueuedRuns now uses a jobKey with replace - Fixed an issue with ZodWorker when using jobKey * Publish improvement docker images * fixed the improvement docker publishing * Downgrade back to prisma 4.16.0 because 5.1.x broke docker builds * Changes to how queued runs work - Split the worker into two different workers, one dedicated to performRunExecution - Schedule performRunExecution in a single place, with a queue and using a round robin manually controlled concurrency - Remove startQueuedRuns - All runs are queued before they are started - Setting the worker maxPoolSize to the same as the worker concurrency - Starting to be able to split the docker image * Remove queue name from startRun graphile job * Make the prisma connection pool stuff configurable through env vars * Hardcode (for now) the max concurrent runs limit * Rewrite performRunExecution to be more performant PerformRunExecutionV2: - Does not create and manage jobRunExecution records - Does not reimplement retrying, uses graphile worker retrying instead I’ve kept around PerformRunExecutionV1 so this works when deploying. Definitely needs LOTS of testing * Fix issues with cached tasks - Limit the size of the cached tasks sent when executing a run, using the knapsack problem dynamic programming approach - Actually USE the cached tasks in IO by using the idempotencyKey instead of the task ID - Remove output from all logs - Added a stress test job catalog * Forgot to commit the logger updates * Never log connectionString * Login to docker hub to get around rate limits * Add additional logging to the graphile workers * Fix the *_ENABLED env vars * Allow adding and removing jobs to be done from the webapp * Don’t set the job to failed if it’s being retried * Deprecated queue options in the job and removed startPosition. Now using the job/env combo as the job queue name * Dequeung jobs doesn’t check if the runner is initialized * Fixed issues with retrying a run getting stuck on a cancelled task, and errors from parsing the results of dequeing a job * Remove queued round robin thing that isn’t used anymore * Added slack to job catalog * Better forwards compat * Added long delay * Fixed lock file
71 lines
2.6 KiB
Docker
71 lines
2.6 KiB
Docker
FROM node:18.16.1-bullseye-slim AS pruner
|
|
|
|
WORKDIR /triggerdotdev
|
|
|
|
COPY --chown=node:node . .
|
|
RUN npx -q turbo@1.10.9 prune --scope=webapp --docker
|
|
RUN find . -name "node_modules" -type d -prune -exec rm -rf '{}' +
|
|
|
|
# Base strategy to have layer caching
|
|
FROM node:18.16.1-bullseye-slim AS base
|
|
RUN apt-get update && apt-get install -y openssl dumb-init
|
|
WORKDIR /triggerdotdev
|
|
COPY --chown=node:node .gitignore .gitignore
|
|
COPY --from=pruner --chown=node:node /triggerdotdev/out/json/ .
|
|
COPY --from=pruner --chown=node:node /triggerdotdev/out/pnpm-lock.yaml ./pnpm-lock.yaml
|
|
COPY --from=pruner --chown=node:node /triggerdotdev/out/pnpm-workspace.yaml ./pnpm-workspace.yaml
|
|
|
|
## Dev deps
|
|
FROM base AS dev-deps
|
|
WORKDIR /triggerdotdev
|
|
# Corepack is used to install pnpm
|
|
RUN corepack enable
|
|
ENV NODE_ENV development
|
|
RUN pnpm install --ignore-scripts --no-frozen-lockfile
|
|
|
|
## Production deps
|
|
FROM base AS production-deps
|
|
WORKDIR /triggerdotdev
|
|
# Corepack is used to install pnpm
|
|
RUN corepack enable
|
|
ENV NODE_ENV production
|
|
RUN pnpm install --prod --no-frozen-lockfile
|
|
COPY --from=pruner --chown=node:node /triggerdotdev/packages/database/prisma/schema.prisma /triggerdotdev/packages/database/prisma/schema.prisma
|
|
# RUN pnpm add @prisma/client@5.1.1 -w
|
|
RUN pnpx prisma@4.16.0 generate --schema /triggerdotdev/packages/database/prisma/schema.prisma
|
|
|
|
## Builder (builds the webapp)
|
|
FROM base AS builder
|
|
WORKDIR /triggerdotdev
|
|
# Corepack is used to install pnpm
|
|
RUN corepack enable
|
|
COPY --from=pruner --chown=node:node /triggerdotdev/out/full/ .
|
|
COPY --from=dev-deps --chown=node:node /triggerdotdev/ .
|
|
COPY --chown=node:node turbo.json turbo.json
|
|
COPY --chown=node:node docker/scripts ./scripts
|
|
RUN chmod +x ./scripts/wait-for-it.sh
|
|
RUN chmod +x ./scripts/entrypoint.sh
|
|
RUN pnpm run generate
|
|
RUN pnpm run build --filter=webapp...
|
|
|
|
# Runner
|
|
FROM node:18.16.1-bullseye-slim AS runner
|
|
RUN apt-get update && apt-get install -y openssl
|
|
WORKDIR /triggerdotdev
|
|
RUN corepack enable
|
|
ENV NODE_ENV production
|
|
|
|
COPY --from=base /usr/bin/dumb-init /usr/bin/dumb-init
|
|
COPY --from=pruner --chown=node:node /triggerdotdev/out/full/ .
|
|
COPY --from=production-deps --chown=node:node /triggerdotdev .
|
|
COPY --from=builder --chown=node:node /triggerdotdev/apps/webapp/build/server.js ./apps/webapp/build/server.js
|
|
COPY --from=builder --chown=node:node /triggerdotdev/apps/webapp/build ./apps/webapp/build
|
|
COPY --from=builder --chown=node:node /triggerdotdev/apps/webapp/public ./apps/webapp/public
|
|
COPY --from=builder --chown=node:node /triggerdotdev/apps/webapp/prisma/seed.js ./apps/webapp/prisma/seed.js
|
|
COPY --from=builder --chown=node:node /triggerdotdev/scripts ./scripts
|
|
|
|
EXPOSE 3000
|
|
|
|
USER node
|
|
CMD ["./scripts/entrypoint.sh"]
|