Eric Allam
b1b9321ad2
* WIP job run performance improvements - Added a `perf` tool to better measure job run performance under heavy load - Removed `runFinished` job (not really needed) - startQueuedRuns now uses a jobKey with replace - Fixed an issue with ZodWorker when using jobKey * Publish improvement docker images * fixed the improvement docker publishing * Downgrade back to prisma 4.16.0 because 5.1.x broke docker builds * Changes to how queued runs work - Split the worker into two different workers, one dedicated to performRunExecution - Schedule performRunExecution in a single place, with a queue and using a round robin manually controlled concurrency - Remove startQueuedRuns - All runs are queued before they are started - Setting the worker maxPoolSize to the same as the worker concurrency - Starting to be able to split the docker image * Remove queue name from startRun graphile job * Make the prisma connection pool stuff configurable through env vars * Hardcode (for now) the max concurrent runs limit * Rewrite performRunExecution to be more performant PerformRunExecutionV2: - Does not create and manage jobRunExecution records - Does not reimplement retrying, uses graphile worker retrying instead I’ve kept around PerformRunExecutionV1 so this works when deploying. Definitely needs LOTS of testing * Fix issues with cached tasks - Limit the size of the cached tasks sent when executing a run, using the knapsack problem dynamic programming approach - Actually USE the cached tasks in IO by using the idempotencyKey instead of the task ID - Remove output from all logs - Added a stress test job catalog * Forgot to commit the logger updates * Never log connectionString * Login to docker hub to get around rate limits * Add additional logging to the graphile workers * Fix the *_ENABLED env vars * Allow adding and removing jobs to be done from the webapp * Don’t set the job to failed if it’s being retried * Deprecated queue options in the job and removed startPosition. Now using the job/env combo as the job queue name * Dequeung jobs doesn’t check if the runner is initialized * Fixed issues with retrying a run getting stuck on a cancelled task, and errors from parsing the results of dequeing a job * Remove queued round robin thing that isn’t used anymore * Added slack to job catalog * Better forwards compat * Added long delay * Fixed lock file
47 lines
2.0 KiB
TypeScript
47 lines
2.0 KiB
TypeScript
import { z } from "zod";
|
|
import { SecretStoreOptionsSchema } from "./services/secrets/secretStore.server";
|
|
|
|
const EnvironmentSchema = z.object({
|
|
NODE_ENV: z.union([z.literal("development"), z.literal("production"), z.literal("test")]),
|
|
DATABASE_URL: z.string(),
|
|
DATABASE_CONNECTION_LIMIT: z.coerce.number().int().default(10),
|
|
DATABASE_POOL_TIMEOUT: z.coerce.number().int().default(60),
|
|
DIRECT_URL: z.string(),
|
|
SESSION_SECRET: z.string(),
|
|
MAGIC_LINK_SECRET: z.string(),
|
|
ENCRYPTION_KEY: z.string(),
|
|
REMIX_APP_PORT: z.string().optional(),
|
|
LOGIN_ORIGIN: z.string().default("http://localhost:3030"),
|
|
APP_ORIGIN: z.string().default("http://localhost:3030"),
|
|
APP_ENV: z
|
|
.union([
|
|
z.literal("development"),
|
|
z.literal("production"),
|
|
z.literal("test"),
|
|
z.literal("staging"),
|
|
])
|
|
.default(process.env.NODE_ENV),
|
|
SECRET_STORE: SecretStoreOptionsSchema.default("DATABASE"),
|
|
POSTHOG_PROJECT_KEY: z.string().optional(),
|
|
TELEMETRY_TRIGGER_API_KEY: z.string().optional(),
|
|
TELEMETRY_TRIGGER_API_URL: z.string().optional(),
|
|
HIGHLIGHT_PROJECT_ID: z.string().optional(),
|
|
AUTH_GITHUB_CLIENT_ID: z.string().optional(),
|
|
AUTH_GITHUB_CLIENT_SECRET: z.string().optional(),
|
|
FROM_EMAIL: z.string().optional(),
|
|
REPLY_TO_EMAIL: z.string().optional(),
|
|
RESEND_API_KEY: z.string().optional(),
|
|
PLAIN_API_KEY: z.string().optional(),
|
|
RUNTIME_PLATFORM: z.enum(["docker-compose", "ecs", "local"]).default("local"),
|
|
WORKER_SCHEMA: z.string().default("graphile_worker"),
|
|
WORKER_CONCURRENCY: z.coerce.number().int().default(10),
|
|
WORKER_POLL_INTERVAL: z.coerce.number().int().default(1000),
|
|
EXECUTION_WORKER_CONCURRENCY: z.coerce.number().int().default(10),
|
|
EXECUTION_WORKER_POLL_INTERVAL: z.coerce.number().int().default(1000),
|
|
WORKER_ENABLED: z.string().default("true"),
|
|
EXECUTION_WORKER_ENABLED: z.string().default("true"),
|
|
});
|
|
|
|
export type Environment = z.infer<typeof EnvironmentSchema>;
|
|
export const env = EnvironmentSchema.parse(process.env);
|