nicktrn
2c82d4c4d1
Adds an in-process backpressure signal that pauses dequeuing when the
Kubernetes cluster is saturated, so work overflows cheaply in the queue
instead of piling up as unschedulable pods. Saturation is read by
scraping the apiserver's total pod-object count
(`apiserver_storage_objects{resource="pods"}`) and applying an
engage/release threshold with hysteresis - a single lightweight
aggregate scrape, not a pod listing.
Backpressure sources are now evaluated independently and OR'd: each
source has its own enable and dry-run flag, and the supervisor engages
if any enabled source trips. This adds the pod-count source alongside
the existing one without changing it, and is extensible to more sources
later. Off by default.
The scrape uses the in-cluster kubeconfig over `https` so TLS verifies
against the cluster CA (the fetch-options helper attaches the CA as an
`https.Agent`, which the global `fetch` ignores - that path silently
dropped the CA). Enabling the pod-count source requires the supervisor's
service account to be granted `get` on the `/metrics` non-resource URL;
that RBAC and the per-deployment env wiring are operator-side and live
elsewhere.
New config (pod-count source):
`TRIGGER_DEQUEUE_BACKPRESSURE_POD_COUNT_ENABLED` (default false),
`_POD_COUNT_DRY_RUN` (default true), `_POD_COUNT_ENGAGE` /
`_POD_COUNT_RELEASE` (hysteresis thresholds), `_POD_COUNT_REFRESH_MS`
(scrape interval, default 5s). The existing source's flags are
unchanged.
Observability: a `supervisor_cluster_pod_count` gauge, and the pod-count
monitor's metrics are namespaced (`supervisor_backpressure_pod_count_*`)
so the existing backpressure metrics keep their names.
65 lines
1.9 KiB
TypeScript
65 lines
1.9 KiB
TypeScript
import { describe, it, expect, vi } from "vitest";
|
|
|
|
// Mock std-env before importing env.ts so the module-level `Env.parse(stdEnv)`
|
|
// doesn't fail in a test environment that lacks required vars.
|
|
vi.mock("std-env", () => ({
|
|
env: {
|
|
TRIGGER_API_URL: "http://localhost:3030",
|
|
TRIGGER_WORKER_TOKEN: "test-token",
|
|
MANAGED_WORKER_SECRET: "test-secret",
|
|
OTEL_EXPORTER_OTLP_ENDPOINT: "http://localhost:4318",
|
|
},
|
|
}));
|
|
|
|
const { Env } = await import("./env.js");
|
|
|
|
// Minimal env that satisfies all required fields; everything else has defaults.
|
|
const base = {
|
|
TRIGGER_API_URL: "http://localhost:3030",
|
|
TRIGGER_WORKER_TOKEN: "test-token",
|
|
MANAGED_WORKER_SECRET: "test-secret",
|
|
OTEL_EXPORTER_OTLP_ENDPOINT: "http://localhost:4318",
|
|
};
|
|
|
|
describe("Env superRefine - backpressure source awareness", () => {
|
|
it("pod-count source can be enabled without a Redis host", () => {
|
|
expect(() =>
|
|
Env.parse({
|
|
...base,
|
|
TRIGGER_DEQUEUE_BACKPRESSURE_POD_COUNT_ENABLED: "true",
|
|
})
|
|
).not.toThrow();
|
|
});
|
|
|
|
it("redis source requires a Redis host", () => {
|
|
expect(() =>
|
|
Env.parse({
|
|
...base,
|
|
TRIGGER_DEQUEUE_BACKPRESSURE_ENABLED: "true",
|
|
})
|
|
).toThrow();
|
|
});
|
|
|
|
it("both sources can be enabled together (with a Redis host)", () => {
|
|
expect(() =>
|
|
Env.parse({
|
|
...base,
|
|
TRIGGER_DEQUEUE_BACKPRESSURE_ENABLED: "true",
|
|
TRIGGER_DEQUEUE_BACKPRESSURE_REDIS_HOST: "localhost",
|
|
TRIGGER_DEQUEUE_BACKPRESSURE_POD_COUNT_ENABLED: "true",
|
|
})
|
|
).not.toThrow();
|
|
});
|
|
|
|
it("rejects pod-count release >= engage when the source is enabled", () => {
|
|
expect(() =>
|
|
Env.parse({
|
|
...base,
|
|
TRIGGER_DEQUEUE_BACKPRESSURE_POD_COUNT_ENABLED: "true",
|
|
TRIGGER_DEQUEUE_BACKPRESSURE_POD_COUNT_ENGAGE: "100",
|
|
TRIGGER_DEQUEUE_BACKPRESSURE_POD_COUNT_RELEASE: "100",
|
|
})
|
|
).toThrow();
|
|
});
|
|
});
|