A unit-test shard recently failed on a timing race rather than a real
regression - a run-engine waitpoint test sleeps 1250ms waiting on a
1000ms timeout that's processed by a ~1000ms worker poll, so on a
CPU-starved shard the margin evaporates and the whole matrix goes red.
Because `fail-fast` defaults on, that one flake cancels the sibling
shards, and the only recovery is re-running the entire matrix "just to
be sure" - which is itself slow.
This is the low-risk first pass at that pain:
- `fail-fast: false` on the webapp and internal shard matrices, so one
flaky shard no longer cancels its siblings. "Re-run failed jobs" now
re-runs just the failed shard instead of the whole matrix.
- CI-scoped `retry: process.env.CI ? 2 : 0` on the timing-sensitive
packages (`run-engine`, `redis-worker`, `schedule-engine`). Flakes
self-heal in CI; local runs stay at `retry: 0` so they still surface in
dev. A stopgap until the timing tests are made deterministic.
- `fetch-depth: 1` on the unit-test checkouts - they don't use git
history, so the full clone was wasted setup time across ~20 jobs.
- Reconcile the pre-pull image tags with what testcontainers actually
pulls (`redis:7-alpine` -> `redis:7.2`, `ryuk:0.11.0` -> `ryuk:0.14.0`)
and add `minio/minio:latest` to the webapp pre-pull. Otherwise those
images pull unauthenticated at test time and risk Docker Hub rate-limit
flakes (worst on fork PRs, where the authenticated pre-pull is skipped
entirely).
Deeper follow-ups - bigger runners, turbo remote cache, runtime-weighted
sharding, and the real root-cause fix (container reuse / template-DB
isolation + deterministic timing tests) - are tracked under TRI-10484.
nicktrn
707bf1adb4