-
[OPIK-6308] [BE] feat: fast-fail with HTTP 503 when Python BE pool is saturated (#7091)
frostbyte_neo 发布于
2026-06-15 08:03:53 +00:00 -
[OPIK-6308] [BE] feat: fast-fail with HTTP 503 when Python BE pool is saturated
-
fix(executor): initialize tracer before pre-warm so the pool is actually populated
DockerExecutor.init set self.tracer AFTER _pre_warm_container_pool() and
_start_pool_monitor(). create_container opens a span on self.tracer, so the
pre-warm futures raised AttributeError, the ThreadPoolExecutor swallowed them,
and the pool started empty. The background scheduler then refilled it on its
next tick.The original get_container masked this with a 10-second blocking retry, so
callers waited for the scheduler to catch up. Iteration 1 of this PR replaced
that with a fast-fail TimeoutError → HTTP 503, exposing the latent bug:
every fresh DockerExecutor would return 503 for the first ~3s of its life,
including in CI where the Docker executor tests now fail with the saturation
body.Hoist the tracer assignment above the two pre-warm callers so create_container
has a usable span object from the first invocation.- fix(executor): distinguish shutdown-during-wait from saturation in run_scoring
Address Baz review (PR #7046): the get_*() pre-check covered the
"shutdown already in progress before call" case, but stop_event can also
fire between that pre-check and the bounded Queue.get. The resulting
Empty -> TimeoutError was being collapsed into SATURATED_ERROR by
run_scoring, double-counting the shutdown as a pool-saturation event in
metrics and giving operators a false-positive saturation alert.- Docker run_scoring re-checks stop_event in the
except TimeoutError
branch; on shutdown it returns SHUTDOWN_ERROR and skips the
saturated outcome counter tick. - Process run_scoring re-checks stop_event symmetrically.
- Add a parametrized-style race test on both executors that patches
get_container/get_worker to set stop_event and then raise the
bounded-wait TimeoutError, confirming the response collapses to
SHUTDOWN_ERROR and the Docker saturated outcome counter does not fire.
-
refactor(executor): address baz review
-
refactor(executor): address Thiago's comment plus other improvements
下载附件
-