Tha.Les
378c64fbc4
The error path destroyed all evidence: rmtree on failure threw away the demucs stderr, the stage, and the device, leaving "Audio processing failed" as the only artifact -- undebuggable after the fact. - Failed jobs now move to jobs/failed/<id> with an error.txt recording stage, device, model, classified cause, stage timings, and the demucs stderr tail. Heavy payloads (source, stems, video) are stripped first so quarantines stay KB-scale. Expired after 7 days by a new sweep that runs even on persistent-library deployments (failure evidence is diagnostics, not library content). The TTL sweep skips failed/. - New app/pipeline/errors.py: SeparationError carries the stderr tail + device out of separate(); classify_failure() maps failure text to out-of-memory / unsupported-device / disk-full / bad-input / unknown. The classified cause surfaces as Job.error_detail, shown in the studio as a muted secondary line under the generic error message. - Per-stage wall-clock timings (download/prepare, analyze, separate, post) recorded on the job, written to metadata.json, included in error.txt, and emitted as a one-line completion summary with the compute device -- performance regressions and the CPU-vs-GPU question are now answerable from logs. Closes #277 Closes #294 Closes #293 Co-authored-by: Thales <>
44 lines
1.5 KiB
Python
44 lines
1.5 KiB
Python
"""Tests for the pipeline failure classifier and SeparationError (#294, #277)."""
|
|
|
|
from __future__ import annotations
|
|
|
|
import pytest
|
|
|
|
from app.pipeline.errors import SeparationError, classify_failure
|
|
|
|
|
|
@pytest.mark.parametrize(
|
|
("text", "expected"),
|
|
[
|
|
("RuntimeError: CUDA out of memory. Tried to allocate 2.00 GiB", "out-of-memory"),
|
|
("RuntimeError: MPS backend out of memory (MPS allocated: 5.2 GB)", "out-of-memory"),
|
|
("OSError: cannot allocate memory", "out-of-memory"),
|
|
("RuntimeError: no kernel image is available for execution", "unsupported-device"),
|
|
("AssertionError: Torch not compiled with CUDA enabled", "unsupported-device"),
|
|
("OSError: [Errno 28] No space left on device", "disk-full"),
|
|
("Invalid data found when processing input", "bad-input"),
|
|
("RuntimeError: no stems produced by demucs", "bad-input"),
|
|
("something entirely novel went wrong", "unknown"),
|
|
("", "unknown"),
|
|
],
|
|
)
|
|
def test_classify_failure(text: str, expected: str):
|
|
assert classify_failure(text) == expected
|
|
|
|
|
|
def test_classify_is_case_insensitive():
|
|
assert classify_failure("CUDA OUT OF MEMORY") == "out-of-memory"
|
|
|
|
|
|
def test_separation_error_carries_evidence():
|
|
err = SeparationError("demucs failed: boom", tail=["line1", "boom"], device="mps")
|
|
assert isinstance(err, RuntimeError)
|
|
assert err.tail == ["line1", "boom"]
|
|
assert err.device == "mps"
|
|
|
|
|
|
def test_separation_error_defaults():
|
|
err = SeparationError("plain")
|
|
assert err.tail == []
|
|
assert err.device is None
|