fix(jobs): unstick audio-edit jobs — waveform OOM, retry skip, screens handoff (#1049)

## Summary
Job `2c1b922e` (Dave Matthews Band – Why I Am, 24-bit/96kHz FLAC, audio
edit requested) stalled twice in prod. Three bugs:

1. **Waveform OOM** — `WaveformGenerator.generate_data_only` decoded the
full source at native resolution through pydub. For a 6-min hi-res FLAC
that peaked at ~1 GB RSS and OOM-killed the 1 GiB `audio-download-job`
while pre-generating editor assets. It now decodes straight to 16 kHz
mono PCM via ffmpeg and computes the envelope with numpy: **156 MB vs
1.07 GB peak** on the real file, amplitudes within 0.003 of the old
output. The same function backs the editor's on-demand waveform in the
API.
2. **Retry skipped unfinished audio-edit prep** — the Cloud Run
auto-retry saw `downloading`, treated the download as done and exited 0,
leaving the job at `downloading` forever. A retry that finds an
audio-edit job still at `downloading` now resumes the editor prep. That
path triggers nothing downstream, so it's safe. Non-edit jobs keep the
existing skip (downstream may already be triggered).
3. **Screens handoff ignored `audio_edit_complete`** —
`advance_to_screens_if_ready` (added in #1002) and the
`recover-stuck-jobs` lost-screens sweep only accepted `downloading`.
Audio-edited jobs prep at `audio_edit_complete`, so **every audio-edited
job since Sept 15 stalled** after lyrics and separation finished (log:
`screens not (re)triggered — status=audio_edit_complete`). Both now use
a shared `PREP_PHASE_STATUSES`.

2c1b922e was unstuck manually (admin reset → `awaiting_audio_edit`, then
a direct `/api/internal/workers/screens` dispatch). Added a
TROUBLESHOOTING entry covering both signatures and the manual fixes.

## Testing
- New tests: hi-res 96k/24-bit decode path (low-rate mono args, envelope
matches pydub reference, silent → 0, undecodable → error);
download-worker resume vs skip; handoff at `audio_edit_complete`;
watchdog recovers `audio_edit_complete` and queries both statuses.
- `pytest tests/unit/` → 1824 passed, 8 skipped
- `pytest backend/tests/ --ignore=backend/tests/emulator` → 4586 passed,
36 skipped
- Version 0.238.4 → 0.238.5

🤖 Generated with [Claude Code](https://claude.com/claude-code)

<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit

* **Bug Fixes**
* Audio-edit jobs can resume preparation after an interruption without
redownloading existing audio.
* Screen generation can proceed after audio editing is complete, and
stalled screen-generation triggers can be recovered.
* Waveform previews are more reliable for high-resolution and silent
audio. Invalid audio and decoding timeouts now produce clear errors.
* **Documentation**
* Added troubleshooting guidance for jobs stuck during audio editing or
before screen generation.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
Andrew Beveridge ☄️andClaude Opus 5.5 authored and GitHub committed 2026-09-24 23:39:25 -04:00
1 parent daa46347f7
commit bcdfd6b6bb
10 files changed
+379 -63

No files matched your search

@@ -12,6 +12,7 @@ Tests cover:
import os
import pytest
from pathlib import Path
from pydub import AudioSegment
from karaoke_gen.instrumental_review import (
WaveformGenerator,
@@ -413,3 +414,78 @@ class TestCustomDimensions:
)
assert os.path.exists(output_path)
class TestDataOnlyHiResDecode:
"""generate_data_only must not decode the source at native resolution.
A 6-minute 24-bit/96kHz stereo FLAC peaked at ~1 GB through pydub and
OOM-killed the 1 GiB audio-download Cloud Run Job (job 2c1b922e).
"""
@pytest.fixture
def hires_audio_path(self, temp_dir):
import subprocess
path = os.path.join(temp_dir, "hires.flac")
subprocess.run(
[AudioSegment.converter, "-nostdin", "-v", "error", "-y",
"-f", "lavfi", "-i", "sine=frequency=440:sample_rate=96000:duration=5",
"-af", "volume=-10dB", "-ac", "2",
"-sample_fmt", "s32", "-bits_per_raw_sample", "24", path],
check=True,
)
return path
def test_decodes_at_low_rate_mono(self, hires_audio_path):
import subprocess
from unittest.mock import patch
real_run = subprocess.run
with patch("karaoke_gen.instrumental_review.waveform.subprocess.run",
side_effect=real_run) as mock_run:
WaveformGenerator().generate_data_only(hires_audio_path, num_points=100)
cmd = mock_run.call_args.args[0]
assert cmd[cmd.index("-ac") + 1] == "1"
assert cmd[cmd.index("-ar") + 1] == "16000"
def test_matches_native_resolution_envelope(self, hires_audio_path):
"""Low-rate envelope ≈ the pydub native-resolution RMS envelope."""
import math
amplitudes, duration = WaveformGenerator().generate_data_only(
hires_audio_path, num_points=100
)
audio = AudioSegment.from_file(hires_audio_path).set_channels(1)
window_ms = len(audio) // 100
expected = []
for start in range(0, len(audio), window_ms):
w = audio[start:start + window_ms]
db = 20 * math.log10(w.rms / w.max_possible_amplitude) if w.rms else -100.0
expected.append(max(0.0, min(1.0, (db + 60) / 60)))
assert abs(duration - 5.0) < 0.01
assert len(amplitudes) == len(expected)
assert max(abs(a - e) for a, e in zip(amplitudes, expected)) < 0.01
def test_silent_audio_is_zero(self, silent_audio_path):
amplitudes, _ = WaveformGenerator().generate_data_only(silent_audio_path, num_points=50)
assert amplitudes and all(a == 0.0 for a in amplitudes)
def test_undecodable_file_raises(self, temp_dir):
path = os.path.join(temp_dir, "garbage.flac")
with open(path, "wb") as f:
f.write(b"not audio at all")
with pytest.raises(RuntimeError, match="ffmpeg failed"):
WaveformGenerator().generate_data_only(path)
def test_decode_timeout_raises_runtime_error(self, loud_audio_path):
import subprocess
from unittest.mock import patch
with patch("karaoke_gen.instrumental_review.waveform.subprocess.run",
side_effect=subprocess.TimeoutExpired(cmd="ffmpeg", timeout=300)):
with pytest.raises(RuntimeError, match="timed out after 300"):
WaveformGenerator().generate_data_only(loud_audio_path)