mirror of
https://github.com/nomadkaraoke/karaoke-gen.git
synced 2026-11-20 13:07:57 +09:00
fix(jobs): unstick audio-edit jobs — waveform OOM, retry skip, screens handoff (#1049)
## Summary Job `2c1b922e` (Dave Matthews Band – Why I Am, 24-bit/96kHz FLAC, audio edit requested) stalled twice in prod. Three bugs: 1. **Waveform OOM** — `WaveformGenerator.generate_data_only` decoded the full source at native resolution through pydub. For a 6-min hi-res FLAC that peaked at ~1 GB RSS and OOM-killed the 1 GiB `audio-download-job` while pre-generating editor assets. It now decodes straight to 16 kHz mono PCM via ffmpeg and computes the envelope with numpy: **156 MB vs 1.07 GB peak** on the real file, amplitudes within 0.003 of the old output. The same function backs the editor's on-demand waveform in the API. 2. **Retry skipped unfinished audio-edit prep** — the Cloud Run auto-retry saw `downloading`, treated the download as done and exited 0, leaving the job at `downloading` forever. A retry that finds an audio-edit job still at `downloading` now resumes the editor prep. That path triggers nothing downstream, so it's safe. Non-edit jobs keep the existing skip (downstream may already be triggered). 3. **Screens handoff ignored `audio_edit_complete`** — `advance_to_screens_if_ready` (added in #1002) and the `recover-stuck-jobs` lost-screens sweep only accepted `downloading`. Audio-edited jobs prep at `audio_edit_complete`, so **every audio-edited job since Sept 15 stalled** after lyrics and separation finished (log: `screens not (re)triggered — status=audio_edit_complete`). Both now use a shared `PREP_PHASE_STATUSES`. 2c1b922e was unstuck manually (admin reset → `awaiting_audio_edit`, then a direct `/api/internal/workers/screens` dispatch). Added a TROUBLESHOOTING entry covering both signatures and the manual fixes. ## Testing - New tests: hi-res 96k/24-bit decode path (low-rate mono args, envelope matches pydub reference, silent → 0, undecodable → error); download-worker resume vs skip; handoff at `audio_edit_complete`; watchdog recovers `audio_edit_complete` and queries both statuses. - `pytest tests/unit/` → 1824 passed, 8 skipped - `pytest backend/tests/ --ignore=backend/tests/emulator` → 4586 passed, 36 skipped - Version 0.238.4 → 0.238.5 🤖 Generated with [Claude Code](https://claude.com/claude-code) <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit * **Bug Fixes** * Audio-edit jobs can resume preparation after an interruption without redownloading existing audio. * Screen generation can proceed after audio editing is complete, and stalled screen-generation triggers can be recovered. * Waveform previews are more reliable for high-resolution and silent audio. Invalid audio and decoding timeouts now produce clear errors. * **Documentation** * Added troubleshooting guidance for jobs stuck during audio editing or before screen generation. <!-- end of auto-generated comment: release notes by coderabbit.ai --> --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
1 parent
daa46347f7
commit
bcdfd6b6bb
10 files changed
+379
-63
No files matched your search
@@ -12,6 +12,7 @@ Tests cover:
|
||||
import os
|
||||
import pytest
|
||||
from pathlib import Path
|
||||
from pydub import AudioSegment
|
||||
|
||||
from karaoke_gen.instrumental_review import (
|
||||
WaveformGenerator,
|
||||
@@ -413,3 +414,78 @@ class TestCustomDimensions:
|
||||
)
|
||||
|
||||
assert os.path.exists(output_path)
|
||||
|
||||
|
||||
class TestDataOnlyHiResDecode:
|
||||
"""generate_data_only must not decode the source at native resolution.
|
||||
|
||||
A 6-minute 24-bit/96kHz stereo FLAC peaked at ~1 GB through pydub and
|
||||
OOM-killed the 1 GiB audio-download Cloud Run Job (job 2c1b922e).
|
||||
"""
|
||||
|
||||
@pytest.fixture
|
||||
def hires_audio_path(self, temp_dir):
|
||||
import subprocess
|
||||
|
||||
path = os.path.join(temp_dir, "hires.flac")
|
||||
subprocess.run(
|
||||
[AudioSegment.converter, "-nostdin", "-v", "error", "-y",
|
||||
"-f", "lavfi", "-i", "sine=frequency=440:sample_rate=96000:duration=5",
|
||||
"-af", "volume=-10dB", "-ac", "2",
|
||||
"-sample_fmt", "s32", "-bits_per_raw_sample", "24", path],
|
||||
check=True,
|
||||
)
|
||||
return path
|
||||
|
||||
def test_decodes_at_low_rate_mono(self, hires_audio_path):
|
||||
import subprocess
|
||||
from unittest.mock import patch
|
||||
|
||||
real_run = subprocess.run
|
||||
with patch("karaoke_gen.instrumental_review.waveform.subprocess.run",
|
||||
side_effect=real_run) as mock_run:
|
||||
WaveformGenerator().generate_data_only(hires_audio_path, num_points=100)
|
||||
|
||||
cmd = mock_run.call_args.args[0]
|
||||
assert cmd[cmd.index("-ac") + 1] == "1"
|
||||
assert cmd[cmd.index("-ar") + 1] == "16000"
|
||||
|
||||
def test_matches_native_resolution_envelope(self, hires_audio_path):
|
||||
"""Low-rate envelope ≈ the pydub native-resolution RMS envelope."""
|
||||
import math
|
||||
|
||||
amplitudes, duration = WaveformGenerator().generate_data_only(
|
||||
hires_audio_path, num_points=100
|
||||
)
|
||||
|
||||
audio = AudioSegment.from_file(hires_audio_path).set_channels(1)
|
||||
window_ms = len(audio) // 100
|
||||
expected = []
|
||||
for start in range(0, len(audio), window_ms):
|
||||
w = audio[start:start + window_ms]
|
||||
db = 20 * math.log10(w.rms / w.max_possible_amplitude) if w.rms else -100.0
|
||||
expected.append(max(0.0, min(1.0, (db + 60) / 60)))
|
||||
|
||||
assert abs(duration - 5.0) < 0.01
|
||||
assert len(amplitudes) == len(expected)
|
||||
assert max(abs(a - e) for a, e in zip(amplitudes, expected)) < 0.01
|
||||
|
||||
def test_silent_audio_is_zero(self, silent_audio_path):
|
||||
amplitudes, _ = WaveformGenerator().generate_data_only(silent_audio_path, num_points=50)
|
||||
assert amplitudes and all(a == 0.0 for a in amplitudes)
|
||||
|
||||
def test_undecodable_file_raises(self, temp_dir):
|
||||
path = os.path.join(temp_dir, "garbage.flac")
|
||||
with open(path, "wb") as f:
|
||||
f.write(b"not audio at all")
|
||||
with pytest.raises(RuntimeError, match="ffmpeg failed"):
|
||||
WaveformGenerator().generate_data_only(path)
|
||||
|
||||
def test_decode_timeout_raises_runtime_error(self, loud_audio_path):
|
||||
import subprocess
|
||||
from unittest.mock import patch
|
||||
|
||||
with patch("karaoke_gen.instrumental_review.waveform.subprocess.run",
|
||||
side_effect=subprocess.TimeoutExpired(cmd="ffmpeg", timeout=300)):
|
||||
with pytest.raises(RuntimeError, match="timed out after 300"):
|
||||
WaveformGenerator().generate_data_only(loud_audio_path)
|
||||
Reference in new issue
Block a user