Files
karaoke-gen/tests
Andrew Beveridge ☄️andClaude Opus 5.5 bcdfd6b6bb fix(jobs): unstick audio-edit jobs — waveform OOM, retry skip, screens handoff (#1049)
## Summary
Job `2c1b922e` (Dave Matthews Band – Why I Am, 24-bit/96kHz FLAC, audio
edit requested) stalled twice in prod. Three bugs:

1. **Waveform OOM** — `WaveformGenerator.generate_data_only` decoded the
full source at native resolution through pydub. For a 6-min hi-res FLAC
that peaked at ~1 GB RSS and OOM-killed the 1 GiB `audio-download-job`
while pre-generating editor assets. It now decodes straight to 16 kHz
mono PCM via ffmpeg and computes the envelope with numpy: **156 MB vs
1.07 GB peak** on the real file, amplitudes within 0.003 of the old
output. The same function backs the editor's on-demand waveform in the
API.
2. **Retry skipped unfinished audio-edit prep** — the Cloud Run
auto-retry saw `downloading`, treated the download as done and exited 0,
leaving the job at `downloading` forever. A retry that finds an
audio-edit job still at `downloading` now resumes the editor prep. That
path triggers nothing downstream, so it's safe. Non-edit jobs keep the
existing skip (downstream may already be triggered).
3. **Screens handoff ignored `audio_edit_complete`** —
`advance_to_screens_if_ready` (added in #1002) and the
`recover-stuck-jobs` lost-screens sweep only accepted `downloading`.
Audio-edited jobs prep at `audio_edit_complete`, so **every audio-edited
job since Sept 15 stalled** after lyrics and separation finished (log:
`screens not (re)triggered — status=audio_edit_complete`). Both now use
a shared `PREP_PHASE_STATUSES`.

2c1b922e was unstuck manually (admin reset → `awaiting_audio_edit`, then
a direct `/api/internal/workers/screens` dispatch). Added a
TROUBLESHOOTING entry covering both signatures and the manual fixes.

## Testing
- New tests: hi-res 96k/24-bit decode path (low-rate mono args, envelope
matches pydub reference, silent → 0, undecodable → error);
download-worker resume vs skip; handoff at `audio_edit_complete`;
watchdog recovers `audio_edit_complete` and queries both statuses.
- `pytest tests/unit/` → 1824 passed, 8 skipped
- `pytest backend/tests/ --ignore=backend/tests/emulator` → 4586 passed,
36 skipped
- Version 0.238.4 → 0.238.5

🤖 Generated with [Claude Code](https://claude.com/claude-code)

<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit

* **Bug Fixes**
* Audio-edit jobs can resume preparation after an interruption without
redownloading existing audio.
* Screen generation can proceed after audio editing is complete, and
stalled screen-generation triggers can be recovered.
* Waveform previews are more reliable for high-resolution and silent
audio. Invalid audio and decoding timeouts now produce clear errors.
* **Documentation**
* Added troubleshooting guidance for jobs stuck during audio editing or
before screen generation.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-09-24 23:39:25 -04:00
..