From 0d0605d7b557e249034cdbb98a73e30467f327cc Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Andrew=20Beveridge=20=E2=98=84=EF=B8=8F?= Date: Mon, 19 Oct 2026 02:21:52 -0400 Subject: [PATCH] fix: final encode waits for stem separation instead of stranding the job; pin audio-separator 0.44.2 (v0.278.11) (#1191) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit ## Problem The daily E2E Stage 2 failed on 2026-10-17 (run #216) and 2026-10-18 (#217). When a customer finishes review **before** stem separation does, `video-encoding-job` fails its prerequisites (`Missing instrumental: instrumental_clean`) 3× within seconds. The job then sits at `instrumental_selected` ("Starting final encode") forever: it's never marked failed, nothing alerts, and separation finishing doesn't re-trigger the encode. The race became common when the GPU image's unpinned `pip install "audio-separator[gpu]"` picked up **0.47.0** on the 10-17 rebuild. On L4, the mel-band RoFormer **karaoke** models run 3–5× slower than on 0.44.2, so total separation went from ~5 to ~10–16 min: | model (~3-min track) | 0.44.2 | 0.47.0 | |---|---|---| | karaoke_aufr33_viperx | ~25s | 75s | | karaoke_gabox_v2 | ~26s | 149s | | karaoke_becruily | ~27s | 157s | | instrumental fv7z / resurrection | ~50–60s | ~62s | The model YAMLs still say `num_overlap: 8, batch_size: 1`, so this isn't the #299 YAML-defaults change; the cause is somewhere in 0.45–0.47's RoFormer inference changes. That root cause is tracked separately. ## Fix - **Pin** `audio-separator[gpu]==0.44.2` in `backend/Dockerfile.gpu-base` (`ARG AUDIO_SEPARATOR_VERSION`). Changing this file triggers the GPU base image rebuild in CI. - **Video worker parks instead of failing.** When the selected stem is missing *and* separation is still running (`audio_complete` false, valid status), it sets `state_data.video_waiting_for_stems`, puts a user-visible message ("Waiting for audio separation to finish; the final video starts automatically") and exits 0, so Cloud Run doesn't retry. It re-checks after parking, and an atomic Firestore test-and-clear (`claim_job_state_flag`) means exactly one of the video worker and the audio worker proceeds if the stems land mid-park. - **Audio worker resumes it.** After `mark_audio_complete`, `resume_video_if_waiting_for_stems` re-triggers a parked encode. The fake-workers pipeline does the same, so the local stack matches prod. - **Backstop in `recover_stuck_jobs`.** `instrumental_selected` for more than 10 min with the stem present → re-trigger the video worker (covers a lost resume or lost dispatch). More than 60 min with separation never finishing → `mark_job_failed` (`stage=video_waiting_for_stems`), so the job fails loudly. Jobs older than 24h are never revived. (No jobs are in `instrumental_selected` in prod right now.) ## Testing - New `backend/tests/test_video_waits_for_stems.py` (16 tests): pending detection (including that a wrong status never parks), park / continue / claim-race outcomes, and resume (flag claimed / no flag / job moved on). - `test_recover_stuck_jobs_repark.py`: 5 new sweep tests (re-trigger, fresh, still separating, never-ready → fail, >24h ignored). - Full `backend/tests` (excluding emulator): 6455 passed. - Local CodeRabbit review: 0 findings. ## Verify after deploy - The GPU job logs `Separator version 0.44.2`, and karaoke-model separation is back to ~20–30s. - The next daily E2E Stage 2 passes. @coderabbitai ignore 🤖 Generated with [Claude Code](https://claude.com/claude-code) https://claude.ai/code/session_019pmG2XAgtUguEJWLFpkKHN Co-authored-by: Claude Opus 5.5 --- backend/Dockerfile.gpu-base | 10 +- backend/api/routes/internal.py | 46 ++++++ backend/services/fake_workers/pipeline.py | 2 + backend/services/firestore_service.py | 21 +++ backend/services/job_health_service.py | 18 +++ backend/services/job_manager.py | 35 +++++ .../tests/test_recover_stuck_jobs_repark.py | 77 +++++++++- backend/tests/test_video_waits_for_stems.py | 132 ++++++++++++++++++ backend/workers/audio_worker.py | 9 ++ backend/workers/video_worker.py | 42 +++++- docs/TROUBLESHOOTING.md | 17 +++ pyproject.toml | 2 +- 12 files changed, 405 insertions(+), 6 deletions(-) create mode 100644 backend/tests/test_video_waits_for_stems.py diff --git a/backend/Dockerfile.gpu-base b/backend/Dockerfile.gpu-base index d190d996..0f4cfc24 100644 --- a/backend/Dockerfile.gpu-base +++ b/backend/Dockerfile.gpu-base @@ -102,9 +102,17 @@ RUN --mount=type=cache,target=/root/.cache/pip \ # have NVIDIA driver 570 which only supports up to CUDA 12.8. PyTorch cu130 # fails with "NVIDIA driver is too old". PyTorch cu126 works with 12.8 drivers. # Installing torch first means audio-separator[gpu] sees it as already satisfied. +# +# PINNED to an exact version: an unpinned install silently picked up 0.47.0 on the +# 2026-10-17 rebuild, and on L4 its mel-band RoFormer karaoke models run 3-5x slower +# than 0.44.2 (gabox_v2 ~26s -> 149s, becruily ~27s -> 157s for a ~3-min track; +# YAML overlap/batch are unchanged, so it is in the 0.45-0.47 inference changes). +# Total separation went from ~5 to ~10-16 min and the daily E2E started failing. +# Only bump this after timing a real job on the new version (see BACKLOG). +ARG AUDIO_SEPARATOR_VERSION=0.44.2 RUN --mount=type=cache,target=/root/.cache/pip \ pip install torch==2.6.0+cu126 torchvision==0.21.0+cu126 --index-url https://download.pytorch.org/whl/cu126 && \ - pip install "audio-separator[gpu]" + pip install "audio-separator[gpu]==${AUDIO_SEPARATOR_VERSION}" # Now copy actual source code (AFTER deps installed, so code changes don't # invalidate the expensive pip install layer in registry cache) diff --git a/backend/api/routes/internal.py b/backend/api/routes/internal.py index cf8bca6c..4a2fdccd 100644 --- a/backend/api/routes/internal.py +++ b/backend/api/routes/internal.py @@ -1362,6 +1362,47 @@ def _recover_stuck_jobs_sync(http_request: Request, loop: asyncio.AbstractEventL ) stems_restore_failed.append(job_id) + # --- INSTRUMENTAL_SELECTED with no encode running --- + # The video worker moves to GENERATING_VIDEO as soon as it starts, so a job + # still here after 10 min has no encode underway. Either it's parked waiting + # for stem separation (the audio worker resumes it; this catches a lost + # resume) or its video dispatch was lost. Re-trigger once the stems exist; + # if separation never finishes, fail loudly so the job doesn't sit silently + # (the 2026-10-17/18 stuck "Starting final encode" jobs). Bounded to 24h so + # ancient abandoned jobs are never revived. + from backend.services.job_health_service import selected_stem_pending + video_retriggered = [] + stems_wait_failed = [] + is_query = jobs_ref.where( + filter=FieldFilter("status", "==", JobStatus.INSTRUMENTAL_SELECTED.value) + ).limit(SCREENS_RECOVERY_SCAN_LIMIT).stream() + for doc in is_query: + if len(video_retriggered) >= RENDER_RETRIGGERS_PER_TICK: + break + job_id = (doc.to_dict() or {}).get("job_id", doc.id) + job = job_manager.get_job(job_id) + age = _job_updated_age_seconds(job) if job else None + if age is None or not (INSTRUMENTAL_SELECTED_STALL_SECONDS < age < INSTRUMENTAL_SELECTED_MAX_AGE_SECONDS): + continue + if selected_stem_pending(job): + if age > STEMS_WAIT_FAIL_SECONDS: + logger.error(f"[job:{job_id}] Final encode waited >60 min for stem separation that never finished — failing") + job_manager.mark_job_failed( + job_id=job_id, + error_message="Audio separation didn't finish, so the final video couldn't be made. Retry the job to try again.", + error_details={"stage": "video_waiting_for_stems", "reason": "stems_never_ready"}, + ) + stems_wait_failed.append(job_id) + continue + logger.warning(f"[job:{job_id}] INSTRUMENTAL_SELECTED stalled >10 min with stems ready — re-triggering video worker") + try: + # Clear a parked flag first so a late audio-worker resume can't double-trigger. + job_manager.claim_video_stems_wait(job_id) + if run_on_loop(loop, worker_service.trigger_video_worker(job_id)): + video_retriggered.append(job_id) + except Exception as e: # noqa: BLE001 + logger.warning(f"[job:{job_id}] video re-trigger failed: {e}") + logger.info( f"RECOVER_STUCK_JOBS complete: recovered={len(recovered)} " f"download_parked={len(download_parked)} download_retried={len(download_retried)} " @@ -1392,6 +1433,8 @@ def _recover_stuck_jobs_sync(http_request: Request, loop: asyncio.AbstractEventL "screens_retriggered_count": len(screens_retriggered), "stems_restore_failed_jobs": stems_restore_failed, "regen_screens_retriggered_jobs": regen_screens_retriggered, + "video_retriggered_jobs": video_retriggered, + "stems_wait_failed_jobs": stems_wait_failed, } @@ -1400,6 +1443,9 @@ RENDER_RETRIGGERS_PER_TICK = 10 # bound the blast radius per 5-min tick PREP_SCREENS_STALL_SECONDS = 10 * 60 # screens normally start within seconds of lyrics done SCREENS_RETRIGGERS_PER_TICK = 10 # bound the blast radius per 5-min tick SCREENS_RECOVERY_SCAN_LIMIT = 500 # cap DOWNLOADING docs scanned per tick (transient state) +INSTRUMENTAL_SELECTED_STALL_SECONDS = 10 * 60 # video worker leaves this state within seconds +STEMS_WAIT_FAIL_SECONDS = 60 * 60 # separation normally lands within ~15 min of review +INSTRUMENTAL_SELECTED_MAX_AGE_SECONDS = 24 * 3600 # never revive ancient abandoned jobs def _job_updated_age_seconds(job): diff --git a/backend/services/fake_workers/pipeline.py b/backend/services/fake_workers/pipeline.py index ae580399..de261346 100644 --- a/backend/services/fake_workers/pipeline.py +++ b/backend/services/fake_workers/pipeline.py @@ -134,6 +134,8 @@ async def run_audio(job_id: str) -> bool: job_manager.mark_audio_complete(job_id) await job_manager.advance_to_screens_if_ready(job_id) + # Same as the real audio worker: resume a final encode parked on these stems. + await job_manager.resume_video_if_waiting_for_stems(job_id) return True diff --git a/backend/services/firestore_service.py b/backend/services/firestore_service.py index 8f86eeba..bbd0d14b 100644 --- a/backend/services/firestore_service.py +++ b/backend/services/firestore_service.py @@ -131,6 +131,27 @@ class FirestoreService: logger.error(f"Error updating job {job_id}: {e}") raise + def claim_job_state_flag(self, job_id: str, key: str) -> bool: + """Atomically test-and-clear ``state_data.``. + + Returns True for exactly one caller while the flag is truthy (it is set + to False in the same transaction), so two workers racing to act on the + same flag can't both proceed. + """ + doc_ref = self.db.collection(self.collection).document(job_id) + + @firestore.transactional + def _claim(transaction, ref) -> bool: + snapshot = ref.get(transaction=transaction) + if not snapshot.exists: + return False + if not ((snapshot.to_dict() or {}).get("state_data") or {}).get(key): + return False + transaction.update(ref, {f"state_data.{key}": False, "updated_at": datetime.utcnow()}) + return True + + return _claim(self.db.transaction(), doc_ref) + def update_job_status( self, job_id: str, diff --git a/backend/services/job_health_service.py b/backend/services/job_health_service.py index eeb526ef..5b66e39b 100644 --- a/backend/services/job_health_service.py +++ b/backend/services/job_health_service.py @@ -287,3 +287,21 @@ def validate_worker_can_run(worker_name: str, job: Job) -> Optional[str]: ) return None + + +def selected_stem_pending(job) -> bool: + """True when the selected instrumental stem hasn't been produced yet but + stem separation is still running, i.e. the job should wait, not fail. + + Review can finish before separation does (audio separation is decoupled from + the lyrics review path), so this is a normal ordering, not an error. + """ + if validate_worker_can_run("video_worker", job): + return False # wrong status (duplicate/late trigger): never park, let the normal checks reject it + selection = (job.state_data or {}).get('instrumental_selection') + if selection not in ('clean', 'with_backing'): + return False # 'custom' uses an uploaded file / mute-region stem + stem_key = 'instrumental_clean' if selection == 'clean' else 'instrumental_with_backing' + if (job.file_urls or {}).get('stems', {}).get(stem_key): + return False + return not (job.state_data or {}).get('audio_complete', False) diff --git a/backend/services/job_manager.py b/backend/services/job_manager.py index df45fce3..fd0c9f25 100644 --- a/backend/services/job_manager.py +++ b/backend/services/job_manager.py @@ -1305,6 +1305,41 @@ class JobManager: self.update_state_data(job_id, 'audio_complete', True) logger.info(f"Job {job_id}: Audio processing complete (separation finished)") + # Set by the video worker when review finished before stem separation did + # (the selected instrumental stem doesn't exist yet). The audio worker + # resumes the encode when separation completes; recover_stuck_jobs is the + # backstop. Before this, the video worker just failed its prerequisites 3x + # and the job sat at instrumental_selected forever (2026-10-17/18). + VIDEO_WAITING_FOR_STEMS_KEY = 'video_waiting_for_stems' + + def park_video_for_stems(self, job_id: str) -> None: + """Record that the final encode is waiting for stem separation.""" + self.update_job(job_id, { + f'state_data.{self.VIDEO_WAITING_FOR_STEMS_KEY}': True, + 'message': 'Waiting for audio separation to finish; the final video starts automatically', + }) + + def claim_video_stems_wait(self, job_id: str) -> bool: + """Atomically clear the waiting flag; True for exactly one claimant.""" + return self.firestore.claim_job_state_flag(job_id, self.VIDEO_WAITING_FOR_STEMS_KEY) + + async def resume_video_if_waiting_for_stems(self, job_id: str) -> bool: + """Re-trigger the video worker for a job parked waiting on stems. + + Called by the audio worker right after ``mark_audio_complete``. No-op + unless the video worker parked the job (flag set) and it's still in a + state the video worker accepts. Returns True iff a dispatch was issued. + """ + if not self.claim_video_stems_wait(job_id): + return False + job = self.get_job(job_id) + if not job or job.status not in (JobStatus.INSTRUMENTAL_SELECTED, JobStatus.REVIEW_COMPLETE): + logger.info(f"Job {job_id}: stems ready but job is {getattr(job, 'status', None)}; not resuming video") + return False + from backend.services.worker_service import get_worker_service + logger.info(f"Job {job_id}: stems ready — resuming parked video worker") + return await get_worker_service().trigger_video_worker(job_id) + def mark_lyrics_complete(self, job_id: str) -> None: """ Mark lyrics processing as complete. diff --git a/backend/tests/test_recover_stuck_jobs_repark.py b/backend/tests/test_recover_stuck_jobs_repark.py index 35c9d95d..aa2df018 100644 --- a/backend/tests/test_recover_stuck_jobs_repark.py +++ b/backend/tests/test_recover_stuck_jobs_repark.py @@ -56,7 +56,7 @@ def _job(job_id, status, *, minutes_stale=0, source_name=None, download_retry=No def _mock_jm(*, downloading=None, pending=None, rendering=None, review_complete=None, - downloading_prep=None, jobs=None): + downloading_prep=None, instrumental_selected=None, jobs=None): """Wire a JobManager mock for the endpoint's 4 status queries. downloading/rendering/review_complete use .where().stream(); pending uses @@ -69,11 +69,13 @@ def _mock_jm(*, downloading=None, pending=None, rendering=None, review_complete= where.stream.side_effect = [ iter(downloading or []), iter(rendering or []), iter(review_complete or []), ] - # Three .limit().stream() calls in order: download_pending_retry, then downloading + # Four .limit().stream() calls in order: download_pending_retry, then downloading # (lost-screens-trigger sweep, bounded by SCREENS_RECOVERY_SCAN_LIMIT), then - # lyrics_complete (stalled storage-retention stems restore) + # lyrics_complete (stalled storage-retention stems restore), then + # instrumental_selected (final encode parked on stems / lost video trigger) where.limit.return_value.stream.side_effect = [ iter(pending or []), iter(downloading_prep or []), iter([]), + iter(instrumental_selected or []), ] mock_jm.get_job.side_effect = lambda jid: (jobs or {}).get(jid) mock_jm.transition_to_state.return_value = True @@ -284,3 +286,72 @@ def test_fresh_review_complete_is_left_alone(client): assert resp.status_code == 200 assert resp.json()["render_retriggered_jobs"] == [] mock_ws.trigger_render_video_worker.assert_not_awaited() + + +def _is_job(job_id, *, minutes_stale, audio_complete, stem_ready, selection="clean"): + job = _job(job_id, JobStatus.INSTRUMENTAL_SELECTED, minutes_stale=minutes_stale) + job.state_data.update({"instrumental_selection": selection, "audio_complete": audio_complete}) + if stem_ready: + job.file_urls = {"stems": {"instrumental_clean": "gs://b/clean.flac", + "instrumental_with_backing": "gs://b/backing.flac"}} + return job + + +def _run_is(client, job): + mock_jm = _mock_jm( + instrumental_selected=[_doc(job.job_id, JobStatus.INSTRUMENTAL_SELECTED)], + jobs={job.job_id: job}, + ) + mock_ws = MagicMock() + mock_ws.trigger_video_worker = AsyncMock(return_value=True) + with patch("backend.api.routes.internal.JobManager", return_value=mock_jm), \ + patch("backend.services.worker_service.get_worker_service", return_value=mock_ws): + resp = client.post("/api/internal/recover-stuck-jobs") + assert resp.status_code == 200 + return resp.json(), mock_jm, mock_ws + + +def test_instrumental_selected_with_stems_ready_retriggers_video(client): + """Parked/lost video trigger: stems landed but no encode started -> re-trigger.""" + body, mock_jm, mock_ws = _run_is( + client, _is_job("is1", minutes_stale=15, audio_complete=True, stem_ready=True)) + assert body["video_retriggered_jobs"] == ["is1"] + mock_ws.trigger_video_worker.assert_awaited_once_with("is1") + mock_jm.claim_video_stems_wait.assert_called_once_with("is1") + + +def test_instrumental_selected_fresh_is_left_alone(client): + body, _, mock_ws = _run_is( + client, _is_job("is2", minutes_stale=2, audio_complete=True, stem_ready=True)) + assert body["video_retriggered_jobs"] == [] + mock_ws.trigger_video_worker.assert_not_awaited() + + +def test_instrumental_selected_still_separating_waits(client): + """Separation still running (<60 min): neither re-trigger nor fail.""" + body, mock_jm, mock_ws = _run_is( + client, _is_job("is3", minutes_stale=30, audio_complete=False, stem_ready=False)) + assert body["video_retriggered_jobs"] == [] + assert body["stems_wait_failed_jobs"] == [] + mock_ws.trigger_video_worker.assert_not_awaited() + mock_jm.mark_job_failed.assert_not_called() + + +def test_instrumental_selected_stems_never_ready_fails_loudly(client): + """Separation never finished (>60 min): fail so the job isn't stuck silently.""" + body, mock_jm, mock_ws = _run_is( + client, _is_job("is4", minutes_stale=90, audio_complete=False, stem_ready=False)) + assert body["stems_wait_failed_jobs"] == ["is4"] + mock_jm.mark_job_failed.assert_called_once() + assert mock_jm.mark_job_failed.call_args.kwargs["error_details"]["stage"] == "video_waiting_for_stems" + mock_ws.trigger_video_worker.assert_not_awaited() + + +def test_instrumental_selected_older_than_24h_is_never_revived(client): + body, mock_jm, mock_ws = _run_is( + client, _is_job("is5", minutes_stale=25 * 60, audio_complete=True, stem_ready=True)) + assert body["video_retriggered_jobs"] == [] + assert body["stems_wait_failed_jobs"] == [] + mock_ws.trigger_video_worker.assert_not_awaited() + mock_jm.mark_job_failed.assert_not_called() + diff --git a/backend/tests/test_video_waits_for_stems.py b/backend/tests/test_video_waits_for_stems.py new file mode 100644 index 00000000..8b763ab3 --- /dev/null +++ b/backend/tests/test_video_waits_for_stems.py @@ -0,0 +1,132 @@ +"""Final encode waits for stem separation instead of failing (2026-10-19). + +Review can finish before separation does. The video worker used to fail its +prerequisites ("Missing instrumental: instrumental_clean") 3x and leave the job at +instrumental_selected forever. Now it parks the job, the audio worker resumes it +when the stems land, and an atomic claim on the flag makes the park/resume race +safe. +""" + +from datetime import datetime, timezone +from unittest.mock import AsyncMock, MagicMock, patch + +import pytest + +from backend.models.job import Job, JobStatus +from backend.services.job_health_service import selected_stem_pending +from backend.services.job_manager import JobManager +from backend.workers.video_worker import _park_until_stems_ready + + +def _job(*, status=JobStatus.INSTRUMENTAL_SELECTED, selection="clean", audio_complete=False, + stems=None, waiting=None): + sd = {"instrumental_selection": selection, "audio_complete": audio_complete} + if waiting is not None: + sd["video_waiting_for_stems"] = waiting + return Job( + job_id="j1", artist="A", title="B", status=status, + created_at=datetime.now(timezone.utc), updated_at=datetime.now(timezone.utc), + state_data=sd, file_urls={"stems": stems or {}}, + ) + + +READY = {"instrumental_clean": "gs://b/c.flac", "instrumental_with_backing": "gs://b/w.flac"} + + +class TestSelectedStemPending: + def test_pending_when_stem_missing_and_separation_running(self): + assert selected_stem_pending(_job()) is True + + def test_with_backing_selection_checks_its_own_stem(self): + assert selected_stem_pending(_job(selection="with_backing", + stems={"instrumental_clean": "x"})) is True + + def test_not_pending_once_stem_exists(self): + assert selected_stem_pending(_job(stems=READY)) is False + + def test_not_pending_when_separation_finished_without_stem(self): + """audio_complete but no stem = a real failure; let prerequisites reject it.""" + assert selected_stem_pending(_job(audio_complete=True)) is False + + def test_custom_instrumental_never_waits(self): + assert selected_stem_pending(_job(selection="custom")) is False + + @pytest.mark.parametrize("status", [JobStatus.GENERATING_VIDEO, JobStatus.COMPLETE, JobStatus.FAILED]) + def test_wrong_status_never_parks(self, status): + assert selected_stem_pending(_job(status=status)) is False + + +class TestParkUntilStemsReady: + def _jm(self, refreshed, claim=False): + jm = MagicMock() + jm.get_job.return_value = refreshed + jm.claim_video_stems_wait.return_value = claim + return jm + + def test_not_pending_continues_without_touching_the_flag(self): + jm = self._jm(None) + parked, job = _park_until_stems_ready("j1", _job(stems=READY), jm, MagicMock()) + assert parked is False and job.file_urls["stems"] == READY + jm.park_video_for_stems.assert_not_called() + + def test_still_pending_after_refresh_parks(self): + jm = self._jm(_job()) + parked, _ = _park_until_stems_ready("j1", _job(), jm, MagicMock()) + assert parked is True + jm.park_video_for_stems.assert_called_once_with("j1") + jm.claim_video_stems_wait.assert_not_called() + + def test_stems_landed_while_parking_and_we_win_the_claim_continues(self): + jm = self._jm(_job(stems=READY, audio_complete=True), claim=True) + parked, job = _park_until_stems_ready("j1", _job(), jm, MagicMock()) + assert parked is False + assert job.file_urls["stems"] == READY # continues with the refreshed job + + def test_stems_landed_but_audio_worker_already_resumed_exits(self): + jm = self._jm(_job(stems=READY, audio_complete=True), claim=False) + parked, _ = _park_until_stems_ready("j1", _job(), jm, MagicMock()) + assert parked is True # the audio worker dispatched a fresh run + + +class TestResumeVideoIfWaitingForStems: + def _jm(self, *, claimed, job): + jm = JobManager.__new__(JobManager) + jm.firestore = MagicMock() + jm.firestore.claim_job_state_flag.return_value = claimed + jm.get_job = MagicMock(return_value=job) + return jm + + @pytest.mark.asyncio + async def test_resumes_parked_job(self): + jm = self._jm(claimed=True, job=_job(stems=READY, audio_complete=True)) + ws = MagicMock(trigger_video_worker=AsyncMock(return_value=True)) + with patch("backend.services.worker_service.get_worker_service", return_value=ws): + assert await jm.resume_video_if_waiting_for_stems("j1") is True + jm.firestore.claim_job_state_flag.assert_called_once_with("j1", "video_waiting_for_stems") + ws.trigger_video_worker.assert_awaited_once_with("j1") + + @pytest.mark.asyncio + async def test_no_flag_no_trigger(self): + """Normal ordering (stems before review): nothing parked, nothing to do.""" + jm = self._jm(claimed=False, job=_job()) + ws = MagicMock(trigger_video_worker=AsyncMock(return_value=True)) + with patch("backend.services.worker_service.get_worker_service", return_value=ws): + assert await jm.resume_video_if_waiting_for_stems("j1") is False + ws.trigger_video_worker.assert_not_awaited() + + @pytest.mark.asyncio + async def test_job_moved_on_is_not_resumed(self): + jm = self._jm(claimed=True, job=_job(status=JobStatus.FAILED)) + ws = MagicMock(trigger_video_worker=AsyncMock(return_value=True)) + with patch("backend.services.worker_service.get_worker_service", return_value=ws): + assert await jm.resume_video_if_waiting_for_stems("j1") is False + ws.trigger_video_worker.assert_not_awaited() + + +def test_park_sets_flag_and_user_facing_message(): + jm = JobManager.__new__(JobManager) + jm.update_job = MagicMock() + jm.park_video_for_stems("j1") + updates = jm.update_job.call_args.args[1] + assert updates["state_data.video_waiting_for_stems"] is True + assert "starts automatically" in updates["message"] diff --git a/backend/workers/audio_worker.py b/backend/workers/audio_worker.py index fbf4c47c..3de19c71 100644 --- a/backend/workers/audio_worker.py +++ b/backend/workers/audio_worker.py @@ -412,6 +412,15 @@ async def process_audio_separation(job_id: str) -> bool: # normal run never double-triggers. await job_manager.advance_to_screens_if_ready(job_id) + # If review already finished and the video worker parked the job + # waiting for these stems, resume the final encode now. + try: + if await job_manager.resume_video_if_waiting_for_stems(job_id): + job_log.info("Stems ready: resumed the parked final encode") + except Exception as e: + # recover_stuck_jobs re-triggers a parked job within ~10 min + logger.error(f"[job:{job_id}] Failed to resume parked video worker: {e}") + # Auto-approval second chance: when lyrics/screens finished BEFORE # audio, the job is already parked in AWAITING_REVIEW and the # screens-worker scoring ran without backing analysis. Now that diff --git a/backend/workers/video_worker.py b/backend/workers/video_worker.py index 0ae918d5..453b4f25 100644 --- a/backend/workers/video_worker.py +++ b/backend/workers/video_worker.py @@ -47,7 +47,7 @@ from backend.services.admin_rerender_service import ( ) from backend.services.regenerate_service import active_regenerate from backend.utils.existing_instrumental import resolve_existing_instrumental -from backend.services.job_health_service import validate_worker_can_run +from backend.services.job_health_service import selected_stem_pending, validate_worker_can_run from backend.services.rclone_service import get_rclone_service from backend.services.youtube_service import get_youtube_service from backend.services.youtube_description import translated_language_name @@ -272,6 +272,12 @@ async def generate_video_orchestrated(job_id: str) -> bool: except Exception as e: # noqa: BLE001 - the render stage already reported a missing mix logger.error(f"[job:{job_id}] Input recovery failed in video worker: {e}") + # Review can finish before stem separation does: wait for the stems instead + # of failing prerequisites (which used to strand the job forever). + parked, job = _park_until_stems_ready(job_id, job, job_manager, job_log) + if parked: + return True + # Validate prerequisites if not _validate_prerequisites(job): logger.error(f"[job:{job_id}] Prerequisites not met for video generation") @@ -868,6 +874,12 @@ async def generate_video_legacy(job_id: str) -> bool: logger.error(f"[job:{job_id}] Job not found") return False + # Review can finish before stem separation does: wait for the stems instead + # of failing prerequisites (which used to strand the job forever). + parked, job = _park_until_stems_ready(job_id, job, job_manager, job_log) + if parked: + return True + # Validate prerequisites if not _validate_prerequisites(job): logger.error(f"[job:{job_id}] Prerequisites not met for video generation") @@ -1536,6 +1548,34 @@ def _store_video_processing_metadata( logger.warning(f"Job {job_id}: Failed to store video processing metadata (non-fatal): {e}") +def _park_until_stems_ready(job_id: str, job, job_manager: JobManager, job_log): + """Park the job if its selected stem is still being separated. + + Returns ``(parked, job)``. When ``parked`` is True the worker should exit + successfully: the audio worker re-triggers it once separation completes. + When False, carry on with the returned (possibly refreshed) job. + + Race with the audio worker: we set the flag, then re-read. If separation + finished in between, exactly one of us wins the atomic claim on the flag — + we continue in-process, or the audio worker has already dispatched a fresh + run and we exit. + """ + if not selected_stem_pending(job): + return False, job + job_manager.park_video_for_stems(job_id) + refreshed = job_manager.get_job(job_id) or job + if selected_stem_pending(refreshed): + msg = "Stem separation still running — final encode parked; it resumes automatically when stems are ready" + logger.info(f"[job:{job_id}] {msg}") + job_log.info(msg) + return True, refreshed + if job_manager.claim_video_stems_wait(job_id): + logger.info(f"[job:{job_id}] Stems landed while parking — continuing") + return False, refreshed + logger.info(f"[job:{job_id}] Stems landed while parking; audio worker already resumed the encode") + return True, refreshed + + def _validate_prerequisites(job) -> bool: """Validate that all prerequisites are met for video generation.""" # Validate job status is appropriate for video worker diff --git a/docs/TROUBLESHOOTING.md b/docs/TROUBLESHOOTING.md index 96dde4e2..46240237 100644 --- a/docs/TROUBLESHOOTING.md +++ b/docs/TROUBLESHOOTING.md @@ -276,6 +276,23 @@ curl -X POST "https://api.nomadkaraoke.com/api/internal/workers/screens" \ -d '{"job_id": ""}' ``` +## Job at `instrumental_selected` "Waiting for audio separation to finish" + +This is normal when review finishes before stem separation does. The video worker +parks the job (`state_data.video_waiting_for_stems=true`) and exits cleanly. The +audio worker resumes the final encode as soon as the stems land +(`resume_video_if_waiting_for_stems`). Backstop: `recover_stuck_jobs` re-triggers +the video worker once a job has sat in `instrumental_selected` for 10+ min with its +stem present. If separation never finishes, it fails the job after 60 min +(`error_details.stage = "video_waiting_for_stems"`), so it never stays stuck silently. + +Before 2026-10-19 (v0.278.11) the video worker failed its prerequisites instead +("Missing instrumental: instrumental_clean") and the job stayed at "Starting final +encode" forever. That got much more likely when the GPU image picked up +audio-separator 0.47.0 (karaoke models 3–5× slower on L4). The version is now pinned +in `backend/Dockerfile.gpu-base` (`AUDIO_SEPARATOR_VERSION`). Time a real job before +bumping it. + ## Job stuck at `encoding` status **Cause:** A Cloud Run deployment killed the poller mid-encoding. The GCE worker finished but nobody received the result. The health service will flag this as `encoding_stuck` after 50 minutes. diff --git a/pyproject.toml b/pyproject.toml index 1124af99..a6eca6d6 100644 --- a/pyproject.toml +++ b/pyproject.toml @@ -1,6 +1,6 @@ [tool.poetry] name = "karaoke-gen" -version = "0.278.10" +version = "0.278.11" description = "Generate karaoke videos with synchronized lyrics. Handles the entire process from downloading audio and lyrics to creating the final video with title screens." authors = ["Andrew Beveridge "] license = "MIT"