fix: final encode waits for stem separation instead of stranding the job; pin audio-separator 0.44.2 (v0.278.11) (#1191)

## Problem
The daily E2E Stage 2 failed on 2026-10-17 (run #216) and 2026-10-18
(#217). When a customer finishes review **before** stem separation does,
`video-encoding-job` fails its prerequisites (`Missing instrumental:
instrumental_clean`) 3× within seconds. The job then sits at
`instrumental_selected` ("Starting final encode") forever: it's never
marked failed, nothing alerts, and separation finishing doesn't
re-trigger the encode.

The race became common when the GPU image's unpinned `pip install
"audio-separator[gpu]"` picked up **0.47.0** on the 10-17 rebuild. On
L4, the mel-band RoFormer **karaoke** models run 3–5× slower than on
0.44.2, so total separation went from ~5 to ~10–16 min:

| model (~3-min track) | 0.44.2 | 0.47.0 |
|---|---|---|
| karaoke_aufr33_viperx | ~25s | 75s |
| karaoke_gabox_v2 | ~26s | 149s |
| karaoke_becruily | ~27s | 157s |
| instrumental fv7z / resurrection | ~50–60s | ~62s |

The model YAMLs still say `num_overlap: 8, batch_size: 1`, so this isn't
the #299 YAML-defaults change; the cause is somewhere in 0.45–0.47's
RoFormer inference changes. That root cause is tracked separately.

## Fix
- **Pin** `audio-separator[gpu]==0.44.2` in
`backend/Dockerfile.gpu-base` (`ARG AUDIO_SEPARATOR_VERSION`). Changing
this file triggers the GPU base image rebuild in CI.
- **Video worker parks instead of failing.** When the selected stem is
missing *and* separation is still running (`audio_complete` false, valid
status), it sets `state_data.video_waiting_for_stems`, puts a
user-visible message ("Waiting for audio separation to finish; the final
video starts automatically") and exits 0, so Cloud Run doesn't retry. It
re-checks after parking, and an atomic Firestore test-and-clear
(`claim_job_state_flag`) means exactly one of the video worker and the
audio worker proceeds if the stems land mid-park.
- **Audio worker resumes it.** After `mark_audio_complete`,
`resume_video_if_waiting_for_stems` re-triggers a parked encode. The
fake-workers pipeline does the same, so the local stack matches prod.
- **Backstop in `recover_stuck_jobs`.** `instrumental_selected` for more
than 10 min with the stem present → re-trigger the video worker (covers
a lost resume or lost dispatch). More than 60 min with separation never
finishing → `mark_job_failed` (`stage=video_waiting_for_stems`), so the
job fails loudly. Jobs older than 24h are never revived. (No jobs are in
`instrumental_selected` in prod right now.)

## Testing
- New `backend/tests/test_video_waits_for_stems.py` (16 tests): pending
detection (including that a wrong status never parks), park / continue /
claim-race outcomes, and resume (flag claimed / no flag / job moved on).
- `test_recover_stuck_jobs_repark.py`: 5 new sweep tests (re-trigger,
fresh, still separating, never-ready → fail, >24h ignored).
- Full `backend/tests` (excluding emulator): 6455 passed.
- Local CodeRabbit review: 0 findings.

## Verify after deploy
- The GPU job logs `Separator version 0.44.2`, and karaoke-model
separation is back to ~20–30s.
- The next daily E2E Stage 2 passes.

@coderabbitai ignore

🤖 Generated with [Claude Code](https://claude.com/claude-code)

https://claude.ai/code/session_019pmG2XAgtUguEJWLFpkKHN

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
Andrew Beveridge ☄️andClaude Opus 5.5 authored and GitHub committed 2026-10-19 02:21:52 -04:00
1 parent 70b2f4279a
commit 0d0605d7b5
12 files changed
+405 -6

No files matched your search

+9 -1
View File
@@ -102,9 +102,17 @@ RUN --mount=type=cache,target=/root/.cache/pip \
# have NVIDIA driver 570 which only supports up to CUDA 12.8. PyTorch cu130
# fails with "NVIDIA driver is too old". PyTorch cu126 works with 12.8 drivers.
# Installing torch first means audio-separator[gpu] sees it as already satisfied.
#
# PINNED to an exact version: an unpinned install silently picked up 0.47.0 on the
# 2026-10-17 rebuild, and on L4 its mel-band RoFormer karaoke models run 3-5x slower
# than 0.44.2 (gabox_v2 ~26s -> 149s, becruily ~27s -> 157s for a ~3-min track;
# YAML overlap/batch are unchanged, so it is in the 0.45-0.47 inference changes).
# Total separation went from ~5 to ~10-16 min and the daily E2E started failing.
# Only bump this after timing a real job on the new version (see BACKLOG).
ARG AUDIO_SEPARATOR_VERSION=0.44.2
RUN --mount=type=cache,target=/root/.cache/pip \
pip install torch==2.6.0+cu126 torchvision==0.21.0+cu126 --index-url https://download.pytorch.org/whl/cu126 && \
pip install "audio-separator[gpu]"
pip install "audio-separator[gpu]==${AUDIO_SEPARATOR_VERSION}"
# Now copy actual source code (AFTER deps installed, so code changes don't
# invalidate the expensive pip install layer in registry cache)
+46
View File
@@ -1362,6 +1362,47 @@ def _recover_stuck_jobs_sync(http_request: Request, loop: asyncio.AbstractEventL
)
stems_restore_failed.append(job_id)
# --- INSTRUMENTAL_SELECTED with no encode running ---
# The video worker moves to GENERATING_VIDEO as soon as it starts, so a job
# still here after 10 min has no encode underway. Either it's parked waiting
# for stem separation (the audio worker resumes it; this catches a lost
# resume) or its video dispatch was lost. Re-trigger once the stems exist;
# if separation never finishes, fail loudly so the job doesn't sit silently
# (the 2026-10-17/18 stuck "Starting final encode" jobs). Bounded to 24h so
# ancient abandoned jobs are never revived.
from backend.services.job_health_service import selected_stem_pending
video_retriggered = []
stems_wait_failed = []
is_query = jobs_ref.where(
filter=FieldFilter("status", "==", JobStatus.INSTRUMENTAL_SELECTED.value)
).limit(SCREENS_RECOVERY_SCAN_LIMIT).stream()
for doc in is_query:
if len(video_retriggered) >= RENDER_RETRIGGERS_PER_TICK:
break
job_id = (doc.to_dict() or {}).get("job_id", doc.id)
job = job_manager.get_job(job_id)
age = _job_updated_age_seconds(job) if job else None
if age is None or not (INSTRUMENTAL_SELECTED_STALL_SECONDS < age < INSTRUMENTAL_SELECTED_MAX_AGE_SECONDS):
continue
if selected_stem_pending(job):
if age > STEMS_WAIT_FAIL_SECONDS:
logger.error(f"[job:{job_id}] Final encode waited >60 min for stem separation that never finished — failing")
job_manager.mark_job_failed(
job_id=job_id,
error_message="Audio separation didn't finish, so the final video couldn't be made. Retry the job to try again.",
error_details={"stage": "video_waiting_for_stems", "reason": "stems_never_ready"},
)
stems_wait_failed.append(job_id)
continue
logger.warning(f"[job:{job_id}] INSTRUMENTAL_SELECTED stalled >10 min with stems ready — re-triggering video worker")
try:
# Clear a parked flag first so a late audio-worker resume can't double-trigger.
job_manager.claim_video_stems_wait(job_id)
if run_on_loop(loop, worker_service.trigger_video_worker(job_id)):
video_retriggered.append(job_id)
except Exception as e: # noqa: BLE001
logger.warning(f"[job:{job_id}] video re-trigger failed: {e}")
logger.info(
f"RECOVER_STUCK_JOBS complete: recovered={len(recovered)} "
f"download_parked={len(download_parked)} download_retried={len(download_retried)} "
@@ -1392,6 +1433,8 @@ def _recover_stuck_jobs_sync(http_request: Request, loop: asyncio.AbstractEventL
"screens_retriggered_count": len(screens_retriggered),
"stems_restore_failed_jobs": stems_restore_failed,
"regen_screens_retriggered_jobs": regen_screens_retriggered,
"video_retriggered_jobs": video_retriggered,
"stems_wait_failed_jobs": stems_wait_failed,
}
@@ -1400,6 +1443,9 @@ RENDER_RETRIGGERS_PER_TICK = 10 # bound the blast radius per 5-min tick
PREP_SCREENS_STALL_SECONDS = 10 * 60 # screens normally start within seconds of lyrics done
SCREENS_RETRIGGERS_PER_TICK = 10 # bound the blast radius per 5-min tick
SCREENS_RECOVERY_SCAN_LIMIT = 500 # cap DOWNLOADING docs scanned per tick (transient state)
INSTRUMENTAL_SELECTED_STALL_SECONDS = 10 * 60 # video worker leaves this state within seconds
STEMS_WAIT_FAIL_SECONDS = 60 * 60 # separation normally lands within ~15 min of review
INSTRUMENTAL_SELECTED_MAX_AGE_SECONDS = 24 * 3600 # never revive ancient abandoned jobs
def _job_updated_age_seconds(job):
@@ -134,6 +134,8 @@ async def run_audio(job_id: str) -> bool:
job_manager.mark_audio_complete(job_id)
await job_manager.advance_to_screens_if_ready(job_id)
# Same as the real audio worker: resume a final encode parked on these stems.
await job_manager.resume_video_if_waiting_for_stems(job_id)
return True
+21
View File
@@ -131,6 +131,27 @@ class FirestoreService:
logger.error(f"Error updating job {job_id}: {e}")
raise
def claim_job_state_flag(self, job_id: str, key: str) -> bool:
"""Atomically test-and-clear ``state_data.<key>``.
Returns True for exactly one caller while the flag is truthy (it is set
to False in the same transaction), so two workers racing to act on the
same flag can't both proceed.
"""
doc_ref = self.db.collection(self.collection).document(job_id)
@firestore.transactional
def _claim(transaction, ref) -> bool:
snapshot = ref.get(transaction=transaction)
if not snapshot.exists:
return False
if not ((snapshot.to_dict() or {}).get("state_data") or {}).get(key):
return False
transaction.update(ref, {f"state_data.{key}": False, "updated_at": datetime.utcnow()})
return True
return _claim(self.db.transaction(), doc_ref)
def update_job_status(
self,
job_id: str,
+18
View File
@@ -287,3 +287,21 @@ def validate_worker_can_run(worker_name: str, job: Job) -> Optional[str]:
)
return None
def selected_stem_pending(job) -> bool:
"""True when the selected instrumental stem hasn't been produced yet but
stem separation is still running, i.e. the job should wait, not fail.
Review can finish before separation does (audio separation is decoupled from
the lyrics review path), so this is a normal ordering, not an error.
"""
if validate_worker_can_run("video_worker", job):
return False # wrong status (duplicate/late trigger): never park, let the normal checks reject it
selection = (job.state_data or {}).get('instrumental_selection')
if selection not in ('clean', 'with_backing'):
return False # 'custom' uses an uploaded file / mute-region stem
stem_key = 'instrumental_clean' if selection == 'clean' else 'instrumental_with_backing'
if (job.file_urls or {}).get('stems', {}).get(stem_key):
return False
return not (job.state_data or {}).get('audio_complete', False)
+35
View File
@@ -1305,6 +1305,41 @@ class JobManager:
self.update_state_data(job_id, 'audio_complete', True)
logger.info(f"Job {job_id}: Audio processing complete (separation finished)")
# Set by the video worker when review finished before stem separation did
# (the selected instrumental stem doesn't exist yet). The audio worker
# resumes the encode when separation completes; recover_stuck_jobs is the
# backstop. Before this, the video worker just failed its prerequisites 3x
# and the job sat at instrumental_selected forever (2026-10-17/18).
VIDEO_WAITING_FOR_STEMS_KEY = 'video_waiting_for_stems'
def park_video_for_stems(self, job_id: str) -> None:
"""Record that the final encode is waiting for stem separation."""
self.update_job(job_id, {
f'state_data.{self.VIDEO_WAITING_FOR_STEMS_KEY}': True,
'message': 'Waiting for audio separation to finish; the final video starts automatically',
})
def claim_video_stems_wait(self, job_id: str) -> bool:
"""Atomically clear the waiting flag; True for exactly one claimant."""
return self.firestore.claim_job_state_flag(job_id, self.VIDEO_WAITING_FOR_STEMS_KEY)
async def resume_video_if_waiting_for_stems(self, job_id: str) -> bool:
"""Re-trigger the video worker for a job parked waiting on stems.
Called by the audio worker right after ``mark_audio_complete``. No-op
unless the video worker parked the job (flag set) and it's still in a
state the video worker accepts. Returns True iff a dispatch was issued.
"""
if not self.claim_video_stems_wait(job_id):
return False
job = self.get_job(job_id)
if not job or job.status not in (JobStatus.INSTRUMENTAL_SELECTED, JobStatus.REVIEW_COMPLETE):
logger.info(f"Job {job_id}: stems ready but job is {getattr(job, 'status', None)}; not resuming video")
return False
from backend.services.worker_service import get_worker_service
logger.info(f"Job {job_id}: stems ready — resuming parked video worker")
return await get_worker_service().trigger_video_worker(job_id)
def mark_lyrics_complete(self, job_id: str) -> None:
"""
Mark lyrics processing as complete.
@@ -56,7 +56,7 @@ def _job(job_id, status, *, minutes_stale=0, source_name=None, download_retry=No
def _mock_jm(*, downloading=None, pending=None, rendering=None, review_complete=None,
downloading_prep=None, jobs=None):
downloading_prep=None, instrumental_selected=None, jobs=None):
"""Wire a JobManager mock for the endpoint's 4 status queries.
downloading/rendering/review_complete use .where().stream(); pending uses
@@ -69,11 +69,13 @@ def _mock_jm(*, downloading=None, pending=None, rendering=None, review_complete=
where.stream.side_effect = [
iter(downloading or []), iter(rendering or []), iter(review_complete or []),
]
# Three .limit().stream() calls in order: download_pending_retry, then downloading
# Four .limit().stream() calls in order: download_pending_retry, then downloading
# (lost-screens-trigger sweep, bounded by SCREENS_RECOVERY_SCAN_LIMIT), then
# lyrics_complete (stalled storage-retention stems restore)
# lyrics_complete (stalled storage-retention stems restore), then
# instrumental_selected (final encode parked on stems / lost video trigger)
where.limit.return_value.stream.side_effect = [
iter(pending or []), iter(downloading_prep or []), iter([]),
iter(instrumental_selected or []),
]
mock_jm.get_job.side_effect = lambda jid: (jobs or {}).get(jid)
mock_jm.transition_to_state.return_value = True
@@ -284,3 +286,72 @@ def test_fresh_review_complete_is_left_alone(client):
assert resp.status_code == 200
assert resp.json()["render_retriggered_jobs"] == []
mock_ws.trigger_render_video_worker.assert_not_awaited()
def _is_job(job_id, *, minutes_stale, audio_complete, stem_ready, selection="clean"):
job = _job(job_id, JobStatus.INSTRUMENTAL_SELECTED, minutes_stale=minutes_stale)
job.state_data.update({"instrumental_selection": selection, "audio_complete": audio_complete})
if stem_ready:
job.file_urls = {"stems": {"instrumental_clean": "gs://b/clean.flac",
"instrumental_with_backing": "gs://b/backing.flac"}}
return job
def _run_is(client, job):
mock_jm = _mock_jm(
instrumental_selected=[_doc(job.job_id, JobStatus.INSTRUMENTAL_SELECTED)],
jobs={job.job_id: job},
)
mock_ws = MagicMock()
mock_ws.trigger_video_worker = AsyncMock(return_value=True)
with patch("backend.api.routes.internal.JobManager", return_value=mock_jm), \
patch("backend.services.worker_service.get_worker_service", return_value=mock_ws):
resp = client.post("/api/internal/recover-stuck-jobs")
assert resp.status_code == 200
return resp.json(), mock_jm, mock_ws
def test_instrumental_selected_with_stems_ready_retriggers_video(client):
"""Parked/lost video trigger: stems landed but no encode started -> re-trigger."""
body, mock_jm, mock_ws = _run_is(
client, _is_job("is1", minutes_stale=15, audio_complete=True, stem_ready=True))
assert body["video_retriggered_jobs"] == ["is1"]
mock_ws.trigger_video_worker.assert_awaited_once_with("is1")
mock_jm.claim_video_stems_wait.assert_called_once_with("is1")
def test_instrumental_selected_fresh_is_left_alone(client):
body, _, mock_ws = _run_is(
client, _is_job("is2", minutes_stale=2, audio_complete=True, stem_ready=True))
assert body["video_retriggered_jobs"] == []
mock_ws.trigger_video_worker.assert_not_awaited()
def test_instrumental_selected_still_separating_waits(client):
"""Separation still running (<60 min): neither re-trigger nor fail."""
body, mock_jm, mock_ws = _run_is(
client, _is_job("is3", minutes_stale=30, audio_complete=False, stem_ready=False))
assert body["video_retriggered_jobs"] == []
assert body["stems_wait_failed_jobs"] == []
mock_ws.trigger_video_worker.assert_not_awaited()
mock_jm.mark_job_failed.assert_not_called()
def test_instrumental_selected_stems_never_ready_fails_loudly(client):
"""Separation never finished (>60 min): fail so the job isn't stuck silently."""
body, mock_jm, mock_ws = _run_is(
client, _is_job("is4", minutes_stale=90, audio_complete=False, stem_ready=False))
assert body["stems_wait_failed_jobs"] == ["is4"]
mock_jm.mark_job_failed.assert_called_once()
assert mock_jm.mark_job_failed.call_args.kwargs["error_details"]["stage"] == "video_waiting_for_stems"
mock_ws.trigger_video_worker.assert_not_awaited()
def test_instrumental_selected_older_than_24h_is_never_revived(client):
body, mock_jm, mock_ws = _run_is(
client, _is_job("is5", minutes_stale=25 * 60, audio_complete=True, stem_ready=True))
assert body["video_retriggered_jobs"] == []
assert body["stems_wait_failed_jobs"] == []
mock_ws.trigger_video_worker.assert_not_awaited()
mock_jm.mark_job_failed.assert_not_called()
+132
View File
@@ -0,0 +1,132 @@
"""Final encode waits for stem separation instead of failing (2026-10-19).
Review can finish before separation does. The video worker used to fail its
prerequisites ("Missing instrumental: instrumental_clean") 3x and leave the job at
instrumental_selected forever. Now it parks the job, the audio worker resumes it
when the stems land, and an atomic claim on the flag makes the park/resume race
safe.
"""
from datetime import datetime, timezone
from unittest.mock import AsyncMock, MagicMock, patch
import pytest
from backend.models.job import Job, JobStatus
from backend.services.job_health_service import selected_stem_pending
from backend.services.job_manager import JobManager
from backend.workers.video_worker import _park_until_stems_ready
def _job(*, status=JobStatus.INSTRUMENTAL_SELECTED, selection="clean", audio_complete=False,
stems=None, waiting=None):
sd = {"instrumental_selection": selection, "audio_complete": audio_complete}
if waiting is not None:
sd["video_waiting_for_stems"] = waiting
return Job(
job_id="j1", artist="A", title="B", status=status,
created_at=datetime.now(timezone.utc), updated_at=datetime.now(timezone.utc),
state_data=sd, file_urls={"stems": stems or {}},
)
READY = {"instrumental_clean": "gs://b/c.flac", "instrumental_with_backing": "gs://b/w.flac"}
class TestSelectedStemPending:
def test_pending_when_stem_missing_and_separation_running(self):
assert selected_stem_pending(_job()) is True
def test_with_backing_selection_checks_its_own_stem(self):
assert selected_stem_pending(_job(selection="with_backing",
stems={"instrumental_clean": "x"})) is True
def test_not_pending_once_stem_exists(self):
assert selected_stem_pending(_job(stems=READY)) is False
def test_not_pending_when_separation_finished_without_stem(self):
"""audio_complete but no stem = a real failure; let prerequisites reject it."""
assert selected_stem_pending(_job(audio_complete=True)) is False
def test_custom_instrumental_never_waits(self):
assert selected_stem_pending(_job(selection="custom")) is False
@pytest.mark.parametrize("status", [JobStatus.GENERATING_VIDEO, JobStatus.COMPLETE, JobStatus.FAILED])
def test_wrong_status_never_parks(self, status):
assert selected_stem_pending(_job(status=status)) is False
class TestParkUntilStemsReady:
def _jm(self, refreshed, claim=False):
jm = MagicMock()
jm.get_job.return_value = refreshed
jm.claim_video_stems_wait.return_value = claim
return jm
def test_not_pending_continues_without_touching_the_flag(self):
jm = self._jm(None)
parked, job = _park_until_stems_ready("j1", _job(stems=READY), jm, MagicMock())
assert parked is False and job.file_urls["stems"] == READY
jm.park_video_for_stems.assert_not_called()
def test_still_pending_after_refresh_parks(self):
jm = self._jm(_job())
parked, _ = _park_until_stems_ready("j1", _job(), jm, MagicMock())
assert parked is True
jm.park_video_for_stems.assert_called_once_with("j1")
jm.claim_video_stems_wait.assert_not_called()
def test_stems_landed_while_parking_and_we_win_the_claim_continues(self):
jm = self._jm(_job(stems=READY, audio_complete=True), claim=True)
parked, job = _park_until_stems_ready("j1", _job(), jm, MagicMock())
assert parked is False
assert job.file_urls["stems"] == READY # continues with the refreshed job
def test_stems_landed_but_audio_worker_already_resumed_exits(self):
jm = self._jm(_job(stems=READY, audio_complete=True), claim=False)
parked, _ = _park_until_stems_ready("j1", _job(), jm, MagicMock())
assert parked is True # the audio worker dispatched a fresh run
class TestResumeVideoIfWaitingForStems:
def _jm(self, *, claimed, job):
jm = JobManager.__new__(JobManager)
jm.firestore = MagicMock()
jm.firestore.claim_job_state_flag.return_value = claimed
jm.get_job = MagicMock(return_value=job)
return jm
@pytest.mark.asyncio
async def test_resumes_parked_job(self):
jm = self._jm(claimed=True, job=_job(stems=READY, audio_complete=True))
ws = MagicMock(trigger_video_worker=AsyncMock(return_value=True))
with patch("backend.services.worker_service.get_worker_service", return_value=ws):
assert await jm.resume_video_if_waiting_for_stems("j1") is True
jm.firestore.claim_job_state_flag.assert_called_once_with("j1", "video_waiting_for_stems")
ws.trigger_video_worker.assert_awaited_once_with("j1")
@pytest.mark.asyncio
async def test_no_flag_no_trigger(self):
"""Normal ordering (stems before review): nothing parked, nothing to do."""
jm = self._jm(claimed=False, job=_job())
ws = MagicMock(trigger_video_worker=AsyncMock(return_value=True))
with patch("backend.services.worker_service.get_worker_service", return_value=ws):
assert await jm.resume_video_if_waiting_for_stems("j1") is False
ws.trigger_video_worker.assert_not_awaited()
@pytest.mark.asyncio
async def test_job_moved_on_is_not_resumed(self):
jm = self._jm(claimed=True, job=_job(status=JobStatus.FAILED))
ws = MagicMock(trigger_video_worker=AsyncMock(return_value=True))
with patch("backend.services.worker_service.get_worker_service", return_value=ws):
assert await jm.resume_video_if_waiting_for_stems("j1") is False
ws.trigger_video_worker.assert_not_awaited()
def test_park_sets_flag_and_user_facing_message():
jm = JobManager.__new__(JobManager)
jm.update_job = MagicMock()
jm.park_video_for_stems("j1")
updates = jm.update_job.call_args.args[1]
assert updates["state_data.video_waiting_for_stems"] is True
assert "starts automatically" in updates["message"]
+9
View File
@@ -412,6 +412,15 @@ async def process_audio_separation(job_id: str) -> bool:
# normal run never double-triggers.
await job_manager.advance_to_screens_if_ready(job_id)
# If review already finished and the video worker parked the job
# waiting for these stems, resume the final encode now.
try:
if await job_manager.resume_video_if_waiting_for_stems(job_id):
job_log.info("Stems ready: resumed the parked final encode")
except Exception as e:
# recover_stuck_jobs re-triggers a parked job within ~10 min
logger.error(f"[job:{job_id}] Failed to resume parked video worker: {e}")
# Auto-approval second chance: when lyrics/screens finished BEFORE
# audio, the job is already parked in AWAITING_REVIEW and the
# screens-worker scoring ran without backing analysis. Now that
+41 -1
View File
@@ -47,7 +47,7 @@ from backend.services.admin_rerender_service import (
)
from backend.services.regenerate_service import active_regenerate
from backend.utils.existing_instrumental import resolve_existing_instrumental
from backend.services.job_health_service import validate_worker_can_run
from backend.services.job_health_service import selected_stem_pending, validate_worker_can_run
from backend.services.rclone_service import get_rclone_service
from backend.services.youtube_service import get_youtube_service
from backend.services.youtube_description import translated_language_name
@@ -272,6 +272,12 @@ async def generate_video_orchestrated(job_id: str) -> bool:
except Exception as e: # noqa: BLE001 - the render stage already reported a missing mix
logger.error(f"[job:{job_id}] Input recovery failed in video worker: {e}")
# Review can finish before stem separation does: wait for the stems instead
# of failing prerequisites (which used to strand the job forever).
parked, job = _park_until_stems_ready(job_id, job, job_manager, job_log)
if parked:
return True
# Validate prerequisites
if not _validate_prerequisites(job):
logger.error(f"[job:{job_id}] Prerequisites not met for video generation")
@@ -868,6 +874,12 @@ async def generate_video_legacy(job_id: str) -> bool:
logger.error(f"[job:{job_id}] Job not found")
return False
# Review can finish before stem separation does: wait for the stems instead
# of failing prerequisites (which used to strand the job forever).
parked, job = _park_until_stems_ready(job_id, job, job_manager, job_log)
if parked:
return True
# Validate prerequisites
if not _validate_prerequisites(job):
logger.error(f"[job:{job_id}] Prerequisites not met for video generation")
@@ -1536,6 +1548,34 @@ def _store_video_processing_metadata(
logger.warning(f"Job {job_id}: Failed to store video processing metadata (non-fatal): {e}")
def _park_until_stems_ready(job_id: str, job, job_manager: JobManager, job_log):
"""Park the job if its selected stem is still being separated.
Returns ``(parked, job)``. When ``parked`` is True the worker should exit
successfully: the audio worker re-triggers it once separation completes.
When False, carry on with the returned (possibly refreshed) job.
Race with the audio worker: we set the flag, then re-read. If separation
finished in between, exactly one of us wins the atomic claim on the flag —
we continue in-process, or the audio worker has already dispatched a fresh
run and we exit.
"""
if not selected_stem_pending(job):
return False, job
job_manager.park_video_for_stems(job_id)
refreshed = job_manager.get_job(job_id) or job
if selected_stem_pending(refreshed):
msg = "Stem separation still running — final encode parked; it resumes automatically when stems are ready"
logger.info(f"[job:{job_id}] {msg}")
job_log.info(msg)
return True, refreshed
if job_manager.claim_video_stems_wait(job_id):
logger.info(f"[job:{job_id}] Stems landed while parking — continuing")
return False, refreshed
logger.info(f"[job:{job_id}] Stems landed while parking; audio worker already resumed the encode")
return True, refreshed
def _validate_prerequisites(job) -> bool:
"""Validate that all prerequisites are met for video generation."""
# Validate job status is appropriate for video worker
+17
View File
@@ -276,6 +276,23 @@ curl -X POST "https://api.nomadkaraoke.com/api/internal/workers/screens" \
-d '{"job_id": "<job_id>"}'
```
## Job at `instrumental_selected` "Waiting for audio separation to finish"
This is normal when review finishes before stem separation does. The video worker
parks the job (`state_data.video_waiting_for_stems=true`) and exits cleanly. The
audio worker resumes the final encode as soon as the stems land
(`resume_video_if_waiting_for_stems`). Backstop: `recover_stuck_jobs` re-triggers
the video worker once a job has sat in `instrumental_selected` for 10+ min with its
stem present. If separation never finishes, it fails the job after 60 min
(`error_details.stage = "video_waiting_for_stems"`), so it never stays stuck silently.
Before 2026-10-19 (v0.278.11) the video worker failed its prerequisites instead
("Missing instrumental: instrumental_clean") and the job stayed at "Starting final
encode" forever. That got much more likely when the GPU image picked up
audio-separator 0.47.0 (karaoke models 3–5× slower on L4). The version is now pinned
in `backend/Dockerfile.gpu-base` (`AUDIO_SEPARATOR_VERSION`). Time a real job before
bumping it.
## Job stuck at `encoding` status
**Cause:** A Cloud Run deployment killed the poller mid-encoding. The GCE worker finished but nobody received the result. The health service will flag this as `encoding_stuck` after 50 minutes.
+1 -1
View File
@@ -1,6 +1,6 @@
[tool.poetry]
name = "karaoke-gen"
version = "0.278.10"
version = "0.278.11"
description = "Generate karaoke videos with synchronized lyrics. Handles the entire process from downloading audio and lyrics to creating the final video with title screens."
authors = ["Andrew Beveridge <andrew@beveridge.uk>"]
license = "MIT"