mirror of
https://github.com/nomadkaraoke/karaoke-gen.git
synced 2026-11-20 13:07:57 +09:00
fix: final encode waits for stem separation instead of stranding the job; pin audio-separator 0.44.2 (v0.278.11) (#1191)
## Problem The daily E2E Stage 2 failed on 2026-10-17 (run #216) and 2026-10-18 (#217). When a customer finishes review **before** stem separation does, `video-encoding-job` fails its prerequisites (`Missing instrumental: instrumental_clean`) 3× within seconds. The job then sits at `instrumental_selected` ("Starting final encode") forever: it's never marked failed, nothing alerts, and separation finishing doesn't re-trigger the encode. The race became common when the GPU image's unpinned `pip install "audio-separator[gpu]"` picked up **0.47.0** on the 10-17 rebuild. On L4, the mel-band RoFormer **karaoke** models run 3–5× slower than on 0.44.2, so total separation went from ~5 to ~10–16 min: | model (~3-min track) | 0.44.2 | 0.47.0 | |---|---|---| | karaoke_aufr33_viperx | ~25s | 75s | | karaoke_gabox_v2 | ~26s | 149s | | karaoke_becruily | ~27s | 157s | | instrumental fv7z / resurrection | ~50–60s | ~62s | The model YAMLs still say `num_overlap: 8, batch_size: 1`, so this isn't the #299 YAML-defaults change; the cause is somewhere in 0.45–0.47's RoFormer inference changes. That root cause is tracked separately. ## Fix - **Pin** `audio-separator[gpu]==0.44.2` in `backend/Dockerfile.gpu-base` (`ARG AUDIO_SEPARATOR_VERSION`). Changing this file triggers the GPU base image rebuild in CI. - **Video worker parks instead of failing.** When the selected stem is missing *and* separation is still running (`audio_complete` false, valid status), it sets `state_data.video_waiting_for_stems`, puts a user-visible message ("Waiting for audio separation to finish; the final video starts automatically") and exits 0, so Cloud Run doesn't retry. It re-checks after parking, and an atomic Firestore test-and-clear (`claim_job_state_flag`) means exactly one of the video worker and the audio worker proceeds if the stems land mid-park. - **Audio worker resumes it.** After `mark_audio_complete`, `resume_video_if_waiting_for_stems` re-triggers a parked encode. The fake-workers pipeline does the same, so the local stack matches prod. - **Backstop in `recover_stuck_jobs`.** `instrumental_selected` for more than 10 min with the stem present → re-trigger the video worker (covers a lost resume or lost dispatch). More than 60 min with separation never finishing → `mark_job_failed` (`stage=video_waiting_for_stems`), so the job fails loudly. Jobs older than 24h are never revived. (No jobs are in `instrumental_selected` in prod right now.) ## Testing - New `backend/tests/test_video_waits_for_stems.py` (16 tests): pending detection (including that a wrong status never parks), park / continue / claim-race outcomes, and resume (flag claimed / no flag / job moved on). - `test_recover_stuck_jobs_repark.py`: 5 new sweep tests (re-trigger, fresh, still separating, never-ready → fail, >24h ignored). - Full `backend/tests` (excluding emulator): 6455 passed. - Local CodeRabbit review: 0 findings. ## Verify after deploy - The GPU job logs `Separator version 0.44.2`, and karaoke-model separation is back to ~20–30s. - The next daily E2E Stage 2 passes. @coderabbitai ignore 🤖 Generated with [Claude Code](https://claude.com/claude-code) https://claude.ai/code/session_019pmG2XAgtUguEJWLFpkKHN Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
1 parent
70b2f4279a
commit
0d0605d7b5
12 files changed
+405
-6
No files matched your search
@@ -102,9 +102,17 @@ RUN --mount=type=cache,target=/root/.cache/pip \
|
||||
# have NVIDIA driver 570 which only supports up to CUDA 12.8. PyTorch cu130
|
||||
# fails with "NVIDIA driver is too old". PyTorch cu126 works with 12.8 drivers.
|
||||
# Installing torch first means audio-separator[gpu] sees it as already satisfied.
|
||||
#
|
||||
# PINNED to an exact version: an unpinned install silently picked up 0.47.0 on the
|
||||
# 2026-10-17 rebuild, and on L4 its mel-band RoFormer karaoke models run 3-5x slower
|
||||
# than 0.44.2 (gabox_v2 ~26s -> 149s, becruily ~27s -> 157s for a ~3-min track;
|
||||
# YAML overlap/batch are unchanged, so it is in the 0.45-0.47 inference changes).
|
||||
# Total separation went from ~5 to ~10-16 min and the daily E2E started failing.
|
||||
# Only bump this after timing a real job on the new version (see BACKLOG).
|
||||
ARG AUDIO_SEPARATOR_VERSION=0.44.2
|
||||
RUN --mount=type=cache,target=/root/.cache/pip \
|
||||
pip install torch==2.6.0+cu126 torchvision==0.21.0+cu126 --index-url https://download.pytorch.org/whl/cu126 && \
|
||||
pip install "audio-separator[gpu]"
|
||||
pip install "audio-separator[gpu]==${AUDIO_SEPARATOR_VERSION}"
|
||||
|
||||
# Now copy actual source code (AFTER deps installed, so code changes don't
|
||||
# invalidate the expensive pip install layer in registry cache)
|
||||
|
||||
@@ -1362,6 +1362,47 @@ def _recover_stuck_jobs_sync(http_request: Request, loop: asyncio.AbstractEventL
|
||||
)
|
||||
stems_restore_failed.append(job_id)
|
||||
|
||||
# --- INSTRUMENTAL_SELECTED with no encode running ---
|
||||
# The video worker moves to GENERATING_VIDEO as soon as it starts, so a job
|
||||
# still here after 10 min has no encode underway. Either it's parked waiting
|
||||
# for stem separation (the audio worker resumes it; this catches a lost
|
||||
# resume) or its video dispatch was lost. Re-trigger once the stems exist;
|
||||
# if separation never finishes, fail loudly so the job doesn't sit silently
|
||||
# (the 2026-10-17/18 stuck "Starting final encode" jobs). Bounded to 24h so
|
||||
# ancient abandoned jobs are never revived.
|
||||
from backend.services.job_health_service import selected_stem_pending
|
||||
video_retriggered = []
|
||||
stems_wait_failed = []
|
||||
is_query = jobs_ref.where(
|
||||
filter=FieldFilter("status", "==", JobStatus.INSTRUMENTAL_SELECTED.value)
|
||||
).limit(SCREENS_RECOVERY_SCAN_LIMIT).stream()
|
||||
for doc in is_query:
|
||||
if len(video_retriggered) >= RENDER_RETRIGGERS_PER_TICK:
|
||||
break
|
||||
job_id = (doc.to_dict() or {}).get("job_id", doc.id)
|
||||
job = job_manager.get_job(job_id)
|
||||
age = _job_updated_age_seconds(job) if job else None
|
||||
if age is None or not (INSTRUMENTAL_SELECTED_STALL_SECONDS < age < INSTRUMENTAL_SELECTED_MAX_AGE_SECONDS):
|
||||
continue
|
||||
if selected_stem_pending(job):
|
||||
if age > STEMS_WAIT_FAIL_SECONDS:
|
||||
logger.error(f"[job:{job_id}] Final encode waited >60 min for stem separation that never finished — failing")
|
||||
job_manager.mark_job_failed(
|
||||
job_id=job_id,
|
||||
error_message="Audio separation didn't finish, so the final video couldn't be made. Retry the job to try again.",
|
||||
error_details={"stage": "video_waiting_for_stems", "reason": "stems_never_ready"},
|
||||
)
|
||||
stems_wait_failed.append(job_id)
|
||||
continue
|
||||
logger.warning(f"[job:{job_id}] INSTRUMENTAL_SELECTED stalled >10 min with stems ready — re-triggering video worker")
|
||||
try:
|
||||
# Clear a parked flag first so a late audio-worker resume can't double-trigger.
|
||||
job_manager.claim_video_stems_wait(job_id)
|
||||
if run_on_loop(loop, worker_service.trigger_video_worker(job_id)):
|
||||
video_retriggered.append(job_id)
|
||||
except Exception as e: # noqa: BLE001
|
||||
logger.warning(f"[job:{job_id}] video re-trigger failed: {e}")
|
||||
|
||||
logger.info(
|
||||
f"RECOVER_STUCK_JOBS complete: recovered={len(recovered)} "
|
||||
f"download_parked={len(download_parked)} download_retried={len(download_retried)} "
|
||||
@@ -1392,6 +1433,8 @@ def _recover_stuck_jobs_sync(http_request: Request, loop: asyncio.AbstractEventL
|
||||
"screens_retriggered_count": len(screens_retriggered),
|
||||
"stems_restore_failed_jobs": stems_restore_failed,
|
||||
"regen_screens_retriggered_jobs": regen_screens_retriggered,
|
||||
"video_retriggered_jobs": video_retriggered,
|
||||
"stems_wait_failed_jobs": stems_wait_failed,
|
||||
}
|
||||
|
||||
|
||||
@@ -1400,6 +1443,9 @@ RENDER_RETRIGGERS_PER_TICK = 10 # bound the blast radius per 5-min tick
|
||||
PREP_SCREENS_STALL_SECONDS = 10 * 60 # screens normally start within seconds of lyrics done
|
||||
SCREENS_RETRIGGERS_PER_TICK = 10 # bound the blast radius per 5-min tick
|
||||
SCREENS_RECOVERY_SCAN_LIMIT = 500 # cap DOWNLOADING docs scanned per tick (transient state)
|
||||
INSTRUMENTAL_SELECTED_STALL_SECONDS = 10 * 60 # video worker leaves this state within seconds
|
||||
STEMS_WAIT_FAIL_SECONDS = 60 * 60 # separation normally lands within ~15 min of review
|
||||
INSTRUMENTAL_SELECTED_MAX_AGE_SECONDS = 24 * 3600 # never revive ancient abandoned jobs
|
||||
|
||||
|
||||
def _job_updated_age_seconds(job):
|
||||
|
||||
@@ -134,6 +134,8 @@ async def run_audio(job_id: str) -> bool:
|
||||
|
||||
job_manager.mark_audio_complete(job_id)
|
||||
await job_manager.advance_to_screens_if_ready(job_id)
|
||||
# Same as the real audio worker: resume a final encode parked on these stems.
|
||||
await job_manager.resume_video_if_waiting_for_stems(job_id)
|
||||
return True
|
||||
|
||||
|
||||
|
||||
@@ -131,6 +131,27 @@ class FirestoreService:
|
||||
logger.error(f"Error updating job {job_id}: {e}")
|
||||
raise
|
||||
|
||||
def claim_job_state_flag(self, job_id: str, key: str) -> bool:
|
||||
"""Atomically test-and-clear ``state_data.<key>``.
|
||||
|
||||
Returns True for exactly one caller while the flag is truthy (it is set
|
||||
to False in the same transaction), so two workers racing to act on the
|
||||
same flag can't both proceed.
|
||||
"""
|
||||
doc_ref = self.db.collection(self.collection).document(job_id)
|
||||
|
||||
@firestore.transactional
|
||||
def _claim(transaction, ref) -> bool:
|
||||
snapshot = ref.get(transaction=transaction)
|
||||
if not snapshot.exists:
|
||||
return False
|
||||
if not ((snapshot.to_dict() or {}).get("state_data") or {}).get(key):
|
||||
return False
|
||||
transaction.update(ref, {f"state_data.{key}": False, "updated_at": datetime.utcnow()})
|
||||
return True
|
||||
|
||||
return _claim(self.db.transaction(), doc_ref)
|
||||
|
||||
def update_job_status(
|
||||
self,
|
||||
job_id: str,
|
||||
|
||||
@@ -287,3 +287,21 @@ def validate_worker_can_run(worker_name: str, job: Job) -> Optional[str]:
|
||||
)
|
||||
|
||||
return None
|
||||
|
||||
|
||||
def selected_stem_pending(job) -> bool:
|
||||
"""True when the selected instrumental stem hasn't been produced yet but
|
||||
stem separation is still running, i.e. the job should wait, not fail.
|
||||
|
||||
Review can finish before separation does (audio separation is decoupled from
|
||||
the lyrics review path), so this is a normal ordering, not an error.
|
||||
"""
|
||||
if validate_worker_can_run("video_worker", job):
|
||||
return False # wrong status (duplicate/late trigger): never park, let the normal checks reject it
|
||||
selection = (job.state_data or {}).get('instrumental_selection')
|
||||
if selection not in ('clean', 'with_backing'):
|
||||
return False # 'custom' uses an uploaded file / mute-region stem
|
||||
stem_key = 'instrumental_clean' if selection == 'clean' else 'instrumental_with_backing'
|
||||
if (job.file_urls or {}).get('stems', {}).get(stem_key):
|
||||
return False
|
||||
return not (job.state_data or {}).get('audio_complete', False)
|
||||
@@ -1305,6 +1305,41 @@ class JobManager:
|
||||
self.update_state_data(job_id, 'audio_complete', True)
|
||||
logger.info(f"Job {job_id}: Audio processing complete (separation finished)")
|
||||
|
||||
# Set by the video worker when review finished before stem separation did
|
||||
# (the selected instrumental stem doesn't exist yet). The audio worker
|
||||
# resumes the encode when separation completes; recover_stuck_jobs is the
|
||||
# backstop. Before this, the video worker just failed its prerequisites 3x
|
||||
# and the job sat at instrumental_selected forever (2026-10-17/18).
|
||||
VIDEO_WAITING_FOR_STEMS_KEY = 'video_waiting_for_stems'
|
||||
|
||||
def park_video_for_stems(self, job_id: str) -> None:
|
||||
"""Record that the final encode is waiting for stem separation."""
|
||||
self.update_job(job_id, {
|
||||
f'state_data.{self.VIDEO_WAITING_FOR_STEMS_KEY}': True,
|
||||
'message': 'Waiting for audio separation to finish; the final video starts automatically',
|
||||
})
|
||||
|
||||
def claim_video_stems_wait(self, job_id: str) -> bool:
|
||||
"""Atomically clear the waiting flag; True for exactly one claimant."""
|
||||
return self.firestore.claim_job_state_flag(job_id, self.VIDEO_WAITING_FOR_STEMS_KEY)
|
||||
|
||||
async def resume_video_if_waiting_for_stems(self, job_id: str) -> bool:
|
||||
"""Re-trigger the video worker for a job parked waiting on stems.
|
||||
|
||||
Called by the audio worker right after ``mark_audio_complete``. No-op
|
||||
unless the video worker parked the job (flag set) and it's still in a
|
||||
state the video worker accepts. Returns True iff a dispatch was issued.
|
||||
"""
|
||||
if not self.claim_video_stems_wait(job_id):
|
||||
return False
|
||||
job = self.get_job(job_id)
|
||||
if not job or job.status not in (JobStatus.INSTRUMENTAL_SELECTED, JobStatus.REVIEW_COMPLETE):
|
||||
logger.info(f"Job {job_id}: stems ready but job is {getattr(job, 'status', None)}; not resuming video")
|
||||
return False
|
||||
from backend.services.worker_service import get_worker_service
|
||||
logger.info(f"Job {job_id}: stems ready — resuming parked video worker")
|
||||
return await get_worker_service().trigger_video_worker(job_id)
|
||||
|
||||
def mark_lyrics_complete(self, job_id: str) -> None:
|
||||
"""
|
||||
Mark lyrics processing as complete.
|
||||
|
||||
@@ -56,7 +56,7 @@ def _job(job_id, status, *, minutes_stale=0, source_name=None, download_retry=No
|
||||
|
||||
|
||||
def _mock_jm(*, downloading=None, pending=None, rendering=None, review_complete=None,
|
||||
downloading_prep=None, jobs=None):
|
||||
downloading_prep=None, instrumental_selected=None, jobs=None):
|
||||
"""Wire a JobManager mock for the endpoint's 4 status queries.
|
||||
|
||||
downloading/rendering/review_complete use .where().stream(); pending uses
|
||||
@@ -69,11 +69,13 @@ def _mock_jm(*, downloading=None, pending=None, rendering=None, review_complete=
|
||||
where.stream.side_effect = [
|
||||
iter(downloading or []), iter(rendering or []), iter(review_complete or []),
|
||||
]
|
||||
# Three .limit().stream() calls in order: download_pending_retry, then downloading
|
||||
# Four .limit().stream() calls in order: download_pending_retry, then downloading
|
||||
# (lost-screens-trigger sweep, bounded by SCREENS_RECOVERY_SCAN_LIMIT), then
|
||||
# lyrics_complete (stalled storage-retention stems restore)
|
||||
# lyrics_complete (stalled storage-retention stems restore), then
|
||||
# instrumental_selected (final encode parked on stems / lost video trigger)
|
||||
where.limit.return_value.stream.side_effect = [
|
||||
iter(pending or []), iter(downloading_prep or []), iter([]),
|
||||
iter(instrumental_selected or []),
|
||||
]
|
||||
mock_jm.get_job.side_effect = lambda jid: (jobs or {}).get(jid)
|
||||
mock_jm.transition_to_state.return_value = True
|
||||
@@ -284,3 +286,72 @@ def test_fresh_review_complete_is_left_alone(client):
|
||||
assert resp.status_code == 200
|
||||
assert resp.json()["render_retriggered_jobs"] == []
|
||||
mock_ws.trigger_render_video_worker.assert_not_awaited()
|
||||
|
||||
|
||||
def _is_job(job_id, *, minutes_stale, audio_complete, stem_ready, selection="clean"):
|
||||
job = _job(job_id, JobStatus.INSTRUMENTAL_SELECTED, minutes_stale=minutes_stale)
|
||||
job.state_data.update({"instrumental_selection": selection, "audio_complete": audio_complete})
|
||||
if stem_ready:
|
||||
job.file_urls = {"stems": {"instrumental_clean": "gs://b/clean.flac",
|
||||
"instrumental_with_backing": "gs://b/backing.flac"}}
|
||||
return job
|
||||
|
||||
|
||||
def _run_is(client, job):
|
||||
mock_jm = _mock_jm(
|
||||
instrumental_selected=[_doc(job.job_id, JobStatus.INSTRUMENTAL_SELECTED)],
|
||||
jobs={job.job_id: job},
|
||||
)
|
||||
mock_ws = MagicMock()
|
||||
mock_ws.trigger_video_worker = AsyncMock(return_value=True)
|
||||
with patch("backend.api.routes.internal.JobManager", return_value=mock_jm), \
|
||||
patch("backend.services.worker_service.get_worker_service", return_value=mock_ws):
|
||||
resp = client.post("/api/internal/recover-stuck-jobs")
|
||||
assert resp.status_code == 200
|
||||
return resp.json(), mock_jm, mock_ws
|
||||
|
||||
|
||||
def test_instrumental_selected_with_stems_ready_retriggers_video(client):
|
||||
"""Parked/lost video trigger: stems landed but no encode started -> re-trigger."""
|
||||
body, mock_jm, mock_ws = _run_is(
|
||||
client, _is_job("is1", minutes_stale=15, audio_complete=True, stem_ready=True))
|
||||
assert body["video_retriggered_jobs"] == ["is1"]
|
||||
mock_ws.trigger_video_worker.assert_awaited_once_with("is1")
|
||||
mock_jm.claim_video_stems_wait.assert_called_once_with("is1")
|
||||
|
||||
|
||||
def test_instrumental_selected_fresh_is_left_alone(client):
|
||||
body, _, mock_ws = _run_is(
|
||||
client, _is_job("is2", minutes_stale=2, audio_complete=True, stem_ready=True))
|
||||
assert body["video_retriggered_jobs"] == []
|
||||
mock_ws.trigger_video_worker.assert_not_awaited()
|
||||
|
||||
|
||||
def test_instrumental_selected_still_separating_waits(client):
|
||||
"""Separation still running (<60 min): neither re-trigger nor fail."""
|
||||
body, mock_jm, mock_ws = _run_is(
|
||||
client, _is_job("is3", minutes_stale=30, audio_complete=False, stem_ready=False))
|
||||
assert body["video_retriggered_jobs"] == []
|
||||
assert body["stems_wait_failed_jobs"] == []
|
||||
mock_ws.trigger_video_worker.assert_not_awaited()
|
||||
mock_jm.mark_job_failed.assert_not_called()
|
||||
|
||||
|
||||
def test_instrumental_selected_stems_never_ready_fails_loudly(client):
|
||||
"""Separation never finished (>60 min): fail so the job isn't stuck silently."""
|
||||
body, mock_jm, mock_ws = _run_is(
|
||||
client, _is_job("is4", minutes_stale=90, audio_complete=False, stem_ready=False))
|
||||
assert body["stems_wait_failed_jobs"] == ["is4"]
|
||||
mock_jm.mark_job_failed.assert_called_once()
|
||||
assert mock_jm.mark_job_failed.call_args.kwargs["error_details"]["stage"] == "video_waiting_for_stems"
|
||||
mock_ws.trigger_video_worker.assert_not_awaited()
|
||||
|
||||
|
||||
def test_instrumental_selected_older_than_24h_is_never_revived(client):
|
||||
body, mock_jm, mock_ws = _run_is(
|
||||
client, _is_job("is5", minutes_stale=25 * 60, audio_complete=True, stem_ready=True))
|
||||
assert body["video_retriggered_jobs"] == []
|
||||
assert body["stems_wait_failed_jobs"] == []
|
||||
mock_ws.trigger_video_worker.assert_not_awaited()
|
||||
mock_jm.mark_job_failed.assert_not_called()
|
||||
|
||||
@@ -0,0 +1,132 @@
|
||||
"""Final encode waits for stem separation instead of failing (2026-10-19).
|
||||
|
||||
Review can finish before separation does. The video worker used to fail its
|
||||
prerequisites ("Missing instrumental: instrumental_clean") 3x and leave the job at
|
||||
instrumental_selected forever. Now it parks the job, the audio worker resumes it
|
||||
when the stems land, and an atomic claim on the flag makes the park/resume race
|
||||
safe.
|
||||
"""
|
||||
|
||||
from datetime import datetime, timezone
|
||||
from unittest.mock import AsyncMock, MagicMock, patch
|
||||
|
||||
import pytest
|
||||
|
||||
from backend.models.job import Job, JobStatus
|
||||
from backend.services.job_health_service import selected_stem_pending
|
||||
from backend.services.job_manager import JobManager
|
||||
from backend.workers.video_worker import _park_until_stems_ready
|
||||
|
||||
|
||||
def _job(*, status=JobStatus.INSTRUMENTAL_SELECTED, selection="clean", audio_complete=False,
|
||||
stems=None, waiting=None):
|
||||
sd = {"instrumental_selection": selection, "audio_complete": audio_complete}
|
||||
if waiting is not None:
|
||||
sd["video_waiting_for_stems"] = waiting
|
||||
return Job(
|
||||
job_id="j1", artist="A", title="B", status=status,
|
||||
created_at=datetime.now(timezone.utc), updated_at=datetime.now(timezone.utc),
|
||||
state_data=sd, file_urls={"stems": stems or {}},
|
||||
)
|
||||
|
||||
|
||||
READY = {"instrumental_clean": "gs://b/c.flac", "instrumental_with_backing": "gs://b/w.flac"}
|
||||
|
||||
|
||||
class TestSelectedStemPending:
|
||||
def test_pending_when_stem_missing_and_separation_running(self):
|
||||
assert selected_stem_pending(_job()) is True
|
||||
|
||||
def test_with_backing_selection_checks_its_own_stem(self):
|
||||
assert selected_stem_pending(_job(selection="with_backing",
|
||||
stems={"instrumental_clean": "x"})) is True
|
||||
|
||||
def test_not_pending_once_stem_exists(self):
|
||||
assert selected_stem_pending(_job(stems=READY)) is False
|
||||
|
||||
def test_not_pending_when_separation_finished_without_stem(self):
|
||||
"""audio_complete but no stem = a real failure; let prerequisites reject it."""
|
||||
assert selected_stem_pending(_job(audio_complete=True)) is False
|
||||
|
||||
def test_custom_instrumental_never_waits(self):
|
||||
assert selected_stem_pending(_job(selection="custom")) is False
|
||||
|
||||
@pytest.mark.parametrize("status", [JobStatus.GENERATING_VIDEO, JobStatus.COMPLETE, JobStatus.FAILED])
|
||||
def test_wrong_status_never_parks(self, status):
|
||||
assert selected_stem_pending(_job(status=status)) is False
|
||||
|
||||
|
||||
class TestParkUntilStemsReady:
|
||||
def _jm(self, refreshed, claim=False):
|
||||
jm = MagicMock()
|
||||
jm.get_job.return_value = refreshed
|
||||
jm.claim_video_stems_wait.return_value = claim
|
||||
return jm
|
||||
|
||||
def test_not_pending_continues_without_touching_the_flag(self):
|
||||
jm = self._jm(None)
|
||||
parked, job = _park_until_stems_ready("j1", _job(stems=READY), jm, MagicMock())
|
||||
assert parked is False and job.file_urls["stems"] == READY
|
||||
jm.park_video_for_stems.assert_not_called()
|
||||
|
||||
def test_still_pending_after_refresh_parks(self):
|
||||
jm = self._jm(_job())
|
||||
parked, _ = _park_until_stems_ready("j1", _job(), jm, MagicMock())
|
||||
assert parked is True
|
||||
jm.park_video_for_stems.assert_called_once_with("j1")
|
||||
jm.claim_video_stems_wait.assert_not_called()
|
||||
|
||||
def test_stems_landed_while_parking_and_we_win_the_claim_continues(self):
|
||||
jm = self._jm(_job(stems=READY, audio_complete=True), claim=True)
|
||||
parked, job = _park_until_stems_ready("j1", _job(), jm, MagicMock())
|
||||
assert parked is False
|
||||
assert job.file_urls["stems"] == READY # continues with the refreshed job
|
||||
|
||||
def test_stems_landed_but_audio_worker_already_resumed_exits(self):
|
||||
jm = self._jm(_job(stems=READY, audio_complete=True), claim=False)
|
||||
parked, _ = _park_until_stems_ready("j1", _job(), jm, MagicMock())
|
||||
assert parked is True # the audio worker dispatched a fresh run
|
||||
|
||||
|
||||
class TestResumeVideoIfWaitingForStems:
|
||||
def _jm(self, *, claimed, job):
|
||||
jm = JobManager.__new__(JobManager)
|
||||
jm.firestore = MagicMock()
|
||||
jm.firestore.claim_job_state_flag.return_value = claimed
|
||||
jm.get_job = MagicMock(return_value=job)
|
||||
return jm
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_resumes_parked_job(self):
|
||||
jm = self._jm(claimed=True, job=_job(stems=READY, audio_complete=True))
|
||||
ws = MagicMock(trigger_video_worker=AsyncMock(return_value=True))
|
||||
with patch("backend.services.worker_service.get_worker_service", return_value=ws):
|
||||
assert await jm.resume_video_if_waiting_for_stems("j1") is True
|
||||
jm.firestore.claim_job_state_flag.assert_called_once_with("j1", "video_waiting_for_stems")
|
||||
ws.trigger_video_worker.assert_awaited_once_with("j1")
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_no_flag_no_trigger(self):
|
||||
"""Normal ordering (stems before review): nothing parked, nothing to do."""
|
||||
jm = self._jm(claimed=False, job=_job())
|
||||
ws = MagicMock(trigger_video_worker=AsyncMock(return_value=True))
|
||||
with patch("backend.services.worker_service.get_worker_service", return_value=ws):
|
||||
assert await jm.resume_video_if_waiting_for_stems("j1") is False
|
||||
ws.trigger_video_worker.assert_not_awaited()
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_job_moved_on_is_not_resumed(self):
|
||||
jm = self._jm(claimed=True, job=_job(status=JobStatus.FAILED))
|
||||
ws = MagicMock(trigger_video_worker=AsyncMock(return_value=True))
|
||||
with patch("backend.services.worker_service.get_worker_service", return_value=ws):
|
||||
assert await jm.resume_video_if_waiting_for_stems("j1") is False
|
||||
ws.trigger_video_worker.assert_not_awaited()
|
||||
|
||||
|
||||
def test_park_sets_flag_and_user_facing_message():
|
||||
jm = JobManager.__new__(JobManager)
|
||||
jm.update_job = MagicMock()
|
||||
jm.park_video_for_stems("j1")
|
||||
updates = jm.update_job.call_args.args[1]
|
||||
assert updates["state_data.video_waiting_for_stems"] is True
|
||||
assert "starts automatically" in updates["message"]
|
||||
@@ -412,6 +412,15 @@ async def process_audio_separation(job_id: str) -> bool:
|
||||
# normal run never double-triggers.
|
||||
await job_manager.advance_to_screens_if_ready(job_id)
|
||||
|
||||
# If review already finished and the video worker parked the job
|
||||
# waiting for these stems, resume the final encode now.
|
||||
try:
|
||||
if await job_manager.resume_video_if_waiting_for_stems(job_id):
|
||||
job_log.info("Stems ready: resumed the parked final encode")
|
||||
except Exception as e:
|
||||
# recover_stuck_jobs re-triggers a parked job within ~10 min
|
||||
logger.error(f"[job:{job_id}] Failed to resume parked video worker: {e}")
|
||||
|
||||
# Auto-approval second chance: when lyrics/screens finished BEFORE
|
||||
# audio, the job is already parked in AWAITING_REVIEW and the
|
||||
# screens-worker scoring ran without backing analysis. Now that
|
||||
|
||||
@@ -47,7 +47,7 @@ from backend.services.admin_rerender_service import (
|
||||
)
|
||||
from backend.services.regenerate_service import active_regenerate
|
||||
from backend.utils.existing_instrumental import resolve_existing_instrumental
|
||||
from backend.services.job_health_service import validate_worker_can_run
|
||||
from backend.services.job_health_service import selected_stem_pending, validate_worker_can_run
|
||||
from backend.services.rclone_service import get_rclone_service
|
||||
from backend.services.youtube_service import get_youtube_service
|
||||
from backend.services.youtube_description import translated_language_name
|
||||
@@ -272,6 +272,12 @@ async def generate_video_orchestrated(job_id: str) -> bool:
|
||||
except Exception as e: # noqa: BLE001 - the render stage already reported a missing mix
|
||||
logger.error(f"[job:{job_id}] Input recovery failed in video worker: {e}")
|
||||
|
||||
# Review can finish before stem separation does: wait for the stems instead
|
||||
# of failing prerequisites (which used to strand the job forever).
|
||||
parked, job = _park_until_stems_ready(job_id, job, job_manager, job_log)
|
||||
if parked:
|
||||
return True
|
||||
|
||||
# Validate prerequisites
|
||||
if not _validate_prerequisites(job):
|
||||
logger.error(f"[job:{job_id}] Prerequisites not met for video generation")
|
||||
@@ -868,6 +874,12 @@ async def generate_video_legacy(job_id: str) -> bool:
|
||||
logger.error(f"[job:{job_id}] Job not found")
|
||||
return False
|
||||
|
||||
# Review can finish before stem separation does: wait for the stems instead
|
||||
# of failing prerequisites (which used to strand the job forever).
|
||||
parked, job = _park_until_stems_ready(job_id, job, job_manager, job_log)
|
||||
if parked:
|
||||
return True
|
||||
|
||||
# Validate prerequisites
|
||||
if not _validate_prerequisites(job):
|
||||
logger.error(f"[job:{job_id}] Prerequisites not met for video generation")
|
||||
@@ -1536,6 +1548,34 @@ def _store_video_processing_metadata(
|
||||
logger.warning(f"Job {job_id}: Failed to store video processing metadata (non-fatal): {e}")
|
||||
|
||||
|
||||
def _park_until_stems_ready(job_id: str, job, job_manager: JobManager, job_log):
|
||||
"""Park the job if its selected stem is still being separated.
|
||||
|
||||
Returns ``(parked, job)``. When ``parked`` is True the worker should exit
|
||||
successfully: the audio worker re-triggers it once separation completes.
|
||||
When False, carry on with the returned (possibly refreshed) job.
|
||||
|
||||
Race with the audio worker: we set the flag, then re-read. If separation
|
||||
finished in between, exactly one of us wins the atomic claim on the flag —
|
||||
we continue in-process, or the audio worker has already dispatched a fresh
|
||||
run and we exit.
|
||||
"""
|
||||
if not selected_stem_pending(job):
|
||||
return False, job
|
||||
job_manager.park_video_for_stems(job_id)
|
||||
refreshed = job_manager.get_job(job_id) or job
|
||||
if selected_stem_pending(refreshed):
|
||||
msg = "Stem separation still running — final encode parked; it resumes automatically when stems are ready"
|
||||
logger.info(f"[job:{job_id}] {msg}")
|
||||
job_log.info(msg)
|
||||
return True, refreshed
|
||||
if job_manager.claim_video_stems_wait(job_id):
|
||||
logger.info(f"[job:{job_id}] Stems landed while parking — continuing")
|
||||
return False, refreshed
|
||||
logger.info(f"[job:{job_id}] Stems landed while parking; audio worker already resumed the encode")
|
||||
return True, refreshed
|
||||
|
||||
|
||||
def _validate_prerequisites(job) -> bool:
|
||||
"""Validate that all prerequisites are met for video generation."""
|
||||
# Validate job status is appropriate for video worker
|
||||
|
||||
@@ -276,6 +276,23 @@ curl -X POST "https://api.nomadkaraoke.com/api/internal/workers/screens" \
|
||||
-d '{"job_id": "<job_id>"}'
|
||||
```
|
||||
|
||||
## Job at `instrumental_selected` "Waiting for audio separation to finish"
|
||||
|
||||
This is normal when review finishes before stem separation does. The video worker
|
||||
parks the job (`state_data.video_waiting_for_stems=true`) and exits cleanly. The
|
||||
audio worker resumes the final encode as soon as the stems land
|
||||
(`resume_video_if_waiting_for_stems`). Backstop: `recover_stuck_jobs` re-triggers
|
||||
the video worker once a job has sat in `instrumental_selected` for 10+ min with its
|
||||
stem present. If separation never finishes, it fails the job after 60 min
|
||||
(`error_details.stage = "video_waiting_for_stems"`), so it never stays stuck silently.
|
||||
|
||||
Before 2026-10-19 (v0.278.11) the video worker failed its prerequisites instead
|
||||
("Missing instrumental: instrumental_clean") and the job stayed at "Starting final
|
||||
encode" forever. That got much more likely when the GPU image picked up
|
||||
audio-separator 0.47.0 (karaoke models 3–5× slower on L4). The version is now pinned
|
||||
in `backend/Dockerfile.gpu-base` (`AUDIO_SEPARATOR_VERSION`). Time a real job before
|
||||
bumping it.
|
||||
|
||||
## Job stuck at `encoding` status
|
||||
|
||||
**Cause:** A Cloud Run deployment killed the poller mid-encoding. The GCE worker finished but nobody received the result. The health service will flag this as `encoding_stuck` after 50 minutes.
|
||||
|
||||
+1
-1
@@ -1,6 +1,6 @@
|
||||
[tool.poetry]
|
||||
name = "karaoke-gen"
|
||||
version = "0.278.10"
|
||||
version = "0.278.11"
|
||||
description = "Generate karaoke videos with synchronized lyrics. Handles the entire process from downloading audio and lyrics to creating the final video with title screens."
|
||||
authors = ["Andrew Beveridge <andrew@beveridge.uk>"]
|
||||
license = "MIT"
|
||||
|
||||
Reference in new issue
Block a user