mirror of
https://github.com/nomadkaraoke/karaoke-gen.git
synced 2026-11-20 13:07:57 +09:00
fix: run render-video worker as Cloud Run Job to survive backend deploys (#994)
## Summary Tenant E2E run #107 (`singa`) failed with job `41e06b90` frozen at `rendering_video` — even though the GCE encoder **finished the render and uploaded every output**. Root cause: a backend deploy landed mid-render and killed the render poll, which ran as a FastAPI BackgroundTask on the Cloud Run service. Full investigation: `docs/archive/2026-09-13-render-deploy-survival-investigation.md` ### The failure chain (4 layers, each structurally broken) 1. **BackgroundTask execution** — the render endpoint returns 200 immediately and polls post-response; Cloud Run SIGKILLs old-revision instances ~10s after SIGTERM on every deploy. 2. **Unreachable shutdown parking** — uvicorn waits for background tasks *before* running lifespan shutdown, so `park_active_render_jobs_for_shutdown()` never runs in exactly the scenario it was written for. Its comment claimed a 600s grace period; Cloud Run gives 10s. Verified in logs: the old revision's last lines were uvicorn's `Waiting for background tasks to complete.` — then SIGKILL. 3. **Cloud Tasks retry defeated** — the queue got its 200 at dispatch; `dispatch_deadline=1800s` protects nothing. 4. **45-min stuck sweep** — reachable but slower than the E2E's 30-min budget and a ~50-60 min silent stall + wasted re-render for a real customer, on every deploy that lands mid-render (deploys go out on every merge to main). This is the **identical bug class fixed for the video worker in 2026-03** (`docs/archive/2026-03-08-fix-encoding-deploy-survival.md`, `USE_CLOUD_RUN_JOBS_FOR_VIDEO`) — the render worker was never migrated. ### Changes - **Route `trigger_render_video_worker` through a Cloud Run Job** — reuses `video-encoding-job` with args override `python -m backend.workers.render_video_worker --job-id <id>`, behind `USE_CLOUD_RUN_JOBS_FOR_RENDER` (enabled in deploy env). Job executions run to completion; deploys never touch them. - **New CLI entrypoint** in `render_video_worker.py`. Exit-code contract differs from video_worker deliberately: clean returns (True AND False) exit 0 because `process_render_video` parks/fails/supersedes internally; only an escaped crash exits 1 (job template has `max_retries=2`). - **Worker-side idempotency gate** (review finding): the CRJ path bypasses the HTTP endpoint's `_check_worker_idempotency`, and CRJ retries could re-enter a job in `RENDERING_VIDEO` (hard-failing it via the same-state transition guard) or resurrect a terminal job. The worker now skips terminal + already-rendering jobs cleanly regardless of dispatch path. - **Surface failed video-worker dispatch** (review finding): the ignored `trigger_video_worker` return after render completion now logs at ERROR instead of silently stalling the job at `instrumental_selected`. - **Pulumi**: `video-encoding-job` env gains both `USE_CLOUD_RUN_JOBS_FOR_*` flags so the render execution's in-process video-worker trigger also dispatches a CRJ. **Already applied to prod** (`pulumi up` run locally, verified on the job). - **`main.py`**: corrected the false 600s comment; shrank the unreachable 480s lifespan wait to 5s. Parking hook kept as legacy/flag-off net. ### Rollback Remove `USE_CLOUD_RUN_JOBS_FOR_RENDER=true` from the service env → falls back to the legacy Cloud Tasks path. ## Testing - Backend: 4422 unit + 151 emulator tests pass locally. - New tests: CRJ routing (flag on/off), CLI exit-code contract, worker-side idempotency gate (terminal skip, already-rendering skip). - Post-deploy: will dispatch the Tenant Portals E2E workflow (the failing scenario) against prod. 🤖 Generated with [Claude Code](https://claude.com/claude-code) <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit - **New Features** - Video rendering now runs as durable background jobs, allowing renders to continue through service deployments. - Added safeguards to prevent duplicate processing and safely handle completed or in-progress jobs. - Improved failure reporting when a video-processing job cannot be started. - **Bug Fixes** - Fixed render jobs becoming stuck when deployments interrupted background processing. - **Documentation** - Added troubleshooting guidance and incident lessons for interrupted or stuck renders. <!-- end of auto-generated comment: release notes by coderabbit.ai --> --------- Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
This commit is contained in:
1 parent
b41664a2fc
commit
3cdb62a058
15 files changed
+523
-23
No files matched your search
@@ -1994,7 +1994,7 @@ jobs:
|
||||
--min-instances 1 \
|
||||
--vpc-connector projects/nomadkaraoke/locations/us-central1/connectors/cloud-run-connector \
|
||||
--vpc-egress private-ranges-only \
|
||||
--set-env-vars "GOOGLE_CLOUD_PROJECT=${{ env.PROJECT_ID }},GCS_BUCKET_NAME=karaoke-gen-storage-${{ env.PROJECT_ID }},FIRESTORE_COLLECTION=jobs,ENVIRONMENT=production,KARAOKE_GEN_VERSION=$VERSION,DEFAULT_DROPBOX_PATH=/MediaUnsynced/Karaoke/Tracks-Organized,DEFAULT_GDRIVE_FOLDER_ID=1laRKAyxo0v817SstfM5XkpbWiNKNAMSX,ENABLE_CLOUD_TASKS=true,CLOUD_RUN_SERVICE_URL=https://api.nomadkaraoke.com,GCP_REGION=us-central1,DEFAULT_ENABLE_YOUTUBE_UPLOAD=true,DEFAULT_BRAND_PREFIX=NOMAD,DEFAULT_PRIVATE_DROPBOX_PATH=/MediaUnsynced/Karaoke/Tracks-NonPublished,DEFAULT_PRIVATE_BRAND_PREFIX=NOMADNP,USE_GCE_ENCODING=true,USE_GCE_PREVIEW_ENCODING=true,ENABLE_PUSH_NOTIFICATIONS=true,USE_CLOUD_RUN_JOBS_FOR_VIDEO=true,EMAIL_PROVIDER=postmark,AUTO_CORRECT_MODEL=claude-opus-4-8,AUTO_CORRECT_COMPARE_MODELS=claude-opus-4-8;gemini-3.1-pro-preview,AUTO_CORRECT_PROACTIVE_ENABLED=true,AUTO_APPROVAL_BACKING_KEEP_ENABLED=true,COMMUNITY_DAILY_PICK_ENABLED=true,EDGE_AUTH_MODE=enforce,COMMIT_SHA=${{ steps.build-meta.outputs.sha }},PR_NUMBER=${{ steps.build-meta.outputs.pr_number }},PR_TITLE=${{ steps.build-meta.outputs.pr_title }}" \
|
||||
--set-env-vars "GOOGLE_CLOUD_PROJECT=${{ env.PROJECT_ID }},GCS_BUCKET_NAME=karaoke-gen-storage-${{ env.PROJECT_ID }},FIRESTORE_COLLECTION=jobs,ENVIRONMENT=production,KARAOKE_GEN_VERSION=$VERSION,DEFAULT_DROPBOX_PATH=/MediaUnsynced/Karaoke/Tracks-Organized,DEFAULT_GDRIVE_FOLDER_ID=1laRKAyxo0v817SstfM5XkpbWiNKNAMSX,ENABLE_CLOUD_TASKS=true,CLOUD_RUN_SERVICE_URL=https://api.nomadkaraoke.com,GCP_REGION=us-central1,DEFAULT_ENABLE_YOUTUBE_UPLOAD=true,DEFAULT_BRAND_PREFIX=NOMAD,DEFAULT_PRIVATE_DROPBOX_PATH=/MediaUnsynced/Karaoke/Tracks-NonPublished,DEFAULT_PRIVATE_BRAND_PREFIX=NOMADNP,USE_GCE_ENCODING=true,USE_GCE_PREVIEW_ENCODING=true,ENABLE_PUSH_NOTIFICATIONS=true,USE_CLOUD_RUN_JOBS_FOR_VIDEO=true,USE_CLOUD_RUN_JOBS_FOR_RENDER=true,EMAIL_PROVIDER=postmark,AUTO_CORRECT_MODEL=claude-opus-4-8,AUTO_CORRECT_COMPARE_MODELS=claude-opus-4-8;gemini-3.1-pro-preview,AUTO_CORRECT_PROACTIVE_ENABLED=true,AUTO_APPROVAL_BACKING_KEEP_ENABLED=true,COMMUNITY_DAILY_PICK_ENABLED=true,EDGE_AUTH_MODE=enforce,COMMIT_SHA=${{ steps.build-meta.outputs.sha }},PR_NUMBER=${{ steps.build-meta.outputs.pr_number }},PR_TITLE=${{ steps.build-meta.outputs.pr_title }}" \
|
||||
--set-secrets "ANTHROPIC_API_KEY=anthropic-api-key:latest,AUDIOSHAKE_API_TOKEN=audioshake-api-key:latest,GENIUS_API_TOKEN=genius-api-key:latest,SPOTIFY_COOKIE_SP_DC=spotify-cookie:latest,RAPIDAPI_KEY=rapidapi-key:latest,DEFAULT_DISCORD_WEBHOOK_URL=discord-releases-webhook:latest,ADMIN_TOKENS=admin-tokens:latest,RED_API_KEY=red-api-key:latest,RED_API_URL=red-api-url:latest,OPS_API_KEY=ops-api-key:latest,OPS_API_URL=ops-api-url:latest,FLACFETCH_API_URL=flacfetch-api-url:latest,FLACFETCH_API_KEY=flacfetch-api-key:latest,POSTMARK_SERVER_TOKEN=postmark-server-token:latest,STRIPE_SECRET_KEY=stripe-secret-key:latest,STRIPE_WEBHOOK_SECRET=stripe-webhook-secret:latest,LANGFUSE_PUBLIC_KEY=langfuse-public-key:latest,LANGFUSE_SECRET_KEY=langfuse-secret-key:latest,LANGFUSE_HOST=langfuse-host:latest,ENCODING_WORKER_URL=encoding-worker-url:latest,ENCODING_WORKER_API_KEY=encoding-worker-api-key:latest,ENCODING_WORKER_FALLBACK_VMS=encoding-worker-fallback-vms:latest,VAPID_PUBLIC_KEY=vapid-public-key:latest,VAPID_PRIVATE_KEY=vapid-private-key:latest,E2E_BYPASS_KEY=e2e-bypass-key:latest,GDRIVE_VALIDATOR_URL=gdrive-validator-url:latest,EDGE_ORIGIN_SECRET=edge-origin-secret:latest"
|
||||
|
||||
# R3/R4 — smoke the tagged candidate revision BEFORE it serves any traffic.
|
||||
|
||||
@@ -200,6 +200,15 @@ class Settings(BaseSettings):
|
||||
# Default is false - Cloud Tasks is sufficient for most videos (15-20 min).
|
||||
use_cloud_run_jobs_for_video: bool = os.getenv("USE_CLOUD_RUN_JOBS_FOR_VIDEO", "false").lower() in ("true", "1", "yes")
|
||||
|
||||
# When enabled AND enable_cloud_tasks is true, the post-review render worker
|
||||
# also runs as a Cloud Run Job (reusing video-encoding-job with an args
|
||||
# override). As a BackgroundTask on the service it was killed by every
|
||||
# deploy rollout that landed mid-render — Cloud Run gives instances only
|
||||
# 10s after SIGTERM, far less than a render takes, so the job froze at
|
||||
# rendering_video until the 45-min stuck-job sweep (incident 2026-09-13,
|
||||
# job 41e06b90; same class as incident 2026-03-08 for the video worker).
|
||||
use_cloud_run_jobs_for_render: bool = os.getenv("USE_CLOUD_RUN_JOBS_FOR_RENDER", "false").lower() in ("true", "1", "yes")
|
||||
|
||||
# GCE Encoding Worker (for high-performance video encoding)
|
||||
# When enabled, video encoding is offloaded to a dedicated C4 GCE instance
|
||||
# with faster CPU (Intel Granite Rapids 3.9 GHz) instead of Cloud Run.
|
||||
|
||||
+17
-8
@@ -111,19 +111,28 @@ async def lifespan(app: FastAPI):
|
||||
|
||||
yield
|
||||
|
||||
# Shutdown - wait for any active workers to complete before terminating
|
||||
# This prevents Cloud Run from killing workers mid-processing.
|
||||
# Shutdown - best-effort parking of any still-registered workers.
|
||||
#
|
||||
# Cloud Run gen2 default termination grace period is 600s. We wait up to
|
||||
# 480s, leaving ~120s for the cleanup pass below to write Firestore state
|
||||
# before SIGKILL. Without that headroom, render jobs that miss the wait
|
||||
# would sit in `rendering_video` forever waiting for an operator.
|
||||
# Reality check (incident 2026-09-13, job 41e06b90): Cloud Run services
|
||||
# SIGKILL instances ~10 seconds after SIGTERM — NOT the 600s this comment
|
||||
# previously claimed — and uvicorn only runs this lifespan-shutdown code
|
||||
# AFTER all BackgroundTasks have completed ("Waiting for background tasks
|
||||
# to complete"). A long-running worker task therefore keeps uvicorn stuck
|
||||
# in that wait until SIGKILL, and this hook never executes at all. It can
|
||||
# NOT be relied on to protect in-flight renders; that protection comes
|
||||
# from running them as Cloud Run Jobs instead (USE_CLOUD_RUN_JOBS_FOR_RENDER,
|
||||
# mirroring USE_CLOUD_RUN_JOBS_FOR_VIDEO from incident 2026-03-08).
|
||||
#
|
||||
# This hook is kept as a cheap safety net for the legacy flag-off path and
|
||||
# for the narrow case where workers finish right as SIGTERM lands. The wait
|
||||
# is capped well inside the 10s kill window so the parking pass below still
|
||||
# gets a chance to write Firestore state.
|
||||
logger.info("Shutdown requested, checking for active workers...")
|
||||
if worker_registry.has_active_workers():
|
||||
active = worker_registry.get_active_workers()
|
||||
logger.info(f"Active workers found: {active}")
|
||||
logger.info("Waiting for workers to complete (timeout: 480s)...")
|
||||
completed = await worker_registry.wait_for_completion(timeout=480)
|
||||
logger.info("Waiting for workers to complete (timeout: 5s)...")
|
||||
completed = await worker_registry.wait_for_completion(timeout=5)
|
||||
if not completed:
|
||||
logger.error(
|
||||
"Shutdown timeout - some workers may not have completed cleanly. "
|
||||
|
||||
@@ -690,9 +690,25 @@ class WorkerService:
|
||||
return False
|
||||
|
||||
async def trigger_render_video_worker(self, job_id: str) -> bool:
|
||||
"""Trigger render video worker (post-review)."""
|
||||
"""
|
||||
Trigger render video worker (post-review).
|
||||
|
||||
When USE_CLOUD_RUN_JOBS_FOR_RENDER=true and ENABLE_CLOUD_TASKS=true,
|
||||
runs as a Cloud Run Job (reusing video-encoding-job with an args
|
||||
override) so the render orchestration survives Cloud Run Service
|
||||
deployment rollouts. As a BackgroundTask it was killed ~10s after
|
||||
SIGTERM whenever a deploy landed mid-render, freezing the job at
|
||||
rendering_video (incident 2026-09-13, job 41e06b90 — same class as
|
||||
incident 2026-03-08 which migrated the video worker to Cloud Run Jobs).
|
||||
"""
|
||||
self._bump_worker_generation(job_id)
|
||||
self._warmup_encoding_worker(job_id)
|
||||
if self._use_cloud_tasks and self.settings.use_cloud_run_jobs_for_render:
|
||||
return await self._trigger_worker_cloud_run_job(
|
||||
job_id=job_id,
|
||||
cloud_run_job_name="video-encoding-job",
|
||||
worker_module="render_video_worker",
|
||||
)
|
||||
return await self.trigger_worker("render-video", job_id)
|
||||
|
||||
def _bump_worker_generation(self, job_id: str) -> None:
|
||||
|
||||
@@ -0,0 +1,71 @@
|
||||
"""
|
||||
Tests for the render video worker CLI entry point (Cloud Run Jobs execution).
|
||||
|
||||
The render worker runs as a Cloud Run Job (video-encoding-job with an args
|
||||
override) so it survives Cloud Run Service deployment rollouts — as a
|
||||
BackgroundTask it was killed ~10s after SIGTERM whenever a deploy landed
|
||||
mid-render (incident 2026-09-13, job 41e06b90).
|
||||
|
||||
Exit-code contract (deliberately different from video_worker): the Cloud Run
|
||||
Job template has max_retries=2 and process_render_video handles every expected
|
||||
failure internally (park / fail_job / supersede-discard). A clean return —
|
||||
True OR False — must exit 0 so Cloud Run never re-runs the worker against a
|
||||
job that already moved to a non-render state. Only an escaped exception
|
||||
(crash before state was handled) exits 1 to request a Cloud Run retry.
|
||||
"""
|
||||
import sys
|
||||
from unittest.mock import AsyncMock, MagicMock, patch
|
||||
|
||||
import pytest
|
||||
|
||||
from backend.workers.render_video_worker import main
|
||||
|
||||
|
||||
def _run_main_with_args(job_id="job123"):
|
||||
return patch.object(sys, "argv", ["render_video_worker", "--job-id", job_id])
|
||||
|
||||
|
||||
class TestRenderVideoWorkerCli:
|
||||
def test_main_exits_zero_on_success(self):
|
||||
with _run_main_with_args(), \
|
||||
patch("backend.workers.render_video_worker.process_render_video",
|
||||
new=AsyncMock(return_value=True)):
|
||||
with pytest.raises(SystemExit) as exc_info:
|
||||
main()
|
||||
assert exc_info.value.code == 0
|
||||
|
||||
def test_main_exits_zero_on_handled_failure(self):
|
||||
"""False means the worker already parked/failed/superseded the job —
|
||||
a Cloud Run retry would re-enter a job in a non-render state."""
|
||||
with _run_main_with_args(), \
|
||||
patch("backend.workers.render_video_worker.process_render_video",
|
||||
new=AsyncMock(return_value=False)):
|
||||
with pytest.raises(SystemExit) as exc_info:
|
||||
main()
|
||||
assert exc_info.value.code == 0
|
||||
|
||||
def test_main_exits_nonzero_on_crash(self):
|
||||
"""An escaped exception means job state was NOT handled — exit 1 so
|
||||
Cloud Run's max_retries can re-run the worker."""
|
||||
with _run_main_with_args(), \
|
||||
patch("backend.workers.render_video_worker.process_render_video",
|
||||
new=AsyncMock(side_effect=RuntimeError("boom"))):
|
||||
with pytest.raises(SystemExit) as exc_info:
|
||||
main()
|
||||
assert exc_info.value.code == 1
|
||||
|
||||
def test_main_passes_job_id(self):
|
||||
mock_worker = AsyncMock(return_value=True)
|
||||
with _run_main_with_args("abc999"), \
|
||||
patch("backend.workers.render_video_worker.process_render_video",
|
||||
new=mock_worker):
|
||||
with pytest.raises(SystemExit):
|
||||
main()
|
||||
mock_worker.assert_awaited_once_with("abc999")
|
||||
|
||||
def test_main_requires_job_id(self):
|
||||
with patch.object(sys, "argv", ["render_video_worker"]):
|
||||
with pytest.raises(SystemExit) as exc_info:
|
||||
main()
|
||||
# argparse exits 2 on missing required argument
|
||||
assert exc_info.value.code == 2
|
||||
@@ -104,6 +104,9 @@ class TestIsDuetPropagation:
|
||||
job.style_assets = {}
|
||||
job.prep_only = False
|
||||
job.subtitle_offset_ms = 0
|
||||
# Real dispatch state — the worker itself transitions to
|
||||
# RENDERING_VIDEO (and skips jobs already in it / terminal).
|
||||
job.status = JobStatus.REVIEW_COMPLETE
|
||||
return job
|
||||
|
||||
def test_is_duet_true_passed_to_output_config_local_path(self):
|
||||
|
||||
@@ -23,7 +23,11 @@ from backend.models.job import JobStatus
|
||||
from backend.workers.registry import worker_registry
|
||||
|
||||
|
||||
def _build_minimal_job(status=JobStatus.RENDERING_VIDEO):
|
||||
def _build_minimal_job(status=JobStatus.REVIEW_COMPLETE):
|
||||
# Default matches the real entry state: the worker is dispatched with the
|
||||
# job in REVIEW_COMPLETE and transitions to RENDERING_VIDEO itself. A job
|
||||
# already in RENDERING_VIDEO at entry is skipped by the worker-side
|
||||
# idempotency gate (duplicate trigger / Cloud Run Job retry protection).
|
||||
job = MagicMock()
|
||||
job.artist = "Test Artist"
|
||||
job.title = "Test Title"
|
||||
@@ -61,7 +65,15 @@ async def test_worker_registers_and_unregisters_on_success():
|
||||
|
||||
mock_job_manager = MagicMock()
|
||||
mock_job_manager.get_job.return_value = job
|
||||
mock_job_manager.transition_to_state.return_value = True
|
||||
|
||||
# Mirror real transitions on the shared mock so the supersession fence's
|
||||
# mid-render re-reads see RENDERING_VIDEO (a frozen REVIEW_COMPLETE would
|
||||
# read as "status reset" and discard the render).
|
||||
def _transition(job_id=None, new_status=None, **kwargs):
|
||||
job.status = new_status
|
||||
return True
|
||||
|
||||
mock_job_manager.transition_to_state.side_effect = _transition
|
||||
|
||||
mock_encoding_service = MagicMock()
|
||||
mock_encoding_service.is_enabled = True
|
||||
@@ -226,3 +238,56 @@ def test_park_active_render_jobs_continues_after_per_job_failure():
|
||||
|
||||
# The successful job should still have been parked
|
||||
assert parked == 1
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_worker_skips_job_in_terminal_state():
|
||||
"""A stale/duplicate trigger for a finished job must never resurrect it.
|
||||
|
||||
The Cloud Run Job dispatch path bypasses the HTTP endpoint's
|
||||
_check_worker_idempotency guard, so the worker enforces this itself.
|
||||
"""
|
||||
from backend.workers import render_video_worker as rvw
|
||||
|
||||
mock_job_manager = MagicMock()
|
||||
mock_job_manager.get_job.return_value = _build_minimal_job(JobStatus.COMPLETE)
|
||||
|
||||
with patch.object(rvw, "JobManager", return_value=mock_job_manager), \
|
||||
patch.object(rvw, "StorageService"), \
|
||||
patch.object(rvw, "get_settings"), \
|
||||
patch.object(rvw, "create_job_logger", return_value=MagicMock()), \
|
||||
patch.object(rvw, "setup_job_logging", return_value=MagicMock()):
|
||||
result = await rvw.process_render_video("done-job")
|
||||
|
||||
assert result is False
|
||||
# Must not touch job state at all
|
||||
mock_job_manager.transition_to_state.assert_not_called()
|
||||
mock_job_manager.fail_job.assert_not_called()
|
||||
mock_job_manager.update_state_data.assert_not_called()
|
||||
# And must not leak a registry entry
|
||||
assert worker_registry.get_active_workers() == {}
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_worker_skips_job_already_rendering():
|
||||
"""Re-entry on a job already in RENDERING_VIDEO (duplicate trigger, or a
|
||||
Cloud Run Job max_retries re-run after a crash) must be a clean no-op —
|
||||
NOT a hard fail via the same-state InvalidStateTransition path. A live
|
||||
render must not be disrupted; a dead one is recovered by the
|
||||
rendering_video_stuck sweep."""
|
||||
from backend.workers import render_video_worker as rvw
|
||||
|
||||
mock_job_manager = MagicMock()
|
||||
mock_job_manager.get_job.return_value = _build_minimal_job(JobStatus.RENDERING_VIDEO)
|
||||
|
||||
with patch.object(rvw, "JobManager", return_value=mock_job_manager), \
|
||||
patch.object(rvw, "StorageService"), \
|
||||
patch.object(rvw, "get_settings"), \
|
||||
patch.object(rvw, "create_job_logger", return_value=MagicMock()), \
|
||||
patch.object(rvw, "setup_job_logging", return_value=MagicMock()):
|
||||
result = await rvw.process_render_video("mid-render-job")
|
||||
|
||||
assert result is False
|
||||
mock_job_manager.transition_to_state.assert_not_called()
|
||||
mock_job_manager.fail_job.assert_not_called()
|
||||
assert worker_registry.get_active_workers() == {}
|
||||
@@ -512,6 +512,82 @@ class TestWorkerService:
|
||||
assert "--job-id" in str(override_args)
|
||||
assert "test123" in str(override_args)
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_trigger_render_video_worker_routes_to_cloud_run_job(self):
|
||||
"""Test that render worker routes to Cloud Run Jobs when enabled.
|
||||
|
||||
When USE_CLOUD_RUN_JOBS_FOR_RENDER=true and ENABLE_CLOUD_TASKS=true,
|
||||
trigger_render_video_worker should dispatch a Cloud Run Job (reusing
|
||||
video-encoding-job with a render_video_worker args override) instead
|
||||
of posting to the internal HTTP endpoint. As a BackgroundTask on the
|
||||
service, the render poll was killed ~10s after SIGTERM whenever a
|
||||
deploy rollout landed mid-render, freezing the job at rendering_video
|
||||
(see incident 2026-09-13, job 41e06b90).
|
||||
"""
|
||||
from backend.services.worker_service import WorkerService, reset_worker_service
|
||||
reset_worker_service()
|
||||
|
||||
with patch('backend.services.worker_service.get_settings') as mock_settings:
|
||||
mock_settings.return_value.admin_tokens = "test-token"
|
||||
mock_settings.return_value.google_cloud_project = "test-project"
|
||||
mock_settings.return_value.enable_cloud_tasks = True
|
||||
mock_settings.return_value.gcp_region = "us-central1"
|
||||
mock_settings.return_value.use_cloud_run_jobs_for_render = True
|
||||
|
||||
mock_jobs_client = MagicMock()
|
||||
mock_operation = MagicMock()
|
||||
mock_operation.metadata = "test-metadata"
|
||||
mock_jobs_client.run_job.return_value = mock_operation
|
||||
|
||||
mock_run_v2 = MagicMock()
|
||||
mock_run_v2.JobsClient.return_value = mock_jobs_client
|
||||
mock_run_v2.RunJobRequest = MagicMock()
|
||||
mock_run_v2.RunJobRequest.Overrides = MagicMock()
|
||||
mock_run_v2.RunJobRequest.Overrides.ContainerOverride = MagicMock()
|
||||
|
||||
import google.cloud
|
||||
with patch.dict('sys.modules', {'google.cloud.run_v2': mock_run_v2}), \
|
||||
patch.object(google.cloud, 'run_v2', mock_run_v2, create=True):
|
||||
service = WorkerService()
|
||||
result = await service.trigger_render_video_worker("test456")
|
||||
|
||||
assert result is True
|
||||
mock_jobs_client.run_job.assert_called_once()
|
||||
|
||||
# Reuses the video-encoding-job Cloud Run Job resource
|
||||
call_args = mock_run_v2.RunJobRequest.call_args
|
||||
assert call_args is not None
|
||||
assert "video-encoding-job" in str(call_args)
|
||||
|
||||
# But overrides the module to the render worker with the job id
|
||||
override_args = mock_run_v2.RunJobRequest.Overrides.ContainerOverride.call_args
|
||||
assert "backend.workers.render_video_worker" in str(override_args)
|
||||
assert "--job-id" in str(override_args)
|
||||
assert "test456" in str(override_args)
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_trigger_render_video_worker_flag_off_uses_legacy_path(self):
|
||||
"""With USE_CLOUD_RUN_JOBS_FOR_RENDER off, the legacy Cloud Tasks /
|
||||
HTTP path is used and no Cloud Run Job is dispatched (rollback path)."""
|
||||
from backend.services.worker_service import WorkerService, reset_worker_service
|
||||
reset_worker_service()
|
||||
|
||||
with patch('backend.services.worker_service.get_settings') as mock_settings:
|
||||
mock_settings.return_value.admin_tokens = "test-token"
|
||||
mock_settings.return_value.google_cloud_project = "test-project"
|
||||
mock_settings.return_value.enable_cloud_tasks = True
|
||||
mock_settings.return_value.gcp_region = "us-central1"
|
||||
mock_settings.return_value.use_cloud_run_jobs_for_render = False
|
||||
|
||||
service = WorkerService()
|
||||
with patch.object(service, '_trigger_worker_cloud_run_job', new=AsyncMock()) as mock_crj, \
|
||||
patch.object(service, 'trigger_worker', new=AsyncMock(return_value=True)) as mock_legacy:
|
||||
result = await service.trigger_render_video_worker("test789")
|
||||
|
||||
assert result is True
|
||||
mock_crj.assert_not_called()
|
||||
mock_legacy.assert_called_once_with("render-video", "test789")
|
||||
|
||||
|
||||
class TestWorkerServiceWarmup:
|
||||
"""Test encoding worker warmup in trigger methods."""
|
||||
|
||||
@@ -123,7 +123,11 @@ def test_bump_worker_generation_is_best_effort():
|
||||
# render worker (GCE path) supersession behaviour
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
def _render_job(generation=1):
|
||||
def _render_job(generation=1, status=JobStatus.RENDERING_VIDEO):
|
||||
# Entry snapshots (job_start) must use REVIEW_COMPLETE — the real dispatch
|
||||
# state. A job already in RENDERING_VIDEO at entry is skipped by the
|
||||
# worker-side idempotency gate (duplicate trigger / CRJ retry protection).
|
||||
# Mid-render re-read snapshots (job_after) stay RENDERING_VIDEO.
|
||||
job = MagicMock()
|
||||
job.artist = "John Maus"
|
||||
job.title = "The Fear"
|
||||
@@ -134,7 +138,7 @@ def _render_job(generation=1):
|
||||
job.prep_only = False
|
||||
job.state_data = {"worker_generation": generation, "is_duet": False}
|
||||
job.file_urls = {}
|
||||
job.status = JobStatus.RENDERING_VIDEO
|
||||
job.status = status
|
||||
return job
|
||||
|
||||
|
||||
@@ -187,7 +191,7 @@ async def _run_gce_render(job_start, job_after, transition_side_effect=None):
|
||||
@pytest.mark.asyncio
|
||||
async def test_render_superseded_by_status_reset_does_not_fail_job():
|
||||
"""Admin reset moved the job to awaiting_review mid-render → discard, don't fail."""
|
||||
job_start = _render_job(generation=1)
|
||||
job_start = _render_job(generation=1, status=JobStatus.REVIEW_COMPLETE)
|
||||
job_after = _render_job(generation=1)
|
||||
job_after.status = "awaiting_review" # reset out from under the render
|
||||
|
||||
@@ -207,7 +211,7 @@ async def test_render_superseded_by_status_reset_does_not_fail_job():
|
||||
@pytest.mark.asyncio
|
||||
async def test_render_superseded_by_generation_bump_does_not_fail_job():
|
||||
"""A newer render was triggered (generation bumped) → discard the stale run."""
|
||||
job_start = _render_job(generation=1)
|
||||
job_start = _render_job(generation=1, status=JobStatus.REVIEW_COMPLETE)
|
||||
job_after = _render_job(generation=2) # newer run took over
|
||||
|
||||
result, jm = await _run_gce_render(job_start, job_after)
|
||||
@@ -225,7 +229,7 @@ async def test_invalid_terminal_transition_is_graceful_when_superseded():
|
||||
reset moved the job to awaiting_review). The worker must bail gracefully,
|
||||
exactly reproducing and fixing the 7f457087 incident.
|
||||
"""
|
||||
job_start = _render_job(generation=1)
|
||||
job_start = _render_job(generation=1, status=JobStatus.REVIEW_COMPLETE)
|
||||
job_after = _render_job(generation=1)
|
||||
job_after.status = "awaiting_review" # reset confirmed on re-read in the handler
|
||||
|
||||
@@ -252,7 +256,7 @@ async def test_invalid_transition_when_not_superseded_still_fails():
|
||||
workflow defect — it must still fail the job (loudly) rather than be silently
|
||||
swallowed and leave the job stuck. Guards against over-broad error suppression.
|
||||
"""
|
||||
job_start = _render_job(generation=1)
|
||||
job_start = _render_job(generation=1, status=JobStatus.REVIEW_COMPLETE)
|
||||
job_after = _render_job(generation=1) # same gen + still rendering_video => NOT superseded
|
||||
|
||||
invalid = InvalidStateTransitionError(
|
||||
@@ -272,7 +276,7 @@ async def test_progress_callback_skips_update_when_superseded():
|
||||
"""A stale render's progress ticks must not drag a reset job's progress bar."""
|
||||
from backend.workers import render_video_worker as rvw
|
||||
|
||||
job_start = _render_job(generation=1)
|
||||
job_start = _render_job(generation=1, status=JobStatus.REVIEW_COMPLETE)
|
||||
superseded = _render_job(generation=2) # reset bumped the fence mid-render
|
||||
|
||||
jm = MagicMock()
|
||||
@@ -318,7 +322,7 @@ async def test_progress_callback_skips_update_when_superseded():
|
||||
@pytest.mark.asyncio
|
||||
async def test_normal_render_still_completes():
|
||||
"""Un-superseded render must still write outputs and transition to INSTRUMENTAL_SELECTED."""
|
||||
job_start = _render_job(generation=1)
|
||||
job_start = _render_job(generation=1, status=JobStatus.REVIEW_COMPLETE)
|
||||
job_after = _render_job(generation=1) # same gen, still rendering_video
|
||||
|
||||
result, jm = await _run_gce_render(job_start, job_after)
|
||||
|
||||
@@ -123,6 +123,47 @@ async def process_render_video(job_id: str) -> bool:
|
||||
if not job:
|
||||
logger.error(f"[job:{job_id}] Job not found in Firestore")
|
||||
job_log.error(f"Job {job_id} not found in Firestore!")
|
||||
await worker_registry.unregister(job_id, "render-video")
|
||||
return False
|
||||
|
||||
# Hard idempotency gate (worker-side). The legacy HTTP endpoint enforces
|
||||
# this via _check_worker_idempotency at dispatch time, but the Cloud Run
|
||||
# Job path invokes this worker directly — and Cloud Run may re-run the
|
||||
# execution (max_retries=2) after an OOM/timeout/hard kill. The worker
|
||||
# must therefore protect itself regardless of dispatch path:
|
||||
# - terminal job: never resurrect. A stale or duplicate trigger racing
|
||||
# a job that just finished would otherwise re-render and drag a
|
||||
# complete job back to instrumental_selected.
|
||||
# - already RENDERING_VIDEO: either another execution owns the render
|
||||
# (duplicate trigger) or a previous execution died mid-render (Cloud
|
||||
# Run Job retry). Never re-enter: a live render must not be disrupted,
|
||||
# and a dead one is recovered by the rendering_video_stuck sweep →
|
||||
# re-park → auto-retry. Re-entering would hit the same-state
|
||||
# transition guard and hard-fail the job with a spurious
|
||||
# "Invalid state transition" error.
|
||||
# (Legacy path unaffected: the worker enters with status
|
||||
# REVIEW_COMPLETE — RENDERING_VIDEO is only set below, by this worker.)
|
||||
terminal_statuses = {"failed", "cancelled", "complete", "prep_complete", "error"}
|
||||
status_value = job.status.value if hasattr(job.status, "value") else str(job.status)
|
||||
if status_value in terminal_statuses:
|
||||
logger.info(
|
||||
f"[job:{job_id}] WORKER_END worker=render-video status=skipped "
|
||||
f"reason=terminal_state:{status_value}"
|
||||
)
|
||||
job_log.info(f"Job in terminal state '{status_value}' — skipping render")
|
||||
await worker_registry.unregister(job_id, "render-video")
|
||||
return False
|
||||
if job.status == JobStatus.RENDERING_VIDEO:
|
||||
logger.warning(
|
||||
f"[job:{job_id}] WORKER_END worker=render-video status=skipped "
|
||||
f"reason=already_rendering"
|
||||
)
|
||||
job_log.warning(
|
||||
"Job already in rendering_video — another execution owns this render "
|
||||
"(or a previous one died and the rendering_video_stuck sweep will "
|
||||
"recover it); not re-entering"
|
||||
)
|
||||
await worker_registry.unregister(job_id, "render-video")
|
||||
return False
|
||||
|
||||
# Capture the supersession fence at start. If an admin reset (or a newer
|
||||
@@ -334,7 +375,20 @@ async def process_render_video(job_id: str) -> bool:
|
||||
from backend.services.worker_service import get_worker_service
|
||||
worker_service = get_worker_service()
|
||||
job_log.info("Triggering video worker for final encoding...")
|
||||
await worker_service.trigger_video_worker(job_id)
|
||||
video_triggered = await worker_service.trigger_video_worker(job_id)
|
||||
if not video_triggered:
|
||||
# No exception path reaches here (trigger returns False
|
||||
# instead of raising) — surface it loudly, or the job
|
||||
# sits at instrumental_selected with no error and no
|
||||
# sweep watching that state.
|
||||
job_log.error(
|
||||
"Video worker dispatch FAILED after render complete — "
|
||||
"job stalls at instrumental_selected until manually retried"
|
||||
)
|
||||
logger.error(
|
||||
f"[job:{job_id}] Video worker dispatch failed after GCE "
|
||||
f"render complete (job stalled at instrumental_selected)"
|
||||
)
|
||||
|
||||
job_manager.update_state_data(job_id, 'render_progress', {'stage': 'complete'})
|
||||
return True
|
||||
@@ -637,7 +691,16 @@ async def process_render_video(job_id: str) -> bool:
|
||||
|
||||
worker_service = get_worker_service()
|
||||
job_log.info("Triggering video worker for final encoding...")
|
||||
await worker_service.trigger_video_worker(job_id)
|
||||
video_triggered = await worker_service.trigger_video_worker(job_id)
|
||||
if not video_triggered:
|
||||
job_log.error(
|
||||
"Video worker dispatch FAILED after render complete — "
|
||||
"job stalls at instrumental_selected until manually retried"
|
||||
)
|
||||
logger.error(
|
||||
f"[job:{job_id}] Video worker dispatch failed after local "
|
||||
f"render complete (job stalled at instrumental_selected)"
|
||||
)
|
||||
|
||||
# Mark render progress as complete for idempotency
|
||||
# This allows the worker to be re-triggered after admin reset
|
||||
@@ -843,3 +906,59 @@ def park_active_render_jobs_for_shutdown() -> int:
|
||||
|
||||
# For compatibility with worker service
|
||||
render_video_worker = process_render_video
|
||||
|
||||
|
||||
# ==================== CLI Entry Point for Cloud Run Jobs ====================
|
||||
# Allows the render worker to run as a standalone Cloud Run Job execution
|
||||
# (video-encoding-job with an args override), immune to Cloud Run Service
|
||||
# deployment rollouts. As a BackgroundTask on the service it was killed ~10s
|
||||
# after SIGTERM whenever a deploy landed mid-render (incident 2026-09-13).
|
||||
# Usage: python -m backend.workers.render_video_worker --job-id <job_id>
|
||||
|
||||
def main():
|
||||
"""CLI entry point for running the render worker as a Cloud Run Job.
|
||||
|
||||
Exit-code contract (differs from video_worker deliberately): the Cloud Run
|
||||
Job template has max_retries=2, and ``process_render_video`` handles every
|
||||
expected failure internally — it parks the job (capacity/shutdown), fails
|
||||
it with a user-facing message, or discards a superseded result. Re-running
|
||||
the worker after any of those would re-enter a job that has already moved
|
||||
to a non-render state. So a clean return (True OR False) exits 0, and only
|
||||
an escaped exception (crash before state was handled) exits 1 to request a
|
||||
Cloud Run retry.
|
||||
"""
|
||||
import argparse
|
||||
import asyncio
|
||||
import sys
|
||||
|
||||
parser = argparse.ArgumentParser(
|
||||
description="Post-review render video worker for karaoke generation"
|
||||
)
|
||||
parser.add_argument("--job-id", required=True, help="Job ID to process")
|
||||
args = parser.parse_args()
|
||||
job_id = args.job_id
|
||||
|
||||
logging.basicConfig(
|
||||
level=logging.INFO,
|
||||
format="%(asctime)s - %(name)s - %(levelname)s - %(message)s"
|
||||
)
|
||||
|
||||
logger.info(f"Starting render video worker CLI for job {job_id}")
|
||||
|
||||
try:
|
||||
success = asyncio.run(process_render_video(job_id))
|
||||
if success:
|
||||
logger.info(f"Render video completed successfully for job {job_id}")
|
||||
else:
|
||||
logger.error(
|
||||
f"Render video did not complete for job {job_id} "
|
||||
"(job state already handled by the worker — parked, failed, or superseded)"
|
||||
)
|
||||
sys.exit(0)
|
||||
except Exception as e:
|
||||
logger.error(f"Render video worker crashed: {e}", exc_info=True)
|
||||
sys.exit(1)
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
main()
|
||||
@@ -1883,3 +1883,60 @@ gated to deployed envs, skips test/dev.
|
||||
- Detect silent *partial success* at the **publish boundary** (what *should* have
|
||||
shipped vs what did), and make the metadata backstop validate a just-published
|
||||
track's **own** completeness same-run, not only via global-max gap math (D2).
|
||||
|
||||
## BackgroundTasks Cannot Survive Deploys — And the Shutdown Hook That "Protects" Them Never Runs (Sep 2026, v0.224.0)
|
||||
|
||||
**Context:** Tenant-E2E run #107 failed: job `41e06b90` froze at `rendering_video`
|
||||
even though the GCE encoder finished the render and uploaded every output. A backend
|
||||
deploy landed mid-render. The render worker ran as a FastAPI BackgroundTask on the
|
||||
Cloud Run service; the deploy retired the old revision, SIGTERM arrived, and its last
|
||||
log lines were uvicorn's `Shutting down` / `Waiting for background tasks to complete`
|
||||
— then nothing. Nobody ever polled the encoder again.
|
||||
|
||||
**The load-bearing wrong belief:** `main.py`'s lifespan shutdown hook (worker-registry
|
||||
wait + `park_active_render_jobs_for_shutdown()`) was built for exactly this scenario
|
||||
and commented "Cloud Run gen2 default termination grace period is 600s. We wait up to
|
||||
480s…". Both halves were wrong:
|
||||
|
||||
1. **Cloud Run services SIGKILL ~10 seconds after SIGTERM**, not 600s.
|
||||
2. **Uvicorn runs lifespan shutdown only AFTER all background tasks complete.** A
|
||||
render task with 30 minutes left keeps uvicorn stuck in its bg-task wait until
|
||||
SIGKILL — so the parking hook is architecturally unreachable in the one scenario
|
||||
it exists for. It had never once fired in anger.
|
||||
|
||||
Three other layers also failed to save the job, each for a structural reason:
|
||||
- **Cloud Tasks retry** — defeated by design: the worker endpoint returns 200
|
||||
immediately and does the work post-response, so the queue considers the task done
|
||||
seconds after dispatch (`dispatch_deadline=1800s` protects nothing).
|
||||
- **`recover-stuck-jobs`** — keyed on `updated_at` staleness > 45 min; correct as a
|
||||
slow safety net, but slower than the tenant E2E's 30-min render budget and a
|
||||
~50-60 min stall + wasted re-render for a real customer.
|
||||
- **Worker-side completion** — the GCE encoder finished and uploaded to GCS, but
|
||||
results are pull-only (backend polls); there is no completion push, so a dead
|
||||
poller means a done-but-unrecorded render.
|
||||
|
||||
**Fix:** run the render worker as a **Cloud Run Job** (reuse `video-encoding-job`
|
||||
with a `render_video_worker` args override; flag `USE_CLOUD_RUN_JOBS_FOR_RENDER`,
|
||||
mirroring the 2026-03 `USE_CLOUD_RUN_JOBS_FOR_VIDEO` fix for the identical incident
|
||||
one stage later in the pipeline). Job executions run to completion; service deploys
|
||||
can't touch them. The job env also sets both `USE_CLOUD_RUN_JOBS_FOR_*` flags so the
|
||||
render execution's in-process trigger of the video worker dispatches a Cloud Run Job
|
||||
too instead of regressing to the vulnerable path.
|
||||
|
||||
**Principles for next time:**
|
||||
- **Any work that outlives its HTTP response on Cloud Run is presumed dead on every
|
||||
deploy.** BackgroundTasks are only acceptable for work that completes in seconds.
|
||||
Long orchestration belongs in Cloud Run Jobs (audio-download, lyrics, separation,
|
||||
video encode, and now render all migrated for this exact reason — when one worker
|
||||
gets this fix, audit the *other* workers for the same pattern; render sat
|
||||
vulnerable for 6 months after video was fixed).
|
||||
- **A graceful-shutdown hook is untested until it has provably fired.** Uvicorn's
|
||||
shutdown ordering (bg tasks → lifespan) plus Cloud Run's 10s kill window made ours
|
||||
dead code. If a safety net matters, verify it in logs during a real deploy.
|
||||
- **Returning 200-then-working silently discards the queue's retry semantics.** If
|
||||
you accept work via Cloud Tasks, either do the work in-request or make sure some
|
||||
other mechanism owns retries.
|
||||
- **Exit codes are API when Cloud Run Jobs retry:** `process_render_video` handles
|
||||
its own failures (park/fail/supersede), so the CLI exits 0 on any clean return and
|
||||
1 only on an escaped crash — otherwise `max_retries=2` would re-enter jobs that
|
||||
already moved to a non-render state.
|
||||
@@ -325,6 +325,8 @@ gcloud compute instances describe encoding-worker-a \
|
||||
|
||||
**Auto-recovery (v0.192.3+):** `recover-stuck-jobs` (every 5 min) now flags a job in `rendering_video` with no progress for **>45 min** (`rendering_video_stuck`) and re-parks it into `render_pending_capacity`, so the existing `retry-pending-render-jobs` cron resets it to `review_complete` and re-renders (same 24h ceiling). No manual action needed.
|
||||
|
||||
**Root-cause fix (v0.224.0+):** the dominant cause of this state was **backend deploys**: the render worker ran as a FastAPI BackgroundTask on the Cloud Run *service*, and every deploy rollout that landed mid-render SIGKILLed it ~10s after SIGTERM (the lifespan "park on shutdown" hook never runs in that scenario — uvicorn waits for background tasks *before* running lifespan shutdown, so it dies waiting; incident 2026-09-13, job `41e06b90`, tenant-E2E run #107). Since v0.224.0 the render worker runs as a **Cloud Run Job** (`video-encoding-job` with a `render_video_worker` args override, flag `USE_CLOUD_RUN_JOBS_FOR_RENDER`), which runs to completion regardless of service deploys — same fix the video worker got in 2026-03 (`USE_CLOUD_RUN_JOBS_FOR_VIDEO`). The 45-min sweep remains as the safety net for the remaining causes (job execution crash/OOM/timeout, lost GCE VM). Rollback: remove `USE_CLOUD_RUN_JOBS_FOR_RENDER=true` from the service env to fall back to the legacy Cloud Tasks path.
|
||||
|
||||
**Manual unstick (if needed):** don't re-trigger the render worker while status is `rendering_video` (it starts mid-state and fails with "Invalid state transition"). Instead let it fail (or re-park it), then use the admin **Retry** which resumes cleanly from `review_complete`. If a ghost `render_progress.stage='running'` blocks re-trigger, clear it first (`update({'state_data.render_progress': {'stage': 'pending'}})`).
|
||||
|
||||
---
|
||||
|
||||
@@ -0,0 +1,54 @@
|
||||
# Investigation: Tenant E2E #107 failure — deploy killed render poll, job froze at `rendering_video`
|
||||
|
||||
**Date:** 2026-09-13
|
||||
**Incident:** Tenant Portals E2E run #107 (`singa`) failed `job reached complete before render timeout` (expected `complete`, got `rendering_video`). Job `41e06b90`.
|
||||
**Fix:** v0.224.0 — render worker migrated to Cloud Run Jobs (`USE_CLOUD_RUN_JOBS_FOR_RENDER`).
|
||||
|
||||
## Timeline (all UTC, 2026-09-13)
|
||||
|
||||
| Time | Event |
|
||||
|------|-------|
|
||||
| 04:58:44 | New backend revision `karaoke-backend-00973-xac` starts (deploy in progress) |
|
||||
| 04:58:52 | Review complete → Cloud Task created for render-video worker (`dispatch_deadline=1800s`) |
|
||||
| 04:58:53 | Cloud Task POST hits **old** revision `00971-xad`; endpoint returns 200 immediately, `process_render_video` continues as FastAPI BackgroundTask; job → `rendering_video`; render submitted to GCE `encoding-worker-a` |
|
||||
| 04:59:03 | Backend records "Encoding progress: 55%" (last Firestore write, `updated_at` freezes here) |
|
||||
| 04:59:13.867 | Old revision gets SIGTERM: uvicorn logs `Shutting down` then `Waiting for background tasks to complete.` — **its last-ever log lines** |
|
||||
| ~04:59:24 | SIGKILL (Cloud Run gives ~10s, not the 600s the code comment claimed). Lifespan shutdown hook (worker-registry wait + `park_active_render_jobs_for_shutdown`) **never ran** |
|
||||
| 04:59:57 | GHA runner logs "Deploy - Backend (Cloud Run) completed: Succeeded" |
|
||||
| 05:00:05 | **GCE worker completes the render** and uploads `with_vocals.mkv` + karaoke.ass/lrc/corrected.txt to GCS. Nobody polls it again; result never recorded |
|
||||
| 05:00, 05:05, … | `RECOVER_STUCK_JOBS` runs every 5 min on the new revision: `render_retriggered=0` — `rendering_video_stuck` needs `updated_at` stale > **45 min** |
|
||||
| 05:15:10 | `encoding-worker-idle-shutdown` correctly stops the now-idle VM (confirmed `active_jobs=0` first — its fail-safe worked; red herring) |
|
||||
| 05:30:06 | E2E 30-min render deadline expires; test cleanup deletes job, refunds credit; run #107 fails |
|
||||
|
||||
Recovery *would* have fired at ~05:44 (45-min staleness) and re-rendered from scratch. Real-customer impact of this class: ~50-60 min silent stall + a wasted re-render, on every deploy that lands mid-render (deploys go out on every merge to main).
|
||||
|
||||
## Root cause chain (4 independent layers, each structurally broken for this case)
|
||||
|
||||
1. **BackgroundTask execution** — the render endpoint returns 200 immediately and polls the GCE worker post-response. Cloud Run kills old-revision instances ~10s after SIGTERM on every deploy; a mid-render poll loop cannot survive.
|
||||
2. **Unreachable shutdown parking** — uvicorn's shutdown sequence waits for background tasks *before* running FastAPI lifespan shutdown. With a long render task in flight, SIGKILL always arrives first, so `park_active_render_jobs_for_shutdown()` is dead code in exactly the scenario it was written for. The comment justifying it ("600s grace, wait 480s") was factually wrong.
|
||||
3. **Cloud Tasks retry defeated** — the queue got its 200 at 04:58:53; `dispatch_deadline` protects nothing when the handler is fire-and-forget.
|
||||
4. **Slow watchdog** — `rendering_video_stuck` = `updated_at` stale > 45 min. Correct as a last-ditch net, but the only net that was actually reachable, and slower than both the E2E budget and any reasonable customer expectation.
|
||||
|
||||
## Why this was a known-class bug
|
||||
|
||||
`docs/archive/2026-03-08-fix-encoding-deploy-survival.md` diagnosed the **identical** failure for the *video worker* (final encode) — same BackgroundTask pattern, same deploy-kill, same conclusion — and fixed it by routing through the `video-encoding-job` Cloud Run Job (`USE_CLOUD_RUN_JOBS_FOR_VIDEO=true`, live since March). The render worker — one pipeline stage earlier, same shape of work — was never migrated. Related-but-distinct prior fixes that did NOT cover this: GCE-worker drain on deploy (protects the *encoder*, not the orchestrator), poll pinning v0.184.2 (protects against *worker* primary swaps), `recover-stuck-jobs` re-park v0.192.3 (the 45-min net that was too slow here).
|
||||
|
||||
## Fix (v0.224.0)
|
||||
|
||||
- `trigger_render_video_worker` routes to a Cloud Run Job execution — reusing `video-encoding-job` with args override `python -m backend.workers.render_video_worker --job-id <id>` — when `USE_CLOUD_RUN_JOBS_FOR_RENDER=true` and Cloud Tasks enabled. Job executions run to completion; deploys update the job template image but never touch running executions.
|
||||
- New CLI entrypoint in `render_video_worker.py`. Exit-code contract differs from video_worker deliberately: clean returns (True *and* False) exit 0 because `process_render_video` already parked/failed/superseded the job internally; only an escaped crash exits 1 (job template has `max_retries=2`).
|
||||
- Pulumi: `video-encoding-job` env gains `USE_CLOUD_RUN_JOBS_FOR_VIDEO=true` + `USE_CLOUD_RUN_JOBS_FOR_RENDER=true` so the render execution's in-process `trigger_video_worker` also dispatches a Cloud Run Job instead of regressing to the vulnerable path.
|
||||
- ci.yml: service deploy env gains `USE_CLOUD_RUN_JOBS_FOR_RENDER=true`.
|
||||
- `main.py` lifespan comment corrected (10s reality, uvicorn ordering) and the pointless 480s wait shrunk to 5s; the parking hook remains as a legacy/flag-off net.
|
||||
- 45-min `rendering_video_stuck` sweep unchanged — still the net for execution crash/OOM/timeout and lost-VM cases.
|
||||
|
||||
**Rollback:** remove `USE_CLOUD_RUN_JOBS_FOR_RENDER=true` from the service env (falls back to the legacy Cloud Tasks → BackgroundTask path).
|
||||
|
||||
## Deliberately not done (and why)
|
||||
|
||||
- **Atomic (transactional) REVIEW_COMPLETE → RENDERING_VIDEO claim** (CodeRabbit finding on PR #994): two executions starting within the same sub-second window could both pass the worker-side gate and double-render. The identical read-then-write window exists in the legacy endpoint's `_check_worker_idempotency` (check stage → mark 'running' is not transactional), the worker-generation fence guarantees the stale result is discarded so final state is always correct, and the consequence is bounded wasted compute on a rare² race (requires two dispatches for one job AND <1s start skew). A conditional-write claim belongs in `JobManager`/`FirestoreService.update_job_status` — shared machinery for every worker — and deserves its own change, not a rider on this one.
|
||||
|
||||
- **Adopt-completed-render on recovery** (query the pinned worker's `/status` during the stuck sweep and harvest finished outputs instead of re-rendering): meaningful only for the deploy-kill case this PR eliminates at the source; remaining orphan causes mostly kill the GCE worker too. Skipped to keep the change focused.
|
||||
- **Render-progress heartbeats + faster stuck threshold**: same reasoning — with the poller in a Cloud Run Job, poller death becomes rare enough that the 45-min net is acceptable.
|
||||
- **uvicorn `--timeout-graceful-shutdown` + CancelledError parking**: would make the lifespan hook genuinely reachable, but it's belt-and-braces once renders no longer run on the service; adds signal-handling risk to every request path.
|
||||
- **Same audit for `lyrics`/`screens`/`audio` BackgroundTasks**: lyrics + separation + audio-download already run as Cloud Run Jobs; screens/preview tasks are short. Noted in LESSONS-LEARNED as the pattern to check when touching those workers.
|
||||
@@ -668,6 +668,21 @@ def create_video_encoding_job(
|
||||
name="USE_GCE_ENCODING",
|
||||
value="true",
|
||||
),
|
||||
# The render worker also runs in this job (args
|
||||
# override → backend.workers.render_video_worker)
|
||||
# and triggers the video worker in-process when the
|
||||
# render completes. Without these flags that
|
||||
# downstream trigger would fall back to the
|
||||
# deploy-vulnerable Cloud Tasks → BackgroundTask
|
||||
# path this job exists to avoid.
|
||||
cloudrunv2.JobTemplateTemplateContainerEnvArgs(
|
||||
name="USE_CLOUD_RUN_JOBS_FOR_VIDEO",
|
||||
value="true",
|
||||
),
|
||||
cloudrunv2.JobTemplateTemplateContainerEnvArgs(
|
||||
name="USE_CLOUD_RUN_JOBS_FOR_RENDER",
|
||||
value="true",
|
||||
),
|
||||
],
|
||||
)
|
||||
],
|
||||
|
||||
+1
-1
@@ -1,6 +1,6 @@
|
||||
[tool.poetry]
|
||||
name = "karaoke-gen"
|
||||
version = "0.223.2"
|
||||
version = "0.224.0"
|
||||
description = "Generate karaoke videos with synchronized lyrics. Handles the entire process from downloading audio and lyrics to creating the final video with title screens."
|
||||
authors = ["Andrew Beveridge <andrew@beveridge.uk>"]
|
||||
license = "MIT"
|
||||
|
||||
Reference in new issue
Block a user