Compare commits

...
Author SHA1 Message Date
Andrew BeveridgeandClaude Fable 5 5222be688a docs: record deferred atomic-claim follow-up (CodeRabbit review)
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-09-13 02:47:42 -04:00
Andrew BeveridgeandClaude Fable 5 42c6874257 fix: worker-side idempotency gate + surface failed video-worker dispatch
Review findings on the Cloud Run Job migration:

1. The CRJ dispatch path bypasses the HTTP endpoint's
   _check_worker_idempotency guard, and Cloud Run may re-run an execution
   (max_retries=2) after an OOM/timeout/hard kill. Re-entering
   process_render_video on a job already in RENDERING_VIDEO hit the
   same-state transition guard and hard-failed the job with a spurious
   "Invalid state transition" — worse, a stale trigger racing a job that
   just finished could resurrect a terminal job. The worker now protects
   itself regardless of dispatch path: terminal states and RENDERING_VIDEO
   are skipped cleanly (live renders undisturbed; dead ones recovered by
   the rendering_video_stuck sweep). Legacy path unaffected — the worker
   enters with REVIEW_COMPLETE.

2. trigger_video_worker's bool return was silently discarded after render
   completion (both GCE and local paths); a failed CRJ dispatch left the
   job stalled at instrumental_selected with no error anywhere. Now logged
   at ERROR so the error monitor sees it.

Also fixes a pre-existing registry leak on the job-not-found early return,
and updates test fixtures to enter the worker in REVIEW_COMPLETE (the real
dispatch state) instead of RENDERING_VIDEO.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-09-13 02:35:03 -04:00
Andrew BeveridgeandClaude Fable 5 2cc68a0bba fix: run render-video worker as Cloud Run Job to survive backend deploys
Tenant E2E #107 (job 41e06b90): a deploy landed mid-render, Cloud Run
SIGKILLed the old instance ~10s after SIGTERM, and the render poll — a
FastAPI BackgroundTask — died silently. The GCE encoder finished the
render and uploaded all outputs, but nobody polled it again; the job
froze at rendering_video until the 45-min stuck-job sweep (past the
E2E's 30-min budget, and a ~50-60 min stall + wasted re-render for a
real customer).

The lifespan "park on shutdown" hook never runs in this scenario:
uvicorn waits for background tasks BEFORE running lifespan shutdown, so
with a long render in flight it dies in that wait. Its comment claimed
a 600s grace period; Cloud Run services give 10s.

Fix — same shape as the 2026-03-08 video-worker fix (which this worker
missed): route trigger_render_video_worker through the existing
video-encoding-job Cloud Run Job with a render_video_worker args
override, behind USE_CLOUD_RUN_JOBS_FOR_RENDER (enabled in the deploy
env). Job executions run to completion; service deploys never touch
them.

- render_video_worker: new CLI entrypoint. Exit-code contract differs
  from video_worker deliberately: clean returns (True AND False) exit 0
  because process_render_video parks/fails/supersedes internally; only
  an escaped crash exits 1 (job template has max_retries=2).
- Pulumi: video-encoding-job env gains both USE_CLOUD_RUN_JOBS_FOR_*
  flags so the render execution's in-process video-worker trigger also
  dispatches a Cloud Run Job (applied to prod ahead of merge).
- main.py: corrected the false 600s comment; shrank the unreachable
  480s lifespan wait to 5s. Parking hook kept as legacy/flag-off net.
- 45-min rendering_video_stuck sweep unchanged (net for crash/OOM/
  timeout/lost-VM).

Rollback: drop USE_CLOUD_RUN_JOBS_FOR_RENDER=true from the service env.

Investigation: docs/archive/2026-09-13-render-deploy-survival-investigation.md

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-09-13 02:16:47 -04:00
15 changed files with 523 additions and 23 deletions

No files matched your search

+1 -1
View File
@@ -1994,7 +1994,7 @@ jobs:
--min-instances 1 \
--vpc-connector projects/nomadkaraoke/locations/us-central1/connectors/cloud-run-connector \
--vpc-egress private-ranges-only \
--set-env-vars "GOOGLE_CLOUD_PROJECT=${{ env.PROJECT_ID }},GCS_BUCKET_NAME=karaoke-gen-storage-${{ env.PROJECT_ID }},FIRESTORE_COLLECTION=jobs,ENVIRONMENT=production,KARAOKE_GEN_VERSION=$VERSION,DEFAULT_DROPBOX_PATH=/MediaUnsynced/Karaoke/Tracks-Organized,DEFAULT_GDRIVE_FOLDER_ID=1laRKAyxo0v817SstfM5XkpbWiNKNAMSX,ENABLE_CLOUD_TASKS=true,CLOUD_RUN_SERVICE_URL=https://api.nomadkaraoke.com,GCP_REGION=us-central1,DEFAULT_ENABLE_YOUTUBE_UPLOAD=true,DEFAULT_BRAND_PREFIX=NOMAD,DEFAULT_PRIVATE_DROPBOX_PATH=/MediaUnsynced/Karaoke/Tracks-NonPublished,DEFAULT_PRIVATE_BRAND_PREFIX=NOMADNP,USE_GCE_ENCODING=true,USE_GCE_PREVIEW_ENCODING=true,ENABLE_PUSH_NOTIFICATIONS=true,USE_CLOUD_RUN_JOBS_FOR_VIDEO=true,EMAIL_PROVIDER=postmark,AUTO_CORRECT_MODEL=claude-opus-4-8,AUTO_CORRECT_COMPARE_MODELS=claude-opus-4-8;gemini-3.1-pro-preview,AUTO_CORRECT_PROACTIVE_ENABLED=true,AUTO_APPROVAL_BACKING_KEEP_ENABLED=true,COMMUNITY_DAILY_PICK_ENABLED=true,EDGE_AUTH_MODE=enforce,COMMIT_SHA=${{ steps.build-meta.outputs.sha }},PR_NUMBER=${{ steps.build-meta.outputs.pr_number }},PR_TITLE=${{ steps.build-meta.outputs.pr_title }}" \
--set-env-vars "GOOGLE_CLOUD_PROJECT=${{ env.PROJECT_ID }},GCS_BUCKET_NAME=karaoke-gen-storage-${{ env.PROJECT_ID }},FIRESTORE_COLLECTION=jobs,ENVIRONMENT=production,KARAOKE_GEN_VERSION=$VERSION,DEFAULT_DROPBOX_PATH=/MediaUnsynced/Karaoke/Tracks-Organized,DEFAULT_GDRIVE_FOLDER_ID=1laRKAyxo0v817SstfM5XkpbWiNKNAMSX,ENABLE_CLOUD_TASKS=true,CLOUD_RUN_SERVICE_URL=https://api.nomadkaraoke.com,GCP_REGION=us-central1,DEFAULT_ENABLE_YOUTUBE_UPLOAD=true,DEFAULT_BRAND_PREFIX=NOMAD,DEFAULT_PRIVATE_DROPBOX_PATH=/MediaUnsynced/Karaoke/Tracks-NonPublished,DEFAULT_PRIVATE_BRAND_PREFIX=NOMADNP,USE_GCE_ENCODING=true,USE_GCE_PREVIEW_ENCODING=true,ENABLE_PUSH_NOTIFICATIONS=true,USE_CLOUD_RUN_JOBS_FOR_VIDEO=true,USE_CLOUD_RUN_JOBS_FOR_RENDER=true,EMAIL_PROVIDER=postmark,AUTO_CORRECT_MODEL=claude-opus-4-8,AUTO_CORRECT_COMPARE_MODELS=claude-opus-4-8;gemini-3.1-pro-preview,AUTO_CORRECT_PROACTIVE_ENABLED=true,AUTO_APPROVAL_BACKING_KEEP_ENABLED=true,COMMUNITY_DAILY_PICK_ENABLED=true,EDGE_AUTH_MODE=enforce,COMMIT_SHA=${{ steps.build-meta.outputs.sha }},PR_NUMBER=${{ steps.build-meta.outputs.pr_number }},PR_TITLE=${{ steps.build-meta.outputs.pr_title }}" \
--set-secrets "ANTHROPIC_API_KEY=anthropic-api-key:latest,AUDIOSHAKE_API_TOKEN=audioshake-api-key:latest,GENIUS_API_TOKEN=genius-api-key:latest,SPOTIFY_COOKIE_SP_DC=spotify-cookie:latest,RAPIDAPI_KEY=rapidapi-key:latest,DEFAULT_DISCORD_WEBHOOK_URL=discord-releases-webhook:latest,ADMIN_TOKENS=admin-tokens:latest,RED_API_KEY=red-api-key:latest,RED_API_URL=red-api-url:latest,OPS_API_KEY=ops-api-key:latest,OPS_API_URL=ops-api-url:latest,FLACFETCH_API_URL=flacfetch-api-url:latest,FLACFETCH_API_KEY=flacfetch-api-key:latest,POSTMARK_SERVER_TOKEN=postmark-server-token:latest,STRIPE_SECRET_KEY=stripe-secret-key:latest,STRIPE_WEBHOOK_SECRET=stripe-webhook-secret:latest,LANGFUSE_PUBLIC_KEY=langfuse-public-key:latest,LANGFUSE_SECRET_KEY=langfuse-secret-key:latest,LANGFUSE_HOST=langfuse-host:latest,ENCODING_WORKER_URL=encoding-worker-url:latest,ENCODING_WORKER_API_KEY=encoding-worker-api-key:latest,ENCODING_WORKER_FALLBACK_VMS=encoding-worker-fallback-vms:latest,VAPID_PUBLIC_KEY=vapid-public-key:latest,VAPID_PRIVATE_KEY=vapid-private-key:latest,E2E_BYPASS_KEY=e2e-bypass-key:latest,GDRIVE_VALIDATOR_URL=gdrive-validator-url:latest,EDGE_ORIGIN_SECRET=edge-origin-secret:latest"
# R3/R4 — smoke the tagged candidate revision BEFORE it serves any traffic.
+9
View File
@@ -200,6 +200,15 @@ class Settings(BaseSettings):
# Default is false - Cloud Tasks is sufficient for most videos (15-20 min).
use_cloud_run_jobs_for_video: bool = os.getenv("USE_CLOUD_RUN_JOBS_FOR_VIDEO", "false").lower() in ("true", "1", "yes")
# When enabled AND enable_cloud_tasks is true, the post-review render worker
# also runs as a Cloud Run Job (reusing video-encoding-job with an args
# override). As a BackgroundTask on the service it was killed by every
# deploy rollout that landed mid-render — Cloud Run gives instances only
# 10s after SIGTERM, far less than a render takes, so the job froze at
# rendering_video until the 45-min stuck-job sweep (incident 2026-09-13,
# job 41e06b90; same class as incident 2026-03-08 for the video worker).
use_cloud_run_jobs_for_render: bool = os.getenv("USE_CLOUD_RUN_JOBS_FOR_RENDER", "false").lower() in ("true", "1", "yes")
# GCE Encoding Worker (for high-performance video encoding)
# When enabled, video encoding is offloaded to a dedicated C4 GCE instance
# with faster CPU (Intel Granite Rapids 3.9 GHz) instead of Cloud Run.
+17 -8
View File
@@ -111,19 +111,28 @@ async def lifespan(app: FastAPI):
yield
# Shutdown - wait for any active workers to complete before terminating
# This prevents Cloud Run from killing workers mid-processing.
# Shutdown - best-effort parking of any still-registered workers.
#
# Cloud Run gen2 default termination grace period is 600s. We wait up to
# 480s, leaving ~120s for the cleanup pass below to write Firestore state
# before SIGKILL. Without that headroom, render jobs that miss the wait
# would sit in `rendering_video` forever waiting for an operator.
# Reality check (incident 2026-09-13, job 41e06b90): Cloud Run services
# SIGKILL instances ~10 seconds after SIGTERM — NOT the 600s this comment
# previously claimed — and uvicorn only runs this lifespan-shutdown code
# AFTER all BackgroundTasks have completed ("Waiting for background tasks
# to complete"). A long-running worker task therefore keeps uvicorn stuck
# in that wait until SIGKILL, and this hook never executes at all. It can
# NOT be relied on to protect in-flight renders; that protection comes
# from running them as Cloud Run Jobs instead (USE_CLOUD_RUN_JOBS_FOR_RENDER,
# mirroring USE_CLOUD_RUN_JOBS_FOR_VIDEO from incident 2026-03-08).
#
# This hook is kept as a cheap safety net for the legacy flag-off path and
# for the narrow case where workers finish right as SIGTERM lands. The wait
# is capped well inside the 10s kill window so the parking pass below still
# gets a chance to write Firestore state.
logger.info("Shutdown requested, checking for active workers...")
if worker_registry.has_active_workers():
active = worker_registry.get_active_workers()
logger.info(f"Active workers found: {active}")
logger.info("Waiting for workers to complete (timeout: 480s)...")
completed = await worker_registry.wait_for_completion(timeout=480)
logger.info("Waiting for workers to complete (timeout: 5s)...")
completed = await worker_registry.wait_for_completion(timeout=5)
if not completed:
logger.error(
"Shutdown timeout - some workers may not have completed cleanly. "
+17 -1
View File
@@ -690,9 +690,25 @@ class WorkerService:
return False
async def trigger_render_video_worker(self, job_id: str) -> bool:
"""Trigger render video worker (post-review)."""
"""
Trigger render video worker (post-review).
When USE_CLOUD_RUN_JOBS_FOR_RENDER=true and ENABLE_CLOUD_TASKS=true,
runs as a Cloud Run Job (reusing video-encoding-job with an args
override) so the render orchestration survives Cloud Run Service
deployment rollouts. As a BackgroundTask it was killed ~10s after
SIGTERM whenever a deploy landed mid-render, freezing the job at
rendering_video (incident 2026-09-13, job 41e06b90 — same class as
incident 2026-03-08 which migrated the video worker to Cloud Run Jobs).
"""
self._bump_worker_generation(job_id)
self._warmup_encoding_worker(job_id)
if self._use_cloud_tasks and self.settings.use_cloud_run_jobs_for_render:
return await self._trigger_worker_cloud_run_job(
job_id=job_id,
cloud_run_job_name="video-encoding-job",
worker_module="render_video_worker",
)
return await self.trigger_worker("render-video", job_id)
def _bump_worker_generation(self, job_id: str) -> None:
@@ -0,0 +1,71 @@
"""
Tests for the render video worker CLI entry point (Cloud Run Jobs execution).
The render worker runs as a Cloud Run Job (video-encoding-job with an args
override) so it survives Cloud Run Service deployment rollouts — as a
BackgroundTask it was killed ~10s after SIGTERM whenever a deploy landed
mid-render (incident 2026-09-13, job 41e06b90).
Exit-code contract (deliberately different from video_worker): the Cloud Run
Job template has max_retries=2 and process_render_video handles every expected
failure internally (park / fail_job / supersede-discard). A clean return —
True OR False — must exit 0 so Cloud Run never re-runs the worker against a
job that already moved to a non-render state. Only an escaped exception
(crash before state was handled) exits 1 to request a Cloud Run retry.
"""
import sys
from unittest.mock import AsyncMock, MagicMock, patch
import pytest
from backend.workers.render_video_worker import main
def _run_main_with_args(job_id="job123"):
return patch.object(sys, "argv", ["render_video_worker", "--job-id", job_id])
class TestRenderVideoWorkerCli:
def test_main_exits_zero_on_success(self):
with _run_main_with_args(), \
patch("backend.workers.render_video_worker.process_render_video",
new=AsyncMock(return_value=True)):
with pytest.raises(SystemExit) as exc_info:
main()
assert exc_info.value.code == 0
def test_main_exits_zero_on_handled_failure(self):
"""False means the worker already parked/failed/superseded the job —
a Cloud Run retry would re-enter a job in a non-render state."""
with _run_main_with_args(), \
patch("backend.workers.render_video_worker.process_render_video",
new=AsyncMock(return_value=False)):
with pytest.raises(SystemExit) as exc_info:
main()
assert exc_info.value.code == 0
def test_main_exits_nonzero_on_crash(self):
"""An escaped exception means job state was NOT handled — exit 1 so
Cloud Run's max_retries can re-run the worker."""
with _run_main_with_args(), \
patch("backend.workers.render_video_worker.process_render_video",
new=AsyncMock(side_effect=RuntimeError("boom"))):
with pytest.raises(SystemExit) as exc_info:
main()
assert exc_info.value.code == 1
def test_main_passes_job_id(self):
mock_worker = AsyncMock(return_value=True)
with _run_main_with_args("abc999"), \
patch("backend.workers.render_video_worker.process_render_video",
new=mock_worker):
with pytest.raises(SystemExit):
main()
mock_worker.assert_awaited_once_with("abc999")
def test_main_requires_job_id(self):
with patch.object(sys, "argv", ["render_video_worker"]):
with pytest.raises(SystemExit) as exc_info:
main()
# argparse exits 2 on missing required argument
assert exc_info.value.code == 2
@@ -104,6 +104,9 @@ class TestIsDuetPropagation:
job.style_assets = {}
job.prep_only = False
job.subtitle_offset_ms = 0
# Real dispatch state — the worker itself transitions to
# RENDERING_VIDEO (and skips jobs already in it / terminal).
job.status = JobStatus.REVIEW_COMPLETE
return job
def test_is_duet_true_passed_to_output_config_local_path(self):
@@ -23,7 +23,11 @@ from backend.models.job import JobStatus
from backend.workers.registry import worker_registry
def _build_minimal_job(status=JobStatus.RENDERING_VIDEO):
def _build_minimal_job(status=JobStatus.REVIEW_COMPLETE):
# Default matches the real entry state: the worker is dispatched with the
# job in REVIEW_COMPLETE and transitions to RENDERING_VIDEO itself. A job
# already in RENDERING_VIDEO at entry is skipped by the worker-side
# idempotency gate (duplicate trigger / Cloud Run Job retry protection).
job = MagicMock()
job.artist = "Test Artist"
job.title = "Test Title"
@@ -61,7 +65,15 @@ async def test_worker_registers_and_unregisters_on_success():
mock_job_manager = MagicMock()
mock_job_manager.get_job.return_value = job
mock_job_manager.transition_to_state.return_value = True
# Mirror real transitions on the shared mock so the supersession fence's
# mid-render re-reads see RENDERING_VIDEO (a frozen REVIEW_COMPLETE would
# read as "status reset" and discard the render).
def _transition(job_id=None, new_status=None, **kwargs):
job.status = new_status
return True
mock_job_manager.transition_to_state.side_effect = _transition
mock_encoding_service = MagicMock()
mock_encoding_service.is_enabled = True
@@ -226,3 +238,56 @@ def test_park_active_render_jobs_continues_after_per_job_failure():
# The successful job should still have been parked
assert parked == 1
@pytest.mark.asyncio
async def test_worker_skips_job_in_terminal_state():
"""A stale/duplicate trigger for a finished job must never resurrect it.
The Cloud Run Job dispatch path bypasses the HTTP endpoint's
_check_worker_idempotency guard, so the worker enforces this itself.
"""
from backend.workers import render_video_worker as rvw
mock_job_manager = MagicMock()
mock_job_manager.get_job.return_value = _build_minimal_job(JobStatus.COMPLETE)
with patch.object(rvw, "JobManager", return_value=mock_job_manager), \
patch.object(rvw, "StorageService"), \
patch.object(rvw, "get_settings"), \
patch.object(rvw, "create_job_logger", return_value=MagicMock()), \
patch.object(rvw, "setup_job_logging", return_value=MagicMock()):
result = await rvw.process_render_video("done-job")
assert result is False
# Must not touch job state at all
mock_job_manager.transition_to_state.assert_not_called()
mock_job_manager.fail_job.assert_not_called()
mock_job_manager.update_state_data.assert_not_called()
# And must not leak a registry entry
assert worker_registry.get_active_workers() == {}
@pytest.mark.asyncio
async def test_worker_skips_job_already_rendering():
"""Re-entry on a job already in RENDERING_VIDEO (duplicate trigger, or a
Cloud Run Job max_retries re-run after a crash) must be a clean no-op —
NOT a hard fail via the same-state InvalidStateTransition path. A live
render must not be disrupted; a dead one is recovered by the
rendering_video_stuck sweep."""
from backend.workers import render_video_worker as rvw
mock_job_manager = MagicMock()
mock_job_manager.get_job.return_value = _build_minimal_job(JobStatus.RENDERING_VIDEO)
with patch.object(rvw, "JobManager", return_value=mock_job_manager), \
patch.object(rvw, "StorageService"), \
patch.object(rvw, "get_settings"), \
patch.object(rvw, "create_job_logger", return_value=MagicMock()), \
patch.object(rvw, "setup_job_logging", return_value=MagicMock()):
result = await rvw.process_render_video("mid-render-job")
assert result is False
mock_job_manager.transition_to_state.assert_not_called()
mock_job_manager.fail_job.assert_not_called()
assert worker_registry.get_active_workers() == {}
+76
View File
@@ -512,6 +512,82 @@ class TestWorkerService:
assert "--job-id" in str(override_args)
assert "test123" in str(override_args)
@pytest.mark.asyncio
async def test_trigger_render_video_worker_routes_to_cloud_run_job(self):
"""Test that render worker routes to Cloud Run Jobs when enabled.
When USE_CLOUD_RUN_JOBS_FOR_RENDER=true and ENABLE_CLOUD_TASKS=true,
trigger_render_video_worker should dispatch a Cloud Run Job (reusing
video-encoding-job with a render_video_worker args override) instead
of posting to the internal HTTP endpoint. As a BackgroundTask on the
service, the render poll was killed ~10s after SIGTERM whenever a
deploy rollout landed mid-render, freezing the job at rendering_video
(see incident 2026-09-13, job 41e06b90).
"""
from backend.services.worker_service import WorkerService, reset_worker_service
reset_worker_service()
with patch('backend.services.worker_service.get_settings') as mock_settings:
mock_settings.return_value.admin_tokens = "test-token"
mock_settings.return_value.google_cloud_project = "test-project"
mock_settings.return_value.enable_cloud_tasks = True
mock_settings.return_value.gcp_region = "us-central1"
mock_settings.return_value.use_cloud_run_jobs_for_render = True
mock_jobs_client = MagicMock()
mock_operation = MagicMock()
mock_operation.metadata = "test-metadata"
mock_jobs_client.run_job.return_value = mock_operation
mock_run_v2 = MagicMock()
mock_run_v2.JobsClient.return_value = mock_jobs_client
mock_run_v2.RunJobRequest = MagicMock()
mock_run_v2.RunJobRequest.Overrides = MagicMock()
mock_run_v2.RunJobRequest.Overrides.ContainerOverride = MagicMock()
import google.cloud
with patch.dict('sys.modules', {'google.cloud.run_v2': mock_run_v2}), \
patch.object(google.cloud, 'run_v2', mock_run_v2, create=True):
service = WorkerService()
result = await service.trigger_render_video_worker("test456")
assert result is True
mock_jobs_client.run_job.assert_called_once()
# Reuses the video-encoding-job Cloud Run Job resource
call_args = mock_run_v2.RunJobRequest.call_args
assert call_args is not None
assert "video-encoding-job" in str(call_args)
# But overrides the module to the render worker with the job id
override_args = mock_run_v2.RunJobRequest.Overrides.ContainerOverride.call_args
assert "backend.workers.render_video_worker" in str(override_args)
assert "--job-id" in str(override_args)
assert "test456" in str(override_args)
@pytest.mark.asyncio
async def test_trigger_render_video_worker_flag_off_uses_legacy_path(self):
"""With USE_CLOUD_RUN_JOBS_FOR_RENDER off, the legacy Cloud Tasks /
HTTP path is used and no Cloud Run Job is dispatched (rollback path)."""
from backend.services.worker_service import WorkerService, reset_worker_service
reset_worker_service()
with patch('backend.services.worker_service.get_settings') as mock_settings:
mock_settings.return_value.admin_tokens = "test-token"
mock_settings.return_value.google_cloud_project = "test-project"
mock_settings.return_value.enable_cloud_tasks = True
mock_settings.return_value.gcp_region = "us-central1"
mock_settings.return_value.use_cloud_run_jobs_for_render = False
service = WorkerService()
with patch.object(service, '_trigger_worker_cloud_run_job', new=AsyncMock()) as mock_crj, \
patch.object(service, 'trigger_worker', new=AsyncMock(return_value=True)) as mock_legacy:
result = await service.trigger_render_video_worker("test789")
assert result is True
mock_crj.assert_not_called()
mock_legacy.assert_called_once_with("render-video", "test789")
class TestWorkerServiceWarmup:
"""Test encoding worker warmup in trigger methods."""
+12 -8
View File
@@ -123,7 +123,11 @@ def test_bump_worker_generation_is_best_effort():
# render worker (GCE path) supersession behaviour
# ---------------------------------------------------------------------------
def _render_job(generation=1):
def _render_job(generation=1, status=JobStatus.RENDERING_VIDEO):
# Entry snapshots (job_start) must use REVIEW_COMPLETE — the real dispatch
# state. A job already in RENDERING_VIDEO at entry is skipped by the
# worker-side idempotency gate (duplicate trigger / CRJ retry protection).
# Mid-render re-read snapshots (job_after) stay RENDERING_VIDEO.
job = MagicMock()
job.artist = "John Maus"
job.title = "The Fear"
@@ -134,7 +138,7 @@ def _render_job(generation=1):
job.prep_only = False
job.state_data = {"worker_generation": generation, "is_duet": False}
job.file_urls = {}
job.status = JobStatus.RENDERING_VIDEO
job.status = status
return job
@@ -187,7 +191,7 @@ async def _run_gce_render(job_start, job_after, transition_side_effect=None):
@pytest.mark.asyncio
async def test_render_superseded_by_status_reset_does_not_fail_job():
"""Admin reset moved the job to awaiting_review mid-render → discard, don't fail."""
job_start = _render_job(generation=1)
job_start = _render_job(generation=1, status=JobStatus.REVIEW_COMPLETE)
job_after = _render_job(generation=1)
job_after.status = "awaiting_review" # reset out from under the render
@@ -207,7 +211,7 @@ async def test_render_superseded_by_status_reset_does_not_fail_job():
@pytest.mark.asyncio
async def test_render_superseded_by_generation_bump_does_not_fail_job():
"""A newer render was triggered (generation bumped) → discard the stale run."""
job_start = _render_job(generation=1)
job_start = _render_job(generation=1, status=JobStatus.REVIEW_COMPLETE)
job_after = _render_job(generation=2) # newer run took over
result, jm = await _run_gce_render(job_start, job_after)
@@ -225,7 +229,7 @@ async def test_invalid_terminal_transition_is_graceful_when_superseded():
reset moved the job to awaiting_review). The worker must bail gracefully,
exactly reproducing and fixing the 7f457087 incident.
"""
job_start = _render_job(generation=1)
job_start = _render_job(generation=1, status=JobStatus.REVIEW_COMPLETE)
job_after = _render_job(generation=1)
job_after.status = "awaiting_review" # reset confirmed on re-read in the handler
@@ -252,7 +256,7 @@ async def test_invalid_transition_when_not_superseded_still_fails():
workflow defect — it must still fail the job (loudly) rather than be silently
swallowed and leave the job stuck. Guards against over-broad error suppression.
"""
job_start = _render_job(generation=1)
job_start = _render_job(generation=1, status=JobStatus.REVIEW_COMPLETE)
job_after = _render_job(generation=1) # same gen + still rendering_video => NOT superseded
invalid = InvalidStateTransitionError(
@@ -272,7 +276,7 @@ async def test_progress_callback_skips_update_when_superseded():
"""A stale render's progress ticks must not drag a reset job's progress bar."""
from backend.workers import render_video_worker as rvw
job_start = _render_job(generation=1)
job_start = _render_job(generation=1, status=JobStatus.REVIEW_COMPLETE)
superseded = _render_job(generation=2) # reset bumped the fence mid-render
jm = MagicMock()
@@ -318,7 +322,7 @@ async def test_progress_callback_skips_update_when_superseded():
@pytest.mark.asyncio
async def test_normal_render_still_completes():
"""Un-superseded render must still write outputs and transition to INSTRUMENTAL_SELECTED."""
job_start = _render_job(generation=1)
job_start = _render_job(generation=1, status=JobStatus.REVIEW_COMPLETE)
job_after = _render_job(generation=1) # same gen, still rendering_video
result, jm = await _run_gce_render(job_start, job_after)
+121 -2
View File
@@ -123,6 +123,47 @@ async def process_render_video(job_id: str) -> bool:
if not job:
logger.error(f"[job:{job_id}] Job not found in Firestore")
job_log.error(f"Job {job_id} not found in Firestore!")
await worker_registry.unregister(job_id, "render-video")
return False
# Hard idempotency gate (worker-side). The legacy HTTP endpoint enforces
# this via _check_worker_idempotency at dispatch time, but the Cloud Run
# Job path invokes this worker directly — and Cloud Run may re-run the
# execution (max_retries=2) after an OOM/timeout/hard kill. The worker
# must therefore protect itself regardless of dispatch path:
# - terminal job: never resurrect. A stale or duplicate trigger racing
# a job that just finished would otherwise re-render and drag a
# complete job back to instrumental_selected.
# - already RENDERING_VIDEO: either another execution owns the render
# (duplicate trigger) or a previous execution died mid-render (Cloud
# Run Job retry). Never re-enter: a live render must not be disrupted,
# and a dead one is recovered by the rendering_video_stuck sweep →
# re-park → auto-retry. Re-entering would hit the same-state
# transition guard and hard-fail the job with a spurious
# "Invalid state transition" error.
# (Legacy path unaffected: the worker enters with status
# REVIEW_COMPLETE — RENDERING_VIDEO is only set below, by this worker.)
terminal_statuses = {"failed", "cancelled", "complete", "prep_complete", "error"}
status_value = job.status.value if hasattr(job.status, "value") else str(job.status)
if status_value in terminal_statuses:
logger.info(
f"[job:{job_id}] WORKER_END worker=render-video status=skipped "
f"reason=terminal_state:{status_value}"
)
job_log.info(f"Job in terminal state '{status_value}' — skipping render")
await worker_registry.unregister(job_id, "render-video")
return False
if job.status == JobStatus.RENDERING_VIDEO:
logger.warning(
f"[job:{job_id}] WORKER_END worker=render-video status=skipped "
f"reason=already_rendering"
)
job_log.warning(
"Job already in rendering_video — another execution owns this render "
"(or a previous one died and the rendering_video_stuck sweep will "
"recover it); not re-entering"
)
await worker_registry.unregister(job_id, "render-video")
return False
# Capture the supersession fence at start. If an admin reset (or a newer
@@ -334,7 +375,20 @@ async def process_render_video(job_id: str) -> bool:
from backend.services.worker_service import get_worker_service
worker_service = get_worker_service()
job_log.info("Triggering video worker for final encoding...")
await worker_service.trigger_video_worker(job_id)
video_triggered = await worker_service.trigger_video_worker(job_id)
if not video_triggered:
# No exception path reaches here (trigger returns False
# instead of raising) — surface it loudly, or the job
# sits at instrumental_selected with no error and no
# sweep watching that state.
job_log.error(
"Video worker dispatch FAILED after render complete — "
"job stalls at instrumental_selected until manually retried"
)
logger.error(
f"[job:{job_id}] Video worker dispatch failed after GCE "
f"render complete (job stalled at instrumental_selected)"
)
job_manager.update_state_data(job_id, 'render_progress', {'stage': 'complete'})
return True
@@ -637,7 +691,16 @@ async def process_render_video(job_id: str) -> bool:
worker_service = get_worker_service()
job_log.info("Triggering video worker for final encoding...")
await worker_service.trigger_video_worker(job_id)
video_triggered = await worker_service.trigger_video_worker(job_id)
if not video_triggered:
job_log.error(
"Video worker dispatch FAILED after render complete — "
"job stalls at instrumental_selected until manually retried"
)
logger.error(
f"[job:{job_id}] Video worker dispatch failed after local "
f"render complete (job stalled at instrumental_selected)"
)
# Mark render progress as complete for idempotency
# This allows the worker to be re-triggered after admin reset
@@ -843,3 +906,59 @@ def park_active_render_jobs_for_shutdown() -> int:
# For compatibility with worker service
render_video_worker = process_render_video
# ==================== CLI Entry Point for Cloud Run Jobs ====================
# Allows the render worker to run as a standalone Cloud Run Job execution
# (video-encoding-job with an args override), immune to Cloud Run Service
# deployment rollouts. As a BackgroundTask on the service it was killed ~10s
# after SIGTERM whenever a deploy landed mid-render (incident 2026-09-13).
# Usage: python -m backend.workers.render_video_worker --job-id <job_id>
def main():
"""CLI entry point for running the render worker as a Cloud Run Job.
Exit-code contract (differs from video_worker deliberately): the Cloud Run
Job template has max_retries=2, and ``process_render_video`` handles every
expected failure internally — it parks the job (capacity/shutdown), fails
it with a user-facing message, or discards a superseded result. Re-running
the worker after any of those would re-enter a job that has already moved
to a non-render state. So a clean return (True OR False) exits 0, and only
an escaped exception (crash before state was handled) exits 1 to request a
Cloud Run retry.
"""
import argparse
import asyncio
import sys
parser = argparse.ArgumentParser(
description="Post-review render video worker for karaoke generation"
)
parser.add_argument("--job-id", required=True, help="Job ID to process")
args = parser.parse_args()
job_id = args.job_id
logging.basicConfig(
level=logging.INFO,
format="%(asctime)s - %(name)s - %(levelname)s - %(message)s"
)
logger.info(f"Starting render video worker CLI for job {job_id}")
try:
success = asyncio.run(process_render_video(job_id))
if success:
logger.info(f"Render video completed successfully for job {job_id}")
else:
logger.error(
f"Render video did not complete for job {job_id} "
"(job state already handled by the worker — parked, failed, or superseded)"
)
sys.exit(0)
except Exception as e:
logger.error(f"Render video worker crashed: {e}", exc_info=True)
sys.exit(1)
if __name__ == "__main__":
main()
+57
View File
@@ -1883,3 +1883,60 @@ gated to deployed envs, skips test/dev.
- Detect silent *partial success* at the **publish boundary** (what *should* have
shipped vs what did), and make the metadata backstop validate a just-published
track's **own** completeness same-run, not only via global-max gap math (D2).
## BackgroundTasks Cannot Survive Deploys — And the Shutdown Hook That "Protects" Them Never Runs (Sep 2026, v0.224.0)
**Context:** Tenant-E2E run #107 failed: job `41e06b90` froze at `rendering_video`
even though the GCE encoder finished the render and uploaded every output. A backend
deploy landed mid-render. The render worker ran as a FastAPI BackgroundTask on the
Cloud Run service; the deploy retired the old revision, SIGTERM arrived, and its last
log lines were uvicorn's `Shutting down` / `Waiting for background tasks to complete`
— then nothing. Nobody ever polled the encoder again.
**The load-bearing wrong belief:** `main.py`'s lifespan shutdown hook (worker-registry
wait + `park_active_render_jobs_for_shutdown()`) was built for exactly this scenario
and commented "Cloud Run gen2 default termination grace period is 600s. We wait up to
480s…". Both halves were wrong:
1. **Cloud Run services SIGKILL ~10 seconds after SIGTERM**, not 600s.
2. **Uvicorn runs lifespan shutdown only AFTER all background tasks complete.** A
render task with 30 minutes left keeps uvicorn stuck in its bg-task wait until
SIGKILL — so the parking hook is architecturally unreachable in the one scenario
it exists for. It had never once fired in anger.
Three other layers also failed to save the job, each for a structural reason:
- **Cloud Tasks retry** — defeated by design: the worker endpoint returns 200
immediately and does the work post-response, so the queue considers the task done
seconds after dispatch (`dispatch_deadline=1800s` protects nothing).
- **`recover-stuck-jobs`** — keyed on `updated_at` staleness > 45 min; correct as a
slow safety net, but slower than the tenant E2E's 30-min render budget and a
~50-60 min stall + wasted re-render for a real customer.
- **Worker-side completion** — the GCE encoder finished and uploaded to GCS, but
results are pull-only (backend polls); there is no completion push, so a dead
poller means a done-but-unrecorded render.
**Fix:** run the render worker as a **Cloud Run Job** (reuse `video-encoding-job`
with a `render_video_worker` args override; flag `USE_CLOUD_RUN_JOBS_FOR_RENDER`,
mirroring the 2026-03 `USE_CLOUD_RUN_JOBS_FOR_VIDEO` fix for the identical incident
one stage later in the pipeline). Job executions run to completion; service deploys
can't touch them. The job env also sets both `USE_CLOUD_RUN_JOBS_FOR_*` flags so the
render execution's in-process trigger of the video worker dispatches a Cloud Run Job
too instead of regressing to the vulnerable path.
**Principles for next time:**
- **Any work that outlives its HTTP response on Cloud Run is presumed dead on every
deploy.** BackgroundTasks are only acceptable for work that completes in seconds.
Long orchestration belongs in Cloud Run Jobs (audio-download, lyrics, separation,
video encode, and now render all migrated for this exact reason — when one worker
gets this fix, audit the *other* workers for the same pattern; render sat
vulnerable for 6 months after video was fixed).
- **A graceful-shutdown hook is untested until it has provably fired.** Uvicorn's
shutdown ordering (bg tasks → lifespan) plus Cloud Run's 10s kill window made ours
dead code. If a safety net matters, verify it in logs during a real deploy.
- **Returning 200-then-working silently discards the queue's retry semantics.** If
you accept work via Cloud Tasks, either do the work in-request or make sure some
other mechanism owns retries.
- **Exit codes are API when Cloud Run Jobs retry:** `process_render_video` handles
its own failures (park/fail/supersede), so the CLI exits 0 on any clean return and
1 only on an escaped crash — otherwise `max_retries=2` would re-enter jobs that
already moved to a non-render state.
+2
View File
@@ -325,6 +325,8 @@ gcloud compute instances describe encoding-worker-a \
**Auto-recovery (v0.192.3+):** `recover-stuck-jobs` (every 5 min) now flags a job in `rendering_video` with no progress for **>45 min** (`rendering_video_stuck`) and re-parks it into `render_pending_capacity`, so the existing `retry-pending-render-jobs` cron resets it to `review_complete` and re-renders (same 24h ceiling). No manual action needed.
**Root-cause fix (v0.224.0+):** the dominant cause of this state was **backend deploys**: the render worker ran as a FastAPI BackgroundTask on the Cloud Run *service*, and every deploy rollout that landed mid-render SIGKILLed it ~10s after SIGTERM (the lifespan "park on shutdown" hook never runs in that scenario — uvicorn waits for background tasks *before* running lifespan shutdown, so it dies waiting; incident 2026-09-13, job `41e06b90`, tenant-E2E run #107). Since v0.224.0 the render worker runs as a **Cloud Run Job** (`video-encoding-job` with a `render_video_worker` args override, flag `USE_CLOUD_RUN_JOBS_FOR_RENDER`), which runs to completion regardless of service deploys — same fix the video worker got in 2026-03 (`USE_CLOUD_RUN_JOBS_FOR_VIDEO`). The 45-min sweep remains as the safety net for the remaining causes (job execution crash/OOM/timeout, lost GCE VM). Rollback: remove `USE_CLOUD_RUN_JOBS_FOR_RENDER=true` from the service env to fall back to the legacy Cloud Tasks path.
**Manual unstick (if needed):** don't re-trigger the render worker while status is `rendering_video` (it starts mid-state and fails with "Invalid state transition"). Instead let it fail (or re-park it), then use the admin **Retry** which resumes cleanly from `review_complete`. If a ghost `render_progress.stage='running'` blocks re-trigger, clear it first (`update({'state_data.render_progress': {'stage': 'pending'}})`).
---
@@ -0,0 +1,54 @@
# Investigation: Tenant E2E #107 failure — deploy killed render poll, job froze at `rendering_video`
**Date:** 2026-09-13
**Incident:** Tenant Portals E2E run #107 (`singa`) failed `job reached complete before render timeout` (expected `complete`, got `rendering_video`). Job `41e06b90`.
**Fix:** v0.224.0 — render worker migrated to Cloud Run Jobs (`USE_CLOUD_RUN_JOBS_FOR_RENDER`).
## Timeline (all UTC, 2026-09-13)
| Time | Event |
|------|-------|
| 04:58:44 | New backend revision `karaoke-backend-00973-xac` starts (deploy in progress) |
| 04:58:52 | Review complete → Cloud Task created for render-video worker (`dispatch_deadline=1800s`) |
| 04:58:53 | Cloud Task POST hits **old** revision `00971-xad`; endpoint returns 200 immediately, `process_render_video` continues as FastAPI BackgroundTask; job → `rendering_video`; render submitted to GCE `encoding-worker-a` |
| 04:59:03 | Backend records "Encoding progress: 55%" (last Firestore write, `updated_at` freezes here) |
| 04:59:13.867 | Old revision gets SIGTERM: uvicorn logs `Shutting down` then `Waiting for background tasks to complete.` — **its last-ever log lines** |
| ~04:59:24 | SIGKILL (Cloud Run gives ~10s, not the 600s the code comment claimed). Lifespan shutdown hook (worker-registry wait + `park_active_render_jobs_for_shutdown`) **never ran** |
| 04:59:57 | GHA runner logs "Deploy - Backend (Cloud Run) completed: Succeeded" |
| 05:00:05 | **GCE worker completes the render** and uploads `with_vocals.mkv` + karaoke.ass/lrc/corrected.txt to GCS. Nobody polls it again; result never recorded |
| 05:00, 05:05, … | `RECOVER_STUCK_JOBS` runs every 5 min on the new revision: `render_retriggered=0` — `rendering_video_stuck` needs `updated_at` stale > **45 min** |
| 05:15:10 | `encoding-worker-idle-shutdown` correctly stops the now-idle VM (confirmed `active_jobs=0` first — its fail-safe worked; red herring) |
| 05:30:06 | E2E 30-min render deadline expires; test cleanup deletes job, refunds credit; run #107 fails |
Recovery *would* have fired at ~05:44 (45-min staleness) and re-rendered from scratch. Real-customer impact of this class: ~50-60 min silent stall + a wasted re-render, on every deploy that lands mid-render (deploys go out on every merge to main).
## Root cause chain (4 independent layers, each structurally broken for this case)
1. **BackgroundTask execution** — the render endpoint returns 200 immediately and polls the GCE worker post-response. Cloud Run kills old-revision instances ~10s after SIGTERM on every deploy; a mid-render poll loop cannot survive.
2. **Unreachable shutdown parking** — uvicorn's shutdown sequence waits for background tasks *before* running FastAPI lifespan shutdown. With a long render task in flight, SIGKILL always arrives first, so `park_active_render_jobs_for_shutdown()` is dead code in exactly the scenario it was written for. The comment justifying it ("600s grace, wait 480s") was factually wrong.
3. **Cloud Tasks retry defeated** — the queue got its 200 at 04:58:53; `dispatch_deadline` protects nothing when the handler is fire-and-forget.
4. **Slow watchdog** — `rendering_video_stuck` = `updated_at` stale > 45 min. Correct as a last-ditch net, but the only net that was actually reachable, and slower than both the E2E budget and any reasonable customer expectation.
## Why this was a known-class bug
`docs/archive/2026-03-08-fix-encoding-deploy-survival.md` diagnosed the **identical** failure for the *video worker* (final encode) — same BackgroundTask pattern, same deploy-kill, same conclusion — and fixed it by routing through the `video-encoding-job` Cloud Run Job (`USE_CLOUD_RUN_JOBS_FOR_VIDEO=true`, live since March). The render worker — one pipeline stage earlier, same shape of work — was never migrated. Related-but-distinct prior fixes that did NOT cover this: GCE-worker drain on deploy (protects the *encoder*, not the orchestrator), poll pinning v0.184.2 (protects against *worker* primary swaps), `recover-stuck-jobs` re-park v0.192.3 (the 45-min net that was too slow here).
## Fix (v0.224.0)
- `trigger_render_video_worker` routes to a Cloud Run Job execution — reusing `video-encoding-job` with args override `python -m backend.workers.render_video_worker --job-id <id>` — when `USE_CLOUD_RUN_JOBS_FOR_RENDER=true` and Cloud Tasks enabled. Job executions run to completion; deploys update the job template image but never touch running executions.
- New CLI entrypoint in `render_video_worker.py`. Exit-code contract differs from video_worker deliberately: clean returns (True *and* False) exit 0 because `process_render_video` already parked/failed/superseded the job internally; only an escaped crash exits 1 (job template has `max_retries=2`).
- Pulumi: `video-encoding-job` env gains `USE_CLOUD_RUN_JOBS_FOR_VIDEO=true` + `USE_CLOUD_RUN_JOBS_FOR_RENDER=true` so the render execution's in-process `trigger_video_worker` also dispatches a Cloud Run Job instead of regressing to the vulnerable path.
- ci.yml: service deploy env gains `USE_CLOUD_RUN_JOBS_FOR_RENDER=true`.
- `main.py` lifespan comment corrected (10s reality, uvicorn ordering) and the pointless 480s wait shrunk to 5s; the parking hook remains as a legacy/flag-off net.
- 45-min `rendering_video_stuck` sweep unchanged — still the net for execution crash/OOM/timeout and lost-VM cases.
**Rollback:** remove `USE_CLOUD_RUN_JOBS_FOR_RENDER=true` from the service env (falls back to the legacy Cloud Tasks → BackgroundTask path).
## Deliberately not done (and why)
- **Atomic (transactional) REVIEW_COMPLETE → RENDERING_VIDEO claim** (CodeRabbit finding on PR #994): two executions starting within the same sub-second window could both pass the worker-side gate and double-render. The identical read-then-write window exists in the legacy endpoint's `_check_worker_idempotency` (check stage → mark 'running' is not transactional), the worker-generation fence guarantees the stale result is discarded so final state is always correct, and the consequence is bounded wasted compute on a rare² race (requires two dispatches for one job AND <1s start skew). A conditional-write claim belongs in `JobManager`/`FirestoreService.update_job_status` — shared machinery for every worker — and deserves its own change, not a rider on this one.
- **Adopt-completed-render on recovery** (query the pinned worker's `/status` during the stuck sweep and harvest finished outputs instead of re-rendering): meaningful only for the deploy-kill case this PR eliminates at the source; remaining orphan causes mostly kill the GCE worker too. Skipped to keep the change focused.
- **Render-progress heartbeats + faster stuck threshold**: same reasoning — with the poller in a Cloud Run Job, poller death becomes rare enough that the 45-min net is acceptable.
- **uvicorn `--timeout-graceful-shutdown` + CancelledError parking**: would make the lifespan hook genuinely reachable, but it's belt-and-braces once renders no longer run on the service; adds signal-handling risk to every request path.
- **Same audit for `lyrics`/`screens`/`audio` BackgroundTasks**: lyrics + separation + audio-download already run as Cloud Run Jobs; screens/preview tasks are short. Noted in LESSONS-LEARNED as the pattern to check when touching those workers.
+15
View File
@@ -668,6 +668,21 @@ def create_video_encoding_job(
name="USE_GCE_ENCODING",
value="true",
),
# The render worker also runs in this job (args
# override → backend.workers.render_video_worker)
# and triggers the video worker in-process when the
# render completes. Without these flags that
# downstream trigger would fall back to the
# deploy-vulnerable Cloud Tasks → BackgroundTask
# path this job exists to avoid.
cloudrunv2.JobTemplateTemplateContainerEnvArgs(
name="USE_CLOUD_RUN_JOBS_FOR_VIDEO",
value="true",
),
cloudrunv2.JobTemplateTemplateContainerEnvArgs(
name="USE_CLOUD_RUN_JOBS_FOR_RENDER",
value="true",
),
],
)
],
+1 -1
View File
@@ -1,6 +1,6 @@
[tool.poetry]
name = "karaoke-gen"
version = "0.223.2"
version = "0.224.0"
description = "Generate karaoke videos with synchronized lyrics. Handles the entire process from downloading audio and lyrics to creating the final video with title screens."
authors = ["Andrew Beveridge <andrew@beveridge.uk>"]
license = "MIT"