mirror of
https://github.com/nomadkaraoke/karaoke-gen.git
synced 2026-11-20 13:07:57 +09:00
## Summary Tenant E2E run #107 (`singa`) failed with job `41e06b90` frozen at `rendering_video` — even though the GCE encoder **finished the render and uploaded every output**. Root cause: a backend deploy landed mid-render and killed the render poll, which ran as a FastAPI BackgroundTask on the Cloud Run service. Full investigation: `docs/archive/2026-09-13-render-deploy-survival-investigation.md` ### The failure chain (4 layers, each structurally broken) 1. **BackgroundTask execution** — the render endpoint returns 200 immediately and polls post-response; Cloud Run SIGKILLs old-revision instances ~10s after SIGTERM on every deploy. 2. **Unreachable shutdown parking** — uvicorn waits for background tasks *before* running lifespan shutdown, so `park_active_render_jobs_for_shutdown()` never runs in exactly the scenario it was written for. Its comment claimed a 600s grace period; Cloud Run gives 10s. Verified in logs: the old revision's last lines were uvicorn's `Waiting for background tasks to complete.` — then SIGKILL. 3. **Cloud Tasks retry defeated** — the queue got its 200 at dispatch; `dispatch_deadline=1800s` protects nothing. 4. **45-min stuck sweep** — reachable but slower than the E2E's 30-min budget and a ~50-60 min silent stall + wasted re-render for a real customer, on every deploy that lands mid-render (deploys go out on every merge to main). This is the **identical bug class fixed for the video worker in 2026-03** (`docs/archive/2026-03-08-fix-encoding-deploy-survival.md`, `USE_CLOUD_RUN_JOBS_FOR_VIDEO`) — the render worker was never migrated. ### Changes - **Route `trigger_render_video_worker` through a Cloud Run Job** — reuses `video-encoding-job` with args override `python -m backend.workers.render_video_worker --job-id <id>`, behind `USE_CLOUD_RUN_JOBS_FOR_RENDER` (enabled in deploy env). Job executions run to completion; deploys never touch them. - **New CLI entrypoint** in `render_video_worker.py`. Exit-code contract differs from video_worker deliberately: clean returns (True AND False) exit 0 because `process_render_video` parks/fails/supersedes internally; only an escaped crash exits 1 (job template has `max_retries=2`). - **Worker-side idempotency gate** (review finding): the CRJ path bypasses the HTTP endpoint's `_check_worker_idempotency`, and CRJ retries could re-enter a job in `RENDERING_VIDEO` (hard-failing it via the same-state transition guard) or resurrect a terminal job. The worker now skips terminal + already-rendering jobs cleanly regardless of dispatch path. - **Surface failed video-worker dispatch** (review finding): the ignored `trigger_video_worker` return after render completion now logs at ERROR instead of silently stalling the job at `instrumental_selected`. - **Pulumi**: `video-encoding-job` env gains both `USE_CLOUD_RUN_JOBS_FOR_*` flags so the render execution's in-process video-worker trigger also dispatches a CRJ. **Already applied to prod** (`pulumi up` run locally, verified on the job). - **`main.py`**: corrected the false 600s comment; shrank the unreachable 480s lifespan wait to 5s. Parking hook kept as legacy/flag-off net. ### Rollback Remove `USE_CLOUD_RUN_JOBS_FOR_RENDER=true` from the service env → falls back to the legacy Cloud Tasks path. ## Testing - Backend: 4422 unit + 151 emulator tests pass locally. - New tests: CRJ routing (flag on/off), CLI exit-code contract, worker-side idempotency gate (terminal skip, already-rendering skip). - Post-deploy: will dispatch the Tenant Portals E2E workflow (the failing scenario) against prod. 🤖 Generated with [Claude Code](https://claude.com/claude-code) <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit - **New Features** - Video rendering now runs as durable background jobs, allowing renders to continue through service deployments. - Added safeguards to prevent duplicate processing and safely handle completed or in-progress jobs. - Improved failure reporting when a video-processing job cannot be started. - **Bug Fixes** - Fixed render jobs becoming stuck when deployments interrupted background processing. - **Documentation** - Added troubleshooting guidance and incident lessons for interrupted or stuck renders. <!-- end of auto-generated comment: release notes by coderabbit.ai --> --------- Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
72 lines
3.0 KiB
Python
72 lines
3.0 KiB
Python
"""
|
|
Tests for the render video worker CLI entry point (Cloud Run Jobs execution).
|
|
|
|
The render worker runs as a Cloud Run Job (video-encoding-job with an args
|
|
override) so it survives Cloud Run Service deployment rollouts — as a
|
|
BackgroundTask it was killed ~10s after SIGTERM whenever a deploy landed
|
|
mid-render (incident 2026-09-13, job 41e06b90).
|
|
|
|
Exit-code contract (deliberately different from video_worker): the Cloud Run
|
|
Job template has max_retries=2 and process_render_video handles every expected
|
|
failure internally (park / fail_job / supersede-discard). A clean return —
|
|
True OR False — must exit 0 so Cloud Run never re-runs the worker against a
|
|
job that already moved to a non-render state. Only an escaped exception
|
|
(crash before state was handled) exits 1 to request a Cloud Run retry.
|
|
"""
|
|
import sys
|
|
from unittest.mock import AsyncMock, MagicMock, patch
|
|
|
|
import pytest
|
|
|
|
from backend.workers.render_video_worker import main
|
|
|
|
|
|
def _run_main_with_args(job_id="job123"):
|
|
return patch.object(sys, "argv", ["render_video_worker", "--job-id", job_id])
|
|
|
|
|
|
class TestRenderVideoWorkerCli:
|
|
def test_main_exits_zero_on_success(self):
|
|
with _run_main_with_args(), \
|
|
patch("backend.workers.render_video_worker.process_render_video",
|
|
new=AsyncMock(return_value=True)):
|
|
with pytest.raises(SystemExit) as exc_info:
|
|
main()
|
|
assert exc_info.value.code == 0
|
|
|
|
def test_main_exits_zero_on_handled_failure(self):
|
|
"""False means the worker already parked/failed/superseded the job —
|
|
a Cloud Run retry would re-enter a job in a non-render state."""
|
|
with _run_main_with_args(), \
|
|
patch("backend.workers.render_video_worker.process_render_video",
|
|
new=AsyncMock(return_value=False)):
|
|
with pytest.raises(SystemExit) as exc_info:
|
|
main()
|
|
assert exc_info.value.code == 0
|
|
|
|
def test_main_exits_nonzero_on_crash(self):
|
|
"""An escaped exception means job state was NOT handled — exit 1 so
|
|
Cloud Run's max_retries can re-run the worker."""
|
|
with _run_main_with_args(), \
|
|
patch("backend.workers.render_video_worker.process_render_video",
|
|
new=AsyncMock(side_effect=RuntimeError("boom"))):
|
|
with pytest.raises(SystemExit) as exc_info:
|
|
main()
|
|
assert exc_info.value.code == 1
|
|
|
|
def test_main_passes_job_id(self):
|
|
mock_worker = AsyncMock(return_value=True)
|
|
with _run_main_with_args("abc999"), \
|
|
patch("backend.workers.render_video_worker.process_render_video",
|
|
new=mock_worker):
|
|
with pytest.raises(SystemExit):
|
|
main()
|
|
mock_worker.assert_awaited_once_with("abc999")
|
|
|
|
def test_main_requires_job_id(self):
|
|
with patch.object(sys, "argv", ["render_video_worker"]):
|
|
with pytest.raises(SystemExit) as exc_info:
|
|
main()
|
|
# argparse exits 2 on missing required argument
|
|
assert exc_info.value.code == 2
|