fix: run render-video worker as Cloud Run Job to survive backend deploys (#994)

## Summary

Tenant E2E run #107 (`singa`) failed with job `41e06b90` frozen at
`rendering_video` — even though the GCE encoder **finished the render
and uploaded every output**. Root cause: a backend deploy landed
mid-render and killed the render poll, which ran as a FastAPI
BackgroundTask on the Cloud Run service.

Full investigation:
`docs/archive/2026-09-13-render-deploy-survival-investigation.md`

### The failure chain (4 layers, each structurally broken)

1. **BackgroundTask execution** — the render endpoint returns 200
immediately and polls post-response; Cloud Run SIGKILLs old-revision
instances ~10s after SIGTERM on every deploy.
2. **Unreachable shutdown parking** — uvicorn waits for background tasks
*before* running lifespan shutdown, so
`park_active_render_jobs_for_shutdown()` never runs in exactly the
scenario it was written for. Its comment claimed a 600s grace period;
Cloud Run gives 10s. Verified in logs: the old revision's last lines
were uvicorn's `Waiting for background tasks to complete.` — then
SIGKILL.
3. **Cloud Tasks retry defeated** — the queue got its 200 at dispatch;
`dispatch_deadline=1800s` protects nothing.
4. **45-min stuck sweep** — reachable but slower than the E2E's 30-min
budget and a ~50-60 min silent stall + wasted re-render for a real
customer, on every deploy that lands mid-render (deploys go out on every
merge to main).

This is the **identical bug class fixed for the video worker in
2026-03** (`docs/archive/2026-03-08-fix-encoding-deploy-survival.md`,
`USE_CLOUD_RUN_JOBS_FOR_VIDEO`) — the render worker was never migrated.

### Changes

- **Route `trigger_render_video_worker` through a Cloud Run Job** —
reuses `video-encoding-job` with args override `python -m
backend.workers.render_video_worker --job-id <id>`, behind
`USE_CLOUD_RUN_JOBS_FOR_RENDER` (enabled in deploy env). Job executions
run to completion; deploys never touch them.
- **New CLI entrypoint** in `render_video_worker.py`. Exit-code contract
differs from video_worker deliberately: clean returns (True AND False)
exit 0 because `process_render_video` parks/fails/supersedes internally;
only an escaped crash exits 1 (job template has `max_retries=2`).
- **Worker-side idempotency gate** (review finding): the CRJ path
bypasses the HTTP endpoint's `_check_worker_idempotency`, and CRJ
retries could re-enter a job in `RENDERING_VIDEO` (hard-failing it via
the same-state transition guard) or resurrect a terminal job. The worker
now skips terminal + already-rendering jobs cleanly regardless of
dispatch path.
- **Surface failed video-worker dispatch** (review finding): the ignored
`trigger_video_worker` return after render completion now logs at ERROR
instead of silently stalling the job at `instrumental_selected`.
- **Pulumi**: `video-encoding-job` env gains both
`USE_CLOUD_RUN_JOBS_FOR_*` flags so the render execution's in-process
video-worker trigger also dispatches a CRJ. **Already applied to prod**
(`pulumi up` run locally, verified on the job).
- **`main.py`**: corrected the false 600s comment; shrank the
unreachable 480s lifespan wait to 5s. Parking hook kept as
legacy/flag-off net.

### Rollback

Remove `USE_CLOUD_RUN_JOBS_FOR_RENDER=true` from the service env → falls
back to the legacy Cloud Tasks path.

## Testing

- Backend: 4422 unit + 151 emulator tests pass locally.
- New tests: CRJ routing (flag on/off), CLI exit-code contract,
worker-side idempotency gate (terminal skip, already-rendering skip).
- Post-deploy: will dispatch the Tenant Portals E2E workflow (the
failing scenario) against prod.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit

- **New Features**
- Video rendering now runs as durable background jobs, allowing renders
to continue through service deployments.
- Added safeguards to prevent duplicate processing and safely handle
completed or in-progress jobs.
- Improved failure reporting when a video-processing job cannot be
started.

- **Bug Fixes**
- Fixed render jobs becoming stuck when deployments interrupted
background processing.

- **Documentation**
- Added troubleshooting guidance and incident lessons for interrupted or
stuck renders.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->

---------

Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
This commit is contained in:
Andrew Beveridge ☄️andClaude Fable 5 authored and GitHub committed 2026-09-13 03:02:15 -04:00
1 parent b41664a2fc
commit 3cdb62a058
15 files changed
+523 -23

No files matched your search

+1 -1
View File
@@ -1994,7 +1994,7 @@ jobs:
--min-instances 1 \
--vpc-connector projects/nomadkaraoke/locations/us-central1/connectors/cloud-run-connector \
--vpc-egress private-ranges-only \
--set-env-vars "GOOGLE_CLOUD_PROJECT=${{ env.PROJECT_ID }},GCS_BUCKET_NAME=karaoke-gen-storage-${{ env.PROJECT_ID }},FIRESTORE_COLLECTION=jobs,ENVIRONMENT=production,KARAOKE_GEN_VERSION=$VERSION,DEFAULT_DROPBOX_PATH=/MediaUnsynced/Karaoke/Tracks-Organized,DEFAULT_GDRIVE_FOLDER_ID=1laRKAyxo0v817SstfM5XkpbWiNKNAMSX,ENABLE_CLOUD_TASKS=true,CLOUD_RUN_SERVICE_URL=https://api.nomadkaraoke.com,GCP_REGION=us-central1,DEFAULT_ENABLE_YOUTUBE_UPLOAD=true,DEFAULT_BRAND_PREFIX=NOMAD,DEFAULT_PRIVATE_DROPBOX_PATH=/MediaUnsynced/Karaoke/Tracks-NonPublished,DEFAULT_PRIVATE_BRAND_PREFIX=NOMADNP,USE_GCE_ENCODING=true,USE_GCE_PREVIEW_ENCODING=true,ENABLE_PUSH_NOTIFICATIONS=true,USE_CLOUD_RUN_JOBS_FOR_VIDEO=true,EMAIL_PROVIDER=postmark,AUTO_CORRECT_MODEL=claude-opus-4-8,AUTO_CORRECT_COMPARE_MODELS=claude-opus-4-8;gemini-3.1-pro-preview,AUTO_CORRECT_PROACTIVE_ENABLED=true,AUTO_APPROVAL_BACKING_KEEP_ENABLED=true,COMMUNITY_DAILY_PICK_ENABLED=true,EDGE_AUTH_MODE=enforce,COMMIT_SHA=${{ steps.build-meta.outputs.sha }},PR_NUMBER=${{ steps.build-meta.outputs.pr_number }},PR_TITLE=${{ steps.build-meta.outputs.pr_title }}" \
--set-env-vars "GOOGLE_CLOUD_PROJECT=${{ env.PROJECT_ID }},GCS_BUCKET_NAME=karaoke-gen-storage-${{ env.PROJECT_ID }},FIRESTORE_COLLECTION=jobs,ENVIRONMENT=production,KARAOKE_GEN_VERSION=$VERSION,DEFAULT_DROPBOX_PATH=/MediaUnsynced/Karaoke/Tracks-Organized,DEFAULT_GDRIVE_FOLDER_ID=1laRKAyxo0v817SstfM5XkpbWiNKNAMSX,ENABLE_CLOUD_TASKS=true,CLOUD_RUN_SERVICE_URL=https://api.nomadkaraoke.com,GCP_REGION=us-central1,DEFAULT_ENABLE_YOUTUBE_UPLOAD=true,DEFAULT_BRAND_PREFIX=NOMAD,DEFAULT_PRIVATE_DROPBOX_PATH=/MediaUnsynced/Karaoke/Tracks-NonPublished,DEFAULT_PRIVATE_BRAND_PREFIX=NOMADNP,USE_GCE_ENCODING=true,USE_GCE_PREVIEW_ENCODING=true,ENABLE_PUSH_NOTIFICATIONS=true,USE_CLOUD_RUN_JOBS_FOR_VIDEO=true,USE_CLOUD_RUN_JOBS_FOR_RENDER=true,EMAIL_PROVIDER=postmark,AUTO_CORRECT_MODEL=claude-opus-4-8,AUTO_CORRECT_COMPARE_MODELS=claude-opus-4-8;gemini-3.1-pro-preview,AUTO_CORRECT_PROACTIVE_ENABLED=true,AUTO_APPROVAL_BACKING_KEEP_ENABLED=true,COMMUNITY_DAILY_PICK_ENABLED=true,EDGE_AUTH_MODE=enforce,COMMIT_SHA=${{ steps.build-meta.outputs.sha }},PR_NUMBER=${{ steps.build-meta.outputs.pr_number }},PR_TITLE=${{ steps.build-meta.outputs.pr_title }}" \
--set-secrets "ANTHROPIC_API_KEY=anthropic-api-key:latest,AUDIOSHAKE_API_TOKEN=audioshake-api-key:latest,GENIUS_API_TOKEN=genius-api-key:latest,SPOTIFY_COOKIE_SP_DC=spotify-cookie:latest,RAPIDAPI_KEY=rapidapi-key:latest,DEFAULT_DISCORD_WEBHOOK_URL=discord-releases-webhook:latest,ADMIN_TOKENS=admin-tokens:latest,RED_API_KEY=red-api-key:latest,RED_API_URL=red-api-url:latest,OPS_API_KEY=ops-api-key:latest,OPS_API_URL=ops-api-url:latest,FLACFETCH_API_URL=flacfetch-api-url:latest,FLACFETCH_API_KEY=flacfetch-api-key:latest,POSTMARK_SERVER_TOKEN=postmark-server-token:latest,STRIPE_SECRET_KEY=stripe-secret-key:latest,STRIPE_WEBHOOK_SECRET=stripe-webhook-secret:latest,LANGFUSE_PUBLIC_KEY=langfuse-public-key:latest,LANGFUSE_SECRET_KEY=langfuse-secret-key:latest,LANGFUSE_HOST=langfuse-host:latest,ENCODING_WORKER_URL=encoding-worker-url:latest,ENCODING_WORKER_API_KEY=encoding-worker-api-key:latest,ENCODING_WORKER_FALLBACK_VMS=encoding-worker-fallback-vms:latest,VAPID_PUBLIC_KEY=vapid-public-key:latest,VAPID_PRIVATE_KEY=vapid-private-key:latest,E2E_BYPASS_KEY=e2e-bypass-key:latest,GDRIVE_VALIDATOR_URL=gdrive-validator-url:latest,EDGE_ORIGIN_SECRET=edge-origin-secret:latest"
# R3/R4 — smoke the tagged candidate revision BEFORE it serves any traffic.