mirror of
https://github.com/nomadkaraoke/karaoke-gen.git
synced 2026-11-20 13:07:57 +09:00
## Problem: a regression from #1191, caught by #alerts triage At 07:03 UTC on 2026-10-19, two new freeze alerts fired: **52s with 29 requests affected** (culprit `firestore_service.append_log_to_subcollection`, during CDG packaging) and **81s with 33 requests** (culprit `dropbox_service.upload_file`). The stacks show `video_worker.generate_video` running **inside karaoke-backend**. Cause: in today's daily E2E run (job 5a5e5181), #1191's park/resume worked as designed (parked 06:55:04, resumed 06:57:50). But the resume runs in `audio-separation-job`, which doesn't set `USE_CLOUD_RUN_JOBS_FOR_VIDEO`. So `trigger_video_worker` fell back to Cloud Tasks, which runs it as a `BackgroundTask` on the API, putting the whole video worker on the API's event loop. The same latent bug already applied to the audio worker's auto-approval → render trigger. ## Fix - `backend/config.py`: inside a Cloud Run Job (`CLOUD_RUN_JOB` is set by the platform), `use_cloud_run_jobs_for_video` and `use_cloud_run_jobs_for_render` default to **true**. An explicit env value still wins. This covers every worker job (audio, lyrics, download, ...) with no infra change, through the normal image deploy. - Error monitor normalizer: `requests_affected=N` → `requests_affected=<N>`, so a stall culprit is one alert pattern instead of one per request count. ## Testing - New `backend/tests/test_config_cloud_run_job_dispatch.py`: inside a job → true; outside → unchanged; explicit env wins. - New normalizer test. - `backend/tests` + `tests/unit/services/error_monitor`: 6871 passed. - Local CodeRabbit review: 0 findings. ## Verify after deploy The next parked-then-resumed job logs `Started Cloud Run Job video-encoding-job` from audio-separation-job, and there are no `EVENT_LOOP_STALL` lines on karaoke-backend with `video_worker` in the stack. @coderabbitai ignore 🤖 Generated with [Claude Code](https://claude.com/claude-code) https://claude.ai/code/session_019pmG2XAgtUguEJWLFpkKHN --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>