mirror of
https://github.com/nomadkaraoke/karaoke-gen.git
synced 2026-11-20 13:07:57 +09:00
## Summary Every gen job waited **2–5 min** before `audio-download-job` (and then `lyrics-transcription-job`) even started. That blocked kjbox's goal of singing within about 4 minutes of making a song. **Root cause: Cloud Run Jobs in us-central1 are slow to start on Google's side.** Our image, CPU and code aren't the cause. - The `Started` condition fires about 10 s before the first Python log, so our process startup is fine. - decide's 203 MB image on 4 CPU is just as slow in us-central1. - A throwaway job running Google's tiny sample image, 4 runs per region: **us-central1 4m–5m07s**, **us-east4 8–19 s**, **us-east1 8–25 s**. All four us-central1 runs started in the same second, as if released from a queue. Quotas are far from their limits. ## Changes - **backend**: new `CPU_JOBS_REGION` setting (default `us-east4`) for triggering `audio-download-job`, `lyrics-transcription-job` and `bulk-search-job`. `GCP_REGION` is unchanged. `video-encoding-job` stays in us-central1, because it pulls multi-GB outputs from the us-central1 bucket. - **infra (Pulumi, already applied)**: the three jobs are created in us-east4 as new resources. The legacy us-central1 copies are kept as the rollback path, with deletion protection turned off so a follow-up can remove them. The preview after applying showed 289 unchanged, so there's no drift. - **ci**: the CPU app image is also pushed to `us-east4-docker.pkg.dev/nomadkaraoke/karaoke-backend-gpu/karaoke-backend-cpu`, so jobs pull from their own region. The lyrics job image update now targets us-east4. After each deploy, CI starts a no-op `audio-download-job` execution so the one-time ~2.5 min image import doesn't land on the first customer. - `scripts/rollback.sh`: lyrics job moves to the us-east4 image, and jobs whose target tag is missing are skipped instead of aborting the loop. - Docs: TROUBLESHOOTING runbook (measure + rollback), ARCHITECTURE, LESSONS-LEARNED. ## Testing - `make test`: all backend tests pass, plus 1422 frontend tests. - New tests in `backend/tests/test_worker_service_cpu_jobs_region.py`: routing of each moved job, video job stays in `GCP_REGION`, env override for rollback, and the Pulumi/backend region constants must match. - Prod smoke test of the new us-east4 `audio-download-job` with the real image (`import backend.workers.audio_download_worker`): **Started in 7.0 s, 8.6 s, 10.0 s** (first run of a new digest adds a one-time ~2.5 min import, which the CI step covers). - Local review via a reviewer agent: no must-fix findings; the rollback-path suggestions are addressed. ## Rollout One-time image seed (Cloud Build gcrane copy) and `pulumi up` are done. After merge, CI deploys the backend, which starts triggering the us-east4 jobs. 🤖 Generated with [Claude Code](https://claude.com/claude-code) <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit * **Performance** * Audio downloads, bulk searches, and lyric transcriptions now run in a region with faster job startup times, reducing delays before processing begins. Video encoding continues to run in its existing region. * **Reliability** * Rollbacks now skip jobs when the target image is unavailable and continue with the remaining jobs. Other image lookup errors still stop the rollback. * **Documentation** * Updated architecture and troubleshooting guides with regional job details, startup-delay guidance, and rollback instructions. <!-- end of auto-generated comment: release notes by coderabbit.ai --> --------- Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com>