Files
karaoke-gen/.github
Andrew Beveridge ☄️andClaude Opus 5.5 3e855ff65d perf(jobs): run download/lyrics/bulk-search Cloud Run Jobs in us-east4 — cut 2-5 min job start latency (v0.247.1) (#1077)
## Summary
Every gen job waited **2–5 min** before `audio-download-job` (and then
`lyrics-transcription-job`) even started. That blocked kjbox's goal of
singing within about 4 minutes of making a song.

**Root cause: Cloud Run Jobs in us-central1 are slow to start on
Google's side.** Our image, CPU and code aren't the cause.
- The `Started` condition fires about 10 s before the first Python log,
so our process startup is fine.
- decide's 203 MB image on 4 CPU is just as slow in us-central1.
- A throwaway job running Google's tiny sample image, 4 runs per region:
**us-central1 4m–5m07s**, **us-east4 8–19 s**, **us-east1 8–25 s**. All
four us-central1 runs started in the same second, as if released from a
queue. Quotas are far from their limits.

## Changes
- **backend**: new `CPU_JOBS_REGION` setting (default `us-east4`) for
triggering `audio-download-job`, `lyrics-transcription-job` and
`bulk-search-job`. `GCP_REGION` is unchanged. `video-encoding-job` stays
in us-central1, because it pulls multi-GB outputs from the us-central1
bucket.
- **infra (Pulumi, already applied)**: the three jobs are created in
us-east4 as new resources. The legacy us-central1 copies are kept as the
rollback path, with deletion protection turned off so a follow-up can
remove them. The preview after applying showed 289 unchanged, so there's
no drift.
- **ci**: the CPU app image is also pushed to
`us-east4-docker.pkg.dev/nomadkaraoke/karaoke-backend-gpu/karaoke-backend-cpu`,
so jobs pull from their own region. The lyrics job image update now
targets us-east4. After each deploy, CI starts a no-op
`audio-download-job` execution so the one-time ~2.5 min image import
doesn't land on the first customer.
- `scripts/rollback.sh`: lyrics job moves to the us-east4 image, and
jobs whose target tag is missing are skipped instead of aborting the
loop.
- Docs: TROUBLESHOOTING runbook (measure + rollback), ARCHITECTURE,
LESSONS-LEARNED.

## Testing
- `make test`: all backend tests pass, plus 1422 frontend tests.
- New tests in `backend/tests/test_worker_service_cpu_jobs_region.py`:
routing of each moved job, video job stays in `GCP_REGION`, env override
for rollback, and the Pulumi/backend region constants must match.
- Prod smoke test of the new us-east4 `audio-download-job` with the real
image (`import backend.workers.audio_download_worker`): **Started in 7.0
s, 8.6 s, 10.0 s** (first run of a new digest adds a one-time ~2.5 min
import, which the CI step covers).
- Local review via a reviewer agent: no must-fix findings; the
rollback-path suggestions are addressed.

## Rollout
One-time image seed (Cloud Build gcrane copy) and `pulumi up` are done.
After merge, CI deploys the backend, which starts triggering the
us-east4 jobs.

🤖 Generated with [Claude Code](https://claude.com/claude-code)


<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit

* **Performance**
* Audio downloads, bulk searches, and lyric transcriptions now run in a
region with faster job startup times, reducing delays before processing
begins. Video encoding continues to run in its existing region.
* **Reliability**
* Rollbacks now skip jobs when the target image is unavailable and
continue with the remaining jobs. Other image lookup errors still stop
the rollback.
* **Documentation**
* Updated architecture and troubleshooting guides with regional job
details, startup-delay guidance, and rollback instructions.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->

---------

Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-09-29 01:02:19 -04:00
..
2024-09-17 12:18:12 -04:00
2025-12-11 03:49:32 -05:00