perf(jobs): run download/lyrics/bulk-search Cloud Run Jobs in us-east4 — cut 2-5 min job start latency (v0.247.1) (#1077)

## Summary
Every gen job waited **2–5 min** before `audio-download-job` (and then
`lyrics-transcription-job`) even started. That blocked kjbox's goal of
singing within about 4 minutes of making a song.

**Root cause: Cloud Run Jobs in us-central1 are slow to start on
Google's side.** Our image, CPU and code aren't the cause.
- The `Started` condition fires about 10 s before the first Python log,
so our process startup is fine.
- decide's 203 MB image on 4 CPU is just as slow in us-central1.
- A throwaway job running Google's tiny sample image, 4 runs per region:
**us-central1 4m–5m07s**, **us-east4 8–19 s**, **us-east1 8–25 s**. All
four us-central1 runs started in the same second, as if released from a
queue. Quotas are far from their limits.

## Changes
- **backend**: new `CPU_JOBS_REGION` setting (default `us-east4`) for
triggering `audio-download-job`, `lyrics-transcription-job` and
`bulk-search-job`. `GCP_REGION` is unchanged. `video-encoding-job` stays
in us-central1, because it pulls multi-GB outputs from the us-central1
bucket.
- **infra (Pulumi, already applied)**: the three jobs are created in
us-east4 as new resources. The legacy us-central1 copies are kept as the
rollback path, with deletion protection turned off so a follow-up can
remove them. The preview after applying showed 289 unchanged, so there's
no drift.
- **ci**: the CPU app image is also pushed to
`us-east4-docker.pkg.dev/nomadkaraoke/karaoke-backend-gpu/karaoke-backend-cpu`,
so jobs pull from their own region. The lyrics job image update now
targets us-east4. After each deploy, CI starts a no-op
`audio-download-job` execution so the one-time ~2.5 min image import
doesn't land on the first customer.
- `scripts/rollback.sh`: lyrics job moves to the us-east4 image, and
jobs whose target tag is missing are skipped instead of aborting the
loop.
- Docs: TROUBLESHOOTING runbook (measure + rollback), ARCHITECTURE,
LESSONS-LEARNED.

## Testing
- `make test`: all backend tests pass, plus 1422 frontend tests.
- New tests in `backend/tests/test_worker_service_cpu_jobs_region.py`:
routing of each moved job, video job stays in `GCP_REGION`, env override
for rollback, and the Pulumi/backend region constants must match.
- Prod smoke test of the new us-east4 `audio-download-job` with the real
image (`import backend.workers.audio_download_worker`): **Started in 7.0
s, 8.6 s, 10.0 s** (first run of a new digest adds a one-time ~2.5 min
import, which the CI step covers).
- Local review via a reviewer agent: no must-fix findings; the
rollback-path suggestions are addressed.

## Rollout
One-time image seed (Cloud Build gcrane copy) and `pulumi up` are done.
After merge, CI deploys the backend, which starts triggering the
us-east4 jobs.

🤖 Generated with [Claude Code](https://claude.com/claude-code)


<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit

* **Performance**
* Audio downloads, bulk searches, and lyric transcriptions now run in a
region with faster job startup times, reducing delays before processing
begins. Video encoding continues to run in its existing region.
* **Reliability**
* Rollbacks now skip jobs when the target image is unavailable and
continue with the remaining jobs. Other image lookup errors still stop
the rollback.
* **Documentation**
* Updated architecture and troubleshooting guides with regional job
details, startup-delay guidance, and rollback instructions.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->

---------

Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
This commit is contained in:
Andrew Beveridge ☄️andClaude Opus 5.5 authored and GitHub committed 2026-09-29 01:02:19 -04:00
1 parent 518126727b
commit 3e855ff65d
11 files changed
+287 -39

No files matched your search

+34 -3
View File
@@ -1106,6 +1106,13 @@ jobs:
GPU_REPOSITORY: karaoke-backend-gpu
GPU_BASE_IMAGE_NAME: karaoke-backend-gpu-base
GPU_APP_IMAGE_NAME: karaoke-backend
# CPU app image copy for the latency-critical CPU jobs (audio-download,
# lyrics-transcription, bulk-search) in us-east4, where Cloud Run Jobs start
# in ~15s instead of 2-5 min. Lives in the existing us-east4 repo so the jobs
# pull same-region; only changed app layers cross regions on each push.
CPU_JOBS_REGISTRY: us-east4-docker.pkg.dev
CPU_JOBS_REPOSITORY: karaoke-backend-gpu
CPU_JOBS_IMAGE_NAME: karaoke-backend-cpu
steps:
- name: Check disk space (aggressive for Docker builds)
run: |
@@ -1859,12 +1866,32 @@ jobs:
${{ env.REGISTRY }}/${{ env.PROJECT_ID }}/${{ env.REPOSITORY }}/${{ env.APP_IMAGE_NAME }}:latest
${{ env.REGISTRY }}/${{ env.PROJECT_ID }}/${{ env.REPOSITORY }}/${{ env.APP_IMAGE_NAME }}:${{ github.sha }}
${{ env.REGISTRY }}/${{ env.PROJECT_ID }}/${{ env.REPOSITORY }}/${{ env.APP_IMAGE_NAME }}:v${{ steps.version_check.outputs.version }}
${{ env.CPU_JOBS_REGISTRY }}/${{ env.PROJECT_ID }}/${{ env.CPU_JOBS_REPOSITORY }}/${{ env.CPU_JOBS_IMAGE_NAME }}:latest
${{ env.CPU_JOBS_REGISTRY }}/${{ env.PROJECT_ID }}/${{ env.CPU_JOBS_REPOSITORY }}/${{ env.CPU_JOBS_IMAGE_NAME }}:${{ github.sha }}
${{ env.CPU_JOBS_REGISTRY }}/${{ env.PROJECT_ID }}/${{ env.CPU_JOBS_REPOSITORY }}/${{ env.CPU_JOBS_IMAGE_NAME }}:v${{ steps.version_check.outputs.version }}
# Registry cache for app layers
# ignore-error=true allows build to continue if cache doesn't exist yet
cache-from: type=registry,ref=${{ env.REGISTRY }}/${{ env.PROJECT_ID }}/${{ env.REPOSITORY }}/${{ env.APP_IMAGE_NAME }}:cache,ignore-error=true
cache-to: type=registry,ref=${{ env.REGISTRY }}/${{ env.PROJECT_ID }}/${{ env.REPOSITORY }}/${{ env.APP_IMAGE_NAME }}:cache,mode=max,ignore-error=true
provenance: false
- name: Pre-import CPU image for us-east4 jobs
# A new image digest costs a one-time ~2.5 min "Container image import" on
# its FIRST execution in a region (measured 2026-09-29); later executions
# of any job using that digest start in ~8s. audio-download-job runs
# :latest (just pushed), so a no-op execution now performs the import
# while the service deploy + smoke test run, instead of on the first
# customer download. Async on purpose: the import overlaps the rest of
# the deploy. Non-fatal: worst case the first download pays the import.
run: |
gcloud run jobs execute audio-download-job \
--region us-east4 \
--project ${{ env.PROJECT_ID }} \
--args="python,-c,pass" \
--async \
--quiet \
|| echo "::warning::us-east4 CPU image pre-import execution failed to start (non-fatal)"
# ============================================
# GPU Image Build (for audio worker with L4 GPU)
# Same pattern as base image: hash check → rebuild if needed.
@@ -2127,6 +2154,7 @@ jobs:
run: |
IMAGE="${{ env.REGISTRY }}/${{ env.PROJECT_ID }}/${{ env.REPOSITORY }}/${{ env.APP_IMAGE_NAME }}:${{ github.sha }}"
GPU_IMAGE="${{ env.GPU_REGISTRY }}/${{ env.PROJECT_ID }}/${{ env.GPU_REPOSITORY }}/${{ env.GPU_APP_IMAGE_NAME }}:${{ github.sha }}"
CPU_JOBS_IMAGE="${{ env.CPU_JOBS_REGISTRY }}/${{ env.PROJECT_ID }}/${{ env.CPU_JOBS_REPOSITORY }}/${{ env.CPU_JOBS_IMAGE_NAME }}:${{ github.sha }}"
echo "📦 Updating Cloud Run Jobs..."
@@ -2137,7 +2165,10 @@ jobs:
--project ${{ env.PROJECT_ID }} \
--quiet
# Lyrics transcription job (CPU, us-central1)
# Lyrics transcription job (CPU, us-east4 — see CPU_JOBS_REGION in
# backend/config.py). The us-east4 job is created by Pulumi, so
# `pulumi up` must run BEFORE merging a change that introduces it. audio-download-job and bulk-search-job track the
# :latest tag pushed above, so they need no update here.
# NOTE: image-only update. Adding --update-secrets here crashed gcloud
# ("Invalid secret path ... in annotation") because this job's existing
# secrets are stored via Cloud Run secret *aliases* that --update-secrets
@@ -2147,8 +2178,8 @@ jobs:
# doc); until then the proactive flag stays unset and the worker's
# _run_proactive_auto_correct() returns early (no-op).
gcloud run jobs update lyrics-transcription-job \
--image "$IMAGE" \
--region us-central1 \
--image "$CPU_JOBS_IMAGE" \
--region us-east4 \
--project ${{ env.PROJECT_ID }} \
--quiet