mirror of
https://github.com/nomadkaraoke/karaoke-gen.git
synced 2026-11-20 13:07:57 +09:00
perf(jobs): run download/lyrics/bulk-search Cloud Run Jobs in us-east4 — cut 2-5 min job start latency (v0.247.1) (#1077)
## Summary Every gen job waited **2–5 min** before `audio-download-job` (and then `lyrics-transcription-job`) even started. That blocked kjbox's goal of singing within about 4 minutes of making a song. **Root cause: Cloud Run Jobs in us-central1 are slow to start on Google's side.** Our image, CPU and code aren't the cause. - The `Started` condition fires about 10 s before the first Python log, so our process startup is fine. - decide's 203 MB image on 4 CPU is just as slow in us-central1. - A throwaway job running Google's tiny sample image, 4 runs per region: **us-central1 4m–5m07s**, **us-east4 8–19 s**, **us-east1 8–25 s**. All four us-central1 runs started in the same second, as if released from a queue. Quotas are far from their limits. ## Changes - **backend**: new `CPU_JOBS_REGION` setting (default `us-east4`) for triggering `audio-download-job`, `lyrics-transcription-job` and `bulk-search-job`. `GCP_REGION` is unchanged. `video-encoding-job` stays in us-central1, because it pulls multi-GB outputs from the us-central1 bucket. - **infra (Pulumi, already applied)**: the three jobs are created in us-east4 as new resources. The legacy us-central1 copies are kept as the rollback path, with deletion protection turned off so a follow-up can remove them. The preview after applying showed 289 unchanged, so there's no drift. - **ci**: the CPU app image is also pushed to `us-east4-docker.pkg.dev/nomadkaraoke/karaoke-backend-gpu/karaoke-backend-cpu`, so jobs pull from their own region. The lyrics job image update now targets us-east4. After each deploy, CI starts a no-op `audio-download-job` execution so the one-time ~2.5 min image import doesn't land on the first customer. - `scripts/rollback.sh`: lyrics job moves to the us-east4 image, and jobs whose target tag is missing are skipped instead of aborting the loop. - Docs: TROUBLESHOOTING runbook (measure + rollback), ARCHITECTURE, LESSONS-LEARNED. ## Testing - `make test`: all backend tests pass, plus 1422 frontend tests. - New tests in `backend/tests/test_worker_service_cpu_jobs_region.py`: routing of each moved job, video job stays in `GCP_REGION`, env override for rollback, and the Pulumi/backend region constants must match. - Prod smoke test of the new us-east4 `audio-download-job` with the real image (`import backend.workers.audio_download_worker`): **Started in 7.0 s, 8.6 s, 10.0 s** (first run of a new digest adds a one-time ~2.5 min import, which the CI step covers). - Local review via a reviewer agent: no must-fix findings; the rollback-path suggestions are addressed. ## Rollout One-time image seed (Cloud Build gcrane copy) and `pulumi up` are done. After merge, CI deploys the backend, which starts triggering the us-east4 jobs. 🤖 Generated with [Claude Code](https://claude.com/claude-code) <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit * **Performance** * Audio downloads, bulk searches, and lyric transcriptions now run in a region with faster job startup times, reducing delays before processing begins. Video encoding continues to run in its existing region. * **Reliability** * Rollbacks now skip jobs when the target image is unavailable and continue with the remaining jobs. Other image lookup errors still stop the rollback. * **Documentation** * Updated architecture and troubleshooting guides with regional job details, startup-delay guidance, and rollback instructions. <!-- end of auto-generated comment: release notes by coderabbit.ai --> --------- Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
This commit is contained in:
1 parent
518126727b
commit
3e855ff65d
11 files changed
+287
-39
No files matched your search
@@ -1106,6 +1106,13 @@ jobs:
|
||||
GPU_REPOSITORY: karaoke-backend-gpu
|
||||
GPU_BASE_IMAGE_NAME: karaoke-backend-gpu-base
|
||||
GPU_APP_IMAGE_NAME: karaoke-backend
|
||||
# CPU app image copy for the latency-critical CPU jobs (audio-download,
|
||||
# lyrics-transcription, bulk-search) in us-east4, where Cloud Run Jobs start
|
||||
# in ~15s instead of 2-5 min. Lives in the existing us-east4 repo so the jobs
|
||||
# pull same-region; only changed app layers cross regions on each push.
|
||||
CPU_JOBS_REGISTRY: us-east4-docker.pkg.dev
|
||||
CPU_JOBS_REPOSITORY: karaoke-backend-gpu
|
||||
CPU_JOBS_IMAGE_NAME: karaoke-backend-cpu
|
||||
steps:
|
||||
- name: Check disk space (aggressive for Docker builds)
|
||||
run: |
|
||||
@@ -1859,12 +1866,32 @@ jobs:
|
||||
${{ env.REGISTRY }}/${{ env.PROJECT_ID }}/${{ env.REPOSITORY }}/${{ env.APP_IMAGE_NAME }}:latest
|
||||
${{ env.REGISTRY }}/${{ env.PROJECT_ID }}/${{ env.REPOSITORY }}/${{ env.APP_IMAGE_NAME }}:${{ github.sha }}
|
||||
${{ env.REGISTRY }}/${{ env.PROJECT_ID }}/${{ env.REPOSITORY }}/${{ env.APP_IMAGE_NAME }}:v${{ steps.version_check.outputs.version }}
|
||||
${{ env.CPU_JOBS_REGISTRY }}/${{ env.PROJECT_ID }}/${{ env.CPU_JOBS_REPOSITORY }}/${{ env.CPU_JOBS_IMAGE_NAME }}:latest
|
||||
${{ env.CPU_JOBS_REGISTRY }}/${{ env.PROJECT_ID }}/${{ env.CPU_JOBS_REPOSITORY }}/${{ env.CPU_JOBS_IMAGE_NAME }}:${{ github.sha }}
|
||||
${{ env.CPU_JOBS_REGISTRY }}/${{ env.PROJECT_ID }}/${{ env.CPU_JOBS_REPOSITORY }}/${{ env.CPU_JOBS_IMAGE_NAME }}:v${{ steps.version_check.outputs.version }}
|
||||
# Registry cache for app layers
|
||||
# ignore-error=true allows build to continue if cache doesn't exist yet
|
||||
cache-from: type=registry,ref=${{ env.REGISTRY }}/${{ env.PROJECT_ID }}/${{ env.REPOSITORY }}/${{ env.APP_IMAGE_NAME }}:cache,ignore-error=true
|
||||
cache-to: type=registry,ref=${{ env.REGISTRY }}/${{ env.PROJECT_ID }}/${{ env.REPOSITORY }}/${{ env.APP_IMAGE_NAME }}:cache,mode=max,ignore-error=true
|
||||
provenance: false
|
||||
|
||||
- name: Pre-import CPU image for us-east4 jobs
|
||||
# A new image digest costs a one-time ~2.5 min "Container image import" on
|
||||
# its FIRST execution in a region (measured 2026-09-29); later executions
|
||||
# of any job using that digest start in ~8s. audio-download-job runs
|
||||
# :latest (just pushed), so a no-op execution now performs the import
|
||||
# while the service deploy + smoke test run, instead of on the first
|
||||
# customer download. Async on purpose: the import overlaps the rest of
|
||||
# the deploy. Non-fatal: worst case the first download pays the import.
|
||||
run: |
|
||||
gcloud run jobs execute audio-download-job \
|
||||
--region us-east4 \
|
||||
--project ${{ env.PROJECT_ID }} \
|
||||
--args="python,-c,pass" \
|
||||
--async \
|
||||
--quiet \
|
||||
|| echo "::warning::us-east4 CPU image pre-import execution failed to start (non-fatal)"
|
||||
|
||||
# ============================================
|
||||
# GPU Image Build (for audio worker with L4 GPU)
|
||||
# Same pattern as base image: hash check → rebuild if needed.
|
||||
@@ -2127,6 +2154,7 @@ jobs:
|
||||
run: |
|
||||
IMAGE="${{ env.REGISTRY }}/${{ env.PROJECT_ID }}/${{ env.REPOSITORY }}/${{ env.APP_IMAGE_NAME }}:${{ github.sha }}"
|
||||
GPU_IMAGE="${{ env.GPU_REGISTRY }}/${{ env.PROJECT_ID }}/${{ env.GPU_REPOSITORY }}/${{ env.GPU_APP_IMAGE_NAME }}:${{ github.sha }}"
|
||||
CPU_JOBS_IMAGE="${{ env.CPU_JOBS_REGISTRY }}/${{ env.PROJECT_ID }}/${{ env.CPU_JOBS_REPOSITORY }}/${{ env.CPU_JOBS_IMAGE_NAME }}:${{ github.sha }}"
|
||||
|
||||
echo "📦 Updating Cloud Run Jobs..."
|
||||
|
||||
@@ -2137,7 +2165,10 @@ jobs:
|
||||
--project ${{ env.PROJECT_ID }} \
|
||||
--quiet
|
||||
|
||||
# Lyrics transcription job (CPU, us-central1)
|
||||
# Lyrics transcription job (CPU, us-east4 — see CPU_JOBS_REGION in
|
||||
# backend/config.py). The us-east4 job is created by Pulumi, so
|
||||
# `pulumi up` must run BEFORE merging a change that introduces it. audio-download-job and bulk-search-job track the
|
||||
# :latest tag pushed above, so they need no update here.
|
||||
# NOTE: image-only update. Adding --update-secrets here crashed gcloud
|
||||
# ("Invalid secret path ... in annotation") because this job's existing
|
||||
# secrets are stored via Cloud Run secret *aliases* that --update-secrets
|
||||
@@ -2147,8 +2178,8 @@ jobs:
|
||||
# doc); until then the proactive flag stays unset and the worker's
|
||||
# _run_proactive_auto_correct() returns early (no-op).
|
||||
gcloud run jobs update lyrics-transcription-job \
|
||||
--image "$IMAGE" \
|
||||
--region us-central1 \
|
||||
--image "$CPU_JOBS_IMAGE" \
|
||||
--region us-east4 \
|
||||
--project ${{ env.PROJECT_ID }} \
|
||||
--quiet
|
||||
|
||||
|
||||
Reference in new issue
Block a user