perf: Cloud Run burst headroom — concurrency 32 + min-instances 2 (#1035)

## Summary

Final tuning from the 16-tab test diagnosis (#1033 fixed the edge rate
limit eating the requests; this fixes the remaining latency). With
nothing failing anymore, the warm 16-tab run still showed every tab at
~17–19s — server logs pinpointed why: the one warm instance answered
`correction-data` in **1.2s** while the three instances the burst
spawned answered in **15.8–18.8s** — Cloud Run holds requests during the
~16s app boot.

Concurrency 16 (set in #1028 while the event loop still had blocking
hazards) now overshoots: the loop is clean, one instance juggles dozens
of light requests fine, and forcing scale-out per burst just converts
load into cold-start waits.

## Change

`--concurrency 16 → 32` and `--min-instances 1 → 2`: 64 warm request
slots, so a 16-tab reload (~48 concurrent API calls) is absorbed with
zero cold starts. Still far below the original 80 that allowed the
2026-09-18 single-instance wedge.

## 💰 Cost note (Andrew)

The second min-instance idles at Cloud Run idle rates — roughly
**$15/month**. Single-flag revert if you'd rather not pay it; without it
the first burst after idle pays one ~16s cold-start wave for the
overflow beyond 32 concurrent slots.

## Follow-up worth considering (not in this PR)

App cold start is ~16s — heavy import graph (matplotlib, pydub, GCP
clients). Lazy-importing the heavy tails would benefit every scale-out.

## Testing

Deploy-flag-only change. Post-deploy: re-run
`scripts/load-test-review.py --tabs 16` — expect single-digit-second
tabs with no cold-start waves.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
This commit is contained in:
Andrew Beveridge ☄️andClaude Fable 5 authored and GitHub committed 2026-09-22 18:26:41 -04:00
1 parent 438efb751d
commit 6514673fde
2 files changed
+12 -9

No files matched your search

+11 -8
View File
@@ -1976,12 +1976,15 @@ jobs:
echo "Version: $VERSION"
echo "Previously-serving revision: ${PREV_REVISION:-<unknown>}"
# --concurrency 16 (was 80): the backend is one uvicorn process per
# 2-vCPU instance; review-page loads are heavy (audio byte-serving,
# waveform work). At 80 a 10-tab burst all landed on the single
# min-instance and wedged it ("no available instance" aborts,
# 2026-09-18 investigation). 16 makes bursts trigger scale-out while
# staying comfortably above normal per-instance load.
# --concurrency 32 (was 80 → 16 → 32): at 80 a 10-tab burst all
# landed on the single min-instance and wedged its then-blocking
# event loop ("no available instance" aborts, 2026-09-18). 16 fixed
# that but overshot once the loop was made non-blocking: every burst
# forced scale-out, and each overflow request waited out a ~16s app
# cold start (2026-09-22 16-tab test: warm instance served in 1.2s,
# freshly-spawned ones in 16-19s). 32 + --min-instances 2 gives 64
# warm slots so a 16-tab reload is absorbed without cold starts,
# while staying far below the wedge-era 80.
gcloud run deploy karaoke-backend \
--image $IMAGE \
--no-traffic \
@@ -1995,9 +1998,9 @@ jobs:
--cpu-throttling \
--execution-environment gen2 \
--timeout 1800 \
--concurrency 16 \
--concurrency 32 \
--max-instances 20 \
--min-instances 1 \
--min-instances 2 \
--vpc-connector projects/nomadkaraoke/locations/us-central1/connectors/cloud-run-connector \
--vpc-egress private-ranges-only \
--set-env-vars "GOOGLE_CLOUD_PROJECT=${{ env.PROJECT_ID }},GCS_BUCKET_NAME=karaoke-gen-storage-${{ env.PROJECT_ID }},FIRESTORE_COLLECTION=jobs,ENVIRONMENT=production,KARAOKE_GEN_VERSION=$VERSION,DEFAULT_DROPBOX_PATH=/MediaUnsynced/Karaoke/Tracks-Organized,DEFAULT_GDRIVE_FOLDER_ID=1laRKAyxo0v817SstfM5XkpbWiNKNAMSX,ENABLE_CLOUD_TASKS=true,CLOUD_RUN_SERVICE_URL=https://api.nomadkaraoke.com,GCP_REGION=us-central1,DEFAULT_ENABLE_YOUTUBE_UPLOAD=true,DEFAULT_BRAND_PREFIX=NOMAD,DEFAULT_PRIVATE_DROPBOX_PATH=/MediaUnsynced/Karaoke/Tracks-NonPublished,DEFAULT_PRIVATE_BRAND_PREFIX=NOMADNP,USE_GCE_ENCODING=true,USE_GCE_PREVIEW_ENCODING=true,ENABLE_PUSH_NOTIFICATIONS=true,USE_CLOUD_RUN_JOBS_FOR_VIDEO=true,USE_CLOUD_RUN_JOBS_FOR_RENDER=true,EMAIL_PROVIDER=postmark,AUTO_CORRECT_MODEL=claude-opus-4-8,AUTO_CORRECT_COMPARE_MODELS=claude-opus-4-8;gemini-3.1-pro-preview,AUTO_CORRECT_PROACTIVE_ENABLED=true,AUTO_APPROVAL_BACKING_KEEP_ENABLED=true,COMMUNITY_DAILY_PICK_ENABLED=true,EDGE_AUTH_MODE=enforce,COMMIT_SHA=${{ steps.build-meta.outputs.sha }},PR_NUMBER=${{ steps.build-meta.outputs.pr_number }},PR_TITLE=${{ steps.build-meta.outputs.pr_title }}" \
+1 -1
View File
@@ -1,6 +1,6 @@
[tool.poetry]
name = "karaoke-gen"
version = "0.234.0"
version = "0.234.1"
description = "Generate karaoke videos with synchronized lyrics. Handles the entire process from downloading audio and lyrics to creating the final video with title screens."
authors = ["Andrew Beveridge <andrew@beveridge.uk>"]
license = "MIT"