mirror of
https://github.com/nomadkaraoke/karaoke-gen.git
synced 2026-11-20 13:07:57 +09:00
## Summary
A freshly-spawned `karaoke-backend` Cloud Run instance took **~16s
before serving its first request** — Cloud Run holds routed requests
during boot, so every scale-out was a 16s latency cliff (measured
2026-09-22 during the 16-tab load test: warm instance 1.2s vs fresh
instances 15.8–18.8s for the same request). Handoff spec:
`docs/archive/2026-09-22-gen-cold-start-reduction-handoff.md`
(workspace).
Measured breakdown: **~8–9s Python imports** + **~7.5s lifespan
preloads** (spaCy ~1.0s, NLTK cmudict+punkt ~2.3s, Langfuse ~3s,
credential validation ~1.6s — all gating readiness, from prod logs).
## Changes
**Lazy imports, fixed at the source module** so every importer benefits
and the CLI keeps working (imports now happen at first use inside worker
code paths):
| Module | Deferred | Cost removed |
|---|---|---|
| `karaoke_gen/.../utils/syllable_counter.py` | spacy, nltk, pyphen,
syllables → `__init__`/methods | ~2.6s |
| `karaoke_gen/.../correction/phrase_analyzer.py` | spacy → `__init__`
(was also pulling **torch** via thinc) | ~1.7s |
| `karaoke_gen/audio_processor.py` | audio_separator
(torch/onnxruntime/librosa), pydub | ~2.9s |
| `backend/config.py` | google.cloud.secretmanager → `get_secret()` |
~1.7s |
| `backend/services/{custom_lyrics,auto_correct}/service.py` |
google.genai → `_call_gemini()` | ~1.2s |
| audioshake/whisper transcribers, cdgmaker composer | pydub → using
methods | small |
`audio_processor` keeps `Separator` / `AudioSeparatorAPIClient` /
`REMOTE_API_AVAILABLE` / `SEPARATOR_AVAILABLE` as (lazy) module
attributes via PEP 562 `__getattr__`, so existing test patch targets are
unchanged; intra-module code uses helpers because LOAD_GLOBAL bypasses
module `__getattr__`. Same pattern keeps `custom_lyrics.service.genai`
patchable.
**Lifespan** (`backend/main.py`): the spaCy/NLTK/Langfuse preloads +
credential validation moved into a daemon thread
(`_run_background_warmup`). They warm caches for the *first
lyrics-processing job* — they never needed to gate HTTP readiness.
Workers racing the warmup fall back to the preloaders' existing lazy
paths. Heavy imports happen in this thread, not in async handlers
(event-loop-blocking gotcha from v0.230.0/v0.231.1).
## Results
- Local `import backend.main`: **10.63s → 2.75s** (remaining is
dominated by `google.cloud.firestore` ~0.8s, a genuine serving
dependency)
- Lifespan startup: ~7.5s → instant ("Application startup complete"
same-second)
- Expected prod cold start: **~16s → ~3–4s** (to be verified post-deploy
with `scripts/load-test-review.py --tabs 16` + fresh-instance request
latency)
## Tests
- **New:** `backend/tests/test_cold_start_imports.py` —
subprocess-imports `backend.main`, asserts none of the heavy libraries
(torch, spacy, nltk, audio_separator, matplotlib, pydub, google.genai,
secretmanager, librosa, onnxruntime) leak into `sys.modules`;
`backend/tests/test_main_startup_warmup.py` — warmup calls all preloads,
survives individual failures, and doesn't block readiness.
- Full backend + package suites: **4541 passed** (incl. the
audio_processor/custom_lyrics/auto_correct patch-target suites). Ruff:
no new findings on changed files.
## Follow-up (separate PR after prod verification)
If fresh-instance first-request latency lands <5s, drop `--min-instances
2 → 1` in `ci.yml` — the second warm instance was the cold-start
mitigation (~$15/mo).
🤖 Generated with [Claude Code](https://claude.com/claude-code)
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
- **Performance**
- Reduced backend cold-start time by deferring heavy dependencies until
they are needed.
- Startup readiness is no longer blocked by model, language-resource,
analytics, or credential warmups; these now run in the background.
- Secret Manager and audio, transcription, lyric, and correction tools
load on demand.
- **Documentation**
- Added guidance and measured results for diagnosing and reducing
backend cold starts.
- **Quality**
- Added automated checks to ensure startup remains lightweight and
background warmups continue safely after individual failures.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
---------
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>