mirror of
https://github.com/nomadkaraoke/karaoke-gen.git
synced 2026-11-20 13:07:57 +09:00
@coderabbitai ignore ## Why LRCLIB now applies stricter rate limits to unregistered clients (https://lrclib.net/docs#implementation-requirements-rate-limiting), and Cloud Run shares egress IPs. On 10-19, prod jobs got intermittent 503s, and job 80e7ffdf lost its LRCLIB lyrics. LRCLIB is our only free source; it supplied lyrics for 165 of 220 jobs in 30 days. Andrew registered a server-side app. ## What - **`karaoke_gen/lyrics_transcriber/lyrics/lrclib_client.py`** (new): - Sends `Lrclib-App-Id` + `Lrclib-Api-Key` from `LRCLIB_APP_ID` / `LRCLIB_SECRET_KEY`. - Applies LRCLIB's rules: every 429 sets a **process-wide cooldown** of `Retry-After` plus up to 30% jitter. The wait is never shortened, and overlapping 429s keep the latest expiry. Nothing is sent until the cooldown has passed. It waits and retries once only when the wait is ≤30s. - Retries 5xx after 1s/2s, honouring a 5xx `Retry-After`. - A **401 on our credentials** logs a WARNING and drops to anonymous access for the process, so a revoked key degrades instead of losing LRCLIB. - `LRCLIBProvider` (both `get` and `search`) uses it. The `quick_version` renderer uses it with `retry=False`, so the fast path gets the auth and the cooldown but never blocks. - **Infra:** - Pulumi secrets `lrclib-app-id` / `lrclib-api-key`. - Env vars on the `lyrics-transcription-job`. - The backend service deploy's `--set-secrets` (`ci.yml`). The karaoke-backend SA already has project-wide `secretAccessor`. ## Deploy order (do NOT merge before steps 1–2) 1. `pulumi up` to create the two secrets and the job env. Needs `/gcp-admin`. 2. Add the secret versions from Andrew's `.envrc` values. 3. Merge. The CI backend deploy references the secrets. ## Testing - New `tests/unit/test_lrclib_client.py` (14 tests): headers; anonymous use; Retry-After with jitter; a long Retry-After is not shortened and blocks further calls; no-retry mode; a second 429; 5xx backoff; 5xx Retry-After; 401 fallback; overlap merge (mutation-checked); provider routes through the client. - 164 passed across the client, `quick_version` and `search_lyrics` tests. - **Live** with the real app key: Dar Williams - Iowa and Wolfmother - Where Eagles Have Been were found. One 503 was recovered by the retry. - Local CodeRabbit, 3 cycles. I fixed the Retry-After cap, the cooldown after a final 429, the `quick_version` 401 path, the overlap merge and 5xx Retry-After. I skipped one false positive about the tests' fake clock (the retry path sleeps, then sends without re-checking the clock). 🤖 Generated with [Claude Code](https://claude.com/claude-code) https://claude.ai/code/session_01Jw21syo4HizJJjYw2mg2Vp --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Backend Development Setup
One-Time Setup
1. Create Virtual Environment
cd /Users/andrew/Projects/karaoke-gen
python3 -m venv backend/venv
2. Activate Virtual Environment
source backend/venv/bin/activate
3. Install Dependencies
IMPORTANT: Use Python 3.12 (not 3.13/3.14 - pydantic has compatibility issues)
# Backend dependencies
pip install -r backend/requirements.txt
# Install karaoke_gen in development mode
pip install -e .
# Install lyrics_transcriber submodule
pip install -e lyrics_transcriber_local/
# Install Google API client (for YouTube upload)
pip install google-api-python-client google-auth-httplib2 google-auth-oauthlib
4. Add Shell Alias (Optional)
Add to ~/.zshrc:
alias backend='cd /Users/andrew/Projects/karaoke-gen && source backend/venv/bin/activate'
Then you can just run:
backend
# Now you're in the project directory with venv activated!
Daily Workflow
Before Making Changes
# Activate venv
source backend/venv/bin/activate
# Or if you set up the alias:
backend
After Making Changes
# 1. Run full validation (catches import errors!)
python3 backend/validate.py
# 2. If validation passes, deploy
./scripts/deploy.sh
Validation Options
Full Validation (Recommended - Requires venv)
source backend/venv/bin/activate
python3 backend/validate.py
Catches:
- ✅ Syntax errors
- ✅ Import errors (like the processing_service issue!)
- ✅ Configuration issues
- ✅ FastAPI app creation problems
- ✅ Missing dependencies
This would have caught the ModuleNotFoundError!
Quick Check (No venv needed)
./backend/quick-check.sh
Catches:
- ✅ Syntax errors
- ✅ Missing files
- ✅ Some obvious import issues
Good for quick checks, but won't catch all import errors
Example: Complete Development Session
# 1. Activate environment
backend # or: source backend/venv/bin/activate
# 2. Make your changes
vim backend/api/routes/jobs.py
# 3. Validate (catches import errors!)
python3 backend/validate.py
# ✅ All validations passed!
# 4. Deploy
./scripts/deploy.sh
# 5. Test
export BACKEND_URL="https://karaoke-backend-ipzqd2k4yq-uc.a.run.app"
export AUTH_TOKEN=$(gcloud auth print-identity-token)
curl -H "Authorization: Bearer $AUTH_TOKEN" $BACKEND_URL/api/health
What About karaoke_gen Imports?
The backend imports from karaoke_gen (the CLI package):
from karaoke_gen.karaoke_gen import KaraokePrep
from karaoke_gen.audio_processor import AudioProcessor
Options:
Option 1: Install in development mode (Recommended)
source backend/venv/bin/activate
pip install -e .
This makes karaoke_gen available in the venv without copying files.
Option 2: Use system Python
Your system Python already has karaoke_gen installed, so validation will work even if the venv doesn't have it (Python will fall back to system packages).
Running the Backend Locally
# Activate venv
source backend/venv/bin/activate
# Set required environment variables
export GOOGLE_CLOUD_PROJECT="nomadkaraoke"
export GCS_BUCKET_NAME="karaoke-gen-storage-nomadkaraoke"
# Optional: Point to local credentials
export GOOGLE_APPLICATION_CREDENTIALS="/path/to/service-account-key.json"
# Run the server
cd backend
uvicorn main:app --reload --port 8080
Visit: http://localhost:8080/api/health
Troubleshooting
"No module named 'fastapi'"
Fix:
source backend/venv/bin/activate
pip install -r backend/requirements.txt
"No module named 'karaoke_gen'"
Fix:
source backend/venv/bin/activate
cd /Users/andrew/Projects/karaoke-gen
pip install -e .
Validation passes locally but fails in Cloud Run
Check:
- Is
requirements.txtup to date? - Did you add a new dependency without adding it to
requirements.txt? - Run:
pip freeze | grep <package-name>
Updating Dependencies
When you add new packages:
# 1. Install in venv
source backend/venv/bin/activate
pip install new-package
# 2. Update requirements.txt
pip freeze > backend/requirements.txt
# 3. Or manually add just the new package:
echo "new-package==1.2.3" >> backend/requirements.txt
# 4. Validate
python3 backend/validate.py
# 5. Deploy
./scripts/deploy.sh
Summary
Best practice workflow:
- One-time setup: Create venv and install dependencies
- Daily: Activate venv when working on backend
- Before deploy: Run
python3 backend/validate.py(catches import errors!) - Deploy: Use
./scripts/deploy.sh
The import error would have been caught if you ran validation in a venv! ✅