@coderabbitai ignore ## Why LRCLIB now applies stricter rate limits to unregistered clients (https://lrclib.net/docs#implementation-requirements-rate-limiting), and Cloud Run shares egress IPs. On 10-19, prod jobs got intermittent 503s, and job 80e7ffdf lost its LRCLIB lyrics. LRCLIB is our only free source; it supplied lyrics for 165 of 220 jobs in 30 days. Andrew registered a server-side app. ## What - **`karaoke_gen/lyrics_transcriber/lyrics/lrclib_client.py`** (new): - Sends `Lrclib-App-Id` + `Lrclib-Api-Key` from `LRCLIB_APP_ID` / `LRCLIB_SECRET_KEY`. - Applies LRCLIB's rules: every 429 sets a **process-wide cooldown** of `Retry-After` plus up to 30% jitter. The wait is never shortened, and overlapping 429s keep the latest expiry. Nothing is sent until the cooldown has passed. It waits and retries once only when the wait is ≤30s. - Retries 5xx after 1s/2s, honouring a 5xx `Retry-After`. - A **401 on our credentials** logs a WARNING and drops to anonymous access for the process, so a revoked key degrades instead of losing LRCLIB. - `LRCLIBProvider` (both `get` and `search`) uses it. The `quick_version` renderer uses it with `retry=False`, so the fast path gets the auth and the cooldown but never blocks. - **Infra:** - Pulumi secrets `lrclib-app-id` / `lrclib-api-key`. - Env vars on the `lyrics-transcription-job`. - The backend service deploy's `--set-secrets` (`ci.yml`). The karaoke-backend SA already has project-wide `secretAccessor`. ## Deploy order (do NOT merge before steps 1–2) 1. `pulumi up` to create the two secrets and the job env. Needs `/gcp-admin`. 2. Add the secret versions from Andrew's `.envrc` values. 3. Merge. The CI backend deploy references the secrets. ## Testing - New `tests/unit/test_lrclib_client.py` (14 tests): headers; anonymous use; Retry-After with jitter; a long Retry-After is not shortened and blocks further calls; no-retry mode; a second 429; 5xx backoff; 5xx Retry-After; 401 fallback; overlap merge (mutation-checked); provider routes through the client. - 164 passed across the client, `quick_version` and `search_lyrics` tests. - **Live** with the real app key: Dar Williams - Iowa and Wolfmother - Where Eagles Have Been were found. One 503 was recovered by the retry. - Local CodeRabbit, 3 cycles. I fixed the Retry-After cap, the cooldown after a final 429, the `quick_version` 401 path, the overlap merge and 5xx Retry-After. I skipped one false positive about the tests' fake clock (the retry path sleeps, then sends without re-checking the clock). 🤖 Generated with [Claude Code](https://claude.com/claude-code) https://claude.ai/code/session_01Jw21syo4HizJJjYw2mg2Vp --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Infrastructure as Code with Pulumi
This directory is the home of Nomad Karaoke's shared platform IaC. It holds two Pulumi projects:
| Project | Path | Stack | Manages |
|---|---|---|---|
karaoke-gen-infrastructure |
infrastructure/ |
prod |
The nomadkaraoke GCP project: gen's backend plus shared services (runner manager, divebar, KN sync, error monitor, monitoring, backups, IAM, secrets), and gen's Cloudflare edge security on api.nomadkaraoke.com |
nomad-cloudflare |
infrastructure/cloudflare/ |
nomadkaraoke/prod |
The Nomad Karaoke Cloudflare account: zones, DNS, rulesets, email routing, tunnels, Zero Trust, Pages/Worker domains. See cloudflare/README.md |
Boundary rule: infra that isn't specific to one app lives here. App repos keep only
app-specific stacks (e.g. karaoke-decide's Cloud Run and its API-proxy Worker + route). Every
Cloudflare zone ruleset phase has exactly one owner; see the exclusions table in cloudflare/README.md.
CI never runs pulumi up for either project: apply locally, then merge.
Directory Structure
infrastructure/
├── __main__.py # Entry point - wires modules together (~340 lines)
├── config.py # Centralized configuration constants
├── Pulumi.yaml # Pulumi project config
├── Pulumi.prod.yaml # Production stack config
├── requirements.txt # Python dependencies
│
├── modules/ # Core GCP resources
│ ├── database.py # Firestore database + indexes
│ ├── storage.py # GCS bucket + lifecycle rules
│ ├── artifact_registry.py # Docker repository
│ ├── secrets.py # Secret Manager secrets
│ ├── cloud_tasks.py # Task queues
│ ├── cloud_run.py # Domain mapping, Cloud Run Jobs
│ ├── monitoring.py # Alert policies
│ └── iam/ # IAM by purpose
│ ├── backend_sa.py # karaoke-backend service account
│ ├── github_actions_sa.py # WIF + deployer permissions
│ ├── claude_automation_sa.py # Claude Code automation
│ └── worker_sas.py # VM service accounts
│
├── compute/ # VM-based resources
│ ├── flacfetch_vm.py # Audio download service
│ ├── encoding_worker_vm.py # Video encoding VM
│ ├── github_runners.py # Self-hosted CI runners
│ └── startup_scripts/ # VM startup scripts (.sh)
│ ├── flacfetch.sh
│ ├── encoding_worker.sh
│ └── github_runner.sh
│
├── functions/ # Cloud Functions
│ └── gdrive_validator/
│ ├── main.py
│ ├── requirements.txt
│ └── deploy.sh # Source upload script
│
└── monitoring/ # Monitoring configs
├── dashboard.json
└── job-inspector.html
Resources Managed
| Category | Resources |
|---|---|
| Database | Firestore (Native mode), 10 indexes, TTL policy |
| Storage | GCS bucket with lifecycle rules |
| Registry | Artifact Registry for Docker images |
| Secrets | 18 Secret Manager secrets |
| IAM | 7 service accounts, WIF, 50+ IAM bindings |
| Compute | Encoding worker, Flacfetch, 10 GitHub runners |
| Tasks | 5 Cloud Tasks queues |
| Monitoring | 4 alert policies |
| Functions | GDrive validator + scheduler |
Quick Start
cd infrastructure
# Preview changes
pulumi preview --diff
# Apply changes
pulumi up
# View outputs
pulumi stack output
Prerequisites
- Pulumi CLI:
brew install pulumi - Pulumi Account: Sign up at https://app.pulumi.com
- Google Cloud SDK: With authenticated account
- Python 3.10+: For running Pulumi
Module Guide
config.py
Centralized configuration for the entire infrastructure:
- Project ID, region, zone
- Machine types and disk sizes
- Google Drive folder ID
- Number of GitHub runners
modules/database.py
Firestore database and composite indexes. Indexes are required for queries that filter/order by multiple fields.
modules/storage.py
GCS bucket with lifecycle rules:
- 7-day cleanup of staging data
- 24-hour cleanup of raw audio files
modules/iam/
Service accounts organized by purpose:
backend_sa.py: Cloud Run backend permissionsgithub_actions_sa.py: CI/CD via Workload Identity Federationworker_sas.py: VM service accounts (flacfetch, encoding, runners)
compute/
VM definitions with startup scripts as separate files. Each module creates:
- Static IP (where needed)
- VM instance
- Firewall rules
Adding Secret Values
After pulumi up, add actual secret values:
# API keys
echo -n "key" | gcloud secrets versions add audioshake-api-key --data-file=-
echo -n "key" | gcloud secrets versions add genius-api-key --data-file=-
# Private tracker credentials
echo -n "key" | gcloud secrets versions add red-api-key --data-file=-
echo -n "url" | gcloud secrets versions add red-api-url --data-file=-
# GitHub runner PAT (admin:org scope for org-level runners)
echo -n "ghp_xxx" | gcloud secrets versions add github-runner-pat --data-file=-
GDrive Validator
The Cloud Function validates your public Google Drive folder daily:
- Checks for duplicate sequence numbers
- Validates filename formats
- Detects sequence gaps
- Sends Pushbullet notifications
Deploy:
cd functions/gdrive_validator
./deploy.sh
Note: deploy.sh uploads the source ZIP to GCS. Pulumi creates the function resources but doesn't handle source upload.
Self-Hosted GitHub Runners
20 GCP VMs serve as organization-level self-hosted runners, available to all repos in the nomadkaraoke org:
- 200GB SSD (vs 14GB on GitHub-hosted)
- Pre-installed: Docker, Python 3.13, Node 20, Java 21, FFmpeg
- Labels:
self-hosted,linux,x64,gcp,large-disk
PAT Requirements: The github-runner-pat secret requires admin:org scope (specifically manage_runners:org) for organization-level runner registration.
Usage in workflows (any nomadkaraoke repo):
runs-on: [self-hosted, linux, gcp]
Troubleshooting:
# View startup logs
gcloud compute ssh github-runner-1 --zone=us-central1-a -- \
"sudo cat /var/log/github-runner-startup.log"
# Check runner service
gcloud compute ssh github-runner-1 --zone=us-central1-a -- \
"sudo systemctl status actions.runner.*"
Encoding Worker
High-performance VM for video encoding:
- Machine: c4-standard-8 (8 vCPU, 32GB RAM)
- Disk: 50GB boot disk (hyperdisk-balanced at the free 3000 IOPS / 140 MB/s baseline; pd-balanced on c2d/n2/n2d fallbacks). See docs/archive/2026-09-26-encoding-worker-disk-cost.md
- Pre-installed: FFmpeg (static build), Python 3.13, Noto fonts
API Endpoints:
POST /encode- Full encoding jobPOST /encode-preview- Quick 480p previewGET /status/{job_id}- Job statusGET /health- Health check
Troubleshooting:
gcloud compute ssh encoding-worker --zone=us-central1-c -- \
"sudo systemctl status encoding-worker"
State Management
Pulumi state is stored in Pulumi Cloud. To sync with actual cloud resources:
pulumi refresh
Multi-Environment
pulumi stack init dev
pulumi config set gcp:project nomadkaraoke-dev
pulumi up
pulumi stack select prod # Switch stacks
Troubleshooting
"Resource already exists"
Import manually-created resources:
pulumi import gcp:storage/bucket:Bucket karaoke-storage karaoke-gen-storage-nomadkaraoke
State conflicts
pulumi refresh # Sync state with cloud
Permissions errors
Ensure your gcloud account has Owner or Editor role, or specific roles like Cloud Build Editor, Storage Admin.
Future Improvements
Phase 5: Custom Machine Images (Planned)
The encoding worker currently takes ~10 minutes to start because it builds Python 3.13 from source on every boot. A Packer-built custom image would reduce this to ~30 seconds.
See: docs/PHASE5-PACKER-IMAGE-PLAN.md for the implementation plan.
Benefits:
- Startup time: 10 min → 30 sec
- Pre-baked: Python 3.13, FFmpeg, fonts
- Hot updates: karaoke-gen wheel still downloaded at runtime