Files
karaoke-gen/infrastructure
Andrew Beveridge ☄️andClaude Opus 5.5 4286d98dbe fix: LRCLIB as a registered server-side app; honour Retry-After, retry 5xx (v0.279.2) (#1200)
@coderabbitai ignore

## Why
LRCLIB now applies stricter rate limits to unregistered clients
(https://lrclib.net/docs#implementation-requirements-rate-limiting), and
Cloud Run shares egress IPs. On 10-19, prod jobs got intermittent 503s,
and job 80e7ffdf lost its LRCLIB lyrics. LRCLIB is our only free source;
it supplied lyrics for 165 of 220 jobs in 30 days. Andrew registered a
server-side app.

## What
- **`karaoke_gen/lyrics_transcriber/lyrics/lrclib_client.py`** (new):
- Sends `Lrclib-App-Id` + `Lrclib-Api-Key` from `LRCLIB_APP_ID` /
`LRCLIB_SECRET_KEY`.
- Applies LRCLIB's rules: every 429 sets a **process-wide cooldown** of
`Retry-After` plus up to 30% jitter. The wait is never shortened, and
overlapping 429s keep the latest expiry. Nothing is sent until the
cooldown has passed. It waits and retries once only when the wait is
≤30s.
  - Retries 5xx after 1s/2s, honouring a 5xx `Retry-After`.
- A **401 on our credentials** logs a WARNING and drops to anonymous
access for the process, so a revoked key degrades instead of losing
LRCLIB.
- `LRCLIBProvider` (both `get` and `search`) uses it. The
`quick_version` renderer uses it with `retry=False`, so the fast path
gets the auth and the cooldown but never blocks.
- **Infra:**
  - Pulumi secrets `lrclib-app-id` / `lrclib-api-key`.
  - Env vars on the `lyrics-transcription-job`.
- The backend service deploy's `--set-secrets` (`ci.yml`). The
karaoke-backend SA already has project-wide `secretAccessor`.

## Deploy order (do NOT merge before steps 1–2)
1. `pulumi up` to create the two secrets and the job env. Needs
`/gcp-admin`.
2. Add the secret versions from Andrew's `.envrc` values.
3. Merge. The CI backend deploy references the secrets.

## Testing
- New `tests/unit/test_lrclib_client.py` (14 tests): headers; anonymous
use; Retry-After with jitter; a long Retry-After is not shortened and
blocks further calls; no-retry mode; a second 429; 5xx backoff; 5xx
Retry-After; 401 fallback; overlap merge (mutation-checked); provider
routes through the client.
- 164 passed across the client, `quick_version` and `search_lyrics`
tests.
- **Live** with the real app key: Dar Williams - Iowa and Wolfmother -
Where Eagles Have Been were found. One 503 was recovered by the retry.
- Local CodeRabbit, 3 cycles. I fixed the Retry-After cap, the cooldown
after a final 429, the `quick_version` 401 path, the overlap merge and
5xx Retry-After. I skipped one false positive about the tests' fake
clock (the retry path sleeps, then sends without re-checking the clock).

🤖 Generated with [Claude Code](https://claude.com/claude-code)

https://claude.ai/code/session_01Jw21syo4HizJJjYw2mg2Vp

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-19 17:33:19 -04:00
..

Infrastructure as Code with Pulumi

This directory is the home of Nomad Karaoke's shared platform IaC. It holds two Pulumi projects:

Project Path Stack Manages
karaoke-gen-infrastructure infrastructure/ prod The nomadkaraoke GCP project: gen's backend plus shared services (runner manager, divebar, KN sync, error monitor, monitoring, backups, IAM, secrets), and gen's Cloudflare edge security on api.nomadkaraoke.com
nomad-cloudflare infrastructure/cloudflare/ nomadkaraoke/prod The Nomad Karaoke Cloudflare account: zones, DNS, rulesets, email routing, tunnels, Zero Trust, Pages/Worker domains. See cloudflare/README.md

Boundary rule: infra that isn't specific to one app lives here. App repos keep only app-specific stacks (e.g. karaoke-decide's Cloud Run and its API-proxy Worker + route). Every Cloudflare zone ruleset phase has exactly one owner; see the exclusions table in cloudflare/README.md. CI never runs pulumi up for either project: apply locally, then merge.

Directory Structure

infrastructure/
├── __main__.py              # Entry point - wires modules together (~340 lines)
├── config.py                # Centralized configuration constants
├── Pulumi.yaml              # Pulumi project config
├── Pulumi.prod.yaml         # Production stack config
├── requirements.txt         # Python dependencies
│
├── modules/                 # Core GCP resources
│   ├── database.py         # Firestore database + indexes
│   ├── storage.py          # GCS bucket + lifecycle rules
│   ├── artifact_registry.py # Docker repository
│   ├── secrets.py          # Secret Manager secrets
│   ├── cloud_tasks.py      # Task queues
│   ├── cloud_run.py        # Domain mapping, Cloud Run Jobs
│   ├── monitoring.py       # Alert policies
│   └── iam/                # IAM by purpose
│       ├── backend_sa.py   # karaoke-backend service account
│       ├── github_actions_sa.py # WIF + deployer permissions
│       ├── claude_automation_sa.py # Claude Code automation
│       └── worker_sas.py   # VM service accounts
│
├── compute/                 # VM-based resources
│   ├── flacfetch_vm.py     # Audio download service
│   ├── encoding_worker_vm.py # Video encoding VM
│   ├── github_runners.py   # Self-hosted CI runners
│   └── startup_scripts/    # VM startup scripts (.sh)
│       ├── flacfetch.sh
│       ├── encoding_worker.sh
│       └── github_runner.sh
│
├── functions/               # Cloud Functions
│   └── gdrive_validator/
│       ├── main.py
│       ├── requirements.txt
│       └── deploy.sh       # Source upload script
│
└── monitoring/              # Monitoring configs
    ├── dashboard.json
    └── job-inspector.html

Resources Managed

Category Resources
Database Firestore (Native mode), 10 indexes, TTL policy
Storage GCS bucket with lifecycle rules
Registry Artifact Registry for Docker images
Secrets 18 Secret Manager secrets
IAM 7 service accounts, WIF, 50+ IAM bindings
Compute Encoding worker, Flacfetch, 10 GitHub runners
Tasks 5 Cloud Tasks queues
Monitoring 4 alert policies
Functions GDrive validator + scheduler

Quick Start

cd infrastructure

# Preview changes
pulumi preview --diff

# Apply changes
pulumi up

# View outputs
pulumi stack output

Prerequisites

  1. Pulumi CLI: brew install pulumi
  2. Pulumi Account: Sign up at https://app.pulumi.com
  3. Google Cloud SDK: With authenticated account
  4. Python 3.10+: For running Pulumi

Module Guide

config.py

Centralized configuration for the entire infrastructure:

  • Project ID, region, zone
  • Machine types and disk sizes
  • Google Drive folder ID
  • Number of GitHub runners

modules/database.py

Firestore database and composite indexes. Indexes are required for queries that filter/order by multiple fields.

modules/storage.py

GCS bucket with lifecycle rules:

  • 7-day cleanup of staging data
  • 24-hour cleanup of raw audio files

modules/iam/

Service accounts organized by purpose:

  • backend_sa.py: Cloud Run backend permissions
  • github_actions_sa.py: CI/CD via Workload Identity Federation
  • worker_sas.py: VM service accounts (flacfetch, encoding, runners)

compute/

VM definitions with startup scripts as separate files. Each module creates:

  • Static IP (where needed)
  • VM instance
  • Firewall rules

Adding Secret Values

After pulumi up, add actual secret values:

# API keys
echo -n "key" | gcloud secrets versions add audioshake-api-key --data-file=-
echo -n "key" | gcloud secrets versions add genius-api-key --data-file=-

# Private tracker credentials
echo -n "key" | gcloud secrets versions add red-api-key --data-file=-
echo -n "url" | gcloud secrets versions add red-api-url --data-file=-

# GitHub runner PAT (admin:org scope for org-level runners)
echo -n "ghp_xxx" | gcloud secrets versions add github-runner-pat --data-file=-

GDrive Validator

The Cloud Function validates your public Google Drive folder daily:

  • Checks for duplicate sequence numbers
  • Validates filename formats
  • Detects sequence gaps
  • Sends Pushbullet notifications

Deploy:

cd functions/gdrive_validator
./deploy.sh

Note: deploy.sh uploads the source ZIP to GCS. Pulumi creates the function resources but doesn't handle source upload.

Self-Hosted GitHub Runners

20 GCP VMs serve as organization-level self-hosted runners, available to all repos in the nomadkaraoke org:

  • 200GB SSD (vs 14GB on GitHub-hosted)
  • Pre-installed: Docker, Python 3.13, Node 20, Java 21, FFmpeg
  • Labels: self-hosted, linux, x64, gcp, large-disk

PAT Requirements: The github-runner-pat secret requires admin:org scope (specifically manage_runners:org) for organization-level runner registration.

Usage in workflows (any nomadkaraoke repo):

runs-on: [self-hosted, linux, gcp]

Troubleshooting:

# View startup logs
gcloud compute ssh github-runner-1 --zone=us-central1-a -- \
  "sudo cat /var/log/github-runner-startup.log"

# Check runner service
gcloud compute ssh github-runner-1 --zone=us-central1-a -- \
  "sudo systemctl status actions.runner.*"

Encoding Worker

High-performance VM for video encoding:

  • Machine: c4-standard-8 (8 vCPU, 32GB RAM)
  • Disk: 50GB boot disk (hyperdisk-balanced at the free 3000 IOPS / 140 MB/s baseline; pd-balanced on c2d/n2/n2d fallbacks). See docs/archive/2026-09-26-encoding-worker-disk-cost.md
  • Pre-installed: FFmpeg (static build), Python 3.13, Noto fonts

API Endpoints:

  • POST /encode - Full encoding job
  • POST /encode-preview - Quick 480p preview
  • GET /status/{job_id} - Job status
  • GET /health - Health check

Troubleshooting:

gcloud compute ssh encoding-worker --zone=us-central1-c -- \
  "sudo systemctl status encoding-worker"

State Management

Pulumi state is stored in Pulumi Cloud. To sync with actual cloud resources:

pulumi refresh

Multi-Environment

pulumi stack init dev
pulumi config set gcp:project nomadkaraoke-dev
pulumi up

pulumi stack select prod  # Switch stacks

Troubleshooting

"Resource already exists"

Import manually-created resources:

pulumi import gcp:storage/bucket:Bucket karaoke-storage karaoke-gen-storage-nomadkaraoke

State conflicts

pulumi refresh  # Sync state with cloud

Permissions errors

Ensure your gcloud account has Owner or Editor role, or specific roles like Cloud Build Editor, Storage Admin.

Future Improvements

Phase 5: Custom Machine Images (Planned)

The encoding worker currently takes ~10 minutes to start because it builds Python 3.13 from source on every boot. A Packer-built custom image would reduce this to ~30 seconds.

See: docs/PHASE5-PACKER-IMAGE-PLAN.md for the implementation plan.

Benefits:

  • Startup time: 10 min → 30 sec
  • Pre-baked: Python 3.13, FFmpeg, fonts
  • Hot updates: karaoke-gen wheel still downloaded at runtime