Files
karaoke-gen/infrastructure
Andrew Beveridge ☄️andClaude Opus 5.5 8ec789a9aa fix(bulk): album lookup reads our MusicBrainz mirror, never the live API (v0.277.0) (#1176)
## Summary
Bulk → By album → "Britney Spears" showed **"Lookup failed. Please try
again."** `/api/bulk/album/artists` returned 502 because every attempt
to `musicbrainz.org/ws/2` from Cloud Run failed with
`httpx.ConnectError('')`. The same request from a home IP returns 200.
It last worked on 9/29. Most likely MusicBrainz is refusing our shared
GCP egress.

Per Andrew, we should only query our own infra, with no live-API
fallback. `musicbrainz_service` now reads **only our BigQuery
MusicBrainz mirror** (`karaoke_decide.mb_*`, which karaoke-decide's
`mb-refresh` rebuilds weekly from the official dumps).

## Changes
- **`backend/services/musicbrainz_service.py`**
- The HTTP client, throttle and retries for musicbrainz.org are removed.
Queries go through the BigQuery REST `jobs.query` endpoint (httpx + ADC;
no new dependency).
- Queries are parameterized, capped with `maximumBytesBilled` = 2 GiB,
and cached in memory for 10 min.
  - Lookups by table:
- artists: `mb_artists_normalized`. Ranking is exact match (including
"The "+q), then prefix, then substring, then popularity.
    - albums: `mb_artist_credit_artists` → `mb_release_groups`.
    - editions: `mb_releases`.
- tracks: `mb_release_tracks`, the new table in karaoke-decide#158,
clustered by release group + release.
- Measured against prod data: about 0.6–1 s per query, MB to 0.4 GB
billed.
- Canonical-pressing selection now ranks an edition by its best
release-event country.
- **Route:** passes `release_group_mbid` through on edition switches. An
unknown release returns **404** instead of 502.
- **Frontend:** `switchEdition` sends the album's `release_group_mbid`
so the lookup stays cluster-pruned. Older clients still work through a
one-off lookup that scans about 0.2 GB.
- **Infra:** the `karaoke-backend` SA gets project-level
`bigquery.dataViewer` + `bigquery.jobUser`. decide's Pulumi owns the
dataset ACL, so a dataset-scoped grant would fight it.
- **Docs:** `docs/API.md` and
`docs/archive/2026-10-16-bulk-album-musicbrainz-mirror.md`.

## Rollout (order matters)
1. Merge karaoke-decide#158, then build `mb_release_tracks`: by hand
(SQL in the archive doc, about $0.08) or at Sunday's refresh.
2. Run `pulumi up` for the IAM change. The agent's credentials are
read-only, so this needs Andrew.
3. Merge this PR.

## Testing
- `make test`: all passed (backend + 1735 frontend tests).
- New and rewritten unit tests cover normalization, BigQuery row
decoding, the REST request shape (no musicbrainz.org, billing cap,
params, cache), the HTTP/timeout errors, every lookup, the release-group
lookup path, and the 404.
- Ran the service live against the prod mirror: Britney Spears → 39
albums → …Baby One More Time (11 tracks, 6 variants); also checked
"beatles", "Beyoncé", "ac/dc", "Sigur Rós".
- CodeRabbit CLI: 1 minor finding (404 vs 502), fixed.

@coderabbitai ignore

🤖 Generated with [Claude Code](https://claude.com/claude-code)

https://claude.ai/code/session_01C1sMXk2MaxBLbep5jgiFab

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-17 00:06:16 -04:00
..

Infrastructure as Code with Pulumi

This directory is the home of Nomad Karaoke's shared platform IaC. It holds two Pulumi projects:

Project Path Stack Manages
karaoke-gen-infrastructure infrastructure/ prod The nomadkaraoke GCP project: gen's backend plus shared services (runner manager, divebar, KN sync, error monitor, monitoring, backups, IAM, secrets), and gen's Cloudflare edge security on api.nomadkaraoke.com
nomad-cloudflare infrastructure/cloudflare/ nomadkaraoke/prod The Nomad Karaoke Cloudflare account: zones, DNS, rulesets, email routing, tunnels, Zero Trust, Pages/Worker domains. See cloudflare/README.md

Boundary rule: infra that isn't specific to one app lives here. App repos keep only app-specific stacks (e.g. karaoke-decide's Cloud Run and its API-proxy Worker + route). Every Cloudflare zone ruleset phase has exactly one owner; see the exclusions table in cloudflare/README.md. CI never runs pulumi up for either project: apply locally, then merge.

Directory Structure

infrastructure/
├── __main__.py              # Entry point - wires modules together (~340 lines)
├── config.py                # Centralized configuration constants
├── Pulumi.yaml              # Pulumi project config
├── Pulumi.prod.yaml         # Production stack config
├── requirements.txt         # Python dependencies
│
├── modules/                 # Core GCP resources
│   ├── database.py         # Firestore database + indexes
│   ├── storage.py          # GCS bucket + lifecycle rules
│   ├── artifact_registry.py # Docker repository
│   ├── secrets.py          # Secret Manager secrets
│   ├── cloud_tasks.py      # Task queues
│   ├── cloud_run.py        # Domain mapping, Cloud Run Jobs
│   ├── monitoring.py       # Alert policies
│   └── iam/                # IAM by purpose
│       ├── backend_sa.py   # karaoke-backend service account
│       ├── github_actions_sa.py # WIF + deployer permissions
│       ├── claude_automation_sa.py # Claude Code automation
│       └── worker_sas.py   # VM service accounts
│
├── compute/                 # VM-based resources
│   ├── flacfetch_vm.py     # Audio download service
│   ├── encoding_worker_vm.py # Video encoding VM
│   ├── github_runners.py   # Self-hosted CI runners
│   └── startup_scripts/    # VM startup scripts (.sh)
│       ├── flacfetch.sh
│       ├── encoding_worker.sh
│       └── github_runner.sh
│
├── functions/               # Cloud Functions
│   └── gdrive_validator/
│       ├── main.py
│       ├── requirements.txt
│       └── deploy.sh       # Source upload script
│
└── monitoring/              # Monitoring configs
    ├── dashboard.json
    └── job-inspector.html

Resources Managed

Category Resources
Database Firestore (Native mode), 10 indexes, TTL policy
Storage GCS bucket with lifecycle rules
Registry Artifact Registry for Docker images
Secrets 18 Secret Manager secrets
IAM 7 service accounts, WIF, 50+ IAM bindings
Compute Encoding worker, Flacfetch, 10 GitHub runners
Tasks 5 Cloud Tasks queues
Monitoring 4 alert policies
Functions GDrive validator + scheduler

Quick Start

cd infrastructure

# Preview changes
pulumi preview --diff

# Apply changes
pulumi up

# View outputs
pulumi stack output

Prerequisites

  1. Pulumi CLI: brew install pulumi
  2. Pulumi Account: Sign up at https://app.pulumi.com
  3. Google Cloud SDK: With authenticated account
  4. Python 3.10+: For running Pulumi

Module Guide

config.py

Centralized configuration for the entire infrastructure:

  • Project ID, region, zone
  • Machine types and disk sizes
  • Google Drive folder ID
  • Number of GitHub runners

modules/database.py

Firestore database and composite indexes. Indexes are required for queries that filter/order by multiple fields.

modules/storage.py

GCS bucket with lifecycle rules:

  • 7-day cleanup of staging data
  • 24-hour cleanup of raw audio files

modules/iam/

Service accounts organized by purpose:

  • backend_sa.py: Cloud Run backend permissions
  • github_actions_sa.py: CI/CD via Workload Identity Federation
  • worker_sas.py: VM service accounts (flacfetch, encoding, runners)

compute/

VM definitions with startup scripts as separate files. Each module creates:

  • Static IP (where needed)
  • VM instance
  • Firewall rules

Adding Secret Values

After pulumi up, add actual secret values:

# API keys
echo -n "key" | gcloud secrets versions add audioshake-api-key --data-file=-
echo -n "key" | gcloud secrets versions add genius-api-key --data-file=-

# Private tracker credentials
echo -n "key" | gcloud secrets versions add red-api-key --data-file=-
echo -n "url" | gcloud secrets versions add red-api-url --data-file=-

# GitHub runner PAT (admin:org scope for org-level runners)
echo -n "ghp_xxx" | gcloud secrets versions add github-runner-pat --data-file=-

GDrive Validator

The Cloud Function validates your public Google Drive folder daily:

  • Checks for duplicate sequence numbers
  • Validates filename formats
  • Detects sequence gaps
  • Sends Pushbullet notifications

Deploy:

cd functions/gdrive_validator
./deploy.sh

Note: deploy.sh uploads the source ZIP to GCS. Pulumi creates the function resources but doesn't handle source upload.

Self-Hosted GitHub Runners

20 GCP VMs serve as organization-level self-hosted runners, available to all repos in the nomadkaraoke org:

  • 200GB SSD (vs 14GB on GitHub-hosted)
  • Pre-installed: Docker, Python 3.13, Node 20, Java 21, FFmpeg
  • Labels: self-hosted, linux, x64, gcp, large-disk

PAT Requirements: The github-runner-pat secret requires admin:org scope (specifically manage_runners:org) for organization-level runner registration.

Usage in workflows (any nomadkaraoke repo):

runs-on: [self-hosted, linux, gcp]

Troubleshooting:

# View startup logs
gcloud compute ssh github-runner-1 --zone=us-central1-a -- \
  "sudo cat /var/log/github-runner-startup.log"

# Check runner service
gcloud compute ssh github-runner-1 --zone=us-central1-a -- \
  "sudo systemctl status actions.runner.*"

Encoding Worker

High-performance VM for video encoding:

  • Machine: c4-standard-8 (8 vCPU, 32GB RAM)
  • Disk: 50GB boot disk (hyperdisk-balanced at the free 3000 IOPS / 140 MB/s baseline; pd-balanced on c2d/n2/n2d fallbacks). See docs/archive/2026-09-26-encoding-worker-disk-cost.md
  • Pre-installed: FFmpeg (static build), Python 3.13, Noto fonts

API Endpoints:

  • POST /encode - Full encoding job
  • POST /encode-preview - Quick 480p preview
  • GET /status/{job_id} - Job status
  • GET /health - Health check

Troubleshooting:

gcloud compute ssh encoding-worker --zone=us-central1-c -- \
  "sudo systemctl status encoding-worker"

State Management

Pulumi state is stored in Pulumi Cloud. To sync with actual cloud resources:

pulumi refresh

Multi-Environment

pulumi stack init dev
pulumi config set gcp:project nomadkaraoke-dev
pulumi up

pulumi stack select prod  # Switch stacks

Troubleshooting

"Resource already exists"

Import manually-created resources:

pulumi import gcp:storage/bucket:Bucket karaoke-storage karaoke-gen-storage-nomadkaraoke

State conflicts

pulumi refresh  # Sync state with cloud

Permissions errors

Ensure your gcloud account has Owner or Editor role, or specific roles like Cloud Build Editor, Storage Admin.

Future Improvements

Phase 5: Custom Machine Images (Planned)

The encoding worker currently takes ~10 minutes to start because it builds Python 3.13 from source on every boot. A Packer-built custom image would reduce this to ~30 seconds.

See: docs/PHASE5-PACKER-IMAGE-PLAN.md for the implementation plan.

Benefits:

  • Startup time: 10 min → 30 sec
  • Pre-baked: Python 3.13, FFmpeg, fonts
  • Hot updates: karaoke-gen wheel still downloaded at runtime