Files
karaoke-gen/backend
Andrew Beveridge ☄️andClaude Opus 5.5 8ec789a9aa fix(bulk): album lookup reads our MusicBrainz mirror, never the live API (v0.277.0) (#1176)
## Summary
Bulk → By album → "Britney Spears" showed **"Lookup failed. Please try
again."** `/api/bulk/album/artists` returned 502 because every attempt
to `musicbrainz.org/ws/2` from Cloud Run failed with
`httpx.ConnectError('')`. The same request from a home IP returns 200.
It last worked on 9/29. Most likely MusicBrainz is refusing our shared
GCP egress.

Per Andrew, we should only query our own infra, with no live-API
fallback. `musicbrainz_service` now reads **only our BigQuery
MusicBrainz mirror** (`karaoke_decide.mb_*`, which karaoke-decide's
`mb-refresh` rebuilds weekly from the official dumps).

## Changes
- **`backend/services/musicbrainz_service.py`**
- The HTTP client, throttle and retries for musicbrainz.org are removed.
Queries go through the BigQuery REST `jobs.query` endpoint (httpx + ADC;
no new dependency).
- Queries are parameterized, capped with `maximumBytesBilled` = 2 GiB,
and cached in memory for 10 min.
  - Lookups by table:
- artists: `mb_artists_normalized`. Ranking is exact match (including
"The "+q), then prefix, then substring, then popularity.
    - albums: `mb_artist_credit_artists` → `mb_release_groups`.
    - editions: `mb_releases`.
- tracks: `mb_release_tracks`, the new table in karaoke-decide#158,
clustered by release group + release.
- Measured against prod data: about 0.6–1 s per query, MB to 0.4 GB
billed.
- Canonical-pressing selection now ranks an edition by its best
release-event country.
- **Route:** passes `release_group_mbid` through on edition switches. An
unknown release returns **404** instead of 502.
- **Frontend:** `switchEdition` sends the album's `release_group_mbid`
so the lookup stays cluster-pruned. Older clients still work through a
one-off lookup that scans about 0.2 GB.
- **Infra:** the `karaoke-backend` SA gets project-level
`bigquery.dataViewer` + `bigquery.jobUser`. decide's Pulumi owns the
dataset ACL, so a dataset-scoped grant would fight it.
- **Docs:** `docs/API.md` and
`docs/archive/2026-10-16-bulk-album-musicbrainz-mirror.md`.

## Rollout (order matters)
1. Merge karaoke-decide#158, then build `mb_release_tracks`: by hand
(SQL in the archive doc, about $0.08) or at Sunday's refresh.
2. Run `pulumi up` for the IAM change. The agent's credentials are
read-only, so this needs Andrew.
3. Merge this PR.

## Testing
- `make test`: all passed (backend + 1735 frontend tests).
- New and rewritten unit tests cover normalization, BigQuery row
decoding, the REST request shape (no musicbrainz.org, billing cap,
params, cache), the HTTP/timeout errors, every lookup, the release-group
lookup path, and the 404.
- Ran the service live against the prod mirror: Britney Spears → 39
albums → …Baby One More Time (11 tracks, 6 variants); also checked
"beatles", "Beyoncé", "ac/dc", "Sigur Rós".
- CodeRabbit CLI: 1 minor finding (404 vs 502), fixed.

@coderabbitai ignore

🤖 Generated with [Claude Code](https://claude.com/claude-code)

https://claude.ai/code/session_01C1sMXk2MaxBLbep5jgiFab

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-17 00:06:16 -04:00
..
…

Backend Development Setup

One-Time Setup

1. Create Virtual Environment

cd /Users/andrew/Projects/karaoke-gen
python3 -m venv backend/venv

2. Activate Virtual Environment

source backend/venv/bin/activate

3. Install Dependencies

IMPORTANT: Use Python 3.12 (not 3.13/3.14 - pydantic has compatibility issues)

# Backend dependencies
pip install -r backend/requirements.txt

# Install karaoke_gen in development mode
pip install -e .

# Install lyrics_transcriber submodule
pip install -e lyrics_transcriber_local/

# Install Google API client (for YouTube upload)
pip install google-api-python-client google-auth-httplib2 google-auth-oauthlib

4. Add Shell Alias (Optional)

Add to ~/.zshrc:

alias backend='cd /Users/andrew/Projects/karaoke-gen && source backend/venv/bin/activate'

Then you can just run:

backend
# Now you're in the project directory with venv activated!

Daily Workflow

Before Making Changes

# Activate venv
source backend/venv/bin/activate

# Or if you set up the alias:
backend

After Making Changes

# 1. Run full validation (catches import errors!)
python3 backend/validate.py

# 2. If validation passes, deploy
./scripts/deploy.sh

Validation Options

source backend/venv/bin/activate
python3 backend/validate.py

Catches:

  • ✅ Syntax errors
  • ✅ Import errors (like the processing_service issue!)
  • ✅ Configuration issues
  • ✅ FastAPI app creation problems
  • ✅ Missing dependencies

This would have caught the ModuleNotFoundError!

Quick Check (No venv needed)

./backend/quick-check.sh

Catches:

  • ✅ Syntax errors
  • ✅ Missing files
  • ✅ Some obvious import issues

Good for quick checks, but won't catch all import errors


Example: Complete Development Session

# 1. Activate environment
backend  # or: source backend/venv/bin/activate

# 2. Make your changes
vim backend/api/routes/jobs.py

# 3. Validate (catches import errors!)
python3 backend/validate.py
# ✅ All validations passed!

# 4. Deploy
./scripts/deploy.sh

# 5. Test
export BACKEND_URL="https://karaoke-backend-ipzqd2k4yq-uc.a.run.app"
export AUTH_TOKEN=$(gcloud auth print-identity-token)
curl -H "Authorization: Bearer $AUTH_TOKEN" $BACKEND_URL/api/health

What About karaoke_gen Imports?

The backend imports from karaoke_gen (the CLI package):

from karaoke_gen.karaoke_gen import KaraokePrep
from karaoke_gen.audio_processor import AudioProcessor

Options:

source backend/venv/bin/activate
pip install -e .

This makes karaoke_gen available in the venv without copying files.

Option 2: Use system Python

Your system Python already has karaoke_gen installed, so validation will work even if the venv doesn't have it (Python will fall back to system packages).


Running the Backend Locally

# Activate venv
source backend/venv/bin/activate

# Set required environment variables
export GOOGLE_CLOUD_PROJECT="nomadkaraoke"
export GCS_BUCKET_NAME="karaoke-gen-storage-nomadkaraoke"

# Optional: Point to local credentials
export GOOGLE_APPLICATION_CREDENTIALS="/path/to/service-account-key.json"

# Run the server
cd backend
uvicorn main:app --reload --port 8080

Visit: http://localhost:8080/api/health


Troubleshooting

"No module named 'fastapi'"

Fix:

source backend/venv/bin/activate
pip install -r backend/requirements.txt

"No module named 'karaoke_gen'"

Fix:

source backend/venv/bin/activate
cd /Users/andrew/Projects/karaoke-gen
pip install -e .

Validation passes locally but fails in Cloud Run

Check:

  1. Is requirements.txt up to date?
  2. Did you add a new dependency without adding it to requirements.txt?
  3. Run: pip freeze | grep <package-name>

Updating Dependencies

When you add new packages:

# 1. Install in venv
source backend/venv/bin/activate
pip install new-package

# 2. Update requirements.txt
pip freeze > backend/requirements.txt

# 3. Or manually add just the new package:
echo "new-package==1.2.3" >> backend/requirements.txt

# 4. Validate
python3 backend/validate.py

# 5. Deploy
./scripts/deploy.sh

Summary

Best practice workflow:

  1. One-time setup: Create venv and install dependencies
  2. Daily: Activate venv when working on backend
  3. Before deploy: Run python3 backend/validate.py (catches import errors!)
  4. Deploy: Use ./scripts/deploy.sh

The import error would have been caught if you ran validation in a venv! ✅