Files
karaoke-gen/backend
Andrew Beveridge ☄️andClaude Fable 5 70ea964ca9 feat(dev): full local backend for external contributors (no GCP creds needed) (#946)
## Summary

Motivated by external contributor Martín (netux) asking "is there an
easy way to run the full backend without GCS?" — he couldn't test PR
#895's backend changes locally. This PR makes the whole backend runnable
locally against emulators with **zero GCP credentials and zero paid API
keys**.

### What was broken for contributors
- Workers migrated to Cloud Run Jobs for prod robustness
(audio-download, audio-separation, lyrics-transcription, bulk-search)
silently bypassed the direct-HTTP dev mode → local jobs stalled at
PENDING forever
- Signed URL generation called `google.auth.default()` → crashed with no
credentials (review audio, downloads, theme previews, upload URLs)
- Magic-link login 503'd locally (console email provider counted as "not
configured")
- Fresh emulator had no default theme → every job create returned 422
- `docs/DEVELOPMENT.md` told people to `cp .env.example .env` but the
file didn't exist — and nothing loaded `.env` anyway
- `run-backend-local.sh` defaulted to the production GCP project

### Changes
- **worker_service**: CRJ-based workers run as local subprocesses
(`python -m backend.workers.X --job-id Y`, exact CRJ parity) when
`ENABLE_CLOUD_TASKS=false` **and** not `is_production()`
(regression-tested both ways)
- **storage_service**: signed URLs → plain fake-gcs-server URLs when
`STORAGE_EMULATOR_HOST` set (GET via JSON API download path; PUT via
unvalidated-signature path, enabled by `-public-host` on the emulator)
- **users route**: magic-link 503 only in production; dev prints the
link to the console
- **file_upload bugfix**: a theme with style params but no asset files
never got `style_params_gcs_path` written (empty-dict falsiness in `elif
theme_style_assets:`) — surfaced by the minimal seed theme, latent in
prod
- **scripts/seed-local-data.py** (new): creates bucket + default theme;
wired into `run-backend-local.sh --with-emulators`, which now uses
`test-project`
- **config**: `load_dotenv()` so `.env` actually works; new root
`.env.example`
- **CONTRIBUTING.md** (new): no-credentials quickstart; README +
DEVELOPMENT.md link to it

### Verified end-to-end locally (no GCP credentials)
Upload job → local audio separation (real models) → transcription →
screens → `awaiting_review`; review endpoints all working including
`/api/review/{job}/audio/vocals` (the PR #895 endpoint) and 302 →
emulator audio URLs. Magic-link login round-trip via console link.

### Tests
- 224 tests across affected files pass; full backend unit suite: 3969
passed
- `backend/tests/test_emulator_integration.py` has 8 pre-existing
failures that reproduce identically on unmodified `main` locally (broken
theme-service mock in that file) — unrelated to this PR
- New tests: dev-mode subprocess dispatch (audio + bulk-search),
prod-mode never-Popen guard, emulator URL generation (GET + PUT)

@coderabbitai ignore

---------

Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
2026-08-27 06:48:00 +00:00
..

Backend Development Setup

One-Time Setup

1. Create Virtual Environment

cd /Users/andrew/Projects/karaoke-gen
python3 -m venv backend/venv

2. Activate Virtual Environment

source backend/venv/bin/activate

3. Install Dependencies

IMPORTANT: Use Python 3.12 (not 3.13/3.14 - pydantic has compatibility issues)

# Backend dependencies
pip install -r backend/requirements.txt

# Install karaoke_gen in development mode
pip install -e .

# Install lyrics_transcriber submodule
pip install -e lyrics_transcriber_local/

# Install Google API client (for YouTube upload)
pip install google-api-python-client google-auth-httplib2 google-auth-oauthlib

4. Add Shell Alias (Optional)

Add to ~/.zshrc:

alias backend='cd /Users/andrew/Projects/karaoke-gen && source backend/venv/bin/activate'

Then you can just run:

backend
# Now you're in the project directory with venv activated!

Daily Workflow

Before Making Changes

# Activate venv
source backend/venv/bin/activate

# Or if you set up the alias:
backend

After Making Changes

# 1. Run full validation (catches import errors!)
python3 backend/validate.py

# 2. If validation passes, deploy
./scripts/deploy.sh

Validation Options

source backend/venv/bin/activate
python3 backend/validate.py

Catches:

  • ✅ Syntax errors
  • ✅ Import errors (like the processing_service issue!)
  • ✅ Configuration issues
  • ✅ FastAPI app creation problems
  • ✅ Missing dependencies

This would have caught the ModuleNotFoundError!

Quick Check (No venv needed)

./backend/quick-check.sh

Catches:

  • ✅ Syntax errors
  • ✅ Missing files
  • ✅ Some obvious import issues

Good for quick checks, but won't catch all import errors


Example: Complete Development Session

# 1. Activate environment
backend  # or: source backend/venv/bin/activate

# 2. Make your changes
vim backend/api/routes/jobs.py

# 3. Validate (catches import errors!)
python3 backend/validate.py
# ✅ All validations passed!

# 4. Deploy
./scripts/deploy.sh

# 5. Test
export BACKEND_URL="https://karaoke-backend-ipzqd2k4yq-uc.a.run.app"
export AUTH_TOKEN=$(gcloud auth print-identity-token)
curl -H "Authorization: Bearer $AUTH_TOKEN" $BACKEND_URL/api/health

What About karaoke_gen Imports?

The backend imports from karaoke_gen (the CLI package):

from karaoke_gen.karaoke_gen import KaraokePrep
from karaoke_gen.audio_processor import AudioProcessor

Options:

source backend/venv/bin/activate
pip install -e .

This makes karaoke_gen available in the venv without copying files.

Option 2: Use system Python

Your system Python already has karaoke_gen installed, so validation will work even if the venv doesn't have it (Python will fall back to system packages).


Running the Backend Locally

# Activate venv
source backend/venv/bin/activate

# Set required environment variables
export GOOGLE_CLOUD_PROJECT="nomadkaraoke"
export GCS_BUCKET_NAME="karaoke-gen-storage-nomadkaraoke"

# Optional: Point to local credentials
export GOOGLE_APPLICATION_CREDENTIALS="/path/to/service-account-key.json"

# Run the server
cd backend
uvicorn main:app --reload --port 8080

Visit: http://localhost:8080/api/health


Troubleshooting

"No module named 'fastapi'"

Fix:

source backend/venv/bin/activate
pip install -r backend/requirements.txt

"No module named 'karaoke_gen'"

Fix:

source backend/venv/bin/activate
cd /Users/andrew/Projects/karaoke-gen
pip install -e .

Validation passes locally but fails in Cloud Run

Check:

  1. Is requirements.txt up to date?
  2. Did you add a new dependency without adding it to requirements.txt?
  3. Run: pip freeze | grep <package-name>

Updating Dependencies

When you add new packages:

# 1. Install in venv
source backend/venv/bin/activate
pip install new-package

# 2. Update requirements.txt
pip freeze > backend/requirements.txt

# 3. Or manually add just the new package:
echo "new-package==1.2.3" >> backend/requirements.txt

# 4. Validate
python3 backend/validate.py

# 5. Deploy
./scripts/deploy.sh

Summary

Best practice workflow:

  1. One-time setup: Create venv and install dependencies
  2. Daily: Activate venv when working on backend
  3. Before deploy: Run python3 backend/validate.py (catches import errors!)
  4. Deploy: Use ./scripts/deploy.sh

The import error would have been caught if you ran validation in a venv! ✅