Files
karaoke-gen/CONTRIBUTING.md
Andrew Beveridge ☄️andClaude Fable 5 70ea964ca9 feat(dev): full local backend for external contributors (no GCP creds needed) (#946)
## Summary

Motivated by external contributor Martín (netux) asking "is there an
easy way to run the full backend without GCS?" — he couldn't test PR
#895's backend changes locally. This PR makes the whole backend runnable
locally against emulators with **zero GCP credentials and zero paid API
keys**.

### What was broken for contributors
- Workers migrated to Cloud Run Jobs for prod robustness
(audio-download, audio-separation, lyrics-transcription, bulk-search)
silently bypassed the direct-HTTP dev mode → local jobs stalled at
PENDING forever
- Signed URL generation called `google.auth.default()` → crashed with no
credentials (review audio, downloads, theme previews, upload URLs)
- Magic-link login 503'd locally (console email provider counted as "not
configured")
- Fresh emulator had no default theme → every job create returned 422
- `docs/DEVELOPMENT.md` told people to `cp .env.example .env` but the
file didn't exist — and nothing loaded `.env` anyway
- `run-backend-local.sh` defaulted to the production GCP project

### Changes
- **worker_service**: CRJ-based workers run as local subprocesses
(`python -m backend.workers.X --job-id Y`, exact CRJ parity) when
`ENABLE_CLOUD_TASKS=false` **and** not `is_production()`
(regression-tested both ways)
- **storage_service**: signed URLs → plain fake-gcs-server URLs when
`STORAGE_EMULATOR_HOST` set (GET via JSON API download path; PUT via
unvalidated-signature path, enabled by `-public-host` on the emulator)
- **users route**: magic-link 503 only in production; dev prints the
link to the console
- **file_upload bugfix**: a theme with style params but no asset files
never got `style_params_gcs_path` written (empty-dict falsiness in `elif
theme_style_assets:`) — surfaced by the minimal seed theme, latent in
prod
- **scripts/seed-local-data.py** (new): creates bucket + default theme;
wired into `run-backend-local.sh --with-emulators`, which now uses
`test-project`
- **config**: `load_dotenv()` so `.env` actually works; new root
`.env.example`
- **CONTRIBUTING.md** (new): no-credentials quickstart; README +
DEVELOPMENT.md link to it

### Verified end-to-end locally (no GCP credentials)
Upload job → local audio separation (real models) → transcription →
screens → `awaiting_review`; review endpoints all working including
`/api/review/{job}/audio/vocals` (the PR #895 endpoint) and 302 →
emulator audio URLs. Magic-link login round-trip via console link.

### Tests
- 224 tests across affected files pass; full backend unit suite: 3969
passed
- `backend/tests/test_emulator_integration.py` has 8 pre-existing
failures that reproduce identically on unmodified `main` locally (broken
theme-service mock in that file) — unrelated to this PR
- New tests: dev-mode subprocess dispatch (audio + bulk-search),
prod-mode never-Popen guard, emulator URL generation (GET + PUT)

@coderabbitai ignore

---------

Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
2026-08-27 06:48:00 +00:00

4.1 KiB

Contributing to karaoke-gen

Thanks for your interest in contributing! This guide focuses on getting a working local development environment with no Google Cloud credentials or paid API keys — everything runs against local emulators.

What runs locally (no credentials needed)

Component Local story
FastAPI backend Runs directly via uvicorn
Firestore Google's Firestore emulator
Cloud Storage fake-gcs-server (Docker)
Background workers (audio, lyrics, screens, video…) Run as local subprocesses / in-process tasks
Audio separation Local CPU/GPU via audio-separator (set MODEL_DIR; models download on first use)
Lyrics transcription Local Whisper (poetry install --extras local-whisper); AudioShake used only if you set a token
Login (magic links) Printed to the backend console instead of emailed
Signed URLs (review audio, downloads, uploads) Rewritten to plain emulator URLs
Frontend (Next.js) npm run dev, pointed at the local backend

Things that do NOT run locally: payments (Stripe), production email (Postmark), YouTube/Drive/Dropbox distribution, and the flacfetch URL-download service — jobs from URLs won't process locally; use file upload instead.

Prerequisites

  • Python 3.12/3.13 + Poetry
  • Node.js 20+ (for the frontend)
  • Docker (for the GCS emulator)
  • Java 21+ and the Google Cloud SDK with the Firestore emulator: gcloud components install cloud-firestore-emulator (no login/account needed — the emulator runs entirely offline)
  • ffmpeg (brew install ffmpeg / apt install ffmpeg)

Backend quickstart

poetry install                              # heavy: torch etc, ~10 min first time
./scripts/run-backend-local.sh --with-emulators

This starts the Firestore emulator (port 8080), fake-gcs-server (port 4443), creates the bucket, seeds a default theme (scripts/seed-local-data.py), and runs the backend at http://localhost:8000 (API docs at /docs).

Optional API keys go in .env (see .env.example) — none are required to boot.

Try it

# Health
curl http://localhost:8000/api/health

# Create a job from a local audio file (admin token is "local-dev-token")
curl -X POST http://localhost:8000/api/jobs/upload \
  -H 'Authorization: Bearer local-dev-token' \
  -F "file=@/path/to/song.flac" -F "artist=Some Artist" -F "title=Some Song"

# Watch it process; when status hits awaiting_review, open the review UI
curl http://localhost:8000/api/jobs/<job_id> -H 'Authorization: Bearer local-dev-token'

For audio separation to run, set MODEL_DIR (e.g. MODEL_DIR=./models) — separation models (~2 GB) are downloaded on first use. Without it the job fails at the separation step with a clear error.

Frontend quickstart

cd frontend
npm install
NEXT_PUBLIC_API_URL=http://localhost:8000 npx next dev

Then open http://localhost:3000. To log in, enter any email address — the magic link is printed in the backend console (look for EMAIL TO:); open that link in your browser.

Review UI without running the full pipeline

If you're working on the lyrics review UI specifically, there's a fixtures harness that serves real correction data through the review API shape without processing any audio:

python scripts/review_test_fixtures.py   # serves on :8765

Tests

make test          # everything (backend + frontend)
make test-backend  # backend only (unit + emulator tests)

Emulator-backed integration tests live in backend/tests/emulator/ and use the same two emulators as the local dev setup.

Pull requests

  • Keep PRs focused; include tests for behavior changes.
  • Run make test before pushing — CI runs the same suite.
  • Bump tool.poetry.version in pyproject.toml for code changes.
  • CI must pass before merge; a maintainer will review from there.

See docs/DEVELOPMENT.md for deeper docs (architecture, deployment, observability) — most of that file concerns the production GCP deployment, which contributors don't need.