## Summary Motivated by external contributor Martín (netux) asking "is there an easy way to run the full backend without GCS?" — he couldn't test PR #895's backend changes locally. This PR makes the whole backend runnable locally against emulators with **zero GCP credentials and zero paid API keys**. ### What was broken for contributors - Workers migrated to Cloud Run Jobs for prod robustness (audio-download, audio-separation, lyrics-transcription, bulk-search) silently bypassed the direct-HTTP dev mode → local jobs stalled at PENDING forever - Signed URL generation called `google.auth.default()` → crashed with no credentials (review audio, downloads, theme previews, upload URLs) - Magic-link login 503'd locally (console email provider counted as "not configured") - Fresh emulator had no default theme → every job create returned 422 - `docs/DEVELOPMENT.md` told people to `cp .env.example .env` but the file didn't exist — and nothing loaded `.env` anyway - `run-backend-local.sh` defaulted to the production GCP project ### Changes - **worker_service**: CRJ-based workers run as local subprocesses (`python -m backend.workers.X --job-id Y`, exact CRJ parity) when `ENABLE_CLOUD_TASKS=false` **and** not `is_production()` (regression-tested both ways) - **storage_service**: signed URLs → plain fake-gcs-server URLs when `STORAGE_EMULATOR_HOST` set (GET via JSON API download path; PUT via unvalidated-signature path, enabled by `-public-host` on the emulator) - **users route**: magic-link 503 only in production; dev prints the link to the console - **file_upload bugfix**: a theme with style params but no asset files never got `style_params_gcs_path` written (empty-dict falsiness in `elif theme_style_assets:`) — surfaced by the minimal seed theme, latent in prod - **scripts/seed-local-data.py** (new): creates bucket + default theme; wired into `run-backend-local.sh --with-emulators`, which now uses `test-project` - **config**: `load_dotenv()` so `.env` actually works; new root `.env.example` - **CONTRIBUTING.md** (new): no-credentials quickstart; README + DEVELOPMENT.md link to it ### Verified end-to-end locally (no GCP credentials) Upload job → local audio separation (real models) → transcription → screens → `awaiting_review`; review endpoints all working including `/api/review/{job}/audio/vocals` (the PR #895 endpoint) and 302 → emulator audio URLs. Magic-link login round-trip via console link. ### Tests - 224 tests across affected files pass; full backend unit suite: 3969 passed - `backend/tests/test_emulator_integration.py` has 8 pre-existing failures that reproduce identically on unmodified `main` locally (broken theme-service mock in that file) — unrelated to this PR - New tests: dev-mode subprocess dispatch (audio + bulk-search), prod-mode never-Popen guard, emulator URL generation (GET + PUT) @coderabbitai ignore --------- Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
13 KiB
Development Guide
External contributor? See CONTRIBUTING.md at the repo root — the backend runs fully locally against emulators with no GCP credentials or paid API keys. This file covers the full setup including production/deployment concerns.
Quick Start
Prerequisites
- Python 3.12
- Node.js 20+
- Docker (for GCS emulator)
- Google Cloud SDK with Firestore emulator
# Install Firestore emulator
gcloud components install cloud-firestore-emulator
Backend Setup
# Create and activate venv
python3.12 -m venv venv
source venv/bin/activate
# Install dependencies
poetry install
# Optional: copy environment template for API keys / overrides
# (nothing in it is required to run locally with emulators)
cp .env.example .env
Frontend Setup
cd frontend
npm install
cp .env.example .env.local
# Edit .env.local with KARAOKE_ACCESS_TOKEN
Running Locally
Backend with Emulators
# Start emulators, create the bucket, seed a default theme, run the backend
./scripts/run-backend-local.sh --with-emulators
# Backend runs at http://localhost:8000
# API docs at http://localhost:8000/docs
In emulator mode no GCP credentials are needed: workers run as local
subprocesses, signed URLs are rewritten to emulator URLs, and magic-link
login emails are printed to the console. Jobs must be created via file
upload (URL jobs require the flacfetch cloud service). Set MODEL_DIR to
enable local audio separation. See CONTRIBUTING.md for
the full walkthrough.
Frontend
cd frontend
npm run dev
# Runs at http://localhost:3000
Testing
See TESTING.md for complete testing guidance.
Quick reference (run before committing):
make test 2>&1 | tail -n 500
This runs ALL tests (backend + frontend) and installs dependencies automatically.
For faster iteration, run subsets:
make test-backend- Backend only (~2 min)make test-frontend- Frontend only (~3 min)
TESTING.md covers: test types, CI requirements, Playwright usage, mocking guidelines, and coverage enforcement.
For ad-hoc production debugging, see the "Ad-Hoc Production Debugging" section in TESTING.md. Use the template at frontend/e2e/helpers/debug-prod-template.mjs.
Deployment
Automatic (CI/CD)
Push to main triggers GitHub Actions:
- Run tests
- Build Docker image
- Deploy to Cloud Run
- Deploy frontend to Cloudflare Pages
Infrastructure Changes (Pulumi)
All infrastructure changes must go through PRs. The CI pipeline automatically deploys Pulumi changes when PRs merge to main.
IMPORTANT: Always deploy infrastructure changes locally first. CI deploys run on spot VMs that can be preempted mid-apply, leaving Pulumi state partially applied (e.g., resources deleted but not recreated). Running pulumi up locally before merging ensures changes work correctly and avoids painful state recovery.
cd infrastructure
# 1. Preview changes
pulumi preview --stack nomadkaraoke/karaoke-gen-infrastructure/prod
# 2. Apply locally FIRST (ensures clean apply before CI)
pulumi up --stack nomadkaraoke/karaoke-gen-infrastructure/prod
# 3. Then create PR and merge (CI will run pulumi up again as a no-op)
If CI deploy fails or is interrupted:
# Cancel stale lock from interrupted run
pulumi cancel --stack nomadkaraoke/karaoke-gen-infrastructure/prod
# Sync state with actual GCP resources
pulumi refresh --stack nomadkaraoke/karaoke-gen-infrastructure/prod --yes
# Re-apply (may need `pulumi import` for resources created but not in state)
pulumi up --stack nomadkaraoke/karaoke-gen-infrastructure/prod
See docs/LESSONS-LEARNED.md for Pulumi CI gotchas and recovery examples.
What's managed by Pulumi:
- Firestore database and indexes
- GCS buckets
- Cloud Run service configuration
- Cloud Tasks queues
- Secret Manager secrets
- Service accounts and IAM bindings
- Workload Identity Federation
- GCE instances (GitHub runners x3 spot + 1 on-demand build runner, encoding worker)
Workflow:
- Make changes in the appropriate module under
infrastructure/modules/- Core GCP resources (database, storage, secrets, IAM, etc.)compute/- VM definitions and startup scriptsconfig.py- Shared configuration constants__main__.py- Entry point that wires modules together
- Run
pulumi previewlocally to verify - Run
pulumi uplocally to apply - Create PR and merge (CI re-runs
pulumi upas a no-op confirmation)
Manual Frontend Deploy
cd frontend
npm run build
npx wrangler pages deploy out
Requires CLOUDFLARE_API_TOKEN and CLOUDFLARE_ACCOUNT_ID environment variables (set via .envrc or exported manually). The wrangler.toml in the repo root configures the project name and build output directory.
Testing Tenant Portals Locally
To test a tenant portal on localhost:3000, add a ?tenant= query parameter:
http://localhost:3000/?tenant=singa # Singa portal (green/black)
http://localhost:3000/?tenant=vocalstar # Vocal Star portal (yellow/blue)
http://localhost:3000/ # Default Nomad Karaoke (no tenant)
The frontend's detectTenantFromUrl() checks for the tenant query param on localhost. In production, tenants are detected from the subdomain (e.g., singa.nomadkaraoke.com).
Environment Variables
Backend (.env)
# Required
AUDIOSHAKE_API_TOKEN=...
GENIUS_ACCESS_TOKEN=...
# GCP (auto-configured in Cloud Run)
GOOGLE_CLOUD_PROJECT=...
GCS_BUCKET_NAME=...
# Auth
ADMIN_TOKENS=token1,token2
Frontend (.env.local)
NEXT_PUBLIC_API_URL=http://localhost:8000
# For E2E tests (see frontend/.env.local.example)
KARAOKE_ACCESS_TOKEN=... # Skip email enrollment
TESTMAIL_API_KEY=... # testmail.app API key
TESTMAIL_NAMESPACE=... # testmail.app namespace
Key Scripts
| Script | Purpose |
|---|---|
make test |
Run all backend tests |
./scripts/run-backend-local.sh |
Start backend with emulators |
./scripts/start-emulators.sh |
Start emulators only |
./scripts/stop-emulators.sh |
Stop emulators |
./scripts/debug-job.sh <job_id> |
Debug a cloud job (status, logs, files) |
./scripts/get_job.py <job_id> |
Fetch job data from Firestore |
./scripts/fetch_job_logs.py <job_id> |
Fetch cloud job logs to local file |
./scripts/analyze_log_timing.py <log> |
Profile karaoke-gen performance from logs |
./scripts/benchmark_ffmpeg.py |
Isolated FFmpeg encoding benchmark |
./scripts/compare_local_vs_remote.py |
Compare local vs cloud karaoke-gen performance |
./scripts/setup-singa-tenant.py |
Generate Singa tenant assets and upload to GCS |
./scripts/setup-vocalstar-tenant.py |
Generate Vocal Star tenant assets and upload to GCS |
Encoding Worker Operations
The encoding worker is a blue-green primary pair (encoding-worker-a/-b, c4d) plus a ranked pool of 8 stopped fallback VMs spanning 6 machine families (c4d/c4/n4d/c2d/n2d/n2) for stockout resilience — 10 VMs total. Candidate ordering (runtime + deploy) is the shared backend/services/encoding_worker_preference.py. See ARCHITECTURE.md for the full design.
Seed Firestore config (initial setup or reset)
python scripts/seed-encoding-worker-config.py <ip-a> <ip-b>
Check VM status
gcloud compute instances describe encoding-worker-a \
--zone=us-central1-c --project=nomadkaraoke \
--format='value(status)'
Manually start / stop a VM
# Start
gcloud compute instances start encoding-worker-a \
--zone=us-central1-c --project=nomadkaraoke
# Stop
gcloud compute instances stop encoding-worker-a \
--zone=us-central1-c --project=nomadkaraoke
Check worker health
IP=$(gcloud compute instances describe encoding-worker-a \
--zone=us-central1-c --project=nomadkaraoke \
--format='value(networkInterfaces[0].accessConfigs[0].natIP)')
curl http://$IP:8080/health
Manually trigger idle shutdown
Go to Cloud Scheduler in the GCP console and click "Force run" on the encoding-worker-idle-shutdown job, or:
gcloud scheduler jobs run encoding-worker-idle-shutdown \
--location=us-central1 --project=nomadkaraoke
Roll back a deploy
If a deploy went wrong, swap the Firestore config back to the old primary:
python3 << 'EOF'
import os
os.environ['GOOGLE_CLOUD_PROJECT'] = 'nomadkaraoke'
from google.cloud import firestore
db = firestore.Client(project='nomadkaraoke')
db.collection('config').document('encoding-worker').update({'primary': 'a'}) # or 'b'
print("Rolled back")
EOF
The backend's 30-second URL cache will pick up the change within 30 seconds.
Project Structure
karaoke-gen/
├── backend/
│ ├── api/routes/ # FastAPI endpoints
│ ├── models/ # Pydantic models
│ ├── services/ # Business logic
│ ├── workers/ # Background workers
│ └── tests/ # Backend tests
├── frontend/
│ ├── src/
│ │ ├── app/ # Next.js pages
│ │ ├── components/ # React components
│ │ └── lib/ # Utilities
│ └── tests/ # Frontend tests
├── infrastructure/ # Pulumi IaC
├── karaoke_gen/ # CLI package
└── scripts/ # Dev scripts
Observability & Debugging
The backend exports OpenTelemetry traces to Google Cloud Trace. Use these for investigating production issues.
Accessing Cloud Trace
Cloud Console (easiest for browsing):
https://console.cloud.google.com/traces/list?project=nomadkaraoke
CLI queries:
# List recent traces (requires `gcloud alpha` for trace commands)
# Easier to use the Console UI for trace browsing
# Find traces by job_id in logs, then correlate with Cloud Trace
gcloud logging read 'resource.type="cloud_run_revision" textPayload=~"JOB_ID"' \
--project=nomadkaraoke --limit=50
Key Traced Spans
| Span Name | What It Measures | Key Attributes |
|---|---|---|
lyrics_corrector.run |
Full lyrics correction pipeline | word_count, reference_count |
lyrics_corrector.find_anchors_and_gaps |
Anchor sequence matching | anchor_count, gap_count |
lyrics_corrector.process_corrections |
Agentic correction processing | gap_count, total_proposals |
anchor_search.find_anchors |
N-gram parallel processing | transcription.text_length, timeout_seconds |
agentic.propose_for_gap |
AI gap classification | gap_id, word_count, gap_category |
add-lyrics |
Adding reference lyrics | job_id, source |
generate-preview-video |
Video preview generation | job_id, use_gce |
Investigating Slow Operations
- Find the trace: Go to Cloud Trace console, filter by time range around the failure
- Look at waterfall: Identify which span took longest
- Check attributes: Each span has metadata (word counts, durations, etc.)
- Correlate with logs: Use timestamp and job_id to find related logs
Example Investigation Workflow
# 1. Find job failures in logs
gcloud logging read 'resource.type="cloud_run_revision" severity>=ERROR' \
--project=nomadkaraoke --limit=20 --format="table(timestamp,textPayload)"
# 2. Get timeline for specific job
./scripts/debug-job.sh <job_id>
# 3. Check anchor search progress (time-based logging every 30s)
gcloud logging read 'resource.type="cloud_run_revision" "ANCHOR SEARCH: Progress"' \
--project=nomadkaraoke --limit=20
# 4. Open Cloud Trace to see detailed timing breakdown
# Filter traces by time window when the job ran
Debugging Scripts
| Script | Purpose |
|---|---|
./scripts/debug-job.sh <job_id> |
Show job status, timeline, logs, GCS files |
./scripts/get_job.py <job_id> |
Fetch raw job data from Firestore |
./scripts/fetch_job_logs.py <job_id> |
Download all logs for a job to local file |
Adding New Traces
When adding new operations that may need investigation:
from karaoke_gen.lyrics_transcriber.utils.tracing import create_span, add_span_attribute
with create_span("operation_name", {"initial": "attributes"}) as span:
# ... do work ...
if span:
span.set_attribute("result_count", len(results))
For backend code:
from backend.services.tracing import create_span
with create_span("operation_name", {"job_id": job_id}) as span:
# ... do work ...
Troubleshooting
Emulators Won't Start
# Check if already running
lsof -i :8080 # Firestore
lsof -i :4443 # GCS
# Stop them
./scripts/stop-emulators.sh
Docker Issues
# Ensure Docker is running
docker info
# For GCS emulator
docker pull fsouza/fake-gcs-server
Test Failures
See TESTING.md for debugging test failures, coverage issues, and CI troubleshooting.