Files
karaoke-gen/docs/DEVELOPMENT.md
T
Andrew Beveridge ☄️andClaude Opus 5.5 c277bc8ee3 feat: local E2E stack (fake workers + test API) + email-code sign-in (v0.275.0) (#1165)
## Summary
- **Local stack:** `scripts/local-stack.sh up --frontend` runs the
**real** backend on the Firestore + GCS emulators with `FAKE_WORKERS=1`.
In-process fake workers move jobs through their real states in seconds,
using tiny real media, so web Playwright and mobile Maestro full-journey
tests can run offline against the same endpoints.
- **Email-code sign-in** (nomad-mobile spec §4.1): `POST
/api/users/auth/send-code` + `verify-code`. The code logic is shared
with kjbox via `LoginCodeStore`, and sends are throttled per email and
per IP.
- **Bug fix found by the new suite:** `confirm-duration` moved jobs to
`separating_stage1`, from which they can never reach review. It now
resumes at `downloading`. No prod job had taken this path yet.

## Changes
- `backend/services/fake_workers/`: dispatch seam in
`WorkerService.trigger_*`, plus fake search, download, checkout and
outbox, scenarios, and per-user config. The real stem upload, transcode,
backing-vocals analysis and preview render are reused. Every check is
strict (`is True`), so a MagicMock settings object never enables the
fakes, and startup **refuses FAKE_WORKERS in production**.
- `/api/test/*` router (mounted only in that mode): `status`,
`scenario`, `fake-config`, `emails`, `reset`, plus the fake Stripe
checkout page. Paying it runs the real webhook handler.
- `frontend/e2e/local-stack/` (9 specs, ~50s):
  - sign-up → make → review → instrumental → download
  - buy and cancel credits
  - duration confirm, failed, complete and review states
  - code sign-in and lockout
- New required CI job `local-stack-e2e`.
- Docs: TESTING.md § Local stack, DEVELOPMENT.md, API.md, README,
LESSONS-LEARNED.
- `MAX_SIGNUPS_PER_IP` can now be set by env. Only the local stack
raises it; production keeps the default of 2.

## Testing
- [x] Backend unit: 6175 passed
- [x] Emulator: 152 passed
- [x] Frontend Jest: 1706 passed
- [x] Local-stack Playwright: 9/9 passed (CI mode)

## Review
- [x] CodeRabbit CLI review completed locally. 1 minor finding declined:
it suggested swallowing infrastructure errors in send-code as fake
"sent" responses, which would hide real outages from users.

@coderabbitai ignore

---
🤖 Generated with [Claude Code](https://claude.com/claude-code)

https://claude.ai/code/session_01AGuxtgoscF6RJGtpjHPg3W

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-15 16:00:24 -04:00

14 KiB

Development Guide

External contributor? See CONTRIBUTING.md at the repo root — the backend runs fully locally against emulators with no GCP credentials or paid API keys. This file covers the full setup including production/deployment concerns.

Quick Start

Prerequisites

  • Python 3.12
  • Node.js 20+
  • Docker (for GCS emulator)
  • Google Cloud SDK with Firestore emulator
# Install Firestore emulator
gcloud components install cloud-firestore-emulator

Backend Setup

# Create and activate venv
python3.12 -m venv venv
source venv/bin/activate

# Install dependencies
poetry install

# Optional: copy environment template for API keys / overrides
# (nothing in it is required to run locally with emulators)
cp .env.example .env

Frontend Setup

cd frontend
npm install
cp .env.example .env.local
# Edit .env.local with KARAOKE_ACCESS_TOKEN

Running Locally

Backend with Emulators

# Start emulators, create the bucket, seed a default theme, run the backend
./scripts/run-backend-local.sh --with-emulators

# Backend runs at http://localhost:8000
# API docs at http://localhost:8000/docs

In emulator mode no GCP credentials are needed: workers run as local subprocesses, signed URLs are rewritten to emulator URLs, and magic-link login emails are printed to the console. Jobs must be created via file upload (URL jobs require the flacfetch cloud service). Set MODEL_DIR to enable local audio separation. See CONTRIBUTING.md for the full walkthrough.

Local stack (full app, fake workers)

scripts/local-stack.sh up --frontend
# API http://127.0.0.1:8010 (FAKE_WORKERS=1), web http://127.0.0.1:3010

The whole product runs offline: search, job creation, review, credits, fake Stripe checkout, emails (read them at /api/test/emails) and downloads. Jobs move through their real states in seconds via in-process fake workers, and POST /api/test/scenario seeds users and jobs in any state. This is what the frontend/e2e/local-stack Playwright suite and the mobile app's Maestro flows run against. See TESTING.md § Local stack.

Frontend

cd frontend
npm run dev
# Runs at http://localhost:3000

Testing

See TESTING.md for complete testing guidance.

Quick reference (run before committing):

make test 2>&1 | tail -n 500

This runs ALL tests (backend + frontend) and installs dependencies automatically.

For faster iteration, run subsets:

  • make test-backend - Backend only (~2 min)
  • make test-frontend - Frontend only (~3 min)

TESTING.md covers: test types, CI requirements, Playwright usage, mocking guidelines, and coverage enforcement.

For ad-hoc production debugging, see the "Ad-Hoc Production Debugging" section in TESTING.md. Use the template at frontend/e2e/helpers/debug-prod-template.mjs.

Deployment

Automatic (CI/CD)

Push to main triggers GitHub Actions:

  1. Run tests
  2. Build Docker image
  3. Deploy to Cloud Run
  4. Deploy frontend to Cloudflare Pages

Infrastructure Changes (Pulumi)

All infrastructure changes must go through PRs. The CI pipeline automatically deploys Pulumi changes when PRs merge to main.

IMPORTANT: Always deploy infrastructure changes locally first. CI deploys run on spot VMs that can be preempted mid-apply, leaving Pulumi state partially applied (e.g., resources deleted but not recreated). Running pulumi up locally before merging ensures changes work correctly and avoids painful state recovery.

cd infrastructure

# 1. Preview changes
pulumi preview --stack nomadkaraoke/karaoke-gen-infrastructure/prod

# 2. Apply locally FIRST (ensures clean apply before CI)
pulumi up --stack nomadkaraoke/karaoke-gen-infrastructure/prod

# 3. Then create PR and merge (CI will run pulumi up again as a no-op)

If CI deploy fails or is interrupted:

# Cancel stale lock from interrupted run
pulumi cancel --stack nomadkaraoke/karaoke-gen-infrastructure/prod

# Sync state with actual GCP resources
pulumi refresh --stack nomadkaraoke/karaoke-gen-infrastructure/prod --yes

# Re-apply (may need `pulumi import` for resources created but not in state)
pulumi up --stack nomadkaraoke/karaoke-gen-infrastructure/prod

See docs/LESSONS-LEARNED.md for Pulumi CI gotchas and recovery examples.

What's managed by Pulumi:

  • Firestore database and indexes
  • GCS buckets
  • Cloud Run service configuration
  • Cloud Tasks queues
  • Secret Manager secrets
  • Service accounts and IAM bindings
  • Workload Identity Federation
  • GCE instances (GitHub runners x3 spot + 1 on-demand build runner, encoding worker)

Workflow:

  1. Make changes in the appropriate module under infrastructure/
    • modules/ - Core GCP resources (database, storage, secrets, IAM, etc.)
    • compute/ - VM definitions and startup scripts
    • config.py - Shared configuration constants
    • __main__.py - Entry point that wires modules together
  2. Run pulumi preview locally to verify
  3. Run pulumi up locally to apply
  4. Create PR and merge (CI re-runs pulumi up as a no-op confirmation)

Manual Frontend Deploy

cd frontend
npm run build
npx wrangler pages deploy out

Requires CLOUDFLARE_API_TOKEN and CLOUDFLARE_ACCOUNT_ID environment variables (set via .envrc or exported manually). Since 2026-10-14 these must point at the dedicated Nomad Karaoke Cloudflare account (4585249dcf5d2f9fb4e075c5382cd071) — the workspace .envrc does this (the personal account is PERSONAL_CLOUDFLARE_* there), as do the CI GitHub secrets of the same names (Pages-only token). The wrangler.toml in the repo root configures the project name (karaoke-gen, i.e. karaoke-gen-9k4.pages.dev, custom domain gen.nomadkaraoke.com) and build output directory. Tenant portals deploy to the separate karaoke-gen-tenant project (karaoke-gen-tenant-x0h.pages.dev); the pages.dev names carry suffixes because they are globally unique and the projects were recreated in the new account.

Testing Tenant Portals Locally

To test a tenant portal on localhost:3000, add a ?tenant= query parameter:

http://localhost:3000/?tenant=singa      # Singa portal (green/black)
http://localhost:3000/?tenant=vocalstar  # Vocal Star portal (yellow/blue)
http://localhost:3000/                   # Default Nomad Karaoke (no tenant)

The frontend's detectTenantFromUrl() checks for the tenant query param on localhost. In production, tenants are detected from the subdomain (e.g., singa.nomadkaraoke.com).

Environment Variables

Backend (.env)

# Required
AUDIOSHAKE_API_TOKEN=...
GENIUS_ACCESS_TOKEN=...

# GCP (auto-configured in Cloud Run)
GOOGLE_CLOUD_PROJECT=...
GCS_BUCKET_NAME=...

# Auth
ADMIN_TOKENS=token1,token2

Frontend (.env.local)

NEXT_PUBLIC_API_URL=http://localhost:8000

# For E2E tests (see frontend/.env.local.example)
KARAOKE_ACCESS_TOKEN=...     # Skip email enrollment
TESTMAIL_API_KEY=...         # testmail.app API key
TESTMAIL_NAMESPACE=...       # testmail.app namespace

Key Scripts

Script Purpose
make test Run all backend tests
./scripts/run-backend-local.sh Start backend with emulators
./scripts/start-emulators.sh Start emulators only
./scripts/stop-emulators.sh Stop emulators
./scripts/debug-job.sh <job_id> Debug a cloud job (status, logs, files)
./scripts/get_job.py <job_id> Fetch job data from Firestore
./scripts/fetch_job_logs.py <job_id> Fetch cloud job logs to local file
./scripts/analyze_log_timing.py <log> Profile karaoke-gen performance from logs
./scripts/benchmark_ffmpeg.py Isolated FFmpeg encoding benchmark
./scripts/compare_local_vs_remote.py Compare local vs cloud karaoke-gen performance
./scripts/setup-singa-tenant.py Generate Singa tenant assets and upload to GCS
./scripts/setup-vocalstar-tenant.py Generate Vocal Star tenant assets and upload to GCS

Encoding Worker Operations

The encoding worker is a Spot blue-green primary pair (encoding-worker-a/-b, c4d-highcpu-16) plus one stopped on-demand fallback (encoding-worker-fallback-c2df, c2d-highcpu-16, us-central1-f) — 3 VMs total since 2026-09-26. CI only blue-green redeploys the worker when a path in infrastructure/encoding-worker/worker_code_paths.txt changed since the serving version (or via Actions → CI → Run workflow with "Force a blue-green redeploy of the GCE encoding worker"). Candidate ordering (runtime + deploy) is the shared backend/services/encoding_worker_preference.py. See ARCHITECTURE.md for the full design.

Seed Firestore config (initial setup or reset)

python scripts/seed-encoding-worker-config.py <ip-a> <ip-b>

Check VM status

gcloud compute instances describe encoding-worker-a \
  --zone=us-central1-c --project=nomadkaraoke \
  --format='value(status)'

Manually start / stop a VM

# Start
gcloud compute instances start encoding-worker-a \
  --zone=us-central1-c --project=nomadkaraoke

# Stop
gcloud compute instances stop encoding-worker-a \
  --zone=us-central1-c --project=nomadkaraoke

Check worker health

IP=$(gcloud compute instances describe encoding-worker-a \
  --zone=us-central1-c --project=nomadkaraoke \
  --format='value(networkInterfaces[0].accessConfigs[0].natIP)')
curl http://$IP:8080/health

Manually trigger idle shutdown

Go to Cloud Scheduler in the GCP console and click "Force run" on the encoding-worker-idle-shutdown job, or:

gcloud scheduler jobs run encoding-worker-idle-shutdown \
  --location=us-central1 --project=nomadkaraoke

Roll back a deploy

If a deploy went wrong, swap the Firestore config back to the old primary:

python3 << 'EOF'
import os
os.environ['GOOGLE_CLOUD_PROJECT'] = 'nomadkaraoke'
from google.cloud import firestore
db = firestore.Client(project='nomadkaraoke')
db.collection('config').document('encoding-worker').update({'primary': 'a'})  # or 'b'
print("Rolled back")
EOF

The backend's 30-second URL cache will pick up the change within 30 seconds.

Project Structure

karaoke-gen/
├── backend/
│   ├── api/routes/       # FastAPI endpoints
│   ├── models/           # Pydantic models
│   ├── services/         # Business logic
│   ├── workers/          # Background workers
│   └── tests/            # Backend tests
├── frontend/
│   ├── src/
│   │   ├── app/          # Next.js pages
│   │   ├── components/   # React components
│   │   └── lib/          # Utilities
│   └── tests/            # Frontend tests
├── infrastructure/       # Pulumi IaC
├── karaoke_gen/          # CLI package
└── scripts/              # Dev scripts

Observability & Debugging

The backend exports OpenTelemetry traces to Google Cloud Trace. Use these for investigating production issues.

Accessing Cloud Trace

Cloud Console (easiest for browsing):

https://console.cloud.google.com/traces/list?project=nomadkaraoke

CLI queries:

# List recent traces (requires `gcloud alpha` for trace commands)
# Easier to use the Console UI for trace browsing

# Find traces by job_id in logs, then correlate with Cloud Trace
gcloud logging read 'resource.type="cloud_run_revision" textPayload=~"JOB_ID"' \
  --project=nomadkaraoke --limit=50

Key Traced Spans

Span Name What It Measures Key Attributes
lyrics_corrector.run Full lyrics correction pipeline word_count, reference_count
lyrics_corrector.find_anchors_and_gaps Anchor sequence matching anchor_count, gap_count
lyrics_corrector.process_corrections Agentic correction processing gap_count, total_proposals
anchor_search.find_anchors N-gram parallel processing transcription.text_length, timeout_seconds
agentic.propose_for_gap AI gap classification gap_id, word_count, gap_category
add-lyrics Adding reference lyrics job_id, source
generate-preview-video Video preview generation job_id, use_gce

Investigating Slow Operations

  1. Find the trace: Go to Cloud Trace console, filter by time range around the failure
  2. Look at waterfall: Identify which span took longest
  3. Check attributes: Each span has metadata (word counts, durations, etc.)
  4. Correlate with logs: Use timestamp and job_id to find related logs

Example Investigation Workflow

# 1. Find job failures in logs
gcloud logging read 'resource.type="cloud_run_revision" severity>=ERROR' \
  --project=nomadkaraoke --limit=20 --format="table(timestamp,textPayload)"

# 2. Get timeline for specific job
./scripts/debug-job.sh <job_id>

# 3. Check anchor search progress (time-based logging every 30s)
gcloud logging read 'resource.type="cloud_run_revision" "ANCHOR SEARCH: Progress"' \
  --project=nomadkaraoke --limit=20

# 4. Open Cloud Trace to see detailed timing breakdown
# Filter traces by time window when the job ran

Debugging Scripts

Script Purpose
./scripts/debug-job.sh <job_id> Show job status, timeline, logs, GCS files
./scripts/get_job.py <job_id> Fetch raw job data from Firestore
./scripts/fetch_job_logs.py <job_id> Download all logs for a job to local file

Adding New Traces

When adding new operations that may need investigation:

from karaoke_gen.lyrics_transcriber.utils.tracing import create_span, add_span_attribute

with create_span("operation_name", {"initial": "attributes"}) as span:
    # ... do work ...
    if span:
        span.set_attribute("result_count", len(results))

For backend code:

from backend.services.tracing import create_span

with create_span("operation_name", {"job_id": job_id}) as span:
    # ... do work ...

Troubleshooting

Emulators Won't Start

# Check if already running
lsof -i :8080  # Firestore
lsof -i :4443  # GCS

# Stop them
./scripts/stop-emulators.sh

Docker Issues

# Ensure Docker is running
docker info

# For GCS emulator
docker pull fsouza/fake-gcs-server

Test Failures

See TESTING.md for debugging test failures, coverage issues, and CI troubleshooting.