## Why Since 2026-10-31 the AWS EC2 worker (`aws-encoding-worker-1`) has handled every encode and distribution job. 27 jobs went to it, with no Cloud Run fallbacks, no `encoding_stuck`, and nothing encoding-related in #alerts. The c4d Spot pair (`encoding-worker-a`/`-b`) now only costs ~$8/mo of Hyperdisk. `encoding-worker-fallback-c2df` stays as the GCP fallback. ## What Deleting the pair needs code changes first, because several paths assumed it exists: - **`encoding_worker_manager`** - The primary is optional: an empty `primary_vm` means none. - `ranked_candidates` skips a missing primary and always sets `is_primary`, so with no primary a successful start records the AWS/c2df override. Previously the legacy index-0 rule would have cleared it. - `ensure_primary_running` and `ensure_preferred_running` start a ranked candidate. If there are no candidates they raise `NO_CANDIDATES`. - `demote_active_worker` looks up the override's `machine_type`, so the AWS box can now be cooled down. It then clears the override so the retry goes elsewhere. - **`encoding_service`** - When `active_url` is empty, it targets the top-ranked candidate. - It also invalidates its cached URL after a single-candidate fallback. - **CI deploy (`ci.yml` + `deploy_promote.py`)** - While AWS serves, a c2df green is validate-only: it writes `validated_version`, keeps the AWS override, and stops the green. Before this, it would have replaced the AWS override with c2df. - The worker-change gate now diffs against `validated_version`, because `primary_version` stops moving. - An in-place restart of a serving override first waits for it to drain (15-min cap). If it is still busy, the restart is skipped. - **Idle function:** a missing VM is logged as a WARNING returning `NOT_FOUND`, not an ERROR. - **Error monitor:** watches c2df instead of a/b. - **Pulumi:** drops the a/b VMs and static IPs. The prod preview shows exactly 4 deletes. A separate, unrelated `youtube-comment-autoreply-scheduler` drift shows up too; it will not be applied. - **Docs:** CLAUDE.md, ARCHITECTURE, DEVELOPMENT and TROUBLESHOOTING. ## Rollout (after merge + deploy) 1. Back up `config/encoding-worker`, then set `primary_vm`, `primary_ip`, `secondary_vm` and `secondary_ip` to `""`. 2. Apply `pulumi destroy --target` on the 4 a/b URNs only. 3. Redeploy the idle function with `deploy.sh`. 4. Verify: one prod encode on AWS, warmup, and the logs. **Rollback:** in Firestore, delete `speed_rank_overrides` and set `aws_video_worker=false`. Restoring the pair would need a revert plus `pulumi up`, which gives new static IPs. ## Testing - New `backend/tests/test_encoding_worker_no_primary.py` (13 tests), plus new deploy_promote, idle-function and error-monitor tests. - Full backend suite: 7188 passed. The 8 `test_emulator_integration` failures also fail on main, because of a local emulator from another session. - CodeRabbit CLI: no findings. An independent review found 2 issues, both fixed: the demote retry path, and the in-place restart drain. 🤖 Generated with [Claude Code](https://claude.com/claude-code) https://claude.ai/code/session_012RYLVTkatGnrYFhCzq4PTe @coderabbitai ignore Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
14 KiB
Development Guide
External contributor? See CONTRIBUTING.md at the repo root — the backend runs fully locally against emulators with no GCP credentials or paid API keys. This file covers the full setup including production/deployment concerns.
Quick Start
Prerequisites
- Python 3.12
- Node.js 20+
- Docker (for GCS emulator)
- Google Cloud SDK with Firestore emulator
# Install Firestore emulator
gcloud components install cloud-firestore-emulator
Backend Setup
# Create and activate venv
python3.12 -m venv venv
source venv/bin/activate
# Install dependencies
poetry install
# Optional: copy environment template for API keys / overrides
# (nothing in it is required to run locally with emulators)
cp .env.example .env
Frontend Setup
cd frontend
npm install
cp .env.example .env.local
# Edit .env.local with KARAOKE_ACCESS_TOKEN
Running Locally
Backend with Emulators
# Start emulators, create the bucket, seed a default theme, run the backend
./scripts/run-backend-local.sh --with-emulators
# Backend runs at http://localhost:8000
# API docs at http://localhost:8000/docs
In emulator mode no GCP credentials are needed: workers run as local
subprocesses, signed URLs are rewritten to emulator URLs, and magic-link
login emails are printed to the console. Jobs must be created via file
upload (URL jobs require the flacfetch cloud service). Set MODEL_DIR to
enable local audio separation. See CONTRIBUTING.md for
the full walkthrough.
Local stack (full app, fake workers)
scripts/local-stack.sh up --frontend
# API http://127.0.0.1:8010 (FAKE_WORKERS=1), web http://127.0.0.1:3010
The whole product runs offline: search, job creation, review, credits, fake
Stripe checkout, emails (read them at /api/test/emails) and downloads. Jobs
move through their real states in seconds via in-process fake workers, and
POST /api/test/scenario seeds users and jobs in any state. This is what the
frontend/e2e/local-stack Playwright suite and the mobile app's Maestro flows
run against. See TESTING.md § Local stack.
Frontend
cd frontend
npm run dev
# Runs at http://localhost:3000
Testing
See TESTING.md for complete testing guidance.
Quick reference (run before committing):
make test 2>&1 | tail -n 500
This runs ALL tests (backend + frontend) and installs dependencies automatically.
For faster iteration, run subsets:
make test-backend- Backend only (~2 min)make test-frontend- Frontend only (~3 min)
TESTING.md covers: test types, CI requirements, Playwright usage, mocking guidelines, and coverage enforcement.
For ad-hoc production debugging, see the "Ad-Hoc Production Debugging" section in TESTING.md. Use the template at frontend/e2e/helpers/debug-prod-template.mjs.
Deployment
Automatic (CI/CD)
Push to main triggers GitHub Actions:
- Run tests
- Build Docker image
- Deploy to Cloud Run
- Deploy frontend to Cloudflare Pages
Infrastructure Changes (Pulumi)
All infrastructure changes must go through PRs. The CI pipeline automatically deploys Pulumi changes when PRs merge to main.
IMPORTANT: Always deploy infrastructure changes locally first. CI deploys run on spot VMs that can be preempted mid-apply, leaving Pulumi state partially applied (e.g., resources deleted but not recreated). Running pulumi up locally before merging ensures changes work correctly and avoids painful state recovery.
cd infrastructure
# 1. Preview changes
pulumi preview --stack nomadkaraoke/karaoke-gen-infrastructure/prod
# 2. Apply locally FIRST (ensures clean apply before CI)
pulumi up --stack nomadkaraoke/karaoke-gen-infrastructure/prod
# 3. Then create PR and merge (CI will run pulumi up again as a no-op)
If CI deploy fails or is interrupted:
# Cancel stale lock from interrupted run
pulumi cancel --stack nomadkaraoke/karaoke-gen-infrastructure/prod
# Sync state with actual GCP resources
pulumi refresh --stack nomadkaraoke/karaoke-gen-infrastructure/prod --yes
# Re-apply (may need `pulumi import` for resources created but not in state)
pulumi up --stack nomadkaraoke/karaoke-gen-infrastructure/prod
See docs/LESSONS-LEARNED.md for Pulumi CI gotchas and recovery examples.
What's managed by Pulumi:
- Firestore database and indexes
- GCS buckets
- Cloud Run service configuration
- Cloud Tasks queues
- Secret Manager secrets
- Service accounts and IAM bindings
- Workload Identity Federation
- GCE instances (GitHub runners x3 spot + 1 on-demand build runner, encoding worker)
Workflow:
- Make changes in the appropriate module under
infrastructure/modules/- Core GCP resources (database, storage, secrets, IAM, etc.)compute/- VM definitions and startup scriptsconfig.py- Shared configuration constants__main__.py- Entry point that wires modules together
- Run
pulumi previewlocally to verify - Run
pulumi uplocally to apply - Create PR and merge (CI re-runs
pulumi upas a no-op confirmation)
Manual Frontend Deploy
cd frontend
npm run build
npx wrangler pages deploy out
Requires CLOUDFLARE_API_TOKEN and CLOUDFLARE_ACCOUNT_ID environment variables (set via .envrc or exported manually). Since 2026-10-14 these must point at the dedicated Nomad Karaoke Cloudflare account (4585249dcf5d2f9fb4e075c5382cd071) — the workspace .envrc does this (the personal account is PERSONAL_CLOUDFLARE_* there), as do the CI GitHub secrets of the same names (Pages-only token). The wrangler.toml in the repo root configures the project name (karaoke-gen, i.e. karaoke-gen-9k4.pages.dev, custom domain gen.nomadkaraoke.com) and build output directory. Tenant portals deploy to the separate karaoke-gen-tenant project (karaoke-gen-tenant-x0h.pages.dev); the pages.dev names carry suffixes because they are globally unique and the projects were recreated in the new account.
Testing Tenant Portals Locally
To test a tenant portal on localhost:3000, add a ?tenant= query parameter:
http://localhost:3000/?tenant=singa # Singa portal (green/black)
http://localhost:3000/?tenant=vocalstar # Vocal Star portal (yellow/blue)
http://localhost:3000/ # Default Nomad Karaoke (no tenant)
The frontend's detectTenantFromUrl() checks for the tenant query param on localhost. In production, tenants are detected from the subdomain (e.g., singa.nomadkaraoke.com).
Environment Variables
Backend (.env)
# Required
AUDIOSHAKE_API_TOKEN=...
GENIUS_ACCESS_TOKEN=...
# GCP (auto-configured in Cloud Run)
GOOGLE_CLOUD_PROJECT=...
GCS_BUCKET_NAME=...
# Auth
ADMIN_TOKENS=token1,token2
Frontend (.env.local)
NEXT_PUBLIC_API_URL=http://localhost:8000
# For E2E tests (see frontend/.env.local.example)
KARAOKE_ACCESS_TOKEN=... # Skip email enrollment
TESTMAIL_API_KEY=... # testmail.app API key
TESTMAIL_NAMESPACE=... # testmail.app namespace
Key Scripts
| Script | Purpose |
|---|---|
make test |
Run all backend tests |
./scripts/run-backend-local.sh |
Start backend with emulators |
./scripts/start-emulators.sh |
Start emulators only |
./scripts/stop-emulators.sh |
Stop emulators |
./scripts/debug-job.sh <job_id> |
Debug a cloud job (status, logs, files) |
./scripts/get_job.py <job_id> |
Fetch job data from Firestore |
./scripts/fetch_job_logs.py <job_id> |
Fetch cloud job logs to local file |
./scripts/analyze_log_timing.py <log> |
Profile karaoke-gen performance from logs |
./scripts/benchmark_ffmpeg.py |
Isolated FFmpeg encoding benchmark |
./scripts/compare_local_vs_remote.py |
Compare local vs cloud karaoke-gen performance |
./scripts/setup-singa-tenant.py |
Generate Singa tenant assets and upload to GCS |
./scripts/setup-vocalstar-tenant.py |
Generate Vocal Star tenant assets and upload to GCS |
Encoding Worker Operations
The encoding worker is the AWS EC2 box aws-encoding-worker-1 (preferred, also runs distribution) plus one stopped on-demand GCE fallback (encoding-worker-fallback-c2df, c2d-highcpu-16, us-central1-f). The c4d Spot pair (encoding-worker-a/-b) was retired and deleted in 2026-11; examples below that name it should use c2df (--zone=us-central1-f). CI only blue-green redeploys the worker when a path in infrastructure/encoding-worker/worker_code_paths.txt changed since the serving version (or via Actions → CI → Run workflow with "Force a blue-green redeploy of the GCE encoding worker"). Candidate ordering (runtime + deploy) is the shared backend/services/encoding_worker_preference.py. See ARCHITECTURE.md for the full design.
Seed Firestore config (initial setup or reset)
python scripts/seed-encoding-worker-config.py <ip-a> <ip-b>
Check VM status
gcloud compute instances describe encoding-worker-a \
--zone=us-central1-c --project=nomadkaraoke \
--format='value(status)'
Manually start / stop a VM
# Start
gcloud compute instances start encoding-worker-a \
--zone=us-central1-c --project=nomadkaraoke
# Stop
gcloud compute instances stop encoding-worker-a \
--zone=us-central1-c --project=nomadkaraoke
Check worker health
IP=$(gcloud compute instances describe encoding-worker-a \
--zone=us-central1-c --project=nomadkaraoke \
--format='value(networkInterfaces[0].accessConfigs[0].natIP)')
curl http://$IP:8080/health
Manually trigger idle shutdown
Go to Cloud Scheduler in the GCP console and click "Force run" on the encoding-worker-idle-shutdown job, or:
gcloud scheduler jobs run encoding-worker-idle-shutdown \
--location=us-central1 --project=nomadkaraoke
Roll back a deploy
If a deploy went wrong, swap the Firestore config back to the old primary:
python3 << 'EOF'
import os
os.environ['GOOGLE_CLOUD_PROJECT'] = 'nomadkaraoke'
from google.cloud import firestore
db = firestore.Client(project='nomadkaraoke')
db.collection('config').document('encoding-worker').update({'primary': 'a'}) # or 'b'
print("Rolled back")
EOF
The backend's 30-second URL cache will pick up the change within 30 seconds.
Project Structure
karaoke-gen/
├── backend/
│ ├── api/routes/ # FastAPI endpoints
│ ├── models/ # Pydantic models
│ ├── services/ # Business logic
│ ├── workers/ # Background workers
│ └── tests/ # Backend tests
├── frontend/
│ ├── src/
│ │ ├── app/ # Next.js pages
│ │ ├── components/ # React components
│ │ └── lib/ # Utilities
│ └── tests/ # Frontend tests
├── infrastructure/ # Pulumi IaC
├── karaoke_gen/ # CLI package
└── scripts/ # Dev scripts
Observability & Debugging
The backend exports OpenTelemetry traces to Google Cloud Trace. Use these for investigating production issues.
Accessing Cloud Trace
Cloud Console (easiest for browsing):
https://console.cloud.google.com/traces/list?project=nomadkaraoke
CLI queries:
# List recent traces (requires `gcloud alpha` for trace commands)
# Easier to use the Console UI for trace browsing
# Find traces by job_id in logs, then correlate with Cloud Trace
gcloud logging read 'resource.type="cloud_run_revision" textPayload=~"JOB_ID"' \
--project=nomadkaraoke --limit=50
Key Traced Spans
| Span Name | What It Measures | Key Attributes |
|---|---|---|
lyrics_corrector.run |
Full lyrics correction pipeline | word_count, reference_count |
lyrics_corrector.find_anchors_and_gaps |
Anchor sequence matching | anchor_count, gap_count |
lyrics_corrector.process_corrections |
Agentic correction processing | gap_count, total_proposals |
anchor_search.find_anchors |
N-gram parallel processing | transcription.text_length, timeout_seconds |
agentic.propose_for_gap |
AI gap classification | gap_id, word_count, gap_category |
add-lyrics |
Adding reference lyrics | job_id, source |
generate-preview-video |
Video preview generation | job_id, use_gce |
Investigating Slow Operations
- Find the trace: Go to Cloud Trace console, filter by time range around the failure
- Look at waterfall: Identify which span took longest
- Check attributes: Each span has metadata (word counts, durations, etc.)
- Correlate with logs: Use timestamp and job_id to find related logs
Example Investigation Workflow
# 1. Find job failures in logs
gcloud logging read 'resource.type="cloud_run_revision" severity>=ERROR' \
--project=nomadkaraoke --limit=20 --format="table(timestamp,textPayload)"
# 2. Get timeline for specific job
./scripts/debug-job.sh <job_id>
# 3. Check anchor search progress (time-based logging every 30s)
gcloud logging read 'resource.type="cloud_run_revision" "ANCHOR SEARCH: Progress"' \
--project=nomadkaraoke --limit=20
# 4. Open Cloud Trace to see detailed timing breakdown
# Filter traces by time window when the job ran
Debugging Scripts
| Script | Purpose |
|---|---|
./scripts/debug-job.sh <job_id> |
Show job status, timeline, logs, GCS files |
./scripts/get_job.py <job_id> |
Fetch raw job data from Firestore |
./scripts/fetch_job_logs.py <job_id> |
Download all logs for a job to local file |
Adding New Traces
When adding new operations that may need investigation:
from karaoke_gen.lyrics_transcriber.utils.tracing import create_span, add_span_attribute
with create_span("operation_name", {"initial": "attributes"}) as span:
# ... do work ...
if span:
span.set_attribute("result_count", len(results))
For backend code:
from backend.services.tracing import create_span
with create_span("operation_name", {"job_id": job_id}) as span:
# ... do work ...
Troubleshooting
Emulators Won't Start
# Check if already running
lsof -i :8080 # Firestore
lsof -i :4443 # GCS
# Stop them
./scripts/stop-emulators.sh
Docker Issues
# Ensure Docker is running
docker info
# For GCS emulator
docker pull fsouza/fake-gcs-server
Test Failures
See TESTING.md for debugging test failures, coverage issues, and CI troubleshooting.