## Summary Fixes the deployment failure from PR #297 by making lyrics_transcriber a proper subpackage of karaoke_gen instead of a separate vendored library. ### Changes - Move `libs/lyrics_transcriber/` to `karaoke_gen/lyrics_transcriber/` - Update all imports from `from lyrics_transcriber.X` to `from karaoke_gen.lyrics_transcriber.X` - Simplify Dockerfile (no separate `libs/` COPY needed) - Update pyproject.toml to remove separate package entry - Update test mocks and logger names ### Why this is better - Cleaner project structure - one package containing everything - Simpler Dockerfile - just COPY karaoke_gen/ - No confusion about vendored vs internal code ## Test plan - [x] `poetry install` succeeds - [x] All imports work correctly - [x] All 2614 tests pass - [ ] Backend deployment succeeds @coderabbitai ignore Co-authored-by: Claude Opus 4.5 <noreply@anthropic.com>
8.3 KiB
Development Guide
Quick Start
Prerequisites
- Python 3.12
- Node.js 20+
- Docker (for GCS emulator)
- Google Cloud SDK with Firestore emulator
# Install Firestore emulator
gcloud components install cloud-firestore-emulator
Backend Setup
# Create and activate venv
python3.12 -m venv venv
source venv/bin/activate
# Install dependencies
poetry install
# Copy environment template
cp .env.example .env
# Edit .env with required API keys
Frontend Setup
cd frontend
npm install
cp .env.example .env.local
# Edit .env.local with KARAOKE_ACCESS_TOKEN
Running Locally
Backend with Emulators
# Start emulators and backend
./scripts/run-backend-local.sh --with-emulators
# Backend runs at http://localhost:8000
# API docs at http://localhost:8000/docs
Frontend
cd frontend
npm run dev
# Runs at http://localhost:3000
Testing
See TESTING.md for complete testing guidance.
Quick reference (run before committing):
make test 2>&1 | tail -n 500
This runs ALL tests (backend + frontend) and installs dependencies automatically.
For faster iteration, run subsets:
make test-backend- Backend only (~2 min)make test-frontend- Frontend only (~3 min)
TESTING.md covers: test types, CI requirements, Playwright usage, mocking guidelines, and coverage enforcement.
Deployment
Automatic (CI/CD)
Push to main triggers GitHub Actions:
- Run tests
- Build Docker image
- Deploy to Cloud Run
- Deploy frontend to Cloudflare Pages
Infrastructure Changes (Pulumi)
All infrastructure changes must go through PRs. The CI pipeline automatically deploys Pulumi changes when PRs merge to main.
# Preview changes locally (requires GCP auth)
cd infrastructure
pulumi preview --stack nomadkaraoke/karaoke-gen-infrastructure/prod
# DO NOT run `pulumi up` locally - let CI handle deployment
# CI uses --skip-preview due to limited service account permissions
What's managed by Pulumi:
- Firestore database and indexes
- GCS buckets
- Cloud Run service configuration
- Cloud Tasks queues
- Secret Manager secrets
- Service accounts and IAM bindings
- Workload Identity Federation
- GCE instances (GitHub runners, encoding worker)
Workflow:
- Make changes in the appropriate module under
infrastructure/modules/- Core GCP resources (database, storage, secrets, IAM, etc.)compute/- VM definitions and startup scriptsconfig.py- Shared configuration constants__main__.py- Entry point that wires modules together
- Run
pulumi previewlocally to verify - Create PR and merge
- CI deploys automatically
See docs/LESSONS-LEARNED.md for Pulumi CI gotchas.
Manual Frontend Deploy
cd frontend
npm run build
npx wrangler pages deploy out
Environment Variables
Backend (.env)
# Required
AUDIOSHAKE_API_TOKEN=...
GENIUS_ACCESS_TOKEN=...
AUDIO_SEPARATOR_API_URL=... # Modal API endpoint
# GCP (auto-configured in Cloud Run)
GOOGLE_CLOUD_PROJECT=...
GCS_BUCKET_NAME=...
# Auth
ADMIN_TOKENS=token1,token2
Frontend (.env.local)
NEXT_PUBLIC_API_URL=http://localhost:8000
# For E2E tests (see frontend/.env.local.example)
KARAOKE_ACCESS_TOKEN=... # Skip email enrollment
TESTMAIL_API_KEY=... # testmail.app API key
TESTMAIL_NAMESPACE=... # testmail.app namespace
Key Scripts
| Script | Purpose |
|---|---|
make test |
Run all backend tests |
./scripts/run-backend-local.sh |
Start backend with emulators |
./scripts/start-emulators.sh |
Start emulators only |
./scripts/stop-emulators.sh |
Stop emulators |
./scripts/debug-job.sh <job_id> |
Debug a cloud job (status, logs, files) |
./scripts/get_job.py <job_id> |
Fetch job data from Firestore |
./scripts/fetch_job_logs.py <job_id> |
Fetch cloud job logs to local file |
./scripts/analyze_log_timing.py <log> |
Profile karaoke-gen performance from logs |
./scripts/benchmark_ffmpeg.py |
Isolated FFmpeg encoding benchmark |
./scripts/compare_local_vs_remote.py |
Compare local vs cloud karaoke-gen performance |
Project Structure
karaoke-gen/
├── backend/
│ ├── api/routes/ # FastAPI endpoints
│ ├── models/ # Pydantic models
│ ├── services/ # Business logic
│ ├── workers/ # Background workers
│ └── tests/ # Backend tests
├── frontend/
│ ├── src/
│ │ ├── app/ # Next.js pages
│ │ ├── components/ # React components
│ │ └── lib/ # Utilities
│ └── tests/ # Frontend tests
├── infrastructure/ # Pulumi IaC
├── karaoke_gen/ # CLI package
└── scripts/ # Dev scripts
Observability & Debugging
The backend exports OpenTelemetry traces to Google Cloud Trace. Use these for investigating production issues.
Accessing Cloud Trace
Cloud Console (easiest for browsing):
https://console.cloud.google.com/traces/list?project=nomadkaraoke
CLI queries:
# List recent traces (requires `gcloud alpha` for trace commands)
# Easier to use the Console UI for trace browsing
# Find traces by job_id in logs, then correlate with Cloud Trace
gcloud logging read 'resource.type="cloud_run_revision" textPayload=~"JOB_ID"' \
--project=nomadkaraoke --limit=50
Key Traced Spans
| Span Name | What It Measures | Key Attributes |
|---|---|---|
lyrics_corrector.run |
Full lyrics correction pipeline | word_count, reference_count |
lyrics_corrector.find_anchors_and_gaps |
Anchor sequence matching | anchor_count, gap_count |
lyrics_corrector.process_corrections |
Agentic correction processing | gap_count, total_proposals |
anchor_search.find_anchors |
N-gram parallel processing | transcription.text_length, timeout_seconds |
agentic.propose_for_gap |
AI gap classification | gap_id, word_count, gap_category |
add-lyrics |
Adding reference lyrics | job_id, source |
generate-preview-video |
Video preview generation | job_id, use_gce |
Investigating Slow Operations
- Find the trace: Go to Cloud Trace console, filter by time range around the failure
- Look at waterfall: Identify which span took longest
- Check attributes: Each span has metadata (word counts, durations, etc.)
- Correlate with logs: Use timestamp and job_id to find related logs
Example Investigation Workflow
# 1. Find job failures in logs
gcloud logging read 'resource.type="cloud_run_revision" severity>=ERROR' \
--project=nomadkaraoke --limit=20 --format="table(timestamp,textPayload)"
# 2. Get timeline for specific job
./scripts/debug-job.sh <job_id>
# 3. Check anchor search progress (time-based logging every 30s)
gcloud logging read 'resource.type="cloud_run_revision" "ANCHOR SEARCH: Progress"' \
--project=nomadkaraoke --limit=20
# 4. Open Cloud Trace to see detailed timing breakdown
# Filter traces by time window when the job ran
Debugging Scripts
| Script | Purpose |
|---|---|
./scripts/debug-job.sh <job_id> |
Show job status, timeline, logs, GCS files |
./scripts/get_job.py <job_id> |
Fetch raw job data from Firestore |
./scripts/fetch_job_logs.py <job_id> |
Download all logs for a job to local file |
Adding New Traces
When adding new operations that may need investigation:
from karaoke_gen.lyrics_transcriber.utils.tracing import create_span, add_span_attribute
with create_span("operation_name", {"initial": "attributes"}) as span:
# ... do work ...
if span:
span.set_attribute("result_count", len(results))
For backend code:
from backend.services.tracing import create_span
with create_span("operation_name", {"job_id": job_id}) as span:
# ... do work ...
Troubleshooting
Emulators Won't Start
# Check if already running
lsof -i :8080 # Firestore
lsof -i :4443 # GCS
# Stop them
./scripts/stop-emulators.sh
Docker Issues
# Ensure Docker is running
docker info
# For GCS emulator
docker pull fsouza/fake-gcs-server
Test Failures
See TESTING.md for debugging test failures, coverage issues, and CI troubleshooting.