## Summary
Phase 1 of moving encoding to AWS. EC2 is paid for by the AWS Activate
credits (acct 790139457751, expiring 2028-10-31).
- **Plan:** `docs/archive/2026-10-29-aws-encoding-worker-plan.md`.
- **Benchmark** (3 real prod jobs replayed): an EC2 c7a.4xlarge matched
or beat GCE c4d-highcpu-16.
The EC2 box runs the **unchanged** `gce_encoding` worker service and
joins the existing candidate pool as `provider: aws`. The existing
failover, active override, lost-job detection and stockout cooldowns all
apply to it as they are.
## Changes
- **`backend/services/aws_encoding_compute.py` (new):**
`AwsComputeAdapter` looks to the manager like the
`compute_v1.InstancesClient` methods it uses.
- EC2 states map to GCE statuses.
- EC2 capacity errors are added to `CAPACITY_ERROR_CODES`.
- A missing instance raises `NotFound`.
- Controller credentials come from Secret Manager
`aws-encoding-controller-credentials`.
- **`EncodingWorkerManager`:**
- `_compute_for(vm)` sends AWS VMs to the adapter; the GCE paths are
untouched.
- A shared `parse_fallback_vms` passes `provider`/`instance_id`/`region`
through.
- `ranked_candidates()` is used by both the encode path and the warmups.
- `ensure_preferred_running()` behaves **exactly like
`ensure_primary_running` whenever the GCE primary ranks first**, which
is today's case.
- An AWS control-plane fault fails over to the GCE candidates.
- **Ranking:** `SPEED_RANK["c7a"] = 90`, so AWS is the last resort by
default. Firestore `config/encoding-worker.speed_rank_overrides =
{"c7a": 5}` makes it preferred, with no deploy.
- **Isolation from GCE-only tooling:**
- The override now records `active_override_provider`.
- The CI blue-green deploy, `deploy_promote` and the idle Cloud Function
all ignore the AWS worker; it hot-updates its wheel per job.
- A GCE promotion clears any stale provider tag.
- **Worker:**
- Uses `gcloud storage` on AWS, because gsutil can't use WIF credential
files. GCE keeps gsutil.
- `infrastructure/encoding-worker/aws/` adds:
- `provision.sh`, as EC2 user-data;
- `startup-aws.sh`, uploaded by CI;
- `idle-watchdog.py`, an idle self-stop that runs as its own systemd
timer.
- **Infra (GCP), already applied with `/gcp-admin`:**
- The WIF pool `aws-encoding`, plus an AWS provider that only admits the
`nomad-encoding-worker` role, federated to `encoding-worker@`.
- The secret `aws-encoding-controller-credentials`.
- Preview and apply: 4 creates, 288 unchanged.
- **Infra (AWS):** a new Pulumi project `infrastructure/aws` (stack
`nomadkaraoke/prod`):
- the c7a.8xlarge **persistent Spot** worker, which stops on
interruption or shutdown;
- an Elastic IP;
- a security group on 8080 (API key, same as the GCE firewall);
- the worker role with SSM;
- a tag-scoped `nomad-encoding-controller` user.
- Preview: 10 creates. **I apply it after merge**, because the box
provisions from the deployed wheel and boot scripts.
- `boto3` is now an explicit dependency.
- CI also runs the `encoding_worker_idle` function tests, in the job
renamed "Infra - Cloud Function tests".
## Rollout (after merge)
1. **Stage A:** `pulumi up` the AWS stack. Store the controller key in
the secret, then add the worker's entry to
`encoding-worker-fallback-vms`. It's inert unless both c4d and c2d stock
out.
2. **Stage B:** a supervised `speed_rank_overrides {"c7a": 5}` window.
Run real jobs, then compare timings and outputs, including a visual
check of the portrait video.
3. **Stage C:** keep it preferred. After a clean week, delete the c4d
pair.
Phase 2, running distribution (Dropbox/YouTube/Drive uploads) from AWS,
follows in a separate PR.
## Testing
- **New tests:**
- adapter: botocore `Stubber`, covering the state map, capacity and
NotFound handling;
- manager routing and ranking with overrides;
- `ensure_preferred_running` branches;
- failover on an AWS control-plane fault;
- provider tagging;
- deploy_promote exclusions;
- the idle function skipping AWS;
- watchdog decisions;
- the gcs CLI choice.
- **Updated** the candidate and warmup-route tests to the new seams.
- **Full backend + unit suites** have no new failures. The local
emulator and cross-suite failures are identical on main.
- **Local CodeRabbit:** 2 findings, both fixed (failover on AWS faults;
stale provider tag).
@coderabbitai ignore
🤖 Generated with [Claude Code](https://claude.com/claude-code)
https://claude.ai/code/session_012RYLVSR5qgwsxTQtvGimKF
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
11 KiB
Karaoke-Gen AI Assistant Guidelines
Project Overview
Karaoke video generation platform - CLI tool and web service that creates professional karaoke videos with synchronized lyrics.
- Production: https://gen.nomadkaraoke.com (frontend), https://api.nomadkaraoke.com (backend)
- CLI:
karaoke-gen(local),karaoke-gen-remote(cloud) - Repo: https://github.com/nomadkaraoke/karaoke-gen
Quick Reference
| What | Where |
|---|---|
| Product vision & goals | docs/PRODUCT-VISION.md |
| Current status | docs/README.md |
| Architecture | docs/ARCHITECTURE.md |
| Dev setup & testing | docs/DEVELOPMENT.md |
| Testing & code quality | docs/TESTING.md |
| API reference | docs/API.md |
| Past learnings | docs/LESSONS-LEARNED.md |
| Operational runbooks | docs/TROUBLESHOOTING.md |
| GDrive validator & gaps | docs/GDRIVE-VALIDATOR.md |
| Product communication | docs/PRODUCT-COMMUNICATION-GUIDE.md |
| Brand style guide | docs/BRAND-STYLE-GUIDE.md |
| Making medleys (multi-song karaoke) | docs/MEDLEYS.md + scripts/medley/make_medley.py |
Tech Stack
- Backend: FastAPI on Cloud Run, Firestore, GCS, Secret Manager
- Frontend: Next.js on Cloudflare Pages
- Processing: Cloud Run GPU (L4 audio separation), AudioShake/Whisper (transcription)
- Infra: Pulumi IaC, GitHub Actions CI/CD
Essential Rules
Git Workflow
- Never commit directly to
main- use/new-worktree <description>to start - Follow global workflow - see
~/.claude/CLAUDE.mdfor command sequence - Create PR with summary, changes, testing info
- Merge only after CI passes
Testing (Required)
Before planning any implementation work, read docs/TESTING.md and ensure your plan includes a thorough testing strategy that follows the guidance there. This includes:
- Which test types are needed (unit, integration, E2E)
- Where tests should be placed
- What mocking approach to use
- Coverage expectations
- Production E2E tests for user-facing features - see below
Critical: If your plan includes "Manual Testing" steps, convert them to automated Playwright E2E tests. Production E2E tests (in frontend/e2e/production/) are valuable even if only run once after deployment - they codify expected behavior and verify the deployed code works. See docs/TESTING.md for details.
Before committing, run all tests:
make test 2>&1 | tail -n 500
This single command:
- Installs dependencies automatically (backend + frontend)
- Runs all backend tests (unit, integration, emulator)
- Runs all frontend tests (Jest unit + Playwright E2E)
- Takes ~5-10 minutes
All tests must pass. Don't dismiss failures as "pre-existing" - investigate and fix them.
Testing in Production (for Agents)
When asked to "test it yourself in prod" or verify a fix in production:
# Get admin token
export KARAOKE_ADMIN_TOKEN=$(gcloud secrets versions access latest --secret=admin-tokens --project=nomadkaraoke | cut -d',' -f1)
# Option 1: Use the debug template directly
cd frontend && node e2e/helpers/debug-prod-template.mjs
# Option 2: Copy template for customization (gitignored)
cp frontend/e2e/helpers/debug-prod-template.mjs frontend/test-my-issue.local.mjs
# Edit the script, then run it
node test-my-issue.local.mjs
# Option 3: Run existing production E2E tests
KARAOKE_ADMIN_TOKEN=$KARAOKE_ADMIN_TOKEN npx playwright test e2e/production/admin-dashboard.spec.ts
Files matching test-*.local.* and debug-*.local.* are gitignored - you can hardcode tokens in them safely.
See docs/TESTING.md § "Ad-Hoc Production Debugging" for full details.
Version Bumping
- Bump
tool.poetry.versioninpyproject.tomlfor code changes - Skip for docs-only changes
Infrastructure
- All GCP changes via Pulumi PRs - but CI does not run Pulumi (see the note in
.github/workflows/ci.yml): a mergedinfrastructure/change is not live until someone runspulumi uplocally. Claude sessions have read-only ADC, so they need Andrew's/gcp-admingrant to apply. Ask for it early, and vet with a targetedpulumi preview --difffirst infrastructure/is the shared platform IaC home: the GCP project (infrastructure/) and the Nomad Karaoke Cloudflare account (infrastructure/cloudflare/, own Pulumi project + stack, see its README). Never hand-edit Cloudflare config that stack manages; app repos keep only app-specific stacks- Run
pulumi uplocally BEFORE merging infra PRs - CI only validates; it never applies. If you merge first, main and the live state drift until someone applies - Never modify GCP resources directly via console or
gcloudCLI gcloudCLI for reading/debugging only (e.g., checking logs, SSH to VMs)- Stop and notify user on auth issues
- GCS ops on macOS: never use
gsutil -m— its fork-based multiprocessing segfaults on macOS (crash-report popups + the parent process hangs forever). Usegcloud storageinstead (gcloud storage cp/rm/rsync …) — no fork, and faster. Plaingsutil(no-m) is safe but slower.~/.botopinsparallel_process_count=1as a safety net; don't remove it.
Internationalization (i18n)
- All user-facing strings live in
frontend/messages/{locale}.json(33 locales), NOT in components - Backend strings live in
backend/translations/{locale}.json(currently 3 locales: en, es, de) - Components use
useTranslations('namespace')from next-intl - Pages are under
frontend/app/[locale]/— locale-aware routing (customer-facing:/,/app/*,/order/*,/auth/*,/payment/*,/r/*) - Non-locale routes at those same paths (e.g.
frontend/app/app/jobs/[[...slug]]/page.tsx) are<LocaleRedirect />shims that redirect into[locale]— they are NOT English-only surfaces - The
/app/jobs/#/{id}/review,/instrumental, and/audio-editreview UIs are fully multilingual (components undercomponents/lyrics-review/,components/instrumental-review/,components/audio-editor/useuseTranslationsheavily) - Only
/admin/*is English-only — it has no[locale]/admincounterpart and inheritsDefaultIntlProviderfrom the root layout - Internal links use
Linkfrom@/i18n/routing(locale-aware) - After adding/changing English strings:
python frontend/scripts/translate.py --messages-dir frontend/messages --target all - Translation uses Gemini 3.8 Flash via the Gemini Developer API (key:
GEMINI_API_KEY, else Secret Managergemini-api-keyvia your gcloud login —frontend/scripts/gemini_client.py) with GCS cache for efficiency. Vertex AI is disabled in the GCP project — never usevertexai=True; backend code usesbackend/services/gemini_client.get_genai_client(). - CI validates all locale files have matching keys — PR will fail if translations are missing
- Pre-commit hook auto-translates when en.json changes (enable:
git config core.hooksPath .githooks) - Don't hardcode user-facing strings — add to
messages/en.jsonand uset('key')
API Authentication (for Agents)
When you need to call backend APIs programmatically:
Internal Worker Endpoints
Use the X-Admin-Token header with the token from Secret Manager:
# Trigger video worker
curl -X POST "https://api.nomadkaraoke.com/api/internal/workers/video" \
-H "X-Admin-Token: $(gcloud secrets versions access latest --secret=admin-tokens --project=nomadkaraoke)" \
-H "Content-Type: application/json" \
-d '{"job_id": "YOUR_JOB_ID"}'
# Other internal endpoints follow the same pattern
Firestore Direct Access
# Set the correct project
export GOOGLE_CLOUD_PROJECT=nomadkaraoke
# Use Python with google-cloud-firestore
python3 << 'EOF'
import os
os.environ['GOOGLE_CLOUD_PROJECT'] = 'nomadkaraoke'
from google.cloud import firestore
db = firestore.Client(project='nomadkaraoke')
# Query/update documents...
EOF
GCE Encoding Worker
The worker is a Spot c4d-highcpu-16 blue-green primary pair (encoding-worker-a/-b,
us-central1-c) plus ONE stopped on-demand fallback (encoding-worker-fallback-c2df,
c2d-highcpu-16, us-central1-f) — 3 VMs since 2026-09-26. Idle VMs stop after 5 min.
CI blue-green redeploys the worker only when worker code changed
(infrastructure/encoding-worker/worker_code_paths.txt); force it via Actions → CI →
Run workflow → "Force a blue-green redeploy of the GCE encoding worker". The currently-serving
VM is active_override_vm in config/encoding-worker if set, else primary_vm.
Use that VM name + its zone below (there is no VM literally named encoding-worker):
# Check health (includes wheel_version). Example uses the primary pair's VM.
gcloud compute ssh encoding-worker-a --zone=us-central1-c --project=nomadkaraoke \
--command="curl -s http://localhost:8080/health"
# Restart the service (pick up new worker-side code after a deploy)
gcloud compute ssh encoding-worker-a --zone=us-central1-c --project=nomadkaraoke \
--command="sudo systemctl restart encoding-worker"
For job stuck at encoding status, see docs/TROUBLESHOOTING.md.
AWS encoding worker (aws-encoding-worker-1, EC2 c7a.8xlarge Spot, us-east-2) is a pool
candidate too (provider: aws in encoding-worker-fallback-vms): last resort by default,
preferred when Firestore config/encoding-worker.speed_rank_overrides = {"c7a": 5}. Not
blue-green deployed, idle-stops itself, shell via SSM, logs are on the box (journald). See
infrastructure/aws/README.md.
What Doesn't Work
gcloud auth print-identity-token- Wrong token type for internal endpoints- Cloud Tasks without proper service account - Gets 401 from Cloud Run
Documentation Maintenance
Structure
docs/
├── README.md # Current status + navigation (UPDATE THIS)
├── ARCHITECTURE.md # System design
├── DEVELOPMENT.md # Dev setup, testing, deployment
├── API.md # Backend API reference
├── LESSONS-LEARNED.md # Key insights for future agents
└── archive/ # Historical docs (YYYY-MM-DD-topic.md)
Rules
- Before merging PRs: Run
/docs-reviewto check if docs need updating - Periodically: Run
/docs-maintainto verify docs are organized - For significant work: Create
docs/archive/YYYY-MM-DD-topic.md - Add learnings: Update
docs/LESSONS-LEARNED.mdwith insights - Keep status current: Update
docs/README.mdwhen project state changes
What Goes Where
- README.md: Current status, known issues, quick links
- ARCHITECTURE.md: System design, data flow, tech decisions
- DEVELOPMENT.md: Setup, testing, deployment, CI/CD
- API.md: Endpoints, request/response formats
- LESSONS-LEARNED.md: What worked, what didn't, gotchas
- archive/: Completed work, session summaries, old plans
Related Projects
flacfetch
Location: /Users/andrew/Projects/flacfetch
Workflow: Make changes on main, bump version, push, wait for CI, then poetry update flacfetch in karaoke-gen.
Pre-PR Checklist
Follow global workflow (~/.claude/CLAUDE.md):
- Tests pass:
/test(ormake test/npm run test:all) - Version bumped (if code changed)
- Docs checked:
/docs-review - Code reviewed:
/review(CodeRabbit CLI) - PR created:
/pr(adds @coderabbitai ignore)