Files
karaoke-gen/backend
Andrew Beveridge ☄️andClaude Opus 5.5 a1cbe1c0d4 feat(encoding): retire the GCE c4d pair; run on AWS + c2df only (0.293.0) (#1255)
## Why
Since 2026-10-31 the AWS EC2 worker (`aws-encoding-worker-1`) has
handled every encode and distribution job. 27 jobs went to it, with no
Cloud Run fallbacks, no `encoding_stuck`, and nothing encoding-related
in #alerts. The c4d Spot pair (`encoding-worker-a`/`-b`) now only costs
~$8/mo of Hyperdisk. `encoding-worker-fallback-c2df` stays as the GCP
fallback.

## What
Deleting the pair needs code changes first, because several paths
assumed it exists:
- **`encoding_worker_manager`**
  - The primary is optional: an empty `primary_vm` means none.
- `ranked_candidates` skips a missing primary and always sets
`is_primary`, so with no primary a successful start records the AWS/c2df
override. Previously the legacy index-0 rule would have cleared it.
- `ensure_primary_running` and `ensure_preferred_running` start a ranked
candidate. If there are no candidates they raise `NO_CANDIDATES`.
- `demote_active_worker` looks up the override's `machine_type`, so the
AWS box can now be cooled down. It then clears the override so the retry
goes elsewhere.
- **`encoding_service`**
  - When `active_url` is empty, it targets the top-ranked candidate.
- It also invalidates its cached URL after a single-candidate fallback.
- **CI deploy (`ci.yml` + `deploy_promote.py`)**
- While AWS serves, a c2df green is validate-only: it writes
`validated_version`, keeps the AWS override, and stops the green. Before
this, it would have replaced the AWS override with c2df.
- The worker-change gate now diffs against `validated_version`, because
`primary_version` stops moving.
- An in-place restart of a serving override first waits for it to drain
(15-min cap). If it is still busy, the restart is skipped.
- **Idle function:** a missing VM is logged as a WARNING returning
`NOT_FOUND`, not an ERROR.
- **Error monitor:** watches c2df instead of a/b.
- **Pulumi:** drops the a/b VMs and static IPs. The prod preview shows
exactly 4 deletes. A separate, unrelated
`youtube-comment-autoreply-scheduler` drift shows up too; it will not be
applied.
- **Docs:** CLAUDE.md, ARCHITECTURE, DEVELOPMENT and TROUBLESHOOTING.

## Rollout (after merge + deploy)
1. Back up `config/encoding-worker`, then set `primary_vm`,
`primary_ip`, `secondary_vm` and `secondary_ip` to `""`.
2. Apply `pulumi destroy --target` on the 4 a/b URNs only.
3. Redeploy the idle function with `deploy.sh`.
4. Verify: one prod encode on AWS, warmup, and the logs.

**Rollback:** in Firestore, delete `speed_rank_overrides` and set
`aws_video_worker=false`. Restoring the pair would need a revert plus
`pulumi up`, which gives new static IPs.

## Testing
- New `backend/tests/test_encoding_worker_no_primary.py` (13 tests),
plus new deploy_promote, idle-function and error-monitor tests.
- Full backend suite: 7188 passed. The 8 `test_emulator_integration`
failures also fail on main, because of a local emulator from another
session.
- CodeRabbit CLI: no findings. An independent review found 2 issues,
both fixed: the demote retry path, and the in-place restart drain.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

https://claude.ai/code/session_012RYLVTkatGnrYFhCzq4PTe

@coderabbitai ignore

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-05 18:50:54 -05:00
..

Backend Development Setup

One-Time Setup

1. Create Virtual Environment

cd /Users/andrew/Projects/karaoke-gen
python3 -m venv backend/venv

2. Activate Virtual Environment

source backend/venv/bin/activate

3. Install Dependencies

IMPORTANT: Use Python 3.12 (not 3.13/3.14 - pydantic has compatibility issues)

# Backend dependencies
pip install -r backend/requirements.txt

# Install karaoke_gen in development mode
pip install -e .

# Install lyrics_transcriber submodule
pip install -e lyrics_transcriber_local/

# Install Google API client (for YouTube upload)
pip install google-api-python-client google-auth-httplib2 google-auth-oauthlib

4. Add Shell Alias (Optional)

Add to ~/.zshrc:

alias backend='cd /Users/andrew/Projects/karaoke-gen && source backend/venv/bin/activate'

Then you can just run:

backend
# Now you're in the project directory with venv activated!

Daily Workflow

Before Making Changes

# Activate venv
source backend/venv/bin/activate

# Or if you set up the alias:
backend

After Making Changes

# 1. Run full validation (catches import errors!)
python3 backend/validate.py

# 2. If validation passes, deploy
./scripts/deploy.sh

Validation Options

source backend/venv/bin/activate
python3 backend/validate.py

Catches:

  • ✅ Syntax errors
  • ✅ Import errors (like the processing_service issue!)
  • ✅ Configuration issues
  • ✅ FastAPI app creation problems
  • ✅ Missing dependencies

This would have caught the ModuleNotFoundError!

Quick Check (No venv needed)

./backend/quick-check.sh

Catches:

  • ✅ Syntax errors
  • ✅ Missing files
  • ✅ Some obvious import issues

Good for quick checks, but won't catch all import errors


Example: Complete Development Session

# 1. Activate environment
backend  # or: source backend/venv/bin/activate

# 2. Make your changes
vim backend/api/routes/jobs.py

# 3. Validate (catches import errors!)
python3 backend/validate.py
# ✅ All validations passed!

# 4. Deploy
./scripts/deploy.sh

# 5. Test
export BACKEND_URL="https://karaoke-backend-ipzqd2k4yq-uc.a.run.app"
export AUTH_TOKEN=$(gcloud auth print-identity-token)
curl -H "Authorization: Bearer $AUTH_TOKEN" $BACKEND_URL/api/health

What About karaoke_gen Imports?

The backend imports from karaoke_gen (the CLI package):

from karaoke_gen.karaoke_gen import KaraokePrep
from karaoke_gen.audio_processor import AudioProcessor

Options:

source backend/venv/bin/activate
pip install -e .

This makes karaoke_gen available in the venv without copying files.

Option 2: Use system Python

Your system Python already has karaoke_gen installed, so validation will work even if the venv doesn't have it (Python will fall back to system packages).


Running the Backend Locally

# Activate venv
source backend/venv/bin/activate

# Set required environment variables
export GOOGLE_CLOUD_PROJECT="nomadkaraoke"
export GCS_BUCKET_NAME="karaoke-gen-storage-nomadkaraoke"

# Optional: Point to local credentials
export GOOGLE_APPLICATION_CREDENTIALS="/path/to/service-account-key.json"

# Run the server
cd backend
uvicorn main:app --reload --port 8080

Visit: http://localhost:8080/api/health


Troubleshooting

"No module named 'fastapi'"

Fix:

source backend/venv/bin/activate
pip install -r backend/requirements.txt

"No module named 'karaoke_gen'"

Fix:

source backend/venv/bin/activate
cd /Users/andrew/Projects/karaoke-gen
pip install -e .

Validation passes locally but fails in Cloud Run

Check:

  1. Is requirements.txt up to date?
  2. Did you add a new dependency without adding it to requirements.txt?
  3. Run: pip freeze | grep <package-name>

Updating Dependencies

When you add new packages:

# 1. Install in venv
source backend/venv/bin/activate
pip install new-package

# 2. Update requirements.txt
pip freeze > backend/requirements.txt

# 3. Or manually add just the new package:
echo "new-package==1.2.3" >> backend/requirements.txt

# 4. Validate
python3 backend/validate.py

# 5. Deploy
./scripts/deploy.sh

Summary

Best practice workflow:

  1. One-time setup: Create venv and install dependencies
  2. Daily: Activate venv when working on backend
  3. Before deploy: Run python3 backend/validate.py (catches import errors!)
  4. Deploy: Use ./scripts/deploy.sh

The import error would have been caught if you ran validation in a venv! ✅