Files
karaoke-gen/backend
Andrew Beveridge ☄️andClaude Opus 5.5 78264fa23a feat(encoding): run the video worker (distribution) on the AWS box (v0.289.0) (#1240)
## Summary
Phase 2 of moving encoding to AWS; Phase 1 was #1236 and #1239. Design
and rationale are in
`docs/archive/2026-10-29-aws-encoding-worker-plan.md` § Phase 2.

When Firestore `config/encoding-worker.aws_video_worker` is true **and**
the AWS worker ranks first, `trigger_video_worker` runs the video worker
**on the AWS box** instead of starting the Cloud Run job
`video-encoding-job`.
- **Uploads move to AWS:** the Dropbox/YouTube/Drive uploads (~187
GB/mo, ~$18/mo of Cloud Run internet egress) leave from AWS, on credits.
- **No Cloud Run CPU** for the job (~$11/mo).
- **Off by default.** Any failure falls back to the Cloud Run job, so
this path can't drop a job.

## Changes
- **`POST /run-worker` (worker):**
- Spawns `python -m backend.workers.video_worker --job-id …` with the
box's video-worker env.
  - Status is reported via `/status/{id}`.
- A run counts in `/health` `active_jobs` from the moment it's accepted,
so the idle watchdog won't stop the box mid-run.
  - Returns 503 unless the env file exists.
  - `job_id` is pattern-validated.
  - Any startup failure marks the run failed.
- **`backend/services/local_encoded_outputs.py`:** a same-box handoff.
- When the co-located video worker asks (`stash_local_outputs`, set only
where `LOCAL_ENCODED_OUTPUTS_DIR` exists), the encoder keeps local
copies of the finals, keyed by blob path.
- The video worker takes those instead of downloading them back out of
GCS. Without this, the move would *add* ~400 MB/job of GCS egress.
  - Copies are made off the event loop and purged after 6 h.
  - No change on GCE or Cloud Run.
- **`video-worker-env.py` (box, every boot):**
- Mirrors the **live** `video-encoding-job` env, with secrets resolved,
so nothing is hand-copied and nothing drifts.
- Runs as the job's identity, `karaoke-backend@`, via a second WIF
binding (Pulumi `aws-encoding-worker-wif-backend-user`).
- On failure it writes no env file, so `/run-worker` answers 503 and the
Cloud Run job is used.
- **`WorkerService._trigger_video_worker_on_aws`:** checks the flag and
that the AWS worker ranks first, ensures it's running, waits for health,
then POSTs `/run-worker`. Returns False on any doubt.
- **Also:** `startup-aws.sh` creates the backend credential config and
runs the env helper; CI uploads `video-worker-env.py`;
`worker_code_paths.txt` now gates `local_encoded_outputs.py`.

## Rollout (needs `/gcp-admin`)
1. **Before merge:** targeted `pulumi up` of the backend WIF binding.
It's additive.
2. **After stage B of Phase 1** (AWS worker preferred, timings
verified): stop/start the box once so the new `startup-aws.sh` writes
the env, then set `aws_video_worker: true`. Set it to `false` to roll
back.

## Testing
- **27 new tests:**
- handoff stash/take/purge, path-escape guard, and the per-request stash
flag;
- `/run-worker`: disabled, bad module, dedupe, active immediately, spawn
args/env, failure log tail, wheel-not-ready, safe start, unsafe ids;
  - env builder;
- the trigger's AWS branch: off, not first, happy path, didn't start,
refusals, errors.
- **Full backend + unit suites:** no new failures (the 13 local failures
are identical on main).
- **Local CodeRabbit:** 5 findings, all fixed. The code-paths gate test
also caught a missing worker path, now fixed.

@coderabbitai ignore

🤖 Generated with [Claude Code](https://claude.com/claude-code)

https://claude.ai/code/session_012RYLVSR5qgwsxTQtvGimKF

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-31 15:02:42 -04:00
..

Backend Development Setup

One-Time Setup

1. Create Virtual Environment

cd /Users/andrew/Projects/karaoke-gen
python3 -m venv backend/venv

2. Activate Virtual Environment

source backend/venv/bin/activate

3. Install Dependencies

IMPORTANT: Use Python 3.12 (not 3.13/3.14 - pydantic has compatibility issues)

# Backend dependencies
pip install -r backend/requirements.txt

# Install karaoke_gen in development mode
pip install -e .

# Install lyrics_transcriber submodule
pip install -e lyrics_transcriber_local/

# Install Google API client (for YouTube upload)
pip install google-api-python-client google-auth-httplib2 google-auth-oauthlib

4. Add Shell Alias (Optional)

Add to ~/.zshrc:

alias backend='cd /Users/andrew/Projects/karaoke-gen && source backend/venv/bin/activate'

Then you can just run:

backend
# Now you're in the project directory with venv activated!

Daily Workflow

Before Making Changes

# Activate venv
source backend/venv/bin/activate

# Or if you set up the alias:
backend

After Making Changes

# 1. Run full validation (catches import errors!)
python3 backend/validate.py

# 2. If validation passes, deploy
./scripts/deploy.sh

Validation Options

source backend/venv/bin/activate
python3 backend/validate.py

Catches:

  • ✅ Syntax errors
  • ✅ Import errors (like the processing_service issue!)
  • ✅ Configuration issues
  • ✅ FastAPI app creation problems
  • ✅ Missing dependencies

This would have caught the ModuleNotFoundError!

Quick Check (No venv needed)

./backend/quick-check.sh

Catches:

  • ✅ Syntax errors
  • ✅ Missing files
  • ✅ Some obvious import issues

Good for quick checks, but won't catch all import errors


Example: Complete Development Session

# 1. Activate environment
backend  # or: source backend/venv/bin/activate

# 2. Make your changes
vim backend/api/routes/jobs.py

# 3. Validate (catches import errors!)
python3 backend/validate.py
# ✅ All validations passed!

# 4. Deploy
./scripts/deploy.sh

# 5. Test
export BACKEND_URL="https://karaoke-backend-ipzqd2k4yq-uc.a.run.app"
export AUTH_TOKEN=$(gcloud auth print-identity-token)
curl -H "Authorization: Bearer $AUTH_TOKEN" $BACKEND_URL/api/health

What About karaoke_gen Imports?

The backend imports from karaoke_gen (the CLI package):

from karaoke_gen.karaoke_gen import KaraokePrep
from karaoke_gen.audio_processor import AudioProcessor

Options:

source backend/venv/bin/activate
pip install -e .

This makes karaoke_gen available in the venv without copying files.

Option 2: Use system Python

Your system Python already has karaoke_gen installed, so validation will work even if the venv doesn't have it (Python will fall back to system packages).


Running the Backend Locally

# Activate venv
source backend/venv/bin/activate

# Set required environment variables
export GOOGLE_CLOUD_PROJECT="nomadkaraoke"
export GCS_BUCKET_NAME="karaoke-gen-storage-nomadkaraoke"

# Optional: Point to local credentials
export GOOGLE_APPLICATION_CREDENTIALS="/path/to/service-account-key.json"

# Run the server
cd backend
uvicorn main:app --reload --port 8080

Visit: http://localhost:8080/api/health


Troubleshooting

"No module named 'fastapi'"

Fix:

source backend/venv/bin/activate
pip install -r backend/requirements.txt

"No module named 'karaoke_gen'"

Fix:

source backend/venv/bin/activate
cd /Users/andrew/Projects/karaoke-gen
pip install -e .

Validation passes locally but fails in Cloud Run

Check:

  1. Is requirements.txt up to date?
  2. Did you add a new dependency without adding it to requirements.txt?
  3. Run: pip freeze | grep <package-name>

Updating Dependencies

When you add new packages:

# 1. Install in venv
source backend/venv/bin/activate
pip install new-package

# 2. Update requirements.txt
pip freeze > backend/requirements.txt

# 3. Or manually add just the new package:
echo "new-package==1.2.3" >> backend/requirements.txt

# 4. Validate
python3 backend/validate.py

# 5. Deploy
./scripts/deploy.sh

Summary

Best practice workflow:

  1. One-time setup: Create venv and install dependencies
  2. Daily: Activate venv when working on backend
  3. Before deploy: Run python3 backend/validate.py (catches import errors!)
  4. Deploy: Use ./scripts/deploy.sh

The import error would have been caught if you ran validation in a venv! ✅