Files
Andrew Beveridge ☄️andClaude Opus 5.5 78264fa23a feat(encoding): run the video worker (distribution) on the AWS box (v0.289.0) (#1240)
## Summary
Phase 2 of moving encoding to AWS; Phase 1 was #1236 and #1239. Design
and rationale are in
`docs/archive/2026-10-29-aws-encoding-worker-plan.md` § Phase 2.

When Firestore `config/encoding-worker.aws_video_worker` is true **and**
the AWS worker ranks first, `trigger_video_worker` runs the video worker
**on the AWS box** instead of starting the Cloud Run job
`video-encoding-job`.
- **Uploads move to AWS:** the Dropbox/YouTube/Drive uploads (~187
GB/mo, ~$18/mo of Cloud Run internet egress) leave from AWS, on credits.
- **No Cloud Run CPU** for the job (~$11/mo).
- **Off by default.** Any failure falls back to the Cloud Run job, so
this path can't drop a job.

## Changes
- **`POST /run-worker` (worker):**
- Spawns `python -m backend.workers.video_worker --job-id …` with the
box's video-worker env.
  - Status is reported via `/status/{id}`.
- A run counts in `/health` `active_jobs` from the moment it's accepted,
so the idle watchdog won't stop the box mid-run.
  - Returns 503 unless the env file exists.
  - `job_id` is pattern-validated.
  - Any startup failure marks the run failed.
- **`backend/services/local_encoded_outputs.py`:** a same-box handoff.
- When the co-located video worker asks (`stash_local_outputs`, set only
where `LOCAL_ENCODED_OUTPUTS_DIR` exists), the encoder keeps local
copies of the finals, keyed by blob path.
- The video worker takes those instead of downloading them back out of
GCS. Without this, the move would *add* ~400 MB/job of GCS egress.
  - Copies are made off the event loop and purged after 6 h.
  - No change on GCE or Cloud Run.
- **`video-worker-env.py` (box, every boot):**
- Mirrors the **live** `video-encoding-job` env, with secrets resolved,
so nothing is hand-copied and nothing drifts.
- Runs as the job's identity, `karaoke-backend@`, via a second WIF
binding (Pulumi `aws-encoding-worker-wif-backend-user`).
- On failure it writes no env file, so `/run-worker` answers 503 and the
Cloud Run job is used.
- **`WorkerService._trigger_video_worker_on_aws`:** checks the flag and
that the AWS worker ranks first, ensures it's running, waits for health,
then POSTs `/run-worker`. Returns False on any doubt.
- **Also:** `startup-aws.sh` creates the backend credential config and
runs the env helper; CI uploads `video-worker-env.py`;
`worker_code_paths.txt` now gates `local_encoded_outputs.py`.

## Rollout (needs `/gcp-admin`)
1. **Before merge:** targeted `pulumi up` of the backend WIF binding.
It's additive.
2. **After stage B of Phase 1** (AWS worker preferred, timings
verified): stop/start the box once so the new `startup-aws.sh` writes
the env, then set `aws_video_worker: true`. Set it to `false` to roll
back.

## Testing
- **27 new tests:**
- handoff stash/take/purge, path-escape guard, and the per-request stash
flag;
- `/run-worker`: disabled, bad module, dedupe, active immediately, spawn
args/env, failure log tail, wheel-not-ready, safe start, unsafe ids;
  - env builder;
- the trigger's AWS branch: off, not first, happy path, didn't start,
refusals, errors.
- **Full backend + unit suites:** no new failures (the 13 local failures
are identical on main).
- **Local CodeRabbit:** 5 findings, all fixed. The code-paths gate test
also caught a missing worker path, now fixed.

@coderabbitai ignore

🤖 Generated with [Claude Code](https://claude.com/claude-code)

https://claude.ai/code/session_012RYLVSR5qgwsxTQtvGimKF

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-31 15:02:42 -04:00
..

nomad-aws: Nomad Karaoke's AWS account (790139457751)

This stack holds the AWS encoding worker, an on-demand EC2 instance running the same gce_encoding HTTP service as the GCE encoding VMs. It is paid for by AWS Activate credits ($5k, expiring 2028-10-31). The plan and benchmark are in docs/archive/2026-10-29-aws-encoding-worker-plan.md.

What's here

Resource Purpose
aws-encoding-worker-1 c7a.4xlarge on-demand, us-east-2b. Stops on OS shutdown (idle watchdog); the EBS root survives. Spot capacity was too thin (see __main__.py). The Elastic IP serves :8080.
nomad-encoding-worker role The box's identity. GCP federates it to encoding-worker@nomadkaraoke (WIF pool aws-encoding, in the GCP stack's modules/aws_encoding_wif.py). It also has SSM for a shell.
nomad-encoding-controller user Backend rights: start/stop instances tagged Project=nomad-encoding, plus describe.

provision.sh (in infrastructure/encoding-worker/aws/) runs once as user-data. Every boot after that runs bootstrap.sh → startup-aws.sh, which CI uploads to GCS. The new wheel installs at boot and per job, so there is no blue-green deploy for this box. idle-watchdog.py stops the box when it's idle.

Apply

pulumi up runs locally and never in CI, with Nomad's AWS keys from the workspace .envrc:

cd infrastructure/aws && python3 -m venv venv && ./venv/bin/pip install -r requirements.txt
AWS_ACCESS_KEY_ID=$NOMAD_AWS_ACCESS_KEY_ID AWS_SECRET_ACCESS_KEY=$NOMAD_AWS_SECRET_ACCESS_KEY \
  pulumi up --stack nomadkaraoke/prod

ami and user_data are ignore_changes, so a newer Ubuntu or an edited provision.sh never replaces the worker's disk. To re-provision on purpose, use pulumi up --replace <instance urn>.

One-time wiring after the first up

  1. Controller key → GCP secret (needs /gcp-admin):

    aws iam create-access-key --user-name nomad-encoding-controller \
      | jq -c '{access_key_id:.AccessKey.AccessKeyId, secret_access_key:.AccessKey.SecretAccessKey}' \
      | scripts/gcp-admin gcloud secrets versions add aws-encoding-controller-credentials --data-file=- --project=nomadkaraoke
    
  2. Register the worker in the pool: append pulumi stack output fallback_vms_entry to the JSON list in secret encoding-worker-fallback-vms. The backend picks up :latest on new instances, and the next deploy picks it up everywhere.

  3. Rank: by default the AWS worker is a last resort (SPEED_RANK["c7a"] = 90). To prefer it, set Firestore config/encoding-worker.speed_rank_overrides = {"c7a": 5}. Delete the field to roll back; no deploy is needed either way.

  4. Distribution on AWS (Phase 2): once the AWS worker is preferred (step 3), set Firestore config/encoding-worker.aws_video_worker = true. The video worker (Dropbox/YouTube/Drive uploads) then runs on the box via /run-worker, with the Cloud Run job as an automatic fallback. Set it to false to roll back. Run logs are in /var/log/encoding-worker-runs/.

Operating it

  • Shell: aws ssm start-session --target <instance_id> --region us-east-2.
  • Logs: journalctl -u encoding-worker, /var/log/encoding-worker-startup.log, journalctl -u encoding-worker-idle. They are not in Cloud Logging.
  • Health: curl http://<eip>:8080/health.
  • Capacity: if c7a capacity is short, the start fails with InsufficientInstanceCapacity; that is recorded in capacity_state, demoted for 15 min, and GCE takes over.