## Summary Phase 2 of moving encoding to AWS; Phase 1 was #1236 and #1239. Design and rationale are in `docs/archive/2026-10-29-aws-encoding-worker-plan.md` § Phase 2. When Firestore `config/encoding-worker.aws_video_worker` is true **and** the AWS worker ranks first, `trigger_video_worker` runs the video worker **on the AWS box** instead of starting the Cloud Run job `video-encoding-job`. - **Uploads move to AWS:** the Dropbox/YouTube/Drive uploads (~187 GB/mo, ~$18/mo of Cloud Run internet egress) leave from AWS, on credits. - **No Cloud Run CPU** for the job (~$11/mo). - **Off by default.** Any failure falls back to the Cloud Run job, so this path can't drop a job. ## Changes - **`POST /run-worker` (worker):** - Spawns `python -m backend.workers.video_worker --job-id …` with the box's video-worker env. - Status is reported via `/status/{id}`. - A run counts in `/health` `active_jobs` from the moment it's accepted, so the idle watchdog won't stop the box mid-run. - Returns 503 unless the env file exists. - `job_id` is pattern-validated. - Any startup failure marks the run failed. - **`backend/services/local_encoded_outputs.py`:** a same-box handoff. - When the co-located video worker asks (`stash_local_outputs`, set only where `LOCAL_ENCODED_OUTPUTS_DIR` exists), the encoder keeps local copies of the finals, keyed by blob path. - The video worker takes those instead of downloading them back out of GCS. Without this, the move would *add* ~400 MB/job of GCS egress. - Copies are made off the event loop and purged after 6 h. - No change on GCE or Cloud Run. - **`video-worker-env.py` (box, every boot):** - Mirrors the **live** `video-encoding-job` env, with secrets resolved, so nothing is hand-copied and nothing drifts. - Runs as the job's identity, `karaoke-backend@`, via a second WIF binding (Pulumi `aws-encoding-worker-wif-backend-user`). - On failure it writes no env file, so `/run-worker` answers 503 and the Cloud Run job is used. - **`WorkerService._trigger_video_worker_on_aws`:** checks the flag and that the AWS worker ranks first, ensures it's running, waits for health, then POSTs `/run-worker`. Returns False on any doubt. - **Also:** `startup-aws.sh` creates the backend credential config and runs the env helper; CI uploads `video-worker-env.py`; `worker_code_paths.txt` now gates `local_encoded_outputs.py`. ## Rollout (needs `/gcp-admin`) 1. **Before merge:** targeted `pulumi up` of the backend WIF binding. It's additive. 2. **After stage B of Phase 1** (AWS worker preferred, timings verified): stop/start the box once so the new `startup-aws.sh` writes the env, then set `aws_video_worker: true`. Set it to `false` to roll back. ## Testing - **27 new tests:** - handoff stash/take/purge, path-escape guard, and the per-request stash flag; - `/run-worker`: disabled, bad module, dedupe, active immediately, spawn args/env, failure log tail, wheel-not-ready, safe start, unsafe ids; - env builder; - the trigger's AWS branch: off, not first, happy path, didn't start, refusals, errors. - **Full backend + unit suites:** no new failures (the 13 local failures are identical on main). - **Local CodeRabbit:** 5 findings, all fixed. The code-paths gate test also caught a missing worker path, now fixed. @coderabbitai ignore 🤖 Generated with [Claude Code](https://claude.com/claude-code) https://claude.ai/code/session_012RYLVSR5qgwsxTQtvGimKF --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
nomad-aws: Nomad Karaoke's AWS account (790139457751)
This stack holds the AWS encoding worker, an on-demand EC2 instance running the same gce_encoding HTTP service as the GCE encoding VMs. It is paid for by AWS Activate credits ($5k, expiring 2028-10-31). The plan and benchmark are in docs/archive/2026-10-29-aws-encoding-worker-plan.md.
What's here
| Resource | Purpose |
|---|---|
aws-encoding-worker-1 |
c7a.4xlarge on-demand, us-east-2b. Stops on OS shutdown (idle watchdog); the EBS root survives. Spot capacity was too thin (see __main__.py). The Elastic IP serves :8080. |
nomad-encoding-worker role |
The box's identity. GCP federates it to encoding-worker@nomadkaraoke (WIF pool aws-encoding, in the GCP stack's modules/aws_encoding_wif.py). It also has SSM for a shell. |
nomad-encoding-controller user |
Backend rights: start/stop instances tagged Project=nomad-encoding, plus describe. |
provision.sh (in infrastructure/encoding-worker/aws/) runs once as user-data. Every boot after that runs bootstrap.sh → startup-aws.sh, which CI uploads to GCS. The new wheel installs at boot and per job, so there is no blue-green deploy for this box. idle-watchdog.py stops the box when it's idle.
Apply
pulumi up runs locally and never in CI, with Nomad's AWS keys from the workspace .envrc:
cd infrastructure/aws && python3 -m venv venv && ./venv/bin/pip install -r requirements.txt
AWS_ACCESS_KEY_ID=$NOMAD_AWS_ACCESS_KEY_ID AWS_SECRET_ACCESS_KEY=$NOMAD_AWS_SECRET_ACCESS_KEY \
pulumi up --stack nomadkaraoke/prod
ami and user_data are ignore_changes, so a newer Ubuntu or an edited provision.sh never replaces the worker's disk. To re-provision on purpose, use pulumi up --replace <instance urn>.
One-time wiring after the first up
-
Controller key → GCP secret (needs
/gcp-admin):aws iam create-access-key --user-name nomad-encoding-controller \ | jq -c '{access_key_id:.AccessKey.AccessKeyId, secret_access_key:.AccessKey.SecretAccessKey}' \ | scripts/gcp-admin gcloud secrets versions add aws-encoding-controller-credentials --data-file=- --project=nomadkaraoke -
Register the worker in the pool: append
pulumi stack output fallback_vms_entryto the JSON list in secretencoding-worker-fallback-vms. The backend picks up:lateston new instances, and the next deploy picks it up everywhere. -
Rank: by default the AWS worker is a last resort (
SPEED_RANK["c7a"] = 90). To prefer it, set Firestoreconfig/encoding-worker.speed_rank_overrides = {"c7a": 5}. Delete the field to roll back; no deploy is needed either way. -
Distribution on AWS (Phase 2): once the AWS worker is preferred (step 3), set Firestore
config/encoding-worker.aws_video_worker = true. The video worker (Dropbox/YouTube/Drive uploads) then runs on the box via/run-worker, with the Cloud Run job as an automatic fallback. Set it tofalseto roll back. Run logs are in/var/log/encoding-worker-runs/.
Operating it
- Shell:
aws ssm start-session --target <instance_id> --region us-east-2. - Logs:
journalctl -u encoding-worker,/var/log/encoding-worker-startup.log,journalctl -u encoding-worker-idle. They are not in Cloud Logging. - Health:
curl http://<eip>:8080/health. - Capacity: if c7a capacity is short, the start fails with
InsufficientInstanceCapacity; that is recorded incapacity_state, demoted for 15 min, and GCE takes over.