## Summary
Phase 1 of moving encoding to AWS. EC2 is paid for by the AWS Activate
credits (acct 790139457751, expiring 2028-10-31).
- **Plan:** `docs/archive/2026-10-29-aws-encoding-worker-plan.md`.
- **Benchmark** (3 real prod jobs replayed): an EC2 c7a.4xlarge matched
or beat GCE c4d-highcpu-16.
The EC2 box runs the **unchanged** `gce_encoding` worker service and
joins the existing candidate pool as `provider: aws`. The existing
failover, active override, lost-job detection and stockout cooldowns all
apply to it as they are.
## Changes
- **`backend/services/aws_encoding_compute.py` (new):**
`AwsComputeAdapter` looks to the manager like the
`compute_v1.InstancesClient` methods it uses.
- EC2 states map to GCE statuses.
- EC2 capacity errors are added to `CAPACITY_ERROR_CODES`.
- A missing instance raises `NotFound`.
- Controller credentials come from Secret Manager
`aws-encoding-controller-credentials`.
- **`EncodingWorkerManager`:**
- `_compute_for(vm)` sends AWS VMs to the adapter; the GCE paths are
untouched.
- A shared `parse_fallback_vms` passes `provider`/`instance_id`/`region`
through.
- `ranked_candidates()` is used by both the encode path and the warmups.
- `ensure_preferred_running()` behaves **exactly like
`ensure_primary_running` whenever the GCE primary ranks first**, which
is today's case.
- An AWS control-plane fault fails over to the GCE candidates.
- **Ranking:** `SPEED_RANK["c7a"] = 90`, so AWS is the last resort by
default. Firestore `config/encoding-worker.speed_rank_overrides =
{"c7a": 5}` makes it preferred, with no deploy.
- **Isolation from GCE-only tooling:**
- The override now records `active_override_provider`.
- The CI blue-green deploy, `deploy_promote` and the idle Cloud Function
all ignore the AWS worker; it hot-updates its wheel per job.
- A GCE promotion clears any stale provider tag.
- **Worker:**
- Uses `gcloud storage` on AWS, because gsutil can't use WIF credential
files. GCE keeps gsutil.
- `infrastructure/encoding-worker/aws/` adds:
- `provision.sh`, as EC2 user-data;
- `startup-aws.sh`, uploaded by CI;
- `idle-watchdog.py`, an idle self-stop that runs as its own systemd
timer.
- **Infra (GCP), already applied with `/gcp-admin`:**
- The WIF pool `aws-encoding`, plus an AWS provider that only admits the
`nomad-encoding-worker` role, federated to `encoding-worker@`.
- The secret `aws-encoding-controller-credentials`.
- Preview and apply: 4 creates, 288 unchanged.
- **Infra (AWS):** a new Pulumi project `infrastructure/aws` (stack
`nomadkaraoke/prod`):
- the c7a.8xlarge **persistent Spot** worker, which stops on
interruption or shutdown;
- an Elastic IP;
- a security group on 8080 (API key, same as the GCE firewall);
- the worker role with SSM;
- a tag-scoped `nomad-encoding-controller` user.
- Preview: 10 creates. **I apply it after merge**, because the box
provisions from the deployed wheel and boot scripts.
- `boto3` is now an explicit dependency.
- CI also runs the `encoding_worker_idle` function tests, in the job
renamed "Infra - Cloud Function tests".
## Rollout (after merge)
1. **Stage A:** `pulumi up` the AWS stack. Store the controller key in
the secret, then add the worker's entry to
`encoding-worker-fallback-vms`. It's inert unless both c4d and c2d stock
out.
2. **Stage B:** a supervised `speed_rank_overrides {"c7a": 5}` window.
Run real jobs, then compare timings and outputs, including a visual
check of the portrait video.
3. **Stage C:** keep it preferred. After a clean week, delete the c4d
pair.
Phase 2, running distribution (Dropbox/YouTube/Drive uploads) from AWS,
follows in a separate PR.
## Testing
- **New tests:**
- adapter: botocore `Stubber`, covering the state map, capacity and
NotFound handling;
- manager routing and ranking with overrides;
- `ensure_preferred_running` branches;
- failover on an AWS control-plane fault;
- provider tagging;
- deploy_promote exclusions;
- the idle function skipping AWS;
- watchdog decisions;
- the gcs CLI choice.
- **Updated** the candidate and warmup-route tests to the new seams.
- **Full backend + unit suites** have no new failures. The local
emulator and cross-suite failures are identical on main.
- **Local CodeRabbit:** 2 findings, both fixed (failover on AWS faults;
stale provider tag).
@coderabbitai ignore
🤖 Generated with [Claude Code](https://claude.com/claude-code)
https://claude.ai/code/session_012RYLVSR5qgwsxTQtvGimKF
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
3.2 KiB
nomad-aws: Nomad Karaoke's AWS account (790139457751)
This stack holds the AWS encoding worker, an EC2 Spot instance running the same gce_encoding HTTP service as the GCE encoding VMs. It is paid for by AWS Activate credits ($5k, expiring 2028-10-31). The plan and benchmark are in docs/archive/2026-10-29-aws-encoding-worker-plan.md.
What's here
| Resource | Purpose |
|---|---|
aws-encoding-worker-1 |
c7a.8xlarge persistent Spot, us-east-2b. Stops on interruption and on OS shutdown; the EBS root survives. The Elastic IP serves :8080. |
nomad-encoding-worker role |
The box's identity. GCP federates it to encoding-worker@nomadkaraoke (WIF pool aws-encoding, in the GCP stack's modules/aws_encoding_wif.py). It also has SSM for a shell. |
nomad-encoding-controller user |
Backend rights: start/stop instances tagged Project=nomad-encoding, plus describe. |
provision.sh (in infrastructure/encoding-worker/aws/) runs once as user-data. Every boot after that runs bootstrap.sh → startup-aws.sh, which CI uploads to GCS. The new wheel installs at boot and per job, so there is no blue-green deploy for this box. idle-watchdog.py stops the box when it's idle.
Apply
pulumi up runs locally and never in CI, with Nomad's AWS keys from the workspace .envrc:
cd infrastructure/aws && python3 -m venv venv && ./venv/bin/pip install -r requirements.txt
AWS_ACCESS_KEY_ID=$NOMAD_AWS_ACCESS_KEY_ID AWS_SECRET_ACCESS_KEY=$NOMAD_AWS_SECRET_ACCESS_KEY \
pulumi up --stack nomadkaraoke/prod
ami and user_data are ignore_changes, so a newer Ubuntu or an edited provision.sh never replaces the worker's disk. To re-provision on purpose, use pulumi up --replace <instance urn>.
One-time wiring after the first up
- Controller key → GCP secret (needs
/gcp-admin):aws iam create-access-key --user-name nomad-encoding-controller \ | jq -c '{access_key_id:.AccessKey.AccessKeyId, secret_access_key:.AccessKey.SecretAccessKey}' \ | scripts/gcp-admin gcloud secrets versions add aws-encoding-controller-credentials --data-file=- --project=nomadkaraoke - Register the worker in the pool: append
pulumi stack output fallback_vms_entryto the JSON list in secretencoding-worker-fallback-vms. The backend picks up:lateston new instances, and the next deploy picks it up everywhere. - Rank: by default the AWS worker is a last resort (
SPEED_RANK["c7a"] = 90). To prefer it, set Firestoreconfig/encoding-worker.speed_rank_overrides = {"c7a": 5}. Delete the field to roll back; no deploy is needed either way.
Operating it
- Shell:
aws ssm start-session --target <instance_id> --region us-east-2. - Logs:
journalctl -u encoding-worker,/var/log/encoding-worker-startup.log,journalctl -u encoding-worker-idle. They are not in Cloud Logging. - Health:
curl http://<eip>:8080/health. - Spot interruption: EC2 stops the box. An in-flight encode/render is detected as lost (the box is no longer RUNNING) and resubmitted. If c7a capacity is short, the start fails with
InsufficientInstanceCapacity; that is recorded incapacity_state, demoted for 15 min, and GCE takes over.