Files
karaoke-gen/infrastructure/encoding-worker
Andrew Beveridge ☄️andClaude Opus 5.5 de5dfcd50d feat(encoding): AWS EC2 encoding worker as a pool candidate (v0.288.0) (#1236)
## Summary
Phase 1 of moving encoding to AWS. EC2 is paid for by the AWS Activate
credits (acct 790139457751, expiring 2028-10-31).
- **Plan:** `docs/archive/2026-10-29-aws-encoding-worker-plan.md`.
- **Benchmark** (3 real prod jobs replayed): an EC2 c7a.4xlarge matched
or beat GCE c4d-highcpu-16.

The EC2 box runs the **unchanged** `gce_encoding` worker service and
joins the existing candidate pool as `provider: aws`. The existing
failover, active override, lost-job detection and stockout cooldowns all
apply to it as they are.

## Changes
- **`backend/services/aws_encoding_compute.py` (new):**
`AwsComputeAdapter` looks to the manager like the
`compute_v1.InstancesClient` methods it uses.
  - EC2 states map to GCE statuses.
  - EC2 capacity errors are added to `CAPACITY_ERROR_CODES`.
  - A missing instance raises `NotFound`.
- Controller credentials come from Secret Manager
`aws-encoding-controller-credentials`.
- **`EncodingWorkerManager`:**
- `_compute_for(vm)` sends AWS VMs to the adapter; the GCE paths are
untouched.
- A shared `parse_fallback_vms` passes `provider`/`instance_id`/`region`
through.
- `ranked_candidates()` is used by both the encode path and the warmups.
- `ensure_preferred_running()` behaves **exactly like
`ensure_primary_running` whenever the GCE primary ranks first**, which
is today's case.
  - An AWS control-plane fault fails over to the GCE candidates.
- **Ranking:** `SPEED_RANK["c7a"] = 90`, so AWS is the last resort by
default. Firestore `config/encoding-worker.speed_rank_overrides =
{"c7a": 5}` makes it preferred, with no deploy.
- **Isolation from GCE-only tooling:**
  - The override now records `active_override_provider`.
- The CI blue-green deploy, `deploy_promote` and the idle Cloud Function
all ignore the AWS worker; it hot-updates its wheel per job.
  - A GCE promotion clears any stale provider tag.
- **Worker:**
- Uses `gcloud storage` on AWS, because gsutil can't use WIF credential
files. GCE keeps gsutil.
  - `infrastructure/encoding-worker/aws/` adds:
    - `provision.sh`, as EC2 user-data;
    - `startup-aws.sh`, uploaded by CI;
- `idle-watchdog.py`, an idle self-stop that runs as its own systemd
timer.
- **Infra (GCP), already applied with `/gcp-admin`:**
- The WIF pool `aws-encoding`, plus an AWS provider that only admits the
`nomad-encoding-worker` role, federated to `encoding-worker@`.
  - The secret `aws-encoding-controller-credentials`.
  - Preview and apply: 4 creates, 288 unchanged.
- **Infra (AWS):** a new Pulumi project `infrastructure/aws` (stack
`nomadkaraoke/prod`):
- the c7a.8xlarge **persistent Spot** worker, which stops on
interruption or shutdown;
  - an Elastic IP;
  - a security group on 8080 (API key, same as the GCE firewall);
  - the worker role with SSM;
  - a tag-scoped `nomad-encoding-controller` user.
- Preview: 10 creates. **I apply it after merge**, because the box
provisions from the deployed wheel and boot scripts.
- `boto3` is now an explicit dependency.
- CI also runs the `encoding_worker_idle` function tests, in the job
renamed "Infra - Cloud Function tests".

## Rollout (after merge)
1. **Stage A:** `pulumi up` the AWS stack. Store the controller key in
the secret, then add the worker's entry to
`encoding-worker-fallback-vms`. It's inert unless both c4d and c2d stock
out.
2. **Stage B:** a supervised `speed_rank_overrides {"c7a": 5}` window.
Run real jobs, then compare timings and outputs, including a visual
check of the portrait video.
3. **Stage C:** keep it preferred. After a clean week, delete the c4d
pair.

Phase 2, running distribution (Dropbox/YouTube/Drive uploads) from AWS,
follows in a separate PR.

## Testing
- **New tests:**
- adapter: botocore `Stubber`, covering the state map, capacity and
NotFound handling;
  - manager routing and ranking with overrides;
  - `ensure_preferred_running` branches;
  - failover on an AWS control-plane fault;
  - provider tagging;
  - deploy_promote exclusions;
  - the idle function skipping AWS;
  - watchdog decisions;
  - the gcs CLI choice.
- **Updated** the candidate and warmup-route tests to the new seams.
- **Full backend + unit suites** have no new failures. The local
emulator and cross-suite failures are identical on main.
- **Local CodeRabbit:** 2 findings, both fixed (failover on AWS faults;
stale provider tag).

@coderabbitai ignore

🤖 Generated with [Claude Code](https://claude.com/claude-code)

https://claude.ai/code/session_012RYLVSR5qgwsxTQtvGimKF

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-29 17:42:35 -04:00
..

Encoding Worker Deployment

This directory contains the scripts for the GCE encoding worker's immutable deployment pattern.

Architecture

┌─────────────────────────────────────────────────────────────────┐
│                    PACKER IMAGE (rarely changes)                │
├─────────────────────────────────────────────────────────────────┤
│ - Python 3.13, FFmpeg, fonts (static dependencies)             │
│ - bootstrap.sh (downloads startup.sh from GCS)                 │
│ - systemd service definition                                    │
└─────────────────────────────────────────────────────────────────┘
                              │
                              ▼ downloads on every service start
┌─────────────────────────────────────────────────────────────────┐
│                    GCS (CI-managed, always current)             │
├─────────────────────────────────────────────────────────────────┤
│ gs://karaoke-gen-storage-nomadkaraoke/                         │
│ ├── encoding-worker/                                            │
│ │   ├── startup.sh      ← Main logic (this file)               │
│ │   └── version.txt     ← Expected version number              │
│ └── wheels/                                                     │
│     └── karaoke_gen-current.whl  ← Latest wheel (fixed path)   │
└─────────────────────────────────────────────────────────────────┘

Files

File Location Purpose
bootstrap.sh Baked in Packer image Downloads and runs startup.sh
startup.sh Deployed to GCS by CI Main startup logic
version.txt Written to GCS by CI Expected version for verification

Deployment Flow

  1. CI builds wheel → uploads to wheels/karaoke_gen-current.whl
  2. CI uploads startup.sh → encoding-worker/startup.sh
  3. CI writes version → encoding-worker/version.txt
  4. CI restarts service → triggers bootstrap.sh
  5. bootstrap.sh downloads fresh startup.sh from GCS
  6. startup.sh downloads wheel, installs, verifies version
  7. CI verifies health endpoint returns correct version
  8. CI FAILS if version mismatch

Why This Pattern?

Before (problematic)

  • startup.sh baked into Packer image
  • Required image rebuild + VM recreation for logic changes
  • 51 versioned wheels, sorting could fail
  • CI warned on version mismatch but continued

After (robust)

  • startup.sh downloaded from GCS on every start
  • Logic changes via CI, no image rebuild needed
  • Fixed path karaoke_gen-current.whl, no sorting
  • CI FAILS on version mismatch

Modifying Startup Logic

  1. Edit infrastructure/encoding-worker/startup.sh
  2. Push to main
  3. CI automatically deploys to GCS
  4. Service restart picks up new script

No Packer image rebuild needed!

When to Rebuild Packer Image

Only rebuild when changing:

  • Python version
  • FFmpeg version
  • System packages/fonts
  • bootstrap.sh (should be extremely rare)

Manual Operations

# Check current version on VM
gcloud compute ssh encoding-worker --zone=us-central1-c --project=nomadkaraoke \
  --tunnel-through-iap \
  --command="curl -s http://localhost:8080/health | jq '.wheel_version'"

# View startup logs
gcloud compute ssh encoding-worker --zone=us-central1-c --project=nomadkaraoke \
  --tunnel-through-iap \
  --command="sudo journalctl -u encoding-worker -n 50"

# Force service restart (triggers fresh startup.sh download)
gcloud compute ssh encoding-worker --zone=us-central1-c --project=nomadkaraoke \
  --tunnel-through-iap \
  --command="sudo systemctl restart encoding-worker"

Troubleshooting

Version mismatch after deployment

  1. Check CI logs for GCS upload success
  2. Check /var/log/encoding-worker-startup.log on VM
  3. Verify version.txt in GCS matches expected

Service won't start

  1. Check bootstrap.sh can reach GCS: gsutil ls gs://karaoke-gen-storage-nomadkaraoke/
  2. Check Secret Manager access for API key
  3. Check startup.sh logs: cat /var/log/encoding-worker-startup.log