Files
karaoke-gen/infrastructure/encoding-worker/aws/startup-aws.sh
T
Andrew Beveridge ☄️andClaude Opus 5.5 de5dfcd50d feat(encoding): AWS EC2 encoding worker as a pool candidate (v0.288.0) (#1236)
## Summary
Phase 1 of moving encoding to AWS. EC2 is paid for by the AWS Activate
credits (acct 790139457751, expiring 2028-10-31).
- **Plan:** `docs/archive/2026-10-29-aws-encoding-worker-plan.md`.
- **Benchmark** (3 real prod jobs replayed): an EC2 c7a.4xlarge matched
or beat GCE c4d-highcpu-16.

The EC2 box runs the **unchanged** `gce_encoding` worker service and
joins the existing candidate pool as `provider: aws`. The existing
failover, active override, lost-job detection and stockout cooldowns all
apply to it as they are.

## Changes
- **`backend/services/aws_encoding_compute.py` (new):**
`AwsComputeAdapter` looks to the manager like the
`compute_v1.InstancesClient` methods it uses.
  - EC2 states map to GCE statuses.
  - EC2 capacity errors are added to `CAPACITY_ERROR_CODES`.
  - A missing instance raises `NotFound`.
- Controller credentials come from Secret Manager
`aws-encoding-controller-credentials`.
- **`EncodingWorkerManager`:**
- `_compute_for(vm)` sends AWS VMs to the adapter; the GCE paths are
untouched.
- A shared `parse_fallback_vms` passes `provider`/`instance_id`/`region`
through.
- `ranked_candidates()` is used by both the encode path and the warmups.
- `ensure_preferred_running()` behaves **exactly like
`ensure_primary_running` whenever the GCE primary ranks first**, which
is today's case.
  - An AWS control-plane fault fails over to the GCE candidates.
- **Ranking:** `SPEED_RANK["c7a"] = 90`, so AWS is the last resort by
default. Firestore `config/encoding-worker.speed_rank_overrides =
{"c7a": 5}` makes it preferred, with no deploy.
- **Isolation from GCE-only tooling:**
  - The override now records `active_override_provider`.
- The CI blue-green deploy, `deploy_promote` and the idle Cloud Function
all ignore the AWS worker; it hot-updates its wheel per job.
  - A GCE promotion clears any stale provider tag.
- **Worker:**
- Uses `gcloud storage` on AWS, because gsutil can't use WIF credential
files. GCE keeps gsutil.
  - `infrastructure/encoding-worker/aws/` adds:
    - `provision.sh`, as EC2 user-data;
    - `startup-aws.sh`, uploaded by CI;
- `idle-watchdog.py`, an idle self-stop that runs as its own systemd
timer.
- **Infra (GCP), already applied with `/gcp-admin`:**
- The WIF pool `aws-encoding`, plus an AWS provider that only admits the
`nomad-encoding-worker` role, federated to `encoding-worker@`.
  - The secret `aws-encoding-controller-credentials`.
  - Preview and apply: 4 creates, 288 unchanged.
- **Infra (AWS):** a new Pulumi project `infrastructure/aws` (stack
`nomadkaraoke/prod`):
- the c7a.8xlarge **persistent Spot** worker, which stops on
interruption or shutdown;
  - an Elastic IP;
  - a security group on 8080 (API key, same as the GCE firewall);
  - the worker role with SSM;
  - a tag-scoped `nomad-encoding-controller` user.
- Preview: 10 creates. **I apply it after merge**, because the box
provisions from the deployed wheel and boot scripts.
- `boto3` is now an explicit dependency.
- CI also runs the `encoding_worker_idle` function tests, in the job
renamed "Infra - Cloud Function tests".

## Rollout (after merge)
1. **Stage A:** `pulumi up` the AWS stack. Store the controller key in
the secret, then add the worker's entry to
`encoding-worker-fallback-vms`. It's inert unless both c4d and c2d stock
out.
2. **Stage B:** a supervised `speed_rank_overrides {"c7a": 5}` window.
Run real jobs, then compare timings and outputs, including a visual
check of the portrait video.
3. **Stage C:** keep it preferred. After a clean week, delete the c4d
pair.

Phase 2, running distribution (Dropbox/YouTube/Drive uploads) from AWS,
follows in a separate PR.

## Testing
- **New tests:**
- adapter: botocore `Stubber`, covering the state map, capacity and
NotFound handling;
  - manager routing and ranking with overrides;
  - `ensure_preferred_running` branches;
  - failover on an AWS control-plane fault;
  - provider tagging;
  - deploy_promote exclusions;
  - the idle function skipping AWS;
  - watchdog decisions;
  - the gcs CLI choice.
- **Updated** the candidate and warmup-route tests to the new seams.
- **Full backend + unit suites** have no new failures. The local
emulator and cross-suite failures are identical on main.
- **Local CodeRabbit:** 2 findings, both fixed (failover on AWS faults;
stale provider tag).

@coderabbitai ignore

🤖 Generated with [Claude Code](https://claude.com/claude-code)

https://claude.ai/code/session_012RYLVSR5qgwsxTQtvGimKF

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-29 17:42:35 -04:00

66 lines
2.8 KiB
Bash
Executable File

#!/bin/bash
# AWS encoding worker startup (run by bootstrap.sh on every service start).
#
# AWS counterpart of ../startup.sh. Differences: no GCE metadata server (GCP
# auth is Workload Identity Federation via gcp-wif.json + the instance's IMDSv2
# role), `gcloud storage` instead of gsutil (gsutil can't use WIF credential
# files), and the VM name comes from /opt/encoding-worker/vm-name.
#
# CI uploads this to gs://karaoke-gen-storage-nomadkaraoke/encoding-worker/startup-aws.sh
set -e
BUCKET="gs://karaoke-gen-storage-nomadkaraoke"
WORKER_DIR="/opt/encoding-worker"
VENV="${WORKER_DIR}/venv"
TMP_DIR="/tmp/encoding-worker-startup"
export GOOGLE_APPLICATION_CREDENTIALS="${WORKER_DIR}/gcp-wif.json"
exec > >(tee -a /var/log/encoding-worker-startup.log) 2>&1
echo "=== AWS encoding worker startup: $(date) ==="
rm -rf "${TMP_DIR}" && mkdir -p "${TMP_DIR}"
echo "[1/5] Fetching API key from Secret Manager (WIF)..."
ENCODING_API_KEY=""
for attempt in $(seq 1 10); do
ENCODING_API_KEY=$(gcloud secrets versions access latest --secret=encoding-worker-api-key --project=nomadkaraoke 2>/dev/null || echo "")
[ -n "$ENCODING_API_KEY" ] && break
echo "API key fetch failed (attempt ${attempt}/10), retrying in 5s..."
sleep 5
done
if [ -z "$ENCODING_API_KEY" ]; then
echo "FATAL: No API key from Secret Manager; exiting so systemd retries"
exit 1
fi
VM_NAME=$(cat "${WORKER_DIR}/vm-name" 2>/dev/null || hostname)
cat > "${WORKER_DIR}/env" <<ENV
ENCODING_API_KEY=${ENCODING_API_KEY}
ENCODING_WORKER_VM_NAME=${VM_NAME}
ENCODING_WORKER_PROVIDER=aws
GOOGLE_APPLICATION_CREDENTIALS=${WORKER_DIR}/gcp-wif.json
ENV
chmod 600 "${WORKER_DIR}/env"
# Keep the idle watchdog current with the deployed code (best effort).
gcloud storage cp "${BUCKET}/encoding-worker/idle-watchdog.py" "${WORKER_DIR}/idle-watchdog.py" || true
echo "[2/5] Downloading version manifest..."
EXPECTED_VERSION=$(gcloud storage cat "${BUCKET}/encoding-worker/version.txt" 2>/dev/null | tr -d '[:space:]' || echo "")
echo "Expected version: ${EXPECTED_VERSION:-unknown}"
echo "[3/5] Downloading current wheel..."
WHEEL_PATH="${TMP_DIR}/karaoke_gen-${EXPECTED_VERSION:-0.0.0}-py3-none-any.whl"
gcloud storage cp "${BUCKET}/wheels/karaoke_gen-current.whl" "${WHEEL_PATH}"
echo "[4/5] Installing wheel (code only; deps are baked by provision.sh)..."
"${VENV}/bin/pip" install --upgrade --quiet --force-reinstall --no-deps "${WHEEL_PATH}"
echo "[5/5] Verifying installed version..."
INSTALLED_VERSION=$("${VENV}/bin/pip" show karaoke-gen 2>/dev/null | grep "^Version:" | cut -d' ' -f2 | tr -d '[:space:]')
if [ -n "$EXPECTED_VERSION" ] && [ "${INSTALLED_VERSION}" != "${EXPECTED_VERSION}" ]; then
echo "FATAL: version mismatch (expected ${EXPECTED_VERSION}, installed ${INSTALLED_VERSION})"
exit 1
fi
rm -rf "${TMP_DIR}"
echo "=== Startup complete: karaoke-gen v${INSTALLED_VERSION} on ${VM_NAME} ==="