Files
karaoke-gen/backend/services/aws_encoding_compute.py
T
Andrew Beveridge ☄️andClaude Opus 5.5 de5dfcd50d feat(encoding): AWS EC2 encoding worker as a pool candidate (v0.288.0) (#1236)
## Summary
Phase 1 of moving encoding to AWS. EC2 is paid for by the AWS Activate
credits (acct 790139457751, expiring 2028-10-31).
- **Plan:** `docs/archive/2026-10-29-aws-encoding-worker-plan.md`.
- **Benchmark** (3 real prod jobs replayed): an EC2 c7a.4xlarge matched
or beat GCE c4d-highcpu-16.

The EC2 box runs the **unchanged** `gce_encoding` worker service and
joins the existing candidate pool as `provider: aws`. The existing
failover, active override, lost-job detection and stockout cooldowns all
apply to it as they are.

## Changes
- **`backend/services/aws_encoding_compute.py` (new):**
`AwsComputeAdapter` looks to the manager like the
`compute_v1.InstancesClient` methods it uses.
  - EC2 states map to GCE statuses.
  - EC2 capacity errors are added to `CAPACITY_ERROR_CODES`.
  - A missing instance raises `NotFound`.
- Controller credentials come from Secret Manager
`aws-encoding-controller-credentials`.
- **`EncodingWorkerManager`:**
- `_compute_for(vm)` sends AWS VMs to the adapter; the GCE paths are
untouched.
- A shared `parse_fallback_vms` passes `provider`/`instance_id`/`region`
through.
- `ranked_candidates()` is used by both the encode path and the warmups.
- `ensure_preferred_running()` behaves **exactly like
`ensure_primary_running` whenever the GCE primary ranks first**, which
is today's case.
  - An AWS control-plane fault fails over to the GCE candidates.
- **Ranking:** `SPEED_RANK["c7a"] = 90`, so AWS is the last resort by
default. Firestore `config/encoding-worker.speed_rank_overrides =
{"c7a": 5}` makes it preferred, with no deploy.
- **Isolation from GCE-only tooling:**
  - The override now records `active_override_provider`.
- The CI blue-green deploy, `deploy_promote` and the idle Cloud Function
all ignore the AWS worker; it hot-updates its wheel per job.
  - A GCE promotion clears any stale provider tag.
- **Worker:**
- Uses `gcloud storage` on AWS, because gsutil can't use WIF credential
files. GCE keeps gsutil.
  - `infrastructure/encoding-worker/aws/` adds:
    - `provision.sh`, as EC2 user-data;
    - `startup-aws.sh`, uploaded by CI;
- `idle-watchdog.py`, an idle self-stop that runs as its own systemd
timer.
- **Infra (GCP), already applied with `/gcp-admin`:**
- The WIF pool `aws-encoding`, plus an AWS provider that only admits the
`nomad-encoding-worker` role, federated to `encoding-worker@`.
  - The secret `aws-encoding-controller-credentials`.
  - Preview and apply: 4 creates, 288 unchanged.
- **Infra (AWS):** a new Pulumi project `infrastructure/aws` (stack
`nomadkaraoke/prod`):
- the c7a.8xlarge **persistent Spot** worker, which stops on
interruption or shutdown;
  - an Elastic IP;
  - a security group on 8080 (API key, same as the GCE firewall);
  - the worker role with SSM;
  - a tag-scoped `nomad-encoding-controller` user.
- Preview: 10 creates. **I apply it after merge**, because the box
provisions from the deployed wheel and boot scripts.
- `boto3` is now an explicit dependency.
- CI also runs the `encoding_worker_idle` function tests, in the job
renamed "Infra - Cloud Function tests".

## Rollout (after merge)
1. **Stage A:** `pulumi up` the AWS stack. Store the controller key in
the secret, then add the worker's entry to
`encoding-worker-fallback-vms`. It's inert unless both c4d and c2d stock
out.
2. **Stage B:** a supervised `speed_rank_overrides {"c7a": 5}` window.
Run real jobs, then compare timings and outputs, including a visual
check of the portrait video.
3. **Stage C:** keep it preferred. After a clean week, delete the c4d
pair.

Phase 2, running distribution (Dropbox/YouTube/Drive uploads) from AWS,
follows in a separate PR.

## Testing
- **New tests:**
- adapter: botocore `Stubber`, covering the state map, capacity and
NotFound handling;
  - manager routing and ranking with overrides;
  - `ensure_preferred_running` branches;
  - failover on an AWS control-plane fault;
  - provider tagging;
  - deploy_promote exclusions;
  - the idle function skipping AWS;
  - watchdog decisions;
  - the gcs CLI choice.
- **Updated** the candidate and warmup-route tests to the new seams.
- **Full backend + unit suites** have no new failures. The local
emulator and cross-suite failures are identical on main.
- **Local CodeRabbit:** 2 findings, both fixed (failover on AWS faults;
stale provider tag).

@coderabbitai ignore

🤖 Generated with [Claude Code](https://claude.com/claude-code)

https://claude.ai/code/session_012RYLVSR5qgwsxTQtvGimKF

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-29 17:42:35 -04:00

168 lines
6.4 KiB
Python

"""EC2 adapter for the AWS encoding worker.
The AWS worker (an EC2 Spot instance running the same ``gce_encoding`` HTTP
service) is just another candidate in the encoding-worker pool. Rather than
teach every caller about EC2, this module mimics the slice of
``google.cloud.compute_v1.InstancesClient`` that ``EncodingWorkerManager`` uses
(``get`` / ``start`` / ``stop``), so the manager routes AWS VMs here and the
GCE code paths stay untouched:
- EC2 states map to GCE statuses (``running`` → ``RUNNING``, ``stopped`` →
``TERMINATED``, …) so the existing RUNNING/STAGING/stopped checks just work.
- ``start`` returns an operation object shaped like a compute ExtendedOperation
(``result()``, ``error_code``, ``error_message``); EC2 capacity errors surface
as codes in ``CAPACITY_ERROR_CODES`` so stockout cooldowns apply unchanged.
- A missing instance raises ``google.api_core.exceptions.NotFound``, which the
manager already treats as "skip this candidate".
Credentials: an IAM user limited to Start/Stop/Describe on instances tagged
``Project=nomad-encoding``, stored as JSON ``{"access_key_id", "secret_access_key"}``
in Secret Manager ``aws-encoding-controller-credentials`` (env
``AWS_ENCODING_CONTROLLER_CREDENTIALS`` overrides, for tests/local runs).
"""
from __future__ import annotations
import json
import logging
import os
import threading
from dataclasses import dataclass
from typing import Any, Dict, Optional
from google.api_core import exceptions as gcp_exceptions
logger = logging.getLogger(__name__)
CREDENTIALS_SECRET = "aws-encoding-controller-credentials"
CREDENTIALS_ENV = "AWS_ENCODING_CONTROLLER_CREDENTIALS"
# EC2 instance state → GCE instance status.
_EC2_TO_GCE_STATUS = {
"pending": "STAGING",
"running": "RUNNING",
"stopping": "STOPPING",
"stopped": "TERMINATED",
"shutting-down": "STOPPING",
"terminated": "TERMINATED",
}
@dataclass(frozen=True)
class AwsVm:
"""Where an AWS worker lives. Built from a fallback-pool entry."""
instance_id: str
region: str
@dataclass
class _Instance:
status: str
class AwsStartOperation:
"""Duck-types a compute ExtendedOperation for ``_wait_for_compute_operation``."""
def __init__(self, run):
self._run = run
self.error_code = ""
self.error_message = ""
def result(self, timeout: Optional[float] = None) -> None:
from botocore.exceptions import ClientError
try:
self._run()
except ClientError as e:
err = e.response.get("Error", {})
self.error_code = err.get("Code", "") or "ClientError"
self.error_message = err.get("Message", "") or str(e)
def _load_credentials(project_id: str) -> Dict[str, str]:
raw = os.environ.get(CREDENTIALS_ENV)
if not raw:
from google.cloud import secretmanager
client = secretmanager.SecretManagerServiceClient()
name = f"projects/{project_id}/secrets/{CREDENTIALS_SECRET}/versions/latest"
raw = client.access_secret_version(name=name).payload.data.decode("utf-8")
creds = json.loads(raw)
return {"access_key_id": creds["access_key_id"], "secret_access_key": creds["secret_access_key"]}
class AwsComputeAdapter:
"""``get``/``start``/``stop`` for registered AWS worker VMs, by VM name."""
def __init__(self, vms: Dict[str, AwsVm], project_id: str = "nomadkaraoke", client_factory=None):
self._vms = dict(vms)
self._project_id = project_id
self._client_factory = client_factory
self._clients: Dict[str, Any] = {}
self._lock = threading.Lock()
def handles(self, vm_name: str) -> bool:
return vm_name in self._vms
def _client(self, region: str):
with self._lock:
if region not in self._clients:
if self._client_factory is not None:
self._clients[region] = self._client_factory(region)
else:
import boto3
creds = _load_credentials(self._project_id)
self._clients[region] = boto3.client(
"ec2",
region_name=region,
aws_access_key_id=creds["access_key_id"],
aws_secret_access_key=creds["secret_access_key"],
)
return self._clients[region]
def _vm(self, instance: str) -> AwsVm:
vm = self._vms.get(instance)
if vm is None:
raise gcp_exceptions.NotFound(f"AWS worker {instance} is not registered")
return vm
def get(self, project: str = "", zone: str = "", instance: str = "") -> _Instance:
from botocore.exceptions import ClientError
vm = self._vm(instance)
try:
resp = self._client(vm.region).describe_instances(InstanceIds=[vm.instance_id])
except ClientError as e:
if e.response.get("Error", {}).get("Code", "").startswith("InvalidInstanceID"):
raise gcp_exceptions.NotFound(f"AWS worker {instance} ({vm.instance_id}) not found") from e
raise
reservations = resp.get("Reservations") or []
if not reservations or not reservations[0].get("Instances"):
raise gcp_exceptions.NotFound(f"AWS worker {instance} ({vm.instance_id}) not found")
state = reservations[0]["Instances"][0]["State"]["Name"]
return _Instance(status=_EC2_TO_GCE_STATUS.get(state, state.upper()))
def start(self, project: str = "", zone: str = "", instance: str = "") -> AwsStartOperation:
vm = self._vm(instance)
logger.info("Starting AWS worker %s (%s in %s)", instance, vm.instance_id, vm.region)
return AwsStartOperation(lambda: self._client(vm.region).start_instances(InstanceIds=[vm.instance_id]))
def stop(self, project: str = "", zone: str = "", instance: str = "") -> None:
vm = self._vm(instance)
logger.info("Stopping AWS worker %s (%s in %s)", instance, vm.instance_id, vm.region)
self._client(vm.region).stop_instances(InstanceIds=[vm.instance_id])
def aws_vms_from_pool(entries) -> Dict[str, AwsVm]:
"""``{vm_name: AwsVm}`` for the ``provider == "aws"`` entries of a fallback pool."""
vms: Dict[str, AwsVm] = {}
for e in entries or []:
if (e.get("provider") or "").lower() != "aws":
continue
name = e.get("vm") or e.get("vm_name")
if name and e.get("instance_id") and e.get("region"):
vms[name] = AwsVm(instance_id=e["instance_id"], region=e["region"])
return vms