Files
karaoke-gen/backend/services/encoding_worker_preference.py
T
Andrew Beveridge ☄️andClaude Opus 5.5 de5dfcd50d feat(encoding): AWS EC2 encoding worker as a pool candidate (v0.288.0) (#1236)
## Summary
Phase 1 of moving encoding to AWS. EC2 is paid for by the AWS Activate
credits (acct 790139457751, expiring 2028-10-31).
- **Plan:** `docs/archive/2026-10-29-aws-encoding-worker-plan.md`.
- **Benchmark** (3 real prod jobs replayed): an EC2 c7a.4xlarge matched
or beat GCE c4d-highcpu-16.

The EC2 box runs the **unchanged** `gce_encoding` worker service and
joins the existing candidate pool as `provider: aws`. The existing
failover, active override, lost-job detection and stockout cooldowns all
apply to it as they are.

## Changes
- **`backend/services/aws_encoding_compute.py` (new):**
`AwsComputeAdapter` looks to the manager like the
`compute_v1.InstancesClient` methods it uses.
  - EC2 states map to GCE statuses.
  - EC2 capacity errors are added to `CAPACITY_ERROR_CODES`.
  - A missing instance raises `NotFound`.
- Controller credentials come from Secret Manager
`aws-encoding-controller-credentials`.
- **`EncodingWorkerManager`:**
- `_compute_for(vm)` sends AWS VMs to the adapter; the GCE paths are
untouched.
- A shared `parse_fallback_vms` passes `provider`/`instance_id`/`region`
through.
- `ranked_candidates()` is used by both the encode path and the warmups.
- `ensure_preferred_running()` behaves **exactly like
`ensure_primary_running` whenever the GCE primary ranks first**, which
is today's case.
  - An AWS control-plane fault fails over to the GCE candidates.
- **Ranking:** `SPEED_RANK["c7a"] = 90`, so AWS is the last resort by
default. Firestore `config/encoding-worker.speed_rank_overrides =
{"c7a": 5}` makes it preferred, with no deploy.
- **Isolation from GCE-only tooling:**
  - The override now records `active_override_provider`.
- The CI blue-green deploy, `deploy_promote` and the idle Cloud Function
all ignore the AWS worker; it hot-updates its wheel per job.
  - A GCE promotion clears any stale provider tag.
- **Worker:**
- Uses `gcloud storage` on AWS, because gsutil can't use WIF credential
files. GCE keeps gsutil.
  - `infrastructure/encoding-worker/aws/` adds:
    - `provision.sh`, as EC2 user-data;
    - `startup-aws.sh`, uploaded by CI;
- `idle-watchdog.py`, an idle self-stop that runs as its own systemd
timer.
- **Infra (GCP), already applied with `/gcp-admin`:**
- The WIF pool `aws-encoding`, plus an AWS provider that only admits the
`nomad-encoding-worker` role, federated to `encoding-worker@`.
  - The secret `aws-encoding-controller-credentials`.
  - Preview and apply: 4 creates, 288 unchanged.
- **Infra (AWS):** a new Pulumi project `infrastructure/aws` (stack
`nomadkaraoke/prod`):
- the c7a.8xlarge **persistent Spot** worker, which stops on
interruption or shutdown;
  - an Elastic IP;
  - a security group on 8080 (API key, same as the GCE firewall);
  - the worker role with SSM;
  - a tag-scoped `nomad-encoding-controller` user.
- Preview: 10 creates. **I apply it after merge**, because the box
provisions from the deployed wheel and boot scripts.
- `boto3` is now an explicit dependency.
- CI also runs the `encoding_worker_idle` function tests, in the job
renamed "Infra - Cloud Function tests".

## Rollout (after merge)
1. **Stage A:** `pulumi up` the AWS stack. Store the controller key in
the secret, then add the worker's entry to
`encoding-worker-fallback-vms`. It's inert unless both c4d and c2d stock
out.
2. **Stage B:** a supervised `speed_rank_overrides {"c7a": 5}` window.
Run real jobs, then compare timings and outputs, including a visual
check of the portrait video.
3. **Stage C:** keep it preferred. After a clean week, delete the c4d
pair.

Phase 2, running distribution (Dropbox/YouTube/Drive uploads) from AWS,
follows in a separate PR.

## Testing
- **New tests:**
- adapter: botocore `Stubber`, covering the state map, capacity and
NotFound handling;
  - manager routing and ranking with overrides;
  - `ensure_preferred_running` branches;
  - failover on an AWS control-plane fault;
  - provider tagging;
  - deploy_promote exclusions;
  - the idle function skipping AWS;
  - watchdog decisions;
  - the gcs CLI choice.
- **Updated** the candidate and warmup-route tests to the new seams.
- **Full backend + unit suites** have no new failures. The local
emulator and cross-suite failures are identical on main.
- **Local CodeRabbit:** 2 findings, both fixed (failover on AWS faults;
stale provider tag).

@coderabbitai ignore

🤖 Generated with [Claude Code](https://claude.com/claude-code)

https://claude.ai/code/session_012RYLVSR5qgwsxTQtvGimKF

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-29 17:42:35 -04:00

204 lines
8.7 KiB
Python
Raw Blame History

This file contains ambiguous Unicode characters
This file contains Unicode characters that might be confused with other characters. If you think that this is intentional, you can safely ignore this warning. Use the Escape button to reveal them.
"""Shared, pure preference logic for encoding-worker candidate ordering.
This is the SINGLE SOURCE OF TRUTH for how encode-worker VM candidates are ranked,
used by BOTH:
* runtime worker selection —
``backend/services/encoding_service._build_worker_candidates`` →
``encoding_worker_manager.ensure_any_running``
* deploy green (staging) selection —
``infrastructure/encoding-worker/deploy_promote.select_green_candidates``
Keeping the ordering in one place stops the two call-sites from drifting (they
previously used different ad-hoc heuristics: runtime = secret order, deploy =
"n2 before c4d").
Design:
* **Fastest-first base order.** Candidates sort by ``SPEED_RANK`` (lower =
faster), so the fastest available type (c4d) is always preferred when it can
start — the whole reason c4d was chosen. No behaviour change on a healthy day.
* **Availability-aware cooldown.** A type that recently failed to start with a
capacity/stockout error (recorded in Firestore ``capacity_state``) is
*demoted to the back* for ``COOLDOWN_SECONDS`` — long enough to stop paying
the ~2-min-per-attempt probe cost on a known-dead type, short enough to
re-probe and snap back to the faster type the moment capacity returns. It is
demoted, never dropped, so the pool always stays fully available.
This module is intentionally **dependency-free (stdlib only)** so that
``deploy_promote`` — which is loaded by path in CI with no backend package deps —
can import it without pulling the heavy backend tree.
"""
from __future__ import annotations
from datetime import datetime, timezone
from typing import Any, Dict, List, Optional
# Lower rank = faster = preferred. Keyed by machine FAMILY (the prefix before the
# first "-"), so the ranking is independent of the vCPU size — the fleet moved
# from *-highcpu-32 to *-highcpu-16 on 2026-09-26 and could move again. Seeded
# from CPU architecture; refine with the measured medians from
# infrastructure/encoding-worker/benchmark_types.py.
# c4d AMD EPYC Turin (Zen5) — Spot blue-green primary pair, fastest
# c4 Intel Emerald Rapids — newest Intel perf tier
# n4d AMD (Titanium) — flexible/deep AMD
# c2d AMD Milan (Zen3) — mature, deep pool (current on-demand fallback)
# n2d AMD Rome/Milan — deep pool
# n2 Intel Cascade/Ice Lake — deep pool
SPEED_RANK: Dict[str, int] = {
"c4d": 10,
"c4": 20,
"n4d": 30,
"c2d": 50,
"n2d": 60,
"n2": 70,
# AWS EC2 c7a (AMD Genoa, us-east-2) — the AWS encoding worker
# (provider "aws"; machine_type "c7a-8xlarge", "." → "-" for Firestore-safe
# cooldown keys). Last resort by default; make it preferred at runtime with
# config/encoding-worker.speed_rank_overrides = {"c7a": 5}.
"c7a": 90,
}
# Rank for a candidate whose machine type is unknown and cannot be inferred:
# treat as slowest-but-usable so it sorts LAST without ever being dropped.
UNKNOWN_RANK = 1000
# How long a type that just stocked out is demoted before it is re-probed.
COOLDOWN_SECONDS = 900 # 15 minutes
# Machine type of the blue-green primary/secondary pair (encoding-worker-a/-b).
# They are always c4d (Spot); used when a candidate carries no explicit
# machine_type. Keep in sync with infrastructure/config.py MachineTypes.ENCODING_WORKER.
PRIMARY_MACHINE_TYPE = "c4d-highcpu-16"
# Current vCPU size of the fleet — used only to turn a family inferred from a VM
# name into a full machine type (for display + cooldown keys).
_FLEET_SIZE_SUFFIX = "-highcpu-16"
# vm-name substring → machine family, for backward-compat inference when a
# fallback entry predates the machine_type field. Checked in order — longer /
# more-specific family tokens FIRST so "n2d" is not shadowed by "n2", nor "c4d"
# by "c4".
_NAME_TOKEN_TO_FAMILY = ("c4d", "n4d", "n2d", "c2d", "c4", "n2")
def machine_family(machine_type: Optional[str]) -> Optional[str]:
"""``"c4d-highcpu-16"`` → ``"c4d"``; None/empty → None."""
if not machine_type:
return None
return machine_type.split("-", 1)[0].lower()
def _candidate_name(candidate: Dict[str, Any]) -> str:
"""VM name from either dict shape ({"vm"} for deploy, {"vm_name"} for runtime)."""
return (candidate.get("vm") or candidate.get("vm_name") or "")
def infer_machine_type(candidate: Dict[str, Any]) -> Optional[str]:
"""Best-effort machine type: explicit ``machine_type`` field, else vm-name token.
Fallback secret entries created before the ``machine_type`` field carry only
vm/zone/ip; the VM-name family token (…-fallback-c2df) is the only hint and
is expanded to the current fleet size. Returns None when nothing matches
(caller treats as UNKNOWN_RANK).
"""
explicit = candidate.get("machine_type")
if explicit:
return explicit
name = _candidate_name(candidate).lower()
for family in _NAME_TOKEN_TO_FAMILY:
if family in name:
return f"{family}{_FLEET_SIZE_SUFFIX}"
return None
def _speed_rank(candidate: Dict[str, Any], rank_overrides: Optional[Dict[str, Any]] = None) -> int:
family = machine_family(infer_machine_type(candidate))
if rank_overrides and family in rank_overrides:
try:
return int(rank_overrides[family])
except (TypeError, ValueError):
pass # a malformed override must never break worker selection
return SPEED_RANK.get(family, UNKNOWN_RANK)
def cooldown_key(candidate: Dict[str, Any]) -> Optional[str]:
"""Firestore ``capacity_state`` key for a candidate: ``"<machine_type>@<zone>"``.
Zone-scoped because a stockout is per (machine family × zone). Returns None
when the machine type can't be determined (no cooldown tracking possible).
NOTE: keys deliberately avoid ``.`` so they are safe as Firestore map keys.
"""
mtype = infer_machine_type(candidate)
if not mtype:
return None
zone = candidate.get("zone")
return f"{mtype}@{zone}" if zone else mtype
def _parse_iso(value: Any) -> Optional[datetime]:
if not value:
return None
try:
dt = datetime.fromisoformat(value)
except (ValueError, TypeError):
return None
if dt.tzinfo is None: # tolerate naive timestamps — assume UTC
dt = dt.replace(tzinfo=timezone.utc)
return dt
def _recently_stocked_out(
candidate: Dict[str, Any],
capacity_state: Optional[Dict[str, Any]],
now: datetime,
cooldown_seconds: float,
) -> bool:
"""True if this candidate's (type@zone) — or its coarser type-only key — is in cooldown."""
if not capacity_state:
return False
key = cooldown_key(candidate)
if not key:
return False
ts = _parse_iso(capacity_state.get(key))
if ts is None:
# Fall back to a type-only signal (a coarser "this family is short").
mtype = infer_machine_type(candidate)
if mtype:
ts = _parse_iso(capacity_state.get(mtype))
if ts is None:
return False
return (now - ts).total_seconds() < cooldown_seconds
def ordered_candidates(
pool: List[Dict[str, Any]],
capacity_state: Optional[Dict[str, Any]] = None,
now: Optional[datetime] = None,
cooldown_seconds: float = COOLDOWN_SECONDS,
rank_overrides: Optional[Dict[str, Any]] = None,
) -> List[Dict[str, Any]]:
"""Rank ``pool`` fastest-first, demoting recently-stocked-out types to the back.
Args:
pool: candidate dicts, each ``{vm|vm_name, zone, ip, machine_type?, kind?}``.
Objects are returned as-is (not copied), only reordered — so callers can
round-trip their own richer objects through a dict view.
capacity_state: Firestore map ``{"<type>@<zone>"|"<type>": last_stockout_iso}``.
Absent/empty ⇒ pure fastest-first order (today's behaviour).
now: current UTC time (injectable for tests); defaults to ``datetime.now(UTC)``.
cooldown_seconds: demotion window; defaults to ``COOLDOWN_SECONDS``.
rank_overrides: ``{family: rank}`` from Firestore
``config/encoding-worker.speed_rank_overrides``; replaces SPEED_RANK
for those families (e.g. ``{"c7a": 5}`` prefers the AWS worker).
The speed sort is STABLE, so equal-rank candidates keep their input order —
callers express a secondary preference (e.g. zone spread) purely via input
ordering. The hot/cold partition is likewise stable within each group.
"""
if now is None:
now = datetime.now(timezone.utc)
ranked = sorted(pool, key=lambda c: _speed_rank(c, rank_overrides)) # stable: fastest first
hot = [c for c in ranked if not _recently_stocked_out(c, capacity_state, now, cooldown_seconds)]
cold = [c for c in ranked if _recently_stocked_out(c, capacity_state, now, cooldown_seconds)]
return hot + cold