mirror of
https://github.com/nomadkaraoke/karaoke-gen.git
synced 2026-11-20 13:07:57 +09:00
## Summary
Phase 1 of moving encoding to AWS. EC2 is paid for by the AWS Activate
credits (acct 790139457751, expiring 2028-10-31).
- **Plan:** `docs/archive/2026-10-29-aws-encoding-worker-plan.md`.
- **Benchmark** (3 real prod jobs replayed): an EC2 c7a.4xlarge matched
or beat GCE c4d-highcpu-16.
The EC2 box runs the **unchanged** `gce_encoding` worker service and
joins the existing candidate pool as `provider: aws`. The existing
failover, active override, lost-job detection and stockout cooldowns all
apply to it as they are.
## Changes
- **`backend/services/aws_encoding_compute.py` (new):**
`AwsComputeAdapter` looks to the manager like the
`compute_v1.InstancesClient` methods it uses.
- EC2 states map to GCE statuses.
- EC2 capacity errors are added to `CAPACITY_ERROR_CODES`.
- A missing instance raises `NotFound`.
- Controller credentials come from Secret Manager
`aws-encoding-controller-credentials`.
- **`EncodingWorkerManager`:**
- `_compute_for(vm)` sends AWS VMs to the adapter; the GCE paths are
untouched.
- A shared `parse_fallback_vms` passes `provider`/`instance_id`/`region`
through.
- `ranked_candidates()` is used by both the encode path and the warmups.
- `ensure_preferred_running()` behaves **exactly like
`ensure_primary_running` whenever the GCE primary ranks first**, which
is today's case.
- An AWS control-plane fault fails over to the GCE candidates.
- **Ranking:** `SPEED_RANK["c7a"] = 90`, so AWS is the last resort by
default. Firestore `config/encoding-worker.speed_rank_overrides =
{"c7a": 5}` makes it preferred, with no deploy.
- **Isolation from GCE-only tooling:**
- The override now records `active_override_provider`.
- The CI blue-green deploy, `deploy_promote` and the idle Cloud Function
all ignore the AWS worker; it hot-updates its wheel per job.
- A GCE promotion clears any stale provider tag.
- **Worker:**
- Uses `gcloud storage` on AWS, because gsutil can't use WIF credential
files. GCE keeps gsutil.
- `infrastructure/encoding-worker/aws/` adds:
- `provision.sh`, as EC2 user-data;
- `startup-aws.sh`, uploaded by CI;
- `idle-watchdog.py`, an idle self-stop that runs as its own systemd
timer.
- **Infra (GCP), already applied with `/gcp-admin`:**
- The WIF pool `aws-encoding`, plus an AWS provider that only admits the
`nomad-encoding-worker` role, federated to `encoding-worker@`.
- The secret `aws-encoding-controller-credentials`.
- Preview and apply: 4 creates, 288 unchanged.
- **Infra (AWS):** a new Pulumi project `infrastructure/aws` (stack
`nomadkaraoke/prod`):
- the c7a.8xlarge **persistent Spot** worker, which stops on
interruption or shutdown;
- an Elastic IP;
- a security group on 8080 (API key, same as the GCE firewall);
- the worker role with SSM;
- a tag-scoped `nomad-encoding-controller` user.
- Preview: 10 creates. **I apply it after merge**, because the box
provisions from the deployed wheel and boot scripts.
- `boto3` is now an explicit dependency.
- CI also runs the `encoding_worker_idle` function tests, in the job
renamed "Infra - Cloud Function tests".
## Rollout (after merge)
1. **Stage A:** `pulumi up` the AWS stack. Store the controller key in
the secret, then add the worker's entry to
`encoding-worker-fallback-vms`. It's inert unless both c4d and c2d stock
out.
2. **Stage B:** a supervised `speed_rank_overrides {"c7a": 5}` window.
Run real jobs, then compare timings and outputs, including a visual
check of the portrait video.
3. **Stage C:** keep it preferred. After a clean week, delete the c4d
pair.
Phase 2, running distribution (Dropbox/YouTube/Drive uploads) from AWS,
follows in a separate PR.
## Testing
- **New tests:**
- adapter: botocore `Stubber`, covering the state map, capacity and
NotFound handling;
- manager routing and ranking with overrides;
- `ensure_preferred_running` branches;
- failover on an AWS control-plane fault;
- provider tagging;
- deploy_promote exclusions;
- the idle function skipping AWS;
- watchdog decisions;
- the gcs CLI choice.
- **Updated** the candidate and warmup-route tests to the new seams.
- **Full backend + unit suites** have no new failures. The local
emulator and cross-suite failures are identical on main.
- **Local CodeRabbit:** 2 findings, both fixed (failover on AWS faults;
stale provider tag).
@coderabbitai ignore
🤖 Generated with [Claude Code](https://claude.com/claude-code)
https://claude.ai/code/session_012RYLVSR5qgwsxTQtvGimKF
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
204 lines
8.7 KiB
Python
204 lines
8.7 KiB
Python
"""Shared, pure preference logic for encoding-worker candidate ordering.
|
||
|
||
This is the SINGLE SOURCE OF TRUTH for how encode-worker VM candidates are ranked,
|
||
used by BOTH:
|
||
|
||
* runtime worker selection —
|
||
``backend/services/encoding_service._build_worker_candidates`` →
|
||
``encoding_worker_manager.ensure_any_running``
|
||
* deploy green (staging) selection —
|
||
``infrastructure/encoding-worker/deploy_promote.select_green_candidates``
|
||
|
||
Keeping the ordering in one place stops the two call-sites from drifting (they
|
||
previously used different ad-hoc heuristics: runtime = secret order, deploy =
|
||
"n2 before c4d").
|
||
|
||
Design:
|
||
* **Fastest-first base order.** Candidates sort by ``SPEED_RANK`` (lower =
|
||
faster), so the fastest available type (c4d) is always preferred when it can
|
||
start — the whole reason c4d was chosen. No behaviour change on a healthy day.
|
||
* **Availability-aware cooldown.** A type that recently failed to start with a
|
||
capacity/stockout error (recorded in Firestore ``capacity_state``) is
|
||
*demoted to the back* for ``COOLDOWN_SECONDS`` — long enough to stop paying
|
||
the ~2-min-per-attempt probe cost on a known-dead type, short enough to
|
||
re-probe and snap back to the faster type the moment capacity returns. It is
|
||
demoted, never dropped, so the pool always stays fully available.
|
||
|
||
This module is intentionally **dependency-free (stdlib only)** so that
|
||
``deploy_promote`` — which is loaded by path in CI with no backend package deps —
|
||
can import it without pulling the heavy backend tree.
|
||
"""
|
||
from __future__ import annotations
|
||
|
||
from datetime import datetime, timezone
|
||
from typing import Any, Dict, List, Optional
|
||
|
||
# Lower rank = faster = preferred. Keyed by machine FAMILY (the prefix before the
|
||
# first "-"), so the ranking is independent of the vCPU size — the fleet moved
|
||
# from *-highcpu-32 to *-highcpu-16 on 2026-09-26 and could move again. Seeded
|
||
# from CPU architecture; refine with the measured medians from
|
||
# infrastructure/encoding-worker/benchmark_types.py.
|
||
# c4d AMD EPYC Turin (Zen5) — Spot blue-green primary pair, fastest
|
||
# c4 Intel Emerald Rapids — newest Intel perf tier
|
||
# n4d AMD (Titanium) — flexible/deep AMD
|
||
# c2d AMD Milan (Zen3) — mature, deep pool (current on-demand fallback)
|
||
# n2d AMD Rome/Milan — deep pool
|
||
# n2 Intel Cascade/Ice Lake — deep pool
|
||
SPEED_RANK: Dict[str, int] = {
|
||
"c4d": 10,
|
||
"c4": 20,
|
||
"n4d": 30,
|
||
"c2d": 50,
|
||
"n2d": 60,
|
||
"n2": 70,
|
||
# AWS EC2 c7a (AMD Genoa, us-east-2) — the AWS encoding worker
|
||
# (provider "aws"; machine_type "c7a-8xlarge", "." → "-" for Firestore-safe
|
||
# cooldown keys). Last resort by default; make it preferred at runtime with
|
||
# config/encoding-worker.speed_rank_overrides = {"c7a": 5}.
|
||
"c7a": 90,
|
||
}
|
||
|
||
# Rank for a candidate whose machine type is unknown and cannot be inferred:
|
||
# treat as slowest-but-usable so it sorts LAST without ever being dropped.
|
||
UNKNOWN_RANK = 1000
|
||
|
||
# How long a type that just stocked out is demoted before it is re-probed.
|
||
COOLDOWN_SECONDS = 900 # 15 minutes
|
||
|
||
# Machine type of the blue-green primary/secondary pair (encoding-worker-a/-b).
|
||
# They are always c4d (Spot); used when a candidate carries no explicit
|
||
# machine_type. Keep in sync with infrastructure/config.py MachineTypes.ENCODING_WORKER.
|
||
PRIMARY_MACHINE_TYPE = "c4d-highcpu-16"
|
||
|
||
# Current vCPU size of the fleet — used only to turn a family inferred from a VM
|
||
# name into a full machine type (for display + cooldown keys).
|
||
_FLEET_SIZE_SUFFIX = "-highcpu-16"
|
||
|
||
# vm-name substring → machine family, for backward-compat inference when a
|
||
# fallback entry predates the machine_type field. Checked in order — longer /
|
||
# more-specific family tokens FIRST so "n2d" is not shadowed by "n2", nor "c4d"
|
||
# by "c4".
|
||
_NAME_TOKEN_TO_FAMILY = ("c4d", "n4d", "n2d", "c2d", "c4", "n2")
|
||
|
||
|
||
def machine_family(machine_type: Optional[str]) -> Optional[str]:
|
||
"""``"c4d-highcpu-16"`` → ``"c4d"``; None/empty → None."""
|
||
if not machine_type:
|
||
return None
|
||
return machine_type.split("-", 1)[0].lower()
|
||
|
||
|
||
def _candidate_name(candidate: Dict[str, Any]) -> str:
|
||
"""VM name from either dict shape ({"vm"} for deploy, {"vm_name"} for runtime)."""
|
||
return (candidate.get("vm") or candidate.get("vm_name") or "")
|
||
|
||
|
||
def infer_machine_type(candidate: Dict[str, Any]) -> Optional[str]:
|
||
"""Best-effort machine type: explicit ``machine_type`` field, else vm-name token.
|
||
|
||
Fallback secret entries created before the ``machine_type`` field carry only
|
||
vm/zone/ip; the VM-name family token (…-fallback-c2df) is the only hint and
|
||
is expanded to the current fleet size. Returns None when nothing matches
|
||
(caller treats as UNKNOWN_RANK).
|
||
"""
|
||
explicit = candidate.get("machine_type")
|
||
if explicit:
|
||
return explicit
|
||
name = _candidate_name(candidate).lower()
|
||
for family in _NAME_TOKEN_TO_FAMILY:
|
||
if family in name:
|
||
return f"{family}{_FLEET_SIZE_SUFFIX}"
|
||
return None
|
||
|
||
|
||
def _speed_rank(candidate: Dict[str, Any], rank_overrides: Optional[Dict[str, Any]] = None) -> int:
|
||
family = machine_family(infer_machine_type(candidate))
|
||
if rank_overrides and family in rank_overrides:
|
||
try:
|
||
return int(rank_overrides[family])
|
||
except (TypeError, ValueError):
|
||
pass # a malformed override must never break worker selection
|
||
return SPEED_RANK.get(family, UNKNOWN_RANK)
|
||
|
||
|
||
def cooldown_key(candidate: Dict[str, Any]) -> Optional[str]:
|
||
"""Firestore ``capacity_state`` key for a candidate: ``"<machine_type>@<zone>"``.
|
||
|
||
Zone-scoped because a stockout is per (machine family × zone). Returns None
|
||
when the machine type can't be determined (no cooldown tracking possible).
|
||
NOTE: keys deliberately avoid ``.`` so they are safe as Firestore map keys.
|
||
"""
|
||
mtype = infer_machine_type(candidate)
|
||
if not mtype:
|
||
return None
|
||
zone = candidate.get("zone")
|
||
return f"{mtype}@{zone}" if zone else mtype
|
||
|
||
|
||
def _parse_iso(value: Any) -> Optional[datetime]:
|
||
if not value:
|
||
return None
|
||
try:
|
||
dt = datetime.fromisoformat(value)
|
||
except (ValueError, TypeError):
|
||
return None
|
||
if dt.tzinfo is None: # tolerate naive timestamps — assume UTC
|
||
dt = dt.replace(tzinfo=timezone.utc)
|
||
return dt
|
||
|
||
|
||
def _recently_stocked_out(
|
||
candidate: Dict[str, Any],
|
||
capacity_state: Optional[Dict[str, Any]],
|
||
now: datetime,
|
||
cooldown_seconds: float,
|
||
) -> bool:
|
||
"""True if this candidate's (type@zone) — or its coarser type-only key — is in cooldown."""
|
||
if not capacity_state:
|
||
return False
|
||
key = cooldown_key(candidate)
|
||
if not key:
|
||
return False
|
||
ts = _parse_iso(capacity_state.get(key))
|
||
if ts is None:
|
||
# Fall back to a type-only signal (a coarser "this family is short").
|
||
mtype = infer_machine_type(candidate)
|
||
if mtype:
|
||
ts = _parse_iso(capacity_state.get(mtype))
|
||
if ts is None:
|
||
return False
|
||
return (now - ts).total_seconds() < cooldown_seconds
|
||
|
||
|
||
def ordered_candidates(
|
||
pool: List[Dict[str, Any]],
|
||
capacity_state: Optional[Dict[str, Any]] = None,
|
||
now: Optional[datetime] = None,
|
||
cooldown_seconds: float = COOLDOWN_SECONDS,
|
||
rank_overrides: Optional[Dict[str, Any]] = None,
|
||
) -> List[Dict[str, Any]]:
|
||
"""Rank ``pool`` fastest-first, demoting recently-stocked-out types to the back.
|
||
|
||
Args:
|
||
pool: candidate dicts, each ``{vm|vm_name, zone, ip, machine_type?, kind?}``.
|
||
Objects are returned as-is (not copied), only reordered — so callers can
|
||
round-trip their own richer objects through a dict view.
|
||
capacity_state: Firestore map ``{"<type>@<zone>"|"<type>": last_stockout_iso}``.
|
||
Absent/empty ⇒ pure fastest-first order (today's behaviour).
|
||
now: current UTC time (injectable for tests); defaults to ``datetime.now(UTC)``.
|
||
cooldown_seconds: demotion window; defaults to ``COOLDOWN_SECONDS``.
|
||
rank_overrides: ``{family: rank}`` from Firestore
|
||
``config/encoding-worker.speed_rank_overrides``; replaces SPEED_RANK
|
||
for those families (e.g. ``{"c7a": 5}`` prefers the AWS worker).
|
||
|
||
The speed sort is STABLE, so equal-rank candidates keep their input order —
|
||
callers express a secondary preference (e.g. zone spread) purely via input
|
||
ordering. The hot/cold partition is likewise stable within each group.
|
||
"""
|
||
if now is None:
|
||
now = datetime.now(timezone.utc)
|
||
ranked = sorted(pool, key=lambda c: _speed_rank(c, rank_overrides)) # stable: fastest first
|
||
hot = [c for c in ranked if not _recently_stocked_out(c, capacity_state, now, cooldown_seconds)]
|
||
cold = [c for c in ranked if _recently_stocked_out(c, capacity_state, now, cooldown_seconds)]
|
||
return hot + cold
|