Files
karaoke-gen/backend/services/encoding_worker_preference.py
Andrew Beveridge ☄️andClaude Opus 5.5 20b957958c infra(aws): encoding worker on-demand c7a.4xlarge; stage A status (#1239)
Follow-up to #1236.

- **Instance type:** the AWS encoding worker is now **on-demand
c7a.4xlarge** instead of Spot c7a.8xlarge.
- The Spot create failed with `InsufficientInstanceCapacity`, and Spot
placement scores were 1-3/10 for c7a/c7i/c6a/c8a 4-8xlarge across
us-east-1/2.
- A worker that stops and starts on demand has to win capacity on every
start.
- Cost is ~$82/mo of credits. The 4xlarge matched or beat GCE in the
replay benchmark.
- **Already applied** (instance `i-0c21505d7017356bb`, EIP
3.128.70.236).
- **Docs and plan:** stage A status, plus the live end-to-end test
through the worker HTTP API (render 117s; all 9 outputs; portrait SSIM
0.993, the only difference being anti-aliasing of the header text).

The only code-file change is a comment in
`encoding_worker_preference.py`, so there is no version bump.

No local CodeRabbit run (IaC and docs only), so the bot review stays on.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

https://claude.ai/code/session_012RYLVSR5qgwsxTQtvGimKF

<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
<!-- coderabbit-ai-generated:v1 -->

## Summary by CodeRabbit

* **Operational Updates**
* The AWS encoding worker now runs on an on-demand `c7a.4xlarge`
instance rather than a Spot `c7a.8xlarge`.
* The worker stops when the operating system shuts down, while its root
volume is preserved.
* If AWS cannot start the worker because capacity is insufficient, it is
temporarily deprioritized for 15 minutes, allowing processing to fall
back to GCE.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-31 12:16:19 -04:00

204 lines
8.7 KiB
Python
Raw Permalink Blame History

This file contains ambiguous Unicode characters
This file contains Unicode characters that might be confused with other characters. If you think that this is intentional, you can safely ignore this warning. Use the Escape button to reveal them.
"""Shared, pure preference logic for encoding-worker candidate ordering.
This is the SINGLE SOURCE OF TRUTH for how encode-worker VM candidates are ranked,
used by BOTH:
* runtime worker selection —
``backend/services/encoding_service._build_worker_candidates`` →
``encoding_worker_manager.ensure_any_running``
* deploy green (staging) selection —
``infrastructure/encoding-worker/deploy_promote.select_green_candidates``
Keeping the ordering in one place stops the two call-sites from drifting (they
previously used different ad-hoc heuristics: runtime = secret order, deploy =
"n2 before c4d").
Design:
* **Fastest-first base order.** Candidates sort by ``SPEED_RANK`` (lower =
faster), so the fastest available type (c4d) is always preferred when it can
start — the whole reason c4d was chosen. No behaviour change on a healthy day.
* **Availability-aware cooldown.** A type that recently failed to start with a
capacity/stockout error (recorded in Firestore ``capacity_state``) is
*demoted to the back* for ``COOLDOWN_SECONDS`` — long enough to stop paying
the ~2-min-per-attempt probe cost on a known-dead type, short enough to
re-probe and snap back to the faster type the moment capacity returns. It is
demoted, never dropped, so the pool always stays fully available.
This module is intentionally **dependency-free (stdlib only)** so that
``deploy_promote`` — which is loaded by path in CI with no backend package deps —
can import it without pulling the heavy backend tree.
"""
from __future__ import annotations
from datetime import datetime, timezone
from typing import Any, Dict, List, Optional
# Lower rank = faster = preferred. Keyed by machine FAMILY (the prefix before the
# first "-"), so the ranking is independent of the vCPU size — the fleet moved
# from *-highcpu-32 to *-highcpu-16 on 2026-09-26 and could move again. Seeded
# from CPU architecture; refine with the measured medians from
# infrastructure/encoding-worker/benchmark_types.py.
# c4d AMD EPYC Turin (Zen5) — Spot blue-green primary pair, fastest
# c4 Intel Emerald Rapids — newest Intel perf tier
# n4d AMD (Titanium) — flexible/deep AMD
# c2d AMD Milan (Zen3) — mature, deep pool (current on-demand fallback)
# n2d AMD Rome/Milan — deep pool
# n2 Intel Cascade/Ice Lake — deep pool
SPEED_RANK: Dict[str, int] = {
"c4d": 10,
"c4": 20,
"n4d": 30,
"c2d": 50,
"n2d": 60,
"n2": 70,
# AWS EC2 c7a (AMD Genoa, us-east-2) — the AWS encoding worker
# (provider "aws"; machine_type "c7a-4xlarge", "." → "-" for Firestore-safe
# cooldown keys). Last resort by default; make it preferred at runtime with
# config/encoding-worker.speed_rank_overrides = {"c7a": 5}.
"c7a": 90,
}
# Rank for a candidate whose machine type is unknown and cannot be inferred:
# treat as slowest-but-usable so it sorts LAST without ever being dropped.
UNKNOWN_RANK = 1000
# How long a type that just stocked out is demoted before it is re-probed.
COOLDOWN_SECONDS = 900 # 15 minutes
# Machine type of the blue-green primary/secondary pair (encoding-worker-a/-b).
# They are always c4d (Spot); used when a candidate carries no explicit
# machine_type. Keep in sync with infrastructure/config.py MachineTypes.ENCODING_WORKER.
PRIMARY_MACHINE_TYPE = "c4d-highcpu-16"
# Current vCPU size of the fleet — used only to turn a family inferred from a VM
# name into a full machine type (for display + cooldown keys).
_FLEET_SIZE_SUFFIX = "-highcpu-16"
# vm-name substring → machine family, for backward-compat inference when a
# fallback entry predates the machine_type field. Checked in order — longer /
# more-specific family tokens FIRST so "n2d" is not shadowed by "n2", nor "c4d"
# by "c4".
_NAME_TOKEN_TO_FAMILY = ("c4d", "n4d", "n2d", "c2d", "c4", "n2")
def machine_family(machine_type: Optional[str]) -> Optional[str]:
"""``"c4d-highcpu-16"`` → ``"c4d"``; None/empty → None."""
if not machine_type:
return None
return machine_type.split("-", 1)[0].lower()
def _candidate_name(candidate: Dict[str, Any]) -> str:
"""VM name from either dict shape ({"vm"} for deploy, {"vm_name"} for runtime)."""
return (candidate.get("vm") or candidate.get("vm_name") or "")
def infer_machine_type(candidate: Dict[str, Any]) -> Optional[str]:
"""Best-effort machine type: explicit ``machine_type`` field, else vm-name token.
Fallback secret entries created before the ``machine_type`` field carry only
vm/zone/ip; the VM-name family token (…-fallback-c2df) is the only hint and
is expanded to the current fleet size. Returns None when nothing matches
(caller treats as UNKNOWN_RANK).
"""
explicit = candidate.get("machine_type")
if explicit:
return explicit
name = _candidate_name(candidate).lower()
for family in _NAME_TOKEN_TO_FAMILY:
if family in name:
return f"{family}{_FLEET_SIZE_SUFFIX}"
return None
def _speed_rank(candidate: Dict[str, Any], rank_overrides: Optional[Dict[str, Any]] = None) -> int:
family = machine_family(infer_machine_type(candidate))
if rank_overrides and family in rank_overrides:
try:
return int(rank_overrides[family])
except (TypeError, ValueError):
pass # a malformed override must never break worker selection
return SPEED_RANK.get(family, UNKNOWN_RANK)
def cooldown_key(candidate: Dict[str, Any]) -> Optional[str]:
"""Firestore ``capacity_state`` key for a candidate: ``"<machine_type>@<zone>"``.
Zone-scoped because a stockout is per (machine family × zone). Returns None
when the machine type can't be determined (no cooldown tracking possible).
NOTE: keys deliberately avoid ``.`` so they are safe as Firestore map keys.
"""
mtype = infer_machine_type(candidate)
if not mtype:
return None
zone = candidate.get("zone")
return f"{mtype}@{zone}" if zone else mtype
def _parse_iso(value: Any) -> Optional[datetime]:
if not value:
return None
try:
dt = datetime.fromisoformat(value)
except (ValueError, TypeError):
return None
if dt.tzinfo is None: # tolerate naive timestamps — assume UTC
dt = dt.replace(tzinfo=timezone.utc)
return dt
def _recently_stocked_out(
candidate: Dict[str, Any],
capacity_state: Optional[Dict[str, Any]],
now: datetime,
cooldown_seconds: float,
) -> bool:
"""True if this candidate's (type@zone) — or its coarser type-only key — is in cooldown."""
if not capacity_state:
return False
key = cooldown_key(candidate)
if not key:
return False
ts = _parse_iso(capacity_state.get(key))
if ts is None:
# Fall back to a type-only signal (a coarser "this family is short").
mtype = infer_machine_type(candidate)
if mtype:
ts = _parse_iso(capacity_state.get(mtype))
if ts is None:
return False
return (now - ts).total_seconds() < cooldown_seconds
def ordered_candidates(
pool: List[Dict[str, Any]],
capacity_state: Optional[Dict[str, Any]] = None,
now: Optional[datetime] = None,
cooldown_seconds: float = COOLDOWN_SECONDS,
rank_overrides: Optional[Dict[str, Any]] = None,
) -> List[Dict[str, Any]]:
"""Rank ``pool`` fastest-first, demoting recently-stocked-out types to the back.
Args:
pool: candidate dicts, each ``{vm|vm_name, zone, ip, machine_type?, kind?}``.
Objects are returned as-is (not copied), only reordered — so callers can
round-trip their own richer objects through a dict view.
capacity_state: Firestore map ``{"<type>@<zone>"|"<type>": last_stockout_iso}``.
Absent/empty ⇒ pure fastest-first order (today's behaviour).
now: current UTC time (injectable for tests); defaults to ``datetime.now(UTC)``.
cooldown_seconds: demotion window; defaults to ``COOLDOWN_SECONDS``.
rank_overrides: ``{family: rank}`` from Firestore
``config/encoding-worker.speed_rank_overrides``; replaces SPEED_RANK
for those families (e.g. ``{"c7a": 5}`` prefers the AWS worker).
The speed sort is STABLE, so equal-rank candidates keep their input order —
callers express a secondary preference (e.g. zone spread) purely via input
ordering. The hot/cold partition is likewise stable within each group.
"""
if now is None:
now = datetime.now(timezone.utc)
ranked = sorted(pool, key=lambda c: _speed_rank(c, rank_overrides)) # stable: fastest first
hot = [c for c in ranked if not _recently_stocked_out(c, capacity_state, now, cooldown_seconds)]
cold = [c for c in ranked if _recently_stocked_out(c, capacity_state, now, cooldown_seconds)]
return hot + cold