mirror of
https://github.com/nomadkaraoke/karaoke-gen.git
synced 2026-11-20 13:07:57 +09:00
Follow-up to #1236. - **Instance type:** the AWS encoding worker is now **on-demand c7a.4xlarge** instead of Spot c7a.8xlarge. - The Spot create failed with `InsufficientInstanceCapacity`, and Spot placement scores were 1-3/10 for c7a/c7i/c6a/c8a 4-8xlarge across us-east-1/2. - A worker that stops and starts on demand has to win capacity on every start. - Cost is ~$82/mo of credits. The 4xlarge matched or beat GCE in the replay benchmark. - **Already applied** (instance `i-0c21505d7017356bb`, EIP 3.128.70.236). - **Docs and plan:** stage A status, plus the live end-to-end test through the worker HTTP API (render 117s; all 9 outputs; portrait SSIM 0.993, the only difference being anti-aliasing of the header text). The only code-file change is a comment in `encoding_worker_preference.py`, so there is no version bump. No local CodeRabbit run (IaC and docs only), so the bot review stays on. 🤖 Generated with [Claude Code](https://claude.com/claude-code) https://claude.ai/code/session_012RYLVSR5qgwsxTQtvGimKF <!-- This is an auto-generated comment: release notes by coderabbit.ai --> <!-- coderabbit-ai-generated:v1 --> ## Summary by CodeRabbit * **Operational Updates** * The AWS encoding worker now runs on an on-demand `c7a.4xlarge` instance rather than a Spot `c7a.8xlarge`. * The worker stops when the operating system shuts down, while its root volume is preserved. * If AWS cannot start the worker because capacity is insufficient, it is temporarily deprioritized for 15 minutes, allowing processing to fall back to GCE. <!-- end of auto-generated comment: release notes by coderabbit.ai --> --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
204 lines
8.7 KiB
Python
204 lines
8.7 KiB
Python
"""Shared, pure preference logic for encoding-worker candidate ordering.
|
||
|
||
This is the SINGLE SOURCE OF TRUTH for how encode-worker VM candidates are ranked,
|
||
used by BOTH:
|
||
|
||
* runtime worker selection —
|
||
``backend/services/encoding_service._build_worker_candidates`` →
|
||
``encoding_worker_manager.ensure_any_running``
|
||
* deploy green (staging) selection —
|
||
``infrastructure/encoding-worker/deploy_promote.select_green_candidates``
|
||
|
||
Keeping the ordering in one place stops the two call-sites from drifting (they
|
||
previously used different ad-hoc heuristics: runtime = secret order, deploy =
|
||
"n2 before c4d").
|
||
|
||
Design:
|
||
* **Fastest-first base order.** Candidates sort by ``SPEED_RANK`` (lower =
|
||
faster), so the fastest available type (c4d) is always preferred when it can
|
||
start — the whole reason c4d was chosen. No behaviour change on a healthy day.
|
||
* **Availability-aware cooldown.** A type that recently failed to start with a
|
||
capacity/stockout error (recorded in Firestore ``capacity_state``) is
|
||
*demoted to the back* for ``COOLDOWN_SECONDS`` — long enough to stop paying
|
||
the ~2-min-per-attempt probe cost on a known-dead type, short enough to
|
||
re-probe and snap back to the faster type the moment capacity returns. It is
|
||
demoted, never dropped, so the pool always stays fully available.
|
||
|
||
This module is intentionally **dependency-free (stdlib only)** so that
|
||
``deploy_promote`` — which is loaded by path in CI with no backend package deps —
|
||
can import it without pulling the heavy backend tree.
|
||
"""
|
||
from __future__ import annotations
|
||
|
||
from datetime import datetime, timezone
|
||
from typing import Any, Dict, List, Optional
|
||
|
||
# Lower rank = faster = preferred. Keyed by machine FAMILY (the prefix before the
|
||
# first "-"), so the ranking is independent of the vCPU size — the fleet moved
|
||
# from *-highcpu-32 to *-highcpu-16 on 2026-09-26 and could move again. Seeded
|
||
# from CPU architecture; refine with the measured medians from
|
||
# infrastructure/encoding-worker/benchmark_types.py.
|
||
# c4d AMD EPYC Turin (Zen5) — Spot blue-green primary pair, fastest
|
||
# c4 Intel Emerald Rapids — newest Intel perf tier
|
||
# n4d AMD (Titanium) — flexible/deep AMD
|
||
# c2d AMD Milan (Zen3) — mature, deep pool (current on-demand fallback)
|
||
# n2d AMD Rome/Milan — deep pool
|
||
# n2 Intel Cascade/Ice Lake — deep pool
|
||
SPEED_RANK: Dict[str, int] = {
|
||
"c4d": 10,
|
||
"c4": 20,
|
||
"n4d": 30,
|
||
"c2d": 50,
|
||
"n2d": 60,
|
||
"n2": 70,
|
||
# AWS EC2 c7a (AMD Genoa, us-east-2) — the AWS encoding worker
|
||
# (provider "aws"; machine_type "c7a-4xlarge", "." → "-" for Firestore-safe
|
||
# cooldown keys). Last resort by default; make it preferred at runtime with
|
||
# config/encoding-worker.speed_rank_overrides = {"c7a": 5}.
|
||
"c7a": 90,
|
||
}
|
||
|
||
# Rank for a candidate whose machine type is unknown and cannot be inferred:
|
||
# treat as slowest-but-usable so it sorts LAST without ever being dropped.
|
||
UNKNOWN_RANK = 1000
|
||
|
||
# How long a type that just stocked out is demoted before it is re-probed.
|
||
COOLDOWN_SECONDS = 900 # 15 minutes
|
||
|
||
# Machine type of the blue-green primary/secondary pair (encoding-worker-a/-b).
|
||
# They are always c4d (Spot); used when a candidate carries no explicit
|
||
# machine_type. Keep in sync with infrastructure/config.py MachineTypes.ENCODING_WORKER.
|
||
PRIMARY_MACHINE_TYPE = "c4d-highcpu-16"
|
||
|
||
# Current vCPU size of the fleet — used only to turn a family inferred from a VM
|
||
# name into a full machine type (for display + cooldown keys).
|
||
_FLEET_SIZE_SUFFIX = "-highcpu-16"
|
||
|
||
# vm-name substring → machine family, for backward-compat inference when a
|
||
# fallback entry predates the machine_type field. Checked in order — longer /
|
||
# more-specific family tokens FIRST so "n2d" is not shadowed by "n2", nor "c4d"
|
||
# by "c4".
|
||
_NAME_TOKEN_TO_FAMILY = ("c4d", "n4d", "n2d", "c2d", "c4", "n2")
|
||
|
||
|
||
def machine_family(machine_type: Optional[str]) -> Optional[str]:
|
||
"""``"c4d-highcpu-16"`` → ``"c4d"``; None/empty → None."""
|
||
if not machine_type:
|
||
return None
|
||
return machine_type.split("-", 1)[0].lower()
|
||
|
||
|
||
def _candidate_name(candidate: Dict[str, Any]) -> str:
|
||
"""VM name from either dict shape ({"vm"} for deploy, {"vm_name"} for runtime)."""
|
||
return (candidate.get("vm") or candidate.get("vm_name") or "")
|
||
|
||
|
||
def infer_machine_type(candidate: Dict[str, Any]) -> Optional[str]:
|
||
"""Best-effort machine type: explicit ``machine_type`` field, else vm-name token.
|
||
|
||
Fallback secret entries created before the ``machine_type`` field carry only
|
||
vm/zone/ip; the VM-name family token (…-fallback-c2df) is the only hint and
|
||
is expanded to the current fleet size. Returns None when nothing matches
|
||
(caller treats as UNKNOWN_RANK).
|
||
"""
|
||
explicit = candidate.get("machine_type")
|
||
if explicit:
|
||
return explicit
|
||
name = _candidate_name(candidate).lower()
|
||
for family in _NAME_TOKEN_TO_FAMILY:
|
||
if family in name:
|
||
return f"{family}{_FLEET_SIZE_SUFFIX}"
|
||
return None
|
||
|
||
|
||
def _speed_rank(candidate: Dict[str, Any], rank_overrides: Optional[Dict[str, Any]] = None) -> int:
|
||
family = machine_family(infer_machine_type(candidate))
|
||
if rank_overrides and family in rank_overrides:
|
||
try:
|
||
return int(rank_overrides[family])
|
||
except (TypeError, ValueError):
|
||
pass # a malformed override must never break worker selection
|
||
return SPEED_RANK.get(family, UNKNOWN_RANK)
|
||
|
||
|
||
def cooldown_key(candidate: Dict[str, Any]) -> Optional[str]:
|
||
"""Firestore ``capacity_state`` key for a candidate: ``"<machine_type>@<zone>"``.
|
||
|
||
Zone-scoped because a stockout is per (machine family × zone). Returns None
|
||
when the machine type can't be determined (no cooldown tracking possible).
|
||
NOTE: keys deliberately avoid ``.`` so they are safe as Firestore map keys.
|
||
"""
|
||
mtype = infer_machine_type(candidate)
|
||
if not mtype:
|
||
return None
|
||
zone = candidate.get("zone")
|
||
return f"{mtype}@{zone}" if zone else mtype
|
||
|
||
|
||
def _parse_iso(value: Any) -> Optional[datetime]:
|
||
if not value:
|
||
return None
|
||
try:
|
||
dt = datetime.fromisoformat(value)
|
||
except (ValueError, TypeError):
|
||
return None
|
||
if dt.tzinfo is None: # tolerate naive timestamps — assume UTC
|
||
dt = dt.replace(tzinfo=timezone.utc)
|
||
return dt
|
||
|
||
|
||
def _recently_stocked_out(
|
||
candidate: Dict[str, Any],
|
||
capacity_state: Optional[Dict[str, Any]],
|
||
now: datetime,
|
||
cooldown_seconds: float,
|
||
) -> bool:
|
||
"""True if this candidate's (type@zone) — or its coarser type-only key — is in cooldown."""
|
||
if not capacity_state:
|
||
return False
|
||
key = cooldown_key(candidate)
|
||
if not key:
|
||
return False
|
||
ts = _parse_iso(capacity_state.get(key))
|
||
if ts is None:
|
||
# Fall back to a type-only signal (a coarser "this family is short").
|
||
mtype = infer_machine_type(candidate)
|
||
if mtype:
|
||
ts = _parse_iso(capacity_state.get(mtype))
|
||
if ts is None:
|
||
return False
|
||
return (now - ts).total_seconds() < cooldown_seconds
|
||
|
||
|
||
def ordered_candidates(
|
||
pool: List[Dict[str, Any]],
|
||
capacity_state: Optional[Dict[str, Any]] = None,
|
||
now: Optional[datetime] = None,
|
||
cooldown_seconds: float = COOLDOWN_SECONDS,
|
||
rank_overrides: Optional[Dict[str, Any]] = None,
|
||
) -> List[Dict[str, Any]]:
|
||
"""Rank ``pool`` fastest-first, demoting recently-stocked-out types to the back.
|
||
|
||
Args:
|
||
pool: candidate dicts, each ``{vm|vm_name, zone, ip, machine_type?, kind?}``.
|
||
Objects are returned as-is (not copied), only reordered — so callers can
|
||
round-trip their own richer objects through a dict view.
|
||
capacity_state: Firestore map ``{"<type>@<zone>"|"<type>": last_stockout_iso}``.
|
||
Absent/empty ⇒ pure fastest-first order (today's behaviour).
|
||
now: current UTC time (injectable for tests); defaults to ``datetime.now(UTC)``.
|
||
cooldown_seconds: demotion window; defaults to ``COOLDOWN_SECONDS``.
|
||
rank_overrides: ``{family: rank}`` from Firestore
|
||
``config/encoding-worker.speed_rank_overrides``; replaces SPEED_RANK
|
||
for those families (e.g. ``{"c7a": 5}`` prefers the AWS worker).
|
||
|
||
The speed sort is STABLE, so equal-rank candidates keep their input order —
|
||
callers express a secondary preference (e.g. zone spread) purely via input
|
||
ordering. The hot/cold partition is likewise stable within each group.
|
||
"""
|
||
if now is None:
|
||
now = datetime.now(timezone.utc)
|
||
ranked = sorted(pool, key=lambda c: _speed_rank(c, rank_overrides)) # stable: fastest first
|
||
hot = [c for c in ranked if not _recently_stocked_out(c, capacity_state, now, cooldown_seconds)]
|
||
cold = [c for c in ranked if _recently_stocked_out(c, capacity_state, now, cooldown_seconds)]
|
||
return hot + cold
|