mirror of
https://github.com/nomadkaraoke/karaoke-gen.git
synced 2026-11-20 13:07:57 +09:00
## Why Since 2026-10-31 the AWS EC2 worker (`aws-encoding-worker-1`) has handled every encode and distribution job. 27 jobs went to it, with no Cloud Run fallbacks, no `encoding_stuck`, and nothing encoding-related in #alerts. The c4d Spot pair (`encoding-worker-a`/`-b`) now only costs ~$8/mo of Hyperdisk. `encoding-worker-fallback-c2df` stays as the GCP fallback. ## What Deleting the pair needs code changes first, because several paths assumed it exists: - **`encoding_worker_manager`** - The primary is optional: an empty `primary_vm` means none. - `ranked_candidates` skips a missing primary and always sets `is_primary`, so with no primary a successful start records the AWS/c2df override. Previously the legacy index-0 rule would have cleared it. - `ensure_primary_running` and `ensure_preferred_running` start a ranked candidate. If there are no candidates they raise `NO_CANDIDATES`. - `demote_active_worker` looks up the override's `machine_type`, so the AWS box can now be cooled down. It then clears the override so the retry goes elsewhere. - **`encoding_service`** - When `active_url` is empty, it targets the top-ranked candidate. - It also invalidates its cached URL after a single-candidate fallback. - **CI deploy (`ci.yml` + `deploy_promote.py`)** - While AWS serves, a c2df green is validate-only: it writes `validated_version`, keeps the AWS override, and stops the green. Before this, it would have replaced the AWS override with c2df. - The worker-change gate now diffs against `validated_version`, because `primary_version` stops moving. - An in-place restart of a serving override first waits for it to drain (15-min cap). If it is still busy, the restart is skipped. - **Idle function:** a missing VM is logged as a WARNING returning `NOT_FOUND`, not an ERROR. - **Error monitor:** watches c2df instead of a/b. - **Pulumi:** drops the a/b VMs and static IPs. The prod preview shows exactly 4 deletes. A separate, unrelated `youtube-comment-autoreply-scheduler` drift shows up too; it will not be applied. - **Docs:** CLAUDE.md, ARCHITECTURE, DEVELOPMENT and TROUBLESHOOTING. ## Rollout (after merge + deploy) 1. Back up `config/encoding-worker`, then set `primary_vm`, `primary_ip`, `secondary_vm` and `secondary_ip` to `""`. 2. Apply `pulumi destroy --target` on the 4 a/b URNs only. 3. Redeploy the idle function with `deploy.sh`. 4. Verify: one prod encode on AWS, warmup, and the logs. **Rollback:** in Firestore, delete `speed_rank_overrides` and set `aws_video_worker=false`. Restoring the pair would need a revert plus `pulumi up`, which gives new static IPs. ## Testing - New `backend/tests/test_encoding_worker_no_primary.py` (13 tests), plus new deploy_promote, idle-function and error-monitor tests. - Full backend suite: 7188 passed. The 8 `test_emulator_integration` failures also fail on main, because of a local emulator from another session. - CodeRabbit CLI: no findings. An independent review found 2 issues, both fixed: the demote retry path, and the in-place restart drain. 🤖 Generated with [Claude Code](https://claude.com/claude-code) https://claude.ai/code/session_012RYLVTkatGnrYFhCzq4PTe @coderabbitai ignore Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
121 lines
5.2 KiB
Python
121 lines
5.2 KiB
Python
"""Invariants for the encoding-worker capacity-fallback fleet config.
|
|
|
|
Guards the machine-family diversification added 2026-08-12 after a region-wide
|
|
c4d-highcpu-32 ZONE_RESOURCE_POOL_EXHAUSTED stockout (us-central1-a/-b/-c at once)
|
|
exhausted every same-family lane and forced slow local encoding (→ 524 on
|
|
preview, parked renders).
|
|
|
|
Run locally with: `pytest infrastructure/test_encoding_worker_config.py`
|
|
(infrastructure is not part of the CI backend test gate — Pulumi validates the
|
|
resource graph via `pulumi preview`).
|
|
"""
|
|
|
|
from config import EncodingWorkerConfig, MachineTypes
|
|
|
|
|
|
def test_fleet_is_single_on_demand_fallback():
|
|
"""Since 2026-11 the GCE fleet is ONE on-demand fallback (c2df): the c4d Spot
|
|
a/b pair was retired for the AWS EC2 worker. Growing this again should be a
|
|
deliberate decision (update the fallback secret + docs), not an accident."""
|
|
assert not hasattr(EncodingWorkerConfig, "VM_NAMES")
|
|
assert not hasattr(MachineTypes, "ENCODING_WORKER")
|
|
assert len(EncodingWorkerConfig.FALLBACKS) == 1
|
|
for fb in EncodingWorkerConfig.FALLBACKS:
|
|
assert fb["machine_type"].endswith("-16"), fb
|
|
|
|
|
|
def test_idle_shutdown_is_fast_but_outlives_review_heartbeat():
|
|
"""Idle VMs stop after 5 min, checked every 2 min. The lyrics-review page
|
|
heartbeats every 2 min (frontend REVIEW_HEARTBEAT_INTERVAL_MS) so an active
|
|
review keeps the warm VM alive — the timeout must leave margin over it."""
|
|
assert EncodingWorkerConfig.IDLE_TIMEOUT_MINUTES == 5
|
|
assert EncodingWorkerConfig.IDLE_CHECK_SCHEDULE == "*/2 * * * *"
|
|
review_heartbeat_minutes = 2
|
|
assert EncodingWorkerConfig.IDLE_TIMEOUT_MINUTES >= 2 * review_heartbeat_minutes
|
|
|
|
|
|
def test_n2_fallbacks_use_pd_balanced_disk():
|
|
"""n2 does not support hyperdisk-balanced; those VMs must use pd-balanced,
|
|
else Pulumi/GCE rejects the instance at create time."""
|
|
for fb in EncodingWorkerConfig.FALLBACKS:
|
|
if fb["machine_type"].startswith("n2"):
|
|
assert fb["disk_type"] == "pd-balanced", fb
|
|
|
|
|
|
# Machine families and the ONLY boot-disk type each supports for these VMs.
|
|
# Next-gen Titanium families (c4/c4d/n4/n4d) support hyperdisk-balanced only;
|
|
# older families (c2d/n2/n2d) use pd-balanced. Mismatched disk_type = GCE rejects
|
|
# the instance at create time (the exact failure the fallback fleet must avoid).
|
|
_FAMILY_DISK_TYPE = {
|
|
"c4d": "hyperdisk-balanced",
|
|
"c4": "hyperdisk-balanced",
|
|
"n4d": "hyperdisk-balanced",
|
|
"n4": "hyperdisk-balanced",
|
|
"c2d": "pd-balanced",
|
|
"n2d": "pd-balanced",
|
|
"n2": "pd-balanced",
|
|
}
|
|
|
|
|
|
def _family(machine_type: str) -> str:
|
|
# Longest prefix wins so "n2d"/"c4d" aren't shadowed by "n2"/"c4".
|
|
return max(
|
|
(fam for fam in _FAMILY_DISK_TYPE if machine_type.startswith(fam)),
|
|
key=len,
|
|
default="",
|
|
)
|
|
|
|
|
|
def test_every_fallback_disk_type_matches_its_family_capability():
|
|
"""Each fallback's disk_type must be the one its machine family supports."""
|
|
for fb in EncodingWorkerConfig.FALLBACKS:
|
|
fam = _family(fb["machine_type"])
|
|
assert fam, f"unknown family for {fb['machine_type']}"
|
|
assert fb["disk_type"] == _FAMILY_DISK_TYPE[fam], fb
|
|
|
|
|
|
def test_zone_spread_avoids_same_type_same_zone():
|
|
"""No machine type should sit twice in the same zone (correlated stockout)."""
|
|
seen = set()
|
|
for fb in EncodingWorkerConfig.FALLBACKS:
|
|
key = (fb["machine_type"], fb["zone_suffix"])
|
|
assert key not in seen, f"duplicate (type,zone): {key}"
|
|
seen.add(key)
|
|
|
|
|
|
def test_fallback_names_and_ips_are_unique_and_aligned():
|
|
"""Names/IPs are zipped by position with FALLBACKS — they must stay aligned
|
|
and unique so no two VMs collide on a resource name or static IP."""
|
|
fb = EncodingWorkerConfig.FALLBACKS
|
|
suffixes = [f["suffix"] for f in fb]
|
|
assert len(set(suffixes)) == len(suffixes), "duplicate fallback suffix"
|
|
assert EncodingWorkerConfig.FALLBACK_VM_NAMES == [
|
|
EncodingWorkerConfig.fallback_vm_name(s) for s in suffixes
|
|
]
|
|
assert EncodingWorkerConfig.FALLBACK_IP_NAMES == [
|
|
EncodingWorkerConfig.fallback_ip_name(s) for s in suffixes
|
|
]
|
|
assert len(set(EncodingWorkerConfig.FALLBACK_VM_NAMES)) == len(fb)
|
|
assert len(set(EncodingWorkerConfig.FALLBACK_IP_NAMES)) == len(fb)
|
|
|
|
|
|
def test_worker_boot_disk_fits_packer_image_size():
|
|
"""GCE cannot create a boot disk smaller than its source image's disk size.
|
|
The Packer image's `disk_size` must stay <= DiskSizes.ENCODING_WORKER, or
|
|
every worker (re)creation from the new image fails. Also keeps the
|
|
ENCODING_WORKER_IMAGE mirror constant honest."""
|
|
import re
|
|
from pathlib import Path
|
|
|
|
from config import DiskSizes
|
|
|
|
hcl = (Path(__file__).parent / "packer" / "encoding-worker.pkr.hcl").read_text()
|
|
match = re.search(r"^\s*disk_size\s*=\s*(\d+)", hcl, re.MULTILINE)
|
|
assert match, "disk_size not found in encoding-worker.pkr.hcl"
|
|
image_size = int(match.group(1))
|
|
assert image_size == DiskSizes.ENCODING_WORKER_IMAGE
|
|
assert image_size <= DiskSizes.ENCODING_WORKER
|
|
# Headroom floor: OS + baked venv is ~12-15 GB and per-job scratch lives in
|
|
# /tmp on the boot disk. Don't let a future "cost cut" starve encodes.
|
|
assert DiskSizes.ENCODING_WORKER >= 40
|