mirror of
https://github.com/nomadkaraoke/karaoke-gen.git
synced 2026-11-20 13:07:57 +09:00
## Summary Approved GCP cost cuts for the encoding pipeline (target: whole project < $300/mo; encoding VMs were ~$300/mo for ~5-20 jobs/day). Details + before/after table: `docs/archive/2026-09-26-encoding-cost-cuts.md`. | | Before | After | |---|---|---| | Primary pair a/b | c4d-highcpu-32 on-demand | **c4d-highcpu-16 Spot** (STOP on preemption) | | Fallbacks | 8 VMs / 6 families | **1 on-demand** `encoding-worker-fallback-c2df` (c2d-highcpu-16, us-central1-f) | | Idle shutdown | 15 min / check every 5 | **5 min / check every 2** | | CI worker blue-green | every push to main | only when `infrastructure/encoding-worker/worker_code_paths.txt` paths changed since the serving version (+ `workflow_dispatch` `force_encoding_worker_deploy`) | | Nomad Dropbox upload | whole folder | minus lossless 4K **MP4** (tenant folders unchanged) | **Infra already applied locally (targeted Pulumi):** 7 fallback VMs + disks + 7 static IPs deleted; a/b + c2df resized in place; idle function env + scheduler updated. The Spot switch was done in place with `gcloud compute instances set-scheduling` on the stopped VMs + `pulumi refresh` (pulumi-gcp treats a `scheduling` diff as *replace*, and recreating a c4d VM risks a stockout). `encoding-worker-fallback-vms` secret → v6 (c2df only). CI's pulumi run should be a no-op for these resources. ### Scope change (per Andrew) **Task 5 dropped:** review previews stay on the encoding VM (`USE_GCE_PREVIEW_ENCODING=true` unchanged) and the review-page warmup is kept, since it deliberately warms the VM for the preview and the final render. To make sure the new 5-min idle timeout doesn't undermine that, the review page now also heartbeats **every 2 min** while the tab is visible and was used in the last 15 min, and re-warms on tab refocus (`useEncodingWorkerKeepAlive`). Before this, the heartbeat only fired on lyric edits. The idle checker keeps the routed VM alive while `last_activity_at` < 5 min, and both warmup and heartbeat write it. Tests cover both sides. ### Code changes - **Spot preemption mid-encode**: when a poll fails, `wait_for_completion` checks the pinned worker VM. If it is STOPPING/TERMINATED it raises `EncodingJobLostError`, so `run_with_lost_job_resubmit` resubmits the encode, and that resubmit restarts the Spot VM or falls back to c2df. Before, the failure showed up after ~7 min as a "lost contact" error that was not resubmitted. - **Stale fallback lists**: a candidate VM that no longer exists (`NotFound`) is skipped instead of aborting the whole failover. This protects the window before running instances pick up the new secret. - `SPEED_RANK` is now keyed by machine family, so the ranking is independent of VM size. - **Idle-stop vs. in-flight encode** (review follow-up): the idle function's `/health active_jobs` only counts RUNNING jobs, and `last_activity_at` was only written when a VM was started. With a 5-min window, a long encode's VM could be stopped in the seconds between the job finishing and the next poll. The poll loop now refreshes `last_activity_at` at most once a minute. A VM seen in STAGING/PROVISIONING after it accepted a job also counts as lost. CI `Read Firestore config` now writes empty values instead of "None". - **Worker deploy gate**: `backend/tests/test_encoding_worker_code_paths.py` derives the worker's transitive `backend.*` import set and fails if `worker_code_paths.txt` doesn't cover it. Every deploy still uploads the wheel and `startup.sh`, so a cold-booted VM always runs the latest code. - **Dropbox**: nothing automated reads lossless files from Dropbox. The lossless **MKV** is still uploaded because the completion email and the live Fiverr auto-reply bot both tell customers it is in the Dropbox folder, and the VocalStar tenant expects lossless files. To drop the MKV as well, update those two texts first, then add the MKV suffix to `DROPBOX_SKIP_OUTPUT_SUFFIXES`. ## Testing - New/updated unit tests: preemption → lost job → resubmit, NotFound candidate skip, family ranking, idle 5-min/heartbeat window, fleet-shape invariants, worker-path coverage, Dropbox exclusion, keep-alive hook (Jest, fake timers). - `pytest backend/tests --ignore=backend/tests/emulator`: 4802 passed. The only failures are 6 tests that need the spaCy `en_core_web_sm` model, which isn't installed in my local venv; unrelated to this PR. - Jest lyrics-review + lib/lyrics-review: 423 passed. Infra tests (`test_encoding_worker_config.py`, idle function): all pass. 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
106 lines
4.0 KiB
Python
106 lines
4.0 KiB
Python
"""Guard for the CI encoding-worker deploy gate.
|
|
|
|
CI only blue-green redeploys the GCE encoding worker when a path listed in
|
|
infrastructure/encoding-worker/worker_code_paths.txt changed since the serving
|
|
worker's version. If the worker starts importing a backend module that is NOT
|
|
listed, changes to it would ship to the worker only on its next cold boot and
|
|
without the pre-promote test encode. This test derives the worker's import
|
|
closure (every ``backend.*`` module reachable from the worker entrypoint and the
|
|
karaoke_gen package) and asserts each one is covered by the path list.
|
|
"""
|
|
import ast
|
|
from pathlib import Path
|
|
|
|
REPO = Path(__file__).resolve().parents[2]
|
|
PATHS_FILE = REPO / "infrastructure" / "encoding-worker" / "worker_code_paths.txt"
|
|
|
|
|
|
def _pathspecs():
|
|
specs = []
|
|
for line in PATHS_FILE.read_text().splitlines():
|
|
line = line.strip()
|
|
if line and not line.startswith("#"):
|
|
specs.append(line)
|
|
return specs
|
|
|
|
|
|
def _covered(rel_path: str, specs) -> bool:
|
|
return any(rel_path == s or (s.endswith("/") and rel_path.startswith(s)) for s in specs)
|
|
|
|
|
|
def _backend_imports(py_file: Path):
|
|
"""All ``backend.*`` module names imported anywhere in the file (incl. lazily)."""
|
|
tree = ast.parse(py_file.read_text(), filename=str(py_file))
|
|
mods = set()
|
|
for node in ast.walk(tree):
|
|
if isinstance(node, ast.ImportFrom) and node.module and node.level == 0:
|
|
if node.module == "backend" or node.module.startswith("backend."):
|
|
mods.add(node.module)
|
|
for alias in node.names: # `from backend.services import x` → module x
|
|
mods.add(f"{node.module}.{alias.name}")
|
|
elif isinstance(node, ast.Import):
|
|
for alias in node.names:
|
|
if alias.name.startswith("backend."):
|
|
mods.add(alias.name)
|
|
return mods
|
|
|
|
|
|
def _module_file(mod: str):
|
|
base = REPO / Path(*mod.split("."))
|
|
for candidate in (base.with_suffix(".py"), base / "__init__.py"):
|
|
if candidate.is_file():
|
|
return candidate
|
|
return None # a name imported from a module, not a module itself
|
|
|
|
|
|
def _worker_closure():
|
|
roots = list((REPO / "karaoke_gen").rglob("*.py"))
|
|
roots += list((REPO / "backend" / "services" / "gce_encoding").rglob("*.py"))
|
|
seen_files = set()
|
|
queue = list(roots)
|
|
backend_files = set()
|
|
while queue:
|
|
f = queue.pop()
|
|
if f in seen_files:
|
|
continue
|
|
seen_files.add(f)
|
|
for mod in _backend_imports(f):
|
|
mf = _module_file(mod)
|
|
if not mf:
|
|
continue
|
|
# Importing backend.a.b also executes backend/__init__.py and
|
|
# backend/a/__init__.py — they're part of the worker's code too.
|
|
parts = mod.split(".")
|
|
pkg_inits = [_module_file(".".join(parts[:i])) for i in range(1, len(parts))]
|
|
for dep in [mf, *pkg_inits]:
|
|
if dep and dep not in seen_files:
|
|
backend_files.add(dep)
|
|
queue.append(dep)
|
|
return backend_files
|
|
|
|
|
|
def test_worker_code_paths_file_is_well_formed():
|
|
specs = _pathspecs()
|
|
assert "karaoke_gen/" in specs
|
|
assert "backend/services/gce_encoding/" in specs
|
|
# pyproject.toml changes on EVERY PR (version bump) — listing it would make
|
|
# the gate fire on every push, defeating it. Dependency changes are caught
|
|
# via poetry.lock.
|
|
assert "pyproject.toml" not in specs
|
|
assert "poetry.lock" in specs
|
|
for s in specs:
|
|
assert (REPO / s.rstrip("/")).exists(), f"stale path in {PATHS_FILE.name}: {s}"
|
|
|
|
|
|
def test_every_backend_module_the_worker_imports_is_gated():
|
|
specs = _pathspecs()
|
|
missing = sorted(
|
|
str(f.relative_to(REPO)) for f in _worker_closure()
|
|
if not _covered(str(f.relative_to(REPO)), specs)
|
|
)
|
|
assert not missing, (
|
|
"The encoding worker imports backend modules not listed in "
|
|
f"{PATHS_FILE.relative_to(REPO)} — add them so CI redeploys the worker "
|
|
f"when they change: {missing}"
|
|
)
|