mirror of
https://github.com/nomadkaraoke/karaoke-gen.git
synced 2026-11-20 13:07:57 +09:00
## Summary
Phase 1 of moving encoding to AWS. EC2 is paid for by the AWS Activate
credits (acct 790139457751, expiring 2028-10-31).
- **Plan:** `docs/archive/2026-10-29-aws-encoding-worker-plan.md`.
- **Benchmark** (3 real prod jobs replayed): an EC2 c7a.4xlarge matched
or beat GCE c4d-highcpu-16.
The EC2 box runs the **unchanged** `gce_encoding` worker service and
joins the existing candidate pool as `provider: aws`. The existing
failover, active override, lost-job detection and stockout cooldowns all
apply to it as they are.
## Changes
- **`backend/services/aws_encoding_compute.py` (new):**
`AwsComputeAdapter` looks to the manager like the
`compute_v1.InstancesClient` methods it uses.
- EC2 states map to GCE statuses.
- EC2 capacity errors are added to `CAPACITY_ERROR_CODES`.
- A missing instance raises `NotFound`.
- Controller credentials come from Secret Manager
`aws-encoding-controller-credentials`.
- **`EncodingWorkerManager`:**
- `_compute_for(vm)` sends AWS VMs to the adapter; the GCE paths are
untouched.
- A shared `parse_fallback_vms` passes `provider`/`instance_id`/`region`
through.
- `ranked_candidates()` is used by both the encode path and the warmups.
- `ensure_preferred_running()` behaves **exactly like
`ensure_primary_running` whenever the GCE primary ranks first**, which
is today's case.
- An AWS control-plane fault fails over to the GCE candidates.
- **Ranking:** `SPEED_RANK["c7a"] = 90`, so AWS is the last resort by
default. Firestore `config/encoding-worker.speed_rank_overrides =
{"c7a": 5}` makes it preferred, with no deploy.
- **Isolation from GCE-only tooling:**
- The override now records `active_override_provider`.
- The CI blue-green deploy, `deploy_promote` and the idle Cloud Function
all ignore the AWS worker; it hot-updates its wheel per job.
- A GCE promotion clears any stale provider tag.
- **Worker:**
- Uses `gcloud storage` on AWS, because gsutil can't use WIF credential
files. GCE keeps gsutil.
- `infrastructure/encoding-worker/aws/` adds:
- `provision.sh`, as EC2 user-data;
- `startup-aws.sh`, uploaded by CI;
- `idle-watchdog.py`, an idle self-stop that runs as its own systemd
timer.
- **Infra (GCP), already applied with `/gcp-admin`:**
- The WIF pool `aws-encoding`, plus an AWS provider that only admits the
`nomad-encoding-worker` role, federated to `encoding-worker@`.
- The secret `aws-encoding-controller-credentials`.
- Preview and apply: 4 creates, 288 unchanged.
- **Infra (AWS):** a new Pulumi project `infrastructure/aws` (stack
`nomadkaraoke/prod`):
- the c7a.8xlarge **persistent Spot** worker, which stops on
interruption or shutdown;
- an Elastic IP;
- a security group on 8080 (API key, same as the GCE firewall);
- the worker role with SSM;
- a tag-scoped `nomad-encoding-controller` user.
- Preview: 10 creates. **I apply it after merge**, because the box
provisions from the deployed wheel and boot scripts.
- `boto3` is now an explicit dependency.
- CI also runs the `encoding_worker_idle` function tests, in the job
renamed "Infra - Cloud Function tests".
## Rollout (after merge)
1. **Stage A:** `pulumi up` the AWS stack. Store the controller key in
the secret, then add the worker's entry to
`encoding-worker-fallback-vms`. It's inert unless both c4d and c2d stock
out.
2. **Stage B:** a supervised `speed_rank_overrides {"c7a": 5}` window.
Run real jobs, then compare timings and outputs, including a visual
check of the portrait video.
3. **Stage C:** keep it preferred. After a clean week, delete the c4d
pair.
Phase 2, running distribution (Dropbox/YouTube/Drive uploads) from AWS,
follows in a separate PR.
## Testing
- **New tests:**
- adapter: botocore `Stubber`, covering the state map, capacity and
NotFound handling;
- manager routing and ranking with overrides;
- `ensure_preferred_running` branches;
- failover on an AWS control-plane fault;
- provider tagging;
- deploy_promote exclusions;
- the idle function skipping AWS;
- watchdog decisions;
- the gcs CLI choice.
- **Updated** the candidate and warmup-route tests to the new seams.
- **Full backend + unit suites** have no new failures. The local
emulator and cross-suite failures are identical on main.
- **Local CodeRabbit:** 2 findings, both fixed (failover on AWS faults;
stale provider tag).
@coderabbitai ignore
🤖 Generated with [Claude Code](https://claude.com/claude-code)
https://claude.ai/code/session_012RYLVSR5qgwsxTQtvGimKF
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
203 lines
9.1 KiB
Python
203 lines
9.1 KiB
Python
"""Pure decision logic for the encoding-worker blue-green deploy (used by CI).
|
|
|
|
The GitHub Actions workflow (`.github/workflows/ci.yml`) does all the GCP I/O
|
|
(`gcloud compute instances start/stop`, health `curl`s, Firestore writes). It
|
|
delegates the *decisions* to the pure functions here so the branching — the part
|
|
that's easy to get wrong and impossible to run locally in CI — has real unit
|
|
tests (`backend/tests/test_deploy_promote.py`).
|
|
|
|
Background: the c4d primary/secondary are Spot VMs (since 2026-09-26) and c4d
|
|
capacity in us-central1 can run out, in which case traffic runs on the on-demand
|
|
fallback recorded as `active_override_vm`. A naive "start the c4d secondary and
|
|
swap" deploy can't start the secondary and never refreshes the serving fallback,
|
|
so worker-side changes never reach production. These helpers make the deploy pick
|
|
a *fresh* green target (c4d secondary if it starts, else a fallback) and then
|
|
promote that validated green to serving with zero downtime. CI only runs this when
|
|
worker code changed (infrastructure/encoding-worker/worker_code_paths.txt).
|
|
"""
|
|
from __future__ import annotations
|
|
|
|
from typing import Any, Dict, List, Optional
|
|
|
|
# c4d/n2 primary+secondary live here; the worker manager uses the same zone.
|
|
DEFAULT_C4D_ZONE = "us-central1-c"
|
|
|
|
# Shared, pure candidate-ranking logic — the SINGLE SOURCE OF TRUTH also used by
|
|
# the runtime worker manager, so deploy green selection and runtime selection can
|
|
# never drift. Preferred import is the normal package path; when that isn't
|
|
# importable (this file is loaded by path in CI/tests, and an installed `backend`
|
|
# package can even shadow the repo working tree), fall back to loading the module
|
|
# straight from its file. It is stdlib-only, so by-path exec has no side effects.
|
|
try:
|
|
from backend.services.encoding_worker_preference import (
|
|
PRIMARY_MACHINE_TYPE,
|
|
ordered_candidates,
|
|
)
|
|
except ImportError: # pragma: no cover - exercised in the bare CI heredoc / shadowed env
|
|
import importlib.util
|
|
import pathlib
|
|
|
|
_pref_path = (
|
|
pathlib.Path(__file__).resolve().parents[2]
|
|
/ "backend" / "services" / "encoding_worker_preference.py"
|
|
)
|
|
_spec = importlib.util.spec_from_file_location("encoding_worker_preference", _pref_path)
|
|
_pref = importlib.util.module_from_spec(_spec)
|
|
_spec.loader.exec_module(_pref)
|
|
PRIMARY_MACHINE_TYPE = _pref.PRIMARY_MACHINE_TYPE
|
|
ordered_candidates = _pref.ordered_candidates
|
|
|
|
|
|
def select_green_candidates(
|
|
config: Dict[str, Any],
|
|
fallback_vms: List[Dict[str, Any]],
|
|
) -> List[Dict[str, Optional[str]]]:
|
|
"""Ranked list of candidate green (staging) targets for the workflow to try.
|
|
|
|
Pool = the c4d ``secondary_vm`` (kind="secondary") + all fallback VMs
|
|
(kind="fallback"), EXCLUDING the current ``active_override_vm``. The pool is
|
|
ranked by the shared preference logic — fastest-first, demoting any type that
|
|
recently stocked out (per ``config["capacity_state"]``). So:
|
|
* when c4d has capacity, the c4d secondary sorts first → normal blue-green;
|
|
* during a c4d stockout, the secondary is demoted and a deep-pool fallback
|
|
(n2d/n2/c2d) leads, instead of wasting ~2 min probing a c4d that won't boot.
|
|
|
|
Excluding the current override is what makes the deploy zero-downtime: the
|
|
green is always a *different*, freshly-booted worker, so the override keeps
|
|
serving until the workflow atomically switches to the validated green.
|
|
|
|
Each candidate: {"vm", "ip", "zone", "machine_type", "kind"}.
|
|
"""
|
|
pool: List[Dict[str, Optional[str]]] = []
|
|
|
|
secondary_vm = config.get("secondary_vm")
|
|
if secondary_vm:
|
|
pool.append({
|
|
"vm": secondary_vm,
|
|
"ip": config.get("secondary_ip"),
|
|
"zone": config.get("secondary_zone") or DEFAULT_C4D_ZONE,
|
|
# secondary is always the c4d pair unless the config says otherwise.
|
|
"machine_type": config.get("secondary_machine_type") or PRIMARY_MACHINE_TYPE,
|
|
"kind": "secondary",
|
|
})
|
|
|
|
override_vm = config.get("active_override_vm")
|
|
# Only complete entries are usable — a fallback missing vm/zone/ip would
|
|
# produce an unstartable candidate (e.g. `gcloud ... --zone=None`) and
|
|
# silently drop capacity from the selection path.
|
|
# AWS workers (provider "aws") are excluded: this workflow starts/stops
|
|
# greens with `gcloud compute`, and the AWS worker hot-updates its wheel per
|
|
# job instead of being blue-green deployed.
|
|
usable = [
|
|
f for f in (fallback_vms or [])
|
|
if f.get("vm") and f.get("zone") and f.get("ip") and f.get("vm") != override_vm
|
|
and (f.get("provider") or "").lower() != "aws"
|
|
]
|
|
for f in usable:
|
|
pool.append({
|
|
"vm": f["vm"],
|
|
"ip": f.get("ip"),
|
|
"zone": f.get("zone"),
|
|
"machine_type": f.get("machine_type"), # inferred from vm name if absent
|
|
"kind": "fallback",
|
|
})
|
|
|
|
return ordered_candidates(pool, capacity_state=config.get("capacity_state"))
|
|
|
|
|
|
def _is_aws_override(config: Dict[str, Any]) -> bool:
|
|
return (config.get("active_override_provider") or "").lower() == "aws"
|
|
|
|
|
|
def promote_plan(
|
|
green: Dict[str, Any],
|
|
config: Dict[str, Any],
|
|
*,
|
|
now: str,
|
|
version: str,
|
|
) -> Dict[str, Any]:
|
|
"""Given a *validated* green worker, decide how to make it the serving worker.
|
|
|
|
Returns {"firestore_updates": {...}, "drain_and_stop": [{"vm","ip","zone"}, ...]}.
|
|
The workflow applies the Firestore updates in one transaction, then drains
|
|
(waits for active_jobs==0, bounded) and stops each VM in `drain_and_stop`.
|
|
|
|
Two cases:
|
|
* green.kind == "secondary": standard blue-green swap (green→primary, old
|
|
primary→secondary); ALSO clear any `active_override_*` so traffic returns
|
|
to the fresh c4d primary. Drain+stop the old primary and the old override.
|
|
* green.kind == "fallback": the green becomes the `active_override` (the
|
|
serving worker). Leave primary/secondary untouched so c4d resumes as
|
|
primary when capacity returns. Drain+stop the previous override (if any,
|
|
and different from the green).
|
|
"""
|
|
drain: List[Dict[str, Optional[str]]] = []
|
|
|
|
def _finalize(updates: Dict[str, Any]) -> Dict[str, Any]:
|
|
# Never drain/stop the worker we just promoted, whatever the config held
|
|
# (e.g. a stale override that pointed at the green, or primary==secondary).
|
|
safe = [t for t in drain if t.get("vm") and t["vm"] != green["vm"]]
|
|
return {"firestore_updates": updates, "drain_and_stop": safe}
|
|
|
|
if green["kind"] == "secondary":
|
|
updates: Dict[str, Any] = {
|
|
"primary_vm": green["vm"],
|
|
"primary_ip": green["ip"],
|
|
"primary_version": version,
|
|
"primary_deployed_at": now,
|
|
"secondary_vm": config.get("primary_vm"),
|
|
"secondary_ip": config.get("primary_ip"),
|
|
"secondary_version": config.get("primary_version"),
|
|
"secondary_deployed_at": now,
|
|
"last_swap_at": now,
|
|
"deploy_in_progress": False,
|
|
"deploy_in_progress_since": None,
|
|
}
|
|
old_primary = config.get("primary_vm")
|
|
if old_primary:
|
|
drain.append({
|
|
"vm": old_primary,
|
|
"ip": config.get("primary_ip"),
|
|
"zone": config.get("primary_zone") or DEFAULT_C4D_ZONE,
|
|
})
|
|
# A stale override must be cleared so active_url follows the fresh primary —
|
|
# except an AWS override, which is serving by preference (rank override),
|
|
# can't be gcloud-stopped, and picks up the new wheel on its next job.
|
|
if config.get("active_override_vm") and not _is_aws_override(config):
|
|
updates.update({
|
|
"active_override_vm": None,
|
|
"active_override_ip": None,
|
|
"active_override_zone": None,
|
|
"active_override_set_at": None,
|
|
"active_override_version": None,
|
|
"active_override_provider": None,
|
|
})
|
|
drain.append({
|
|
"vm": config["active_override_vm"],
|
|
"ip": config.get("active_override_ip"),
|
|
"zone": config.get("active_override_zone"),
|
|
})
|
|
return _finalize(updates)
|
|
|
|
# kind == "fallback": the green becomes the serving override.
|
|
updates = {
|
|
"active_override_vm": green["vm"],
|
|
"active_override_ip": green["ip"],
|
|
"active_override_zone": green["zone"],
|
|
"active_override_set_at": now,
|
|
"active_override_version": version,
|
|
# The green is a GCE VM: never leave a stale "aws" tag on it (CI hides
|
|
# an AWS override from this flow, so it may be the override replaced).
|
|
"active_override_provider": None,
|
|
"deploy_in_progress": False,
|
|
"deploy_in_progress_since": None,
|
|
}
|
|
old_override = config.get("active_override_vm")
|
|
if old_override and old_override != green["vm"] and not _is_aws_override(config):
|
|
drain.append({
|
|
"vm": old_override,
|
|
"ip": config.get("active_override_ip"),
|
|
"zone": config.get("active_override_zone"),
|
|
})
|
|
return _finalize(updates)
|