mirror of
https://github.com/nomadkaraoke/karaoke-gen.git
synced 2026-11-20 13:07:57 +09:00
## Summary
Phase 1 of moving encoding to AWS. EC2 is paid for by the AWS Activate
credits (acct 790139457751, expiring 2028-10-31).
- **Plan:** `docs/archive/2026-10-29-aws-encoding-worker-plan.md`.
- **Benchmark** (3 real prod jobs replayed): an EC2 c7a.4xlarge matched
or beat GCE c4d-highcpu-16.
The EC2 box runs the **unchanged** `gce_encoding` worker service and
joins the existing candidate pool as `provider: aws`. The existing
failover, active override, lost-job detection and stockout cooldowns all
apply to it as they are.
## Changes
- **`backend/services/aws_encoding_compute.py` (new):**
`AwsComputeAdapter` looks to the manager like the
`compute_v1.InstancesClient` methods it uses.
- EC2 states map to GCE statuses.
- EC2 capacity errors are added to `CAPACITY_ERROR_CODES`.
- A missing instance raises `NotFound`.
- Controller credentials come from Secret Manager
`aws-encoding-controller-credentials`.
- **`EncodingWorkerManager`:**
- `_compute_for(vm)` sends AWS VMs to the adapter; the GCE paths are
untouched.
- A shared `parse_fallback_vms` passes `provider`/`instance_id`/`region`
through.
- `ranked_candidates()` is used by both the encode path and the warmups.
- `ensure_preferred_running()` behaves **exactly like
`ensure_primary_running` whenever the GCE primary ranks first**, which
is today's case.
- An AWS control-plane fault fails over to the GCE candidates.
- **Ranking:** `SPEED_RANK["c7a"] = 90`, so AWS is the last resort by
default. Firestore `config/encoding-worker.speed_rank_overrides =
{"c7a": 5}` makes it preferred, with no deploy.
- **Isolation from GCE-only tooling:**
- The override now records `active_override_provider`.
- The CI blue-green deploy, `deploy_promote` and the idle Cloud Function
all ignore the AWS worker; it hot-updates its wheel per job.
- A GCE promotion clears any stale provider tag.
- **Worker:**
- Uses `gcloud storage` on AWS, because gsutil can't use WIF credential
files. GCE keeps gsutil.
- `infrastructure/encoding-worker/aws/` adds:
- `provision.sh`, as EC2 user-data;
- `startup-aws.sh`, uploaded by CI;
- `idle-watchdog.py`, an idle self-stop that runs as its own systemd
timer.
- **Infra (GCP), already applied with `/gcp-admin`:**
- The WIF pool `aws-encoding`, plus an AWS provider that only admits the
`nomad-encoding-worker` role, federated to `encoding-worker@`.
- The secret `aws-encoding-controller-credentials`.
- Preview and apply: 4 creates, 288 unchanged.
- **Infra (AWS):** a new Pulumi project `infrastructure/aws` (stack
`nomadkaraoke/prod`):
- the c7a.8xlarge **persistent Spot** worker, which stops on
interruption or shutdown;
- an Elastic IP;
- a security group on 8080 (API key, same as the GCE firewall);
- the worker role with SSM;
- a tag-scoped `nomad-encoding-controller` user.
- Preview: 10 creates. **I apply it after merge**, because the box
provisions from the deployed wheel and boot scripts.
- `boto3` is now an explicit dependency.
- CI also runs the `encoding_worker_idle` function tests, in the job
renamed "Infra - Cloud Function tests".
## Rollout (after merge)
1. **Stage A:** `pulumi up` the AWS stack. Store the controller key in
the secret, then add the worker's entry to
`encoding-worker-fallback-vms`. It's inert unless both c4d and c2d stock
out.
2. **Stage B:** a supervised `speed_rank_overrides {"c7a": 5}` window.
Run real jobs, then compare timings and outputs, including a visual
check of the portrait video.
3. **Stage C:** keep it preferred. After a clean week, delete the c4d
pair.
Phase 2, running distribution (Dropbox/YouTube/Drive uploads) from AWS,
follows in a separate PR.
## Testing
- **New tests:**
- adapter: botocore `Stubber`, covering the state map, capacity and
NotFound handling;
- manager routing and ranking with overrides;
- `ensure_preferred_running` branches;
- failover on an AWS control-plane fault;
- provider tagging;
- deploy_promote exclusions;
- the idle function skipping AWS;
- watchdog decisions;
- the gcs CLI choice.
- **Updated** the candidate and warmup-route tests to the new seams.
- **Full backend + unit suites** have no new failures. The local
emulator and cross-suite failures are identical on main.
- **Local CodeRabbit:** 2 findings, both fixed (failover on AWS faults;
stale provider tag).
@coderabbitai ignore
🤖 Generated with [Claude Code](https://claude.com/claude-code)
https://claude.ai/code/session_012RYLVSR5qgwsxTQtvGimKF
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
258 lines
9.9 KiB
Python
258 lines
9.9 KiB
Python
"""
|
|
Encoding Worker Idle Shutdown Cloud Function.
|
|
|
|
Triggered by Cloud Scheduler every 2 minutes. Checks all encoding worker
|
|
VMs (primary, secondary, AND multi-zone fallbacks) and stops any that
|
|
are idle. Without checking the fallbacks, a fallback VM started during a
|
|
capacity event could be left running indefinitely after the event clears
|
|
— observed 2026-05-07 with `encoding-worker-fallback-a` running 13+ hours
|
|
idle (~$15 cost) because the previous version of this function only
|
|
iterated primary/secondary in the default zone.
|
|
|
|
Idle criteria (all must be true to stop a VM):
|
|
- No active encoding jobs (from /health endpoint active_jobs)
|
|
- No recent user activity (from Firestore last_activity_at > IDLE_TIMEOUT_MINUTES)
|
|
- Only applies to the *currently routed* VM: `active_override_vm` if
|
|
set, else primary. Secondary and non-active fallbacks always stop on
|
|
idle so they don't leak after a capacity event.
|
|
- No deploy in progress (from Firestore deploy_in_progress flag)
|
|
|
|
Environment variables:
|
|
- GCP_PROJECT: GCP project ID (default: nomadkaraoke)
|
|
- GCP_ZONE: Default zone for primary/secondary (default: us-central1-c)
|
|
- IDLE_TIMEOUT_MINUTES: Minutes of inactivity before shutdown (default: 5;
|
|
Pulumi sets it from EncodingWorkerConfig.IDLE_TIMEOUT_MINUTES)
|
|
- ENCODING_WORKER_FALLBACK_VMS: JSON list of {vm, zone, ip} for fallback
|
|
VMs in alternate zones. Same secret the backend uses for multi-zone
|
|
failover. Empty/missing → no fallbacks checked.
|
|
"""
|
|
|
|
import json
|
|
import logging
|
|
import os
|
|
import time
|
|
from datetime import datetime, timezone, timedelta
|
|
|
|
import functions_framework
|
|
import requests
|
|
from google.cloud import compute_v1, firestore
|
|
|
|
logger = logging.getLogger(__name__)
|
|
|
|
PROJECT_ID = os.environ.get("GCP_PROJECT", "nomadkaraoke")
|
|
ZONE = os.environ.get("GCP_ZONE", "us-central1-c")
|
|
IDLE_TIMEOUT_MINUTES = int(os.environ.get("IDLE_TIMEOUT_MINUTES", "5"))
|
|
DEPLOY_STALE_TIMEOUT_MINUTES = 20
|
|
ENCODING_WORKER_PORT = 8080
|
|
|
|
_compute_client = None
|
|
_firestore_client = None
|
|
|
|
|
|
def get_compute_client():
|
|
global _compute_client
|
|
if _compute_client is None:
|
|
_compute_client = compute_v1.InstancesClient()
|
|
return _compute_client
|
|
|
|
|
|
def get_firestore_client():
|
|
global _firestore_client
|
|
if _firestore_client is None:
|
|
_firestore_client = firestore.Client(project=PROJECT_ID)
|
|
return _firestore_client
|
|
|
|
|
|
def get_vm_status(vm_name, zone=ZONE):
|
|
client = get_compute_client()
|
|
try:
|
|
instance = client.get(project=PROJECT_ID, zone=zone, instance=vm_name)
|
|
return instance.status
|
|
except Exception as e:
|
|
logger.error(f"Failed to get status for {vm_name} in {zone}: {e}")
|
|
return "UNKNOWN"
|
|
|
|
|
|
def get_vm_ip(vm_name, zone=ZONE):
|
|
client = get_compute_client()
|
|
try:
|
|
instance = client.get(project=PROJECT_ID, zone=zone, instance=vm_name)
|
|
for iface in instance.network_interfaces:
|
|
for access in iface.access_configs:
|
|
if access.nat_i_p:
|
|
return access.nat_i_p
|
|
except Exception:
|
|
pass
|
|
return None
|
|
|
|
|
|
def check_active_jobs(vm_ip, attempts=3):
|
|
"""Return the worker's active_jobs count, or None if it can't be confirmed.
|
|
|
|
Returns None (NOT 0) on any failure — connection error, timeout, non-200,
|
|
or unparseable body — so the caller can fail SAFE. A busy encoder running a
|
|
4K render pins all 32 cores and shares uvicorn with the encode subprocess,
|
|
so /health can miss the 5s window even though the VM is hard at work.
|
|
Treating that unreachable/slow /health as "0 active jobs" previously let the
|
|
idle check stop a VM mid-render, orphaning the in-memory job and surfacing
|
|
as "lost contact with worker ... not found" (incident 2026-06-15).
|
|
|
|
A genuinely idle VM answers /health quickly, so the retry loop is cheap and
|
|
a truly-idle VM is still reclaimed; only ambiguous (busy/unreachable) VMs
|
|
are kept alive for this cycle.
|
|
"""
|
|
last_err = None
|
|
for attempt in range(attempts):
|
|
try:
|
|
resp = requests.get(f"http://{vm_ip}:{ENCODING_WORKER_PORT}/health", timeout=5)
|
|
if resp.status_code == 200:
|
|
return int(resp.json().get("active_jobs", 0))
|
|
last_err = f"HTTP {resp.status_code}"
|
|
except Exception as e: # noqa: BLE001 - any failure means "unconfirmed"
|
|
last_err = f"{type(e).__name__}: {e}"
|
|
if attempt < attempts - 1:
|
|
time.sleep(2)
|
|
logger.warning(
|
|
f"Could not confirm active_jobs for {vm_ip} after {attempts} attempts "
|
|
f"({last_err}); treating as unknown (assume busy)"
|
|
)
|
|
return None
|
|
|
|
|
|
def stop_vm(vm_name, zone=ZONE):
|
|
client = get_compute_client()
|
|
logger.info(f"Stopping idle VM: {vm_name} in {zone}")
|
|
client.stop(project=PROJECT_ID, zone=zone, instance=vm_name)
|
|
|
|
|
|
def _parse_fallback_vms():
|
|
"""Return list of (vm_name, zone) tuples from ENCODING_WORKER_FALLBACK_VMS env.
|
|
|
|
Schema mirrors the backend: JSON list of {"vm","zone","ip"}. Returns
|
|
empty list when missing or malformed (worst case: we don't check
|
|
fallbacks, primary/secondary still get checked).
|
|
"""
|
|
raw = os.environ.get("ENCODING_WORKER_FALLBACK_VMS", "")
|
|
if not raw:
|
|
return []
|
|
try:
|
|
parsed = json.loads(raw)
|
|
except (ValueError, TypeError) as exc:
|
|
logger.warning(f"Invalid ENCODING_WORKER_FALLBACK_VMS JSON: {exc}")
|
|
return []
|
|
|
|
fallbacks = []
|
|
for item in parsed:
|
|
try:
|
|
if (item.get("provider") or "").lower() == "aws":
|
|
# The AWS worker is not a GCE VM; it idle-stops itself
|
|
# (infrastructure/encoding-worker/aws/idle-watchdog.py).
|
|
continue
|
|
fallbacks.append((item["vm"], item["zone"]))
|
|
except (KeyError, TypeError) as exc:
|
|
logger.warning(f"Skipping malformed fallback entry {item}: {exc}")
|
|
continue
|
|
return fallbacks
|
|
|
|
|
|
@functions_framework.http
|
|
def idle_shutdown(request):
|
|
now = datetime.now(timezone.utc)
|
|
idle_cutoff = now - timedelta(minutes=IDLE_TIMEOUT_MINUTES)
|
|
stale_cutoff = now - timedelta(minutes=DEPLOY_STALE_TIMEOUT_MINUTES)
|
|
|
|
db = get_firestore_client()
|
|
doc = db.collection("config").document("encoding-worker").get()
|
|
if not doc.exists:
|
|
logger.warning("Encoding worker config not found in Firestore")
|
|
return json.dumps({"status": "no_config"}), 200
|
|
|
|
config = doc.to_dict()
|
|
|
|
# Check deploy_in_progress
|
|
deploy_in_progress = config.get("deploy_in_progress", False)
|
|
if deploy_in_progress:
|
|
deploy_since = config.get("deploy_in_progress_since")
|
|
if deploy_since:
|
|
deploy_time = datetime.fromisoformat(deploy_since)
|
|
if deploy_time < stale_cutoff:
|
|
logger.warning(
|
|
f"Clearing stale deploy_in_progress flag (started {deploy_since})"
|
|
)
|
|
db.collection("config").document("encoding-worker").update({
|
|
"deploy_in_progress": False,
|
|
"deploy_in_progress_since": None,
|
|
})
|
|
deploy_in_progress = False
|
|
else:
|
|
logger.info("Deploy in progress, skipping idle check")
|
|
return json.dumps({"status": "deploy_in_progress"}), 200
|
|
|
|
# Parse last_activity_at
|
|
last_activity = config.get("last_activity_at")
|
|
has_recent_activity = False
|
|
if last_activity:
|
|
activity_time = datetime.fromisoformat(last_activity)
|
|
has_recent_activity = activity_time > idle_cutoff
|
|
|
|
# The currently-routed VM gets keep-alive-on-recent-activity treatment:
|
|
# capacity-fallback override wins, else primary. Secondary and non-active
|
|
# fallbacks always stop when idle so we don't leak them after a capacity
|
|
# event clears.
|
|
primary_vm = config.get("primary_vm")
|
|
active_override_vm = config.get("active_override_vm")
|
|
active_vm = active_override_vm or primary_vm
|
|
|
|
# Build the full candidate list: primary + secondary in default zone,
|
|
# plus each multi-zone fallback in its own zone.
|
|
candidates = [
|
|
(primary_vm, ZONE),
|
|
(config.get("secondary_vm"), ZONE),
|
|
]
|
|
candidates.extend(_parse_fallback_vms())
|
|
|
|
results = {}
|
|
for vm_name, vm_zone in candidates:
|
|
if not vm_name:
|
|
continue
|
|
|
|
status = get_vm_status(vm_name, vm_zone)
|
|
if status != "RUNNING":
|
|
results[vm_name] = f"already_{status.lower()}"
|
|
continue
|
|
|
|
# Fail SAFE: never stop a VM unless /health gives us a *confirmed*
|
|
# active_jobs == 0. If we can't reach the VM or confirm its job count,
|
|
# keep it alive this cycle (the function reruns every 2 min, so a
|
|
# genuinely idle VM is reclaimed once /health responds). This prevents
|
|
# stopping a VM mid-render — the cause of "lost contact with worker"
|
|
# (incident 2026-06-15).
|
|
vm_ip = get_vm_ip(vm_name, vm_zone)
|
|
if not vm_ip:
|
|
results[vm_name] = "no_ip_kept_alive"
|
|
logger.warning(f"{vm_name} is RUNNING but has no reachable IP; keeping alive (fail-safe)")
|
|
continue
|
|
|
|
active_jobs = check_active_jobs(vm_ip)
|
|
if active_jobs is None:
|
|
results[vm_name] = "health_unknown_kept_alive"
|
|
logger.warning(f"{vm_name} /health unconfirmed; keeping alive (fail-safe)")
|
|
continue
|
|
if active_jobs > 0:
|
|
results[vm_name] = f"active_jobs={active_jobs}"
|
|
logger.info(f"{vm_name} has {active_jobs} active jobs, keeping alive")
|
|
continue
|
|
|
|
# active_jobs confirmed 0. Only the currently-routed VM (primary, or
|
|
# active_override fallback) gets the extra recent-activity keep-alive.
|
|
# Other VMs stop on idle.
|
|
if vm_name == active_vm and has_recent_activity:
|
|
results[vm_name] = "active_session"
|
|
logger.info(f"{vm_name} (active) kept alive due to recent activity")
|
|
continue
|
|
|
|
stop_vm(vm_name, vm_zone)
|
|
results[vm_name] = "stopped"
|
|
|
|
return json.dumps({"status": "checked", "results": results}), 200
|