Files
karaoke-gen/infrastructure/functions/encoding_worker_idle/main.py
T
Andrew Beveridge ☄️andClaude Opus 5.5 de5dfcd50d feat(encoding): AWS EC2 encoding worker as a pool candidate (v0.288.0) (#1236)
## Summary
Phase 1 of moving encoding to AWS. EC2 is paid for by the AWS Activate
credits (acct 790139457751, expiring 2028-10-31).
- **Plan:** `docs/archive/2026-10-29-aws-encoding-worker-plan.md`.
- **Benchmark** (3 real prod jobs replayed): an EC2 c7a.4xlarge matched
or beat GCE c4d-highcpu-16.

The EC2 box runs the **unchanged** `gce_encoding` worker service and
joins the existing candidate pool as `provider: aws`. The existing
failover, active override, lost-job detection and stockout cooldowns all
apply to it as they are.

## Changes
- **`backend/services/aws_encoding_compute.py` (new):**
`AwsComputeAdapter` looks to the manager like the
`compute_v1.InstancesClient` methods it uses.
  - EC2 states map to GCE statuses.
  - EC2 capacity errors are added to `CAPACITY_ERROR_CODES`.
  - A missing instance raises `NotFound`.
- Controller credentials come from Secret Manager
`aws-encoding-controller-credentials`.
- **`EncodingWorkerManager`:**
- `_compute_for(vm)` sends AWS VMs to the adapter; the GCE paths are
untouched.
- A shared `parse_fallback_vms` passes `provider`/`instance_id`/`region`
through.
- `ranked_candidates()` is used by both the encode path and the warmups.
- `ensure_preferred_running()` behaves **exactly like
`ensure_primary_running` whenever the GCE primary ranks first**, which
is today's case.
  - An AWS control-plane fault fails over to the GCE candidates.
- **Ranking:** `SPEED_RANK["c7a"] = 90`, so AWS is the last resort by
default. Firestore `config/encoding-worker.speed_rank_overrides =
{"c7a": 5}` makes it preferred, with no deploy.
- **Isolation from GCE-only tooling:**
  - The override now records `active_override_provider`.
- The CI blue-green deploy, `deploy_promote` and the idle Cloud Function
all ignore the AWS worker; it hot-updates its wheel per job.
  - A GCE promotion clears any stale provider tag.
- **Worker:**
- Uses `gcloud storage` on AWS, because gsutil can't use WIF credential
files. GCE keeps gsutil.
  - `infrastructure/encoding-worker/aws/` adds:
    - `provision.sh`, as EC2 user-data;
    - `startup-aws.sh`, uploaded by CI;
- `idle-watchdog.py`, an idle self-stop that runs as its own systemd
timer.
- **Infra (GCP), already applied with `/gcp-admin`:**
- The WIF pool `aws-encoding`, plus an AWS provider that only admits the
`nomad-encoding-worker` role, federated to `encoding-worker@`.
  - The secret `aws-encoding-controller-credentials`.
  - Preview and apply: 4 creates, 288 unchanged.
- **Infra (AWS):** a new Pulumi project `infrastructure/aws` (stack
`nomadkaraoke/prod`):
- the c7a.8xlarge **persistent Spot** worker, which stops on
interruption or shutdown;
  - an Elastic IP;
  - a security group on 8080 (API key, same as the GCE firewall);
  - the worker role with SSM;
  - a tag-scoped `nomad-encoding-controller` user.
- Preview: 10 creates. **I apply it after merge**, because the box
provisions from the deployed wheel and boot scripts.
- `boto3` is now an explicit dependency.
- CI also runs the `encoding_worker_idle` function tests, in the job
renamed "Infra - Cloud Function tests".

## Rollout (after merge)
1. **Stage A:** `pulumi up` the AWS stack. Store the controller key in
the secret, then add the worker's entry to
`encoding-worker-fallback-vms`. It's inert unless both c4d and c2d stock
out.
2. **Stage B:** a supervised `speed_rank_overrides {"c7a": 5}` window.
Run real jobs, then compare timings and outputs, including a visual
check of the portrait video.
3. **Stage C:** keep it preferred. After a clean week, delete the c4d
pair.

Phase 2, running distribution (Dropbox/YouTube/Drive uploads) from AWS,
follows in a separate PR.

## Testing
- **New tests:**
- adapter: botocore `Stubber`, covering the state map, capacity and
NotFound handling;
  - manager routing and ranking with overrides;
  - `ensure_preferred_running` branches;
  - failover on an AWS control-plane fault;
  - provider tagging;
  - deploy_promote exclusions;
  - the idle function skipping AWS;
  - watchdog decisions;
  - the gcs CLI choice.
- **Updated** the candidate and warmup-route tests to the new seams.
- **Full backend + unit suites** have no new failures. The local
emulator and cross-suite failures are identical on main.
- **Local CodeRabbit:** 2 findings, both fixed (failover on AWS faults;
stale provider tag).

@coderabbitai ignore

🤖 Generated with [Claude Code](https://claude.com/claude-code)

https://claude.ai/code/session_012RYLVSR5qgwsxTQtvGimKF

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-29 17:42:35 -04:00

258 lines
9.9 KiB
Python

"""
Encoding Worker Idle Shutdown Cloud Function.
Triggered by Cloud Scheduler every 2 minutes. Checks all encoding worker
VMs (primary, secondary, AND multi-zone fallbacks) and stops any that
are idle. Without checking the fallbacks, a fallback VM started during a
capacity event could be left running indefinitely after the event clears
— observed 2026-05-07 with `encoding-worker-fallback-a` running 13+ hours
idle (~$15 cost) because the previous version of this function only
iterated primary/secondary in the default zone.
Idle criteria (all must be true to stop a VM):
- No active encoding jobs (from /health endpoint active_jobs)
- No recent user activity (from Firestore last_activity_at > IDLE_TIMEOUT_MINUTES)
- Only applies to the *currently routed* VM: `active_override_vm` if
set, else primary. Secondary and non-active fallbacks always stop on
idle so they don't leak after a capacity event.
- No deploy in progress (from Firestore deploy_in_progress flag)
Environment variables:
- GCP_PROJECT: GCP project ID (default: nomadkaraoke)
- GCP_ZONE: Default zone for primary/secondary (default: us-central1-c)
- IDLE_TIMEOUT_MINUTES: Minutes of inactivity before shutdown (default: 5;
Pulumi sets it from EncodingWorkerConfig.IDLE_TIMEOUT_MINUTES)
- ENCODING_WORKER_FALLBACK_VMS: JSON list of {vm, zone, ip} for fallback
VMs in alternate zones. Same secret the backend uses for multi-zone
failover. Empty/missing → no fallbacks checked.
"""
import json
import logging
import os
import time
from datetime import datetime, timezone, timedelta
import functions_framework
import requests
from google.cloud import compute_v1, firestore
logger = logging.getLogger(__name__)
PROJECT_ID = os.environ.get("GCP_PROJECT", "nomadkaraoke")
ZONE = os.environ.get("GCP_ZONE", "us-central1-c")
IDLE_TIMEOUT_MINUTES = int(os.environ.get("IDLE_TIMEOUT_MINUTES", "5"))
DEPLOY_STALE_TIMEOUT_MINUTES = 20
ENCODING_WORKER_PORT = 8080
_compute_client = None
_firestore_client = None
def get_compute_client():
global _compute_client
if _compute_client is None:
_compute_client = compute_v1.InstancesClient()
return _compute_client
def get_firestore_client():
global _firestore_client
if _firestore_client is None:
_firestore_client = firestore.Client(project=PROJECT_ID)
return _firestore_client
def get_vm_status(vm_name, zone=ZONE):
client = get_compute_client()
try:
instance = client.get(project=PROJECT_ID, zone=zone, instance=vm_name)
return instance.status
except Exception as e:
logger.error(f"Failed to get status for {vm_name} in {zone}: {e}")
return "UNKNOWN"
def get_vm_ip(vm_name, zone=ZONE):
client = get_compute_client()
try:
instance = client.get(project=PROJECT_ID, zone=zone, instance=vm_name)
for iface in instance.network_interfaces:
for access in iface.access_configs:
if access.nat_i_p:
return access.nat_i_p
except Exception:
pass
return None
def check_active_jobs(vm_ip, attempts=3):
"""Return the worker's active_jobs count, or None if it can't be confirmed.
Returns None (NOT 0) on any failure — connection error, timeout, non-200,
or unparseable body — so the caller can fail SAFE. A busy encoder running a
4K render pins all 32 cores and shares uvicorn with the encode subprocess,
so /health can miss the 5s window even though the VM is hard at work.
Treating that unreachable/slow /health as "0 active jobs" previously let the
idle check stop a VM mid-render, orphaning the in-memory job and surfacing
as "lost contact with worker ... not found" (incident 2026-06-15).
A genuinely idle VM answers /health quickly, so the retry loop is cheap and
a truly-idle VM is still reclaimed; only ambiguous (busy/unreachable) VMs
are kept alive for this cycle.
"""
last_err = None
for attempt in range(attempts):
try:
resp = requests.get(f"http://{vm_ip}:{ENCODING_WORKER_PORT}/health", timeout=5)
if resp.status_code == 200:
return int(resp.json().get("active_jobs", 0))
last_err = f"HTTP {resp.status_code}"
except Exception as e: # noqa: BLE001 - any failure means "unconfirmed"
last_err = f"{type(e).__name__}: {e}"
if attempt < attempts - 1:
time.sleep(2)
logger.warning(
f"Could not confirm active_jobs for {vm_ip} after {attempts} attempts "
f"({last_err}); treating as unknown (assume busy)"
)
return None
def stop_vm(vm_name, zone=ZONE):
client = get_compute_client()
logger.info(f"Stopping idle VM: {vm_name} in {zone}")
client.stop(project=PROJECT_ID, zone=zone, instance=vm_name)
def _parse_fallback_vms():
"""Return list of (vm_name, zone) tuples from ENCODING_WORKER_FALLBACK_VMS env.
Schema mirrors the backend: JSON list of {"vm","zone","ip"}. Returns
empty list when missing or malformed (worst case: we don't check
fallbacks, primary/secondary still get checked).
"""
raw = os.environ.get("ENCODING_WORKER_FALLBACK_VMS", "")
if not raw:
return []
try:
parsed = json.loads(raw)
except (ValueError, TypeError) as exc:
logger.warning(f"Invalid ENCODING_WORKER_FALLBACK_VMS JSON: {exc}")
return []
fallbacks = []
for item in parsed:
try:
if (item.get("provider") or "").lower() == "aws":
# The AWS worker is not a GCE VM; it idle-stops itself
# (infrastructure/encoding-worker/aws/idle-watchdog.py).
continue
fallbacks.append((item["vm"], item["zone"]))
except (KeyError, TypeError) as exc:
logger.warning(f"Skipping malformed fallback entry {item}: {exc}")
continue
return fallbacks
@functions_framework.http
def idle_shutdown(request):
now = datetime.now(timezone.utc)
idle_cutoff = now - timedelta(minutes=IDLE_TIMEOUT_MINUTES)
stale_cutoff = now - timedelta(minutes=DEPLOY_STALE_TIMEOUT_MINUTES)
db = get_firestore_client()
doc = db.collection("config").document("encoding-worker").get()
if not doc.exists:
logger.warning("Encoding worker config not found in Firestore")
return json.dumps({"status": "no_config"}), 200
config = doc.to_dict()
# Check deploy_in_progress
deploy_in_progress = config.get("deploy_in_progress", False)
if deploy_in_progress:
deploy_since = config.get("deploy_in_progress_since")
if deploy_since:
deploy_time = datetime.fromisoformat(deploy_since)
if deploy_time < stale_cutoff:
logger.warning(
f"Clearing stale deploy_in_progress flag (started {deploy_since})"
)
db.collection("config").document("encoding-worker").update({
"deploy_in_progress": False,
"deploy_in_progress_since": None,
})
deploy_in_progress = False
else:
logger.info("Deploy in progress, skipping idle check")
return json.dumps({"status": "deploy_in_progress"}), 200
# Parse last_activity_at
last_activity = config.get("last_activity_at")
has_recent_activity = False
if last_activity:
activity_time = datetime.fromisoformat(last_activity)
has_recent_activity = activity_time > idle_cutoff
# The currently-routed VM gets keep-alive-on-recent-activity treatment:
# capacity-fallback override wins, else primary. Secondary and non-active
# fallbacks always stop when idle so we don't leak them after a capacity
# event clears.
primary_vm = config.get("primary_vm")
active_override_vm = config.get("active_override_vm")
active_vm = active_override_vm or primary_vm
# Build the full candidate list: primary + secondary in default zone,
# plus each multi-zone fallback in its own zone.
candidates = [
(primary_vm, ZONE),
(config.get("secondary_vm"), ZONE),
]
candidates.extend(_parse_fallback_vms())
results = {}
for vm_name, vm_zone in candidates:
if not vm_name:
continue
status = get_vm_status(vm_name, vm_zone)
if status != "RUNNING":
results[vm_name] = f"already_{status.lower()}"
continue
# Fail SAFE: never stop a VM unless /health gives us a *confirmed*
# active_jobs == 0. If we can't reach the VM or confirm its job count,
# keep it alive this cycle (the function reruns every 2 min, so a
# genuinely idle VM is reclaimed once /health responds). This prevents
# stopping a VM mid-render — the cause of "lost contact with worker"
# (incident 2026-06-15).
vm_ip = get_vm_ip(vm_name, vm_zone)
if not vm_ip:
results[vm_name] = "no_ip_kept_alive"
logger.warning(f"{vm_name} is RUNNING but has no reachable IP; keeping alive (fail-safe)")
continue
active_jobs = check_active_jobs(vm_ip)
if active_jobs is None:
results[vm_name] = "health_unknown_kept_alive"
logger.warning(f"{vm_name} /health unconfirmed; keeping alive (fail-safe)")
continue
if active_jobs > 0:
results[vm_name] = f"active_jobs={active_jobs}"
logger.info(f"{vm_name} has {active_jobs} active jobs, keeping alive")
continue
# active_jobs confirmed 0. Only the currently-routed VM (primary, or
# active_override fallback) gets the extra recent-activity keep-alive.
# Other VMs stop on idle.
if vm_name == active_vm and has_recent_activity:
results[vm_name] = "active_session"
logger.info(f"{vm_name} (active) kept alive due to recent activity")
continue
stop_vm(vm_name, vm_zone)
results[vm_name] = "stopped"
return json.dumps({"status": "checked", "results": results}), 200