Files
karaoke-gen/backend/api/routes/encoding_worker.py
T
Andrew Beveridge ☄️andClaude Opus 5.5 de5dfcd50d feat(encoding): AWS EC2 encoding worker as a pool candidate (v0.288.0) (#1236)
## Summary
Phase 1 of moving encoding to AWS. EC2 is paid for by the AWS Activate
credits (acct 790139457751, expiring 2028-10-31).
- **Plan:** `docs/archive/2026-10-29-aws-encoding-worker-plan.md`.
- **Benchmark** (3 real prod jobs replayed): an EC2 c7a.4xlarge matched
or beat GCE c4d-highcpu-16.

The EC2 box runs the **unchanged** `gce_encoding` worker service and
joins the existing candidate pool as `provider: aws`. The existing
failover, active override, lost-job detection and stockout cooldowns all
apply to it as they are.

## Changes
- **`backend/services/aws_encoding_compute.py` (new):**
`AwsComputeAdapter` looks to the manager like the
`compute_v1.InstancesClient` methods it uses.
  - EC2 states map to GCE statuses.
  - EC2 capacity errors are added to `CAPACITY_ERROR_CODES`.
  - A missing instance raises `NotFound`.
- Controller credentials come from Secret Manager
`aws-encoding-controller-credentials`.
- **`EncodingWorkerManager`:**
- `_compute_for(vm)` sends AWS VMs to the adapter; the GCE paths are
untouched.
- A shared `parse_fallback_vms` passes `provider`/`instance_id`/`region`
through.
- `ranked_candidates()` is used by both the encode path and the warmups.
- `ensure_preferred_running()` behaves **exactly like
`ensure_primary_running` whenever the GCE primary ranks first**, which
is today's case.
  - An AWS control-plane fault fails over to the GCE candidates.
- **Ranking:** `SPEED_RANK["c7a"] = 90`, so AWS is the last resort by
default. Firestore `config/encoding-worker.speed_rank_overrides =
{"c7a": 5}` makes it preferred, with no deploy.
- **Isolation from GCE-only tooling:**
  - The override now records `active_override_provider`.
- The CI blue-green deploy, `deploy_promote` and the idle Cloud Function
all ignore the AWS worker; it hot-updates its wheel per job.
  - A GCE promotion clears any stale provider tag.
- **Worker:**
- Uses `gcloud storage` on AWS, because gsutil can't use WIF credential
files. GCE keeps gsutil.
  - `infrastructure/encoding-worker/aws/` adds:
    - `provision.sh`, as EC2 user-data;
    - `startup-aws.sh`, uploaded by CI;
- `idle-watchdog.py`, an idle self-stop that runs as its own systemd
timer.
- **Infra (GCP), already applied with `/gcp-admin`:**
- The WIF pool `aws-encoding`, plus an AWS provider that only admits the
`nomad-encoding-worker` role, federated to `encoding-worker@`.
  - The secret `aws-encoding-controller-credentials`.
  - Preview and apply: 4 creates, 288 unchanged.
- **Infra (AWS):** a new Pulumi project `infrastructure/aws` (stack
`nomadkaraoke/prod`):
- the c7a.8xlarge **persistent Spot** worker, which stops on
interruption or shutdown;
  - an Elastic IP;
  - a security group on 8080 (API key, same as the GCE firewall);
  - the worker role with SSM;
  - a tag-scoped `nomad-encoding-controller` user.
- Preview: 10 creates. **I apply it after merge**, because the box
provisions from the deployed wheel and boot scripts.
- `boto3` is now an explicit dependency.
- CI also runs the `encoding_worker_idle` function tests, in the job
renamed "Infra - Cloud Function tests".

## Rollout (after merge)
1. **Stage A:** `pulumi up` the AWS stack. Store the controller key in
the secret, then add the worker's entry to
`encoding-worker-fallback-vms`. It's inert unless both c4d and c2d stock
out.
2. **Stage B:** a supervised `speed_rank_overrides {"c7a": 5}` window.
Run real jobs, then compare timings and outputs, including a visual
check of the portrait video.
3. **Stage C:** keep it preferred. After a clean week, delete the c4d
pair.

Phase 2, running distribution (Dropbox/YouTube/Drive uploads) from AWS,
follows in a separate PR.

## Testing
- **New tests:**
- adapter: botocore `Stubber`, covering the state map, capacity and
NotFound handling;
  - manager routing and ranking with overrides;
  - `ensure_preferred_running` branches;
  - failover on an AWS control-plane fault;
  - provider tagging;
  - deploy_promote exclusions;
  - the idle function skipping AWS;
  - watchdog decisions;
  - the gcs CLI choice.
- **Updated** the candidate and warmup-route tests to the new seams.
- **Full backend + unit suites** have no new failures. The local
emulator and cross-suite failures are identical on main.
- **Local CodeRabbit:** 2 findings, both fixed (failover on AWS faults;
stale provider tag).

@coderabbitai ignore

🤖 Generated with [Claude Code](https://claude.com/claude-code)

https://claude.ai/code/session_012RYLVSR5qgwsxTQtvGimKF

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-29 17:42:35 -04:00

105 lines
4.2 KiB
Python

"""
Encoding worker lifecycle endpoints.
Provides warmup and heartbeat endpoints for the blue-green
encoding worker VMs. Called by the lyrics-review frontend to ensure the
primary VM is running before the reviewer previews a render.
Auth: these are gated on ``require_review_auth`` (review access to a specific
job), NOT ``require_admin``. The callers are ordinary customers on the
lyrics-review page — gating on admin makes every non-admin caller 403, which
silently defeats the JIT pre-warm. Scoping to the job being reviewed keeps the
abuse posture identical to the rest of the review surface: you can only warm
the shared encoding VM if you already have review access to a real job.
"""
import asyncio
import logging
from typing import Tuple
from fastapi import APIRouter, Depends
from backend.api.dependencies import require_review_auth
from backend.services.encoding_worker_manager import EncodingWorkerManager
from backend.services.encoding_errors import EncodingWorkerStartError
logger = logging.getLogger(__name__)
router = APIRouter(prefix="/internal/encoding-worker", tags=["encoding-worker"])
# Lazy-loaded singleton
_manager = None
def get_worker_manager():
global _manager
if _manager is None:
from google.cloud import compute_v1, firestore
from backend.config import get_settings
settings = get_settings()
db = firestore.Client(project=settings.google_cloud_project)
compute_client = compute_v1.InstancesClient()
_manager = EncodingWorkerManager(
db=db,
compute_client=compute_client,
project_id=settings.google_cloud_project,
)
return _manager
@router.post("/warmup/{job_id}")
async def warmup_encoding_worker(
job_id: str,
_auth: Tuple[str, str] = Depends(require_review_auth),
manager=Depends(get_worker_manager),
):
"""Start the preferred encoding worker if it's stopped (normally the GCE primary).
Called on lyrics-review page load so the VM is warm by the time the
reviewer previews a render. ``job_id`` scopes auth to the job under review.
"""
try:
# ensure_primary_running does Firestore reads + GCE compute-API calls
# (instances.get / instances.start) that can take 10s+. Run it off the
# event loop: on the single-process server a sync call here froze the
# WHOLE instance for ~11s on every review-page open with a cold VM,
# stalling that same page's audio/lyrics requests behind it
# (2026-09-19, follow-up to the concurrent-load investigation).
result = await asyncio.to_thread(manager.ensure_preferred_running)
if result["started"]:
logger.info(f"Started encoding worker VM: {result['vm_name']}")
return result
except EncodingWorkerStartError as e:
# The primary VM couldn't start (capacity exhaustion or a transient
# 503 SERVICE_UNAVAILABLE from the GCE backend). This is NOT fatal:
# the encoding flow's own warmup (ensure_any_running) falls back to
# the alternate-zone workers, so the job still completes. Log at
# WARNING so it doesn't trip the production error monitor and page us
# for self-healing capacity events.
logger.warning(
f"Primary encoding worker warmup failed ({e}); "
f"encoding will fall back to an alternate-zone worker"
)
return {"started": False, "error": str(e)}
except Exception as e:
logger.error(f"Failed to warm up encoding worker: {e}")
return {"started": False, "error": str(e)}
@router.post("/heartbeat/{job_id}")
async def heartbeat_encoding_worker(
job_id: str,
_auth: Tuple[str, str] = Depends(require_review_auth),
manager=Depends(get_worker_manager),
):
"""Update activity timestamp to prevent idle shutdown.
Called periodically while the reviewer works. ``job_id`` scopes auth to the
job under review.
"""
try:
# Sync Firestore write — keep it off the event loop (see warmup above).
await asyncio.to_thread(manager.update_activity)
return {"status": "ok"}
except Exception as e:
logger.error(f"Failed to update encoding worker heartbeat: {e}")
return {"status": "error", "error": str(e)}