Files
karaoke-gen/backend/services/encoding_errors.py
T
Andrew Beveridge ☄️andClaude Opus 5.5 de5dfcd50d feat(encoding): AWS EC2 encoding worker as a pool candidate (v0.288.0) (#1236)
## Summary
Phase 1 of moving encoding to AWS. EC2 is paid for by the AWS Activate
credits (acct 790139457751, expiring 2028-10-31).
- **Plan:** `docs/archive/2026-10-29-aws-encoding-worker-plan.md`.
- **Benchmark** (3 real prod jobs replayed): an EC2 c7a.4xlarge matched
or beat GCE c4d-highcpu-16.

The EC2 box runs the **unchanged** `gce_encoding` worker service and
joins the existing candidate pool as `provider: aws`. The existing
failover, active override, lost-job detection and stockout cooldowns all
apply to it as they are.

## Changes
- **`backend/services/aws_encoding_compute.py` (new):**
`AwsComputeAdapter` looks to the manager like the
`compute_v1.InstancesClient` methods it uses.
  - EC2 states map to GCE statuses.
  - EC2 capacity errors are added to `CAPACITY_ERROR_CODES`.
  - A missing instance raises `NotFound`.
- Controller credentials come from Secret Manager
`aws-encoding-controller-credentials`.
- **`EncodingWorkerManager`:**
- `_compute_for(vm)` sends AWS VMs to the adapter; the GCE paths are
untouched.
- A shared `parse_fallback_vms` passes `provider`/`instance_id`/`region`
through.
- `ranked_candidates()` is used by both the encode path and the warmups.
- `ensure_preferred_running()` behaves **exactly like
`ensure_primary_running` whenever the GCE primary ranks first**, which
is today's case.
  - An AWS control-plane fault fails over to the GCE candidates.
- **Ranking:** `SPEED_RANK["c7a"] = 90`, so AWS is the last resort by
default. Firestore `config/encoding-worker.speed_rank_overrides =
{"c7a": 5}` makes it preferred, with no deploy.
- **Isolation from GCE-only tooling:**
  - The override now records `active_override_provider`.
- The CI blue-green deploy, `deploy_promote` and the idle Cloud Function
all ignore the AWS worker; it hot-updates its wheel per job.
  - A GCE promotion clears any stale provider tag.
- **Worker:**
- Uses `gcloud storage` on AWS, because gsutil can't use WIF credential
files. GCE keeps gsutil.
  - `infrastructure/encoding-worker/aws/` adds:
    - `provision.sh`, as EC2 user-data;
    - `startup-aws.sh`, uploaded by CI;
- `idle-watchdog.py`, an idle self-stop that runs as its own systemd
timer.
- **Infra (GCP), already applied with `/gcp-admin`:**
- The WIF pool `aws-encoding`, plus an AWS provider that only admits the
`nomad-encoding-worker` role, federated to `encoding-worker@`.
  - The secret `aws-encoding-controller-credentials`.
  - Preview and apply: 4 creates, 288 unchanged.
- **Infra (AWS):** a new Pulumi project `infrastructure/aws` (stack
`nomadkaraoke/prod`):
- the c7a.8xlarge **persistent Spot** worker, which stops on
interruption or shutdown;
  - an Elastic IP;
  - a security group on 8080 (API key, same as the GCE firewall);
  - the worker role with SSM;
  - a tag-scoped `nomad-encoding-controller` user.
- Preview: 10 creates. **I apply it after merge**, because the box
provisions from the deployed wheel and boot scripts.
- `boto3` is now an explicit dependency.
- CI also runs the `encoding_worker_idle` function tests, in the job
renamed "Infra - Cloud Function tests".

## Rollout (after merge)
1. **Stage A:** `pulumi up` the AWS stack. Store the controller key in
the secret, then add the worker's entry to
`encoding-worker-fallback-vms`. It's inert unless both c4d and c2d stock
out.
2. **Stage B:** a supervised `speed_rank_overrides {"c7a": 5}` window.
Run real jobs, then compare timings and outputs, including a visual
check of the portrait video.
3. **Stage C:** keep it preferred. After a clean week, delete the c4d
pair.

Phase 2, running distribution (Dropbox/YouTube/Drive uploads) from AWS,
follows in a separate PR.

## Testing
- **New tests:**
- adapter: botocore `Stubber`, covering the state map, capacity and
NotFound handling;
  - manager routing and ranking with overrides;
  - `ensure_preferred_running` branches;
  - failover on an AWS control-plane fault;
  - provider tagging;
  - deploy_promote exclusions;
  - the idle function skipping AWS;
  - watchdog decisions;
  - the gcs CLI choice.
- **Updated** the candidate and warmup-route tests to the new seams.
- **Full backend + unit suites** have no new failures. The local
emulator and cross-suite failures are identical on main.
- **Local CodeRabbit:** 2 findings, both fixed (failover on AWS faults;
stale provider tag).

@coderabbitai ignore

🤖 Generated with [Claude Code](https://claude.com/claude-code)

https://claude.ai/code/session_012RYLVSR5qgwsxTQtvGimKF

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-29 17:42:35 -04:00

151 lines
5.8 KiB
Python

"""Typed exceptions for encoding worker lifecycle failures.
These let callers distinguish between transient capacity issues (retry will
likely help) and unexpected start failures (something is actually broken).
"""
from typing import FrozenSet
CAPACITY_ERROR_CODES: FrozenSet[str] = frozenset({
"ZONE_RESOURCE_POOL_EXHAUSTED",
"ZONE_RESOURCE_POOL_EXHAUSTED_WITH_DETAILS",
"STOCKOUT",
"QUOTA_EXCEEDED",
# AWS EC2 (the AWS encoding worker; see aws_encoding_compute.py)
"InsufficientInstanceCapacity",
"InsufficientHostCapacity",
"InsufficientCapacity",
"VcpuLimitExceeded",
"MaxSpotInstanceCountExceeded",
})
# Sentinel code stamped on an EncodingWorkerInfraError so log filters and the
# park/retry path can tell a runtime worker-infra failure apart from a real
# GCE VM-start failure.
WORKER_INFRA_FAILURE_CODE = "worker_infra_failure"
# Substrings (matched case-insensitively) that identify a worker-side *infrastructure*
# failure — the VM booted and accepted the job but then could not talk to GCP to do
# real work. The canonical case (job 6452888e, 2026-08-31) was a fallback VM that
# could not reach the metadata server to fetch its default service-account token:
# "Failed to retrieve https://metadata.google.internal/.../service-accounts/default/
# ... Compute Engine Metadata server unavailable ... SSLCertVerificationError ..."
# These are properties of the *VM*, not the encode job — the same work will succeed on
# a healthy worker, so they must be retried/re-dispatched rather than failing the job.
# Kept deliberately specific so a genuine encode error (bad codec, corrupt input) never
# matches. Compared lowercase.
WORKER_INFRA_ERROR_MARKERS: FrozenSet[str] = frozenset({
"metadata.google.internal",
"metadata server unavailable",
"compute engine metadata",
"service-accounts/default",
"sslcertverificationerror",
"certificate_verify_failed",
"could not automatically determine credentials",
"defaultcredentialserror",
})
def is_worker_infra_error(message: str) -> bool:
"""True if a worker-reported error string looks like a VM-level infra/auth
failure (metadata server unreachable, SSL/cert, missing credentials) rather
than a genuine problem with the encode job itself."""
if not message:
return False
lowered = message.lower()
return any(marker in lowered for marker in WORKER_INFRA_ERROR_MARKERS)
class EncodingWorkerStartError(Exception):
"""Raised when an attempt to start an encoding worker VM fails.
Carries the GCE error code so callers can react to specific failure modes.
"""
def __init__(
self,
message: str,
*,
vm_name: str = "",
zone: str = "",
code: str = "",
) -> None:
super().__init__(message)
self.vm_name = vm_name
self.zone = zone
self.code = code
class EncodingWorkerCapacityError(EncodingWorkerStartError):
"""A capacity-related GCE failure (e.g. ZONE_RESOURCE_POOL_EXHAUSTED).
The zone temporarily cannot allocate the requested machine type. Callers
should treat this as recoverable — retrying after a wait, or trying an
alternate zone, is likely to succeed.
"""
class EncodingWorkerInfraError(EncodingWorkerStartError):
"""A worker VM accepted the job but then hit a VM-level infrastructure/auth
failure mid-run (e.g. it could not reach the GCE metadata server to fetch its
service-account token — job 6452888e, 2026-08-31).
Subclasses ``EncodingWorkerStartError`` on purpose: the failure is a property
of the *worker*, not the encode job, so it should flow through the same
recoverable path as a VM-start failure (the render worker parks the job for
auto-retry; the final-encode Cloud Run Job retries). Before raising this,
callers demote the offending VM so the retry lands on a healthy worker rather
than looping on the broken one.
"""
# Marker the encoding worker writes into a job's status when it was interrupted
# by a worker-process restart (OOM, deploy, crash). Kept in sync with
# gce_encoding/persistence.py `_RESTART_FAIL_CODE`.
ENCODING_RESTART_FAILURE_CODE = "encoding_worker_restart"
class EncodingJobNotFoundError(RuntimeError):
"""The encoding worker returned HTTP 404 for a job status poll.
Subclasses RuntimeError so existing `except RuntimeError` handlers and
`"not found" in str(e)` checks keep working, while giving callers a precise
type to branch on (e.g. distinguishing a lost job from other RuntimeErrors)
without parsing the message text. Keep the message containing "not found".
"""
class EncodingJobLostError(Exception):
"""The encoding worker lost a job mid-run (restart wiped its in-memory state).
Distinct from a transient poll blip: the ffmpeg process and temp work dir
are gone, so the *same* job will never complete. The only recovery is to
resubmit the work as a fresh job. Deliberately NOT a subclass of
RuntimeError so it escapes the poll-failure tolerance in
`wait_for_completion` instead of being swallowed as a retryable blip.
"""
def __init__(self, message: str, *, job_id: str = "") -> None:
super().__init__(message)
self.job_id = job_id
def classify_gce_error(code: str, message: str, *, vm_name: str, zone: str) -> EncodingWorkerStartError:
"""Wrap a GCE error code/message in the appropriate typed exception."""
if code in CAPACITY_ERROR_CODES:
return EncodingWorkerCapacityError(
f"VM {vm_name} could not be started in {zone}: {code} — {message}",
vm_name=vm_name,
zone=zone,
code=code,
)
return EncodingWorkerStartError(
f"VM {vm_name} start failed in {zone}: {code or 'unknown error'} — {message or 'no message'}",
vm_name=vm_name,
zone=zone,
code=code,
)