mirror of
https://github.com/nomadkaraoke/karaoke-gen.git
synced 2026-11-20 13:07:57 +09:00
## Summary
Phase 1 of moving encoding to AWS. EC2 is paid for by the AWS Activate
credits (acct 790139457751, expiring 2028-10-31).
- **Plan:** `docs/archive/2026-10-29-aws-encoding-worker-plan.md`.
- **Benchmark** (3 real prod jobs replayed): an EC2 c7a.4xlarge matched
or beat GCE c4d-highcpu-16.
The EC2 box runs the **unchanged** `gce_encoding` worker service and
joins the existing candidate pool as `provider: aws`. The existing
failover, active override, lost-job detection and stockout cooldowns all
apply to it as they are.
## Changes
- **`backend/services/aws_encoding_compute.py` (new):**
`AwsComputeAdapter` looks to the manager like the
`compute_v1.InstancesClient` methods it uses.
- EC2 states map to GCE statuses.
- EC2 capacity errors are added to `CAPACITY_ERROR_CODES`.
- A missing instance raises `NotFound`.
- Controller credentials come from Secret Manager
`aws-encoding-controller-credentials`.
- **`EncodingWorkerManager`:**
- `_compute_for(vm)` sends AWS VMs to the adapter; the GCE paths are
untouched.
- A shared `parse_fallback_vms` passes `provider`/`instance_id`/`region`
through.
- `ranked_candidates()` is used by both the encode path and the warmups.
- `ensure_preferred_running()` behaves **exactly like
`ensure_primary_running` whenever the GCE primary ranks first**, which
is today's case.
- An AWS control-plane fault fails over to the GCE candidates.
- **Ranking:** `SPEED_RANK["c7a"] = 90`, so AWS is the last resort by
default. Firestore `config/encoding-worker.speed_rank_overrides =
{"c7a": 5}` makes it preferred, with no deploy.
- **Isolation from GCE-only tooling:**
- The override now records `active_override_provider`.
- The CI blue-green deploy, `deploy_promote` and the idle Cloud Function
all ignore the AWS worker; it hot-updates its wheel per job.
- A GCE promotion clears any stale provider tag.
- **Worker:**
- Uses `gcloud storage` on AWS, because gsutil can't use WIF credential
files. GCE keeps gsutil.
- `infrastructure/encoding-worker/aws/` adds:
- `provision.sh`, as EC2 user-data;
- `startup-aws.sh`, uploaded by CI;
- `idle-watchdog.py`, an idle self-stop that runs as its own systemd
timer.
- **Infra (GCP), already applied with `/gcp-admin`:**
- The WIF pool `aws-encoding`, plus an AWS provider that only admits the
`nomad-encoding-worker` role, federated to `encoding-worker@`.
- The secret `aws-encoding-controller-credentials`.
- Preview and apply: 4 creates, 288 unchanged.
- **Infra (AWS):** a new Pulumi project `infrastructure/aws` (stack
`nomadkaraoke/prod`):
- the c7a.8xlarge **persistent Spot** worker, which stops on
interruption or shutdown;
- an Elastic IP;
- a security group on 8080 (API key, same as the GCE firewall);
- the worker role with SSM;
- a tag-scoped `nomad-encoding-controller` user.
- Preview: 10 creates. **I apply it after merge**, because the box
provisions from the deployed wheel and boot scripts.
- `boto3` is now an explicit dependency.
- CI also runs the `encoding_worker_idle` function tests, in the job
renamed "Infra - Cloud Function tests".
## Rollout (after merge)
1. **Stage A:** `pulumi up` the AWS stack. Store the controller key in
the secret, then add the worker's entry to
`encoding-worker-fallback-vms`. It's inert unless both c4d and c2d stock
out.
2. **Stage B:** a supervised `speed_rank_overrides {"c7a": 5}` window.
Run real jobs, then compare timings and outputs, including a visual
check of the portrait video.
3. **Stage C:** keep it preferred. After a clean week, delete the c4d
pair.
Phase 2, running distribution (Dropbox/YouTube/Drive uploads) from AWS,
follows in a separate PR.
## Testing
- **New tests:**
- adapter: botocore `Stubber`, covering the state map, capacity and
NotFound handling;
- manager routing and ranking with overrides;
- `ensure_preferred_running` branches;
- failover on an AWS control-plane fault;
- provider tagging;
- deploy_promote exclusions;
- the idle function skipping AWS;
- watchdog decisions;
- the gcs CLI choice.
- **Updated** the candidate and warmup-route tests to the new seams.
- **Full backend + unit suites** have no new failures. The local
emulator and cross-suite failures are identical on main.
- **Local CodeRabbit:** 2 findings, both fixed (failover on AWS faults;
stale provider tag).
@coderabbitai ignore
🤖 Generated with [Claude Code](https://claude.com/claude-code)
https://claude.ai/code/session_012RYLVSR5qgwsxTQtvGimKF
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
151 lines
5.8 KiB
Python
151 lines
5.8 KiB
Python
"""Typed exceptions for encoding worker lifecycle failures.
|
|
|
|
These let callers distinguish between transient capacity issues (retry will
|
|
likely help) and unexpected start failures (something is actually broken).
|
|
"""
|
|
|
|
from typing import FrozenSet
|
|
|
|
|
|
CAPACITY_ERROR_CODES: FrozenSet[str] = frozenset({
|
|
"ZONE_RESOURCE_POOL_EXHAUSTED",
|
|
"ZONE_RESOURCE_POOL_EXHAUSTED_WITH_DETAILS",
|
|
"STOCKOUT",
|
|
"QUOTA_EXCEEDED",
|
|
# AWS EC2 (the AWS encoding worker; see aws_encoding_compute.py)
|
|
"InsufficientInstanceCapacity",
|
|
"InsufficientHostCapacity",
|
|
"InsufficientCapacity",
|
|
"VcpuLimitExceeded",
|
|
"MaxSpotInstanceCountExceeded",
|
|
})
|
|
|
|
|
|
# Sentinel code stamped on an EncodingWorkerInfraError so log filters and the
|
|
# park/retry path can tell a runtime worker-infra failure apart from a real
|
|
# GCE VM-start failure.
|
|
WORKER_INFRA_FAILURE_CODE = "worker_infra_failure"
|
|
|
|
|
|
# Substrings (matched case-insensitively) that identify a worker-side *infrastructure*
|
|
# failure — the VM booted and accepted the job but then could not talk to GCP to do
|
|
# real work. The canonical case (job 6452888e, 2026-08-31) was a fallback VM that
|
|
# could not reach the metadata server to fetch its default service-account token:
|
|
# "Failed to retrieve https://metadata.google.internal/.../service-accounts/default/
|
|
# ... Compute Engine Metadata server unavailable ... SSLCertVerificationError ..."
|
|
# These are properties of the *VM*, not the encode job — the same work will succeed on
|
|
# a healthy worker, so they must be retried/re-dispatched rather than failing the job.
|
|
# Kept deliberately specific so a genuine encode error (bad codec, corrupt input) never
|
|
# matches. Compared lowercase.
|
|
WORKER_INFRA_ERROR_MARKERS: FrozenSet[str] = frozenset({
|
|
"metadata.google.internal",
|
|
"metadata server unavailable",
|
|
"compute engine metadata",
|
|
"service-accounts/default",
|
|
"sslcertverificationerror",
|
|
"certificate_verify_failed",
|
|
"could not automatically determine credentials",
|
|
"defaultcredentialserror",
|
|
})
|
|
|
|
|
|
def is_worker_infra_error(message: str) -> bool:
|
|
"""True if a worker-reported error string looks like a VM-level infra/auth
|
|
failure (metadata server unreachable, SSL/cert, missing credentials) rather
|
|
than a genuine problem with the encode job itself."""
|
|
if not message:
|
|
return False
|
|
lowered = message.lower()
|
|
return any(marker in lowered for marker in WORKER_INFRA_ERROR_MARKERS)
|
|
|
|
|
|
class EncodingWorkerStartError(Exception):
|
|
"""Raised when an attempt to start an encoding worker VM fails.
|
|
|
|
Carries the GCE error code so callers can react to specific failure modes.
|
|
"""
|
|
|
|
def __init__(
|
|
self,
|
|
message: str,
|
|
*,
|
|
vm_name: str = "",
|
|
zone: str = "",
|
|
code: str = "",
|
|
) -> None:
|
|
super().__init__(message)
|
|
self.vm_name = vm_name
|
|
self.zone = zone
|
|
self.code = code
|
|
|
|
|
|
class EncodingWorkerCapacityError(EncodingWorkerStartError):
|
|
"""A capacity-related GCE failure (e.g. ZONE_RESOURCE_POOL_EXHAUSTED).
|
|
|
|
The zone temporarily cannot allocate the requested machine type. Callers
|
|
should treat this as recoverable — retrying after a wait, or trying an
|
|
alternate zone, is likely to succeed.
|
|
"""
|
|
|
|
|
|
class EncodingWorkerInfraError(EncodingWorkerStartError):
|
|
"""A worker VM accepted the job but then hit a VM-level infrastructure/auth
|
|
failure mid-run (e.g. it could not reach the GCE metadata server to fetch its
|
|
service-account token — job 6452888e, 2026-08-31).
|
|
|
|
Subclasses ``EncodingWorkerStartError`` on purpose: the failure is a property
|
|
of the *worker*, not the encode job, so it should flow through the same
|
|
recoverable path as a VM-start failure (the render worker parks the job for
|
|
auto-retry; the final-encode Cloud Run Job retries). Before raising this,
|
|
callers demote the offending VM so the retry lands on a healthy worker rather
|
|
than looping on the broken one.
|
|
"""
|
|
|
|
|
|
# Marker the encoding worker writes into a job's status when it was interrupted
|
|
# by a worker-process restart (OOM, deploy, crash). Kept in sync with
|
|
# gce_encoding/persistence.py `_RESTART_FAIL_CODE`.
|
|
ENCODING_RESTART_FAILURE_CODE = "encoding_worker_restart"
|
|
|
|
|
|
class EncodingJobNotFoundError(RuntimeError):
|
|
"""The encoding worker returned HTTP 404 for a job status poll.
|
|
|
|
Subclasses RuntimeError so existing `except RuntimeError` handlers and
|
|
`"not found" in str(e)` checks keep working, while giving callers a precise
|
|
type to branch on (e.g. distinguishing a lost job from other RuntimeErrors)
|
|
without parsing the message text. Keep the message containing "not found".
|
|
"""
|
|
|
|
|
|
class EncodingJobLostError(Exception):
|
|
"""The encoding worker lost a job mid-run (restart wiped its in-memory state).
|
|
|
|
Distinct from a transient poll blip: the ffmpeg process and temp work dir
|
|
are gone, so the *same* job will never complete. The only recovery is to
|
|
resubmit the work as a fresh job. Deliberately NOT a subclass of
|
|
RuntimeError so it escapes the poll-failure tolerance in
|
|
`wait_for_completion` instead of being swallowed as a retryable blip.
|
|
"""
|
|
|
|
def __init__(self, message: str, *, job_id: str = "") -> None:
|
|
super().__init__(message)
|
|
self.job_id = job_id
|
|
|
|
|
|
def classify_gce_error(code: str, message: str, *, vm_name: str, zone: str) -> EncodingWorkerStartError:
|
|
"""Wrap a GCE error code/message in the appropriate typed exception."""
|
|
if code in CAPACITY_ERROR_CODES:
|
|
return EncodingWorkerCapacityError(
|
|
f"VM {vm_name} could not be started in {zone}: {code} — {message}",
|
|
vm_name=vm_name,
|
|
zone=zone,
|
|
code=code,
|
|
)
|
|
return EncodingWorkerStartError(
|
|
f"VM {vm_name} start failed in {zone}: {code or 'unknown error'} — {message or 'no message'}",
|
|
vm_name=vm_name,
|
|
zone=zone,
|
|
code=code,
|
|
)
|