Files
karaoke-gen/backend/services/encoding_errors.py
T
Andrew Beveridge ☄️andClaude Opus 4.8 51dbf1cbfd fix(encoding): serialize heavy encodes (OOM-proof) + auto-resubmit lost renders (#908)
## Problem

Three Arctic Monkeys renders (`a3f340a2`, `374dec26`, `590630c0`) failed
within ~2 min on 2026-08-15 with `lost contact with worker after 5
consecutive poll failures: Encoding job … not found`.

**Root cause (confirmed from live forensics):** Both 60 GB c4d primary
encoding workers were down (Spot stockout), so all render traffic
funneled onto the single 32 GB `n2-highcpu-32` fallback. The worker ran
`ThreadPoolExecutor(max_workers=4)`, so the batch put **3 concurrent 4K
ffmpeg encodes** (~30 GB) on it → **kernel OOM-killer fired** →
`encoding-worker` service restarted → in-memory `jobs` registry wiped →
every poll `404`'d → all three failed. ffmpeg already saturates all
cores for one job, so the parallelism bought **no throughput**, only
multiplied peak RAM.

## Fix

**Worker — serialize the heavy lane**
(`backend/services/gce_encoding/main.py`)
- Split the single pool into `heavy_executor`
(`max_workers=ENCODING_HEAVY_CONCURRENCY`, default **1**, clamped to
`[1,4]`) for renders/encodes and `light_executor` (previews + wheel
installs). One heavy ffmpeg at a time = OOM-proof on any machine type;
extras queue as `pending`.
- `/status` now exposes `queue_position` (+ `restart_failure_code`);
jobs tagged by `kind`.

**Client + workers — auto-resubmit a lost job**
- `wait_for_completion` raises the typed `EncodingJobLostError` on a
**404-after-the-job-was-seen** (a pinned poll can only 404 if the worker
lost it) or a terminal failure carrying `restart_failure_code`. 404 is
now a typed `EncodingJobNotFoundError` (no message parsing).
- Time spent `pending` counts against a generous `queue_timeout`, not
the per-run `timeout`.
- `run_with_lost_job_resubmit` resubmits a lost job as
`<id>_retry_<hex8>` (bounded, `ENCODING_RESUBMIT_MAX=2`), wired into
`render_video_worker`, `video_worker`, and `video_worker_orchestrator`
(incl. the stale-cache path). Idempotent by deterministic output paths
(overwrite, no dupes).
- `GCEEncodingBackend.encode` re-raises the lost error instead of
flattening it to `success=False`, so the resubmit actually fires.

## Testing
- New tests: worker lane serialization + `queue_position`;
concurrency-clamp matrix; client lost-job detection (404-after-seen,
restart code, pending-not-timeout); resubmit wrapper (fresh id, bounded,
non-retry of other errors); backend re-raise; preview queue-timeout cap.
- Full backend suite: **3766 passed, 139 skipped**.
- Local CodeRabbit review completed; feedback addressed (typed 404,
bounded env parsing, preview queue cap, idempotency doc).

## Deploy note
`main.py` ships in the wheel, but the running uvicorn process only picks
up worker-side changes on a **service restart** — the CI deploy /
blue-green swap handles this. Post-deploy verification: re-render the 3
failed jobs **concurrently** (the exact OOM trigger) and confirm no OOM
(`journalctl -k`), `NRestarts` doesn't increase, and all three complete.

@coderabbitai ignore

---------

Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
2026-08-15 15:26:25 -04:00

93 lines
3.1 KiB
Python

"""Typed exceptions for encoding worker lifecycle failures.
These let callers distinguish between transient capacity issues (retry will
likely help) and unexpected start failures (something is actually broken).
"""
from typing import FrozenSet
CAPACITY_ERROR_CODES: FrozenSet[str] = frozenset({
"ZONE_RESOURCE_POOL_EXHAUSTED",
"ZONE_RESOURCE_POOL_EXHAUSTED_WITH_DETAILS",
"STOCKOUT",
"QUOTA_EXCEEDED",
})
class EncodingWorkerStartError(Exception):
"""Raised when an attempt to start an encoding worker VM fails.
Carries the GCE error code so callers can react to specific failure modes.
"""
def __init__(
self,
message: str,
*,
vm_name: str = "",
zone: str = "",
code: str = "",
) -> None:
super().__init__(message)
self.vm_name = vm_name
self.zone = zone
self.code = code
class EncodingWorkerCapacityError(EncodingWorkerStartError):
"""A capacity-related GCE failure (e.g. ZONE_RESOURCE_POOL_EXHAUSTED).
The zone temporarily cannot allocate the requested machine type. Callers
should treat this as recoverable — retrying after a wait, or trying an
alternate zone, is likely to succeed.
"""
# Marker the encoding worker writes into a job's status when it was interrupted
# by a worker-process restart (OOM, deploy, crash). Kept in sync with
# gce_encoding/persistence.py `_RESTART_FAIL_CODE`.
ENCODING_RESTART_FAILURE_CODE = "encoding_worker_restart"
class EncodingJobNotFoundError(RuntimeError):
"""The encoding worker returned HTTP 404 for a job status poll.
Subclasses RuntimeError so existing `except RuntimeError` handlers and
`"not found" in str(e)` checks keep working, while giving callers a precise
type to branch on (e.g. distinguishing a lost job from other RuntimeErrors)
without parsing the message text. Keep the message containing "not found".
"""
class EncodingJobLostError(Exception):
"""The encoding worker lost a job mid-run (restart wiped its in-memory state).
Distinct from a transient poll blip: the ffmpeg process and temp work dir
are gone, so the *same* job will never complete. The only recovery is to
resubmit the work as a fresh job. Deliberately NOT a subclass of
RuntimeError so it escapes the poll-failure tolerance in
`wait_for_completion` instead of being swallowed as a retryable blip.
"""
def __init__(self, message: str, *, job_id: str = "") -> None:
super().__init__(message)
self.job_id = job_id
def classify_gce_error(code: str, message: str, *, vm_name: str, zone: str) -> EncodingWorkerStartError:
"""Wrap a GCE error code/message in the appropriate typed exception."""
if code in CAPACITY_ERROR_CODES:
return EncodingWorkerCapacityError(
f"VM {vm_name} could not be started in {zone}: {code} — {message}",
vm_name=vm_name,
zone=zone,
code=code,
)
return EncodingWorkerStartError(
f"VM {vm_name} start failed in {zone}: {code or 'unknown error'} — {message or 'no message'}",
vm_name=vm_name,
zone=zone,
code=code,
)