mirror of
https://github.com/nomadkaraoke/karaoke-gen.git
synced 2026-11-20 13:07:57 +09:00
## Problem Three Arctic Monkeys renders (`a3f340a2`, `374dec26`, `590630c0`) failed within ~2 min on 2026-08-15 with `lost contact with worker after 5 consecutive poll failures: Encoding job … not found`. **Root cause (confirmed from live forensics):** Both 60 GB c4d primary encoding workers were down (Spot stockout), so all render traffic funneled onto the single 32 GB `n2-highcpu-32` fallback. The worker ran `ThreadPoolExecutor(max_workers=4)`, so the batch put **3 concurrent 4K ffmpeg encodes** (~30 GB) on it → **kernel OOM-killer fired** → `encoding-worker` service restarted → in-memory `jobs` registry wiped → every poll `404`'d → all three failed. ffmpeg already saturates all cores for one job, so the parallelism bought **no throughput**, only multiplied peak RAM. ## Fix **Worker — serialize the heavy lane** (`backend/services/gce_encoding/main.py`) - Split the single pool into `heavy_executor` (`max_workers=ENCODING_HEAVY_CONCURRENCY`, default **1**, clamped to `[1,4]`) for renders/encodes and `light_executor` (previews + wheel installs). One heavy ffmpeg at a time = OOM-proof on any machine type; extras queue as `pending`. - `/status` now exposes `queue_position` (+ `restart_failure_code`); jobs tagged by `kind`. **Client + workers — auto-resubmit a lost job** - `wait_for_completion` raises the typed `EncodingJobLostError` on a **404-after-the-job-was-seen** (a pinned poll can only 404 if the worker lost it) or a terminal failure carrying `restart_failure_code`. 404 is now a typed `EncodingJobNotFoundError` (no message parsing). - Time spent `pending` counts against a generous `queue_timeout`, not the per-run `timeout`. - `run_with_lost_job_resubmit` resubmits a lost job as `<id>_retry_<hex8>` (bounded, `ENCODING_RESUBMIT_MAX=2`), wired into `render_video_worker`, `video_worker`, and `video_worker_orchestrator` (incl. the stale-cache path). Idempotent by deterministic output paths (overwrite, no dupes). - `GCEEncodingBackend.encode` re-raises the lost error instead of flattening it to `success=False`, so the resubmit actually fires. ## Testing - New tests: worker lane serialization + `queue_position`; concurrency-clamp matrix; client lost-job detection (404-after-seen, restart code, pending-not-timeout); resubmit wrapper (fresh id, bounded, non-retry of other errors); backend re-raise; preview queue-timeout cap. - Full backend suite: **3766 passed, 139 skipped**. - Local CodeRabbit review completed; feedback addressed (typed 404, bounded env parsing, preview queue cap, idempotency doc). ## Deploy note `main.py` ships in the wheel, but the running uvicorn process only picks up worker-side changes on a **service restart** — the CI deploy / blue-green swap handles this. Post-deploy verification: re-render the 3 failed jobs **concurrently** (the exact OOM trigger) and confirm no OOM (`journalctl -k`), `NRestarts` doesn't increase, and all three complete. @coderabbitai ignore --------- Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
93 lines
3.1 KiB
Python
93 lines
3.1 KiB
Python
"""Typed exceptions for encoding worker lifecycle failures.
|
|
|
|
These let callers distinguish between transient capacity issues (retry will
|
|
likely help) and unexpected start failures (something is actually broken).
|
|
"""
|
|
|
|
from typing import FrozenSet
|
|
|
|
|
|
CAPACITY_ERROR_CODES: FrozenSet[str] = frozenset({
|
|
"ZONE_RESOURCE_POOL_EXHAUSTED",
|
|
"ZONE_RESOURCE_POOL_EXHAUSTED_WITH_DETAILS",
|
|
"STOCKOUT",
|
|
"QUOTA_EXCEEDED",
|
|
})
|
|
|
|
|
|
class EncodingWorkerStartError(Exception):
|
|
"""Raised when an attempt to start an encoding worker VM fails.
|
|
|
|
Carries the GCE error code so callers can react to specific failure modes.
|
|
"""
|
|
|
|
def __init__(
|
|
self,
|
|
message: str,
|
|
*,
|
|
vm_name: str = "",
|
|
zone: str = "",
|
|
code: str = "",
|
|
) -> None:
|
|
super().__init__(message)
|
|
self.vm_name = vm_name
|
|
self.zone = zone
|
|
self.code = code
|
|
|
|
|
|
class EncodingWorkerCapacityError(EncodingWorkerStartError):
|
|
"""A capacity-related GCE failure (e.g. ZONE_RESOURCE_POOL_EXHAUSTED).
|
|
|
|
The zone temporarily cannot allocate the requested machine type. Callers
|
|
should treat this as recoverable — retrying after a wait, or trying an
|
|
alternate zone, is likely to succeed.
|
|
"""
|
|
|
|
|
|
# Marker the encoding worker writes into a job's status when it was interrupted
|
|
# by a worker-process restart (OOM, deploy, crash). Kept in sync with
|
|
# gce_encoding/persistence.py `_RESTART_FAIL_CODE`.
|
|
ENCODING_RESTART_FAILURE_CODE = "encoding_worker_restart"
|
|
|
|
|
|
class EncodingJobNotFoundError(RuntimeError):
|
|
"""The encoding worker returned HTTP 404 for a job status poll.
|
|
|
|
Subclasses RuntimeError so existing `except RuntimeError` handlers and
|
|
`"not found" in str(e)` checks keep working, while giving callers a precise
|
|
type to branch on (e.g. distinguishing a lost job from other RuntimeErrors)
|
|
without parsing the message text. Keep the message containing "not found".
|
|
"""
|
|
|
|
|
|
class EncodingJobLostError(Exception):
|
|
"""The encoding worker lost a job mid-run (restart wiped its in-memory state).
|
|
|
|
Distinct from a transient poll blip: the ffmpeg process and temp work dir
|
|
are gone, so the *same* job will never complete. The only recovery is to
|
|
resubmit the work as a fresh job. Deliberately NOT a subclass of
|
|
RuntimeError so it escapes the poll-failure tolerance in
|
|
`wait_for_completion` instead of being swallowed as a retryable blip.
|
|
"""
|
|
|
|
def __init__(self, message: str, *, job_id: str = "") -> None:
|
|
super().__init__(message)
|
|
self.job_id = job_id
|
|
|
|
|
|
def classify_gce_error(code: str, message: str, *, vm_name: str, zone: str) -> EncodingWorkerStartError:
|
|
"""Wrap a GCE error code/message in the appropriate typed exception."""
|
|
if code in CAPACITY_ERROR_CODES:
|
|
return EncodingWorkerCapacityError(
|
|
f"VM {vm_name} could not be started in {zone}: {code} — {message}",
|
|
vm_name=vm_name,
|
|
zone=zone,
|
|
code=code,
|
|
)
|
|
return EncodingWorkerStartError(
|
|
f"VM {vm_name} start failed in {zone}: {code or 'unknown error'} — {message or 'no message'}",
|
|
vm_name=vm_name,
|
|
zone=zone,
|
|
code=code,
|
|
)
|