mirror of
https://github.com/nomadkaraoke/karaoke-gen.git
synced 2026-11-20 13:07:57 +09:00
## Why Prod alert burst (2026-08-31): a paying customer's job (`6452888e`) was **terminally failed** — its render worker error was: > Encoding job 6452888e failed: Failed to retrieve `https://metadata.google.internal/.../service-accounts/default/` … **Compute Engine Metadata server unavailable** … `SSLCertVerificationError` Root cause: the c4d primary pair was in a zonal stockout (503 on start), so the render fell back to `encoding-worker-fallback-c4a`. That fallback VM **booted and accepted the job** but then couldn't reach the GCE metadata server to fetch its service-account token, so it couldn't touch GCS and died mid-encode. This surfaced as a bare `RuntimeError`, hit `render_video_worker`'s generic `except Exception` → `fail_job`, and lost the job with **no retry** — even though the exact same work succeeds on any healthy worker. The same alert burst also included the daily E2E happy-path failure (the encoder being offline made the preview step flaky) and a benign `match-judge AI call failed; returning no-suggestion` red-error page. All three are addressed here. ## What **1. Encoding infra/auth failures are now recoverable (primary fix)** - `encoding_errors.py`: `is_worker_infra_error()` classifier (metadata / SSL-cert / missing-credentials markers, deliberately specific so a real ffmpeg/input error never matches) + new `EncodingWorkerInfraError` — a **subclass of `EncodingWorkerStartError`** so it flows through the existing park-for-auto-retry path with zero changes to the render worker's `except` (and the final-encode Cloud Run Job retries too). - `encoding_service.wait_for_completion()`: on a worker-reported infra/auth failure, **demote the broken worker + invalidate the URL cache**, then raise the recoverable typed error instead of `RuntimeError`. - `encoding_worker_manager.demote_active_worker()`: records a capacity-state cooldown for the serving fallback's `machine_type@zone` and clears `active_override`, so the retry re-selects a *different, healthy* VM. Scoped to **fallbacks only** — it never family-demotes the fast c4d primary on a rare blip. **2. match-judge noise** — the two graceful-degradation `logger.exception` calls become `logger.warning(exc_info=True)`, so a transient Vertex/Gemini blip that safely returns no-suggestion stops paging as a red "new error pattern". **3. Happy-path E2E robustness** — tolerate slow encoder cold-starts at the preview step: scope the alert check to the modal (kills the spurious empty-`[role=alert]` "Preview error:" from run #151) and recover the "Proceed to Instrumental" button via reload+reopen. The final assertion stays authoritative, so genuinely broken generation still fails the smoke test. ## Testing - New unit coverage: classifier positive/negative + subclass (`test_encoding_errors.py`), `demote_active_worker` (fallback demote+clear, primary no-op, never-raises), `wait_for_completion` infra classification (+ demote/invalidate, and no-manager path), and render worker **parks** (not fails) on `EncodingWorkerInfraError`. - `python -m pytest` across the encoding / worker / match-judge suites: **177 passed** locally. ## Not in this PR (infra, needs Andrew / GCP write) - **Fault A**: `encoding-worker-a` (c4d, us-central1-c) is chronically 503 on start (zonal stockout) — that's why the "Encoder offline" footer showed in the E2E. This change makes the fallback path robust to it, but the underlying c4d capacity/zone choice is an infra decision (secret/VM change; my GCP access is read-only). Worth deciding whether to repin the primary family/zone or promote a fallback. 🤖 Generated with [Claude Code](https://claude.com/claude-code) <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit * **New Features** * Detects infrastructure and authentication failures on encoding workers. * Demotes affected workers and retries jobs using alternate capacity when available. * **Bug Fixes** * Prevents recoverable worker failures from becoming terminal job errors. * Reduces misleading error-level logging for gracefully handled match-judge failures. * Improves preview-flow recovery by waiting for actions to become enabled. * **Documentation** * Clarified that retries prefer, but do not guarantee, a different worker. * **Chores** * Updated the application version to 0.217.0. <!-- end of auto-generated comment: release notes by coderabbit.ai --> --------- Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
145 lines
5.6 KiB
Python
145 lines
5.6 KiB
Python
"""Typed exceptions for encoding worker lifecycle failures.
|
|
|
|
These let callers distinguish between transient capacity issues (retry will
|
|
likely help) and unexpected start failures (something is actually broken).
|
|
"""
|
|
|
|
from typing import FrozenSet
|
|
|
|
|
|
CAPACITY_ERROR_CODES: FrozenSet[str] = frozenset({
|
|
"ZONE_RESOURCE_POOL_EXHAUSTED",
|
|
"ZONE_RESOURCE_POOL_EXHAUSTED_WITH_DETAILS",
|
|
"STOCKOUT",
|
|
"QUOTA_EXCEEDED",
|
|
})
|
|
|
|
|
|
# Sentinel code stamped on an EncodingWorkerInfraError so log filters and the
|
|
# park/retry path can tell a runtime worker-infra failure apart from a real
|
|
# GCE VM-start failure.
|
|
WORKER_INFRA_FAILURE_CODE = "worker_infra_failure"
|
|
|
|
|
|
# Substrings (matched case-insensitively) that identify a worker-side *infrastructure*
|
|
# failure — the VM booted and accepted the job but then could not talk to GCP to do
|
|
# real work. The canonical case (job 6452888e, 2026-08-31) was a fallback VM that
|
|
# could not reach the metadata server to fetch its default service-account token:
|
|
# "Failed to retrieve https://metadata.google.internal/.../service-accounts/default/
|
|
# ... Compute Engine Metadata server unavailable ... SSLCertVerificationError ..."
|
|
# These are properties of the *VM*, not the encode job — the same work will succeed on
|
|
# a healthy worker, so they must be retried/re-dispatched rather than failing the job.
|
|
# Kept deliberately specific so a genuine encode error (bad codec, corrupt input) never
|
|
# matches. Compared lowercase.
|
|
WORKER_INFRA_ERROR_MARKERS: FrozenSet[str] = frozenset({
|
|
"metadata.google.internal",
|
|
"metadata server unavailable",
|
|
"compute engine metadata",
|
|
"service-accounts/default",
|
|
"sslcertverificationerror",
|
|
"certificate_verify_failed",
|
|
"could not automatically determine credentials",
|
|
"defaultcredentialserror",
|
|
})
|
|
|
|
|
|
def is_worker_infra_error(message: str) -> bool:
|
|
"""True if a worker-reported error string looks like a VM-level infra/auth
|
|
failure (metadata server unreachable, SSL/cert, missing credentials) rather
|
|
than a genuine problem with the encode job itself."""
|
|
if not message:
|
|
return False
|
|
lowered = message.lower()
|
|
return any(marker in lowered for marker in WORKER_INFRA_ERROR_MARKERS)
|
|
|
|
|
|
class EncodingWorkerStartError(Exception):
|
|
"""Raised when an attempt to start an encoding worker VM fails.
|
|
|
|
Carries the GCE error code so callers can react to specific failure modes.
|
|
"""
|
|
|
|
def __init__(
|
|
self,
|
|
message: str,
|
|
*,
|
|
vm_name: str = "",
|
|
zone: str = "",
|
|
code: str = "",
|
|
) -> None:
|
|
super().__init__(message)
|
|
self.vm_name = vm_name
|
|
self.zone = zone
|
|
self.code = code
|
|
|
|
|
|
class EncodingWorkerCapacityError(EncodingWorkerStartError):
|
|
"""A capacity-related GCE failure (e.g. ZONE_RESOURCE_POOL_EXHAUSTED).
|
|
|
|
The zone temporarily cannot allocate the requested machine type. Callers
|
|
should treat this as recoverable — retrying after a wait, or trying an
|
|
alternate zone, is likely to succeed.
|
|
"""
|
|
|
|
|
|
class EncodingWorkerInfraError(EncodingWorkerStartError):
|
|
"""A worker VM accepted the job but then hit a VM-level infrastructure/auth
|
|
failure mid-run (e.g. it could not reach the GCE metadata server to fetch its
|
|
service-account token — job 6452888e, 2026-08-31).
|
|
|
|
Subclasses ``EncodingWorkerStartError`` on purpose: the failure is a property
|
|
of the *worker*, not the encode job, so it should flow through the same
|
|
recoverable path as a VM-start failure (the render worker parks the job for
|
|
auto-retry; the final-encode Cloud Run Job retries). Before raising this,
|
|
callers demote the offending VM so the retry lands on a healthy worker rather
|
|
than looping on the broken one.
|
|
"""
|
|
|
|
|
|
# Marker the encoding worker writes into a job's status when it was interrupted
|
|
# by a worker-process restart (OOM, deploy, crash). Kept in sync with
|
|
# gce_encoding/persistence.py `_RESTART_FAIL_CODE`.
|
|
ENCODING_RESTART_FAILURE_CODE = "encoding_worker_restart"
|
|
|
|
|
|
class EncodingJobNotFoundError(RuntimeError):
|
|
"""The encoding worker returned HTTP 404 for a job status poll.
|
|
|
|
Subclasses RuntimeError so existing `except RuntimeError` handlers and
|
|
`"not found" in str(e)` checks keep working, while giving callers a precise
|
|
type to branch on (e.g. distinguishing a lost job from other RuntimeErrors)
|
|
without parsing the message text. Keep the message containing "not found".
|
|
"""
|
|
|
|
|
|
class EncodingJobLostError(Exception):
|
|
"""The encoding worker lost a job mid-run (restart wiped its in-memory state).
|
|
|
|
Distinct from a transient poll blip: the ffmpeg process and temp work dir
|
|
are gone, so the *same* job will never complete. The only recovery is to
|
|
resubmit the work as a fresh job. Deliberately NOT a subclass of
|
|
RuntimeError so it escapes the poll-failure tolerance in
|
|
`wait_for_completion` instead of being swallowed as a retryable blip.
|
|
"""
|
|
|
|
def __init__(self, message: str, *, job_id: str = "") -> None:
|
|
super().__init__(message)
|
|
self.job_id = job_id
|
|
|
|
|
|
def classify_gce_error(code: str, message: str, *, vm_name: str, zone: str) -> EncodingWorkerStartError:
|
|
"""Wrap a GCE error code/message in the appropriate typed exception."""
|
|
if code in CAPACITY_ERROR_CODES:
|
|
return EncodingWorkerCapacityError(
|
|
f"VM {vm_name} could not be started in {zone}: {code} — {message}",
|
|
vm_name=vm_name,
|
|
zone=zone,
|
|
code=code,
|
|
)
|
|
return EncodingWorkerStartError(
|
|
f"VM {vm_name} start failed in {zone}: {code or 'unknown error'} — {message or 'no message'}",
|
|
vm_name=vm_name,
|
|
zone=zone,
|
|
code=code,
|
|
)
|