Files
karaoke-gen/backend/services/encoding_errors.py
T
Andrew Beveridge ☄️andClaude Opus 4.8 d39dfd6778 Encoding worker infra/auth failures: retry instead of terminal-failing the job (#972)
## Why

Prod alert burst (2026-08-31): a paying customer's job (`6452888e`) was
**terminally failed** — its render worker error was:

> Encoding job 6452888e failed: Failed to retrieve
`https://metadata.google.internal/.../service-accounts/default/` …
**Compute Engine Metadata server unavailable** …
`SSLCertVerificationError`

Root cause: the c4d primary pair was in a zonal stockout (503 on start),
so the render fell back to `encoding-worker-fallback-c4a`. That fallback
VM **booted and accepted the job** but then couldn't reach the GCE
metadata server to fetch its service-account token, so it couldn't touch
GCS and died mid-encode. This surfaced as a bare `RuntimeError`, hit
`render_video_worker`'s generic `except Exception` → `fail_job`, and
lost the job with **no retry** — even though the exact same work
succeeds on any healthy worker.

The same alert burst also included the daily E2E happy-path failure (the
encoder being offline made the preview step flaky) and a benign
`match-judge AI call failed; returning no-suggestion` red-error page.
All three are addressed here.

## What

**1. Encoding infra/auth failures are now recoverable (primary fix)**
- `encoding_errors.py`: `is_worker_infra_error()` classifier (metadata /
SSL-cert / missing-credentials markers, deliberately specific so a real
ffmpeg/input error never matches) + new `EncodingWorkerInfraError` — a
**subclass of `EncodingWorkerStartError`** so it flows through the
existing park-for-auto-retry path with zero changes to the render
worker's `except` (and the final-encode Cloud Run Job retries too).
- `encoding_service.wait_for_completion()`: on a worker-reported
infra/auth failure, **demote the broken worker + invalidate the URL
cache**, then raise the recoverable typed error instead of
`RuntimeError`.
- `encoding_worker_manager.demote_active_worker()`: records a
capacity-state cooldown for the serving fallback's `machine_type@zone`
and clears `active_override`, so the retry re-selects a *different,
healthy* VM. Scoped to **fallbacks only** — it never family-demotes the
fast c4d primary on a rare blip.

**2. match-judge noise** — the two graceful-degradation
`logger.exception` calls become `logger.warning(exc_info=True)`, so a
transient Vertex/Gemini blip that safely returns no-suggestion stops
paging as a red "new error pattern".

**3. Happy-path E2E robustness** — tolerate slow encoder cold-starts at
the preview step: scope the alert check to the modal (kills the spurious
empty-`[role=alert]` "Preview error:" from run #151) and recover the
"Proceed to Instrumental" button via reload+reopen. The final assertion
stays authoritative, so genuinely broken generation still fails the
smoke test.

## Testing
- New unit coverage: classifier positive/negative + subclass
(`test_encoding_errors.py`), `demote_active_worker` (fallback
demote+clear, primary no-op, never-raises), `wait_for_completion` infra
classification (+ demote/invalidate, and no-manager path), and render
worker **parks** (not fails) on `EncodingWorkerInfraError`.
- `python -m pytest` across the encoding / worker / match-judge suites:
**177 passed** locally.

## Not in this PR (infra, needs Andrew / GCP write)
- **Fault A**: `encoding-worker-a` (c4d, us-central1-c) is chronically
503 on start (zonal stockout) — that's why the "Encoder offline" footer
showed in the E2E. This change makes the fallback path robust to it, but
the underlying c4d capacity/zone choice is an infra decision (secret/VM
change; my GCP access is read-only). Worth deciding whether to repin the
primary family/zone or promote a fallback.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit

* **New Features**
* Detects infrastructure and authentication failures on encoding
workers.
* Demotes affected workers and retries jobs using alternate capacity
when available.

* **Bug Fixes**
* Prevents recoverable worker failures from becoming terminal job
errors.
* Reduces misleading error-level logging for gracefully handled
match-judge failures.
* Improves preview-flow recovery by waiting for actions to become
enabled.

* **Documentation**
* Clarified that retries prefer, but do not guarantee, a different
worker.

* **Chores**
  * Updated the application version to 0.217.0.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->

---------

Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
2026-08-31 18:59:59 -04:00

145 lines
5.6 KiB
Python

"""Typed exceptions for encoding worker lifecycle failures.
These let callers distinguish between transient capacity issues (retry will
likely help) and unexpected start failures (something is actually broken).
"""
from typing import FrozenSet
CAPACITY_ERROR_CODES: FrozenSet[str] = frozenset({
"ZONE_RESOURCE_POOL_EXHAUSTED",
"ZONE_RESOURCE_POOL_EXHAUSTED_WITH_DETAILS",
"STOCKOUT",
"QUOTA_EXCEEDED",
})
# Sentinel code stamped on an EncodingWorkerInfraError so log filters and the
# park/retry path can tell a runtime worker-infra failure apart from a real
# GCE VM-start failure.
WORKER_INFRA_FAILURE_CODE = "worker_infra_failure"
# Substrings (matched case-insensitively) that identify a worker-side *infrastructure*
# failure — the VM booted and accepted the job but then could not talk to GCP to do
# real work. The canonical case (job 6452888e, 2026-08-31) was a fallback VM that
# could not reach the metadata server to fetch its default service-account token:
# "Failed to retrieve https://metadata.google.internal/.../service-accounts/default/
# ... Compute Engine Metadata server unavailable ... SSLCertVerificationError ..."
# These are properties of the *VM*, not the encode job — the same work will succeed on
# a healthy worker, so they must be retried/re-dispatched rather than failing the job.
# Kept deliberately specific so a genuine encode error (bad codec, corrupt input) never
# matches. Compared lowercase.
WORKER_INFRA_ERROR_MARKERS: FrozenSet[str] = frozenset({
"metadata.google.internal",
"metadata server unavailable",
"compute engine metadata",
"service-accounts/default",
"sslcertverificationerror",
"certificate_verify_failed",
"could not automatically determine credentials",
"defaultcredentialserror",
})
def is_worker_infra_error(message: str) -> bool:
"""True if a worker-reported error string looks like a VM-level infra/auth
failure (metadata server unreachable, SSL/cert, missing credentials) rather
than a genuine problem with the encode job itself."""
if not message:
return False
lowered = message.lower()
return any(marker in lowered for marker in WORKER_INFRA_ERROR_MARKERS)
class EncodingWorkerStartError(Exception):
"""Raised when an attempt to start an encoding worker VM fails.
Carries the GCE error code so callers can react to specific failure modes.
"""
def __init__(
self,
message: str,
*,
vm_name: str = "",
zone: str = "",
code: str = "",
) -> None:
super().__init__(message)
self.vm_name = vm_name
self.zone = zone
self.code = code
class EncodingWorkerCapacityError(EncodingWorkerStartError):
"""A capacity-related GCE failure (e.g. ZONE_RESOURCE_POOL_EXHAUSTED).
The zone temporarily cannot allocate the requested machine type. Callers
should treat this as recoverable — retrying after a wait, or trying an
alternate zone, is likely to succeed.
"""
class EncodingWorkerInfraError(EncodingWorkerStartError):
"""A worker VM accepted the job but then hit a VM-level infrastructure/auth
failure mid-run (e.g. it could not reach the GCE metadata server to fetch its
service-account token — job 6452888e, 2026-08-31).
Subclasses ``EncodingWorkerStartError`` on purpose: the failure is a property
of the *worker*, not the encode job, so it should flow through the same
recoverable path as a VM-start failure (the render worker parks the job for
auto-retry; the final-encode Cloud Run Job retries). Before raising this,
callers demote the offending VM so the retry lands on a healthy worker rather
than looping on the broken one.
"""
# Marker the encoding worker writes into a job's status when it was interrupted
# by a worker-process restart (OOM, deploy, crash). Kept in sync with
# gce_encoding/persistence.py `_RESTART_FAIL_CODE`.
ENCODING_RESTART_FAILURE_CODE = "encoding_worker_restart"
class EncodingJobNotFoundError(RuntimeError):
"""The encoding worker returned HTTP 404 for a job status poll.
Subclasses RuntimeError so existing `except RuntimeError` handlers and
`"not found" in str(e)` checks keep working, while giving callers a precise
type to branch on (e.g. distinguishing a lost job from other RuntimeErrors)
without parsing the message text. Keep the message containing "not found".
"""
class EncodingJobLostError(Exception):
"""The encoding worker lost a job mid-run (restart wiped its in-memory state).
Distinct from a transient poll blip: the ffmpeg process and temp work dir
are gone, so the *same* job will never complete. The only recovery is to
resubmit the work as a fresh job. Deliberately NOT a subclass of
RuntimeError so it escapes the poll-failure tolerance in
`wait_for_completion` instead of being swallowed as a retryable blip.
"""
def __init__(self, message: str, *, job_id: str = "") -> None:
super().__init__(message)
self.job_id = job_id
def classify_gce_error(code: str, message: str, *, vm_name: str, zone: str) -> EncodingWorkerStartError:
"""Wrap a GCE error code/message in the appropriate typed exception."""
if code in CAPACITY_ERROR_CODES:
return EncodingWorkerCapacityError(
f"VM {vm_name} could not be started in {zone}: {code} — {message}",
vm_name=vm_name,
zone=zone,
code=code,
)
return EncodingWorkerStartError(
f"VM {vm_name} start failed in {zone}: {code or 'unknown error'} — {message or 'no message'}",
vm_name=vm_name,
zone=zone,
code=code,
)