mirror of
https://github.com/nomadkaraoke/karaoke-gen.git
synced 2026-11-20 13:07:57 +09:00
## Why Prod alert burst (2026-08-31): a paying customer's job (`6452888e`) was **terminally failed** — its render worker error was: > Encoding job 6452888e failed: Failed to retrieve `https://metadata.google.internal/.../service-accounts/default/` … **Compute Engine Metadata server unavailable** … `SSLCertVerificationError` Root cause: the c4d primary pair was in a zonal stockout (503 on start), so the render fell back to `encoding-worker-fallback-c4a`. That fallback VM **booted and accepted the job** but then couldn't reach the GCE metadata server to fetch its service-account token, so it couldn't touch GCS and died mid-encode. This surfaced as a bare `RuntimeError`, hit `render_video_worker`'s generic `except Exception` → `fail_job`, and lost the job with **no retry** — even though the exact same work succeeds on any healthy worker. The same alert burst also included the daily E2E happy-path failure (the encoder being offline made the preview step flaky) and a benign `match-judge AI call failed; returning no-suggestion` red-error page. All three are addressed here. ## What **1. Encoding infra/auth failures are now recoverable (primary fix)** - `encoding_errors.py`: `is_worker_infra_error()` classifier (metadata / SSL-cert / missing-credentials markers, deliberately specific so a real ffmpeg/input error never matches) + new `EncodingWorkerInfraError` — a **subclass of `EncodingWorkerStartError`** so it flows through the existing park-for-auto-retry path with zero changes to the render worker's `except` (and the final-encode Cloud Run Job retries too). - `encoding_service.wait_for_completion()`: on a worker-reported infra/auth failure, **demote the broken worker + invalidate the URL cache**, then raise the recoverable typed error instead of `RuntimeError`. - `encoding_worker_manager.demote_active_worker()`: records a capacity-state cooldown for the serving fallback's `machine_type@zone` and clears `active_override`, so the retry re-selects a *different, healthy* VM. Scoped to **fallbacks only** — it never family-demotes the fast c4d primary on a rare blip. **2. match-judge noise** — the two graceful-degradation `logger.exception` calls become `logger.warning(exc_info=True)`, so a transient Vertex/Gemini blip that safely returns no-suggestion stops paging as a red "new error pattern". **3. Happy-path E2E robustness** — tolerate slow encoder cold-starts at the preview step: scope the alert check to the modal (kills the spurious empty-`[role=alert]` "Preview error:" from run #151) and recover the "Proceed to Instrumental" button via reload+reopen. The final assertion stays authoritative, so genuinely broken generation still fails the smoke test. ## Testing - New unit coverage: classifier positive/negative + subclass (`test_encoding_errors.py`), `demote_active_worker` (fallback demote+clear, primary no-op, never-raises), `wait_for_completion` infra classification (+ demote/invalidate, and no-manager path), and render worker **parks** (not fails) on `EncodingWorkerInfraError`. - `python -m pytest` across the encoding / worker / match-judge suites: **177 passed** locally. ## Not in this PR (infra, needs Andrew / GCP write) - **Fault A**: `encoding-worker-a` (c4d, us-central1-c) is chronically 503 on start (zonal stockout) — that's why the "Encoder offline" footer showed in the E2E. This change makes the fallback path robust to it, but the underlying c4d capacity/zone choice is an infra decision (secret/VM change; my GCP access is read-only). Worth deciding whether to repin the primary family/zone or promote a fallback. 🤖 Generated with [Claude Code](https://claude.com/claude-code) <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit * **New Features** * Detects infrastructure and authentication failures on encoding workers. * Demotes affected workers and retries jobs using alternate capacity when available. * **Bug Fixes** * Prevents recoverable worker failures from becoming terminal job errors. * Reduces misleading error-level logging for gracefully handled match-judge failures. * Improves preview-flow recovery by waiting for actions to become enabled. * **Documentation** * Clarified that retries prefer, but do not guarantee, a different worker. * **Chores** * Updated the application version to 0.217.0. <!-- end of auto-generated comment: release notes by coderabbit.ai --> --------- Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
62 lines
2.6 KiB
Python
62 lines
2.6 KiB
Python
"""Tests for encoding worker error classification helpers."""
|
|
|
|
from backend.services.encoding_errors import (
|
|
EncodingWorkerInfraError,
|
|
EncodingWorkerStartError,
|
|
is_worker_infra_error,
|
|
)
|
|
|
|
|
|
# The verbatim worker-reported error from the prod incident (job 6452888e).
|
|
_METADATA_FAILURE = (
|
|
"Failed to retrieve https://metadata.google.internal/computeMetadata/v1/instance/"
|
|
"service-accounts/default/?recursive=true from the Google Compute Engine metadata "
|
|
"service. Compute Engine Metadata server unavailable. Last exception: "
|
|
"HTTPSConnectionPool(host='metadata.google.internal', port=443): Max retries "
|
|
"exceeded with url: /computeMetadata/v1/instance/service-accounts/default/"
|
|
"?recursive=true (Caused by SSLError(SSLCertVerificationError(1, '[SSL: "
|
|
"CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get local issuer "
|
|
"certificate (_ssl.c:1018)')))"
|
|
)
|
|
|
|
|
|
class TestIsWorkerInfraError:
|
|
def test_metadata_failure_is_infra(self):
|
|
assert is_worker_infra_error(_METADATA_FAILURE) is True
|
|
|
|
def test_matches_are_case_insensitive(self):
|
|
assert is_worker_infra_error(_METADATA_FAILURE.upper()) is True
|
|
|
|
def test_individual_markers(self):
|
|
for msg in (
|
|
"Compute Engine Metadata server unavailable",
|
|
"SSLCertVerificationError: certificate_verify_failed",
|
|
"google.auth.exceptions.DefaultCredentialsError: could not automatically "
|
|
"determine credentials",
|
|
"GET /computeMetadata/v1/instance/service-accounts/default/ failed",
|
|
):
|
|
assert is_worker_infra_error(msg) is True, msg
|
|
|
|
def test_genuine_encode_error_is_not_infra(self):
|
|
# Real ffmpeg / input problems must stay terminal, never masquerade as infra.
|
|
for msg in (
|
|
"ffmpeg exploded",
|
|
"Invalid data found when processing input",
|
|
"Unknown encoder 'libx265'",
|
|
"Output file is empty, nothing was encoded",
|
|
"",
|
|
):
|
|
assert is_worker_infra_error(msg) is False, msg
|
|
|
|
def test_none_is_not_infra(self):
|
|
assert is_worker_infra_error(None) is False # type: ignore[arg-type]
|
|
|
|
|
|
class TestEncodingWorkerInfraError:
|
|
def test_is_recoverable_start_error_subclass(self):
|
|
"""Subclassing EncodingWorkerStartError is what routes it through the render
|
|
worker's park-for-auto-retry path (and the final-encode Cloud Run Job retry)."""
|
|
err = EncodingWorkerInfraError("boom", vm_name="encoding-worker-fallback-c4a")
|
|
assert isinstance(err, EncodingWorkerStartError)
|
|
assert err.vm_name == "encoding-worker-fallback-c4a"
|