Files
karaoke-gen/backend/tests/test_encoding_errors.py
T
Andrew Beveridge ☄️andClaude Opus 4.8 d39dfd6778 Encoding worker infra/auth failures: retry instead of terminal-failing the job (#972)
## Why

Prod alert burst (2026-08-31): a paying customer's job (`6452888e`) was
**terminally failed** — its render worker error was:

> Encoding job 6452888e failed: Failed to retrieve
`https://metadata.google.internal/.../service-accounts/default/` …
**Compute Engine Metadata server unavailable** …
`SSLCertVerificationError`

Root cause: the c4d primary pair was in a zonal stockout (503 on start),
so the render fell back to `encoding-worker-fallback-c4a`. That fallback
VM **booted and accepted the job** but then couldn't reach the GCE
metadata server to fetch its service-account token, so it couldn't touch
GCS and died mid-encode. This surfaced as a bare `RuntimeError`, hit
`render_video_worker`'s generic `except Exception` → `fail_job`, and
lost the job with **no retry** — even though the exact same work
succeeds on any healthy worker.

The same alert burst also included the daily E2E happy-path failure (the
encoder being offline made the preview step flaky) and a benign
`match-judge AI call failed; returning no-suggestion` red-error page.
All three are addressed here.

## What

**1. Encoding infra/auth failures are now recoverable (primary fix)**
- `encoding_errors.py`: `is_worker_infra_error()` classifier (metadata /
SSL-cert / missing-credentials markers, deliberately specific so a real
ffmpeg/input error never matches) + new `EncodingWorkerInfraError` — a
**subclass of `EncodingWorkerStartError`** so it flows through the
existing park-for-auto-retry path with zero changes to the render
worker's `except` (and the final-encode Cloud Run Job retries too).
- `encoding_service.wait_for_completion()`: on a worker-reported
infra/auth failure, **demote the broken worker + invalidate the URL
cache**, then raise the recoverable typed error instead of
`RuntimeError`.
- `encoding_worker_manager.demote_active_worker()`: records a
capacity-state cooldown for the serving fallback's `machine_type@zone`
and clears `active_override`, so the retry re-selects a *different,
healthy* VM. Scoped to **fallbacks only** — it never family-demotes the
fast c4d primary on a rare blip.

**2. match-judge noise** — the two graceful-degradation
`logger.exception` calls become `logger.warning(exc_info=True)`, so a
transient Vertex/Gemini blip that safely returns no-suggestion stops
paging as a red "new error pattern".

**3. Happy-path E2E robustness** — tolerate slow encoder cold-starts at
the preview step: scope the alert check to the modal (kills the spurious
empty-`[role=alert]` "Preview error:" from run #151) and recover the
"Proceed to Instrumental" button via reload+reopen. The final assertion
stays authoritative, so genuinely broken generation still fails the
smoke test.

## Testing
- New unit coverage: classifier positive/negative + subclass
(`test_encoding_errors.py`), `demote_active_worker` (fallback
demote+clear, primary no-op, never-raises), `wait_for_completion` infra
classification (+ demote/invalidate, and no-manager path), and render
worker **parks** (not fails) on `EncodingWorkerInfraError`.
- `python -m pytest` across the encoding / worker / match-judge suites:
**177 passed** locally.

## Not in this PR (infra, needs Andrew / GCP write)
- **Fault A**: `encoding-worker-a` (c4d, us-central1-c) is chronically
503 on start (zonal stockout) — that's why the "Encoder offline" footer
showed in the E2E. This change makes the fallback path robust to it, but
the underlying c4d capacity/zone choice is an infra decision (secret/VM
change; my GCP access is read-only). Worth deciding whether to repin the
primary family/zone or promote a fallback.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit

* **New Features**
* Detects infrastructure and authentication failures on encoding
workers.
* Demotes affected workers and retries jobs using alternate capacity
when available.

* **Bug Fixes**
* Prevents recoverable worker failures from becoming terminal job
errors.
* Reduces misleading error-level logging for gracefully handled
match-judge failures.
* Improves preview-flow recovery by waiting for actions to become
enabled.

* **Documentation**
* Clarified that retries prefer, but do not guarantee, a different
worker.

* **Chores**
  * Updated the application version to 0.217.0.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->

---------

Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
2026-08-31 18:59:59 -04:00

62 lines
2.6 KiB
Python

"""Tests for encoding worker error classification helpers."""
from backend.services.encoding_errors import (
EncodingWorkerInfraError,
EncodingWorkerStartError,
is_worker_infra_error,
)
# The verbatim worker-reported error from the prod incident (job 6452888e).
_METADATA_FAILURE = (
"Failed to retrieve https://metadata.google.internal/computeMetadata/v1/instance/"
"service-accounts/default/?recursive=true from the Google Compute Engine metadata "
"service. Compute Engine Metadata server unavailable. Last exception: "
"HTTPSConnectionPool(host='metadata.google.internal', port=443): Max retries "
"exceeded with url: /computeMetadata/v1/instance/service-accounts/default/"
"?recursive=true (Caused by SSLError(SSLCertVerificationError(1, '[SSL: "
"CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get local issuer "
"certificate (_ssl.c:1018)')))"
)
class TestIsWorkerInfraError:
def test_metadata_failure_is_infra(self):
assert is_worker_infra_error(_METADATA_FAILURE) is True
def test_matches_are_case_insensitive(self):
assert is_worker_infra_error(_METADATA_FAILURE.upper()) is True
def test_individual_markers(self):
for msg in (
"Compute Engine Metadata server unavailable",
"SSLCertVerificationError: certificate_verify_failed",
"google.auth.exceptions.DefaultCredentialsError: could not automatically "
"determine credentials",
"GET /computeMetadata/v1/instance/service-accounts/default/ failed",
):
assert is_worker_infra_error(msg) is True, msg
def test_genuine_encode_error_is_not_infra(self):
# Real ffmpeg / input problems must stay terminal, never masquerade as infra.
for msg in (
"ffmpeg exploded",
"Invalid data found when processing input",
"Unknown encoder 'libx265'",
"Output file is empty, nothing was encoded",
"",
):
assert is_worker_infra_error(msg) is False, msg
def test_none_is_not_infra(self):
assert is_worker_infra_error(None) is False # type: ignore[arg-type]
class TestEncodingWorkerInfraError:
def test_is_recoverable_start_error_subclass(self):
"""Subclassing EncodingWorkerStartError is what routes it through the render
worker's park-for-auto-retry path (and the final-encode Cloud Run Job retry)."""
err = EncodingWorkerInfraError("boom", vm_name="encoding-worker-fallback-c4a")
assert isinstance(err, EncodingWorkerStartError)
assert err.vm_name == "encoding-worker-fallback-c4a"