## Why
Prod alert burst (2026-08-31): a paying customer's job (`6452888e`) was
**terminally failed** — its render worker error was:
> Encoding job 6452888e failed: Failed to retrieve
`https://metadata.google.internal/.../service-accounts/default/` …
**Compute Engine Metadata server unavailable** …
`SSLCertVerificationError`
Root cause: the c4d primary pair was in a zonal stockout (503 on start),
so the render fell back to `encoding-worker-fallback-c4a`. That fallback
VM **booted and accepted the job** but then couldn't reach the GCE
metadata server to fetch its service-account token, so it couldn't touch
GCS and died mid-encode. This surfaced as a bare `RuntimeError`, hit
`render_video_worker`'s generic `except Exception` → `fail_job`, and
lost the job with **no retry** — even though the exact same work
succeeds on any healthy worker.
The same alert burst also included the daily E2E happy-path failure (the
encoder being offline made the preview step flaky) and a benign
`match-judge AI call failed; returning no-suggestion` red-error page.
All three are addressed here.
## What
**1. Encoding infra/auth failures are now recoverable (primary fix)**
- `encoding_errors.py`: `is_worker_infra_error()` classifier (metadata /
SSL-cert / missing-credentials markers, deliberately specific so a real
ffmpeg/input error never matches) + new `EncodingWorkerInfraError` — a
**subclass of `EncodingWorkerStartError`** so it flows through the
existing park-for-auto-retry path with zero changes to the render
worker's `except` (and the final-encode Cloud Run Job retries too).
- `encoding_service.wait_for_completion()`: on a worker-reported
infra/auth failure, **demote the broken worker + invalidate the URL
cache**, then raise the recoverable typed error instead of
`RuntimeError`.
- `encoding_worker_manager.demote_active_worker()`: records a
capacity-state cooldown for the serving fallback's `machine_type@zone`
and clears `active_override`, so the retry re-selects a *different,
healthy* VM. Scoped to **fallbacks only** — it never family-demotes the
fast c4d primary on a rare blip.
**2. match-judge noise** — the two graceful-degradation
`logger.exception` calls become `logger.warning(exc_info=True)`, so a
transient Vertex/Gemini blip that safely returns no-suggestion stops
paging as a red "new error pattern".
**3. Happy-path E2E robustness** — tolerate slow encoder cold-starts at
the preview step: scope the alert check to the modal (kills the spurious
empty-`[role=alert]` "Preview error:" from run #151) and recover the
"Proceed to Instrumental" button via reload+reopen. The final assertion
stays authoritative, so genuinely broken generation still fails the
smoke test.
## Testing
- New unit coverage: classifier positive/negative + subclass
(`test_encoding_errors.py`), `demote_active_worker` (fallback
demote+clear, primary no-op, never-raises), `wait_for_completion` infra
classification (+ demote/invalidate, and no-manager path), and render
worker **parks** (not fails) on `EncodingWorkerInfraError`.
- `python -m pytest` across the encoding / worker / match-judge suites:
**177 passed** locally.
## Not in this PR (infra, needs Andrew / GCP write)
- **Fault A**: `encoding-worker-a` (c4d, us-central1-c) is chronically
503 on start (zonal stockout) — that's why the "Encoder offline" footer
showed in the E2E. This change makes the fallback path robust to it, but
the underlying c4d capacity/zone choice is an infra decision (secret/VM
change; my GCP access is read-only). Worth deciding whether to repin the
primary family/zone or promote a fallback.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
* **New Features**
* Detects infrastructure and authentication failures on encoding
workers.
* Demotes affected workers and retries jobs using alternate capacity
when available.
* **Bug Fixes**
* Prevents recoverable worker failures from becoming terminal job
errors.
* Reduces misleading error-level logging for gracefully handled
match-judge failures.
* Improves preview-flow recovery by waiting for actions to become
enabled.
* **Documentation**
* Clarified that retries prefer, but do not guarantee, a different
worker.
* **Chores**
* Updated the application version to 0.217.0.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
---------
Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>