Files
karaoke-gen/backend/api/routes/encoding_worker.py
T
Andrew Beveridge ☄️andClaude Fable 5 7cb5de729a fix: encoding-worker warmup/heartbeat off the event loop (11s instance freeze on review open) (#1029)
## Summary

Follow-up to the concurrent-load reliability work (#1024, #1028),
triggered by Andrew observing a ~5s "Loading audio…" stall opening job
`7a16be60`'s review page.

**Diagnosis from prod request logs (2026-09-20 00:25Z):**
correction-data returned in 0.57s, then the page's parallel batch —
`audio/vocals` (7.5MB), playback audio 302, `sessions`, and the
encoding-worker `warmup` + `heartbeat` — ALL took a uniform ~12s on the
same warm instance, with an **11-second gap containing zero log lines**
(00:25:38.05 → 00:25:49.13). The last logger before the gap and the
first after it: `encoding_worker_manager`.

**Root cause:** `POST /api/internal/encoding-worker/warmup/{job_id}`
(fired by every lyrics-review page open, per its JIT pre-warm design)
called `manager.ensure_primary_running()` — Firestore reads + GCE
compute `instances.get`/`instances.start` (10s+ when the VM is
cold/slow) — **synchronously inside an async handler**. On the
single-process/single-event-loop server this froze the whole instance,
stalling the very page load that fired it. Same bug class as the
v0.230.0 review-path fixes, one router over.

## Changes

- `warmup_encoding_worker` and `heartbeat_encoding_worker` run their
manager calls via `asyncio.to_thread` (behavior otherwise unchanged,
including the soft-fail on `EncodingWorkerStartError`).
- New `backend/tests/test_encoding_worker_routes.py` —
success/soft-fail/heartbeat passthrough plus a ticker-based test proving
the event loop keeps running during a slow warmup (would fail on the old
code).

## Testing

Full backend suite: **4510 passed**. Post-deploy I'll re-check a
review-page open against a cold encoding VM in the request logs (the
audio requests should no longer correlate with warmup latency).

🤖 Generated with [Claude Code](https://claude.com/claude-code)

<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->

## Summary by CodeRabbit

- **Bug Fixes**
- Improved encoding-worker startup and heartbeat responsiveness by
preventing slow operations from blocking other asynchronous activity.
- Encoding-worker startup failures now return a clear soft-failure
response while allowing the service to continue operating.

- **Maintenance**
  - Updated the package version to 0.231.1.

<!-- end of auto-generated comment: release notes by coderabbit.ai -->

Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
2026-09-19 20:47:38 -04:00

105 lines
4.2 KiB
Python

"""
Encoding worker lifecycle endpoints.
Provides warmup and heartbeat endpoints for the blue-green
encoding worker VMs. Called by the lyrics-review frontend to ensure the
primary VM is running before the reviewer previews a render.
Auth: these are gated on ``require_review_auth`` (review access to a specific
job), NOT ``require_admin``. The callers are ordinary customers on the
lyrics-review page — gating on admin makes every non-admin caller 403, which
silently defeats the JIT pre-warm. Scoping to the job being reviewed keeps the
abuse posture identical to the rest of the review surface: you can only warm
the shared encoding VM if you already have review access to a real job.
"""
import asyncio
import logging
from typing import Tuple
from fastapi import APIRouter, Depends
from backend.api.dependencies import require_review_auth
from backend.services.encoding_worker_manager import EncodingWorkerManager
from backend.services.encoding_errors import EncodingWorkerStartError
logger = logging.getLogger(__name__)
router = APIRouter(prefix="/internal/encoding-worker", tags=["encoding-worker"])
# Lazy-loaded singleton
_manager = None
def get_worker_manager():
global _manager
if _manager is None:
from google.cloud import compute_v1, firestore
from backend.config import get_settings
settings = get_settings()
db = firestore.Client(project=settings.google_cloud_project)
compute_client = compute_v1.InstancesClient()
_manager = EncodingWorkerManager(
db=db,
compute_client=compute_client,
project_id=settings.google_cloud_project,
)
return _manager
@router.post("/warmup/{job_id}")
async def warmup_encoding_worker(
job_id: str,
_auth: Tuple[str, str] = Depends(require_review_auth),
manager=Depends(get_worker_manager),
):
"""Start the primary encoding worker VM if it's stopped.
Called on lyrics-review page load so the VM is warm by the time the
reviewer previews a render. ``job_id`` scopes auth to the job under review.
"""
try:
# ensure_primary_running does Firestore reads + GCE compute-API calls
# (instances.get / instances.start) that can take 10s+. Run it off the
# event loop: on the single-process server a sync call here froze the
# WHOLE instance for ~11s on every review-page open with a cold VM,
# stalling that same page's audio/lyrics requests behind it
# (2026-09-19, follow-up to the concurrent-load investigation).
result = await asyncio.to_thread(manager.ensure_primary_running)
if result["started"]:
logger.info(f"Started encoding worker VM: {result['vm_name']}")
return result
except EncodingWorkerStartError as e:
# The primary VM couldn't start (capacity exhaustion or a transient
# 503 SERVICE_UNAVAILABLE from the GCE backend). This is NOT fatal:
# the encoding flow's own warmup (ensure_any_running) falls back to
# the alternate-zone workers, so the job still completes. Log at
# WARNING so it doesn't trip the production error monitor and page us
# for self-healing capacity events.
logger.warning(
f"Primary encoding worker warmup failed ({e}); "
f"encoding will fall back to an alternate-zone worker"
)
return {"started": False, "error": str(e)}
except Exception as e:
logger.error(f"Failed to warm up encoding worker: {e}")
return {"started": False, "error": str(e)}
@router.post("/heartbeat/{job_id}")
async def heartbeat_encoding_worker(
job_id: str,
_auth: Tuple[str, str] = Depends(require_review_auth),
manager=Depends(get_worker_manager),
):
"""Update activity timestamp to prevent idle shutdown.
Called periodically while the reviewer works. ``job_id`` scopes auth to the
job under review.
"""
try:
# Sync Firestore write — keep it off the event loop (see warmup above).
await asyncio.to_thread(manager.update_activity)
return {"status": "ok"}
except Exception as e:
logger.error(f"Failed to update encoding worker heartbeat: {e}")
return {"status": "error", "error": str(e)}