Files
karaoke-gen/backend/api/routes/encoding_worker.py
T
Andrew Beveridge ☄️andClaude Opus 4.8 107542ac39 fix(encoding): scope worker warmup/heartbeat to review-auth, not admin (#862)
## Summary

The lyrics-review page pre-warms the encoding VM on load (and heartbeats
while the reviewer edits) so it's booted by the time they preview a
render. Both endpoints — `POST /api/internal/encoding-worker/warmup` and
`/heartbeat` — required `require_admin` since they were introduced (Mar
25 2026, #595). So **every non-admin customer got a 403 and the JIT
pre-warm silently never fired**. Renders still completed (the encoding
flow's own `ensure_any_running()` boots the VM on demand), but customers
paid the full cold-boot latency the pre-warm was meant to hide. Reported
via 403s in the review-screen devtools console.

## Root cause

`require_admin` is an admin-only gate; the callers are ordinary
customers on a customer-facing page. Admin was the wrong altitude — the
rest of the review surface (`submitCorrections`, `handlers`,
annotations, audio-edit…) uses `require_review_auth`, which the review
page's token already satisfies.

## Fix

Re-scope both endpoints to `require_review_auth` with a `{job_id}` path
param:

- **Backend** (`encoding_worker.py`): `POST /warmup/{job_id}` and
`/heartbeat/{job_id}`, gated on `require_review_auth`. Warming the
shared VM now authorizes exactly like the review page's other calls, so
warmup/heartbeat succeed **precisely when** the review page works —
owners, admins (bypass), and `review_token` users included. Abuse
posture is unchanged: you can only warm the VM if you already have
review access to a real job.
- **Frontend** (`api.ts`, `LyricsAnalyzer.tsx`): thread `jobId` through
`warmupEncodingWorker`/`heartbeatEncodingWorker`, only fire in cloud
mode (`jobId` present — no cloud VM to warm in local mode).

## Changes

- `backend/api/routes/encoding_worker.py` — review-auth + `{job_id}`
path
- `frontend/lib/api.ts` — job-scoped URLs, `jobId` param + guard
- `frontend/components/lyrics-review/LyricsAnalyzer.tsx` — pass `jobId`,
gate on it
- `backend/tests/test_encoding_worker_routes.py` — review-auth contract
+ regression guard that these routes are **not** admin-gated
- `docs/TROUBLESHOOTING.md` — note the auth change and 403 diagnosis
- `pyproject.toml` — 0.188.8 → 0.188.9

## Testing

- Backend: `test_encoding_worker_routes.py` (11) +
manager/persistence/system-status/retry (67 total) +
`test_api_routes.py` (17) — all green. New regression test asserts the
routes depend on `require_review_auth` and **never** `require_admin`,
plus 401-on-no-auth and 404-on-jobless-path.
- Frontend: `jest` lyrics-review + api (319) green; `tsc` clean on both
changed files.
- Prod verification to follow after deploy: open a review page as a
non-admin customer and confirm warmup/heartbeat return 200 (not 403).

@coderabbitai ignore

Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-01 22:25:05 -04:00

97 lines
3.6 KiB
Python

"""
Encoding worker lifecycle endpoints.
Provides warmup and heartbeat endpoints for the blue-green
encoding worker VMs. Called by the lyrics-review frontend to ensure the
primary VM is running before the reviewer previews a render.
Auth: these are gated on ``require_review_auth`` (review access to a specific
job), NOT ``require_admin``. The callers are ordinary customers on the
lyrics-review page — gating on admin makes every non-admin caller 403, which
silently defeats the JIT pre-warm. Scoping to the job being reviewed keeps the
abuse posture identical to the rest of the review surface: you can only warm
the shared encoding VM if you already have review access to a real job.
"""
import logging
from typing import Tuple
from fastapi import APIRouter, Depends
from backend.api.dependencies import require_review_auth
from backend.services.encoding_worker_manager import EncodingWorkerManager
from backend.services.encoding_errors import EncodingWorkerStartError
logger = logging.getLogger(__name__)
router = APIRouter(prefix="/internal/encoding-worker", tags=["encoding-worker"])
# Lazy-loaded singleton
_manager = None
def get_worker_manager():
global _manager
if _manager is None:
from google.cloud import compute_v1, firestore
from backend.config import get_settings
settings = get_settings()
db = firestore.Client(project=settings.google_cloud_project)
compute_client = compute_v1.InstancesClient()
_manager = EncodingWorkerManager(
db=db,
compute_client=compute_client,
project_id=settings.google_cloud_project,
)
return _manager
@router.post("/warmup/{job_id}")
async def warmup_encoding_worker(
job_id: str,
_auth: Tuple[str, str] = Depends(require_review_auth),
manager=Depends(get_worker_manager),
):
"""Start the primary encoding worker VM if it's stopped.
Called on lyrics-review page load so the VM is warm by the time the
reviewer previews a render. ``job_id`` scopes auth to the job under review.
"""
try:
result = manager.ensure_primary_running()
if result["started"]:
logger.info(f"Started encoding worker VM: {result['vm_name']}")
return result
except EncodingWorkerStartError as e:
# The primary VM couldn't start (capacity exhaustion or a transient
# 503 SERVICE_UNAVAILABLE from the GCE backend). This is NOT fatal:
# the encoding flow's own warmup (ensure_any_running) falls back to
# the alternate-zone workers, so the job still completes. Log at
# WARNING so it doesn't trip the production error monitor and page us
# for self-healing capacity events.
logger.warning(
f"Primary encoding worker warmup failed ({e}); "
f"encoding will fall back to an alternate-zone worker"
)
return {"started": False, "error": str(e)}
except Exception as e:
logger.error(f"Failed to warm up encoding worker: {e}")
return {"started": False, "error": str(e)}
@router.post("/heartbeat/{job_id}")
async def heartbeat_encoding_worker(
job_id: str,
_auth: Tuple[str, str] = Depends(require_review_auth),
manager=Depends(get_worker_manager),
):
"""Update activity timestamp to prevent idle shutdown.
Called periodically while the reviewer works. ``job_id`` scopes auth to the
job under review.
"""
try:
manager.update_activity()
return {"status": "ok"}
except Exception as e:
logger.error(f"Failed to update encoding worker heartbeat: {e}")
return {"status": "error", "error": str(e)}