mirror of
https://github.com/nomadkaraoke/karaoke-gen.git
synced 2026-11-20 13:07:57 +09:00
## Summary The lyrics-review page pre-warms the encoding VM on load (and heartbeats while the reviewer edits) so it's booted by the time they preview a render. Both endpoints — `POST /api/internal/encoding-worker/warmup` and `/heartbeat` — required `require_admin` since they were introduced (Mar 25 2026, #595). So **every non-admin customer got a 403 and the JIT pre-warm silently never fired**. Renders still completed (the encoding flow's own `ensure_any_running()` boots the VM on demand), but customers paid the full cold-boot latency the pre-warm was meant to hide. Reported via 403s in the review-screen devtools console. ## Root cause `require_admin` is an admin-only gate; the callers are ordinary customers on a customer-facing page. Admin was the wrong altitude — the rest of the review surface (`submitCorrections`, `handlers`, annotations, audio-edit…) uses `require_review_auth`, which the review page's token already satisfies. ## Fix Re-scope both endpoints to `require_review_auth` with a `{job_id}` path param: - **Backend** (`encoding_worker.py`): `POST /warmup/{job_id}` and `/heartbeat/{job_id}`, gated on `require_review_auth`. Warming the shared VM now authorizes exactly like the review page's other calls, so warmup/heartbeat succeed **precisely when** the review page works — owners, admins (bypass), and `review_token` users included. Abuse posture is unchanged: you can only warm the VM if you already have review access to a real job. - **Frontend** (`api.ts`, `LyricsAnalyzer.tsx`): thread `jobId` through `warmupEncodingWorker`/`heartbeatEncodingWorker`, only fire in cloud mode (`jobId` present — no cloud VM to warm in local mode). ## Changes - `backend/api/routes/encoding_worker.py` — review-auth + `{job_id}` path - `frontend/lib/api.ts` — job-scoped URLs, `jobId` param + guard - `frontend/components/lyrics-review/LyricsAnalyzer.tsx` — pass `jobId`, gate on it - `backend/tests/test_encoding_worker_routes.py` — review-auth contract + regression guard that these routes are **not** admin-gated - `docs/TROUBLESHOOTING.md` — note the auth change and 403 diagnosis - `pyproject.toml` — 0.188.8 → 0.188.9 ## Testing - Backend: `test_encoding_worker_routes.py` (11) + manager/persistence/system-status/retry (67 total) + `test_api_routes.py` (17) — all green. New regression test asserts the routes depend on `require_review_auth` and **never** `require_admin`, plus 401-on-no-auth and 404-on-jobless-path. - Frontend: `jest` lyrics-review + api (319) green; `tsc` clean on both changed files. - Prod verification to follow after deploy: open a review page as a non-admin customer and confirm warmup/heartbeat return 200 (not 403). @coderabbitai ignore Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
97 lines
3.6 KiB
Python
97 lines
3.6 KiB
Python
"""
|
|
Encoding worker lifecycle endpoints.
|
|
|
|
Provides warmup and heartbeat endpoints for the blue-green
|
|
encoding worker VMs. Called by the lyrics-review frontend to ensure the
|
|
primary VM is running before the reviewer previews a render.
|
|
|
|
Auth: these are gated on ``require_review_auth`` (review access to a specific
|
|
job), NOT ``require_admin``. The callers are ordinary customers on the
|
|
lyrics-review page — gating on admin makes every non-admin caller 403, which
|
|
silently defeats the JIT pre-warm. Scoping to the job being reviewed keeps the
|
|
abuse posture identical to the rest of the review surface: you can only warm
|
|
the shared encoding VM if you already have review access to a real job.
|
|
"""
|
|
|
|
import logging
|
|
from typing import Tuple
|
|
from fastapi import APIRouter, Depends
|
|
from backend.api.dependencies import require_review_auth
|
|
from backend.services.encoding_worker_manager import EncodingWorkerManager
|
|
from backend.services.encoding_errors import EncodingWorkerStartError
|
|
|
|
logger = logging.getLogger(__name__)
|
|
|
|
router = APIRouter(prefix="/internal/encoding-worker", tags=["encoding-worker"])
|
|
|
|
# Lazy-loaded singleton
|
|
_manager = None
|
|
|
|
|
|
def get_worker_manager():
|
|
global _manager
|
|
if _manager is None:
|
|
from google.cloud import compute_v1, firestore
|
|
from backend.config import get_settings
|
|
settings = get_settings()
|
|
db = firestore.Client(project=settings.google_cloud_project)
|
|
compute_client = compute_v1.InstancesClient()
|
|
_manager = EncodingWorkerManager(
|
|
db=db,
|
|
compute_client=compute_client,
|
|
project_id=settings.google_cloud_project,
|
|
)
|
|
return _manager
|
|
|
|
|
|
@router.post("/warmup/{job_id}")
|
|
async def warmup_encoding_worker(
|
|
job_id: str,
|
|
_auth: Tuple[str, str] = Depends(require_review_auth),
|
|
manager=Depends(get_worker_manager),
|
|
):
|
|
"""Start the primary encoding worker VM if it's stopped.
|
|
|
|
Called on lyrics-review page load so the VM is warm by the time the
|
|
reviewer previews a render. ``job_id`` scopes auth to the job under review.
|
|
"""
|
|
try:
|
|
result = manager.ensure_primary_running()
|
|
if result["started"]:
|
|
logger.info(f"Started encoding worker VM: {result['vm_name']}")
|
|
return result
|
|
except EncodingWorkerStartError as e:
|
|
# The primary VM couldn't start (capacity exhaustion or a transient
|
|
# 503 SERVICE_UNAVAILABLE from the GCE backend). This is NOT fatal:
|
|
# the encoding flow's own warmup (ensure_any_running) falls back to
|
|
# the alternate-zone workers, so the job still completes. Log at
|
|
# WARNING so it doesn't trip the production error monitor and page us
|
|
# for self-healing capacity events.
|
|
logger.warning(
|
|
f"Primary encoding worker warmup failed ({e}); "
|
|
f"encoding will fall back to an alternate-zone worker"
|
|
)
|
|
return {"started": False, "error": str(e)}
|
|
except Exception as e:
|
|
logger.error(f"Failed to warm up encoding worker: {e}")
|
|
return {"started": False, "error": str(e)}
|
|
|
|
|
|
@router.post("/heartbeat/{job_id}")
|
|
async def heartbeat_encoding_worker(
|
|
job_id: str,
|
|
_auth: Tuple[str, str] = Depends(require_review_auth),
|
|
manager=Depends(get_worker_manager),
|
|
):
|
|
"""Update activity timestamp to prevent idle shutdown.
|
|
|
|
Called periodically while the reviewer works. ``job_id`` scopes auth to the
|
|
job under review.
|
|
"""
|
|
try:
|
|
manager.update_activity()
|
|
return {"status": "ok"}
|
|
except Exception as e:
|
|
logger.error(f"Failed to update encoding worker heartbeat: {e}")
|
|
return {"status": "error", "error": str(e)}
|