mirror of
https://github.com/nomadkaraoke/karaoke-gen.git
synced 2026-11-20 13:07:57 +09:00
feat: concurrent-reliability phases 2-5 — Cloud Run shape, degradation telemetry, banner tuning, load-test guard (#1028)
## Summary Phases 2–5 of the concurrent-load reliability initiative (Phase 1 = #1024, verified live: the 10-tab meltdown went from 9/30 failed requests to 30/30 OK). Verbatim spec: `docs/archive/2026-09-17-concurrent-review-reliability-verbatim-record.md`; plan: `docs/archive/2026-09-18-concurrent-review-reliability-investigation-and-plan.md`. ## P2 — Cloud Run shape - **`--concurrency 80 → 16`** for karaoke-backend: the service is one uvicorn process per 2-vCPU instance, and review loads are heavy; at 80 a burst never triggers scale-out and all lands on the single min-instance. 16 makes bursts fan out while staying well above normal per-instance load. (Deploy flags live in ci.yml — the service is not Pulumi-managed.) - **Local preview ffmpeg fallback bounded** (`LOCAL_PREVIEW_MAX_CONCURRENCY=1`, 900s queue timeout, fast RuntimeError after that): the GCE-outage fallback runs ffmpeg ON the API instance; unbounded, a few concurrent Preview clicks during an encoder outage pegged both vCPUs, stalled the health probe, and showed everyone the orange banner (Andrew's banner-during-preview screenshot). ## P3 — degradation telemetry (the explicit ask) - New **`POST /api/client-events`** (unauthenticated so review-token users report too; 30/min/IP rate limit; bot-filtered; detail capped; URLs query-stripped). Each event → Firestore `client_events` collection **and** a structured `client_event type=…` log line, so "how often are users seeing this?" is reviewable via Firestore aggregation or Cloud Logging correlation. - Frontend reporter `lib/degradation-events.ts` (60s per-type throttle, localhost no-op, never throws), wired into: - `BackendStatusBanner` → `banner_reconnecting` / `banner_unavailable` with stall age, in-flight count, probe state - review lyrics-load failure → `lyrics_load_failed` with message/status - `VocalsAudioDataLoader` → `waveform_slow` (>20s to first strips) and `waveform_failed` (gave up) with poll/retry counts ## P4 — banner tuning - Escalating to the full "We're having trouble reaching our servers" card now requires **two consecutive failed health probes** (`UNAVAILABLE_PROBE_FAILURES=2`). One failed 4s probe against a briefly-pegged instance shows only the calm "Reconnecting…" pill. Single-failure behavior for the pill is unchanged, as is instant clearing on recovery. ## P5 — regression guard - **`scripts/load-test-review.py`** checked in: replays the 10-tab incident (Firestore-ADC review tokens, curl request trios, pass/fail thresholds, exit code). Documented in `docs/TESTING.md`. Validated against prod post-Phase-1: 9/9 requests OK, tabs < 3s. ## Testing - Backend: 6 new tests for client-events (persist/validate/rate-limit/bot/fail-soft/detail-cap) + local-preview bound test — full suite **4517 passed**. - Frontend: new degradation-events suite, two-probe escalation + debug-snapshot tests; one existing api-fetch test updated to the new escalation contract — full suite **1298 passed**. - Live validation of the load-test tool against prod (small 3-tab run): PASS. ## Notes for review - Cloud Run concurrency change means bursts now spin up extra instances (billable during requests only; max-instances unchanged at 20). - Firestore `client_events` has no TTL policy yet (needs a gcloud/console action our read-only ADC can't do); volume is tiny (throttled per client) — can add later if it ever matters. 🤖 Generated with [Claude Code](https://claude.com/claude-code) <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit - **New Features** - Added automatic reporting for degraded experiences, including reconnecting or unavailable backend status, failed lyrics loads, and slow or failed waveform loads. - Added concurrent review-page load testing to help verify performance under multiple open tabs. - **Improvements** - Backend status now remains in “reconnecting” until multiple health checks fail, reducing premature unavailable messages. - Local preview processing is limited during heavy demand to improve stability. - **Documentation** - Added API documentation for client telemetry and guidance for running concurrent-load regression tests. <!-- end of auto-generated comment: release notes by coderabbit.ai --> --------- Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
This commit is contained in:
1 parent
e0f3d1bb45
commit
30cf41e7d6
18 files changed
+912
-18
No files matched your search
@@ -1976,6 +1976,12 @@ jobs:
|
||||
echo "Version: $VERSION"
|
||||
echo "Previously-serving revision: ${PREV_REVISION:-<unknown>}"
|
||||
|
||||
# --concurrency 16 (was 80): the backend is one uvicorn process per
|
||||
# 2-vCPU instance; review-page loads are heavy (audio byte-serving,
|
||||
# waveform work). At 80 a 10-tab burst all landed on the single
|
||||
# min-instance and wedged it ("no available instance" aborts,
|
||||
# 2026-09-18 investigation). 16 makes bursts trigger scale-out while
|
||||
# staying comfortably above normal per-instance load.
|
||||
gcloud run deploy karaoke-backend \
|
||||
--image $IMAGE \
|
||||
--no-traffic \
|
||||
@@ -1989,7 +1995,7 @@ jobs:
|
||||
--cpu-throttling \
|
||||
--execution-environment gen2 \
|
||||
--timeout 1800 \
|
||||
--concurrency 80 \
|
||||
--concurrency 16 \
|
||||
--max-instances 20 \
|
||||
--min-instances 1 \
|
||||
--vpc-connector projects/nomadkaraoke/locations/us-central1/connectors/cloud-run-connector \
|
||||
|
||||
@@ -0,0 +1,135 @@
|
||||
"""POST /api/client-events — persistent telemetry for user-visible degradation.
|
||||
|
||||
Records every time the frontend shows a user a degraded-service surface (the
|
||||
orange "trouble reaching our servers" banner, the stronger "temporarily
|
||||
unavailable" error, a failed lyrics load, a slow/failed waveform), with enough
|
||||
metadata to review frequency and context later. Two sinks per event:
|
||||
|
||||
1. A structured INFO log line (``client_event type=…``) — queryable in Cloud
|
||||
Logging / Log Analytics alongside backend symptoms from the same window.
|
||||
2. A Firestore doc in ``client_events`` — survives log retention, cheap to
|
||||
aggregate for "how often are users seeing this?" reviews.
|
||||
|
||||
Unauthenticated (review-token users must be able to report) and rate-limited
|
||||
per IP, mirroring /api/client-errors. The limiter is process-local, so the
|
||||
effective ceiling is 120/min/IP × live instances — acceptable for this threat
|
||||
model (non-malicious browsers, 60s client-side per-type throttle); Cloudflare's
|
||||
zone-wide rate limit sits in front as a flood backstop. Events are dropped,
|
||||
never queued, when the limiter trips — telemetry, not a delivery guarantee.
|
||||
"""
|
||||
from __future__ import annotations
|
||||
|
||||
import logging
|
||||
import time
|
||||
from datetime import datetime, timezone
|
||||
from typing import Any, Literal, Optional
|
||||
|
||||
from fastapi import APIRouter, HTTPException, Request
|
||||
from pydantic import BaseModel, Field, field_validator
|
||||
|
||||
from backend.services.error_monitor.frontend_ingestion import (
|
||||
RateLimiter,
|
||||
is_bot_user_agent,
|
||||
sanitize_url,
|
||||
)
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
router = APIRouter(prefix="/client-events", tags=["client-events"])
|
||||
|
||||
# Sized for a legitimate worst case from ONE IP: a many-tab burst (e.g. 16 review
|
||||
# tabs at once) where every tab reports a banner + a waveform event — that must be
|
||||
# fully recorded, not self-suppressed. Each tab throttles per-type client-side.
|
||||
_limiter = RateLimiter(max_per_minute=120)
|
||||
_db_singleton = None
|
||||
|
||||
# The full closed set of reportable events. Reject anything else so the
|
||||
# collection stays reviewable (no free-form types accumulating).
|
||||
EventType = Literal[
|
||||
"banner_reconnecting",
|
||||
"banner_unavailable",
|
||||
"lyrics_load_failed",
|
||||
"waveform_slow",
|
||||
"waveform_failed",
|
||||
]
|
||||
|
||||
|
||||
def _get_db():
|
||||
global _db_singleton
|
||||
if _db_singleton is None:
|
||||
from google.cloud import firestore # type: ignore[import]
|
||||
|
||||
_db_singleton = firestore.Client(project="nomadkaraoke")
|
||||
return _db_singleton
|
||||
|
||||
|
||||
class ClientEventPayload(BaseModel):
|
||||
type: EventType
|
||||
url: str = Field("", max_length=2048)
|
||||
job_id: Optional[str] = Field(None, max_length=64)
|
||||
user_email: Optional[str] = Field(None, max_length=320)
|
||||
locale: str = Field("en", max_length=10)
|
||||
release: str = Field("", max_length=64)
|
||||
# Small free-form context (stall age, http status, elapsed ms, probe state…).
|
||||
# Size-capped server-side; values are stored as-is.
|
||||
detail: Optional[dict] = None
|
||||
|
||||
@field_validator("url", "locale", "release")
|
||||
@classmethod
|
||||
def strip(cls, v: str) -> str:
|
||||
return (v or "").strip()
|
||||
|
||||
|
||||
def _capped_detail(detail: Optional[dict]) -> dict[str, Any]:
|
||||
"""Keep detail reviewable: at most 12 keys, scalar-ish values, short strings."""
|
||||
if not isinstance(detail, dict):
|
||||
return {}
|
||||
out: dict[str, Any] = {}
|
||||
for key, value in list(detail.items())[:12]:
|
||||
k = str(key)[:40]
|
||||
if isinstance(value, (int, float, bool)) or value is None:
|
||||
out[k] = value
|
||||
else:
|
||||
out[k] = str(value)[:200]
|
||||
return out
|
||||
|
||||
|
||||
@router.post("", status_code=202)
|
||||
def report_client_event(payload: ClientEventPayload, request: Request) -> dict:
|
||||
client_ip = request.client.host if request.client else "unknown"
|
||||
if not _limiter.allow(client_ip, time.monotonic()):
|
||||
raise HTTPException(status_code=429, detail="too many events")
|
||||
|
||||
user_agent = request.headers.get("user-agent", "")[:1024]
|
||||
if is_bot_user_agent(user_agent):
|
||||
return {"status": "ignored"}
|
||||
|
||||
event = {
|
||||
"type": payload.type,
|
||||
"url": sanitize_url(payload.url),
|
||||
"job_id": payload.job_id,
|
||||
"user_email": payload.user_email,
|
||||
"locale": payload.locale,
|
||||
"release": payload.release,
|
||||
"user_agent": user_agent,
|
||||
"detail": _capped_detail(payload.detail),
|
||||
"created_at": datetime.now(timezone.utc),
|
||||
}
|
||||
|
||||
# Sink 1: structured log line — correlate with backend symptoms in Logging.
|
||||
logger.info(
|
||||
"client_event type=%s job_id=%s user=%s url=%s release=%s detail=%s",
|
||||
event["type"],
|
||||
event["job_id"],
|
||||
event["user_email"],
|
||||
event["url"],
|
||||
event["release"],
|
||||
event["detail"],
|
||||
)
|
||||
|
||||
# Sink 2: Firestore — best-effort; telemetry must never error back to users.
|
||||
try:
|
||||
_get_db().collection("client_events").add(event)
|
||||
except Exception:
|
||||
logger.exception("Failed to persist client event (non-fatal)")
|
||||
|
||||
return {"status": "recorded"}
|
||||
@@ -13,6 +13,7 @@ Usage:
|
||||
The baseApiUrl includes the job_id, and all endpoints are relative to that.
|
||||
"""
|
||||
import asyncio
|
||||
import concurrent.futures
|
||||
import logging
|
||||
import hashlib
|
||||
import json
|
||||
@@ -278,12 +279,65 @@ def _prepare_preview_inputs(job, temp_dir: str, storage, is_duet: bool):
|
||||
return correction_result, audio_path, output_config
|
||||
|
||||
|
||||
def _render_preview_locally(job, updated_data: Dict[str, Any], is_duet: bool) -> str:
|
||||
"""Render the full 360p preview locally and upload it to GCS.
|
||||
# At most this many local preview renders at once. The local path runs ffmpeg
|
||||
# ON the 2-vCPU API instance (it's the fallback when GCE preview encoding is
|
||||
# down); unbounded, a GCE outage plus a few users clicking Preview saturates
|
||||
# the instance's CPU, stalls the health probe, and shows everyone the orange
|
||||
# "trouble reaching our servers" banner. Serializing renders keeps the API
|
||||
# responsive — previews just queue a little longer during an encoder outage.
|
||||
_LOCAL_PREVIEW_MAX_CONCURRENCY = max(1, int(os.getenv("LOCAL_PREVIEW_MAX_CONCURRENCY", "1")))
|
||||
_LOCAL_PREVIEW_QUEUE_TIMEOUT_S = int(os.getenv("LOCAL_PREVIEW_QUEUE_TIMEOUT_S", "900"))
|
||||
_local_preview_slots = asyncio.Semaphore(_LOCAL_PREVIEW_MAX_CONCURRENCY)
|
||||
# Dedicated executor: admitted renders never compete with the shared to_thread
|
||||
# pool, and its concurrent.futures-level done callbacks fire when the WORKER
|
||||
# THREAD finishes — the asyncio wrapper future can be "cancelled" while the
|
||||
# thread is still running, so it must not drive the slot release.
|
||||
_local_preview_executor = concurrent.futures.ThreadPoolExecutor(
|
||||
max_workers=_LOCAL_PREVIEW_MAX_CONCURRENCY, thread_name_prefix="local-preview"
|
||||
)
|
||||
|
||||
Blocking (downloads + ffmpeg) — call via asyncio.to_thread. Used when GCE
|
||||
preview encoding is disabled and as the fallback when it fails.
|
||||
|
||||
async def _render_preview_locally(job, updated_data: Dict[str, Any], is_duet: bool) -> str:
|
||||
"""Render the full 360p preview locally (bounded) and upload it to GCS.
|
||||
|
||||
Used when GCE preview encoding is disabled and as the fallback when it
|
||||
fails. Admission happens HERE on the event loop (async semaphore +
|
||||
wait_for) so queued renders wait as cheap coroutines — only an ADMITTED
|
||||
render occupies a thread-pool worker for its blocking downloads + ffmpeg.
|
||||
"""
|
||||
try:
|
||||
await asyncio.wait_for(
|
||||
_local_preview_slots.acquire(), timeout=_LOCAL_PREVIEW_QUEUE_TIMEOUT_S
|
||||
)
|
||||
except asyncio.TimeoutError:
|
||||
raise RuntimeError(
|
||||
"Local preview encoder is busy (waited "
|
||||
f"{_LOCAL_PREVIEW_QUEUE_TIMEOUT_S}s for a slot) — try again shortly"
|
||||
)
|
||||
def _run_render():
|
||||
return _render_preview_locally_unbounded(job, updated_data, is_duet)
|
||||
|
||||
loop = asyncio.get_running_loop()
|
||||
worker_future = _local_preview_executor.submit(_run_render)
|
||||
|
||||
def _release_slot(fut: "concurrent.futures.Future") -> None:
|
||||
# Runs when the WORKER THREAD actually finishes. Releasing from an
|
||||
# async finally (or the asyncio wrapper's callback) would free the slot
|
||||
# the moment the awaiting request is cancelled — while the ffmpeg
|
||||
# render thread is still consuming CPU — letting a second render start
|
||||
# and defeating the concurrency bound.
|
||||
if not fut.cancelled():
|
||||
fut.exception() # mark retrieved when the caller was cancelled/abandoned
|
||||
try:
|
||||
loop.call_soon_threadsafe(_local_preview_slots.release)
|
||||
except RuntimeError:
|
||||
pass # loop already closed (shutdown) — nothing left to bound
|
||||
|
||||
worker_future.add_done_callback(_release_slot)
|
||||
return await asyncio.wrap_future(worker_future)
|
||||
|
||||
|
||||
def _render_preview_locally_unbounded(job, updated_data: Dict[str, Any], is_duet: bool) -> str:
|
||||
job_id = job.job_id
|
||||
storage = StorageService()
|
||||
with tempfile.TemporaryDirectory() as temp_dir:
|
||||
@@ -349,7 +403,7 @@ async def _encode_preview_in_background(
|
||||
logger.warning(f"Job {job_id}: GCE preview encoding failed, falling back to local: {gce_error}")
|
||||
|
||||
try:
|
||||
await asyncio.to_thread(_render_preview_locally, job, updated_data, is_duet)
|
||||
await _render_preview_locally(job, updated_data, is_duet)
|
||||
logger.info(f"Job {job_id}: Preview generated locally after GCE failure: {preview_hash}")
|
||||
except Exception as local_error:
|
||||
logger.error(
|
||||
@@ -1873,9 +1927,7 @@ async def generate_preview_video(
|
||||
|
||||
# Local path (GCE preview disabled, or GCE setup failed): render
|
||||
# the full preview synchronously — no cold-start problem, but slow.
|
||||
preview_hash = await asyncio.to_thread(
|
||||
_render_preview_locally, job, updated_data, is_duet
|
||||
)
|
||||
preview_hash = await _render_preview_locally(job, updated_data, is_duet)
|
||||
|
||||
logger.info(f"Job {job_id}: Preview video generated: {preview_hash}")
|
||||
span.set_attribute("preview_hash", preview_hash)
|
||||
|
||||
+2
-1
@@ -10,7 +10,7 @@ from fastapi import FastAPI
|
||||
from fastapi.middleware.cors import CORSMiddleware
|
||||
|
||||
from backend.config import settings, validate_production_config
|
||||
from backend.api.routes import health, jobs, internal, file_upload, review, auth, audio_search, themes, users, admin, tenant, tenant_admin, tenant_bulk, rate_limits, push, catalog, encoding_worker, client_errors, bulk, parse_titles
|
||||
from backend.api.routes import health, jobs, internal, file_upload, review, auth, audio_search, themes, users, admin, tenant, tenant_admin, tenant_bulk, rate_limits, push, catalog, encoding_worker, client_errors, client_events, bulk, parse_titles
|
||||
from backend.services.tracing import setup_tracing, instrument_app, get_current_trace_id
|
||||
from backend.services.structured_logging import setup_structured_logging
|
||||
from backend.services.spacy_preloader import preload_spacy_model
|
||||
@@ -213,6 +213,7 @@ app.include_router(bulk.router, prefix="/api") # Bulk Mode: multi-job submissio
|
||||
from backend.api.routes import requests_board
|
||||
app.include_router(requests_board.router, prefix="/api") # Public song-request voting board
|
||||
app.include_router(client_errors.router, prefix="/api") # Frontend crash reports
|
||||
app.include_router(client_events.router, prefix="/api") # Frontend degradation telemetry (banner/unavailable/waveform events)
|
||||
app.include_router(tenant.router) # Tenant/white-label configuration (no /api prefix, router has it)
|
||||
app.include_router(tenant_admin.router) # Admin tenant provisioning (router has /api prefix)
|
||||
app.include_router(tenant_bulk.router) # Tenant bulk-upload filename analysis (router has /api prefix)
|
||||
|
||||
@@ -0,0 +1,85 @@
|
||||
"""Tests for POST /api/client-events (degradation telemetry)."""
|
||||
|
||||
from unittest.mock import MagicMock, patch
|
||||
|
||||
import pytest
|
||||
from fastapi import FastAPI
|
||||
from fastapi.testclient import TestClient
|
||||
|
||||
from backend.api.routes import client_events
|
||||
|
||||
|
||||
@pytest.fixture
|
||||
def client():
|
||||
app = FastAPI()
|
||||
app.include_router(client_events.router, prefix="/api")
|
||||
return TestClient(app)
|
||||
|
||||
|
||||
@pytest.fixture(autouse=True)
|
||||
def fresh_limiter_and_db(monkeypatch):
|
||||
monkeypatch.setattr(client_events, "_limiter", client_events.RateLimiter(max_per_minute=30))
|
||||
db = MagicMock()
|
||||
monkeypatch.setattr(client_events, "_db_singleton", db)
|
||||
return db
|
||||
|
||||
|
||||
def _event(**overrides):
|
||||
payload = {
|
||||
"type": "banner_unavailable",
|
||||
"url": "https://gen.nomadkaraoke.com/app/jobs?token=SECRET#/abc123/review",
|
||||
"job_id": "abc123",
|
||||
"locale": "en",
|
||||
"release": "deadbeef",
|
||||
"detail": {"stall_ms": 21000, "probe_ok": False},
|
||||
}
|
||||
payload.update(overrides)
|
||||
return payload
|
||||
|
||||
|
||||
class TestReportClientEvent:
|
||||
def test_valid_event_persists_to_firestore(self, client, fresh_limiter_and_db):
|
||||
resp = client.post("/api/client-events", json=_event())
|
||||
|
||||
assert resp.status_code == 202
|
||||
assert resp.json() == {"status": "recorded"}
|
||||
fresh_limiter_and_db.collection.assert_called_once_with("client_events")
|
||||
(doc,) = fresh_limiter_and_db.collection.return_value.add.call_args[0]
|
||||
assert doc["type"] == "banner_unavailable"
|
||||
assert doc["job_id"] == "abc123"
|
||||
assert doc["detail"] == {"stall_ms": 21000, "probe_ok": False}
|
||||
# Query string (may carry tokens) must be stripped before persisting.
|
||||
assert "SECRET" not in doc["url"]
|
||||
|
||||
def test_unknown_event_type_rejected(self, client):
|
||||
resp = client.post("/api/client-events", json=_event(type="something_else"))
|
||||
assert resp.status_code == 422
|
||||
|
||||
def test_rate_limited(self, client, monkeypatch):
|
||||
monkeypatch.setattr(client_events, "_limiter", client_events.RateLimiter(max_per_minute=2))
|
||||
assert client.post("/api/client-events", json=_event()).status_code == 202
|
||||
assert client.post("/api/client-events", json=_event()).status_code == 202
|
||||
assert client.post("/api/client-events", json=_event()).status_code == 429
|
||||
|
||||
def test_bot_user_agent_ignored(self, client, fresh_limiter_and_db):
|
||||
resp = client.post(
|
||||
"/api/client-events",
|
||||
json=_event(),
|
||||
headers={"User-Agent": "Googlebot/2.1 (+http://www.google.com/bot.html)"},
|
||||
)
|
||||
assert resp.status_code == 202
|
||||
assert resp.json() == {"status": "ignored"}
|
||||
fresh_limiter_and_db.collection.assert_not_called()
|
||||
|
||||
def test_firestore_failure_still_returns_202(self, client, fresh_limiter_and_db):
|
||||
fresh_limiter_and_db.collection.return_value.add.side_effect = Exception("firestore down")
|
||||
resp = client.post("/api/client-events", json=_event())
|
||||
assert resp.status_code == 202 # telemetry must never error back to users
|
||||
|
||||
def test_detail_is_capped(self):
|
||||
big = {f"key{i}": "x" * 500 for i in range(30)}
|
||||
capped = client_events._capped_detail(big)
|
||||
assert len(capped) == 12
|
||||
assert all(len(v) <= 200 for v in capped.values())
|
||||
assert client_events._capped_detail(None) == {}
|
||||
assert client_events._capped_detail("not a dict") == {}
|
||||
@@ -1825,7 +1825,7 @@ class TestEncodePreviewInBackground:
|
||||
encoding_service = MagicMock()
|
||||
encoding_service.encode_preview_video = AsyncMock(side_effect=RuntimeError("stockout"))
|
||||
storage = MagicMock()
|
||||
local_render = MagicMock(return_value="h1")
|
||||
local_render = AsyncMock(return_value="h1")
|
||||
self._run(mock_job, encoding_service, storage, local_render)
|
||||
local_render.assert_called_once()
|
||||
storage.upload_json.assert_not_called()
|
||||
@@ -1836,7 +1836,7 @@ class TestEncodePreviewInBackground:
|
||||
encoding_service = MagicMock()
|
||||
encoding_service.encode_preview_video = AsyncMock(side_effect=RuntimeError("stockout"))
|
||||
storage = MagicMock()
|
||||
local_render = MagicMock(side_effect=RuntimeError("ffmpeg died"))
|
||||
local_render = AsyncMock(side_effect=RuntimeError("ffmpeg died"))
|
||||
self._run(mock_job, encoding_service, storage, local_render)
|
||||
storage.upload_json.assert_called_once_with(
|
||||
"jobs/job-bg/previews/h1.error.json",
|
||||
@@ -2041,3 +2041,101 @@ class TestReviewAudioSingleFlightAndByteCap:
|
||||
assert "a" not in review_module._review_audio_cache
|
||||
assert review_module._review_audio_cache_bytes <= 100
|
||||
assert set(review_module._review_audio_cache) == {"b", "c", "d", "e"}
|
||||
|
||||
|
||||
class TestLocalPreviewRenderBound:
|
||||
"""The local preview ffmpeg fallback must be bounded so a GCE outage plus a
|
||||
few concurrent Preview clicks can't saturate the API instance's CPU.
|
||||
Admission is an async semaphore on the event loop (waiters are cheap
|
||||
coroutines, not parked executor threads)."""
|
||||
|
||||
def test_serializes_and_times_out_admission(self, monkeypatch):
|
||||
import asyncio
|
||||
import threading
|
||||
from backend.api.routes import review as review_module
|
||||
|
||||
monkeypatch.setattr(review_module, "_LOCAL_PREVIEW_QUEUE_TIMEOUT_S", 0.2)
|
||||
monkeypatch.setattr(review_module, "_local_preview_slots", asyncio.Semaphore(1))
|
||||
|
||||
started = threading.Event()
|
||||
release = threading.Event()
|
||||
|
||||
def slow_render(job, updated_data, is_duet):
|
||||
started.set()
|
||||
release.wait(timeout=5)
|
||||
return "hash"
|
||||
|
||||
monkeypatch.setattr(review_module, "_render_preview_locally_unbounded", slow_render)
|
||||
|
||||
async def scenario():
|
||||
first = asyncio.create_task(
|
||||
review_module._render_preview_locally(MagicMock(), {}, False)
|
||||
)
|
||||
await asyncio.to_thread(started.wait, 5)
|
||||
|
||||
# Slot held by the first render — the second caller's admission wait
|
||||
# times out fast instead of piling more ffmpeg onto the instance.
|
||||
with pytest.raises(RuntimeError, match="busy"):
|
||||
await review_module._render_preview_locally(MagicMock(), {}, False)
|
||||
|
||||
release.set()
|
||||
assert await first == "hash"
|
||||
|
||||
# Slot released — a fresh render is admitted immediately.
|
||||
monkeypatch.setattr(
|
||||
review_module, "_render_preview_locally_unbounded",
|
||||
lambda job, updated_data, is_duet: "hash2",
|
||||
)
|
||||
assert await review_module._render_preview_locally(MagicMock(), {}, False) == "hash2"
|
||||
|
||||
asyncio.run(scenario())
|
||||
|
||||
def test_cancelled_caller_does_not_free_slot_while_render_runs(self, monkeypatch):
|
||||
"""Cancelling the awaiting request must NOT release the slot while the
|
||||
ffmpeg worker thread is still running — otherwise a second render can
|
||||
start and defeat the concurrency bound."""
|
||||
import asyncio
|
||||
import threading
|
||||
from backend.api.routes import review as review_module
|
||||
|
||||
monkeypatch.setattr(review_module, "_LOCAL_PREVIEW_QUEUE_TIMEOUT_S", 0.2)
|
||||
monkeypatch.setattr(review_module, "_local_preview_slots", asyncio.Semaphore(1))
|
||||
|
||||
started = threading.Event()
|
||||
release = threading.Event()
|
||||
|
||||
def slow_render(job, updated_data, is_duet):
|
||||
started.set()
|
||||
release.wait(timeout=5)
|
||||
return "hash"
|
||||
|
||||
monkeypatch.setattr(review_module, "_render_preview_locally_unbounded", slow_render)
|
||||
|
||||
async def scenario():
|
||||
first = asyncio.create_task(
|
||||
review_module._render_preview_locally(MagicMock(), {}, False)
|
||||
)
|
||||
await asyncio.to_thread(started.wait, 5)
|
||||
|
||||
first.cancel()
|
||||
with pytest.raises(asyncio.CancelledError):
|
||||
await first
|
||||
|
||||
# Worker thread is still rendering — the slot must still be held.
|
||||
with pytest.raises(RuntimeError, match="busy"):
|
||||
await review_module._render_preview_locally(MagicMock(), {}, False)
|
||||
|
||||
# Worker finishes → done-callback releases the slot.
|
||||
release.set()
|
||||
for _ in range(50):
|
||||
if not review_module._local_preview_slots.locked():
|
||||
break
|
||||
await asyncio.sleep(0.05)
|
||||
|
||||
monkeypatch.setattr(
|
||||
review_module, "_render_preview_locally_unbounded",
|
||||
lambda job, updated_data, is_duet: "hash2",
|
||||
)
|
||||
assert await review_module._render_preview_locally(MagicMock(), {}, False) == "hash2"
|
||||
|
||||
asyncio.run(scenario())
|
||||
+27
@@ -2669,3 +2669,30 @@ search returned nothing usable.
|
||||
|
||||
Stripe webhooks implemented at `/api/users/webhooks/stripe`.
|
||||
Job status webhooks not yet implemented.
|
||||
|
||||
## Client Telemetry
|
||||
|
||||
### Degradation Events
|
||||
|
||||
```http
|
||||
POST /api/client-events
|
||||
```
|
||||
|
||||
Unauthenticated, rate-limited (120/min/IP — sized so a many-tab burst from one IP is fully recorded), bot-filtered. The frontend reports every
|
||||
user-visible degraded-service surface: `banner_reconnecting`, `banner_unavailable`
|
||||
(the connectivity banner — reported by `BackendStatusBanner`), `lyrics_load_failed`
|
||||
(review page "temporarily unavailable"), `waveform_slow` / `waveform_failed`
|
||||
(Waveforms-mode strips late or missing). Body: `{type, url, job_id?, user_email?,
|
||||
locale, release, detail?}` — `detail` is size-capped server-side (≤12 keys, ≤200
|
||||
chars per value); URLs are stored query-stripped.
|
||||
|
||||
Each event emits a structured `client_event type=…` INFO log line and is
|
||||
best-effort persisted to the Firestore `client_events` collection (a Firestore
|
||||
write failure is logged but still returns `202` — telemetry never errors back to
|
||||
the client), so frequency can be reviewed later
|
||||
(Firestore aggregation, or Cloud Logging correlated with backend symptoms from the
|
||||
same window). Reporter: `frontend/lib/degradation-events.ts` (60s per-type throttle,
|
||||
localhost no-op). Added in v0.231.0 — see
|
||||
`docs/archive/2026-09-18-concurrent-review-reliability-investigation-and-plan.md`.
|
||||
|
||||
Crash reports are the separate `POST /api/client-errors` (error-monitor pipeline).
|
||||
@@ -456,6 +456,25 @@ The production tests cover:
|
||||
- Final encoding and download
|
||||
- Admin dashboard functionality
|
||||
|
||||
## Concurrent-Load Regression Test (review pages)
|
||||
|
||||
`scripts/load-test-review.py` replays the 2026-09-18 "10 tabs melt the backend"
|
||||
incident: N simultaneous review-tab loads, each firing the page's real GET trio
|
||||
(correction-data, waveform-data, ranged audio/vocals), authenticated with per-job
|
||||
review tokens read from Firestore via ADC (no admin secret needed).
|
||||
|
||||
```bash
|
||||
python scripts/load-test-review.py # 10 tabs vs prod, warm thresholds
|
||||
python scripts/load-test-review.py --tabs 20
|
||||
python scripts/load-test-review.py --max-tab-seconds 45 # cold caches (first-ever waveforms)
|
||||
```
|
||||
|
||||
Pass = every request 2xx and every tab under the threshold; exit 1 otherwise.
|
||||
Run it after changes to the review hot path (`backend/api/routes/review.py`,
|
||||
`audio_transcoding_service`, `audio_analysis_service`) or Cloud Run deploy flags.
|
||||
Background + before/after numbers:
|
||||
`docs/archive/2026-09-18-concurrent-review-reliability-investigation-and-plan.md`.
|
||||
|
||||
## Ad-Hoc Production Debugging (for Agents)
|
||||
|
||||
When asked to "test it yourself in prod" or debug a specific production issue, use the debug script template:
|
||||
|
||||
@@ -131,10 +131,16 @@ describe('apiFetch', () => {
|
||||
// stays hidden (a slow endpoint must not alarm the user without evidence).
|
||||
await jest.advanceTimersByTimeAsync(STALL_RECONNECTING_MS)
|
||||
expect(getBackendStatus()).toBe('online')
|
||||
// Probe times out (~4s) → outage confirmed → the hint appears.
|
||||
// Probe times out (~4s) → outage suspected → the hint appears.
|
||||
await jest.advanceTimersByTimeAsync(5_000)
|
||||
expect(getBackendStatus()).toBe('reconnecting')
|
||||
// The stall alone now qualifies for "unavailable", but escalation needs a
|
||||
// SECOND consecutive probe failure (one failed probe can just be a briefly
|
||||
// pegged instance) — until then the calm pill is all the user sees.
|
||||
await jest.advanceTimersByTimeAsync(STALL_UNAVAILABLE_MS - STALL_RECONNECTING_MS - 5_000)
|
||||
expect(getBackendStatus()).toBe('reconnecting')
|
||||
// Verdict goes stale (~24s) → probe #2 fires → times out (~28s) → confirmed.
|
||||
await jest.advanceTimersByTimeAsync(9_000)
|
||||
expect(getBackendStatus()).toBe('unavailable')
|
||||
|
||||
settleGet(mockResponse(200, { ok: true }))
|
||||
|
||||
@@ -8,6 +8,7 @@
|
||||
|
||||
import {
|
||||
getBackendStatus,
|
||||
getBackendStatusDebug,
|
||||
beginRequest,
|
||||
endRequest,
|
||||
configureHealthProbe,
|
||||
@@ -158,4 +159,53 @@ describe('backend-status store (health-probe confirmation)', () => {
|
||||
await jest.advanceTimersByTimeAsync(3000) // verdict lands
|
||||
expect(getBackendStatus()).toBe('reconnecting')
|
||||
})
|
||||
|
||||
it('needs TWO consecutive failed probes before escalating to unavailable', async () => {
|
||||
// Each probe takes 3s to fail — realistic for a probe racing its own
|
||||
// timeout against a briefly-pegged (not down) instance.
|
||||
const probe = jest.fn(
|
||||
() => new Promise<boolean>((res) => setTimeout(() => res(false), 3000)),
|
||||
)
|
||||
configureHealthProbe(probe)
|
||||
|
||||
open.push(beginRequest())
|
||||
// t=10s: probe #1 starts. t=13s: failure #1 recorded.
|
||||
// t=20s: the stall alone now qualifies for "unavailable", but only one
|
||||
// probe failure has been recorded (verdict fresh until t=23s) —
|
||||
// the calm reconnecting pill must be all the user sees.
|
||||
await jest.advanceTimersByTimeAsync(STALL_UNAVAILABLE_MS + 1000) // t=21s
|
||||
expect(getBackendStatus()).toBe('reconnecting')
|
||||
expect(getBackendStatusDebug().consecutiveProbeFailures).toBe(1)
|
||||
|
||||
// t=23s: verdict stale → probe #2 starts. t=26s: failure #2 → escalate.
|
||||
await jest.advanceTimersByTimeAsync(6000) // t=27s
|
||||
expect(getBackendStatus()).toBe('unavailable')
|
||||
expect(getBackendStatusDebug().consecutiveProbeFailures).toBe(2)
|
||||
})
|
||||
|
||||
it('a probe success resets the consecutive-failure count', async () => {
|
||||
let reachable = false
|
||||
const probe = jest.fn(() => Promise.resolve(reachable))
|
||||
configureHealthProbe(probe)
|
||||
|
||||
open.push(beginRequest())
|
||||
await jest.advanceTimersByTimeAsync(STALL_UNAVAILABLE_MS + PROBE_FRESH_MS)
|
||||
expect(getBackendStatusDebug().consecutiveProbeFailures).toBeGreaterThanOrEqual(2)
|
||||
|
||||
reachable = true
|
||||
await jest.advanceTimersByTimeAsync(PROBE_FRESH_MS + 2000)
|
||||
expect(getBackendStatus()).toBe('online')
|
||||
expect(getBackendStatusDebug().consecutiveProbeFailures).toBe(0)
|
||||
})
|
||||
|
||||
it('exposes a diagnostic snapshot for telemetry', async () => {
|
||||
configureHealthProbe(() => Promise.resolve(false))
|
||||
open.push(beginRequest())
|
||||
await jest.advanceTimersByTimeAsync(STALL_RECONNECTING_MS + 1000)
|
||||
|
||||
const debug = getBackendStatusDebug()
|
||||
expect(debug.inFlightCount).toBe(1)
|
||||
expect(debug.oldestStallMs).toBeGreaterThanOrEqual(STALL_RECONNECTING_MS)
|
||||
expect(debug.lastProbeOk).toBe(false)
|
||||
})
|
||||
})
|
||||
@@ -1,6 +1,7 @@
|
||||
"use client"
|
||||
|
||||
import { useEffect, useState, useCallback, useRef } from "react"
|
||||
import { reportDegradationEvent } from "@/lib/degradation-events"
|
||||
import { useAuth } from "@/lib/auth"
|
||||
import { api, Job, createLyricsReviewApiClient, lyricsReviewApi } from "@/lib/api"
|
||||
import { Spinner } from "@/components/ui/spinner"
|
||||
@@ -334,6 +335,10 @@ function LyricsReviewWrapper({ job, isLocalMode = false, isReplay = false }: { j
|
||||
setCorrectionData(data)
|
||||
} catch (err) {
|
||||
console.error("Failed to load correction data:", err)
|
||||
reportDegradationEvent("lyrics_load_failed", {
|
||||
message: err instanceof Error ? err.message : String(err),
|
||||
status: (err as { status?: number })?.status ?? null,
|
||||
})
|
||||
setError(err instanceof Error ? err.message : "Failed to load lyrics data")
|
||||
} finally {
|
||||
setIsLoading(false)
|
||||
|
||||
@@ -5,8 +5,10 @@ import { useTranslations } from "next-intl"
|
||||
import { Loader2, CloudOff, X } from "lucide-react"
|
||||
import {
|
||||
useBackendStatus,
|
||||
getBackendStatusDebug,
|
||||
__installBackendStatusDevHook,
|
||||
} from "@/lib/backend-status"
|
||||
import { reportDegradationEvent } from "@/lib/degradation-events"
|
||||
|
||||
/**
|
||||
* App-wide, non-blocking banner that reacts to backend connectivity (see
|
||||
@@ -37,6 +39,26 @@ export function BackendStatusBanner() {
|
||||
if (status === "online") setDismissed(false)
|
||||
}, [status])
|
||||
|
||||
// Persist every banner display server-side (Andrew's telemetry ask): this is
|
||||
// the single global banner instance, so reporting here == "shown to a user".
|
||||
// The reporter throttles per type, so a flapping status can't spam.
|
||||
useEffect(() => {
|
||||
if (status === "online") return
|
||||
// A dismissed unavailable card renders nothing — reporting it would count a
|
||||
// banner the user never saw. (dismiss only re-arms after passing online.)
|
||||
if (status === "unavailable" && dismissed) return
|
||||
const debug = getBackendStatusDebug()
|
||||
reportDegradationEvent(
|
||||
status === "unavailable" ? "banner_unavailable" : "banner_reconnecting",
|
||||
{
|
||||
stall_ms: debug.oldestStallMs,
|
||||
in_flight: debug.inFlightCount,
|
||||
probe_ok: debug.lastProbeOk,
|
||||
probe_failures: debug.consecutiveProbeFailures,
|
||||
},
|
||||
)
|
||||
}, [status, dismissed])
|
||||
|
||||
if (status === "online") return null
|
||||
|
||||
if (status === "reconnecting") {
|
||||
|
||||
@@ -1,6 +1,12 @@
|
||||
import { AudioData, AudioNotReadyError, fetchAudioData, isTransientAudioError } from '@/lib/audio-data'
|
||||
import { reportDegradationEvent } from '@/lib/degradation-events'
|
||||
import { createContext, PropsWithChildren, useEffect, useState } from 'react'
|
||||
|
||||
// A first paint whose waveform strips took longer than this to appear counts as
|
||||
// a degraded experience worth recording (Andrew's "wait 1+ minutes" complaint).
|
||||
// Excludes legitimate separation-in-progress waits shorter than one 202 poll.
|
||||
const SLOW_LOAD_REPORT_MS = 20_000
|
||||
|
||||
export const VocalsAudioDataLoaderContext = createContext<{ audioData: AudioData | null }>({ audioData: null })
|
||||
|
||||
export interface AudioDataLoaderProps extends PropsWithChildren {
|
||||
@@ -30,11 +36,21 @@ export const VocalsAudioDataLoader = ({ audioUrl, children }: AudioDataLoaderPro
|
||||
// audioUrl has already been set.
|
||||
let cancelled = false
|
||||
let retryTimer: ReturnType<typeof setTimeout> | undefined
|
||||
const startedAt = Date.now()
|
||||
|
||||
const load = (attempt: number, transientAttempt: number) => {
|
||||
fetchAudioData(audioUrl)
|
||||
.then((audioData) => {
|
||||
if (!cancelled) setAudioData(audioData)
|
||||
if (cancelled) return
|
||||
const elapsedMs = Date.now() - startedAt
|
||||
if (elapsedMs >= SLOW_LOAD_REPORT_MS) {
|
||||
reportDegradationEvent('waveform_slow', {
|
||||
elapsed_ms: elapsedMs,
|
||||
not_ready_polls: attempt,
|
||||
transient_retries: transientAttempt,
|
||||
})
|
||||
}
|
||||
setAudioData(audioData)
|
||||
})
|
||||
.catch((error) => {
|
||||
if (cancelled) return
|
||||
@@ -49,6 +65,12 @@ export const VocalsAudioDataLoader = ({ audioUrl, children }: AudioDataLoaderPro
|
||||
return
|
||||
}
|
||||
console.error('Failed to load vocals audio data', error)
|
||||
reportDegradationEvent('waveform_failed', {
|
||||
message: error instanceof Error ? error.message : String(error),
|
||||
elapsed_ms: Date.now() - startedAt,
|
||||
not_ready_polls: attempt,
|
||||
transient_retries: transientAttempt,
|
||||
})
|
||||
setAudioData(null)
|
||||
})
|
||||
}
|
||||
|
||||
@@ -0,0 +1,64 @@
|
||||
/**
|
||||
* @jest-environment jsdom
|
||||
* @jest-environment-options {"url": "https://gen.nomadkaraoke.com/app/jobs?x=1#/abc12345/review"}
|
||||
*/
|
||||
|
||||
import {
|
||||
reportDegradationEvent,
|
||||
jobIdFromLocation,
|
||||
__resetDegradationEventsForTest,
|
||||
} from '@/lib/degradation-events'
|
||||
|
||||
describe('degradation-events reporter', () => {
|
||||
let fetchMock: jest.Mock
|
||||
|
||||
beforeEach(() => {
|
||||
__resetDegradationEventsForTest()
|
||||
fetchMock = jest.fn(() => Promise.resolve({ ok: true }))
|
||||
global.fetch = fetchMock as unknown as typeof fetch
|
||||
})
|
||||
|
||||
it('POSTs the event with sanitized url, job id and detail', () => {
|
||||
reportDegradationEvent('banner_unavailable', { stall_ms: 21000, probe_ok: false })
|
||||
|
||||
expect(fetchMock).toHaveBeenCalledTimes(1)
|
||||
const [url, init] = fetchMock.mock.calls[0]
|
||||
expect(url).toBe('https://api.nomadkaraoke.com/api/client-events')
|
||||
const body = JSON.parse((init as RequestInit).body as string)
|
||||
expect(body.type).toBe('banner_unavailable')
|
||||
expect(body.url).not.toContain('x=1') // query stripped
|
||||
expect(body.job_id).toBe('abc12345')
|
||||
expect(body.detail).toEqual({ stall_ms: 21000, probe_ok: false })
|
||||
expect((init as RequestInit).keepalive).toBe(true)
|
||||
})
|
||||
|
||||
it('throttles repeat reports of the same type', () => {
|
||||
reportDegradationEvent('banner_reconnecting')
|
||||
reportDegradationEvent('banner_reconnecting')
|
||||
expect(fetchMock).toHaveBeenCalledTimes(1)
|
||||
|
||||
// A different type is NOT throttled by the first one.
|
||||
reportDegradationEvent('waveform_failed')
|
||||
expect(fetchMock).toHaveBeenCalledTimes(2)
|
||||
})
|
||||
|
||||
it('never throws when fetch rejects', () => {
|
||||
fetchMock.mockImplementation(() => Promise.reject(new Error('offline')))
|
||||
expect(() => reportDegradationEvent('waveform_slow')).not.toThrow()
|
||||
})
|
||||
})
|
||||
|
||||
describe('jobIdFromLocation', () => {
|
||||
it('parses hash-router review URLs', () => {
|
||||
expect(jobIdFromLocation('https://gen.nomadkaraoke.com/app/jobs#/d93747cd/review')).toBe('d93747cd')
|
||||
})
|
||||
|
||||
it('parses path-style job URLs', () => {
|
||||
expect(jobIdFromLocation('https://gen.nomadkaraoke.com/en/jobs/0370244a')).toBe('0370244a')
|
||||
})
|
||||
|
||||
it('returns null when no job id is present', () => {
|
||||
expect(jobIdFromLocation('https://gen.nomadkaraoke.com/')).toBeNull()
|
||||
expect(jobIdFromLocation('not a url')).toBeNull()
|
||||
})
|
||||
})
|
||||
@@ -47,6 +47,11 @@ export const STALL_RECONNECTING_MS = 10_000
|
||||
export const STALL_UNAVAILABLE_MS = 20_000
|
||||
/** A health-probe verdict is trusted this long before we re-probe during a stall. */
|
||||
export const PROBE_FRESH_MS = 10_000
|
||||
/** Escalating to the full "unavailable" card needs this many CONSECUTIVE failed
|
||||
* probes — one failed 4s probe against a briefly-busy instance (e.g. a heavy
|
||||
* transcode pinning the CPU) is not an outage. The reconnecting pill still
|
||||
* shows after a single failure; the assertive message waits for confirmation. */
|
||||
export const UNAVAILABLE_PROBE_FAILURES = 2
|
||||
|
||||
let nextId = 1
|
||||
/** id -> startedAt (ms) for every tracked backend GET currently in flight. */
|
||||
@@ -65,6 +70,8 @@ let probeInFlight = false
|
||||
/** Last probe verdict (null = no verdict yet) and when it settled. */
|
||||
let lastProbeOk: boolean | null = null
|
||||
let lastProbeAt = 0
|
||||
/** Failed probes in a row (reset by any success or reconfiguration). */
|
||||
let consecutiveProbeFailures = 0
|
||||
/** Bumped on configureHealthProbe so a probe from a previous config can't land. */
|
||||
let probeEpoch = 0
|
||||
|
||||
@@ -98,10 +105,15 @@ function maybeStartProbe() {
|
||||
if (lastProbeOk !== null && Date.now() - lastProbeAt < PROBE_FRESH_MS) return
|
||||
probeInFlight = true
|
||||
const epoch = probeEpoch
|
||||
const record = (ok: boolean) => {
|
||||
if (epoch !== probeEpoch) return
|
||||
lastProbeOk = ok
|
||||
consecutiveProbeFailures = ok ? 0 : consecutiveProbeFailures + 1
|
||||
}
|
||||
healthProbe()
|
||||
.then(
|
||||
(ok) => (epoch === probeEpoch ? (lastProbeOk = ok) : undefined),
|
||||
() => (epoch === probeEpoch ? (lastProbeOk = false) : undefined),
|
||||
(ok) => record(ok),
|
||||
() => record(false),
|
||||
)
|
||||
.finally(() => {
|
||||
if (epoch !== probeEpoch) return
|
||||
@@ -118,7 +130,13 @@ function recompute() {
|
||||
// read is treated as a legitimately slow endpoint and the banner stays hidden.
|
||||
if (devOverride === null && next !== 'online' && healthProbe) {
|
||||
maybeStartProbe()
|
||||
if (lastProbeOk !== false) next = 'online'
|
||||
if (lastProbeOk !== false) {
|
||||
next = 'online'
|
||||
} else if (next === 'unavailable' && consecutiveProbeFailures < UNAVAILABLE_PROBE_FAILURES) {
|
||||
// One failed probe can just be a briefly-pegged instance; keep the calm
|
||||
// pill until a second consecutive failure confirms real unreachability.
|
||||
next = 'reconnecting'
|
||||
}
|
||||
}
|
||||
setStatus(next)
|
||||
// Stop the clock once nothing is outstanding (and no dev override needs it).
|
||||
@@ -163,6 +181,26 @@ export function configureHealthProbe(probe: HealthProbe | null): void {
|
||||
probeInFlight = false
|
||||
lastProbeOk = null
|
||||
lastProbeAt = 0
|
||||
consecutiveProbeFailures = 0
|
||||
}
|
||||
|
||||
/** Diagnostic snapshot for degradation telemetry (see lib/degradation-events). */
|
||||
export function getBackendStatusDebug(): {
|
||||
oldestStallMs: number
|
||||
inFlightCount: number
|
||||
lastProbeOk: boolean | null
|
||||
consecutiveProbeFailures: number
|
||||
} {
|
||||
let oldest = Infinity
|
||||
for (const startedAt of inFlight.values()) {
|
||||
if (startedAt < oldest) oldest = startedAt
|
||||
}
|
||||
return {
|
||||
oldestStallMs: inFlight.size === 0 ? 0 : Date.now() - oldest,
|
||||
inFlightCount: inFlight.size,
|
||||
lastProbeOk,
|
||||
consecutiveProbeFailures,
|
||||
}
|
||||
}
|
||||
|
||||
export function getBackendStatus(): BackendStatus {
|
||||
|
||||
@@ -0,0 +1,106 @@
|
||||
/**
|
||||
* Degradation-event reporter: fire-and-forget telemetry for every user-visible
|
||||
* degraded-service surface (connectivity banner, "temporarily unavailable"
|
||||
* lyrics failure, slow/failed waveform). POSTs to /api/client-events, which
|
||||
* persists to Cloud Logging + Firestore so frequency can be reviewed later.
|
||||
*
|
||||
* - Never throws; failures are swallowed (telemetry must not cause UX noise).
|
||||
* - Per-type throttle so a flapping status can't spam the API.
|
||||
* - No-ops on localhost (would otherwise report dev blips to prod).
|
||||
*
|
||||
* NOTE: intentionally does NOT import from lib/api.ts — api.ts imports
|
||||
* backend-status.ts, and this module is called from banner-adjacent code, so
|
||||
* importing api.ts here risks a module cycle. The base URL is derived the same
|
||||
* way api.ts derives it.
|
||||
*/
|
||||
|
||||
export type DegradationEventType =
|
||||
| 'banner_reconnecting'
|
||||
| 'banner_unavailable'
|
||||
| 'lyrics_load_failed'
|
||||
| 'waveform_slow'
|
||||
| 'waveform_failed'
|
||||
|
||||
/** Minimum gap between two reports of the same event type. */
|
||||
const THROTTLE_MS = 60_000
|
||||
const lastSentAt = new Map<DegradationEventType, number>()
|
||||
|
||||
export function __resetDegradationEventsForTest() {
|
||||
lastSentAt.clear()
|
||||
}
|
||||
|
||||
function apiBaseUrl(): string {
|
||||
if (typeof window === 'undefined') return ''
|
||||
const h = window.location.hostname
|
||||
if (h === 'localhost' || h === '127.0.0.1') return '' // guarded below anyway
|
||||
return process.env.NEXT_PUBLIC_API_URL || 'https://api.nomadkaraoke.com'
|
||||
}
|
||||
|
||||
function isLocalhost(): boolean {
|
||||
if (typeof window === 'undefined') return true
|
||||
const h = window.location.hostname
|
||||
return h === 'localhost' || h === '127.0.0.1' || h === '[::1]' || h === '::1'
|
||||
}
|
||||
|
||||
/** Best-effort job id from review-style URLs (/app/jobs#/{id}/… or /jobs/{id}). */
|
||||
export function jobIdFromLocation(href: string): string | null {
|
||||
try {
|
||||
const u = new URL(href)
|
||||
const hashMatch = u.hash.match(/^#\/([0-9a-f-]{6,64})(\/|$)/i)
|
||||
if (hashMatch) return hashMatch[1]
|
||||
const pathMatch = u.pathname.match(/\/jobs\/([0-9a-f-]{6,64})(\/|$)/i)
|
||||
if (pathMatch) return pathMatch[1]
|
||||
return null
|
||||
} catch {
|
||||
return null
|
||||
}
|
||||
}
|
||||
|
||||
function sanitizedHref(): string {
|
||||
try {
|
||||
const u = new URL(window.location.href)
|
||||
u.search = ''
|
||||
return u.toString().slice(0, 2048)
|
||||
} catch {
|
||||
return ''
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
* Report one degradation event. Safe to call from anywhere client-side;
|
||||
* throttled per type, silent on failure.
|
||||
*/
|
||||
export function reportDegradationEvent(
|
||||
type: DegradationEventType,
|
||||
detail?: Record<string, unknown>,
|
||||
): void {
|
||||
try {
|
||||
if (typeof window === 'undefined' || isLocalhost()) return
|
||||
|
||||
const now = Date.now()
|
||||
const last = lastSentAt.get(type)
|
||||
if (last && now - last < THROTTLE_MS) return
|
||||
lastSentAt.set(type, now)
|
||||
|
||||
const body = {
|
||||
type,
|
||||
url: sanitizedHref(),
|
||||
job_id: jobIdFromLocation(window.location.href),
|
||||
user_email: null as string | null,
|
||||
locale: document.documentElement.lang || 'en',
|
||||
release: process.env.NEXT_PUBLIC_COMMIT_SHA || '',
|
||||
detail: detail ?? null,
|
||||
}
|
||||
|
||||
void fetch(`${apiBaseUrl()}/api/client-events`, {
|
||||
method: 'POST',
|
||||
keepalive: true,
|
||||
headers: { 'Content-Type': 'application/json' },
|
||||
body: JSON.stringify(body),
|
||||
}).catch(() => {
|
||||
/* swallow — telemetry must never surface */
|
||||
})
|
||||
} catch {
|
||||
/* swallow */
|
||||
}
|
||||
}
|
||||
+1
-1
@@ -1,6 +1,6 @@
|
||||
[tool.poetry]
|
||||
name = "karaoke-gen"
|
||||
version = "0.230.3"
|
||||
version = "0.231.0"
|
||||
description = "Generate karaoke videos with synchronized lyrics. Handles the entire process from downloading audio and lyrics to creating the final video with title screens."
|
||||
authors = ["Andrew Beveridge <andrew@beveridge.uk>"]
|
||||
license = "MIT"
|
||||
|
||||
@@ -0,0 +1,158 @@
|
||||
#!/usr/bin/env python3
|
||||
"""Concurrent review-page load test (the "10 tabs" regression guard).
|
||||
|
||||
Reproduces the 2026-09-18 incident scenario: N simultaneous lyrics-review tab
|
||||
loads, each firing the page's real GET trio (correction-data, waveform-data,
|
||||
audio/vocals with a Range header). Before v0.230.0 this melted the backend
|
||||
(9/30 requests 500'd, Cloud Run "no available instance"); after, all requests
|
||||
succeed with warm tabs completing in a few seconds.
|
||||
|
||||
Auth: per-job review tokens read from Firestore via ADC (read-only SA works) —
|
||||
no admin secret needed. Requests go through curl (browser-ish UA) because the
|
||||
Cloudflare WAF blocks python-urllib.
|
||||
|
||||
Usage:
|
||||
python scripts/load-test-review.py # 10 tabs vs prod
|
||||
python scripts/load-test-review.py --tabs 20
|
||||
python scripts/load-test-review.py --max-tab-seconds 45 # cold-cache run
|
||||
|
||||
Exit code 0 = pass (no failed requests, all tabs under --max-tab-seconds);
|
||||
1 = regression. Prints a per-request table either way.
|
||||
"""
|
||||
from __future__ import annotations
|
||||
|
||||
import argparse
|
||||
import concurrent.futures
|
||||
import json
|
||||
import os
|
||||
import subprocess
|
||||
import sys
|
||||
import time
|
||||
|
||||
API_DEFAULT = "https://api.nomadkaraoke.com"
|
||||
UA = (
|
||||
"Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 "
|
||||
"(KHTML, like Gecko) Chrome/128 Safari/537.36"
|
||||
)
|
||||
|
||||
|
||||
def fetch_review_jobs(limit: int) -> list[dict]:
|
||||
"""In-review jobs + review tokens from Firestore (ADC)."""
|
||||
os.environ.setdefault("GOOGLE_CLOUD_PROJECT", "nomadkaraoke")
|
||||
from google.cloud import firestore # type: ignore[import]
|
||||
|
||||
db = firestore.Client(project="nomadkaraoke")
|
||||
query = db.collection("jobs").where(
|
||||
filter=firestore.FieldFilter("status", "in", ["in_review", "awaiting_review"])
|
||||
).limit(limit * 2)
|
||||
jobs = []
|
||||
for doc in query.stream():
|
||||
data = doc.to_dict()
|
||||
token = data.get("review_token") or (data.get("state_data") or {}).get("review_token")
|
||||
if token:
|
||||
jobs.append({"job_id": doc.id, "token": token})
|
||||
if len(jobs) >= limit:
|
||||
break
|
||||
return jobs
|
||||
|
||||
|
||||
def curl(url: str, extra_args: list[str] | None = None, timeout: int = 90) -> dict:
|
||||
"""One GET via curl; returns {status, seconds, bytes}."""
|
||||
cmd = [
|
||||
"curl", "-sS", "-o", "/dev/null", "-A", UA, "--max-time", str(timeout),
|
||||
"-w", "%{http_code} %{time_total} %{size_download}",
|
||||
*(extra_args or []),
|
||||
url,
|
||||
]
|
||||
started = time.monotonic()
|
||||
proc = subprocess.run(cmd, capture_output=True, text=True)
|
||||
if proc.returncode != 0 and not proc.stdout.strip():
|
||||
return {"status": 0, "seconds": time.monotonic() - started, "bytes": 0}
|
||||
status, seconds, size = proc.stdout.strip().split()[:3]
|
||||
return {"status": int(status), "seconds": float(seconds), "bytes": int(size)}
|
||||
|
||||
|
||||
def tab_requests(api: str, job: dict) -> list[tuple[str, str, list[str]]]:
|
||||
base = f"{api}/api/review/{job['job_id']}"
|
||||
tok = f"review_token={job['token']}"
|
||||
return [
|
||||
("correction-data", f"{base}/correction-data?{tok}", []),
|
||||
("waveform-data", f"{base}/waveform-data?{tok}", []),
|
||||
("vocals", f"{base}/audio/vocals?{tok}", ["-r", "0-131071"]),
|
||||
]
|
||||
|
||||
|
||||
def simulate_tab(api: str, job: dict) -> dict:
|
||||
started = time.monotonic()
|
||||
results = {}
|
||||
with concurrent.futures.ThreadPoolExecutor(max_workers=3) as pool:
|
||||
futures = {
|
||||
pool.submit(curl, url, extra): name
|
||||
for name, url, extra in tab_requests(api, job)
|
||||
}
|
||||
for fut in concurrent.futures.as_completed(futures):
|
||||
results[futures[fut]] = fut.result()
|
||||
return {"job_id": job["job_id"], "seconds": time.monotonic() - started, "requests": results}
|
||||
|
||||
|
||||
def main() -> int:
|
||||
parser = argparse.ArgumentParser(description=__doc__)
|
||||
parser.add_argument("--tabs", type=int, default=10)
|
||||
parser.add_argument("--api", default=API_DEFAULT)
|
||||
parser.add_argument(
|
||||
"--max-tab-seconds", type=float, default=20.0,
|
||||
help="Fail if any simulated tab takes longer than this (default 20; "
|
||||
"use ~45 for a cold-cache run where waveforms compute for the first time)",
|
||||
)
|
||||
parser.add_argument(
|
||||
"--jobs-file", default=None,
|
||||
help="JSON file of [{job_id, token}] to skip the Firestore lookup",
|
||||
)
|
||||
args = parser.parse_args()
|
||||
if args.tabs < 1:
|
||||
parser.error("--tabs must be >= 1")
|
||||
|
||||
if args.jobs_file:
|
||||
jobs = json.load(open(args.jobs_file))[: args.tabs]
|
||||
else:
|
||||
print(f"Fetching up to {args.tabs} in-review jobs from Firestore…")
|
||||
jobs = fetch_review_jobs(args.tabs)
|
||||
if len(jobs) < args.tabs:
|
||||
# A partial run would "PASS" without reproducing the requested burst.
|
||||
print(
|
||||
f"Only {len(jobs)} in-review jobs available but --tabs {args.tabs} requested. "
|
||||
"Aborting — re-run with a smaller --tabs to test at reduced scale."
|
||||
)
|
||||
return 1
|
||||
print(f"Simulating {len(jobs)} concurrent review tabs against {args.api}\n")
|
||||
|
||||
with concurrent.futures.ThreadPoolExecutor(max_workers=len(jobs)) as pool:
|
||||
tabs = list(pool.map(lambda j: simulate_tab(args.api, j), jobs))
|
||||
|
||||
failed_requests = 0
|
||||
slow_tabs = 0
|
||||
for tab in sorted(tabs, key=lambda t: t["seconds"]):
|
||||
parts = []
|
||||
for name in ("correction-data", "waveform-data", "vocals"):
|
||||
r = tab["requests"][name]
|
||||
ok = 200 <= r["status"] < 300
|
||||
failed_requests += 0 if ok else 1
|
||||
parts.append(f"{name}={r['status']}({r['seconds']:.1f}s)")
|
||||
slow = tab["seconds"] > args.max_tab_seconds
|
||||
slow_tabs += 1 if slow else 0
|
||||
flag = " ← SLOW" if slow else ""
|
||||
print(f" {tab['job_id']} tab={tab['seconds']:5.1f}s {' '.join(parts)}{flag}")
|
||||
|
||||
total = len(jobs) * 3
|
||||
print(f"\n{total - failed_requests}/{total} requests OK; "
|
||||
f"{slow_tabs} tab(s) over {args.max_tab_seconds}s")
|
||||
if failed_requests or slow_tabs:
|
||||
print("RESULT: FAIL — concurrent review-load regression "
|
||||
"(see docs/archive/2026-09-18-concurrent-review-reliability-investigation-and-plan.md)")
|
||||
return 1
|
||||
print("RESULT: PASS")
|
||||
return 0
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
sys.exit(main())
|
||||
Reference in new issue
Block a user