From 30cf41e7d63db63e3808bdacb7a97da6eda3b788 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Andrew=20Beveridge=20=E2=98=84=EF=B8=8F?= Date: Sat, 19 Sep 2026 12:48:21 -0400 Subject: [PATCH] =?UTF-8?q?feat:=20concurrent-reliability=20phases=202-5?= =?UTF-8?q?=20=E2=80=94=20Cloud=20Run=20shape,=20degradation=20telemetry,?= =?UTF-8?q?=20banner=20tuning,=20load-test=20guard=20(#1028)?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit ## Summary Phases 2–5 of the concurrent-load reliability initiative (Phase 1 = #1024, verified live: the 10-tab meltdown went from 9/30 failed requests to 30/30 OK). Verbatim spec: `docs/archive/2026-09-17-concurrent-review-reliability-verbatim-record.md`; plan: `docs/archive/2026-09-18-concurrent-review-reliability-investigation-and-plan.md`. ## P2 — Cloud Run shape - **`--concurrency 80 → 16`** for karaoke-backend: the service is one uvicorn process per 2-vCPU instance, and review loads are heavy; at 80 a burst never triggers scale-out and all lands on the single min-instance. 16 makes bursts fan out while staying well above normal per-instance load. (Deploy flags live in ci.yml — the service is not Pulumi-managed.) - **Local preview ffmpeg fallback bounded** (`LOCAL_PREVIEW_MAX_CONCURRENCY=1`, 900s queue timeout, fast RuntimeError after that): the GCE-outage fallback runs ffmpeg ON the API instance; unbounded, a few concurrent Preview clicks during an encoder outage pegged both vCPUs, stalled the health probe, and showed everyone the orange banner (Andrew's banner-during-preview screenshot). ## P3 — degradation telemetry (the explicit ask) - New **`POST /api/client-events`** (unauthenticated so review-token users report too; 30/min/IP rate limit; bot-filtered; detail capped; URLs query-stripped). Each event → Firestore `client_events` collection **and** a structured `client_event type=…` log line, so "how often are users seeing this?" is reviewable via Firestore aggregation or Cloud Logging correlation. - Frontend reporter `lib/degradation-events.ts` (60s per-type throttle, localhost no-op, never throws), wired into: - `BackendStatusBanner` → `banner_reconnecting` / `banner_unavailable` with stall age, in-flight count, probe state - review lyrics-load failure → `lyrics_load_failed` with message/status - `VocalsAudioDataLoader` → `waveform_slow` (>20s to first strips) and `waveform_failed` (gave up) with poll/retry counts ## P4 — banner tuning - Escalating to the full "We're having trouble reaching our servers" card now requires **two consecutive failed health probes** (`UNAVAILABLE_PROBE_FAILURES=2`). One failed 4s probe against a briefly-pegged instance shows only the calm "Reconnecting…" pill. Single-failure behavior for the pill is unchanged, as is instant clearing on recovery. ## P5 — regression guard - **`scripts/load-test-review.py`** checked in: replays the 10-tab incident (Firestore-ADC review tokens, curl request trios, pass/fail thresholds, exit code). Documented in `docs/TESTING.md`. Validated against prod post-Phase-1: 9/9 requests OK, tabs < 3s. ## Testing - Backend: 6 new tests for client-events (persist/validate/rate-limit/bot/fail-soft/detail-cap) + local-preview bound test — full suite **4517 passed**. - Frontend: new degradation-events suite, two-probe escalation + debug-snapshot tests; one existing api-fetch test updated to the new escalation contract — full suite **1298 passed**. - Live validation of the load-test tool against prod (small 3-tab run): PASS. ## Notes for review - Cloud Run concurrency change means bursts now spin up extra instances (billable during requests only; max-instances unchanged at 20). - Firestore `client_events` has no TTL policy yet (needs a gcloud/console action our read-only ADC can't do); volume is tiny (throttled per client) — can add later if it ever matters. 🤖 Generated with [Claude Code](https://claude.com/claude-code) ## Summary by CodeRabbit - **New Features** - Added automatic reporting for degraded experiences, including reconnecting or unavailable backend status, failed lyrics loads, and slow or failed waveform loads. - Added concurrent review-page load testing to help verify performance under multiple open tabs. - **Improvements** - Backend status now remains in “reconnecting” until multiple health checks fail, reducing premature unavailable messages. - Local preview processing is limited during heavy demand to improve stability. - **Documentation** - Added API documentation for client telemetry and guidance for running concurrent-load regression tests. --------- Co-authored-by: Claude Fable 5 --- .github/workflows/ci.yml | 8 +- backend/api/routes/client_events.py | 135 +++++++++++++++ backend/api/routes/review.py | 68 +++++++- backend/main.py | 3 +- backend/tests/test_client_events.py | 85 ++++++++++ backend/tests/test_routes_review.py | 102 ++++++++++- docs/API.md | 27 +++ docs/TESTING.md | 19 +++ frontend/__tests__/api-fetch.test.ts | 8 +- frontend/__tests__/backend-status.test.ts | 50 ++++++ .../[locale]/app/jobs/[[...slug]]/client.tsx | 5 + frontend/components/backend-status-banner.tsx | 22 +++ .../lyrics-review/VocalsAudioDataLoader.tsx | 24 ++- .../lib/__tests__/degradation-events.test.ts | 64 +++++++ frontend/lib/backend-status.ts | 44 ++++- frontend/lib/degradation-events.ts | 106 ++++++++++++ pyproject.toml | 2 +- scripts/load-test-review.py | 158 ++++++++++++++++++ 18 files changed, 912 insertions(+), 18 deletions(-) create mode 100644 backend/api/routes/client_events.py create mode 100644 backend/tests/test_client_events.py create mode 100644 frontend/lib/__tests__/degradation-events.test.ts create mode 100644 frontend/lib/degradation-events.ts create mode 100644 scripts/load-test-review.py diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml index 0e1e71cb..23a1a9fa 100644 --- a/.github/workflows/ci.yml +++ b/.github/workflows/ci.yml @@ -1976,6 +1976,12 @@ jobs: echo "Version: $VERSION" echo "Previously-serving revision: ${PREV_REVISION:-}" + # --concurrency 16 (was 80): the backend is one uvicorn process per + # 2-vCPU instance; review-page loads are heavy (audio byte-serving, + # waveform work). At 80 a 10-tab burst all landed on the single + # min-instance and wedged it ("no available instance" aborts, + # 2026-09-18 investigation). 16 makes bursts trigger scale-out while + # staying comfortably above normal per-instance load. gcloud run deploy karaoke-backend \ --image $IMAGE \ --no-traffic \ @@ -1989,7 +1995,7 @@ jobs: --cpu-throttling \ --execution-environment gen2 \ --timeout 1800 \ - --concurrency 80 \ + --concurrency 16 \ --max-instances 20 \ --min-instances 1 \ --vpc-connector projects/nomadkaraoke/locations/us-central1/connectors/cloud-run-connector \ diff --git a/backend/api/routes/client_events.py b/backend/api/routes/client_events.py new file mode 100644 index 00000000..933611c8 --- /dev/null +++ b/backend/api/routes/client_events.py @@ -0,0 +1,135 @@ +"""POST /api/client-events — persistent telemetry for user-visible degradation. + +Records every time the frontend shows a user a degraded-service surface (the +orange "trouble reaching our servers" banner, the stronger "temporarily +unavailable" error, a failed lyrics load, a slow/failed waveform), with enough +metadata to review frequency and context later. Two sinks per event: + + 1. A structured INFO log line (``client_event type=…``) — queryable in Cloud + Logging / Log Analytics alongside backend symptoms from the same window. + 2. A Firestore doc in ``client_events`` — survives log retention, cheap to + aggregate for "how often are users seeing this?" reviews. + +Unauthenticated (review-token users must be able to report) and rate-limited +per IP, mirroring /api/client-errors. The limiter is process-local, so the +effective ceiling is 120/min/IP × live instances — acceptable for this threat +model (non-malicious browsers, 60s client-side per-type throttle); Cloudflare's +zone-wide rate limit sits in front as a flood backstop. Events are dropped, +never queued, when the limiter trips — telemetry, not a delivery guarantee. +""" +from __future__ import annotations + +import logging +import time +from datetime import datetime, timezone +from typing import Any, Literal, Optional + +from fastapi import APIRouter, HTTPException, Request +from pydantic import BaseModel, Field, field_validator + +from backend.services.error_monitor.frontend_ingestion import ( + RateLimiter, + is_bot_user_agent, + sanitize_url, +) + +logger = logging.getLogger(__name__) +router = APIRouter(prefix="/client-events", tags=["client-events"]) + +# Sized for a legitimate worst case from ONE IP: a many-tab burst (e.g. 16 review +# tabs at once) where every tab reports a banner + a waveform event — that must be +# fully recorded, not self-suppressed. Each tab throttles per-type client-side. +_limiter = RateLimiter(max_per_minute=120) +_db_singleton = None + +# The full closed set of reportable events. Reject anything else so the +# collection stays reviewable (no free-form types accumulating). +EventType = Literal[ + "banner_reconnecting", + "banner_unavailable", + "lyrics_load_failed", + "waveform_slow", + "waveform_failed", +] + + +def _get_db(): + global _db_singleton + if _db_singleton is None: + from google.cloud import firestore # type: ignore[import] + + _db_singleton = firestore.Client(project="nomadkaraoke") + return _db_singleton + + +class ClientEventPayload(BaseModel): + type: EventType + url: str = Field("", max_length=2048) + job_id: Optional[str] = Field(None, max_length=64) + user_email: Optional[str] = Field(None, max_length=320) + locale: str = Field("en", max_length=10) + release: str = Field("", max_length=64) + # Small free-form context (stall age, http status, elapsed ms, probe state…). + # Size-capped server-side; values are stored as-is. + detail: Optional[dict] = None + + @field_validator("url", "locale", "release") + @classmethod + def strip(cls, v: str) -> str: + return (v or "").strip() + + +def _capped_detail(detail: Optional[dict]) -> dict[str, Any]: + """Keep detail reviewable: at most 12 keys, scalar-ish values, short strings.""" + if not isinstance(detail, dict): + return {} + out: dict[str, Any] = {} + for key, value in list(detail.items())[:12]: + k = str(key)[:40] + if isinstance(value, (int, float, bool)) or value is None: + out[k] = value + else: + out[k] = str(value)[:200] + return out + + +@router.post("", status_code=202) +def report_client_event(payload: ClientEventPayload, request: Request) -> dict: + client_ip = request.client.host if request.client else "unknown" + if not _limiter.allow(client_ip, time.monotonic()): + raise HTTPException(status_code=429, detail="too many events") + + user_agent = request.headers.get("user-agent", "")[:1024] + if is_bot_user_agent(user_agent): + return {"status": "ignored"} + + event = { + "type": payload.type, + "url": sanitize_url(payload.url), + "job_id": payload.job_id, + "user_email": payload.user_email, + "locale": payload.locale, + "release": payload.release, + "user_agent": user_agent, + "detail": _capped_detail(payload.detail), + "created_at": datetime.now(timezone.utc), + } + + # Sink 1: structured log line — correlate with backend symptoms in Logging. + logger.info( + "client_event type=%s job_id=%s user=%s url=%s release=%s detail=%s", + event["type"], + event["job_id"], + event["user_email"], + event["url"], + event["release"], + event["detail"], + ) + + # Sink 2: Firestore — best-effort; telemetry must never error back to users. + try: + _get_db().collection("client_events").add(event) + except Exception: + logger.exception("Failed to persist client event (non-fatal)") + + return {"status": "recorded"} diff --git a/backend/api/routes/review.py b/backend/api/routes/review.py index eb73fbf7..a349356b 100644 --- a/backend/api/routes/review.py +++ b/backend/api/routes/review.py @@ -13,6 +13,7 @@ Usage: The baseApiUrl includes the job_id, and all endpoints are relative to that. """ import asyncio +import concurrent.futures import logging import hashlib import json @@ -278,12 +279,65 @@ def _prepare_preview_inputs(job, temp_dir: str, storage, is_duet: bool): return correction_result, audio_path, output_config -def _render_preview_locally(job, updated_data: Dict[str, Any], is_duet: bool) -> str: - """Render the full 360p preview locally and upload it to GCS. +# At most this many local preview renders at once. The local path runs ffmpeg +# ON the 2-vCPU API instance (it's the fallback when GCE preview encoding is +# down); unbounded, a GCE outage plus a few users clicking Preview saturates +# the instance's CPU, stalls the health probe, and shows everyone the orange +# "trouble reaching our servers" banner. Serializing renders keeps the API +# responsive — previews just queue a little longer during an encoder outage. +_LOCAL_PREVIEW_MAX_CONCURRENCY = max(1, int(os.getenv("LOCAL_PREVIEW_MAX_CONCURRENCY", "1"))) +_LOCAL_PREVIEW_QUEUE_TIMEOUT_S = int(os.getenv("LOCAL_PREVIEW_QUEUE_TIMEOUT_S", "900")) +_local_preview_slots = asyncio.Semaphore(_LOCAL_PREVIEW_MAX_CONCURRENCY) +# Dedicated executor: admitted renders never compete with the shared to_thread +# pool, and its concurrent.futures-level done callbacks fire when the WORKER +# THREAD finishes — the asyncio wrapper future can be "cancelled" while the +# thread is still running, so it must not drive the slot release. +_local_preview_executor = concurrent.futures.ThreadPoolExecutor( + max_workers=_LOCAL_PREVIEW_MAX_CONCURRENCY, thread_name_prefix="local-preview" +) - Blocking (downloads + ffmpeg) — call via asyncio.to_thread. Used when GCE - preview encoding is disabled and as the fallback when it fails. + +async def _render_preview_locally(job, updated_data: Dict[str, Any], is_duet: bool) -> str: + """Render the full 360p preview locally (bounded) and upload it to GCS. + + Used when GCE preview encoding is disabled and as the fallback when it + fails. Admission happens HERE on the event loop (async semaphore + + wait_for) so queued renders wait as cheap coroutines — only an ADMITTED + render occupies a thread-pool worker for its blocking downloads + ffmpeg. """ + try: + await asyncio.wait_for( + _local_preview_slots.acquire(), timeout=_LOCAL_PREVIEW_QUEUE_TIMEOUT_S + ) + except asyncio.TimeoutError: + raise RuntimeError( + "Local preview encoder is busy (waited " + f"{_LOCAL_PREVIEW_QUEUE_TIMEOUT_S}s for a slot) — try again shortly" + ) + def _run_render(): + return _render_preview_locally_unbounded(job, updated_data, is_duet) + + loop = asyncio.get_running_loop() + worker_future = _local_preview_executor.submit(_run_render) + + def _release_slot(fut: "concurrent.futures.Future") -> None: + # Runs when the WORKER THREAD actually finishes. Releasing from an + # async finally (or the asyncio wrapper's callback) would free the slot + # the moment the awaiting request is cancelled — while the ffmpeg + # render thread is still consuming CPU — letting a second render start + # and defeating the concurrency bound. + if not fut.cancelled(): + fut.exception() # mark retrieved when the caller was cancelled/abandoned + try: + loop.call_soon_threadsafe(_local_preview_slots.release) + except RuntimeError: + pass # loop already closed (shutdown) — nothing left to bound + + worker_future.add_done_callback(_release_slot) + return await asyncio.wrap_future(worker_future) + + +def _render_preview_locally_unbounded(job, updated_data: Dict[str, Any], is_duet: bool) -> str: job_id = job.job_id storage = StorageService() with tempfile.TemporaryDirectory() as temp_dir: @@ -349,7 +403,7 @@ async def _encode_preview_in_background( logger.warning(f"Job {job_id}: GCE preview encoding failed, falling back to local: {gce_error}") try: - await asyncio.to_thread(_render_preview_locally, job, updated_data, is_duet) + await _render_preview_locally(job, updated_data, is_duet) logger.info(f"Job {job_id}: Preview generated locally after GCE failure: {preview_hash}") except Exception as local_error: logger.error( @@ -1873,9 +1927,7 @@ async def generate_preview_video( # Local path (GCE preview disabled, or GCE setup failed): render # the full preview synchronously — no cold-start problem, but slow. - preview_hash = await asyncio.to_thread( - _render_preview_locally, job, updated_data, is_duet - ) + preview_hash = await _render_preview_locally(job, updated_data, is_duet) logger.info(f"Job {job_id}: Preview video generated: {preview_hash}") span.set_attribute("preview_hash", preview_hash) diff --git a/backend/main.py b/backend/main.py index 1343f69b..55c5902e 100644 --- a/backend/main.py +++ b/backend/main.py @@ -10,7 +10,7 @@ from fastapi import FastAPI from fastapi.middleware.cors import CORSMiddleware from backend.config import settings, validate_production_config -from backend.api.routes import health, jobs, internal, file_upload, review, auth, audio_search, themes, users, admin, tenant, tenant_admin, tenant_bulk, rate_limits, push, catalog, encoding_worker, client_errors, bulk, parse_titles +from backend.api.routes import health, jobs, internal, file_upload, review, auth, audio_search, themes, users, admin, tenant, tenant_admin, tenant_bulk, rate_limits, push, catalog, encoding_worker, client_errors, client_events, bulk, parse_titles from backend.services.tracing import setup_tracing, instrument_app, get_current_trace_id from backend.services.structured_logging import setup_structured_logging from backend.services.spacy_preloader import preload_spacy_model @@ -213,6 +213,7 @@ app.include_router(bulk.router, prefix="/api") # Bulk Mode: multi-job submissio from backend.api.routes import requests_board app.include_router(requests_board.router, prefix="/api") # Public song-request voting board app.include_router(client_errors.router, prefix="/api") # Frontend crash reports +app.include_router(client_events.router, prefix="/api") # Frontend degradation telemetry (banner/unavailable/waveform events) app.include_router(tenant.router) # Tenant/white-label configuration (no /api prefix, router has it) app.include_router(tenant_admin.router) # Admin tenant provisioning (router has /api prefix) app.include_router(tenant_bulk.router) # Tenant bulk-upload filename analysis (router has /api prefix) diff --git a/backend/tests/test_client_events.py b/backend/tests/test_client_events.py new file mode 100644 index 00000000..a15d610a --- /dev/null +++ b/backend/tests/test_client_events.py @@ -0,0 +1,85 @@ +"""Tests for POST /api/client-events (degradation telemetry).""" + +from unittest.mock import MagicMock, patch + +import pytest +from fastapi import FastAPI +from fastapi.testclient import TestClient + +from backend.api.routes import client_events + + +@pytest.fixture +def client(): + app = FastAPI() + app.include_router(client_events.router, prefix="/api") + return TestClient(app) + + +@pytest.fixture(autouse=True) +def fresh_limiter_and_db(monkeypatch): + monkeypatch.setattr(client_events, "_limiter", client_events.RateLimiter(max_per_minute=30)) + db = MagicMock() + monkeypatch.setattr(client_events, "_db_singleton", db) + return db + + +def _event(**overrides): + payload = { + "type": "banner_unavailable", + "url": "https://gen.nomadkaraoke.com/app/jobs?token=SECRET#/abc123/review", + "job_id": "abc123", + "locale": "en", + "release": "deadbeef", + "detail": {"stall_ms": 21000, "probe_ok": False}, + } + payload.update(overrides) + return payload + + +class TestReportClientEvent: + def test_valid_event_persists_to_firestore(self, client, fresh_limiter_and_db): + resp = client.post("/api/client-events", json=_event()) + + assert resp.status_code == 202 + assert resp.json() == {"status": "recorded"} + fresh_limiter_and_db.collection.assert_called_once_with("client_events") + (doc,) = fresh_limiter_and_db.collection.return_value.add.call_args[0] + assert doc["type"] == "banner_unavailable" + assert doc["job_id"] == "abc123" + assert doc["detail"] == {"stall_ms": 21000, "probe_ok": False} + # Query string (may carry tokens) must be stripped before persisting. + assert "SECRET" not in doc["url"] + + def test_unknown_event_type_rejected(self, client): + resp = client.post("/api/client-events", json=_event(type="something_else")) + assert resp.status_code == 422 + + def test_rate_limited(self, client, monkeypatch): + monkeypatch.setattr(client_events, "_limiter", client_events.RateLimiter(max_per_minute=2)) + assert client.post("/api/client-events", json=_event()).status_code == 202 + assert client.post("/api/client-events", json=_event()).status_code == 202 + assert client.post("/api/client-events", json=_event()).status_code == 429 + + def test_bot_user_agent_ignored(self, client, fresh_limiter_and_db): + resp = client.post( + "/api/client-events", + json=_event(), + headers={"User-Agent": "Googlebot/2.1 (+http://www.google.com/bot.html)"}, + ) + assert resp.status_code == 202 + assert resp.json() == {"status": "ignored"} + fresh_limiter_and_db.collection.assert_not_called() + + def test_firestore_failure_still_returns_202(self, client, fresh_limiter_and_db): + fresh_limiter_and_db.collection.return_value.add.side_effect = Exception("firestore down") + resp = client.post("/api/client-events", json=_event()) + assert resp.status_code == 202 # telemetry must never error back to users + + def test_detail_is_capped(self): + big = {f"key{i}": "x" * 500 for i in range(30)} + capped = client_events._capped_detail(big) + assert len(capped) == 12 + assert all(len(v) <= 200 for v in capped.values()) + assert client_events._capped_detail(None) == {} + assert client_events._capped_detail("not a dict") == {} diff --git a/backend/tests/test_routes_review.py b/backend/tests/test_routes_review.py index 7eefcb59..2c14faa8 100644 --- a/backend/tests/test_routes_review.py +++ b/backend/tests/test_routes_review.py @@ -1825,7 +1825,7 @@ class TestEncodePreviewInBackground: encoding_service = MagicMock() encoding_service.encode_preview_video = AsyncMock(side_effect=RuntimeError("stockout")) storage = MagicMock() - local_render = MagicMock(return_value="h1") + local_render = AsyncMock(return_value="h1") self._run(mock_job, encoding_service, storage, local_render) local_render.assert_called_once() storage.upload_json.assert_not_called() @@ -1836,7 +1836,7 @@ class TestEncodePreviewInBackground: encoding_service = MagicMock() encoding_service.encode_preview_video = AsyncMock(side_effect=RuntimeError("stockout")) storage = MagicMock() - local_render = MagicMock(side_effect=RuntimeError("ffmpeg died")) + local_render = AsyncMock(side_effect=RuntimeError("ffmpeg died")) self._run(mock_job, encoding_service, storage, local_render) storage.upload_json.assert_called_once_with( "jobs/job-bg/previews/h1.error.json", @@ -2041,3 +2041,101 @@ class TestReviewAudioSingleFlightAndByteCap: assert "a" not in review_module._review_audio_cache assert review_module._review_audio_cache_bytes <= 100 assert set(review_module._review_audio_cache) == {"b", "c", "d", "e"} + + +class TestLocalPreviewRenderBound: + """The local preview ffmpeg fallback must be bounded so a GCE outage plus a + few concurrent Preview clicks can't saturate the API instance's CPU. + Admission is an async semaphore on the event loop (waiters are cheap + coroutines, not parked executor threads).""" + + def test_serializes_and_times_out_admission(self, monkeypatch): + import asyncio + import threading + from backend.api.routes import review as review_module + + monkeypatch.setattr(review_module, "_LOCAL_PREVIEW_QUEUE_TIMEOUT_S", 0.2) + monkeypatch.setattr(review_module, "_local_preview_slots", asyncio.Semaphore(1)) + + started = threading.Event() + release = threading.Event() + + def slow_render(job, updated_data, is_duet): + started.set() + release.wait(timeout=5) + return "hash" + + monkeypatch.setattr(review_module, "_render_preview_locally_unbounded", slow_render) + + async def scenario(): + first = asyncio.create_task( + review_module._render_preview_locally(MagicMock(), {}, False) + ) + await asyncio.to_thread(started.wait, 5) + + # Slot held by the first render — the second caller's admission wait + # times out fast instead of piling more ffmpeg onto the instance. + with pytest.raises(RuntimeError, match="busy"): + await review_module._render_preview_locally(MagicMock(), {}, False) + + release.set() + assert await first == "hash" + + # Slot released — a fresh render is admitted immediately. + monkeypatch.setattr( + review_module, "_render_preview_locally_unbounded", + lambda job, updated_data, is_duet: "hash2", + ) + assert await review_module._render_preview_locally(MagicMock(), {}, False) == "hash2" + + asyncio.run(scenario()) + + def test_cancelled_caller_does_not_free_slot_while_render_runs(self, monkeypatch): + """Cancelling the awaiting request must NOT release the slot while the + ffmpeg worker thread is still running — otherwise a second render can + start and defeat the concurrency bound.""" + import asyncio + import threading + from backend.api.routes import review as review_module + + monkeypatch.setattr(review_module, "_LOCAL_PREVIEW_QUEUE_TIMEOUT_S", 0.2) + monkeypatch.setattr(review_module, "_local_preview_slots", asyncio.Semaphore(1)) + + started = threading.Event() + release = threading.Event() + + def slow_render(job, updated_data, is_duet): + started.set() + release.wait(timeout=5) + return "hash" + + monkeypatch.setattr(review_module, "_render_preview_locally_unbounded", slow_render) + + async def scenario(): + first = asyncio.create_task( + review_module._render_preview_locally(MagicMock(), {}, False) + ) + await asyncio.to_thread(started.wait, 5) + + first.cancel() + with pytest.raises(asyncio.CancelledError): + await first + + # Worker thread is still rendering — the slot must still be held. + with pytest.raises(RuntimeError, match="busy"): + await review_module._render_preview_locally(MagicMock(), {}, False) + + # Worker finishes → done-callback releases the slot. + release.set() + for _ in range(50): + if not review_module._local_preview_slots.locked(): + break + await asyncio.sleep(0.05) + + monkeypatch.setattr( + review_module, "_render_preview_locally_unbounded", + lambda job, updated_data, is_duet: "hash2", + ) + assert await review_module._render_preview_locally(MagicMock(), {}, False) == "hash2" + + asyncio.run(scenario()) diff --git a/docs/API.md b/docs/API.md index 7ec7ba55..9ca3e364 100644 --- a/docs/API.md +++ b/docs/API.md @@ -2669,3 +2669,30 @@ search returned nothing usable. Stripe webhooks implemented at `/api/users/webhooks/stripe`. Job status webhooks not yet implemented. + +## Client Telemetry + +### Degradation Events + +```http +POST /api/client-events +``` + +Unauthenticated, rate-limited (120/min/IP — sized so a many-tab burst from one IP is fully recorded), bot-filtered. The frontend reports every +user-visible degraded-service surface: `banner_reconnecting`, `banner_unavailable` +(the connectivity banner — reported by `BackendStatusBanner`), `lyrics_load_failed` +(review page "temporarily unavailable"), `waveform_slow` / `waveform_failed` +(Waveforms-mode strips late or missing). Body: `{type, url, job_id?, user_email?, +locale, release, detail?}` — `detail` is size-capped server-side (≤12 keys, ≤200 +chars per value); URLs are stored query-stripped. + +Each event emits a structured `client_event type=…` INFO log line and is +best-effort persisted to the Firestore `client_events` collection (a Firestore +write failure is logged but still returns `202` — telemetry never errors back to +the client), so frequency can be reviewed later +(Firestore aggregation, or Cloud Logging correlated with backend symptoms from the +same window). Reporter: `frontend/lib/degradation-events.ts` (60s per-type throttle, +localhost no-op). Added in v0.231.0 — see +`docs/archive/2026-09-18-concurrent-review-reliability-investigation-and-plan.md`. + +Crash reports are the separate `POST /api/client-errors` (error-monitor pipeline). diff --git a/docs/TESTING.md b/docs/TESTING.md index 8e2930ca..31d787f9 100644 --- a/docs/TESTING.md +++ b/docs/TESTING.md @@ -456,6 +456,25 @@ The production tests cover: - Final encoding and download - Admin dashboard functionality +## Concurrent-Load Regression Test (review pages) + +`scripts/load-test-review.py` replays the 2026-09-18 "10 tabs melt the backend" +incident: N simultaneous review-tab loads, each firing the page's real GET trio +(correction-data, waveform-data, ranged audio/vocals), authenticated with per-job +review tokens read from Firestore via ADC (no admin secret needed). + +```bash +python scripts/load-test-review.py # 10 tabs vs prod, warm thresholds +python scripts/load-test-review.py --tabs 20 +python scripts/load-test-review.py --max-tab-seconds 45 # cold caches (first-ever waveforms) +``` + +Pass = every request 2xx and every tab under the threshold; exit 1 otherwise. +Run it after changes to the review hot path (`backend/api/routes/review.py`, +`audio_transcoding_service`, `audio_analysis_service`) or Cloud Run deploy flags. +Background + before/after numbers: +`docs/archive/2026-09-18-concurrent-review-reliability-investigation-and-plan.md`. + ## Ad-Hoc Production Debugging (for Agents) When asked to "test it yourself in prod" or debug a specific production issue, use the debug script template: diff --git a/frontend/__tests__/api-fetch.test.ts b/frontend/__tests__/api-fetch.test.ts index b4cf33d2..33ef2c4d 100644 --- a/frontend/__tests__/api-fetch.test.ts +++ b/frontend/__tests__/api-fetch.test.ts @@ -131,10 +131,16 @@ describe('apiFetch', () => { // stays hidden (a slow endpoint must not alarm the user without evidence). await jest.advanceTimersByTimeAsync(STALL_RECONNECTING_MS) expect(getBackendStatus()).toBe('online') - // Probe times out (~4s) → outage confirmed → the hint appears. + // Probe times out (~4s) → outage suspected → the hint appears. await jest.advanceTimersByTimeAsync(5_000) expect(getBackendStatus()).toBe('reconnecting') + // The stall alone now qualifies for "unavailable", but escalation needs a + // SECOND consecutive probe failure (one failed probe can just be a briefly + // pegged instance) — until then the calm pill is all the user sees. await jest.advanceTimersByTimeAsync(STALL_UNAVAILABLE_MS - STALL_RECONNECTING_MS - 5_000) + expect(getBackendStatus()).toBe('reconnecting') + // Verdict goes stale (~24s) → probe #2 fires → times out (~28s) → confirmed. + await jest.advanceTimersByTimeAsync(9_000) expect(getBackendStatus()).toBe('unavailable') settleGet(mockResponse(200, { ok: true })) diff --git a/frontend/__tests__/backend-status.test.ts b/frontend/__tests__/backend-status.test.ts index 182d8f6d..ce0a6e71 100644 --- a/frontend/__tests__/backend-status.test.ts +++ b/frontend/__tests__/backend-status.test.ts @@ -8,6 +8,7 @@ import { getBackendStatus, + getBackendStatusDebug, beginRequest, endRequest, configureHealthProbe, @@ -158,4 +159,53 @@ describe('backend-status store (health-probe confirmation)', () => { await jest.advanceTimersByTimeAsync(3000) // verdict lands expect(getBackendStatus()).toBe('reconnecting') }) + + it('needs TWO consecutive failed probes before escalating to unavailable', async () => { + // Each probe takes 3s to fail — realistic for a probe racing its own + // timeout against a briefly-pegged (not down) instance. + const probe = jest.fn( + () => new Promise((res) => setTimeout(() => res(false), 3000)), + ) + configureHealthProbe(probe) + + open.push(beginRequest()) + // t=10s: probe #1 starts. t=13s: failure #1 recorded. + // t=20s: the stall alone now qualifies for "unavailable", but only one + // probe failure has been recorded (verdict fresh until t=23s) — + // the calm reconnecting pill must be all the user sees. + await jest.advanceTimersByTimeAsync(STALL_UNAVAILABLE_MS + 1000) // t=21s + expect(getBackendStatus()).toBe('reconnecting') + expect(getBackendStatusDebug().consecutiveProbeFailures).toBe(1) + + // t=23s: verdict stale → probe #2 starts. t=26s: failure #2 → escalate. + await jest.advanceTimersByTimeAsync(6000) // t=27s + expect(getBackendStatus()).toBe('unavailable') + expect(getBackendStatusDebug().consecutiveProbeFailures).toBe(2) + }) + + it('a probe success resets the consecutive-failure count', async () => { + let reachable = false + const probe = jest.fn(() => Promise.resolve(reachable)) + configureHealthProbe(probe) + + open.push(beginRequest()) + await jest.advanceTimersByTimeAsync(STALL_UNAVAILABLE_MS + PROBE_FRESH_MS) + expect(getBackendStatusDebug().consecutiveProbeFailures).toBeGreaterThanOrEqual(2) + + reachable = true + await jest.advanceTimersByTimeAsync(PROBE_FRESH_MS + 2000) + expect(getBackendStatus()).toBe('online') + expect(getBackendStatusDebug().consecutiveProbeFailures).toBe(0) + }) + + it('exposes a diagnostic snapshot for telemetry', async () => { + configureHealthProbe(() => Promise.resolve(false)) + open.push(beginRequest()) + await jest.advanceTimersByTimeAsync(STALL_RECONNECTING_MS + 1000) + + const debug = getBackendStatusDebug() + expect(debug.inFlightCount).toBe(1) + expect(debug.oldestStallMs).toBeGreaterThanOrEqual(STALL_RECONNECTING_MS) + expect(debug.lastProbeOk).toBe(false) + }) }) diff --git a/frontend/app/[locale]/app/jobs/[[...slug]]/client.tsx b/frontend/app/[locale]/app/jobs/[[...slug]]/client.tsx index 4739ce5b..0bc97531 100644 --- a/frontend/app/[locale]/app/jobs/[[...slug]]/client.tsx +++ b/frontend/app/[locale]/app/jobs/[[...slug]]/client.tsx @@ -1,6 +1,7 @@ "use client" import { useEffect, useState, useCallback, useRef } from "react" +import { reportDegradationEvent } from "@/lib/degradation-events" import { useAuth } from "@/lib/auth" import { api, Job, createLyricsReviewApiClient, lyricsReviewApi } from "@/lib/api" import { Spinner } from "@/components/ui/spinner" @@ -334,6 +335,10 @@ function LyricsReviewWrapper({ job, isLocalMode = false, isReplay = false }: { j setCorrectionData(data) } catch (err) { console.error("Failed to load correction data:", err) + reportDegradationEvent("lyrics_load_failed", { + message: err instanceof Error ? err.message : String(err), + status: (err as { status?: number })?.status ?? null, + }) setError(err instanceof Error ? err.message : "Failed to load lyrics data") } finally { setIsLoading(false) diff --git a/frontend/components/backend-status-banner.tsx b/frontend/components/backend-status-banner.tsx index b05c08b1..3699c895 100644 --- a/frontend/components/backend-status-banner.tsx +++ b/frontend/components/backend-status-banner.tsx @@ -5,8 +5,10 @@ import { useTranslations } from "next-intl" import { Loader2, CloudOff, X } from "lucide-react" import { useBackendStatus, + getBackendStatusDebug, __installBackendStatusDevHook, } from "@/lib/backend-status" +import { reportDegradationEvent } from "@/lib/degradation-events" /** * App-wide, non-blocking banner that reacts to backend connectivity (see @@ -37,6 +39,26 @@ export function BackendStatusBanner() { if (status === "online") setDismissed(false) }, [status]) + // Persist every banner display server-side (Andrew's telemetry ask): this is + // the single global banner instance, so reporting here == "shown to a user". + // The reporter throttles per type, so a flapping status can't spam. + useEffect(() => { + if (status === "online") return + // A dismissed unavailable card renders nothing — reporting it would count a + // banner the user never saw. (dismiss only re-arms after passing online.) + if (status === "unavailable" && dismissed) return + const debug = getBackendStatusDebug() + reportDegradationEvent( + status === "unavailable" ? "banner_unavailable" : "banner_reconnecting", + { + stall_ms: debug.oldestStallMs, + in_flight: debug.inFlightCount, + probe_ok: debug.lastProbeOk, + probe_failures: debug.consecutiveProbeFailures, + }, + ) + }, [status, dismissed]) + if (status === "online") return null if (status === "reconnecting") { diff --git a/frontend/components/lyrics-review/VocalsAudioDataLoader.tsx b/frontend/components/lyrics-review/VocalsAudioDataLoader.tsx index bc085305..4b6d8c18 100644 --- a/frontend/components/lyrics-review/VocalsAudioDataLoader.tsx +++ b/frontend/components/lyrics-review/VocalsAudioDataLoader.tsx @@ -1,6 +1,12 @@ import { AudioData, AudioNotReadyError, fetchAudioData, isTransientAudioError } from '@/lib/audio-data' +import { reportDegradationEvent } from '@/lib/degradation-events' import { createContext, PropsWithChildren, useEffect, useState } from 'react' +// A first paint whose waveform strips took longer than this to appear counts as +// a degraded experience worth recording (Andrew's "wait 1+ minutes" complaint). +// Excludes legitimate separation-in-progress waits shorter than one 202 poll. +const SLOW_LOAD_REPORT_MS = 20_000 + export const VocalsAudioDataLoaderContext = createContext<{ audioData: AudioData | null }>({ audioData: null }) export interface AudioDataLoaderProps extends PropsWithChildren { @@ -30,11 +36,21 @@ export const VocalsAudioDataLoader = ({ audioUrl, children }: AudioDataLoaderPro // audioUrl has already been set. let cancelled = false let retryTimer: ReturnType | undefined + const startedAt = Date.now() const load = (attempt: number, transientAttempt: number) => { fetchAudioData(audioUrl) .then((audioData) => { - if (!cancelled) setAudioData(audioData) + if (cancelled) return + const elapsedMs = Date.now() - startedAt + if (elapsedMs >= SLOW_LOAD_REPORT_MS) { + reportDegradationEvent('waveform_slow', { + elapsed_ms: elapsedMs, + not_ready_polls: attempt, + transient_retries: transientAttempt, + }) + } + setAudioData(audioData) }) .catch((error) => { if (cancelled) return @@ -49,6 +65,12 @@ export const VocalsAudioDataLoader = ({ audioUrl, children }: AudioDataLoaderPro return } console.error('Failed to load vocals audio data', error) + reportDegradationEvent('waveform_failed', { + message: error instanceof Error ? error.message : String(error), + elapsed_ms: Date.now() - startedAt, + not_ready_polls: attempt, + transient_retries: transientAttempt, + }) setAudioData(null) }) } diff --git a/frontend/lib/__tests__/degradation-events.test.ts b/frontend/lib/__tests__/degradation-events.test.ts new file mode 100644 index 00000000..9c78ae5e --- /dev/null +++ b/frontend/lib/__tests__/degradation-events.test.ts @@ -0,0 +1,64 @@ +/** + * @jest-environment jsdom + * @jest-environment-options {"url": "https://gen.nomadkaraoke.com/app/jobs?x=1#/abc12345/review"} + */ + +import { + reportDegradationEvent, + jobIdFromLocation, + __resetDegradationEventsForTest, +} from '@/lib/degradation-events' + +describe('degradation-events reporter', () => { + let fetchMock: jest.Mock + + beforeEach(() => { + __resetDegradationEventsForTest() + fetchMock = jest.fn(() => Promise.resolve({ ok: true })) + global.fetch = fetchMock as unknown as typeof fetch + }) + + it('POSTs the event with sanitized url, job id and detail', () => { + reportDegradationEvent('banner_unavailable', { stall_ms: 21000, probe_ok: false }) + + expect(fetchMock).toHaveBeenCalledTimes(1) + const [url, init] = fetchMock.mock.calls[0] + expect(url).toBe('https://api.nomadkaraoke.com/api/client-events') + const body = JSON.parse((init as RequestInit).body as string) + expect(body.type).toBe('banner_unavailable') + expect(body.url).not.toContain('x=1') // query stripped + expect(body.job_id).toBe('abc12345') + expect(body.detail).toEqual({ stall_ms: 21000, probe_ok: false }) + expect((init as RequestInit).keepalive).toBe(true) + }) + + it('throttles repeat reports of the same type', () => { + reportDegradationEvent('banner_reconnecting') + reportDegradationEvent('banner_reconnecting') + expect(fetchMock).toHaveBeenCalledTimes(1) + + // A different type is NOT throttled by the first one. + reportDegradationEvent('waveform_failed') + expect(fetchMock).toHaveBeenCalledTimes(2) + }) + + it('never throws when fetch rejects', () => { + fetchMock.mockImplementation(() => Promise.reject(new Error('offline'))) + expect(() => reportDegradationEvent('waveform_slow')).not.toThrow() + }) +}) + +describe('jobIdFromLocation', () => { + it('parses hash-router review URLs', () => { + expect(jobIdFromLocation('https://gen.nomadkaraoke.com/app/jobs#/d93747cd/review')).toBe('d93747cd') + }) + + it('parses path-style job URLs', () => { + expect(jobIdFromLocation('https://gen.nomadkaraoke.com/en/jobs/0370244a')).toBe('0370244a') + }) + + it('returns null when no job id is present', () => { + expect(jobIdFromLocation('https://gen.nomadkaraoke.com/')).toBeNull() + expect(jobIdFromLocation('not a url')).toBeNull() + }) +}) diff --git a/frontend/lib/backend-status.ts b/frontend/lib/backend-status.ts index af219027..2819858e 100644 --- a/frontend/lib/backend-status.ts +++ b/frontend/lib/backend-status.ts @@ -47,6 +47,11 @@ export const STALL_RECONNECTING_MS = 10_000 export const STALL_UNAVAILABLE_MS = 20_000 /** A health-probe verdict is trusted this long before we re-probe during a stall. */ export const PROBE_FRESH_MS = 10_000 +/** Escalating to the full "unavailable" card needs this many CONSECUTIVE failed + * probes — one failed 4s probe against a briefly-busy instance (e.g. a heavy + * transcode pinning the CPU) is not an outage. The reconnecting pill still + * shows after a single failure; the assertive message waits for confirmation. */ +export const UNAVAILABLE_PROBE_FAILURES = 2 let nextId = 1 /** id -> startedAt (ms) for every tracked backend GET currently in flight. */ @@ -65,6 +70,8 @@ let probeInFlight = false /** Last probe verdict (null = no verdict yet) and when it settled. */ let lastProbeOk: boolean | null = null let lastProbeAt = 0 +/** Failed probes in a row (reset by any success or reconfiguration). */ +let consecutiveProbeFailures = 0 /** Bumped on configureHealthProbe so a probe from a previous config can't land. */ let probeEpoch = 0 @@ -98,10 +105,15 @@ function maybeStartProbe() { if (lastProbeOk !== null && Date.now() - lastProbeAt < PROBE_FRESH_MS) return probeInFlight = true const epoch = probeEpoch + const record = (ok: boolean) => { + if (epoch !== probeEpoch) return + lastProbeOk = ok + consecutiveProbeFailures = ok ? 0 : consecutiveProbeFailures + 1 + } healthProbe() .then( - (ok) => (epoch === probeEpoch ? (lastProbeOk = ok) : undefined), - () => (epoch === probeEpoch ? (lastProbeOk = false) : undefined), + (ok) => record(ok), + () => record(false), ) .finally(() => { if (epoch !== probeEpoch) return @@ -118,7 +130,13 @@ function recompute() { // read is treated as a legitimately slow endpoint and the banner stays hidden. if (devOverride === null && next !== 'online' && healthProbe) { maybeStartProbe() - if (lastProbeOk !== false) next = 'online' + if (lastProbeOk !== false) { + next = 'online' + } else if (next === 'unavailable' && consecutiveProbeFailures < UNAVAILABLE_PROBE_FAILURES) { + // One failed probe can just be a briefly-pegged instance; keep the calm + // pill until a second consecutive failure confirms real unreachability. + next = 'reconnecting' + } } setStatus(next) // Stop the clock once nothing is outstanding (and no dev override needs it). @@ -163,6 +181,26 @@ export function configureHealthProbe(probe: HealthProbe | null): void { probeInFlight = false lastProbeOk = null lastProbeAt = 0 + consecutiveProbeFailures = 0 +} + +/** Diagnostic snapshot for degradation telemetry (see lib/degradation-events). */ +export function getBackendStatusDebug(): { + oldestStallMs: number + inFlightCount: number + lastProbeOk: boolean | null + consecutiveProbeFailures: number +} { + let oldest = Infinity + for (const startedAt of inFlight.values()) { + if (startedAt < oldest) oldest = startedAt + } + return { + oldestStallMs: inFlight.size === 0 ? 0 : Date.now() - oldest, + inFlightCount: inFlight.size, + lastProbeOk, + consecutiveProbeFailures, + } } export function getBackendStatus(): BackendStatus { diff --git a/frontend/lib/degradation-events.ts b/frontend/lib/degradation-events.ts new file mode 100644 index 00000000..6f3dd21a --- /dev/null +++ b/frontend/lib/degradation-events.ts @@ -0,0 +1,106 @@ +/** + * Degradation-event reporter: fire-and-forget telemetry for every user-visible + * degraded-service surface (connectivity banner, "temporarily unavailable" + * lyrics failure, slow/failed waveform). POSTs to /api/client-events, which + * persists to Cloud Logging + Firestore so frequency can be reviewed later. + * + * - Never throws; failures are swallowed (telemetry must not cause UX noise). + * - Per-type throttle so a flapping status can't spam the API. + * - No-ops on localhost (would otherwise report dev blips to prod). + * + * NOTE: intentionally does NOT import from lib/api.ts — api.ts imports + * backend-status.ts, and this module is called from banner-adjacent code, so + * importing api.ts here risks a module cycle. The base URL is derived the same + * way api.ts derives it. + */ + +export type DegradationEventType = + | 'banner_reconnecting' + | 'banner_unavailable' + | 'lyrics_load_failed' + | 'waveform_slow' + | 'waveform_failed' + +/** Minimum gap between two reports of the same event type. */ +const THROTTLE_MS = 60_000 +const lastSentAt = new Map() + +export function __resetDegradationEventsForTest() { + lastSentAt.clear() +} + +function apiBaseUrl(): string { + if (typeof window === 'undefined') return '' + const h = window.location.hostname + if (h === 'localhost' || h === '127.0.0.1') return '' // guarded below anyway + return process.env.NEXT_PUBLIC_API_URL || 'https://api.nomadkaraoke.com' +} + +function isLocalhost(): boolean { + if (typeof window === 'undefined') return true + const h = window.location.hostname + return h === 'localhost' || h === '127.0.0.1' || h === '[::1]' || h === '::1' +} + +/** Best-effort job id from review-style URLs (/app/jobs#/{id}/… or /jobs/{id}). */ +export function jobIdFromLocation(href: string): string | null { + try { + const u = new URL(href) + const hashMatch = u.hash.match(/^#\/([0-9a-f-]{6,64})(\/|$)/i) + if (hashMatch) return hashMatch[1] + const pathMatch = u.pathname.match(/\/jobs\/([0-9a-f-]{6,64})(\/|$)/i) + if (pathMatch) return pathMatch[1] + return null + } catch { + return null + } +} + +function sanitizedHref(): string { + try { + const u = new URL(window.location.href) + u.search = '' + return u.toString().slice(0, 2048) + } catch { + return '' + } +} + +/** + * Report one degradation event. Safe to call from anywhere client-side; + * throttled per type, silent on failure. + */ +export function reportDegradationEvent( + type: DegradationEventType, + detail?: Record, +): void { + try { + if (typeof window === 'undefined' || isLocalhost()) return + + const now = Date.now() + const last = lastSentAt.get(type) + if (last && now - last < THROTTLE_MS) return + lastSentAt.set(type, now) + + const body = { + type, + url: sanitizedHref(), + job_id: jobIdFromLocation(window.location.href), + user_email: null as string | null, + locale: document.documentElement.lang || 'en', + release: process.env.NEXT_PUBLIC_COMMIT_SHA || '', + detail: detail ?? null, + } + + void fetch(`${apiBaseUrl()}/api/client-events`, { + method: 'POST', + keepalive: true, + headers: { 'Content-Type': 'application/json' }, + body: JSON.stringify(body), + }).catch(() => { + /* swallow — telemetry must never surface */ + }) + } catch { + /* swallow */ + } +} diff --git a/pyproject.toml b/pyproject.toml index 892bca7e..e8eb8842 100644 --- a/pyproject.toml +++ b/pyproject.toml @@ -1,6 +1,6 @@ [tool.poetry] name = "karaoke-gen" -version = "0.230.3" +version = "0.231.0" description = "Generate karaoke videos with synchronized lyrics. Handles the entire process from downloading audio and lyrics to creating the final video with title screens." authors = ["Andrew Beveridge "] license = "MIT" diff --git a/scripts/load-test-review.py b/scripts/load-test-review.py new file mode 100644 index 00000000..e9365874 --- /dev/null +++ b/scripts/load-test-review.py @@ -0,0 +1,158 @@ +#!/usr/bin/env python3 +"""Concurrent review-page load test (the "10 tabs" regression guard). + +Reproduces the 2026-09-18 incident scenario: N simultaneous lyrics-review tab +loads, each firing the page's real GET trio (correction-data, waveform-data, +audio/vocals with a Range header). Before v0.230.0 this melted the backend +(9/30 requests 500'd, Cloud Run "no available instance"); after, all requests +succeed with warm tabs completing in a few seconds. + +Auth: per-job review tokens read from Firestore via ADC (read-only SA works) — +no admin secret needed. Requests go through curl (browser-ish UA) because the +Cloudflare WAF blocks python-urllib. + +Usage: + python scripts/load-test-review.py # 10 tabs vs prod + python scripts/load-test-review.py --tabs 20 + python scripts/load-test-review.py --max-tab-seconds 45 # cold-cache run + +Exit code 0 = pass (no failed requests, all tabs under --max-tab-seconds); +1 = regression. Prints a per-request table either way. +""" +from __future__ import annotations + +import argparse +import concurrent.futures +import json +import os +import subprocess +import sys +import time + +API_DEFAULT = "https://api.nomadkaraoke.com" +UA = ( + "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 " + "(KHTML, like Gecko) Chrome/128 Safari/537.36" +) + + +def fetch_review_jobs(limit: int) -> list[dict]: + """In-review jobs + review tokens from Firestore (ADC).""" + os.environ.setdefault("GOOGLE_CLOUD_PROJECT", "nomadkaraoke") + from google.cloud import firestore # type: ignore[import] + + db = firestore.Client(project="nomadkaraoke") + query = db.collection("jobs").where( + filter=firestore.FieldFilter("status", "in", ["in_review", "awaiting_review"]) + ).limit(limit * 2) + jobs = [] + for doc in query.stream(): + data = doc.to_dict() + token = data.get("review_token") or (data.get("state_data") or {}).get("review_token") + if token: + jobs.append({"job_id": doc.id, "token": token}) + if len(jobs) >= limit: + break + return jobs + + +def curl(url: str, extra_args: list[str] | None = None, timeout: int = 90) -> dict: + """One GET via curl; returns {status, seconds, bytes}.""" + cmd = [ + "curl", "-sS", "-o", "/dev/null", "-A", UA, "--max-time", str(timeout), + "-w", "%{http_code} %{time_total} %{size_download}", + *(extra_args or []), + url, + ] + started = time.monotonic() + proc = subprocess.run(cmd, capture_output=True, text=True) + if proc.returncode != 0 and not proc.stdout.strip(): + return {"status": 0, "seconds": time.monotonic() - started, "bytes": 0} + status, seconds, size = proc.stdout.strip().split()[:3] + return {"status": int(status), "seconds": float(seconds), "bytes": int(size)} + + +def tab_requests(api: str, job: dict) -> list[tuple[str, str, list[str]]]: + base = f"{api}/api/review/{job['job_id']}" + tok = f"review_token={job['token']}" + return [ + ("correction-data", f"{base}/correction-data?{tok}", []), + ("waveform-data", f"{base}/waveform-data?{tok}", []), + ("vocals", f"{base}/audio/vocals?{tok}", ["-r", "0-131071"]), + ] + + +def simulate_tab(api: str, job: dict) -> dict: + started = time.monotonic() + results = {} + with concurrent.futures.ThreadPoolExecutor(max_workers=3) as pool: + futures = { + pool.submit(curl, url, extra): name + for name, url, extra in tab_requests(api, job) + } + for fut in concurrent.futures.as_completed(futures): + results[futures[fut]] = fut.result() + return {"job_id": job["job_id"], "seconds": time.monotonic() - started, "requests": results} + + +def main() -> int: + parser = argparse.ArgumentParser(description=__doc__) + parser.add_argument("--tabs", type=int, default=10) + parser.add_argument("--api", default=API_DEFAULT) + parser.add_argument( + "--max-tab-seconds", type=float, default=20.0, + help="Fail if any simulated tab takes longer than this (default 20; " + "use ~45 for a cold-cache run where waveforms compute for the first time)", + ) + parser.add_argument( + "--jobs-file", default=None, + help="JSON file of [{job_id, token}] to skip the Firestore lookup", + ) + args = parser.parse_args() + if args.tabs < 1: + parser.error("--tabs must be >= 1") + + if args.jobs_file: + jobs = json.load(open(args.jobs_file))[: args.tabs] + else: + print(f"Fetching up to {args.tabs} in-review jobs from Firestore…") + jobs = fetch_review_jobs(args.tabs) + if len(jobs) < args.tabs: + # A partial run would "PASS" without reproducing the requested burst. + print( + f"Only {len(jobs)} in-review jobs available but --tabs {args.tabs} requested. " + "Aborting — re-run with a smaller --tabs to test at reduced scale." + ) + return 1 + print(f"Simulating {len(jobs)} concurrent review tabs against {args.api}\n") + + with concurrent.futures.ThreadPoolExecutor(max_workers=len(jobs)) as pool: + tabs = list(pool.map(lambda j: simulate_tab(args.api, j), jobs)) + + failed_requests = 0 + slow_tabs = 0 + for tab in sorted(tabs, key=lambda t: t["seconds"]): + parts = [] + for name in ("correction-data", "waveform-data", "vocals"): + r = tab["requests"][name] + ok = 200 <= r["status"] < 300 + failed_requests += 0 if ok else 1 + parts.append(f"{name}={r['status']}({r['seconds']:.1f}s)") + slow = tab["seconds"] > args.max_tab_seconds + slow_tabs += 1 if slow else 0 + flag = " ← SLOW" if slow else "" + print(f" {tab['job_id']} tab={tab['seconds']:5.1f}s {' '.join(parts)}{flag}") + + total = len(jobs) * 3 + print(f"\n{total - failed_requests}/{total} requests OK; " + f"{slow_tabs} tab(s) over {args.max_tab_seconds}s") + if failed_requests or slow_tabs: + print("RESULT: FAIL — concurrent review-load regression " + "(see docs/archive/2026-09-18-concurrent-review-reliability-investigation-and-plan.md)") + return 1 + print("RESULT: PASS") + return 0 + + +if __name__ == "__main__": + sys.exit(main())