feat: concurrent-reliability phases 2-5 — Cloud Run shape, degradation telemetry, banner tuning, load-test guard (#1028)

## Summary

Phases 2–5 of the concurrent-load reliability initiative (Phase 1 =
#1024, verified live: the 10-tab meltdown went from 9/30 failed requests
to 30/30 OK). Verbatim spec:
`docs/archive/2026-09-17-concurrent-review-reliability-verbatim-record.md`;
plan:
`docs/archive/2026-09-18-concurrent-review-reliability-investigation-and-plan.md`.

## P2 — Cloud Run shape
- **`--concurrency 80 → 16`** for karaoke-backend: the service is one
uvicorn process per 2-vCPU instance, and review loads are heavy; at 80 a
burst never triggers scale-out and all lands on the single min-instance.
16 makes bursts fan out while staying well above normal per-instance
load. (Deploy flags live in ci.yml — the service is not Pulumi-managed.)
- **Local preview ffmpeg fallback bounded**
(`LOCAL_PREVIEW_MAX_CONCURRENCY=1`, 900s queue timeout, fast
RuntimeError after that): the GCE-outage fallback runs ffmpeg ON the API
instance; unbounded, a few concurrent Preview clicks during an encoder
outage pegged both vCPUs, stalled the health probe, and showed everyone
the orange banner (Andrew's banner-during-preview screenshot).

## P3 — degradation telemetry (the explicit ask)
- New **`POST /api/client-events`** (unauthenticated so review-token
users report too; 30/min/IP rate limit; bot-filtered; detail capped;
URLs query-stripped). Each event → Firestore `client_events` collection
**and** a structured `client_event type=…` log line, so "how often are
users seeing this?" is reviewable via Firestore aggregation or Cloud
Logging correlation.
- Frontend reporter `lib/degradation-events.ts` (60s per-type throttle,
localhost no-op, never throws), wired into:
- `BackendStatusBanner` → `banner_reconnecting` / `banner_unavailable`
with stall age, in-flight count, probe state
- review lyrics-load failure → `lyrics_load_failed` with message/status
- `VocalsAudioDataLoader` → `waveform_slow` (>20s to first strips) and
`waveform_failed` (gave up) with poll/retry counts

## P4 — banner tuning
- Escalating to the full "We're having trouble reaching our servers"
card now requires **two consecutive failed health probes**
(`UNAVAILABLE_PROBE_FAILURES=2`). One failed 4s probe against a
briefly-pegged instance shows only the calm "Reconnecting…" pill.
Single-failure behavior for the pill is unchanged, as is instant
clearing on recovery.

## P5 — regression guard
- **`scripts/load-test-review.py`** checked in: replays the 10-tab
incident (Firestore-ADC review tokens, curl request trios, pass/fail
thresholds, exit code). Documented in `docs/TESTING.md`. Validated
against prod post-Phase-1: 9/9 requests OK, tabs < 3s.

## Testing
- Backend: 6 new tests for client-events
(persist/validate/rate-limit/bot/fail-soft/detail-cap) + local-preview
bound test — full suite **4517 passed**.
- Frontend: new degradation-events suite, two-probe escalation +
debug-snapshot tests; one existing api-fetch test updated to the new
escalation contract — full suite **1298 passed**.
- Live validation of the load-test tool against prod (small 3-tab run):
PASS.

## Notes for review
- Cloud Run concurrency change means bursts now spin up extra instances
(billable during requests only; max-instances unchanged at 20).
- Firestore `client_events` has no TTL policy yet (needs a
gcloud/console action our read-only ADC can't do); volume is tiny
(throttled per client) — can add later if it ever matters.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit

- **New Features**
- Added automatic reporting for degraded experiences, including
reconnecting or unavailable backend status, failed lyrics loads, and
slow or failed waveform loads.
- Added concurrent review-page load testing to help verify performance
under multiple open tabs.

- **Improvements**
- Backend status now remains in “reconnecting” until multiple health
checks fail, reducing premature unavailable messages.
- Local preview processing is limited during heavy demand to improve
stability.

- **Documentation**
- Added API documentation for client telemetry and guidance for running
concurrent-load regression tests.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->

---------

Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
This commit is contained in:
Andrew Beveridge ☄️andClaude Fable 5 authored and GitHub committed 2026-09-19 12:48:21 -04:00
1 parent e0f3d1bb45
commit 30cf41e7d6
18 files changed
+912 -18

No files matched your search

+7 -1
View File
@@ -1976,6 +1976,12 @@ jobs:
echo "Version: $VERSION"
echo "Previously-serving revision: ${PREV_REVISION:-<unknown>}"
# --concurrency 16 (was 80): the backend is one uvicorn process per
# 2-vCPU instance; review-page loads are heavy (audio byte-serving,
# waveform work). At 80 a 10-tab burst all landed on the single
# min-instance and wedged it ("no available instance" aborts,
# 2026-09-18 investigation). 16 makes bursts trigger scale-out while
# staying comfortably above normal per-instance load.
gcloud run deploy karaoke-backend \
--image $IMAGE \
--no-traffic \
@@ -1989,7 +1995,7 @@ jobs:
--cpu-throttling \
--execution-environment gen2 \
--timeout 1800 \
--concurrency 80 \
--concurrency 16 \
--max-instances 20 \
--min-instances 1 \
--vpc-connector projects/nomadkaraoke/locations/us-central1/connectors/cloud-run-connector \
+135
View File
@@ -0,0 +1,135 @@
"""POST /api/client-events — persistent telemetry for user-visible degradation.
Records every time the frontend shows a user a degraded-service surface (the
orange "trouble reaching our servers" banner, the stronger "temporarily
unavailable" error, a failed lyrics load, a slow/failed waveform), with enough
metadata to review frequency and context later. Two sinks per event:
1. A structured INFO log line (``client_event type=…``) — queryable in Cloud
Logging / Log Analytics alongside backend symptoms from the same window.
2. A Firestore doc in ``client_events`` — survives log retention, cheap to
aggregate for "how often are users seeing this?" reviews.
Unauthenticated (review-token users must be able to report) and rate-limited
per IP, mirroring /api/client-errors. The limiter is process-local, so the
effective ceiling is 120/min/IP × live instances — acceptable for this threat
model (non-malicious browsers, 60s client-side per-type throttle); Cloudflare's
zone-wide rate limit sits in front as a flood backstop. Events are dropped,
never queued, when the limiter trips — telemetry, not a delivery guarantee.
"""
from __future__ import annotations
import logging
import time
from datetime import datetime, timezone
from typing import Any, Literal, Optional
from fastapi import APIRouter, HTTPException, Request
from pydantic import BaseModel, Field, field_validator
from backend.services.error_monitor.frontend_ingestion import (
RateLimiter,
is_bot_user_agent,
sanitize_url,
)
logger = logging.getLogger(__name__)
router = APIRouter(prefix="/client-events", tags=["client-events"])
# Sized for a legitimate worst case from ONE IP: a many-tab burst (e.g. 16 review
# tabs at once) where every tab reports a banner + a waveform event — that must be
# fully recorded, not self-suppressed. Each tab throttles per-type client-side.
_limiter = RateLimiter(max_per_minute=120)
_db_singleton = None
# The full closed set of reportable events. Reject anything else so the
# collection stays reviewable (no free-form types accumulating).
EventType = Literal[
"banner_reconnecting",
"banner_unavailable",
"lyrics_load_failed",
"waveform_slow",
"waveform_failed",
]
def _get_db():
global _db_singleton
if _db_singleton is None:
from google.cloud import firestore # type: ignore[import]
_db_singleton = firestore.Client(project="nomadkaraoke")
return _db_singleton
class ClientEventPayload(BaseModel):
type: EventType
url: str = Field("", max_length=2048)
job_id: Optional[str] = Field(None, max_length=64)
user_email: Optional[str] = Field(None, max_length=320)
locale: str = Field("en", max_length=10)
release: str = Field("", max_length=64)
# Small free-form context (stall age, http status, elapsed ms, probe state…).
# Size-capped server-side; values are stored as-is.
detail: Optional[dict] = None
@field_validator("url", "locale", "release")
@classmethod
def strip(cls, v: str) -> str:
return (v or "").strip()
def _capped_detail(detail: Optional[dict]) -> dict[str, Any]:
"""Keep detail reviewable: at most 12 keys, scalar-ish values, short strings."""
if not isinstance(detail, dict):
return {}
out: dict[str, Any] = {}
for key, value in list(detail.items())[:12]:
k = str(key)[:40]
if isinstance(value, (int, float, bool)) or value is None:
out[k] = value
else:
out[k] = str(value)[:200]
return out
@router.post("", status_code=202)
def report_client_event(payload: ClientEventPayload, request: Request) -> dict:
client_ip = request.client.host if request.client else "unknown"
if not _limiter.allow(client_ip, time.monotonic()):
raise HTTPException(status_code=429, detail="too many events")
user_agent = request.headers.get("user-agent", "")[:1024]
if is_bot_user_agent(user_agent):
return {"status": "ignored"}
event = {
"type": payload.type,
"url": sanitize_url(payload.url),
"job_id": payload.job_id,
"user_email": payload.user_email,
"locale": payload.locale,
"release": payload.release,
"user_agent": user_agent,
"detail": _capped_detail(payload.detail),
"created_at": datetime.now(timezone.utc),
}
# Sink 1: structured log line — correlate with backend symptoms in Logging.
logger.info(
"client_event type=%s job_id=%s user=%s url=%s release=%s detail=%s",
event["type"],
event["job_id"],
event["user_email"],
event["url"],
event["release"],
event["detail"],
)
# Sink 2: Firestore — best-effort; telemetry must never error back to users.
try:
_get_db().collection("client_events").add(event)
except Exception:
logger.exception("Failed to persist client event (non-fatal)")
return {"status": "recorded"}
+60 -8
View File
@@ -13,6 +13,7 @@ Usage:
The baseApiUrl includes the job_id, and all endpoints are relative to that.
"""
import asyncio
import concurrent.futures
import logging
import hashlib
import json
@@ -278,12 +279,65 @@ def _prepare_preview_inputs(job, temp_dir: str, storage, is_duet: bool):
return correction_result, audio_path, output_config
def _render_preview_locally(job, updated_data: Dict[str, Any], is_duet: bool) -> str:
"""Render the full 360p preview locally and upload it to GCS.
# At most this many local preview renders at once. The local path runs ffmpeg
# ON the 2-vCPU API instance (it's the fallback when GCE preview encoding is
# down); unbounded, a GCE outage plus a few users clicking Preview saturates
# the instance's CPU, stalls the health probe, and shows everyone the orange
# "trouble reaching our servers" banner. Serializing renders keeps the API
# responsive — previews just queue a little longer during an encoder outage.
_LOCAL_PREVIEW_MAX_CONCURRENCY = max(1, int(os.getenv("LOCAL_PREVIEW_MAX_CONCURRENCY", "1")))
_LOCAL_PREVIEW_QUEUE_TIMEOUT_S = int(os.getenv("LOCAL_PREVIEW_QUEUE_TIMEOUT_S", "900"))
_local_preview_slots = asyncio.Semaphore(_LOCAL_PREVIEW_MAX_CONCURRENCY)
# Dedicated executor: admitted renders never compete with the shared to_thread
# pool, and its concurrent.futures-level done callbacks fire when the WORKER
# THREAD finishes — the asyncio wrapper future can be "cancelled" while the
# thread is still running, so it must not drive the slot release.
_local_preview_executor = concurrent.futures.ThreadPoolExecutor(
max_workers=_LOCAL_PREVIEW_MAX_CONCURRENCY, thread_name_prefix="local-preview"
)
Blocking (downloads + ffmpeg) — call via asyncio.to_thread. Used when GCE
preview encoding is disabled and as the fallback when it fails.
async def _render_preview_locally(job, updated_data: Dict[str, Any], is_duet: bool) -> str:
"""Render the full 360p preview locally (bounded) and upload it to GCS.
Used when GCE preview encoding is disabled and as the fallback when it
fails. Admission happens HERE on the event loop (async semaphore +
wait_for) so queued renders wait as cheap coroutines — only an ADMITTED
render occupies a thread-pool worker for its blocking downloads + ffmpeg.
"""
try:
await asyncio.wait_for(
_local_preview_slots.acquire(), timeout=_LOCAL_PREVIEW_QUEUE_TIMEOUT_S
)
except asyncio.TimeoutError:
raise RuntimeError(
"Local preview encoder is busy (waited "
f"{_LOCAL_PREVIEW_QUEUE_TIMEOUT_S}s for a slot) — try again shortly"
)
def _run_render():
return _render_preview_locally_unbounded(job, updated_data, is_duet)
loop = asyncio.get_running_loop()
worker_future = _local_preview_executor.submit(_run_render)
def _release_slot(fut: "concurrent.futures.Future") -> None:
# Runs when the WORKER THREAD actually finishes. Releasing from an
# async finally (or the asyncio wrapper's callback) would free the slot
# the moment the awaiting request is cancelled — while the ffmpeg
# render thread is still consuming CPU — letting a second render start
# and defeating the concurrency bound.
if not fut.cancelled():
fut.exception() # mark retrieved when the caller was cancelled/abandoned
try:
loop.call_soon_threadsafe(_local_preview_slots.release)
except RuntimeError:
pass # loop already closed (shutdown) — nothing left to bound
worker_future.add_done_callback(_release_slot)
return await asyncio.wrap_future(worker_future)
def _render_preview_locally_unbounded(job, updated_data: Dict[str, Any], is_duet: bool) -> str:
job_id = job.job_id
storage = StorageService()
with tempfile.TemporaryDirectory() as temp_dir:
@@ -349,7 +403,7 @@ async def _encode_preview_in_background(
logger.warning(f"Job {job_id}: GCE preview encoding failed, falling back to local: {gce_error}")
try:
await asyncio.to_thread(_render_preview_locally, job, updated_data, is_duet)
await _render_preview_locally(job, updated_data, is_duet)
logger.info(f"Job {job_id}: Preview generated locally after GCE failure: {preview_hash}")
except Exception as local_error:
logger.error(
@@ -1873,9 +1927,7 @@ async def generate_preview_video(
# Local path (GCE preview disabled, or GCE setup failed): render
# the full preview synchronously — no cold-start problem, but slow.
preview_hash = await asyncio.to_thread(
_render_preview_locally, job, updated_data, is_duet
)
preview_hash = await _render_preview_locally(job, updated_data, is_duet)
logger.info(f"Job {job_id}: Preview video generated: {preview_hash}")
span.set_attribute("preview_hash", preview_hash)
+2 -1
View File
@@ -10,7 +10,7 @@ from fastapi import FastAPI
from fastapi.middleware.cors import CORSMiddleware
from backend.config import settings, validate_production_config
from backend.api.routes import health, jobs, internal, file_upload, review, auth, audio_search, themes, users, admin, tenant, tenant_admin, tenant_bulk, rate_limits, push, catalog, encoding_worker, client_errors, bulk, parse_titles
from backend.api.routes import health, jobs, internal, file_upload, review, auth, audio_search, themes, users, admin, tenant, tenant_admin, tenant_bulk, rate_limits, push, catalog, encoding_worker, client_errors, client_events, bulk, parse_titles
from backend.services.tracing import setup_tracing, instrument_app, get_current_trace_id
from backend.services.structured_logging import setup_structured_logging
from backend.services.spacy_preloader import preload_spacy_model
@@ -213,6 +213,7 @@ app.include_router(bulk.router, prefix="/api") # Bulk Mode: multi-job submissio
from backend.api.routes import requests_board
app.include_router(requests_board.router, prefix="/api") # Public song-request voting board
app.include_router(client_errors.router, prefix="/api") # Frontend crash reports
app.include_router(client_events.router, prefix="/api") # Frontend degradation telemetry (banner/unavailable/waveform events)
app.include_router(tenant.router) # Tenant/white-label configuration (no /api prefix, router has it)
app.include_router(tenant_admin.router) # Admin tenant provisioning (router has /api prefix)
app.include_router(tenant_bulk.router) # Tenant bulk-upload filename analysis (router has /api prefix)
+85
View File
@@ -0,0 +1,85 @@
"""Tests for POST /api/client-events (degradation telemetry)."""
from unittest.mock import MagicMock, patch
import pytest
from fastapi import FastAPI
from fastapi.testclient import TestClient
from backend.api.routes import client_events
@pytest.fixture
def client():
app = FastAPI()
app.include_router(client_events.router, prefix="/api")
return TestClient(app)
@pytest.fixture(autouse=True)
def fresh_limiter_and_db(monkeypatch):
monkeypatch.setattr(client_events, "_limiter", client_events.RateLimiter(max_per_minute=30))
db = MagicMock()
monkeypatch.setattr(client_events, "_db_singleton", db)
return db
def _event(**overrides):
payload = {
"type": "banner_unavailable",
"url": "https://gen.nomadkaraoke.com/app/jobs?token=SECRET#/abc123/review",
"job_id": "abc123",
"locale": "en",
"release": "deadbeef",
"detail": {"stall_ms": 21000, "probe_ok": False},
}
payload.update(overrides)
return payload
class TestReportClientEvent:
def test_valid_event_persists_to_firestore(self, client, fresh_limiter_and_db):
resp = client.post("/api/client-events", json=_event())
assert resp.status_code == 202
assert resp.json() == {"status": "recorded"}
fresh_limiter_and_db.collection.assert_called_once_with("client_events")
(doc,) = fresh_limiter_and_db.collection.return_value.add.call_args[0]
assert doc["type"] == "banner_unavailable"
assert doc["job_id"] == "abc123"
assert doc["detail"] == {"stall_ms": 21000, "probe_ok": False}
# Query string (may carry tokens) must be stripped before persisting.
assert "SECRET" not in doc["url"]
def test_unknown_event_type_rejected(self, client):
resp = client.post("/api/client-events", json=_event(type="something_else"))
assert resp.status_code == 422
def test_rate_limited(self, client, monkeypatch):
monkeypatch.setattr(client_events, "_limiter", client_events.RateLimiter(max_per_minute=2))
assert client.post("/api/client-events", json=_event()).status_code == 202
assert client.post("/api/client-events", json=_event()).status_code == 202
assert client.post("/api/client-events", json=_event()).status_code == 429
def test_bot_user_agent_ignored(self, client, fresh_limiter_and_db):
resp = client.post(
"/api/client-events",
json=_event(),
headers={"User-Agent": "Googlebot/2.1 (+http://www.google.com/bot.html)"},
)
assert resp.status_code == 202
assert resp.json() == {"status": "ignored"}
fresh_limiter_and_db.collection.assert_not_called()
def test_firestore_failure_still_returns_202(self, client, fresh_limiter_and_db):
fresh_limiter_and_db.collection.return_value.add.side_effect = Exception("firestore down")
resp = client.post("/api/client-events", json=_event())
assert resp.status_code == 202 # telemetry must never error back to users
def test_detail_is_capped(self):
big = {f"key{i}": "x" * 500 for i in range(30)}
capped = client_events._capped_detail(big)
assert len(capped) == 12
assert all(len(v) <= 200 for v in capped.values())
assert client_events._capped_detail(None) == {}
assert client_events._capped_detail("not a dict") == {}
+100 -2
View File
@@ -1825,7 +1825,7 @@ class TestEncodePreviewInBackground:
encoding_service = MagicMock()
encoding_service.encode_preview_video = AsyncMock(side_effect=RuntimeError("stockout"))
storage = MagicMock()
local_render = MagicMock(return_value="h1")
local_render = AsyncMock(return_value="h1")
self._run(mock_job, encoding_service, storage, local_render)
local_render.assert_called_once()
storage.upload_json.assert_not_called()
@@ -1836,7 +1836,7 @@ class TestEncodePreviewInBackground:
encoding_service = MagicMock()
encoding_service.encode_preview_video = AsyncMock(side_effect=RuntimeError("stockout"))
storage = MagicMock()
local_render = MagicMock(side_effect=RuntimeError("ffmpeg died"))
local_render = AsyncMock(side_effect=RuntimeError("ffmpeg died"))
self._run(mock_job, encoding_service, storage, local_render)
storage.upload_json.assert_called_once_with(
"jobs/job-bg/previews/h1.error.json",
@@ -2041,3 +2041,101 @@ class TestReviewAudioSingleFlightAndByteCap:
assert "a" not in review_module._review_audio_cache
assert review_module._review_audio_cache_bytes <= 100
assert set(review_module._review_audio_cache) == {"b", "c", "d", "e"}
class TestLocalPreviewRenderBound:
"""The local preview ffmpeg fallback must be bounded so a GCE outage plus a
few concurrent Preview clicks can't saturate the API instance's CPU.
Admission is an async semaphore on the event loop (waiters are cheap
coroutines, not parked executor threads)."""
def test_serializes_and_times_out_admission(self, monkeypatch):
import asyncio
import threading
from backend.api.routes import review as review_module
monkeypatch.setattr(review_module, "_LOCAL_PREVIEW_QUEUE_TIMEOUT_S", 0.2)
monkeypatch.setattr(review_module, "_local_preview_slots", asyncio.Semaphore(1))
started = threading.Event()
release = threading.Event()
def slow_render(job, updated_data, is_duet):
started.set()
release.wait(timeout=5)
return "hash"
monkeypatch.setattr(review_module, "_render_preview_locally_unbounded", slow_render)
async def scenario():
first = asyncio.create_task(
review_module._render_preview_locally(MagicMock(), {}, False)
)
await asyncio.to_thread(started.wait, 5)
# Slot held by the first render — the second caller's admission wait
# times out fast instead of piling more ffmpeg onto the instance.
with pytest.raises(RuntimeError, match="busy"):
await review_module._render_preview_locally(MagicMock(), {}, False)
release.set()
assert await first == "hash"
# Slot released — a fresh render is admitted immediately.
monkeypatch.setattr(
review_module, "_render_preview_locally_unbounded",
lambda job, updated_data, is_duet: "hash2",
)
assert await review_module._render_preview_locally(MagicMock(), {}, False) == "hash2"
asyncio.run(scenario())
def test_cancelled_caller_does_not_free_slot_while_render_runs(self, monkeypatch):
"""Cancelling the awaiting request must NOT release the slot while the
ffmpeg worker thread is still running — otherwise a second render can
start and defeat the concurrency bound."""
import asyncio
import threading
from backend.api.routes import review as review_module
monkeypatch.setattr(review_module, "_LOCAL_PREVIEW_QUEUE_TIMEOUT_S", 0.2)
monkeypatch.setattr(review_module, "_local_preview_slots", asyncio.Semaphore(1))
started = threading.Event()
release = threading.Event()
def slow_render(job, updated_data, is_duet):
started.set()
release.wait(timeout=5)
return "hash"
monkeypatch.setattr(review_module, "_render_preview_locally_unbounded", slow_render)
async def scenario():
first = asyncio.create_task(
review_module._render_preview_locally(MagicMock(), {}, False)
)
await asyncio.to_thread(started.wait, 5)
first.cancel()
with pytest.raises(asyncio.CancelledError):
await first
# Worker thread is still rendering — the slot must still be held.
with pytest.raises(RuntimeError, match="busy"):
await review_module._render_preview_locally(MagicMock(), {}, False)
# Worker finishes → done-callback releases the slot.
release.set()
for _ in range(50):
if not review_module._local_preview_slots.locked():
break
await asyncio.sleep(0.05)
monkeypatch.setattr(
review_module, "_render_preview_locally_unbounded",
lambda job, updated_data, is_duet: "hash2",
)
assert await review_module._render_preview_locally(MagicMock(), {}, False) == "hash2"
asyncio.run(scenario())
+27
View File
@@ -2669,3 +2669,30 @@ search returned nothing usable.
Stripe webhooks implemented at `/api/users/webhooks/stripe`.
Job status webhooks not yet implemented.
## Client Telemetry
### Degradation Events
```http
POST /api/client-events
```
Unauthenticated, rate-limited (120/min/IP — sized so a many-tab burst from one IP is fully recorded), bot-filtered. The frontend reports every
user-visible degraded-service surface: `banner_reconnecting`, `banner_unavailable`
(the connectivity banner — reported by `BackendStatusBanner`), `lyrics_load_failed`
(review page "temporarily unavailable"), `waveform_slow` / `waveform_failed`
(Waveforms-mode strips late or missing). Body: `{type, url, job_id?, user_email?,
locale, release, detail?}` — `detail` is size-capped server-side (≤12 keys, ≤200
chars per value); URLs are stored query-stripped.
Each event emits a structured `client_event type=…` INFO log line and is
best-effort persisted to the Firestore `client_events` collection (a Firestore
write failure is logged but still returns `202` — telemetry never errors back to
the client), so frequency can be reviewed later
(Firestore aggregation, or Cloud Logging correlated with backend symptoms from the
same window). Reporter: `frontend/lib/degradation-events.ts` (60s per-type throttle,
localhost no-op). Added in v0.231.0 — see
`docs/archive/2026-09-18-concurrent-review-reliability-investigation-and-plan.md`.
Crash reports are the separate `POST /api/client-errors` (error-monitor pipeline).
+19
View File
@@ -456,6 +456,25 @@ The production tests cover:
- Final encoding and download
- Admin dashboard functionality
## Concurrent-Load Regression Test (review pages)
`scripts/load-test-review.py` replays the 2026-09-18 "10 tabs melt the backend"
incident: N simultaneous review-tab loads, each firing the page's real GET trio
(correction-data, waveform-data, ranged audio/vocals), authenticated with per-job
review tokens read from Firestore via ADC (no admin secret needed).
```bash
python scripts/load-test-review.py # 10 tabs vs prod, warm thresholds
python scripts/load-test-review.py --tabs 20
python scripts/load-test-review.py --max-tab-seconds 45 # cold caches (first-ever waveforms)
```
Pass = every request 2xx and every tab under the threshold; exit 1 otherwise.
Run it after changes to the review hot path (`backend/api/routes/review.py`,
`audio_transcoding_service`, `audio_analysis_service`) or Cloud Run deploy flags.
Background + before/after numbers:
`docs/archive/2026-09-18-concurrent-review-reliability-investigation-and-plan.md`.
## Ad-Hoc Production Debugging (for Agents)
When asked to "test it yourself in prod" or debug a specific production issue, use the debug script template:
+7 -1
View File
@@ -131,10 +131,16 @@ describe('apiFetch', () => {
// stays hidden (a slow endpoint must not alarm the user without evidence).
await jest.advanceTimersByTimeAsync(STALL_RECONNECTING_MS)
expect(getBackendStatus()).toBe('online')
// Probe times out (~4s) → outage confirmed → the hint appears.
// Probe times out (~4s) → outage suspected → the hint appears.
await jest.advanceTimersByTimeAsync(5_000)
expect(getBackendStatus()).toBe('reconnecting')
// The stall alone now qualifies for "unavailable", but escalation needs a
// SECOND consecutive probe failure (one failed probe can just be a briefly
// pegged instance) — until then the calm pill is all the user sees.
await jest.advanceTimersByTimeAsync(STALL_UNAVAILABLE_MS - STALL_RECONNECTING_MS - 5_000)
expect(getBackendStatus()).toBe('reconnecting')
// Verdict goes stale (~24s) → probe #2 fires → times out (~28s) → confirmed.
await jest.advanceTimersByTimeAsync(9_000)
expect(getBackendStatus()).toBe('unavailable')
settleGet(mockResponse(200, { ok: true }))
+50
View File
@@ -8,6 +8,7 @@
import {
getBackendStatus,
getBackendStatusDebug,
beginRequest,
endRequest,
configureHealthProbe,
@@ -158,4 +159,53 @@ describe('backend-status store (health-probe confirmation)', () => {
await jest.advanceTimersByTimeAsync(3000) // verdict lands
expect(getBackendStatus()).toBe('reconnecting')
})
it('needs TWO consecutive failed probes before escalating to unavailable', async () => {
// Each probe takes 3s to fail — realistic for a probe racing its own
// timeout against a briefly-pegged (not down) instance.
const probe = jest.fn(
() => new Promise<boolean>((res) => setTimeout(() => res(false), 3000)),
)
configureHealthProbe(probe)
open.push(beginRequest())
// t=10s: probe #1 starts. t=13s: failure #1 recorded.
// t=20s: the stall alone now qualifies for "unavailable", but only one
// probe failure has been recorded (verdict fresh until t=23s) —
// the calm reconnecting pill must be all the user sees.
await jest.advanceTimersByTimeAsync(STALL_UNAVAILABLE_MS + 1000) // t=21s
expect(getBackendStatus()).toBe('reconnecting')
expect(getBackendStatusDebug().consecutiveProbeFailures).toBe(1)
// t=23s: verdict stale → probe #2 starts. t=26s: failure #2 → escalate.
await jest.advanceTimersByTimeAsync(6000) // t=27s
expect(getBackendStatus()).toBe('unavailable')
expect(getBackendStatusDebug().consecutiveProbeFailures).toBe(2)
})
it('a probe success resets the consecutive-failure count', async () => {
let reachable = false
const probe = jest.fn(() => Promise.resolve(reachable))
configureHealthProbe(probe)
open.push(beginRequest())
await jest.advanceTimersByTimeAsync(STALL_UNAVAILABLE_MS + PROBE_FRESH_MS)
expect(getBackendStatusDebug().consecutiveProbeFailures).toBeGreaterThanOrEqual(2)
reachable = true
await jest.advanceTimersByTimeAsync(PROBE_FRESH_MS + 2000)
expect(getBackendStatus()).toBe('online')
expect(getBackendStatusDebug().consecutiveProbeFailures).toBe(0)
})
it('exposes a diagnostic snapshot for telemetry', async () => {
configureHealthProbe(() => Promise.resolve(false))
open.push(beginRequest())
await jest.advanceTimersByTimeAsync(STALL_RECONNECTING_MS + 1000)
const debug = getBackendStatusDebug()
expect(debug.inFlightCount).toBe(1)
expect(debug.oldestStallMs).toBeGreaterThanOrEqual(STALL_RECONNECTING_MS)
expect(debug.lastProbeOk).toBe(false)
})
})
@@ -1,6 +1,7 @@
"use client"
import { useEffect, useState, useCallback, useRef } from "react"
import { reportDegradationEvent } from "@/lib/degradation-events"
import { useAuth } from "@/lib/auth"
import { api, Job, createLyricsReviewApiClient, lyricsReviewApi } from "@/lib/api"
import { Spinner } from "@/components/ui/spinner"
@@ -334,6 +335,10 @@ function LyricsReviewWrapper({ job, isLocalMode = false, isReplay = false }: { j
setCorrectionData(data)
} catch (err) {
console.error("Failed to load correction data:", err)
reportDegradationEvent("lyrics_load_failed", {
message: err instanceof Error ? err.message : String(err),
status: (err as { status?: number })?.status ?? null,
})
setError(err instanceof Error ? err.message : "Failed to load lyrics data")
} finally {
setIsLoading(false)
@@ -5,8 +5,10 @@ import { useTranslations } from "next-intl"
import { Loader2, CloudOff, X } from "lucide-react"
import {
useBackendStatus,
getBackendStatusDebug,
__installBackendStatusDevHook,
} from "@/lib/backend-status"
import { reportDegradationEvent } from "@/lib/degradation-events"
/**
* App-wide, non-blocking banner that reacts to backend connectivity (see
@@ -37,6 +39,26 @@ export function BackendStatusBanner() {
if (status === "online") setDismissed(false)
}, [status])
// Persist every banner display server-side (Andrew's telemetry ask): this is
// the single global banner instance, so reporting here == "shown to a user".
// The reporter throttles per type, so a flapping status can't spam.
useEffect(() => {
if (status === "online") return
// A dismissed unavailable card renders nothing — reporting it would count a
// banner the user never saw. (dismiss only re-arms after passing online.)
if (status === "unavailable" && dismissed) return
const debug = getBackendStatusDebug()
reportDegradationEvent(
status === "unavailable" ? "banner_unavailable" : "banner_reconnecting",
{
stall_ms: debug.oldestStallMs,
in_flight: debug.inFlightCount,
probe_ok: debug.lastProbeOk,
probe_failures: debug.consecutiveProbeFailures,
},
)
}, [status, dismissed])
if (status === "online") return null
if (status === "reconnecting") {
@@ -1,6 +1,12 @@
import { AudioData, AudioNotReadyError, fetchAudioData, isTransientAudioError } from '@/lib/audio-data'
import { reportDegradationEvent } from '@/lib/degradation-events'
import { createContext, PropsWithChildren, useEffect, useState } from 'react'
// A first paint whose waveform strips took longer than this to appear counts as
// a degraded experience worth recording (Andrew's "wait 1+ minutes" complaint).
// Excludes legitimate separation-in-progress waits shorter than one 202 poll.
const SLOW_LOAD_REPORT_MS = 20_000
export const VocalsAudioDataLoaderContext = createContext<{ audioData: AudioData | null }>({ audioData: null })
export interface AudioDataLoaderProps extends PropsWithChildren {
@@ -30,11 +36,21 @@ export const VocalsAudioDataLoader = ({ audioUrl, children }: AudioDataLoaderPro
// audioUrl has already been set.
let cancelled = false
let retryTimer: ReturnType<typeof setTimeout> | undefined
const startedAt = Date.now()
const load = (attempt: number, transientAttempt: number) => {
fetchAudioData(audioUrl)
.then((audioData) => {
if (!cancelled) setAudioData(audioData)
if (cancelled) return
const elapsedMs = Date.now() - startedAt
if (elapsedMs >= SLOW_LOAD_REPORT_MS) {
reportDegradationEvent('waveform_slow', {
elapsed_ms: elapsedMs,
not_ready_polls: attempt,
transient_retries: transientAttempt,
})
}
setAudioData(audioData)
})
.catch((error) => {
if (cancelled) return
@@ -49,6 +65,12 @@ export const VocalsAudioDataLoader = ({ audioUrl, children }: AudioDataLoaderPro
return
}
console.error('Failed to load vocals audio data', error)
reportDegradationEvent('waveform_failed', {
message: error instanceof Error ? error.message : String(error),
elapsed_ms: Date.now() - startedAt,
not_ready_polls: attempt,
transient_retries: transientAttempt,
})
setAudioData(null)
})
}
@@ -0,0 +1,64 @@
/**
* @jest-environment jsdom
* @jest-environment-options {"url": "https://gen.nomadkaraoke.com/app/jobs?x=1#/abc12345/review"}
*/
import {
reportDegradationEvent,
jobIdFromLocation,
__resetDegradationEventsForTest,
} from '@/lib/degradation-events'
describe('degradation-events reporter', () => {
let fetchMock: jest.Mock
beforeEach(() => {
__resetDegradationEventsForTest()
fetchMock = jest.fn(() => Promise.resolve({ ok: true }))
global.fetch = fetchMock as unknown as typeof fetch
})
it('POSTs the event with sanitized url, job id and detail', () => {
reportDegradationEvent('banner_unavailable', { stall_ms: 21000, probe_ok: false })
expect(fetchMock).toHaveBeenCalledTimes(1)
const [url, init] = fetchMock.mock.calls[0]
expect(url).toBe('https://api.nomadkaraoke.com/api/client-events')
const body = JSON.parse((init as RequestInit).body as string)
expect(body.type).toBe('banner_unavailable')
expect(body.url).not.toContain('x=1') // query stripped
expect(body.job_id).toBe('abc12345')
expect(body.detail).toEqual({ stall_ms: 21000, probe_ok: false })
expect((init as RequestInit).keepalive).toBe(true)
})
it('throttles repeat reports of the same type', () => {
reportDegradationEvent('banner_reconnecting')
reportDegradationEvent('banner_reconnecting')
expect(fetchMock).toHaveBeenCalledTimes(1)
// A different type is NOT throttled by the first one.
reportDegradationEvent('waveform_failed')
expect(fetchMock).toHaveBeenCalledTimes(2)
})
it('never throws when fetch rejects', () => {
fetchMock.mockImplementation(() => Promise.reject(new Error('offline')))
expect(() => reportDegradationEvent('waveform_slow')).not.toThrow()
})
})
describe('jobIdFromLocation', () => {
it('parses hash-router review URLs', () => {
expect(jobIdFromLocation('https://gen.nomadkaraoke.com/app/jobs#/d93747cd/review')).toBe('d93747cd')
})
it('parses path-style job URLs', () => {
expect(jobIdFromLocation('https://gen.nomadkaraoke.com/en/jobs/0370244a')).toBe('0370244a')
})
it('returns null when no job id is present', () => {
expect(jobIdFromLocation('https://gen.nomadkaraoke.com/')).toBeNull()
expect(jobIdFromLocation('not a url')).toBeNull()
})
})
+41 -3
View File
@@ -47,6 +47,11 @@ export const STALL_RECONNECTING_MS = 10_000
export const STALL_UNAVAILABLE_MS = 20_000
/** A health-probe verdict is trusted this long before we re-probe during a stall. */
export const PROBE_FRESH_MS = 10_000
/** Escalating to the full "unavailable" card needs this many CONSECUTIVE failed
* probes — one failed 4s probe against a briefly-busy instance (e.g. a heavy
* transcode pinning the CPU) is not an outage. The reconnecting pill still
* shows after a single failure; the assertive message waits for confirmation. */
export const UNAVAILABLE_PROBE_FAILURES = 2
let nextId = 1
/** id -> startedAt (ms) for every tracked backend GET currently in flight. */
@@ -65,6 +70,8 @@ let probeInFlight = false
/** Last probe verdict (null = no verdict yet) and when it settled. */
let lastProbeOk: boolean | null = null
let lastProbeAt = 0
/** Failed probes in a row (reset by any success or reconfiguration). */
let consecutiveProbeFailures = 0
/** Bumped on configureHealthProbe so a probe from a previous config can't land. */
let probeEpoch = 0
@@ -98,10 +105,15 @@ function maybeStartProbe() {
if (lastProbeOk !== null && Date.now() - lastProbeAt < PROBE_FRESH_MS) return
probeInFlight = true
const epoch = probeEpoch
const record = (ok: boolean) => {
if (epoch !== probeEpoch) return
lastProbeOk = ok
consecutiveProbeFailures = ok ? 0 : consecutiveProbeFailures + 1
}
healthProbe()
.then(
(ok) => (epoch === probeEpoch ? (lastProbeOk = ok) : undefined),
() => (epoch === probeEpoch ? (lastProbeOk = false) : undefined),
(ok) => record(ok),
() => record(false),
)
.finally(() => {
if (epoch !== probeEpoch) return
@@ -118,7 +130,13 @@ function recompute() {
// read is treated as a legitimately slow endpoint and the banner stays hidden.
if (devOverride === null && next !== 'online' && healthProbe) {
maybeStartProbe()
if (lastProbeOk !== false) next = 'online'
if (lastProbeOk !== false) {
next = 'online'
} else if (next === 'unavailable' && consecutiveProbeFailures < UNAVAILABLE_PROBE_FAILURES) {
// One failed probe can just be a briefly-pegged instance; keep the calm
// pill until a second consecutive failure confirms real unreachability.
next = 'reconnecting'
}
}
setStatus(next)
// Stop the clock once nothing is outstanding (and no dev override needs it).
@@ -163,6 +181,26 @@ export function configureHealthProbe(probe: HealthProbe | null): void {
probeInFlight = false
lastProbeOk = null
lastProbeAt = 0
consecutiveProbeFailures = 0
}
/** Diagnostic snapshot for degradation telemetry (see lib/degradation-events). */
export function getBackendStatusDebug(): {
oldestStallMs: number
inFlightCount: number
lastProbeOk: boolean | null
consecutiveProbeFailures: number
} {
let oldest = Infinity
for (const startedAt of inFlight.values()) {
if (startedAt < oldest) oldest = startedAt
}
return {
oldestStallMs: inFlight.size === 0 ? 0 : Date.now() - oldest,
inFlightCount: inFlight.size,
lastProbeOk,
consecutiveProbeFailures,
}
}
export function getBackendStatus(): BackendStatus {
+106
View File
@@ -0,0 +1,106 @@
/**
* Degradation-event reporter: fire-and-forget telemetry for every user-visible
* degraded-service surface (connectivity banner, "temporarily unavailable"
* lyrics failure, slow/failed waveform). POSTs to /api/client-events, which
* persists to Cloud Logging + Firestore so frequency can be reviewed later.
*
* - Never throws; failures are swallowed (telemetry must not cause UX noise).
* - Per-type throttle so a flapping status can't spam the API.
* - No-ops on localhost (would otherwise report dev blips to prod).
*
* NOTE: intentionally does NOT import from lib/api.ts — api.ts imports
* backend-status.ts, and this module is called from banner-adjacent code, so
* importing api.ts here risks a module cycle. The base URL is derived the same
* way api.ts derives it.
*/
export type DegradationEventType =
| 'banner_reconnecting'
| 'banner_unavailable'
| 'lyrics_load_failed'
| 'waveform_slow'
| 'waveform_failed'
/** Minimum gap between two reports of the same event type. */
const THROTTLE_MS = 60_000
const lastSentAt = new Map<DegradationEventType, number>()
export function __resetDegradationEventsForTest() {
lastSentAt.clear()
}
function apiBaseUrl(): string {
if (typeof window === 'undefined') return ''
const h = window.location.hostname
if (h === 'localhost' || h === '127.0.0.1') return '' // guarded below anyway
return process.env.NEXT_PUBLIC_API_URL || 'https://api.nomadkaraoke.com'
}
function isLocalhost(): boolean {
if (typeof window === 'undefined') return true
const h = window.location.hostname
return h === 'localhost' || h === '127.0.0.1' || h === '[::1]' || h === '::1'
}
/** Best-effort job id from review-style URLs (/app/jobs#/{id}/… or /jobs/{id}). */
export function jobIdFromLocation(href: string): string | null {
try {
const u = new URL(href)
const hashMatch = u.hash.match(/^#\/([0-9a-f-]{6,64})(\/|$)/i)
if (hashMatch) return hashMatch[1]
const pathMatch = u.pathname.match(/\/jobs\/([0-9a-f-]{6,64})(\/|$)/i)
if (pathMatch) return pathMatch[1]
return null
} catch {
return null
}
}
function sanitizedHref(): string {
try {
const u = new URL(window.location.href)
u.search = ''
return u.toString().slice(0, 2048)
} catch {
return ''
}
}
/**
* Report one degradation event. Safe to call from anywhere client-side;
* throttled per type, silent on failure.
*/
export function reportDegradationEvent(
type: DegradationEventType,
detail?: Record<string, unknown>,
): void {
try {
if (typeof window === 'undefined' || isLocalhost()) return
const now = Date.now()
const last = lastSentAt.get(type)
if (last && now - last < THROTTLE_MS) return
lastSentAt.set(type, now)
const body = {
type,
url: sanitizedHref(),
job_id: jobIdFromLocation(window.location.href),
user_email: null as string | null,
locale: document.documentElement.lang || 'en',
release: process.env.NEXT_PUBLIC_COMMIT_SHA || '',
detail: detail ?? null,
}
void fetch(`${apiBaseUrl()}/api/client-events`, {
method: 'POST',
keepalive: true,
headers: { 'Content-Type': 'application/json' },
body: JSON.stringify(body),
}).catch(() => {
/* swallow — telemetry must never surface */
})
} catch {
/* swallow */
}
}
+1 -1
View File
@@ -1,6 +1,6 @@
[tool.poetry]
name = "karaoke-gen"
version = "0.230.3"
version = "0.231.0"
description = "Generate karaoke videos with synchronized lyrics. Handles the entire process from downloading audio and lyrics to creating the final video with title screens."
authors = ["Andrew Beveridge <andrew@beveridge.uk>"]
license = "MIT"
+158
View File
@@ -0,0 +1,158 @@
#!/usr/bin/env python3
"""Concurrent review-page load test (the "10 tabs" regression guard).
Reproduces the 2026-09-18 incident scenario: N simultaneous lyrics-review tab
loads, each firing the page's real GET trio (correction-data, waveform-data,
audio/vocals with a Range header). Before v0.230.0 this melted the backend
(9/30 requests 500'd, Cloud Run "no available instance"); after, all requests
succeed with warm tabs completing in a few seconds.
Auth: per-job review tokens read from Firestore via ADC (read-only SA works) —
no admin secret needed. Requests go through curl (browser-ish UA) because the
Cloudflare WAF blocks python-urllib.
Usage:
python scripts/load-test-review.py # 10 tabs vs prod
python scripts/load-test-review.py --tabs 20
python scripts/load-test-review.py --max-tab-seconds 45 # cold-cache run
Exit code 0 = pass (no failed requests, all tabs under --max-tab-seconds);
1 = regression. Prints a per-request table either way.
"""
from __future__ import annotations
import argparse
import concurrent.futures
import json
import os
import subprocess
import sys
import time
API_DEFAULT = "https://api.nomadkaraoke.com"
UA = (
"Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 "
"(KHTML, like Gecko) Chrome/128 Safari/537.36"
)
def fetch_review_jobs(limit: int) -> list[dict]:
"""In-review jobs + review tokens from Firestore (ADC)."""
os.environ.setdefault("GOOGLE_CLOUD_PROJECT", "nomadkaraoke")
from google.cloud import firestore # type: ignore[import]
db = firestore.Client(project="nomadkaraoke")
query = db.collection("jobs").where(
filter=firestore.FieldFilter("status", "in", ["in_review", "awaiting_review"])
).limit(limit * 2)
jobs = []
for doc in query.stream():
data = doc.to_dict()
token = data.get("review_token") or (data.get("state_data") or {}).get("review_token")
if token:
jobs.append({"job_id": doc.id, "token": token})
if len(jobs) >= limit:
break
return jobs
def curl(url: str, extra_args: list[str] | None = None, timeout: int = 90) -> dict:
"""One GET via curl; returns {status, seconds, bytes}."""
cmd = [
"curl", "-sS", "-o", "/dev/null", "-A", UA, "--max-time", str(timeout),
"-w", "%{http_code} %{time_total} %{size_download}",
*(extra_args or []),
url,
]
started = time.monotonic()
proc = subprocess.run(cmd, capture_output=True, text=True)
if proc.returncode != 0 and not proc.stdout.strip():
return {"status": 0, "seconds": time.monotonic() - started, "bytes": 0}
status, seconds, size = proc.stdout.strip().split()[:3]
return {"status": int(status), "seconds": float(seconds), "bytes": int(size)}
def tab_requests(api: str, job: dict) -> list[tuple[str, str, list[str]]]:
base = f"{api}/api/review/{job['job_id']}"
tok = f"review_token={job['token']}"
return [
("correction-data", f"{base}/correction-data?{tok}", []),
("waveform-data", f"{base}/waveform-data?{tok}", []),
("vocals", f"{base}/audio/vocals?{tok}", ["-r", "0-131071"]),
]
def simulate_tab(api: str, job: dict) -> dict:
started = time.monotonic()
results = {}
with concurrent.futures.ThreadPoolExecutor(max_workers=3) as pool:
futures = {
pool.submit(curl, url, extra): name
for name, url, extra in tab_requests(api, job)
}
for fut in concurrent.futures.as_completed(futures):
results[futures[fut]] = fut.result()
return {"job_id": job["job_id"], "seconds": time.monotonic() - started, "requests": results}
def main() -> int:
parser = argparse.ArgumentParser(description=__doc__)
parser.add_argument("--tabs", type=int, default=10)
parser.add_argument("--api", default=API_DEFAULT)
parser.add_argument(
"--max-tab-seconds", type=float, default=20.0,
help="Fail if any simulated tab takes longer than this (default 20; "
"use ~45 for a cold-cache run where waveforms compute for the first time)",
)
parser.add_argument(
"--jobs-file", default=None,
help="JSON file of [{job_id, token}] to skip the Firestore lookup",
)
args = parser.parse_args()
if args.tabs < 1:
parser.error("--tabs must be >= 1")
if args.jobs_file:
jobs = json.load(open(args.jobs_file))[: args.tabs]
else:
print(f"Fetching up to {args.tabs} in-review jobs from Firestore…")
jobs = fetch_review_jobs(args.tabs)
if len(jobs) < args.tabs:
# A partial run would "PASS" without reproducing the requested burst.
print(
f"Only {len(jobs)} in-review jobs available but --tabs {args.tabs} requested. "
"Aborting — re-run with a smaller --tabs to test at reduced scale."
)
return 1
print(f"Simulating {len(jobs)} concurrent review tabs against {args.api}\n")
with concurrent.futures.ThreadPoolExecutor(max_workers=len(jobs)) as pool:
tabs = list(pool.map(lambda j: simulate_tab(args.api, j), jobs))
failed_requests = 0
slow_tabs = 0
for tab in sorted(tabs, key=lambda t: t["seconds"]):
parts = []
for name in ("correction-data", "waveform-data", "vocals"):
r = tab["requests"][name]
ok = 200 <= r["status"] < 300
failed_requests += 0 if ok else 1
parts.append(f"{name}={r['status']}({r['seconds']:.1f}s)")
slow = tab["seconds"] > args.max_tab_seconds
slow_tabs += 1 if slow else 0
flag = " ← SLOW" if slow else ""
print(f" {tab['job_id']} tab={tab['seconds']:5.1f}s {' '.join(parts)}{flag}")
total = len(jobs) * 3
print(f"\n{total - failed_requests}/{total} requests OK; "
f"{slow_tabs} tab(s) over {args.max_tab_seconds}s")
if failed_requests or slow_tabs:
print("RESULT: FAIL — concurrent review-load regression "
"(see docs/archive/2026-09-18-concurrent-review-reliability-investigation-and-plan.md)")
return 1
print("RESULT: PASS")
return 0
if __name__ == "__main__":
sys.exit(main())