Files
karaoke-gen/scripts/load-test-review.py
T
Andrew Beveridge ☄️andClaude Fable 5 30cf41e7d6 feat: concurrent-reliability phases 2-5 — Cloud Run shape, degradation telemetry, banner tuning, load-test guard (#1028)
## Summary

Phases 2–5 of the concurrent-load reliability initiative (Phase 1 =
#1024, verified live: the 10-tab meltdown went from 9/30 failed requests
to 30/30 OK). Verbatim spec:
`docs/archive/2026-09-17-concurrent-review-reliability-verbatim-record.md`;
plan:
`docs/archive/2026-09-18-concurrent-review-reliability-investigation-and-plan.md`.

## P2 — Cloud Run shape
- **`--concurrency 80 → 16`** for karaoke-backend: the service is one
uvicorn process per 2-vCPU instance, and review loads are heavy; at 80 a
burst never triggers scale-out and all lands on the single min-instance.
16 makes bursts fan out while staying well above normal per-instance
load. (Deploy flags live in ci.yml — the service is not Pulumi-managed.)
- **Local preview ffmpeg fallback bounded**
(`LOCAL_PREVIEW_MAX_CONCURRENCY=1`, 900s queue timeout, fast
RuntimeError after that): the GCE-outage fallback runs ffmpeg ON the API
instance; unbounded, a few concurrent Preview clicks during an encoder
outage pegged both vCPUs, stalled the health probe, and showed everyone
the orange banner (Andrew's banner-during-preview screenshot).

## P3 — degradation telemetry (the explicit ask)
- New **`POST /api/client-events`** (unauthenticated so review-token
users report too; 30/min/IP rate limit; bot-filtered; detail capped;
URLs query-stripped). Each event → Firestore `client_events` collection
**and** a structured `client_event type=…` log line, so "how often are
users seeing this?" is reviewable via Firestore aggregation or Cloud
Logging correlation.
- Frontend reporter `lib/degradation-events.ts` (60s per-type throttle,
localhost no-op, never throws), wired into:
- `BackendStatusBanner` → `banner_reconnecting` / `banner_unavailable`
with stall age, in-flight count, probe state
- review lyrics-load failure → `lyrics_load_failed` with message/status
- `VocalsAudioDataLoader` → `waveform_slow` (>20s to first strips) and
`waveform_failed` (gave up) with poll/retry counts

## P4 — banner tuning
- Escalating to the full "We're having trouble reaching our servers"
card now requires **two consecutive failed health probes**
(`UNAVAILABLE_PROBE_FAILURES=2`). One failed 4s probe against a
briefly-pegged instance shows only the calm "Reconnecting…" pill.
Single-failure behavior for the pill is unchanged, as is instant
clearing on recovery.

## P5 — regression guard
- **`scripts/load-test-review.py`** checked in: replays the 10-tab
incident (Firestore-ADC review tokens, curl request trios, pass/fail
thresholds, exit code). Documented in `docs/TESTING.md`. Validated
against prod post-Phase-1: 9/9 requests OK, tabs < 3s.

## Testing
- Backend: 6 new tests for client-events
(persist/validate/rate-limit/bot/fail-soft/detail-cap) + local-preview
bound test — full suite **4517 passed**.
- Frontend: new degradation-events suite, two-probe escalation +
debug-snapshot tests; one existing api-fetch test updated to the new
escalation contract — full suite **1298 passed**.
- Live validation of the load-test tool against prod (small 3-tab run):
PASS.

## Notes for review
- Cloud Run concurrency change means bursts now spin up extra instances
(billable during requests only; max-instances unchanged at 20).
- Firestore `client_events` has no TTL policy yet (needs a
gcloud/console action our read-only ADC can't do); volume is tiny
(throttled per client) — can add later if it ever matters.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit

- **New Features**
- Added automatic reporting for degraded experiences, including
reconnecting or unavailable backend status, failed lyrics loads, and
slow or failed waveform loads.
- Added concurrent review-page load testing to help verify performance
under multiple open tabs.

- **Improvements**
- Backend status now remains in “reconnecting” until multiple health
checks fail, reducing premature unavailable messages.
- Local preview processing is limited during heavy demand to improve
stability.

- **Documentation**
- Added API documentation for client telemetry and guidance for running
concurrent-load regression tests.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->

---------

Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
2026-09-19 12:48:21 -04:00

159 lines
6.1 KiB
Python

#!/usr/bin/env python3
"""Concurrent review-page load test (the "10 tabs" regression guard).
Reproduces the 2026-09-18 incident scenario: N simultaneous lyrics-review tab
loads, each firing the page's real GET trio (correction-data, waveform-data,
audio/vocals with a Range header). Before v0.230.0 this melted the backend
(9/30 requests 500'd, Cloud Run "no available instance"); after, all requests
succeed with warm tabs completing in a few seconds.
Auth: per-job review tokens read from Firestore via ADC (read-only SA works) —
no admin secret needed. Requests go through curl (browser-ish UA) because the
Cloudflare WAF blocks python-urllib.
Usage:
python scripts/load-test-review.py # 10 tabs vs prod
python scripts/load-test-review.py --tabs 20
python scripts/load-test-review.py --max-tab-seconds 45 # cold-cache run
Exit code 0 = pass (no failed requests, all tabs under --max-tab-seconds);
1 = regression. Prints a per-request table either way.
"""
from __future__ import annotations
import argparse
import concurrent.futures
import json
import os
import subprocess
import sys
import time
API_DEFAULT = "https://api.nomadkaraoke.com"
UA = (
"Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 "
"(KHTML, like Gecko) Chrome/128 Safari/537.36"
)
def fetch_review_jobs(limit: int) -> list[dict]:
"""In-review jobs + review tokens from Firestore (ADC)."""
os.environ.setdefault("GOOGLE_CLOUD_PROJECT", "nomadkaraoke")
from google.cloud import firestore # type: ignore[import]
db = firestore.Client(project="nomadkaraoke")
query = db.collection("jobs").where(
filter=firestore.FieldFilter("status", "in", ["in_review", "awaiting_review"])
).limit(limit * 2)
jobs = []
for doc in query.stream():
data = doc.to_dict()
token = data.get("review_token") or (data.get("state_data") or {}).get("review_token")
if token:
jobs.append({"job_id": doc.id, "token": token})
if len(jobs) >= limit:
break
return jobs
def curl(url: str, extra_args: list[str] | None = None, timeout: int = 90) -> dict:
"""One GET via curl; returns {status, seconds, bytes}."""
cmd = [
"curl", "-sS", "-o", "/dev/null", "-A", UA, "--max-time", str(timeout),
"-w", "%{http_code} %{time_total} %{size_download}",
*(extra_args or []),
url,
]
started = time.monotonic()
proc = subprocess.run(cmd, capture_output=True, text=True)
if proc.returncode != 0 and not proc.stdout.strip():
return {"status": 0, "seconds": time.monotonic() - started, "bytes": 0}
status, seconds, size = proc.stdout.strip().split()[:3]
return {"status": int(status), "seconds": float(seconds), "bytes": int(size)}
def tab_requests(api: str, job: dict) -> list[tuple[str, str, list[str]]]:
base = f"{api}/api/review/{job['job_id']}"
tok = f"review_token={job['token']}"
return [
("correction-data", f"{base}/correction-data?{tok}", []),
("waveform-data", f"{base}/waveform-data?{tok}", []),
("vocals", f"{base}/audio/vocals?{tok}", ["-r", "0-131071"]),
]
def simulate_tab(api: str, job: dict) -> dict:
started = time.monotonic()
results = {}
with concurrent.futures.ThreadPoolExecutor(max_workers=3) as pool:
futures = {
pool.submit(curl, url, extra): name
for name, url, extra in tab_requests(api, job)
}
for fut in concurrent.futures.as_completed(futures):
results[futures[fut]] = fut.result()
return {"job_id": job["job_id"], "seconds": time.monotonic() - started, "requests": results}
def main() -> int:
parser = argparse.ArgumentParser(description=__doc__)
parser.add_argument("--tabs", type=int, default=10)
parser.add_argument("--api", default=API_DEFAULT)
parser.add_argument(
"--max-tab-seconds", type=float, default=20.0,
help="Fail if any simulated tab takes longer than this (default 20; "
"use ~45 for a cold-cache run where waveforms compute for the first time)",
)
parser.add_argument(
"--jobs-file", default=None,
help="JSON file of [{job_id, token}] to skip the Firestore lookup",
)
args = parser.parse_args()
if args.tabs < 1:
parser.error("--tabs must be >= 1")
if args.jobs_file:
jobs = json.load(open(args.jobs_file))[: args.tabs]
else:
print(f"Fetching up to {args.tabs} in-review jobs from Firestore…")
jobs = fetch_review_jobs(args.tabs)
if len(jobs) < args.tabs:
# A partial run would "PASS" without reproducing the requested burst.
print(
f"Only {len(jobs)} in-review jobs available but --tabs {args.tabs} requested. "
"Aborting — re-run with a smaller --tabs to test at reduced scale."
)
return 1
print(f"Simulating {len(jobs)} concurrent review tabs against {args.api}\n")
with concurrent.futures.ThreadPoolExecutor(max_workers=len(jobs)) as pool:
tabs = list(pool.map(lambda j: simulate_tab(args.api, j), jobs))
failed_requests = 0
slow_tabs = 0
for tab in sorted(tabs, key=lambda t: t["seconds"]):
parts = []
for name in ("correction-data", "waveform-data", "vocals"):
r = tab["requests"][name]
ok = 200 <= r["status"] < 300
failed_requests += 0 if ok else 1
parts.append(f"{name}={r['status']}({r['seconds']:.1f}s)")
slow = tab["seconds"] > args.max_tab_seconds
slow_tabs += 1 if slow else 0
flag = " ← SLOW" if slow else ""
print(f" {tab['job_id']} tab={tab['seconds']:5.1f}s {' '.join(parts)}{flag}")
total = len(jobs) * 3
print(f"\n{total - failed_requests}/{total} requests OK; "
f"{slow_tabs} tab(s) over {args.max_tab_seconds}s")
if failed_requests or slow_tabs:
print("RESULT: FAIL — concurrent review-load regression "
"(see docs/archive/2026-09-18-concurrent-review-reliability-investigation-and-plan.md)")
return 1
print("RESULT: PASS")
return 0
if __name__ == "__main__":
sys.exit(main())