Files
karaoke-gen/scripts
Andrew Beveridge ☄️andClaude Fable 5 30cf41e7d6 feat: concurrent-reliability phases 2-5 — Cloud Run shape, degradation telemetry, banner tuning, load-test guard (#1028)
## Summary

Phases 2–5 of the concurrent-load reliability initiative (Phase 1 =
#1024, verified live: the 10-tab meltdown went from 9/30 failed requests
to 30/30 OK). Verbatim spec:
`docs/archive/2026-09-17-concurrent-review-reliability-verbatim-record.md`;
plan:
`docs/archive/2026-09-18-concurrent-review-reliability-investigation-and-plan.md`.

## P2 — Cloud Run shape
- **`--concurrency 80 → 16`** for karaoke-backend: the service is one
uvicorn process per 2-vCPU instance, and review loads are heavy; at 80 a
burst never triggers scale-out and all lands on the single min-instance.
16 makes bursts fan out while staying well above normal per-instance
load. (Deploy flags live in ci.yml — the service is not Pulumi-managed.)
- **Local preview ffmpeg fallback bounded**
(`LOCAL_PREVIEW_MAX_CONCURRENCY=1`, 900s queue timeout, fast
RuntimeError after that): the GCE-outage fallback runs ffmpeg ON the API
instance; unbounded, a few concurrent Preview clicks during an encoder
outage pegged both vCPUs, stalled the health probe, and showed everyone
the orange banner (Andrew's banner-during-preview screenshot).

## P3 — degradation telemetry (the explicit ask)
- New **`POST /api/client-events`** (unauthenticated so review-token
users report too; 30/min/IP rate limit; bot-filtered; detail capped;
URLs query-stripped). Each event → Firestore `client_events` collection
**and** a structured `client_event type=…` log line, so "how often are
users seeing this?" is reviewable via Firestore aggregation or Cloud
Logging correlation.
- Frontend reporter `lib/degradation-events.ts` (60s per-type throttle,
localhost no-op, never throws), wired into:
- `BackendStatusBanner` → `banner_reconnecting` / `banner_unavailable`
with stall age, in-flight count, probe state
- review lyrics-load failure → `lyrics_load_failed` with message/status
- `VocalsAudioDataLoader` → `waveform_slow` (>20s to first strips) and
`waveform_failed` (gave up) with poll/retry counts

## P4 — banner tuning
- Escalating to the full "We're having trouble reaching our servers"
card now requires **two consecutive failed health probes**
(`UNAVAILABLE_PROBE_FAILURES=2`). One failed 4s probe against a
briefly-pegged instance shows only the calm "Reconnecting…" pill.
Single-failure behavior for the pill is unchanged, as is instant
clearing on recovery.

## P5 — regression guard
- **`scripts/load-test-review.py`** checked in: replays the 10-tab
incident (Firestore-ADC review tokens, curl request trios, pass/fail
thresholds, exit code). Documented in `docs/TESTING.md`. Validated
against prod post-Phase-1: 9/9 requests OK, tabs < 3s.

## Testing
- Backend: 6 new tests for client-events
(persist/validate/rate-limit/bot/fail-soft/detail-cap) + local-preview
bound test — full suite **4517 passed**.
- Frontend: new degradation-events suite, two-probe escalation +
debug-snapshot tests; one existing api-fetch test updated to the new
escalation contract — full suite **1298 passed**.
- Live validation of the load-test tool against prod (small 3-tab run):
PASS.

## Notes for review
- Cloud Run concurrency change means bursts now spin up extra instances
(billable during requests only; max-instances unchanged at 20).
- Firestore `client_events` has no TTL policy yet (needs a
gcloud/console action our read-only ADC can't do); volume is tiny
(throttled per client) — can add later if it ever matters.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit

- **New Features**
- Added automatic reporting for degraded experiences, including
reconnecting or unavailable backend status, failed lyrics loads, and
slow or failed waveform loads.
- Added concurrent review-page load testing to help verify performance
under multiple open tabs.

- **Improvements**
- Backend status now remains in “reconnecting” until multiple health
checks fail, reducing premature unavailable messages.
- Local preview processing is limited during heavy demand to improve
stability.

- **Documentation**
- Added API documentation for client telemetry and guidance for running
concurrent-load regression tests.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->

---------

Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
2026-09-19 12:48:21 -04:00
..
2026-01-05 01:33:17 +00:00