mirror of
https://github.com/nomadkaraoke/karaoke-gen.git
synced 2026-11-20 13:07:57 +09:00
## Summary The "Reconnecting" pill and the "servers unavailable" banner (164 / 49 events, 2026-09-19 → 10-02) were mostly caused by the single API instance's **event loop freezing for 20–45s**. Every request on the instance, `/api/health` included, completed at the same moment. The full evidence and plan are in `docs/archive/2026-10-03-backend-loop-freezes-telemetry-plan.md`. ### What froze the loop, and the fix for each - **Screens worker:** it ran as a post-response `async def` BackgroundTask doing 4K PIL renders, GCS transfers and ffmpeg transcodes on the loop. Under `--cpu-throttling` it was also CPU-starved. **Fix:** it now runs inline in its Cloud Tasks request (CPU stays allocated, failures still return 200 so Cloud Tasks doesn't double-run it), with the heavy steps in `asyncio.to_thread`. Also offloaded: style load, timing signals, pre-apply. - **Matplotlib font cache:** it was rebuilt on every new instance (17/28 episodes). **Fix:** it's pre-built in `Dockerfile.base` (`MPLCONFIGDIR=/opt/mplconfig`), and pyplot is imported lazily. - **Hourly stale-review cron:** an `async def` with zero awaits doing ≤1500 reads and SMTP sends. **Fix:** it runs in a thread. - **166 `async def` routes that never awaited** (sync Firestore/GCS/email/Gemini on the loop, e.g. magic-link verify). **Fix:** they're now plain `def`, so they run in FastAPI's threadpool. The new AST guard `test_no_blocking_async_routes.py` fails CI on new ones. It also flags sync routes that call `asyncio.create_task`/`get_running_loop`; that caught `confirm_duration` and `trigger_youtube_queue_processing`, which stay async. ### Safety net - `backend/services/loop_watchdog.py`: a heartbeat plus a daemon thread. Any stall over 1s logs `EVENT_LOOP_STALL` with the loop thread's stack and a `culprit=file:line`. Stalls of 5s or more are written to `client_events` as `server_loop_stall`. - Pulumi: the log metric `backend/event_loop_stalls` plus an alert on ≥2 stalls of ≥10s in 30 min. **This still needs a local `pulumi up`.** Pulumi is never applied by CI, and the admin gcloud login has expired. ### Telemetry gaps fixed - **Identity:** `user_email` used to be hard-coded to null. The frontend now sends its session token, and the server resolves `user_email`, `is_admin`, `is_internal` and `is_test`. A client-sent email is ignored. - **New fields:** `device_fingerprint` (the existing FingerprintJS visitorId), `tab_id`, `episode_id`, and `tenant`. - **New event:** `banner_recovered`, carrying `duration_ms` and `peak_status`. - **Report:** `scripts/client_events_report.py` summarises the data, excluding admin/internal/test accounts by default. ## Testing - `make test` passes: backend 5529, plus the emulator suite, karaoke_gen unit tests, and frontend Jest 1623. - New tests: - Watchdog: a deterministic state machine plus detection of a real blocking call. - Event-loop responsiveness: a ticker keeps ticking while the heavy steps run. - Screens inline and failure semantics. - `client_events` identity and fields. - The AST guard. - The report summary. - Frontend reporter and banner episode tests. - Local CodeRabbit CLI review: 0 findings. 🤖 Generated with [Claude Code](https://claude.com/claude-code) @coderabbitai ignore --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
112 lines
5.0 KiB
Python
112 lines
5.0 KiB
Python
#!/usr/bin/env python3
|
|
"""Summarise degradation telemetry (Firestore ``client_events``).
|
|
|
|
Answers "how often are real users seeing the Reconnecting pill / servers
|
|
unavailable banner, for how long, and what froze the backend?"
|
|
|
|
python scripts/client_events_report.py # last 7 days, customers only
|
|
python scripts/client_events_report.py --days 30 --all # include admin/internal/test
|
|
|
|
Needs read access to Firestore project ``nomadkaraoke`` (ADC; the read-only
|
|
``claude-readonly`` SA works). Read-only.
|
|
|
|
Collection fields (since 2026-10-03; older docs have none of the identity
|
|
fields): type, source (client|server), created_at, user_email, is_admin,
|
|
is_internal, is_test, tenant, device_fingerprint, tab_id, episode_id, url,
|
|
release, detail{...}. Server docs (type ``server_loop_stall``) add culprit /
|
|
culprits / stack / revision / instance_id.
|
|
"""
|
|
from __future__ import annotations
|
|
|
|
import argparse
|
|
import collections
|
|
from datetime import datetime, timedelta, timezone
|
|
|
|
|
|
def _tenant_from_url(url: str) -> str | None:
|
|
"""Fallback for pre-2026-10-03 docs that lack the server-derived ``tenant``."""
|
|
host = url.split("://", 1)[-1].split("/", 1)[0].lower()
|
|
if not host.endswith(".nomadkaraoke.com"):
|
|
return None
|
|
sub = host[: -len(".nomadkaraoke.com")]
|
|
return None if (not sub or "." in sub or sub in {"gen", "www", "api", "app"}) else sub
|
|
|
|
|
|
def _who(e: dict) -> str:
|
|
return e.get("user_email") or (
|
|
f"fp:{e['device_fingerprint'][:10]}" if e.get("device_fingerprint") else "(anonymous)"
|
|
)
|
|
|
|
|
|
def _is_noise(e: dict) -> bool:
|
|
return bool(e.get("is_admin") or e.get("is_internal") or e.get("is_test"))
|
|
|
|
|
|
def summarise(events: list[dict], include_noise: bool) -> str:
|
|
out: list[str] = []
|
|
client = [e for e in events if e.get("source", "client") == "client" and e.get("type") != "server_loop_stall"]
|
|
stalls = [e for e in events if e.get("type") == "server_loop_stall"]
|
|
if not include_noise:
|
|
dropped = sum(1 for e in client if _is_noise(e))
|
|
client = [e for e in client if not _is_noise(e)]
|
|
out.append(f"(excluded {dropped} admin/internal/test events; --all to include)")
|
|
|
|
by_type = collections.Counter(e["type"] for e in client)
|
|
out.append("\n== Client events by type")
|
|
for t, n in by_type.most_common():
|
|
out.append(f" {t:22s} {n}")
|
|
|
|
out.append("\n== Banners per day (reconnecting / unavailable / waking)")
|
|
days: dict[str, collections.Counter] = collections.defaultdict(collections.Counter)
|
|
for e in client:
|
|
if e["type"].startswith("banner_") and e["type"] != "banner_recovered":
|
|
days[e["created_at"].strftime("%Y-%m-%d")][e["type"]] += 1
|
|
for d in sorted(days):
|
|
c = days[d]
|
|
out.append(f" {d} {c['banner_reconnecting']:4d} / {c['banner_unavailable']:4d} / {c['banner_waking']:4d}")
|
|
|
|
rec = [e for e in client if e["type"] == "banner_recovered"]
|
|
if rec:
|
|
durs = sorted((e.get("detail") or {}).get("duration_ms") or 0 for e in rec)
|
|
p = lambda q: durs[min(len(durs) - 1, int(q * len(durs)))] / 1000 # noqa: E731
|
|
peaks = collections.Counter((e.get("detail") or {}).get("peak_status") for e in rec)
|
|
out.append(
|
|
f"\n== Episode durations ({len(rec)} recovered): p50 {p(0.5):.1f}s p90 {p(0.9):.1f}s "
|
|
f"max {durs[-1] / 1000:.1f}s peaks {dict(peaks)}"
|
|
)
|
|
|
|
users = collections.Counter(_who(e) for e in client if e["type"] in ("banner_reconnecting", "banner_unavailable"))
|
|
out.append(f"\n== Affected users/devices: {len(users)}")
|
|
for u, n in users.most_common(15):
|
|
out.append(f" {n:4d} {u}")
|
|
|
|
tenants = collections.Counter(e.get("tenant") or _tenant_from_url(e.get("url", "")) or "(consumer)" for e in client)
|
|
out.append("\n== By tenant: " + ", ".join(f"{t}={n}" for t, n in tenants.most_common()))
|
|
|
|
out.append(f"\n== Server event-loop stalls >=5s: {len(stalls)}")
|
|
culprits = collections.Counter((e.get("detail") or {}).get("culprit", "?") for e in stalls)
|
|
for c, n in culprits.most_common(15):
|
|
durs = [(e.get("detail") or {}).get("duration_ms", 0) for e in stalls if (e.get("detail") or {}).get("culprit", "?") == c]
|
|
out.append(f" {n:3d}x max {max(durs) / 1000:5.1f}s {c}")
|
|
return "\n".join(out)
|
|
|
|
|
|
def main() -> None:
|
|
ap = argparse.ArgumentParser(description=__doc__, formatter_class=argparse.RawDescriptionHelpFormatter)
|
|
ap.add_argument("--days", type=int, default=7)
|
|
ap.add_argument("--all", action="store_true", help="include admin/internal/test accounts")
|
|
args = ap.parse_args()
|
|
|
|
from google.cloud import firestore
|
|
from google.cloud.firestore_v1.base_query import FieldFilter
|
|
|
|
since = datetime.now(timezone.utc) - timedelta(days=args.days)
|
|
db = firestore.Client(project="nomadkaraoke")
|
|
events = [d.to_dict() for d in db.collection("client_events").where(filter=FieldFilter("created_at", ">=", since)).stream()]
|
|
print(f"client_events since {since:%Y-%m-%d %H:%M} UTC: {len(events)} docs")
|
|
print(summarise(events, include_noise=args.all))
|
|
|
|
|
|
if __name__ == "__main__":
|
|
main()
|