mirror of
https://github.com/nomadkaraoke/karaoke-gen.git
synced 2026-11-20 13:07:57 +09:00
## Summary The "Reconnecting" pill and the "servers unavailable" banner (164 / 49 events, 2026-09-19 → 10-02) were mostly caused by the single API instance's **event loop freezing for 20–45s**. Every request on the instance, `/api/health` included, completed at the same moment. The full evidence and plan are in `docs/archive/2026-10-03-backend-loop-freezes-telemetry-plan.md`. ### What froze the loop, and the fix for each - **Screens worker:** it ran as a post-response `async def` BackgroundTask doing 4K PIL renders, GCS transfers and ffmpeg transcodes on the loop. Under `--cpu-throttling` it was also CPU-starved. **Fix:** it now runs inline in its Cloud Tasks request (CPU stays allocated, failures still return 200 so Cloud Tasks doesn't double-run it), with the heavy steps in `asyncio.to_thread`. Also offloaded: style load, timing signals, pre-apply. - **Matplotlib font cache:** it was rebuilt on every new instance (17/28 episodes). **Fix:** it's pre-built in `Dockerfile.base` (`MPLCONFIGDIR=/opt/mplconfig`), and pyplot is imported lazily. - **Hourly stale-review cron:** an `async def` with zero awaits doing ≤1500 reads and SMTP sends. **Fix:** it runs in a thread. - **166 `async def` routes that never awaited** (sync Firestore/GCS/email/Gemini on the loop, e.g. magic-link verify). **Fix:** they're now plain `def`, so they run in FastAPI's threadpool. The new AST guard `test_no_blocking_async_routes.py` fails CI on new ones. It also flags sync routes that call `asyncio.create_task`/`get_running_loop`; that caught `confirm_duration` and `trigger_youtube_queue_processing`, which stay async. ### Safety net - `backend/services/loop_watchdog.py`: a heartbeat plus a daemon thread. Any stall over 1s logs `EVENT_LOOP_STALL` with the loop thread's stack and a `culprit=file:line`. Stalls of 5s or more are written to `client_events` as `server_loop_stall`. - Pulumi: the log metric `backend/event_loop_stalls` plus an alert on ≥2 stalls of ≥10s in 30 min. **This still needs a local `pulumi up`.** Pulumi is never applied by CI, and the admin gcloud login has expired. ### Telemetry gaps fixed - **Identity:** `user_email` used to be hard-coded to null. The frontend now sends its session token, and the server resolves `user_email`, `is_admin`, `is_internal` and `is_test`. A client-sent email is ignored. - **New fields:** `device_fingerprint` (the existing FingerprintJS visitorId), `tab_id`, `episode_id`, and `tenant`. - **New event:** `banner_recovered`, carrying `duration_ms` and `peak_status`. - **Report:** `scripts/client_events_report.py` summarises the data, excluding admin/internal/test accounts by default. ## Testing - `make test` passes: backend 5529, plus the emulator suite, karaoke_gen unit tests, and frontend Jest 1623. - New tests: - Watchdog: a deterministic state machine plus detection of a real blocking call. - Event-loop responsiveness: a ticker keeps ticking while the heavy steps run. - Screens inline and failure semantics. - `client_events` identity and fields. - The AST guard. - The report summary. - Frontend reporter and banner episode tests. - Local CodeRabbit CLI review: 0 findings. 🤖 Generated with [Claude Code](https://claude.com/claude-code) @coderabbitai ignore --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
58 lines
1.8 KiB
Python
58 lines
1.8 KiB
Python
"""Per-key re-entrant locks (in-process).
|
|
|
|
Route handlers now run in FastAPI's threadpool (plain ``def``), so two requests
|
|
for the same job/user on one instance can interleave inside non-transactional
|
|
check-then-write code (e.g. a double-clicked Cancel refunding twice). When those
|
|
handlers were never-awaiting ``async def``s they ran start-to-finish on the
|
|
event loop, which serialized them per instance; these locks restore exactly
|
|
that per-instance guarantee. They do NOT coordinate across Cloud Run instances
|
|
(nothing did before either) — that would need Firestore transactions.
|
|
|
|
Usage::
|
|
|
|
with job_lock(job_id):
|
|
...check status, then write...
|
|
"""
|
|
from __future__ import annotations
|
|
|
|
import threading
|
|
from contextlib import contextmanager
|
|
from typing import Dict, Iterator
|
|
|
|
|
|
class KeyedLocks:
|
|
def __init__(self) -> None:
|
|
self._guard = threading.Lock()
|
|
self._locks: Dict[str, threading.RLock] = {}
|
|
self._refs: Dict[str, int] = {}
|
|
|
|
@contextmanager
|
|
def hold(self, key: str) -> Iterator[None]:
|
|
with self._guard:
|
|
lock = self._locks.get(key)
|
|
if lock is None:
|
|
lock = self._locks[key] = threading.RLock()
|
|
self._refs[key] = self._refs.get(key, 0) + 1
|
|
try:
|
|
with lock:
|
|
yield
|
|
finally:
|
|
with self._guard:
|
|
self._refs[key] -= 1
|
|
if self._refs[key] == 0:
|
|
# Nobody holds or waits on it — drop it so the map stays small.
|
|
del self._refs[key]
|
|
del self._locks[key]
|
|
|
|
|
|
_job_locks = KeyedLocks()
|
|
_user_locks = KeyedLocks()
|
|
|
|
|
|
def job_lock(job_id: str):
|
|
return _job_locks.hold(job_id or "")
|
|
|
|
|
|
def user_lock(email: str):
|
|
return _user_locks.hold((email or "").lower())
|