mirror of
https://github.com/nomadkaraoke/karaoke-gen.git
synced 2026-11-20 13:07:57 +09:00
## Summary The "Reconnecting" pill and the "servers unavailable" banner (164 / 49 events, 2026-09-19 → 10-02) were mostly caused by the single API instance's **event loop freezing for 20–45s**. Every request on the instance, `/api/health` included, completed at the same moment. The full evidence and plan are in `docs/archive/2026-10-03-backend-loop-freezes-telemetry-plan.md`. ### What froze the loop, and the fix for each - **Screens worker:** it ran as a post-response `async def` BackgroundTask doing 4K PIL renders, GCS transfers and ffmpeg transcodes on the loop. Under `--cpu-throttling` it was also CPU-starved. **Fix:** it now runs inline in its Cloud Tasks request (CPU stays allocated, failures still return 200 so Cloud Tasks doesn't double-run it), with the heavy steps in `asyncio.to_thread`. Also offloaded: style load, timing signals, pre-apply. - **Matplotlib font cache:** it was rebuilt on every new instance (17/28 episodes). **Fix:** it's pre-built in `Dockerfile.base` (`MPLCONFIGDIR=/opt/mplconfig`), and pyplot is imported lazily. - **Hourly stale-review cron:** an `async def` with zero awaits doing ≤1500 reads and SMTP sends. **Fix:** it runs in a thread. - **166 `async def` routes that never awaited** (sync Firestore/GCS/email/Gemini on the loop, e.g. magic-link verify). **Fix:** they're now plain `def`, so they run in FastAPI's threadpool. The new AST guard `test_no_blocking_async_routes.py` fails CI on new ones. It also flags sync routes that call `asyncio.create_task`/`get_running_loop`; that caught `confirm_duration` and `trigger_youtube_queue_processing`, which stay async. ### Safety net - `backend/services/loop_watchdog.py`: a heartbeat plus a daemon thread. Any stall over 1s logs `EVENT_LOOP_STALL` with the loop thread's stack and a `culprit=file:line`. Stalls of 5s or more are written to `client_events` as `server_loop_stall`. - Pulumi: the log metric `backend/event_loop_stalls` plus an alert on ≥2 stalls of ≥10s in 30 min. **This still needs a local `pulumi up`.** Pulumi is never applied by CI, and the admin gcloud login has expired. ### Telemetry gaps fixed - **Identity:** `user_email` used to be hard-coded to null. The frontend now sends its session token, and the server resolves `user_email`, `is_admin`, `is_internal` and `is_test`. A client-sent email is ignored. - **New fields:** `device_fingerprint` (the existing FingerprintJS visitorId), `tab_id`, `episode_id`, and `tenant`. - **New event:** `banner_recovered`, carrying `duration_ms` and `peak_status`. - **Report:** `scripts/client_events_report.py` summarises the data, excluding admin/internal/test accounts by default. ## Testing - `make test` passes: backend 5529, plus the emulator suite, karaoke_gen unit tests, and frontend Jest 1623. - New tests: - Watchdog: a deterministic state machine plus detection of a real blocking call. - Event-loop responsiveness: a ticker keeps ticking while the heavy steps run. - Screens inline and failure semantics. - `client_events` identity and fields. - The AST guard. - The report summary. - Frontend reporter and banner episode tests. - Local CodeRabbit CLI review: 0 findings. 🤖 Generated with [Claude Code](https://claude.com/claude-code) @coderabbitai ignore --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
69 lines
2.1 KiB
Python
69 lines
2.1 KiB
Python
"""Per-key locks restore per-instance serialization for threadpool route handlers."""
|
|
import threading
|
|
import time
|
|
from unittest.mock import MagicMock
|
|
|
|
from backend.utils.keyed_lock import KeyedLocks
|
|
|
|
|
|
def test_same_key_serializes_different_keys_dont():
|
|
locks = KeyedLocks()
|
|
order = []
|
|
|
|
def worker(key, tag):
|
|
with locks.hold(key):
|
|
order.append(f"{tag}-in")
|
|
time.sleep(0.1)
|
|
order.append(f"{tag}-out")
|
|
|
|
t1 = threading.Thread(target=worker, args=("job1", "a"))
|
|
t2 = threading.Thread(target=worker, args=("job1", "b"))
|
|
t1.start(); time.sleep(0.02); t2.start()
|
|
t1.join(); t2.join()
|
|
assert order == ["a-in", "a-out", "b-in", "b-out"]
|
|
|
|
start = time.monotonic()
|
|
t3 = threading.Thread(target=worker, args=("x", "c"))
|
|
t4 = threading.Thread(target=worker, args=("y", "d"))
|
|
t3.start(); t4.start(); t3.join(); t4.join()
|
|
assert time.monotonic() - start < 0.18 # ran in parallel
|
|
|
|
|
|
def test_reentrant_and_cleans_up():
|
|
locks = KeyedLocks()
|
|
with locks.hold("k"):
|
|
with locks.hold("k"):
|
|
pass
|
|
assert locks._locks == {} and locks._refs == {}
|
|
|
|
|
|
def test_concurrent_cancel_refunds_once():
|
|
"""Two threads cancelling the same job: the second must see CANCELLED and no-op."""
|
|
from backend.models.job import JobStatus
|
|
from backend.services import job_manager as jm_mod
|
|
|
|
state = {"status": JobStatus.AWAITING_REVIEW}
|
|
jm = jm_mod.JobManager.__new__(jm_mod.JobManager)
|
|
|
|
def get_job(_id):
|
|
time.sleep(0.05) # widen the check-then-write window
|
|
job = MagicMock()
|
|
job.status = state["status"]
|
|
return job
|
|
|
|
def update_job_status(job_id, status, message):
|
|
state["status"] = status
|
|
|
|
jm.get_job = get_job
|
|
jm.update_job_status = update_job_status
|
|
jm._refund_credit_for_job = MagicMock()
|
|
|
|
results = []
|
|
threads = [threading.Thread(target=lambda: results.append(jm.cancel_job("j1"))) for _ in range(2)]
|
|
for t in threads:
|
|
t.start()
|
|
for t in threads:
|
|
t.join()
|
|
assert sorted(results) == [False, True]
|
|
jm._refund_credit_for_job.assert_called_once()
|