Files
Andrew Beveridge ☄️andClaude Opus 5.5 4e43449559 fix(backend): stop API event-loop freezes + identify users in degradation telemetry (v0.264.0) (#1120)
## Summary
The "Reconnecting" pill and the "servers unavailable" banner (164 / 49
events, 2026-09-19 → 10-02) were mostly caused by the single API
instance's **event loop freezing for 20–45s**. Every request on the
instance, `/api/health` included, completed at the same moment. The full
evidence and plan are in
`docs/archive/2026-10-03-backend-loop-freezes-telemetry-plan.md`.

### What froze the loop, and the fix for each
- **Screens worker:** it ran as a post-response `async def`
BackgroundTask doing 4K PIL renders, GCS transfers and ffmpeg transcodes
on the loop. Under `--cpu-throttling` it was also CPU-starved. **Fix:**
it now runs inline in its Cloud Tasks request (CPU stays allocated,
failures still return 200 so Cloud Tasks doesn't double-run it), with
the heavy steps in `asyncio.to_thread`. Also offloaded: style load,
timing signals, pre-apply.
- **Matplotlib font cache:** it was rebuilt on every new instance (17/28
episodes). **Fix:** it's pre-built in `Dockerfile.base`
(`MPLCONFIGDIR=/opt/mplconfig`), and pyplot is imported lazily.
- **Hourly stale-review cron:** an `async def` with zero awaits doing
≤1500 reads and SMTP sends. **Fix:** it runs in a thread.
- **166 `async def` routes that never awaited** (sync
Firestore/GCS/email/Gemini on the loop, e.g. magic-link verify).
**Fix:** they're now plain `def`, so they run in FastAPI's threadpool.
The new AST guard `test_no_blocking_async_routes.py` fails CI on new
ones. It also flags sync routes that call
`asyncio.create_task`/`get_running_loop`; that caught `confirm_duration`
and `trigger_youtube_queue_processing`, which stay async.

### Safety net
- `backend/services/loop_watchdog.py`: a heartbeat plus a daemon thread.
Any stall over 1s logs `EVENT_LOOP_STALL` with the loop thread's stack
and a `culprit=file:line`. Stalls of 5s or more are written to
`client_events` as `server_loop_stall`.
- Pulumi: the log metric `backend/event_loop_stalls` plus an alert on ≥2
stalls of ≥10s in 30 min. **This still needs a local `pulumi up`.**
Pulumi is never applied by CI, and the admin gcloud login has expired.

### Telemetry gaps fixed
- **Identity:** `user_email` used to be hard-coded to null. The frontend
now sends its session token, and the server resolves `user_email`,
`is_admin`, `is_internal` and `is_test`. A client-sent email is ignored.
- **New fields:** `device_fingerprint` (the existing FingerprintJS
visitorId), `tab_id`, `episode_id`, and `tenant`.
- **New event:** `banner_recovered`, carrying `duration_ms` and
`peak_status`.
- **Report:** `scripts/client_events_report.py` summarises the data,
excluding admin/internal/test accounts by default.

## Testing
- `make test` passes: backend 5529, plus the emulator suite, karaoke_gen
unit tests, and frontend Jest 1623.
- New tests:
- Watchdog: a deterministic state machine plus detection of a real
blocking call.
- Event-loop responsiveness: a ticker keeps ticking while the heavy
steps run.
  - Screens inline and failure semantics.
  - `client_events` identity and fields.
  - The AST guard.
  - The report summary.
  - Frontend reporter and banner episode tests.
- Local CodeRabbit CLI review: 0 findings.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

@coderabbitai ignore

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-03 14:17:51 -04:00

69 lines
2.1 KiB
Python

"""Per-key locks restore per-instance serialization for threadpool route handlers."""
import threading
import time
from unittest.mock import MagicMock
from backend.utils.keyed_lock import KeyedLocks
def test_same_key_serializes_different_keys_dont():
locks = KeyedLocks()
order = []
def worker(key, tag):
with locks.hold(key):
order.append(f"{tag}-in")
time.sleep(0.1)
order.append(f"{tag}-out")
t1 = threading.Thread(target=worker, args=("job1", "a"))
t2 = threading.Thread(target=worker, args=("job1", "b"))
t1.start(); time.sleep(0.02); t2.start()
t1.join(); t2.join()
assert order == ["a-in", "a-out", "b-in", "b-out"]
start = time.monotonic()
t3 = threading.Thread(target=worker, args=("x", "c"))
t4 = threading.Thread(target=worker, args=("y", "d"))
t3.start(); t4.start(); t3.join(); t4.join()
assert time.monotonic() - start < 0.18 # ran in parallel
def test_reentrant_and_cleans_up():
locks = KeyedLocks()
with locks.hold("k"):
with locks.hold("k"):
pass
assert locks._locks == {} and locks._refs == {}
def test_concurrent_cancel_refunds_once():
"""Two threads cancelling the same job: the second must see CANCELLED and no-op."""
from backend.models.job import JobStatus
from backend.services import job_manager as jm_mod
state = {"status": JobStatus.AWAITING_REVIEW}
jm = jm_mod.JobManager.__new__(jm_mod.JobManager)
def get_job(_id):
time.sleep(0.05) # widen the check-then-write window
job = MagicMock()
job.status = state["status"]
return job
def update_job_status(job_id, status, message):
state["status"] = status
jm.get_job = get_job
jm.update_job_status = update_job_status
jm._refund_credit_for_job = MagicMock()
results = []
threads = [threading.Thread(target=lambda: results.append(jm.cancel_job("j1"))) for _ in range(2)]
for t in threads:
t.start()
for t in threads:
t.join()
assert sorted(results) == [False, True]
jm._refund_credit_for_job.assert_called_once()