Files
Andrew Beveridge ☄️andClaude Opus 5.5 0d817509fa fix(api): event-loop stalls — visibility redistribution off-loop, route warmup, watchdog/Cloud Trace noise (0.286.2) (#1230)
## Summary

Fixes the 2026-10-29 #alerts event-loop stall batch (handoff
`docs/archive/2026-10-29-gen-event-loop-stalls-handoff.md` in the
workspace).

**1. Visibility change off the loop (39s stall, 13 requests incl.
/api/health)**
- `change_to_private` now runs claim, flag write, `redistribute_video`,
public-output delete and the final commit as **one** `asyncio.to_thread`
call (`_redistribute_and_commit_private`). `redistribute_video` runs on
that thread's own loop through the new `redistribute_video_off_loop`
(`asyncio.run(with_notification_drain(...))`).
- Is anything bound to the main loop? No. `redistribute_video` builds
its own `JobManager`/`StorageService`. The Dropbox, YouTube and GDrive
clients are sync. Job-log writes go through `log_write_offload`'s writer
thread. Notification tasks scheduled via `_schedule_notification` would
land on the private loop, so `with_notification_drain` drains them
before the loop closes, as the Cloud Run Job entry points do.
- Why one thread call and not just the redistribution (from CodeRabbit):
a thread can't be cancelled. So a cancelled request can no longer leave
a committed private archive with its public outputs still live, or a
claimed flag with no redistribution.
- Guard: `redistribute_video` is added to `BLOCKING_CALLS`, and
`visibility_change_service.change_to_private`/`change_to_public` are
added to `AWAITED_ROUTE_HELPERS`. I checked that putting back `await
redistribute_video(...)` fails the test.
- **Double Dropbox upload confirmed and fixed.** GCS `finals/` holds
every output twice: short canonical keys written by `_upload_results`
(`lossy_4k_mp4.mp4`, …) and the readable encoder names (verified on job
0f211e97: 17 objects / 525 MiB). Redistribution uploaded both. The
short-named `lossless_4k_mp4.mp4` also slipped past the `" (Final
Karaoke Lossless 4k).mp4"` Dropbox exclusion.
`_dedupe_redistribution_files` now keeps one copy per final, under its
readable name:
- a short name whose readable twin exists (twin found by suffix) is
dropped;
- a short name with no twin is renamed to the readable name (CDG/TXT
zips included).

**2. First-request stalls on new instances**
- Lifespan builds FastAPI's lazy route contexts
(`fastapi.routing.iter_route_contexts` +
`effective_low_priority_routes`) in a thread before readiness, with a
timeout and best-effort error handling. The results are cached by routes
version under a lock, so the OTel `_get_route_details` walk no longer
compiles route regexes on the loop. It is a no-op on FastAPI versions
without these internals. Note: the lock pins 0.128 but prod installs
unpinned (`pip install -e .`), so prod runs a newer FastAPI. I checked
against FastAPI 0.162: the first walk builds 304 contexts, the second
walk and the OTel lookup are cached (<1ms).
- `flacfetch.core.manager`, `flacfetch.downloaders.youtube` and `yt_dlp`
are added to `_LAZY_ROUTE_IMPORTS`.
- `search_audio` auto_download: `select_best` now goes through
`asyncio.to_thread`, with `select_best` added to `BLOCKING_CALLS`.

**3. Loop watchdog false positive**
- `_finish` logs WARNING, not ERROR, when all of these hold: no non-idle
sample was taken, a process freeze was observed (`frozen_s > 0`), and no
request was in flight when the stall began. Every "affected" request
therefore arrived while the instance was thawing. An unsampled stall
with no observed freeze, or with a request in flight at its start, stays
ERROR. The freeze requirement came from CodeRabbit.

**4. Cloud Trace exporter noise**
- `cloud_trace_export` ignore pattern for `Error while writing to Cloud
Trace`. CodeRabbit suggested matching only the transient gRPC detail. I
didn't: the error monitor only sees `jsonPayload.message` (the exception
is a separate field), and a persistent loss of telemetry isn't
#alerts-worthy anyway.

Smaller change: `change_to_private`'s Firestore `job_ref.update` writes
now also run off the loop.

## Testing
- New `backend/tests/test_redistribute_video.py`:
  - dedupe unit tests;
- an end-to-end `redistribute_video` run with mocks, checking the files
the Dropbox stage sees;
  - the off-loop runner uses its calling thread;
- `change_to_private` runs the redistribution on another thread and
loop;
  - a cancelled request still deletes the public outputs.
- `test_loop_watchdog.py`: frozen + unsampled + late arrival gives
WARNING; a request in flight at the start gives ERROR; no observed
freeze gives ERROR; a sampled stall with a late arrival gives ERROR.
- `test_main_startup_warmup.py`: lazy imports list; the route-context
walk; no-op without the API; failure is non-fatal; lifespan warms
off-loop before readiness.
- `test_known_issues.py`: the Cloud Trace pattern matches, and other
trace errors are not ignored.
- Local runs: `pytest backend/tests/ tests/unit/services/error_monitor
--ignore=backend/tests/emulator`: 7369 passed, 29 skipped. Not counted
in that total: `test_emulator_integration.py` fails 8/11 the same way on
the base commit, because a stray emulator from another session is on
:8080 locally. That's environmental, not caused by this PR.
- Local CodeRabbit: 3 cycles. Cycle 1: 3 findings (2 fixed, 1 skipped
with the reason above). Cycle 2: 1 finding, fixed. Cycle 3: 0 findings,
but CodeRabbit reported it reused cached results rather than a fresh
review.

## Not verified
- No prod run of a real public→private change yet. After deploy, the
Dropbox folder of the next one should contain each final exactly once,
and no `EVENT_LOOP_STALL` should appear during it.
- Route-context warmup was checked only against FastAPI 0.162 in a
scratch venv, not the exact prod version. Watch for the `Warmed N route
contexts` log line on the next cold start.
- Existing private Dropbox folders that already hold the duplicates are
not cleaned up by this PR.

Alert:
https://discord.com/channels/1198691101943541770/1265517528608669707/1563719390849929330
Alert:
https://discord.com/channels/1198691101943541770/1265517528608669707/1563009782363721760
Alert:
https://discord.com/channels/1198691101943541770/1265517528608669707/1563039889354784882
Alert:
https://discord.com/channels/1198691101943541770/1265517528608669707/1563221363278348514
Alert:
https://discord.com/channels/1198691101943541770/1265517528608669707/1564308425033252895
Alert:
https://discord.com/channels/1198691101943541770/1265517528608669707/1563160849336762449

@coderabbitai ignore

🤖 Generated with [Claude Code](https://claude.com/claude-code)

https://claude.ai/code/session_012RYLVTiUPKxn3sQPyp3Teh

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-29 02:49:20 -04:00

466 lines
20 KiB
Python

"""Event-loop stall watchdog.
The API runs ONE uvicorn process with ONE asyncio event loop per Cloud Run
instance. Any synchronous work executed on that loop (sync Firestore/GCS/HTTP
calls, PIL/ffmpeg, a first ``import matplotlib.pyplot``…) freezes *every*
request on the instance — including ``/api/health`` — which users see as the
"Reconnecting" pill / "temporarily unavailable" banner. (Analysis 2026-10-03:
docs/archive/2026-10-03-backend-loop-freezes-telemetry-plan.md.)
This module makes such freezes self-diagnosing:
* an asyncio heartbeat task stamps ``last_tick`` every ``tick_s``;
* a daemon thread notices when the stamp goes stale (> ``stall_threshold_s``)
and captures the *event-loop thread's* Python stack via
``sys._current_frames()`` — i.e. the exact line that is blocking — resampling
at ``sample_at_s`` so a long freeze shows where it actually spent its time
(``EVENT_LOOP_STALL in progress`` lines, always WARNING);
* when the loop resumes it logs ``EVENT_LOOP_STALL ended duration_ms=…``
(WARNING, or ERROR at ``error_threshold_s`` — the alerting log metric keys on
that), and for stalls ≥ ``record_threshold_s`` hands a summary to
``recorder`` (Firestore ``client_events`` type ``server_loop_stall`` in prod),
so server-side cause and user-visible impact live in one collection.
The thread only reads frames and a float; it never touches the loop, so it is
safe to run alongside it. Recording happens on the watchdog thread (sync
Firestore is fine there).
"""
from __future__ import annotations
import asyncio
import logging
import os
import sys
import threading
import time
import traceback
from datetime import datetime, timezone
from typing import Any, Callable, Optional
logger = logging.getLogger(__name__)
# Frames from these path fragments are "ours" — the innermost one is reported
# as the culprit (the stdlib/library frame below it is usually just the I/O).
_OWN_CODE_MARKERS = ("/backend/", "/karaoke_gen/")
_MAX_STACK_FRAMES = 30
_MAX_STACK_CHARS = 8000
Recorder = Callable[[dict], None]
class RequestTracker:
"""Counts HTTP requests so the watchdog can tell whether a stall hurt anyone.
Mutated only on the event-loop thread (by ``RequestTrackingMiddleware``) and
read from the watchdog thread; plain ints are safe to read under the GIL.
Why (2026-10-12/13 flapping alert): Cloud Run autoscaling starts extra
instances that serve one request and then sit idle, CPU-throttled, while the
SpaCy/NLTK warmup thread trickles along for 10+ minutes. Those idle instances
produced 10-40s "stalls" (empty culprit — the freeze consumed the sampling
schedule) with no request on them, tripping the >=2-in-30min alert. A stall
with no request in flight and none queued behind it affects nobody.
"""
def __init__(self) -> None:
self.in_flight = 0
self.started = 0
def begin(self) -> None:
self.in_flight += 1
self.started += 1
def end(self) -> None:
self.in_flight = max(0, self.in_flight - 1)
# Process-wide tracker: the middleware is attached at import time, the watchdog
# is created later in the lifespan — both share this instance.
request_tracker = RequestTracker()
class RequestTrackingMiddleware:
"""Pure-ASGI middleware feeding ``request_tracker`` (no per-request allocation
beyond the wrapper call; outermost so even rejected requests count)."""
def __init__(self, app, tracker: Optional[RequestTracker] = None) -> None:
self.app = app
self.tracker = tracker or request_tracker
async def __call__(self, scope, receive, send):
if scope.get("type") != "http":
return await self.app(scope, receive, send)
self.tracker.begin()
try:
return await self.app(scope, receive, send)
finally:
self.tracker.end()
# Innermost frames that mean "the loop is idle, waiting for I/O" — not blocked.
_IDLE_FUNCS = {"select", "poll", "epoll", "kqueue", "control", "_run_once"}
_IDLE_FILES = ("selectors.py", "base_events.py")
def _is_idle_frame(frame) -> bool:
"""True when the loop thread is parked in the selector (idle), not running code."""
if frame is None:
return False
code = frame.f_code
return code.co_name in _IDLE_FUNCS and code.co_filename.endswith(_IDLE_FILES)
def _format_stack(frame) -> tuple[str, str]:
"""Return (stack_text, culprit) for a frame; culprit = innermost own-code frame."""
if frame is None:
return "", ""
summary = traceback.extract_stack(frame)[-_MAX_STACK_FRAMES:]
culprit = ""
for fs in reversed(summary):
if any(m in fs.filename for m in _OWN_CODE_MARKERS) and "loop_watchdog" not in fs.filename:
short = fs.filename.split("/app/")[-1]
culprit = f"{short}:{fs.lineno} in {fs.name}"
break
if not culprit and summary:
fs = summary[-1]
culprit = f"{fs.filename}:{fs.lineno} in {fs.name}"
text = "".join(traceback.format_list(summary))[-_MAX_STACK_CHARS:]
return text, culprit
class LoopWatchdog:
def __init__(
self,
*,
tick_s: float = 0.25,
stall_threshold_s: float = 1.0,
error_threshold_s: float = 10.0,
record_threshold_s: float = 5.0,
sample_at_s: tuple[float, ...] = (1.0, 5.0, 15.0),
recorder: Optional[Recorder] = None,
clock: Callable[[], float] = time.monotonic,
cpu_clock: Callable[[], float] = time.process_time,
frozen_gap_s: float = 0.75,
frozen_cpu_ratio: float = 0.25,
request_tracker: Optional[RequestTracker] = None,
request_grace_s: float = 0.5,
) -> None:
self.tick_s = tick_s
self.stall_threshold_s = stall_threshold_s
self.error_threshold_s = error_threshold_s
self.record_threshold_s = record_threshold_s
self.sample_at_s = tuple(sorted(sample_at_s))
self.recorder = recorder
self._clock = clock
self._cpu_clock = cpu_clock
# Cloud Run --cpu-throttling freezes the WHOLE process between requests —
# sometimes mid-callback, where the "loop parked in select()" check can't
# see it (2026-10-08: a 30s "stall" in asyncio's get_debug on an instance
# with no requests in flight). Signature: this watchdog thread itself
# oversleeps by >= frozen_gap_s while the process burns almost no CPU
# (a blocking I/O call lets this thread wake on time; GIL-hogging CPU
# work burns CPU). That frozen time is discounted from the stall.
# 0.75s = 3 missed ticks: measured wake gaps stay <= 0.27s under sync I/O,
# pure-Python CPU and C-extension CPU on the loop, while partial throttling
# gives short 1-2.5s freezes (126 of 128 asyncio-internal "stalls" on
# 2026-10-08 had no request waiting on them).
self.frozen_gap_s = frozen_gap_s
self.frozen_cpu_ratio = frozen_cpu_ratio
self._last_check: Optional[float] = None
self._last_cpu = 0.0
self._stall_frozen_s = 0.0
# A freeze can carry the stall past every sample_at_s point without a
# sample. If the loop is STILL blocked after the thaw, that is real
# blocking — take one sample then so the stall has a culprit (2026-10-12:
# 10-40s ERROR stalls all had an empty culprit).
self._sample_after_thaw = False
# With a tracker, a stall only escalates to ERROR (the alerting level) if
# a request was in flight when it began or arrived during it. Requests
# queued in the socket during the freeze are accepted right after the loop
# resumes, so the verdict waits ``request_grace_s`` after resume.
self.request_tracker = request_tracker
self.request_grace_s = request_grace_s
self._stall_in_flight = 0
self._stall_started_count = 0
self._resumed_at: Optional[float] = None
self._resumed_duration = 0.0
self._last_tick = clock()
self._loop_thread_id: Optional[int] = None
self._heartbeat_task: Optional[asyncio.Task] = None
self._thread: Optional[threading.Thread] = None
self._stop = threading.Event()
# Current-stall state (only touched by the watchdog thread).
self._in_stall = False
self._stall_started_tick = 0.0
self._samples: list[dict[str, Any]] = []
self._next_sample_idx = 0
# ----- lifecycle -------------------------------------------------------
def start(self) -> None:
"""Start heartbeat + watchdog thread. Must be called from inside the loop."""
if self._thread is not None:
return
self._loop_thread_id = threading.get_ident()
self._last_tick = self._clock()
self._heartbeat_task = asyncio.get_running_loop().create_task(self._heartbeat())
self._stop.clear()
self._thread = threading.Thread(target=self._run, name="loop-watchdog", daemon=True)
self._thread.start()
logger.info(
"Event-loop watchdog started (stall>=%.1fs logged, >=%.1fs recorded, >=%.1fs error)",
self.stall_threshold_s, self.record_threshold_s, self.error_threshold_s,
)
async def stop(self) -> None:
self._stop.set()
if self._heartbeat_task is not None:
self._heartbeat_task.cancel()
try:
await self._heartbeat_task
except (asyncio.CancelledError, Exception):
pass
self._heartbeat_task = None
self._thread = None
async def _heartbeat(self) -> None:
while True:
self._last_tick = self._clock()
await asyncio.sleep(self.tick_s)
def _run(self) -> None:
while not self._stop.wait(self.tick_s):
try:
self.check()
except Exception: # pragma: no cover — must never kill the thread
logger.exception("loop watchdog check failed")
# ----- detection (called from the watchdog thread; unit-testable) ------
def _sample(self, age: float) -> None:
frame = sys._current_frames().get(self._loop_thread_id) if self._loop_thread_id else None
if _is_idle_frame(frame):
# Loop is parked in select(): not blocked. This is what a CPU-throttled
# idle Cloud Run instance looks like when CPU returns (the watchdog
# thread can wake before the heartbeat does). Record it so _finish
# can drop an all-idle "stall" as a false positive.
self._samples.append({"at_s": round(age, 1), "culprit": "", "stack": "", "idle": True})
return
stack, culprit = _format_stack(frame)
if self._samples and self._samples[-1]["stack"] == stack:
return # identical to the previous sample — nothing new to learn
self._samples.append({"at_s": round(age, 1), "culprit": culprit, "stack": stack, "idle": False})
# Always WARNING. The "ended" line is the one ERROR per freeze (>=10s
# blocked with a request affected): it's what the backend/event_loop_stalls
# metric counts, and the error monitor turns each ERROR line shape into its
# own #alerts pattern, so an ERROR here too meant two alerts per freeze.
level = logging.WARNING
logger.log(
level,
"EVENT_LOOP_STALL in progress age_ms=%d culprit=%s\n%s",
int(age * 1000), culprit, stack,
extra={"loop_stall_age_ms": int(age * 1000), "loop_stall_culprit": culprit},
)
def _frozen_gap(self, now: float) -> float:
"""Seconds since the previous check during which the whole process was frozen."""
cpu = self._cpu_clock()
prev_check, prev_cpu = self._last_check, self._last_cpu
self._last_check, self._last_cpu = now, cpu
if prev_check is None:
return 0.0
gap = now - prev_check
if gap >= self.frozen_gap_s and (cpu - prev_cpu) < gap * self.frozen_cpu_ratio:
return gap
return 0.0
def check(self, now: Optional[float] = None) -> None:
now = self._clock() if now is None else now
frozen = self._frozen_gap(now)
last = self._last_tick
age = now - last
if not self._in_stall:
if age >= self.stall_threshold_s:
self._in_stall = True
self._stall_started_tick = last
self._samples = []
self._next_sample_idx = 0
self._stall_frozen_s = 0.0
self._sample_after_thaw = False
self._resumed_at = None
if self.request_tracker is not None:
self._stall_in_flight = self.request_tracker.in_flight
self._stall_started_count = self.request_tracker.started
else:
return
if self._resumed_at is not None:
# Loop already resumed; waiting out the grace window for queued requests.
if self._requests_affected() or now - self._resumed_at >= self.request_grace_s:
self._finish(self._resumed_duration)
return
# Only the part of the frozen gap after the last heartbeat belongs to this stall.
self._stall_frozen_s += min(frozen, age)
# In a stall: has the heartbeat resumed?
if last > self._stall_started_tick:
# The heartbeat that ended the stall fired one tick_s after the
# previous one would have, so subtract the nominal sleep.
duration = max(0.0, last - self._stall_started_tick - self.tick_s)
if self.request_tracker is not None and not self._requests_affected():
self._resumed_at = now
self._resumed_duration = duration
return
self._finish(duration)
return
if frozen:
# Just thawed: the loop's stack shows where the freeze caught it, not
# what blocked it. Skip the samples due during the freeze.
skipped = False
while (
self._next_sample_idx < len(self.sample_at_s)
and age >= self.sample_at_s[self._next_sample_idx]
):
self._next_sample_idx += 1
skipped = True
if skipped:
self._sample_after_thaw = True
return
if self._sample_after_thaw:
# First on-time check after a freeze and the loop still hasn't ticked.
self._sample_after_thaw = False
self._sample(age)
while (
self._next_sample_idx < len(self.sample_at_s)
and age >= self.sample_at_s[self._next_sample_idx]
):
self._next_sample_idx += 1
self._sample(age)
def _requests_affected(self) -> int:
"""Requests that waited on this stall: in flight at its start + started since."""
t = self.request_tracker
if t is None:
return 0
return self._stall_in_flight + max(0, t.started - self._stall_started_count)
def _finish(self, duration: float) -> None:
samples = self._samples
frozen_s = self._stall_frozen_s
affected = self._requests_affected()
tracked = self.request_tracker is not None
in_flight_at_start = self._stall_in_flight
self._resumed_at = None
self._in_stall = False
self._samples = []
self._stall_frozen_s = 0.0
if frozen_s:
logger.debug(
"loop watchdog: discounted %dms of process freeze (CPU throttling) from a %dms gap",
int(frozen_s * 1000), int(duration * 1000),
)
duration = max(0.0, duration - frozen_s)
if duration < self.stall_threshold_s:
return
if samples and all(s.get("idle") for s in samples):
# Every sample saw the loop idle in select() — a throttled/idle gap
# (no CPU allocated between requests), not a blocking call.
logger.debug("loop watchdog: ignored idle gap of %dms", int(duration * 1000))
return
samples = [s for s in samples if not s.get("idle")]
culprits = [s["culprit"] for s in samples if s["culprit"]]
# The longest-lived sample is the most representative of the freeze.
main_culprit = culprits[-1] if culprits else ""
# No request waited on it (idle, typically CPU-throttled instance) → never
# ERROR: the alert counts ERROR lines and nobody saw this freeze.
no_impact = tracked and affected == 0
# A process freeze (CPU throttling of an idle instance) was observed and
# swallowed the whole sampling schedule, so the stall was never seen
# blocking in our code. If nothing was in flight when it began, every
# "affected" request merely arrived while the instance was thawing
# (2026-10-23: 20.7s, empty culprit, its one request arrived 0.3s before
# the end). Not actionable. Without an observed freeze an unsampled stall
# can be real blocking (a GIL hog that starved this thread until the loop
# had already resumed), so that stays eligible for ERROR.
unattributed_late_only = (
tracked and not samples and frozen_s > 0 and in_flight_at_start == 0
)
level = (
logging.ERROR
if duration >= self.error_threshold_s and not no_impact and not unattributed_late_only
else logging.WARNING
)
logger.log(
level,
"EVENT_LOOP_STALL ended duration_ms=%d culprit=%s requests_affected=%s",
int(duration * 1000), main_culprit, affected if tracked else "unknown",
extra={
"loop_stall_duration_ms": int(duration * 1000),
"loop_stall_culprit": main_culprit,
"loop_stall_culprits": culprits,
"loop_stall_requests_affected": affected if tracked else None,
},
)
if self.recorder is not None and duration >= self.record_threshold_s:
try:
self.recorder({
"duration_ms": int(duration * 1000),
"culprit": main_culprit,
"culprits": culprits,
"stack": samples[-1]["stack"] if samples else "",
"requests_affected": affected if tracked else None,
})
except Exception:
logger.warning("Failed to record loop stall (non-fatal)", exc_info=True)
# ----- production recorder ---------------------------------------------------
_instance_id: Optional[str] = None
def _cloud_run_instance_id() -> str:
"""Best-effort Cloud Run instance id from the metadata server (cached)."""
global _instance_id
if _instance_id is not None:
return _instance_id
_instance_id = ""
try:
import requests
resp = requests.get(
"http://metadata.google.internal/computeMetadata/v1/instance/id",
headers={"Metadata-Flavor": "Google"},
timeout=1,
)
if resp.ok:
_instance_id = resp.text.strip()
except Exception:
pass
return _instance_id
def firestore_stall_recorder(stall: dict) -> None:
"""Persist a stall as a ``client_events`` doc (type ``server_loop_stall``)."""
from google.cloud import firestore # type: ignore[import]
from backend.version import VERSION
doc = {
"type": "server_loop_stall",
"source": "server",
"created_at": datetime.now(timezone.utc),
"release": VERSION,
"revision": os.getenv("K_REVISION", ""),
"instance_id": _cloud_run_instance_id(),
"detail": {
"duration_ms": stall["duration_ms"],
"culprit": stall["culprit"][:300],
"requests_affected": stall.get("requests_affected"),
},
"culprits": [c[:300] for c in stall.get("culprits", [])][:5],
"stack": stall.get("stack", "")[:_MAX_STACK_CHARS],
}
firestore.Client(project="nomadkaraoke").collection("client_events").add(doc)