mirror of
https://github.com/nomadkaraoke/karaoke-gen.git
synced 2026-11-20 13:07:57 +09:00
## Summary Fixes the 2026-10-29 #alerts event-loop stall batch (handoff `docs/archive/2026-10-29-gen-event-loop-stalls-handoff.md` in the workspace). **1. Visibility change off the loop (39s stall, 13 requests incl. /api/health)** - `change_to_private` now runs claim, flag write, `redistribute_video`, public-output delete and the final commit as **one** `asyncio.to_thread` call (`_redistribute_and_commit_private`). `redistribute_video` runs on that thread's own loop through the new `redistribute_video_off_loop` (`asyncio.run(with_notification_drain(...))`). - Is anything bound to the main loop? No. `redistribute_video` builds its own `JobManager`/`StorageService`. The Dropbox, YouTube and GDrive clients are sync. Job-log writes go through `log_write_offload`'s writer thread. Notification tasks scheduled via `_schedule_notification` would land on the private loop, so `with_notification_drain` drains them before the loop closes, as the Cloud Run Job entry points do. - Why one thread call and not just the redistribution (from CodeRabbit): a thread can't be cancelled. So a cancelled request can no longer leave a committed private archive with its public outputs still live, or a claimed flag with no redistribution. - Guard: `redistribute_video` is added to `BLOCKING_CALLS`, and `visibility_change_service.change_to_private`/`change_to_public` are added to `AWAITED_ROUTE_HELPERS`. I checked that putting back `await redistribute_video(...)` fails the test. - **Double Dropbox upload confirmed and fixed.** GCS `finals/` holds every output twice: short canonical keys written by `_upload_results` (`lossy_4k_mp4.mp4`, …) and the readable encoder names (verified on job 0f211e97: 17 objects / 525 MiB). Redistribution uploaded both. The short-named `lossless_4k_mp4.mp4` also slipped past the `" (Final Karaoke Lossless 4k).mp4"` Dropbox exclusion. `_dedupe_redistribution_files` now keeps one copy per final, under its readable name: - a short name whose readable twin exists (twin found by suffix) is dropped; - a short name with no twin is renamed to the readable name (CDG/TXT zips included). **2. First-request stalls on new instances** - Lifespan builds FastAPI's lazy route contexts (`fastapi.routing.iter_route_contexts` + `effective_low_priority_routes`) in a thread before readiness, with a timeout and best-effort error handling. The results are cached by routes version under a lock, so the OTel `_get_route_details` walk no longer compiles route regexes on the loop. It is a no-op on FastAPI versions without these internals. Note: the lock pins 0.128 but prod installs unpinned (`pip install -e .`), so prod runs a newer FastAPI. I checked against FastAPI 0.162: the first walk builds 304 contexts, the second walk and the OTel lookup are cached (<1ms). - `flacfetch.core.manager`, `flacfetch.downloaders.youtube` and `yt_dlp` are added to `_LAZY_ROUTE_IMPORTS`. - `search_audio` auto_download: `select_best` now goes through `asyncio.to_thread`, with `select_best` added to `BLOCKING_CALLS`. **3. Loop watchdog false positive** - `_finish` logs WARNING, not ERROR, when all of these hold: no non-idle sample was taken, a process freeze was observed (`frozen_s > 0`), and no request was in flight when the stall began. Every "affected" request therefore arrived while the instance was thawing. An unsampled stall with no observed freeze, or with a request in flight at its start, stays ERROR. The freeze requirement came from CodeRabbit. **4. Cloud Trace exporter noise** - `cloud_trace_export` ignore pattern for `Error while writing to Cloud Trace`. CodeRabbit suggested matching only the transient gRPC detail. I didn't: the error monitor only sees `jsonPayload.message` (the exception is a separate field), and a persistent loss of telemetry isn't #alerts-worthy anyway. Smaller change: `change_to_private`'s Firestore `job_ref.update` writes now also run off the loop. ## Testing - New `backend/tests/test_redistribute_video.py`: - dedupe unit tests; - an end-to-end `redistribute_video` run with mocks, checking the files the Dropbox stage sees; - the off-loop runner uses its calling thread; - `change_to_private` runs the redistribution on another thread and loop; - a cancelled request still deletes the public outputs. - `test_loop_watchdog.py`: frozen + unsampled + late arrival gives WARNING; a request in flight at the start gives ERROR; no observed freeze gives ERROR; a sampled stall with a late arrival gives ERROR. - `test_main_startup_warmup.py`: lazy imports list; the route-context walk; no-op without the API; failure is non-fatal; lifespan warms off-loop before readiness. - `test_known_issues.py`: the Cloud Trace pattern matches, and other trace errors are not ignored. - Local runs: `pytest backend/tests/ tests/unit/services/error_monitor --ignore=backend/tests/emulator`: 7369 passed, 29 skipped. Not counted in that total: `test_emulator_integration.py` fails 8/11 the same way on the base commit, because a stray emulator from another session is on :8080 locally. That's environmental, not caused by this PR. - Local CodeRabbit: 3 cycles. Cycle 1: 3 findings (2 fixed, 1 skipped with the reason above). Cycle 2: 1 finding, fixed. Cycle 3: 0 findings, but CodeRabbit reported it reused cached results rather than a fresh review. ## Not verified - No prod run of a real public→private change yet. After deploy, the Dropbox folder of the next one should contain each final exactly once, and no `EVENT_LOOP_STALL` should appear during it. - Route-context warmup was checked only against FastAPI 0.162 in a scratch venv, not the exact prod version. Watch for the `Warmed N route contexts` log line on the next cold start. - Existing private Dropbox folders that already hold the duplicates are not cleaned up by this PR. Alert: https://discord.com/channels/1198691101943541770/1265517528608669707/1563719390849929330 Alert: https://discord.com/channels/1198691101943541770/1265517528608669707/1563009782363721760 Alert: https://discord.com/channels/1198691101943541770/1265517528608669707/1563039889354784882 Alert: https://discord.com/channels/1198691101943541770/1265517528608669707/1563221363278348514 Alert: https://discord.com/channels/1198691101943541770/1265517528608669707/1564308425033252895 Alert: https://discord.com/channels/1198691101943541770/1265517528608669707/1563160849336762449 @coderabbitai ignore 🤖 Generated with [Claude Code](https://claude.com/claude-code) https://claude.ai/code/session_012RYLVTiUPKxn3sQPyp3Teh --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
466 lines
20 KiB
Python
466 lines
20 KiB
Python
"""Event-loop stall watchdog.
|
|
|
|
The API runs ONE uvicorn process with ONE asyncio event loop per Cloud Run
|
|
instance. Any synchronous work executed on that loop (sync Firestore/GCS/HTTP
|
|
calls, PIL/ffmpeg, a first ``import matplotlib.pyplot``…) freezes *every*
|
|
request on the instance — including ``/api/health`` — which users see as the
|
|
"Reconnecting" pill / "temporarily unavailable" banner. (Analysis 2026-10-03:
|
|
docs/archive/2026-10-03-backend-loop-freezes-telemetry-plan.md.)
|
|
|
|
This module makes such freezes self-diagnosing:
|
|
|
|
* an asyncio heartbeat task stamps ``last_tick`` every ``tick_s``;
|
|
* a daemon thread notices when the stamp goes stale (> ``stall_threshold_s``)
|
|
and captures the *event-loop thread's* Python stack via
|
|
``sys._current_frames()`` — i.e. the exact line that is blocking — resampling
|
|
at ``sample_at_s`` so a long freeze shows where it actually spent its time
|
|
(``EVENT_LOOP_STALL in progress`` lines, always WARNING);
|
|
* when the loop resumes it logs ``EVENT_LOOP_STALL ended duration_ms=…``
|
|
(WARNING, or ERROR at ``error_threshold_s`` — the alerting log metric keys on
|
|
that), and for stalls ≥ ``record_threshold_s`` hands a summary to
|
|
``recorder`` (Firestore ``client_events`` type ``server_loop_stall`` in prod),
|
|
so server-side cause and user-visible impact live in one collection.
|
|
|
|
The thread only reads frames and a float; it never touches the loop, so it is
|
|
safe to run alongside it. Recording happens on the watchdog thread (sync
|
|
Firestore is fine there).
|
|
"""
|
|
from __future__ import annotations
|
|
|
|
import asyncio
|
|
import logging
|
|
import os
|
|
import sys
|
|
import threading
|
|
import time
|
|
import traceback
|
|
from datetime import datetime, timezone
|
|
from typing import Any, Callable, Optional
|
|
|
|
logger = logging.getLogger(__name__)
|
|
|
|
# Frames from these path fragments are "ours" — the innermost one is reported
|
|
# as the culprit (the stdlib/library frame below it is usually just the I/O).
|
|
_OWN_CODE_MARKERS = ("/backend/", "/karaoke_gen/")
|
|
_MAX_STACK_FRAMES = 30
|
|
_MAX_STACK_CHARS = 8000
|
|
|
|
Recorder = Callable[[dict], None]
|
|
|
|
|
|
class RequestTracker:
|
|
"""Counts HTTP requests so the watchdog can tell whether a stall hurt anyone.
|
|
|
|
Mutated only on the event-loop thread (by ``RequestTrackingMiddleware``) and
|
|
read from the watchdog thread; plain ints are safe to read under the GIL.
|
|
|
|
Why (2026-10-12/13 flapping alert): Cloud Run autoscaling starts extra
|
|
instances that serve one request and then sit idle, CPU-throttled, while the
|
|
SpaCy/NLTK warmup thread trickles along for 10+ minutes. Those idle instances
|
|
produced 10-40s "stalls" (empty culprit — the freeze consumed the sampling
|
|
schedule) with no request on them, tripping the >=2-in-30min alert. A stall
|
|
with no request in flight and none queued behind it affects nobody.
|
|
"""
|
|
|
|
def __init__(self) -> None:
|
|
self.in_flight = 0
|
|
self.started = 0
|
|
|
|
def begin(self) -> None:
|
|
self.in_flight += 1
|
|
self.started += 1
|
|
|
|
def end(self) -> None:
|
|
self.in_flight = max(0, self.in_flight - 1)
|
|
|
|
|
|
# Process-wide tracker: the middleware is attached at import time, the watchdog
|
|
# is created later in the lifespan — both share this instance.
|
|
request_tracker = RequestTracker()
|
|
|
|
|
|
class RequestTrackingMiddleware:
|
|
"""Pure-ASGI middleware feeding ``request_tracker`` (no per-request allocation
|
|
beyond the wrapper call; outermost so even rejected requests count)."""
|
|
|
|
def __init__(self, app, tracker: Optional[RequestTracker] = None) -> None:
|
|
self.app = app
|
|
self.tracker = tracker or request_tracker
|
|
|
|
async def __call__(self, scope, receive, send):
|
|
if scope.get("type") != "http":
|
|
return await self.app(scope, receive, send)
|
|
self.tracker.begin()
|
|
try:
|
|
return await self.app(scope, receive, send)
|
|
finally:
|
|
self.tracker.end()
|
|
|
|
# Innermost frames that mean "the loop is idle, waiting for I/O" — not blocked.
|
|
_IDLE_FUNCS = {"select", "poll", "epoll", "kqueue", "control", "_run_once"}
|
|
_IDLE_FILES = ("selectors.py", "base_events.py")
|
|
|
|
|
|
def _is_idle_frame(frame) -> bool:
|
|
"""True when the loop thread is parked in the selector (idle), not running code."""
|
|
if frame is None:
|
|
return False
|
|
code = frame.f_code
|
|
return code.co_name in _IDLE_FUNCS and code.co_filename.endswith(_IDLE_FILES)
|
|
|
|
|
|
def _format_stack(frame) -> tuple[str, str]:
|
|
"""Return (stack_text, culprit) for a frame; culprit = innermost own-code frame."""
|
|
if frame is None:
|
|
return "", ""
|
|
summary = traceback.extract_stack(frame)[-_MAX_STACK_FRAMES:]
|
|
culprit = ""
|
|
for fs in reversed(summary):
|
|
if any(m in fs.filename for m in _OWN_CODE_MARKERS) and "loop_watchdog" not in fs.filename:
|
|
short = fs.filename.split("/app/")[-1]
|
|
culprit = f"{short}:{fs.lineno} in {fs.name}"
|
|
break
|
|
if not culprit and summary:
|
|
fs = summary[-1]
|
|
culprit = f"{fs.filename}:{fs.lineno} in {fs.name}"
|
|
text = "".join(traceback.format_list(summary))[-_MAX_STACK_CHARS:]
|
|
return text, culprit
|
|
|
|
|
|
class LoopWatchdog:
|
|
def __init__(
|
|
self,
|
|
*,
|
|
tick_s: float = 0.25,
|
|
stall_threshold_s: float = 1.0,
|
|
error_threshold_s: float = 10.0,
|
|
record_threshold_s: float = 5.0,
|
|
sample_at_s: tuple[float, ...] = (1.0, 5.0, 15.0),
|
|
recorder: Optional[Recorder] = None,
|
|
clock: Callable[[], float] = time.monotonic,
|
|
cpu_clock: Callable[[], float] = time.process_time,
|
|
frozen_gap_s: float = 0.75,
|
|
frozen_cpu_ratio: float = 0.25,
|
|
request_tracker: Optional[RequestTracker] = None,
|
|
request_grace_s: float = 0.5,
|
|
) -> None:
|
|
self.tick_s = tick_s
|
|
self.stall_threshold_s = stall_threshold_s
|
|
self.error_threshold_s = error_threshold_s
|
|
self.record_threshold_s = record_threshold_s
|
|
self.sample_at_s = tuple(sorted(sample_at_s))
|
|
self.recorder = recorder
|
|
self._clock = clock
|
|
self._cpu_clock = cpu_clock
|
|
# Cloud Run --cpu-throttling freezes the WHOLE process between requests —
|
|
# sometimes mid-callback, where the "loop parked in select()" check can't
|
|
# see it (2026-10-08: a 30s "stall" in asyncio's get_debug on an instance
|
|
# with no requests in flight). Signature: this watchdog thread itself
|
|
# oversleeps by >= frozen_gap_s while the process burns almost no CPU
|
|
# (a blocking I/O call lets this thread wake on time; GIL-hogging CPU
|
|
# work burns CPU). That frozen time is discounted from the stall.
|
|
# 0.75s = 3 missed ticks: measured wake gaps stay <= 0.27s under sync I/O,
|
|
# pure-Python CPU and C-extension CPU on the loop, while partial throttling
|
|
# gives short 1-2.5s freezes (126 of 128 asyncio-internal "stalls" on
|
|
# 2026-10-08 had no request waiting on them).
|
|
self.frozen_gap_s = frozen_gap_s
|
|
self.frozen_cpu_ratio = frozen_cpu_ratio
|
|
self._last_check: Optional[float] = None
|
|
self._last_cpu = 0.0
|
|
self._stall_frozen_s = 0.0
|
|
# A freeze can carry the stall past every sample_at_s point without a
|
|
# sample. If the loop is STILL blocked after the thaw, that is real
|
|
# blocking — take one sample then so the stall has a culprit (2026-10-12:
|
|
# 10-40s ERROR stalls all had an empty culprit).
|
|
self._sample_after_thaw = False
|
|
# With a tracker, a stall only escalates to ERROR (the alerting level) if
|
|
# a request was in flight when it began or arrived during it. Requests
|
|
# queued in the socket during the freeze are accepted right after the loop
|
|
# resumes, so the verdict waits ``request_grace_s`` after resume.
|
|
self.request_tracker = request_tracker
|
|
self.request_grace_s = request_grace_s
|
|
self._stall_in_flight = 0
|
|
self._stall_started_count = 0
|
|
self._resumed_at: Optional[float] = None
|
|
self._resumed_duration = 0.0
|
|
|
|
self._last_tick = clock()
|
|
self._loop_thread_id: Optional[int] = None
|
|
self._heartbeat_task: Optional[asyncio.Task] = None
|
|
self._thread: Optional[threading.Thread] = None
|
|
self._stop = threading.Event()
|
|
|
|
# Current-stall state (only touched by the watchdog thread).
|
|
self._in_stall = False
|
|
self._stall_started_tick = 0.0
|
|
self._samples: list[dict[str, Any]] = []
|
|
self._next_sample_idx = 0
|
|
|
|
# ----- lifecycle -------------------------------------------------------
|
|
def start(self) -> None:
|
|
"""Start heartbeat + watchdog thread. Must be called from inside the loop."""
|
|
if self._thread is not None:
|
|
return
|
|
self._loop_thread_id = threading.get_ident()
|
|
self._last_tick = self._clock()
|
|
self._heartbeat_task = asyncio.get_running_loop().create_task(self._heartbeat())
|
|
self._stop.clear()
|
|
self._thread = threading.Thread(target=self._run, name="loop-watchdog", daemon=True)
|
|
self._thread.start()
|
|
logger.info(
|
|
"Event-loop watchdog started (stall>=%.1fs logged, >=%.1fs recorded, >=%.1fs error)",
|
|
self.stall_threshold_s, self.record_threshold_s, self.error_threshold_s,
|
|
)
|
|
|
|
async def stop(self) -> None:
|
|
self._stop.set()
|
|
if self._heartbeat_task is not None:
|
|
self._heartbeat_task.cancel()
|
|
try:
|
|
await self._heartbeat_task
|
|
except (asyncio.CancelledError, Exception):
|
|
pass
|
|
self._heartbeat_task = None
|
|
self._thread = None
|
|
|
|
async def _heartbeat(self) -> None:
|
|
while True:
|
|
self._last_tick = self._clock()
|
|
await asyncio.sleep(self.tick_s)
|
|
|
|
def _run(self) -> None:
|
|
while not self._stop.wait(self.tick_s):
|
|
try:
|
|
self.check()
|
|
except Exception: # pragma: no cover — must never kill the thread
|
|
logger.exception("loop watchdog check failed")
|
|
|
|
# ----- detection (called from the watchdog thread; unit-testable) ------
|
|
def _sample(self, age: float) -> None:
|
|
frame = sys._current_frames().get(self._loop_thread_id) if self._loop_thread_id else None
|
|
if _is_idle_frame(frame):
|
|
# Loop is parked in select(): not blocked. This is what a CPU-throttled
|
|
# idle Cloud Run instance looks like when CPU returns (the watchdog
|
|
# thread can wake before the heartbeat does). Record it so _finish
|
|
# can drop an all-idle "stall" as a false positive.
|
|
self._samples.append({"at_s": round(age, 1), "culprit": "", "stack": "", "idle": True})
|
|
return
|
|
stack, culprit = _format_stack(frame)
|
|
if self._samples and self._samples[-1]["stack"] == stack:
|
|
return # identical to the previous sample — nothing new to learn
|
|
self._samples.append({"at_s": round(age, 1), "culprit": culprit, "stack": stack, "idle": False})
|
|
# Always WARNING. The "ended" line is the one ERROR per freeze (>=10s
|
|
# blocked with a request affected): it's what the backend/event_loop_stalls
|
|
# metric counts, and the error monitor turns each ERROR line shape into its
|
|
# own #alerts pattern, so an ERROR here too meant two alerts per freeze.
|
|
level = logging.WARNING
|
|
logger.log(
|
|
level,
|
|
"EVENT_LOOP_STALL in progress age_ms=%d culprit=%s\n%s",
|
|
int(age * 1000), culprit, stack,
|
|
extra={"loop_stall_age_ms": int(age * 1000), "loop_stall_culprit": culprit},
|
|
)
|
|
|
|
def _frozen_gap(self, now: float) -> float:
|
|
"""Seconds since the previous check during which the whole process was frozen."""
|
|
cpu = self._cpu_clock()
|
|
prev_check, prev_cpu = self._last_check, self._last_cpu
|
|
self._last_check, self._last_cpu = now, cpu
|
|
if prev_check is None:
|
|
return 0.0
|
|
gap = now - prev_check
|
|
if gap >= self.frozen_gap_s and (cpu - prev_cpu) < gap * self.frozen_cpu_ratio:
|
|
return gap
|
|
return 0.0
|
|
|
|
def check(self, now: Optional[float] = None) -> None:
|
|
now = self._clock() if now is None else now
|
|
frozen = self._frozen_gap(now)
|
|
last = self._last_tick
|
|
age = now - last
|
|
|
|
if not self._in_stall:
|
|
if age >= self.stall_threshold_s:
|
|
self._in_stall = True
|
|
self._stall_started_tick = last
|
|
self._samples = []
|
|
self._next_sample_idx = 0
|
|
self._stall_frozen_s = 0.0
|
|
self._sample_after_thaw = False
|
|
self._resumed_at = None
|
|
if self.request_tracker is not None:
|
|
self._stall_in_flight = self.request_tracker.in_flight
|
|
self._stall_started_count = self.request_tracker.started
|
|
else:
|
|
return
|
|
if self._resumed_at is not None:
|
|
# Loop already resumed; waiting out the grace window for queued requests.
|
|
if self._requests_affected() or now - self._resumed_at >= self.request_grace_s:
|
|
self._finish(self._resumed_duration)
|
|
return
|
|
# Only the part of the frozen gap after the last heartbeat belongs to this stall.
|
|
self._stall_frozen_s += min(frozen, age)
|
|
|
|
# In a stall: has the heartbeat resumed?
|
|
if last > self._stall_started_tick:
|
|
# The heartbeat that ended the stall fired one tick_s after the
|
|
# previous one would have, so subtract the nominal sleep.
|
|
duration = max(0.0, last - self._stall_started_tick - self.tick_s)
|
|
if self.request_tracker is not None and not self._requests_affected():
|
|
self._resumed_at = now
|
|
self._resumed_duration = duration
|
|
return
|
|
self._finish(duration)
|
|
return
|
|
|
|
if frozen:
|
|
# Just thawed: the loop's stack shows where the freeze caught it, not
|
|
# what blocked it. Skip the samples due during the freeze.
|
|
skipped = False
|
|
while (
|
|
self._next_sample_idx < len(self.sample_at_s)
|
|
and age >= self.sample_at_s[self._next_sample_idx]
|
|
):
|
|
self._next_sample_idx += 1
|
|
skipped = True
|
|
if skipped:
|
|
self._sample_after_thaw = True
|
|
return
|
|
|
|
if self._sample_after_thaw:
|
|
# First on-time check after a freeze and the loop still hasn't ticked.
|
|
self._sample_after_thaw = False
|
|
self._sample(age)
|
|
|
|
while (
|
|
self._next_sample_idx < len(self.sample_at_s)
|
|
and age >= self.sample_at_s[self._next_sample_idx]
|
|
):
|
|
self._next_sample_idx += 1
|
|
self._sample(age)
|
|
|
|
def _requests_affected(self) -> int:
|
|
"""Requests that waited on this stall: in flight at its start + started since."""
|
|
t = self.request_tracker
|
|
if t is None:
|
|
return 0
|
|
return self._stall_in_flight + max(0, t.started - self._stall_started_count)
|
|
|
|
def _finish(self, duration: float) -> None:
|
|
samples = self._samples
|
|
frozen_s = self._stall_frozen_s
|
|
affected = self._requests_affected()
|
|
tracked = self.request_tracker is not None
|
|
in_flight_at_start = self._stall_in_flight
|
|
self._resumed_at = None
|
|
self._in_stall = False
|
|
self._samples = []
|
|
self._stall_frozen_s = 0.0
|
|
if frozen_s:
|
|
logger.debug(
|
|
"loop watchdog: discounted %dms of process freeze (CPU throttling) from a %dms gap",
|
|
int(frozen_s * 1000), int(duration * 1000),
|
|
)
|
|
duration = max(0.0, duration - frozen_s)
|
|
if duration < self.stall_threshold_s:
|
|
return
|
|
if samples and all(s.get("idle") for s in samples):
|
|
# Every sample saw the loop idle in select() — a throttled/idle gap
|
|
# (no CPU allocated between requests), not a blocking call.
|
|
logger.debug("loop watchdog: ignored idle gap of %dms", int(duration * 1000))
|
|
return
|
|
samples = [s for s in samples if not s.get("idle")]
|
|
culprits = [s["culprit"] for s in samples if s["culprit"]]
|
|
# The longest-lived sample is the most representative of the freeze.
|
|
main_culprit = culprits[-1] if culprits else ""
|
|
# No request waited on it (idle, typically CPU-throttled instance) → never
|
|
# ERROR: the alert counts ERROR lines and nobody saw this freeze.
|
|
no_impact = tracked and affected == 0
|
|
# A process freeze (CPU throttling of an idle instance) was observed and
|
|
# swallowed the whole sampling schedule, so the stall was never seen
|
|
# blocking in our code. If nothing was in flight when it began, every
|
|
# "affected" request merely arrived while the instance was thawing
|
|
# (2026-10-23: 20.7s, empty culprit, its one request arrived 0.3s before
|
|
# the end). Not actionable. Without an observed freeze an unsampled stall
|
|
# can be real blocking (a GIL hog that starved this thread until the loop
|
|
# had already resumed), so that stays eligible for ERROR.
|
|
unattributed_late_only = (
|
|
tracked and not samples and frozen_s > 0 and in_flight_at_start == 0
|
|
)
|
|
level = (
|
|
logging.ERROR
|
|
if duration >= self.error_threshold_s and not no_impact and not unattributed_late_only
|
|
else logging.WARNING
|
|
)
|
|
logger.log(
|
|
level,
|
|
"EVENT_LOOP_STALL ended duration_ms=%d culprit=%s requests_affected=%s",
|
|
int(duration * 1000), main_culprit, affected if tracked else "unknown",
|
|
extra={
|
|
"loop_stall_duration_ms": int(duration * 1000),
|
|
"loop_stall_culprit": main_culprit,
|
|
"loop_stall_culprits": culprits,
|
|
"loop_stall_requests_affected": affected if tracked else None,
|
|
},
|
|
)
|
|
if self.recorder is not None and duration >= self.record_threshold_s:
|
|
try:
|
|
self.recorder({
|
|
"duration_ms": int(duration * 1000),
|
|
"culprit": main_culprit,
|
|
"culprits": culprits,
|
|
"stack": samples[-1]["stack"] if samples else "",
|
|
"requests_affected": affected if tracked else None,
|
|
})
|
|
except Exception:
|
|
logger.warning("Failed to record loop stall (non-fatal)", exc_info=True)
|
|
|
|
|
|
# ----- production recorder ---------------------------------------------------
|
|
_instance_id: Optional[str] = None
|
|
|
|
|
|
def _cloud_run_instance_id() -> str:
|
|
"""Best-effort Cloud Run instance id from the metadata server (cached)."""
|
|
global _instance_id
|
|
if _instance_id is not None:
|
|
return _instance_id
|
|
_instance_id = ""
|
|
try:
|
|
import requests
|
|
|
|
resp = requests.get(
|
|
"http://metadata.google.internal/computeMetadata/v1/instance/id",
|
|
headers={"Metadata-Flavor": "Google"},
|
|
timeout=1,
|
|
)
|
|
if resp.ok:
|
|
_instance_id = resp.text.strip()
|
|
except Exception:
|
|
pass
|
|
return _instance_id
|
|
|
|
|
|
def firestore_stall_recorder(stall: dict) -> None:
|
|
"""Persist a stall as a ``client_events`` doc (type ``server_loop_stall``)."""
|
|
from google.cloud import firestore # type: ignore[import]
|
|
|
|
from backend.version import VERSION
|
|
|
|
doc = {
|
|
"type": "server_loop_stall",
|
|
"source": "server",
|
|
"created_at": datetime.now(timezone.utc),
|
|
"release": VERSION,
|
|
"revision": os.getenv("K_REVISION", ""),
|
|
"instance_id": _cloud_run_instance_id(),
|
|
"detail": {
|
|
"duration_ms": stall["duration_ms"],
|
|
"culprit": stall["culprit"][:300],
|
|
"requests_affected": stall.get("requests_affected"),
|
|
},
|
|
"culprits": [c[:300] for c in stall.get("culprits", [])][:5],
|
|
"stack": stall.get("stack", "")[:_MAX_STACK_CHARS],
|
|
}
|
|
firestore.Client(project="nomadkaraoke").collection("client_events").add(doc)
|