mirror of
https://github.com/nomadkaraoke/karaoke-gen.git
synced 2026-11-20 13:07:57 +09:00
@coderabbitai ignore ## Why The residual event-loop freezes BACKLOG item. Seven days of `EVENT_LOOP_STALL in progress` stacks (2026-10-12→19, 355 samples), grouped by their outermost app frame, ranked the app-level loop blockers: | Source | Samples | Max | |---|---|---| | in-process screens worker → job-log handler → sync Firestore write per log line | 30 | 2.2s | | `require_auth` → `validate_token_full` (session read on **every** authed request) | 19 | 1.9s | | preview encodes (`_run_preview_encode`/`_encode_preview_in_background`) → job-log write | 15 | 2.4s | | `change_visibility` → sync Dropbox/GCS deletes | 6 | 1.2s | | tenant middleware → `get_tenant_config` (GCS on cache miss) | 5 | 1.2s | | `require_review_auth` → `get_job` | 5 | 2.2s | | `auto_correct_worker._run_suggest` → `get_job` (the only stall since #1192) | 1 | 1.2s | (The render/video in-process stalls stopped with #1192.) ## What - **`backend/services/log_write_offload.py`**: when a job-log write is made on a running event loop in the API process, it goes to one background writer thread. The queue is bounded at 5000 and a drop is logged at WARNING, rate-limited; `atexit` drains the queue. Writes stay inline in threads and Cloud Run Jobs, so the worker code paths behave exactly as before. Logs are read back ordered by their own `timestamp`, so the hand-off doesn't reorder them. - **`asyncio.to_thread`** for `require_auth`/`require_auth_legacy`/`require_review_auth`, the tenant middleware, `rerender_with_current_theme`'s tenant lookup, `auto_correct._run_suggest`, and `change_visibility`'s `get_job`. - **Visibility change:** `_delete_*_outputs` are now plain `def`, and the sync steps run in one worker thread. Moving them off the loop removed the implicit serialisation, so both directions now **claim `visibility_change_in_progress` atomically** before touching anything. They use the new `FirestoreService.try_set_job_state_flag`, a transactional test-and-set. A lost claim returns 409. The regenerate→private continuation passes `holds_claim=True`, because its regenerate already set the flag. The claim is released if the reset fails, and a raising screens trigger rolls back like a `False` one. - **Guard (`test_no_blocking_async_routes.py`):** `BLOCKING_CALLS` gains `validate_token(_full)`, `validate_session`, `get_tenant_config`, `append_log_to_subcollection` and `append_worker_log`. Every `async def` in `api/dependencies.py` and `middleware/` is now checked, not just decorated routes. Mutation-checked: reverting `require_auth` fails the guard. - `docs/LESSONS-LEARNED.md` entry. About 70 direct `get_job`/`update_job` calls in async routes remain. They're single round-trips that rarely show in the watchdog, and the doc says to move them when one does. ## Testing - New: `test_log_write_offload.py` (inline vs offloaded, Cloud Run Job inline, ordering, full-queue drop, Firestore append off the loop thread), claim tests in `test_change_visibility.py` (lost claim, release on failure, chained continuation, trigger exception, route 409), and emulator `test_try_set_job_state_flag.py` (once-only, already set, missing job, one winner among 4 concurrent callers). - Backend: 6478 passed. The 8 failures are `test_emulator_integration.py`, caused by my local environment: another session's emulator setup has an `ssh` forward on :4443 instead of fake-GCS, and they fail identically on untouched main. CI's emulator job covers them and the new emulator test. - Local CodeRabbit: 3 cycles. I fixed the claim race (two findings), trigger-exception rollback, and the guarded claim release. I skipped two minor nitpicks on the pre-existing trigger-failure rollback. 🤖 Generated with [Claude Code](https://claude.com/claude-code) https://claude.ai/code/session_01Jw21syo4HizJJjYw2mg2Vp --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
101 lines
3.2 KiB
Python
101 lines
3.2 KiB
Python
"""
|
|
Keep job-log Firestore writes off the API event loop.
|
|
|
|
Job logs (jobs/{id}/logs/*) are written synchronously by the JobLogHandler /
|
|
JobLogger / job_logging handlers. When that code runs on the API's event loop (the
|
|
screens worker and preview encodes run as in-process background tasks), every log
|
|
line is a 0.1-5s Firestore round-trip that freezes every request on the instance:
|
|
the loop watchdog attributed 45 stalls in a week to append_log_to_subcollection.
|
|
|
|
`submit_or_run` hands the write to one background thread when the caller is on a
|
|
running event loop in the API process, and runs it inline everywhere else (worker
|
|
threads, Cloud Run Jobs, scripts), so ordering-sensitive callers are unaffected.
|
|
Logs are read back ordered by their own timestamp, so the hand-off doesn't reorder
|
|
them. The queue is bounded: under a Firestore outage we drop job-log lines rather
|
|
than grow memory.
|
|
"""
|
|
import asyncio
|
|
import atexit
|
|
import logging
|
|
import os
|
|
import queue
|
|
import threading
|
|
import time
|
|
from typing import Callable
|
|
|
|
logger = logging.getLogger(__name__)
|
|
|
|
_MAX_PENDING = 5000
|
|
_DROP_WARNING_INTERVAL_S = 60.0
|
|
|
|
_queue: "queue.Queue[Callable[[], None]]" = queue.Queue(maxsize=_MAX_PENDING)
|
|
_worker: threading.Thread | None = None
|
|
_worker_lock = threading.Lock()
|
|
_dropped = 0
|
|
_last_drop_warning = 0.0
|
|
|
|
|
|
def _on_event_loop() -> bool:
|
|
try:
|
|
asyncio.get_running_loop()
|
|
except RuntimeError:
|
|
return False
|
|
return True
|
|
|
|
|
|
def _should_offload() -> bool:
|
|
# A Cloud Run Job runs a single job; blocking its own loop affects nobody else,
|
|
# and a process that exits right after its last log line must not lose it.
|
|
return _on_event_loop() and not os.environ.get("CLOUD_RUN_JOB")
|
|
|
|
|
|
def _run_worker() -> None:
|
|
while True:
|
|
write = _queue.get()
|
|
try:
|
|
write()
|
|
except Exception as e: # the write itself already swallows; belt and braces
|
|
logger.debug(f"Offloaded job-log write failed: {e}")
|
|
finally:
|
|
_queue.task_done()
|
|
|
|
|
|
def _ensure_worker() -> None:
|
|
global _worker
|
|
if _worker is not None and _worker.is_alive():
|
|
return
|
|
with _worker_lock:
|
|
if _worker is None or not _worker.is_alive():
|
|
_worker = threading.Thread(target=_run_worker, name="job-log-writer", daemon=True)
|
|
_worker.start()
|
|
|
|
|
|
def submit_or_run(write: Callable[[], None]) -> None:
|
|
"""Run `write` on the background writer if called on the API loop, else inline."""
|
|
global _dropped, _last_drop_warning
|
|
if not _should_offload():
|
|
write()
|
|
return
|
|
_ensure_worker()
|
|
try:
|
|
_queue.put_nowait(write)
|
|
except queue.Full:
|
|
_dropped += 1
|
|
now = time.monotonic()
|
|
if now - _last_drop_warning >= _DROP_WARNING_INTERVAL_S:
|
|
_last_drop_warning = now
|
|
logger.warning(f"Job-log writer queue full ({_MAX_PENDING}); dropped {_dropped} job-log lines so far")
|
|
|
|
|
|
def drain(timeout: float = 10.0) -> bool:
|
|
"""Wait (up to `timeout` s) for queued writes to finish. True if fully drained."""
|
|
deadline = time.monotonic() + timeout
|
|
while _queue.unfinished_tasks:
|
|
if _worker is None or not _worker.is_alive() or time.monotonic() >= deadline:
|
|
return False
|
|
time.sleep(0.01)
|
|
return True
|
|
|
|
|
|
atexit.register(drain)
|