mirror of
https://github.com/nomadkaraoke/karaoke-gen.git
synced 2026-11-20 13:07:57 +09:00
@coderabbitai ignore ## Why The residual event-loop freezes BACKLOG item. Seven days of `EVENT_LOOP_STALL in progress` stacks (2026-10-12→19, 355 samples), grouped by their outermost app frame, ranked the app-level loop blockers: | Source | Samples | Max | |---|---|---| | in-process screens worker → job-log handler → sync Firestore write per log line | 30 | 2.2s | | `require_auth` → `validate_token_full` (session read on **every** authed request) | 19 | 1.9s | | preview encodes (`_run_preview_encode`/`_encode_preview_in_background`) → job-log write | 15 | 2.4s | | `change_visibility` → sync Dropbox/GCS deletes | 6 | 1.2s | | tenant middleware → `get_tenant_config` (GCS on cache miss) | 5 | 1.2s | | `require_review_auth` → `get_job` | 5 | 2.2s | | `auto_correct_worker._run_suggest` → `get_job` (the only stall since #1192) | 1 | 1.2s | (The render/video in-process stalls stopped with #1192.) ## What - **`backend/services/log_write_offload.py`**: when a job-log write is made on a running event loop in the API process, it goes to one background writer thread. The queue is bounded at 5000 and a drop is logged at WARNING, rate-limited; `atexit` drains the queue. Writes stay inline in threads and Cloud Run Jobs, so the worker code paths behave exactly as before. Logs are read back ordered by their own `timestamp`, so the hand-off doesn't reorder them. - **`asyncio.to_thread`** for `require_auth`/`require_auth_legacy`/`require_review_auth`, the tenant middleware, `rerender_with_current_theme`'s tenant lookup, `auto_correct._run_suggest`, and `change_visibility`'s `get_job`. - **Visibility change:** `_delete_*_outputs` are now plain `def`, and the sync steps run in one worker thread. Moving them off the loop removed the implicit serialisation, so both directions now **claim `visibility_change_in_progress` atomically** before touching anything. They use the new `FirestoreService.try_set_job_state_flag`, a transactional test-and-set. A lost claim returns 409. The regenerate→private continuation passes `holds_claim=True`, because its regenerate already set the flag. The claim is released if the reset fails, and a raising screens trigger rolls back like a `False` one. - **Guard (`test_no_blocking_async_routes.py`):** `BLOCKING_CALLS` gains `validate_token(_full)`, `validate_session`, `get_tenant_config`, `append_log_to_subcollection` and `append_worker_log`. Every `async def` in `api/dependencies.py` and `middleware/` is now checked, not just decorated routes. Mutation-checked: reverting `require_auth` fails the guard. - `docs/LESSONS-LEARNED.md` entry. About 70 direct `get_job`/`update_job` calls in async routes remain. They're single round-trips that rarely show in the watchdog, and the doc says to move them when one does. ## Testing - New: `test_log_write_offload.py` (inline vs offloaded, Cloud Run Job inline, ordering, full-queue drop, Firestore append off the loop thread), claim tests in `test_change_visibility.py` (lost claim, release on failure, chained continuation, trigger exception, route 409), and emulator `test_try_set_job_state_flag.py` (once-only, already set, missing job, one winner among 4 concurrent callers). - Backend: 6478 passed. The 8 failures are `test_emulator_integration.py`, caused by my local environment: another session's emulator setup has an `ssh` forward on :4443 instead of fake-GCS, and they fail identically on untouched main. CI's emulator job covers them and the new emulator test. - Local CodeRabbit: 3 cycles. I fixed the claim race (two findings), trigger-exception rollback, and the guarded claim release. I skipped two minor nitpicks on the pre-existing trigger-failure rollback. 🤖 Generated with [Claude Code](https://claude.com/claude-code) https://claude.ai/code/session_01Jw21syo4HizJJjYw2mg2Vp --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
110 lines
3.7 KiB
Python
110 lines
3.7 KiB
Python
"""Job-log Firestore writes must not run on the API event loop (log_write_offload)."""
|
|
import asyncio
|
|
import queue
|
|
import threading
|
|
from unittest.mock import MagicMock
|
|
|
|
import pytest
|
|
|
|
from backend.services import log_write_offload
|
|
|
|
|
|
@pytest.fixture(autouse=True)
|
|
def _not_in_cloud_run_job(monkeypatch):
|
|
monkeypatch.delenv("CLOUD_RUN_JOB", raising=False)
|
|
yield
|
|
assert log_write_offload.drain(timeout=5)
|
|
|
|
|
|
def test_runs_inline_off_the_event_loop():
|
|
ran_on = []
|
|
log_write_offload.submit_or_run(lambda: ran_on.append(threading.get_ident()))
|
|
assert ran_on == [threading.get_ident()]
|
|
|
|
|
|
def test_offloads_from_a_running_event_loop():
|
|
release = threading.Event()
|
|
ran_on = []
|
|
|
|
def slow_write():
|
|
release.wait(5)
|
|
ran_on.append(threading.get_ident())
|
|
|
|
async def on_loop():
|
|
log_write_offload.submit_or_run(slow_write)
|
|
# submit returned without waiting for the (blocked) write
|
|
assert ran_on == []
|
|
return threading.get_ident()
|
|
|
|
loop_thread = asyncio.run(on_loop())
|
|
release.set()
|
|
assert log_write_offload.drain(timeout=5)
|
|
assert len(ran_on) == 1 and ran_on[0] != loop_thread
|
|
|
|
|
|
def test_runs_inline_on_the_loop_inside_a_cloud_run_job(monkeypatch):
|
|
monkeypatch.setenv("CLOUD_RUN_JOB", "video-encoding-job")
|
|
ran_on = []
|
|
|
|
async def on_loop():
|
|
log_write_offload.submit_or_run(lambda: ran_on.append(threading.get_ident()))
|
|
return threading.get_ident()
|
|
|
|
assert ran_on == [asyncio.run(on_loop())]
|
|
|
|
|
|
def test_offloaded_writes_keep_submission_order():
|
|
written = []
|
|
|
|
async def on_loop():
|
|
for i in range(50):
|
|
log_write_offload.submit_or_run(lambda i=i: written.append(i))
|
|
|
|
asyncio.run(on_loop())
|
|
assert log_write_offload.drain(timeout=5)
|
|
assert written == list(range(50))
|
|
|
|
|
|
def test_full_queue_drops_with_a_warning(monkeypatch, caplog):
|
|
release = threading.Event()
|
|
monkeypatch.setattr(log_write_offload, "_queue", queue.Queue(maxsize=1))
|
|
monkeypatch.setattr(log_write_offload, "_worker", None)
|
|
monkeypatch.setattr(log_write_offload, "_dropped", 0)
|
|
monkeypatch.setattr(log_write_offload, "_last_drop_warning", 0.0)
|
|
written = []
|
|
|
|
async def on_loop():
|
|
log_write_offload.submit_or_run(lambda: release.wait(5)) # occupies the writer
|
|
await asyncio.sleep(0.05)
|
|
log_write_offload.submit_or_run(lambda: written.append("queued")) # fills the queue
|
|
log_write_offload.submit_or_run(lambda: written.append("dropped")) # no room
|
|
|
|
with caplog.at_level("WARNING", logger=log_write_offload.__name__):
|
|
asyncio.run(on_loop())
|
|
release.set()
|
|
assert log_write_offload.drain(timeout=5)
|
|
assert written == ["queued"]
|
|
assert log_write_offload._dropped == 1
|
|
assert "dropped 1 job-log lines" in caplog.text
|
|
|
|
|
|
def test_firestore_append_log_does_not_write_on_the_loop_thread():
|
|
from backend.models.worker_log import WorkerLogEntry
|
|
from backend.services.firestore_service import FirestoreService
|
|
|
|
service = FirestoreService.__new__(FirestoreService)
|
|
service.collection = "jobs"
|
|
service.db = MagicMock()
|
|
doc_ref = service.db.collection.return_value.document.return_value.collection.return_value.document.return_value
|
|
set_threads = []
|
|
doc_ref.set.side_effect = lambda data: set_threads.append(threading.get_ident())
|
|
|
|
async def on_loop():
|
|
service.append_log_to_subcollection("job1", WorkerLogEntry.create(job_id="job1", worker="screens", level="INFO", message="hi"))
|
|
return threading.get_ident()
|
|
|
|
loop_thread = asyncio.run(on_loop())
|
|
assert log_write_offload.drain(timeout=5)
|
|
assert len(set_threads) == 1 and set_threads[0] != loop_thread
|
|
assert doc_ref.set.call_args.args[0]["message"] == "hi"
|