Files
karaoke-gen/backend/tests/test_log_write_offload.py
T
Andrew Beveridge ☄️andClaude Opus 5.5 3bbf397dc4 fix: keep auth, tenant lookup and job-log writes off the API event loop (v0.278.14) (#1194)
@coderabbitai ignore

## Why
The residual event-loop freezes BACKLOG item. Seven days of
`EVENT_LOOP_STALL in progress` stacks (2026-10-12→19, 355 samples),
grouped by their outermost app frame, ranked the app-level loop
blockers:

| Source | Samples | Max |
|---|---|---|
| in-process screens worker → job-log handler → sync Firestore write per
log line | 30 | 2.2s |
| `require_auth` → `validate_token_full` (session read on **every**
authed request) | 19 | 1.9s |
| preview encodes
(`_run_preview_encode`/`_encode_preview_in_background`) → job-log write
| 15 | 2.4s |
| `change_visibility` → sync Dropbox/GCS deletes | 6 | 1.2s |
| tenant middleware → `get_tenant_config` (GCS on cache miss) | 5 | 1.2s
|
| `require_review_auth` → `get_job` | 5 | 2.2s |
| `auto_correct_worker._run_suggest` → `get_job` (the only stall since
#1192) | 1 | 1.2s |

(The render/video in-process stalls stopped with #1192.)

## What
- **`backend/services/log_write_offload.py`**: when a job-log write is
made on a running event loop in the API process, it goes to one
background writer thread. The queue is bounded at 5000 and a drop is
logged at WARNING, rate-limited; `atexit` drains the queue. Writes stay
inline in threads and Cloud Run Jobs, so the worker code paths behave
exactly as before. Logs are read back ordered by their own `timestamp`,
so the hand-off doesn't reorder them.
- **`asyncio.to_thread`** for
`require_auth`/`require_auth_legacy`/`require_review_auth`, the tenant
middleware, `rerender_with_current_theme`'s tenant lookup,
`auto_correct._run_suggest`, and `change_visibility`'s `get_job`.
- **Visibility change:** `_delete_*_outputs` are now plain `def`, and
the sync steps run in one worker thread. Moving them off the loop
removed the implicit serialisation, so both directions now **claim
`visibility_change_in_progress` atomically** before touching anything.
They use the new `FirestoreService.try_set_job_state_flag`, a
transactional test-and-set. A lost claim returns 409. The
regenerate→private continuation passes `holds_claim=True`, because its
regenerate already set the flag. The claim is released if the reset
fails, and a raising screens trigger rolls back like a `False` one.
- **Guard (`test_no_blocking_async_routes.py`):** `BLOCKING_CALLS` gains
`validate_token(_full)`, `validate_session`, `get_tenant_config`,
`append_log_to_subcollection` and `append_worker_log`. Every `async def`
in `api/dependencies.py` and `middleware/` is now checked, not just
decorated routes. Mutation-checked: reverting `require_auth` fails the
guard.
- `docs/LESSONS-LEARNED.md` entry. About 70 direct
`get_job`/`update_job` calls in async routes remain. They're single
round-trips that rarely show in the watchdog, and the doc says to move
them when one does.

## Testing
- New: `test_log_write_offload.py` (inline vs offloaded, Cloud Run Job
inline, ordering, full-queue drop, Firestore append off the loop
thread), claim tests in `test_change_visibility.py` (lost claim, release
on failure, chained continuation, trigger exception, route 409), and
emulator `test_try_set_job_state_flag.py` (once-only, already set,
missing job, one winner among 4 concurrent callers).
- Backend: 6478 passed. The 8 failures are
`test_emulator_integration.py`, caused by my local environment: another
session's emulator setup has an `ssh` forward on :4443 instead of
fake-GCS, and they fail identically on untouched main. CI's emulator job
covers them and the new emulator test.
- Local CodeRabbit: 3 cycles. I fixed the claim race (two findings),
trigger-exception rollback, and the guarded claim release. I skipped two
minor nitpicks on the pre-existing trigger-failure rollback.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

https://claude.ai/code/session_01Jw21syo4HizJJjYw2mg2Vp

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-19 14:41:34 -04:00

110 lines
3.7 KiB
Python

"""Job-log Firestore writes must not run on the API event loop (log_write_offload)."""
import asyncio
import queue
import threading
from unittest.mock import MagicMock
import pytest
from backend.services import log_write_offload
@pytest.fixture(autouse=True)
def _not_in_cloud_run_job(monkeypatch):
monkeypatch.delenv("CLOUD_RUN_JOB", raising=False)
yield
assert log_write_offload.drain(timeout=5)
def test_runs_inline_off_the_event_loop():
ran_on = []
log_write_offload.submit_or_run(lambda: ran_on.append(threading.get_ident()))
assert ran_on == [threading.get_ident()]
def test_offloads_from_a_running_event_loop():
release = threading.Event()
ran_on = []
def slow_write():
release.wait(5)
ran_on.append(threading.get_ident())
async def on_loop():
log_write_offload.submit_or_run(slow_write)
# submit returned without waiting for the (blocked) write
assert ran_on == []
return threading.get_ident()
loop_thread = asyncio.run(on_loop())
release.set()
assert log_write_offload.drain(timeout=5)
assert len(ran_on) == 1 and ran_on[0] != loop_thread
def test_runs_inline_on_the_loop_inside_a_cloud_run_job(monkeypatch):
monkeypatch.setenv("CLOUD_RUN_JOB", "video-encoding-job")
ran_on = []
async def on_loop():
log_write_offload.submit_or_run(lambda: ran_on.append(threading.get_ident()))
return threading.get_ident()
assert ran_on == [asyncio.run(on_loop())]
def test_offloaded_writes_keep_submission_order():
written = []
async def on_loop():
for i in range(50):
log_write_offload.submit_or_run(lambda i=i: written.append(i))
asyncio.run(on_loop())
assert log_write_offload.drain(timeout=5)
assert written == list(range(50))
def test_full_queue_drops_with_a_warning(monkeypatch, caplog):
release = threading.Event()
monkeypatch.setattr(log_write_offload, "_queue", queue.Queue(maxsize=1))
monkeypatch.setattr(log_write_offload, "_worker", None)
monkeypatch.setattr(log_write_offload, "_dropped", 0)
monkeypatch.setattr(log_write_offload, "_last_drop_warning", 0.0)
written = []
async def on_loop():
log_write_offload.submit_or_run(lambda: release.wait(5)) # occupies the writer
await asyncio.sleep(0.05)
log_write_offload.submit_or_run(lambda: written.append("queued")) # fills the queue
log_write_offload.submit_or_run(lambda: written.append("dropped")) # no room
with caplog.at_level("WARNING", logger=log_write_offload.__name__):
asyncio.run(on_loop())
release.set()
assert log_write_offload.drain(timeout=5)
assert written == ["queued"]
assert log_write_offload._dropped == 1
assert "dropped 1 job-log lines" in caplog.text
def test_firestore_append_log_does_not_write_on_the_loop_thread():
from backend.models.worker_log import WorkerLogEntry
from backend.services.firestore_service import FirestoreService
service = FirestoreService.__new__(FirestoreService)
service.collection = "jobs"
service.db = MagicMock()
doc_ref = service.db.collection.return_value.document.return_value.collection.return_value.document.return_value
set_threads = []
doc_ref.set.side_effect = lambda data: set_threads.append(threading.get_ident())
async def on_loop():
service.append_log_to_subcollection("job1", WorkerLogEntry.create(job_id="job1", worker="screens", level="INFO", message="hi"))
return threading.get_ident()
loop_thread = asyncio.run(on_loop())
assert log_write_offload.drain(timeout=5)
assert len(set_threads) == 1 and set_threads[0] != loop_thread
assert doc_ref.set.call_args.args[0]["message"] == "hi"