Files
karaoke-gen/backend/tests/test_event_loop_responsiveness.py
Andrew Beveridge ☄️andClaude Opus 5.5 f2612ac110 fix: close edit→delete refund loop, retry lost YouTube uploads, unblock event loop (v0.268.1) (#1135)
## Summary
Investigation of two 2026-10-08 Discord alerts (plus a third that
arrived mid-session) turned up four issues:

1. **Credit-refund loop.** A brand-new welcome-credit account received a
finished video (job 8a10a22d), clicked **Edit** (status back to
`awaiting_review`), then **deleted** the job and got its credit back.
`delete_job` refunds any non-complete status, and Edit makes a delivered
job look "never completed".
2. **YouTube 410 "Gone" dropped the upload.** The resumable session was
lost 1.3s into the upload. `upload_video` called `execute()` once with
no retries, and the orchestrator only queued quota errors, so any other
failure lost the upload.
3. **EVENT_LOOP_STALL 21s in `upload_json`.** `complete_review` (an
async route) wrote the corrections JSON to GCS directly on the event
loop. Ranking a week of watchdog stacks by route showed the same pattern
in `recover_stuck_jobs` (~260 stalls/week), the YouTube queue processor
(its `_process_single_upload` ran a whole upload on the loop),
`create_job_from_search` (theme prep), `_validate_audio_durations`
(15s), and the sync `run_job` / generation bump inside worker triggers.
4. **False 30s "stall" in asyncio `get_debug`.** Cloud Run CPU
throttling froze the whole instance mid-callback with no request in
flight. The watchdog only discounted freezes where the loop was parked
in `select()`.

## Changes
- **Refund:** `_refund_credit_for_job` skips jobs with `edit_count > 0`
(already delivered). This covers delete, cancel, fail and the 48h review
auto-expiry.
- **YouTube:** on 404/410 the upload restarts with a fresh session (3
attempts, with backoff). `execute(num_retries=3)` lets googleapiclient
resume chunks on 5xx/429 within a session. Any other failure is queued
in `youtube_upload_queue` (`reason="upload_error"`) for the existing
processor, which retries up to 5 times. Admins still get a Discord
warning: "queued for automatic retry".
- **Event loop:**
- New `backend/utils/loop_bridge.py`: `run_in_body_thread` runs sync
route bodies on a dedicated 16-thread pool, so they can't exhaust the
default executor the dispatched coroutines need. `run_on_loop`
dispatches coroutines back to the API loop (`run_coroutine_threadsafe`,
120s timeout).
- `complete_review`: its sync save runs via `to_thread`, and the render
trigger stays on the loop.
- `create_job_from_search`, `recover_stuck_jobs` and
`retry_pending_render_jobs`: bodies run in the route-body pool.
`_retrigger_parked_download` now does its writes in that thread.
- `_validate_audio_durations` and the YouTube queue's
`_process_single_upload` keep their async signatures but run in threads.
The queue's quota and Firestore calls go through `to_thread`.
- `worker_service`: the generation bump and `client.run_job` go through
`to_thread`.
- **Watchdog:** if the watchdog thread itself oversleeps by 2s or more
while the process uses less than 25% CPU, the whole process was frozen.
That time is subtracted from the stall and isn't sampled. Blocking I/O
(the watchdog still wakes on time) and GIL-hogging CPU work (CPU keeps
advancing) are still reported.
- Docs: `API.md` (edit/refund note) and `LESSONS-LEARNED.md`
(mostly-sync routes that await one trigger; throttling freezes).

## Testing
- Backend unit suite: **5863 passed**, 36 skipped (`pytest backend/tests
--ignore=backend/tests/emulator`).
- New tests:
- edited-track refund cases (delete in awaiting_review /
lyrics_complete, cancel);
  - YouTube restart on 410, give up after 3 attempts, no restart on 403;
  - orchestrator queues failed uploads and warns when queueing fails;
  - loop-responsiveness ticker tests for each offloaded path;
  - `run_on_loop` result, exception and timeout-with-cancel;
  - `run_in_body_thread` copies context and uses its own pool;
- watchdog: freeze ignored, I/O stall and CPU hog still reported, freeze
subtracted from a real stall.
- Local CodeRabbit: 3 cycles. Fixed the nested-executor deadlock risk,
the missing `run_on_loop` timeout, the parked-download writes on the
loop, and a coroutine leak. The last cycle had 0 findings.

@coderabbitai ignore

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-08 16:15:47 -04:00

372 lines
12 KiB
Python

"""The heavy steps of the screens worker / stale-review cron must not block the loop.
Prod (2026-10-03 analysis): these ran synchronously on the single uvicorn event
loop and froze every request on the instance for 20-45s, surfacing as the
"servers unavailable" banner. Each test patches the heavy sync call with a
``time.sleep`` and asserts a concurrent ticker keeps ticking meanwhile — i.e.
the work really runs in a worker thread.
"""
import asyncio
import time
from unittest.mock import MagicMock, patch
import pytest
BLOCK_S = 0.4
async def _ticks_during(coro) -> int:
"""Run coro while counting 20ms ticks; a blocked loop yields ~0 ticks."""
ticks = 0
done = asyncio.Event()
async def ticker():
nonlocal ticks
while not done.is_set():
await asyncio.sleep(0.02)
ticks += 1
t = asyncio.create_task(ticker())
try:
await coro
finally:
done.set()
await t
return ticks
def _sleep(*_a, **_kw):
time.sleep(BLOCK_S)
return {}
# Blocked for BLOCK_S → ~0-1 ticks; offloaded → ~BLOCK_S/0.02 = 20 ticks.
MIN_TICKS = 8
@pytest.mark.asyncio
async def test_upload_screens_runs_off_loop(tmp_path):
from backend.workers import screens_worker
title = tmp_path / "t.png"
end = tmp_path / "e.png"
title.write_bytes(b"x")
end.write_bytes(b"x")
storage = MagicMock()
storage.upload_file.side_effect = _sleep
ticks = await _ticks_during(
screens_worker._upload_screens("j1", MagicMock(), storage, str(title), str(end))
)
assert storage.upload_file.call_count == 2
assert ticks >= MIN_TICKS
@pytest.mark.asyncio
async def test_review_audio_transcode_runs_off_loop():
from backend.workers import screens_worker
jm = MagicMock()
job = MagicMock()
job.file_urls = {}
job.input_media_gcs_path = None
jm.get_job.return_value = job
with patch("backend.services.audio_transcoding_service.AudioTranscodingService") as svc, \
patch("backend.utils.stems.vocals_stem_path", return_value=None):
svc.return_value.prepare_review_audio_for_job.side_effect = _sleep
ticks = await _ticks_during(
screens_worker._transcode_review_audio("j1", jm, MagicMock())
)
svc.return_value.prepare_review_audio_for_job.assert_called_once()
assert ticks >= MIN_TICKS
@pytest.mark.asyncio
async def test_title_screen_render_runs_off_loop(tmp_path):
from backend.workers import screens_worker
gen = MagicMock()
gen.create_title_video.side_effect = _sleep
style = MagicMock()
style.get_intro_format.return_value = {}
job = MagicMock(artist="A", title="B")
ticks = await _ticks_during(
screens_worker._generate_title_screen("j1", job, gen, style, str(tmp_path))
)
gen.create_title_video.assert_called_once()
assert ticks >= MIN_TICKS
@pytest.mark.asyncio
async def test_style_config_load_runs_off_loop(tmp_path):
from backend.workers import style_helper
job = MagicMock(job_id="j1", style_assets={}, style_params_gcs_path=None)
def slow_load(**_kw):
time.sleep(BLOCK_S)
return (None, {})
with patch.object(style_helper, "load_styles_from_gcs", side_effect=slow_load):
cfg = style_helper.StyleConfig(job, MagicMock(), str(tmp_path))
ticks = await _ticks_during(cfg.load())
assert ticks >= MIN_TICKS
@pytest.mark.asyncio
async def test_stale_review_cron_runs_off_loop():
from backend.workers import stale_review_processor
with patch.object(stale_review_processor, "process_stale_reviews_sync", side_effect=_sleep):
ticks = await _ticks_during(stale_review_processor.process_stale_reviews())
assert ticks >= MIN_TICKS
@pytest.mark.asyncio
async def test_health_encoding_status_builds_service_off_loop():
"""A cold get_encoding_service() (imports compute_v1, ~10s) must not block the loop."""
from backend.api.routes import health
def slow_build():
time.sleep(BLOCK_S)
svc = MagicMock()
svc.is_enabled = False
return svc
with patch.object(health, "get_encoding_service", side_effect=slow_build):
ticks = await _ticks_during(health.check_encoding_worker_status())
assert ticks >= MIN_TICKS
def test_get_encoding_service_builds_once_under_concurrency():
import threading
from backend.services import encoding_service as es
builds = []
class FakeService:
def __init__(self):
builds.append(1)
time.sleep(0.05)
def set_worker_manager(self, m):
pass
with patch.object(es, "_encoding_service", None), patch.object(es, "EncodingService", FakeService):
results = []
threads = [threading.Thread(target=lambda: results.append(es.get_encoding_service())) for _ in range(5)]
for t in threads:
t.start()
for t in threads:
t.join()
assert len(builds) == 1
assert len({id(r) for r in results}) == 1
# --- 2026-10-08: routes/cron stalls traced by the loop watchdog ---------------
# complete_review's corrections upload (21s), create_job_from_search's theme prep
# (5-9s), recover_stuck_jobs' Firestore scans (~260 stalls/week), the YouTube
# queue processor and upload-duration validation all ran sync I/O on the loop.
def test_run_on_loop_runs_coroutine_on_the_given_loop():
from backend.utils.loop_bridge import run_on_loop
async def main():
loop = asyncio.get_running_loop()
seen = {}
async def coro():
seen["loop"] = asyncio.get_running_loop()
return 42
result = await asyncio.to_thread(run_on_loop, loop, coro())
assert result == 42
assert seen["loop"] is loop
asyncio.run(main())
def test_run_on_loop_propagates_exceptions():
from backend.utils.loop_bridge import run_on_loop
async def main():
loop = asyncio.get_running_loop()
async def boom():
raise ValueError("nope")
with pytest.raises(ValueError, match="nope"):
await asyncio.to_thread(run_on_loop, loop, boom())
asyncio.run(main())
def test_run_on_loop_times_out_and_cancels():
from backend.utils.loop_bridge import run_on_loop
import concurrent.futures
async def main():
loop = asyncio.get_running_loop()
cancelled = asyncio.Event()
async def hangs():
try:
await asyncio.sleep(10)
except asyncio.CancelledError:
cancelled.set()
raise
with pytest.raises(concurrent.futures.TimeoutError):
await asyncio.to_thread(run_on_loop, loop, hangs(), 0.05)
await asyncio.wait_for(cancelled.wait(), 1)
asyncio.run(main())
def test_run_in_body_thread_copies_context_and_uses_own_pool():
import contextvars
import threading
from backend.utils.loop_bridge import run_in_body_thread
var = contextvars.ContextVar("v", default=None)
def body():
return var.get(), threading.current_thread().name
async def main():
var.set("request-ctx")
return await run_in_body_thread(body)
value, thread_name = asyncio.run(main())
assert value == "request-ctx"
assert thread_name.startswith("route-body")
@pytest.mark.asyncio
async def test_complete_review_saves_off_loop_and_triggers_render():
from unittest.mock import AsyncMock
from backend.api.routes import review
from backend.models.job import JobStatus
job = MagicMock()
job.status = JobStatus.AWAITING_REVIEW
job.file_urls = {}
job.existing_instrumental_gcs_path = None
job.state_data = {}
jm = MagicMock()
jm.get_job.return_value = job
jm.delete_state_data_keys.return_value = []
storage = MagicMock()
storage.upload_json.side_effect = _sleep
with patch.object(review, "JobManager", return_value=jm), \
patch.object(review, "StorageService", return_value=storage), \
patch("backend.services.worker_service.get_worker_service") as ws:
ws.return_value.trigger_render_video_worker = AsyncMock(return_value=True)
result = {}
async def call():
result.update(await review.complete_review(
job_id="j1",
updated_data={"corrections": [], "instrumental_selection": "clean"},
auth_info=("u@example.com", "job_owner"),
))
ticks = await _ticks_during(call())
storage.upload_json.assert_called_once()
ws.return_value.trigger_render_video_worker.assert_awaited_once_with("j1")
assert result == {"status": "success", "instrumental_selection": "clean"}
assert ticks >= MIN_TICKS
@pytest.mark.asyncio
async def test_create_job_from_search_runs_off_loop():
from backend.api.routes import jobs
with patch.object(jobs, "_create_job_from_search_sync", side_effect=_sleep) as body:
ticks = await _ticks_during(
jobs.create_job_from_search(MagicMock(), MagicMock(), MagicMock(), MagicMock())
)
body.assert_called_once()
assert isinstance(body.call_args.args[-1], asyncio.AbstractEventLoop)
assert ticks >= MIN_TICKS
@pytest.mark.asyncio
@pytest.mark.parametrize("route, body", [
("recover_stuck_jobs", "_recover_stuck_jobs_sync"),
("retry_pending_render_jobs", "_retry_pending_render_jobs_sync"),
])
async def test_internal_crons_run_off_loop(route, body):
from backend.api.routes import internal
with patch.object(internal, body, side_effect=_sleep) as mock_body:
ticks = await _ticks_during(getattr(internal, route)(MagicMock(), MagicMock()))
mock_body.assert_called_once()
assert ticks >= MIN_TICKS
@pytest.mark.asyncio
async def test_upload_duration_validation_runs_off_loop():
from backend.api.routes import file_upload
with patch.object(file_upload, "_validate_audio_durations_sync", side_effect=_sleep):
ticks = await _ticks_during(file_upload._validate_audio_durations(MagicMock(), "a.flac", "i.flac"))
assert ticks >= MIN_TICKS
@pytest.mark.asyncio
async def test_youtube_queue_single_upload_runs_off_loop():
from backend.workers import youtube_queue_processor as qp
with patch.object(qp, "_process_single_upload_sync", side_effect=_sleep):
ticks = await _ticks_during(qp._process_single_upload("j1", {}, MagicMock(), MagicMock()))
assert ticks >= MIN_TICKS
@pytest.mark.asyncio
async def test_youtube_queue_quota_and_queue_reads_run_off_loop():
from backend.workers import youtube_queue_processor as qp
quota = MagicMock()
quota.check_quota_available.side_effect = lambda: (_sleep(), (True, 100, "ok"))[1]
queue = MagicMock()
queue.get_queued_uploads.side_effect = lambda **_kw: (_sleep(), [])[1]
with patch.object(qp, "get_youtube_quota_service", return_value=quota), \
patch.object(qp, "get_youtube_upload_queue_service", return_value=queue):
ticks = await _ticks_during(qp.process_youtube_upload_queue())
# 2 x BLOCK_S of blocking work; offloaded it ticks through both.
assert ticks >= 2 * MIN_TICKS
@pytest.mark.asyncio
async def test_cloud_run_job_dispatch_runs_off_loop():
from backend.services.worker_service import WorkerService
client = MagicMock()
client.run_job.side_effect = _sleep
ws = WorkerService.__new__(WorkerService)
ticks = await _ticks_during(ws._run_job_with_retry(client, MagicMock(), log_prefix="[t]"))
client.run_job.assert_called_once()
assert ticks >= MIN_TICKS
@pytest.mark.asyncio
async def test_render_trigger_bumps_generation_off_loop():
from unittest.mock import AsyncMock
from backend.services.worker_service import WorkerService
ws = WorkerService.__new__(WorkerService)
ws._use_cloud_tasks = False
ws.settings = MagicMock(use_cloud_run_jobs_for_render=False)
with patch.object(WorkerService, "_bump_worker_generation", side_effect=_sleep), \
patch.object(WorkerService, "_start_encoding_worker_warmup"), \
patch.object(WorkerService, "trigger_worker", new=AsyncMock(return_value=True)):
ticks = await _ticks_during(ws.trigger_render_video_worker("j1"))
assert ticks >= MIN_TICKS