mirror of
https://github.com/nomadkaraoke/karaoke-gen.git
synced 2026-11-20 13:07:57 +09:00
## Summary Investigation of two 2026-10-08 Discord alerts (plus a third that arrived mid-session) turned up four issues: 1. **Credit-refund loop.** A brand-new welcome-credit account received a finished video (job 8a10a22d), clicked **Edit** (status back to `awaiting_review`), then **deleted** the job and got its credit back. `delete_job` refunds any non-complete status, and Edit makes a delivered job look "never completed". 2. **YouTube 410 "Gone" dropped the upload.** The resumable session was lost 1.3s into the upload. `upload_video` called `execute()` once with no retries, and the orchestrator only queued quota errors, so any other failure lost the upload. 3. **EVENT_LOOP_STALL 21s in `upload_json`.** `complete_review` (an async route) wrote the corrections JSON to GCS directly on the event loop. Ranking a week of watchdog stacks by route showed the same pattern in `recover_stuck_jobs` (~260 stalls/week), the YouTube queue processor (its `_process_single_upload` ran a whole upload on the loop), `create_job_from_search` (theme prep), `_validate_audio_durations` (15s), and the sync `run_job` / generation bump inside worker triggers. 4. **False 30s "stall" in asyncio `get_debug`.** Cloud Run CPU throttling froze the whole instance mid-callback with no request in flight. The watchdog only discounted freezes where the loop was parked in `select()`. ## Changes - **Refund:** `_refund_credit_for_job` skips jobs with `edit_count > 0` (already delivered). This covers delete, cancel, fail and the 48h review auto-expiry. - **YouTube:** on 404/410 the upload restarts with a fresh session (3 attempts, with backoff). `execute(num_retries=3)` lets googleapiclient resume chunks on 5xx/429 within a session. Any other failure is queued in `youtube_upload_queue` (`reason="upload_error"`) for the existing processor, which retries up to 5 times. Admins still get a Discord warning: "queued for automatic retry". - **Event loop:** - New `backend/utils/loop_bridge.py`: `run_in_body_thread` runs sync route bodies on a dedicated 16-thread pool, so they can't exhaust the default executor the dispatched coroutines need. `run_on_loop` dispatches coroutines back to the API loop (`run_coroutine_threadsafe`, 120s timeout). - `complete_review`: its sync save runs via `to_thread`, and the render trigger stays on the loop. - `create_job_from_search`, `recover_stuck_jobs` and `retry_pending_render_jobs`: bodies run in the route-body pool. `_retrigger_parked_download` now does its writes in that thread. - `_validate_audio_durations` and the YouTube queue's `_process_single_upload` keep their async signatures but run in threads. The queue's quota and Firestore calls go through `to_thread`. - `worker_service`: the generation bump and `client.run_job` go through `to_thread`. - **Watchdog:** if the watchdog thread itself oversleeps by 2s or more while the process uses less than 25% CPU, the whole process was frozen. That time is subtracted from the stall and isn't sampled. Blocking I/O (the watchdog still wakes on time) and GIL-hogging CPU work (CPU keeps advancing) are still reported. - Docs: `API.md` (edit/refund note) and `LESSONS-LEARNED.md` (mostly-sync routes that await one trigger; throttling freezes). ## Testing - Backend unit suite: **5863 passed**, 36 skipped (`pytest backend/tests --ignore=backend/tests/emulator`). - New tests: - edited-track refund cases (delete in awaiting_review / lyrics_complete, cancel); - YouTube restart on 410, give up after 3 attempts, no restart on 403; - orchestrator queues failed uploads and warns when queueing fails; - loop-responsiveness ticker tests for each offloaded path; - `run_on_loop` result, exception and timeout-with-cancel; - `run_in_body_thread` copies context and uses its own pool; - watchdog: freeze ignored, I/O stall and CPU hog still reported, freeze subtracted from a real stall. - Local CodeRabbit: 3 cycles. Fixed the nested-executor deadlock risk, the missing `run_on_loop` timeout, the parked-download writes on the loop, and a coroutine leak. The last cycle had 0 findings. @coderabbitai ignore 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
59 lines
2.4 KiB
Python
59 lines
2.4 KiB
Python
"""Run a blocking route body in a worker thread while its coroutines still run on the loop.
|
|
|
|
The API has one event loop per Cloud Run instance, so sync Firestore / GCS calls
|
|
inside an ``async def`` route freeze every request on the instance (see
|
|
``backend/services/loop_watchdog.py``). Routes that are mostly sync I/O but must
|
|
await a few coroutines (worker triggers) do::
|
|
|
|
return await run_in_body_thread(_do_work_sync, ..., asyncio.get_running_loop())
|
|
|
|
and, inside ``_do_work_sync``, ``run_on_loop(loop, worker_service.trigger_x(job_id))``.
|
|
The coroutine runs on the API loop (so loop APIs like ``create_task`` inside it
|
|
keep working) while the worker thread waits for its result.
|
|
"""
|
|
import asyncio
|
|
import concurrent.futures
|
|
import contextvars
|
|
import functools
|
|
from typing import Any, Callable, Coroutine, TypeVar
|
|
|
|
T = TypeVar("T")
|
|
|
|
# Route bodies block in run_on_loop while the coroutine they dispatched may itself
|
|
# need asyncio.to_thread (the loop's default executor). Running the bodies on
|
|
# their own pool means they can never use up the threads those coroutines wait for.
|
|
_BODY_EXECUTOR = concurrent.futures.ThreadPoolExecutor(max_workers=16, thread_name_prefix="route-body")
|
|
|
|
# Worker triggers finish in seconds (bounded run_job retries); anything this slow
|
|
# is hung, and the cron/route should fail rather than hold its thread forever.
|
|
RUN_ON_LOOP_TIMEOUT_SECONDS = 120
|
|
|
|
|
|
async def run_in_body_thread(func: Callable[..., T], *args: Any) -> T:
|
|
"""Run a sync route body on the dedicated route-body pool and await its result.
|
|
|
|
Like ``asyncio.to_thread``, the caller's contextvars (trace / tenant / request
|
|
context) are copied into the thread.
|
|
"""
|
|
loop = asyncio.get_running_loop()
|
|
ctx = contextvars.copy_context()
|
|
return await loop.run_in_executor(_BODY_EXECUTOR, functools.partial(ctx.run, func, *args))
|
|
|
|
|
|
def run_on_loop(
|
|
loop: asyncio.AbstractEventLoop,
|
|
coro: Coroutine[Any, Any, T],
|
|
timeout: float = RUN_ON_LOOP_TIMEOUT_SECONDS,
|
|
) -> T:
|
|
"""From a worker thread, run ``coro`` on ``loop`` and return (or raise) its result."""
|
|
try:
|
|
future = asyncio.run_coroutine_threadsafe(coro, loop)
|
|
except Exception:
|
|
coro.close() # loop closed (instance shutdown) — don't leak a never-awaited coroutine
|
|
raise
|
|
try:
|
|
return future.result(timeout=timeout)
|
|
except concurrent.futures.TimeoutError:
|
|
future.cancel()
|
|
raise
|