Files
Andrew Beveridge ☄️andClaude Opus 5.5 f2612ac110 fix: close edit→delete refund loop, retry lost YouTube uploads, unblock event loop (v0.268.1) (#1135)
## Summary
Investigation of two 2026-10-08 Discord alerts (plus a third that
arrived mid-session) turned up four issues:

1. **Credit-refund loop.** A brand-new welcome-credit account received a
finished video (job 8a10a22d), clicked **Edit** (status back to
`awaiting_review`), then **deleted** the job and got its credit back.
`delete_job` refunds any non-complete status, and Edit makes a delivered
job look "never completed".
2. **YouTube 410 "Gone" dropped the upload.** The resumable session was
lost 1.3s into the upload. `upload_video` called `execute()` once with
no retries, and the orchestrator only queued quota errors, so any other
failure lost the upload.
3. **EVENT_LOOP_STALL 21s in `upload_json`.** `complete_review` (an
async route) wrote the corrections JSON to GCS directly on the event
loop. Ranking a week of watchdog stacks by route showed the same pattern
in `recover_stuck_jobs` (~260 stalls/week), the YouTube queue processor
(its `_process_single_upload` ran a whole upload on the loop),
`create_job_from_search` (theme prep), `_validate_audio_durations`
(15s), and the sync `run_job` / generation bump inside worker triggers.
4. **False 30s "stall" in asyncio `get_debug`.** Cloud Run CPU
throttling froze the whole instance mid-callback with no request in
flight. The watchdog only discounted freezes where the loop was parked
in `select()`.

## Changes
- **Refund:** `_refund_credit_for_job` skips jobs with `edit_count > 0`
(already delivered). This covers delete, cancel, fail and the 48h review
auto-expiry.
- **YouTube:** on 404/410 the upload restarts with a fresh session (3
attempts, with backoff). `execute(num_retries=3)` lets googleapiclient
resume chunks on 5xx/429 within a session. Any other failure is queued
in `youtube_upload_queue` (`reason="upload_error"`) for the existing
processor, which retries up to 5 times. Admins still get a Discord
warning: "queued for automatic retry".
- **Event loop:**
- New `backend/utils/loop_bridge.py`: `run_in_body_thread` runs sync
route bodies on a dedicated 16-thread pool, so they can't exhaust the
default executor the dispatched coroutines need. `run_on_loop`
dispatches coroutines back to the API loop (`run_coroutine_threadsafe`,
120s timeout).
- `complete_review`: its sync save runs via `to_thread`, and the render
trigger stays on the loop.
- `create_job_from_search`, `recover_stuck_jobs` and
`retry_pending_render_jobs`: bodies run in the route-body pool.
`_retrigger_parked_download` now does its writes in that thread.
- `_validate_audio_durations` and the YouTube queue's
`_process_single_upload` keep their async signatures but run in threads.
The queue's quota and Firestore calls go through `to_thread`.
- `worker_service`: the generation bump and `client.run_job` go through
`to_thread`.
- **Watchdog:** if the watchdog thread itself oversleeps by 2s or more
while the process uses less than 25% CPU, the whole process was frozen.
That time is subtracted from the stall and isn't sampled. Blocking I/O
(the watchdog still wakes on time) and GIL-hogging CPU work (CPU keeps
advancing) are still reported.
- Docs: `API.md` (edit/refund note) and `LESSONS-LEARNED.md`
(mostly-sync routes that await one trigger; throttling freezes).

## Testing
- Backend unit suite: **5863 passed**, 36 skipped (`pytest backend/tests
--ignore=backend/tests/emulator`).
- New tests:
- edited-track refund cases (delete in awaiting_review /
lyrics_complete, cancel);
  - YouTube restart on 410, give up after 3 attempts, no restart on 403;
  - orchestrator queues failed uploads and warns when queueing fails;
  - loop-responsiveness ticker tests for each offloaded path;
  - `run_on_loop` result, exception and timeout-with-cancel;
  - `run_in_body_thread` copies context and uses its own pool;
- watchdog: freeze ignored, I/O stall and CPU hog still reported, freeze
subtracted from a real stall.
- Local CodeRabbit: 3 cycles. Fixed the nested-executor deadlock risk,
the missing `run_on_loop` timeout, the parked-download writes on the
loop, and a coroutine leak. The last cycle had 0 findings.

@coderabbitai ignore

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-08 16:15:47 -04:00

59 lines
2.4 KiB
Python

"""Run a blocking route body in a worker thread while its coroutines still run on the loop.
The API has one event loop per Cloud Run instance, so sync Firestore / GCS calls
inside an ``async def`` route freeze every request on the instance (see
``backend/services/loop_watchdog.py``). Routes that are mostly sync I/O but must
await a few coroutines (worker triggers) do::
return await run_in_body_thread(_do_work_sync, ..., asyncio.get_running_loop())
and, inside ``_do_work_sync``, ``run_on_loop(loop, worker_service.trigger_x(job_id))``.
The coroutine runs on the API loop (so loop APIs like ``create_task`` inside it
keep working) while the worker thread waits for its result.
"""
import asyncio
import concurrent.futures
import contextvars
import functools
from typing import Any, Callable, Coroutine, TypeVar
T = TypeVar("T")
# Route bodies block in run_on_loop while the coroutine they dispatched may itself
# need asyncio.to_thread (the loop's default executor). Running the bodies on
# their own pool means they can never use up the threads those coroutines wait for.
_BODY_EXECUTOR = concurrent.futures.ThreadPoolExecutor(max_workers=16, thread_name_prefix="route-body")
# Worker triggers finish in seconds (bounded run_job retries); anything this slow
# is hung, and the cron/route should fail rather than hold its thread forever.
RUN_ON_LOOP_TIMEOUT_SECONDS = 120
async def run_in_body_thread(func: Callable[..., T], *args: Any) -> T:
"""Run a sync route body on the dedicated route-body pool and await its result.
Like ``asyncio.to_thread``, the caller's contextvars (trace / tenant / request
context) are copied into the thread.
"""
loop = asyncio.get_running_loop()
ctx = contextvars.copy_context()
return await loop.run_in_executor(_BODY_EXECUTOR, functools.partial(ctx.run, func, *args))
def run_on_loop(
loop: asyncio.AbstractEventLoop,
coro: Coroutine[Any, Any, T],
timeout: float = RUN_ON_LOOP_TIMEOUT_SECONDS,
) -> T:
"""From a worker thread, run ``coro`` on ``loop`` and return (or raise) its result."""
try:
future = asyncio.run_coroutine_threadsafe(coro, loop)
except Exception:
coro.close() # loop closed (instance shutdown) — don't leak a never-awaited coroutine
raise
try:
return future.result(timeout=timeout)
except concurrent.futures.TimeoutError:
future.cancel()
raise