Files
karaoke-gen/backend/utils
Andrew Beveridge ☄️andClaude Opus 5.5 f2612ac110 fix: close edit→delete refund loop, retry lost YouTube uploads, unblock event loop (v0.268.1) (#1135)
## Summary
Investigation of two 2026-10-08 Discord alerts (plus a third that
arrived mid-session) turned up four issues:

1. **Credit-refund loop.** A brand-new welcome-credit account received a
finished video (job 8a10a22d), clicked **Edit** (status back to
`awaiting_review`), then **deleted** the job and got its credit back.
`delete_job` refunds any non-complete status, and Edit makes a delivered
job look "never completed".
2. **YouTube 410 "Gone" dropped the upload.** The resumable session was
lost 1.3s into the upload. `upload_video` called `execute()` once with
no retries, and the orchestrator only queued quota errors, so any other
failure lost the upload.
3. **EVENT_LOOP_STALL 21s in `upload_json`.** `complete_review` (an
async route) wrote the corrections JSON to GCS directly on the event
loop. Ranking a week of watchdog stacks by route showed the same pattern
in `recover_stuck_jobs` (~260 stalls/week), the YouTube queue processor
(its `_process_single_upload` ran a whole upload on the loop),
`create_job_from_search` (theme prep), `_validate_audio_durations`
(15s), and the sync `run_job` / generation bump inside worker triggers.
4. **False 30s "stall" in asyncio `get_debug`.** Cloud Run CPU
throttling froze the whole instance mid-callback with no request in
flight. The watchdog only discounted freezes where the loop was parked
in `select()`.

## Changes
- **Refund:** `_refund_credit_for_job` skips jobs with `edit_count > 0`
(already delivered). This covers delete, cancel, fail and the 48h review
auto-expiry.
- **YouTube:** on 404/410 the upload restarts with a fresh session (3
attempts, with backoff). `execute(num_retries=3)` lets googleapiclient
resume chunks on 5xx/429 within a session. Any other failure is queued
in `youtube_upload_queue` (`reason="upload_error"`) for the existing
processor, which retries up to 5 times. Admins still get a Discord
warning: "queued for automatic retry".
- **Event loop:**
- New `backend/utils/loop_bridge.py`: `run_in_body_thread` runs sync
route bodies on a dedicated 16-thread pool, so they can't exhaust the
default executor the dispatched coroutines need. `run_on_loop`
dispatches coroutines back to the API loop (`run_coroutine_threadsafe`,
120s timeout).
- `complete_review`: its sync save runs via `to_thread`, and the render
trigger stays on the loop.
- `create_job_from_search`, `recover_stuck_jobs` and
`retry_pending_render_jobs`: bodies run in the route-body pool.
`_retrigger_parked_download` now does its writes in that thread.
- `_validate_audio_durations` and the YouTube queue's
`_process_single_upload` keep their async signatures but run in threads.
The queue's quota and Firestore calls go through `to_thread`.
- `worker_service`: the generation bump and `client.run_job` go through
`to_thread`.
- **Watchdog:** if the watchdog thread itself oversleeps by 2s or more
while the process uses less than 25% CPU, the whole process was frozen.
That time is subtracted from the stall and isn't sampled. Blocking I/O
(the watchdog still wakes on time) and GIL-hogging CPU work (CPU keeps
advancing) are still reported.
- Docs: `API.md` (edit/refund note) and `LESSONS-LEARNED.md`
(mostly-sync routes that await one trigger; throttling freezes).

## Testing
- Backend unit suite: **5863 passed**, 36 skipped (`pytest backend/tests
--ignore=backend/tests/emulator`).
- New tests:
- edited-track refund cases (delete in awaiting_review /
lyrics_complete, cancel);
  - YouTube restart on 410, give up after 3 attempts, no restart on 403;
  - orchestrator queues failed uploads and warns when queueing fails;
  - loop-responsiveness ticker tests for each offloaded path;
  - `run_on_loop` result, exception and timeout-with-cancel;
  - `run_in_body_thread` copies context and uses its own pool;
- watchdog: freeze ignored, I/O stall and CPU hog still reported, freeze
subtracted from a real stall.
- Local CodeRabbit: 3 cycles. Fixed the nested-executor deadlock risk,
the missing `run_on_loop` timeout, the parked-download writes on the
loop, and a coroutine leak. The last cycle had 0 findings.

@coderabbitai ignore

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-08 16:15:47 -04:00
..