mirror of
https://github.com/nomadkaraoke/karaoke-gen.git
synced 2026-11-20 13:07:57 +09:00
## Summary Investigation of two 2026-10-08 Discord alerts (plus a third that arrived mid-session) turned up four issues: 1. **Credit-refund loop.** A brand-new welcome-credit account received a finished video (job 8a10a22d), clicked **Edit** (status back to `awaiting_review`), then **deleted** the job and got its credit back. `delete_job` refunds any non-complete status, and Edit makes a delivered job look "never completed". 2. **YouTube 410 "Gone" dropped the upload.** The resumable session was lost 1.3s into the upload. `upload_video` called `execute()` once with no retries, and the orchestrator only queued quota errors, so any other failure lost the upload. 3. **EVENT_LOOP_STALL 21s in `upload_json`.** `complete_review` (an async route) wrote the corrections JSON to GCS directly on the event loop. Ranking a week of watchdog stacks by route showed the same pattern in `recover_stuck_jobs` (~260 stalls/week), the YouTube queue processor (its `_process_single_upload` ran a whole upload on the loop), `create_job_from_search` (theme prep), `_validate_audio_durations` (15s), and the sync `run_job` / generation bump inside worker triggers. 4. **False 30s "stall" in asyncio `get_debug`.** Cloud Run CPU throttling froze the whole instance mid-callback with no request in flight. The watchdog only discounted freezes where the loop was parked in `select()`. ## Changes - **Refund:** `_refund_credit_for_job` skips jobs with `edit_count > 0` (already delivered). This covers delete, cancel, fail and the 48h review auto-expiry. - **YouTube:** on 404/410 the upload restarts with a fresh session (3 attempts, with backoff). `execute(num_retries=3)` lets googleapiclient resume chunks on 5xx/429 within a session. Any other failure is queued in `youtube_upload_queue` (`reason="upload_error"`) for the existing processor, which retries up to 5 times. Admins still get a Discord warning: "queued for automatic retry". - **Event loop:** - New `backend/utils/loop_bridge.py`: `run_in_body_thread` runs sync route bodies on a dedicated 16-thread pool, so they can't exhaust the default executor the dispatched coroutines need. `run_on_loop` dispatches coroutines back to the API loop (`run_coroutine_threadsafe`, 120s timeout). - `complete_review`: its sync save runs via `to_thread`, and the render trigger stays on the loop. - `create_job_from_search`, `recover_stuck_jobs` and `retry_pending_render_jobs`: bodies run in the route-body pool. `_retrigger_parked_download` now does its writes in that thread. - `_validate_audio_durations` and the YouTube queue's `_process_single_upload` keep their async signatures but run in threads. The queue's quota and Firestore calls go through `to_thread`. - `worker_service`: the generation bump and `client.run_job` go through `to_thread`. - **Watchdog:** if the watchdog thread itself oversleeps by 2s or more while the process uses less than 25% CPU, the whole process was frozen. That time is subtracted from the stall and isn't sampled. Blocking I/O (the watchdog still wakes on time) and GIL-hogging CPU work (CPU keeps advancing) are still reported. - Docs: `API.md` (edit/refund note) and `LESSONS-LEARNED.md` (mostly-sync routes that await one trigger; throttling freezes). ## Testing - Backend unit suite: **5863 passed**, 36 skipped (`pytest backend/tests --ignore=backend/tests/emulator`). - New tests: - edited-track refund cases (delete in awaiting_review / lyrics_complete, cancel); - YouTube restart on 410, give up after 3 attempts, no restart on 403; - orchestrator queues failed uploads and warns when queueing fails; - loop-responsiveness ticker tests for each offloaded path; - `run_on_loop` result, exception and timeout-with-cancel; - `run_in_body_thread` copies context and uses its own pool; - watchdog: freeze ignored, I/O stall and CPU hog still reported, freeze subtracted from a real stall. - Local CodeRabbit: 3 cycles. Fixed the nested-executor deadlock risk, the missing `run_on_loop` timeout, the parked-download writes on the loop, and a coroutine leak. The last cycle had 0 findings. @coderabbitai ignore 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>