mirror of
https://github.com/nomadkaraoke/karaoke-gen.git
synced 2026-11-20 13:07:57 +09:00
## Summary Fixes the 2026-10-29 #alerts event-loop stall batch (handoff `docs/archive/2026-10-29-gen-event-loop-stalls-handoff.md` in the workspace). **1. Visibility change off the loop (39s stall, 13 requests incl. /api/health)** - `change_to_private` now runs claim, flag write, `redistribute_video`, public-output delete and the final commit as **one** `asyncio.to_thread` call (`_redistribute_and_commit_private`). `redistribute_video` runs on that thread's own loop through the new `redistribute_video_off_loop` (`asyncio.run(with_notification_drain(...))`). - Is anything bound to the main loop? No. `redistribute_video` builds its own `JobManager`/`StorageService`. The Dropbox, YouTube and GDrive clients are sync. Job-log writes go through `log_write_offload`'s writer thread. Notification tasks scheduled via `_schedule_notification` would land on the private loop, so `with_notification_drain` drains them before the loop closes, as the Cloud Run Job entry points do. - Why one thread call and not just the redistribution (from CodeRabbit): a thread can't be cancelled. So a cancelled request can no longer leave a committed private archive with its public outputs still live, or a claimed flag with no redistribution. - Guard: `redistribute_video` is added to `BLOCKING_CALLS`, and `visibility_change_service.change_to_private`/`change_to_public` are added to `AWAITED_ROUTE_HELPERS`. I checked that putting back `await redistribute_video(...)` fails the test. - **Double Dropbox upload confirmed and fixed.** GCS `finals/` holds every output twice: short canonical keys written by `_upload_results` (`lossy_4k_mp4.mp4`, …) and the readable encoder names (verified on job 0f211e97: 17 objects / 525 MiB). Redistribution uploaded both. The short-named `lossless_4k_mp4.mp4` also slipped past the `" (Final Karaoke Lossless 4k).mp4"` Dropbox exclusion. `_dedupe_redistribution_files` now keeps one copy per final, under its readable name: - a short name whose readable twin exists (twin found by suffix) is dropped; - a short name with no twin is renamed to the readable name (CDG/TXT zips included). **2. First-request stalls on new instances** - Lifespan builds FastAPI's lazy route contexts (`fastapi.routing.iter_route_contexts` + `effective_low_priority_routes`) in a thread before readiness, with a timeout and best-effort error handling. The results are cached by routes version under a lock, so the OTel `_get_route_details` walk no longer compiles route regexes on the loop. It is a no-op on FastAPI versions without these internals. Note: the lock pins 0.128 but prod installs unpinned (`pip install -e .`), so prod runs a newer FastAPI. I checked against FastAPI 0.162: the first walk builds 304 contexts, the second walk and the OTel lookup are cached (<1ms). - `flacfetch.core.manager`, `flacfetch.downloaders.youtube` and `yt_dlp` are added to `_LAZY_ROUTE_IMPORTS`. - `search_audio` auto_download: `select_best` now goes through `asyncio.to_thread`, with `select_best` added to `BLOCKING_CALLS`. **3. Loop watchdog false positive** - `_finish` logs WARNING, not ERROR, when all of these hold: no non-idle sample was taken, a process freeze was observed (`frozen_s > 0`), and no request was in flight when the stall began. Every "affected" request therefore arrived while the instance was thawing. An unsampled stall with no observed freeze, or with a request in flight at its start, stays ERROR. The freeze requirement came from CodeRabbit. **4. Cloud Trace exporter noise** - `cloud_trace_export` ignore pattern for `Error while writing to Cloud Trace`. CodeRabbit suggested matching only the transient gRPC detail. I didn't: the error monitor only sees `jsonPayload.message` (the exception is a separate field), and a persistent loss of telemetry isn't #alerts-worthy anyway. Smaller change: `change_to_private`'s Firestore `job_ref.update` writes now also run off the loop. ## Testing - New `backend/tests/test_redistribute_video.py`: - dedupe unit tests; - an end-to-end `redistribute_video` run with mocks, checking the files the Dropbox stage sees; - the off-loop runner uses its calling thread; - `change_to_private` runs the redistribution on another thread and loop; - a cancelled request still deletes the public outputs. - `test_loop_watchdog.py`: frozen + unsampled + late arrival gives WARNING; a request in flight at the start gives ERROR; no observed freeze gives ERROR; a sampled stall with a late arrival gives ERROR. - `test_main_startup_warmup.py`: lazy imports list; the route-context walk; no-op without the API; failure is non-fatal; lifespan warms off-loop before readiness. - `test_known_issues.py`: the Cloud Trace pattern matches, and other trace errors are not ignored. - Local runs: `pytest backend/tests/ tests/unit/services/error_monitor --ignore=backend/tests/emulator`: 7369 passed, 29 skipped. Not counted in that total: `test_emulator_integration.py` fails 8/11 the same way on the base commit, because a stray emulator from another session is on :8080 locally. That's environmental, not caused by this PR. - Local CodeRabbit: 3 cycles. Cycle 1: 3 findings (2 fixed, 1 skipped with the reason above). Cycle 2: 1 finding, fixed. Cycle 3: 0 findings, but CodeRabbit reported it reused cached results rather than a fresh review. ## Not verified - No prod run of a real public→private change yet. After deploy, the Dropbox folder of the next one should contain each final exactly once, and no `EVENT_LOOP_STALL` should appear during it. - Route-context warmup was checked only against FastAPI 0.162 in a scratch venv, not the exact prod version. Watch for the `Warmed N route contexts` log line on the next cold start. - Existing private Dropbox folders that already hold the duplicates are not cleaned up by this PR. Alert: https://discord.com/channels/1198691101943541770/1265517528608669707/1563719390849929330 Alert: https://discord.com/channels/1198691101943541770/1265517528608669707/1563009782363721760 Alert: https://discord.com/channels/1198691101943541770/1265517528608669707/1563039889354784882 Alert: https://discord.com/channels/1198691101943541770/1265517528608669707/1563221363278348514 Alert: https://discord.com/channels/1198691101943541770/1265517528608669707/1564308425033252895 Alert: https://discord.com/channels/1198691101943541770/1265517528608669707/1563160849336762449 @coderabbitai ignore 🤖 Generated with [Claude Code](https://claude.com/claude-code) https://claude.ai/code/session_012RYLVTiUPKxn3sQPyp3Teh --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>