Files
karaoke-gen/tests
Andrew Beveridge ☄️andClaude Opus 5.5 0d817509fa fix(api): event-loop stalls — visibility redistribution off-loop, route warmup, watchdog/Cloud Trace noise (0.286.2) (#1230)
## Summary

Fixes the 2026-10-29 #alerts event-loop stall batch (handoff
`docs/archive/2026-10-29-gen-event-loop-stalls-handoff.md` in the
workspace).

**1. Visibility change off the loop (39s stall, 13 requests incl.
/api/health)**
- `change_to_private` now runs claim, flag write, `redistribute_video`,
public-output delete and the final commit as **one** `asyncio.to_thread`
call (`_redistribute_and_commit_private`). `redistribute_video` runs on
that thread's own loop through the new `redistribute_video_off_loop`
(`asyncio.run(with_notification_drain(...))`).
- Is anything bound to the main loop? No. `redistribute_video` builds
its own `JobManager`/`StorageService`. The Dropbox, YouTube and GDrive
clients are sync. Job-log writes go through `log_write_offload`'s writer
thread. Notification tasks scheduled via `_schedule_notification` would
land on the private loop, so `with_notification_drain` drains them
before the loop closes, as the Cloud Run Job entry points do.
- Why one thread call and not just the redistribution (from CodeRabbit):
a thread can't be cancelled. So a cancelled request can no longer leave
a committed private archive with its public outputs still live, or a
claimed flag with no redistribution.
- Guard: `redistribute_video` is added to `BLOCKING_CALLS`, and
`visibility_change_service.change_to_private`/`change_to_public` are
added to `AWAITED_ROUTE_HELPERS`. I checked that putting back `await
redistribute_video(...)` fails the test.
- **Double Dropbox upload confirmed and fixed.** GCS `finals/` holds
every output twice: short canonical keys written by `_upload_results`
(`lossy_4k_mp4.mp4`, …) and the readable encoder names (verified on job
0f211e97: 17 objects / 525 MiB). Redistribution uploaded both. The
short-named `lossless_4k_mp4.mp4` also slipped past the `" (Final
Karaoke Lossless 4k).mp4"` Dropbox exclusion.
`_dedupe_redistribution_files` now keeps one copy per final, under its
readable name:
- a short name whose readable twin exists (twin found by suffix) is
dropped;
- a short name with no twin is renamed to the readable name (CDG/TXT
zips included).

**2. First-request stalls on new instances**
- Lifespan builds FastAPI's lazy route contexts
(`fastapi.routing.iter_route_contexts` +
`effective_low_priority_routes`) in a thread before readiness, with a
timeout and best-effort error handling. The results are cached by routes
version under a lock, so the OTel `_get_route_details` walk no longer
compiles route regexes on the loop. It is a no-op on FastAPI versions
without these internals. Note: the lock pins 0.128 but prod installs
unpinned (`pip install -e .`), so prod runs a newer FastAPI. I checked
against FastAPI 0.162: the first walk builds 304 contexts, the second
walk and the OTel lookup are cached (<1ms).
- `flacfetch.core.manager`, `flacfetch.downloaders.youtube` and `yt_dlp`
are added to `_LAZY_ROUTE_IMPORTS`.
- `search_audio` auto_download: `select_best` now goes through
`asyncio.to_thread`, with `select_best` added to `BLOCKING_CALLS`.

**3. Loop watchdog false positive**
- `_finish` logs WARNING, not ERROR, when all of these hold: no non-idle
sample was taken, a process freeze was observed (`frozen_s > 0`), and no
request was in flight when the stall began. Every "affected" request
therefore arrived while the instance was thawing. An unsampled stall
with no observed freeze, or with a request in flight at its start, stays
ERROR. The freeze requirement came from CodeRabbit.

**4. Cloud Trace exporter noise**
- `cloud_trace_export` ignore pattern for `Error while writing to Cloud
Trace`. CodeRabbit suggested matching only the transient gRPC detail. I
didn't: the error monitor only sees `jsonPayload.message` (the exception
is a separate field), and a persistent loss of telemetry isn't
#alerts-worthy anyway.

Smaller change: `change_to_private`'s Firestore `job_ref.update` writes
now also run off the loop.

## Testing
- New `backend/tests/test_redistribute_video.py`:
  - dedupe unit tests;
- an end-to-end `redistribute_video` run with mocks, checking the files
the Dropbox stage sees;
  - the off-loop runner uses its calling thread;
- `change_to_private` runs the redistribution on another thread and
loop;
  - a cancelled request still deletes the public outputs.
- `test_loop_watchdog.py`: frozen + unsampled + late arrival gives
WARNING; a request in flight at the start gives ERROR; no observed
freeze gives ERROR; a sampled stall with a late arrival gives ERROR.
- `test_main_startup_warmup.py`: lazy imports list; the route-context
walk; no-op without the API; failure is non-fatal; lifespan warms
off-loop before readiness.
- `test_known_issues.py`: the Cloud Trace pattern matches, and other
trace errors are not ignored.
- Local runs: `pytest backend/tests/ tests/unit/services/error_monitor
--ignore=backend/tests/emulator`: 7369 passed, 29 skipped. Not counted
in that total: `test_emulator_integration.py` fails 8/11 the same way on
the base commit, because a stray emulator from another session is on
:8080 locally. That's environmental, not caused by this PR.
- Local CodeRabbit: 3 cycles. Cycle 1: 3 findings (2 fixed, 1 skipped
with the reason above). Cycle 2: 1 finding, fixed. Cycle 3: 0 findings,
but CodeRabbit reported it reused cached results rather than a fresh
review.

## Not verified
- No prod run of a real public→private change yet. After deploy, the
Dropbox folder of the next one should contain each final exactly once,
and no `EVENT_LOOP_STALL` should appear during it.
- Route-context warmup was checked only against FastAPI 0.162 in a
scratch venv, not the exact prod version. Watch for the `Warmed N route
contexts` log line on the next cold start.
- Existing private Dropbox folders that already hold the duplicates are
not cleaned up by this PR.

Alert:
https://discord.com/channels/1198691101943541770/1265517528608669707/1563719390849929330
Alert:
https://discord.com/channels/1198691101943541770/1265517528608669707/1563009782363721760
Alert:
https://discord.com/channels/1198691101943541770/1265517528608669707/1563039889354784882
Alert:
https://discord.com/channels/1198691101943541770/1265517528608669707/1563221363278348514
Alert:
https://discord.com/channels/1198691101943541770/1265517528608669707/1564308425033252895
Alert:
https://discord.com/channels/1198691101943541770/1265517528608669707/1563160849336762449

@coderabbitai ignore

🤖 Generated with [Claude Code](https://claude.com/claude-code)

https://claude.ai/code/session_012RYLVTiUPKxn3sQPyp3Teh

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-29 02:49:20 -04:00
..