fix(backend): move blocking credential checks + GDrive validation off the event loop (v0.267.1) (#1133)

## Summary
Two recurring `EVENT_LOOP_STALL`s alerted on 2026-10-07:
- **16s** in `mark_uploads_complete` → `check_dropbox_credentials()` →
expired-token refresh (HTTP + Secret Manager `add_secret_version` +
version cleanup), all on the event loop. Also seen 10-06 14:03, 10-07
05:49/06:00 via `create_job_with_upload_urls`.
- **5-10s** after every job completion in
`trigger_gdrive_validation_endpoint` (sync OIDC token fetch +
`requests.post(timeout=60)`). Seen 7× since 10-04.

These routes `await request.json()` etc., so the v0.264.0 "async route
without await" guard didn't catch them.

## Changes
- `file_upload.py` (`upload_and_create_job`,
`create_job_with_upload_urls`, `mark_uploads_complete`,
`create_job_from_url`) and `audio_search.py` (`search_audio`): 18
`check_*_credentials()` calls → `await asyncio.to_thread(...)`.
Plain-`def` routes were already in the threadpool and are unchanged.
- `internal.py`: `trigger_gdrive_validation` → `asyncio.to_thread`.
- `test_no_blocking_async_routes.py`: new `BLOCKING_CALLS` denylist
guard (credential checks, `trigger_gdrive_validation`, `requests.*`,
`time.sleep`) for direct calls inside `async def` routes, plus a
detector unit test. Run against `origin/main` it flags exactly the 19
sites fixed here.
- LESSONS-LEARNED entry; version 0.267.1.

## Testing
- `make test`: backend 5838 passed, 2021 passed (second backend set),
emulator 150 passed, frontend 1654 passed.
- Local CodeRabbit review: no findings.

@coderabbitai ignore

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
Andrew Beveridge ☄️andClaude Opus 5.5 authored and GitHub committed 2026-10-07 21:25:56 -04:00
1 parent 9d04274de7
commit 437e72a29b
6 files changed
+112 -20

No files matched your search

+3 -3
View File
@@ -729,17 +729,17 @@ async def search_audio(
credential_manager = get_credential_manager()
if effective_enable_youtube_upload:
result = credential_manager.check_youtube_credentials()
result = await asyncio.to_thread(credential_manager.check_youtube_credentials)
if result.status != CredentialStatus.VALID:
invalid_services.append(f"youtube ({result.message})")
if effective_dropbox_path:
result = credential_manager.check_dropbox_credentials()
result = await asyncio.to_thread(credential_manager.check_dropbox_credentials)
if result.status != CredentialStatus.VALID:
invalid_services.append(f"dropbox ({result.message})")
if effective_gdrive_folder_id:
result = credential_manager.check_gdrive_credentials()
result = await asyncio.to_thread(credential_manager.check_gdrive_credentials)
if result.status != CredentialStatus.VALID:
invalid_services.append(f"gdrive ({result.message})")
+15 -15
View File
@@ -610,17 +610,17 @@ async def upload_and_create_job(
credential_manager = get_credential_manager()
if effective_enable_youtube_upload and not is_private:
result = credential_manager.check_youtube_credentials()
result = await asyncio.to_thread(credential_manager.check_youtube_credentials)
if result.status != CredentialStatus.VALID:
invalid_services.append(f"youtube ({result.message})")
if dist.dropbox_path:
result = credential_manager.check_dropbox_credentials()
result = await asyncio.to_thread(credential_manager.check_dropbox_credentials)
if result.status != CredentialStatus.VALID:
invalid_services.append(f"dropbox ({result.message})")
if dist.gdrive_folder_id and not is_private:
result = credential_manager.check_gdrive_credentials()
result = await asyncio.to_thread(credential_manager.check_gdrive_credentials)
if result.status != CredentialStatus.VALID:
invalid_services.append(f"gdrive ({result.message})")
@@ -930,7 +930,7 @@ async def upload_and_create_job(
distribution_services: Dict[str, Any] = {}
if dist.dropbox_path:
dropbox_result = credential_manager.check_dropbox_credentials()
dropbox_result = await asyncio.to_thread(credential_manager.check_dropbox_credentials)
distribution_services["dropbox"] = {
"enabled": True,
"path": dist.dropbox_path,
@@ -939,7 +939,7 @@ async def upload_and_create_job(
}
if dist.gdrive_folder_id:
gdrive_result = credential_manager.check_gdrive_credentials()
gdrive_result = await asyncio.to_thread(credential_manager.check_gdrive_credentials)
distribution_services["gdrive"] = {
"enabled": True,
"folder_id": dist.gdrive_folder_id,
@@ -948,7 +948,7 @@ async def upload_and_create_job(
}
if effective_enable_youtube_upload:
youtube_result = credential_manager.check_youtube_credentials()
youtube_result = await asyncio.to_thread(credential_manager.check_youtube_credentials)
distribution_services["youtube"] = {
"enabled": True,
"credentials_valid": youtube_result.status == CredentialStatus.VALID,
@@ -1315,17 +1315,17 @@ async def create_job_with_upload_urls(
credential_manager = get_credential_manager()
if effective_enable_youtube_upload and not effective_is_private:
result = credential_manager.check_youtube_credentials()
result = await asyncio.to_thread(credential_manager.check_youtube_credentials)
if result.status != CredentialStatus.VALID:
invalid_services.append(f"youtube ({result.message})")
if dist.dropbox_path:
result = credential_manager.check_dropbox_credentials()
result = await asyncio.to_thread(credential_manager.check_dropbox_credentials)
if result.status != CredentialStatus.VALID:
invalid_services.append(f"dropbox ({result.message})")
if dist.gdrive_folder_id and not effective_is_private:
result = credential_manager.check_gdrive_credentials()
result = await asyncio.to_thread(credential_manager.check_gdrive_credentials)
if result.status != CredentialStatus.VALID:
invalid_services.append(f"gdrive ({result.message})")
@@ -1695,7 +1695,7 @@ async def mark_uploads_complete(
updated_job = job_manager.get_job(job_id)
if updated_job.dropbox_path:
dropbox_result = credential_manager.check_dropbox_credentials()
dropbox_result = await asyncio.to_thread(credential_manager.check_dropbox_credentials)
distribution_services["dropbox"] = {
"enabled": True,
"path": updated_job.dropbox_path,
@@ -1703,7 +1703,7 @@ async def mark_uploads_complete(
}
if updated_job.gdrive_folder_id:
gdrive_result = credential_manager.check_gdrive_credentials()
gdrive_result = await asyncio.to_thread(credential_manager.check_gdrive_credentials)
distribution_services["gdrive"] = {
"enabled": True,
"folder_id": updated_job.gdrive_folder_id,
@@ -1711,7 +1711,7 @@ async def mark_uploads_complete(
}
if updated_job.enable_youtube_upload:
youtube_result = credential_manager.check_youtube_credentials()
youtube_result = await asyncio.to_thread(credential_manager.check_youtube_credentials)
distribution_services["youtube"] = {
"enabled": True,
"credentials_valid": youtube_result.status == CredentialStatus.VALID,
@@ -2029,17 +2029,17 @@ async def create_job_from_url(
credential_manager = get_credential_manager()
if effective_enable_youtube_upload and not body.is_private:
result = credential_manager.check_youtube_credentials()
result = await asyncio.to_thread(credential_manager.check_youtube_credentials)
if result.status != CredentialStatus.VALID:
invalid_services.append(f"youtube ({result.message})")
if dist.dropbox_path:
result = credential_manager.check_dropbox_credentials()
result = await asyncio.to_thread(credential_manager.check_dropbox_credentials)
if result.status != CredentialStatus.VALID:
invalid_services.append(f"dropbox ({result.message})")
if dist.gdrive_folder_id and not body.is_private:
result = credential_manager.check_gdrive_credentials()
result = await asyncio.to_thread(credential_manager.check_gdrive_credentials)
if result.status != CredentialStatus.VALID:
invalid_services.append(f"gdrive ({result.message})")
+4 -1
View File
@@ -931,7 +931,10 @@ async def trigger_gdrive_validation_endpoint(
add_span_attribute("brand_code", str(brand_code))
try:
result = trigger_gdrive_validation(brand_code=brand_code, expect_cdg=expect_cdg)
# Sync OIDC fetch + HTTP POST (up to 60s) — off the event loop.
result = await asyncio.to_thread(
trigger_gdrive_validation, brand_code=brand_code, expect_cdg=expect_cdg
)
if result is None:
return {"status": "skipped", "message": "GDRIVE_VALIDATOR_URL not configured"}
@@ -191,3 +191,91 @@ async def helper():
await client.sync_ok()
'''
assert find_awaited_sync_routes(caller) == ["sync_ok"]
# Known-blocking calls (network / Secret Manager I/O) that awaiting *something else*
# in the same handler doesn't excuse. On 2026-10-07 the routes below awaited
# request.json() etc. (so test_no_async_route_without_await passed) yet froze the
# loop 5-16s: a Dropbox token refresh (HTTP + Secret Manager write) inside
# check_dropbox_credentials, and the GDrive validator's OIDC fetch + POST.
# Pass these to asyncio.to_thread(fn, ...) — a reference, not a call — instead.
BLOCKING_CALLS = {
"check_youtube_credentials",
"check_dropbox_credentials",
"check_gdrive_credentials",
"check_all_credentials",
"trigger_gdrive_validation",
}
_BLOCKING_MODULE_CALLS = {("requests", "get"), ("requests", "post"), ("requests", "put"),
("requests", "delete"), ("requests", "request"), ("time", "sleep")}
def _blocking_call_name(call: ast.Call) -> str | None:
fn = call.func
if isinstance(fn, ast.Name) and fn.id in BLOCKING_CALLS:
return fn.id
if isinstance(fn, ast.Attribute):
if fn.attr in BLOCKING_CALLS:
return fn.attr
if isinstance(fn.value, ast.Name) and (fn.value.id, fn.attr) in _BLOCKING_MODULE_CALLS:
return f"{fn.value.id}.{fn.attr}"
return None
def find_blocking_calls_in_async_routes(source: str) -> list[str]:
"""`async def` routes that directly call a known-blocking function (excluding
nested sync helpers, which are presumably run via to_thread)."""
tree = ast.parse(source)
hits = []
for fn in ast.walk(tree):
if not (isinstance(fn, ast.AsyncFunctionDef) and _is_route(fn)):
continue
stack = list(fn.body)
while stack:
node = stack.pop()
if isinstance(node, (ast.FunctionDef, ast.Lambda, ast.ClassDef)):
continue
if isinstance(node, ast.Call) and (name := _blocking_call_name(node)):
hits.append(f"{fn.name}: {name}() line {node.lineno}")
stack.extend(ast.iter_child_nodes(node))
return hits
def test_no_known_blocking_calls_in_async_routes():
offenders = []
for path in ROUTE_FILES:
for hit in find_blocking_calls_in_async_routes(path.read_text()):
offenders.append(f"{path.relative_to(BACKEND.parent)}::{hit}")
assert not offenders, (
"These `async def` routes call known-blocking functions directly on the event "
"loop. Use `await asyncio.to_thread(fn, ...)`:\n " + "\n ".join(sorted(offenders))
)
def test_blocking_call_detector():
src = '''
@router.post("/a")
async def bad(request):
await request.json()
cm.check_dropbox_credentials()
requests.post(url)
@router.post("/b")
async def good(request):
await request.json()
await asyncio.to_thread(cm.check_dropbox_credentials)
await asyncio.to_thread(trigger_gdrive_validation, brand_code="X")
@router.post("/c")
def sync_ok():
cm.check_dropbox_credentials()
@router.post("/d")
async def helper_ok():
def _work():
return trigger_gdrive_validation()
await asyncio.to_thread(_work)
'''
hits = find_blocking_calls_in_async_routes(src)
assert sorted(h.split(" line")[0] for h in hits) == [
"bad: check_dropbox_credentials()", "bad: requests.post()"]
+1
View File
@@ -32,6 +32,7 @@ The "servers unavailable" banner (49 episodes in 2 weeks) was mostly the API's o
- The inverse also fails: a `def` route that calls `asyncio.create_task`/`get_running_loop`. In the threadpool, that raises, or silently does nothing if the `RuntimeError` is swallowed (`rate_limits.trigger_youtube_queue_processing` did exactly that).
- **Moving a handler into the threadpool removes the implicit per-instance serialization.** A never-awaiting `async def` used to run start-to-finish uninterrupted, so non-transactional check-then-write code (a double-clicked Cancel refunding twice, a double Complete) was safe on one instance. `backend/utils/keyed_lock.py` restores that: `JobManager.transition_to_state`/`cancel_job` lock per job, `UserService.add_credits` per user. Also grep for `await <converted handler>(`; the guard test checks this now (it found `admin.update_job` awaiting `delete_job_outputs`).
- **Watchdog vs `--cpu-throttling`:** an idle instance gets no CPU between requests, so a gap with the loop parked in `select()` is not a stall. The watchdog drops all-idle samples.
- **An `await` somewhere in the handler doesn't make its other calls safe** (v0.267.1). `mark_uploads_complete`, `create_job_with_upload_urls` and others awaited `request.json()`, so they passed the guard, yet called `check_dropbox_credentials()` inline. When the Dropbox token had expired, that did an HTTP refresh plus a Secret Manager write and version cleanup on the loop, a 16s freeze. `trigger_gdrive_validation_endpoint` froze 5-10s after every job (OIDC fetch + POST). Both are now behind `asyncio.to_thread`, and the guard's `BLOCKING_CALLS` denylist fails CI if a known-blocking function is called directly in an `async def` route. Add to the denylist when the watchdog names a new culprit.
- **Detection:** `backend/services/loop_watchdog.py` logs `EVENT_LOOP_STALL` with the loop thread's stack and a `culprit=file:line`. Grep for it before guessing.
---
+1 -1
View File
@@ -1,6 +1,6 @@
[tool.poetry]
name = "karaoke-gen"
version = "0.267.0"
version = "0.267.1"
description = "Generate karaoke videos with synchronized lyrics. Handles the entire process from downloading audio and lyrics to creating the final video with title screens."
authors = ["Andrew Beveridge <andrew@beveridge.uk>"]
license = "MIT"