mirror of
https://github.com/nomadkaraoke/karaoke-gen.git
synced 2026-11-20 13:07:57 +09:00
fix(backend): move blocking credential checks + GDrive validation off the event loop (v0.267.1) (#1133)
## Summary Two recurring `EVENT_LOOP_STALL`s alerted on 2026-10-07: - **16s** in `mark_uploads_complete` → `check_dropbox_credentials()` → expired-token refresh (HTTP + Secret Manager `add_secret_version` + version cleanup), all on the event loop. Also seen 10-06 14:03, 10-07 05:49/06:00 via `create_job_with_upload_urls`. - **5-10s** after every job completion in `trigger_gdrive_validation_endpoint` (sync OIDC token fetch + `requests.post(timeout=60)`). Seen 7× since 10-04. These routes `await request.json()` etc., so the v0.264.0 "async route without await" guard didn't catch them. ## Changes - `file_upload.py` (`upload_and_create_job`, `create_job_with_upload_urls`, `mark_uploads_complete`, `create_job_from_url`) and `audio_search.py` (`search_audio`): 18 `check_*_credentials()` calls → `await asyncio.to_thread(...)`. Plain-`def` routes were already in the threadpool and are unchanged. - `internal.py`: `trigger_gdrive_validation` → `asyncio.to_thread`. - `test_no_blocking_async_routes.py`: new `BLOCKING_CALLS` denylist guard (credential checks, `trigger_gdrive_validation`, `requests.*`, `time.sleep`) for direct calls inside `async def` routes, plus a detector unit test. Run against `origin/main` it flags exactly the 19 sites fixed here. - LESSONS-LEARNED entry; version 0.267.1. ## Testing - `make test`: backend 5838 passed, 2021 passed (second backend set), emulator 150 passed, frontend 1654 passed. - Local CodeRabbit review: no findings. @coderabbitai ignore 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
1 parent
9d04274de7
commit
437e72a29b
6 files changed
+112
-20
No files matched your search
@@ -729,17 +729,17 @@ async def search_audio(
|
||||
credential_manager = get_credential_manager()
|
||||
|
||||
if effective_enable_youtube_upload:
|
||||
result = credential_manager.check_youtube_credentials()
|
||||
result = await asyncio.to_thread(credential_manager.check_youtube_credentials)
|
||||
if result.status != CredentialStatus.VALID:
|
||||
invalid_services.append(f"youtube ({result.message})")
|
||||
|
||||
if effective_dropbox_path:
|
||||
result = credential_manager.check_dropbox_credentials()
|
||||
result = await asyncio.to_thread(credential_manager.check_dropbox_credentials)
|
||||
if result.status != CredentialStatus.VALID:
|
||||
invalid_services.append(f"dropbox ({result.message})")
|
||||
|
||||
if effective_gdrive_folder_id:
|
||||
result = credential_manager.check_gdrive_credentials()
|
||||
result = await asyncio.to_thread(credential_manager.check_gdrive_credentials)
|
||||
if result.status != CredentialStatus.VALID:
|
||||
invalid_services.append(f"gdrive ({result.message})")
|
||||
|
||||
|
||||
@@ -610,17 +610,17 @@ async def upload_and_create_job(
|
||||
credential_manager = get_credential_manager()
|
||||
|
||||
if effective_enable_youtube_upload and not is_private:
|
||||
result = credential_manager.check_youtube_credentials()
|
||||
result = await asyncio.to_thread(credential_manager.check_youtube_credentials)
|
||||
if result.status != CredentialStatus.VALID:
|
||||
invalid_services.append(f"youtube ({result.message})")
|
||||
|
||||
if dist.dropbox_path:
|
||||
result = credential_manager.check_dropbox_credentials()
|
||||
result = await asyncio.to_thread(credential_manager.check_dropbox_credentials)
|
||||
if result.status != CredentialStatus.VALID:
|
||||
invalid_services.append(f"dropbox ({result.message})")
|
||||
|
||||
if dist.gdrive_folder_id and not is_private:
|
||||
result = credential_manager.check_gdrive_credentials()
|
||||
result = await asyncio.to_thread(credential_manager.check_gdrive_credentials)
|
||||
if result.status != CredentialStatus.VALID:
|
||||
invalid_services.append(f"gdrive ({result.message})")
|
||||
|
||||
@@ -930,7 +930,7 @@ async def upload_and_create_job(
|
||||
distribution_services: Dict[str, Any] = {}
|
||||
|
||||
if dist.dropbox_path:
|
||||
dropbox_result = credential_manager.check_dropbox_credentials()
|
||||
dropbox_result = await asyncio.to_thread(credential_manager.check_dropbox_credentials)
|
||||
distribution_services["dropbox"] = {
|
||||
"enabled": True,
|
||||
"path": dist.dropbox_path,
|
||||
@@ -939,7 +939,7 @@ async def upload_and_create_job(
|
||||
}
|
||||
|
||||
if dist.gdrive_folder_id:
|
||||
gdrive_result = credential_manager.check_gdrive_credentials()
|
||||
gdrive_result = await asyncio.to_thread(credential_manager.check_gdrive_credentials)
|
||||
distribution_services["gdrive"] = {
|
||||
"enabled": True,
|
||||
"folder_id": dist.gdrive_folder_id,
|
||||
@@ -948,7 +948,7 @@ async def upload_and_create_job(
|
||||
}
|
||||
|
||||
if effective_enable_youtube_upload:
|
||||
youtube_result = credential_manager.check_youtube_credentials()
|
||||
youtube_result = await asyncio.to_thread(credential_manager.check_youtube_credentials)
|
||||
distribution_services["youtube"] = {
|
||||
"enabled": True,
|
||||
"credentials_valid": youtube_result.status == CredentialStatus.VALID,
|
||||
@@ -1315,17 +1315,17 @@ async def create_job_with_upload_urls(
|
||||
credential_manager = get_credential_manager()
|
||||
|
||||
if effective_enable_youtube_upload and not effective_is_private:
|
||||
result = credential_manager.check_youtube_credentials()
|
||||
result = await asyncio.to_thread(credential_manager.check_youtube_credentials)
|
||||
if result.status != CredentialStatus.VALID:
|
||||
invalid_services.append(f"youtube ({result.message})")
|
||||
|
||||
if dist.dropbox_path:
|
||||
result = credential_manager.check_dropbox_credentials()
|
||||
result = await asyncio.to_thread(credential_manager.check_dropbox_credentials)
|
||||
if result.status != CredentialStatus.VALID:
|
||||
invalid_services.append(f"dropbox ({result.message})")
|
||||
|
||||
if dist.gdrive_folder_id and not effective_is_private:
|
||||
result = credential_manager.check_gdrive_credentials()
|
||||
result = await asyncio.to_thread(credential_manager.check_gdrive_credentials)
|
||||
if result.status != CredentialStatus.VALID:
|
||||
invalid_services.append(f"gdrive ({result.message})")
|
||||
|
||||
@@ -1695,7 +1695,7 @@ async def mark_uploads_complete(
|
||||
updated_job = job_manager.get_job(job_id)
|
||||
|
||||
if updated_job.dropbox_path:
|
||||
dropbox_result = credential_manager.check_dropbox_credentials()
|
||||
dropbox_result = await asyncio.to_thread(credential_manager.check_dropbox_credentials)
|
||||
distribution_services["dropbox"] = {
|
||||
"enabled": True,
|
||||
"path": updated_job.dropbox_path,
|
||||
@@ -1703,7 +1703,7 @@ async def mark_uploads_complete(
|
||||
}
|
||||
|
||||
if updated_job.gdrive_folder_id:
|
||||
gdrive_result = credential_manager.check_gdrive_credentials()
|
||||
gdrive_result = await asyncio.to_thread(credential_manager.check_gdrive_credentials)
|
||||
distribution_services["gdrive"] = {
|
||||
"enabled": True,
|
||||
"folder_id": updated_job.gdrive_folder_id,
|
||||
@@ -1711,7 +1711,7 @@ async def mark_uploads_complete(
|
||||
}
|
||||
|
||||
if updated_job.enable_youtube_upload:
|
||||
youtube_result = credential_manager.check_youtube_credentials()
|
||||
youtube_result = await asyncio.to_thread(credential_manager.check_youtube_credentials)
|
||||
distribution_services["youtube"] = {
|
||||
"enabled": True,
|
||||
"credentials_valid": youtube_result.status == CredentialStatus.VALID,
|
||||
@@ -2029,17 +2029,17 @@ async def create_job_from_url(
|
||||
credential_manager = get_credential_manager()
|
||||
|
||||
if effective_enable_youtube_upload and not body.is_private:
|
||||
result = credential_manager.check_youtube_credentials()
|
||||
result = await asyncio.to_thread(credential_manager.check_youtube_credentials)
|
||||
if result.status != CredentialStatus.VALID:
|
||||
invalid_services.append(f"youtube ({result.message})")
|
||||
|
||||
if dist.dropbox_path:
|
||||
result = credential_manager.check_dropbox_credentials()
|
||||
result = await asyncio.to_thread(credential_manager.check_dropbox_credentials)
|
||||
if result.status != CredentialStatus.VALID:
|
||||
invalid_services.append(f"dropbox ({result.message})")
|
||||
|
||||
if dist.gdrive_folder_id and not body.is_private:
|
||||
result = credential_manager.check_gdrive_credentials()
|
||||
result = await asyncio.to_thread(credential_manager.check_gdrive_credentials)
|
||||
if result.status != CredentialStatus.VALID:
|
||||
invalid_services.append(f"gdrive ({result.message})")
|
||||
|
||||
|
||||
@@ -931,7 +931,10 @@ async def trigger_gdrive_validation_endpoint(
|
||||
add_span_attribute("brand_code", str(brand_code))
|
||||
|
||||
try:
|
||||
result = trigger_gdrive_validation(brand_code=brand_code, expect_cdg=expect_cdg)
|
||||
# Sync OIDC fetch + HTTP POST (up to 60s) — off the event loop.
|
||||
result = await asyncio.to_thread(
|
||||
trigger_gdrive_validation, brand_code=brand_code, expect_cdg=expect_cdg
|
||||
)
|
||||
if result is None:
|
||||
return {"status": "skipped", "message": "GDRIVE_VALIDATOR_URL not configured"}
|
||||
|
||||
|
||||
@@ -191,3 +191,91 @@ async def helper():
|
||||
await client.sync_ok()
|
||||
'''
|
||||
assert find_awaited_sync_routes(caller) == ["sync_ok"]
|
||||
|
||||
|
||||
# Known-blocking calls (network / Secret Manager I/O) that awaiting *something else*
|
||||
# in the same handler doesn't excuse. On 2026-10-07 the routes below awaited
|
||||
# request.json() etc. (so test_no_async_route_without_await passed) yet froze the
|
||||
# loop 5-16s: a Dropbox token refresh (HTTP + Secret Manager write) inside
|
||||
# check_dropbox_credentials, and the GDrive validator's OIDC fetch + POST.
|
||||
# Pass these to asyncio.to_thread(fn, ...) — a reference, not a call — instead.
|
||||
BLOCKING_CALLS = {
|
||||
"check_youtube_credentials",
|
||||
"check_dropbox_credentials",
|
||||
"check_gdrive_credentials",
|
||||
"check_all_credentials",
|
||||
"trigger_gdrive_validation",
|
||||
}
|
||||
_BLOCKING_MODULE_CALLS = {("requests", "get"), ("requests", "post"), ("requests", "put"),
|
||||
("requests", "delete"), ("requests", "request"), ("time", "sleep")}
|
||||
|
||||
|
||||
def _blocking_call_name(call: ast.Call) -> str | None:
|
||||
fn = call.func
|
||||
if isinstance(fn, ast.Name) and fn.id in BLOCKING_CALLS:
|
||||
return fn.id
|
||||
if isinstance(fn, ast.Attribute):
|
||||
if fn.attr in BLOCKING_CALLS:
|
||||
return fn.attr
|
||||
if isinstance(fn.value, ast.Name) and (fn.value.id, fn.attr) in _BLOCKING_MODULE_CALLS:
|
||||
return f"{fn.value.id}.{fn.attr}"
|
||||
return None
|
||||
|
||||
|
||||
def find_blocking_calls_in_async_routes(source: str) -> list[str]:
|
||||
"""`async def` routes that directly call a known-blocking function (excluding
|
||||
nested sync helpers, which are presumably run via to_thread)."""
|
||||
tree = ast.parse(source)
|
||||
hits = []
|
||||
for fn in ast.walk(tree):
|
||||
if not (isinstance(fn, ast.AsyncFunctionDef) and _is_route(fn)):
|
||||
continue
|
||||
stack = list(fn.body)
|
||||
while stack:
|
||||
node = stack.pop()
|
||||
if isinstance(node, (ast.FunctionDef, ast.Lambda, ast.ClassDef)):
|
||||
continue
|
||||
if isinstance(node, ast.Call) and (name := _blocking_call_name(node)):
|
||||
hits.append(f"{fn.name}: {name}() line {node.lineno}")
|
||||
stack.extend(ast.iter_child_nodes(node))
|
||||
return hits
|
||||
|
||||
|
||||
def test_no_known_blocking_calls_in_async_routes():
|
||||
offenders = []
|
||||
for path in ROUTE_FILES:
|
||||
for hit in find_blocking_calls_in_async_routes(path.read_text()):
|
||||
offenders.append(f"{path.relative_to(BACKEND.parent)}::{hit}")
|
||||
assert not offenders, (
|
||||
"These `async def` routes call known-blocking functions directly on the event "
|
||||
"loop. Use `await asyncio.to_thread(fn, ...)`:\n " + "\n ".join(sorted(offenders))
|
||||
)
|
||||
|
||||
|
||||
def test_blocking_call_detector():
|
||||
src = '''
|
||||
@router.post("/a")
|
||||
async def bad(request):
|
||||
await request.json()
|
||||
cm.check_dropbox_credentials()
|
||||
requests.post(url)
|
||||
|
||||
@router.post("/b")
|
||||
async def good(request):
|
||||
await request.json()
|
||||
await asyncio.to_thread(cm.check_dropbox_credentials)
|
||||
await asyncio.to_thread(trigger_gdrive_validation, brand_code="X")
|
||||
|
||||
@router.post("/c")
|
||||
def sync_ok():
|
||||
cm.check_dropbox_credentials()
|
||||
|
||||
@router.post("/d")
|
||||
async def helper_ok():
|
||||
def _work():
|
||||
return trigger_gdrive_validation()
|
||||
await asyncio.to_thread(_work)
|
||||
'''
|
||||
hits = find_blocking_calls_in_async_routes(src)
|
||||
assert sorted(h.split(" line")[0] for h in hits) == [
|
||||
"bad: check_dropbox_credentials()", "bad: requests.post()"]
|
||||
@@ -32,6 +32,7 @@ The "servers unavailable" banner (49 episodes in 2 weeks) was mostly the API's o
|
||||
- The inverse also fails: a `def` route that calls `asyncio.create_task`/`get_running_loop`. In the threadpool, that raises, or silently does nothing if the `RuntimeError` is swallowed (`rate_limits.trigger_youtube_queue_processing` did exactly that).
|
||||
- **Moving a handler into the threadpool removes the implicit per-instance serialization.** A never-awaiting `async def` used to run start-to-finish uninterrupted, so non-transactional check-then-write code (a double-clicked Cancel refunding twice, a double Complete) was safe on one instance. `backend/utils/keyed_lock.py` restores that: `JobManager.transition_to_state`/`cancel_job` lock per job, `UserService.add_credits` per user. Also grep for `await <converted handler>(`; the guard test checks this now (it found `admin.update_job` awaiting `delete_job_outputs`).
|
||||
- **Watchdog vs `--cpu-throttling`:** an idle instance gets no CPU between requests, so a gap with the loop parked in `select()` is not a stall. The watchdog drops all-idle samples.
|
||||
- **An `await` somewhere in the handler doesn't make its other calls safe** (v0.267.1). `mark_uploads_complete`, `create_job_with_upload_urls` and others awaited `request.json()`, so they passed the guard, yet called `check_dropbox_credentials()` inline. When the Dropbox token had expired, that did an HTTP refresh plus a Secret Manager write and version cleanup on the loop, a 16s freeze. `trigger_gdrive_validation_endpoint` froze 5-10s after every job (OIDC fetch + POST). Both are now behind `asyncio.to_thread`, and the guard's `BLOCKING_CALLS` denylist fails CI if a known-blocking function is called directly in an `async def` route. Add to the denylist when the watchdog names a new culprit.
|
||||
- **Detection:** `backend/services/loop_watchdog.py` logs `EVENT_LOOP_STALL` with the loop thread's stack and a `culprit=file:line`. Grep for it before guessing.
|
||||
|
||||
---
|
||||
|
||||
+1
-1
@@ -1,6 +1,6 @@
|
||||
[tool.poetry]
|
||||
name = "karaoke-gen"
|
||||
version = "0.267.0"
|
||||
version = "0.267.1"
|
||||
description = "Generate karaoke videos with synchronized lyrics. Handles the entire process from downloading audio and lyrics to creating the final video with title screens."
|
||||
authors = ["Andrew Beveridge <andrew@beveridge.uk>"]
|
||||
license = "MIT"
|
||||
|
||||
Reference in new issue
Block a user