mirror of
https://github.com/nomadkaraoke/karaoke-gen.git
synced 2026-11-20 13:07:57 +09:00
fix: keep auth, tenant lookup and job-log writes off the API event loop (v0.278.14) (#1194)
@coderabbitai ignore ## Why The residual event-loop freezes BACKLOG item. Seven days of `EVENT_LOOP_STALL in progress` stacks (2026-10-12→19, 355 samples), grouped by their outermost app frame, ranked the app-level loop blockers: | Source | Samples | Max | |---|---|---| | in-process screens worker → job-log handler → sync Firestore write per log line | 30 | 2.2s | | `require_auth` → `validate_token_full` (session read on **every** authed request) | 19 | 1.9s | | preview encodes (`_run_preview_encode`/`_encode_preview_in_background`) → job-log write | 15 | 2.4s | | `change_visibility` → sync Dropbox/GCS deletes | 6 | 1.2s | | tenant middleware → `get_tenant_config` (GCS on cache miss) | 5 | 1.2s | | `require_review_auth` → `get_job` | 5 | 2.2s | | `auto_correct_worker._run_suggest` → `get_job` (the only stall since #1192) | 1 | 1.2s | (The render/video in-process stalls stopped with #1192.) ## What - **`backend/services/log_write_offload.py`**: when a job-log write is made on a running event loop in the API process, it goes to one background writer thread. The queue is bounded at 5000 and a drop is logged at WARNING, rate-limited; `atexit` drains the queue. Writes stay inline in threads and Cloud Run Jobs, so the worker code paths behave exactly as before. Logs are read back ordered by their own `timestamp`, so the hand-off doesn't reorder them. - **`asyncio.to_thread`** for `require_auth`/`require_auth_legacy`/`require_review_auth`, the tenant middleware, `rerender_with_current_theme`'s tenant lookup, `auto_correct._run_suggest`, and `change_visibility`'s `get_job`. - **Visibility change:** `_delete_*_outputs` are now plain `def`, and the sync steps run in one worker thread. Moving them off the loop removed the implicit serialisation, so both directions now **claim `visibility_change_in_progress` atomically** before touching anything. They use the new `FirestoreService.try_set_job_state_flag`, a transactional test-and-set. A lost claim returns 409. The regenerate→private continuation passes `holds_claim=True`, because its regenerate already set the flag. The claim is released if the reset fails, and a raising screens trigger rolls back like a `False` one. - **Guard (`test_no_blocking_async_routes.py`):** `BLOCKING_CALLS` gains `validate_token(_full)`, `validate_session`, `get_tenant_config`, `append_log_to_subcollection` and `append_worker_log`. Every `async def` in `api/dependencies.py` and `middleware/` is now checked, not just decorated routes. Mutation-checked: reverting `require_auth` fails the guard. - `docs/LESSONS-LEARNED.md` entry. About 70 direct `get_job`/`update_job` calls in async routes remain. They're single round-trips that rarely show in the watchdog, and the doc says to move them when one does. ## Testing - New: `test_log_write_offload.py` (inline vs offloaded, Cloud Run Job inline, ordering, full-queue drop, Firestore append off the loop thread), claim tests in `test_change_visibility.py` (lost claim, release on failure, chained continuation, trigger exception, route 409), and emulator `test_try_set_job_state_flag.py` (once-only, already set, missing job, one winner among 4 concurrent callers). - Backend: 6478 passed. The 8 failures are `test_emulator_integration.py`, caused by my local environment: another session's emulator setup has an `ssh` forward on :4443 instead of fake-GCS, and they fail identically on untouched main. CI's emulator job covers them and the new emulator test. - Local CodeRabbit: 3 cycles. I fixed the claim race (two findings), trigger-exception rollback, and the guarded claim release. I skipped two minor nitpicks on the pre-existing trigger-failure rollback. 🤖 Generated with [Claude Code](https://claude.com/claude-code) https://claude.ai/code/session_01Jw21syo4HizJJjYw2mg2Vp --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
1 parent
b02cce3cc7
commit
3bbf397dc4
14 files changed
+594
-62
No files matched your search
@@ -1,6 +1,7 @@
|
||||
"""
|
||||
FastAPI dependencies for authentication and authorization.
|
||||
"""
|
||||
import asyncio
|
||||
import logging
|
||||
import secrets
|
||||
from datetime import datetime, timedelta, UTC
|
||||
@@ -89,7 +90,8 @@ async def require_auth(
|
||||
)
|
||||
|
||||
# Validate token using the full method
|
||||
auth_result = auth_service.validate_token_full(token_str)
|
||||
# Sync Firestore reads: run off the event loop (every authed request passes here).
|
||||
auth_result = await asyncio.to_thread(auth_service.validate_token_full, token_str)
|
||||
|
||||
# Get request_id from middleware for correlation
|
||||
request_id = getattr(request.state, "request_id", None)
|
||||
@@ -122,7 +124,7 @@ async def require_auth(
|
||||
if not getattr(request.state, "tenant_config", None):
|
||||
from backend.services.tenant_service import get_tenant_service
|
||||
tenant_service = get_tenant_service()
|
||||
tenant_config = tenant_service.get_tenant_config(auth_result.tenant_id)
|
||||
tenant_config = await asyncio.to_thread(tenant_service.get_tenant_config, auth_result.tenant_id)
|
||||
if tenant_config and tenant_config.is_active:
|
||||
request.state.tenant_config = tenant_config
|
||||
|
||||
@@ -165,7 +167,7 @@ async def require_auth_legacy(
|
||||
detail="Authentication required. Provide token via Authorization header or ?token= parameter"
|
||||
)
|
||||
|
||||
is_valid, user_type, remaining_uses, message = auth_service.validate_token(token_str)
|
||||
is_valid, user_type, remaining_uses, message = await asyncio.to_thread(auth_service.validate_token, token_str)
|
||||
|
||||
if not is_valid:
|
||||
raise HTTPException(
|
||||
@@ -284,10 +286,10 @@ def require_review_auth_factory(job_id_param: str = "job_id"):
|
||||
full_token = token
|
||||
|
||||
if full_token:
|
||||
auth_result = auth_service.validate_token_full(full_token)
|
||||
auth_result = await asyncio.to_thread(auth_service.validate_token_full, full_token)
|
||||
if auth_result.is_valid:
|
||||
# For full auth, also verify job ownership
|
||||
job = job_manager.get_job(job_id)
|
||||
job = await asyncio.to_thread(job_manager.get_job, job_id)
|
||||
if not job:
|
||||
raise HTTPException(status_code=404, detail=f"Job {job_id} not found")
|
||||
|
||||
@@ -319,7 +321,7 @@ def require_review_auth_factory(job_id_param: str = "job_id"):
|
||||
|
||||
# Try review token
|
||||
if review_token:
|
||||
job = job_manager.get_job(job_id)
|
||||
job = await asyncio.to_thread(job_manager.get_job, job_id)
|
||||
|
||||
if not job:
|
||||
raise HTTPException(status_code=404, detail=f"Job {job_id} not found")
|
||||
|
||||
@@ -3093,14 +3093,14 @@ async def change_visibility(
|
||||
"""
|
||||
locale = get_locale_from_request(http_request)
|
||||
job_manager = JobManager()
|
||||
job = job_manager.get_job(job_id)
|
||||
job = await asyncio.to_thread(job_manager.get_job, job_id)
|
||||
if not job:
|
||||
raise HTTPException(status_code=404, detail=t(locale, "jobs.notFound"))
|
||||
|
||||
user_email = auth_result.user_email or "unknown"
|
||||
is_admin = auth_result.is_admin
|
||||
|
||||
from backend.services.visibility_change_service import VisibilityChangeService
|
||||
from backend.services.visibility_change_service import VisibilityChangeInProgress, VisibilityChangeService
|
||||
service = VisibilityChangeService(job_manager)
|
||||
|
||||
# Validate (skip job-exists check since we already handled it)
|
||||
@@ -3130,6 +3130,8 @@ async def change_visibility(
|
||||
except RerenderError as e:
|
||||
# Old track whose finals were archived and can't be regenerated right now.
|
||||
raise HTTPException(status_code=e.status_code, detail=str(e))
|
||||
except VisibilityChangeInProgress as e:
|
||||
raise HTTPException(status_code=409, detail=str(e))
|
||||
except Exception as e:
|
||||
logger.error(f"Error changing visibility for job {job_id}: {e}", exc_info=True)
|
||||
raise HTTPException(status_code=500, detail=f"Failed to change visibility: {str(e)}")
|
||||
@@ -3248,7 +3250,7 @@ async def rerender_with_current_theme(
|
||||
if reason:
|
||||
raise HTTPException(status_code=400, detail=reason)
|
||||
|
||||
tenant = get_tenant_service().get_tenant_config(job.tenant_id)
|
||||
tenant = await asyncio.to_thread(get_tenant_service().get_tenant_config, job.tenant_id)
|
||||
if tenant is None or not tenant.is_active:
|
||||
raise HTTPException(status_code=400, detail="This track's portal is no longer active.")
|
||||
# Re-rendering is free work, so (like tenant billing) require the caller to
|
||||
|
||||
@@ -13,6 +13,7 @@ The middleware is non-blocking - if no tenant is detected, the request
|
||||
proceeds as a default Nomad Karaoke request (tenant_id = None).
|
||||
"""
|
||||
|
||||
import asyncio
|
||||
import logging
|
||||
import os
|
||||
from typing import Optional
|
||||
@@ -66,7 +67,8 @@ class TenantMiddleware(BaseHTTPMiddleware):
|
||||
tenant_config: Optional[TenantConfig] = None
|
||||
if tenant_id:
|
||||
tenant_service = get_tenant_service()
|
||||
tenant_config = tenant_service.get_tenant_config(tenant_id)
|
||||
# get_tenant_config can hit GCS on a cache miss: keep it off the event loop.
|
||||
tenant_config = await asyncio.to_thread(tenant_service.get_tenant_config, tenant_id)
|
||||
|
||||
if tenant_config and not tenant_config.is_active:
|
||||
# Tenant exists but is inactive - treat as default
|
||||
|
||||
@@ -12,6 +12,7 @@ from google.cloud.firestore_v1 import FieldFilter
|
||||
from backend.config import settings
|
||||
from backend.models.job import Job, JobStatus, TimelineEvent
|
||||
from backend.models.worker_log import WorkerLogEntry
|
||||
from backend.services.log_write_offload import submit_or_run
|
||||
from backend.models.review_session import ReviewSession
|
||||
from backend.models.audio_edit_session import AudioEditSession
|
||||
|
||||
@@ -152,6 +153,26 @@ class FirestoreService:
|
||||
|
||||
return _claim(self.db.transaction(), doc_ref)
|
||||
|
||||
def try_set_job_state_flag(self, job_id: str, key: str) -> bool:
|
||||
"""Atomically set ``state_data.<key>`` to True if it isn't already truthy.
|
||||
|
||||
The inverse of ``claim_job_state_flag``: returns True for exactly one of
|
||||
several concurrent callers (a guard flag such as visibility_change_in_progress).
|
||||
"""
|
||||
doc_ref = self.db.collection(self.collection).document(job_id)
|
||||
|
||||
@firestore.transactional
|
||||
def _set(transaction, ref) -> bool:
|
||||
snapshot = ref.get(transaction=transaction)
|
||||
if not snapshot.exists:
|
||||
return False
|
||||
if ((snapshot.to_dict() or {}).get("state_data") or {}).get(key):
|
||||
return False
|
||||
transaction.update(ref, {f"state_data.{key}": True, "updated_at": datetime.utcnow()})
|
||||
return True
|
||||
|
||||
return _set(self.db.transaction(), doc_ref)
|
||||
|
||||
def update_job_status(
|
||||
self,
|
||||
job_id: str,
|
||||
@@ -534,9 +555,19 @@ class FirestoreService:
|
||||
# Get subcollection reference
|
||||
logs_ref = self.db.collection(self.collection).document(job_id).collection("logs")
|
||||
|
||||
# Add document with auto-generated ID or use log_entry.id
|
||||
# Add document with auto-generated ID or use log_entry.id.
|
||||
# On the API event loop this hands off to a background thread
|
||||
# (a sync Firestore write there freezes every request on the instance).
|
||||
doc_ref = logs_ref.document(log_entry.id)
|
||||
doc_ref.set(log_entry.to_dict())
|
||||
data = log_entry.to_dict()
|
||||
|
||||
def _write():
|
||||
try:
|
||||
doc_ref.set(data)
|
||||
except Exception as e:
|
||||
logger.debug(f"Error appending log to subcollection for job {job_id}: {e}")
|
||||
|
||||
submit_or_run(_write)
|
||||
|
||||
# Don't log every append - too spammy
|
||||
except Exception as e:
|
||||
|
||||
@@ -0,0 +1,100 @@
|
||||
"""
|
||||
Keep job-log Firestore writes off the API event loop.
|
||||
|
||||
Job logs (jobs/{id}/logs/*) are written synchronously by the JobLogHandler /
|
||||
JobLogger / job_logging handlers. When that code runs on the API's event loop (the
|
||||
screens worker and preview encodes run as in-process background tasks), every log
|
||||
line is a 0.1-5s Firestore round-trip that freezes every request on the instance:
|
||||
the loop watchdog attributed 45 stalls in a week to append_log_to_subcollection.
|
||||
|
||||
`submit_or_run` hands the write to one background thread when the caller is on a
|
||||
running event loop in the API process, and runs it inline everywhere else (worker
|
||||
threads, Cloud Run Jobs, scripts), so ordering-sensitive callers are unaffected.
|
||||
Logs are read back ordered by their own timestamp, so the hand-off doesn't reorder
|
||||
them. The queue is bounded: under a Firestore outage we drop job-log lines rather
|
||||
than grow memory.
|
||||
"""
|
||||
import asyncio
|
||||
import atexit
|
||||
import logging
|
||||
import os
|
||||
import queue
|
||||
import threading
|
||||
import time
|
||||
from typing import Callable
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
_MAX_PENDING = 5000
|
||||
_DROP_WARNING_INTERVAL_S = 60.0
|
||||
|
||||
_queue: "queue.Queue[Callable[[], None]]" = queue.Queue(maxsize=_MAX_PENDING)
|
||||
_worker: threading.Thread | None = None
|
||||
_worker_lock = threading.Lock()
|
||||
_dropped = 0
|
||||
_last_drop_warning = 0.0
|
||||
|
||||
|
||||
def _on_event_loop() -> bool:
|
||||
try:
|
||||
asyncio.get_running_loop()
|
||||
except RuntimeError:
|
||||
return False
|
||||
return True
|
||||
|
||||
|
||||
def _should_offload() -> bool:
|
||||
# A Cloud Run Job runs a single job; blocking its own loop affects nobody else,
|
||||
# and a process that exits right after its last log line must not lose it.
|
||||
return _on_event_loop() and not os.environ.get("CLOUD_RUN_JOB")
|
||||
|
||||
|
||||
def _run_worker() -> None:
|
||||
while True:
|
||||
write = _queue.get()
|
||||
try:
|
||||
write()
|
||||
except Exception as e: # the write itself already swallows; belt and braces
|
||||
logger.debug(f"Offloaded job-log write failed: {e}")
|
||||
finally:
|
||||
_queue.task_done()
|
||||
|
||||
|
||||
def _ensure_worker() -> None:
|
||||
global _worker
|
||||
if _worker is not None and _worker.is_alive():
|
||||
return
|
||||
with _worker_lock:
|
||||
if _worker is None or not _worker.is_alive():
|
||||
_worker = threading.Thread(target=_run_worker, name="job-log-writer", daemon=True)
|
||||
_worker.start()
|
||||
|
||||
|
||||
def submit_or_run(write: Callable[[], None]) -> None:
|
||||
"""Run `write` on the background writer if called on the API loop, else inline."""
|
||||
global _dropped, _last_drop_warning
|
||||
if not _should_offload():
|
||||
write()
|
||||
return
|
||||
_ensure_worker()
|
||||
try:
|
||||
_queue.put_nowait(write)
|
||||
except queue.Full:
|
||||
_dropped += 1
|
||||
now = time.monotonic()
|
||||
if now - _last_drop_warning >= _DROP_WARNING_INTERVAL_S:
|
||||
_last_drop_warning = now
|
||||
logger.warning(f"Job-log writer queue full ({_MAX_PENDING}); dropped {_dropped} job-log lines so far")
|
||||
|
||||
|
||||
def drain(timeout: float = 10.0) -> bool:
|
||||
"""Wait (up to `timeout` s) for queued writes to finish. True if fully drained."""
|
||||
deadline = time.monotonic() + timeout
|
||||
while _queue.unfinished_tasks:
|
||||
if _worker is None or not _worker.is_alive() or time.monotonic() >= deadline:
|
||||
return False
|
||||
time.sleep(0.01)
|
||||
return True
|
||||
|
||||
|
||||
atexit.register(drain)
|
||||
@@ -5,6 +5,7 @@ Handles changing a job's visibility between public and private:
|
||||
- Public -> Private: redistribute existing finals to private destination (fast, ~1-2 min)
|
||||
- Private -> Public: reset styles, regenerate screens, re-render, re-encode (slow, ~15-30 min)
|
||||
"""
|
||||
import asyncio
|
||||
import logging
|
||||
from datetime import datetime, timezone
|
||||
from typing import Optional
|
||||
@@ -18,6 +19,10 @@ from backend.services.storage_service import StorageService
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
|
||||
class VisibilityChangeInProgress(Exception):
|
||||
"""Another visibility change for this job claimed it first."""
|
||||
|
||||
|
||||
class VisibilityChangeService:
|
||||
"""Service for changing job visibility after completion."""
|
||||
|
||||
@@ -53,7 +58,9 @@ class VisibilityChangeService:
|
||||
|
||||
return None
|
||||
|
||||
async def change_to_private(self, job_id: str, job, user_email: str, regenerate_if_missing: bool = True) -> dict:
|
||||
async def change_to_private(
|
||||
self, job_id: str, job, user_email: str, regenerate_if_missing: bool = True, holds_claim: bool = False,
|
||||
) -> dict:
|
||||
"""
|
||||
Change a public job to private (fast path).
|
||||
|
||||
@@ -67,6 +74,9 @@ class VisibilityChangeService:
|
||||
public may be destroyed until the private archive exists. If redistribution
|
||||
fails, the job is reverted to public and left fully intact (no partial
|
||||
deletion, no leaked NOMADNP code — see redistribute_video's failure path).
|
||||
|
||||
``holds_claim``: the caller already owns ``visibility_change_in_progress``
|
||||
(the regenerate-then-private continuation, whose regenerate set it).
|
||||
"""
|
||||
logger.info(f"[job:{job_id}] Starting visibility change: public -> private (by {user_email})")
|
||||
|
||||
@@ -107,6 +117,14 @@ class VisibilityChangeService:
|
||||
# destination. Public outputs remain live until the private archive exists.
|
||||
# No "changed to private" timeline event yet — it is only recorded once the
|
||||
# move actually succeeds (this is reverted if redistribution fails).
|
||||
# Claim atomically: two requests that both read the job before either wrote
|
||||
# the flag must not both redistribute and delete the public outputs.
|
||||
if not holds_claim:
|
||||
claimed = await asyncio.to_thread(
|
||||
self.job_manager.firestore.try_set_job_state_flag, job_id, "visibility_change_in_progress"
|
||||
)
|
||||
if not claimed:
|
||||
raise VisibilityChangeInProgress("A visibility change is already in progress for this job")
|
||||
job_ref.update({
|
||||
"state_data.visibility_change_in_progress": True,
|
||||
"is_private": True,
|
||||
@@ -141,7 +159,8 @@ class VisibilityChangeService:
|
||||
# the public outputs + recycling the public NOMAD number is best-effort
|
||||
# (each failure is logged inside the helper, never raised), so a stray
|
||||
# orphaned public artifact cannot undo the successful private move.
|
||||
await self._delete_public_outputs(job_id, public_outputs)
|
||||
# Sync YouTube/Dropbox/GDrive/GCS deletes: keep them off the API event loop.
|
||||
await asyncio.to_thread(self._delete_public_outputs, job_id, public_outputs)
|
||||
|
||||
# Record the completed move and clear the guard flag. redistribute_video
|
||||
# already updated state_data with the new private brand_code etc.
|
||||
@@ -171,12 +190,61 @@ class VisibilityChangeService:
|
||||
"""
|
||||
logger.info(f"[job:{job_id}] Starting visibility change: private -> public (by {user_email})")
|
||||
|
||||
# Claim the job before any deletes: two concurrent requests that both passed
|
||||
# validate_change must not both recycle the brand code and re-process.
|
||||
claimed = await asyncio.to_thread(
|
||||
self.job_manager.firestore.try_set_job_state_flag, job_id, "visibility_change_in_progress"
|
||||
)
|
||||
if not claimed:
|
||||
raise VisibilityChangeInProgress("A visibility change is already in progress for this job")
|
||||
|
||||
db = self.job_manager.firestore.db
|
||||
job_ref = db.collection("jobs").document(job_id)
|
||||
try:
|
||||
# Steps 1-5 are sync YouTube/Dropbox/GDrive/GCS/Firestore I/O: off the event loop.
|
||||
await asyncio.to_thread(self._reset_job_for_public, job_id, job, user_email, job_ref)
|
||||
except Exception:
|
||||
try:
|
||||
job_ref.update({"state_data.visibility_change_in_progress": DELETE_FIELD})
|
||||
except Exception:
|
||||
# The job now refuses every visibility change until the flag is cleared.
|
||||
logger.exception(f"[job:{job_id}] Failed to release visibility_change_in_progress after a failed reset")
|
||||
raise
|
||||
|
||||
# Step 6: Trigger screens worker
|
||||
from backend.services.worker_service import get_worker_service
|
||||
worker_service = get_worker_service()
|
||||
try:
|
||||
triggered = await worker_service.trigger_screens_worker(job_id)
|
||||
except Exception as e:
|
||||
logger.warning(f"[job:{job_id}] Screens worker trigger raised during visibility change: {e}")
|
||||
triggered = False
|
||||
|
||||
if not triggered:
|
||||
logger.error(f"[job:{job_id}] Failed to trigger screens worker for visibility change")
|
||||
# Restore to complete status on failure
|
||||
job_ref.update({
|
||||
"status": "complete",
|
||||
"is_private": True,
|
||||
"state_data.visibility_change_in_progress": DELETE_FIELD,
|
||||
"state_data.regen_restore_status": DELETE_FIELD,
|
||||
})
|
||||
raise RuntimeError("Failed to trigger screens worker")
|
||||
|
||||
logger.info(f"[job:{job_id}] Visibility change to public initiated, screens worker triggered")
|
||||
return {
|
||||
"status": "processing",
|
||||
"message": "Visibility change to public started. The video will be re-rendered "
|
||||
"with default branding and published publicly. This takes ~15-30 minutes.",
|
||||
"reprocessing_required": True,
|
||||
}
|
||||
|
||||
def _reset_job_for_public(self, job_id: str, job, user_email: str, job_ref) -> None:
|
||||
"""Steps 1-5 of change_to_public (all sync I/O; run via asyncio.to_thread)."""
|
||||
storage = StorageService()
|
||||
|
||||
# Step 1: Delete distributed outputs + GCS finals
|
||||
await self._delete_distributed_outputs(job_id, job, keep_gcs_finals=False)
|
||||
self._delete_distributed_outputs(job_id, job, keep_gcs_finals=False)
|
||||
|
||||
# Step 2: Delete screens and rendered video from GCS
|
||||
for path in [
|
||||
@@ -252,31 +320,7 @@ class VisibilityChangeService:
|
||||
}
|
||||
job_ref.update(update_payload)
|
||||
|
||||
# Step 6: Trigger screens worker
|
||||
from backend.services.worker_service import get_worker_service
|
||||
worker_service = get_worker_service()
|
||||
triggered = await worker_service.trigger_screens_worker(job_id)
|
||||
|
||||
if not triggered:
|
||||
logger.error(f"[job:{job_id}] Failed to trigger screens worker for visibility change")
|
||||
# Restore to complete status on failure
|
||||
job_ref.update({
|
||||
"status": "complete",
|
||||
"is_private": True,
|
||||
"state_data.visibility_change_in_progress": DELETE_FIELD,
|
||||
"state_data.regen_restore_status": DELETE_FIELD,
|
||||
})
|
||||
raise RuntimeError("Failed to trigger screens worker")
|
||||
|
||||
logger.info(f"[job:{job_id}] Visibility change to public initiated, screens worker triggered")
|
||||
return {
|
||||
"status": "processing",
|
||||
"message": "Visibility change to public started. The video will be re-rendered "
|
||||
"with default branding and published publicly. This takes ~15-30 minutes.",
|
||||
"reprocessing_required": True,
|
||||
}
|
||||
|
||||
async def _delete_public_outputs(self, job_id: str, public_outputs: dict):
|
||||
def _delete_public_outputs(self, job_id: str, public_outputs: dict):
|
||||
"""Delete the captured PUBLIC outputs and recycle the public NOMAD number.
|
||||
|
||||
Used by change_to_private AFTER a successful private redistribution. Operates
|
||||
@@ -373,7 +417,7 @@ class VisibilityChangeService:
|
||||
f"public cleanup incomplete (youtube={youtube_ok}, dropbox={dropbox_ok}, gdrive={gdrive_ok})"
|
||||
)
|
||||
|
||||
async def _delete_distributed_outputs(self, job_id: str, job, keep_gcs_finals: bool = False):
|
||||
def _delete_distributed_outputs(self, job_id: str, job, keep_gcs_finals: bool = False):
|
||||
"""Delete distributed outputs (YouTube, Dropbox, GDrive) and recycle brand code."""
|
||||
import re
|
||||
|
||||
|
||||
@@ -0,0 +1,78 @@
|
||||
"""
|
||||
FirestoreService.try_set_job_state_flag is an atomic test-and-set: of several
|
||||
concurrent callers exactly one wins. change_to_public relies on it to claim
|
||||
visibility_change_in_progress before deleting anything.
|
||||
|
||||
Run with: scripts/run-emulator-tests.sh
|
||||
"""
|
||||
|
||||
import threading
|
||||
import uuid
|
||||
from datetime import datetime, UTC
|
||||
|
||||
import pytest
|
||||
|
||||
from backend.tests.emulator.conftest import emulators_running
|
||||
|
||||
pytestmark = pytest.mark.skipif(
|
||||
not emulators_running(),
|
||||
reason="GCP emulators not running. Start with: scripts/start-emulators.sh"
|
||||
)
|
||||
|
||||
if emulators_running():
|
||||
from backend.models.job import Job, JobStatus
|
||||
from backend.services.firestore_service import FirestoreService
|
||||
|
||||
|
||||
def _create_job(service, state_data):
|
||||
job_id = f"tas-{uuid.uuid4().hex[:8]}"
|
||||
service.create_job(Job(
|
||||
job_id=job_id,
|
||||
status=JobStatus.COMPLETE,
|
||||
created_at=datetime.now(UTC),
|
||||
updated_at=datetime.now(UTC),
|
||||
artist="Test",
|
||||
title="Test",
|
||||
state_data=state_data,
|
||||
))
|
||||
return job_id
|
||||
|
||||
|
||||
def test_sets_flag_once():
|
||||
service = FirestoreService()
|
||||
job_id = _create_job(service, {})
|
||||
assert service.try_set_job_state_flag(job_id, "visibility_change_in_progress") is True
|
||||
assert service.try_set_job_state_flag(job_id, "visibility_change_in_progress") is False
|
||||
assert service.get_job(job_id).state_data["visibility_change_in_progress"] is True
|
||||
|
||||
|
||||
def test_already_set_flag_is_not_claimed():
|
||||
service = FirestoreService()
|
||||
job_id = _create_job(service, {"visibility_change_in_progress": True})
|
||||
assert service.try_set_job_state_flag(job_id, "visibility_change_in_progress") is False
|
||||
|
||||
|
||||
def test_missing_job_is_not_claimed():
|
||||
assert FirestoreService().try_set_job_state_flag("no-such-job", "visibility_change_in_progress") is False
|
||||
|
||||
|
||||
def test_exactly_one_concurrent_caller_wins():
|
||||
service = FirestoreService()
|
||||
job_id = _create_job(service, {})
|
||||
barrier = threading.Barrier(4)
|
||||
results = []
|
||||
|
||||
def attempt():
|
||||
barrier.wait()
|
||||
try:
|
||||
results.append(FirestoreService().try_set_job_state_flag(job_id, "visibility_change_in_progress"))
|
||||
except Exception as e: # contention abort after retries: a loss, never a second winner
|
||||
results.append(e)
|
||||
|
||||
threads = [threading.Thread(target=attempt) for _ in range(4)]
|
||||
for t in threads:
|
||||
t.start()
|
||||
for t in threads:
|
||||
t.join()
|
||||
assert len(results) == 4
|
||||
assert [r for r in results if r is True] == [True]
|
||||
@@ -119,7 +119,7 @@ class TestChangeToPrivate:
|
||||
|
||||
call_order = []
|
||||
mock_redist = AsyncMock(return_value=True, side_effect=lambda *a, **k: call_order.append("redist") or True)
|
||||
mock_delete = AsyncMock(side_effect=lambda *a, **k: call_order.append("delete"))
|
||||
mock_delete = MagicMock(side_effect=lambda *a, **k: call_order.append("delete"))
|
||||
|
||||
with patch("backend.services.visibility_change_service.VisibilityChangeService._delete_public_outputs", mock_delete), \
|
||||
patch("backend.workers.video_worker.redistribute_video", mock_redist):
|
||||
@@ -146,7 +146,7 @@ class TestChangeToPrivate:
|
||||
service = VisibilityChangeService(job_manager=mock_job_manager)
|
||||
job = _make_job(is_private=False, state_data={})
|
||||
|
||||
with patch("backend.services.visibility_change_service.VisibilityChangeService._delete_public_outputs", new_callable=AsyncMock), \
|
||||
with patch("backend.services.visibility_change_service.VisibilityChangeService._delete_public_outputs"), \
|
||||
patch("backend.workers.video_worker.redistribute_video", new_callable=AsyncMock, return_value=True):
|
||||
|
||||
await service.change_to_private("test-123", job, "user@example.com")
|
||||
@@ -175,7 +175,7 @@ class TestChangeToPublic:
|
||||
"dropbox_link": "https://dropbox.com/test",
|
||||
})
|
||||
|
||||
with patch("backend.services.visibility_change_service.VisibilityChangeService._delete_distributed_outputs", new_callable=AsyncMock), \
|
||||
with patch("backend.services.visibility_change_service.VisibilityChangeService._delete_distributed_outputs"), \
|
||||
patch("backend.services.visibility_change_service.StorageService") as mock_storage_cls, \
|
||||
patch("backend.api.routes.file_upload._prepare_theme_for_job", return_value=("jobs/test-123/style/style_params.json", {"bg": "bg.png"}, None)) as mock_theme, \
|
||||
patch("backend.services.worker_service.get_worker_service") as mock_worker_svc:
|
||||
@@ -205,7 +205,7 @@ class TestChangeToPublic:
|
||||
service = VisibilityChangeService(job_manager=mock_job_manager)
|
||||
job = _make_job(is_private=True, state_data={})
|
||||
|
||||
with patch("backend.services.visibility_change_service.VisibilityChangeService._delete_distributed_outputs", new_callable=AsyncMock), \
|
||||
with patch("backend.services.visibility_change_service.VisibilityChangeService._delete_distributed_outputs"), \
|
||||
patch("backend.services.visibility_change_service.StorageService") as mock_storage_cls, \
|
||||
patch("backend.api.routes.file_upload._prepare_theme_for_job", return_value=("jobs/test-123/style/style_params.json", {"bg": "bg.png"}, None)), \
|
||||
patch("backend.services.worker_service.get_worker_service") as mock_worker_svc:
|
||||
@@ -233,7 +233,7 @@ class TestChangeToPublic:
|
||||
service = VisibilityChangeService(job_manager=mock_job_manager)
|
||||
job = _make_job(is_private=True, state_data={})
|
||||
|
||||
with patch("backend.services.visibility_change_service.VisibilityChangeService._delete_distributed_outputs", new_callable=AsyncMock), \
|
||||
with patch("backend.services.visibility_change_service.VisibilityChangeService._delete_distributed_outputs"), \
|
||||
patch("backend.services.visibility_change_service.StorageService") as mock_storage_cls, \
|
||||
patch("backend.api.routes.file_upload._prepare_theme_for_job", return_value=("jobs/test-123/style/style_params.json", {"bg": "bg.png"}, None)), \
|
||||
patch("backend.services.worker_service.get_worker_service") as mock_worker_svc:
|
||||
@@ -277,7 +277,7 @@ class TestChangeToPublic:
|
||||
service = VisibilityChangeService(job_manager=mock_job_manager)
|
||||
job = _make_job(is_private=True, state_data={})
|
||||
|
||||
with patch("backend.services.visibility_change_service.VisibilityChangeService._delete_distributed_outputs", new_callable=AsyncMock), \
|
||||
with patch("backend.services.visibility_change_service.VisibilityChangeService._delete_distributed_outputs"), \
|
||||
patch("backend.services.visibility_change_service.StorageService") as mock_storage_cls, \
|
||||
patch("backend.api.routes.file_upload._prepare_theme_for_job", return_value=("p.json", {}, None)), \
|
||||
patch("backend.services.worker_service.get_worker_service") as mock_worker_svc:
|
||||
@@ -301,7 +301,7 @@ class TestChangeToPublic:
|
||||
service = VisibilityChangeService(job_manager=mock_job_manager)
|
||||
job = _make_job(is_private=True, state_data={})
|
||||
|
||||
with patch("backend.services.visibility_change_service.VisibilityChangeService._delete_distributed_outputs", new_callable=AsyncMock), \
|
||||
with patch("backend.services.visibility_change_service.VisibilityChangeService._delete_distributed_outputs"), \
|
||||
patch("backend.services.visibility_change_service.StorageService") as mock_storage_cls, \
|
||||
patch("backend.api.routes.file_upload._prepare_theme_for_job", return_value=("jobs/test-123/style/style_params.json", {}, None)), \
|
||||
patch("backend.services.worker_service.get_worker_service") as mock_worker_svc:
|
||||
@@ -321,6 +321,96 @@ class TestChangeToPublic:
|
||||
assert payload["is_private"] is True
|
||||
|
||||
|
||||
class TestChangeToPublicClaim:
|
||||
"""change_to_public claims visibility_change_in_progress atomically before deleting anything."""
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_lost_claim_deletes_nothing(self):
|
||||
from backend.services.visibility_change_service import VisibilityChangeInProgress
|
||||
|
||||
mock_job_manager = MagicMock()
|
||||
mock_job_manager.firestore.try_set_job_state_flag.return_value = False
|
||||
service = VisibilityChangeService(job_manager=mock_job_manager)
|
||||
job = _make_job(is_private=True, state_data={})
|
||||
|
||||
with patch("backend.services.visibility_change_service.VisibilityChangeService._delete_distributed_outputs") as mock_delete:
|
||||
with pytest.raises(VisibilityChangeInProgress):
|
||||
await service.change_to_public("test-123", job, "user@example.com")
|
||||
mock_delete.assert_not_called()
|
||||
mock_job_manager.firestore.try_set_job_state_flag.assert_called_once_with("test-123", "visibility_change_in_progress")
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_failed_reset_releases_the_claim(self):
|
||||
mock_job_manager = MagicMock()
|
||||
mock_job_manager.firestore.try_set_job_state_flag.return_value = True
|
||||
mock_job_ref = MagicMock()
|
||||
mock_job_manager.firestore.db.collection.return_value.document.return_value = mock_job_ref
|
||||
service = VisibilityChangeService(job_manager=mock_job_manager)
|
||||
job = _make_job(is_private=True, state_data={})
|
||||
|
||||
with patch("backend.services.visibility_change_service.VisibilityChangeService._delete_distributed_outputs",
|
||||
side_effect=RuntimeError("dropbox down")):
|
||||
with pytest.raises(RuntimeError, match="dropbox down"):
|
||||
await service.change_to_public("test-123", job, "user@example.com")
|
||||
released = mock_job_ref.update.call_args_list[-1].args[0]
|
||||
assert list(released) == ["state_data.visibility_change_in_progress"]
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_trigger_exception_rolls_back_like_a_failed_trigger(self):
|
||||
mock_job_manager = MagicMock()
|
||||
mock_job_ref = MagicMock()
|
||||
mock_job_manager.firestore.db.collection.return_value.document.return_value = mock_job_ref
|
||||
service = VisibilityChangeService(job_manager=mock_job_manager)
|
||||
job = _make_job(is_private=True, state_data={})
|
||||
|
||||
with patch("backend.services.visibility_change_service.VisibilityChangeService._delete_distributed_outputs"), \
|
||||
patch("backend.services.visibility_change_service.StorageService"), \
|
||||
patch("backend.api.routes.file_upload._prepare_theme_for_job", return_value=("p", {}, None)), \
|
||||
patch("backend.services.worker_service.get_worker_service") as mock_worker_svc:
|
||||
mock_worker_svc.return_value.trigger_screens_worker = AsyncMock(side_effect=ConnectionError("tasks down"))
|
||||
with pytest.raises(RuntimeError, match="Failed to trigger screens worker"):
|
||||
await service.change_to_public("test-123", job, "user@example.com")
|
||||
rollback = mock_job_ref.update.call_args_list[-1].args[0]
|
||||
assert rollback["status"] == "complete"
|
||||
assert "state_data.visibility_change_in_progress" in rollback
|
||||
|
||||
|
||||
class TestChangeToPrivateClaim:
|
||||
@pytest.mark.asyncio
|
||||
async def test_lost_claim_touches_nothing(self):
|
||||
from backend.services.visibility_change_service import VisibilityChangeInProgress
|
||||
|
||||
mock_job_manager = MagicMock()
|
||||
mock_job_manager.firestore.try_set_job_state_flag.return_value = False
|
||||
mock_job_ref = MagicMock()
|
||||
mock_job_manager.firestore.db.collection.return_value.document.return_value = mock_job_ref
|
||||
service = VisibilityChangeService(job_manager=mock_job_manager)
|
||||
job = _make_job(is_private=False, state_data={"brand_code": "NOMAD-0001"})
|
||||
|
||||
redist = AsyncMock(return_value=True)
|
||||
with patch("backend.workers.video_worker.redistribute_video", redist):
|
||||
with pytest.raises(VisibilityChangeInProgress):
|
||||
await service.change_to_private("test-123", job, "user@example.com")
|
||||
redist.assert_not_awaited()
|
||||
mock_job_ref.update.assert_not_called()
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_chained_continuation_keeps_its_existing_claim(self):
|
||||
mock_job_manager = MagicMock()
|
||||
mock_job_ref = MagicMock()
|
||||
mock_job_manager.firestore.db.collection.return_value.document.return_value = mock_job_ref
|
||||
service = VisibilityChangeService(job_manager=mock_job_manager)
|
||||
job = _make_job(is_private=False, state_data={"brand_code": "NOMAD-0001", "visibility_change_in_progress": True})
|
||||
|
||||
with patch("backend.workers.video_worker.redistribute_video", AsyncMock(return_value=True)), \
|
||||
patch("backend.services.visibility_change_service.VisibilityChangeService._delete_public_outputs"):
|
||||
result = await service.change_to_private(
|
||||
"test-123", job, "user@example.com", regenerate_if_missing=False, holds_claim=True)
|
||||
assert result["status"] == "success"
|
||||
mock_job_manager.firestore.try_set_job_state_flag.assert_not_called()
|
||||
|
||||
|
||||
class TestChangeToPrivateErrors:
|
||||
"""Test error handling in public -> private flow."""
|
||||
|
||||
@@ -339,7 +429,7 @@ class TestChangeToPrivateErrors:
|
||||
"gdrive_files": {"mp4": "id1"},
|
||||
})
|
||||
|
||||
mock_delete = AsyncMock()
|
||||
mock_delete = MagicMock()
|
||||
with patch("backend.services.visibility_change_service.VisibilityChangeService._delete_public_outputs", mock_delete), \
|
||||
patch("backend.workers.video_worker.redistribute_video", new_callable=AsyncMock, return_value=False):
|
||||
|
||||
@@ -373,7 +463,7 @@ class TestDeleteDistributedOutputs:
|
||||
mock_brand_svc = MagicMock()
|
||||
mock_bcs.return_value = mock_brand_svc
|
||||
|
||||
await service._delete_distributed_outputs("test-123", job, keep_gcs_finals=True)
|
||||
service._delete_distributed_outputs("test-123", job, keep_gcs_finals=True)
|
||||
|
||||
mock_brand_svc.recycle_brand_code.assert_called_once_with("NOMAD", 42)
|
||||
|
||||
@@ -398,7 +488,7 @@ class TestDeleteDistributedOutputs:
|
||||
mock_brand_svc = MagicMock()
|
||||
mock_bcs.return_value = mock_brand_svc
|
||||
|
||||
await service._delete_public_outputs("test-123", public_outputs)
|
||||
service._delete_public_outputs("test-123", public_outputs)
|
||||
|
||||
# No public YouTube/Dropbox/GDrive present → all deletions "succeed" →
|
||||
# the public number is recycled and the mirror is cleaned.
|
||||
@@ -432,7 +522,7 @@ class TestDeleteDistributedOutputs:
|
||||
mock_brand_svc = MagicMock()
|
||||
mock_bcs.return_value = mock_brand_svc
|
||||
|
||||
await service._delete_public_outputs("test-123", public_outputs)
|
||||
service._delete_public_outputs("test-123", public_outputs)
|
||||
|
||||
mock_brand_svc.recycle_brand_code.assert_not_called()
|
||||
|
||||
@@ -459,7 +549,7 @@ class TestDeleteDistributedOutputs:
|
||||
mock_yt.return_value = MagicMock(is_configured=False)
|
||||
mock_dbx.return_value = MagicMock(is_configured=False)
|
||||
|
||||
await service._delete_distributed_outputs("test-123", job, keep_gcs_finals=True)
|
||||
service._delete_distributed_outputs("test-123", job, keep_gcs_finals=True)
|
||||
|
||||
update_call = mock_job_ref.update.call_args
|
||||
payload = update_call.args[0]
|
||||
@@ -482,7 +572,7 @@ class TestDeleteDistributedOutputs:
|
||||
mock_storage.delete_folder.return_value = 3
|
||||
mock_storage_cls.return_value = mock_storage
|
||||
|
||||
await service._delete_distributed_outputs("test-123", job, keep_gcs_finals=False)
|
||||
service._delete_distributed_outputs("test-123", job, keep_gcs_finals=False)
|
||||
|
||||
mock_storage.delete_folder.assert_called_once_with("jobs/test-123/finals/")
|
||||
|
||||
@@ -499,7 +589,7 @@ class TestDeleteDistributedOutputs:
|
||||
mock_storage = MagicMock()
|
||||
mock_storage_cls.return_value = mock_storage
|
||||
|
||||
await service._delete_distributed_outputs("test-123", job, keep_gcs_finals=True)
|
||||
service._delete_distributed_outputs("test-123", job, keep_gcs_finals=True)
|
||||
|
||||
mock_storage.delete_folder.assert_not_called()
|
||||
|
||||
@@ -511,3 +601,31 @@ class TestStateTransitions:
|
||||
from backend.models.job import STATE_TRANSITIONS, JobStatus
|
||||
allowed = STATE_TRANSITIONS[JobStatus.COMPLETE]
|
||||
assert JobStatus.LYRICS_COMPLETE in allowed
|
||||
|
||||
|
||||
def test_route_maps_lost_visibility_claim_to_409():
|
||||
from fastapi.testclient import TestClient
|
||||
|
||||
from backend.api.dependencies import require_auth
|
||||
from backend.main import app
|
||||
from backend.services.auth_service import AuthResult, UserType
|
||||
from backend.services.visibility_change_service import VisibilityChangeInProgress
|
||||
|
||||
async def override():
|
||||
return AuthResult(is_valid=True, user_type=UserType.UNLIMITED, remaining_uses=-1, message="Valid",
|
||||
user_email="user@example.com", is_admin=False)
|
||||
|
||||
app.dependency_overrides[require_auth] = override
|
||||
try:
|
||||
job = _make_job(is_private=True, state_data={})
|
||||
job.user_email = "user@example.com"
|
||||
job_manager = MagicMock()
|
||||
job_manager.get_job.return_value = job
|
||||
with patch("backend.api.routes.jobs.JobManager", return_value=job_manager), \
|
||||
patch.object(VisibilityChangeService, "change_to_public",
|
||||
AsyncMock(side_effect=VisibilityChangeInProgress("A visibility change is already in progress for this job"))):
|
||||
resp = TestClient(app).post(f"/api/jobs/{job.job_id}/change-visibility", json={"target_visibility": "public"})
|
||||
finally:
|
||||
app.dependency_overrides.clear()
|
||||
assert resp.status_code == 409, resp.text
|
||||
assert "already in progress" in resp.json()["detail"]
|
||||
@@ -0,0 +1,109 @@
|
||||
"""Job-log Firestore writes must not run on the API event loop (log_write_offload)."""
|
||||
import asyncio
|
||||
import queue
|
||||
import threading
|
||||
from unittest.mock import MagicMock
|
||||
|
||||
import pytest
|
||||
|
||||
from backend.services import log_write_offload
|
||||
|
||||
|
||||
@pytest.fixture(autouse=True)
|
||||
def _not_in_cloud_run_job(monkeypatch):
|
||||
monkeypatch.delenv("CLOUD_RUN_JOB", raising=False)
|
||||
yield
|
||||
assert log_write_offload.drain(timeout=5)
|
||||
|
||||
|
||||
def test_runs_inline_off_the_event_loop():
|
||||
ran_on = []
|
||||
log_write_offload.submit_or_run(lambda: ran_on.append(threading.get_ident()))
|
||||
assert ran_on == [threading.get_ident()]
|
||||
|
||||
|
||||
def test_offloads_from_a_running_event_loop():
|
||||
release = threading.Event()
|
||||
ran_on = []
|
||||
|
||||
def slow_write():
|
||||
release.wait(5)
|
||||
ran_on.append(threading.get_ident())
|
||||
|
||||
async def on_loop():
|
||||
log_write_offload.submit_or_run(slow_write)
|
||||
# submit returned without waiting for the (blocked) write
|
||||
assert ran_on == []
|
||||
return threading.get_ident()
|
||||
|
||||
loop_thread = asyncio.run(on_loop())
|
||||
release.set()
|
||||
assert log_write_offload.drain(timeout=5)
|
||||
assert len(ran_on) == 1 and ran_on[0] != loop_thread
|
||||
|
||||
|
||||
def test_runs_inline_on_the_loop_inside_a_cloud_run_job(monkeypatch):
|
||||
monkeypatch.setenv("CLOUD_RUN_JOB", "video-encoding-job")
|
||||
ran_on = []
|
||||
|
||||
async def on_loop():
|
||||
log_write_offload.submit_or_run(lambda: ran_on.append(threading.get_ident()))
|
||||
return threading.get_ident()
|
||||
|
||||
assert ran_on == [asyncio.run(on_loop())]
|
||||
|
||||
|
||||
def test_offloaded_writes_keep_submission_order():
|
||||
written = []
|
||||
|
||||
async def on_loop():
|
||||
for i in range(50):
|
||||
log_write_offload.submit_or_run(lambda i=i: written.append(i))
|
||||
|
||||
asyncio.run(on_loop())
|
||||
assert log_write_offload.drain(timeout=5)
|
||||
assert written == list(range(50))
|
||||
|
||||
|
||||
def test_full_queue_drops_with_a_warning(monkeypatch, caplog):
|
||||
release = threading.Event()
|
||||
monkeypatch.setattr(log_write_offload, "_queue", queue.Queue(maxsize=1))
|
||||
monkeypatch.setattr(log_write_offload, "_worker", None)
|
||||
monkeypatch.setattr(log_write_offload, "_dropped", 0)
|
||||
monkeypatch.setattr(log_write_offload, "_last_drop_warning", 0.0)
|
||||
written = []
|
||||
|
||||
async def on_loop():
|
||||
log_write_offload.submit_or_run(lambda: release.wait(5)) # occupies the writer
|
||||
await asyncio.sleep(0.05)
|
||||
log_write_offload.submit_or_run(lambda: written.append("queued")) # fills the queue
|
||||
log_write_offload.submit_or_run(lambda: written.append("dropped")) # no room
|
||||
|
||||
with caplog.at_level("WARNING", logger=log_write_offload.__name__):
|
||||
asyncio.run(on_loop())
|
||||
release.set()
|
||||
assert log_write_offload.drain(timeout=5)
|
||||
assert written == ["queued"]
|
||||
assert log_write_offload._dropped == 1
|
||||
assert "dropped 1 job-log lines" in caplog.text
|
||||
|
||||
|
||||
def test_firestore_append_log_does_not_write_on_the_loop_thread():
|
||||
from backend.models.worker_log import WorkerLogEntry
|
||||
from backend.services.firestore_service import FirestoreService
|
||||
|
||||
service = FirestoreService.__new__(FirestoreService)
|
||||
service.collection = "jobs"
|
||||
service.db = MagicMock()
|
||||
doc_ref = service.db.collection.return_value.document.return_value.collection.return_value.document.return_value
|
||||
set_threads = []
|
||||
doc_ref.set.side_effect = lambda data: set_threads.append(threading.get_ident())
|
||||
|
||||
async def on_loop():
|
||||
service.append_log_to_subcollection("job1", WorkerLogEntry.create(job_id="job1", worker="screens", level="INFO", message="hi"))
|
||||
return threading.get_ident()
|
||||
|
||||
loop_thread = asyncio.run(on_loop())
|
||||
assert log_write_offload.drain(timeout=5)
|
||||
assert len(set_threads) == 1 and set_threads[0] != loop_thread
|
||||
assert doc_ref.set.call_args.args[0]["message"] == "hi"
|
||||
@@ -209,6 +209,14 @@ BLOCKING_CALLS = {
|
||||
"upload_json", # GCS upload (complete_review: up to 21s)
|
||||
"_prepare_theme_for_job", # GCS reads/writes (job creation)
|
||||
"generate", # CustomLyricsService.generate → sync Gemini call (151s, 32 requests)
|
||||
# 2026-10-19 watchdog culprits (7 days of EVENT_LOOP_STALL stacks): sync Firestore /
|
||||
# GCS reads in the auth dependencies, tenant middleware and routes.
|
||||
"validate_token_full", # require_auth: every authenticated request (up to 1.9s)
|
||||
"validate_token",
|
||||
"validate_session",
|
||||
"get_tenant_config", # GCS read on a cache miss (tenant middleware, up to 1.2s)
|
||||
"append_log_to_subcollection", # job-log write; on the loop it is offloaded, see log_write_offload
|
||||
"append_worker_log",
|
||||
}
|
||||
_BLOCKING_MODULE_CALLS = {("requests", "get"), ("requests", "post"), ("requests", "put"),
|
||||
("requests", "delete"), ("requests", "request"), ("time", "sleep")}
|
||||
@@ -226,13 +234,14 @@ def _blocking_call_name(call: ast.Call) -> str | None:
|
||||
return None
|
||||
|
||||
|
||||
def find_blocking_calls_in_async_routes(source: str) -> list[str]:
|
||||
"""`async def` routes that directly call a known-blocking function (excluding
|
||||
nested sync helpers, which are presumably run via to_thread)."""
|
||||
def find_blocking_calls_in_async_routes(source: str, routes_only: bool = True) -> list[str]:
|
||||
"""`async def` routes (or, with routes_only=False, any `async def`) that directly
|
||||
call a known-blocking function (excluding nested sync helpers, which are presumably
|
||||
run via to_thread)."""
|
||||
tree = ast.parse(source)
|
||||
hits = []
|
||||
for fn in ast.walk(tree):
|
||||
if not (isinstance(fn, ast.AsyncFunctionDef) and _is_route(fn)):
|
||||
if not (isinstance(fn, ast.AsyncFunctionDef) and (_is_route(fn) or not routes_only)):
|
||||
continue
|
||||
stack = list(fn.body)
|
||||
while stack:
|
||||
@@ -283,3 +292,33 @@ async def helper_ok():
|
||||
hits = find_blocking_calls_in_async_routes(src)
|
||||
assert sorted(h.split(" line")[0] for h in hits) == [
|
||||
"bad: check_dropbox_credentials()", "bad: requests.post()"]
|
||||
|
||||
|
||||
# Auth dependencies and middleware run on the event loop for (nearly) every request, so
|
||||
# every `async def` in them is held to the same rule as a route, not just decorated ones.
|
||||
ALWAYS_ON_LOOP_FILES = [BACKEND / "api" / "dependencies.py"] + sorted((BACKEND / "middleware").glob("*.py"))
|
||||
|
||||
|
||||
def test_no_known_blocking_calls_in_async_dependencies_or_middleware():
|
||||
offenders = []
|
||||
for path in ALWAYS_ON_LOOP_FILES:
|
||||
for hit in find_blocking_calls_in_async_routes(path.read_text(), routes_only=False):
|
||||
offenders.append(f"{path.relative_to(BACKEND.parent)}::{hit}")
|
||||
assert not offenders, (
|
||||
"These `async def` dependencies/middleware call known-blocking functions on the "
|
||||
"event loop, which stalls every request. Use `await asyncio.to_thread(fn, ...)`:\n "
|
||||
+ "\n ".join(sorted(offenders))
|
||||
)
|
||||
|
||||
|
||||
def test_blocking_call_detector_scans_non_route_async_functions():
|
||||
src = '''
|
||||
async def require_auth(request):
|
||||
return auth_service.validate_token_full(token)
|
||||
|
||||
async def fixed(request):
|
||||
return await asyncio.to_thread(auth_service.validate_token_full, token)
|
||||
'''
|
||||
assert find_blocking_calls_in_async_routes(src) == []
|
||||
hits = find_blocking_calls_in_async_routes(src, routes_only=False)
|
||||
assert [h.split(" line")[0] for h in hits] == ["require_auth: validate_token_full()"]
|
||||
@@ -229,7 +229,7 @@ async def _run_suggest(job_id: str, data: dict, segments: list, reference_lyrics
|
||||
from backend.services.auto_correct.settings import AutoCorrectSettings
|
||||
from backend.services.job_manager import JobManager
|
||||
|
||||
job = JobManager().get_job(job_id)
|
||||
job = await asyncio.to_thread(JobManager().get_job, job_id)
|
||||
service = get_auto_correct_service()
|
||||
result = await asyncio.wait_for(
|
||||
asyncio.to_thread(
|
||||
|
||||
@@ -610,7 +610,7 @@ async def _run_chained_change_to_private(job_id: str, job_manager: JobManager, r
|
||||
try:
|
||||
fresh = job_manager.get_job(job_id)
|
||||
await VisibilityChangeService(job_manager).change_to_private(
|
||||
job_id, fresh, requested_by, regenerate_if_missing=False,
|
||||
job_id, fresh, requested_by, regenerate_if_missing=False, holds_claim=True,
|
||||
)
|
||||
job_log.info("Regenerated finals redistributed: visibility changed to private")
|
||||
except Exception as e: # noqa: BLE001 - the regenerate itself succeeded
|
||||
|
||||
@@ -67,6 +67,13 @@ The "servers unavailable" banner (49 episodes in 2 weeks) was mostly the API's o
|
||||
- The 2 user-felt ones were cold-start `importlib` stalls (4-15s) from routes that lazily import modules pulling in `httpx`. Lifespan startup now imports `main._LAZY_ROUTE_IMPORTS` in a worker thread before readiness. A background warmup thread alone would still race the first request.
|
||||
- **An `await` somewhere in the handler doesn't make its other calls safe** (v0.267.1). `mark_uploads_complete`, `create_job_with_upload_urls` and others awaited `request.json()`, so they passed the guard, yet called `check_dropbox_credentials()` inline. When the Dropbox token had expired, that did an HTTP refresh plus a Secret Manager write and version cleanup on the loop, a 16s freeze. `trigger_gdrive_validation_endpoint` froze 5-10s after every job (OIDC fetch + POST). Both are now behind `asyncio.to_thread`, and the guard's `BLOCKING_CALLS` denylist fails CI if a known-blocking function is called directly in an `async def` route. Add to the denylist when the watchdog names a new culprit.
|
||||
- **Mostly-sync routes that await one trigger** (v0.268.1). `complete_review` (21s corrections `upload_json`), `create_job_from_search` (theme prep), and the `recover_stuck_jobs` / `retry_pending_render_jobs` crons (~260 stalls/week from Firestore scans) each awaited a worker trigger, so the guard passed them. Their bodies now run in a worker thread, and the trigger coroutines go back to the API loop with `backend/utils/loop_bridge.run_on_loop` (`run_coroutine_threadsafe`, 120s timeout), so loop APIs inside the triggers still work. Bodies that call `run_on_loop` run on their own pool (`run_in_body_thread`), not the default executor. Otherwise they could hold every default-pool thread while the coroutines they wait on need one, a deadlock. Watch out for an `async def` helper with no awaits inside, such as `_validate_audio_durations` (2 downloads + 2 decodes) or the YouTube queue's `_process_single_upload` (a whole upload): it's sync work on the loop with an async signature. To rank culprits, group `EVENT_LOOP_STALL in progress` stacks by their first `api/routes/` frame.
|
||||
- **Auth, middleware and job logs were blocking the loop too** (v0.278.14). Over 7 days of stall stacks (2026-10-12→19), the top app-level sources were:
|
||||
- the in-process screens worker and preview encodes, through their job-log handlers. Every log line was a sync Firestore write (`append_log_to_subcollection`);
|
||||
- `require_auth`, through `validate_token_full`. That is a sync Firestore session read on **every** authenticated request;
|
||||
- `require_review_auth` (`get_job`);
|
||||
- the tenant middleware (`get_tenant_config`, a GCS read on a cache miss).
|
||||
|
||||
Job-log writes made on the API loop now go to one background thread (`backend/services/log_write_offload.py`). They stay inline in worker threads and Cloud Run Jobs, where nothing else waits on the loop. The auth dependencies and tenant middleware use `asyncio.to_thread`. The guard now holds every `async def` in `api/dependencies.py` and `middleware/` to `BLOCKING_CALLS`, not just decorated routes. Still on the loop: about 70 direct `get_job`/`update_job` calls in `async def` routes. Each is a single 50-200 ms Firestore round-trip and rarely shows up in the watchdog. Move them when one does.
|
||||
- **Detection:** `backend/services/loop_watchdog.py` logs `EVENT_LOOP_STALL` with the loop thread's stack and a `culprit=file:line`. Grep for it before guessing.
|
||||
- **Alert on impact, not on gaps** (v0.268.5). After the CPU-throttle discount, the stall alert still flapped. Cloud Run autoscaling starts extra instances that serve one request, then sit idle and throttled while the startup warmup thread, which spends 10+ minutes loading SpaCy, competes for slices. Every 10-40s ERROR "stall" over 4 days (11 of them) was on an instance with **zero** requests, and all had an empty culprit. A freeze advances the 1/5/15s sample schedule without sampling, so whatever happens after the thaw has no stack. The watchdog now takes a `RequestTracker`, fed by the outermost pure-ASGI `RequestTrackingMiddleware`. A stall reaches ERROR only if a request was in flight at its start or arrived during it. A request that queued during the stall is counted if it starts within the 0.5s grace window after the loop resumes. Check whether anyone waited before chasing a culprit: join `labels.instanceId` between the stall lines and the request logs.
|
||||
- **Async services with sync bodies** (v0.268.7). `PushNotificationService.send_push` was an `async def` with no real awaits: a Firestore read, a `webpush` POST per device, and a Firestore update, all on the loop. The route guard can't see it because it's a service method. The watchdog caught it at 1.2s with a request waiting. The body is now `send_push_sync`, and `send_push` calls `await asyncio.to_thread(...)`. When `requests_affected>0` lines name a `backend/services/*` frame, that service method is the fix site. Also since v0.268.7: if a process freeze skips every sample point and the loop is still blocked once the process thaws, the watchdog takes one sample then, so the stall gets a culprit.
|
||||
|
||||
+1
-1
@@ -1,6 +1,6 @@
|
||||
[tool.poetry]
|
||||
name = "karaoke-gen"
|
||||
version = "0.278.13"
|
||||
version = "0.278.14"
|
||||
description = "Generate karaoke videos with synchronized lyrics. Handles the entire process from downloading audio and lyrics to creating the final video with title screens."
|
||||
authors = ["Andrew Beveridge <andrew@beveridge.uk>"]
|
||||
license = "MIT"
|
||||
|
||||
Reference in new issue
Block a user