Files
Andrew Beveridge ☄️andClaude Opus 5.5 70bdd0e693 fix(alerts): start-review 409, per-cause crash patterns, one alert per job failure (v0.290.1) (#1243)
Section 1 of the 2026-11-03 #alerts triage handoff: a stale push on a
finished song got a 500, and the error monitor could stay silent on a
new crash.

Alert:
https://discord.com/channels/1198691101943541770/1265517528608669707/1566980744448843886
Alert:
https://discord.com/channels/1198691101943541770/1265517528608669707/1566980746394996879
Alert:
https://discord.com/channels/1198691101943541770/1265517528608669707/1566980747905073224
Alert:
https://discord.com/channels/1198691101943541770/1265517528608669707/1565697725280948294
Alert:
https://discord.com/channels/1198691101943541770/1265517528608669707/1565803187334815775
Alert:
https://discord.com/channels/1198691101943541770/1265517528608669707/1565818418072916110
Alert:
https://discord.com/channels/1198691101943541770/1265517528608669707/1566225593354420226
Alert:
https://discord.com/channels/1198691101943541770/1265517528608669707/1566256141074169987
Alert:
https://discord.com/channels/1198691101943541770/1265517528608669707/1567207141994135654
Alert:
https://discord.com/channels/1198691101943541770/1265517528608669707/1567252495535841386
Alert:
https://discord.com/channels/1198691101943541770/1265517528608669707/1567252497620537415

## What changed

**1a. start-review:** a job not in review gets 409
(`jobs.jobNotInReviewState`) with no transition attempted. Losing the
auto-approver race is a 409 too. `correction-data` has the same race,
and its `except Exception` turned it into a 500; it now returns 409
(`except HTTPException: raise` added). Its status gate still returns
400, as before.

**1b. Crash patterns:** `Exception in ASGI application` and
`request_audit_error` become `Unhandled exception: <Type: message>`,
read from the entry's `exception` field. The handoff's suspicion is
confirmed in `upsert_pattern`: an existing `new` pattern returns
`is_new=False`, so a second, different crash stayed silent. Both log
lines of one crash now share one pattern. The path and method go in the
sample only.

I read the line after the last frame rather than the traceback's last
line. pydantic's last line is a docs link shared by every failure, and
group tracebacks end in a separator.

**1c. One alert per failure:**
- YouTube ids are normalized out of signatures, which also fixes the
false "first time" line in `ops_alerts`.
- `Job <id> failed` (both call sites in `job_manager`) logs WARNING,
since the status write sends the job-failed alert.
- `URL download failed:` is ignored in the monitor, not demoted, because
`file_upload.py` belongs to the section 4 PR.
- `[job:<id>] Download failed` logs WARNING for the expected
`DownloadError`/`FlacfetchServiceError` types. This wasn't named in the
handoff, but it is the same duplicate on the audio-search path.
Unexpected exceptions there stay ERROR with a traceback.

**1d. Suppressions and demotions:**
- Zalo's `zaloJSV2 is not defined`: dropped at `/api/client-errors`,
only with a Zalo UA.
- `python -c` experiments on `audio-separation-job`: ignored, using the
handoff's regex.
- Spotify 403 on `/lyrics`: WARNING with `response.text[:200]`. 401 on
`/lyrics` and 401/403 on `/search` stay ERROR.
- Dropbox token probe: WARNING. A rejected refresh used to log nothing,
so it now logs ERROR, as does having no refresh credentials.
- decide-search health aborts: this was a small change. `should_ignore`
takes an optional `request_url` and matches the exact path
`/api/search/health`.

**1e. Cold start:** `anyio._backends._asyncio` is added to
`_LAZY_ROUTE_IMPORTS`. The prod stack from 10-30 confirms the stall was
in that import. Lifespan also sends one `/api/health` request through
the app in a worker thread, bounded by the existing 30s timeout and
best-effort.

## Testing

- TDD: new tests failed first, then passed.
- Affected suites: 826 passed.
- `backend/tests` (excluding `backend/tests/emulator`): 7088 passed, 8
failed, 29 skipped. All 8 are in `test_emulator_integration.py`, and
that file also shows 8 failures at the base commit a8f9618a. I did not
compare the 8 test names one by one.
- `tests/unit` outside the listed files was not run.
`test_audio_fetcher.py` and `test_remote_cli.py` fail to collect in my
environment; I didn't change either file.
- Read-only replay of real prod entries through the new code: the 11-03
crash gave one specific pattern with `POST
/api/jobs/7f89f884/start-review` in the sample, the decide-search health
aborts and the experiment traceback were ignored.
- Local CodeRabbit review completed with 0 findings.

## Not verified

- Nothing is live, so the alert volume and the cold-start stall haven't
been measured. The warm request has only run in tests, not on a real
Cloud Run cold start.
- A request's first-use imports beyond the anyio backend aren't
individually checked.
- My test environment has no `flacfetch`, so its lazy imports only log a
warning here.
- Zalo: the trigger is one report (11-03).

## Known leftovers

- `validate_state_transition` still logs ERROR before raising, so the
auto-approver race (and the existing in_review race) still emits one
`Invalid state transition` ERROR even though the client now gets a clean
response. The stale-push case no longer reaches it. I left this because
other callers rely on that log.
- A crash that also logs its own ERROR first still gives two patterns,
the ERROR and `Unhandled exception`.
- The first bot block after deploy shows "first time" once, because the
normalized signature is new.
- Old `Exception in ASGI application` and `request_audit_error` patterns
stop recurring and will auto-resolve.
- If the section 4 PR rewords `URL download failed:`, the ignore entry
in `known_issues.py` needs the same wording.

## For Andrew (infra and cost, not changed here)

- Cloud Run startup probe and min-instances for `karaoke-backend`, if
cold-start stalls continue after this ships.
- decide-search's min-0 is why its health pings abort. The suppression
hides those, but real search paths will still alert on the same abort.

@coderabbitai ignore

🤖 Generated with [Claude Code](https://claude.com/claude-code)

https://claude.ai/code/session_012RYLVTvqXyL4QyA2EjzQMG

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-03 23:04:04 +00:00

484 lines
21 KiB
Python

"""
FastAPI application entry point for karaoke generation backend.
(Trivial comment touch to trigger backend CI on ephemeral runners after
the 2026-05-17 dispatcher e2 fix; safe to remove on next backend edit.)
"""
import asyncio
import logging
import threading
import time
from contextlib import asynccontextmanager
from typing import Any
from fastapi import FastAPI
from fastapi.middleware.cors import CORSMiddleware
from backend.config import settings, validate_production_config
from backend.services.fake_workers import fake_workers_enabled
from backend.api.routes import health, jobs, internal, file_upload, review, auth, audio_search, themes, users, admin, tenant, tenant_admin, tenant_bulk, tenant_theme, rate_limits, push, catalog, encoding_worker, client_errors, client_events, bulk, parse_titles
from backend.services.tracing import setup_tracing, instrument_app, get_current_trace_id
from backend.services.structured_logging import setup_structured_logging
from backend.services.spacy_preloader import preload_spacy_model
from backend.services.nltk_preloader import preload_all_nltk_resources
from backend.services.langfuse_preloader import preload_langfuse_handler
from backend.middleware.audit_logging import AuditLoggingMiddleware
from backend.middleware.tenant import TenantMiddleware
from backend.middleware.edge_auth import EdgeAuthMiddleware
from backend.workers.registry import worker_registry
from backend.version import VERSION
# Configure structured logging (JSON in Cloud Run, human-readable locally)
# This must happen before any logging calls
setup_structured_logging()
logger = logging.getLogger(__name__)
# Initialize OpenTelemetry tracing (must happen before app creation)
tracing_enabled = setup_tracing(
service_name="karaoke-backend",
service_version=VERSION,
enable_in_dev=False, # Set to True to enable tracing locally
)
def validate_credentials_on_startup():
"""Validate OAuth credentials on startup and send alerts if needed."""
try:
from backend.services.credential_manager import get_credential_manager, CredentialStatus
manager = get_credential_manager()
results = manager.check_all_credentials()
invalid_services = [
result for result in results.values()
if result.status in (CredentialStatus.INVALID, CredentialStatus.EXPIRED)
]
if invalid_services:
logger.warning(f"Some OAuth credentials need attention:")
for result in invalid_services:
logger.warning(f" - {result.service}: {result.message}")
# Try to send Discord alert
discord_url = settings.get_secret("discord-alert-webhook") if hasattr(settings, 'get_secret') else None
if discord_url:
manager.send_credential_alert(invalid_services, discord_url)
logger.info("Sent credential alert to Discord")
else:
logger.info("All OAuth credentials validated successfully")
except Exception as e:
logger.error(f"Failed to validate credentials on startup: {e}")
# Modules that route handlers import lazily. On a fresh instance the first such
# import ran ON the event loop: httpx/httpcore (via flacfetch_client /
# worker_service) froze it 3-15s on cold disk reads (EVENT_LOOP_STALL in
# importlib, 2026-10-08). Lifespan startup imports them in a worker thread before
# readiness — a background thread alone would still race the first request.
_LAZY_ROUTE_IMPORTS = (
"httpx",
# Imported lazily by httpx / worker_service on first use; still froze the loop
# after the list above existed (httpcore ~1s; google.cloud.run_v2 14.9s with
# 7 requests in flight, 2026-10-16).
"httpcore",
"google.cloud.run_v2",
"backend.services.worker_service",
"backend.services.flacfetch_client",
"backend.services.catalog_proxy_service",
"backend.services.ip_geolocation_service",
"backend.services.email_validation_service",
"backend.services.musicbrainz_service",
# audio_search's auto_download path: select_best -> flacfetch.core.manager ->
# yt_dlp import froze the loop ~11-12s on fresh instances (2026-10-23, 10-26).
"flacfetch.core.manager",
"flacfetch.downloaders.youtube",
"yt_dlp",
# starlette/middleware/base.py picks the anyio backend on the first request:
# 17.3s on the loop with 18 requests in flight (2026-10-30).
"anyio._backends._asyncio",
)
LAZY_ROUTE_IMPORT_TIMEOUT_S = 30
def _import_lazy_route_modules() -> None:
import importlib
for module in _LAZY_ROUTE_IMPORTS:
try:
importlib.import_module(module)
except Exception as e:
logger.warning(f"Startup import of {module} failed (will import lazily): {e}")
def _warm_route_contexts(application: FastAPI) -> int:
"""Build FastAPI's per-route contexts (path regexes, pydantic schemas) up front.
Newer FastAPI builds them lazily on the first request, and the OpenTelemetry
FastAPI instrumentation walks every route (``_get_route_details`` ->
``iter_route_contexts``) for each request's span name. On a fresh instance that
first walk compiled hundreds of route regexes ON the event loop (2026-10-23: 11s,
3 requests; 5.1s on 10-22). The results are cached (by routes version, under a
lock), so one walk here, before readiness, covers every later request.
Best-effort: a FastAPI version without these internals just skips it.
Returns the number of route contexts built (0 if unsupported).
"""
try:
from fastapi.routing import iter_route_contexts
except ImportError: # older FastAPI compiles routes eagerly at include time
return 0
count = 0
for _ in iter_route_contexts(application.routes):
count += 1
for route in application.routes:
low_priority = getattr(route, "effective_low_priority_routes", None)
if callable(low_priority):
count += len(low_priority())
return count
def _warm_middleware_stack(application: Any) -> None:
"""Send one request through the app, so the first user's request isn't what builds
the middleware stack and runs its first-use imports on the loop.
Blocking: runs its own event loop, so call it from a worker thread. /api/health is
exempt from the edge lock, skips the audit log and touches no storage. Best-effort.
"""
import httpx
async def _request() -> None:
transport = httpx.ASGITransport(app=application)
async with httpx.AsyncClient(transport=transport, base_url="http://localhost") as client:
await client.get("/api/health")
start = time.time()
try:
asyncio.run(_request())
except Exception as e: # noqa: BLE001 - warmup must never block startup
logger.warning(f"Middleware warm request failed (stack builds on first request): {e}")
return
logger.info(f"Warmed middleware stack in {time.time() - start:.2f}s")
def _warm_route_contexts_safely(application: FastAPI) -> None:
start = time.time()
try:
built = _warm_route_contexts(application)
except Exception as e: # noqa: BLE001 - warmup must never block startup
logger.warning(f"Route-context warmup failed (routes build lazily): {e}")
return
if built:
logger.info(f"Warmed {built} route contexts in {time.time() - start:.2f}s")
def _run_background_warmup():
"""Warm caches that used to block startup (runs in a daemon thread).
Historically these preloads ran inline in lifespan startup, which meant
Cloud Run held every routed request for the full ~7.5s they take (spaCy
~1s, NLTK ~2.5s, Langfuse ~3s, credential checks ~1.5s) on top of import
time. None of them are needed to serve HTTP: they exist to make the FIRST
lyrics-processing job fast (see docs/archive/2026-01-08-performance-
investigation.md). Running them in a background thread right after boot
keeps that warm-cache property (they finish within seconds, long before
any real job runs) without gating readiness. Workers that race the warmup
simply fall back to the preloaders' lazy paths.
"""
warmup_start = time.time()
# 1. SpaCy model (60+ second delay without preload)
try:
preload_spacy_model("en_core_web_sm")
except Exception as e:
logger.warning(f"SpaCy preload failed (will load lazily): {e}")
# 2. NLTK cmudict (50-100+ second delay without preload)
try:
preload_all_nltk_resources()
except Exception as e:
logger.warning(f"NLTK preload failed (will load lazily): {e}")
# 3. Langfuse callback handler (200+ second delay without preload)
try:
preload_langfuse_handler()
except Exception as e:
logger.warning(f"Langfuse preload failed (will initialize lazily): {e}")
if fake_workers_enabled(settings):
# Local stack: no encoding VM and no OAuth credentials to validate.
logger.info(f"Background warmup complete in {time.time() - warmup_start:.2f}s (fake workers)")
return
# 4. Encoding service singleton — its first construction imports
# google.cloud.compute_v1 (~10s on a fresh instance). Build it here, off the
# event loop, before /api/health/detailed or a preview encode needs it.
try:
from backend.services.encoding_service import get_encoding_service
get_encoding_service()
except Exception as e:
logger.warning(f"Encoding service warmup failed (will build lazily): {e}")
# Validate OAuth credentials (alerting only, nothing depends on it)
try:
validate_credentials_on_startup()
except Exception as e:
logger.error(f"Credential validation failed: {e}")
logger.info(f"Background warmup complete in {time.time() - warmup_start:.2f}s")
@asynccontextmanager
async def lifespan(app: FastAPI):
"""Lifespan event handler for startup and shutdown."""
# Startup
logger.info("Starting karaoke generation backend")
logger.info(f"Environment: {settings.environment}")
logger.info(f"GCS Bucket: {settings.gcs_bucket_name}")
logger.info(f"Tracing enabled: {tracing_enabled}")
# Fail fast if required production config is missing, rather than silently
# running on dev defaults (wrong GCP project / GCS bucket / localhost worker
# URL). No-op outside production. (Fallback audit 2026-06-09, Theme 7.)
validate_production_config()
# Off the loop, but before readiness (Cloud Run holds requests meanwhile).
# Bounded so a hung cold-disk read can't keep the instance from ever going
# ready; the import keeps running in its thread and routes fall back to it.
try:
await asyncio.wait_for(
asyncio.to_thread(_import_lazy_route_modules), timeout=LAZY_ROUTE_IMPORT_TIMEOUT_S
)
except asyncio.TimeoutError:
logger.error(
f"Startup import of lazy route modules exceeded {LAZY_ROUTE_IMPORT_TIMEOUT_S}s; "
"continuing startup (first requests may import them on the event loop)"
)
# Same reason: build the route contexts in a thread, before the first request
# (and the OTel route walk) would build them on the loop.
try:
await asyncio.wait_for(
asyncio.to_thread(_warm_route_contexts_safely, app), timeout=LAZY_ROUTE_IMPORT_TIMEOUT_S
)
except asyncio.TimeoutError:
logger.warning(
f"Route-context warmup exceeded {LAZY_ROUTE_IMPORT_TIMEOUT_S}s; continuing startup"
)
# Same reason, for what only the first request through the middleware touches.
try:
await asyncio.wait_for(
asyncio.to_thread(_warm_middleware_stack, app), timeout=LAZY_ROUTE_IMPORT_TIMEOUT_S
)
except asyncio.TimeoutError:
logger.warning(
f"Middleware warmup exceeded {LAZY_ROUTE_IMPORT_TIMEOUT_S}s; continuing startup"
)
# NLP model / credential warmup runs in the background so it doesn't gate
# readiness — Cloud Run holds all routed requests until lifespan startup
# returns, and these preloads were ~7.5s of the ~16s cold start.
threading.Thread(
target=_run_background_warmup, name="startup-warmup", daemon=True
).start()
# asyncio.to_thread's default executor is min(32, cpu+4) ≈ 6 threads on our
# 2 vCPUs. The screens worker, stale-review cron, auto-correct and many
# review endpoints all offload into it (mostly I/O-bound), so size it up —
# otherwise review-page requests queue behind a few long-running jobs.
import concurrent.futures
asyncio.get_running_loop().set_default_executor(
concurrent.futures.ThreadPoolExecutor(max_workers=48, thread_name_prefix="to-thread")
)
# Event-loop stall watchdog: makes "every request froze for 30s" episodes
# self-diagnosing (logs the blocking stack; records >=5s stalls in prod).
watchdog = None
if settings.loop_watchdog_enabled:
from backend.services.loop_watchdog import (
LoopWatchdog,
firestore_stall_recorder,
request_tracker,
)
watchdog = LoopWatchdog(
recorder=firestore_stall_recorder if settings.environment == "production" else None,
# Stalls nobody waited on (idle, CPU-throttled autoscaled instances)
# stay WARNING so they don't trip the >=10s stall alert.
request_tracker=request_tracker,
)
watchdog.start()
yield
if watchdog is not None:
await watchdog.stop()
# Shutdown - best-effort parking of any still-registered workers.
#
# Reality check (incident 2026-09-13, job 41e06b90): Cloud Run services
# SIGKILL instances ~10 seconds after SIGTERM — NOT the 600s this comment
# previously claimed — and uvicorn only runs this lifespan-shutdown code
# AFTER all BackgroundTasks have completed ("Waiting for background tasks
# to complete"). A long-running worker task therefore keeps uvicorn stuck
# in that wait until SIGKILL, and this hook never executes at all. It can
# NOT be relied on to protect in-flight renders; that protection comes
# from running them as Cloud Run Jobs instead (USE_CLOUD_RUN_JOBS_FOR_RENDER,
# mirroring USE_CLOUD_RUN_JOBS_FOR_VIDEO from incident 2026-03-08).
#
# This hook is kept as a cheap safety net for the legacy flag-off path and
# for the narrow case where workers finish right as SIGTERM lands. The wait
# is capped well inside the 10s kill window so the parking pass below still
# gets a chance to write Firestore state.
logger.info("Shutdown requested, checking for active workers...")
if worker_registry.has_active_workers():
active = worker_registry.get_active_workers()
logger.info(f"Active workers found: {active}")
logger.info("Waiting for workers to complete (timeout: 5s)...")
completed = await worker_registry.wait_for_completion(timeout=5)
if not completed:
logger.error(
"Shutdown timeout - some workers may not have completed cleanly. "
f"Remaining workers: {worker_registry.get_active_workers()}"
)
# Park any still-active render jobs so the auto-retry scheduler can
# recover them. Safe to call unconditionally — it's a no-op if nothing
# render-related is in flight.
try:
from backend.workers.render_video_worker import park_active_render_jobs_for_shutdown
parked = park_active_render_jobs_for_shutdown()
if parked:
logger.warning(
f"Parked {parked} in-flight render job(s) for auto-retry "
f"due to shutdown"
)
except Exception as exc:
logger.error(f"Render-job shutdown park failed: {exc!r}")
else:
logger.info("No active workers, proceeding with shutdown")
logger.info("Shutting down karaoke generation backend")
# Create FastAPI app
app = FastAPI(
title="Karaoke Generator API",
description="Backend API for web-based karaoke video generation",
version=VERSION,
lifespan=lifespan
)
# Instrument FastAPI with OpenTelemetry (adds automatic spans for all requests)
if tracing_enabled:
instrument_app(app)
# Configure CORS
app.add_middleware(
CORSMiddleware,
allow_origins=["*"],
allow_credentials=False,
allow_methods=["*"],
allow_headers=["*"],
)
# Add audit logging middleware (captures all requests with request_id for correlation)
app.add_middleware(AuditLoggingMiddleware)
# Add tenant detection middleware (extracts tenant from subdomain/headers)
app.add_middleware(TenantMiddleware)
# Edge origin lock (added last = outermost, runs first). Rejects direct-to-origin
# requests that bypassed the Cloudflare edge, before tenant/audit middleware do
# any work. No-op unless EDGE_AUTH_MODE is warn/enforce. See
# backend/middleware/edge_auth.py.
app.add_middleware(EdgeAuthMiddleware)
# Counts in-flight requests for the loop watchdog (outermost, so every request that
# waited on a stalled loop counts — even ones the edge lock rejects).
from backend.services.loop_watchdog import RequestTrackingMiddleware # noqa: E402
app.add_middleware(RequestTrackingMiddleware)
# Include routers
app.include_router(health.router, prefix="/api")
app.include_router(jobs.router, prefix="/api")
app.include_router(file_upload.router, prefix="/api") # File upload endpoint
app.include_router(internal.router, prefix="/api") # Internal worker endpoints
app.include_router(encoding_worker.router, prefix="/api") # Encoding worker lifecycle
app.include_router(review.router, prefix="/api") # Review UI compatibility endpoints
app.include_router(auth.router, prefix="/api") # OAuth credential management
app.include_router(audio_search.router, prefix="/api") # Audio search (artist+title mode)
app.include_router(themes.router, prefix="/api") # Theme selection for styles
app.include_router(users.router, prefix="/api") # User auth, credits, and Stripe webhooks
from backend.api.routes import referrals
app.include_router(referrals.router, prefix="/api") # Referral links and payouts
app.include_router(admin.router, prefix="/api") # Admin dashboard and management
app.include_router(rate_limits.router, prefix="/api") # Rate limits admin management
app.include_router(push.router, prefix="/api") # Push notification subscription management
app.include_router(catalog.router, prefix="/api") # Catalog proxy for song/artist autocomplete
app.include_router(parse_titles.router, prefix="/api") # kjbox karaoke-filename parser
app.include_router(bulk.router, prefix="/api") # Bulk Mode: multi-job submission
from backend.api.routes import requests_board
app.include_router(requests_board.router, prefix="/api") # Public song-request voting board
app.include_router(client_errors.router, prefix="/api") # Frontend crash reports
app.include_router(client_events.router, prefix="/api") # Frontend degradation telemetry (banner/unavailable/waveform events)
from backend.api.routes import karaokehunt
app.include_router(karaokehunt.router, prefix="/api") # Retired KaraokeHunt app request interceptor
from backend.api.routes import kjbox
app.include_router(kjbox.router, prefix="/api") # kjbox partner API: singer email-code sign-in + show credits
from backend.api.routes import login_code
app.include_router(login_code.router, prefix="/api") # gen email-code sign-in (/users/auth/send-code, verify-code)
app.include_router(tenant.router) # Tenant/white-label configuration (no /api prefix, router has it)
app.include_router(tenant_admin.router) # Admin tenant provisioning (router has /api prefix)
app.include_router(tenant_bulk.router) # Tenant bulk-upload filename analysis (router has /api prefix)
app.include_router(tenant_theme.router) # Tenant self-service theme editor (router has /api prefix)
if fake_workers_enabled(settings):
# Local E2E stack only (startup refuses FAKE_WORKERS in production).
from backend.api.routes import test_support
app.include_router(test_support.router, prefix="/api")
# Exception handlers
from fastapi import Request
from fastapi.responses import JSONResponse
from backend.exceptions import InsufficientCreditsError
from backend.i18n import t, get_locale_from_request
@app.exception_handler(InsufficientCreditsError)
async def insufficient_credits_exception_handler(request: Request, exc: InsufficientCreditsError):
"""Handle insufficient credits errors with 402 Payment Required status."""
locale = get_locale_from_request(request)
return JSONResponse(
status_code=402,
content={
"detail": t(locale, "errors.insufficientCredits"),
"credits_available": exc.credits_available,
"credits_required": exc.credits_required,
"buy_url": t(locale, "errors.insufficientCreditsBuyUrl"),
},
)
@app.get("/")
async def root():
"""Root endpoint."""
return {
"service": "karaoke-gen-backend",
"version": VERSION,
"status": "running"
}
if __name__ == "__main__":
import uvicorn
uvicorn.run(app, host="0.0.0.0", port=8080)