mirror of
https://github.com/nomadkaraoke/karaoke-gen.git
synced 2026-11-20 13:07:57 +09:00
Section 1 of the 2026-11-03 #alerts triage handoff: a stale push on a
finished song got a 500, and the error monitor could stay silent on a
new crash.
Alert:
https://discord.com/channels/1198691101943541770/1265517528608669707/1566980744448843886
Alert:
https://discord.com/channels/1198691101943541770/1265517528608669707/1566980746394996879
Alert:
https://discord.com/channels/1198691101943541770/1265517528608669707/1566980747905073224
Alert:
https://discord.com/channels/1198691101943541770/1265517528608669707/1565697725280948294
Alert:
https://discord.com/channels/1198691101943541770/1265517528608669707/1565803187334815775
Alert:
https://discord.com/channels/1198691101943541770/1265517528608669707/1565818418072916110
Alert:
https://discord.com/channels/1198691101943541770/1265517528608669707/1566225593354420226
Alert:
https://discord.com/channels/1198691101943541770/1265517528608669707/1566256141074169987
Alert:
https://discord.com/channels/1198691101943541770/1265517528608669707/1567207141994135654
Alert:
https://discord.com/channels/1198691101943541770/1265517528608669707/1567252495535841386
Alert:
https://discord.com/channels/1198691101943541770/1265517528608669707/1567252497620537415
## What changed
**1a. start-review:** a job not in review gets 409
(`jobs.jobNotInReviewState`) with no transition attempted. Losing the
auto-approver race is a 409 too. `correction-data` has the same race,
and its `except Exception` turned it into a 500; it now returns 409
(`except HTTPException: raise` added). Its status gate still returns
400, as before.
**1b. Crash patterns:** `Exception in ASGI application` and
`request_audit_error` become `Unhandled exception: <Type: message>`,
read from the entry's `exception` field. The handoff's suspicion is
confirmed in `upsert_pattern`: an existing `new` pattern returns
`is_new=False`, so a second, different crash stayed silent. Both log
lines of one crash now share one pattern. The path and method go in the
sample only.
I read the line after the last frame rather than the traceback's last
line. pydantic's last line is a docs link shared by every failure, and
group tracebacks end in a separator.
**1c. One alert per failure:**
- YouTube ids are normalized out of signatures, which also fixes the
false "first time" line in `ops_alerts`.
- `Job <id> failed` (both call sites in `job_manager`) logs WARNING,
since the status write sends the job-failed alert.
- `URL download failed:` is ignored in the monitor, not demoted, because
`file_upload.py` belongs to the section 4 PR.
- `[job:<id>] Download failed` logs WARNING for the expected
`DownloadError`/`FlacfetchServiceError` types. This wasn't named in the
handoff, but it is the same duplicate on the audio-search path.
Unexpected exceptions there stay ERROR with a traceback.
**1d. Suppressions and demotions:**
- Zalo's `zaloJSV2 is not defined`: dropped at `/api/client-errors`,
only with a Zalo UA.
- `python -c` experiments on `audio-separation-job`: ignored, using the
handoff's regex.
- Spotify 403 on `/lyrics`: WARNING with `response.text[:200]`. 401 on
`/lyrics` and 401/403 on `/search` stay ERROR.
- Dropbox token probe: WARNING. A rejected refresh used to log nothing,
so it now logs ERROR, as does having no refresh credentials.
- decide-search health aborts: this was a small change. `should_ignore`
takes an optional `request_url` and matches the exact path
`/api/search/health`.
**1e. Cold start:** `anyio._backends._asyncio` is added to
`_LAZY_ROUTE_IMPORTS`. The prod stack from 10-30 confirms the stall was
in that import. Lifespan also sends one `/api/health` request through
the app in a worker thread, bounded by the existing 30s timeout and
best-effort.
## Testing
- TDD: new tests failed first, then passed.
- Affected suites: 826 passed.
- `backend/tests` (excluding `backend/tests/emulator`): 7088 passed, 8
failed, 29 skipped. All 8 are in `test_emulator_integration.py`, and
that file also shows 8 failures at the base commit a8f9618a. I did not
compare the 8 test names one by one.
- `tests/unit` outside the listed files was not run.
`test_audio_fetcher.py` and `test_remote_cli.py` fail to collect in my
environment; I didn't change either file.
- Read-only replay of real prod entries through the new code: the 11-03
crash gave one specific pattern with `POST
/api/jobs/7f89f884/start-review` in the sample, the decide-search health
aborts and the experiment traceback were ignored.
- Local CodeRabbit review completed with 0 findings.
## Not verified
- Nothing is live, so the alert volume and the cold-start stall haven't
been measured. The warm request has only run in tests, not on a real
Cloud Run cold start.
- A request's first-use imports beyond the anyio backend aren't
individually checked.
- My test environment has no `flacfetch`, so its lazy imports only log a
warning here.
- Zalo: the trigger is one report (11-03).
## Known leftovers
- `validate_state_transition` still logs ERROR before raising, so the
auto-approver race (and the existing in_review race) still emits one
`Invalid state transition` ERROR even though the client now gets a clean
response. The stale-push case no longer reaches it. I left this because
other callers rely on that log.
- A crash that also logs its own ERROR first still gives two patterns,
the ERROR and `Unhandled exception`.
- The first bot block after deploy shows "first time" once, because the
normalized signature is new.
- Old `Exception in ASGI application` and `request_audit_error` patterns
stop recurring and will auto-resolve.
- If the section 4 PR rewords `URL download failed:`, the ignore entry
in `known_issues.py` needs the same wording.
## For Andrew (infra and cost, not changed here)
- Cloud Run startup probe and min-instances for `karaoke-backend`, if
cold-start stalls continue after this ships.
- decide-search's min-0 is why its health pings abort. The suppression
hides those, but real search paths will still alert on the same abort.
@coderabbitai ignore
🤖 Generated with [Claude Code](https://claude.com/claude-code)
https://claude.ai/code/session_012RYLVTvqXyL4QyA2EjzQMG
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
484 lines
21 KiB
Python
484 lines
21 KiB
Python
"""
|
|
FastAPI application entry point for karaoke generation backend.
|
|
|
|
(Trivial comment touch to trigger backend CI on ephemeral runners after
|
|
the 2026-05-17 dispatcher e2 fix; safe to remove on next backend edit.)
|
|
"""
|
|
import asyncio
|
|
import logging
|
|
import threading
|
|
import time
|
|
from contextlib import asynccontextmanager
|
|
from typing import Any
|
|
from fastapi import FastAPI
|
|
from fastapi.middleware.cors import CORSMiddleware
|
|
|
|
from backend.config import settings, validate_production_config
|
|
from backend.services.fake_workers import fake_workers_enabled
|
|
from backend.api.routes import health, jobs, internal, file_upload, review, auth, audio_search, themes, users, admin, tenant, tenant_admin, tenant_bulk, tenant_theme, rate_limits, push, catalog, encoding_worker, client_errors, client_events, bulk, parse_titles
|
|
from backend.services.tracing import setup_tracing, instrument_app, get_current_trace_id
|
|
from backend.services.structured_logging import setup_structured_logging
|
|
from backend.services.spacy_preloader import preload_spacy_model
|
|
from backend.services.nltk_preloader import preload_all_nltk_resources
|
|
from backend.services.langfuse_preloader import preload_langfuse_handler
|
|
from backend.middleware.audit_logging import AuditLoggingMiddleware
|
|
from backend.middleware.tenant import TenantMiddleware
|
|
from backend.middleware.edge_auth import EdgeAuthMiddleware
|
|
from backend.workers.registry import worker_registry
|
|
|
|
|
|
from backend.version import VERSION
|
|
|
|
|
|
# Configure structured logging (JSON in Cloud Run, human-readable locally)
|
|
# This must happen before any logging calls
|
|
setup_structured_logging()
|
|
logger = logging.getLogger(__name__)
|
|
|
|
# Initialize OpenTelemetry tracing (must happen before app creation)
|
|
tracing_enabled = setup_tracing(
|
|
service_name="karaoke-backend",
|
|
service_version=VERSION,
|
|
enable_in_dev=False, # Set to True to enable tracing locally
|
|
)
|
|
|
|
|
|
def validate_credentials_on_startup():
|
|
"""Validate OAuth credentials on startup and send alerts if needed."""
|
|
try:
|
|
from backend.services.credential_manager import get_credential_manager, CredentialStatus
|
|
|
|
manager = get_credential_manager()
|
|
results = manager.check_all_credentials()
|
|
|
|
invalid_services = [
|
|
result for result in results.values()
|
|
if result.status in (CredentialStatus.INVALID, CredentialStatus.EXPIRED)
|
|
]
|
|
|
|
if invalid_services:
|
|
logger.warning(f"Some OAuth credentials need attention:")
|
|
for result in invalid_services:
|
|
logger.warning(f" - {result.service}: {result.message}")
|
|
|
|
# Try to send Discord alert
|
|
discord_url = settings.get_secret("discord-alert-webhook") if hasattr(settings, 'get_secret') else None
|
|
if discord_url:
|
|
manager.send_credential_alert(invalid_services, discord_url)
|
|
logger.info("Sent credential alert to Discord")
|
|
else:
|
|
logger.info("All OAuth credentials validated successfully")
|
|
|
|
except Exception as e:
|
|
logger.error(f"Failed to validate credentials on startup: {e}")
|
|
|
|
|
|
# Modules that route handlers import lazily. On a fresh instance the first such
|
|
# import ran ON the event loop: httpx/httpcore (via flacfetch_client /
|
|
# worker_service) froze it 3-15s on cold disk reads (EVENT_LOOP_STALL in
|
|
# importlib, 2026-10-08). Lifespan startup imports them in a worker thread before
|
|
# readiness — a background thread alone would still race the first request.
|
|
_LAZY_ROUTE_IMPORTS = (
|
|
"httpx",
|
|
# Imported lazily by httpx / worker_service on first use; still froze the loop
|
|
# after the list above existed (httpcore ~1s; google.cloud.run_v2 14.9s with
|
|
# 7 requests in flight, 2026-10-16).
|
|
"httpcore",
|
|
"google.cloud.run_v2",
|
|
"backend.services.worker_service",
|
|
"backend.services.flacfetch_client",
|
|
"backend.services.catalog_proxy_service",
|
|
"backend.services.ip_geolocation_service",
|
|
"backend.services.email_validation_service",
|
|
"backend.services.musicbrainz_service",
|
|
# audio_search's auto_download path: select_best -> flacfetch.core.manager ->
|
|
# yt_dlp import froze the loop ~11-12s on fresh instances (2026-10-23, 10-26).
|
|
"flacfetch.core.manager",
|
|
"flacfetch.downloaders.youtube",
|
|
"yt_dlp",
|
|
# starlette/middleware/base.py picks the anyio backend on the first request:
|
|
# 17.3s on the loop with 18 requests in flight (2026-10-30).
|
|
"anyio._backends._asyncio",
|
|
)
|
|
|
|
|
|
LAZY_ROUTE_IMPORT_TIMEOUT_S = 30
|
|
|
|
|
|
def _import_lazy_route_modules() -> None:
|
|
import importlib
|
|
|
|
for module in _LAZY_ROUTE_IMPORTS:
|
|
try:
|
|
importlib.import_module(module)
|
|
except Exception as e:
|
|
logger.warning(f"Startup import of {module} failed (will import lazily): {e}")
|
|
|
|
|
|
def _warm_route_contexts(application: FastAPI) -> int:
|
|
"""Build FastAPI's per-route contexts (path regexes, pydantic schemas) up front.
|
|
|
|
Newer FastAPI builds them lazily on the first request, and the OpenTelemetry
|
|
FastAPI instrumentation walks every route (``_get_route_details`` ->
|
|
``iter_route_contexts``) for each request's span name. On a fresh instance that
|
|
first walk compiled hundreds of route regexes ON the event loop (2026-10-23: 11s,
|
|
3 requests; 5.1s on 10-22). The results are cached (by routes version, under a
|
|
lock), so one walk here, before readiness, covers every later request.
|
|
Best-effort: a FastAPI version without these internals just skips it.
|
|
Returns the number of route contexts built (0 if unsupported).
|
|
"""
|
|
try:
|
|
from fastapi.routing import iter_route_contexts
|
|
except ImportError: # older FastAPI compiles routes eagerly at include time
|
|
return 0
|
|
count = 0
|
|
for _ in iter_route_contexts(application.routes):
|
|
count += 1
|
|
for route in application.routes:
|
|
low_priority = getattr(route, "effective_low_priority_routes", None)
|
|
if callable(low_priority):
|
|
count += len(low_priority())
|
|
return count
|
|
|
|
|
|
def _warm_middleware_stack(application: Any) -> None:
|
|
"""Send one request through the app, so the first user's request isn't what builds
|
|
the middleware stack and runs its first-use imports on the loop.
|
|
|
|
Blocking: runs its own event loop, so call it from a worker thread. /api/health is
|
|
exempt from the edge lock, skips the audit log and touches no storage. Best-effort.
|
|
"""
|
|
import httpx
|
|
|
|
async def _request() -> None:
|
|
transport = httpx.ASGITransport(app=application)
|
|
async with httpx.AsyncClient(transport=transport, base_url="http://localhost") as client:
|
|
await client.get("/api/health")
|
|
|
|
start = time.time()
|
|
try:
|
|
asyncio.run(_request())
|
|
except Exception as e: # noqa: BLE001 - warmup must never block startup
|
|
logger.warning(f"Middleware warm request failed (stack builds on first request): {e}")
|
|
return
|
|
logger.info(f"Warmed middleware stack in {time.time() - start:.2f}s")
|
|
|
|
|
|
def _warm_route_contexts_safely(application: FastAPI) -> None:
|
|
start = time.time()
|
|
try:
|
|
built = _warm_route_contexts(application)
|
|
except Exception as e: # noqa: BLE001 - warmup must never block startup
|
|
logger.warning(f"Route-context warmup failed (routes build lazily): {e}")
|
|
return
|
|
if built:
|
|
logger.info(f"Warmed {built} route contexts in {time.time() - start:.2f}s")
|
|
|
|
|
|
def _run_background_warmup():
|
|
"""Warm caches that used to block startup (runs in a daemon thread).
|
|
|
|
Historically these preloads ran inline in lifespan startup, which meant
|
|
Cloud Run held every routed request for the full ~7.5s they take (spaCy
|
|
~1s, NLTK ~2.5s, Langfuse ~3s, credential checks ~1.5s) on top of import
|
|
time. None of them are needed to serve HTTP: they exist to make the FIRST
|
|
lyrics-processing job fast (see docs/archive/2026-01-08-performance-
|
|
investigation.md). Running them in a background thread right after boot
|
|
keeps that warm-cache property (they finish within seconds, long before
|
|
any real job runs) without gating readiness. Workers that race the warmup
|
|
simply fall back to the preloaders' lazy paths.
|
|
"""
|
|
warmup_start = time.time()
|
|
|
|
# 1. SpaCy model (60+ second delay without preload)
|
|
try:
|
|
preload_spacy_model("en_core_web_sm")
|
|
except Exception as e:
|
|
logger.warning(f"SpaCy preload failed (will load lazily): {e}")
|
|
|
|
# 2. NLTK cmudict (50-100+ second delay without preload)
|
|
try:
|
|
preload_all_nltk_resources()
|
|
except Exception as e:
|
|
logger.warning(f"NLTK preload failed (will load lazily): {e}")
|
|
|
|
# 3. Langfuse callback handler (200+ second delay without preload)
|
|
try:
|
|
preload_langfuse_handler()
|
|
except Exception as e:
|
|
logger.warning(f"Langfuse preload failed (will initialize lazily): {e}")
|
|
|
|
if fake_workers_enabled(settings):
|
|
# Local stack: no encoding VM and no OAuth credentials to validate.
|
|
logger.info(f"Background warmup complete in {time.time() - warmup_start:.2f}s (fake workers)")
|
|
return
|
|
|
|
# 4. Encoding service singleton — its first construction imports
|
|
# google.cloud.compute_v1 (~10s on a fresh instance). Build it here, off the
|
|
# event loop, before /api/health/detailed or a preview encode needs it.
|
|
try:
|
|
from backend.services.encoding_service import get_encoding_service
|
|
|
|
get_encoding_service()
|
|
except Exception as e:
|
|
logger.warning(f"Encoding service warmup failed (will build lazily): {e}")
|
|
|
|
# Validate OAuth credentials (alerting only, nothing depends on it)
|
|
try:
|
|
validate_credentials_on_startup()
|
|
except Exception as e:
|
|
logger.error(f"Credential validation failed: {e}")
|
|
|
|
logger.info(f"Background warmup complete in {time.time() - warmup_start:.2f}s")
|
|
|
|
|
|
@asynccontextmanager
|
|
async def lifespan(app: FastAPI):
|
|
"""Lifespan event handler for startup and shutdown."""
|
|
# Startup
|
|
logger.info("Starting karaoke generation backend")
|
|
logger.info(f"Environment: {settings.environment}")
|
|
logger.info(f"GCS Bucket: {settings.gcs_bucket_name}")
|
|
logger.info(f"Tracing enabled: {tracing_enabled}")
|
|
|
|
# Fail fast if required production config is missing, rather than silently
|
|
# running on dev defaults (wrong GCP project / GCS bucket / localhost worker
|
|
# URL). No-op outside production. (Fallback audit 2026-06-09, Theme 7.)
|
|
validate_production_config()
|
|
|
|
# Off the loop, but before readiness (Cloud Run holds requests meanwhile).
|
|
# Bounded so a hung cold-disk read can't keep the instance from ever going
|
|
# ready; the import keeps running in its thread and routes fall back to it.
|
|
try:
|
|
await asyncio.wait_for(
|
|
asyncio.to_thread(_import_lazy_route_modules), timeout=LAZY_ROUTE_IMPORT_TIMEOUT_S
|
|
)
|
|
except asyncio.TimeoutError:
|
|
logger.error(
|
|
f"Startup import of lazy route modules exceeded {LAZY_ROUTE_IMPORT_TIMEOUT_S}s; "
|
|
"continuing startup (first requests may import them on the event loop)"
|
|
)
|
|
|
|
# Same reason: build the route contexts in a thread, before the first request
|
|
# (and the OTel route walk) would build them on the loop.
|
|
try:
|
|
await asyncio.wait_for(
|
|
asyncio.to_thread(_warm_route_contexts_safely, app), timeout=LAZY_ROUTE_IMPORT_TIMEOUT_S
|
|
)
|
|
except asyncio.TimeoutError:
|
|
logger.warning(
|
|
f"Route-context warmup exceeded {LAZY_ROUTE_IMPORT_TIMEOUT_S}s; continuing startup"
|
|
)
|
|
|
|
# Same reason, for what only the first request through the middleware touches.
|
|
try:
|
|
await asyncio.wait_for(
|
|
asyncio.to_thread(_warm_middleware_stack, app), timeout=LAZY_ROUTE_IMPORT_TIMEOUT_S
|
|
)
|
|
except asyncio.TimeoutError:
|
|
logger.warning(
|
|
f"Middleware warmup exceeded {LAZY_ROUTE_IMPORT_TIMEOUT_S}s; continuing startup"
|
|
)
|
|
|
|
# NLP model / credential warmup runs in the background so it doesn't gate
|
|
# readiness — Cloud Run holds all routed requests until lifespan startup
|
|
# returns, and these preloads were ~7.5s of the ~16s cold start.
|
|
threading.Thread(
|
|
target=_run_background_warmup, name="startup-warmup", daemon=True
|
|
).start()
|
|
|
|
# asyncio.to_thread's default executor is min(32, cpu+4) ≈ 6 threads on our
|
|
# 2 vCPUs. The screens worker, stale-review cron, auto-correct and many
|
|
# review endpoints all offload into it (mostly I/O-bound), so size it up —
|
|
# otherwise review-page requests queue behind a few long-running jobs.
|
|
import concurrent.futures
|
|
|
|
asyncio.get_running_loop().set_default_executor(
|
|
concurrent.futures.ThreadPoolExecutor(max_workers=48, thread_name_prefix="to-thread")
|
|
)
|
|
|
|
# Event-loop stall watchdog: makes "every request froze for 30s" episodes
|
|
# self-diagnosing (logs the blocking stack; records >=5s stalls in prod).
|
|
watchdog = None
|
|
if settings.loop_watchdog_enabled:
|
|
from backend.services.loop_watchdog import (
|
|
LoopWatchdog,
|
|
firestore_stall_recorder,
|
|
request_tracker,
|
|
)
|
|
|
|
watchdog = LoopWatchdog(
|
|
recorder=firestore_stall_recorder if settings.environment == "production" else None,
|
|
# Stalls nobody waited on (idle, CPU-throttled autoscaled instances)
|
|
# stay WARNING so they don't trip the >=10s stall alert.
|
|
request_tracker=request_tracker,
|
|
)
|
|
watchdog.start()
|
|
|
|
yield
|
|
|
|
if watchdog is not None:
|
|
await watchdog.stop()
|
|
|
|
# Shutdown - best-effort parking of any still-registered workers.
|
|
#
|
|
# Reality check (incident 2026-09-13, job 41e06b90): Cloud Run services
|
|
# SIGKILL instances ~10 seconds after SIGTERM — NOT the 600s this comment
|
|
# previously claimed — and uvicorn only runs this lifespan-shutdown code
|
|
# AFTER all BackgroundTasks have completed ("Waiting for background tasks
|
|
# to complete"). A long-running worker task therefore keeps uvicorn stuck
|
|
# in that wait until SIGKILL, and this hook never executes at all. It can
|
|
# NOT be relied on to protect in-flight renders; that protection comes
|
|
# from running them as Cloud Run Jobs instead (USE_CLOUD_RUN_JOBS_FOR_RENDER,
|
|
# mirroring USE_CLOUD_RUN_JOBS_FOR_VIDEO from incident 2026-03-08).
|
|
#
|
|
# This hook is kept as a cheap safety net for the legacy flag-off path and
|
|
# for the narrow case where workers finish right as SIGTERM lands. The wait
|
|
# is capped well inside the 10s kill window so the parking pass below still
|
|
# gets a chance to write Firestore state.
|
|
logger.info("Shutdown requested, checking for active workers...")
|
|
if worker_registry.has_active_workers():
|
|
active = worker_registry.get_active_workers()
|
|
logger.info(f"Active workers found: {active}")
|
|
logger.info("Waiting for workers to complete (timeout: 5s)...")
|
|
completed = await worker_registry.wait_for_completion(timeout=5)
|
|
if not completed:
|
|
logger.error(
|
|
"Shutdown timeout - some workers may not have completed cleanly. "
|
|
f"Remaining workers: {worker_registry.get_active_workers()}"
|
|
)
|
|
|
|
# Park any still-active render jobs so the auto-retry scheduler can
|
|
# recover them. Safe to call unconditionally — it's a no-op if nothing
|
|
# render-related is in flight.
|
|
try:
|
|
from backend.workers.render_video_worker import park_active_render_jobs_for_shutdown
|
|
parked = park_active_render_jobs_for_shutdown()
|
|
if parked:
|
|
logger.warning(
|
|
f"Parked {parked} in-flight render job(s) for auto-retry "
|
|
f"due to shutdown"
|
|
)
|
|
except Exception as exc:
|
|
logger.error(f"Render-job shutdown park failed: {exc!r}")
|
|
else:
|
|
logger.info("No active workers, proceeding with shutdown")
|
|
|
|
logger.info("Shutting down karaoke generation backend")
|
|
|
|
|
|
# Create FastAPI app
|
|
app = FastAPI(
|
|
title="Karaoke Generator API",
|
|
description="Backend API for web-based karaoke video generation",
|
|
version=VERSION,
|
|
lifespan=lifespan
|
|
)
|
|
|
|
# Instrument FastAPI with OpenTelemetry (adds automatic spans for all requests)
|
|
if tracing_enabled:
|
|
instrument_app(app)
|
|
|
|
# Configure CORS
|
|
app.add_middleware(
|
|
CORSMiddleware,
|
|
allow_origins=["*"],
|
|
allow_credentials=False,
|
|
allow_methods=["*"],
|
|
allow_headers=["*"],
|
|
)
|
|
|
|
# Add audit logging middleware (captures all requests with request_id for correlation)
|
|
app.add_middleware(AuditLoggingMiddleware)
|
|
|
|
# Add tenant detection middleware (extracts tenant from subdomain/headers)
|
|
app.add_middleware(TenantMiddleware)
|
|
|
|
# Edge origin lock (added last = outermost, runs first). Rejects direct-to-origin
|
|
# requests that bypassed the Cloudflare edge, before tenant/audit middleware do
|
|
# any work. No-op unless EDGE_AUTH_MODE is warn/enforce. See
|
|
# backend/middleware/edge_auth.py.
|
|
app.add_middleware(EdgeAuthMiddleware)
|
|
|
|
# Counts in-flight requests for the loop watchdog (outermost, so every request that
|
|
# waited on a stalled loop counts — even ones the edge lock rejects).
|
|
from backend.services.loop_watchdog import RequestTrackingMiddleware # noqa: E402
|
|
|
|
app.add_middleware(RequestTrackingMiddleware)
|
|
|
|
# Include routers
|
|
app.include_router(health.router, prefix="/api")
|
|
app.include_router(jobs.router, prefix="/api")
|
|
app.include_router(file_upload.router, prefix="/api") # File upload endpoint
|
|
app.include_router(internal.router, prefix="/api") # Internal worker endpoints
|
|
app.include_router(encoding_worker.router, prefix="/api") # Encoding worker lifecycle
|
|
app.include_router(review.router, prefix="/api") # Review UI compatibility endpoints
|
|
app.include_router(auth.router, prefix="/api") # OAuth credential management
|
|
app.include_router(audio_search.router, prefix="/api") # Audio search (artist+title mode)
|
|
app.include_router(themes.router, prefix="/api") # Theme selection for styles
|
|
app.include_router(users.router, prefix="/api") # User auth, credits, and Stripe webhooks
|
|
from backend.api.routes import referrals
|
|
app.include_router(referrals.router, prefix="/api") # Referral links and payouts
|
|
app.include_router(admin.router, prefix="/api") # Admin dashboard and management
|
|
app.include_router(rate_limits.router, prefix="/api") # Rate limits admin management
|
|
app.include_router(push.router, prefix="/api") # Push notification subscription management
|
|
app.include_router(catalog.router, prefix="/api") # Catalog proxy for song/artist autocomplete
|
|
app.include_router(parse_titles.router, prefix="/api") # kjbox karaoke-filename parser
|
|
app.include_router(bulk.router, prefix="/api") # Bulk Mode: multi-job submission
|
|
from backend.api.routes import requests_board
|
|
app.include_router(requests_board.router, prefix="/api") # Public song-request voting board
|
|
app.include_router(client_errors.router, prefix="/api") # Frontend crash reports
|
|
app.include_router(client_events.router, prefix="/api") # Frontend degradation telemetry (banner/unavailable/waveform events)
|
|
from backend.api.routes import karaokehunt
|
|
app.include_router(karaokehunt.router, prefix="/api") # Retired KaraokeHunt app request interceptor
|
|
from backend.api.routes import kjbox
|
|
app.include_router(kjbox.router, prefix="/api") # kjbox partner API: singer email-code sign-in + show credits
|
|
from backend.api.routes import login_code
|
|
app.include_router(login_code.router, prefix="/api") # gen email-code sign-in (/users/auth/send-code, verify-code)
|
|
app.include_router(tenant.router) # Tenant/white-label configuration (no /api prefix, router has it)
|
|
app.include_router(tenant_admin.router) # Admin tenant provisioning (router has /api prefix)
|
|
app.include_router(tenant_bulk.router) # Tenant bulk-upload filename analysis (router has /api prefix)
|
|
app.include_router(tenant_theme.router) # Tenant self-service theme editor (router has /api prefix)
|
|
if fake_workers_enabled(settings):
|
|
# Local E2E stack only (startup refuses FAKE_WORKERS in production).
|
|
from backend.api.routes import test_support
|
|
app.include_router(test_support.router, prefix="/api")
|
|
|
|
|
|
# Exception handlers
|
|
from fastapi import Request
|
|
from fastapi.responses import JSONResponse
|
|
from backend.exceptions import InsufficientCreditsError
|
|
from backend.i18n import t, get_locale_from_request
|
|
|
|
|
|
@app.exception_handler(InsufficientCreditsError)
|
|
async def insufficient_credits_exception_handler(request: Request, exc: InsufficientCreditsError):
|
|
"""Handle insufficient credits errors with 402 Payment Required status."""
|
|
locale = get_locale_from_request(request)
|
|
return JSONResponse(
|
|
status_code=402,
|
|
content={
|
|
"detail": t(locale, "errors.insufficientCredits"),
|
|
"credits_available": exc.credits_available,
|
|
"credits_required": exc.credits_required,
|
|
"buy_url": t(locale, "errors.insufficientCreditsBuyUrl"),
|
|
},
|
|
)
|
|
|
|
|
|
@app.get("/")
|
|
async def root():
|
|
"""Root endpoint."""
|
|
return {
|
|
"service": "karaoke-gen-backend",
|
|
"version": VERSION,
|
|
"status": "running"
|
|
}
|
|
|
|
|
|
if __name__ == "__main__":
|
|
import uvicorn
|
|
uvicorn.run(app, host="0.0.0.0", port=8080)
|
|
|