mirror of
https://github.com/nomadkaraoke/karaoke-gen.git
synced 2026-11-20 13:07:57 +09:00
@coderabbitai ignore ## Why The residual event-loop freezes BACKLOG item. Seven days of `EVENT_LOOP_STALL in progress` stacks (2026-10-12→19, 355 samples), grouped by their outermost app frame, ranked the app-level loop blockers: | Source | Samples | Max | |---|---|---| | in-process screens worker → job-log handler → sync Firestore write per log line | 30 | 2.2s | | `require_auth` → `validate_token_full` (session read on **every** authed request) | 19 | 1.9s | | preview encodes (`_run_preview_encode`/`_encode_preview_in_background`) → job-log write | 15 | 2.4s | | `change_visibility` → sync Dropbox/GCS deletes | 6 | 1.2s | | tenant middleware → `get_tenant_config` (GCS on cache miss) | 5 | 1.2s | | `require_review_auth` → `get_job` | 5 | 2.2s | | `auto_correct_worker._run_suggest` → `get_job` (the only stall since #1192) | 1 | 1.2s | (The render/video in-process stalls stopped with #1192.) ## What - **`backend/services/log_write_offload.py`**: when a job-log write is made on a running event loop in the API process, it goes to one background writer thread. The queue is bounded at 5000 and a drop is logged at WARNING, rate-limited; `atexit` drains the queue. Writes stay inline in threads and Cloud Run Jobs, so the worker code paths behave exactly as before. Logs are read back ordered by their own `timestamp`, so the hand-off doesn't reorder them. - **`asyncio.to_thread`** for `require_auth`/`require_auth_legacy`/`require_review_auth`, the tenant middleware, `rerender_with_current_theme`'s tenant lookup, `auto_correct._run_suggest`, and `change_visibility`'s `get_job`. - **Visibility change:** `_delete_*_outputs` are now plain `def`, and the sync steps run in one worker thread. Moving them off the loop removed the implicit serialisation, so both directions now **claim `visibility_change_in_progress` atomically** before touching anything. They use the new `FirestoreService.try_set_job_state_flag`, a transactional test-and-set. A lost claim returns 409. The regenerate→private continuation passes `holds_claim=True`, because its regenerate already set the flag. The claim is released if the reset fails, and a raising screens trigger rolls back like a `False` one. - **Guard (`test_no_blocking_async_routes.py`):** `BLOCKING_CALLS` gains `validate_token(_full)`, `validate_session`, `get_tenant_config`, `append_log_to_subcollection` and `append_worker_log`. Every `async def` in `api/dependencies.py` and `middleware/` is now checked, not just decorated routes. Mutation-checked: reverting `require_auth` fails the guard. - `docs/LESSONS-LEARNED.md` entry. About 70 direct `get_job`/`update_job` calls in async routes remain. They're single round-trips that rarely show in the watchdog, and the doc says to move them when one does. ## Testing - New: `test_log_write_offload.py` (inline vs offloaded, Cloud Run Job inline, ordering, full-queue drop, Firestore append off the loop thread), claim tests in `test_change_visibility.py` (lost claim, release on failure, chained continuation, trigger exception, route 409), and emulator `test_try_set_job_state_flag.py` (once-only, already set, missing job, one winner among 4 concurrent callers). - Backend: 6478 passed. The 8 failures are `test_emulator_integration.py`, caused by my local environment: another session's emulator setup has an `ssh` forward on :4443 instead of fake-GCS, and they fail identically on untouched main. CI's emulator job covers them and the new emulator test. - Local CodeRabbit: 3 cycles. I fixed the claim race (two findings), trigger-exception rollback, and the guarded claim release. I skipped two minor nitpicks on the pre-existing trigger-failure rollback. 🤖 Generated with [Claude Code](https://claude.com/claude-code) https://claude.ai/code/session_01Jw21syo4HizJJjYw2mg2Vp --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
364 lines
13 KiB
Python
364 lines
13 KiB
Python
"""
|
|
FastAPI dependencies for authentication and authorization.
|
|
"""
|
|
import asyncio
|
|
import logging
|
|
import secrets
|
|
from datetime import datetime, timedelta, UTC
|
|
from typing import Optional, Tuple, Callable, Union
|
|
from fastapi import Depends, HTTPException, Header, Query, Request, Path
|
|
from fastapi.security import HTTPBearer, HTTPAuthorizationCredentials
|
|
|
|
from backend.services.auth_service import get_auth_service, UserType, AuthService, AuthResult
|
|
|
|
|
|
logger = logging.getLogger(__name__)
|
|
|
|
|
|
def generate_review_token() -> str:
|
|
"""Generate a cryptographically secure review token."""
|
|
return secrets.token_urlsafe(32)
|
|
|
|
|
|
def get_review_token_expiry(hours: int = 24) -> datetime:
|
|
"""Get expiry time for a review token."""
|
|
return datetime.now(UTC) + timedelta(hours=hours)
|
|
|
|
|
|
# HTTP Bearer security scheme (Authorization: Bearer <token>)
|
|
security = HTTPBearer(auto_error=False)
|
|
|
|
|
|
async def get_token_from_request(
|
|
request: Request,
|
|
credentials: Optional[HTTPAuthorizationCredentials] = Depends(security),
|
|
token: Optional[str] = Query(None, description="Access token (alternative to Bearer header)")
|
|
) -> Optional[str]:
|
|
"""
|
|
Extract access token from request.
|
|
|
|
Supports three methods (in priority order):
|
|
1. X-Admin-Token header: Used by Cloud Tasks (since OIDC overwrites Authorization)
|
|
2. Authorization header: Authorization: Bearer <token>
|
|
3. Query parameter: ?token=<token> (for download links)
|
|
|
|
Returns:
|
|
Token string or None
|
|
"""
|
|
# Check X-Admin-Token header first (used by Cloud Tasks)
|
|
# Cloud Tasks OIDC token overwrites Authorization header, so we use a custom header
|
|
admin_token = request.headers.get("X-Admin-Token")
|
|
if admin_token:
|
|
return admin_token
|
|
|
|
# Try Authorization header
|
|
if credentials:
|
|
return credentials.credentials
|
|
|
|
# Try query parameter
|
|
if token:
|
|
return token
|
|
|
|
return None
|
|
|
|
|
|
async def require_auth(
|
|
request: Request,
|
|
auth_service: AuthService = Depends(get_auth_service),
|
|
credentials: Optional[HTTPAuthorizationCredentials] = Depends(security),
|
|
token: Optional[str] = Query(None)
|
|
) -> AuthResult:
|
|
"""
|
|
Require authentication for an endpoint.
|
|
|
|
Returns:
|
|
AuthResult with full authentication context including:
|
|
- is_valid, user_type, remaining_uses, message (backward compatible via tuple unpacking)
|
|
- user_email: Email of authenticated user (if session/API key auth)
|
|
- is_admin: Whether user has admin privileges
|
|
|
|
Raises:
|
|
HTTPException: 401 if authentication fails
|
|
"""
|
|
# Get token from request
|
|
token_str = await get_token_from_request(request, credentials, token)
|
|
|
|
if not token_str:
|
|
raise HTTPException(
|
|
status_code=401,
|
|
detail="Authentication required. Provide token via Authorization header or ?token= parameter"
|
|
)
|
|
|
|
# Validate token using the full method
|
|
# Sync Firestore reads: run off the event loop (every authed request passes here).
|
|
auth_result = await asyncio.to_thread(auth_service.validate_token_full, token_str)
|
|
|
|
# Get request_id from middleware for correlation
|
|
request_id = getattr(request.state, "request_id", None)
|
|
|
|
if not auth_result.is_valid:
|
|
# Log auth failure with request_id for correlation
|
|
auth_header = request.headers.get("Authorization", "")
|
|
logger.warning(
|
|
"auth_failed",
|
|
extra={
|
|
"request_id": request_id,
|
|
"audit_type": "auth_event",
|
|
"auth_message": auth_result.message,
|
|
"token_provided": bool(token_str),
|
|
"token_length": len(token_str) if token_str else 0,
|
|
"auth_header_present": bool(auth_header),
|
|
}
|
|
)
|
|
raise HTTPException(
|
|
status_code=401,
|
|
detail=f"Authentication failed: {auth_result.message}"
|
|
)
|
|
|
|
# If token has a tenant_id, override the hostname-based tenant detection.
|
|
# This allows tenant-scoped tokens to enforce tenant context regardless of domain.
|
|
# We must also load tenant_config so downstream handlers (e.g. job creation)
|
|
# that rely on tenant_config for setting tenant_id on jobs work correctly.
|
|
if auth_result.tenant_id:
|
|
request.state.tenant_id = auth_result.tenant_id
|
|
if not getattr(request.state, "tenant_config", None):
|
|
from backend.services.tenant_service import get_tenant_service
|
|
tenant_service = get_tenant_service()
|
|
tenant_config = await asyncio.to_thread(tenant_service.get_tenant_config, auth_result.tenant_id)
|
|
if tenant_config and tenant_config.is_active:
|
|
request.state.tenant_config = tenant_config
|
|
|
|
# Log successful auth with request_id for correlation with request audit
|
|
logger.info(
|
|
"auth_success",
|
|
extra={
|
|
"request_id": request_id,
|
|
"user_email": auth_result.user_email,
|
|
"user_type": auth_result.user_type.value if auth_result.user_type else None,
|
|
"is_admin": auth_result.is_admin,
|
|
"remaining_uses": auth_result.remaining_uses,
|
|
"tenant_id": auth_result.tenant_id,
|
|
"audit_type": "auth_event",
|
|
}
|
|
)
|
|
|
|
return auth_result
|
|
|
|
|
|
async def require_auth_legacy(
|
|
request: Request,
|
|
auth_service: AuthService = Depends(get_auth_service),
|
|
credentials: Optional[HTTPAuthorizationCredentials] = Depends(security),
|
|
token: Optional[str] = Query(None)
|
|
) -> Tuple[str, UserType, int]:
|
|
"""
|
|
Legacy authentication dependency that returns tuple.
|
|
|
|
DEPRECATED: Use require_auth which returns AuthResult instead.
|
|
|
|
Returns:
|
|
(token, user_type, remaining_uses)
|
|
"""
|
|
token_str = await get_token_from_request(request, credentials, token)
|
|
|
|
if not token_str:
|
|
raise HTTPException(
|
|
status_code=401,
|
|
detail="Authentication required. Provide token via Authorization header or ?token= parameter"
|
|
)
|
|
|
|
is_valid, user_type, remaining_uses, message = await asyncio.to_thread(auth_service.validate_token, token_str)
|
|
|
|
if not is_valid:
|
|
raise HTTPException(
|
|
status_code=401,
|
|
detail=f"Authentication failed: {message}"
|
|
)
|
|
|
|
return token_str, user_type, remaining_uses
|
|
|
|
|
|
async def require_admin(
|
|
auth_result: AuthResult = Depends(require_auth)
|
|
) -> AuthResult:
|
|
"""
|
|
Require admin access for an endpoint.
|
|
|
|
Admin access is granted if:
|
|
- Using an admin token from ADMIN_TOKENS env var
|
|
- User email is from admin domain (e.g., @nomadkaraoke.com)
|
|
- User role is set to ADMIN in database
|
|
|
|
Returns:
|
|
AuthResult with admin privileges
|
|
|
|
Raises:
|
|
HTTPException: 403 if user is not admin
|
|
"""
|
|
if not auth_result.is_admin:
|
|
logger.warning(
|
|
f"Admin access denied for {auth_result.user_type} user "
|
|
f"(email: {auth_result.user_email or 'unknown'})"
|
|
)
|
|
raise HTTPException(
|
|
status_code=403,
|
|
detail="Admin access required"
|
|
)
|
|
|
|
return auth_result
|
|
|
|
|
|
def optional_auth(
|
|
auth_service: AuthService = Depends(get_auth_service),
|
|
credentials: Optional[HTTPAuthorizationCredentials] = Depends(security),
|
|
token: Optional[str] = Query(None)
|
|
) -> Optional[Tuple[str, UserType, int]]:
|
|
"""
|
|
Optional authentication - doesn't fail if no token provided.
|
|
|
|
Useful for endpoints that have different behavior for authenticated users.
|
|
|
|
Returns:
|
|
(token, user_type, remaining_uses) if authenticated, None otherwise
|
|
"""
|
|
# Get token
|
|
token_str = None
|
|
if credentials:
|
|
token_str = credentials.credentials
|
|
elif token:
|
|
token_str = token
|
|
|
|
if not token_str:
|
|
return None
|
|
|
|
# Validate token
|
|
is_valid, user_type, remaining_uses, message = auth_service.validate_token(token_str)
|
|
|
|
if not is_valid:
|
|
logger.debug(f"Optional auth failed: {message}")
|
|
return None
|
|
|
|
return token_str, user_type, remaining_uses
|
|
|
|
|
|
def require_review_auth_factory(job_id_param: str = "job_id"):
|
|
"""
|
|
Factory to create a review authentication dependency.
|
|
|
|
Accepts either:
|
|
1. Full user authentication (admin/user token) - also validates job ownership
|
|
2. Job-specific review token (only valid for the specific job)
|
|
|
|
Args:
|
|
job_id_param: Name of the path parameter containing the job ID
|
|
|
|
Returns:
|
|
Dependency function that validates review access
|
|
"""
|
|
async def require_review_auth(
|
|
request: Request,
|
|
job_id: str = Path(...),
|
|
review_token: Optional[str] = Query(None, description="Job-specific review token"),
|
|
auth_service: AuthService = Depends(get_auth_service),
|
|
credentials: Optional[HTTPAuthorizationCredentials] = Depends(security),
|
|
token: Optional[str] = Query(None, alias="token", description="Full access token")
|
|
) -> Tuple[str, str]:
|
|
"""
|
|
Validate review access for a job.
|
|
|
|
Returns:
|
|
(job_id, auth_type) where auth_type is "full" or "review_token"
|
|
|
|
Raises:
|
|
HTTPException: 401 if authentication fails
|
|
HTTPException: 403 if user doesn't own the job (for full auth)
|
|
"""
|
|
# Import here to avoid circular dependency
|
|
from backend.services.job_manager import JobManager
|
|
|
|
job_manager = JobManager()
|
|
|
|
# Try full authentication first
|
|
full_token = None
|
|
if credentials:
|
|
full_token = credentials.credentials
|
|
elif token:
|
|
full_token = token
|
|
|
|
if full_token:
|
|
auth_result = await asyncio.to_thread(auth_service.validate_token_full, full_token)
|
|
if auth_result.is_valid:
|
|
# For full auth, also verify job ownership
|
|
job = await asyncio.to_thread(job_manager.get_job, job_id)
|
|
if not job:
|
|
raise HTTPException(status_code=404, detail=f"Job {job_id} not found")
|
|
|
|
# Check ownership: admin can access all, users only their own jobs
|
|
if not auth_result.is_admin:
|
|
if auth_result.user_email and job.user_email:
|
|
if auth_result.user_email.lower() != job.user_email.lower():
|
|
logger.warning(
|
|
f"Review access denied: user {auth_result.user_email} "
|
|
f"tried to access job {job_id} owned by {job.user_email}"
|
|
)
|
|
raise HTTPException(
|
|
status_code=403,
|
|
detail="You don't have permission to access this job's review"
|
|
)
|
|
elif job.user_email:
|
|
# Token auth without email trying to access a job with owner
|
|
raise HTTPException(
|
|
status_code=403,
|
|
detail="You don't have permission to access this job's review"
|
|
)
|
|
|
|
logger.info(f"Review access granted via full auth ({auth_result.user_type}) for job {job_id}")
|
|
# Who is reviewing — handlers record it on first open
|
|
# (state_data.review_started_by, see JobManager.record_review_started).
|
|
request.state.review_auth_email = auth_result.user_email
|
|
request.state.review_auth_is_admin = bool(auth_result.is_admin)
|
|
return job_id, "full"
|
|
|
|
# Try review token
|
|
if review_token:
|
|
job = await asyncio.to_thread(job_manager.get_job, job_id)
|
|
|
|
if not job:
|
|
raise HTTPException(status_code=404, detail=f"Job {job_id} not found")
|
|
|
|
# Validate review token matches
|
|
if job.review_token and secrets.compare_digest(job.review_token, review_token):
|
|
# Check expiry if set
|
|
if job.review_token_expires_at:
|
|
now = datetime.now(UTC)
|
|
# Handle timezone-naive datetimes from Firestore
|
|
expiry = job.review_token_expires_at
|
|
if expiry.tzinfo is None:
|
|
expiry = expiry.replace(tzinfo=UTC)
|
|
|
|
if now > expiry:
|
|
logger.warning(f"Review token expired for job {job_id}")
|
|
raise HTTPException(
|
|
status_code=401,
|
|
detail="Review token has expired. Please request a new review link."
|
|
)
|
|
|
|
logger.info(f"Review access granted via review_token for job {job_id}")
|
|
return job_id, "review_token"
|
|
else:
|
|
logger.warning(f"Invalid review token for job {job_id}")
|
|
|
|
# No valid authentication
|
|
raise HTTPException(
|
|
status_code=401,
|
|
detail="Authentication required. Provide either a full access token or a valid review_token for this job."
|
|
)
|
|
|
|
return require_review_auth
|
|
|
|
|
|
# Default instance for most review endpoints
|
|
require_review_auth = require_review_auth_factory()
|
|
|
|
|