Files
karaoke-gen/backend/config.py
T
Andrew Beveridge ☄️andClaude Fable 5 70ea964ca9 feat(dev): full local backend for external contributors (no GCP creds needed) (#946)
## Summary

Motivated by external contributor Martín (netux) asking "is there an
easy way to run the full backend without GCS?" — he couldn't test PR
#895's backend changes locally. This PR makes the whole backend runnable
locally against emulators with **zero GCP credentials and zero paid API
keys**.

### What was broken for contributors
- Workers migrated to Cloud Run Jobs for prod robustness
(audio-download, audio-separation, lyrics-transcription, bulk-search)
silently bypassed the direct-HTTP dev mode → local jobs stalled at
PENDING forever
- Signed URL generation called `google.auth.default()` → crashed with no
credentials (review audio, downloads, theme previews, upload URLs)
- Magic-link login 503'd locally (console email provider counted as "not
configured")
- Fresh emulator had no default theme → every job create returned 422
- `docs/DEVELOPMENT.md` told people to `cp .env.example .env` but the
file didn't exist — and nothing loaded `.env` anyway
- `run-backend-local.sh` defaulted to the production GCP project

### Changes
- **worker_service**: CRJ-based workers run as local subprocesses
(`python -m backend.workers.X --job-id Y`, exact CRJ parity) when
`ENABLE_CLOUD_TASKS=false` **and** not `is_production()`
(regression-tested both ways)
- **storage_service**: signed URLs → plain fake-gcs-server URLs when
`STORAGE_EMULATOR_HOST` set (GET via JSON API download path; PUT via
unvalidated-signature path, enabled by `-public-host` on the emulator)
- **users route**: magic-link 503 only in production; dev prints the
link to the console
- **file_upload bugfix**: a theme with style params but no asset files
never got `style_params_gcs_path` written (empty-dict falsiness in `elif
theme_style_assets:`) — surfaced by the minimal seed theme, latent in
prod
- **scripts/seed-local-data.py** (new): creates bucket + default theme;
wired into `run-backend-local.sh --with-emulators`, which now uses
`test-project`
- **config**: `load_dotenv()` so `.env` actually works; new root
`.env.example`
- **CONTRIBUTING.md** (new): no-credentials quickstart; README +
DEVELOPMENT.md link to it

### Verified end-to-end locally (no GCP credentials)
Upload job → local audio separation (real models) → transcription →
screens → `awaiting_review`; review endpoints all working including
`/api/review/{job}/audio/vocals` (the PR #895 endpoint) and 302 →
emulator audio URLs. Magic-link login round-trip via console link.

### Tests
- 224 tests across affected files pass; full backend unit suite: 3969
passed
- `backend/tests/test_emulator_integration.py` has 8 pre-existing
failures that reproduce identically on unmodified `main` locally (broken
theme-service mock in that file) — unrelated to this PR
- New tests: dev-mode subprocess dispatch (audio + bulk-search),
prod-mode never-Popen guard, emulator URL generation (GET + PUT)

@coderabbitai ignore

---------

Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
2026-08-27 06:48:00 +00:00

365 lines
20 KiB
Python

"""
Configuration management for the karaoke generation backend.
"""
import os
import logging
from typing import Optional, Dict
from dotenv import load_dotenv
from pydantic_settings import BaseSettings
from google.cloud import secretmanager
# Load .env for local development (no-op when absent, e.g. in Cloud Run).
# Must run before the Settings class body below, which reads os.getenv at
# class-definition time. Real environment variables take precedence.
load_dotenv()
logger = logging.getLogger(__name__)
class Settings(BaseSettings):
"""Application settings."""
# Google Cloud
google_cloud_project: str = os.getenv("GOOGLE_CLOUD_PROJECT", "")
gcs_bucket_name: str = os.getenv("GCS_BUCKET_NAME", "karaoke-gen-storage")
gcs_temp_bucket: str = os.getenv("GCS_TEMP_BUCKET", "karaoke-gen-temp")
gcs_output_bucket: str = os.getenv("GCS_OUTPUT_BUCKET", "karaoke-gen-outputs")
firestore_collection: str = os.getenv("FIRESTORE_COLLECTION", "jobs")
# Nomad master fast-sync: push freshly-published Nomad 720p masters straight to the
# Divebar GCS mirror so kjbox (which mirrors that folder every 5 min) picks them up
# within minutes instead of waiting for the nightly Drive->GCS VM. Nomad-brand only.
divebar_files_bucket: str = os.getenv("DIVEBAR_FILES_BUCKET", "nomadkaraoke-divebar-files")
nomad_master_gcs_prefix: str = os.getenv("NOMAD_MASTER_GCS_PREFIX", "files/Nomad Karaoke/MP4-720p")
nomad_master_fast_sync_enabled: bool = os.getenv("NOMAD_MASTER_FAST_SYNC_ENABLED", "true").lower() == "true"
# Original-vocals guide: a padded isolated-vocals file (silence[intro] + vocals) pushed to
# a sibling prefix in the same Divebar bucket, which kjbox mirrors into NOMAD-vocals-padded/
# to power the "Original Vocals" sing-along slider. Same brand gating as the master push.
nomad_vocals_guide_gcs_prefix: str = os.getenv("NOMAD_VOCALS_GUIDE_GCS_PREFIX", "files/Nomad Karaoke/vocals-padded")
# Audio Separator API (for GPU processing)
audio_separator_api_url: Optional[str] = os.getenv("AUDIO_SEPARATOR_API_URL")
# External APIs (can be set via env or Secret Manager)
audioshake_api_key: Optional[str] = os.getenv("AUDIOSHAKE_API_KEY")
genius_api_key: Optional[str] = os.getenv("GENIUS_API_KEY")
spotify_cookie: Optional[str] = os.getenv("SPOTIFY_COOKIE_SP_DC")
rapidapi_key: Optional[str] = os.getenv("RAPIDAPI_KEY")
# Authentication
admin_tokens: Optional[str] = os.getenv("ADMIN_TOKENS") # Comma-separated list
scheduler_service_account: str = os.getenv(
"SCHEDULER_SERVICE_ACCOUNT",
"karaoke-backend@nomadkaraoke.iam.gserviceaccount.com"
)
# Application
environment: str = os.getenv("ENVIRONMENT", "development")
log_level: str = os.getenv("LOG_LEVEL", "INFO")
# Processing
max_concurrent_jobs: int = int(os.getenv("MAX_CONCURRENT_JOBS", "5"))
job_timeout_seconds: int = int(os.getenv("JOB_TIMEOUT_SECONDS", "3600"))
# Lyrics Correction Settings
# Skip all auto-correction (both agentic AI and heuristic handlers).
# When true, raw transcription goes directly to human review without any automatic fixes.
# This is the default because auto-correction currently creates more work for reviewers.
# Set SKIP_CORRECTION=false to re-enable auto-correction if quality improves.
skip_correction: bool = os.getenv("SKIP_CORRECTION", "true").lower() in ("true", "1", "yes")
# Agentic AI Correction (for lyrics correction via LLM)
# Only used when skip_correction=false
# When enabled, uses Gemini via Vertex AI for intelligent lyrics correction
use_agentic_ai: bool = os.getenv("USE_AGENTIC_AI", "true").lower() in ("true", "1", "yes")
agentic_ai_model: str = os.getenv("AGENTIC_AI_MODEL", "vertexai/gemini-3-flash-preview")
# Timeout for agentic correction in seconds. If correction takes longer, abort and
# use uncorrected transcription - human review will fix any issues.
agentic_correction_timeout_seconds: int = int(os.getenv("AGENTIC_CORRECTION_TIMEOUT_SECONDS", "180"))
# Lyrics transcription outer (safety-net) timeout, scaled by audio duration.
# Long inputs (e.g. 30+ minute mashups) legitimately take longer to transcribe,
# so the timeout = clamp(floor, duration * per_audio_second, cap). The cap must
# stay under the Cloud Run Job task timeout so we surface a clean RuntimeError
# before the platform hard-kills the task. The lyrics-transcription-job sets the
# task timeout to 3000s and overrides this cap to 2700s via env; the 1700s
# default here is the safe value if that env is ever unset (under an 1800s task).
transcription_timeout_floor_seconds: int = int(os.getenv("TRANSCRIPTION_TIMEOUT_FLOOR_SECONDS", "1200"))
transcription_timeout_cap_seconds: int = int(os.getenv("TRANSCRIPTION_TIMEOUT_CAP_SECONDS", "1700"))
transcription_timeout_per_audio_second: float = float(os.getenv("TRANSCRIPTION_TIMEOUT_PER_AUDIO_SECOND", "1.0"))
# AI Credit Evaluation (anti-abuse gating for free credits)
# When enabled, welcome and feedback credit grants are evaluated by Gemini
# before granting. Fail-CLOSED: if evaluation fails, the decision becomes
# "pending_review" (no auto-grant; an admin is notified to decide).
credit_eval_enabled: bool = os.getenv("CREDIT_EVAL_ENABLED", "true").lower() in ("true", "1", "yes")
credit_eval_model: str = os.getenv("CREDIT_EVAL_MODEL", "gemini-3.1-pro-preview")
# Custom Lyrics (LLM-powered) — used by the Custom Lyrics mode in the
# Edit All Lyrics modal. Stateless service: takes operator-provided
# custom-lyrics text/file and returns transformed lyric lines.
custom_lyrics_model: str = os.getenv("CUSTOM_LYRICS_MODEL", "gemini-3.1-pro-preview")
custom_lyrics_max_file_mb: int = int(os.getenv("CUSTOM_LYRICS_MAX_FILE_MB", "5"))
custom_lyrics_max_input_lines: int = int(os.getenv("CUSTOM_LYRICS_MAX_INPUT_LINES", "500"))
custom_lyrics_max_iterations: int = int(os.getenv("CUSTOM_LYRICS_MAX_ITERATIONS", "4"))
custom_lyrics_default_strictness: str = os.getenv("CUSTOM_LYRICS_DEFAULT_STRICTNESS", "balanced")
custom_lyrics_max_output_lines_multiplier: float = float(os.getenv("CUSTOM_LYRICS_MAX_OUTPUT_LINES_MULTIPLIER", "2.0"))
# AI auto-correct suggestions — opt-in, user-triggered from the lyrics
# review UI. Stateless: one whole-song LLM call returning word-level
# suggestions the reviewer accepts/rejects individually.
auto_correct_model: str = os.getenv("AUTO_CORRECT_MODEL", "gemini-3.1-pro-preview")
# Models used by the multi-model "compare" mode (semicolon-separated —
# Cloud Run --set-env-vars is comma-delimited). Empty = single-model only.
auto_correct_compare_models: str = os.getenv("AUTO_CORRECT_COMPARE_MODELS", "")
# Proactively generate (and cache) auto-correct suggestions once
# transcription + references are ready, so the review UI has them instantly.
# This flag gates whether the lyrics worker fires the trigger; the actual
# multi-model LLM call + caching runs on the API service (auto_correct_worker),
# which holds the model credentials — the lyrics-transcription-job itself does
# NOT need ANTHROPIC_API_KEY / AUTO_CORRECT_COMPARE_MODELS. Best-effort:
# failures never block or fail the job. Default off.
auto_correct_proactive_enabled: bool = os.getenv(
"AUTO_CORRECT_PROACTIVE_ENABLED", "false"
).lower() in ("true", "1", "yes")
# Match judge — artist/title formatting + match-acceptance for the job
# submission flow. Deterministic + catalog layers always run; the light AI
# layer (Vertex Gemini Flash) fires only when those aren't confident.
match_judge_enabled: bool = os.getenv("MATCH_JUDGE_ENABLED", "true").lower() in (
"true", "1", "yes",
)
match_judge_model: str = os.getenv("MATCH_JUDGE_MODEL", "gemini-3.5-flash")
# Bounds the AI call so a hung model never strands the submission UI. Kept
# generous because Vertex returns 504 DEADLINE_EXCEEDED if generation can't
# finish within this deadline (3s was too tight for gemini-3.5-flash); the
# call still runs off the critical path and degrades to "none" on timeout.
match_judge_timeout_ms: int = int(os.getenv("MATCH_JUDGE_TIMEOUT_MS", "12000"))
# Batch karaoke-filename parser (kjbox download-naming). Reuses the Vertex
# Gemini stack. Larger timeout than match_judge because it batches ~200 items.
parse_titles_enabled: bool = os.getenv("PARSE_TITLES_ENABLED", "true").lower() in (
"1", "true", "yes",
)
parse_titles_model: str = os.getenv("PARSE_TITLES_MODEL", "gemini-3.5-flash")
parse_titles_timeout_ms: int = int(os.getenv("PARSE_TITLES_TIMEOUT_MS", "20000"))
parse_titles_max_items: int = int(os.getenv("PARSE_TITLES_MAX_ITEMS", "200"))
# Items per Gemini call: large client batches are split into concurrent
# chunks of this size so one call never outgrows parse_titles_timeout_ms
# (a 100-item single call did, blanking the whole batch).
parse_titles_chunk_size: int = int(os.getenv("PARSE_TITLES_CHUNK_SIZE", "10"))
# Cloud Tasks (for scalable worker coordination)
# When enabled, workers are triggered via Cloud Tasks for guaranteed delivery
# When disabled (default), workers are triggered via direct HTTP (for development)
enable_cloud_tasks: bool = os.getenv("ENABLE_CLOUD_TASKS", "false").lower() in ("true", "1", "yes")
gcp_region: str = os.getenv("GCP_REGION", "us-central1")
# Cloud Run Jobs (for long-running video encoding)
# When enabled AND enable_cloud_tasks is true, video worker uses Cloud Run Jobs
# instead of Cloud Tasks. This supports encoding times >30 minutes (up to 24 hours).
# Default is false - Cloud Tasks is sufficient for most videos (15-20 min).
use_cloud_run_jobs_for_video: bool = os.getenv("USE_CLOUD_RUN_JOBS_FOR_VIDEO", "false").lower() in ("true", "1", "yes")
# GCE Encoding Worker (for high-performance video encoding)
# When enabled, video encoding is offloaded to a dedicated C4 GCE instance
# with faster CPU (Intel Granite Rapids 3.9 GHz) instead of Cloud Run.
# This provides 2-3x faster encoding times for CPU-bound FFmpeg libx264 encoding.
use_gce_encoding: bool = os.getenv("USE_GCE_ENCODING", "false").lower() in ("true", "1", "yes")
encoding_worker_url: Optional[str] = os.getenv("ENCODING_WORKER_URL") # e.g., http://136.119.50.148:8080
encoding_worker_api_key: Optional[str] = os.getenv("ENCODING_WORKER_API_KEY")
# Optional capacity-fallback VMs in alternate zones. JSON list of objects:
# '[{"vm":"encoding-worker-fallback-a","zone":"us-central1-a","ip":"34.x.x.x"}]'
# Empty by default. After `pulumi up` provisions the fallback VMs (Phase 3
# of the capacity-resilience plan), populate this var with their public
# IPs to enable multi-zone failover when us-central1-c is exhausted.
encoding_worker_fallback_vms: Optional[str] = os.getenv("ENCODING_WORKER_FALLBACK_VMS")
# GCE Preview Encoding (for faster preview video generation)
# When enabled, preview video encoding during lyrics review is offloaded to the GCE worker.
# This reduces preview generation time from 60+ seconds to ~15-20 seconds.
# Requires use_gce_encoding to be enabled and the GCE worker to support /encode-preview endpoint.
use_gce_preview_encoding: bool = os.getenv("USE_GCE_PREVIEW_ENCODING", "false").lower() in ("true", "1", "yes")
# Karaoke-Decide catalog API (for song/artist autocomplete)
karaoke_decide_api_url: str = os.getenv("KARAOKE_DECIDE_API_URL", "https://decide.nomadkaraoke.com")
# Storage paths
temp_dir: str = os.getenv("TEMP_DIR", "/tmp/karaoke-gen")
# Worker logs storage mode
# When enabled, worker logs are stored in a Firestore subcollection (jobs/{job_id}/logs)
# instead of an embedded array. This avoids the 1MB document size limit.
# Default is true for new deployments.
use_log_subcollection: bool = os.getenv("USE_LOG_SUBCOLLECTION", "true").lower() in ("true", "1", "yes")
# Flacfetch remote service (for torrent downloads)
# When configured, audio search uses the remote flacfetch HTTP API instead of local flacfetch.
# This is required for torrent downloads since Cloud Run doesn't support BitTorrent.
flacfetch_api_url: Optional[str] = os.getenv("FLACFETCH_API_URL") # e.g., http://10.0.0.5:8080
flacfetch_api_key: Optional[str] = os.getenv("FLACFETCH_API_KEY")
# Flacfetch client retry settings
# Retries help handle transient network failures when calling the flacfetch service
flacfetch_retry_max_attempts: int = int(os.getenv("FLACFETCH_RETRY_MAX_ATTEMPTS", "9")) # Total attempts (1 initial + 8 retries)
flacfetch_retry_min_wait: float = float(os.getenv("FLACFETCH_RETRY_MIN_WAIT", "10.0")) # seconds
flacfetch_retry_max_wait: float = float(os.getenv("FLACFETCH_RETRY_MAX_WAIT", "60.0")) # seconds
# Default distribution settings (can be overridden per-request)
default_dropbox_path: Optional[str] = os.getenv("DEFAULT_DROPBOX_PATH")
default_gdrive_folder_id: Optional[str] = os.getenv("DEFAULT_GDRIVE_FOLDER_ID")
# Strip whitespace/newlines from webhook URL - common issue when env vars are set with trailing newlines
default_discord_webhook_url: Optional[str] = (
os.getenv("DEFAULT_DISCORD_WEBHOOK_URL", "").strip() or None
)
# Default values for web service jobs (YouTube/Dropbox distribution)
default_enable_youtube_upload: bool = os.getenv("DEFAULT_ENABLE_YOUTUBE_UPLOAD", "false").lower() in ("true", "1", "yes")
default_brand_prefix: Optional[str] = os.getenv("DEFAULT_BRAND_PREFIX")
# Private (non-published) track distribution settings
default_private_dropbox_path: Optional[str] = os.getenv("DEFAULT_PRIVATE_DROPBOX_PATH")
default_private_brand_prefix: Optional[str] = os.getenv("DEFAULT_PRIVATE_BRAND_PREFIX", "NOMADNP")
# YouTube Data API v3 Quota Configuration
# Daily quota limit (YouTube grants 10,000 units/day by default)
youtube_quota_daily_limit: int = int(os.getenv("YOUTUBE_QUOTA_DAILY_LIMIT", "10000"))
# Estimated units consumed per upload (videos.insert=100 + search.list=100 + thumbnails.set=50 + buffer)
youtube_quota_upload_cost: int = int(os.getenv("YOUTUBE_QUOTA_UPLOAD_COST", "300"))
# Safety margin to reserve (avoid hitting exact limit)
youtube_quota_safety_margin: int = int(os.getenv("YOUTUBE_QUOTA_SAFETY_MARGIN", "500"))
# E2E test bypass key for rate limiting (set via secret in production)
e2e_bypass_key: str = os.getenv("E2E_BYPASS_KEY", "")
default_youtube_description: str = os.getenv(
"DEFAULT_YOUTUBE_DESCRIPTION",
"Karaoke video created with Nomad Karaoke (https://nomadkaraoke.com)\n\n"
"AI-powered vocal separation and synchronized lyrics.\n\n"
"#karaoke #music #singing #instrumental #lyrics"
)
# Default CDG/TXT generation settings
# When True, CDG and TXT packages are generated by default (when a theme is set)
# These can be overridden per-request via explicit enable_cdg/enable_txt parameters
default_enable_cdg: bool = os.getenv("DEFAULT_ENABLE_CDG", "true").lower() in ("true", "1", "yes")
default_enable_txt: bool = os.getenv("DEFAULT_ENABLE_TXT", "true").lower() in ("true", "1", "yes")
# Push Notifications Configuration
# When enabled, users can subscribe to push notifications for job status updates
enable_push_notifications: bool = os.getenv("ENABLE_PUSH_NOTIFICATIONS", "false").lower() in ("true", "1", "yes")
# Maximum number of push subscriptions per user (oldest removed when exceeded)
max_push_subscriptions_per_user: int = int(os.getenv("MAX_PUSH_SUBSCRIPTIONS_PER_USER", "5"))
# VAPID subject (email or URL for push service to contact)
vapid_subject: str = os.getenv("VAPID_SUBJECT", "mailto:gen@nomadkaraoke.com")
# Secret Manager cache
_secret_cache: Dict[str, str] = {}
class Config:
env_file = ".env"
case_sensitive = False
def get_secret(self, secret_id: str) -> Optional[str]:
"""
Get a secret from Google Secret Manager.
Caches secrets in memory to avoid repeated API calls.
Falls back to environment variables if Secret Manager unavailable.
Args:
secret_id: Secret name (e.g., "audioshake-api-key")
Returns:
Secret value or None if not found
"""
# Check cache first
if secret_id in self._secret_cache:
return self._secret_cache[secret_id]
# Check environment variable (development mode)
env_var = secret_id.upper().replace('-', '_')
env_value = os.getenv(env_var)
if env_value:
logger.debug(f"Using {secret_id} from environment variable")
self._secret_cache[secret_id] = env_value
return env_value
# Try Secret Manager (production mode)
if not self.google_cloud_project:
logger.warning(f"Cannot fetch secret {secret_id}: No GCP project configured")
return None
try:
client = secretmanager.SecretManagerServiceClient()
name = f"projects/{self.google_cloud_project}/secrets/{secret_id}/versions/latest"
response = client.access_secret_version(request={"name": name})
# Strip whitespace/newlines - common issue when secrets are created with trailing newlines
secret_value = response.payload.data.decode('UTF-8').strip()
# Cache it
self._secret_cache[secret_id] = secret_value
logger.info(f"Loaded secret {secret_id} from Secret Manager")
return secret_value
except Exception as e:
logger.error(f"Failed to load secret {secret_id} from Secret Manager: {e}")
return None
# Global settings instance
settings = Settings()
def get_settings() -> Settings:
"""Get the global settings instance."""
return settings
def is_production() -> bool:
"""True when running in a deployed/production environment.
Cloud Run automatically sets ``K_SERVICE``, so we treat its presence as
production even if ``ENVIRONMENT`` were somehow unset — a production-safe
default. Local dev/test have no ``K_SERVICE`` and a non-"production"
``ENVIRONMENT``. (Fallback audit 2026-06-09, Theme 7.)
"""
env = (os.environ.get("ENVIRONMENT") or os.environ.get("ENV") or "").lower()
if env == "production":
return True
return bool(os.environ.get("K_SERVICE"))
# Env vars that MUST be explicitly set in production. A missing one used to
# silently fall back to a dev default and produce wrong behaviour with only a
# warning (wrong GCP project for secret lookups, wrong GCS bucket, localhost
# worker URL). We only require vars that infra confirms are set on every service
# (GCS temp/output buckets are intentionally excluded — their defaults are
# load-bearing and not set via env). (Fallback audit 2026-06-09, Theme 7.)
_REQUIRED_PRODUCTION_ENV_VARS = (
"GOOGLE_CLOUD_PROJECT",
"GCS_BUCKET_NAME",
"CLOUD_RUN_SERVICE_URL",
)
def validate_production_config() -> None:
"""Fail fast at startup if required production env vars are missing.
In production each var in ``_REQUIRED_PRODUCTION_ENV_VARS`` must be explicitly
set; otherwise the service would silently run with a dev default (wrong GCP
project / wrong GCS bucket / localhost worker URL) and appear to work while
doing the wrong thing. Raises RuntimeError listing every missing var. No-op
outside production. (Fallback audit 2026-06-09, Theme 7.)
"""
if not is_production():
return
missing = [v for v in _REQUIRED_PRODUCTION_ENV_VARS if not (os.environ.get(v) or "").strip()]
if missing:
raise RuntimeError(
"Production misconfiguration: required environment variable(s) not set: "
+ ", ".join(missing)
+ ". Refusing to start with silent dev defaults (wrong GCP project / "
"GCS bucket / localhost worker URL)."
)