mirror of
https://github.com/nomadkaraoke/karaoke-gen.git
synced 2026-11-20 13:07:57 +09:00
## Summary Motivated by external contributor Martín (netux) asking "is there an easy way to run the full backend without GCS?" — he couldn't test PR #895's backend changes locally. This PR makes the whole backend runnable locally against emulators with **zero GCP credentials and zero paid API keys**. ### What was broken for contributors - Workers migrated to Cloud Run Jobs for prod robustness (audio-download, audio-separation, lyrics-transcription, bulk-search) silently bypassed the direct-HTTP dev mode → local jobs stalled at PENDING forever - Signed URL generation called `google.auth.default()` → crashed with no credentials (review audio, downloads, theme previews, upload URLs) - Magic-link login 503'd locally (console email provider counted as "not configured") - Fresh emulator had no default theme → every job create returned 422 - `docs/DEVELOPMENT.md` told people to `cp .env.example .env` but the file didn't exist — and nothing loaded `.env` anyway - `run-backend-local.sh` defaulted to the production GCP project ### Changes - **worker_service**: CRJ-based workers run as local subprocesses (`python -m backend.workers.X --job-id Y`, exact CRJ parity) when `ENABLE_CLOUD_TASKS=false` **and** not `is_production()` (regression-tested both ways) - **storage_service**: signed URLs → plain fake-gcs-server URLs when `STORAGE_EMULATOR_HOST` set (GET via JSON API download path; PUT via unvalidated-signature path, enabled by `-public-host` on the emulator) - **users route**: magic-link 503 only in production; dev prints the link to the console - **file_upload bugfix**: a theme with style params but no asset files never got `style_params_gcs_path` written (empty-dict falsiness in `elif theme_style_assets:`) — surfaced by the minimal seed theme, latent in prod - **scripts/seed-local-data.py** (new): creates bucket + default theme; wired into `run-backend-local.sh --with-emulators`, which now uses `test-project` - **config**: `load_dotenv()` so `.env` actually works; new root `.env.example` - **CONTRIBUTING.md** (new): no-credentials quickstart; README + DEVELOPMENT.md link to it ### Verified end-to-end locally (no GCP credentials) Upload job → local audio separation (real models) → transcription → screens → `awaiting_review`; review endpoints all working including `/api/review/{job}/audio/vocals` (the PR #895 endpoint) and 302 → emulator audio URLs. Magic-link login round-trip via console link. ### Tests - 224 tests across affected files pass; full backend unit suite: 3969 passed - `backend/tests/test_emulator_integration.py` has 8 pre-existing failures that reproduce identically on unmodified `main` locally (broken theme-service mock in that file) — unrelated to this PR - New tests: dev-mode subprocess dispatch (audio + bulk-search), prod-mode never-Popen guard, emulator URL generation (GET + PUT) @coderabbitai ignore --------- Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
82 lines
2.6 KiB
Python
82 lines
2.6 KiB
Python
#!/usr/bin/env python3
|
|
"""
|
|
Seed the local GCS emulator (fake-gcs-server) with the minimum data the
|
|
backend needs to create jobs: the bucket itself plus a default theme.
|
|
|
|
Run after starting the emulators (scripts/start-emulators.sh); it is invoked
|
|
automatically by scripts/run-backend-local.sh --with-emulators. Idempotent —
|
|
safe to run repeatedly.
|
|
|
|
Usage:
|
|
STORAGE_EMULATOR_HOST=http://127.0.0.1:4443 python scripts/seed-local-data.py
|
|
|
|
Environment:
|
|
STORAGE_EMULATOR_HOST Required — refuses to run against real GCS.
|
|
GCS_BUCKET_NAME Bucket to create/seed (default: test-bucket)
|
|
GOOGLE_CLOUD_PROJECT Project for the emulator client (default: test-project)
|
|
"""
|
|
|
|
import json
|
|
import os
|
|
import sys
|
|
from pathlib import Path
|
|
|
|
sys.path.insert(0, str(Path(__file__).parent.parent))
|
|
|
|
DEFAULT_THEME_ID = "local-default"
|
|
|
|
|
|
def main() -> int:
|
|
emulator_host = os.getenv("STORAGE_EMULATOR_HOST")
|
|
if not emulator_host:
|
|
print("ERROR: STORAGE_EMULATOR_HOST is not set. This script only seeds the")
|
|
print("local GCS emulator — it refuses to touch real GCS buckets.")
|
|
return 1
|
|
|
|
project = os.getenv("GOOGLE_CLOUD_PROJECT", "test-project")
|
|
bucket_name = os.getenv("GCS_BUCKET_NAME", "test-bucket")
|
|
|
|
from google.cloud import storage
|
|
|
|
client = storage.Client(project=project)
|
|
|
|
bucket = client.bucket(bucket_name)
|
|
if not bucket.exists():
|
|
client.create_bucket(bucket_name)
|
|
print(f"Created bucket {bucket_name}")
|
|
else:
|
|
print(f"Bucket {bucket_name} already exists")
|
|
|
|
# Theme registry with one default theme (jobs can't be created without one)
|
|
registry = {
|
|
"version": 1,
|
|
"themes": [
|
|
{
|
|
"id": DEFAULT_THEME_ID,
|
|
"name": "Local Default",
|
|
"description": "Default theme seeded for local development",
|
|
"is_default": True,
|
|
"created_at": "2026-01-01T00:00:00Z",
|
|
}
|
|
],
|
|
}
|
|
bucket.blob("themes/_metadata.json").upload_from_string(
|
|
json.dumps(registry, indent=2), content_type="application/json"
|
|
)
|
|
print("Seeded themes/_metadata.json")
|
|
|
|
from karaoke_gen.style_loader import get_default_style_params
|
|
|
|
style_params = get_default_style_params()
|
|
bucket.blob(f"themes/{DEFAULT_THEME_ID}/style_params.json").upload_from_string(
|
|
json.dumps(style_params, indent=2), content_type="application/json"
|
|
)
|
|
print(f"Seeded themes/{DEFAULT_THEME_ID}/style_params.json")
|
|
|
|
print("\nLocal seed data ready — the backend can now create jobs.")
|
|
return 0
|
|
|
|
|
|
if __name__ == "__main__":
|
|
sys.exit(main())
|