mirror of
https://github.com/nomadkaraoke/karaoke-gen.git
synced 2026-11-20 13:07:57 +09:00
feat(dev): full local backend for external contributors (no GCP creds needed) (#946)
## Summary Motivated by external contributor Martín (netux) asking "is there an easy way to run the full backend without GCS?" — he couldn't test PR #895's backend changes locally. This PR makes the whole backend runnable locally against emulators with **zero GCP credentials and zero paid API keys**. ### What was broken for contributors - Workers migrated to Cloud Run Jobs for prod robustness (audio-download, audio-separation, lyrics-transcription, bulk-search) silently bypassed the direct-HTTP dev mode → local jobs stalled at PENDING forever - Signed URL generation called `google.auth.default()` → crashed with no credentials (review audio, downloads, theme previews, upload URLs) - Magic-link login 503'd locally (console email provider counted as "not configured") - Fresh emulator had no default theme → every job create returned 422 - `docs/DEVELOPMENT.md` told people to `cp .env.example .env` but the file didn't exist — and nothing loaded `.env` anyway - `run-backend-local.sh` defaulted to the production GCP project ### Changes - **worker_service**: CRJ-based workers run as local subprocesses (`python -m backend.workers.X --job-id Y`, exact CRJ parity) when `ENABLE_CLOUD_TASKS=false` **and** not `is_production()` (regression-tested both ways) - **storage_service**: signed URLs → plain fake-gcs-server URLs when `STORAGE_EMULATOR_HOST` set (GET via JSON API download path; PUT via unvalidated-signature path, enabled by `-public-host` on the emulator) - **users route**: magic-link 503 only in production; dev prints the link to the console - **file_upload bugfix**: a theme with style params but no asset files never got `style_params_gcs_path` written (empty-dict falsiness in `elif theme_style_assets:`) — surfaced by the minimal seed theme, latent in prod - **scripts/seed-local-data.py** (new): creates bucket + default theme; wired into `run-backend-local.sh --with-emulators`, which now uses `test-project` - **config**: `load_dotenv()` so `.env` actually works; new root `.env.example` - **CONTRIBUTING.md** (new): no-credentials quickstart; README + DEVELOPMENT.md link to it ### Verified end-to-end locally (no GCP credentials) Upload job → local audio separation (real models) → transcription → screens → `awaiting_review`; review endpoints all working including `/api/review/{job}/audio/vocals` (the PR #895 endpoint) and 302 → emulator audio URLs. Magic-link login round-trip via console link. ### Tests - 224 tests across affected files pass; full backend unit suite: 3969 passed - `backend/tests/test_emulator_integration.py` has 8 pre-existing failures that reproduce identically on unmodified `main` locally (broken theme-service mock in that file) — unrelated to this PR - New tests: dev-mode subprocess dispatch (audio + bulk-search), prod-mode never-Popen guard, emulator URL generation (GET + PUT) @coderabbitai ignore --------- Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
This commit is contained in:
1 parent
79b7c80264
commit
70ea964ca9
16 files changed
+466
-27
No files matched your search
@@ -0,0 +1,42 @@
|
||||
# karaoke-gen backend environment — copy to .env and adjust.
|
||||
#
|
||||
# NOTE: scripts/run-backend-local.sh --with-emulators sets all the required
|
||||
# variables itself; you only need a .env for optional API keys or overrides.
|
||||
# None of the keys below are required to boot the backend locally.
|
||||
|
||||
# --- Core (defaults shown are what run-backend-local.sh uses in emulator mode) ---
|
||||
# ENVIRONMENT=development
|
||||
# GOOGLE_CLOUD_PROJECT=test-project
|
||||
# GCS_BUCKET_NAME=test-bucket
|
||||
# FIRESTORE_COLLECTION=test-jobs
|
||||
# ADMIN_TOKENS=local-dev-token
|
||||
# PORT=8000
|
||||
# FRONTEND_URL=http://localhost:3000
|
||||
|
||||
# --- Emulators (set automatically by run-backend-local.sh --with-emulators) ---
|
||||
# FIRESTORE_EMULATOR_HOST=127.0.0.1:8080
|
||||
# STORAGE_EMULATOR_HOST=http://127.0.0.1:4443
|
||||
|
||||
# --- Audio separation (pick one; without either, jobs stall at separation) ---
|
||||
# Local CPU/GPU separation via the audio-separator package. Models are
|
||||
# downloaded into this directory on first use (~2GB). CPU works but is slow.
|
||||
# MODEL_DIR=./models
|
||||
# ...or point at a remote audio-separator API service:
|
||||
# AUDIO_SEPARATOR_API_URL=
|
||||
|
||||
# --- Lyrics transcription ---
|
||||
# The lyrics worker reads AUDIOSHAKE_API_TOKEN (not AUDIOSHAKE_API_KEY).
|
||||
# Without it, transcription falls back to local Whisper if installed
|
||||
# (poetry install --extras local-whisper).
|
||||
# AUDIOSHAKE_API_TOKEN=
|
||||
|
||||
# --- Lyrics sources (optional; improves reference lyrics) ---
|
||||
# GENIUS_API_KEY=
|
||||
# SPOTIFY_COOKIE_SP_DC=
|
||||
|
||||
# --- Email (optional; without it, emails are logged to the console) ---
|
||||
# POSTMARK_SERVER_TOKEN=
|
||||
|
||||
# --- Payments (optional; only needed to exercise payment endpoints) ---
|
||||
# STRIPE_SECRET_KEY=
|
||||
# STRIPE_WEBHOOK_SECRET=
|
||||
@@ -57,4 +57,7 @@ regexes = [
|
||||
# Firebase / GCP client API keys — public by design (identify the project,
|
||||
# access is gated by Firebase Security Rules).
|
||||
'''AIzaSy[A-Za-z0-9_-]{33}''',
|
||||
|
||||
# Documented local-development placeholder token (CONTRIBUTING.md, dev scripts)
|
||||
'''local-dev-token''',
|
||||
]
|
||||
+109
@@ -0,0 +1,109 @@
|
||||
# Contributing to karaoke-gen
|
||||
|
||||
Thanks for your interest in contributing! This guide focuses on getting a
|
||||
working **local development environment with no Google Cloud credentials or
|
||||
paid API keys** — everything runs against local emulators.
|
||||
|
||||
## What runs locally (no credentials needed)
|
||||
|
||||
| Component | Local story |
|
||||
|-----------|-------------|
|
||||
| FastAPI backend | Runs directly via uvicorn |
|
||||
| Firestore | Google's Firestore emulator |
|
||||
| Cloud Storage | [fake-gcs-server](https://github.com/fsouza/fake-gcs-server) (Docker) |
|
||||
| Background workers (audio, lyrics, screens, video…) | Run as local subprocesses / in-process tasks |
|
||||
| Audio separation | Local CPU/GPU via `audio-separator` (set `MODEL_DIR`; models download on first use) |
|
||||
| Lyrics transcription | Local Whisper (`poetry install --extras local-whisper`); AudioShake used only if you set a token |
|
||||
| Login (magic links) | Printed to the backend console instead of emailed |
|
||||
| Signed URLs (review audio, downloads, uploads) | Rewritten to plain emulator URLs |
|
||||
| Frontend (Next.js) | `npm run dev`, pointed at the local backend |
|
||||
|
||||
Things that do NOT run locally: payments (Stripe), production email
|
||||
(Postmark), YouTube/Drive/Dropbox distribution, and the `flacfetch`
|
||||
URL-download service — **jobs from URLs won't process locally; use file
|
||||
upload instead.**
|
||||
|
||||
## Prerequisites
|
||||
|
||||
- Python 3.12/3.13 + [Poetry](https://python-poetry.org/)
|
||||
- Node.js 20+ (for the frontend)
|
||||
- Docker (for the GCS emulator)
|
||||
- Java 21+ and the Google Cloud SDK with the Firestore emulator:
|
||||
`gcloud components install cloud-firestore-emulator`
|
||||
(no login/account needed — the emulator runs entirely offline)
|
||||
- ffmpeg (`brew install ffmpeg` / `apt install ffmpeg`)
|
||||
|
||||
## Backend quickstart
|
||||
|
||||
```bash
|
||||
poetry install # heavy: torch etc, ~10 min first time
|
||||
./scripts/run-backend-local.sh --with-emulators
|
||||
```
|
||||
|
||||
This starts the Firestore emulator (port 8080), fake-gcs-server (port 4443),
|
||||
creates the bucket, seeds a default theme (`scripts/seed-local-data.py`), and
|
||||
runs the backend at <http://localhost:8000> (API docs at `/docs`).
|
||||
|
||||
Optional API keys go in `.env` (see `.env.example`) — none are required to boot.
|
||||
|
||||
### Try it
|
||||
|
||||
```bash
|
||||
# Health
|
||||
curl http://localhost:8000/api/health
|
||||
|
||||
# Create a job from a local audio file (admin token is "local-dev-token")
|
||||
curl -X POST http://localhost:8000/api/jobs/upload \
|
||||
-H 'Authorization: Bearer local-dev-token' \
|
||||
-F "file=@/path/to/song.flac" -F "artist=Some Artist" -F "title=Some Song"
|
||||
|
||||
# Watch it process; when status hits awaiting_review, open the review UI
|
||||
curl http://localhost:8000/api/jobs/<job_id> -H 'Authorization: Bearer local-dev-token'
|
||||
```
|
||||
|
||||
For audio separation to run, set `MODEL_DIR` (e.g. `MODEL_DIR=./models`) —
|
||||
separation models (~2 GB) are downloaded on first use. Without it the job
|
||||
fails at the separation step with a clear error.
|
||||
|
||||
## Frontend quickstart
|
||||
|
||||
```bash
|
||||
cd frontend
|
||||
npm install
|
||||
NEXT_PUBLIC_API_URL=http://localhost:8000 npx next dev
|
||||
```
|
||||
|
||||
Then open <http://localhost:3000>. To log in, enter any email address — the
|
||||
magic link is printed in the **backend** console (look for `EMAIL TO:`); open
|
||||
that link in your browser.
|
||||
|
||||
### Review UI without running the full pipeline
|
||||
|
||||
If you're working on the lyrics review UI specifically, there's a fixtures
|
||||
harness that serves real correction data through the review API shape without
|
||||
processing any audio:
|
||||
|
||||
```bash
|
||||
python scripts/review_test_fixtures.py # serves on :8765
|
||||
```
|
||||
|
||||
## Tests
|
||||
|
||||
```bash
|
||||
make test # everything (backend + frontend)
|
||||
make test-backend # backend only (unit + emulator tests)
|
||||
```
|
||||
|
||||
Emulator-backed integration tests live in `backend/tests/emulator/` and use
|
||||
the same two emulators as the local dev setup.
|
||||
|
||||
## Pull requests
|
||||
|
||||
- Keep PRs focused; include tests for behavior changes.
|
||||
- Run `make test` before pushing — CI runs the same suite.
|
||||
- Bump `tool.poetry.version` in `pyproject.toml` for code changes.
|
||||
- CI must pass before merge; a maintainer will review from there.
|
||||
|
||||
See `docs/DEVELOPMENT.md` for deeper docs (architecture, deployment,
|
||||
observability) — most of that file concerns the production GCP deployment,
|
||||
which contributors don't need.
|
||||
@@ -675,4 +675,6 @@ MIT
|
||||
|
||||
## 🤝 Contributing
|
||||
|
||||
Contributions are welcome! Please see our contributing guidelines.
|
||||
Contributions are welcome! See [CONTRIBUTING.md](CONTRIBUTING.md) — the whole
|
||||
backend runs locally against emulators with no Google Cloud credentials or
|
||||
paid API keys required.
|
||||
@@ -814,8 +814,10 @@ async def upload_and_create_job(
|
||||
update_data['style_assets'] = style_assets
|
||||
if 'style_params' in style_assets:
|
||||
update_data['style_params_gcs_path'] = style_assets['style_params']
|
||||
elif theme_style_assets:
|
||||
# Theme style assets (no custom uploads)
|
||||
elif theme_style_assets or theme_style_params_path:
|
||||
# Theme style assets (no custom uploads). A theme may have style
|
||||
# params but no asset files (e.g. the minimal local-dev seed theme),
|
||||
# so the params path must be written even when the assets dict is empty.
|
||||
update_data['style_assets'] = theme_style_assets
|
||||
if theme_style_params_path:
|
||||
update_data['style_params_gcs_path'] = theme_style_params_path
|
||||
|
||||
@@ -36,6 +36,7 @@ def _mask_email(email: str) -> str:
|
||||
from fastapi import APIRouter, HTTPException, Depends, Request, Header
|
||||
from pydantic import BaseModel, EmailStr
|
||||
|
||||
from backend.config import is_production
|
||||
from backend.i18n import t, get_locale_from_request, get_full_locale_from_request
|
||||
from backend.services.email_validation_service import get_email_validation_service
|
||||
from backend.models.user import (
|
||||
@@ -274,8 +275,10 @@ async def send_magic_link(
|
||||
message=t(locale, "users.magicLinkSent")
|
||||
)
|
||||
|
||||
# Check if email service is configured
|
||||
if not email_service.is_configured():
|
||||
# Check if email service is configured. In development the console provider
|
||||
# logs the magic link to the backend console instead of emailing it, so
|
||||
# contributors can log in locally without a Postmark token.
|
||||
if not email_service.is_configured() and is_production():
|
||||
logger.error("Email service not configured - cannot send magic links")
|
||||
raise HTTPException(
|
||||
status_code=503,
|
||||
|
||||
@@ -4,9 +4,14 @@ Configuration management for the karaoke generation backend.
|
||||
import os
|
||||
import logging
|
||||
from typing import Optional, Dict
|
||||
from dotenv import load_dotenv
|
||||
from pydantic_settings import BaseSettings
|
||||
from google.cloud import secretmanager
|
||||
|
||||
# Load .env for local development (no-op when absent, e.g. in Cloud Run).
|
||||
# Must run before the Settings class body below, which reads os.getenv at
|
||||
# class-definition time. Real environment variables take precedence.
|
||||
load_dotenv()
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
|
||||
@@ -155,7 +155,28 @@ class StorageService:
|
||||
"""Internal method to generate signed URLs for GET or PUT operations."""
|
||||
import google.auth
|
||||
from google.auth.transport import requests
|
||||
|
||||
|
||||
# Local development against fake-gcs-server: real URL signing requires
|
||||
# service-account credentials, which contributors don't have. The emulator
|
||||
# serves objects unauthenticated, so return a plain emulator URL instead.
|
||||
emulator_host = os.getenv("STORAGE_EMULATOR_HOST")
|
||||
if emulator_host:
|
||||
from urllib.parse import quote
|
||||
|
||||
base = emulator_host.rstrip("/")
|
||||
if method == "GET":
|
||||
url = f"{base}/download/storage/v1/b/{self.bucket.name}/o/{quote(blob_path, safe='')}?alt=media"
|
||||
else:
|
||||
# fake-gcs-server treats a PUT with X-Goog signature params as a
|
||||
# signed-URL upload without validating the signature. Requires the
|
||||
# server to run with -public-host matching this host.
|
||||
url = (
|
||||
f"{base}/{self.bucket.name}/{quote(blob_path)}"
|
||||
"?X-Goog-Algorithm=GOOG4-RSA-SHA256&X-Goog-Signature=local-dev"
|
||||
)
|
||||
logger.info(f"Generated emulator {method} URL for {blob_path} (STORAGE_EMULATOR_HOST set)")
|
||||
return url
|
||||
|
||||
try:
|
||||
blob = self.bucket.blob(blob_path)
|
||||
|
||||
|
||||
@@ -24,7 +24,7 @@ import httpx
|
||||
from google.api_core import exceptions as gcp_exceptions
|
||||
from google.protobuf import duration_pb2
|
||||
|
||||
from backend.config import get_settings
|
||||
from backend.config import get_settings, is_production
|
||||
from backend.services.tracing import inject_trace_context
|
||||
|
||||
|
||||
@@ -404,6 +404,10 @@ class WorkerService:
|
||||
in AWAITING_AUDIO_SELECTION. Uses a Cloud Run Job (not BackgroundTasks) so
|
||||
the batch survives API instance scale-down and the user can close the tab.
|
||||
"""
|
||||
if not self._use_cloud_tasks and not is_production():
|
||||
return self._run_worker_module_locally(
|
||||
"bulk_search_worker", ["--batch-id", batch_id], log_prefix=f"[batch:{batch_id}]"
|
||||
)
|
||||
try:
|
||||
from google.cloud import run_v2
|
||||
|
||||
@@ -584,6 +588,29 @@ class WorkerService:
|
||||
# Exhausted transient retries — re-raise so the caller logs + returns False.
|
||||
raise last_exc
|
||||
|
||||
def _run_worker_module_locally(
|
||||
self, worker_module: str, args: list[str], log_prefix: str = ""
|
||||
) -> bool:
|
||||
"""
|
||||
Run a worker module as a detached local subprocess (development mode).
|
||||
|
||||
Mirrors what the corresponding Cloud Run Job would execute
|
||||
(``python -m backend.workers.<module> <args>``), so contributors can run
|
||||
the full pipeline locally against the emulators without GCP credentials.
|
||||
Worker output is streamed to the backend console.
|
||||
"""
|
||||
import subprocess
|
||||
import sys
|
||||
|
||||
cmd = [sys.executable, "-m", f"backend.workers.{worker_module}", *args]
|
||||
try:
|
||||
subprocess.Popen(cmd, env=os.environ.copy())
|
||||
logger.info(f"{log_prefix} Started local worker subprocess: {' '.join(cmd)}")
|
||||
return True
|
||||
except Exception as e:
|
||||
logger.error(f"{log_prefix} Failed to start local worker {worker_module}: {e}", exc_info=True)
|
||||
return False
|
||||
|
||||
async def _trigger_worker_cloud_run_job(
|
||||
self,
|
||||
job_id: str,
|
||||
@@ -607,6 +634,15 @@ class WorkerService:
|
||||
Returns:
|
||||
True if job was triggered successfully, False otherwise
|
||||
"""
|
||||
# Local development (ENABLE_CLOUD_TASKS=false): run the worker module as a
|
||||
# local subprocess with the exact args the Cloud Run Job would receive.
|
||||
# The is_production() guard ensures a deployed environment that omits
|
||||
# ENABLE_CLOUD_TASKS still dispatches to Cloud Run Jobs, never Popen.
|
||||
if not self._use_cloud_tasks and not is_production():
|
||||
return self._run_worker_module_locally(
|
||||
worker_module, ["--job-id", job_id], log_prefix=f"[job:{job_id}]"
|
||||
)
|
||||
|
||||
try:
|
||||
from google.cloud import run_v2
|
||||
|
||||
|
||||
@@ -168,7 +168,7 @@ class TestWorkerService:
|
||||
with patch('backend.services.worker_service.get_settings') as mock_settings:
|
||||
mock_settings.return_value.admin_tokens = "test-token"
|
||||
mock_settings.return_value.google_cloud_project = "test-project"
|
||||
mock_settings.return_value.enable_cloud_tasks = False
|
||||
mock_settings.return_value.enable_cloud_tasks = True
|
||||
mock_settings.return_value.gcp_region = "us-central1"
|
||||
mock_settings.return_value.use_cloud_run_jobs_for_video = False
|
||||
|
||||
@@ -211,7 +211,7 @@ class TestWorkerService:
|
||||
with patch('backend.services.worker_service.get_settings') as mock_settings:
|
||||
mock_settings.return_value.admin_tokens = "test-token"
|
||||
mock_settings.return_value.google_cloud_project = "test-project"
|
||||
mock_settings.return_value.enable_cloud_tasks = False
|
||||
mock_settings.return_value.enable_cloud_tasks = True
|
||||
mock_settings.return_value.gcp_region = "us-central1"
|
||||
mock_settings.return_value.use_cloud_run_jobs_for_video = False
|
||||
|
||||
@@ -245,6 +245,78 @@ class TestWorkerService:
|
||||
assert call_args is not None
|
||||
assert "lyrics-transcription-job" in str(call_args)
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_trigger_audio_worker_local_subprocess_in_dev_mode(self):
|
||||
"""With ENABLE_CLOUD_TASKS=false, CRJ-based workers run as local subprocesses."""
|
||||
from backend.services.worker_service import WorkerService, reset_worker_service
|
||||
reset_worker_service()
|
||||
|
||||
with patch('backend.services.worker_service.get_settings') as mock_settings:
|
||||
mock_settings.return_value.admin_tokens = "test-token"
|
||||
mock_settings.return_value.google_cloud_project = "test-project"
|
||||
mock_settings.return_value.enable_cloud_tasks = False
|
||||
mock_settings.return_value.gcp_region = "us-central1"
|
||||
mock_settings.return_value.use_cloud_run_jobs_for_video = False
|
||||
|
||||
with patch('subprocess.Popen') as mock_popen:
|
||||
service = WorkerService()
|
||||
result = await service.trigger_audio_worker("test123")
|
||||
|
||||
assert result is True
|
||||
mock_popen.assert_called_once()
|
||||
cmd = mock_popen.call_args[0][0]
|
||||
assert cmd[1:] == ["-m", "backend.workers.audio_worker", "--job-id", "test123"]
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_no_local_subprocess_in_production_even_without_cloud_tasks(self):
|
||||
"""A prod deploy that omits ENABLE_CLOUD_TASKS must still use Cloud Run Jobs."""
|
||||
from backend.services.worker_service import WorkerService, reset_worker_service
|
||||
reset_worker_service()
|
||||
|
||||
with patch('backend.services.worker_service.get_settings') as mock_settings, \
|
||||
patch('backend.services.worker_service.is_production', return_value=True):
|
||||
mock_settings.return_value.admin_tokens = "test-token"
|
||||
mock_settings.return_value.google_cloud_project = "test-project"
|
||||
mock_settings.return_value.enable_cloud_tasks = False
|
||||
mock_settings.return_value.gcp_region = "us-central1"
|
||||
mock_settings.return_value.use_cloud_run_jobs_for_video = False
|
||||
|
||||
mock_operation = MagicMock()
|
||||
mock_operation.metadata = "test-metadata"
|
||||
mock_run_v2, mock_jobs_client = self._mock_run_v2([mock_operation])
|
||||
|
||||
import google.cloud
|
||||
with patch('subprocess.Popen') as mock_popen, \
|
||||
patch.dict('sys.modules', {'google.cloud.run_v2': mock_run_v2}), \
|
||||
patch.object(google.cloud, 'run_v2', mock_run_v2, create=True):
|
||||
service = WorkerService()
|
||||
result = await service.trigger_audio_worker("test123")
|
||||
|
||||
assert result is True
|
||||
mock_popen.assert_not_called()
|
||||
mock_jobs_client.run_job.assert_called_once()
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_trigger_bulk_search_worker_local_subprocess_in_dev_mode(self):
|
||||
"""Bulk search worker also falls back to a local subprocess in dev mode."""
|
||||
from backend.services.worker_service import WorkerService, reset_worker_service
|
||||
reset_worker_service()
|
||||
|
||||
with patch('backend.services.worker_service.get_settings') as mock_settings:
|
||||
mock_settings.return_value.admin_tokens = "test-token"
|
||||
mock_settings.return_value.google_cloud_project = "test-project"
|
||||
mock_settings.return_value.enable_cloud_tasks = False
|
||||
mock_settings.return_value.gcp_region = "us-central1"
|
||||
mock_settings.return_value.use_cloud_run_jobs_for_video = False
|
||||
|
||||
with patch('subprocess.Popen') as mock_popen:
|
||||
service = WorkerService()
|
||||
result = await service.trigger_bulk_search_worker("batch-1")
|
||||
|
||||
assert result is True
|
||||
cmd = mock_popen.call_args[0][0]
|
||||
assert cmd[1:] == ["-m", "backend.workers.bulk_search_worker", "--batch-id", "batch-1"]
|
||||
|
||||
def _mock_run_v2(self, run_job_side_effect):
|
||||
"""Build a mocked run_v2 module whose JobsClient.run_job uses side_effect."""
|
||||
mock_jobs_client = MagicMock()
|
||||
@@ -269,7 +341,7 @@ class TestWorkerService:
|
||||
patch('backend.services.worker_service.asyncio.sleep', new=AsyncMock()) as mock_sleep:
|
||||
mock_settings.return_value.admin_tokens = "test-token"
|
||||
mock_settings.return_value.google_cloud_project = "test-project"
|
||||
mock_settings.return_value.enable_cloud_tasks = False
|
||||
mock_settings.return_value.enable_cloud_tasks = True
|
||||
mock_settings.return_value.gcp_region = "us-central1"
|
||||
mock_settings.return_value.use_cloud_run_jobs_for_video = False
|
||||
|
||||
@@ -304,7 +376,7 @@ class TestWorkerService:
|
||||
patch('backend.services.worker_service.asyncio.sleep', new=AsyncMock()):
|
||||
mock_settings.return_value.admin_tokens = "test-token"
|
||||
mock_settings.return_value.google_cloud_project = "test-project"
|
||||
mock_settings.return_value.enable_cloud_tasks = False
|
||||
mock_settings.return_value.enable_cloud_tasks = True
|
||||
mock_settings.return_value.gcp_region = "us-central1"
|
||||
mock_settings.return_value.use_cloud_run_jobs_for_video = False
|
||||
|
||||
@@ -332,7 +404,7 @@ class TestWorkerService:
|
||||
patch('backend.services.worker_service.asyncio.sleep', new=AsyncMock()) as mock_sleep:
|
||||
mock_settings.return_value.admin_tokens = "test-token"
|
||||
mock_settings.return_value.google_cloud_project = "test-project"
|
||||
mock_settings.return_value.enable_cloud_tasks = False
|
||||
mock_settings.return_value.enable_cloud_tasks = True
|
||||
mock_settings.return_value.gcp_region = "us-central1"
|
||||
mock_settings.return_value.use_cloud_run_jobs_for_video = False
|
||||
|
||||
|
||||
@@ -448,7 +448,14 @@ class TestStorageServiceFileOperations:
|
||||
|
||||
class TestStorageServiceSignedUrls:
|
||||
"""Test signed URL generation."""
|
||||
|
||||
|
||||
@pytest.fixture(autouse=True)
|
||||
def _no_emulator_env(self, monkeypatch):
|
||||
# Other test modules (e.g. test_emulator_integration) export
|
||||
# STORAGE_EMULATOR_HOST at import time, which would divert these tests
|
||||
# into the emulator URL fallback instead of real signing.
|
||||
monkeypatch.delenv("STORAGE_EMULATOR_HOST", raising=False)
|
||||
|
||||
@patch("backend.services.storage_service.storage.Client")
|
||||
@patch("backend.services.storage_service.settings")
|
||||
def test_generate_signed_url(self, mock_settings, mock_client_class):
|
||||
@@ -541,3 +548,42 @@ class TestStorageServiceSignedUrls:
|
||||
assert call_kwargs["service_account_email"] == "sa@project.iam.gserviceaccount.com"
|
||||
assert call_kwargs["access_token"] == "access-token-123"
|
||||
|
||||
|
||||
|
||||
class TestStorageServiceEmulatorUrls:
|
||||
"""Test emulator URL fallback when STORAGE_EMULATOR_HOST is set (local dev)."""
|
||||
|
||||
@patch("backend.services.storage_service.storage.Client")
|
||||
@patch("backend.services.storage_service.settings")
|
||||
def test_get_url_uses_emulator_download_path(self, mock_settings, mock_client_class, monkeypatch):
|
||||
mock_settings.google_cloud_project = "test-project"
|
||||
mock_settings.gcs_bucket_name = "test-bucket"
|
||||
mock_bucket = Mock()
|
||||
mock_bucket.name = "test-bucket"
|
||||
mock_client_class.return_value.bucket.return_value = mock_bucket
|
||||
monkeypatch.setenv("STORAGE_EMULATOR_HOST", "http://127.0.0.1:4443")
|
||||
|
||||
from backend.services.storage_service import StorageService
|
||||
url = StorageService().generate_signed_url("jobs/abc/review-audio/song.ogg")
|
||||
|
||||
assert url == (
|
||||
"http://127.0.0.1:4443/download/storage/v1/b/test-bucket/o/"
|
||||
"jobs%2Fabc%2Freview-audio%2Fsong.ogg?alt=media"
|
||||
)
|
||||
|
||||
@patch("backend.services.storage_service.storage.Client")
|
||||
@patch("backend.services.storage_service.settings")
|
||||
def test_put_url_uses_emulator_signed_style_path(self, mock_settings, mock_client_class, monkeypatch):
|
||||
mock_settings.google_cloud_project = "test-project"
|
||||
mock_settings.gcs_bucket_name = "test-bucket"
|
||||
mock_bucket = Mock()
|
||||
mock_bucket.name = "test-bucket"
|
||||
mock_client_class.return_value.bucket.return_value = mock_bucket
|
||||
monkeypatch.setenv("STORAGE_EMULATOR_HOST", "http://127.0.0.1:4443")
|
||||
|
||||
from backend.services.storage_service import StorageService
|
||||
url = StorageService().generate_signed_upload_url("uploads/x/file.bin")
|
||||
|
||||
# fake-gcs-server accepts PUTs with (unvalidated) X-Goog signature params
|
||||
assert url.startswith("http://127.0.0.1:4443/test-bucket/uploads/x/file.bin?")
|
||||
assert "X-Goog-Signature=" in url
|
||||
+15
-3
@@ -1,5 +1,10 @@
|
||||
# Development Guide
|
||||
|
||||
> **External contributor?** See [CONTRIBUTING.md](../CONTRIBUTING.md) at the repo
|
||||
> root — the backend runs fully locally against emulators with **no GCP
|
||||
> credentials or paid API keys**. This file covers the full setup including
|
||||
> production/deployment concerns.
|
||||
|
||||
## Quick Start
|
||||
|
||||
### Prerequisites
|
||||
@@ -24,9 +29,9 @@ source venv/bin/activate
|
||||
# Install dependencies
|
||||
poetry install
|
||||
|
||||
# Copy environment template
|
||||
# Optional: copy environment template for API keys / overrides
|
||||
# (nothing in it is required to run locally with emulators)
|
||||
cp .env.example .env
|
||||
# Edit .env with required API keys
|
||||
```
|
||||
|
||||
### Frontend Setup
|
||||
@@ -43,13 +48,20 @@ cp .env.example .env.local
|
||||
### Backend with Emulators
|
||||
|
||||
```bash
|
||||
# Start emulators and backend
|
||||
# Start emulators, create the bucket, seed a default theme, run the backend
|
||||
./scripts/run-backend-local.sh --with-emulators
|
||||
|
||||
# Backend runs at http://localhost:8000
|
||||
# API docs at http://localhost:8000/docs
|
||||
```
|
||||
|
||||
In emulator mode no GCP credentials are needed: workers run as local
|
||||
subprocesses, signed URLs are rewritten to emulator URLs, and magic-link
|
||||
login emails are printed to the console. Jobs must be created via **file
|
||||
upload** (URL jobs require the flacfetch cloud service). Set `MODEL_DIR` to
|
||||
enable local audio separation. See [CONTRIBUTING.md](../CONTRIBUTING.md) for
|
||||
the full walkthrough.
|
||||
|
||||
### Frontend
|
||||
|
||||
```bash
|
||||
|
||||
+1
-1
@@ -1,6 +1,6 @@
|
||||
[tool.poetry]
|
||||
name = "karaoke-gen"
|
||||
version = "0.200.1"
|
||||
version = "0.201.0"
|
||||
description = "Generate karaoke videos with synchronized lyrics. Handles the entire process from downloading audio and lyrics to creating the final video with title screens."
|
||||
authors = ["Andrew Beveridge <andrew@beveridge.uk>"]
|
||||
license = "MIT"
|
||||
|
||||
@@ -21,19 +21,22 @@ export PORT="${PORT:-8000}"
|
||||
if [[ "$1" == "--with-emulators" ]]; then
|
||||
echo "📦 Starting GCP emulators..."
|
||||
"$SCRIPT_DIR/start-emulators.sh"
|
||||
|
||||
|
||||
export FIRESTORE_EMULATOR_HOST="127.0.0.1:8080"
|
||||
export STORAGE_EMULATOR_HOST="http://127.0.0.1:4443"
|
||||
export GOOGLE_CLOUD_PROJECT="test-project"
|
||||
export GCS_BUCKET_NAME="test-bucket"
|
||||
export FIRESTORE_COLLECTION="test-jobs"
|
||||
|
||||
# Create test bucket
|
||||
curl -s -X POST "http://127.0.0.1:4443/storage/v1/b" \
|
||||
-H "Content-Type: application/json" \
|
||||
-d '{"name":"test-bucket"}' \
|
||||
--data-urlencode "project=test-project" || true
|
||||
|
||||
export FRONTEND_URL="${FRONTEND_URL:-http://localhost:3000}"
|
||||
|
||||
# Create the bucket and seed a default theme (idempotent) so jobs can be created
|
||||
poetry run python "$SCRIPT_DIR/seed-local-data.py"
|
||||
|
||||
echo "✅ Emulators running"
|
||||
else
|
||||
echo "⚠️ Running WITHOUT emulators: this targets the real GCP project"
|
||||
echo " ($GOOGLE_CLOUD_PROJECT) and requires GCP credentials."
|
||||
echo " External contributors should use: $0 --with-emulators"
|
||||
fi
|
||||
|
||||
# Change to project root
|
||||
|
||||
@@ -0,0 +1,81 @@
|
||||
#!/usr/bin/env python3
|
||||
"""
|
||||
Seed the local GCS emulator (fake-gcs-server) with the minimum data the
|
||||
backend needs to create jobs: the bucket itself plus a default theme.
|
||||
|
||||
Run after starting the emulators (scripts/start-emulators.sh); it is invoked
|
||||
automatically by scripts/run-backend-local.sh --with-emulators. Idempotent —
|
||||
safe to run repeatedly.
|
||||
|
||||
Usage:
|
||||
STORAGE_EMULATOR_HOST=http://127.0.0.1:4443 python scripts/seed-local-data.py
|
||||
|
||||
Environment:
|
||||
STORAGE_EMULATOR_HOST Required — refuses to run against real GCS.
|
||||
GCS_BUCKET_NAME Bucket to create/seed (default: test-bucket)
|
||||
GOOGLE_CLOUD_PROJECT Project for the emulator client (default: test-project)
|
||||
"""
|
||||
|
||||
import json
|
||||
import os
|
||||
import sys
|
||||
from pathlib import Path
|
||||
|
||||
sys.path.insert(0, str(Path(__file__).parent.parent))
|
||||
|
||||
DEFAULT_THEME_ID = "local-default"
|
||||
|
||||
|
||||
def main() -> int:
|
||||
emulator_host = os.getenv("STORAGE_EMULATOR_HOST")
|
||||
if not emulator_host:
|
||||
print("ERROR: STORAGE_EMULATOR_HOST is not set. This script only seeds the")
|
||||
print("local GCS emulator — it refuses to touch real GCS buckets.")
|
||||
return 1
|
||||
|
||||
project = os.getenv("GOOGLE_CLOUD_PROJECT", "test-project")
|
||||
bucket_name = os.getenv("GCS_BUCKET_NAME", "test-bucket")
|
||||
|
||||
from google.cloud import storage
|
||||
|
||||
client = storage.Client(project=project)
|
||||
|
||||
bucket = client.bucket(bucket_name)
|
||||
if not bucket.exists():
|
||||
client.create_bucket(bucket_name)
|
||||
print(f"Created bucket {bucket_name}")
|
||||
else:
|
||||
print(f"Bucket {bucket_name} already exists")
|
||||
|
||||
# Theme registry with one default theme (jobs can't be created without one)
|
||||
registry = {
|
||||
"version": 1,
|
||||
"themes": [
|
||||
{
|
||||
"id": DEFAULT_THEME_ID,
|
||||
"name": "Local Default",
|
||||
"description": "Default theme seeded for local development",
|
||||
"is_default": True,
|
||||
"created_at": "2026-01-01T00:00:00Z",
|
||||
}
|
||||
],
|
||||
}
|
||||
bucket.blob("themes/_metadata.json").upload_from_string(
|
||||
json.dumps(registry, indent=2), content_type="application/json"
|
||||
)
|
||||
print("Seeded themes/_metadata.json")
|
||||
|
||||
from karaoke_gen.style_loader import get_default_style_params
|
||||
|
||||
style_params = get_default_style_params()
|
||||
bucket.blob(f"themes/{DEFAULT_THEME_ID}/style_params.json").upload_from_string(
|
||||
json.dumps(style_params, indent=2), content_type="application/json"
|
||||
)
|
||||
print(f"Seeded themes/{DEFAULT_THEME_ID}/style_params.json")
|
||||
|
||||
print("\nLocal seed data ready — the backend can now create jobs.")
|
||||
return 0
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
sys.exit(main())
|
||||
@@ -43,14 +43,16 @@ else
|
||||
# Stop any existing container
|
||||
docker rm -f test-gcs-emulator >/dev/null 2>&1 || true
|
||||
|
||||
# Start fake-gcs-server
|
||||
# Start fake-gcs-server. -public-host makes it accept signed-URL-style
|
||||
# PUT uploads on this host (see StorageService emulator URL fallback).
|
||||
docker run -d \
|
||||
--name test-gcs-emulator \
|
||||
-p 4443:4443 \
|
||||
fsouza/fake-gcs-server:latest \
|
||||
-scheme http \
|
||||
-port 4443 \
|
||||
-external-url http://localhost:4443 \
|
||||
-external-url http://127.0.0.1:4443 \
|
||||
-public-host 127.0.0.1:4443 \
|
||||
> "$EMULATOR_LOG_DIR/gcs.log" 2>&1
|
||||
|
||||
GCS_CONTAINER=$(docker ps -q -f name=test-gcs-emulator)
|
||||
|
||||
Reference in new issue
Block a user