mirror of
https://github.com/nomadkaraoke/karaoke-gen.git
synced 2026-11-20 13:07:57 +09:00
## Summary Motivated by external contributor Martín (netux) asking "is there an easy way to run the full backend without GCS?" — he couldn't test PR #895's backend changes locally. This PR makes the whole backend runnable locally against emulators with **zero GCP credentials and zero paid API keys**. ### What was broken for contributors - Workers migrated to Cloud Run Jobs for prod robustness (audio-download, audio-separation, lyrics-transcription, bulk-search) silently bypassed the direct-HTTP dev mode → local jobs stalled at PENDING forever - Signed URL generation called `google.auth.default()` → crashed with no credentials (review audio, downloads, theme previews, upload URLs) - Magic-link login 503'd locally (console email provider counted as "not configured") - Fresh emulator had no default theme → every job create returned 422 - `docs/DEVELOPMENT.md` told people to `cp .env.example .env` but the file didn't exist — and nothing loaded `.env` anyway - `run-backend-local.sh` defaulted to the production GCP project ### Changes - **worker_service**: CRJ-based workers run as local subprocesses (`python -m backend.workers.X --job-id Y`, exact CRJ parity) when `ENABLE_CLOUD_TASKS=false` **and** not `is_production()` (regression-tested both ways) - **storage_service**: signed URLs → plain fake-gcs-server URLs when `STORAGE_EMULATOR_HOST` set (GET via JSON API download path; PUT via unvalidated-signature path, enabled by `-public-host` on the emulator) - **users route**: magic-link 503 only in production; dev prints the link to the console - **file_upload bugfix**: a theme with style params but no asset files never got `style_params_gcs_path` written (empty-dict falsiness in `elif theme_style_assets:`) — surfaced by the minimal seed theme, latent in prod - **scripts/seed-local-data.py** (new): creates bucket + default theme; wired into `run-backend-local.sh --with-emulators`, which now uses `test-project` - **config**: `load_dotenv()` so `.env` actually works; new root `.env.example` - **CONTRIBUTING.md** (new): no-credentials quickstart; README + DEVELOPMENT.md link to it ### Verified end-to-end locally (no GCP credentials) Upload job → local audio separation (real models) → transcription → screens → `awaiting_review`; review endpoints all working including `/api/review/{job}/audio/vocals` (the PR #895 endpoint) and 302 → emulator audio URLs. Magic-link login round-trip via console link. ### Tests - 224 tests across affected files pass; full backend unit suite: 3969 passed - `backend/tests/test_emulator_integration.py` has 8 pre-existing failures that reproduce identically on unmodified `main` locally (broken theme-service mock in that file) — unrelated to this PR - New tests: dev-mode subprocess dispatch (audio + bulk-search), prod-mode never-Popen guard, emulator URL generation (GET + PUT) @coderabbitai ignore --------- Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
110 lines
3.3 KiB
Bash
Executable File
110 lines
3.3 KiB
Bash
Executable File
#!/bin/bash
|
|
# Start GCP emulators for local testing
|
|
# This script starts Firestore and GCS emulators in the background
|
|
|
|
set -e
|
|
|
|
SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
|
|
PROJECT_ROOT="$(cd "$SCRIPT_DIR/.." && pwd)"
|
|
EMULATOR_LOG_DIR="$PROJECT_ROOT/.emulator-logs"
|
|
|
|
# Create log directory
|
|
mkdir -p "$EMULATOR_LOG_DIR"
|
|
|
|
echo "🚀 Starting GCP emulators for local testing..."
|
|
echo ""
|
|
|
|
# Check if emulators are already running
|
|
if lsof -Pi :8080 -sTCP:LISTEN -t >/dev/null 2>&1; then
|
|
echo "⚠️ Firestore emulator already running on port 8080"
|
|
else
|
|
echo "📦 Starting Firestore emulator on port 8080..."
|
|
gcloud beta emulators firestore start \
|
|
--host-port=127.0.0.1:8080 \
|
|
--quiet \
|
|
> "$EMULATOR_LOG_DIR/firestore.log" 2>&1 &
|
|
|
|
FIRESTORE_PID=$!
|
|
echo $FIRESTORE_PID > "$EMULATOR_LOG_DIR/firestore.pid"
|
|
echo " ✅ Firestore emulator started (PID: $FIRESTORE_PID)"
|
|
fi
|
|
|
|
if lsof -Pi :4443 -sTCP:LISTEN -t >/dev/null 2>&1; then
|
|
echo "⚠️ GCS emulator already running on port 4443"
|
|
else
|
|
echo "📦 Starting GCS emulator (fake-gcs-server) on port 4443..."
|
|
|
|
# Check if Docker is running
|
|
if ! docker info > /dev/null 2>&1; then
|
|
echo "❌ Docker is not running. Please start Docker and try again."
|
|
exit 1
|
|
fi
|
|
|
|
# Stop any existing container
|
|
docker rm -f test-gcs-emulator >/dev/null 2>&1 || true
|
|
|
|
# Start fake-gcs-server. -public-host makes it accept signed-URL-style
|
|
# PUT uploads on this host (see StorageService emulator URL fallback).
|
|
docker run -d \
|
|
--name test-gcs-emulator \
|
|
-p 4443:4443 \
|
|
fsouza/fake-gcs-server:latest \
|
|
-scheme http \
|
|
-port 4443 \
|
|
-external-url http://127.0.0.1:4443 \
|
|
-public-host 127.0.0.1:4443 \
|
|
> "$EMULATOR_LOG_DIR/gcs.log" 2>&1
|
|
|
|
GCS_CONTAINER=$(docker ps -q -f name=test-gcs-emulator)
|
|
echo " ✅ GCS emulator started (Container: $GCS_CONTAINER)"
|
|
fi
|
|
|
|
echo ""
|
|
echo "⏳ Waiting for emulators to be ready..."
|
|
|
|
# Wait for Firestore emulator with retries (up to 30 seconds)
|
|
FIRESTORE_READY=false
|
|
for i in {1..30}; do
|
|
if curl -s http://127.0.0.1:8080 > /dev/null 2>&1; then
|
|
FIRESTORE_READY=true
|
|
echo "✅ Firestore emulator is ready (after ${i}s)"
|
|
break
|
|
fi
|
|
sleep 1
|
|
done
|
|
|
|
if [ "$FIRESTORE_READY" = false ]; then
|
|
echo "❌ Firestore emulator failed to start after 30s"
|
|
echo "Checking emulator logs:"
|
|
tail -20 "$EMULATOR_LOG_DIR/firestore.log" 2>&1 || echo "No logs found"
|
|
exit 1
|
|
fi
|
|
|
|
# Wait for GCS emulator with retries (up to 10 seconds)
|
|
GCS_READY=false
|
|
for i in {1..10}; do
|
|
if curl -s http://127.0.0.1:4443/storage/v1/b > /dev/null 2>&1; then
|
|
GCS_READY=true
|
|
echo "✅ GCS emulator is ready (after ${i}s)"
|
|
break
|
|
fi
|
|
sleep 1
|
|
done
|
|
|
|
if [ "$GCS_READY" = false ]; then
|
|
echo "❌ GCS emulator failed to start after 10s"
|
|
docker logs test-gcs-emulator 2>&1 | tail -20 || echo "No logs found"
|
|
exit 1
|
|
fi
|
|
|
|
echo ""
|
|
echo "🎉 Emulators started successfully!"
|
|
echo ""
|
|
echo "Environment variables to use:"
|
|
echo " export FIRESTORE_EMULATOR_HOST=127.0.0.1:8080"
|
|
echo " export STORAGE_EMULATOR_HOST=http://127.0.0.1:4443"
|
|
echo " export GOOGLE_CLOUD_PROJECT=test-project"
|
|
echo ""
|
|
echo "To stop emulators, run: ./scripts/stop-emulators.sh"
|
|
|