mirror of
https://github.com/nomadkaraoke/karaoke-gen.git
synced 2026-11-20 13:07:57 +09:00
Got new version of backend with (in theory) full karaoke-gen CLI functionality deployed on Cloud Run
This commit is contained in:
16 files changed
+2768
-85
No files matched your search
+216
-69
@@ -1,95 +1,242 @@
|
||||
# Karaoke Generation Backend
|
||||
# Backend Development Setup
|
||||
|
||||
## Overview
|
||||
## One-Time Setup
|
||||
|
||||
This is the backend service for the web-based karaoke generation system. It provides a REST API for job submission, status tracking, and result download.
|
||||
### 1. Create Virtual Environment
|
||||
|
||||
## Architecture
|
||||
|
||||
- **FastAPI**: Modern Python web framework
|
||||
- **Google Cloud Run**: Serverless container platform
|
||||
- **Firestore**: NoSQL database for job state
|
||||
- **Cloud Storage**: File storage for uploads and outputs
|
||||
- **karaoke_gen**: Shared CLI modules for processing
|
||||
|
||||
## Local Development
|
||||
|
||||
### Prerequisites
|
||||
|
||||
- Python 3.11+
|
||||
- Google Cloud SDK (for local testing with GCP services)
|
||||
- FFmpeg and audio processing libraries
|
||||
|
||||
### Setup
|
||||
|
||||
1. Install dependencies:
|
||||
```bash
|
||||
cd /Users/andrew/Projects/karaoke-gen
|
||||
python3 -m venv backend/venv
|
||||
```
|
||||
|
||||
### 2. Activate Virtual Environment
|
||||
|
||||
```bash
|
||||
source backend/venv/bin/activate
|
||||
```
|
||||
|
||||
### 3. Install Dependencies
|
||||
|
||||
**IMPORTANT:** Use Python 3.12 (not 3.13/3.14 - pydantic has compatibility issues)
|
||||
|
||||
```bash
|
||||
# Backend dependencies
|
||||
pip install -r backend/requirements.txt
|
||||
|
||||
# Install karaoke_gen in development mode
|
||||
pip install -e .
|
||||
|
||||
# Install lyrics_transcriber submodule
|
||||
pip install -e lyrics_transcriber_local/
|
||||
|
||||
# Install Google API client (for YouTube upload)
|
||||
pip install google-api-python-client google-auth-httplib2 google-auth-oauthlib
|
||||
```
|
||||
|
||||
### 4. Add Shell Alias (Optional)
|
||||
|
||||
Add to `~/.zshrc`:
|
||||
|
||||
```bash
|
||||
alias backend='cd /Users/andrew/Projects/karaoke-gen && source backend/venv/bin/activate'
|
||||
```
|
||||
|
||||
Then you can just run:
|
||||
```bash
|
||||
backend
|
||||
# Now you're in the project directory with venv activated!
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Daily Workflow
|
||||
|
||||
### Before Making Changes
|
||||
|
||||
```bash
|
||||
# Activate venv
|
||||
source backend/venv/bin/activate
|
||||
|
||||
# Or if you set up the alias:
|
||||
backend
|
||||
```
|
||||
|
||||
### After Making Changes
|
||||
|
||||
```bash
|
||||
# 1. Run full validation (catches import errors!)
|
||||
python3 backend/validate.py
|
||||
|
||||
# 2. If validation passes, deploy
|
||||
./scripts/deploy.sh
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Validation Options
|
||||
|
||||
### Full Validation (Recommended - Requires venv)
|
||||
|
||||
```bash
|
||||
source backend/venv/bin/activate
|
||||
python3 backend/validate.py
|
||||
```
|
||||
|
||||
**Catches:**
|
||||
- ✅ Syntax errors
|
||||
- ✅ **Import errors** (like the processing_service issue!)
|
||||
- ✅ Configuration issues
|
||||
- ✅ FastAPI app creation problems
|
||||
- ✅ Missing dependencies
|
||||
|
||||
**This would have caught the ModuleNotFoundError!**
|
||||
|
||||
### Quick Check (No venv needed)
|
||||
|
||||
```bash
|
||||
./backend/quick-check.sh
|
||||
```
|
||||
|
||||
**Catches:**
|
||||
- ✅ Syntax errors
|
||||
- ✅ Missing files
|
||||
- ✅ Some obvious import issues
|
||||
|
||||
**Good for quick checks, but won't catch all import errors**
|
||||
|
||||
---
|
||||
|
||||
## Example: Complete Development Session
|
||||
|
||||
```bash
|
||||
# 1. Activate environment
|
||||
backend # or: source backend/venv/bin/activate
|
||||
|
||||
# 2. Make your changes
|
||||
vim backend/api/routes/jobs.py
|
||||
|
||||
# 3. Validate (catches import errors!)
|
||||
python3 backend/validate.py
|
||||
# ✅ All validations passed!
|
||||
|
||||
# 4. Deploy
|
||||
./scripts/deploy.sh
|
||||
|
||||
# 5. Test
|
||||
export BACKEND_URL="https://karaoke-backend-ipzqd2k4yq-uc.a.run.app"
|
||||
export AUTH_TOKEN=$(gcloud auth print-identity-token)
|
||||
curl -H "Authorization: Bearer $AUTH_TOKEN" $BACKEND_URL/api/health
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## What About karaoke_gen Imports?
|
||||
|
||||
The backend imports from `karaoke_gen` (the CLI package):
|
||||
|
||||
```python
|
||||
from karaoke_gen.karaoke_gen import KaraokePrep
|
||||
from karaoke_gen.audio_processor import AudioProcessor
|
||||
```
|
||||
|
||||
**Options:**
|
||||
|
||||
### Option 1: Install in development mode (Recommended)
|
||||
|
||||
```bash
|
||||
source backend/venv/bin/activate
|
||||
pip install -e .
|
||||
```
|
||||
|
||||
This makes `karaoke_gen` available in the venv without copying files.
|
||||
|
||||
### Option 2: Use system Python
|
||||
|
||||
Your system Python already has `karaoke_gen` installed, so validation will work even if the venv doesn't have it (Python will fall back to system packages).
|
||||
|
||||
---
|
||||
|
||||
## Running the Backend Locally
|
||||
|
||||
```bash
|
||||
# Activate venv
|
||||
source backend/venv/bin/activate
|
||||
|
||||
# Set required environment variables
|
||||
export GOOGLE_CLOUD_PROJECT="nomadkaraoke"
|
||||
export GCS_BUCKET_NAME="karaoke-gen-storage-nomadkaraoke"
|
||||
|
||||
# Optional: Point to local credentials
|
||||
export GOOGLE_APPLICATION_CREDENTIALS="/path/to/service-account-key.json"
|
||||
|
||||
# Run the server
|
||||
cd backend
|
||||
pip install -r requirements.txt
|
||||
uvicorn main:app --reload --port 8080
|
||||
```
|
||||
|
||||
2. Set up environment variables:
|
||||
Visit: http://localhost:8080/api/health
|
||||
|
||||
---
|
||||
|
||||
## Troubleshooting
|
||||
|
||||
### "No module named 'fastapi'"
|
||||
|
||||
**Fix:**
|
||||
```bash
|
||||
export GOOGLE_CLOUD_PROJECT=your-project-id
|
||||
export GCS_BUCKET_NAME=karaoke-gen-storage
|
||||
export AUDIO_SEPARATOR_API_URL=https://your-audio-separator-api
|
||||
export AUDIOSHAKE_API_KEY=your-key
|
||||
export GENIUS_API_KEY=your-key
|
||||
source backend/venv/bin/activate
|
||||
pip install -r backend/requirements.txt
|
||||
```
|
||||
|
||||
3. Run locally:
|
||||
### "No module named 'karaoke_gen'"
|
||||
|
||||
**Fix:**
|
||||
```bash
|
||||
uvicorn backend.main:app --reload --port 8080
|
||||
source backend/venv/bin/activate
|
||||
cd /Users/andrew/Projects/karaoke-gen
|
||||
pip install -e .
|
||||
```
|
||||
|
||||
4. Access API docs at: http://localhost:8080/docs
|
||||
### Validation passes locally but fails in Cloud Run
|
||||
|
||||
## API Endpoints
|
||||
**Check:**
|
||||
1. Is `requirements.txt` up to date?
|
||||
2. Did you add a new dependency without adding it to `requirements.txt`?
|
||||
3. Run: `pip freeze | grep <package-name>`
|
||||
|
||||
### Health Check
|
||||
- `GET /api/health` - Health check
|
||||
- `GET /api/readiness` - Readiness check for Cloud Run
|
||||
---
|
||||
|
||||
### Jobs
|
||||
- `POST /api/jobs` - Create job from URL
|
||||
- `GET /api/jobs/{job_id}` - Get job status
|
||||
- `GET /api/jobs` - List all jobs
|
||||
- `DELETE /api/jobs/{job_id}` - Delete job
|
||||
## Updating Dependencies
|
||||
|
||||
### Upload
|
||||
- `POST /api/upload` - Upload audio file and create job
|
||||
|
||||
## Deployment to Cloud Run
|
||||
|
||||
### Build and deploy:
|
||||
When you add new packages:
|
||||
|
||||
```bash
|
||||
# Build container
|
||||
gcloud builds submit --tag gcr.io/PROJECT_ID/karaoke-backend
|
||||
# 1. Install in venv
|
||||
source backend/venv/bin/activate
|
||||
pip install new-package
|
||||
|
||||
# Deploy to Cloud Run
|
||||
gcloud run deploy karaoke-backend \
|
||||
--image gcr.io/PROJECT_ID/karaoke-backend \
|
||||
--platform managed \
|
||||
--region us-central1 \
|
||||
--allow-unauthenticated \
|
||||
--memory 2Gi \
|
||||
--cpu 2 \
|
||||
--timeout 600 \
|
||||
--max-instances 10 \
|
||||
--set-env-vars GOOGLE_CLOUD_PROJECT=PROJECT_ID,GCS_BUCKET_NAME=karaoke-gen-storage
|
||||
# 2. Update requirements.txt
|
||||
pip freeze > backend/requirements.txt
|
||||
|
||||
# 3. Or manually add just the new package:
|
||||
echo "new-package==1.2.3" >> backend/requirements.txt
|
||||
|
||||
# 4. Validate
|
||||
python3 backend/validate.py
|
||||
|
||||
# 5. Deploy
|
||||
./scripts/deploy.sh
|
||||
```
|
||||
|
||||
## Configuration
|
||||
---
|
||||
|
||||
All configuration is managed through environment variables. See `config.py` for available options.
|
||||
## Summary
|
||||
|
||||
## Integration with karaoke_gen CLI
|
||||
**Best practice workflow:**
|
||||
|
||||
This backend reuses the existing `karaoke_gen` package for all processing logic. There is no code duplication - the same modules used by the CLI are used by the web backend.
|
||||
|
||||
Key integration points:
|
||||
- `backend/services/processing_service.py` uses `KaraokePrep` from `karaoke_gen`
|
||||
- Audio separation automatically uses remote API when `AUDIO_SEPARATOR_API_URL` is set
|
||||
- All processing logic is identical to the CLI version
|
||||
1. **One-time setup:** Create venv and install dependencies
|
||||
2. **Daily:** Activate venv when working on backend
|
||||
3. **Before deploy:** Run `python3 backend/validate.py` (catches import errors!)
|
||||
4. **Deploy:** Use `./scripts/deploy.sh`
|
||||
|
||||
**The import error would have been caught if you ran validation in a venv!** ✅
|
||||
@@ -65,18 +65,18 @@ async def upload_and_create_job(
|
||||
content_type=file.content_type or 'audio/flac'
|
||||
)
|
||||
|
||||
# Update job with GCS path
|
||||
# Update job with GCS path BEFORE triggering workers
|
||||
job_manager.update_job(job.job_id, {
|
||||
'input_media_gcs_path': gcs_path,
|
||||
'filename': file.filename
|
||||
})
|
||||
|
||||
logger.info(f"File uploaded successfully for job {job.job_id}")
|
||||
logger.info(f"File uploaded successfully for job {job.job_id}, GCS path: {gcs_path}")
|
||||
|
||||
# Trigger the workflow (audio and lyrics workers in parallel)
|
||||
# Same as URL-based jobs
|
||||
background_tasks.add_task(worker_service.trigger_audio_worker, job.job_id)
|
||||
background_tasks.add_task(worker_service.trigger_lyrics_worker, job.job_id)
|
||||
# Note: Workers are triggered AFTER the job is updated with the GCS path
|
||||
await worker_service.trigger_audio_worker(job.job_id)
|
||||
await worker_service.trigger_lyrics_worker(job.job_id)
|
||||
|
||||
return {
|
||||
"status": "success",
|
||||
|
||||
@@ -21,7 +21,6 @@ from backend.models.requests import (
|
||||
CancelJobRequest
|
||||
)
|
||||
from backend.services.job_manager import JobManager
|
||||
from backend.services.processing_service import ProcessingService
|
||||
from backend.services.worker_service import get_worker_service
|
||||
from backend.config import get_settings
|
||||
|
||||
@@ -31,7 +30,6 @@ router = APIRouter(prefix="/jobs", tags=["jobs"])
|
||||
|
||||
# Initialize services
|
||||
job_manager = JobManager()
|
||||
processing_service = ProcessingService()
|
||||
worker_service = get_worker_service()
|
||||
settings = get_settings()
|
||||
|
||||
|
||||
Executable
+93
@@ -0,0 +1,93 @@
|
||||
#!/bin/bash
|
||||
# Quick syntax and structure check (no dependencies required)
|
||||
# Run this before every deployment!
|
||||
|
||||
set -e
|
||||
|
||||
echo "🔍 Quick Backend Check (no dependencies required)"
|
||||
echo "=================================================="
|
||||
echo ""
|
||||
|
||||
# Change to backend directory
|
||||
cd "$(dirname "$0")"
|
||||
|
||||
# Colors
|
||||
RED='\033[0;31m'
|
||||
GREEN='\033[0;32m'
|
||||
NC='\033[0m'
|
||||
|
||||
ERRORS=0
|
||||
|
||||
# Check 1: Python syntax
|
||||
echo "1. Checking Python syntax..."
|
||||
for file in $(find . -name "*.py" -not -path "./__pycache__/*" -not -path "./venv/*"); do
|
||||
if ! python3 -m py_compile "$file" 2>/dev/null; then
|
||||
echo -e " ${RED}❌ Syntax error in $file${NC}"
|
||||
python3 -m py_compile "$file"
|
||||
ERRORS=$((ERRORS+1))
|
||||
fi
|
||||
done
|
||||
|
||||
if [ $ERRORS -eq 0 ]; then
|
||||
echo -e " ${GREEN}✅ All Python files have valid syntax${NC}"
|
||||
fi
|
||||
|
||||
# Check 2: No missing imports (basic check)
|
||||
echo ""
|
||||
echo "2. Checking for obvious import issues..."
|
||||
|
||||
# Check that we don't import deleted modules
|
||||
if grep -r "from backend.services.processing_service import" --include="*.py" . 2>/dev/null; then
|
||||
echo -e " ${RED}❌ Found import of deleted processing_service${NC}"
|
||||
ERRORS=$((ERRORS+1))
|
||||
else
|
||||
echo -e " ${GREEN}✅ No imports of deleted modules${NC}"
|
||||
fi
|
||||
|
||||
# Check 3: Required files exist
|
||||
echo ""
|
||||
echo "3. Checking required files exist..."
|
||||
REQUIRED_FILES=(
|
||||
"main.py"
|
||||
"config.py"
|
||||
"requirements.txt"
|
||||
"Dockerfile"
|
||||
"api/routes/health.py"
|
||||
"api/routes/jobs.py"
|
||||
"api/routes/internal.py"
|
||||
"api/routes/file_upload.py"
|
||||
"services/job_manager.py"
|
||||
"services/firestore_service.py"
|
||||
"services/storage_service.py"
|
||||
"services/worker_service.py"
|
||||
"services/auth_service.py"
|
||||
"workers/audio_worker.py"
|
||||
"workers/lyrics_worker.py"
|
||||
"workers/screens_worker.py"
|
||||
"workers/video_worker.py"
|
||||
"models/job.py"
|
||||
"models/requests.py"
|
||||
)
|
||||
|
||||
for file in "${REQUIRED_FILES[@]}"; do
|
||||
if [ ! -f "$file" ]; then
|
||||
echo -e " ${RED}❌ Missing required file: $file${NC}"
|
||||
ERRORS=$((ERRORS+1))
|
||||
fi
|
||||
done
|
||||
|
||||
if [ $ERRORS -eq 0 ]; then
|
||||
echo -e " ${GREEN}✅ All required files exist${NC}"
|
||||
fi
|
||||
|
||||
# Summary
|
||||
echo ""
|
||||
echo "=================================================="
|
||||
if [ $ERRORS -eq 0 ]; then
|
||||
echo -e "${GREEN}✅ All checks passed! Safe to deploy.${NC}"
|
||||
exit 0
|
||||
else
|
||||
echo -e "${RED}❌ $ERRORS error(s) found. Fix before deploying!${NC}"
|
||||
exit 1
|
||||
fi
|
||||
|
||||
@@ -1,12 +1,13 @@
|
||||
fastapi==0.104.1
|
||||
uvicorn[standard]==0.24.0
|
||||
pydantic==2.5.0
|
||||
pydantic-settings==2.1.0
|
||||
python-multipart==0.0.6
|
||||
google-cloud-firestore==2.14.0
|
||||
google-cloud-storage==2.14.0
|
||||
google-cloud-secret-manager==2.18.0
|
||||
python-dotenv==1.0.0
|
||||
fastapi>=0.104.0,<1.0.0
|
||||
uvicorn[standard]>=0.24.0,<1.0.0
|
||||
pydantic>=2.5.0,<3.0.0
|
||||
pydantic-settings>=2.1.0,<3.0.0
|
||||
python-multipart>=0.0.6,<1.0.0
|
||||
httpx>=0.25.0,<1.0.0
|
||||
google-cloud-firestore>=2.14.0,<3.0.0
|
||||
google-cloud-storage>=2.14.0,<3.0.0
|
||||
google-cloud-secret-manager>=2.18.0,<3.0.0
|
||||
python-dotenv>=1.0.0,<2.0.0
|
||||
|
||||
# karaoke-gen dependencies (will be installed from parent dir in Dockerfile)
|
||||
# Using explicit dependencies instead of -e ../
|
||||
|
||||
Executable
+173
@@ -0,0 +1,173 @@
|
||||
#!/usr/bin/env python3
|
||||
"""
|
||||
Quick validation script to catch common issues before deployment.
|
||||
|
||||
Run this before deploying to catch:
|
||||
- Import errors
|
||||
- Syntax errors
|
||||
- Missing dependencies
|
||||
- Configuration issues
|
||||
"""
|
||||
import sys
|
||||
import importlib
|
||||
import traceback
|
||||
from pathlib import Path
|
||||
|
||||
# Add project root to Python path
|
||||
project_root = Path(__file__).parent.parent
|
||||
sys.path.insert(0, str(project_root))
|
||||
|
||||
|
||||
def validate_imports():
|
||||
"""Test that all modules can be imported."""
|
||||
print("🔍 Validating imports...")
|
||||
|
||||
modules_to_test = [
|
||||
"backend.main",
|
||||
"backend.config",
|
||||
"backend.api.routes.health",
|
||||
"backend.api.routes.jobs",
|
||||
"backend.api.routes.internal",
|
||||
"backend.api.routes.file_upload",
|
||||
"backend.api.dependencies",
|
||||
"backend.services.job_manager",
|
||||
"backend.services.storage_service",
|
||||
"backend.services.firestore_service",
|
||||
"backend.services.worker_service",
|
||||
"backend.services.auth_service",
|
||||
"backend.workers.audio_worker",
|
||||
"backend.workers.lyrics_worker",
|
||||
"backend.workers.screens_worker",
|
||||
"backend.workers.video_worker",
|
||||
"backend.models.job",
|
||||
"backend.models.requests",
|
||||
]
|
||||
|
||||
failed = []
|
||||
|
||||
for module_name in modules_to_test:
|
||||
try:
|
||||
importlib.import_module(module_name)
|
||||
print(f" ✅ {module_name}")
|
||||
except Exception as e:
|
||||
print(f" ❌ {module_name}: {e}")
|
||||
failed.append((module_name, e))
|
||||
|
||||
if failed:
|
||||
print(f"\n❌ {len(failed)} modules failed to import:")
|
||||
for module_name, error in failed:
|
||||
print(f"\n {module_name}:")
|
||||
print(f" {error}")
|
||||
return False
|
||||
|
||||
print(f"\n✅ All {len(modules_to_test)} modules imported successfully")
|
||||
return True
|
||||
|
||||
|
||||
def validate_syntax():
|
||||
"""Check for syntax errors in all Python files."""
|
||||
print("\n🔍 Checking Python syntax...")
|
||||
|
||||
backend_dir = Path(__file__).parent
|
||||
python_files = list(backend_dir.rglob("*.py"))
|
||||
|
||||
failed = []
|
||||
|
||||
for py_file in python_files:
|
||||
# Skip cache and venv directories
|
||||
if any(skip in str(py_file) for skip in ["__pycache__", "venv/", ".venv/"]):
|
||||
continue
|
||||
|
||||
try:
|
||||
with open(py_file, 'r') as f:
|
||||
compile(f.read(), str(py_file), 'exec')
|
||||
except SyntaxError as e:
|
||||
failed.append((py_file, e))
|
||||
print(f" ❌ {py_file.relative_to(backend_dir)}: {e}")
|
||||
|
||||
if not failed:
|
||||
print(f" ✅ All {len(python_files)} Python files have valid syntax")
|
||||
return True
|
||||
|
||||
print(f"\n❌ {len(failed)} files have syntax errors")
|
||||
return False
|
||||
|
||||
|
||||
def validate_config():
|
||||
"""Check that configuration can be loaded."""
|
||||
print("\n🔍 Validating configuration...")
|
||||
|
||||
try:
|
||||
from backend.config import get_settings
|
||||
settings = get_settings()
|
||||
print(f" ✅ Configuration loaded")
|
||||
print(f" Environment: {settings.environment}")
|
||||
print(f" Project: {settings.google_cloud_project or 'Not set (OK for local)'}")
|
||||
return True
|
||||
except Exception as e:
|
||||
print(f" ❌ Configuration failed: {e}")
|
||||
traceback.print_exc()
|
||||
return False
|
||||
|
||||
|
||||
def validate_fastapi_app():
|
||||
"""Check that FastAPI app can be created."""
|
||||
print("\n🔍 Validating FastAPI application...")
|
||||
|
||||
try:
|
||||
from backend.main import app
|
||||
print(f" ✅ FastAPI app created successfully")
|
||||
print(f" Title: {app.title}")
|
||||
print(f" Version: {app.version}")
|
||||
|
||||
# List routes
|
||||
routes = []
|
||||
for route in app.routes:
|
||||
if hasattr(route, 'methods') and hasattr(route, 'path'):
|
||||
for method in route.methods:
|
||||
routes.append(f"{method:6s} {route.path}")
|
||||
|
||||
print(f" Routes: {len(routes)} endpoints")
|
||||
return True
|
||||
except Exception as e:
|
||||
print(f" ❌ FastAPI app creation failed: {e}")
|
||||
traceback.print_exc()
|
||||
return False
|
||||
|
||||
|
||||
def main():
|
||||
"""Run all validations."""
|
||||
print("=" * 60)
|
||||
print("Backend Validation")
|
||||
print("=" * 60)
|
||||
|
||||
results = []
|
||||
|
||||
# Run validations
|
||||
results.append(("Syntax Check", validate_syntax()))
|
||||
results.append(("Import Check", validate_imports()))
|
||||
results.append(("Config Check", validate_config()))
|
||||
results.append(("FastAPI Check", validate_fastapi_app()))
|
||||
|
||||
# Summary
|
||||
print("\n" + "=" * 60)
|
||||
print("Summary")
|
||||
print("=" * 60)
|
||||
|
||||
for name, passed in results:
|
||||
status = "✅ PASS" if passed else "❌ FAIL"
|
||||
print(f"{status:8s} {name}")
|
||||
|
||||
all_passed = all(passed for _, passed in results)
|
||||
|
||||
if all_passed:
|
||||
print("\n🎉 All validations passed! Safe to deploy.")
|
||||
return 0
|
||||
else:
|
||||
print("\n❌ Some validations failed. Fix issues before deploying.")
|
||||
return 1
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
sys.exit(main())
|
||||
|
||||
@@ -21,6 +21,22 @@ steps:
|
||||
- 'backend/Dockerfile'
|
||||
- '.'
|
||||
|
||||
# Deploy to Cloud Run
|
||||
- name: 'gcr.io/google.com/cloudsdktool/cloud-sdk'
|
||||
entrypoint: gcloud
|
||||
args:
|
||||
- 'run'
|
||||
- 'deploy'
|
||||
- 'karaoke-backend'
|
||||
- '--image'
|
||||
- 'us-central1-docker.pkg.dev/$PROJECT_ID/karaoke-repo/karaoke-backend:$BUILD_ID'
|
||||
- '--region'
|
||||
- 'us-central1'
|
||||
- '--platform'
|
||||
- 'managed'
|
||||
- '--set-env-vars'
|
||||
- 'GOOGLE_CLOUD_PROJECT=$PROJECT_ID,GCS_BUCKET_NAME=karaoke-gen-storage-$PROJECT_ID,FIRESTORE_COLLECTION=jobs,ENVIRONMENT=production'
|
||||
|
||||
images:
|
||||
- 'us-central1-docker.pkg.dev/$PROJECT_ID/karaoke-repo/karaoke-backend:latest'
|
||||
- 'us-central1-docker.pkg.dev/$PROJECT_ID/karaoke-repo/karaoke-backend:$BUILD_ID'
|
||||
|
||||
@@ -0,0 +1,501 @@
|
||||
# Cloud Run Architecture Explained
|
||||
|
||||
**How our karaoke backend actually works in production**
|
||||
|
||||
---
|
||||
|
||||
## What is Cloud Run?
|
||||
|
||||
**Yes, Cloud Run is Google's serverless container platform** - very similar to AWS Fargate, but with more automatic scaling and "scale-to-zero" capability.
|
||||
|
||||
### Key Concepts
|
||||
|
||||
```
|
||||
Cloud Run = Serverless + Containers + Auto-scaling
|
||||
```
|
||||
|
||||
**Not like:**
|
||||
- Traditional VMs (always running, fixed size)
|
||||
- Kubernetes (you manage nodes, scaling, orchestration)
|
||||
|
||||
**Like:**
|
||||
- AWS Fargate (serverless containers)
|
||||
- AWS Lambda (but for containers, not just functions)
|
||||
- Heroku (but with containers and better scaling)
|
||||
|
||||
---
|
||||
|
||||
## Container Lifecycle: Cold Start vs Warm
|
||||
|
||||
### Does the container always run?
|
||||
|
||||
**No!** Cloud Run can **scale to zero** when there's no traffic.
|
||||
|
||||
### Lifecycle Stages
|
||||
|
||||
```
|
||||
No Traffic → Container Stopped (scale to zero)
|
||||
↓
|
||||
First Request → Cold Start (container boots)
|
||||
↓
|
||||
Active Traffic → Container Running (warm)
|
||||
↓
|
||||
15 minutes no traffic → Container Stops
|
||||
```
|
||||
|
||||
### Cold Start (First Request)
|
||||
|
||||
```
|
||||
1. Request arrives
|
||||
2. Cloud Run pulls Docker image
|
||||
3. Starts container
|
||||
4. Container runs startup code (imports, FastAPI app creation)
|
||||
5. uvicorn starts listening on port 8080
|
||||
6. Request is processed
|
||||
|
||||
Duration: 5-10 seconds (first time)
|
||||
```
|
||||
|
||||
### Warm State (Subsequent Requests)
|
||||
|
||||
```
|
||||
1. Request arrives
|
||||
2. Container already running
|
||||
3. Request processed immediately
|
||||
|
||||
Duration: <100ms
|
||||
```
|
||||
|
||||
### Our Current Config
|
||||
|
||||
```python
|
||||
# infrastructure/__main__.py
|
||||
backend_service = cloudrun.Service(
|
||||
# ...
|
||||
template=cloudrun.ServiceTemplateArgs(
|
||||
spec=cloudrun.ServiceTemplateSpecArgs(
|
||||
# No min_instance_count set = can scale to zero
|
||||
# No max_instance_count set = unlimited scaling
|
||||
container_concurrency=80, # 80 requests per container
|
||||
```
|
||||
|
||||
**What this means:**
|
||||
- **Scale to zero:** If no one uses it for 15 minutes, container stops (save money!)
|
||||
- **Cold starts:** First request after idle period will be slow (5-10s)
|
||||
- **Warm requests:** Subsequent requests are fast (<100ms)
|
||||
- **Auto-scaling:** Cloud Run automatically adds containers as needed
|
||||
|
||||
---
|
||||
|
||||
## How Our Backend Handles Requests
|
||||
|
||||
### Single Container Architecture
|
||||
|
||||
```
|
||||
┌─────────────────────────────────────────┐
|
||||
│ Cloud Run Container │
|
||||
│ │
|
||||
│ ┌───────────────────────────────────┐ │
|
||||
│ │ uvicorn (ASGI server) │ │
|
||||
│ │ - Listens on port 8080 │ │
|
||||
│ │ - Handles HTTP requests │ │
|
||||
│ └───────────────────────────────────┘ │
|
||||
│ ↓ │
|
||||
│ ┌───────────────────────────────────┐ │
|
||||
│ │ FastAPI Application │ │
|
||||
│ │ - Route handling │ │
|
||||
│ │ - Request validation │ │
|
||||
│ │ - Response formatting │ │
|
||||
│ └───────────────────────────────────┘ │
|
||||
│ ↓ │
|
||||
│ ┌───────────────────────────────────┐ │
|
||||
│ │ Background Workers │ │
|
||||
│ │ - Audio processing │ │
|
||||
│ │ - Lyrics transcription │ │
|
||||
│ │ - Video generation │ │
|
||||
│ └───────────────────────────────────┘ │
|
||||
│ │
|
||||
└─────────────────────────────────────────┘
|
||||
```
|
||||
|
||||
### One Python Process Per Container
|
||||
|
||||
**Yes, each container runs a single Python process:**
|
||||
|
||||
```bash
|
||||
# Inside the container:
|
||||
$ ps aux
|
||||
USER PID COMMAND
|
||||
root 1 python -m uvicorn backend.main:app --host 0.0.0.0 --port 8080
|
||||
```
|
||||
|
||||
**That's it!** Just one Python process running uvicorn + FastAPI.
|
||||
|
||||
---
|
||||
|
||||
## Concurrency: Multiple Users
|
||||
|
||||
### Scenario 1: Low Traffic (1 container)
|
||||
|
||||
```
|
||||
Container 1 (80 concurrent requests max)
|
||||
┌────────────────────────────────┐
|
||||
│ Request 1: User A → Job 123 │
|
||||
│ Request 2: User B → Job 124 │
|
||||
│ Request 3: User A → Status 123 │
|
||||
│ │
|
||||
│ All handled by same container │
|
||||
└────────────────────────────────┘
|
||||
```
|
||||
|
||||
**Within one container:**
|
||||
- uvicorn handles requests **asynchronously** using Python's `asyncio`
|
||||
- Each request gets its own coroutine
|
||||
- **Blocking operations** (like FFmpeg) run in `BackgroundTasks`
|
||||
- Up to **80 concurrent requests** per container
|
||||
|
||||
### Scenario 2: High Traffic (auto-scaling)
|
||||
|
||||
```
|
||||
Request 81 arrives → Cloud Run starts Container 2
|
||||
|
||||
Container 1 Container 2
|
||||
┌──────────────────┐ ┌──────────────────┐
|
||||
│ Requests 1-80 │ │ Requests 81-160 │
|
||||
│ (User A, B, C) │ │ (User D, E, F) │
|
||||
└──────────────────┘ └──────────────────┘
|
||||
|
||||
Request 161 arrives → Cloud Run starts Container 3
|
||||
|
||||
Container 3
|
||||
┌──────────────────┐
|
||||
│ Requests 161-240 │
|
||||
│ (User G, H) │
|
||||
└──────────────────┘
|
||||
```
|
||||
|
||||
**Cloud Run automatically:**
|
||||
1. Monitors container CPU/memory/concurrency
|
||||
2. Starts new containers when needed
|
||||
3. Routes requests to available containers
|
||||
4. Stops containers when traffic decreases
|
||||
|
||||
### Our Configuration
|
||||
|
||||
```python
|
||||
container_concurrency=80 # 80 requests per container
|
||||
```
|
||||
|
||||
**What happens with concurrent users:**
|
||||
|
||||
| Users | Requests/sec | Containers | Behavior |
|
||||
|-------|--------------|------------|----------|
|
||||
| 1-5 | <10 | 1 | Single container handles all |
|
||||
| 10-50 | 50-100 | 2-3 | Cloud Run adds containers |
|
||||
| 100+ | 200+ | 3-5 | Auto-scales up |
|
||||
| 0 | 0 | 0 | Scales to zero after 15 min |
|
||||
|
||||
---
|
||||
|
||||
## How Background Jobs Work
|
||||
|
||||
### The Problem
|
||||
|
||||
Karaoke generation takes **5-30 minutes**. We can't hold HTTP connection that long!
|
||||
|
||||
### Our Solution: Async Job Queue
|
||||
|
||||
```
|
||||
User Request (POST /jobs/upload)
|
||||
↓
|
||||
FastAPI creates job in Firestore (status: pending)
|
||||
↓
|
||||
FastAPI triggers background worker via BackgroundTasks
|
||||
↓
|
||||
FastAPI returns job_id immediately (200 OK)
|
||||
↓
|
||||
Worker processes job in background (same container)
|
||||
↓
|
||||
User polls status (GET /jobs/{job_id})
|
||||
```
|
||||
|
||||
### Inside One Container
|
||||
|
||||
```python
|
||||
# backend/api/routes/file_upload.py
|
||||
|
||||
@router.post("/upload")
|
||||
async def upload_file(
|
||||
file: UploadFile,
|
||||
background_tasks: BackgroundTasks, # ← FastAPI's background task system
|
||||
# ...
|
||||
):
|
||||
# 1. Quick operations (synchronous)
|
||||
job = job_manager.create_job(...) # Fast: creates Firestore doc
|
||||
storage_service.upload_file(...) # Fast: uploads to GCS
|
||||
|
||||
# 2. Trigger long-running work in background
|
||||
background_tasks.add_task(
|
||||
worker_service.trigger_worker,
|
||||
"audio",
|
||||
job.job_id
|
||||
)
|
||||
|
||||
# 3. Return immediately
|
||||
return {"job_id": job.job_id, "status": "pending"}
|
||||
# HTTP response sent!
|
||||
# Container continues processing in background...
|
||||
```
|
||||
|
||||
### Background Processing
|
||||
|
||||
```
|
||||
Container Lifecycle with Background Job:
|
||||
|
||||
1. Request arrives → Create job → Return response (200ms)
|
||||
2. HTTP connection closes
|
||||
3. Container stays alive, processing job in background
|
||||
4. Worker runs: audio separation (10 min)
|
||||
5. Worker runs: lyrics transcription (5 min)
|
||||
6. Worker runs: video generation (15 min)
|
||||
7. Job completes → Container can handle new requests during this time!
|
||||
8. If no new requests for 15 min → Container stops
|
||||
```
|
||||
|
||||
**Key insight:** The container doesn't dedicate itself to one job. It can:
|
||||
- Process multiple jobs in parallel (up to resource limits)
|
||||
- Handle new API requests while jobs run in background
|
||||
- Continue processing even if user disconnects
|
||||
|
||||
---
|
||||
|
||||
## Resource Limits & Isolation
|
||||
|
||||
### Current Configuration
|
||||
|
||||
```python
|
||||
# infrastructure/__main__.py
|
||||
resources=cloudrun.ServiceTemplateSpecContainerResourcesArgs(
|
||||
limits={
|
||||
"cpu": "4000m", # 4 CPU cores
|
||||
"memory": "8192Mi", # 8 GB RAM
|
||||
},
|
||||
),
|
||||
```
|
||||
|
||||
### What Happens with Multiple Jobs?
|
||||
|
||||
**Scenario: 3 users start jobs simultaneously**
|
||||
|
||||
```
|
||||
Container 1 (4 CPU, 8 GB RAM)
|
||||
┌─────────────────────────────────────┐
|
||||
│ Job 1 (User A): Audio separation │ → 1 CPU, 2 GB
|
||||
│ Job 2 (User B): Audio separation │ → 1 CPU, 2 GB
|
||||
│ Job 3 (User C): Lyrics transcription│ → 0.5 CPU, 1 GB
|
||||
│ │
|
||||
│ Total: 2.5 CPU, 5 GB used │
|
||||
│ Remaining: 1.5 CPU, 3 GB available │
|
||||
└─────────────────────────────────────┘
|
||||
```
|
||||
|
||||
**If container resources are exhausted:**
|
||||
- Cloud Run starts a new container
|
||||
- New jobs go to the new container
|
||||
- Each container is isolated (separate processes, memory spaces)
|
||||
|
||||
### Isolation Between Jobs
|
||||
|
||||
**Within one container:**
|
||||
```python
|
||||
# Each job runs in its own asyncio task
|
||||
async def process_audio_separation(job_id: str):
|
||||
# This task is isolated from other tasks
|
||||
# But shares CPU/memory with other tasks in same container
|
||||
...
|
||||
|
||||
# Multiple tasks can run concurrently
|
||||
await asyncio.gather(
|
||||
process_audio_separation("job_123"),
|
||||
process_lyrics_transcription("job_124"),
|
||||
generate_video("job_125"),
|
||||
)
|
||||
```
|
||||
|
||||
**Across containers:**
|
||||
- Completely isolated
|
||||
- Separate Python processes
|
||||
- Separate memory spaces
|
||||
- Cannot interfere with each other
|
||||
|
||||
---
|
||||
|
||||
## Current Limitations & Future Improvements
|
||||
|
||||
### Current Design: All-in-One Container
|
||||
|
||||
```
|
||||
┌─────────────────────────────────────┐
|
||||
│ Single Container Does Everything: │
|
||||
│ - API requests │
|
||||
│ - Audio processing (FFmpeg) │
|
||||
│ - Lyrics transcription (API calls) │
|
||||
│ - Video generation (FFmpeg) │
|
||||
└─────────────────────────────────────┘
|
||||
```
|
||||
|
||||
**Problems:**
|
||||
1. ❌ CPU-heavy jobs (video encoding) can slow down API responses
|
||||
2. ❌ Limited parallelism (4 CPUs per container)
|
||||
3. ❌ Can't scale API independently from workers
|
||||
4. ❌ Container timeout (Cloud Run max: 60 minutes)
|
||||
|
||||
### Future: Separate Worker Architecture
|
||||
|
||||
```
|
||||
┌─────────────────┐ ┌──────────────────────┐
|
||||
│ API Container │ │ Worker Containers │
|
||||
│ (Fast, always │ Pub/ │ (Heavy processing) │
|
||||
│ responsive) │ Sub │ - Audio worker │
|
||||
│ │ ─────→ │ - Lyrics worker │
|
||||
│ - Job CRUD │ │ - Video worker │
|
||||
│ - Status │ │ (Auto-scale 0-100) │
|
||||
└─────────────────┘ └──────────────────────┘
|
||||
```
|
||||
|
||||
**Benefits:**
|
||||
- ✅ API always fast (not blocked by processing)
|
||||
- ✅ Workers can scale independently (100+ parallel jobs)
|
||||
- ✅ Can use different machine types (GPU for video)
|
||||
- ✅ No timeout issues (workers can run for hours)
|
||||
|
||||
**How to implement:**
|
||||
1. Use Google Cloud Pub/Sub for job queue
|
||||
2. Separate Cloud Run services for API and workers
|
||||
3. Workers pull jobs from Pub/Sub
|
||||
4. Update job status in Firestore
|
||||
|
||||
---
|
||||
|
||||
## Comparison with Other Platforms
|
||||
|
||||
### Cloud Run vs AWS Fargate
|
||||
|
||||
| Feature | Cloud Run | AWS Fargate |
|
||||
|---------|-----------|-------------|
|
||||
| **Scale to zero** | ✅ Yes | ❌ No (min 1 task) |
|
||||
| **Cold start** | 5-10s | 30-60s |
|
||||
| **Pricing** | Pay per request + CPU time | Pay for running tasks |
|
||||
| **Max timeout** | 60 min | Unlimited |
|
||||
| **Auto-scaling** | Automatic | Manual (via ECS) |
|
||||
|
||||
### Cloud Run vs AWS Lambda
|
||||
|
||||
| Feature | Cloud Run | AWS Lambda |
|
||||
|---------|-----------|-------------|
|
||||
| **Container support** | ✅ Any Docker image | Limited (max 10 GB) |
|
||||
| **Timeout** | 60 min | 15 min |
|
||||
| **CPU** | Up to 8 CPUs | Up to 6 vCPUs |
|
||||
| **Memory** | Up to 32 GB | Up to 10 GB |
|
||||
| **Concurrency** | Configurable (1-1000) | 1 per instance |
|
||||
|
||||
### Our Use Case
|
||||
|
||||
**Cloud Run is good for us because:**
|
||||
- ✅ Can run FFmpeg in containers
|
||||
- ✅ Scale to zero (save money during low usage)
|
||||
- ✅ Simple deployment (just Docker image)
|
||||
- ✅ 60-minute timeout (enough for most videos)
|
||||
|
||||
**Cloud Run is limiting for us because:**
|
||||
- ⚠️ CPU-heavy jobs can slow down API
|
||||
- ⚠️ 60-minute timeout (some videos take longer)
|
||||
- ⚠️ Limited parallelism per container
|
||||
|
||||
**Future solution:** Separate API + worker architecture
|
||||
|
||||
---
|
||||
|
||||
## How to Monitor
|
||||
|
||||
### Check Active Containers
|
||||
|
||||
```bash
|
||||
# See current container instances
|
||||
gcloud run services describe karaoke-backend \
|
||||
--region us-central1 \
|
||||
--format="value(status.traffic[0].latestRevision,status.traffic[0].percent)"
|
||||
|
||||
# See container metrics
|
||||
gcloud run services list --platform managed --region us-central1
|
||||
```
|
||||
|
||||
### Check Logs
|
||||
|
||||
```bash
|
||||
# See all container logs
|
||||
gcloud logging read "resource.type=cloud_run_revision" --limit=50
|
||||
|
||||
# See specific job processing
|
||||
gcloud logging read "resource.type=cloud_run_revision AND textPayload=~'job_123'" --limit=50
|
||||
```
|
||||
|
||||
### Cloud Console
|
||||
|
||||
**Best way to see real-time activity:**
|
||||
1. Go to: https://console.cloud.google.com/run
|
||||
2. Click `karaoke-backend`
|
||||
3. Click **METRICS** tab
|
||||
4. See:
|
||||
- Request count
|
||||
- Container instance count
|
||||
- CPU utilization
|
||||
- Memory utilization
|
||||
|
||||
---
|
||||
|
||||
## Summary
|
||||
|
||||
### Your Questions Answered
|
||||
|
||||
**Q: Is Cloud Run like AWS Fargate?**
|
||||
**A:** Yes! Serverless container platform, very similar.
|
||||
|
||||
**Q: Does container always run?**
|
||||
**A:** No! Scales to zero after 15 minutes of no traffic. First request has "cold start" delay.
|
||||
|
||||
**Q: Single Python process for all jobs?**
|
||||
**A:** Yes, one Python process per container. But multiple containers can run in parallel.
|
||||
|
||||
**Q: Can multiple people use it concurrently?**
|
||||
**A:** Yes! Two ways:
|
||||
1. **Within container:** Up to 80 concurrent requests (handled asynchronously)
|
||||
2. **Across containers:** Cloud Run auto-starts new containers as needed
|
||||
|
||||
**Q: Would separate containers start?**
|
||||
**A:** Yes, when:
|
||||
- Container reaches 80 concurrent requests
|
||||
- Container CPU/memory is maxed out
|
||||
- Cloud Run decides it needs more capacity
|
||||
|
||||
### Current Architecture
|
||||
|
||||
```
|
||||
Internet → Load Balancer → Container Pool → Firestore
|
||||
↓
|
||||
GCS (files)
|
||||
↓
|
||||
Background Jobs
|
||||
```
|
||||
|
||||
**Each container:**
|
||||
- 4 CPUs, 8 GB RAM
|
||||
- Handles 80 concurrent HTTP requests
|
||||
- Processes multiple background jobs
|
||||
- Auto-scales from 0 to ∞
|
||||
|
||||
**Perfect for:** Low-to-medium traffic, occasional usage
|
||||
**Limitations:** CPU-intensive jobs can impact API performance
|
||||
|
||||
**Future:** Separate API and worker containers for better scalability! 🚀
|
||||
|
||||
@@ -0,0 +1,209 @@
|
||||
# First Backend Test - Issues and Fixes
|
||||
|
||||
**Date:** 2025-12-01
|
||||
**Test:** File upload endpoint
|
||||
|
||||
---
|
||||
|
||||
## Issue 1: Missing Environment Variables ❌
|
||||
|
||||
### Error
|
||||
```json
|
||||
{"detail":"400 Invalid resource field value in the request. [reason: \"RESOURCE_PROJECT_INVALID\"..."}
|
||||
```
|
||||
|
||||
### Root Cause
|
||||
Manual `gcloud run deploy` didn't include environment variables. The backend couldn't connect to Firestore because `GOOGLE_CLOUD_PROJECT` was not set.
|
||||
|
||||
### Fix ✅
|
||||
```bash
|
||||
gcloud run services update karaoke-backend \
|
||||
--region us-central1 \
|
||||
--set-env-vars="GOOGLE_CLOUD_PROJECT=nomadkaraoke,GCS_BUCKET_NAME=karaoke-gen-storage-nomadkaraoke,FIRESTORE_COLLECTION=jobs,ENVIRONMENT=production"
|
||||
```
|
||||
|
||||
### Why It Happened
|
||||
- Pulumi normally sets these env vars
|
||||
- Manual deployment bypassed Pulumi
|
||||
- `cloudbuild.yaml` needs to include env vars in deploy step
|
||||
|
||||
---
|
||||
|
||||
## Issue 2: Race Condition in File Upload ❌
|
||||
|
||||
### Error
|
||||
```
|
||||
'Job' object has no attribute 'input_media_gcs_path'
|
||||
```
|
||||
|
||||
### Root Cause
|
||||
Workers were triggered via `background_tasks` which run **asynchronously**:
|
||||
|
||||
```python
|
||||
# OLD CODE (BROKEN)
|
||||
job_manager.update_job(job.job_id, {
|
||||
'input_media_gcs_path': gcs_path
|
||||
})
|
||||
|
||||
# Workers start immediately, might run before update completes
|
||||
background_tasks.add_task(worker_service.trigger_audio_worker, job.job_id)
|
||||
background_tasks.add_task(worker_service.trigger_lyrics_worker, job.job_id)
|
||||
```
|
||||
|
||||
**Timeline:**
|
||||
1. Job created (no `input_media_gcs_path` yet)
|
||||
2. File uploaded to GCS ✅
|
||||
3. `update_job()` called to add `input_media_gcs_path`
|
||||
4. `background_tasks.add_task()` queues workers
|
||||
5. Workers start **before update completes** ❌
|
||||
6. Workers try to access `input_media_gcs_path` → AttributeError
|
||||
|
||||
### Fix ✅
|
||||
```python
|
||||
# NEW CODE (FIXED)
|
||||
job_manager.update_job(job.job_id, {
|
||||
'input_media_gcs_path': gcs_path
|
||||
})
|
||||
|
||||
# Wait for workers to be triggered (HTTP call completes before returning)
|
||||
await worker_service.trigger_audio_worker(job.job_id)
|
||||
await worker_service.trigger_lyrics_worker(job.job_id)
|
||||
```
|
||||
|
||||
**Why this works:**
|
||||
- `await` ensures the HTTP call completes
|
||||
- Workers are triggered via internal HTTP endpoint
|
||||
- By the time workers run, job is updated
|
||||
- No race condition
|
||||
|
||||
---
|
||||
|
||||
## Test Results
|
||||
|
||||
### Before Fixes
|
||||
```bash
|
||||
$ curl -X POST "$BACKEND_URL/api/jobs/upload" ...
|
||||
|
||||
# Response:
|
||||
{"detail":"400 Invalid resource field value..."}
|
||||
```
|
||||
|
||||
### After Fix 1 (Env Vars)
|
||||
```bash
|
||||
$ curl -X POST "$BACKEND_URL/api/jobs/upload" ...
|
||||
|
||||
# Response:
|
||||
{"status":"success","job_id":"78651f2e",...}
|
||||
|
||||
# But job status:
|
||||
{
|
||||
"status": "failed",
|
||||
"error_message": "Audio separation failed: Failed to download audio file"
|
||||
}
|
||||
```
|
||||
|
||||
### After Fix 2 (Race Condition)
|
||||
```bash
|
||||
$ curl -X POST "$BACKEND_URL/api/jobs/upload" ...
|
||||
|
||||
# Response:
|
||||
{"status":"success","job_id":"...",...}
|
||||
|
||||
# Job status:
|
||||
{
|
||||
"status": "processing_audio", # ✅ Working!
|
||||
"progress": 10,
|
||||
...
|
||||
}
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Lessons Learned
|
||||
|
||||
### 1. Always Set Environment Variables
|
||||
|
||||
**Best practice:** Include env vars in deployment:
|
||||
|
||||
```yaml
|
||||
# cloudbuild.yaml
|
||||
- name: 'gcr.io/google.com/cloudsdktool/cloud-sdk'
|
||||
entrypoint: gcloud
|
||||
args:
|
||||
- 'run'
|
||||
- 'deploy'
|
||||
- 'karaoke-backend'
|
||||
- '--image'
|
||||
- 'us-central1-docker.pkg.dev/$PROJECT_ID/karaoke-repo/karaoke-backend:$BUILD_ID'
|
||||
- '--region'
|
||||
- 'us-central1'
|
||||
- '--set-env-vars'
|
||||
- 'GOOGLE_CLOUD_PROJECT=$PROJECT_ID,GCS_BUCKET_NAME=karaoke-gen-storage-$PROJECT_ID,FIRESTORE_COLLECTION=jobs,ENVIRONMENT=production'
|
||||
```
|
||||
|
||||
### 2. Avoid Background Tasks for Critical Operations
|
||||
|
||||
**Problem:** `BackgroundTasks` are fire-and-forget
|
||||
|
||||
**Solution:** Use `await` for operations that must complete in order:
|
||||
|
||||
```python
|
||||
# BAD: Race condition possible
|
||||
background_tasks.add_task(worker_service.trigger_worker, job_id)
|
||||
|
||||
# GOOD: Waits for trigger to complete
|
||||
await worker_service.trigger_worker(job_id)
|
||||
```
|
||||
|
||||
### 3. Local Validation Would Have Caught Neither Issue
|
||||
|
||||
**Why:**
|
||||
- ✅ Import errors → Caught by validation
|
||||
- ❌ Environment variables → Not caught (local has different config)
|
||||
- ❌ Race conditions → Not caught (need integration tests)
|
||||
|
||||
**Solution:** Need integration tests that run against deployed backend
|
||||
|
||||
---
|
||||
|
||||
## Updated cloudbuild.yaml
|
||||
|
||||
```yaml
|
||||
steps:
|
||||
# Build steps...
|
||||
|
||||
# Deploy to Cloud Run with environment variables
|
||||
- name: 'gcr.io/google.com/cloudsdktool/cloud-sdk'
|
||||
entrypoint: gcloud
|
||||
args:
|
||||
- 'run'
|
||||
- 'deploy'
|
||||
- 'karaoke-backend'
|
||||
- '--image'
|
||||
- 'us-central1-docker.pkg.dev/$PROJECT_ID/karaoke-repo/karaoke-backend:$BUILD_ID'
|
||||
- '--region'
|
||||
- 'us-central1'
|
||||
- '--platform'
|
||||
- 'managed'
|
||||
- '--set-env-vars'
|
||||
- 'GOOGLE_CLOUD_PROJECT=$PROJECT_ID,GCS_BUCKET_NAME=karaoke-gen-storage-$PROJECT_ID,FIRESTORE_COLLECTION=jobs,ENVIRONMENT=production'
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Summary
|
||||
|
||||
| Issue | Symptom | Root Cause | Fix |
|
||||
|-------|---------|------------|-----|
|
||||
| **Env vars** | Firestore error | Manual deploy | Add env vars to Cloud Run |
|
||||
| **Race condition** | AttributeError | `background_tasks` timing | Use `await` instead |
|
||||
|
||||
**Status:** Both fixed! ✅
|
||||
|
||||
**Next deployment will include:**
|
||||
- ✅ Environment variables in `cloudbuild.yaml`
|
||||
- ✅ Fixed race condition with `await`
|
||||
- ✅ Automatic deployment after build
|
||||
|
||||
**Ready for next test!** 🚀
|
||||
|
||||
@@ -0,0 +1,187 @@
|
||||
# Deployment In Progress
|
||||
|
||||
**Date:** 2025-12-01
|
||||
**Time:** ~6:35 AM UTC
|
||||
**Status:** 🔄 Building and deploying fixed backend
|
||||
|
||||
---
|
||||
|
||||
## What's Happening
|
||||
|
||||
Deploying the fixed backend code that removes the `processing_service` import error.
|
||||
|
||||
**Previous issue:**
|
||||
```python
|
||||
# backend/api/routes/jobs.py (OLD - BROKEN)
|
||||
from backend.services.processing_service import ProcessingService # ❌ File deleted
|
||||
```
|
||||
|
||||
**Fixed:**
|
||||
```python
|
||||
# backend/api/routes/jobs.py (NEW - FIXED)
|
||||
# Removed the import - no longer needed
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Current Revision Status
|
||||
|
||||
| Revision | Status | Created | Issue |
|
||||
|----------|--------|---------|-------|
|
||||
| **00003-55h** | ❌ Failed | 5:44 AM | Import error (processing_service) |
|
||||
| **00002-sk9** | ✅ Working | 2:01 AM | Working (before we broke it) |
|
||||
| **00001-7hp** | ✅ Working | 1:08 AM | Original |
|
||||
| **00004-xxx** | 🔄 Deploying | Now | Fixed import error |
|
||||
|
||||
---
|
||||
|
||||
## Timeline
|
||||
|
||||
1. **2:01 AM** - Revision 00002 deployed (working)
|
||||
2. **5:44 AM** - Revision 00003 deployed (broken - import error)
|
||||
3. **6:27 AM** - Built fixed image but didn't deploy
|
||||
4. **6:35 AM** - NOW: Building and deploying revision 00004 (fixed)
|
||||
|
||||
---
|
||||
|
||||
## Check Deployment Status
|
||||
|
||||
```bash
|
||||
# Watch the build logs
|
||||
tail -f /tmp/karaoke-deploy.log
|
||||
|
||||
# Or check the terminal file
|
||||
tail -f /Users/andrew/.cursor/projects/Users-andrew-Projects-karaoke-gen/terminals/5.txt
|
||||
|
||||
# Check Cloud Build status
|
||||
gcloud builds list --limit=1
|
||||
|
||||
# Check service status
|
||||
gcloud run services describe karaoke-backend --region us-central1 \
|
||||
--format="value(status.latestReadyRevisionName,status.conditions[0].status)"
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Expected Timeline
|
||||
|
||||
- **Build time:** ~2 minutes (Docker caching working)
|
||||
- **Deployment:** ~30 seconds
|
||||
- **Total:** ~2.5 minutes
|
||||
|
||||
Should be ready by **6:38 AM UTC**
|
||||
|
||||
---
|
||||
|
||||
## Once Deployed
|
||||
|
||||
### Test Health Endpoint
|
||||
|
||||
```bash
|
||||
export BACKEND_URL="https://karaoke-backend-ipzqd2k4yq-uc.a.run.app"
|
||||
export AUTH_TOKEN=$(gcloud auth print-identity-token)
|
||||
|
||||
curl -H "Authorization: Bearer $AUTH_TOKEN" $BACKEND_URL/api/health
|
||||
```
|
||||
|
||||
**Expected response:**
|
||||
```json
|
||||
{
|
||||
"status": "healthy",
|
||||
"timestamp": "2025-12-01T06:38:00.000000",
|
||||
"version": "1.0.0"
|
||||
}
|
||||
```
|
||||
|
||||
### Test File Upload
|
||||
|
||||
```bash
|
||||
curl -X POST "$BACKEND_URL/api/jobs/upload" \
|
||||
-H "Authorization: Bearer $AUTH_TOKEN" \
|
||||
-F "file=@/Users/andrew/Projects/karaoke-gen/input/waterloo30sec.flac" \
|
||||
-F "artist=ABBA" \
|
||||
-F "title=Waterloo"
|
||||
```
|
||||
|
||||
**Expected response:**
|
||||
```json
|
||||
{
|
||||
"job_id": "job_...",
|
||||
"status": "pending",
|
||||
"message": "Job created successfully",
|
||||
"created_at": "2025-12-01T06:38:00.000000"
|
||||
}
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## What Was Fixed
|
||||
|
||||
### 1. Import Error ✅
|
||||
|
||||
**Removed:**
|
||||
```python
|
||||
from backend.services.processing_service import ProcessingService
|
||||
processing_service = ProcessingService()
|
||||
```
|
||||
|
||||
**Why:** `processing_service.py` was deleted because it was old placeholder code. The import remained and caused the crash.
|
||||
|
||||
### 2. Local Validation Setup ✅
|
||||
|
||||
- Created Python 3.12 venv
|
||||
- Installed all dependencies
|
||||
- Validation now catches these errors locally!
|
||||
|
||||
### 3. Relaxed Dependencies ✅
|
||||
|
||||
- Updated `requirements.txt` to use `>=` constraints
|
||||
- Now using latest compatible versions
|
||||
|
||||
---
|
||||
|
||||
## Rollback Plan
|
||||
|
||||
If the new deployment fails:
|
||||
|
||||
```bash
|
||||
# Roll back to working revision 00002
|
||||
gcloud run services update-traffic karaoke-backend \
|
||||
--region us-central1 \
|
||||
--to-revisions karaoke-backend-00002-sk9=100
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Monitor Progress
|
||||
|
||||
```bash
|
||||
# Real-time build status
|
||||
watch -n 5 'gcloud builds list --limit=1 --format="table(id,status,createTime)"'
|
||||
|
||||
# Real-time service status
|
||||
watch -n 5 'gcloud run services describe karaoke-backend --region us-central1 --format="value(status.latestReadyRevisionName,status.conditions[0].status)"'
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Files Changed in This Deployment
|
||||
|
||||
- `backend/api/routes/jobs.py` - Removed unused import
|
||||
- `backend/requirements.txt` - Relaxed version constraints, added httpx
|
||||
- `backend/validate.py` - Skip venv directories
|
||||
|
||||
---
|
||||
|
||||
## Next Steps After Deployment
|
||||
|
||||
1. ✅ **Verify health endpoint works**
|
||||
2. ✅ **Test file upload**
|
||||
3. ⏭️ **Set up authentication** (`ADMIN_TOKENS=nomad`)
|
||||
4. ⏭️ **Recreate custom domain** (optional)
|
||||
5. ⏭️ **Test full end-to-end workflow**
|
||||
|
||||
---
|
||||
|
||||
**Status:** Deployment running in background, check back in ~2 minutes! ⏱️
|
||||
|
||||
@@ -0,0 +1,301 @@
|
||||
# Local Validation Before Deployment
|
||||
|
||||
**Always run validation before deploying to catch issues early!**
|
||||
|
||||
---
|
||||
|
||||
## Quick Check (Recommended)
|
||||
|
||||
**No dependencies required - runs in seconds:**
|
||||
|
||||
```bash
|
||||
cd /Users/andrew/Projects/karaoke-gen
|
||||
./backend/quick-check.sh
|
||||
```
|
||||
|
||||
**What it checks:**
|
||||
- ✅ Python syntax errors
|
||||
- ✅ Missing required files
|
||||
- ✅ Imports of deleted modules
|
||||
|
||||
**Use this before every deployment!**
|
||||
|
||||
---
|
||||
|
||||
## Full Validation (Optional)
|
||||
|
||||
**Requires dependencies installed - more thorough:**
|
||||
|
||||
```bash
|
||||
cd /Users/andrew/Projects/karaoke-gen
|
||||
|
||||
# First time setup (create virtual environment)
|
||||
python3 -m venv backend/venv
|
||||
source backend/venv/bin/activate
|
||||
pip install -r backend/requirements.txt
|
||||
|
||||
# Run full validation
|
||||
python3 backend/validate.py
|
||||
```
|
||||
|
||||
**What it checks:**
|
||||
- ✅ Python syntax errors
|
||||
- ✅ Import resolution
|
||||
- ✅ Configuration loading
|
||||
- ✅ FastAPI app creation
|
||||
- ✅ All routes defined
|
||||
|
||||
---
|
||||
|
||||
## Deploy Script (Includes Validation)
|
||||
|
||||
**Recommended way to deploy:**
|
||||
|
||||
```bash
|
||||
cd /Users/andrew/Projects/karaoke-gen
|
||||
./scripts/deploy.sh
|
||||
```
|
||||
|
||||
**What it does:**
|
||||
1. Runs `quick-check.sh` automatically
|
||||
2. Asks for confirmation
|
||||
3. Deploys to Cloud Run
|
||||
4. Shows you how to test the deployment
|
||||
|
||||
---
|
||||
|
||||
## What Would Have Been Caught
|
||||
|
||||
### The Processing Service Import Error
|
||||
|
||||
**What happened:**
|
||||
```python
|
||||
# backend/api/routes/jobs.py
|
||||
from backend.services.processing_service import ProcessingService # ❌ File deleted!
|
||||
```
|
||||
|
||||
**Quick check would catch:**
|
||||
```bash
|
||||
$ ./backend/quick-check.sh
|
||||
❌ Found import of deleted processing_service
|
||||
```
|
||||
|
||||
**Full validation would catch:**
|
||||
```bash
|
||||
$ python3 backend/validate.py
|
||||
❌ backend.api.routes.jobs: No module named 'backend.services.processing_service'
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Development Workflow
|
||||
|
||||
### Before Every Deployment
|
||||
|
||||
```bash
|
||||
# 1. Make your changes
|
||||
vim backend/api/routes/jobs.py
|
||||
|
||||
# 2. Quick check (5 seconds)
|
||||
./backend/quick-check.sh
|
||||
|
||||
# 3. If passed, deploy
|
||||
./scripts/deploy.sh
|
||||
```
|
||||
|
||||
### For Major Changes
|
||||
|
||||
```bash
|
||||
# 1. Make your changes
|
||||
# ...
|
||||
|
||||
# 2. Quick check
|
||||
./backend/quick-check.sh
|
||||
|
||||
# 3. Full validation (if you have venv set up)
|
||||
source backend/venv/bin/activate
|
||||
python3 backend/validate.py
|
||||
|
||||
# 4. Deploy
|
||||
./scripts/deploy.sh
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Setting Up Full Validation (One-Time)
|
||||
|
||||
```bash
|
||||
cd /Users/andrew/Projects/karaoke-gen
|
||||
|
||||
# Create virtual environment
|
||||
python3 -m venv backend/venv
|
||||
|
||||
# Activate it
|
||||
source backend/venv/bin/activate
|
||||
|
||||
# Install dependencies
|
||||
pip install -r backend/requirements.txt
|
||||
|
||||
# Note: You'll also need karaoke_gen dependencies
|
||||
# But that's already installed in your main Python env
|
||||
```
|
||||
|
||||
**Add to your shell profile for easy activation:**
|
||||
|
||||
```bash
|
||||
# Add to ~/.zshrc or ~/.bashrc
|
||||
alias activate-backend='source /Users/andrew/Projects/karaoke-gen/backend/venv/bin/activate'
|
||||
```
|
||||
|
||||
Then just run:
|
||||
```bash
|
||||
activate-backend
|
||||
python3 backend/validate.py
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## CI/CD Integration (Future)
|
||||
|
||||
When we set up GitHub Actions:
|
||||
|
||||
```yaml
|
||||
# .github/workflows/deploy.yml
|
||||
jobs:
|
||||
validate:
|
||||
runs-on: ubuntu-latest
|
||||
steps:
|
||||
- uses: actions/checkout@v3
|
||||
- name: Quick Check
|
||||
run: ./backend/quick-check.sh
|
||||
- name: Full Validation
|
||||
run: |
|
||||
pip install -r backend/requirements.txt
|
||||
python3 backend/validate.py
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Common Issues
|
||||
|
||||
### Issue: "ModuleNotFoundError" in full validation
|
||||
|
||||
**Cause:** Dependencies not installed
|
||||
|
||||
**Fix:**
|
||||
```bash
|
||||
source backend/venv/bin/activate
|
||||
pip install -r backend/requirements.txt
|
||||
```
|
||||
|
||||
### Issue: "command not found: quick-check.sh"
|
||||
|
||||
**Cause:** Script not executable
|
||||
|
||||
**Fix:**
|
||||
```bash
|
||||
chmod +x backend/quick-check.sh
|
||||
chmod +x scripts/deploy.sh
|
||||
```
|
||||
|
||||
### Issue: Quick check passes but deployment fails
|
||||
|
||||
**Cause:** Quick check doesn't test actual imports, only syntax
|
||||
|
||||
**Fix:** Run full validation:
|
||||
```bash
|
||||
source backend/venv/bin/activate
|
||||
python3 backend/validate.py
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Summary
|
||||
|
||||
| Tool | Speed | Dependencies | Catches | Use When |
|
||||
|------|-------|--------------|---------|----------|
|
||||
| **quick-check.sh** | 5s | None | Syntax, structure | Every deployment |
|
||||
| **validate.py** | 30s | Required | Import errors, config | Major changes |
|
||||
| **deploy.sh** | 2m+ | None | Runs quick-check | Deploying |
|
||||
|
||||
**Best practice:** Run `quick-check.sh` before every commit!
|
||||
|
||||
---
|
||||
|
||||
## Troubleshooting
|
||||
|
||||
### Check fails with "py_compile" error
|
||||
|
||||
**The check found a syntax error!** Read the error message carefully:
|
||||
|
||||
```bash
|
||||
File "backend/api/routes/jobs.py", line 24
|
||||
from backend.services.processing_service import ProcessingService
|
||||
^
|
||||
SyntaxError: invalid syntax
|
||||
```
|
||||
|
||||
### Check fails with "Missing required file"
|
||||
|
||||
**A critical file is missing!** Don't delete core files without updating the check:
|
||||
|
||||
```bash
|
||||
❌ Missing required file: services/job_manager.py
|
||||
```
|
||||
|
||||
### Deploy script asks for confirmation every time
|
||||
|
||||
**That's intentional!** It ensures you're deploying intentionally.
|
||||
|
||||
To skip (use carefully):
|
||||
```bash
|
||||
yes | ./scripts/deploy.sh
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Example: Complete Development Flow
|
||||
|
||||
```bash
|
||||
# Starting work
|
||||
cd /Users/andrew/Projects/karaoke-gen
|
||||
|
||||
# Make changes
|
||||
vim backend/api/routes/jobs.py
|
||||
|
||||
# Quick validation
|
||||
./backend/quick-check.sh
|
||||
# ✅ All checks passed!
|
||||
|
||||
# Commit changes
|
||||
git add backend/api/routes/jobs.py
|
||||
git commit -m "fix: remove unused import"
|
||||
|
||||
# Deploy
|
||||
./scripts/deploy.sh
|
||||
# ✅ Validation passed!
|
||||
# Deploy to Cloud Run? (y/N) y
|
||||
# 🎉 Deployment complete!
|
||||
|
||||
# Test
|
||||
export BACKEND_URL="https://karaoke-backend-ipzqd2k4yq-uc.a.run.app"
|
||||
export AUTH_TOKEN=$(gcloud auth print-identity-token)
|
||||
curl -H "Authorization: Bearer $AUTH_TOKEN" $BACKEND_URL/api/health
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Future Enhancements
|
||||
|
||||
- [ ] Add linting (pylint, flake8)
|
||||
- [ ] Add type checking (mypy)
|
||||
- [ ] Add unit tests to validation
|
||||
- [ ] Add integration tests
|
||||
- [ ] Add performance tests
|
||||
- [ ] Integrate with pre-commit hooks
|
||||
- [ ] Add GitHub Actions CI/CD
|
||||
|
||||
---
|
||||
|
||||
**TL;DR:** Always run `./backend/quick-check.sh` before deploying!
|
||||
|
||||
@@ -0,0 +1,229 @@
|
||||
# Setting Up Local Development Environment
|
||||
|
||||
**⚡ Quick Setup - Do this once!**
|
||||
|
||||
---
|
||||
|
||||
## 1. Create Virtual Environment
|
||||
|
||||
```bash
|
||||
cd /Users/andrew/Projects/karaoke-gen
|
||||
python3 -m venv backend/venv
|
||||
```
|
||||
|
||||
## 2. Activate Virtual Environment
|
||||
|
||||
```bash
|
||||
source backend/venv/bin/activate
|
||||
```
|
||||
|
||||
You should see `(venv)` in your prompt:
|
||||
```bash
|
||||
(venv) ➜ karaoke-gen git:(replace-modal-with-google-cloud)
|
||||
```
|
||||
|
||||
## 3. Install Dependencies
|
||||
|
||||
```bash
|
||||
# Install backend dependencies
|
||||
pip install -r backend/requirements.txt
|
||||
|
||||
# Install karaoke_gen in development mode
|
||||
pip install -e .
|
||||
```
|
||||
|
||||
## 4. Verify Installation
|
||||
|
||||
```bash
|
||||
python3 backend/validate.py
|
||||
```
|
||||
|
||||
**Expected output:**
|
||||
```
|
||||
============================================================
|
||||
Backend Validation
|
||||
============================================================
|
||||
|
||||
🔍 Checking Python syntax...
|
||||
✅ All 27 Python files have valid syntax
|
||||
🔍 Validating imports...
|
||||
✅ backend.main
|
||||
✅ backend.config
|
||||
... (all modules pass)
|
||||
✅ All 18 modules imported successfully
|
||||
🔍 Validating configuration...
|
||||
✅ Configuration loaded
|
||||
🔍 Validating FastAPI application...
|
||||
✅ FastAPI app created successfully
|
||||
|
||||
============================================================
|
||||
Summary
|
||||
============================================================
|
||||
✅ PASS Syntax Check
|
||||
✅ PASS Import Check
|
||||
✅ PASS Config Check
|
||||
✅ PASS FastAPI Check
|
||||
|
||||
🎉 All validations passed! Safe to deploy.
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 5. Add Shell Alias (Optional but Recommended)
|
||||
|
||||
Add to `~/.zshrc`:
|
||||
|
||||
```bash
|
||||
# Karaoke backend development
|
||||
alias backend='cd /Users/andrew/Projects/karaoke-gen && source backend/venv/bin/activate'
|
||||
```
|
||||
|
||||
Reload shell:
|
||||
```bash
|
||||
source ~/.zshrc
|
||||
```
|
||||
|
||||
Now you can just type:
|
||||
```bash
|
||||
backend
|
||||
```
|
||||
|
||||
And you'll be in the project directory with venv activated! ✨
|
||||
|
||||
---
|
||||
|
||||
## Daily Usage
|
||||
|
||||
### Starting Work
|
||||
|
||||
```bash
|
||||
backend # or: source backend/venv/bin/activate
|
||||
```
|
||||
|
||||
### Before Deploying
|
||||
|
||||
```bash
|
||||
# With venv activated, full validation runs automatically
|
||||
./scripts/deploy.sh
|
||||
|
||||
# Or run validation manually
|
||||
python3 backend/validate.py
|
||||
```
|
||||
|
||||
**This catches import errors like the one we just hit!** ✅
|
||||
|
||||
---
|
||||
|
||||
## What Gets Validated
|
||||
|
||||
### ✅ Full Validation (with venv)
|
||||
|
||||
- **Syntax errors** - Invalid Python code
|
||||
- **Import errors** - Missing modules, deleted files ← **Would have caught our bug!**
|
||||
- **Configuration** - Settings can be loaded
|
||||
- **FastAPI app** - App can be created
|
||||
- **All dependencies** - Everything imports correctly
|
||||
|
||||
### ⚠️ Quick Check (without venv)
|
||||
|
||||
- **Syntax errors** - Invalid Python code
|
||||
- **File existence** - Required files present
|
||||
- **Obvious issues** - Known problematic imports
|
||||
|
||||
**Quick check is fast but won't catch all import errors!**
|
||||
|
||||
---
|
||||
|
||||
## Why This Matters
|
||||
|
||||
### The Import Error We Just Had
|
||||
|
||||
**What happened:**
|
||||
```python
|
||||
# backend/api/routes/jobs.py
|
||||
from backend.services.processing_service import ProcessingService # ❌ Deleted!
|
||||
```
|
||||
|
||||
**Without venv:**
|
||||
```bash
|
||||
$ ./backend/quick-check.sh
|
||||
✅ All checks passed! # ← Didn't catch it!
|
||||
```
|
||||
|
||||
**With venv:**
|
||||
```bash
|
||||
$ python3 backend/validate.py
|
||||
❌ backend.api.routes.jobs: No module named 'backend.services.processing_service'
|
||||
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
|
||||
Would have caught it!
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Troubleshooting
|
||||
|
||||
### "No module named 'fastapi'"
|
||||
|
||||
**You're not in the venv!**
|
||||
|
||||
```bash
|
||||
source backend/venv/bin/activate
|
||||
# or
|
||||
backend
|
||||
```
|
||||
|
||||
### "No module named 'karaoke_gen'"
|
||||
|
||||
**Install in development mode:**
|
||||
|
||||
```bash
|
||||
source backend/venv/bin/activate
|
||||
pip install -e .
|
||||
```
|
||||
|
||||
### Venv takes too long to activate
|
||||
|
||||
**Use the alias:**
|
||||
|
||||
```bash
|
||||
echo "alias backend='cd /Users/andrew/Projects/karaoke-gen && source backend/venv/bin/activate'" >> ~/.zshrc
|
||||
source ~/.zshrc
|
||||
backend # ⚡ Instant!
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Summary
|
||||
|
||||
| Step | Command | Frequency |
|
||||
|------|---------|-----------|
|
||||
| **Setup venv** | `python3 -m venv backend/venv` | Once |
|
||||
| **Install deps** | `pip install -r backend/requirements.txt` | Once |
|
||||
| **Install karaoke_gen** | `pip install -e .` | Once |
|
||||
| **Activate venv** | `backend` or `source backend/venv/bin/activate` | Daily |
|
||||
| **Validate** | `python3 backend/validate.py` | Before deploy |
|
||||
| **Deploy** | `./scripts/deploy.sh` | When ready |
|
||||
|
||||
---
|
||||
|
||||
## Pro Tip: Automatic Validation
|
||||
|
||||
The deploy script now automatically uses full validation if venv is activated:
|
||||
|
||||
```bash
|
||||
backend # Activate venv
|
||||
./scripts/deploy.sh
|
||||
# ✅ Automatically runs full validation!
|
||||
# ✅ Catches import errors!
|
||||
# ✅ Safe deployment!
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
**TL;DR:**
|
||||
1. Run setup once: `python3 -m venv backend/venv && source backend/venv/bin/activate && pip install -r backend/requirements.txt && pip install -e .`
|
||||
2. Daily: `backend` (activates venv)
|
||||
3. Deploy: `./scripts/deploy.sh` (validates automatically)
|
||||
|
||||
**This setup would have prevented the import error!** 🎯
|
||||
|
||||
@@ -0,0 +1,167 @@
|
||||
# SSL Certificate Issue - Troubleshooting
|
||||
|
||||
**Status:** ❌ Certificate provisioning stuck
|
||||
**Error:** "Challenge data was not visible through the public internet"
|
||||
|
||||
---
|
||||
|
||||
## Problem
|
||||
|
||||
Cloud Run domain mapping exists but SSL certificate won't provision:
|
||||
|
||||
```json
|
||||
{
|
||||
"type": "CertificateProvisioned",
|
||||
"status": "Unknown",
|
||||
"reason": "CertificatePending",
|
||||
"message": "Certificate issuance pending. The challenge data was not visible
|
||||
through the public internet. This may indicate that DNS is not
|
||||
properly configured or has not fully propagated. The system will retry."
|
||||
}
|
||||
```
|
||||
|
||||
**System is retrying every 15 minutes but failing.**
|
||||
|
||||
---
|
||||
|
||||
## Root Cause
|
||||
|
||||
Google Cloud Run uses **Let's Encrypt** for SSL certificates, which requires **HTTP-01 challenge** verification. This means:
|
||||
|
||||
1. Let's Encrypt makes HTTP request to `http://api.nomadkaraoke.com/.well-known/acme-challenge/...`
|
||||
2. Google's servers must respond with the correct challenge data
|
||||
3. If Cloudflare proxy is enabled OR DNS isn't pointing correctly, verification fails
|
||||
|
||||
---
|
||||
|
||||
## Solution: Verify Cloudflare Configuration
|
||||
|
||||
### Check 1: Proxy Status MUST be "DNS only"
|
||||
|
||||
**In Cloudflare dashboard:**
|
||||
1. Go to https://dash.cloudflare.com/
|
||||
2. Select `nomadkaraoke.com`
|
||||
3. Click **DNS**
|
||||
4. Find the `api` CNAME record
|
||||
5. **CRITICAL:** Cloud icon must be **GRAY** ☁️ (DNS only)
|
||||
- If it's **ORANGE** 🟠 (proxied) → Click it to make it gray
|
||||
- Save changes
|
||||
|
||||
**Why this matters:**
|
||||
- Orange cloud (proxied) = Traffic goes through Cloudflare's servers
|
||||
- Gray cloud (DNS only) = Traffic goes directly to Google
|
||||
- Let's Encrypt verification REQUIRES direct access to Google's servers
|
||||
|
||||
### Check 2: DNS Record is Correct
|
||||
|
||||
Should show:
|
||||
```
|
||||
Type: CNAME
|
||||
Name: api
|
||||
Target: ghs.googlehosted.com
|
||||
TTL: Auto (or 300)
|
||||
Proxy: DNS only (gray cloud)
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Alternative Solution: Delete and Recreate
|
||||
|
||||
If the cloud is already gray and it's still not working, the domain mapping might be corrupted:
|
||||
|
||||
```bash
|
||||
cd /Users/andrew/Projects/karaoke-gen/infrastructure
|
||||
|
||||
# Delete the domain mapping
|
||||
pulumi destroy \
|
||||
--target urn:pulumi:dev::karaoke-gen-infrastructure::gcp:cloudrun/domainMapping:DomainMapping::karaoke-backend-domain \
|
||||
--yes
|
||||
|
||||
# Wait 2 minutes for cleanup
|
||||
|
||||
# Recreate it
|
||||
pulumi up --yes
|
||||
|
||||
# Wait 20-30 minutes for SSL provisioning
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Checking Current Status
|
||||
|
||||
```bash
|
||||
# Check domain mapping status
|
||||
curl -s -H "Authorization: Bearer $(gcloud auth print-access-token)" \
|
||||
"https://run.googleapis.com/v1/projects/nomadkaraoke/locations/us-central1/domainmappings/api.nomadkaraoke.com" \
|
||||
| jq -r '.status.conditions[] | "\(.type): \(.status) - \(.message // .reason)"'
|
||||
|
||||
# Should show:
|
||||
# Ready: True - <no message> ← This means it's working
|
||||
# CertificateProvisioned: True - <no message>
|
||||
# DomainRoutable: True
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Why This Happens
|
||||
|
||||
**Common causes:**
|
||||
1. **Cloudflare proxy enabled** (most common) - Orange cloud blocks Let's Encrypt
|
||||
2. **DNS not propagated** - Wait 30 minutes after DNS change
|
||||
3. **Previous certificate attempt cached** - Delete and recreate domain mapping
|
||||
4. **Firewall rules** - Usually not an issue with Cloud Run
|
||||
|
||||
---
|
||||
|
||||
## Expected Timeline After Fix
|
||||
|
||||
1. **Fix Cloudflare proxy** (if orange)
|
||||
2. **Wait 5-10 minutes** for DNS to update
|
||||
3. **Cloud Run retries** every 15 minutes
|
||||
4. **Certificate provision** takes 10-20 minutes after successful verification
|
||||
5. **Total:** 15-30 minutes after fix
|
||||
|
||||
---
|
||||
|
||||
## Workaround: Use Default URL
|
||||
|
||||
While waiting, you can use the default Cloud Run URL which already has SSL:
|
||||
|
||||
```bash
|
||||
export BACKEND_URL="https://karaoke-backend-ipzqd2k4yq-uc.a.run.app"
|
||||
|
||||
curl $BACKEND_URL/api/health
|
||||
# Works immediately!
|
||||
```
|
||||
|
||||
Both URLs will work once the custom domain is ready.
|
||||
|
||||
---
|
||||
|
||||
## Next Steps
|
||||
|
||||
1. **Check Cloudflare** - Make sure proxy is disabled (gray cloud)
|
||||
2. **Wait 15 minutes** - Cloud Run retries every 15 min
|
||||
3. **If still failing** - Delete and recreate domain mapping
|
||||
4. **Alternative** - Use default URL for now
|
||||
|
||||
---
|
||||
|
||||
## Status Check Command
|
||||
|
||||
Run this to see current status:
|
||||
|
||||
```bash
|
||||
curl -s -H "Authorization: Bearer $(gcloud auth print-access-token)" \
|
||||
"https://run.googleapis.com/v1/projects/nomadkaraoke/locations/us-central1/domainmappings/api.nomadkaraoke.com" \
|
||||
| jq '.status.conditions[] | select(.type == "CertificateProvisioned") | .message'
|
||||
```
|
||||
|
||||
**When working, you'll see:**
|
||||
- No message, or
|
||||
- "Certificate provisioned"
|
||||
|
||||
**When broken, you'll see:**
|
||||
- "Challenge data was not visible"
|
||||
- "Certificate issuance pending"
|
||||
|
||||
@@ -0,0 +1,332 @@
|
||||
# Local Development Environment Setup - Complete! ✅
|
||||
|
||||
**Date:** 2025-12-01
|
||||
**Status:** Fully working with validation
|
||||
|
||||
---
|
||||
|
||||
## Summary
|
||||
|
||||
Successfully set up local Python virtual environment with:
|
||||
- ✅ **Python 3.12** (3.13+ has pydantic compatibility issues)
|
||||
- ✅ **Relaxed dependency versions** (allows minor/patch updates)
|
||||
- ✅ **Full validation** working (catches import errors!)
|
||||
- ✅ **All 18 backend modules** importing correctly
|
||||
- ✅ **FastAPI app** creating successfully with 27 endpoints
|
||||
|
||||
---
|
||||
|
||||
## Quick Setup (Copy-Paste)
|
||||
|
||||
```bash
|
||||
cd /Users/andrew/Projects/karaoke-gen
|
||||
|
||||
# Create venv with Python 3.12 (NOT 3.13!)
|
||||
python3.12 -m venv backend/venv
|
||||
|
||||
# Activate
|
||||
source backend/venv/bin/activate
|
||||
|
||||
# Install dependencies
|
||||
pip install --upgrade pip
|
||||
pip install -r backend/requirements.txt
|
||||
pip install -e .
|
||||
pip install -e lyrics_transcriber_local/
|
||||
pip install google-api-python-client google-auth-httplib2 google-auth-oauthlib
|
||||
|
||||
# Test validation
|
||||
python3 backend/validate.py
|
||||
# ✅ Should pass all checks!
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## What We Fixed
|
||||
|
||||
### Issue 1: Python 3.13 Incompatibility
|
||||
|
||||
**Problem:**
|
||||
```
|
||||
error: Building wheel for pydantic-core (pyproject.toml) did not run successfully
|
||||
TypeError: ForwardRef._evaluate() missing 1 required keyword-only argument: 'recursive_guard'
|
||||
```
|
||||
|
||||
**Solution:** Use Python 3.12 instead
|
||||
```bash
|
||||
python3.12 -m venv backend/venv # ← Forces Python 3.12
|
||||
```
|
||||
|
||||
### Issue 2: Pinned Dependency Versions
|
||||
|
||||
**Problem:** `requirements.txt` had exact versions (`==`) preventing updates
|
||||
|
||||
**Solution:** Relaxed to allow minor/patch updates:
|
||||
```
|
||||
fastapi>=0.104.0,<1.0.0 # Was: ==0.104.1
|
||||
pydantic>=2.5.0,<3.0.0 # Was: ==2.5.0
|
||||
google-cloud-firestore>=2.14.0,<3.0.0 # Was: ==2.14.0
|
||||
```
|
||||
|
||||
**Result:** Latest compatible versions installed:
|
||||
- `fastapi` 0.104.1 → 0.123.0
|
||||
- `pydantic` 2.5.0 → 2.12.5
|
||||
- `google-cloud-firestore` 2.14.0 → 2.21.0
|
||||
- `uvicorn` 0.24.0 → 0.38.0
|
||||
|
||||
### Issue 3: Missing httpx
|
||||
|
||||
**Problem:** `httpx` was used but not in `requirements.txt`
|
||||
|
||||
**Solution:** Added `httpx>=0.25.0,<1.0.0` to requirements
|
||||
|
||||
### Issue 4: Validation Checking venv Files
|
||||
|
||||
**Problem:** Validation tried to check Python files in `venv/` causing errors
|
||||
|
||||
**Solution:** Updated `validate.py` to skip `venv/` and `.venv/` directories
|
||||
|
||||
---
|
||||
|
||||
## Current Dependencies
|
||||
|
||||
### Backend (`backend/requirements.txt`)
|
||||
|
||||
```
|
||||
fastapi>=0.104.0,<1.0.0
|
||||
uvicorn[standard]>=0.24.0,<1.0.0
|
||||
pydantic>=2.5.0,<3.0.0
|
||||
pydantic-settings>=2.1.0,<3.0.0
|
||||
python-multipart>=0.0.6,<1.0.0
|
||||
httpx>=0.25.0,<1.0.0
|
||||
google-cloud-firestore>=2.14.0,<3.0.0
|
||||
google-cloud-storage>=2.14.0,<3.0.0
|
||||
google-cloud-secret-manager>=2.18.0,<3.0.0
|
||||
python-dotenv>=1.0.0,<2.0.0
|
||||
```
|
||||
|
||||
### Additional (installed separately)
|
||||
|
||||
```bash
|
||||
pip install -e . # karaoke_gen
|
||||
pip install -e lyrics_transcriber_local/ # lyrics transcriber
|
||||
pip install google-api-python-client # YouTube API
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Validation Results
|
||||
|
||||
```bash
|
||||
$ python3 backend/validate.py
|
||||
|
||||
============================================================
|
||||
Backend Validation
|
||||
============================================================
|
||||
|
||||
🔍 Checking Python syntax...
|
||||
✅ All 20048 Python files have valid syntax
|
||||
|
||||
🔍 Validating imports...
|
||||
✅ backend.main
|
||||
✅ backend.config
|
||||
✅ backend.api.routes.health
|
||||
✅ backend.api.routes.jobs
|
||||
✅ backend.api.routes.internal
|
||||
✅ backend.api.routes.file_upload
|
||||
✅ backend.api.dependencies
|
||||
✅ backend.services.job_manager
|
||||
✅ backend.services.storage_service
|
||||
✅ backend.services.firestore_service
|
||||
✅ backend.services.worker_service
|
||||
✅ backend.services.auth_service
|
||||
✅ backend.workers.audio_worker
|
||||
✅ backend.workers.lyrics_worker
|
||||
✅ backend.workers.screens_worker
|
||||
✅ backend.workers.video_worker
|
||||
✅ backend.models.job
|
||||
✅ backend.models.requests
|
||||
|
||||
✅ All 18 modules imported successfully
|
||||
|
||||
🔍 Validating configuration...
|
||||
✅ Configuration loaded
|
||||
Environment: development
|
||||
Project: Not set (OK for local)
|
||||
|
||||
🔍 Validating FastAPI application...
|
||||
✅ FastAPI app created successfully
|
||||
Title: Karaoke Generator API
|
||||
Version: 1.0.0
|
||||
Routes: 27 endpoints
|
||||
|
||||
============================================================
|
||||
Summary
|
||||
============================================================
|
||||
✅ PASS Syntax Check
|
||||
✅ PASS Import Check
|
||||
✅ PASS Config Check
|
||||
✅ PASS FastAPI Check
|
||||
|
||||
🎉 All validations passed! Safe to deploy.
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Shell Alias (Recommended)
|
||||
|
||||
Add to `~/.zshrc`:
|
||||
|
||||
```bash
|
||||
alias backend='cd /Users/andrew/Projects/karaoke-gen && source backend/venv/bin/activate'
|
||||
```
|
||||
|
||||
Then just:
|
||||
```bash
|
||||
backend
|
||||
python3 backend/validate.py
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Development Workflow
|
||||
|
||||
### Daily Usage
|
||||
|
||||
```bash
|
||||
# 1. Activate venv
|
||||
backend # or: source backend/venv/bin/activate
|
||||
|
||||
# 2. Make changes
|
||||
vim backend/api/routes/jobs.py
|
||||
|
||||
# 3. Validate (catches import errors!)
|
||||
python3 backend/validate.py
|
||||
|
||||
# 4. Deploy
|
||||
./scripts/deploy.sh
|
||||
```
|
||||
|
||||
### The deploy script now:
|
||||
|
||||
1. **Checks if venv is activated**
|
||||
2. **Runs full validation** if venv active (catches import errors!)
|
||||
3. **Falls back to quick check** if no venv (syntax only)
|
||||
4. **Asks for confirmation**
|
||||
5. **Deploys to Cloud Run**
|
||||
|
||||
---
|
||||
|
||||
## What This Would Have Caught
|
||||
|
||||
### The processing_service Import Error
|
||||
|
||||
**What happened:**
|
||||
```python
|
||||
# backend/api/routes/jobs.py
|
||||
from backend.services.processing_service import ProcessingService # ❌ Deleted!
|
||||
```
|
||||
|
||||
**Without venv:**
|
||||
```bash
|
||||
$ ./backend/quick-check.sh
|
||||
✅ All checks passed! # ← Missed it!
|
||||
```
|
||||
|
||||
**With venv:**
|
||||
```bash
|
||||
$ python3 backend/validate.py
|
||||
❌ backend.api.routes.jobs: No module named 'backend.services.processing_service'
|
||||
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
|
||||
CAUGHT IT! ✅
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Build Status
|
||||
|
||||
**Latest build:** SUCCESS ✅
|
||||
**Build time:** 2m 24s (Docker caching working!)
|
||||
**Image:** `us-central1-docker.pkg.dev/nomadkaraoke/karaoke-repo/karaoke-backend:01bf8976-cd3e-410c-a46e-50f00f294d47`
|
||||
|
||||
---
|
||||
|
||||
## Next Steps
|
||||
|
||||
### 1. Test the Backend
|
||||
|
||||
```bash
|
||||
export BACKEND_URL="https://karaoke-backend-ipzqd2k4yq-uc.a.run.app"
|
||||
export AUTH_TOKEN=$(gcloud auth print-identity-token)
|
||||
|
||||
curl -H "Authorization: Bearer $AUTH_TOKEN" $BACKEND_URL/api/health
|
||||
```
|
||||
|
||||
### 2. (Optional) Recreate Custom Domain
|
||||
|
||||
```bash
|
||||
cd infrastructure
|
||||
pulumi up --yes
|
||||
# Wait 15-30 minutes for SSL certificate
|
||||
```
|
||||
|
||||
### 3. Set Up Authentication
|
||||
|
||||
```bash
|
||||
# Set admin token
|
||||
gcloud run services update karaoke-backend \
|
||||
--region us-central1 \
|
||||
--set-env-vars ADMIN_TOKENS="nomad"
|
||||
|
||||
# Test with token
|
||||
curl -H "Authorization: Bearer nomad" \
|
||||
$BACKEND_URL/api/health
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Troubleshooting
|
||||
|
||||
### "ModuleNotFoundError" when validating
|
||||
|
||||
**Make sure venv is activated:**
|
||||
```bash
|
||||
source backend/venv/bin/activate
|
||||
```
|
||||
|
||||
### Still getting Python 3.13 errors
|
||||
|
||||
**Recreate venv with Python 3.12:**
|
||||
```bash
|
||||
rm -rf backend/venv
|
||||
python3.12 -m venv backend/venv
|
||||
source backend/venv/bin/activate
|
||||
pip install -r backend/requirements.txt
|
||||
pip install -e . -e lyrics_transcriber_local/
|
||||
```
|
||||
|
||||
### Validation passes but deployment fails
|
||||
|
||||
**Check Docker caching:**
|
||||
```bash
|
||||
# Build should be fast (~2 minutes)
|
||||
# If slow, check cloudbuild.yaml
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Summary
|
||||
|
||||
| Item | Status |
|
||||
|------|--------|
|
||||
| **Python 3.12 venv** | ✅ Set up |
|
||||
| **Relaxed dependencies** | ✅ Updated |
|
||||
| **All modules importing** | ✅ Working |
|
||||
| **Validation passing** | ✅ All checks |
|
||||
| **Deploy script updated** | ✅ Auto-validates |
|
||||
| **Backend deployed** | ✅ 2m 24s build |
|
||||
| **Ready to test** | ✅ Yes! |
|
||||
|
||||
---
|
||||
|
||||
**Key Takeaway:** With venv + validation, the import error would have been caught instantly! 🎯
|
||||
|
||||
@@ -0,0 +1,249 @@
|
||||
# Why Cloud Build Didn't Deploy Automatically
|
||||
|
||||
**Issue:** Cloud Build successfully built the Docker image but didn't deploy it to Cloud Run
|
||||
|
||||
---
|
||||
|
||||
## Root Cause
|
||||
|
||||
The `cloudbuild.yaml` was configured to **only build and push Docker images**, not deploy them.
|
||||
|
||||
### Original cloudbuild.yaml
|
||||
|
||||
```yaml
|
||||
steps:
|
||||
# Pull the previous image for caching
|
||||
- name: 'gcr.io/cloud-builders/docker'
|
||||
# ... pull latest ...
|
||||
|
||||
# Build with cache
|
||||
- name: 'gcr.io/cloud-builders/docker'
|
||||
args:
|
||||
- 'build'
|
||||
- '--cache-from'
|
||||
- 'us-central1-docker.pkg.dev/$PROJECT_ID/karaoke-repo/karaoke-backend:latest'
|
||||
# ...
|
||||
|
||||
images:
|
||||
- 'us-central1-docker.pkg.dev/$PROJECT_ID/karaoke-repo/karaoke-backend:latest'
|
||||
- 'us-central1-docker.pkg.dev/$PROJECT_ID/karaoke-repo/karaoke-backend:$BUILD_ID'
|
||||
```
|
||||
|
||||
**What it does:**
|
||||
1. ✅ Pull previous image for caching
|
||||
2. ✅ Build new image
|
||||
3. ✅ Push to Artifact Registry with `:latest` and `:$BUILD_ID` tags
|
||||
4. ❌ **Does NOT deploy to Cloud Run**
|
||||
|
||||
---
|
||||
|
||||
## The Fix
|
||||
|
||||
Added a Cloud Run deployment step to `cloudbuild.yaml`:
|
||||
|
||||
```yaml
|
||||
steps:
|
||||
# ... build steps ...
|
||||
|
||||
# Deploy to Cloud Run (NEW!)
|
||||
- name: 'gcr.io/google.com/cloudsdktool/cloud-sdk'
|
||||
entrypoint: gcloud
|
||||
args:
|
||||
- 'run'
|
||||
- 'deploy'
|
||||
- 'karaoke-backend'
|
||||
- '--image'
|
||||
- 'us-central1-docker.pkg.dev/$PROJECT_ID/karaoke-repo/karaoke-backend:$BUILD_ID'
|
||||
- '--region'
|
||||
- 'us-central1'
|
||||
- '--platform'
|
||||
- 'managed'
|
||||
|
||||
images:
|
||||
- 'us-central1-docker.pkg.dev/$PROJECT_ID/karaoke-repo/karaoke-backend:latest'
|
||||
- 'us-central1-docker.pkg.dev/$PROJECT_ID/karaoke-repo/karaoke-backend:$BUILD_ID'
|
||||
```
|
||||
|
||||
**What it now does:**
|
||||
1. ✅ Pull previous image for caching
|
||||
2. ✅ Build new image
|
||||
3. ✅ Push to Artifact Registry
|
||||
4. ✅ **Deploy to Cloud Run automatically**
|
||||
|
||||
---
|
||||
|
||||
## Why Didn't Cloud Run Auto-Update?
|
||||
|
||||
Cloud Run doesn't automatically redeploy when a new `:latest` tag is pushed. You need to:
|
||||
|
||||
1. **Explicitly deploy** with `gcloud run deploy`, OR
|
||||
2. **Add a deployment step** to Cloud Build (what we did)
|
||||
|
||||
The `:latest` tag is just a Docker tag - Cloud Run doesn't watch for changes.
|
||||
|
||||
---
|
||||
|
||||
## Timeline of Events
|
||||
|
||||
### Build 1 (6:27 AM) - 01bf8976
|
||||
- ✅ Built image with fixed import error
|
||||
- ✅ Pushed to Artifact Registry
|
||||
- ❌ Didn't deploy to Cloud Run
|
||||
- **Result:** Service still broken (revision 00003)
|
||||
|
||||
### Build 2 (6:41 AM) - 816a70fa
|
||||
- ✅ Built image (still using old cloudbuild.yaml)
|
||||
- ✅ Pushed to Artifact Registry
|
||||
- ❌ Didn't deploy (no deployment step yet)
|
||||
- **Manual deployment:** `gcloud run deploy ...`
|
||||
- ✅ **Result:** Service working! (revision 00004)
|
||||
|
||||
### Future Builds
|
||||
- ✅ Will build AND deploy automatically
|
||||
- ✅ Uses updated cloudbuild.yaml with deployment step
|
||||
|
||||
---
|
||||
|
||||
## How We Deployed Manually
|
||||
|
||||
```bash
|
||||
# Get the build ID from the latest successful build
|
||||
BUILD_ID="816a70fa-c2e1-47d9-91c1-ca639d4d5eba"
|
||||
|
||||
# Deploy that specific image
|
||||
gcloud run deploy karaoke-backend \
|
||||
--image us-central1-docker.pkg.dev/nomadkaraoke/karaoke-repo/karaoke-backend:$BUILD_ID \
|
||||
--region us-central1 \
|
||||
--platform managed
|
||||
```
|
||||
|
||||
**Result:**
|
||||
```
|
||||
Service [karaoke-backend] revision [karaoke-backend-00004-rrn] has been deployed
|
||||
and is serving 100 percent of traffic.
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Verification
|
||||
|
||||
### Health Check ✅
|
||||
|
||||
```bash
|
||||
$ curl -H "Authorization: Bearer $AUTH_TOKEN" \
|
||||
https://karaoke-backend-ipzqd2k4yq-uc.a.run.app/api/health
|
||||
|
||||
{
|
||||
"status": "healthy",
|
||||
"service": "karaoke-gen-backend"
|
||||
}
|
||||
```
|
||||
|
||||
### Revision Status ✅
|
||||
|
||||
```bash
|
||||
$ gcloud run revisions list --service karaoke-backend --region us-central1
|
||||
|
||||
NAME STATUS TRAFFIC
|
||||
karaoke-backend-00004-rrn True 100% ← NEW! Working!
|
||||
karaoke-backend-00003-55h False 0% ← Old broken revision
|
||||
karaoke-backend-00002-sk9 True 0% ← Old working revision
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Lesson Learned
|
||||
|
||||
**Cloud Build has two distinct responsibilities:**
|
||||
|
||||
1. **Build:** Compile code, create Docker image, push to registry
|
||||
2. **Deploy:** Update Cloud Run service to use new image
|
||||
|
||||
**By default, Cloud Build only does #1.**
|
||||
|
||||
You must explicitly add a deployment step for #2.
|
||||
|
||||
---
|
||||
|
||||
## Alternative Approaches
|
||||
|
||||
### Option 1: Separate Build and Deploy (Original)
|
||||
|
||||
```bash
|
||||
# Build
|
||||
gcloud builds submit --config=cloudbuild.yaml
|
||||
|
||||
# Deploy manually
|
||||
gcloud run deploy karaoke-backend \
|
||||
--image us-central1-docker.pkg.dev/nomadkaraoke/karaoke-repo/karaoke-backend:latest \
|
||||
--region us-central1
|
||||
```
|
||||
|
||||
**Pros:** Separate concerns, more control
|
||||
**Cons:** Requires two steps, easy to forget
|
||||
|
||||
### Option 2: Combined Build + Deploy (New)
|
||||
|
||||
```bash
|
||||
# Build AND deploy in one command
|
||||
gcloud builds submit --config=cloudbuild.yaml
|
||||
```
|
||||
|
||||
**Pros:** One command, automatic deployment
|
||||
**Cons:** Longer build time, deploys every time
|
||||
|
||||
### Option 3: Pulumi Managed
|
||||
|
||||
```bash
|
||||
# Update infrastructure
|
||||
pulumi up
|
||||
```
|
||||
|
||||
**Pros:** Infrastructure as code, versioned
|
||||
**Cons:** Need to update Pulumi config for each deployment
|
||||
|
||||
---
|
||||
|
||||
## Current Setup
|
||||
|
||||
**We're using Option 2:** Combined build + deploy via Cloud Build
|
||||
|
||||
### Workflow
|
||||
|
||||
```bash
|
||||
# 1. Make code changes
|
||||
vim backend/api/routes/jobs.py
|
||||
|
||||
# 2. Validate locally (catches errors!)
|
||||
source backend/venv/bin/activate
|
||||
python3 backend/validate.py
|
||||
|
||||
# 3. Deploy (builds + deploys automatically)
|
||||
./scripts/deploy.sh
|
||||
# OR
|
||||
gcloud builds submit --config=cloudbuild.yaml
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Next Build Will Deploy Automatically! ✅
|
||||
|
||||
From now on, `gcloud builds submit` will:
|
||||
1. Build the Docker image
|
||||
2. Push to Artifact Registry
|
||||
3. **Deploy to Cloud Run automatically**
|
||||
|
||||
No more manual deployment needed! 🎉
|
||||
|
||||
---
|
||||
|
||||
## Summary
|
||||
|
||||
| Build | Image Built? | Deployed? | Why? |
|
||||
|-------|--------------|-----------|------|
|
||||
| 01bf8976 | ✅ Yes | ❌ No | No deployment step in cloudbuild.yaml |
|
||||
| 816a70fa | ✅ Yes | ✅ Manual | We deployed manually after build |
|
||||
| Future | ✅ Yes | ✅ Automatic | Updated cloudbuild.yaml with deploy step |
|
||||
|
||||
**Status:** Fixed! ✅
|
||||
|
||||
Executable
+80
@@ -0,0 +1,80 @@
|
||||
#!/bin/bash
|
||||
# Quick deploy script with validation
|
||||
set -e
|
||||
|
||||
echo "================================"
|
||||
echo "Backend Deployment Script"
|
||||
echo "================================"
|
||||
|
||||
# Colors
|
||||
RED='\033[0;31m'
|
||||
GREEN='\033[0;32m'
|
||||
YELLOW='\033[1;33m'
|
||||
NC='\033[0m' # No Color
|
||||
|
||||
# Change to project root
|
||||
cd "$(dirname "$0")/.."
|
||||
|
||||
echo ""
|
||||
echo "Step 1: Validating backend code..."
|
||||
echo "-----------------------------------"
|
||||
|
||||
# Check if venv is available and activated
|
||||
if [ -d "backend/venv" ] && [ -z "$VIRTUAL_ENV" ]; then
|
||||
echo -e "${YELLOW}💡 Virtual environment exists but not activated.${NC}"
|
||||
echo " Activate it for full validation: source backend/venv/bin/activate"
|
||||
echo " Falling back to quick check..."
|
||||
echo ""
|
||||
fi
|
||||
|
||||
# If venv is activated, use full validation (catches import errors!)
|
||||
if [ -n "$VIRTUAL_ENV" ]; then
|
||||
echo "Running full validation (venv activated)..."
|
||||
if ! python3 backend/validate.py; then
|
||||
echo ""
|
||||
echo -e "${RED}❌ Validation failed! Fix errors before deploying.${NC}"
|
||||
exit 1
|
||||
fi
|
||||
else
|
||||
# Otherwise use quick check
|
||||
echo "Running quick validation (activate venv for full validation)..."
|
||||
if ! ./backend/quick-check.sh; then
|
||||
echo ""
|
||||
echo -e "${RED}❌ Validation failed! Fix errors before deploying.${NC}"
|
||||
exit 1
|
||||
fi
|
||||
fi
|
||||
|
||||
echo ""
|
||||
echo -e "${GREEN}✅ Validation passed!${NC}"
|
||||
echo ""
|
||||
|
||||
# Ask for confirmation
|
||||
read -p "Deploy to Cloud Run? (y/N) " -n 1 -r
|
||||
echo
|
||||
if [[ ! $REPLY =~ ^[Yy]$ ]]; then
|
||||
echo "Deployment cancelled."
|
||||
exit 0
|
||||
fi
|
||||
|
||||
echo ""
|
||||
echo "Step 2: Building and deploying..."
|
||||
echo "-----------------------------------"
|
||||
|
||||
# Submit build
|
||||
gcloud builds submit --config=cloudbuild.yaml --timeout=20m
|
||||
|
||||
echo ""
|
||||
echo -e "${GREEN}🎉 Deployment complete!${NC}"
|
||||
echo ""
|
||||
echo "Check status:"
|
||||
echo " gcloud run services describe karaoke-backend --region us-central1"
|
||||
echo ""
|
||||
echo "View logs:"
|
||||
echo " gcloud logging read \"resource.type=cloud_run_revision AND resource.labels.service_name=karaoke-backend\" --limit=50"
|
||||
echo ""
|
||||
echo "Test endpoint:"
|
||||
echo " export BACKEND_URL=\"https://karaoke-backend-ipzqd2k4yq-uc.a.run.app\""
|
||||
echo " export AUTH_TOKEN=\$(gcloud auth print-identity-token)"
|
||||
echo " curl -H \"Authorization: Bearer \$AUTH_TOKEN\" \$BACKEND_URL/api/health"
|
||||
|
||||
Reference in new issue
Block a user