fix: use session with auth header for file downloads

The download_file_via_url method was using requests.get() directly
without the Authorization header, causing all downloads to fail with
401/403 errors even though the job completed successfully.

Now uses self.session.get() which includes the auth token from headers.

Bump version to 0.75.51
This commit is contained in:
Andrew Beveridge committed 2025-12-25 11:54:26 -05:00
1 parent 5c29fabd0f
commit 81dca5c673
7 files changed
+707 -2

No files matched your search

+171
View File
@@ -0,0 +1,171 @@
# CLI + Backend Integration Testing
## The Problem
We had multiple bugs that unit tests didn't catch:
- Content-type mismatch in signed URL uploads (403 errors)
- Style file path handling issues
These bugs required:
1. Full CI/CD deploy (~12 min)
2. Full karaoke generation (~10 min)
3. Manual inspection of logs
**Total: 20+ minutes per iteration** 😱
## The Solution
New **CLI + Backend Integration Tests** that:
- Run the ACTUAL CLI code against a LOCAL backend
- Use GCP emulators (Firestore + GCS)
- Test the full signed URL upload flow
- Run in **< 30 seconds**
## Test Architecture
```
┌─────────────────────────────────────┐
│ Unit Tests (Mocked) │ ← Fast, isolated
│ tests/unit/ │ ~0.5s, 100+ tests
├─────────────────────────────────────┤
│ Backend Emulator Tests │ ← Backend-only integration
│ backend/tests/emulator/ │ ~2s, 11 tests
├─────────────────────────────────────┤
│ CLI + Backend Integration Tests │ ← NEW! Full flow
│ tests/integration/ │ ~20s, catches real bugs
│ test_cli_backend_integration.py │
├─────────────────────────────────────┤
│ Cloud Integration Tests │ ← Against deployed backend
│ backend/tests/test_api_integration│ ~10min, pre-prod only
└─────────────────────────────────────┘
```
## Quick Start
### Run All Integration Tests
```bash
./scripts/run-cli-integration-tests.sh
```
This automatically:
1. Starts GCP emulators
2. Starts local backend
3. Runs tests
4. Cleans up
### Interactive Development Mode
```bash
# Terminal 1: Start environment
./scripts/start-local-test-env.sh
# Terminal 2: Run specific tests repeatedly
pytest tests/integration/test_cli_backend_integration.py -v -k "test_style_params"
```
## What These Tests Catch
### ✅ Content-Type Mismatches
```python
def test_style_params_json_upload(self, test_style_files, cli_config):
"""Test uploading style_params.json via signed URL."""
# This would have caught the Path('.json').suffix bug!
result = client.search_audio(
artist="Test Artist",
title="Test Song",
style_params_path=test_style_files['style_params_path'],
)
# If we get 403, the test fails with a clear message
```
### ✅ Request/Response Format Mismatches
Tests verify CLI and backend agree on API contract.
### ✅ GCS Path Issues
Uploads go to real (emulated) GCS, catches path/encoding bugs.
### ✅ Auth Token Handling
Tests run with actual token flow.
## When to Run
| Situation | Run This |
|-----------|----------|
| Quick check before commit | `./scripts/run-emulator-tests.sh` |
| Testing CLI changes | `./scripts/run-cli-integration-tests.sh` |
| Testing backend API changes | `./scripts/run-cli-integration-tests.sh` |
| Before creating PR | Both of the above |
| Pre-production | Cloud integration tests |
## Test Files
### `tests/integration/test_cli_backend_integration.py`
Key test classes:
- `TestSignedUrlUploadRoundTrip` - Full upload flow
- `TestContentTypeMatching` - Catches the specific bug
- `TestAudioSearchWithStyles` - API contract tests
### Test Fixtures
- `test_style_files` - Creates minimal valid PNG/TTF/JSON files
- `cli_config` - Points CLI at local backend
## Extending Tests
### Adding a New Integration Test
```python
class TestMyNewFeature:
def test_my_feature(self, cli_config, tmp_path):
from karaoke_gen.utils.remote_cli import RemoteKaraokeClient, RemoteConfig
import logging
config = RemoteConfig(
service_url=cli_config['service_url'],
auth_token=cli_config['auth_token'],
environment='test'
)
client = RemoteKaraokeClient(config, logging.getLogger("test"))
# Test your feature
result = client.some_method(...)
assert result['expected_field'] == expected_value
```
## Prerequisites
1. **Docker** (for fake-gcs-server)
2. **gcloud CLI** with Firestore emulator:
```bash
gcloud components install cloud-firestore-emulator
```
## Troubleshooting
### "Emulators not running"
```bash
./scripts/start-emulators.sh
```
### "Backend failed to start"
Check if port 8000 is already in use:
```bash
lsof -i :8000
```
### Tests pass locally but fail in CI
The integration tests require Docker, which may not be available in all CI environments. Consider:
- Running in a CI job with Docker support
- Skipping in CI and running locally before merge
## Benefits
| Before | After |
|--------|-------|
| 20+ min per iteration | < 30 sec per iteration |
| Manual log inspection | Automated assertions |
| Bugs found in production | Bugs caught locally |
| Expensive cloud resources | Free local emulators |
Submodule
+1
Submodule flacfetch added at 04f633cce9.
+2 -1
View File
@@ -1095,7 +1095,8 @@ class RemoteKaraokeClient:
if url.startswith('/'):
url = f"{self.config.service_url}{url}"
response = requests.get(url, stream=True, timeout=600)
# Use session headers (includes Authorization) for authenticated downloads
response = self.session.get(url, stream=True, timeout=600)
if response.status_code != 200:
return False
+1 -1
View File
@@ -1,6 +1,6 @@
[tool.poetry]
name = "karaoke-gen"
version = "0.75.50"
version = "0.75.51"
description = "Generate karaoke videos with synchronized lyrics. Handles the entire process from downloading audio and lyrics to creating the final video with title screens."
authors = ["Andrew Beveridge <andrew@beveridge.uk>"]
license = "MIT"
+111
View File
@@ -0,0 +1,111 @@
#!/bin/bash
# Run CLI + Backend integration tests
#
# This script:
# 1. Starts emulators
# 2. Starts local backend
# 3. Runs integration tests
# 4. Cleans up
#
# Usage:
# ./scripts/run-cli-integration-tests.sh
set -e
SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
PROJECT_ROOT="$(cd "$SCRIPT_DIR/.." && pwd)"
echo "🧪 Running CLI + Backend Integration Tests"
echo "==========================================="
echo ""
# Cleanup function
cleanup() {
echo ""
echo "Cleaning up..."
# Kill backend if running
if [ ! -z "$BACKEND_PID" ]; then
kill $BACKEND_PID 2>/dev/null || true
echo " Stopped backend (PID $BACKEND_PID)"
fi
# Stop emulators
"$SCRIPT_DIR/stop-emulators.sh" 2>/dev/null || true
}
trap cleanup EXIT
# Step 1: Start emulators
echo "Step 1/4: Starting GCP emulators..."
"$SCRIPT_DIR/start-emulators.sh"
echo ""
# Step 2: Set environment
echo "Step 2/4: Setting environment..."
export FIRESTORE_EMULATOR_HOST="127.0.0.1:8080"
export STORAGE_EMULATOR_HOST="http://127.0.0.1:4443"
export GOOGLE_CLOUD_PROJECT="test-project"
export GCS_BUCKET_NAME="test-bucket"
export FIRESTORE_COLLECTION="jobs"
export ENVIRONMENT="test"
export ADMIN_TOKENS="test-admin-token"
# Disable external services for tests
export FLACFETCH_API_URL=""
export AUDIO_SEPARATOR_API_URL=""
export AUDIOSHAKE_API_TOKEN=""
export GENIUS_API_TOKEN=""
export SPOTIFY_COOKIE=""
echo " Environment configured for local testing"
echo ""
# Step 3: Start backend
echo "Step 3/4: Starting local backend..."
cd "$PROJECT_ROOT"
if command -v poetry &> /dev/null; then
poetry run uvicorn backend.main:app --host 127.0.0.1 --port 8000 &
else
python -m uvicorn backend.main:app --host 127.0.0.1 --port 8000 &
fi
BACKEND_PID=$!
# Wait for backend
echo " Waiting for backend..."
for i in {1..30}; do
if curl -s http://localhost:8000/api/health > /dev/null 2>&1; then
echo " Backend ready!"
break
fi
if [ $i -eq 30 ]; then
echo " ❌ Backend failed to start"
exit 1
fi
sleep 1
done
echo ""
# Step 4: Run tests
echo "Step 4/4: Running integration tests..."
echo ""
cd "$PROJECT_ROOT"
if command -v poetry &> /dev/null; then
PYTEST_CMD="poetry run pytest"
else
PYTEST_CMD="pytest"
fi
if $PYTEST_CMD tests/integration/test_cli_backend_integration.py -v --tb=short; then
echo ""
echo "✅ All CLI + Backend integration tests passed!"
exit 0
else
echo ""
echo "❌ Some integration tests failed"
exit 1
fi
+101
View File
@@ -0,0 +1,101 @@
#!/bin/bash
# Start local test environment with emulators and backend
#
# This sets up everything needed to run CLI+Backend integration tests locally.
# Much faster than deploying to cloud for testing!
#
# Usage:
# ./scripts/start-local-test-env.sh
# # In another terminal:
# pytest tests/integration/test_cli_backend_integration.py -v
set -e
SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
PROJECT_ROOT="$(cd "$SCRIPT_DIR/.." && pwd)"
echo "🚀 Starting local test environment"
echo "=================================="
echo ""
# Step 1: Start emulators
echo "Step 1/2: Starting GCP emulators..."
"$SCRIPT_DIR/start-emulators.sh"
echo ""
# Set environment variables for local backend
export FIRESTORE_EMULATOR_HOST="127.0.0.1:8080"
export STORAGE_EMULATOR_HOST="http://127.0.0.1:4443"
export GOOGLE_CLOUD_PROJECT="test-project"
export GCS_BUCKET_NAME="test-bucket"
export FIRESTORE_COLLECTION="jobs"
export ENVIRONMENT="test"
export ADMIN_TOKENS="test-admin-token"
# Also export for any external services to be mocked/disabled
export FLACFETCH_API_URL=""
export AUDIO_SEPARATOR_API_URL=""
export AUDIOSHAKE_API_TOKEN=""
export GENIUS_API_TOKEN=""
export SPOTIFY_COOKIE=""
echo "Step 2/2: Starting local backend..."
echo ""
echo "Environment:"
echo " FIRESTORE_EMULATOR_HOST=$FIRESTORE_EMULATOR_HOST"
echo " STORAGE_EMULATOR_HOST=$STORAGE_EMULATOR_HOST"
echo " GOOGLE_CLOUD_PROJECT=$GOOGLE_CLOUD_PROJECT"
echo " ADMIN_TOKENS=test-admin-token"
echo ""
cd "$PROJECT_ROOT"
# Start backend in background
if command -v poetry &> /dev/null; then
echo "Starting backend with poetry..."
poetry run uvicorn backend.main:app --host 0.0.0.0 --port 8000 --reload &
else
echo "Starting backend with Python..."
python -m uvicorn backend.main:app --host 0.0.0.0 --port 8000 --reload &
fi
BACKEND_PID=$!
echo "Backend PID: $BACKEND_PID"
# Wait for backend to be ready
echo ""
echo "Waiting for backend to be ready..."
for i in {1..30}; do
if curl -s http://localhost:8000/api/health > /dev/null 2>&1; then
echo "✅ Backend is ready!"
break
fi
if [ $i -eq 30 ]; then
echo "❌ Backend failed to start"
kill $BACKEND_PID 2>/dev/null || true
"$SCRIPT_DIR/stop-emulators.sh"
exit 1
fi
sleep 1
done
echo ""
echo "=============================================="
echo "✅ Local test environment is running!"
echo ""
echo "Run integration tests with:"
echo " pytest tests/integration/test_cli_backend_integration.py -v"
echo ""
echo "Or run all CLI+Backend tests:"
echo " ./scripts/run-cli-integration-tests.sh"
echo ""
echo "To stop:"
echo " kill $BACKEND_PID"
echo " ./scripts/stop-emulators.sh"
echo ""
echo "Backend PID: $BACKEND_PID"
echo "=============================================="
# Keep script running (for foreground mode)
wait $BACKEND_PID
@@ -0,0 +1,320 @@
"""
CLI + Backend Integration Tests
These tests run the ACTUAL CLI code against a LOCAL backend with emulators.
This catches bugs like content-type mismatches that unit tests with mocks miss.
Run with:
./scripts/run-cli-integration-tests.sh
Prerequisites:
- GCS emulator running (fake-gcs-server)
- Firestore emulator running
- Backend running locally against emulators
"""
import os
import sys
import json
import pytest
import tempfile
import requests
from pathlib import Path
from unittest.mock import patch
# Add project root to path
PROJECT_ROOT = Path(__file__).parent.parent.parent
sys.path.insert(0, str(PROJECT_ROOT))
def emulators_and_backend_running() -> bool:
"""Check if emulators and local backend are running."""
try:
# Check GCS emulator
gcs_response = requests.get("http://localhost:4443/storage/v1/b", timeout=2)
if gcs_response.status_code != 200:
return False
# Check Firestore emulator
firestore_response = requests.get("http://localhost:8080", timeout=2)
# Check local backend
backend_response = requests.get("http://localhost:8000/api/health", timeout=2)
if backend_response.status_code != 200:
return False
return True
except requests.exceptions.RequestException:
return False
# Skip all tests if infrastructure isn't running
pytestmark = pytest.mark.skipif(
not emulators_and_backend_running(),
reason="Emulators and local backend not running. Run: ./scripts/start-local-test-env.sh"
)
@pytest.fixture
def test_style_files(tmp_path):
"""Create test style files for upload testing."""
# Create a minimal style params JSON
style_params = {
"intro": {
"background_color": "#000000",
"background_image": str(tmp_path / "intro_bg.png"),
"font": str(tmp_path / "test_font.ttf"),
"title_color": "#ffffff",
"artist_color": "#ffdf6b"
},
"karaoke": {
"background_color": "#000000",
"background_image": str(tmp_path / "karaoke_bg.png"),
"font_path": str(tmp_path / "test_font.ttf"),
"primary_color": "112, 112, 247, 255",
"secondary_color": "255, 255, 255, 255"
},
"cdg": {
"background_color": "#000000",
"instrumental_background": str(tmp_path / "cdg_inst_bg.png"),
"title_screen_background": str(tmp_path / "cdg_title_bg.png"),
"font_path": str(tmp_path / "test_font.ttf")
}
}
style_params_path = tmp_path / "styles.json"
with open(style_params_path, 'w') as f:
json.dump(style_params, f)
# Create minimal test image (1x1 pixel PNG)
# PNG header + minimal IHDR + IDAT + IEND
minimal_png = bytes([
0x89, 0x50, 0x4E, 0x47, 0x0D, 0x0A, 0x1A, 0x0A, # PNG signature
0x00, 0x00, 0x00, 0x0D, 0x49, 0x48, 0x44, 0x52, # IHDR chunk
0x00, 0x00, 0x00, 0x01, 0x00, 0x00, 0x00, 0x01, # 1x1
0x08, 0x02, 0x00, 0x00, 0x00, 0x90, 0x77, 0x53,
0xDE, 0x00, 0x00, 0x00, 0x0C, 0x49, 0x44, 0x41, # IDAT chunk
0x54, 0x08, 0xD7, 0x63, 0xF8, 0xFF, 0xFF, 0x3F,
0x00, 0x05, 0xFE, 0x02, 0xFE, 0xDC, 0xCC, 0x59,
0xE7, 0x00, 0x00, 0x00, 0x00, 0x49, 0x45, 0x4E, # IEND chunk
0x44, 0xAE, 0x42, 0x60, 0x82
])
# Create test images
for name in ["intro_bg.png", "karaoke_bg.png", "cdg_inst_bg.png", "cdg_title_bg.png"]:
(tmp_path / name).write_bytes(minimal_png)
# Create minimal TTF font (just header)
minimal_ttf = bytes([0x00, 0x01, 0x00, 0x00] + [0x00] * 100)
(tmp_path / "test_font.ttf").write_bytes(minimal_ttf)
return {
'style_params_path': str(style_params_path),
'tmp_path': tmp_path
}
@pytest.fixture
def cli_config():
"""Configuration for CLI to connect to local backend."""
return {
'service_url': 'http://localhost:8000',
'auth_token': 'test-admin-token',
}
class TestSignedUrlUploadRoundTrip:
"""
Test the full signed URL upload flow:
1. CLI requests signed URLs from backend
2. CLI uploads files directly to GCS using signed URLs
3. Backend can read the uploaded files
This is the flow that broke with the content-type mismatch bug.
"""
def test_style_params_json_upload(self, test_style_files, cli_config):
"""Test uploading style_params.json via signed URL."""
from karaoke_gen.utils.remote_cli import RemoteKaraokeClient, RemoteConfig
import logging
logger = logging.getLogger("test")
config = RemoteConfig(
service_url=cli_config['service_url'],
auth_token=cli_config['auth_token'],
environment='test'
)
client = RemoteKaraokeClient(config, logger)
# This should:
# 1. Send request with style_files metadata
# 2. Get back signed URLs
# 3. Upload files to those URLs
# 4. Return job info
try:
result = client.search_audio(
artist="Test Artist",
title="Test Song",
style_params_path=test_style_files['style_params_path'],
auto_download=False
)
# If we get here without 403, the upload worked!
assert 'job_id' in result
except Exception as e:
# Check if it's a content-type mismatch (403)
if '403' in str(e):
pytest.fail(f"Signed URL upload failed with 403 - likely content-type mismatch: {e}")
raise
def test_image_file_upload(self, test_style_files, cli_config):
"""Test uploading PNG image via signed URL."""
from karaoke_gen.utils.remote_cli import RemoteKaraokeClient, RemoteConfig
import logging
logger = logging.getLogger("test")
config = RemoteConfig(
service_url=cli_config['service_url'],
auth_token=cli_config['auth_token'],
environment='test'
)
client = RemoteKaraokeClient(config, logger)
try:
result = client.search_audio(
artist="Test Artist",
title="Test Song",
style_params_path=test_style_files['style_params_path'],
auto_download=False
)
# Verify job was created
assert 'job_id' in result
# Verify style assets were recorded
job_id = result['job_id']
job_response = requests.get(
f"{cli_config['service_url']}/api/jobs/{job_id}",
headers={'Authorization': f"Bearer {cli_config['auth_token']}"}
)
assert job_response.status_code == 200
job = job_response.json()
# Check style assets exist
assert 'style_assets' in job or job.get('style_params_gcs_path')
except Exception as e:
if '403' in str(e):
pytest.fail(f"Image upload failed with 403 - content-type mismatch: {e}")
raise
def test_font_file_upload(self, test_style_files, cli_config):
"""Test uploading TTF font via signed URL."""
from karaoke_gen.utils.remote_cli import RemoteKaraokeClient, RemoteConfig
import logging
logger = logging.getLogger("test")
config = RemoteConfig(
service_url=cli_config['service_url'],
auth_token=cli_config['auth_token'],
environment='test'
)
client = RemoteKaraokeClient(config, logger)
try:
result = client.search_audio(
artist="Test Artist",
title="Test Song",
style_params_path=test_style_files['style_params_path'],
auto_download=False
)
assert 'job_id' in result
except Exception as e:
if '403' in str(e):
pytest.fail(f"Font upload failed with 403 - content-type mismatch: {e}")
raise
class TestContentTypeMatching:
"""
Direct tests for content-type handling.
These catch the specific bug we found.
"""
def test_get_content_type_with_full_path(self):
"""Test _get_content_type works with full paths."""
from karaoke_gen.utils.remote_cli import RemoteKaraokeClient, RemoteConfig
import logging
logger = logging.getLogger("test")
config = RemoteConfig(
service_url='http://localhost:8000',
auth_token='test',
environment='test'
)
client = RemoteKaraokeClient(config, logger)
# Test various file paths
assert client._get_content_type('/path/to/file.json') == 'application/json'
assert client._get_content_type('/path/to/file.png') == 'image/png'
assert client._get_content_type('/path/to/file.ttf') == 'font/ttf'
assert client._get_content_type('/path/to/file.flac') == 'audio/flac'
def test_get_content_type_with_extension_only_bug(self):
"""
Test that passing just an extension doesn't work.
This documents the bug we found.
"""
from pathlib import Path
# This is the bug: Path('.json').suffix returns empty string!
assert Path('.json').suffix == ''
assert Path('.png').suffix == ''
# But full paths work correctly
assert Path('/path/to/file.json').suffix == '.json'
assert Path('/path/to/file.png').suffix == '.png'
class TestAudioSearchWithStyles:
"""Test audio search endpoint with style file handling."""
def test_audio_search_returns_style_upload_urls(self, cli_config, tmp_path):
"""Test that audio search returns signed URLs for style files."""
# Create minimal style params
style_params = {"karaoke": {"background_color": "#000000"}}
style_path = tmp_path / "styles.json"
with open(style_path, 'w') as f:
json.dump(style_params, f)
# Make direct API call to check response format
response = requests.post(
f"{cli_config['service_url']}/api/audio-search/search",
headers={'Authorization': f"Bearer {cli_config['auth_token']}"},
json={
'artist': 'Test Artist',
'title': 'Test Song',
'auto_download': False,
'style_files': [
{
'filename': 'styles.json',
'content_type': 'application/json',
'file_type': 'style_params'
}
]
}
)
# Should get 200 or 404 (no results), not 500
assert response.status_code in [200, 404], f"Unexpected status: {response.status_code} - {response.text}"
if response.status_code == 200:
data = response.json()
# If style files were provided, should get upload URLs back
if 'style_upload_urls' in data:
assert len(data['style_upload_urls']) > 0
for url_info in data['style_upload_urls']:
assert 'upload_url' in url_info
assert 'file_type' in url_info