Compare commits

..
Author SHA1 Message Date
Andrew Beveridge 3f483f67cc Updates 2025-12-24 22:47:16 -05:00
871 changed files with 225 additions and 365739 deletions

No files matched your search

-74
View File
@@ -1,74 +0,0 @@
---
description: Periodic documentation maintenance and organization check
allowed-tools: Read, Glob, Grep, Edit, Write, Bash
---
# Documentation Maintenance Check
You are performing periodic maintenance on the project documentation.
## Checks to Perform
### 1. Structure Compliance
Verify docs/ follows the correct structure:
```
docs/
├── README.md # Current status + navigation
├── ARCHITECTURE.md # System design
├── DEVELOPMENT.md # Dev setup, testing, deployment
├── API.md # Backend API reference
├── LESSONS-LEARNED.md # Accumulated wisdom
└── archive/ # Historical docs (YYYY-MM-DD-topic.md)
```
Report any files that don't belong or are misplaced.
### 2. Freshness Check
- Read `docs/README.md` - is the status section current?
- Check recent git commits - do they suggest docs should be updated?
- Look for stale information that contradicts current code
### 3. Archive Organization
- Verify all archive files have YYYY-MM-DD prefix
- Check for any docs that should be archived (completed work, old plans)
- Remove truly obsolete content (empty files, duplicate info)
### 4. Cross-Reference Check
- Verify CLAUDE.md points to current doc locations
- Check that docs reference each other correctly
- Ensure no broken internal links
### 5. Content Quality
- Flag any docs over 500 lines (may need splitting)
- Identify duplicate information across docs
- Note sections that seem outdated
## Actions to Take
1. **Fix structural issues** - Move/rename misplaced files
2. **Update stale content** - Refresh outdated information
3. **Archive completed work** - Move old docs to archive/
4. **Report findings** - Summarize what was found and fixed
## Output Format
Provide a maintenance report:
```
## Documentation Maintenance Report - YYYY-MM-DD
### Issues Found
- [list of issues]
### Actions Taken
- [list of fixes made]
### Recommendations
- [suggestions for future attention]
### Status
- [ ] Structure: OK/Fixed
- [ ] Freshness: OK/Updated
- [ ] Archive: OK/Organized
- [ ] Cross-refs: OK/Fixed
- [ ] Quality: OK/Flagged
```
-43
View File
@@ -1,43 +0,0 @@
---
description: Review current session for documentation updates before merging PR
allowed-tools: Read, Glob, Grep, Edit, Write
---
# Pre-Merge Documentation Review
You are reviewing this chat session to identify documentation updates needed before merging the current PR.
## Your Task
1. **Analyze the current session** - What work was done? What was learned? What changed?
2. **Check if docs need updates:**
- Does `docs/README.md` status section need updating?
- Should anything be added to `docs/LESSONS-LEARNED.md`?
- Does `docs/ARCHITECTURE.md` need changes (if architecture changed)?
- Does `docs/DEVELOPMENT.md` need changes (if dev workflow changed)?
- Does `docs/API.md` need changes (if API changed)?
3. **For significant completed work:**
- Create `docs/archive/YYYY-MM-DD-topic.md` summarizing the work
- Use today's date and a descriptive topic name
4. **Update docs/README.md** with:
- Current status if it changed
- Any new known issues discovered
- Link to new archive doc if created
## Guidelines
- Keep updates concise and factual
- Focus on information useful for future AI agents
- Don't duplicate information that's in the code
- Archive docs should capture "what was done and why" not implementation details
## Output
After reviewing, either:
- Make the necessary documentation updates
- Report "No documentation updates needed" with brief explanation why
Remember: Good documentation helps the next agent continue where you left off.
+21
View File
@@ -0,0 +1,21 @@
---
description: Git branching and PR workflow
alwaysApply: true
---
# Branching Workflow
**Never commit directly to `main`.** Always work on a feature branch.
Create descriptive branch names like `feature/add-discord-notifications`, `fix/audio-sync-issue`, or `refactor/cleanup-api-routes`.
# Creating a Pull Request
Once all changes are complete, tests pass, and commits are pushed, create a PR with:
1. **Summary** - Brief overview of the changes
2. **Changes Made** - List of specific modifications
3. **Testing** - How changes were tested (unit tests, manual testing)
4. **Related Issues** - Reference any GitHub issues (e.g., "Fixes #123")
Only merge after review and CI checks pass.
+31
View File
@@ -0,0 +1,31 @@
---
description: Infrastructure as Code requirements using Pulumi
alwaysApply: true
---
# Infrastructure Changes
All Google Cloud infrastructure (IAM, Cloud Storage, Cloud Run, Secret Manager, Artifact Registry, etc.) must be managed via **Pulumi IaC** in the `infrastructure/` folder.
## Rules
- **Reading/querying** with `gcloud` CLI is fine for investigation and debugging
- **Temporary changes** via `gcloud` CLI or Console are acceptable for urgent fixes or experimentation, but **must be cleaned up** and properly codified in the IaC afterward
- **Permanent changes** must go through Pulumi — never leave `gcloud` or Console changes as the final state
## Workflow
1. Make changes in `infrastructure/`
2. Run `pulumi preview` and verify the output
3. Run `pulumi up` to apply
## Authentication Issues
If you encounter any authentication issues with `gcloud` or `pulumi` (e.g., expired tokens, permission errors), **stop and notify the user** so they can run the appropriate auth commands in their external terminal:
```bash
gcloud auth login
gcloud auth application-default login
```
Do not attempt to work around auth issues or make assumptions about credentials.
+27
View File
@@ -0,0 +1,27 @@
---
description: Version bumping rule for commits
alwaysApply: true
---
# Version Management
When committing and pushing changes, always bump the patch version in `pyproject.toml`.
The version is located at `tool.poetry.version` in `pyproject.toml`. For example, if the current version is `0.70.3`, bump it to `0.70.4`.
This ensures every commit pushed to the repository has a unique version number for tracking and deployment purposes.
# Testing Requirements
Before committing and pushing changes, ensure all tests pass:
```bash
python -m pytest tests/ backend/tests/ -v -x --ignore=tests/integration
```
Ensure all code touched is thoroughly covered by unit tests, we have a 70% coverage minimum in CI:
```bash
poetry run pytest tests/unit/ -v --cov=karaoke_gen --cov-report=xml --cov-report=term-missing --cov-fail-under=70
```
Do not commit or push code that fails tests. Fix any test failures before proceeding.
-83
View File
@@ -1,83 +0,0 @@
# .dockerignore - Reduce Docker build context size
# This is critical for fast builds as it reduces upload time and cache invalidation
# Git
.git/
.gitignore
.gitmodules
# Node.js / Frontend (not needed for backend image)
node_modules/
frontend/
frontend-react/
*.log
npm-debug.log*
yarn-debug.log*
yarn-error.log*
# Python virtual environments (use Docker's Python instead)
__pycache__/
*.py[cod]
*$py.class
*.so
.Python
.venv/
venv/
backend/venv/
ENV/
env/
*.egg-info/
.eggs/
dist/
build/
*.egg
# IDE
.idea/
.vscode/
*.swp
*.swo
*~
.DS_Store
# Testing (not needed in production image)
.pytest_cache/
.coverage
htmlcov/
.tox/
.nox/
coverage.xml
*.cover
tests/
backend/tests/
# Documentation (not needed in Docker image)
docs/
*.md
!README.md
# Infrastructure (not needed for Docker build)
infrastructure/
# CI/CD configs
.github/
# Cloud Build configs (keep for reference but not in image)
cloudbuild*.yaml
# Local development files
lyrics_transcriber_local/
*.local
.env
.env.*
*.bak
*.tmp
Thumbs.db
# Large test/sample files
tests/data/
*.mp3
*.wav
*.mp4
*.mkv
*.flac
+146 -358
View File
@@ -17,120 +17,12 @@ concurrency:
jobs:
# ============================================
# PATH FILTERING - Determine what changed
# Only needed for PRs (main branch skips tests)
# ============================================
changes:
name: "Detect Changes"
runs-on: ubuntu-latest
if: github.ref != 'refs/heads/main'
outputs:
frontend: ${{ steps.filter.outputs.frontend }}
backend: ${{ steps.filter.outputs.backend }}
package: ${{ steps.filter.outputs.package }}
infra: ${{ steps.filter.outputs.infra }}
deps: ${{ steps.filter.outputs.deps }}
steps:
- uses: actions/checkout@v4
- uses: dorny/paths-filter@v3
id: filter
with:
filters: |
frontend:
- 'frontend/**'
backend:
- 'backend/**'
package:
- 'karaoke_gen/**'
- 'lyrics_transcriber_temp/**'
- 'tests/**'
infra:
- 'infrastructure/**'
deps:
- 'poetry.lock'
- name: Summary
run: |
echo "### Change Detection Results" >> $GITHUB_STEP_SUMMARY
echo "| Area | Changed |" >> $GITHUB_STEP_SUMMARY
echo "|------|---------|" >> $GITHUB_STEP_SUMMARY
echo "| Frontend | ${{ steps.filter.outputs.frontend }} |" >> $GITHUB_STEP_SUMMARY
echo "| Backend | ${{ steps.filter.outputs.backend }} |" >> $GITHUB_STEP_SUMMARY
echo "| Package | ${{ steps.filter.outputs.package }} |" >> $GITHUB_STEP_SUMMARY
echo "| Infrastructure | ${{ steps.filter.outputs.infra }} |" >> $GITHUB_STEP_SUMMARY
echo "| Dependencies | ${{ steps.filter.outputs.deps }} |" >> $GITHUB_STEP_SUMMARY
# ============================================
# FRONTEND TESTS (PRs only - main skips tests)
# ============================================
frontend-unit-tests:
name: "Frontend - Unit Tests"
runs-on: ubuntu-latest
needs: changes
if: github.ref != 'refs/heads/main' && needs.changes.outputs.frontend == 'true'
steps:
- uses: actions/checkout@v4
- name: Setup Node.js with caching
uses: actions/setup-node@v4
with:
node-version: '20'
cache: 'npm'
cache-dependency-path: frontend/package-lock.json
- name: Install dependencies
working-directory: frontend
run: npm ci --legacy-peer-deps
- name: Run unit tests with coverage
working-directory: frontend
run: npm run test:ci
- name: Upload coverage report
if: always()
uses: actions/upload-artifact@v4
with:
name: frontend-coverage
path: frontend/coverage/
retention-days: 7
frontend-build:
name: "Frontend - Build"
runs-on: ubuntu-latest
needs: changes
if: github.ref != 'refs/heads/main' && needs.changes.outputs.frontend == 'true'
steps:
- uses: actions/checkout@v4
- name: Setup Node.js with caching
uses: actions/setup-node@v4
with:
node-version: '20'
cache: 'npm'
cache-dependency-path: frontend/package-lock.json
- name: Install dependencies
working-directory: frontend
run: npm ci --legacy-peer-deps
- name: Build frontend
working-directory: frontend
env:
NEXT_PUBLIC_API_URL: https://api.nomadkaraoke.com
run: npm run build
# ============================================
# PACKAGE TESTS (PRs only - main skips tests)
# PACKAGE TESTS (karaoke-gen Python CLI)
# ============================================
package-unit-tests:
name: "Package - Unit Tests (70% coverage)"
runs-on: ubuntu-latest
needs: changes
if: github.ref != 'refs/heads/main' && (needs.changes.outputs.package == 'true' || needs.changes.outputs.deps == 'true')
steps:
- uses: actions/checkout@v4
with:
@@ -147,14 +39,6 @@ jobs:
virtualenvs-create: true
virtualenvs-in-project: true
- name: Cache Poetry dependencies
uses: actions/cache@v4
with:
path: .venv
key: venv-package-${{ runner.os }}-py3.13-${{ hashFiles('poetry.lock') }}
restore-keys: |
venv-package-${{ runner.os }}-py3.13-
- name: Install dependencies
run: poetry install --with dev
@@ -163,18 +47,12 @@ jobs:
sudo apt-get update
sudo apt-get install -y ffmpeg
- name: Cache spaCy model
uses: actions/cache@v4
with:
path: ~/.cache/spacy
key: spacy-en_core_web_sm-${{ runner.os }}
- name: Install spaCy language models
run: poetry run python -m spacy download en_core_web_sm
- name: Run unit tests
run: |
poetry run pytest tests/unit/ -v --cov=karaoke_gen --cov-report=xml --cov-report=term-missing --cov-fail-under=69
poetry run pytest tests/unit/ -v --cov=karaoke_gen --cov-report=xml --cov-report=term-missing --cov-fail-under=70
- name: Upload coverage reports
uses: codecov/codecov-action@v5
@@ -187,8 +65,6 @@ jobs:
package-integration-tests:
name: "Package - Integration Tests"
runs-on: ubuntu-latest
needs: changes
if: github.ref != 'refs/heads/main' && (needs.changes.outputs.package == 'true' || needs.changes.outputs.deps == 'true')
steps:
- uses: actions/checkout@v4
with:
@@ -205,14 +81,6 @@ jobs:
virtualenvs-create: true
virtualenvs-in-project: true
- name: Cache Poetry dependencies
uses: actions/cache@v4
with:
path: .venv
key: venv-package-${{ runner.os }}-py3.13-${{ hashFiles('poetry.lock') }}
restore-keys: |
venv-package-${{ runner.os }}-py3.13-
- name: Install dependencies
run: poetry install --with dev
@@ -221,27 +89,16 @@ jobs:
sudo apt-get update
sudo apt-get install -y ffmpeg
- name: Cache spaCy model
uses: actions/cache@v4
with:
path: ~/.cache/spacy
key: spacy-en_core_web_sm-${{ runner.os }}
- name: Install spaCy language models
run: poetry run python -m spacy download en_core_web_sm
- name: Run integration tests
run: poetry run pytest tests/integration/ -v
package-build-test:
name: "Package - Build & Install (Test)"
runs-on: [self-hosted, linux, gcp]
needs: [changes, package-unit-tests, package-integration-tests]
if: >-
github.ref != 'refs/heads/main' && always() &&
(needs.package-unit-tests.result == 'success' || needs.package-unit-tests.result == 'skipped') &&
(needs.package-integration-tests.result == 'success' || needs.package-integration-tests.result == 'skipped') &&
(needs.changes.outputs.package == 'true' || needs.changes.outputs.deps == 'true')
package-build:
name: "Package - Build & Install"
runs-on: ubuntu-latest
needs: [package-unit-tests, package-integration-tests]
steps:
- uses: actions/checkout@v4
with:
@@ -261,20 +118,36 @@ jobs:
- name: Build package
run: poetry build
- name: Free disk space
uses: jlumbroso/free-disk-space@main
with:
tool-cache: false
android: true
haskell: true
dotnet: false
large-packages: false
docker-images: false
swap-storage: false
- name: Test package installation
run: |
pip install dist/*.whl
python -m karaoke_gen.utils.gen_cli --help
- name: Upload build artifacts
uses: actions/upload-artifact@v4
with:
name: package-dist
path: dist/
retention-days: 1
# ============================================
# BACKEND TESTS (PRs only - main skips tests)
# BACKEND TESTS (Cloud Run API service)
# ============================================
backend-unit-tests:
name: "Backend - Unit Tests"
runs-on: [self-hosted, linux, gcp]
needs: changes
if: github.ref != 'refs/heads/main' && (needs.changes.outputs.backend == 'true' || needs.changes.outputs.package == 'true' || needs.changes.outputs.deps == 'true')
runs-on: ubuntu-latest
steps:
- name: Checkout code
uses: actions/checkout@v4
@@ -295,7 +168,7 @@ jobs:
- name: Install dependencies
run: poetry install
- name: Run all backend unit tests
- name: Run unit tests
env:
GOOGLE_CLOUD_PROJECT: test-project
GCS_BUCKET_NAME: test-bucket
@@ -303,16 +176,23 @@ jobs:
ENVIRONMENT: test
run: |
poetry run pytest \
backend/tests/ \
--ignore=backend/tests/emulator \
backend/tests/test_models.py \
backend/tests/test_job_manager.py \
backend/tests/test_file_upload.py \
backend/tests/test_services.py \
-v --tb=short
backend-emulator-tests:
name: "Backend - Emulator Integration Tests"
runs-on: [self-hosted, linux, gcp]
needs: changes
if: github.ref != 'refs/heads/main' && (needs.changes.outputs.backend == 'true' || needs.changes.outputs.package == 'true' || needs.changes.outputs.deps == 'true')
runs-on: ubuntu-latest
steps:
- name: Free disk space
run: |
sudo rm -rf /usr/share/dotnet
sudo rm -rf /opt/ghc
sudo rm -rf /usr/local/share/boost
df -h
- name: Checkout code
uses: actions/checkout@v4
with:
@@ -335,6 +215,14 @@ jobs:
distribution: "temurin"
java-version: "21"
- name: Set up Cloud SDK
uses: google-github-actions/setup-gcloud@v2
with:
version: "latest"
- name: Install Firestore emulator
run: gcloud components install beta cloud-firestore-emulator --quiet
- name: Install dependencies
run: poetry install
@@ -357,8 +245,6 @@ jobs:
backend-lint:
name: "Backend - Code Quality"
runs-on: ubuntu-latest
needs: changes
if: github.ref != 'refs/heads/main' && (needs.changes.outputs.backend == 'true' || needs.changes.outputs.package == 'true')
steps:
- name: Checkout code
uses: actions/checkout@v4
@@ -376,14 +262,6 @@ jobs:
virtualenvs-create: true
virtualenvs-in-project: true
- name: Cache Poetry dependencies
uses: actions/cache@v4
with:
path: .venv
key: venv-lint-${{ runner.os }}-py3.13-${{ hashFiles('poetry.lock') }}
restore-keys: |
venv-lint-${{ runner.os }}-py3.13-
- name: Install dependencies
run: poetry install
@@ -392,79 +270,10 @@ jobs:
poetry run python -m py_compile backend/main.py backend/config.py
echo "✅ Basic syntax validation passed"
# ============================================
# CI GATE - Required status check for PRs
# Only runs on PRs, not on main
# ============================================
ci-gate:
name: "CI Gate"
runs-on: ubuntu-latest
needs: [frontend-unit-tests, frontend-build, package-unit-tests, package-integration-tests, backend-unit-tests, backend-emulator-tests, backend-lint]
if: github.ref != 'refs/heads/main' && always()
steps:
- name: Check results
run: |
results=(
"${{ needs.frontend-unit-tests.result }}"
"${{ needs.frontend-build.result }}"
"${{ needs.package-unit-tests.result }}"
"${{ needs.package-integration-tests.result }}"
"${{ needs.backend-unit-tests.result }}"
"${{ needs.backend-emulator-tests.result }}"
"${{ needs.backend-lint.result }}"
)
for result in "${results[@]}"; do
if [[ "$result" == "failure" || "$result" == "cancelled" ]]; then
echo "❌ CI failed"
exit 1
fi
done
echo "✅ CI passed"
# ============================================
# DEPLOYMENTS (main branch only)
# Tests already passed on PR, so we skip them here
# ============================================
package-build:
name: "Package - Build"
runs-on: ubuntu-latest
if: github.ref == 'refs/heads/main' && github.event_name == 'push'
steps:
- uses: actions/checkout@v4
with:
submodules: true
- name: Set up Python 3.13
uses: actions/setup-python@v5
with:
python-version: "3.13"
- name: Install Poetry
uses: snok/install-poetry@v1
with:
virtualenvs-create: true
virtualenvs-in-project: true
- name: Cache Poetry dependencies
uses: actions/cache@v4
with:
path: .venv
key: venv-package-${{ runner.os }}-py3.13-${{ hashFiles('poetry.lock') }}
restore-keys: |
venv-package-${{ runner.os }}-py3.13-
- name: Build package
run: poetry build
- name: Upload build artifacts
uses: actions/upload-artifact@v4
with:
name: package-dist
path: dist/
retention-days: 1
package-publish:
name: "Deploy - Publish to PyPI"
runs-on: ubuntu-latest
@@ -556,16 +365,17 @@ jobs:
name: "Deploy - Frontend (GitHub Pages)"
runs-on: ubuntu-latest
if: github.ref == 'refs/heads/main' && github.event_name == 'push'
environment:
name: github-pages
url: ${{ steps.deployment.outputs.page_url }}
steps:
- name: Checkout
uses: actions/checkout@v4
- name: Setup Node.js with caching
- name: Setup Node.js
uses: actions/setup-node@v4
with:
node-version: '20'
cache: 'npm'
cache-dependency-path: frontend/package-lock.json
- name: Enable Yarn (for lyrics frontend)
run: corepack enable
@@ -582,14 +392,6 @@ jobs:
run: npm run build
# ===== Build Lyrics Review Frontend (Vite) =====
- name: Cache Yarn dependencies (lyrics frontend)
uses: actions/cache@v4
with:
path: lyrics_transcriber_temp/lyrics_transcriber/frontend/node_modules
key: yarn-lyrics-${{ runner.os }}-${{ hashFiles('lyrics_transcriber_temp/lyrics_transcriber/frontend/yarn.lock') }}
restore-keys: |
yarn-lyrics-${{ runner.os }}-
- name: Install lyrics frontend dependencies
working-directory: lyrics_transcriber_temp/lyrics_transcriber/frontend
run: yarn install --frozen-lockfile
@@ -626,28 +428,20 @@ jobs:
deploy-backend:
name: "Deploy - Backend (Cloud Run)"
runs-on: [self-hosted, linux, gcp]
runs-on: ubuntu-latest
needs: [backend-unit-tests, backend-emulator-tests, backend-lint]
if: github.ref == 'refs/heads/main' && github.event_name == 'push'
permissions:
contents: read
id-token: write
env:
REGISTRY: us-central1-docker.pkg.dev
PROJECT_ID: nomadkaraoke
REPOSITORY: karaoke-repo
BASE_IMAGE_NAME: karaoke-backend-base
APP_IMAGE_NAME: karaoke-backend
steps:
# ============================================
# Check if deployment is needed FIRST
# ============================================
- name: Checkout code
uses: actions/checkout@v4
with:
submodules: false
fetch-depth: 0 # Need full history for tag check
- name: Set up Python 3.13
- name: Set up Python (for version extraction)
uses: actions/setup-python@v5
with:
python-version: "3.13"
@@ -661,7 +455,7 @@ jobs:
VERSION=$(poetry version --short)
echo "version=$VERSION" >> $GITHUB_OUTPUT
echo "📋 Current version: $VERSION"
if git ls-remote --tags origin | grep -q "refs/tags/v$VERSION$"; then
echo "needs_deploy=false" >> $GITHUB_OUTPUT
echo "⏭️ Tag v$VERSION already exists - skipping backend deployment"
@@ -671,8 +465,6 @@ jobs:
echo "✨ Tag v$VERSION does not exist - proceeding with deployment"
fi
# No disk cleanup needed - self-hosted runner has 200GB SSD
- name: Authenticate to Google Cloud (Workload Identity)
if: steps.version_check.outputs.needs_deploy == 'true'
id: auth
@@ -687,37 +479,21 @@ jobs:
with:
version: "latest"
- name: Configure Docker for Artifact Registry
if: steps.version_check.outputs.needs_deploy == 'true'
run: gcloud auth configure-docker ${{ env.REGISTRY }} --quiet
# ============================================
# Docker Build with BuildKit + Registry Cache
# This replaces expensive Cloud Build with free GitHub Actions
# Registry cache persists between builds for fast rebuilds
# ============================================
- name: Set up Docker Buildx
if: steps.version_check.outputs.needs_deploy == 'true'
uses: docker/setup-buildx-action@v3
- name: Calculate base image hash
if: steps.version_check.outputs.needs_deploy == 'true'
id: base-hash
run: |
# Hash both poetry.lock AND Dockerfile.base - rebuild if either changes
CURRENT_HASH=$(cat poetry.lock backend/Dockerfile.base | sha256sum | cut -d' ' -f1)
echo "hash=$CURRENT_HASH" >> $GITHUB_OUTPUT
echo "📋 Base content hash: $CURRENT_HASH"
- name: Check if base image exists and is current
- name: Check if base image needs rebuild
if: steps.version_check.outputs.needs_deploy == 'true'
id: check-base
run: |
BASE_IMAGE="${{ env.REGISTRY }}/${{ env.PROJECT_ID }}/${{ env.REPOSITORY }}/${{ env.BASE_IMAGE_NAME }}:latest"
echo "🔍 Checking if base image needs rebuild..."
# Try to get the existing base image's hash label
# Hash both poetry.lock AND Dockerfile.base - rebuild if either changes
CURRENT_HASH=$(cat poetry.lock backend/Dockerfile.base | sha256sum | cut -d' ' -f1)
echo "Current combined hash (poetry.lock + Dockerfile.base): $CURRENT_HASH"
gcloud auth configure-docker us-central1-docker.pkg.dev --quiet
BASE_IMAGE="us-central1-docker.pkg.dev/nomadkaraoke/karaoke-repo/karaoke-backend-base:latest"
EXISTING_HASH=""
if docker pull $BASE_IMAGE 2>/dev/null; then
EXISTING_HASH=$(docker inspect $BASE_IMAGE --format='{{index .Config.Labels "base.content.hash"}}' 2>/dev/null || echo "")
# Fallback to old label name for backwards compatibility
@@ -726,10 +502,10 @@ jobs:
fi
echo "Existing base image hash: $EXISTING_HASH"
else
echo "⚠️ No existing base image found - will build"
echo "⚠️ No existing base image found"
fi
if [ "${{ steps.base-hash.outputs.hash }}" != "$EXISTING_HASH" ]; then
if [ "$CURRENT_HASH" != "$EXISTING_HASH" ]; then
echo "🔄 Base image needs rebuild (poetry.lock or Dockerfile.base changed)"
echo "needs_rebuild=true" >> $GITHUB_OUTPUT
else
@@ -737,73 +513,88 @@ jobs:
echo "needs_rebuild=false" >> $GITHUB_OUTPUT
fi
- name: Build and push base image (if needed)
- name: Rebuild base image (if needed)
if: steps.version_check.outputs.needs_deploy == 'true' && steps.check-base.outputs.needs_rebuild == 'true'
uses: docker/build-push-action@v6
with:
context: .
file: backend/Dockerfile.base
push: true
tags: |
${{ env.REGISTRY }}/${{ env.PROJECT_ID }}/${{ env.REPOSITORY }}/${{ env.BASE_IMAGE_NAME }}:latest
${{ env.REGISTRY }}/${{ env.PROJECT_ID }}/${{ env.REPOSITORY }}/${{ env.BASE_IMAGE_NAME }}:${{ github.sha }}
# Registry cache for ALL layers (mode=max) - persists between builds
# ignore-error=true allows build to continue if cache doesn't exist yet
cache-from: type=registry,ref=${{ env.REGISTRY }}/${{ env.PROJECT_ID }}/${{ env.REPOSITORY }}/${{ env.BASE_IMAGE_NAME }}:cache,ignore-error=true
cache-to: type=registry,ref=${{ env.REGISTRY }}/${{ env.PROJECT_ID }}/${{ env.REPOSITORY }}/${{ env.BASE_IMAGE_NAME }}:cache,mode=max,ignore-error=true
build-args: |
BUILD_DATE=${{ github.run_id }}
BUILD_ID=${{ github.sha }}
BASE_CONTENT_HASH=${{ steps.base-hash.outputs.hash }}
# Provenance disabled for smaller image and faster push
provenance: false
run: |
echo "🏗️ Rebuilding base image with updated dependencies..."
- name: Build and push app image
if: steps.version_check.outputs.needs_deploy == 'true'
uses: docker/build-push-action@v6
with:
context: .
file: backend/Dockerfile
push: true
tags: |
${{ env.REGISTRY }}/${{ env.PROJECT_ID }}/${{ env.REPOSITORY }}/${{ env.APP_IMAGE_NAME }}:latest
${{ env.REGISTRY }}/${{ env.PROJECT_ID }}/${{ env.REPOSITORY }}/${{ env.APP_IMAGE_NAME }}:${{ github.sha }}
${{ env.REGISTRY }}/${{ env.PROJECT_ID }}/${{ env.REPOSITORY }}/${{ env.APP_IMAGE_NAME }}:v${{ steps.version_check.outputs.version }}
# Registry cache for app layers
# ignore-error=true allows build to continue if cache doesn't exist yet
cache-from: type=registry,ref=${{ env.REGISTRY }}/${{ env.PROJECT_ID }}/${{ env.REPOSITORY }}/${{ env.APP_IMAGE_NAME }}:cache,ignore-error=true
cache-to: type=registry,ref=${{ env.REGISTRY }}/${{ env.PROJECT_ID }}/${{ env.REPOSITORY }}/${{ env.APP_IMAGE_NAME }}:cache,mode=max,ignore-error=true
provenance: false
BASE_BUILD_ID=$(gcloud builds submit \
--config=cloudbuild-base.yaml \
--project=nomadkaraoke \
--format='value(id)' \
--async \
.)
# ============================================
# Deploy to Cloud Run
# ============================================
echo "Base Build ID: $BASE_BUILD_ID"
echo "Build URL: https://console.cloud.google.com/cloud-build/builds/$BASE_BUILD_ID?project=nomadkaraoke"
- name: Deploy to Cloud Run
echo "Waiting for base image build to complete..."
while true; do
BUILD_STATUS=$(gcloud builds describe $BASE_BUILD_ID --project=nomadkaraoke --format='value(status)')
echo "Base build status: $BUILD_STATUS"
if [ "$BUILD_STATUS" = "SUCCESS" ]; then
echo "✅ Base image build completed successfully"
break
elif [ "$BUILD_STATUS" = "FAILURE" ] || [ "$BUILD_STATUS" = "CANCELLED" ] || [ "$BUILD_STATUS" = "TIMEOUT" ]; then
echo "❌ Base image build failed with status: $BUILD_STATUS"
echo ""
echo "==================== BUILD LOGS ===================="
gcloud logging read "resource.type=build AND resource.labels.build_id=$BASE_BUILD_ID" \
--project=nomadkaraoke \
--format="value(textPayload)" \
--limit=200 \
--order=asc || echo "Failed to retrieve logs from Cloud Logging"
echo "===================================================="
gcloud builds log $BASE_BUILD_ID --project=nomadkaraoke 2>&1 || true
exit 1
fi
sleep 30
done
- name: Trigger Cloud Build
if: steps.version_check.outputs.needs_deploy == 'true'
run: |
VERSION="${{ steps.version_check.outputs.version }}"
IMAGE="${{ env.REGISTRY }}/${{ env.PROJECT_ID }}/${{ env.REPOSITORY }}/${{ env.APP_IMAGE_NAME }}:${{ github.sha }}"
echo "🚀 Triggering Cloud Build for deployment..."
echo "Commit: ${{ github.sha }}"
echo "Version: ${{ steps.version_check.outputs.version }}"
echo "🚀 Deploying to Cloud Run..."
echo "Image: $IMAGE"
echo "Version: $VERSION"
BUILD_ID=$(gcloud builds submit \
--config=cloudbuild.yaml \
--project=nomadkaraoke \
--format='value(id)' \
--async \
.)
gcloud run deploy karaoke-backend \
--image $IMAGE \
--region us-central1 \
--platform managed \
--allow-unauthenticated \
--memory 16Gi \
--cpu 8 \
--cpu-boost \
--execution-environment gen2 \
--timeout 1800 \
--concurrency 1 \
--max-instances 50 \
--min-instances 0 \
--set-env-vars "GOOGLE_CLOUD_PROJECT=${{ env.PROJECT_ID }},GCS_BUCKET_NAME=karaoke-gen-storage-${{ env.PROJECT_ID }},FIRESTORE_COLLECTION=jobs,ENVIRONMENT=production,AUDIO_SEPARATOR_API_URL=https://nomadkaraoke--audio-separator-api.modal.run,KARAOKE_GEN_VERSION=$VERSION,DEFAULT_DROPBOX_PATH=/MediaUnsynced/Karaoke/Tracks-Organized,DEFAULT_GDRIVE_FOLDER_ID=1laRKAyxo0v817SstfM5XkpbWiNKNAMSX,ENABLE_CLOUD_TASKS=true,CLOUD_RUN_SERVICE_URL=https://api.nomadkaraoke.com,GCP_REGION=us-central1,DEFAULT_ENABLE_YOUTUBE_UPLOAD=true,DEFAULT_BRAND_PREFIX=NOMAD" \
--set-secrets "AUDIOSHAKE_API_TOKEN=audioshake-api-key:latest,GENIUS_API_TOKEN=genius-api-key:latest,SPOTIFY_COOKIE_SP_DC=spotify-cookie:latest,RAPIDAPI_KEY=rapidapi-key:latest,DEFAULT_DISCORD_WEBHOOK_URL=discord-releases-webhook:latest,ADMIN_TOKENS=admin-tokens:latest,RED_API_KEY=red-api-key:latest,RED_API_URL=red-api-url:latest,OPS_API_KEY=ops-api-key:latest,OPS_API_URL=ops-api-url:latest,FLACFETCH_API_URL=flacfetch-api-url:latest,FLACFETCH_API_KEY=flacfetch-api-key:latest"
echo "Build ID: $BUILD_ID"
echo "Build URL: https://console.cloud.google.com/cloud-build/builds/$BUILD_ID?project=nomadkaraoke"
echo "Waiting for build to complete..."
while true; do
BUILD_STATUS=$(gcloud builds describe $BUILD_ID --project=nomadkaraoke --format='value(status)')
echo "Build status: $BUILD_STATUS"
if [ "$BUILD_STATUS" = "SUCCESS" ]; then
echo "✅ Build completed successfully"
break
elif [ "$BUILD_STATUS" = "FAILURE" ] || [ "$BUILD_STATUS" = "CANCELLED" ] || [ "$BUILD_STATUS" = "TIMEOUT" ]; then
echo "❌ Build failed with status: $BUILD_STATUS"
echo ""
echo "==================== BUILD LOGS ===================="
gcloud logging read "resource.type=build AND resource.labels.build_id=$BUILD_ID" \
--project=nomadkaraoke \
--format="value(textPayload)" \
--limit=200 \
--order=asc || echo "Failed to retrieve logs from Cloud Logging"
echo "===================================================="
echo ""
gcloud builds log $BUILD_ID --project=nomadkaraoke 2>&1 || true
exit 1
fi
sleep 30
done
- name: Verify deployment and version
if: steps.version_check.outputs.needs_deploy == 'true'
@@ -815,22 +606,22 @@ jobs:
echo "Service URL: $SERVICE_URL"
echo "Expected version: $EXPECTED_VERSION"
# Wait for new revision to be ready
echo "Waiting for new revision to be ready..."
sleep 15
# Basic health check
curl -f "$SERVICE_URL/api/health" || exit 1
echo "✅ Service is healthy"
# Version verification with retries (Cloud Run may take time to route to new revision)
MAX_RETRIES=6
RETRY_DELAY=10
for i in $(seq 1 $MAX_RETRIES); do
echo "Checking deployed version (attempt $i/$MAX_RETRIES)..."
DEPLOYED_VERSION=$(curl -sf "$SERVICE_URL/api/health/detailed" | jq -r '.version // empty')
if [ "$DEPLOYED_VERSION" = "$EXPECTED_VERSION" ]; then
echo "✅ Version verified: $DEPLOYED_VERSION"
break
@@ -854,10 +645,7 @@ jobs:
echo "Commit: ${{ github.sha }}"
echo "Branch: ${{ github.ref_name }}"
echo "Run: ${{ github.run_number }}"
echo ""
echo "📊 Built on self-hosted GCP runner (200GB SSD, no disk space issues)"
echo " Runner: gcp-runner with 4 vCPU, 16GB RAM"
- name: Skipped deployment summary
if: steps.version_check.outputs.needs_deploy == 'false'
run: |
-10
View File
@@ -12,9 +12,6 @@ tests/outputs
/input
/output/
# Cloud Infra
.emulator-logs
# Next.js / React Frontend (./frontend)
node_modules/
frontend/node_modules/
@@ -44,9 +41,6 @@ frontend/.pnpm-debug.log*
# Testing
frontend/coverage/
frontend/test-results/
frontend/playwright-report/
frontend/playwright/.cache/
# TypeScript
frontend/*.tsbuildinfo
@@ -226,7 +220,3 @@ cython_debug/
# and can be added to the global gitignore or merged into this file. For a more nuclear
# option (not recommended) you can uncomment the following to ignore the entire idea folder.
#.idea/
flacfetch
# Test output directories
Rick Astley*/
-97
View File
@@ -1,97 +0,0 @@
# Karaoke-Gen AI Assistant Guidelines
## Project Overview
**Karaoke video generation platform** - CLI tool and web service that creates professional karaoke videos with synchronized lyrics.
- **Production**: <https://gen.nomadkaraoke.com> (frontend), <https://api.nomadkaraoke.com> (backend)
- **CLI**: `karaoke-gen` (local), `karaoke-gen-remote` (cloud)
- **Repo**: <https://github.com/nomadkaraoke/karaoke-gen>
## Quick Reference
| What | Where |
|------|-------|
| Current status | `docs/README.md` |
| Architecture | `docs/ARCHITECTURE.md` |
| Dev setup & testing | `docs/DEVELOPMENT.md` |
| API reference | `docs/API.md` |
| Past learnings | `docs/LESSONS-LEARNED.md` |
## Tech Stack
- **Backend**: FastAPI on Cloud Run, Firestore, GCS, Secret Manager
- **Frontend**: Next.js on Cloudflare Pages
- **Processing**: Modal (GPU audio separation), AudioShake/Whisper (transcription)
- **Infra**: Pulumi IaC, GitHub Actions CI/CD
## Essential Rules
### Git Workflow
- **Never commit directly to `main`** - always use feature branches
- **Use git worktrees** for all changes:
```bash
git worktree add -b feature/xyz ../karaoke-gen-feat-xyz main
```
- Create PR with summary, changes, testing info
- Merge only after CI passes
### Testing (Required Before Commit)
```bash
# Backend (10 min timeout, limit output)
make test 2>&1 | tail -n 500
# Frontend
cd frontend && npm run test:all 2>&1 | tail -n 200
```
### Version Bumping
- Bump `tool.poetry.version` in `pyproject.toml` for code changes
- Skip for docs-only changes
### Infrastructure
- All GCP changes via Pulumi in `infrastructure/`
- `gcloud` CLI for reading/debugging only
- Stop and notify user on auth issues
## Documentation Maintenance
### Structure
```text
docs/
├── README.md # Current status + navigation (UPDATE THIS)
├── ARCHITECTURE.md # System design
├── DEVELOPMENT.md # Dev setup, testing, deployment
├── API.md # Backend API reference
├── LESSONS-LEARNED.md # Key insights for future agents
└── archive/ # Historical docs (YYYY-MM-DD-topic.md)
```
### Rules
1. **Before merging PRs**: Run `/docs-review` to check if docs need updating
2. **Periodically**: Run `/docs-maintain` to verify docs are organized
3. **For significant work**: Create `docs/archive/YYYY-MM-DD-topic.md`
4. **Add learnings**: Update `docs/LESSONS-LEARNED.md` with insights
5. **Keep status current**: Update `docs/README.md` when project state changes
### What Goes Where
- **README.md**: Current status, known issues, quick links
- **ARCHITECTURE.md**: System design, data flow, tech decisions
- **DEVELOPMENT.md**: Setup, testing, deployment, CI/CD
- **API.md**: Endpoints, request/response formats
- **LESSONS-LEARNED.md**: What worked, what didn't, gotchas
- **archive/**: Completed work, session summaries, old plans
## Related Projects
### flacfetch
Location: `/Users/andrew/Projects/flacfetch`
Workflow: Make changes on `main`, bump version, push, wait for CI, then `poetry update flacfetch` in karaoke-gen.
## Pre-Commit Checklist
- [ ] Tests pass (`make test` / `npm run test:all`)
- [ ] Version bumped (if code changed)
- [ ] `/docs-review` run (check for doc updates)
- [ ] PR created with clear description
-21
View File
@@ -1,21 +0,0 @@
MIT License
Copyright (c) 2024 Nomad Karaoke LLC
Permission is hereby granted, free of charge, to any person obtaining a copy
of this software and associated documentation files (the "Software"), to deal
in the Software without restriction, including without limitation the rights
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
copies of the Software, and to permit persons to whom the Software is
furnished to do so, subject to the following conditions:
The above copyright notice and this permission notice shall be included in all
copies or substantial portions of the Software.
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
SOFTWARE.
-77
View File
@@ -1,77 +0,0 @@
# Makefile for karaoke-gen project
# Run `make help` to see available commands
.PHONY: help test test-unit test-backend test-integration test-e2e test-all lint clean emulators-start emulators-stop
# Default target
help:
@echo "Available commands:"
@echo " make test - Run all tests (unit + backend + emulator-based)"
@echo " make test-unit - Run unit tests only (karaoke_gen)"
@echo " make test-backend - Run backend unit tests only"
@echo " make test-e2e - Run E2E tests with emulators (starts/stops automatically)"
@echo " make test-all - Alias for 'make test'"
@echo " make lint - Run linter checks"
@echo " make emulators-start - Start GCP emulators for local development"
@echo " make emulators-stop - Stop GCP emulators"
@echo ""
@echo "Before committing, run: make test"
# Run unit tests for karaoke_gen package
test-unit:
@echo "=== Running karaoke_gen unit tests ==="
poetry run pytest tests/unit/ -v --cov=karaoke_gen --cov-report=term-missing --cov-fail-under=69
# Run backend unit tests
test-backend:
@echo "=== Running backend unit tests ==="
poetry run pytest backend/tests/ --ignore=backend/tests/emulator -v
# Run E2E integration tests with emulators
test-e2e:
@echo "=== Running E2E integration tests with emulators ==="
@./scripts/start-emulators.sh || (echo "Failed to start emulators" && exit 1)
@export FIRESTORE_EMULATOR_HOST=127.0.0.1:8080 && \
export STORAGE_EMULATOR_HOST=http://127.0.0.1:4443 && \
export GOOGLE_CLOUD_PROJECT=test-project && \
export GCS_BUCKET_NAME=test-bucket && \
export ADMIN_TOKENS=test-admin-token && \
export ENVIRONMENT=test && \
poetry run pytest backend/tests/emulator/ -v; \
TEST_RESULT=$$?; \
./scripts/stop-emulators.sh; \
exit $$TEST_RESULT
# Run all tests (use this before committing!)
test: test-unit test-backend test-e2e
@echo ""
@echo "✅ All tests passed! Ready to commit."
# Alias for test
test-all: test
# Lint checks
lint:
@echo "=== Running lint checks ==="
poetry run ruff check karaoke_gen/ backend/
poetry run ruff format --check karaoke_gen/ backend/
# Start emulators for local development
emulators-start:
@echo "=== Starting GCP emulators ==="
./scripts/start-emulators.sh
# Stop emulators
emulators-stop:
@echo "=== Stopping GCP emulators ==="
./scripts/stop-emulators.sh
# Clean up temporary files
clean:
find . -type d -name "__pycache__" -exec rm -rf {} + 2>/dev/null || true
find . -type d -name ".pytest_cache" -exec rm -rf {} + 2>/dev/null || true
find . -type d -name ".coverage" -exec rm -rf {} + 2>/dev/null || true
find . -type f -name "*.pyc" -delete 2>/dev/null || true
rm -rf htmlcov/ .coverage coverage.xml 2>/dev/null || true
@echo "Cleaned up temporary files"
-693
View File
@@ -1,693 +0,0 @@
# Karaoke Generator 🎶 🎥 🚀
![PyPI - Version](https://img.shields.io/pypi/v/karaoke-gen)
![PyPI - Python Version](https://img.shields.io/pypi/pyversions/karaoke-gen)
![Tests](https://github.com/nomadkaraoke/karaoke-gen/workflows/Test%20and%20Publish/badge.svg)
![Test Coverage](https://codecov.io/gh/nomadkaraoke/karaoke-gen/branch/main/graph/badge.svg)
Generate professional karaoke videos with instrumental audio and synchronized lyrics. Available as a **local CLI** (`karaoke-gen`) or **cloud-based CLI** (`karaoke-gen-remote`) that offloads processing to Google Cloud.
## ✨ Two Ways to Generate Karaoke
### 1. Local CLI (`karaoke-gen`)
Run all processing locally on your machine. Requires GPU for optimal audio separation performance.
```bash
karaoke-gen "ABBA" "Waterloo"
```
### 2. Remote CLI (`karaoke-gen-remote`)
Offload all processing to a cloud backend. No GPU required - just authenticate and submit jobs.
```bash
karaoke-gen-remote ./song.flac "ABBA" "Waterloo"
```
Both CLIs produce identical outputs: 4K karaoke videos, CDG+MP3 packages, audio stems, and more.
---
## 🎯 Features
### Core Pipeline
- **Audio Separation**: AI-powered vocal/instrumental separation using MDX and Demucs models
- **Lyrics Transcription**: Word-level timestamps via AudioShake API
- **Lyrics Correction**: Match transcription against online lyrics (Genius, Spotify, Musixmatch)
- **Human Review**: Interactive UI for correcting lyrics before final render
- **Video Rendering**: High-quality 4K karaoke videos with customizable styles
- **Multiple Outputs**: MP4 (4K lossless/lossy, 720p), MKV, CDG+MP3, TXT+MP3
### Distribution Features
- **YouTube Upload**: Automatic upload to your YouTube channel
- **Dropbox Integration**: Organize output in brand-coded folders
- **Google Drive**: Upload to public share folders
- **Discord Notifications**: Webhook notifications on completion
---
## 📦 Installation
```bash
pip install karaoke-gen
```
This installs both `karaoke-gen` (local) and `karaoke-gen-remote` (cloud) CLIs.
### Requirements
- Python 3.10-3.13
- FFmpeg
- For local processing: CUDA-capable GPU or Apple Silicon CPU recommended
### Transcription Provider Setup
**Transcription is required** for creating karaoke videos with synchronized lyrics. The system needs word-level timing data to display lyrics in sync with the music.
#### Option 1: AudioShake (Recommended)
Commercial service with high-quality transcription. Best for production use.
```bash
export AUDIOSHAKE_API_TOKEN="your_audioshake_token"
```
Get an API key at [https://www.audioshake.ai/](https://www.audioshake.ai/) - business only, at time of writing this.
#### Option 2: Local Whisper (No Cloud Required)
Run Whisper directly on your local machine using whisper-timestamped. Works on CPU, NVIDIA GPU (CUDA), or Apple Silicon.
```bash
# Install with local Whisper support
pip install "karaoke-gen[local-whisper]"
# Optional: Configure model size (tiny, base, small, medium, large)
export WHISPER_MODEL_SIZE="medium"
# Optional: Force specific device (cpu, cuda, mps)
export WHISPER_DEVICE="cpu"
```
**Model Size Guide:**
| Model | VRAM | Speed | Quality |
|-------|------|-------|---------|
| tiny | ~1GB | Fast | Lower |
| base | ~1GB | Fast | Basic |
| small | ~2GB | Medium | Good |
| medium | ~5GB | Slower | Better |
| large | ~10GB | Slowest | Best |
**CPU-Only Installation** (no GPU required):
```bash
# Pre-install CPU-only PyTorch first
pip install torch torchaudio --index-url https://download.pytorch.org/whl/cpu
pip install "karaoke-gen[local-whisper]"
```
Local Whisper runs automatically as a fallback when no cloud transcription services are configured.
#### Option 3: Whisper via RunPod
Cloud-based alternative using OpenAI's Whisper model on RunPod infrastructure.
```bash
export RUNPOD_API_KEY="your_runpod_key"
export WHISPER_RUNPOD_ID="your_whisper_endpoint_id"
```
Set up a Whisper endpoint at [https://www.runpod.io/](https://www.runpod.io/)
#### Without Transcription (Instrumental Only)
If you don't need synchronized lyrics, use the `--skip-lyrics` flag:
```bash
karaoke-gen --skip-lyrics "Artist" "Title"
```
This creates an instrumental-only karaoke video without lyrics overlay.
> **Note:** See `lyrics_transcriber_temp/README.md` for detailed transcription provider configuration options.
---
## 🖥️ Local CLI (`karaoke-gen`)
### Basic Usage
```bash
# Generate from local audio file
karaoke-gen ./song.mp3 "Artist Name" "Song Title"
# Search and download audio automatically
karaoke-gen "Rick Astley" "Never Gonna Give You Up"
# Process from YouTube URL
karaoke-gen "https://www.youtube.com/watch?v=dQw4w9WgXcQ" "Rick Astley" "Never Gonna Give You Up"
```
### Remote Audio Separation (Optional)
Offload just the GPU-intensive audio separation to Modal.com while keeping other processing local:
```bash
export AUDIO_SEPARATOR_API_URL="https://USERNAME--audio-separator-api.modal.run"
karaoke-gen "Artist" "Title"
```
### Key Options
```bash
# Custom styling
karaoke-gen --style_params_json="./styles.json" "Artist" "Title"
# Generate CDG and TXT packages
karaoke-gen --enable_cdg --enable_txt "Artist" "Title"
# YouTube upload
karaoke-gen --enable_youtube_upload --youtube_description_file="./desc.txt" "Artist" "Title"
# Full production run
karaoke-gen \
--style_params_json="./branding.json" \
--enable_cdg \
--enable_txt \
--brand_prefix="BRAND" \
--enable_youtube_upload \
--youtube_description_file="./description.txt" \
"Artist" "Title"
```
### Full Options Reference
```bash
karaoke-gen --help
```
---
## ☁️ Remote CLI (`karaoke-gen-remote`)
The remote CLI submits jobs to a Google Cloud backend that handles all processing. You don't need a GPU or any audio processing libraries installed locally.
### Setup
1. **Set the backend URL:**
```bash
export KARAOKE_GEN_URL="https://api.nomadkaraoke.com" # Or your own backend
```
2. **Authenticate with Google Cloud:**
```bash
gcloud auth login
```
### Basic Usage
```bash
# Submit a job
karaoke-gen-remote ./song.flac "ABBA" "Waterloo"
# The CLI will:
# 1. Upload your audio file
# 2. Monitor processing progress
# 3. Open lyrics review UI when ready
# 4. Prompt for instrumental selection
# 5. Download all outputs when complete
```
### Job Management
```bash
# List all jobs
karaoke-gen-remote --list
# Resume monitoring an existing job
karaoke-gen-remote --resume abc12345
# Cancel a running job
karaoke-gen-remote --cancel abc12345
# Delete a job and its files
karaoke-gen-remote --delete abc12345
```
### Full Production Run
```bash
karaoke-gen-remote \
--style_params_json="./karaoke-styles.json" \
--enable_cdg \
--enable_txt \
--brand_prefix=NOMAD \
--enable_youtube_upload \
--youtube_description_file="./youtube-description.txt" \
./song.flac "Artist" "Title"
```
### Environment Variables
| Variable | Description | Default |
|----------|-------------|---------|
| `KARAOKE_GEN_URL` | Backend service URL | Required |
| `KARAOKE_GEN_AUTH_TOKEN` | Admin auth token (for protected endpoints) | Optional |
| `REVIEW_UI_URL` | Lyrics review UI URL | `https://gen.nomadkaraoke.com/lyrics/` |
| `POLL_INTERVAL` | Seconds between status polls | `5` |
**Note:** The `REVIEW_UI_URL` defaults to the hosted lyrics review UI. For local development, set it to `http://localhost:5173` if you're running the frontend dev server.
### Authentication
The backend uses token-based authentication for admin operations (bulk delete, internal worker triggers). For basic job submission and monitoring, authentication is optional.
**For admin access:**
```bash
export KARAOKE_GEN_AUTH_TOKEN="your-admin-token"
```
The token must match one of the tokens configured in the backend's `ADMIN_TOKENS` environment variable.
### Non-Interactive Mode
For automated/CI usage:
```bash
karaoke-gen-remote -y ./song.flac "Artist" "Title"
```
The `-y` flag auto-accepts default corrections and selects clean instrumental.
---
## 🎨 Style Configuration
Create a `styles.json` file to customize the karaoke video appearance:
```json
{
"intro": {
"video_duration": 5,
"background_image": "/path/to/title-background.png",
"font": "/path/to/Font.ttf",
"artist_color": "#ffdf6b",
"title_color": "#ffffff"
},
"karaoke": {
"background_image": "/path/to/karaoke-background.png",
"font_path": "/path/to/Font.ttf"
},
"end": {
"background_image": "/path/to/end-background.png"
},
"cdg": {
"font_path": "/path/to/Font.ttf",
"instrumental_background": "/path/to/cdg-background.png"
}
}
```
When using `karaoke-gen-remote`, all referenced files are automatically uploaded with your job.
---
## 📤 Output Files
A completed job produces:
```
BRAND-1234 - Artist - Title/
├── Artist - Title (Final Karaoke Lossless 4k).mp4 # ProRes 4K
├── Artist - Title (Final Karaoke Lossless 4k).mkv # FLAC audio 4K
├── Artist - Title (Final Karaoke Lossy 4k).mp4 # H.264 4K
├── Artist - Title (Final Karaoke Lossy 720p).mp4 # H.264 720p
├── Artist - Title (Final Karaoke CDG).zip # CDG+MP3 package
├── Artist - Title (Final Karaoke TXT).zip # TXT+MP3 package
├── Artist - Title (Karaoke).cdg # Individual CDG
├── Artist - Title (Karaoke).mp3 # Karaoke audio
├── Artist - Title (Karaoke).lrc # LRC lyrics
├── Artist - Title (Karaoke).ass # ASS subtitles
├── Artist - Title (Title).mov # Title screen video
├── Artist - Title (End).mov # End screen video
├── Artist - Title (Instrumental...).flac # Clean instrumental
├── Artist - Title (Instrumental +BV...).flac # With backing vocals
└── stems/ # All audio stems
├── ...Vocals....flac
├── ...Bass....flac
├── ...Drums....flac
└── ...
```
---
## 🏗️ Deploy Your Own Backend
The cloud backend runs on Google Cloud Platform using:
- **Cloud Run**: Serverless API hosting
- **Firestore**: Job state management
- **Cloud Storage**: File uploads and outputs
- **Modal.com**: GPU-accelerated audio separation
- **AudioShake**: Lyrics transcription API
### Prerequisites
- Google Cloud account with billing enabled
- [Pulumi CLI](https://www.pulumi.com/docs/install/)
- Modal.com account (for audio separation)
- AudioShake API key
### Infrastructure Setup
```bash
cd infrastructure
# Install dependencies
pip install -r requirements.txt
# Login to Pulumi
pulumi login
# Create a stack
pulumi stack init prod
# Configure GCP project
pulumi config set gcp:project your-project-id
pulumi config set gcp:region us-central1
# Deploy infrastructure
pulumi up
```
This creates:
- Firestore database
- Cloud Storage bucket
- Artifact Registry
- Service account with IAM roles
- Secret Manager secrets (you add values)
### Add Secret Values
```bash
# AudioShake API key
echo -n "your-audioshake-key" | gcloud secrets versions add audioshake-api-key --data-file=-
# Genius API key
echo -n "your-genius-key" | gcloud secrets versions add genius-api-key --data-file=-
# Modal API URL
echo -n "https://your-modal-url" | gcloud secrets versions add audio-separator-api-url --data-file=-
# YouTube OAuth credentials (JSON)
gcloud secrets versions add youtube-oauth-credentials --data-file=./youtube-creds.json
# Dropbox OAuth credentials (JSON)
gcloud secrets versions add dropbox-oauth-credentials --data-file=./dropbox-creds.json
# Google Drive service account (JSON)
gcloud secrets versions add gdrive-service-account --data-file=./gdrive-sa.json
```
### Deploy Cloud Run
```bash
# Build and deploy
gcloud builds submit --config=cloudbuild.yaml
# Get outputs from Pulumi
SA_EMAIL=$(pulumi stack output service_account_email)
BUCKET_NAME=$(pulumi stack output bucket_name)
# Deploy Cloud Run service
gcloud run deploy karaoke-backend \
--image us-central1-docker.pkg.dev/YOUR-PROJECT/karaoke-repo/karaoke-backend:latest \
--platform managed \
--region us-central1 \
--allow-unauthenticated \
--service-account $SA_EMAIL \
--memory 2Gi \
--cpu 2 \
--timeout 600 \
--set-env-vars="GOOGLE_CLOUD_PROJECT=YOUR-PROJECT,GCS_BUCKET_NAME=$BUCKET_NAME"
```
### Point CLI to Your Backend
```bash
export KARAOKE_GEN_URL="https://your-backend.run.app"
karaoke-gen-remote ./song.flac "Artist" "Title"
```
---
## 🔌 Backend API Reference
The backend exposes a REST API for job management.
### Job Submission
**POST** `/api/jobs/upload`
Submit a new karaoke generation job with audio file and options.
```bash
curl -X POST "https://api.example.com/api/jobs/upload" \
-F "file=@song.flac" \
-F "artist=ABBA" \
-F "title=Waterloo" \
-F "enable_cdg=true" \
-F "enable_txt=true" \
-F "brand_prefix=NOMAD" \
-F "style_params=@styles.json" \
-F "style_karaoke_background=@background.png"
```
### Job Status
**GET** `/api/jobs/{job_id}`
Get job status and details.
```bash
curl "https://api.example.com/api/jobs/abc12345"
```
### List Jobs
**GET** `/api/jobs`
List all jobs with optional status filter.
```bash
curl "https://api.example.com/api/jobs?status=complete&limit=10"
```
### Cancel Job
**POST** `/api/jobs/{job_id}/cancel`
Cancel a running job.
```bash
curl -X POST "https://api.example.com/api/jobs/abc12345/cancel" \
-H "Content-Type: application/json" \
-d '{"reason": "User cancelled"}'
```
### Delete Job
**DELETE** `/api/jobs/{job_id}`
Delete a job and its files.
```bash
curl -X DELETE "https://api.example.com/api/jobs/abc12345?delete_files=true"
```
### Lyrics Review
**GET** `/api/review/{job_id}/correction-data`
Get correction data for lyrics review.
**POST** `/api/review/{job_id}/complete`
Submit corrected lyrics and trigger video rendering.
### Instrumental Selection
**GET** `/api/jobs/{job_id}/instrumental-options`
Get available instrumental options.
**POST** `/api/jobs/{job_id}/select-instrumental`
Submit instrumental selection (clean or with_backing).
```bash
curl -X POST "https://api.example.com/api/jobs/abc12345/select-instrumental" \
-H "Content-Type: application/json" \
-d '{"selection": "clean"}'
```
### Download Files
**GET** `/api/jobs/{job_id}/download-urls`
Get download URLs for all output files.
**GET** `/api/jobs/{job_id}/download/{category}/{file_key}`
Stream download a specific file.
### Health Check
**GET** `/api/health`
Check backend health status.
---
## 🔧 Troubleshooting
### "No suitable files found for processing"
This error occurs during the finalisation step when the `(With Vocals).mkv` file is missing. This file is created during lyrics transcription.
**Most common cause:** No transcription provider configured.
**Quick fix:**
1. Check if transcription providers are configured:
```bash
echo $AUDIOSHAKE_API_TOKEN
echo $RUNPOD_API_KEY
```
2. If both are empty, set up a provider (see [Transcription Provider Setup](#transcription-provider-setup))
3. Or use `--skip-lyrics` for instrumental-only karaoke:
```bash
karaoke-gen --skip-lyrics "Artist" "Title"
```
**Other causes:**
- Invalid API credentials - verify your tokens are correct and active
- API service unavailable - check service status pages
- Network connectivity issues - ensure you can reach the API endpoints
- Transcription timeout - try again or use a different provider
### Transcription Fails Silently
If karaoke-gen runs without errors but produces no synchronized lyrics:
1. **Check logs** - Run with `--log_level debug` for detailed output:
```bash
karaoke-gen --log_level debug "Artist" "Title"
```
2. **Verify environment variables** - Ensure API tokens are exported in your shell:
```bash
# Check if set
printenv | grep -E "(AUDIOSHAKE|RUNPOD|WHISPER)"
# Set in current session
export AUDIOSHAKE_API_TOKEN="your_token"
```
3. **Test API connectivity** - Verify you can reach the transcription service
### "No lyrics found from any source"
This warning means no reference lyrics were fetched from online sources (Genius, Spotify, Musixmatch). The transcription will still work, but auto-correction may be less accurate.
**To fix:**
- Set `GENIUS_API_TOKEN` for Genius lyrics
- Set `SPOTIFY_COOKIE_SP_DC` for Spotify lyrics
- Set `RAPIDAPI_KEY` for Musixmatch lyrics
- Or provide lyrics manually with `--lyrics_file /path/to/lyrics.txt`
### Video Quality Issues
If the output video has quality problems:
- Ensure FFmpeg is properly installed: `ffmpeg -version`
- Check available codecs: `ffmpeg -codecs`
- For 4K output, ensure sufficient disk space (10GB+ per track)
### Local Whisper Issues
#### GPU Out of Memory
If you get CUDA out of memory errors:
```bash
# Use a smaller model
export WHISPER_MODEL_SIZE="small" # or "tiny"
# Or force CPU mode
export WHISPER_DEVICE="cpu"
```
#### Slow Transcription on CPU
CPU transcription is significantly slower than GPU. For faster processing:
- Use a smaller model (`tiny` or `base`)
- Consider using cloud transcription (AudioShake or RunPod)
- On Apple Silicon, the `small` model offers good speed/quality balance
#### Model Download Issues
Whisper models are downloaded on first use (~1-3GB depending on size). If downloads fail:
- Check your internet connection
- Set a custom cache directory: `export WHISPER_CACHE_DIR="/path/with/space"`
- Models are cached in `~/.cache/whisper/` by default
#### whisper-timestamped Not Found
If you get "whisper-timestamped is not installed":
```bash
pip install "karaoke-gen[local-whisper]"
# Or install directly:
pip install whisper-timestamped
```
#### Disabling Local Whisper
If you want to disable local Whisper (e.g., to force cloud transcription):
```bash
export ENABLE_LOCAL_WHISPER="false"
```
---
## 🧪 Development
### Running Tests
```bash
# Run all tests
pytest tests/ backend/tests/ -v
# Run only unit tests
pytest tests/unit/ -v
# Run with coverage
pytest tests/unit/ -v --cov=karaoke_gen --cov-report=term-missing
```
### Project Structure
```
karaoke-gen/
├── karaoke_gen/ # Core CLI package
│ ├── utils/
│ │ ├── gen_cli.py # Local CLI (karaoke-gen)
│ │ └── remote_cli.py # Remote CLI (karaoke-gen-remote)
│ ├── karaoke_finalise/ # Video encoding, packaging, distribution
│ └── style_loader.py # Unified style configuration
├── backend/ # Cloud backend (FastAPI)
│ ├── api/routes/ # API endpoints
│ ├── workers/ # Background processing workers
│ └── services/ # Business logic services
├── infrastructure/ # Pulumi IaC for GCP
├── docs/ # Documentation
└── tests/ # Test suite
```
---
## 📄 License
MIT
---
## 🤝 Contributing
Contributions are welcome! Please see our contributing guidelines.
File renamed without changes.
-43
View File
@@ -1,43 +0,0 @@
# Fast application image - uses pre-built base with all dependencies
#
# The base image (karaoke-backend-base) contains:
# - Python 3.11, ffmpeg, sox, system deps
# - All pip dependencies from requirements-frozen.txt (NO code)
#
# This Dockerfile adds the actual application code on top.
# Build time: ~1-2 minutes (deps already cached in base)
#
# If the base image doesn't exist yet, build it first:
# gcloud builds submit --config=cloudbuild-base.yaml --project=nomadkaraoke
# Use pre-built base image with all dependencies
# Override with --build-arg BASE_IMAGE=karaoke-backend-base:local for local testing
ARG BASE_IMAGE=us-central1-docker.pkg.dev/nomadkaraoke/karaoke-repo/karaoke-backend-base:latest
FROM ${BASE_IMAGE}
# Set working directory (should already exist from base)
WORKDIR /app
# Copy backend code (this is the only thing that changes frequently)
COPY backend /app/backend
# Copy and reinstall karaoke_gen from local source
# This ensures new modules (like style_loader.py) are included without rebuilding base image
# Note: lyrics_transcriber is vendored in lyrics_transcriber_temp/ and bundled via pyproject.toml
COPY karaoke_gen /app/karaoke_gen
COPY lyrics_transcriber_temp /app/lyrics_transcriber_temp
COPY pyproject.toml README.md LICENSE /app/
RUN pip install --no-cache-dir -e .
# Set environment variables
ENV PYTHONUNBUFFERED=1
ENV PORT=8080
# Expose port
EXPOSE 8080
# Run uvicorn directly
# Note: Transmission daemon removed - torrent downloads now handled by
# dedicated flacfetch VM service (see docs/00-current-plan/TORRENT-SERVICE-PLAN.md)
# Using --loop asyncio (instead of uvloop) to support nest_asyncio for remote flacfetch calls
CMD ["uvicorn", "backend.main:app", "--host", "0.0.0.0", "--port", "8080", "--loop", "asyncio"]
-45
View File
@@ -1,45 +0,0 @@
# Base image with all Python dependencies pre-installed
# This image is rebuilt only when poetry.lock changes
#
# Build with: gcloud builds submit --config=cloudbuild-base.yaml
# Or manually: docker build -f backend/Dockerfile.base -t karaoke-backend-base .
FROM python:3.11-slim
# Install system dependencies required by karaoke-gen
# Note: Transmission daemon removed - torrent downloads now handled by
# dedicated flacfetch VM service (see docs/00-current-plan/TORRENT-SERVICE-PLAN.md)
RUN apt-get update && apt-get install -y \
ffmpeg \
libsndfile1 \
libsox-dev \
sox \
build-essential \
curl \
&& rm -rf /var/lib/apt/lists/*
# Set working directory
WORKDIR /app
# Copy dependency files and source code needed for pip install
COPY pyproject.toml README.md LICENSE /app/
COPY karaoke_gen /app/karaoke_gen
COPY lyrics_transcriber_temp /app/lyrics_transcriber_temp
# Install all Python dependencies (the slow part - cached in base image)
RUN pip install --no-cache-dir --upgrade pip && \
pip install --no-cache-dir -e .
# Download spaCy language model for lyrics transcription
# This must be done at build time, not runtime (Cloud Run network restrictions)
RUN python -m spacy download en_core_web_sm
# Label with build info
ARG BUILD_DATE
ARG BUILD_ID
ARG BASE_CONTENT_HASH
LABEL org.opencontainers.image.created="${BUILD_DATE}"
LABEL org.opencontainers.image.revision="${BUILD_ID}"
LABEL org.opencontainers.image.title="karaoke-backend-base"
LABEL org.opencontainers.image.description="Base image with Python dependencies and system tools for karaoke-backend"
LABEL base.content.hash="${BASE_CONTENT_HASH}"
-242
View File
@@ -1,242 +0,0 @@
# Backend Development Setup
## One-Time Setup
### 1. Create Virtual Environment
```bash
cd /Users/andrew/Projects/karaoke-gen
python3 -m venv backend/venv
```
### 2. Activate Virtual Environment
```bash
source backend/venv/bin/activate
```
### 3. Install Dependencies
**IMPORTANT:** Use Python 3.12 (not 3.13/3.14 - pydantic has compatibility issues)
```bash
# Backend dependencies
pip install -r backend/requirements.txt
# Install karaoke_gen in development mode
pip install -e .
# Install lyrics_transcriber submodule
pip install -e lyrics_transcriber_local/
# Install Google API client (for YouTube upload)
pip install google-api-python-client google-auth-httplib2 google-auth-oauthlib
```
### 4. Add Shell Alias (Optional)
Add to `~/.zshrc`:
```bash
alias backend='cd /Users/andrew/Projects/karaoke-gen && source backend/venv/bin/activate'
```
Then you can just run:
```bash
backend
# Now you're in the project directory with venv activated!
```
---
## Daily Workflow
### Before Making Changes
```bash
# Activate venv
source backend/venv/bin/activate
# Or if you set up the alias:
backend
```
### After Making Changes
```bash
# 1. Run full validation (catches import errors!)
python3 backend/validate.py
# 2. If validation passes, deploy
./scripts/deploy.sh
```
---
## Validation Options
### Full Validation (Recommended - Requires venv)
```bash
source backend/venv/bin/activate
python3 backend/validate.py
```
**Catches:**
- ✅ Syntax errors
- ✅ **Import errors** (like the processing_service issue!)
- ✅ Configuration issues
- ✅ FastAPI app creation problems
- ✅ Missing dependencies
**This would have caught the ModuleNotFoundError!**
### Quick Check (No venv needed)
```bash
./backend/quick-check.sh
```
**Catches:**
- ✅ Syntax errors
- ✅ Missing files
- ✅ Some obvious import issues
**Good for quick checks, but won't catch all import errors**
---
## Example: Complete Development Session
```bash
# 1. Activate environment
backend # or: source backend/venv/bin/activate
# 2. Make your changes
vim backend/api/routes/jobs.py
# 3. Validate (catches import errors!)
python3 backend/validate.py
# ✅ All validations passed!
# 4. Deploy
./scripts/deploy.sh
# 5. Test
export BACKEND_URL="https://karaoke-backend-ipzqd2k4yq-uc.a.run.app"
export AUTH_TOKEN=$(gcloud auth print-identity-token)
curl -H "Authorization: Bearer $AUTH_TOKEN" $BACKEND_URL/api/health
```
---
## What About karaoke_gen Imports?
The backend imports from `karaoke_gen` (the CLI package):
```python
from karaoke_gen.karaoke_gen import KaraokePrep
from karaoke_gen.audio_processor import AudioProcessor
```
**Options:**
### Option 1: Install in development mode (Recommended)
```bash
source backend/venv/bin/activate
pip install -e .
```
This makes `karaoke_gen` available in the venv without copying files.
### Option 2: Use system Python
Your system Python already has `karaoke_gen` installed, so validation will work even if the venv doesn't have it (Python will fall back to system packages).
---
## Running the Backend Locally
```bash
# Activate venv
source backend/venv/bin/activate
# Set required environment variables
export GOOGLE_CLOUD_PROJECT="nomadkaraoke"
export GCS_BUCKET_NAME="karaoke-gen-storage-nomadkaraoke"
# Optional: Point to local credentials
export GOOGLE_APPLICATION_CREDENTIALS="/path/to/service-account-key.json"
# Run the server
cd backend
uvicorn main:app --reload --port 8080
```
Visit: http://localhost:8080/api/health
---
## Troubleshooting
### "No module named 'fastapi'"
**Fix:**
```bash
source backend/venv/bin/activate
pip install -r backend/requirements.txt
```
### "No module named 'karaoke_gen'"
**Fix:**
```bash
source backend/venv/bin/activate
cd /Users/andrew/Projects/karaoke-gen
pip install -e .
```
### Validation passes locally but fails in Cloud Run
**Check:**
1. Is `requirements.txt` up to date?
2. Did you add a new dependency without adding it to `requirements.txt`?
3. Run: `pip freeze | grep <package-name>`
---
## Updating Dependencies
When you add new packages:
```bash
# 1. Install in venv
source backend/venv/bin/activate
pip install new-package
# 2. Update requirements.txt
pip freeze > backend/requirements.txt
# 3. Or manually add just the new package:
echo "new-package==1.2.3" >> backend/requirements.txt
# 4. Validate
python3 backend/validate.py
# 5. Deploy
./scripts/deploy.sh
```
---
## Summary
**Best practice workflow:**
1. **One-time setup:** Create venv and install dependencies
2. **Daily:** Activate venv when working on backend
3. **Before deploy:** Run `python3 backend/validate.py` (catches import errors!)
4. **Deploy:** Use `./scripts/deploy.sh`
**The import error would have been caught if you ran validation in a venv!** ✅
View File
Whitespace-only changes.
View File
Whitespace-only changes.
-345
View File
@@ -1,345 +0,0 @@
"""
FastAPI dependencies for authentication and authorization.
"""
import logging
import secrets
from datetime import datetime, timedelta, UTC
from typing import Optional, Tuple, Callable
from fastapi import Depends, HTTPException, Header, Query, Request, Path
from fastapi.security import HTTPBearer, HTTPAuthorizationCredentials
from backend.services.auth_service import get_auth_service, UserType, AuthService
logger = logging.getLogger(__name__)
def generate_review_token() -> str:
"""Generate a cryptographically secure review token."""
return secrets.token_urlsafe(32)
def get_review_token_expiry(hours: int = 24) -> datetime:
"""Get expiry time for a review token."""
return datetime.now(UTC) + timedelta(hours=hours)
# HTTP Bearer security scheme (Authorization: Bearer <token>)
security = HTTPBearer(auto_error=False)
async def get_token_from_request(
request: Request,
credentials: Optional[HTTPAuthorizationCredentials] = Depends(security),
token: Optional[str] = Query(None, description="Access token (alternative to Bearer header)")
) -> Optional[str]:
"""
Extract access token from request.
Supports three methods (in priority order):
1. X-Admin-Token header: Used by Cloud Tasks (since OIDC overwrites Authorization)
2. Authorization header: Authorization: Bearer <token>
3. Query parameter: ?token=<token> (for download links)
Returns:
Token string or None
"""
# Check X-Admin-Token header first (used by Cloud Tasks)
# Cloud Tasks OIDC token overwrites Authorization header, so we use a custom header
admin_token = request.headers.get("X-Admin-Token")
if admin_token:
return admin_token
# Try Authorization header
if credentials:
return credentials.credentials
# Try query parameter
if token:
return token
return None
async def require_auth(
request: Request,
auth_service: AuthService = Depends(get_auth_service),
credentials: Optional[HTTPAuthorizationCredentials] = Depends(security),
token: Optional[str] = Query(None)
) -> Tuple[str, UserType, int]:
"""
Require authentication for an endpoint.
Returns:
(token, user_type, remaining_uses)
Raises:
HTTPException: 401 if authentication fails
"""
# Get token from request
token_str = await get_token_from_request(request, credentials, token)
if not token_str:
raise HTTPException(
status_code=401,
detail="Authentication required. Provide token via Authorization header or ?token= parameter"
)
# Validate token
is_valid, user_type, remaining_uses, message = auth_service.validate_token(token_str)
if not is_valid:
# Log more details for debugging token issues
auth_header = request.headers.get("Authorization", "")
logger.warning(
f"Authentication failed: {message}. "
f"Token provided: {bool(token_str)}, "
f"Token length: {len(token_str) if token_str else 0}, "
f"Auth header present: {bool(auth_header)}, "
f"Header prefix: {auth_header[:20] if auth_header else 'none'}..."
)
raise HTTPException(
status_code=401,
detail=f"Authentication failed: {message}"
)
logger.info(f"Authenticated as {user_type} (remaining: {remaining_uses})")
return token_str, user_type, remaining_uses
async def require_admin(
auth_data: Tuple[str, UserType, int] = Depends(require_auth)
) -> Tuple[str, UserType, int]:
"""
Require admin access for an endpoint.
Returns:
(token, user_type, remaining_uses)
Raises:
HTTPException: 403 if user is not admin
"""
token, user_type, remaining_uses = auth_data
if user_type != UserType.ADMIN:
logger.warning(f"Admin access denied for {user_type} user")
raise HTTPException(
status_code=403,
detail="Admin access required"
)
return token, user_type, remaining_uses
def optional_auth(
auth_service: AuthService = Depends(get_auth_service),
credentials: Optional[HTTPAuthorizationCredentials] = Depends(security),
token: Optional[str] = Query(None)
) -> Optional[Tuple[str, UserType, int]]:
"""
Optional authentication - doesn't fail if no token provided.
Useful for endpoints that have different behavior for authenticated users.
Returns:
(token, user_type, remaining_uses) if authenticated, None otherwise
"""
# Get token
token_str = None
if credentials:
token_str = credentials.credentials
elif token:
token_str = token
if not token_str:
return None
# Validate token
is_valid, user_type, remaining_uses, message = auth_service.validate_token(token_str)
if not is_valid:
logger.debug(f"Optional auth failed: {message}")
return None
return token_str, user_type, remaining_uses
def require_review_auth_factory(job_id_param: str = "job_id"):
"""
Factory to create a review authentication dependency.
Accepts either:
1. Full user authentication (admin/user token)
2. Job-specific review token (only valid for the specific job)
Args:
job_id_param: Name of the path parameter containing the job ID
Returns:
Dependency function that validates review access
"""
async def require_review_auth(
request: Request,
job_id: str = Path(...),
review_token: Optional[str] = Query(None, description="Job-specific review token"),
auth_service: AuthService = Depends(get_auth_service),
credentials: Optional[HTTPAuthorizationCredentials] = Depends(security),
token: Optional[str] = Query(None, alias="token", description="Full access token")
) -> Tuple[str, str]:
"""
Validate review access for a job.
Returns:
(job_id, auth_type) where auth_type is "full" or "review_token"
Raises:
HTTPException: 401 if authentication fails
"""
# Import here to avoid circular dependency
from backend.services.job_manager import JobManager
# Try full authentication first
full_token = None
if credentials:
full_token = credentials.credentials
elif token:
full_token = token
if full_token:
is_valid, user_type, remaining_uses, message = auth_service.validate_token(full_token)
if is_valid:
logger.info(f"Review access granted via full auth ({user_type}) for job {job_id}")
return job_id, "full"
# Try review token
if review_token:
job_manager = JobManager()
job = job_manager.get_job(job_id)
if not job:
raise HTTPException(status_code=404, detail=f"Job {job_id} not found")
# Validate review token matches
if job.review_token and secrets.compare_digest(job.review_token, review_token):
# Check expiry if set
if job.review_token_expires_at:
now = datetime.now(UTC)
# Handle timezone-naive datetimes from Firestore
expiry = job.review_token_expires_at
if expiry.tzinfo is None:
expiry = expiry.replace(tzinfo=UTC)
if now > expiry:
logger.warning(f"Review token expired for job {job_id}")
raise HTTPException(
status_code=401,
detail="Review token has expired. Please request a new review link."
)
logger.info(f"Review access granted via review_token for job {job_id}")
return job_id, "review_token"
else:
logger.warning(f"Invalid review token for job {job_id}")
# No valid authentication
raise HTTPException(
status_code=401,
detail="Authentication required. Provide either a full access token or a valid review_token for this job."
)
return require_review_auth
# Default instance for most review endpoints
require_review_auth = require_review_auth_factory()
def require_instrumental_auth_factory(job_id_param: str = "job_id"):
"""
Factory to create an instrumental review authentication dependency.
Accepts either:
1. Full user authentication (admin/user token)
2. Job-specific instrumental token (only valid for the specific job)
Args:
job_id_param: Name of the path parameter containing the job ID
Returns:
Dependency function that validates instrumental review access
"""
async def require_instrumental_auth(
request: Request,
job_id: str = Path(...),
instrumental_token: Optional[str] = Query(None, description="Job-specific instrumental review token"),
auth_service: AuthService = Depends(get_auth_service),
credentials: Optional[HTTPAuthorizationCredentials] = Depends(security),
token: Optional[str] = Query(None, alias="token", description="Full access token")
) -> Tuple[str, str]:
"""
Validate instrumental review access for a job.
Returns:
(job_id, auth_type) where auth_type is "full" or "instrumental_token"
Raises:
HTTPException: 401 if authentication fails
"""
# Import here to avoid circular dependency
from backend.services.job_manager import JobManager
# Try full authentication first
full_token = None
if credentials:
full_token = credentials.credentials
elif token:
full_token = token
if full_token:
is_valid, user_type, remaining_uses, message = auth_service.validate_token(full_token)
if is_valid:
logger.info(f"Instrumental access granted via full auth ({user_type}) for job {job_id}")
return job_id, "full"
# Try instrumental token
if instrumental_token:
job_manager = JobManager()
job = job_manager.get_job(job_id)
if not job:
raise HTTPException(status_code=404, detail=f"Job {job_id} not found")
# Validate instrumental token matches
if job.instrumental_token and secrets.compare_digest(job.instrumental_token, instrumental_token):
# Check expiry if set
if job.instrumental_token_expires_at:
now = datetime.now(UTC)
# Handle timezone-naive datetimes from Firestore
expiry = job.instrumental_token_expires_at
if expiry.tzinfo is None:
expiry = expiry.replace(tzinfo=UTC)
if now > expiry:
logger.warning(f"Instrumental token expired for job {job_id}")
raise HTTPException(
status_code=401,
detail="Instrumental review token has expired. Please request a new review link."
)
logger.info(f"Instrumental access granted via instrumental_token for job {job_id}")
return job_id, "instrumental_token"
else:
logger.warning(f"Invalid instrumental token for job {job_id}")
# No valid authentication
raise HTTPException(
status_code=401,
detail="Authentication required. Provide either a full access token or a valid instrumental_token for this job."
)
return require_instrumental_auth
# Default instance for most instrumental review endpoints
require_instrumental_auth = require_instrumental_auth_factory()
View File
Whitespace-only changes.
-853
View File
@@ -1,853 +0,0 @@
"""
Audio search API routes for artist+title search mode.
This module provides endpoints for:
1. Creating a job with audio search (artist+title without file)
2. Getting search results for a job
3. Selecting an audio source to download
The flow is:
1. POST /api/audio-search/search - Create job and search for audio
2. GET /api/audio-search/{job_id}/results - Get search results
3. POST /api/audio-search/{job_id}/select - Select audio source
"""
import asyncio
import logging
import os
import tempfile
from typing import Optional, List, Dict, Any, Tuple
from fastapi import APIRouter, HTTPException, BackgroundTasks, Request, Depends
from pydantic import BaseModel, Field
from backend.models.job import JobCreate, JobStatus
from backend.services.job_manager import JobManager
from backend.services.storage_service import StorageService
from backend.services.worker_service import get_worker_service
from backend.services.credential_manager import get_credential_manager, CredentialStatus
from backend.services.audio_search_service import (
get_audio_search_service,
AudioSearchResult,
AudioSearchError,
NoResultsError,
DownloadError,
)
from backend.config import get_settings
from backend.version import VERSION
from backend.api.dependencies import require_auth
from backend.services.auth_service import UserType
from pathlib import Path
logger = logging.getLogger(__name__)
# Valid style file types (from file_upload.py)
STYLE_FILE_TYPES = {
'style_params': {'.json'},
'style_intro_background': {'.jpg', '.jpeg', '.png'},
'style_karaoke_background': {'.jpg', '.jpeg', '.png'},
'style_end_background': {'.jpg', '.jpeg', '.png'},
'style_font': {'.ttf', '.otf'},
'style_cdg_instrumental_background': {'.jpg', '.jpeg', '.png'},
'style_cdg_title_background': {'.jpg', '.jpeg', '.png'},
'style_cdg_outro_background': {'.jpg', '.jpeg', '.png'},
}
router = APIRouter(tags=["audio-search"])
# Initialize services
job_manager = JobManager()
storage_service = StorageService()
worker_service = get_worker_service()
# ============================================================================
# Pydantic models
# ============================================================================
class StyleFileRequest(BaseModel):
"""Information about a style file to be uploaded."""
filename: str = Field(..., description="Original filename with extension")
content_type: str = Field(..., description="MIME type of the file")
file_type: str = Field(..., description="Type of file: 'style_params', 'style_intro_background', 'style_karaoke_background', 'style_end_background', 'style_font', 'style_cdg_instrumental_background', 'style_cdg_title_background', 'style_cdg_outro_background'")
class StyleUploadUrl(BaseModel):
"""Signed URL for uploading a style file."""
file_type: str = Field(..., description="Type of file being uploaded")
gcs_path: str = Field(..., description="Destination path in GCS")
upload_url: str = Field(..., description="Signed URL to PUT the file to")
class AudioSearchRequest(BaseModel):
"""Request to search for audio by artist and title."""
artist: str = Field(..., description="Artist name to search for")
title: str = Field(..., description="Song title to search for")
# Auto-download mode
auto_download: bool = Field(False, description="Automatically select best result and download")
# Theme configuration
theme_id: Optional[str] = Field(None, description="Theme ID to use (e.g., 'nomad', 'default'). If set, CDG/TXT are enabled by default.")
color_overrides: Optional[Dict[str, str]] = Field(None, description="Color overrides: artist_color, title_color, sung_lyrics_color, unsung_lyrics_color (hex #RRGGBB)")
# Processing options (CDG/TXT require style config or theme, disabled by default unless theme is set)
enable_cdg: Optional[bool] = Field(None, description="Generate CDG+MP3 package. Default: True if theme_id set, False otherwise")
enable_txt: Optional[bool] = Field(None, description="Generate TXT+MP3 package. Default: True if theme_id set, False otherwise")
# Finalisation options
brand_prefix: Optional[str] = Field(None, description="Brand code prefix (e.g., NOMAD)")
enable_youtube_upload: bool = Field(False, description="Upload to YouTube")
youtube_description: Optional[str] = Field(None, description="YouTube video description text")
discord_webhook_url: Optional[str] = Field(None, description="Discord webhook URL for notifications")
# Distribution options
dropbox_path: Optional[str] = Field(None, description="Dropbox folder path for organized output")
gdrive_folder_id: Optional[str] = Field(None, description="Google Drive folder ID for public share uploads")
# Lyrics configuration
lyrics_artist: Optional[str] = Field(None, description="Override artist name for lyrics search")
lyrics_title: Optional[str] = Field(None, description="Override title for lyrics search")
subtitle_offset_ms: int = Field(0, description="Subtitle timing offset in milliseconds")
# Audio separation model configuration
clean_instrumental_model: Optional[str] = Field(None, description="Model for clean instrumental separation")
backing_vocals_models: Optional[List[str]] = Field(None, description="Models for backing vocals separation")
other_stems_models: Optional[List[str]] = Field(None, description="Models for other stems")
# Style file uploads (optional)
# Style params JSON and related assets (background images, fonts)
style_files: Optional[List[StyleFileRequest]] = Field(None, description="Style files to upload (style_params JSON, backgrounds, fonts)")
class AudioSearchResultResponse(BaseModel):
"""A single audio search result.
Contains all fields needed for rich display using flacfetch's
shared formatting functions.
"""
index: int
title: str
artist: str
provider: str # Maps to source_name in Release
url: Optional[str] = None # Maps to download_url in Release (may be None for remote)
duration: Optional[int] = None # Maps to duration_seconds in Release
quality: Optional[str] = None # Stringified quality
source_id: Optional[str] = None # Maps to info_hash in Release
seeders: Optional[int] = None
target_file: Optional[str] = None
# Additional fields for rich display (from Release.to_dict())
year: Optional[int] = None
label: Optional[str] = None
edition_info: Optional[str] = None
release_type: Optional[str] = None
channel: Optional[str] = None # For YouTube
view_count: Optional[int] = None # For YouTube
size_bytes: Optional[int] = None
target_file_size: Optional[int] = None
track_pattern: Optional[str] = None
match_score: Optional[float] = None
# Pre-computed display fields
formatted_size: Optional[str] = None
formatted_duration: Optional[str] = None
formatted_views: Optional[str] = None
is_lossless: Optional[bool] = None
quality_str: Optional[str] = None
# Full quality object for Release.from_dict()
quality_data: Optional[Dict[str, Any]] = None
class AudioSearchResponse(BaseModel):
"""Response from audio search."""
status: str
job_id: str
message: str
results: Optional[List[AudioSearchResultResponse]] = None
results_count: int = 0
auto_download: bool = False
server_version: str
# Style file upload URLs (when style_files are specified in request)
style_upload_urls: Optional[List[StyleUploadUrl]] = None
class AudioSelectRequest(BaseModel):
"""Request to select an audio source."""
selection_index: int = Field(..., description="Index of the selected audio source from search results")
class AudioSelectResponse(BaseModel):
"""Response from audio source selection."""
status: str
job_id: str
message: str
selected_index: int
selected_title: str
selected_artist: str
selected_provider: str
def _resolve_cdg_txt_defaults(
theme_id: Optional[str],
enable_cdg: Optional[bool],
enable_txt: Optional[bool]
) -> Tuple[bool, bool]:
"""
Resolve CDG/TXT settings based on theme and explicit settings.
When a theme is selected, CDG and TXT are enabled by default.
Explicit True/False values always override the default.
Args:
theme_id: Theme identifier (if any)
enable_cdg: Explicit CDG setting (None means use default)
enable_txt: Explicit TXT setting (None means use default)
Returns:
Tuple of (resolved_enable_cdg, resolved_enable_txt)
"""
# Default based on whether theme is set
default_enabled = theme_id is not None
# Explicit values override defaults, None uses default
resolved_cdg = enable_cdg if enable_cdg is not None else default_enabled
resolved_txt = enable_txt if enable_txt is not None else default_enabled
return resolved_cdg, resolved_txt
def extract_request_metadata(request: Request, created_from: str = "audio_search") -> Dict[str, Any]:
"""Extract metadata from request for job tracking."""
headers = dict(request.headers)
client_ip = headers.get('x-forwarded-for', '').split(',')[0].strip()
if not client_ip and request.client:
client_ip = request.client.host
user_agent = headers.get('user-agent', '')
environment = headers.get('x-environment', '')
client_id = headers.get('x-client-id', '')
custom_headers = {}
for key, value in headers.items():
if key.lower().startswith('x-') and key.lower() not in ('x-forwarded-for', 'x-forwarded-proto', 'x-forwarded-host'):
custom_headers[key] = value
return {
'client_ip': client_ip,
'user_agent': user_agent,
'environment': environment,
'client_id': client_id,
'server_version': VERSION,
'created_from': created_from,
'custom_headers': custom_headers,
}
async def _trigger_workers_parallel(job_id: str) -> None:
"""Trigger both audio and lyrics workers in parallel."""
await asyncio.gather(
worker_service.trigger_audio_worker(job_id),
worker_service.trigger_lyrics_worker(job_id)
)
async def _download_and_start_processing(
job_id: str,
selection_index: int,
audio_search_service,
background_tasks: BackgroundTasks,
) -> Dict[str, Any]:
"""
Download selected audio and start job processing.
This is called either:
- Immediately after search if auto_download=True
- When user calls the select endpoint
For remote flacfetch downloads (torrent sources), the file is downloaded
on the flacfetch VM and uploaded directly to GCS. For local downloads
(YouTube), the file is downloaded locally and then uploaded to GCS.
"""
job = job_manager.get_job(job_id)
if not job:
raise HTTPException(status_code=404, detail=f"Job {job_id} not found")
# Get search results from state_data
search_results = job.state_data.get('audio_search_results', [])
if not search_results:
raise HTTPException(status_code=400, detail="No search results available")
if selection_index < 0 or selection_index >= len(search_results):
raise HTTPException(
status_code=400,
detail=f"Invalid selection index {selection_index}. Valid range: 0-{len(search_results)-1}"
)
selected = search_results[selection_index]
# Get remote_search_id from state_data (stored during initial search)
remote_search_id = job.state_data.get('remote_search_id')
# Transition to downloading state
job_manager.transition_to_state(
job_id=job_id,
new_status=JobStatus.DOWNLOADING_AUDIO,
progress=10,
message=f"Downloading from {selected['provider']}: {selected['artist']} - {selected['title']}",
state_data_updates={
'selected_audio_index': selection_index,
'selected_audio_provider': selected['provider'],
}
)
try:
# Determine if this is a remote torrent download
is_torrent_source = selected.get('provider') in ['RED', 'OPS']
is_remote_enabled = audio_search_service.is_remote_enabled()
# Check if we can use download_by_id (preferred - avoids re-searching)
source_id = selected.get('source_id')
source_name = selected.get('provider')
target_file = selected.get('target_file')
download_url = selected.get('url')
# For remote torrent downloads, have flacfetch VM upload directly to GCS
if is_torrent_source and is_remote_enabled:
# Generate GCS path for remote upload
gcs_destination = f"uploads/{job_id}/audio/"
# Use download_by_id if we have source_id (preferred - no re-search needed)
if source_id and source_name:
logger.info(f"Using download_by_id for {source_name} ID={source_id} with GCS upload to: {gcs_destination}")
result = audio_search_service.download_by_id(
source_name=source_name,
source_id=source_id,
output_dir="", # Not used for remote
target_file=target_file,
download_url=download_url,
gcs_path=gcs_destination,
)
else:
# Fallback to search-based download (requires re-search)
logger.info(f"No source_id available, falling back to search-based download to: {gcs_destination}")
result = audio_search_service.download(
result_index=selection_index,
output_dir="", # Not used for remote
gcs_path=gcs_destination,
remote_search_id=remote_search_id,
)
# For remote downloads, filepath is already the GCS path
if result.filepath.startswith("gs://"):
# Extract the path portion after the bucket name
# Format: gs://bucket/uploads/job_id/audio/filename.flac
parts = result.filepath.replace("gs://", "").split("/", 1)
if len(parts) == 2:
audio_gcs_path = parts[1]
else:
audio_gcs_path = result.filepath
filename = os.path.basename(result.filepath)
else:
# Fallback: treat as local path (shouldn't happen for remote)
filename = os.path.basename(result.filepath)
audio_gcs_path = f"uploads/{job_id}/audio/{filename}"
logger.warning(f"Remote download returned local path: {result.filepath}, uploading manually")
with open(result.filepath, 'rb') as f:
storage_service.upload_fileobj(f, audio_gcs_path, content_type='audio/flac')
logger.info(f"Remote download complete, GCS path: {audio_gcs_path}")
else:
# Local download (YouTube or fallback)
temp_dir = tempfile.mkdtemp(prefix=f"audio_download_{job_id}_")
# Use download_by_id if we have source_id and remote is enabled
if source_id and source_name and is_remote_enabled:
logger.info(f"Using download_by_id for local download: {source_name} ID={source_id}")
result = audio_search_service.download_by_id(
source_name=source_name,
source_id=source_id,
output_dir=temp_dir,
target_file=target_file,
download_url=download_url,
)
else:
# Fallback to search-based download
result = audio_search_service.download(
result_index=selection_index,
output_dir=temp_dir,
remote_search_id=remote_search_id, # Pass for potential fallback scenarios
)
# Upload to GCS
filename = os.path.basename(result.filepath)
audio_gcs_path = f"uploads/{job_id}/audio/{filename}"
with open(result.filepath, 'rb') as f:
storage_service.upload_fileobj(
f,
audio_gcs_path,
content_type='audio/flac' # flacfetch typically returns FLAC
)
logger.info(f"Uploaded audio to GCS: {audio_gcs_path}")
# Clean up temp file
try:
os.remove(result.filepath)
os.rmdir(temp_dir)
except Exception as e:
logger.warning(f"Failed to clean up temp files: {e}")
# Update job with GCS path and transition to DOWNLOADING
job_manager.update_job(job_id, {
'input_media_gcs_path': audio_gcs_path,
'filename': filename,
})
job_manager.transition_to_state(
job_id=job_id,
new_status=JobStatus.DOWNLOADING,
progress=15,
message="Audio downloaded, starting processing"
)
# Trigger workers
background_tasks.add_task(_trigger_workers_parallel, job_id)
return {
'selected_index': selection_index,
'selected_title': selected['title'],
'selected_artist': selected['artist'],
'selected_provider': selected['provider'],
}
except DownloadError as e:
job_manager.fail_job(job_id, f"Audio download failed: {e}")
raise HTTPException(status_code=500, detail=f"Download failed: {e}")
except Exception as e:
job_manager.fail_job(job_id, f"Audio download failed: {e}")
raise HTTPException(status_code=500, detail=f"Download failed: {e}")
@router.post("/audio-search/search", response_model=AudioSearchResponse)
async def search_audio(
request: Request,
background_tasks: BackgroundTasks,
body: AudioSearchRequest,
auth_data: Tuple[str, UserType, int] = Depends(require_auth)
):
"""
Search for audio by artist and title, creating a new job.
This endpoint:
1. Creates a job in PENDING state
2. Searches for audio using flacfetch
3. Either returns search results for user selection, or
4. If auto_download=True, automatically selects best and starts processing
Use cases:
- Interactive mode (default): Returns results, user calls /select endpoint
- Auto mode (auto_download=True): Automatically selects and downloads best
"""
try:
# Apply default distribution settings
settings = get_settings()
effective_dropbox_path = body.dropbox_path or settings.default_dropbox_path
effective_gdrive_folder_id = body.gdrive_folder_id or settings.default_gdrive_folder_id
effective_discord_webhook_url = body.discord_webhook_url or settings.default_discord_webhook_url
# Apply defaults for YouTube/Dropbox distribution (for web service)
effective_enable_youtube_upload = body.enable_youtube_upload or settings.default_enable_youtube_upload
effective_brand_prefix = body.brand_prefix or settings.default_brand_prefix
effective_youtube_description = body.youtube_description or settings.default_youtube_description
# Validate credentials if distribution services are requested
invalid_services = []
credential_manager = get_credential_manager()
if effective_enable_youtube_upload:
result = credential_manager.check_youtube_credentials()
if result.status != CredentialStatus.VALID:
invalid_services.append(f"youtube ({result.message})")
if effective_dropbox_path:
result = credential_manager.check_dropbox_credentials()
if result.status != CredentialStatus.VALID:
invalid_services.append(f"dropbox ({result.message})")
if effective_gdrive_folder_id:
result = credential_manager.check_gdrive_credentials()
if result.status != CredentialStatus.VALID:
invalid_services.append(f"gdrive ({result.message})")
if invalid_services:
raise HTTPException(
status_code=400,
detail={
"error": "credentials_invalid",
"message": f"Distribution services need re-authorization: {', '.join(invalid_services)}",
"invalid_services": invalid_services,
}
)
# Extract request metadata
request_metadata = extract_request_metadata(request, created_from="audio_search")
# Resolve CDG/TXT defaults based on theme
resolved_cdg, resolved_txt = _resolve_cdg_txt_defaults(
body.theme_id, body.enable_cdg, body.enable_txt
)
# Create job
job_create = JobCreate(
artist=body.artist,
title=body.title,
theme_id=body.theme_id,
color_overrides=body.color_overrides or {},
enable_cdg=resolved_cdg,
enable_txt=resolved_txt,
brand_prefix=effective_brand_prefix,
enable_youtube_upload=effective_enable_youtube_upload,
youtube_description=effective_youtube_description,
youtube_description_template=effective_youtube_description, # video_worker reads this field
discord_webhook_url=effective_discord_webhook_url,
dropbox_path=effective_dropbox_path,
gdrive_folder_id=effective_gdrive_folder_id,
lyrics_artist=body.lyrics_artist,
lyrics_title=body.lyrics_title,
subtitle_offset_ms=body.subtitle_offset_ms,
clean_instrumental_model=body.clean_instrumental_model,
backing_vocals_models=body.backing_vocals_models,
other_stems_models=body.other_stems_models,
audio_search_artist=body.artist,
audio_search_title=body.title,
auto_download=body.auto_download,
request_metadata=request_metadata,
)
job = job_manager.create_job(job_create)
job_id = job.job_id
logger.info(f"Created job {job_id} for audio search: {body.artist} - {body.title}")
# Update job with audio search fields
job_manager.update_job(job_id, {
'audio_search_artist': body.artist,
'audio_search_title': body.title,
'auto_download': body.auto_download,
})
# If theme is set and no custom style files are being uploaded, prepare theme style now
# This copies the theme's style_params.json to the job folder so LyricsTranscriber
# can access the style configuration for preview videos
if body.theme_id and not body.style_files:
from backend.api.routes.file_upload import _prepare_theme_for_job
try:
style_params_path, theme_style_assets, youtube_desc = _prepare_theme_for_job(
job_id, body.theme_id, body.color_overrides
)
theme_update = {
'style_params_gcs_path': style_params_path,
'style_assets': theme_style_assets,
}
if youtube_desc and not effective_youtube_description:
theme_update['youtube_description_template'] = youtube_desc
job_manager.update_job(job_id, theme_update)
logger.info(f"Applied theme '{body.theme_id}' to job {job_id}")
except Exception as e:
logger.warning(f"Failed to prepare theme '{body.theme_id}' for job {job_id}: {e}")
# Continue without theme - job can still be processed with defaults
# Handle style file uploads if provided
style_upload_urls: List[StyleUploadUrl] = []
style_assets = {}
if body.style_files:
logger.info(f"Processing {len(body.style_files)} style file uploads for job {job_id}")
for file_info in body.style_files:
# Validate file type
if file_info.file_type not in STYLE_FILE_TYPES:
raise HTTPException(
status_code=400,
detail=f"Invalid file type '{file_info.file_type}'. Must be one of: {', '.join(STYLE_FILE_TYPES.keys())}"
)
# Validate extension
ext = Path(file_info.filename).suffix.lower()
allowed_exts = STYLE_FILE_TYPES[file_info.file_type]
if ext not in allowed_exts:
raise HTTPException(
status_code=400,
detail=f"Invalid extension '{ext}' for {file_info.file_type}. Allowed: {', '.join(allowed_exts)}"
)
# Generate GCS path
if file_info.file_type == 'style_params':
gcs_path = f"uploads/{job_id}/style/style_params.json"
else:
# style_intro_background -> intro_background, etc.
asset_key = file_info.file_type.replace('style_', '')
gcs_path = f"uploads/{job_id}/style/{asset_key}{ext}"
# Generate signed upload URL
signed_url = storage_service.generate_signed_upload_url(
gcs_path,
content_type=file_info.content_type,
expiration_minutes=60
)
style_upload_urls.append(StyleUploadUrl(
file_type=file_info.file_type,
gcs_path=gcs_path,
upload_url=signed_url
))
# Track the expected asset paths
if file_info.file_type == 'style_params':
style_assets['style_params'] = gcs_path
else:
asset_key = file_info.file_type.replace('style_', '')
style_assets[asset_key] = gcs_path
# Update job with style asset expectations
style_update = {'style_assets': style_assets}
if 'style_params' in style_assets:
style_update['style_params_gcs_path'] = style_assets['style_params']
job_manager.update_job(job_id, style_update)
logger.info(f"Generated {len(style_upload_urls)} style upload URLs for job {job_id}")
# Transition to searching state
job_manager.transition_to_state(
job_id=job_id,
new_status=JobStatus.SEARCHING_AUDIO,
progress=5,
message=f"Searching for: {body.artist} - {body.title}"
)
# Perform search
audio_search_service = get_audio_search_service()
try:
search_results = audio_search_service.search(body.artist, body.title)
except NoResultsError as e:
job_manager.fail_job(job_id, f"No audio sources found for: {body.artist} - {body.title}")
raise HTTPException(
status_code=404,
detail={
"error": "no_results",
"message": str(e),
"job_id": job_id,
}
)
except AudioSearchError as e:
job_manager.fail_job(job_id, f"Audio search failed: {e}")
raise HTTPException(status_code=500, detail=f"Search failed: {e}")
# Store results in job state_data, including remote_search_id if available
results_dicts = [r.to_dict() for r in search_results]
state_data_update = {
'audio_search_results': results_dicts,
'audio_search_count': len(results_dicts),
}
# Store remote_search_id for use during download (important for concurrent requests)
if audio_search_service.last_remote_search_id:
state_data_update['remote_search_id'] = audio_search_service.last_remote_search_id
job_manager.update_job(job_id, {
'state_data': state_data_update
})
# If auto_download, select best and start processing
if body.auto_download:
best_index = audio_search_service.select_best(search_results)
logger.info(f"Auto-download enabled, selecting result {best_index}")
selection_info = await _download_and_start_processing(
job_id=job_id,
selection_index=best_index,
audio_search_service=audio_search_service,
background_tasks=background_tasks,
)
return AudioSearchResponse(
status="success",
job_id=job_id,
message=f"Audio found and download started: {selection_info['selected_artist']} - {selection_info['selected_title']} ({selection_info['selected_provider']})",
results=None, # Don't return results in auto mode
results_count=len(search_results),
auto_download=True,
server_version=VERSION,
style_upload_urls=style_upload_urls if style_upload_urls else None,
)
# Interactive mode: return results for user selection
job_manager.transition_to_state(
job_id=job_id,
new_status=JobStatus.AWAITING_AUDIO_SELECTION,
progress=10,
message=f"Found {len(search_results)} audio sources. Waiting for selection."
)
# Convert to response format with full Release data for rich display
result_responses = []
for r in search_results:
# Get full serialized data from raw_result if available
# raw_result can be either:
# - A dict (from remote flacfetch API)
# - A Release object (from local flacfetch)
raw_dict = {}
if r.raw_result:
if isinstance(r.raw_result, dict):
# Remote flacfetch API returns dicts directly
raw_dict = r.raw_result
else:
# Local flacfetch returns Release objects
try:
raw_dict = r.raw_result.to_dict()
except AttributeError:
pass # Not a Release object
result_responses.append(
AudioSearchResultResponse(
index=r.index,
title=r.title,
artist=r.artist,
provider=r.provider,
url=r.url,
duration=r.duration,
quality=r.quality,
source_id=r.source_id,
seeders=raw_dict.get('seeders') or r.seeders,
target_file=raw_dict.get('target_file') or r.target_file,
# Additional fields for rich display
year=raw_dict.get('year'),
label=raw_dict.get('label'),
edition_info=raw_dict.get('edition_info'),
release_type=raw_dict.get('release_type'),
channel=raw_dict.get('channel'),
view_count=raw_dict.get('view_count'),
size_bytes=raw_dict.get('size_bytes'),
target_file_size=raw_dict.get('target_file_size'),
track_pattern=raw_dict.get('track_pattern'),
match_score=raw_dict.get('match_score'),
# Pre-computed display fields
formatted_size=raw_dict.get('formatted_size'),
formatted_duration=raw_dict.get('formatted_duration'),
formatted_views=raw_dict.get('formatted_views'),
is_lossless=raw_dict.get('is_lossless'),
quality_str=raw_dict.get('quality_str'),
# Full quality object for Release.from_dict()
# Remote API uses 'quality_data', local uses 'quality' for the dict
quality_data=raw_dict.get('quality_data') or raw_dict.get('quality'),
)
)
return AudioSearchResponse(
status="awaiting_selection",
job_id=job_id,
message=f"Found {len(search_results)} audio sources. Call /api/audio-search/{job_id}/select to choose one.",
results=result_responses,
results_count=len(search_results),
auto_download=False,
server_version=VERSION,
style_upload_urls=style_upload_urls if style_upload_urls else None,
)
except HTTPException:
raise
except Exception as e:
logger.error(f"Error in audio search: {e}", exc_info=True)
raise HTTPException(status_code=500, detail=str(e))
@router.get("/audio-search/{job_id}/results")
async def get_audio_search_results(
job_id: str,
auth_data: Tuple[str, UserType, int] = Depends(require_auth)
):
"""
Get audio search results for a job.
Returns the cached search results so user can select one.
"""
job = job_manager.get_job(job_id)
if not job:
raise HTTPException(status_code=404, detail=f"Job {job_id} not found")
search_results = job.state_data.get('audio_search_results', [])
if not search_results:
raise HTTPException(
status_code=400,
detail="No search results available for this job"
)
return {
"status": "success",
"job_id": job_id,
"job_status": job.status,
"artist": job.audio_search_artist or job.artist,
"title": job.audio_search_title or job.title,
"results": search_results,
"results_count": len(search_results),
}
@router.post("/audio-search/{job_id}/select", response_model=AudioSelectResponse)
async def select_audio_source(
job_id: str,
background_tasks: BackgroundTasks,
body: AudioSelectRequest,
auth_data: Tuple[str, UserType, int] = Depends(require_auth)
):
"""
Select an audio source and start job processing.
This endpoint:
1. Validates the job is awaiting selection
2. Downloads the selected audio
3. Uploads to GCS
4. Triggers audio and lyrics workers
"""
job = job_manager.get_job(job_id)
if not job:
raise HTTPException(status_code=404, detail=f"Job {job_id} not found")
# Verify job is awaiting selection
if job.status != JobStatus.AWAITING_AUDIO_SELECTION:
raise HTTPException(
status_code=400,
detail=f"Job is not awaiting audio selection (status: {job.status})"
)
# Get search service instance
# Note: With download_by_id, we no longer need to re-search to populate the cache.
# The source_id stored in job.state_data['audio_search_results'] is sufficient.
audio_search_service = get_audio_search_service()
# Validate search results exist in job state_data
search_results = job.state_data.get('audio_search_results', [])
if not search_results:
raise HTTPException(status_code=400, detail="No search results cached for this job")
selection_info = await _download_and_start_processing(
job_id=job_id,
selection_index=body.selection_index,
audio_search_service=audio_search_service,
background_tasks=background_tasks,
)
return AudioSelectResponse(
status="success",
job_id=job_id,
message="Audio selected and download started",
selected_index=selection_info['selected_index'],
selected_title=selection_info['selected_title'],
selected_artist=selection_info['selected_artist'],
selected_provider=selection_info['selected_provider'],
)
-348
View File
@@ -1,348 +0,0 @@
"""
OAuth credential management API routes.
Provides endpoints for:
1. Checking credential status
2. Device authorization flow for re-authentication
3. Credential validation before job submission
"""
import logging
from typing import Optional
from datetime import datetime
from fastapi import APIRouter, HTTPException, BackgroundTasks
from pydantic import BaseModel
from backend.services.credential_manager import (
get_credential_manager,
CredentialStatus,
CredentialCheckResult,
)
from backend.config import get_settings
logger = logging.getLogger(__name__)
router = APIRouter(prefix="/auth", tags=["auth"])
# ===========================================================================
# Request/Response Models
# ===========================================================================
class CredentialStatusResponse(BaseModel):
"""Response model for credential status."""
service: str
status: str
message: str
last_checked: datetime
expires_at: Optional[datetime] = None
class Config:
from_attributes = True
class AllCredentialsStatusResponse(BaseModel):
"""Response model for all credentials status."""
youtube: CredentialStatusResponse
gdrive: CredentialStatusResponse
dropbox: CredentialStatusResponse
all_valid: bool
services_needing_auth: list[str]
class DeviceAuthStartRequest(BaseModel):
"""Request to start device authorization flow."""
client_id: Optional[str] = None # Optional - reads from Secret Manager if not provided
client_secret: Optional[str] = None # Optional - reads from Secret Manager if not provided
class DeviceAuthStartResponse(BaseModel):
"""Response with device authorization info."""
device_code: str
user_code: str
verification_url: str
expires_in: int
interval: int
instructions: str
class DeviceAuthPollResponse(BaseModel):
"""Response from polling device authorization."""
status: str # "pending", "complete", "expired", "error"
message: Optional[str] = None
credentials_saved: bool = False
class CredentialValidationRequest(BaseModel):
"""Request to validate credentials for specific services."""
youtube: bool = False
gdrive: bool = False
dropbox: bool = False
class CredentialValidationResponse(BaseModel):
"""Response from credential validation."""
valid: bool
invalid_services: list[str]
message: str
# ===========================================================================
# Endpoints
# ===========================================================================
@router.get("/status", response_model=AllCredentialsStatusResponse)
async def get_credentials_status():
"""
Get the status of all OAuth credentials.
Returns the validation status for YouTube, Google Drive, and Dropbox
credentials, including whether they are valid, expired, or need
re-authorization.
"""
manager = get_credential_manager()
results = manager.check_all_credentials()
def to_response(result: CredentialCheckResult) -> CredentialStatusResponse:
return CredentialStatusResponse(
service=result.service,
status=result.status.value,
message=result.message,
last_checked=result.last_checked,
expires_at=result.expires_at
)
services_needing_auth = [
name for name, result in results.items()
if result.status in (CredentialStatus.INVALID, CredentialStatus.EXPIRED, CredentialStatus.NOT_CONFIGURED)
]
return AllCredentialsStatusResponse(
youtube=to_response(results["youtube"]),
gdrive=to_response(results["gdrive"]),
dropbox=to_response(results["dropbox"]),
all_valid=len(services_needing_auth) == 0,
services_needing_auth=services_needing_auth
)
@router.get("/status/{service}", response_model=CredentialStatusResponse)
async def get_service_credential_status(service: str):
"""
Get the status of a specific service's OAuth credentials.
Args:
service: One of "youtube", "gdrive", "dropbox"
"""
manager = get_credential_manager()
if service == "youtube":
result = manager.check_youtube_credentials()
elif service == "gdrive":
result = manager.check_gdrive_credentials()
elif service == "dropbox":
result = manager.check_dropbox_credentials()
else:
raise HTTPException(status_code=400, detail=f"Unknown service: {service}")
return CredentialStatusResponse(
service=result.service,
status=result.status.value,
message=result.message,
last_checked=result.last_checked,
expires_at=result.expires_at
)
@router.post("/validate", response_model=CredentialValidationResponse)
async def validate_credentials(request: CredentialValidationRequest):
"""
Validate that credentials are available for requested services.
Use this before submitting a job to ensure all required
credentials are valid.
"""
manager = get_credential_manager()
invalid_services = []
if request.youtube:
result = manager.check_youtube_credentials()
if result.status != CredentialStatus.VALID:
invalid_services.append("youtube")
if request.gdrive:
result = manager.check_gdrive_credentials()
if result.status != CredentialStatus.VALID:
invalid_services.append("gdrive")
if request.dropbox:
result = manager.check_dropbox_credentials()
if result.status != CredentialStatus.VALID:
invalid_services.append("dropbox")
if invalid_services:
return CredentialValidationResponse(
valid=False,
invalid_services=invalid_services,
message=f"The following services need re-authorization: {', '.join(invalid_services)}"
)
return CredentialValidationResponse(
valid=True,
invalid_services=[],
message="All requested credentials are valid"
)
# ===========================================================================
# Device Authorization Flow Endpoints
# ===========================================================================
@router.post("/youtube/device", response_model=DeviceAuthStartResponse)
async def start_youtube_device_auth(request: Optional[DeviceAuthStartRequest] = None):
"""
Start YouTube device authorization flow.
This initiates a device auth flow that allows authorization
from any device. The user must visit the verification URL
and enter the user code.
Client credentials are loaded from Secret Manager ('youtube-client-credentials')
unless explicitly provided in the request body.
After starting, poll /auth/youtube/device/{device_code} to
check for completion.
"""
manager = get_credential_manager()
try:
device_info = manager.start_youtube_device_auth(
client_id=request.client_id if request else None,
client_secret=request.client_secret if request else None
)
return DeviceAuthStartResponse(
device_code=device_info.device_code,
user_code=device_info.user_code,
verification_url=device_info.verification_url,
expires_in=device_info.expires_in,
interval=device_info.interval,
instructions=f"Visit {device_info.verification_url} and enter code: {device_info.user_code}"
)
except Exception as e:
logger.error(f"Failed to start YouTube device auth: {e}")
raise HTTPException(status_code=500, detail=str(e))
@router.get("/youtube/device/{device_code}", response_model=DeviceAuthPollResponse)
async def poll_youtube_device_auth(device_code: str):
"""
Poll for YouTube device authorization completion.
Call this endpoint at the interval specified in the device auth
start response. Keep polling while status is "pending".
Status values:
- pending: User has not yet authorized
- complete: Authorization successful, credentials saved
- expired: Device code expired, start new flow
- error: An error occurred
"""
manager = get_credential_manager()
status, data = manager.poll_device_auth("youtube", device_code)
return DeviceAuthPollResponse(
status=status,
message=data.get("message") if data else None,
credentials_saved=(status == "complete")
)
@router.post("/gdrive/device", response_model=DeviceAuthStartResponse)
async def start_gdrive_device_auth(request: Optional[DeviceAuthStartRequest] = None):
"""
Start Google Drive device authorization flow.
Client credentials are loaded from Secret Manager ('gdrive-client-credentials')
unless explicitly provided in the request body.
After starting, poll /auth/gdrive/device/{device_code} to check for completion.
"""
manager = get_credential_manager()
try:
device_info = manager.start_gdrive_device_auth(
client_id=request.client_id if request else None,
client_secret=request.client_secret if request else None
)
return DeviceAuthStartResponse(
device_code=device_info.device_code,
user_code=device_info.user_code,
verification_url=device_info.verification_url,
expires_in=device_info.expires_in,
interval=device_info.interval,
instructions=f"Visit {device_info.verification_url} and enter code: {device_info.user_code}"
)
except Exception as e:
logger.error(f"Failed to start Google Drive device auth: {e}")
raise HTTPException(status_code=500, detail=str(e))
@router.get("/gdrive/device/{device_code}", response_model=DeviceAuthPollResponse)
async def poll_gdrive_device_auth(device_code: str):
"""
Poll for Google Drive device authorization completion.
"""
manager = get_credential_manager()
status, data = manager.poll_device_auth("gdrive", device_code)
return DeviceAuthPollResponse(
status=status,
message=data.get("message") if data else None,
credentials_saved=(status == "complete")
)
# ===========================================================================
# Alert Testing
# ===========================================================================
@router.post("/test-alert")
async def test_credential_alert(background_tasks: BackgroundTasks):
"""
Test the credential alert mechanism.
Sends a test alert to Discord if configured.
"""
settings = get_settings()
discord_url = settings.get_secret("discord-alert-webhook")
if not discord_url:
raise HTTPException(
status_code=400,
detail="Discord alert webhook not configured"
)
manager = get_credential_manager()
# Create a fake invalid result for testing
from backend.services.credential_manager import CredentialCheckResult
test_results = [
CredentialCheckResult(
service="test",
status=CredentialStatus.INVALID,
message="This is a test alert",
last_checked=datetime.utcnow()
)
]
background_tasks.add_task(
manager.send_credential_alert,
test_results,
discord_url
)
return {"message": "Test alert queued"}
File diff suppressed because it is too large. Load diff
-141
View File
@@ -1,141 +0,0 @@
"""
Health check routes.
"""
import os
import logging
from fastapi import APIRouter
from typing import Dict, Any, Optional
from backend.version import VERSION
from backend.services.flacfetch_client import get_flacfetch_client
router = APIRouter()
logger = logging.getLogger(__name__)
def check_transmission_status() -> Dict[str, Any]:
"""Check if Transmission daemon is available and responsive."""
try:
import transmission_rpc
host = os.environ.get("TRANSMISSION_HOST", "localhost")
port = int(os.environ.get("TRANSMISSION_PORT", "9091"))
client = transmission_rpc.Client(host=host, port=port, timeout=5)
stats = client.session_stats()
# Get detailed torrent info
torrents = client.get_torrents()
torrent_details = []
for t in torrents:
torrent_info = {
"name": t.name,
"progress": round(t.progress, 1),
"status": str(t.status),
"peers": t.peers_connected if hasattr(t, 'peers_connected') else 0,
"download_speed": round(t.rate_download / 1024, 1) if hasattr(t, 'rate_download') else 0, # KB/s
}
# Check if stalled (downloading but no progress and no peers)
if t.status.downloading and t.progress < 100 and torrent_info['peers'] == 0:
torrent_info['stalled'] = True
torrent_details.append(torrent_info)
return {
"available": True,
"host": host,
"port": port,
"download_dir": stats.download_dir if hasattr(stats, 'download_dir') else None,
"active_torrent_count": stats.active_torrent_count if hasattr(stats, 'active_torrent_count') else 0,
"torrents": torrent_details,
}
except ImportError as e:
return {
"available": False,
"error": f"transmission_rpc not installed: {e}",
}
except Exception as e:
return {
"available": False,
"host": os.environ.get("TRANSMISSION_HOST", "localhost"),
"port": int(os.environ.get("TRANSMISSION_PORT", "9091")),
"error": str(e),
}
@router.get("/health")
async def health_check() -> Dict[str, str]:
"""Health check endpoint."""
return {
"status": "healthy",
"service": "karaoke-gen-backend"
}
async def check_flacfetch_service_status() -> Dict[str, Any]:
"""Check if remote flacfetch service is available and healthy."""
client = get_flacfetch_client()
if not client:
return {
"configured": False,
"message": "Remote flacfetch service not configured (FLACFETCH_API_URL not set)",
}
try:
health = await client.health_check()
return {
"configured": True,
"available": True,
"status": health.get("status"),
"version": health.get("version"),
"transmission": health.get("transmission", {}),
"disk": health.get("disk", {}),
"providers": health.get("providers", {}),
}
except Exception as e:
return {
"configured": True,
"available": False,
"error": str(e),
}
@router.get("/health/detailed")
async def detailed_health_check() -> Dict[str, Any]:
"""
Detailed health check including dependencies.
Use this to debug issues with Transmission, flacfetch service, etc.
"""
transmission_status = check_transmission_status()
flacfetch_status = await check_flacfetch_service_status()
# Log for debugging
if not transmission_status.get("available"):
logger.warning(f"Local Transmission not available: {transmission_status.get('error')}")
else:
logger.info(f"Local Transmission available at {transmission_status.get('host')}:{transmission_status.get('port')}")
if flacfetch_status.get("configured") and not flacfetch_status.get("available"):
logger.warning(f"Remote flacfetch service not available: {flacfetch_status.get('error')}")
elif flacfetch_status.get("available"):
logger.info(f"Remote flacfetch service healthy: {flacfetch_status.get('status')}")
return {
"status": "healthy",
"service": "karaoke-gen-backend",
"version": VERSION,
"dependencies": {
"transmission_local": transmission_status,
"flacfetch_remote": flacfetch_status,
}
}
@router.get("/readiness")
async def readiness_check() -> Dict[str, str]:
"""Readiness check endpoint for Cloud Run."""
return {
"status": "ready",
"service": "karaoke-gen-backend"
}
-330
View File
@@ -1,330 +0,0 @@
"""
Internal API routes for worker coordination.
These endpoints are for internal use only (backend → workers).
They are protected by admin authentication.
With Cloud Tasks integration, these endpoints may be called multiple times
(retry on failure). Idempotency checks prevent duplicate processing.
Observability:
- Extracts trace context from incoming requests (propagated via Cloud Tasks)
- Creates worker spans linked to the original request trace
- All logs include job_id for easy filtering in Cloud Logging
"""
import logging
import asyncio
import time
from typing import Tuple, Optional
from fastapi import APIRouter, HTTPException, BackgroundTasks, Depends, Request
from pydantic import BaseModel
from backend.workers.audio_worker import process_audio_separation
from backend.workers.lyrics_worker import process_lyrics_transcription
from backend.workers.screens_worker import generate_screens
from backend.workers.video_worker import generate_video
from backend.workers.render_video_worker import process_render_video
from backend.api.dependencies import require_admin
from backend.services.auth_service import UserType
from backend.services.job_manager import JobManager
from backend.services.tracing import (
extract_trace_context,
start_span_with_context,
add_span_attribute,
add_span_event,
)
logger = logging.getLogger(__name__)
router = APIRouter(prefix="/internal", tags=["internal"])
class WorkerRequest(BaseModel):
"""Request to trigger a worker."""
job_id: str
class WorkerResponse(BaseModel):
"""Response from worker trigger."""
status: str
job_id: str
message: str
def _check_worker_idempotency(job_id: str, worker_name: str) -> Optional[WorkerResponse]:
"""
Check if a worker is already running or completed for this job.
This provides idempotency for Cloud Tasks retries - if a task is retried
but the worker is already running or has completed, we skip processing.
Args:
job_id: Job ID to check
worker_name: Worker name (audio, lyrics, screens, render, video)
Returns:
WorkerResponse if should skip (already running/complete), None to proceed
"""
job_manager = JobManager()
job = job_manager.get_job(job_id)
if not job:
logger.warning(f"[job:{job_id}] Job not found for {worker_name} worker")
return WorkerResponse(
status="not_found",
job_id=job_id,
message=f"Job {job_id} not found"
)
# Check worker-specific progress in state_data
progress_key = f"{worker_name}_progress"
worker_progress = job.state_data.get(progress_key, {})
stage = worker_progress.get('stage')
if stage == 'running':
logger.info(f"[job:{job_id}] {worker_name.capitalize()} worker already running, skipping")
return WorkerResponse(
status="already_running",
job_id=job_id,
message=f"{worker_name.capitalize()} worker already in progress"
)
if stage == 'complete':
logger.info(f"[job:{job_id}] {worker_name.capitalize()} worker already complete, skipping")
return WorkerResponse(
status="already_complete",
job_id=job_id,
message=f"{worker_name.capitalize()} worker already completed"
)
# Mark as running before starting (for idempotency on next retry)
job_manager.update_state_data(job_id, progress_key, {'stage': 'running'})
return None
@router.post("/workers/audio", response_model=WorkerResponse)
async def trigger_audio_worker(
request: WorkerRequest,
http_request: Request,
background_tasks: BackgroundTasks,
auth_data: Tuple[str, UserType, int] = Depends(require_admin)
):
"""
Trigger audio separation worker for a job.
This endpoint is called internally after job creation to start
the audio processing track (parallel with lyrics processing).
Idempotency: If worker is already running or complete, returns early.
The worker runs in the background and updates job state as it progresses.
"""
job_id = request.job_id
# Extract trace context from incoming request (propagated via Cloud Tasks)
trace_context = extract_trace_context(dict(http_request.headers))
logger.info(f"[job:{job_id}] WORKER_TRIGGER worker=audio")
add_span_attribute("job_id", job_id)
add_span_attribute("worker", "audio")
# Idempotency check
skip_response = _check_worker_idempotency(job_id, "audio")
if skip_response:
add_span_event("worker_skipped", {"reason": skip_response.status})
return skip_response
# Add task to background tasks
# This allows the HTTP response to return immediately
# while the worker continues processing
background_tasks.add_task(process_audio_separation, job_id)
add_span_event("worker_started")
return WorkerResponse(
status="started",
job_id=job_id,
message="Audio separation worker started"
)
@router.post("/workers/lyrics", response_model=WorkerResponse)
async def trigger_lyrics_worker(
request: WorkerRequest,
http_request: Request,
background_tasks: BackgroundTasks,
auth_data: Tuple[str, UserType, int] = Depends(require_admin)
):
"""
Trigger lyrics transcription worker for a job.
This endpoint is called internally after job creation to start
the lyrics processing track (parallel with audio processing).
Idempotency: If worker is already running or complete, returns early.
The worker runs in the background and updates job state as it progresses.
"""
job_id = request.job_id
# Extract trace context from incoming request
trace_context = extract_trace_context(dict(http_request.headers))
logger.info(f"[job:{job_id}] WORKER_TRIGGER worker=lyrics")
add_span_attribute("job_id", job_id)
add_span_attribute("worker", "lyrics")
# Idempotency check
skip_response = _check_worker_idempotency(job_id, "lyrics")
if skip_response:
add_span_event("worker_skipped", {"reason": skip_response.status})
return skip_response
# Add task to background tasks
background_tasks.add_task(process_lyrics_transcription, job_id)
add_span_event("worker_started")
return WorkerResponse(
status="started",
job_id=job_id,
message="Lyrics transcription worker started"
)
@router.post("/workers/screens", response_model=WorkerResponse)
async def trigger_screens_worker(
request: WorkerRequest,
http_request: Request,
background_tasks: BackgroundTasks,
auth_data: Tuple[str, UserType, int] = Depends(require_admin)
):
"""
Trigger title/end screen generation worker.
This is called automatically when both audio and lyrics are complete.
Idempotency: If worker is already running or complete, returns early.
"""
job_id = request.job_id
# Extract trace context from incoming request
trace_context = extract_trace_context(dict(http_request.headers))
logger.info(f"[job:{job_id}] WORKER_TRIGGER worker=screens")
add_span_attribute("job_id", job_id)
add_span_attribute("worker", "screens")
# Idempotency check
skip_response = _check_worker_idempotency(job_id, "screens")
if skip_response:
add_span_event("worker_skipped", {"reason": skip_response.status})
return skip_response
# Add task to background tasks
background_tasks.add_task(generate_screens, job_id)
add_span_event("worker_started")
return WorkerResponse(
status="started",
job_id=job_id,
message="Screens generation worker started"
)
@router.post("/workers/video", response_model=WorkerResponse)
async def trigger_video_worker(
request: WorkerRequest,
http_request: Request,
background_tasks: BackgroundTasks,
auth_data: Tuple[str, UserType, int] = Depends(require_admin)
):
"""
Trigger final video generation and encoding worker.
This is called after user selects their preferred instrumental.
This is the longest-running stage (15-20 minutes).
Idempotency: If worker is already running or complete, returns early.
"""
job_id = request.job_id
# Extract trace context from incoming request
trace_context = extract_trace_context(dict(http_request.headers))
logger.info(f"[job:{job_id}] WORKER_TRIGGER worker=video")
add_span_attribute("job_id", job_id)
add_span_attribute("worker", "video")
# Idempotency check
skip_response = _check_worker_idempotency(job_id, "video")
if skip_response:
add_span_event("worker_skipped", {"reason": skip_response.status})
return skip_response
# Add task to background tasks
background_tasks.add_task(generate_video, job_id)
add_span_event("worker_started")
return WorkerResponse(
status="started",
job_id=job_id,
message="Video generation worker started"
)
@router.post("/workers/render-video", response_model=WorkerResponse)
async def trigger_render_video_worker(
request: WorkerRequest,
http_request: Request,
background_tasks: BackgroundTasks,
auth_data: Tuple[str, UserType, int] = Depends(require_admin)
):
"""
Trigger render video worker (post-review).
This is called after human review is complete.
Uses OutputGenerator from LyricsTranscriber to generate the karaoke video
with the corrected lyrics.
Idempotency: If worker is already running or complete, returns early.
Output: with_vocals.mkv in GCS
Next state: AWAITING_INSTRUMENTAL_SELECTION
"""
job_id = request.job_id
# Extract trace context from incoming request
trace_context = extract_trace_context(dict(http_request.headers))
logger.info(f"[job:{job_id}] WORKER_TRIGGER worker=render-video")
add_span_attribute("job_id", job_id)
add_span_attribute("worker", "render-video")
# Idempotency check
skip_response = _check_worker_idempotency(job_id, "render")
if skip_response:
add_span_event("worker_skipped", {"reason": skip_response.status})
return skip_response
# Add task to background tasks
background_tasks.add_task(process_render_video, job_id)
add_span_event("worker_started")
return WorkerResponse(
status="started",
job_id=job_id,
message="Render video worker started (post-review)"
)
@router.get("/health")
async def internal_health(
auth_data: Tuple[str, UserType, int] = Depends(require_admin)
):
"""
Internal health check endpoint.
Used to verify the internal API is responsive.
Requires admin authentication.
"""
return {"status": "healthy", "service": "karaoke-backend-internal"}
File diff suppressed because it is too large. Load diff
-570
View File
@@ -1,570 +0,0 @@
"""
Review API routes - Compatible with LyricsTranscriber frontend.
These endpoints match the API that the LyricsTranscriber review frontend expects,
allowing us to use the existing React review UI with our cloud backend.
Usage:
Frontend URL: http://localhost:5173/?baseApiUrl=http://localhost:8000/api/review/{job_id}
The baseApiUrl includes the job_id, and all endpoints are relative to that.
"""
import asyncio
import logging
import hashlib
import json
import os
import tempfile
from pathlib import Path
from typing import Dict, Any, Set, Tuple
from fastapi import APIRouter, HTTPException, Request, Depends
from fastapi.responses import FileResponse, StreamingResponse
from starlette.background import BackgroundTask
from backend.models.job import JobStatus
from backend.services.job_manager import JobManager
from backend.services.storage_service import StorageService
from backend.services.job_logging import job_log_context, JobLogger
from backend.services.tracing import create_span, add_span_attribute, add_span_event
from backend.api.dependencies import require_auth, require_review_auth
from backend.services.auth_service import UserType
# LyricsTranscriber imports for preview generation
from lyrics_transcriber.types import CorrectionResult
from lyrics_transcriber.core.config import OutputConfig
from lyrics_transcriber.correction.operations import CorrectionOperations
# Import from the unified style loader
from karaoke_gen.style_loader import load_styles_from_gcs
logger = logging.getLogger(__name__)
router = APIRouter(prefix="/review", tags=["review"])
# Store job context for the session
# In production, this would be handled differently (e.g., session tokens)
_job_contexts: Dict[str, Dict[str, Any]] = {}
# Store preview video paths for serving
_preview_videos: Dict[str, Dict[str, str]] = {}
# Keep references to background tasks to prevent garbage collection
_background_tasks: Set[asyncio.Task] = set()
def _get_audio_hash(job_id: str) -> str:
"""Generate a consistent audio hash for a job."""
return hashlib.md5(job_id.encode()).hexdigest()
@router.get("/{job_id}/ping")
async def ping(job_id: str):
"""Health check endpoint expected by frontend."""
return {"status": "ok"}
@router.get("/{job_id}/correction-data")
async def get_correction_data(
job_id: str,
auth_info: Tuple[str, str] = Depends(require_review_auth)
):
"""
Get correction data for the review interface.
Returns the CorrectionResult data that the frontend needs to render
the lyrics review UI.
"""
job_manager = JobManager()
storage = StorageService()
job = job_manager.get_job(job_id)
if not job:
raise HTTPException(status_code=404, detail="Job not found")
if job.status not in [JobStatus.AWAITING_REVIEW, JobStatus.IN_REVIEW]:
raise HTTPException(
status_code=400,
detail=f"Job not ready for review (current status: {job.status})"
)
# Get corrections URL from file_urls
corrections_gcs = job.file_urls.get('lyrics', {}).get('corrections')
if not corrections_gcs:
# Try direct path
corrections_gcs = f"jobs/{job_id}/lyrics/corrections.json"
if not storage.file_exists(corrections_gcs):
raise HTTPException(
status_code=404,
detail="Corrections data not found. Lyrics processing may not be complete."
)
# Download and return corrections data
try:
corrections_data = storage.download_json(corrections_gcs)
# Add audio hash for the frontend
audio_hash = _get_audio_hash(job_id)
if 'metadata' not in corrections_data:
corrections_data['metadata'] = {}
corrections_data['metadata']['audio_hash'] = audio_hash
corrections_data['metadata']['artist'] = job.artist
corrections_data['metadata']['title'] = job.title
# Store context for audio serving
_job_contexts[job_id] = {
'audio_hash': audio_hash,
'audio_gcs_path': job.input_media_gcs_path
}
# Transition to IN_REVIEW if not already
if job.status == JobStatus.AWAITING_REVIEW:
job_manager.transition_to_state(
job_id=job_id,
new_status=JobStatus.IN_REVIEW,
message="User opened review interface"
)
logger.info(f"Job {job_id}: Serving correction data for review")
return corrections_data
except Exception as e:
logger.error(f"Job {job_id}: Error loading corrections: {e}", exc_info=True)
raise HTTPException(status_code=500, detail=f"Error loading corrections: {str(e)}")
@router.get("/{job_id}/audio/{audio_hash}")
async def get_audio_with_hash(
job_id: str,
audio_hash: str,
auth_info: Tuple[str, str] = Depends(require_review_auth)
):
"""Stream the audio file for playback (with hash parameter)."""
return await _stream_audio(job_id)
@router.get("/{job_id}/audio/")
@router.get("/{job_id}/audio")
async def get_audio_no_hash(
job_id: str,
auth_info: Tuple[str, str] = Depends(require_review_auth)
):
"""Stream the audio file for playback (without hash parameter)."""
return await _stream_audio(job_id)
async def _stream_audio(job_id: str):
"""
Stream the audio file for playback in the review interface.
"""
job_manager = JobManager()
storage = StorageService()
job = job_manager.get_job(job_id)
if not job:
raise HTTPException(status_code=404, detail="Job not found")
audio_gcs_path = job.input_media_gcs_path
if not audio_gcs_path:
raise HTTPException(status_code=404, detail="Audio file not found")
# Download to temp file and stream
try:
# Create temp file
with tempfile.NamedTemporaryFile(delete=False, suffix=".flac") as tmp:
tmp_path = tmp.name
storage.download_file(audio_gcs_path, tmp_path)
# Determine content type
if audio_gcs_path.endswith('.flac'):
media_type = "audio/flac"
elif audio_gcs_path.endswith('.wav'):
media_type = "audio/wav"
elif audio_gcs_path.endswith('.mp3'):
media_type = "audio/mpeg"
else:
media_type = "audio/mpeg"
logger.info(f"Job {job_id}: Streaming audio for review")
return FileResponse(
tmp_path,
media_type=media_type,
filename=os.path.basename(audio_gcs_path),
background=BackgroundTask(os.unlink, tmp_path),
)
except Exception as e:
logger.error(f"Job {job_id}: Error streaming audio: {e}", exc_info=True)
raise HTTPException(status_code=500, detail=f"Error streaming audio: {str(e)}")
@router.post("/{job_id}/complete")
async def complete_review(
job_id: str,
updated_data: Dict[str, Any],
auth_info: Tuple[str, str] = Depends(require_review_auth)
):
"""
Complete the review and save corrected lyrics.
This endpoint receives the updated correction data from the frontend
and saves it, then triggers the render video worker.
"""
job_manager = JobManager()
storage = StorageService()
job = job_manager.get_job(job_id)
if not job:
raise HTTPException(status_code=404, detail="Job not found")
if job.status not in [JobStatus.AWAITING_REVIEW, JobStatus.IN_REVIEW]:
raise HTTPException(
status_code=400,
detail=f"Job not in review state (current status: {job.status})"
)
try:
# Save updated corrections to GCS
corrections_gcs_path = f"jobs/{job_id}/lyrics/corrections_updated.json"
storage.upload_json(corrections_gcs_path, updated_data)
job_manager.update_file_url(job_id, 'lyrics', 'corrections_updated', corrections_gcs_path)
logger.info(f"Job {job_id}: Saved updated corrections")
# Transition to REVIEW_COMPLETE
job_manager.transition_to_state(
job_id=job_id,
new_status=JobStatus.REVIEW_COMPLETE,
progress=70,
message="Review complete, rendering video with corrected lyrics"
)
# Trigger render video worker
from backend.services.worker_service import get_worker_service
worker_service = get_worker_service()
# Run in background, keep reference to prevent garbage collection
task = asyncio.create_task(worker_service.trigger_render_video_worker(job_id))
_background_tasks.add(task)
task.add_done_callback(_background_tasks.discard)
logger.info(f"Job {job_id}: Review complete, triggered render video worker")
return {"status": "success"}
except Exception as e:
logger.error(f"Job {job_id}: Error completing review: {e}", exc_info=True)
raise HTTPException(status_code=500, detail=f"Error completing review: {str(e)}")
@router.post("/{job_id}/handlers")
async def update_handlers(
job_id: str,
enabled_handlers: list,
auth_info: Tuple[str, str] = Depends(require_review_auth)
):
"""
Update enabled correction handlers (optional feature).
For now, just acknowledge the request - full handler support
would require re-running correction.
"""
logger.info(f"Job {job_id}: Handler update requested (not implemented)")
return {"status": "success", "message": "Handler updates not yet implemented"}
@router.post("/{job_id}/add-lyrics")
async def add_lyrics(
job_id: str,
data: Dict[str, str],
auth_info: Tuple[str, str] = Depends(require_review_auth)
):
"""
Add custom lyrics source and rerun correction.
Uses the LyricsTranscriber's CorrectionOperations to add a new lyrics source
and regenerate corrections with the new source included.
"""
job_manager = JobManager()
storage = StorageService()
job = job_manager.get_job(job_id)
if not job:
raise HTTPException(status_code=404, detail="Job not found")
# Job must be in review state to add lyrics
if job.status not in [JobStatus.AWAITING_REVIEW, JobStatus.IN_REVIEW]:
raise HTTPException(
status_code=400,
detail=f"Job not in review state (current status: {job.status})"
)
source = data.get("source", "").strip()
lyrics_text = data.get("lyrics", "").strip()
logger.info(f"Job {job_id}: Adding lyrics source '{source}' with {len(lyrics_text)} characters")
# Use tracing and job_log_context for full observability
with create_span("add-lyrics", {"job_id": job_id, "source": source, "lyrics_length": len(lyrics_text)}) as span:
with job_log_context(job_id, worker="add-lyrics"):
try:
# Create temp directory for this operation
with tempfile.TemporaryDirectory() as temp_dir:
# Download current corrections.json
with create_span("download-corrections") as download_span:
corrections_gcs = f"jobs/{job_id}/lyrics/corrections.json"
corrections_path = os.path.join(temp_dir, "corrections.json")
storage.download_file(corrections_gcs, corrections_path)
download_span.set_attribute("gcs_path", corrections_gcs)
with open(corrections_path, 'r', encoding='utf-8') as f:
original_data = json.load(f)
# Load as CorrectionResult
correction_result = CorrectionResult.from_dict(original_data)
add_span_event("corrections_loaded", {
"segments": len(correction_result.corrected_segments) if correction_result.corrected_segments else 0,
"reference_sources": len(correction_result.reference_lyrics) if correction_result.reference_lyrics else 0,
})
# Set up cache directory
cache_dir = os.path.join(temp_dir, "cache")
os.makedirs(cache_dir, exist_ok=True)
# Add lyrics source using CorrectionOperations (this is the heavy operation)
with create_span("correction-operations-add-lyrics") as correction_span:
correction_span.set_attribute("source_name", source)
updated_result = CorrectionOperations.add_lyrics_source(
correction_result=correction_result,
source=source,
lyrics_text=lyrics_text,
cache_dir=cache_dir,
logger=logger
)
add_span_event("correction_complete", {
"new_segments": len(updated_result.corrected_segments) if updated_result.corrected_segments else 0,
})
# Add audio hash for the frontend
audio_hash = _get_audio_hash(job_id)
if not updated_result.metadata:
updated_result.metadata = {}
updated_result.metadata['audio_hash'] = audio_hash
updated_result.metadata['artist'] = job.artist
updated_result.metadata['title'] = job.title
# Upload updated corrections back to GCS
with create_span("upload-corrections") as upload_span:
updated_data = updated_result.to_dict()
storage.upload_json(corrections_gcs, updated_data)
upload_span.set_attribute("gcs_path", corrections_gcs)
logger.info(f"Job {job_id}: Successfully added lyrics source '{source}'")
span.set_attribute("success", True)
return {"status": "success", "data": updated_data}
except ValueError as e:
# ValueError from CorrectionOperations (e.g., duplicate source name)
logger.warning(f"Job {job_id}: Invalid add lyrics request: {e}")
span.set_attribute("error", str(e))
raise HTTPException(status_code=400, detail=str(e))
except Exception as e:
logger.error(f"Job {job_id}: Failed to add lyrics: {e}", exc_info=True)
span.set_attribute("error", str(e))
raise HTTPException(status_code=500, detail=f"Failed to add lyrics: {str(e)}")
@router.post("/{job_id}/preview-video")
async def generate_preview_video(
job_id: str,
updated_data: Dict[str, Any],
auth_info: Tuple[str, str] = Depends(require_review_auth)
):
"""
Generate a preview video with the current corrections.
Uses the LyricsTranscriber's CorrectionOperations to generate a 360p preview
video with the user's current corrections applied.
"""
job_manager = JobManager()
storage = StorageService()
job = job_manager.get_job(job_id)
if not job:
raise HTTPException(status_code=404, detail="Job not found")
# Job must be in review state to generate preview
if job.status not in [JobStatus.AWAITING_REVIEW, JobStatus.IN_REVIEW]:
raise HTTPException(
status_code=400,
detail=f"Job not in review state (current status: {job.status})"
)
logger.info(f"Job {job_id}: Generating preview video")
# Use tracing and job_log_context for full observability
with create_span("generate-preview-video", {"job_id": job_id}) as span:
with job_log_context(job_id, worker="preview"):
try:
# Create temp directory for this preview operation
with tempfile.TemporaryDirectory() as temp_dir:
# 1. Download original corrections.json (has full structure)
with create_span("download-corrections-and-audio") as download_span:
corrections_gcs = f"jobs/{job_id}/lyrics/corrections.json"
corrections_path = os.path.join(temp_dir, "corrections.json")
storage.download_file(corrections_gcs, corrections_path)
with open(corrections_path, 'r', encoding='utf-8') as f:
original_data = json.load(f)
# 2. Download input audio
audio_path = os.path.join(temp_dir, "audio.flac")
storage.download_file(job.input_media_gcs_path, audio_path)
download_span.set_attribute("audio_gcs_path", job.input_media_gcs_path)
# 3. Load original as CorrectionResult
correction_result = CorrectionResult.from_dict(original_data)
add_span_event("corrections_loaded")
# 4. Get or create styles file for preview using unified style loader
with create_span("load-styles") as styles_span:
styles_path, _ = load_styles_from_gcs(
style_params_gcs_path=job.style_params_gcs_path,
style_assets=job.style_assets,
temp_dir=temp_dir,
download_func=storage.download_file,
logger=logger,
)
styles_span.set_attribute("styles_path", styles_path)
# 5. Set up output config for preview
output_dir = os.path.join(temp_dir, "output")
cache_dir = os.path.join(temp_dir, "cache")
os.makedirs(output_dir, exist_ok=True)
os.makedirs(cache_dir, exist_ok=True)
output_config = OutputConfig(
output_styles_json=styles_path,
output_dir=output_dir,
cache_dir=cache_dir,
video_resolution="360p",
)
# 6. Generate preview video using CorrectionOperations (heavy operation)
with create_span("render-preview-video") as render_span:
render_span.set_attribute("resolution", "360p")
result = CorrectionOperations.generate_preview_video(
correction_result=correction_result,
updated_data=updated_data,
output_config=output_config,
audio_filepath=audio_path,
artist=job.artist,
title=job.title,
logger=logger
)
add_span_event("render_complete")
# 7. Upload preview video to GCS for serving
preview_hash = result["preview_hash"]
video_path = result["video_path"]
with create_span("upload-preview-video") as upload_span:
preview_gcs_path = f"jobs/{job_id}/previews/{preview_hash}.mp4"
storage.upload_file(video_path, preview_gcs_path)
upload_span.set_attribute("gcs_path", preview_gcs_path)
# Store the GCS path for serving
if job_id not in _preview_videos:
_preview_videos[job_id] = {}
_preview_videos[job_id][preview_hash] = preview_gcs_path
logger.info(f"Job {job_id}: Preview video generated: {preview_hash}")
span.set_attribute("preview_hash", preview_hash)
span.set_attribute("success", True)
return {"status": "success", "preview_hash": preview_hash}
except Exception as e:
logger.error(f"Job {job_id}: Failed to generate preview video: {e}", exc_info=True)
span.set_attribute("error", str(e))
raise HTTPException(
status_code=500,
detail=f"Failed to generate preview video: {e}"
) from e
@router.get("/{job_id}/preview-video/{preview_hash}")
async def get_preview_video(
job_id: str,
preview_hash: str,
auth_info: Tuple[str, str] = Depends(require_review_auth)
):
"""Stream the generated preview video."""
storage = StorageService()
# Check in-memory cache first
preview_gcs_path = None
if job_id in _preview_videos and preview_hash in _preview_videos[job_id]:
preview_gcs_path = _preview_videos[job_id][preview_hash]
else:
# Try standard path
preview_gcs_path = f"jobs/{job_id}/previews/{preview_hash}.mp4"
if not storage.file_exists(preview_gcs_path):
raise HTTPException(status_code=404, detail="Preview video not found")
try:
# Download to temp file and stream
with tempfile.NamedTemporaryFile(delete=False, suffix=".mp4") as tmp:
tmp_path = tmp.name
storage.download_file(preview_gcs_path, tmp_path)
logger.info(f"Job {job_id}: Streaming preview video {preview_hash}")
return FileResponse(
tmp_path,
media_type="video/mp4",
filename=f"preview_{preview_hash}.mp4",
headers={
"Accept-Ranges": "bytes",
"Content-Disposition": "inline",
"Cache-Control": "no-cache",
},
background=BackgroundTask(os.unlink, tmp_path),
)
except Exception as e:
logger.error(f"Job {job_id}: Error streaming preview video: {e}", exc_info=True)
raise HTTPException(status_code=500, detail=f"Error streaming preview video: {str(e)}")
@router.post("/{job_id}/v1/annotations")
async def submit_annotation(
job_id: str,
annotation: Dict[str, Any],
auth_info: Tuple[str, str] = Depends(require_review_auth)
):
"""
Submit a correction annotation for ML training data.
For now, just log and acknowledge - full annotation support
would require a database.
"""
logger.info(f"Job {job_id}: Annotation submitted (logged but not stored)")
return {"status": "success", "annotation_id": "stub"}
@router.get("/{job_id}/v1/annotations/stats")
async def get_annotation_stats(
job_id: str,
auth_info: Tuple[str, str] = Depends(require_review_auth)
):
"""Get annotation statistics."""
return {
"total_annotations": 0,
"by_type": {},
"message": "Annotation stats not yet implemented"
}
-162
View File
@@ -1,162 +0,0 @@
"""
Theme API routes for listing and retrieving theme information.
Themes are pre-created style configurations that users can select when creating jobs.
These endpoints are public (no authentication required) to allow theme selection
before job creation.
"""
import logging
from typing import Optional
from fastapi import APIRouter, HTTPException
from backend.models.theme import ThemeDetailResponse, ThemesListResponse
from backend.services.theme_service import get_theme_service
logger = logging.getLogger(__name__)
router = APIRouter(prefix="/themes", tags=["themes"])
@router.get("", response_model=ThemesListResponse)
async def list_themes() -> ThemesListResponse:
"""
List all available themes.
Returns a list of themes with their metadata and preview image URLs.
Preview URLs are signed and valid for 60 minutes.
This endpoint is public and does not require authentication.
"""
try:
theme_service = get_theme_service()
themes = theme_service.list_themes()
return ThemesListResponse(themes=themes)
except Exception as e:
logger.error(f"Error listing themes: {e}")
raise HTTPException(
status_code=500,
detail="Failed to load themes"
)
@router.get("/{theme_id}", response_model=ThemeDetailResponse)
async def get_theme(theme_id: str) -> ThemeDetailResponse:
"""
Get detailed information about a specific theme.
Returns the full theme details including style parameters.
This can be used to preview theme settings or inspect configuration.
Args:
theme_id: The unique identifier of the theme
Returns:
ThemeDetailResponse with full theme details
Raises:
404: Theme not found
"""
try:
theme_service = get_theme_service()
theme = theme_service.get_theme(theme_id)
if theme is None:
raise HTTPException(
status_code=404,
detail=f"Theme not found: {theme_id}"
)
return ThemeDetailResponse(theme=theme)
except HTTPException:
raise
except Exception as e:
logger.error(f"Error getting theme {theme_id}: {e}")
raise HTTPException(
status_code=500,
detail="Failed to load theme"
)
@router.get("/{theme_id}/preview")
async def get_theme_preview(theme_id: str) -> dict:
"""
Get a signed URL for the theme's preview image.
This endpoint is useful when you need just the preview URL
without loading the full theme details.
Args:
theme_id: The unique identifier of the theme
Returns:
Object with preview_url field
Raises:
404: Theme not found or no preview available
"""
try:
theme_service = get_theme_service()
if not theme_service.theme_exists(theme_id):
raise HTTPException(
status_code=404,
detail=f"Theme not found: {theme_id}"
)
theme = theme_service.get_theme(theme_id)
if theme is None or theme.preview_url is None:
raise HTTPException(
status_code=404,
detail=f"No preview available for theme: {theme_id}"
)
return {"preview_url": theme.preview_url}
except HTTPException:
raise
except Exception as e:
logger.error(f"Error getting preview for theme {theme_id}: {e}")
raise HTTPException(
status_code=500,
detail="Failed to load theme preview"
)
@router.get("/{theme_id}/youtube-description")
async def get_theme_youtube_description(theme_id: str) -> dict:
"""
Get the YouTube description template for a theme.
Some themes include a default YouTube video description template
that can be used when uploading videos.
Args:
theme_id: The unique identifier of the theme
Returns:
Object with description field containing the template text,
or null if no template is available
Raises:
404: Theme not found
"""
try:
theme_service = get_theme_service()
if not theme_service.theme_exists(theme_id):
raise HTTPException(
status_code=404,
detail=f"Theme not found: {theme_id}"
)
description = theme_service.get_youtube_description(theme_id)
return {"description": description}
except HTTPException:
raise
except Exception as e:
logger.error(f"Error getting YouTube description for theme {theme_id}: {e}")
raise HTTPException(
status_code=500,
detail="Failed to load YouTube description"
)
-144
View File
@@ -1,144 +0,0 @@
"""
Configuration management for the karaoke generation backend.
"""
import os
import logging
from typing import Optional, Dict
from pydantic_settings import BaseSettings
from google.cloud import secretmanager
logger = logging.getLogger(__name__)
class Settings(BaseSettings):
"""Application settings."""
# Google Cloud
google_cloud_project: str = os.getenv("GOOGLE_CLOUD_PROJECT", "")
gcs_bucket_name: str = os.getenv("GCS_BUCKET_NAME", "karaoke-gen-storage")
gcs_temp_bucket: str = os.getenv("GCS_TEMP_BUCKET", "karaoke-gen-temp")
gcs_output_bucket: str = os.getenv("GCS_OUTPUT_BUCKET", "karaoke-gen-outputs")
firestore_collection: str = os.getenv("FIRESTORE_COLLECTION", "jobs")
# Audio Separator API (for GPU processing)
audio_separator_api_url: Optional[str] = os.getenv("AUDIO_SEPARATOR_API_URL")
# External APIs (can be set via env or Secret Manager)
audioshake_api_key: Optional[str] = os.getenv("AUDIOSHAKE_API_KEY")
genius_api_key: Optional[str] = os.getenv("GENIUS_API_KEY")
spotify_cookie: Optional[str] = os.getenv("SPOTIFY_COOKIE_SP_DC")
rapidapi_key: Optional[str] = os.getenv("RAPIDAPI_KEY")
# Authentication
admin_tokens: Optional[str] = os.getenv("ADMIN_TOKENS") # Comma-separated list
# Application
environment: str = os.getenv("ENVIRONMENT", "development")
log_level: str = os.getenv("LOG_LEVEL", "INFO")
# Processing
max_concurrent_jobs: int = int(os.getenv("MAX_CONCURRENT_JOBS", "5"))
job_timeout_seconds: int = int(os.getenv("JOB_TIMEOUT_SECONDS", "3600"))
# Cloud Tasks (for scalable worker coordination)
# When enabled, workers are triggered via Cloud Tasks for guaranteed delivery
# When disabled (default), workers are triggered via direct HTTP (for development)
enable_cloud_tasks: bool = os.getenv("ENABLE_CLOUD_TASKS", "false").lower() in ("true", "1", "yes")
gcp_region: str = os.getenv("GCP_REGION", "us-central1")
# Cloud Run Jobs (for long-running video encoding)
# When enabled AND enable_cloud_tasks is true, video worker uses Cloud Run Jobs
# instead of Cloud Tasks. This supports encoding times >30 minutes (up to 24 hours).
# Default is false - Cloud Tasks is sufficient for most videos (15-20 min).
use_cloud_run_jobs_for_video: bool = os.getenv("USE_CLOUD_RUN_JOBS_FOR_VIDEO", "false").lower() in ("true", "1", "yes")
# Storage paths
temp_dir: str = os.getenv("TEMP_DIR", "/tmp/karaoke-gen")
# Flacfetch remote service (for torrent downloads)
# When configured, audio search uses the remote flacfetch HTTP API instead of local flacfetch.
# This is required for torrent downloads since Cloud Run doesn't support BitTorrent.
flacfetch_api_url: Optional[str] = os.getenv("FLACFETCH_API_URL") # e.g., http://10.0.0.5:8080
flacfetch_api_key: Optional[str] = os.getenv("FLACFETCH_API_KEY")
# Default distribution settings (can be overridden per-request)
default_dropbox_path: Optional[str] = os.getenv("DEFAULT_DROPBOX_PATH")
default_gdrive_folder_id: Optional[str] = os.getenv("DEFAULT_GDRIVE_FOLDER_ID")
# Strip whitespace/newlines from webhook URL - common issue when env vars are set with trailing newlines
default_discord_webhook_url: Optional[str] = (
os.getenv("DEFAULT_DISCORD_WEBHOOK_URL", "").strip() or None
)
# Default values for web service jobs (YouTube/Dropbox distribution)
default_enable_youtube_upload: bool = os.getenv("DEFAULT_ENABLE_YOUTUBE_UPLOAD", "false").lower() in ("true", "1", "yes")
default_brand_prefix: Optional[str] = os.getenv("DEFAULT_BRAND_PREFIX")
default_youtube_description: str = os.getenv(
"DEFAULT_YOUTUBE_DESCRIPTION",
"Karaoke video created with Nomad Karaoke (https://nomadkaraoke.com)\n\n"
"AI-powered vocal separation and synchronized lyrics.\n\n"
"#karaoke #music #singing #instrumental #lyrics"
)
# Secret Manager cache
_secret_cache: Dict[str, str] = {}
class Config:
env_file = ".env"
case_sensitive = False
def get_secret(self, secret_id: str) -> Optional[str]:
"""
Get a secret from Google Secret Manager.
Caches secrets in memory to avoid repeated API calls.
Falls back to environment variables if Secret Manager unavailable.
Args:
secret_id: Secret name (e.g., "audioshake-api-key")
Returns:
Secret value or None if not found
"""
# Check cache first
if secret_id in self._secret_cache:
return self._secret_cache[secret_id]
# Check environment variable (development mode)
env_var = secret_id.upper().replace('-', '_')
env_value = os.getenv(env_var)
if env_value:
logger.debug(f"Using {secret_id} from environment variable")
self._secret_cache[secret_id] = env_value
return env_value
# Try Secret Manager (production mode)
if not self.google_cloud_project:
logger.warning(f"Cannot fetch secret {secret_id}: No GCP project configured")
return None
try:
client = secretmanager.SecretManagerServiceClient()
name = f"projects/{self.google_cloud_project}/secrets/{secret_id}/versions/latest"
response = client.access_secret_version(request={"name": name})
# Strip whitespace/newlines - common issue when secrets are created with trailing newlines
secret_value = response.payload.data.decode('UTF-8').strip()
# Cache it
self._secret_cache[secret_id] = secret_value
logger.info(f"Loaded secret {secret_id} from Secret Manager")
return secret_value
except Exception as e:
logger.error(f"Failed to load secret {secret_id} from Secret Manager: {e}")
return None
# Global settings instance
settings = Settings()
def get_settings() -> Settings:
"""Get the global settings instance."""
return settings
-127
View File
@@ -1,127 +0,0 @@
"""
FastAPI application entry point for karaoke generation backend.
"""
import logging
from contextlib import asynccontextmanager
from fastapi import FastAPI
from fastapi.middleware.cors import CORSMiddleware
from backend.config import settings
from backend.api.routes import health, jobs, internal, file_upload, review, auth, audio_search, themes
from backend.services.tracing import setup_tracing, instrument_app, get_current_trace_id
from backend.services.structured_logging import setup_structured_logging
from backend.version import VERSION
# Configure structured logging (JSON in Cloud Run, human-readable locally)
# This must happen before any logging calls
setup_structured_logging()
logger = logging.getLogger(__name__)
# Initialize OpenTelemetry tracing (must happen before app creation)
tracing_enabled = setup_tracing(
service_name="karaoke-backend",
service_version=VERSION,
enable_in_dev=False, # Set to True to enable tracing locally
)
async def validate_credentials_on_startup():
"""Validate OAuth credentials on startup and send alerts if needed."""
try:
from backend.services.credential_manager import get_credential_manager, CredentialStatus
manager = get_credential_manager()
results = manager.check_all_credentials()
invalid_services = [
result for result in results.values()
if result.status in (CredentialStatus.INVALID, CredentialStatus.EXPIRED)
]
if invalid_services:
logger.warning(f"Some OAuth credentials need attention:")
for result in invalid_services:
logger.warning(f" - {result.service}: {result.message}")
# Try to send Discord alert
discord_url = settings.get_secret("discord-alert-webhook") if hasattr(settings, 'get_secret') else None
if discord_url:
manager.send_credential_alert(invalid_services, discord_url)
logger.info("Sent credential alert to Discord")
else:
logger.info("All OAuth credentials validated successfully")
except Exception as e:
logger.error(f"Failed to validate credentials on startup: {e}")
@asynccontextmanager
async def lifespan(app: FastAPI):
"""Lifespan event handler for startup and shutdown."""
# Startup
logger.info("Starting karaoke generation backend")
logger.info(f"Environment: {settings.environment}")
logger.info(f"GCS Bucket: {settings.gcs_bucket_name}")
logger.info(f"Tracing enabled: {tracing_enabled}")
# Validate OAuth credentials (non-blocking)
try:
await validate_credentials_on_startup()
except Exception as e:
logger.error(f"Credential validation failed: {e}")
yield
# Shutdown
logger.info("Shutting down karaoke generation backend")
# Create FastAPI app
app = FastAPI(
title="Karaoke Generator API",
description="Backend API for web-based karaoke video generation",
version=VERSION,
lifespan=lifespan
)
# Instrument FastAPI with OpenTelemetry (adds automatic spans for all requests)
if tracing_enabled:
instrument_app(app)
# Configure CORS
app.add_middleware(
CORSMiddleware,
allow_origins=["*"], # Configure this properly for production
allow_credentials=True,
allow_methods=["*"],
allow_headers=["*"],
)
# Include routers
app.include_router(health.router, prefix="/api")
app.include_router(jobs.router, prefix="/api")
app.include_router(file_upload.router, prefix="/api") # File upload endpoint
app.include_router(internal.router, prefix="/api") # Internal worker endpoints
app.include_router(review.router, prefix="/api") # Review UI compatibility endpoints
app.include_router(auth.router, prefix="/api") # OAuth credential management
app.include_router(audio_search.router, prefix="/api") # Audio search (artist+title mode)
app.include_router(themes.router, prefix="/api") # Theme selection for styles
@app.get("/")
async def root():
"""Root endpoint."""
return {
"service": "karaoke-gen-backend",
"version": VERSION,
"status": "running"
}
if __name__ == "__main__":
import uvicorn
uvicorn.run(app, host="0.0.0.0", port=8080)
View File
Whitespace-only changes.
-499
View File
@@ -1,499 +0,0 @@
"""
Job data models for karaoke generation.
This module defines the complete state machine for karaoke generation jobs,
mirroring the 8-stage CLI workflow with human-in-the-loop interaction points.
"""
from typing import Optional, Dict, Any, List
from datetime import datetime
from enum import Enum
from pydantic import BaseModel, Field, validator
class JobStatus(str, Enum):
"""
Job status enumeration - Complete state machine.
The workflow has 8 main stages with 3 human interaction points:
1. Input & Setup (may include audio source selection)
2. Parallel Processing (audio + lyrics)
3. Title/End Screen Generation
4. Countdown Padding Synchronization
5. Human Review (BLOCKING)
6. Instrumental Selection (BLOCKING)
7. Video Finalization
8. Distribution
"""
# Initial states
PENDING = "pending" # Job created, queued for processing
# Audio search states (for artist+title search mode)
SEARCHING_AUDIO = "searching_audio" # Searching for audio sources via flacfetch
AWAITING_AUDIO_SELECTION = "awaiting_audio_selection" # ⚠️ WAITING FOR USER - select audio source
DOWNLOADING_AUDIO = "downloading_audio" # Downloading selected audio from source
DOWNLOADING = "downloading" # Downloading from URL or processing upload
# Stage 2a: Audio separation (parallel track 1)
SEPARATING_STAGE1 = "separating_stage1" # Clean instrumental separation (Modal API)
SEPARATING_STAGE2 = "separating_stage2" # Backing vocals separation (Modal API)
AUDIO_COMPLETE = "audio_complete" # All audio stems ready
# Stage 2b: Lyrics processing (parallel track 2)
TRANSCRIBING = "transcribing" # AudioShake API transcription
CORRECTING = "correcting" # Automatic lyrics correction
LYRICS_COMPLETE = "lyrics_complete" # Corrections JSON ready
# Stage 3: Title/End screens
GENERATING_SCREENS = "generating_screens" # Creating title and end screen videos
# Stage 4: Countdown padding (automatic)
APPLYING_PADDING = "applying_padding" # Synchronizing countdown padding
# Stage 5: Human review (BLOCKING)
AWAITING_REVIEW = "awaiting_review" # ⚠️ WAITING FOR USER - lyrics review needed
IN_REVIEW = "in_review" # User is actively reviewing lyrics
REVIEW_COMPLETE = "review_complete" # User submitted corrected lyrics
# Stage 5.5: Render video with corrected lyrics (post-review)
RENDERING_VIDEO = "rendering_video" # Using OutputGenerator to create with_vocals.mkv
# Stage 6: Instrumental selection (BLOCKING)
AWAITING_INSTRUMENTAL_SELECTION = "awaiting_instrumental_selection" # ⚠️ WAITING FOR USER
INSTRUMENTAL_SELECTED = "instrumental_selected" # User made selection
# Stage 7: Video generation and finalization
GENERATING_VIDEO = "generating_video" # Creating initial karaoke video
ENCODING = "encoding" # Multi-format video encoding (Cloud Build)
PACKAGING = "packaging" # CDG/TXT ZIP generation
# Stage 8: Distribution (optional)
UPLOADING = "uploading" # YouTube/Dropbox upload
NOTIFYING = "notifying" # Discord/Email notifications
# Terminal states
COMPLETE = "complete" # All processing finished successfully
PREP_COMPLETE = "prep_complete" # Prep-only job completed (stops after review)
FAILED = "failed" # Unrecoverable error occurred
CANCELLED = "cancelled" # User cancelled the job
# Legacy compatibility (will be removed)
QUEUED = "queued" # Deprecated: use PENDING
PROCESSING = "processing" # Deprecated: use specific states
READY_FOR_FINALIZATION = "ready_for_finalization" # Deprecated
FINALIZING = "finalizing" # Deprecated: use ENCODING/PACKAGING
ERROR = "error" # Deprecated: use FAILED
# Valid state transitions
STATE_TRANSITIONS = {
# PENDING can go to DOWNLOADING (file upload), SEARCHING_AUDIO (artist+title search), or AWAITING_INSTRUMENTAL_SELECTION (finalise-only)
JobStatus.PENDING: [JobStatus.DOWNLOADING, JobStatus.SEARCHING_AUDIO, JobStatus.AWAITING_INSTRUMENTAL_SELECTION, JobStatus.FAILED, JobStatus.CANCELLED],
# Audio search flow (for artist+title search mode)
JobStatus.SEARCHING_AUDIO: [JobStatus.AWAITING_AUDIO_SELECTION, JobStatus.DOWNLOADING_AUDIO, JobStatus.FAILED],
JobStatus.AWAITING_AUDIO_SELECTION: [JobStatus.DOWNLOADING_AUDIO, JobStatus.FAILED, JobStatus.CANCELLED],
JobStatus.DOWNLOADING_AUDIO: [JobStatus.DOWNLOADING, JobStatus.FAILED],
# DOWNLOADING allows parallel processing (audio + lyrics) and then screens when both complete
JobStatus.DOWNLOADING: [JobStatus.SEPARATING_STAGE1, JobStatus.TRANSCRIBING, JobStatus.GENERATING_SCREENS, JobStatus.FAILED],
# Audio separation flow
JobStatus.SEPARATING_STAGE1: [JobStatus.SEPARATING_STAGE2, JobStatus.FAILED],
JobStatus.SEPARATING_STAGE2: [JobStatus.AUDIO_COMPLETE, JobStatus.FAILED],
JobStatus.AUDIO_COMPLETE: [JobStatus.GENERATING_SCREENS, JobStatus.FAILED],
# Lyrics flow
JobStatus.TRANSCRIBING: [JobStatus.CORRECTING, JobStatus.FAILED],
JobStatus.CORRECTING: [JobStatus.LYRICS_COMPLETE, JobStatus.FAILED],
JobStatus.LYRICS_COMPLETE: [JobStatus.GENERATING_SCREENS, JobStatus.FAILED],
# Post-parallel processing
JobStatus.GENERATING_SCREENS: [JobStatus.APPLYING_PADDING, JobStatus.AWAITING_REVIEW, JobStatus.AWAITING_INSTRUMENTAL_SELECTION, JobStatus.FAILED],
JobStatus.APPLYING_PADDING: [JobStatus.AWAITING_REVIEW, JobStatus.FAILED],
# Human review flow
# AWAITING_REVIEW can go directly to REVIEW_COMPLETE (quick review) or to IN_REVIEW (editing)
JobStatus.AWAITING_REVIEW: [JobStatus.IN_REVIEW, JobStatus.REVIEW_COMPLETE, JobStatus.FAILED, JobStatus.CANCELLED],
JobStatus.IN_REVIEW: [JobStatus.REVIEW_COMPLETE, JobStatus.AWAITING_REVIEW, JobStatus.FAILED],
JobStatus.REVIEW_COMPLETE: [JobStatus.RENDERING_VIDEO, JobStatus.PREP_COMPLETE, JobStatus.FAILED], # PREP_COMPLETE for prep-only jobs
# Video rendering (post-review)
JobStatus.RENDERING_VIDEO: [JobStatus.AWAITING_INSTRUMENTAL_SELECTION, JobStatus.PREP_COMPLETE, JobStatus.FAILED],
# Instrumental selection flow
JobStatus.AWAITING_INSTRUMENTAL_SELECTION: [JobStatus.INSTRUMENTAL_SELECTED, JobStatus.FAILED, JobStatus.CANCELLED],
JobStatus.INSTRUMENTAL_SELECTED: [JobStatus.GENERATING_VIDEO, JobStatus.FAILED],
# Video generation flow
JobStatus.GENERATING_VIDEO: [JobStatus.ENCODING, JobStatus.FAILED],
JobStatus.ENCODING: [JobStatus.PACKAGING, JobStatus.COMPLETE, JobStatus.FAILED],
JobStatus.PACKAGING: [JobStatus.UPLOADING, JobStatus.COMPLETE, JobStatus.FAILED],
# Distribution flow
JobStatus.UPLOADING: [JobStatus.NOTIFYING, JobStatus.COMPLETE, JobStatus.FAILED],
JobStatus.NOTIFYING: [JobStatus.COMPLETE, JobStatus.FAILED],
# Terminal states - COMPLETE, PREP_COMPLETE have no transitions
# FAILED and CANCELLED allow retry transitions to resume from checkpoints
# PREP_COMPLETE allows finalise-only continuation
JobStatus.COMPLETE: [],
JobStatus.PREP_COMPLETE: [JobStatus.AWAITING_INSTRUMENTAL_SELECTION, JobStatus.FAILED], # Finalise-only continues from here
JobStatus.FAILED: [
JobStatus.DOWNLOADING, # Retry from beginning (if input audio exists)
JobStatus.INSTRUMENTAL_SELECTED, # Retry from video generation
JobStatus.REVIEW_COMPLETE, # Retry from render stage
JobStatus.LYRICS_COMPLETE, # Retry from screens generation
],
JobStatus.CANCELLED: [
JobStatus.DOWNLOADING, # Retry from beginning (if input audio exists)
JobStatus.INSTRUMENTAL_SELECTED, # Retry from video generation
JobStatus.REVIEW_COMPLETE, # Retry from render stage
JobStatus.LYRICS_COMPLETE, # Retry from screens generation
],
# Legacy states (for backward compatibility)
JobStatus.QUEUED: [JobStatus.PENDING],
JobStatus.PROCESSING: [JobStatus.SEPARATING_STAGE1, JobStatus.TRANSCRIBING],
JobStatus.READY_FOR_FINALIZATION: [JobStatus.GENERATING_VIDEO],
JobStatus.FINALIZING: [JobStatus.ENCODING],
JobStatus.ERROR: [JobStatus.FAILED],
}
class TimelineEvent(BaseModel):
"""Timeline event for job progress tracking."""
status: str
timestamp: str
progress: Optional[int] = None
message: Optional[str] = None
class LogEntry(BaseModel):
"""Worker log entry for debugging and monitoring."""
timestamp: str
level: str # DEBUG, INFO, WARNING, ERROR
worker: str # audio, lyrics, screens, video, render
message: str
class Job(BaseModel):
"""
Complete job data model.
Tracks the full lifecycle of a karaoke generation job from submission
through all 8 stages to completion or failure.
"""
job_id: str
status: JobStatus
progress: int = 0 # 0-100 percentage for UI display
created_at: datetime
updated_at: datetime
# Input data
url: Optional[str] = None # YouTube URL (if provided)
artist: Optional[str] = None
title: Optional[str] = None
filename: Optional[str] = None # Original uploaded filename
input_media_gcs_path: Optional[str] = None # GCS path to uploaded file
# User preferences
enable_cdg: bool = False # Generate CDG+MP3 package (requires style config)
enable_txt: bool = False # Generate TXT+MP3 package (requires style config)
enable_youtube_upload: bool = False # Upload to YouTube
youtube_description: Optional[str] = None # YouTube video description
webhook_url: Optional[str] = None # Webhook for notifications
user_email: Optional[str] = None # Email for notifications
# Theme configuration (pre-made themes from GCS)
theme_id: Optional[str] = None # Theme identifier (e.g., "nomad", "default")
color_overrides: Dict[str, str] = Field(default_factory=dict)
"""
User color overrides applied on top of theme. Keys:
- artist_color: Hex color for artist name (#RRGGBB)
- title_color: Hex color for song title
- sung_lyrics_color: Hex color for highlighted lyrics
- unsung_lyrics_color: Hex color for unhighlighted lyrics
"""
# Style configuration (uploaded files - used when theme_id is not set)
style_params_gcs_path: Optional[str] = None # GCS path to style_params.json
style_assets: Dict[str, str] = Field(default_factory=dict)
"""
GCS paths to style asset files:
{
"intro_background": "gs://bucket/jobs/{job_id}/style/intro_bg.png",
"karaoke_background": "gs://bucket/jobs/{job_id}/style/karaoke_bg.png",
"end_background": "gs://bucket/jobs/{job_id}/style/end_bg.png",
"font": "gs://bucket/jobs/{job_id}/style/font.ttf",
"cdg_instrumental_background": "gs://bucket/jobs/{job_id}/style/cdg_instr.png",
"cdg_title_background": "gs://bucket/jobs/{job_id}/style/cdg_title.png",
"cdg_outro_background": "gs://bucket/jobs/{job_id}/style/cdg_outro.png"
}
"""
# Finalisation configuration
brand_prefix: Optional[str] = None # Brand code prefix (e.g., "NOMAD")
discord_webhook_url: Optional[str] = None # Discord notification webhook
youtube_description_template: Optional[str] = None # YouTube description template text
# Distribution configuration (native API - for remote CLI)
dropbox_path: Optional[str] = None # Dropbox folder path for organized output (e.g., "/Karaoke/Tracks-Organized")
gdrive_folder_id: Optional[str] = None # Google Drive folder ID for public share uploads
# Legacy distribution configuration (rclone - for local CLI backward compat)
organised_dir_rclone_root: Optional[str] = None # Deprecated: use dropbox_path instead
# Lyrics configuration (overrides for search/transcription)
lyrics_artist: Optional[str] = None # Override artist name for lyrics search
lyrics_title: Optional[str] = None # Override title for lyrics search
lyrics_file_gcs_path: Optional[str] = None # GCS path to user-provided lyrics file
subtitle_offset_ms: int = 0 # Offset for subtitle timing (positive = delay)
# Audio separation model configuration
clean_instrumental_model: Optional[str] = None # Model for clean instrumental separation
backing_vocals_models: Optional[List[str]] = None # Models for backing vocals separation
other_stems_models: Optional[List[str]] = None # Models for other stems (bass, drums, etc.)
# Existing instrumental configuration (Batch 3)
existing_instrumental_gcs_path: Optional[str] = None # GCS path to user-provided instrumental file
# Audio search configuration (Batch 5 - artist+title search mode)
audio_search_artist: Optional[str] = None # Artist name used for audio search
audio_search_title: Optional[str] = None # Title used for audio search
auto_download: bool = False # Auto-select best audio source (skip selection)
# Two-phase workflow configuration (Batch 6)
prep_only: bool = False # Stop after review, don't run finalisation
finalise_only: bool = False # Skip prep, run only finalisation (requires uploaded prep outputs)
keep_brand_code: Optional[str] = None # Preserve existing brand code instead of generating new one
# Review authentication (Batch 7)
review_token: Optional[str] = None # Job-scoped token for lyrics review UI access (generated when entering AWAITING_REVIEW)
review_token_expires_at: Optional[datetime] = None # Token expiry time (optional, for extra security)
instrumental_token: Optional[str] = None # Job-scoped token for instrumental review UI access (generated when entering AWAITING_INSTRUMENTAL_SELECTION)
instrumental_token_expires_at: Optional[datetime] = None # Token expiry time
# Processing state
track_output_dir: Optional[str] = None # Local output directory (temp)
audio_hash: Optional[str] = None # Hash for deduplication
# State-specific data (JSON field for stage-specific metadata)
state_data: Dict[str, Any] = Field(default_factory=dict)
"""
Stage-specific metadata. Examples:
- audio_complete: {"stems": {"clean": "gs://...", "backing": "gs://..."}}
- lyrics_complete: {"corrections_url": "gs://...", "audio_url": "gs://..."}
- review_complete: {"corrected_lyrics": {...}}
- instrumental_selected: {"selection": "clean" | "with_backing"}
- encoding: {"build_id": "...", "progress": 45}
"""
# Timeline tracking
timeline: List[TimelineEvent] = Field(default_factory=list)
# Worker logs for debugging (limited to last N entries to avoid document size issues)
worker_logs: List[LogEntry] = Field(default_factory=list)
# File URLs (GCS storage)
file_urls: Dict[str, Any] = Field(default_factory=dict)
"""
File storage URLs. Structure:
{
"input": "gs://bucket/jobs/{job_id}/input.flac",
"stems": {
"instrumental_clean": "gs://...",
"instrumental_with_backing": "gs://...",
"vocals": "gs://...",
"backing_vocals": "gs://...",
"lead_vocals": "gs://...",
"bass": "gs://...",
"drums": "gs://...",
"guitar": "gs://...",
"piano": "gs://...",
"other": "gs://..."
},
"lyrics": {
"corrections": "gs://bucket/jobs/{job_id}/lyrics/corrections.json",
"audio": "gs://bucket/jobs/{job_id}/lyrics/audio.flac",
"lrc": "gs://...",
"ass": "gs://..."
},
"screens": {
"title": "gs://bucket/jobs/{job_id}/screens/title.mov",
"end": "gs://bucket/jobs/{job_id}/screens/end.mov"
},
"videos": {
"with_vocals": "gs://bucket/jobs/{job_id}/videos/with_vocals.mkv"
},
"finals": {
"lossless_4k_mp4": "gs://...",
"lossless_4k_mkv": "gs://...",
"lossy_4k_mp4": "gs://...",
"lossy_720p_mp4": "gs://..."
},
"packages": {
"cdg_zip": "gs://...",
"txt_zip": "gs://..."
},
"youtube": {
"url": "https://youtube.com/watch?v=...",
"video_id": "..."
}
}
"""
# Results (for backward compatibility, will be deprecated)
output_files: Dict[str, str] = Field(default_factory=dict)
download_urls: Dict[str, str] = Field(default_factory=dict)
# Error handling
error_message: Optional[str] = None
error_details: Optional[Dict[str, Any]] = None # Structured error information
retry_count: int = 0 # Number of retry attempts
# Worker tracking
worker_ids: Dict[str, str] = Field(default_factory=dict)
"""
IDs of background workers/jobs:
{
"audio_worker": "cloud-run-request-id",
"lyrics_worker": "cloud-run-request-id",
"video_encoder": "cloud-build-id",
"distribution": "cloud-run-request-id"
}
"""
# Request metadata (captured at job creation for tracking and filtering)
request_metadata: Dict[str, Any] = Field(default_factory=dict)
"""
Metadata captured from the original API request.
Used for tracking, filtering, and operational management.
Standard fields:
{
"client_ip": "192.168.1.1", # IP address of the client
"user_agent": "karaoke-gen-remote/0.71.0", # User-Agent header
"environment": "test", # From X-Environment header (test/production/development)
"client_id": "cli-user-123", # From X-Client-ID header (customer/user identifier)
"server_version": "0.71.0", # Server version at job creation
"created_from": "upload", # "upload" (file) or "url" (YouTube URL)
"custom_headers": { # All X-* headers for extensibility
"X-Environment": "test",
"X-Client-ID": "cli-user-123",
"X-Request-ID": "abc-123"
}
}
Use cases:
- Filter test vs production jobs
- Track jobs by customer/client
- Debug issues with specific clients
- Bulk cleanup of test jobs
"""
# Note: Status transition validation is handled by JobManager.validate_state_transition()
# which is called before status updates. The Job model does not validate transitions
# because Firestore updates happen directly without reconstructing the model.
class Config:
use_enum_values = True
class JobCreate(BaseModel):
"""
Job creation request.
Either `url` OR file upload is required (file upload handled separately).
Artist and title are optional - will be auto-detected from YouTube if not provided.
"""
url: Optional[str] = None
artist: Optional[str] = None
title: Optional[str] = None
filename: Optional[str] = None # Original uploaded filename
# Optional preferences
enable_cdg: bool = False # Requires style config
enable_txt: bool = False # Requires style config
enable_youtube_upload: bool = False
youtube_description: Optional[str] = None
webhook_url: Optional[str] = None
user_email: Optional[str] = None
# Theme configuration (pre-made themes from GCS)
theme_id: Optional[str] = None # Theme identifier (e.g., "nomad", "default")
color_overrides: Dict[str, str] = Field(default_factory=dict)
"""
User color overrides applied on top of theme. Keys:
- artist_color: Hex color for artist name (#RRGGBB)
- title_color: Hex color for song title
- sung_lyrics_color: Hex color for highlighted lyrics
- unsung_lyrics_color: Hex color for unhighlighted lyrics
"""
# Style configuration (will be populated after file upload, or from theme)
style_params_gcs_path: Optional[str] = None
style_assets: Dict[str, str] = Field(default_factory=dict)
# Finalisation configuration
brand_prefix: Optional[str] = None
discord_webhook_url: Optional[str] = None
youtube_description_template: Optional[str] = None
# Distribution configuration (native API - for remote CLI)
dropbox_path: Optional[str] = None # Dropbox folder path for organized output
gdrive_folder_id: Optional[str] = None # Google Drive folder ID for public share uploads
# Legacy (rclone - deprecated, use dropbox_path instead)
organised_dir_rclone_root: Optional[str] = None
# Lyrics configuration (overrides for search/transcription)
lyrics_artist: Optional[str] = None # Override artist name for lyrics search
lyrics_title: Optional[str] = None # Override title for lyrics search
lyrics_file_gcs_path: Optional[str] = None # GCS path to user-provided lyrics file
subtitle_offset_ms: int = 0 # Offset for subtitle timing (positive = delay)
# Audio separation model configuration
clean_instrumental_model: Optional[str] = None # Model for clean instrumental separation
backing_vocals_models: Optional[List[str]] = None # Models for backing vocals separation
other_stems_models: Optional[List[str]] = None # Models for other stems (bass, drums, etc.)
# Existing instrumental configuration (Batch 3)
existing_instrumental_gcs_path: Optional[str] = None # GCS path to user-provided instrumental file
# Audio search configuration (Batch 5 - artist+title search mode)
audio_search_artist: Optional[str] = None # Artist name used for audio search
audio_search_title: Optional[str] = None # Title used for audio search
auto_download: bool = False # Auto-select best audio source (skip selection)
# Two-phase workflow configuration (Batch 6)
prep_only: bool = False # Stop after review, don't run finalisation
finalise_only: bool = False # Skip prep, run only finalisation
keep_brand_code: Optional[str] = None # Preserve existing brand code instead of generating new one
# Request metadata (set by API endpoint from request headers)
request_metadata: Dict[str, Any] = Field(default_factory=dict)
"""
Populated by the API endpoint with request context:
- client_ip: Client IP address
- user_agent: User-Agent header
- environment: From X-Environment header (test/production/development)
- client_id: From X-Client-ID header
- server_version: Current server version
- custom_headers: All X-* headers
"""
@validator('url', 'artist', 'title')
def validate_inputs(cls, v):
"""Validate string inputs are not empty."""
if v is not None and isinstance(v, str) and not v.strip():
raise ValueError("Field cannot be empty string")
return v.strip() if isinstance(v, str) else v
class JobResponse(BaseModel):
"""Job response model."""
status: str
job_id: str
message: str
-123
View File
@@ -1,123 +0,0 @@
"""
API request models for karaoke generation endpoints.
"""
from typing import Optional, Dict, Any, List
from pydantic import BaseModel, HttpUrl, validator
class URLSubmissionRequest(BaseModel):
"""Request to submit a job from a URL (YouTube, etc.)."""
url: HttpUrl
artist: Optional[str] = None # Auto-detected if not provided
title: Optional[str] = None # Auto-detected if not provided
# Optional preferences
enable_cdg: bool = False # Requires style config
enable_txt: bool = False # Requires style config
enable_youtube_upload: bool = False
youtube_description: Optional[str] = None
webhook_url: Optional[str] = None
user_email: Optional[str] = None
class UploadSubmissionRequest(BaseModel):
"""Request to submit a job from an uploaded file."""
artist: str
title: str
# Optional preferences
enable_cdg: bool = False # Requires style config
enable_txt: bool = False # Requires style config
enable_youtube_upload: bool = False
youtube_description: Optional[str] = None
webhook_url: Optional[str] = None
user_email: Optional[str] = None
class CorrectionsSubmission(BaseModel):
"""
Request to submit corrected lyrics after human review.
This is the critical human-in-the-loop interaction point.
The corrections data comes from the lyrics-transcriber review interface.
"""
corrections: Dict[str, Any] # Full corrections JSON from frontend
user_notes: Optional[str] = None # Optional notes from reviewer
@validator('corrections')
def validate_corrections_format(cls, v):
"""Validate corrections has required fields."""
required_fields = ['lines', 'metadata']
for field in required_fields:
if field not in v:
raise ValueError(f"Corrections must include '{field}' field")
return v
class InstrumentalSelection(BaseModel):
"""
Request to select instrumental audio option.
This is the second critical human-in-the-loop interaction point.
User chooses between clean instrumental, instrumental with backing vocals,
or a custom instrumental (created via create-custom-instrumental endpoint).
"""
selection: str # "clean", "with_backing", or "custom"
@validator('selection')
def validate_selection(cls, v):
"""Validate selection is a valid option."""
valid_options = ['clean', 'with_backing', 'custom']
if v not in valid_options:
raise ValueError(f"Selection must be one of: {valid_options}")
return v
class MuteRegionRequest(BaseModel):
"""A region to mute in the backing vocals."""
start_seconds: float
end_seconds: float
@validator('start_seconds')
def validate_start(cls, v):
if v < 0:
raise ValueError("start_seconds must be non-negative")
return v
@validator('end_seconds')
def validate_end(cls, v, values):
if 'start_seconds' in values and v <= values['start_seconds']:
raise ValueError("end_seconds must be greater than start_seconds")
return v
class CreateCustomInstrumentalRequest(BaseModel):
"""
Request to create a custom instrumental with muted backing vocal regions.
The mute_regions specify time ranges in the backing vocals track that
should be silenced before mixing with the clean instrumental.
"""
mute_regions: List[MuteRegionRequest]
@validator('mute_regions')
def validate_regions(cls, v):
if not v:
raise ValueError("At least one mute region is required")
return v
class StartReviewRequest(BaseModel):
"""Request to mark job as in-review (user opened interface)."""
pass # No body needed, just triggers state transition
class CancelJobRequest(BaseModel):
"""Request to cancel a job."""
reason: Optional[str] = None
class RetryJobRequest(BaseModel):
"""Request to retry a failed job."""
from_stage: Optional[str] = None # Optional: restart from specific stage
-153
View File
@@ -1,153 +0,0 @@
"""
Theme data models for karaoke-gen.
Themes are pre-created style configurations stored in GCS that users can select
when creating karaoke jobs. Each theme includes:
- Style parameters (fonts, colors, backgrounds, layouts)
- Asset files (images, fonts)
- Preview images
- Optional YouTube description template
"""
from datetime import datetime
from typing import Any, Dict, List, Optional
from pydantic import BaseModel, Field
class ThemeSummary(BaseModel):
"""Summary of a theme for listing purposes."""
id: str = Field(..., description="Unique theme identifier (e.g., 'nomad', 'default')")
name: str = Field(..., description="Human-readable theme name")
description: str = Field(..., description="Brief description of the theme style")
preview_url: Optional[str] = Field(None, description="Signed URL for preview image")
thumbnail_url: Optional[str] = Field(None, description="Signed URL for smaller thumbnail")
is_default: bool = Field(False, description="Whether this is the default theme")
class ThemeMetadata(BaseModel):
"""Metadata for a single theme in the registry."""
id: str
name: str
description: str
is_default: bool = False
created_at: Optional[datetime] = None
class ThemeRegistry(BaseModel):
"""Registry of all available themes, stored as _metadata.json in GCS."""
version: int = 1
themes: List[ThemeMetadata] = Field(default_factory=list)
class ThemeDetail(BaseModel):
"""Full theme details including style parameters."""
id: str
name: str
description: str
preview_url: Optional[str] = None
is_default: bool = False
style_params: Dict[str, Any] = Field(default_factory=dict)
has_youtube_description: bool = False
class ColorOverrides(BaseModel):
"""
User-customizable color overrides that can be applied on top of a theme.
All colors are hex format (#RRGGBB). When applied:
- artist_color: Applied to intro.artist_color, end.artist_color, cdg.artist_color
- title_color: Applied to intro.title_color, end.title_color, cdg.title_color
- sung_lyrics_color: Applied to karaoke.primary_color (converted to RGBA), cdg.active_fill
- unsung_lyrics_color: Applied to karaoke.secondary_color (converted to RGBA), cdg.inactive_fill
"""
artist_color: Optional[str] = Field(
None,
pattern=r"^#[0-9A-Fa-f]{6}$",
description="Hex color for artist name on intro/end screens",
)
title_color: Optional[str] = Field(
None,
pattern=r"^#[0-9A-Fa-f]{6}$",
description="Hex color for song title on intro/end screens",
)
sung_lyrics_color: Optional[str] = Field(
None,
pattern=r"^#[0-9A-Fa-f]{6}$",
description="Hex color for lyrics being sung (highlighted)",
)
unsung_lyrics_color: Optional[str] = Field(
None,
pattern=r"^#[0-9A-Fa-f]{6}$",
description="Hex color for lyrics not yet sung",
)
def has_overrides(self) -> bool:
"""Check if any color overrides are set."""
return any(
[
self.artist_color,
self.title_color,
self.sung_lyrics_color,
self.unsung_lyrics_color,
]
)
def to_dict(self) -> Dict[str, str]:
"""Convert to dict, excluding None values."""
return {k: v for k, v in self.model_dump().items() if v is not None}
class ThemesListResponse(BaseModel):
"""Response from GET /api/themes endpoint."""
themes: List[ThemeSummary]
class ThemeDetailResponse(BaseModel):
"""Response from GET /api/themes/{theme_id} endpoint."""
theme: ThemeDetail
# Utility functions for color conversion
def hex_to_rgba(hex_color: str, alpha: int = 255) -> str:
"""
Convert hex color (#RRGGBB) to RGBA format for ASS subtitles.
ASS subtitle format uses "R, G, B, A" string format.
Args:
hex_color: Hex color string like "#7070F7"
alpha: Alpha value 0-255 (default 255 = opaque)
Returns:
RGBA string like "112, 112, 247, 255"
"""
hex_color = hex_color.lstrip("#")
r = int(hex_color[0:2], 16)
g = int(hex_color[2:4], 16)
b = int(hex_color[4:6], 16)
return f"{r}, {g}, {b}, {alpha}"
def rgba_to_hex(rgba_str: str) -> str:
"""
Convert RGBA format to hex color.
Args:
rgba_str: RGBA string like "112, 112, 247, 255"
Returns:
Hex color string like "#7070F7"
"""
parts = [int(x.strip()) for x in rgba_str.split(",")]
r, g, b = parts[0], parts[1], parts[2]
return f"#{r:02x}{g:02x}{b:02x}"
-29
View File
@@ -1,29 +0,0 @@
[tool.pytest.ini_options]
testpaths = ["tests"]
python_files = ["test_*.py"]
python_classes = ["Test*"]
python_functions = ["test_*"]
addopts = [
"-v",
"--tb=short",
"--strict-markers",
"-ra",
]
markers = [
"slow: marks tests as slow (deselect with '-m \"not slow\"')",
"integration: marks tests as integration tests requiring live service",
]
# Timeout for slow tests (in seconds)
timeout = 300
[tool.coverage.run]
source = ["backend"]
omit = ["*/tests/*", "*/test_*"]
[tool.coverage.report]
precision = 2
show_missing = true
skip_covered = false
-93
View File
@@ -1,93 +0,0 @@
#!/bin/bash
# Quick syntax and structure check (no dependencies required)
# Run this before every deployment!
set -e
echo "🔍 Quick Backend Check (no dependencies required)"
echo "=================================================="
echo ""
# Change to backend directory
cd "$(dirname "$0")"
# Colors
RED='\033[0;31m'
GREEN='\033[0;32m'
NC='\033[0m'
ERRORS=0
# Check 1: Python syntax
echo "1. Checking Python syntax..."
for file in $(find . -name "*.py" -not -path "./__pycache__/*" -not -path "./venv/*"); do
if ! python3 -m py_compile "$file" 2>/dev/null; then
echo -e " ${RED}❌ Syntax error in $file${NC}"
python3 -m py_compile "$file"
ERRORS=$((ERRORS+1))
fi
done
if [ $ERRORS -eq 0 ]; then
echo -e " ${GREEN}✅ All Python files have valid syntax${NC}"
fi
# Check 2: No missing imports (basic check)
echo ""
echo "2. Checking for obvious import issues..."
# Check that we don't import deleted modules
if grep -r "from backend.services.processing_service import" --include="*.py" . 2>/dev/null; then
echo -e " ${RED}❌ Found import of deleted processing_service${NC}"
ERRORS=$((ERRORS+1))
else
echo -e " ${GREEN}✅ No imports of deleted modules${NC}"
fi
# Check 3: Required files exist
echo ""
echo "3. Checking required files exist..."
REQUIRED_FILES=(
"main.py"
"config.py"
"requirements.txt"
"Dockerfile"
"api/routes/health.py"
"api/routes/jobs.py"
"api/routes/internal.py"
"api/routes/file_upload.py"
"services/job_manager.py"
"services/firestore_service.py"
"services/storage_service.py"
"services/worker_service.py"
"services/auth_service.py"
"workers/audio_worker.py"
"workers/lyrics_worker.py"
"workers/screens_worker.py"
"workers/video_worker.py"
"models/job.py"
"models/requests.py"
)
for file in "${REQUIRED_FILES[@]}"; do
if [ ! -f "$file" ]; then
echo -e " ${RED}❌ Missing required file: $file${NC}"
ERRORS=$((ERRORS+1))
fi
done
if [ $ERRORS -eq 0 ]; then
echo -e " ${GREEN}✅ All required files exist${NC}"
fi
# Summary
echo ""
echo "=================================================="
if [ $ERRORS -eq 0 ]; then
echo -e "${GREEN}✅ All checks passed! Safe to deploy.${NC}"
exit 0
else
echo -e "${RED}❌ $ERRORS error(s) found. Fix before deploying!${NC}"
exit 1
fi
-29
View File
@@ -1,29 +0,0 @@
# DEPRECATED: This file is no longer used for Docker builds or CI.
# Docker and CI both use poetry/pyproject.toml directly.
# This file is kept for reference only.
fastapi>=0.104.0,<1.0.0
uvicorn[standard]>=0.24.0,<1.0.0
pydantic>=2.5.0,<3.0.0
pydantic-settings>=2.1.0,<3.0.0
python-multipart>=0.0.6,<1.0.0
httpx>=0.25.0,<1.0.0
google-cloud-firestore>=2.14.0,<3.0.0
google-cloud-storage>=2.14.0,<3.0.0
google-cloud-secret-manager>=2.18.0,<3.0.0
google-cloud-tasks>=2.14.0,<3.0.0
google-cloud-run>=0.10.0,<1.0.0
python-dotenv>=1.0.0,<2.0.0
# Audio processing (reuses karaoke_gen library)
audio-separator[remote]>=0.18.0 # Remote Modal API client
pydub>=0.25.1 # Audio manipulation (used by karaoke_gen)
yt-dlp>=2024.0.0 # YouTube downloads (future use)
# Cloud storage native APIs
dropbox>=11.0.0 # Dropbox Python SDK for native API access
# google-api-python-client already installed via karaoke_gen for YouTube
# karaoke-gen dependencies (will be installed from parent dir in Dockerfile)
# Using explicit dependencies instead of -e ../
-60
View File
@@ -1,60 +0,0 @@
#!/bin/bash
# Backend Test Runner
# Runs integration tests against the deployed Cloud Run service
set -e
echo "🧪 Backend Integration Test Suite"
echo "=================================="
echo ""
# Check prerequisites
echo "Checking prerequisites..."
if ! command -v gcloud &> /dev/null; then
echo "❌ gcloud CLI not found. Please install Google Cloud SDK."
exit 1
fi
if ! command -v pytest &> /dev/null; then
echo "❌ pytest not found. Installing test dependencies..."
pip install -r tests/requirements.txt
fi
# Verify authentication
echo "Verifying authentication..."
if ! gcloud auth print-identity-token &> /dev/null; then
echo "❌ Not authenticated with gcloud. Run: gcloud auth login"
exit 1
fi
echo "✅ Prerequisites met"
echo ""
# Run tests
echo "Running integration tests..."
echo ""
# Run fast tests (no slow marker)
echo "📝 Running fast tests..."
pytest tests/test_api_integration.py -v -m "not slow" --tb=short
echo ""
echo "✅ Fast tests complete!"
echo ""
# Ask if user wants to run slow tests
read -p "Run slow/integration tests? These test actual job processing (5-10 min). [y/N] " -n 1 -r
echo
if [[ $REPLY =~ ^[Yy]$ ]]; then
echo "🐌 Running slow integration tests..."
pytest tests/test_api_integration.py -v -m "slow" --tb=short
echo ""
echo "✅ All tests complete!"
else
echo "⏭️ Skipping slow tests"
fi
echo ""
echo "=================================="
echo "Test run complete! 🎉"
View File
Whitespace-only changes.
-243
View File
@@ -1,243 +0,0 @@
"""
Audio analysis service for analyzing backing vocals.
This service wraps the shared karaoke_gen.instrumental_review module
to provide GCS-integrated audio analysis capabilities.
"""
import logging
import os
import tempfile
from typing import Optional
from backend.services.storage_service import StorageService
from karaoke_gen.instrumental_review import (
AudioAnalyzer,
AnalysisResult,
WaveformGenerator,
)
logger = logging.getLogger(__name__)
class AudioAnalysisService:
"""
Service for analyzing backing vocals audio files stored in GCS.
This service acts as a thin wrapper around the shared AudioAnalyzer
and WaveformGenerator classes, handling GCS download/upload operations.
The actual analysis logic is in the shared karaoke_gen.instrumental_review
module, ensuring feature parity between local and remote workflows.
"""
def __init__(
self,
storage_service: Optional[StorageService] = None,
silence_threshold_db: float = -40.0,
min_segment_duration_ms: int = 100,
):
"""
Initialize the audio analysis service.
Args:
storage_service: GCS storage service. If not provided, a new
instance will be created.
silence_threshold_db: Threshold for considering audio as silent.
Default is -40.0 dB.
min_segment_duration_ms: Minimum duration for audible segments.
Default is 100ms.
"""
self.storage_service = storage_service or StorageService()
self.analyzer = AudioAnalyzer(
silence_threshold_db=silence_threshold_db,
min_segment_duration_ms=min_segment_duration_ms,
)
self.waveform_generator = WaveformGenerator()
def analyze_backing_vocals(
self,
gcs_audio_path: str,
job_id: str,
) -> AnalysisResult:
"""
Analyze a backing vocals audio file from GCS.
This method:
1. Downloads the audio file from GCS to a temp file
2. Runs the analysis using the shared AudioAnalyzer
3. Returns the analysis result
Args:
gcs_audio_path: Path to the audio file in GCS
job_id: Job ID for logging
Returns:
AnalysisResult containing analysis data
"""
logger.info(f"[{job_id}] Analyzing backing vocals: {gcs_audio_path}")
with tempfile.TemporaryDirectory() as temp_dir:
# Download audio file
local_audio_path = os.path.join(temp_dir, "backing_vocals.flac")
self.storage_service.download_file(gcs_audio_path, local_audio_path)
# Run analysis
result = self.analyzer.analyze(local_audio_path)
logger.info(
f"[{job_id}] Analysis complete: "
f"has_audible={result.has_audible_content}, "
f"segments={result.segment_count}, "
f"recommendation={result.recommended_selection.value}"
)
return result
def analyze_and_generate_waveform(
self,
gcs_audio_path: str,
job_id: str,
gcs_waveform_destination: str,
) -> tuple[AnalysisResult, str]:
"""
Analyze backing vocals and generate a waveform image.
This method:
1. Downloads the audio file from GCS
2. Runs analysis using AudioAnalyzer
3. Generates waveform image using WaveformGenerator
4. Uploads the waveform image to GCS
5. Returns analysis result and waveform GCS path
Args:
gcs_audio_path: Path to the audio file in GCS
job_id: Job ID for logging
gcs_waveform_destination: Where to upload the waveform image in GCS
Returns:
Tuple of (AnalysisResult, waveform_gcs_path)
"""
logger.info(f"[{job_id}] Analyzing and generating waveform: {gcs_audio_path}")
with tempfile.TemporaryDirectory() as temp_dir:
# Download audio file
local_audio_path = os.path.join(temp_dir, "backing_vocals.flac")
self.storage_service.download_file(gcs_audio_path, local_audio_path)
# Run analysis
result = self.analyzer.analyze(local_audio_path)
# Generate waveform
local_waveform_path = os.path.join(temp_dir, "waveform.png")
self.waveform_generator.generate(
audio_path=local_audio_path,
output_path=local_waveform_path,
segments=result.audible_segments,
show_time_axis=True,
silence_threshold_db=self.analyzer.silence_threshold_db,
)
# Upload waveform to GCS
self.storage_service.upload_file(
local_waveform_path,
gcs_waveform_destination
)
logger.info(
f"[{job_id}] Analysis and waveform generation complete. "
f"Waveform uploaded to: {gcs_waveform_destination}"
)
return result, gcs_waveform_destination
def get_waveform_data(
self,
gcs_audio_path: str,
job_id: str,
num_points: int = 500,
) -> tuple[list[float], float]:
"""
Get waveform data (amplitude envelope) for client-side rendering.
This is useful when the frontend wants to render the waveform
itself using Canvas or SVG, rather than displaying a pre-generated
image.
Args:
gcs_audio_path: Path to the audio file in GCS
job_id: Job ID for logging
num_points: Number of data points to return
Returns:
Tuple of (amplitude_values, duration_seconds)
"""
logger.info(f"[{job_id}] Getting waveform data: {gcs_audio_path}")
with tempfile.TemporaryDirectory() as temp_dir:
# Download audio file
local_audio_path = os.path.join(temp_dir, "backing_vocals.flac")
self.storage_service.download_file(gcs_audio_path, local_audio_path)
# Generate waveform data
amplitudes, duration = self.waveform_generator.generate_data_only(
audio_path=local_audio_path,
num_points=num_points,
)
return amplitudes, duration
def generate_waveform_with_mutes(
self,
gcs_audio_path: str,
job_id: str,
gcs_waveform_destination: str,
mute_regions: list,
) -> str:
"""
Generate a waveform image with mute regions highlighted.
This is useful for showing the user which regions will be muted
in the custom instrumental.
Args:
gcs_audio_path: Path to the audio file in GCS
job_id: Job ID for logging
gcs_waveform_destination: Where to upload the waveform image
mute_regions: List of MuteRegion objects to highlight
Returns:
GCS path to the uploaded waveform image
"""
from karaoke_gen.instrumental_review import MuteRegion
logger.info(
f"[{job_id}] Generating waveform with {len(mute_regions)} mute regions"
)
with tempfile.TemporaryDirectory() as temp_dir:
# Download audio file
local_audio_path = os.path.join(temp_dir, "backing_vocals.flac")
self.storage_service.download_file(gcs_audio_path, local_audio_path)
# First run analysis to get segments
result = self.analyzer.analyze(local_audio_path)
# Generate waveform with mute regions
local_waveform_path = os.path.join(temp_dir, "waveform_with_mutes.png")
self.waveform_generator.generate(
audio_path=local_audio_path,
output_path=local_waveform_path,
segments=result.audible_segments,
mute_regions=mute_regions,
show_time_axis=True,
)
# Upload to GCS
self.storage_service.upload_file(
local_waveform_path,
gcs_waveform_destination
)
return gcs_waveform_destination
-278
View File
@@ -1,278 +0,0 @@
"""
Audio editing service for creating custom instrumentals.
This service wraps the shared karaoke_gen.instrumental_review module
to provide GCS-integrated audio editing capabilities.
"""
import logging
import os
import tempfile
from typing import List, Optional
from backend.services.storage_service import StorageService
from karaoke_gen.instrumental_review import (
AudioEditor,
MuteRegion,
)
from karaoke_gen.instrumental_review.models import CustomInstrumentalResult
logger = logging.getLogger(__name__)
class AudioEditingService:
"""
Service for creating custom instrumental tracks with GCS integration.
This service acts as a thin wrapper around the shared AudioEditor
class, handling GCS download/upload operations.
The actual editing logic is in the shared karaoke_gen.instrumental_review
module, ensuring feature parity between local and remote workflows.
"""
def __init__(
self,
storage_service: Optional[StorageService] = None,
output_format: str = "flac",
):
"""
Initialize the audio editing service.
Args:
storage_service: GCS storage service. If not provided, a new
instance will be created.
output_format: Output audio format. Default is "flac".
"""
self.storage_service = storage_service or StorageService()
self.editor = AudioEditor(output_format=output_format)
def create_custom_instrumental(
self,
gcs_clean_instrumental_path: str,
gcs_backing_vocals_path: str,
mute_regions: List[MuteRegion],
gcs_output_path: str,
job_id: str,
) -> CustomInstrumentalResult:
"""
Create a custom instrumental by muting regions of backing vocals.
This method:
1. Downloads clean instrumental and backing vocals from GCS
2. Applies mute regions to backing vocals
3. Combines with clean instrumental
4. Uploads result to GCS
Args:
gcs_clean_instrumental_path: Path to clean instrumental in GCS
gcs_backing_vocals_path: Path to backing vocals in GCS
mute_regions: List of regions to mute in backing vocals
gcs_output_path: Where to upload the custom instrumental in GCS
job_id: Job ID for logging
Returns:
CustomInstrumentalResult with details about the created file
"""
logger.info(
f"[{job_id}] Creating custom instrumental with "
f"{len(mute_regions)} mute regions"
)
with tempfile.TemporaryDirectory() as temp_dir:
# Download input files
local_clean_path = os.path.join(temp_dir, "clean_instrumental.flac")
local_backing_path = os.path.join(temp_dir, "backing_vocals.flac")
local_output_path = os.path.join(temp_dir, "custom_instrumental.flac")
logger.debug(f"[{job_id}] Downloading clean instrumental")
self.storage_service.download_file(
gcs_clean_instrumental_path,
local_clean_path
)
logger.debug(f"[{job_id}] Downloading backing vocals")
self.storage_service.download_file(
gcs_backing_vocals_path,
local_backing_path
)
# Create custom instrumental using shared editor
result = self.editor.create_custom_instrumental(
clean_instrumental_path=local_clean_path,
backing_vocals_path=local_backing_path,
mute_regions=mute_regions,
output_path=local_output_path,
)
# Upload result to GCS
logger.debug(f"[{job_id}] Uploading custom instrumental to GCS")
self.storage_service.upload_file(local_output_path, gcs_output_path)
# Update result with GCS path
result.output_path = gcs_output_path
logger.info(
f"[{job_id}] Custom instrumental created: {gcs_output_path}, "
f"muted {result.total_muted_duration_seconds:.1f}s across "
f"{len(result.mute_regions_applied)} regions"
)
return result
def create_preview(
self,
gcs_clean_instrumental_path: str,
gcs_backing_vocals_path: str,
mute_regions: List[MuteRegion],
gcs_preview_path: str,
job_id: str,
preview_duration_seconds: Optional[float] = None,
) -> str:
"""
Create a preview of the custom instrumental.
This is similar to create_custom_instrumental but optimized for
quick preview generation (optionally truncated duration).
Args:
gcs_clean_instrumental_path: Path to clean instrumental in GCS
gcs_backing_vocals_path: Path to backing vocals in GCS
mute_regions: List of regions to mute
gcs_preview_path: Where to upload the preview in GCS
job_id: Job ID for logging
preview_duration_seconds: Optional max duration for preview
Returns:
GCS path to the uploaded preview
"""
logger.info(f"[{job_id}] Creating preview with {len(mute_regions)} mute regions")
with tempfile.TemporaryDirectory() as temp_dir:
# Download input files
local_clean_path = os.path.join(temp_dir, "clean_instrumental.flac")
local_backing_path = os.path.join(temp_dir, "backing_vocals.flac")
local_preview_path = os.path.join(temp_dir, "preview.flac")
self.storage_service.download_file(
gcs_clean_instrumental_path,
local_clean_path
)
self.storage_service.download_file(
gcs_backing_vocals_path,
local_backing_path
)
# Generate preview
from pydub import AudioSegment
preview = self.editor.preview_with_mutes(
clean_instrumental_path=local_clean_path,
backing_vocals_path=local_backing_path,
mute_regions=mute_regions,
)
# Optionally truncate
if preview_duration_seconds:
max_ms = int(preview_duration_seconds * 1000)
preview = preview[:max_ms]
# Export and upload
preview.export(local_preview_path, format="flac")
self.storage_service.upload_file(local_preview_path, gcs_preview_path)
logger.info(f"[{job_id}] Preview created: {gcs_preview_path}")
return gcs_preview_path
def mute_backing_vocals_only(
self,
gcs_backing_vocals_path: str,
mute_regions: List[MuteRegion],
gcs_output_path: str,
job_id: str,
) -> str:
"""
Apply mute regions to backing vocals without combining with instrumental.
This is useful for creating an edited backing vocals track that
can be combined with the instrumental later.
Args:
gcs_backing_vocals_path: Path to backing vocals in GCS
mute_regions: List of regions to mute
gcs_output_path: Where to upload the edited backing vocals
job_id: Job ID for logging
Returns:
GCS path to the uploaded edited backing vocals
"""
logger.info(
f"[{job_id}] Muting backing vocals with {len(mute_regions)} regions"
)
with tempfile.TemporaryDirectory() as temp_dir:
local_backing_path = os.path.join(temp_dir, "backing_vocals.flac")
local_output_path = os.path.join(temp_dir, "muted_backing.flac")
self.storage_service.download_file(
gcs_backing_vocals_path,
local_backing_path
)
self.editor.apply_mute_to_single_track(
audio_path=local_backing_path,
mute_regions=mute_regions,
output_path=local_output_path,
)
self.storage_service.upload_file(local_output_path, gcs_output_path)
logger.info(f"[{job_id}] Muted backing vocals: {gcs_output_path}")
return gcs_output_path
def validate_mute_regions(
self,
mute_regions: List[MuteRegion],
total_duration_seconds: float,
) -> List[str]:
"""
Validate mute regions for consistency.
Args:
mute_regions: List of mute regions to validate
total_duration_seconds: Total duration of the audio
Returns:
List of validation error messages (empty if valid)
"""
errors = []
for i, region in enumerate(mute_regions):
if region.start_seconds < 0:
errors.append(
f"Region {i}: start_seconds ({region.start_seconds}) cannot be negative"
)
if region.end_seconds <= region.start_seconds:
errors.append(
f"Region {i}: end_seconds ({region.end_seconds}) must be after "
f"start_seconds ({region.start_seconds})"
)
if region.start_seconds > total_duration_seconds:
errors.append(
f"Region {i}: start_seconds ({region.start_seconds}) exceeds "
f"audio duration ({total_duration_seconds})"
)
if region.end_seconds > total_duration_seconds:
# Not an error, but log a warning - the region will be clamped
logger.warning(
f"Region {i}: end_seconds ({region.end_seconds}) exceeds "
f"audio duration ({total_duration_seconds}), will be clamped"
)
return errors
-681
View File
@@ -1,681 +0,0 @@
"""
Audio search service for finding and downloading audio files.
This service integrates with karaoke_gen.audio_fetcher (which wraps flacfetch)
to search for audio from various sources (YouTube, music trackers, etc.)
and download the selected audio file.
This is a thin wrapper that adds backend-specific functionality:
- Caching raw results for API-based selection flow
- Firestore-compatible serialization
- Singleton pattern for service lifecycle
- Remote flacfetch service integration for torrent downloads
When FLACFETCH_API_URL is configured, this service uses the remote flacfetch
HTTP API for search and download operations. This is required for torrent
downloads since Cloud Run doesn't support BitTorrent peer connections.
"""
import asyncio
import logging
import os
from typing import List, Optional
import nest_asyncio
# Import shared classes from karaoke_gen.audio_fetcher - single source of truth
# Note: Import directly from audio_fetcher module, not karaoke_gen.__init__
# to avoid pulling in heavier dependencies like lyrics_transcriber
from karaoke_gen.audio_fetcher import (
FlacFetcher,
AudioSearchResult,
AudioFetchResult,
AudioFetcherError,
NoResultsError,
DownloadError,
)
from .flacfetch_client import get_flacfetch_client, FlacfetchServiceError
logger = logging.getLogger(__name__)
# Re-export exception under backend's naming convention for compatibility
AudioSearchError = AudioFetcherError
# Also alias AudioFetchResult as AudioDownloadResult for backwards compatibility
AudioDownloadResult = AudioFetchResult
class AudioSearchService:
"""
Service for searching and downloading audio files.
This is a thin wrapper around karaoke_gen.audio_fetcher.FlacFetcher that adds
backend-specific functionality:
- Caches raw results for API-based selection (client picks by index)
- Provides Firestore-compatible serialization
- Manages service lifecycle as singleton
- Remote flacfetch service integration for torrent downloads
When FLACFETCH_API_URL is configured, this service uses the remote flacfetch
HTTP API for search and download operations. This enables torrent downloads
which are not possible in Cloud Run due to network restrictions.
The actual flacfetch integration is in karaoke_gen.audio_fetcher.FlacFetcher,
which is shared between local CLI and cloud backend.
"""
# Sentinel value to indicate "use environment variable"
_USE_ENV = object()
def __init__(
self,
red_api_key: Optional[str] = _USE_ENV,
red_api_url: Optional[str] = _USE_ENV,
ops_api_key: Optional[str] = _USE_ENV,
ops_api_url: Optional[str] = _USE_ENV,
):
"""
Initialize the audio search service.
Args:
red_api_key: API key for RED tracker (optional, uses env if not provided)
red_api_url: Base URL for RED tracker API (optional, uses env if not provided)
ops_api_key: API key for OPS tracker (optional, uses env if not provided)
ops_api_url: Base URL for OPS tracker API (optional, uses env if not provided)
"""
# Use environment variables if not explicitly provided
if red_api_key is self._USE_ENV:
red_api_key = os.environ.get("RED_API_KEY")
if red_api_url is self._USE_ENV:
red_api_url = os.environ.get("RED_API_URL")
if ops_api_key is self._USE_ENV:
ops_api_key = os.environ.get("OPS_API_KEY")
if ops_api_url is self._USE_ENV:
ops_api_url = os.environ.get("OPS_API_URL")
# Check for remote flacfetch client
self._remote_client = get_flacfetch_client()
# Log which mode we're using
if self._remote_client:
logger.info("AudioSearchService using REMOTE flacfetch service (torrent downloads enabled)")
else:
logger.info("AudioSearchService using LOCAL flacfetch (YouTube only in Cloud Run)")
# Delegate to shared FlacFetcher implementation (for local mode or fallback)
self._fetcher = FlacFetcher(
red_api_key=red_api_key,
red_api_url=red_api_url,
ops_api_key=ops_api_key,
ops_api_url=ops_api_url,
)
# Cache search results for API-based selection flow
# Key: index, Value: AudioSearchResult (with raw_result)
self._cached_results: List[AudioSearchResult] = []
# Cache for remote search results (search_id -> results)
self._remote_search_id: Optional[str] = None
def search(self, artist: str, title: str) -> List[AudioSearchResult]:
"""
Search for audio matching the given artist and title.
Results are cached internally for later download via download().
If a remote flacfetch service is configured (FLACFETCH_API_URL), uses that
for better torrent download support. Otherwise falls back to local flacfetch.
Args:
artist: The artist name to search for
title: The track title to search for
Returns:
List of AudioSearchResult objects
Raises:
NoResultsError: If no results are found
AudioSearchError: For other errors (e.g., flacfetch not installed)
"""
try:
# Try remote flacfetch service first if configured
if self._remote_client:
return self._search_remote(artist, title)
# Fallback to local flacfetch
results = self._fetcher.search(artist, title)
# Cache results for later download
self._cached_results = results
self._remote_search_id = None
logger.info(f"Found {len(results)} results for: {artist} - {title}")
return results
except (NoResultsError, AudioFetcherError):
# Re-raise these as-is
raise
except Exception as e:
logger.error(f"Search failed: {e}")
raise AudioSearchError(f"Search failed: {e}") from e
def _search_remote(self, artist: str, title: str) -> List[AudioSearchResult]:
"""
Search using the remote flacfetch service.
Runs async code in a sync context for compatibility with existing API.
"""
try:
# Enable nested event loops (needed when called from FastAPI async context)
# Apply to the current loop if one exists, otherwise it's a no-op
try:
loop = asyncio.get_running_loop()
nest_asyncio.apply(loop)
except RuntimeError:
pass # No running loop, nest_asyncio not needed
# Run async search in sync context
loop = asyncio.new_event_loop()
asyncio.set_event_loop(loop)
try:
response = loop.run_until_complete(
self._remote_client.search(artist, title)
)
finally:
loop.close()
# Check for empty results
if not response.get("results"):
raise NoResultsError(f"No results found for: {artist} - {title}")
# Store search_id for download
self._remote_search_id = response.get("search_id")
# Convert remote results to AudioSearchResult objects
results = []
for item in response.get("results", []):
result = AudioSearchResult(
title=item.get("title", "Unknown"),
artist=item.get("artist", "Unknown"),
url=item.get("download_url", ""), # May be empty for remote
provider=item.get("provider", "Unknown"),
duration=item.get("duration_seconds"),
quality=item.get("quality", ""),
source_id=item.get("info_hash"),
index=item.get("index", 0),
seeders=item.get("seeders"),
target_file=item.get("target_file"),
# Store full remote data in raw_result for rich display
raw_result=item,
)
results.append(result)
# Cache results
self._cached_results = results
logger.info(f"Found {len(results)} results from remote flacfetch for: {artist} - {title}")
return results
except FlacfetchServiceError as e:
logger.warning(f"Remote flacfetch search failed, falling back to local: {e}")
# Fallback to local search
self._remote_search_id = None
results = self._fetcher.search(artist, title)
self._cached_results = results
return results
def select_best(self, results: List[AudioSearchResult]) -> int:
"""
Select the best result from a list of search results.
Uses flacfetch's built-in quality ranking to select the best source.
Args:
results: List of search results
Returns:
Index of the best result
"""
return self._fetcher.select_best(results)
def download(
self,
result_index: int,
output_dir: str,
output_filename: Optional[str] = None,
gcs_path: Optional[str] = None,
remote_search_id: Optional[str] = None,
) -> AudioDownloadResult:
"""
Download audio from a cached search result.
This method uses the cached results from the last search() call.
The API flow is:
1. Client calls search() -> gets list of results
2. Client picks an index
3. Client calls download(index) -> gets downloaded file
If a remote flacfetch service is configured and the search was performed
remotely (for torrent sources), uses the remote service for download.
Args:
result_index: Index of the result to download (from search results)
output_dir: Directory to save the downloaded file
output_filename: Optional filename (without extension)
gcs_path: Optional GCS path for remote uploads (e.g., "uploads/job123/audio/")
remote_search_id: Optional search_id for remote downloads. If provided,
uses this instead of the cached _remote_search_id. This is important
for concurrent requests where the service-level cache may be stale.
Returns:
AudioDownloadResult with the downloaded file path
Raises:
DownloadError: If download fails or no cached result for index
"""
if result_index < 0 or result_index >= len(self._cached_results):
raise DownloadError(
f"No cached result for index {result_index}. "
f"Available indices: 0-{len(self._cached_results) - 1}. "
"Run search() first."
)
result = self._cached_results[result_index]
logger.info(f"Downloading: {result.artist} - {result.title} from {result.provider}")
# Use provided search_id or fall back to service-level cache
effective_search_id = remote_search_id or self._remote_search_id
# Check if this was a remote search (torrent sources need remote download)
if effective_search_id and self._remote_client:
# Check if this is a torrent source that needs remote handling
is_torrent = result.provider in ["RED", "OPS"]
if is_torrent:
return self._download_remote(result_index, output_dir, output_filename, gcs_path, effective_search_id)
# Delegate to shared FlacFetcher (local download)
fetch_result = self._fetcher.download(result, output_dir, output_filename)
logger.info(f"Downloaded to: {fetch_result.filepath}")
return fetch_result
def download_by_id(
self,
source_name: str,
source_id: str,
output_dir: str,
output_filename: Optional[str] = None,
target_file: Optional[str] = None,
download_url: Optional[str] = None,
gcs_path: Optional[str] = None,
) -> AudioDownloadResult:
"""
Download audio directly by source ID (no prior search required).
This is the preferred method when you have stored the source_id from a
previous search and want to download later without re-searching. This
avoids unnecessary API calls to private trackers.
Args:
source_name: Provider name (RED, OPS, YouTube, Spotify)
source_id: Source-specific ID (torrent ID, video ID, track ID)
output_dir: Directory to save the downloaded file
output_filename: Optional filename (without extension)
target_file: For torrents, specific file to extract from the torrent
download_url: For YouTube/Spotify, direct URL (optional)
gcs_path: Optional GCS path for remote uploads
Returns:
AudioDownloadResult with the downloaded file path
Raises:
DownloadError: If download fails
"""
logger.info(f"Download by ID: {source_name} ID={source_id}")
# For torrent sources, must use remote client
if source_name in ["RED", "OPS"]:
if not self._remote_client:
raise DownloadError(
f"Cannot download from {source_name} without remote flacfetch service"
)
return self._download_by_id_remote(
source_name=source_name,
source_id=source_id,
output_dir=output_dir,
output_filename=output_filename,
target_file=target_file,
download_url=download_url,
gcs_path=gcs_path,
)
# For YouTube/Spotify, can use local flacfetch via FetchManager.download_by_id
# But we currently only have remote support, so use remote if available
if self._remote_client:
return self._download_by_id_remote(
source_name=source_name,
source_id=source_id,
output_dir=output_dir,
output_filename=output_filename,
target_file=target_file,
download_url=download_url,
gcs_path=gcs_path,
)
# Local download for YouTube/Spotify (fallback)
# Use the local fetcher's download_by_id if available
raise DownloadError(
f"Local download_by_id not yet implemented for {source_name}. "
"Configure FLACFETCH_API_URL for remote downloads."
)
def _download_by_id_remote(
self,
source_name: str,
source_id: str,
output_dir: str, # Not used for remote downloads (files go to GCS or remote server)
output_filename: Optional[str] = None,
target_file: Optional[str] = None,
download_url: Optional[str] = None,
gcs_path: Optional[str] = None,
) -> AudioDownloadResult:
"""
Download by ID using the remote flacfetch service.
Note: output_dir is accepted for API compatibility but ignored for remote
downloads. Remote downloads either go to GCS (if gcs_path is set) or to
the remote server's download directory.
"""
# output_dir is intentionally unused - remote downloads don't use local paths
_ = output_dir
try:
# Enable nested event loops
try:
running_loop = asyncio.get_running_loop()
nest_asyncio.apply(running_loop)
except RuntimeError:
pass
loop = asyncio.new_event_loop()
asyncio.set_event_loop(loop)
try:
# Start download
download_id = loop.run_until_complete(
self._remote_client.download_by_id(
source_name=source_name,
source_id=source_id,
output_filename=output_filename,
target_file=target_file,
download_url=download_url,
gcs_path=gcs_path,
)
)
logger.info(f"Remote download by ID started: {download_id}")
# Wait for completion
def log_progress(status):
progress = status.get("progress", 0)
speed = status.get("download_speed_kbps", 0)
logger.debug(f"Download progress: {progress:.1f}% ({speed:.1f} KB/s)")
final_status = loop.run_until_complete(
self._remote_client.wait_for_download(
download_id,
timeout=600,
progress_callback=log_progress,
)
)
finally:
loop.close()
# Determine file path
filepath = final_status.get("gcs_path") or final_status.get("output_path")
if not filepath:
raise DownloadError("Remote download completed but no file path returned")
logger.info(f"Remote download by ID complete: {filepath}")
return AudioDownloadResult(
filepath=filepath,
artist="", # Not available without search
title="", # Not available without search
provider=source_name,
quality="", # Not available without search
)
except FlacfetchServiceError as e:
raise DownloadError(f"Remote download by ID failed: {e}") from e
except Exception as e:
logger.error(f"Remote download by ID error: {e}")
raise DownloadError(f"Remote download by ID failed: {e}") from e
def _download_remote(
self,
result_index: int,
output_dir: str,
output_filename: Optional[str] = None,
gcs_path: Optional[str] = None,
search_id: Optional[str] = None,
) -> AudioDownloadResult:
"""
Download using the remote flacfetch service.
The remote service downloads via torrent and optionally uploads to GCS.
Args:
result_index: Index of the result to download
output_dir: Directory to save downloaded file
output_filename: Optional filename
gcs_path: Optional GCS path for remote uploads
search_id: Remote search ID to use (overrides self._remote_search_id)
"""
effective_search_id = search_id or self._remote_search_id
if not effective_search_id:
raise DownloadError("No remote search ID - run search() first")
result = self._cached_results[result_index]
try:
# Enable nested event loops (needed when called from FastAPI async context)
try:
running_loop = asyncio.get_running_loop()
nest_asyncio.apply(running_loop)
except RuntimeError:
pass # No running loop, nest_asyncio not needed
loop = asyncio.new_event_loop()
asyncio.set_event_loop(loop)
try:
# Start download
download_id = loop.run_until_complete(
self._remote_client.download(
search_id=effective_search_id,
result_index=result_index,
output_filename=output_filename,
gcs_path=gcs_path,
)
)
logger.info(f"Remote download started: {download_id}")
# Wait for completion with progress logging
def log_progress(status):
progress = status.get("progress", 0)
speed = status.get("download_speed_kbps", 0)
logger.debug(f"Download progress: {progress:.1f}% ({speed:.1f} KB/s)")
final_status = loop.run_until_complete(
self._remote_client.wait_for_download(
download_id,
timeout=600, # 10 minute timeout
progress_callback=log_progress,
)
)
finally:
loop.close()
# Determine file path
filepath = final_status.get("gcs_path") or final_status.get("output_path")
if not filepath:
raise DownloadError("Remote download completed but no file path returned")
logger.info(f"Remote download complete: {filepath}")
return AudioDownloadResult(
filepath=filepath,
artist=result.artist,
title=result.title,
provider=result.provider,
quality=result.quality,
)
except FlacfetchServiceError as e:
raise DownloadError(f"Remote download failed: {e}") from e
except Exception as e:
logger.error(f"Remote download error: {e}")
raise DownloadError(f"Remote download failed: {e}") from e
def search_and_download_auto(
self,
artist: str,
title: str,
output_dir: str,
output_filename: Optional[str] = None,
gcs_path: Optional[str] = None,
) -> AudioDownloadResult:
"""
Search for audio and automatically download the best result.
This is a convenience method that combines search(), select_best(),
and download() for automated/non-interactive usage.
Args:
artist: The artist name to search for
title: The track title to search for
output_dir: Directory to save the downloaded file
output_filename: Optional filename (without extension)
gcs_path: Optional GCS path for remote uploads
Returns:
AudioDownloadResult with the downloaded file path
Raises:
NoResultsError: If no results are found
DownloadError: If download fails
"""
# Search
results = self.search(artist, title)
# Select best
best_index = self.select_best(results)
best_result = results[best_index]
logger.info(
f"Auto-selected result {best_index}: "
f"{best_result.artist} - {best_result.title} ({best_result.provider})"
)
# Download
return self.download(best_index, output_dir, output_filename, gcs_path)
async def search_async(self, artist: str, title: str) -> List[AudioSearchResult]:
"""
Async version of search for use in async routes.
Note: Currently wraps sync search in executor. Future optimization
could make this fully async.
"""
loop = asyncio.get_event_loop()
return await loop.run_in_executor(None, self.search, artist, title)
async def download_async(
self,
result_index: int,
output_dir: str,
output_filename: Optional[str] = None,
gcs_path: Optional[str] = None,
remote_search_id: Optional[str] = None,
) -> AudioDownloadResult:
"""
Async version of download for use in async routes.
Note: Currently wraps sync download in executor. Future optimization
could make this fully async.
"""
loop = asyncio.get_event_loop()
return await loop.run_in_executor(
None,
lambda: self.download(result_index, output_dir, output_filename, gcs_path, remote_search_id)
)
async def download_by_id_async(
self,
source_name: str,
source_id: str,
output_dir: str,
output_filename: Optional[str] = None,
target_file: Optional[str] = None,
download_url: Optional[str] = None,
gcs_path: Optional[str] = None,
) -> AudioDownloadResult:
"""
Async version of download_by_id for use in async routes.
"""
loop = asyncio.get_event_loop()
return await loop.run_in_executor(
None,
lambda: self.download_by_id(
source_name, source_id, output_dir, output_filename,
target_file, download_url, gcs_path
)
)
def is_remote_enabled(self) -> bool:
"""Check if remote flacfetch service is configured."""
return self._remote_client is not None
@property
def last_remote_search_id(self) -> Optional[str]:
"""Get the search_id from the last remote search.
This should be stored in job state_data after search and passed
back to download() to ensure correct remote download handling.
"""
return self._remote_search_id
async def check_remote_health(self) -> Optional[dict]:
"""
Check health of remote flacfetch service.
Returns:
Health status dict if remote service is configured and healthy,
None if not configured or unhealthy.
"""
if not self._remote_client:
return None
try:
return await self._remote_client.health_check()
except Exception as e:
logger.warning(f"Remote flacfetch health check failed: {e}")
return None
# Singleton instance
_audio_search_service: Optional[AudioSearchService] = None
def get_audio_search_service() -> AudioSearchService:
"""Get the singleton AudioSearchService instance."""
global _audio_search_service
if _audio_search_service is None:
_audio_search_service = AudioSearchService()
return _audio_search_service
-295
View File
@@ -1,295 +0,0 @@
"""
Authentication service for karaoke backend.
Provides token-based authentication with support for:
- Admin tokens (hardcoded in environment)
- User tokens (stored in Firestore)
- Usage tracking and limits
- Token management API
This is designed to be extensible for future Stripe integration
and admin dashboard functionality.
"""
import logging
import hashlib
import time
from enum import Enum
from typing import Optional, Tuple, Dict, Any
from datetime import datetime
from backend.services.firestore_service import FirestoreService
from backend.config import get_settings
logger = logging.getLogger(__name__)
class UserType(str, Enum):
"""User access levels."""
ADMIN = "admin" # Unlimited access, can manage tokens
UNLIMITED = "unlimited" # Unlimited karaoke generation
LIMITED = "limited" # Limited number of uses
STRIPE = "stripe" # Paid access via Stripe (future)
class AuthService:
"""Service for authentication and token management."""
def __init__(self):
"""Initialize auth service."""
self.firestore = FirestoreService()
self.settings = get_settings()
self._init_admin_tokens()
def _init_admin_tokens(self):
"""Load admin tokens from environment variables."""
# Get admin tokens from secret manager or env vars
admin_tokens_str = self.settings.admin_tokens or ""
self.admin_tokens = [t.strip() for t in admin_tokens_str.split(",") if t.strip()]
if self.admin_tokens:
logger.info(f"Loaded {len(self.admin_tokens)} admin token(s)")
else:
logger.warning("No admin tokens configured! Set ADMIN_TOKENS environment variable.")
def validate_token(self, token: str) -> Tuple[bool, UserType, int, str]:
"""
Validate an access token.
Returns:
(is_valid, user_type, remaining_uses, message)
remaining_uses: -1 = unlimited, 0 = exhausted, >0 = remaining count
"""
if not token:
return False, UserType.LIMITED, 0, "No token provided"
# Check for admin tokens (highest priority)
if token in self.admin_tokens:
logger.info("Admin token validated")
return True, UserType.ADMIN, -1, "Admin access granted"
# Debug: Log token comparison info (only first/last few chars for security)
if self.admin_tokens:
expected_prefix = self.admin_tokens[0][:8] if len(self.admin_tokens[0]) >= 8 else self.admin_tokens[0]
provided_prefix = token[:8] if len(token) >= 8 else token
logger.debug(
f"Token mismatch: expected prefix '{expected_prefix}...', "
f"got '{provided_prefix}...', "
f"expected len={len(self.admin_tokens[0])}, got len={len(token)}"
)
# Check stored tokens in Firestore
token_data = self.firestore.get_token(token)
if not token_data:
return False, UserType.LIMITED, 0, "Invalid token"
# Check if token is active
if not token_data.get("active", True):
return False, UserType(token_data["type"]), 0, "Token has been revoked"
token_type = UserType(token_data["type"])
max_uses = token_data.get("max_uses", -1)
# UNLIMITED tokens: no usage limits
if token_type == UserType.UNLIMITED:
return True, token_type, -1, "Unlimited access granted"
# LIMITED tokens: check usage count
if token_type == UserType.LIMITED:
if max_uses <= 0: # -1 means unlimited
return True, token_type, -1, "Limited token with unlimited uses"
current_uses = token_data.get("usage_count", 0)
remaining = max_uses - current_uses
if remaining <= 0:
return False, token_type, 0, "Token usage limit exceeded"
return True, token_type, remaining, f"Limited token: {remaining} uses remaining"
# STRIPE tokens: check expiration and usage
if token_type == UserType.STRIPE:
expires_at = token_data.get("expires_at")
if expires_at:
# expires_at is stored as timestamp
if time.time() > expires_at:
return False, token_type, 0, "Token has expired"
if max_uses > 0:
current_uses = token_data.get("usage_count", 0)
remaining = max_uses - current_uses
if remaining <= 0:
return False, token_type, 0, "Token usage limit exceeded"
return True, token_type, remaining, f"Stripe token: {remaining} uses remaining"
return True, token_type, -1, "Stripe access granted"
return False, UserType.LIMITED, 0, "Unknown token type"
def increment_token_usage(self, token: str, job_id: str) -> bool:
"""
Increment usage count for a token and track the job.
Args:
token: The access token
job_id: The job ID being created
Returns:
True if usage was tracked, False otherwise
"""
# Validate token first
is_valid, user_type, remaining_uses, message = self.validate_token(token)
if not is_valid:
logger.warning(f"Cannot increment usage for invalid token: {message}")
return False
# Don't track usage for admin or unlimited tokens
if user_type in [UserType.ADMIN, UserType.UNLIMITED]:
logger.debug(f"Skipping usage tracking for {user_type} token")
return True
# For admin tokens (not in Firestore), no tracking needed
if token in self.admin_tokens:
return True
# Increment usage in Firestore
try:
self.firestore.increment_token_usage(token, job_id)
logger.info(f"Incremented usage for token (remaining: {remaining_uses - 1})")
return True
except Exception as e:
logger.error(f"Failed to increment token usage: {e}")
return False
def create_token(
self,
token_value: str,
token_type: UserType,
max_uses: int = -1,
expires_at: Optional[float] = None,
created_by: Optional[str] = None
) -> Dict[str, Any]:
"""
Create a new access token.
Args:
token_value: The actual token string (should be secure random string)
token_type: Type of access (admin, unlimited, limited, stripe)
max_uses: Maximum number of uses (-1 = unlimited)
expires_at: Expiration timestamp (None = no expiration)
created_by: Admin token that created this token
Returns:
Token data dictionary
"""
token_data = {
"token": token_value,
"type": token_type.value,
"max_uses": max_uses,
"usage_count": 0,
"active": True,
"created_at": time.time(),
"created_by": created_by,
"expires_at": expires_at,
"jobs": []
}
self.firestore.create_token(token_value, token_data)
logger.info(f"Created new {token_type} token with max_uses={max_uses}")
return token_data
def revoke_token(self, token: str, revoked_by: Optional[str] = None) -> bool:
"""
Revoke an access token.
Args:
token: The token to revoke
revoked_by: Admin token that revoked this token
Returns:
True if revoked, False if token not found
"""
try:
self.firestore.update_token(token, {
"active": False,
"revoked_at": time.time(),
"revoked_by": revoked_by
})
logger.info(f"Revoked token")
return True
except Exception as e:
logger.error(f"Failed to revoke token: {e}")
return False
def list_tokens(self, include_inactive: bool = False) -> list:
"""
List all tokens (admin only).
Args:
include_inactive: Whether to include revoked tokens
Returns:
List of token data dictionaries (with sensitive data masked)
"""
try:
tokens = self.firestore.list_tokens()
# Filter inactive if requested
if not include_inactive:
tokens = [t for t in tokens if t.get("active", True)]
# Mask token values for security (show only first 8 chars)
for token in tokens:
if "token" in token and len(token["token"]) > 8:
token["token"] = token["token"][:8] + "..."
return tokens
except Exception as e:
logger.error(f"Failed to list tokens: {e}")
return []
def get_token_info(self, token: str) -> Optional[Dict[str, Any]]:
"""
Get information about a token (for admin dashboard).
Returns:
Token data or None if not found
"""
# Check if it's an admin token (not stored in Firestore)
if token in self.admin_tokens:
return {
"token": token[:8] + "...",
"type": UserType.ADMIN.value,
"max_uses": -1,
"usage_count": 0,
"active": True,
"source": "environment"
}
# Get from Firestore
token_data = self.firestore.get_token(token)
if token_data and "token" in token_data:
# Mask the full token value
token_data["token"] = token_data["token"][:8] + "..."
return token_data
# Global instance
_auth_service = None
def get_auth_service() -> AuthService:
"""Get the global auth service instance."""
global _auth_service
if _auth_service is None:
_auth_service = AuthService()
return _auth_service
-792
View File
@@ -1,792 +0,0 @@
"""
OAuth Credential Manager for validating and refreshing credentials.
This service provides:
1. Credential validation for YouTube, Dropbox, and Google Drive
2. Device Authorization Flow for re-authentication
3. Proactive monitoring with alerts when credentials expire
"""
import json
import logging
import time
from typing import Optional, Dict, Any, Tuple
from dataclasses import dataclass
from enum import Enum
from datetime import datetime
from google.oauth2.credentials import Credentials as GoogleCredentials
from google.auth.transport.requests import Request as GoogleAuthRequest
from backend.config import get_settings
logger = logging.getLogger(__name__)
class CredentialStatus(str, Enum):
"""Status of OAuth credentials."""
VALID = "valid"
EXPIRED = "expired" # Token expired but refresh may work
INVALID = "invalid" # Refresh failed, re-auth needed
NOT_CONFIGURED = "not_configured" # No credentials stored
ERROR = "error" # Unknown error during validation
@dataclass
class CredentialCheckResult:
"""Result of a credential validation check."""
service: str
status: CredentialStatus
message: str
last_checked: datetime
expires_at: Optional[datetime] = None
@dataclass
class DeviceAuthInfo:
"""Information for device authorization flow."""
device_code: str
user_code: str
verification_url: str
expires_in: int
interval: int
started_at: datetime
class CredentialManager:
"""
Manages OAuth credentials for all external services.
Provides validation, refresh, and device authorization flow for:
- YouTube (Google OAuth)
- Google Drive (Google OAuth)
- Dropbox
"""
# Secret names in Secret Manager
YOUTUBE_SECRET = "youtube-oauth-credentials"
GDRIVE_SECRET = "gdrive-oauth-credentials"
DROPBOX_SECRET = "dropbox-oauth-credentials"
# OAuth client credentials (for device auth flow)
YOUTUBE_CLIENT_SECRET = "youtube-client-credentials"
GDRIVE_CLIENT_SECRET = "gdrive-client-credentials"
# Google OAuth endpoints
GOOGLE_TOKEN_URI = "https://oauth2.googleapis.com/token"
GOOGLE_DEVICE_AUTH_URI = "https://oauth2.googleapis.com/device/code"
# Dropbox OAuth endpoints
DROPBOX_TOKEN_URI = "https://api.dropboxapi.com/oauth2/token"
# Scopes for each service
YOUTUBE_SCOPES = ["https://www.googleapis.com/auth/youtube"]
GDRIVE_SCOPES = ["https://www.googleapis.com/auth/drive.file"]
def __init__(self):
self.settings = get_settings()
self._pending_device_auths: Dict[str, DeviceAuthInfo] = {}
def _get_client_credentials(self, secret_name: str) -> Optional[Dict[str, str]]:
"""
Load OAuth client credentials from Secret Manager.
The secret should contain:
{
"client_id": "...",
"client_secret": "..."
}
"""
try:
creds_json = self.settings.get_secret(secret_name)
if not creds_json:
return None
creds = json.loads(creds_json)
if creds.get("client_id") and creds.get("client_secret"):
return creds
logger.warning(f"{secret_name} missing client_id or client_secret")
return None
except Exception as e:
logger.error(f"Failed to load {secret_name}: {e}")
return None
def get_youtube_client_credentials(self) -> Optional[Dict[str, str]]:
"""Get YouTube OAuth client credentials from Secret Manager."""
return self._get_client_credentials(self.YOUTUBE_CLIENT_SECRET)
def get_gdrive_client_credentials(self) -> Optional[Dict[str, str]]:
"""Get Google Drive OAuth client credentials from Secret Manager."""
return self._get_client_credentials(self.GDRIVE_CLIENT_SECRET)
# =========================================================================
# Credential Validation
# =========================================================================
def check_all_credentials(self) -> Dict[str, CredentialCheckResult]:
"""
Check validity of all OAuth credentials.
Returns:
Dictionary mapping service name to check result
"""
results = {}
results["youtube"] = self.check_youtube_credentials()
results["gdrive"] = self.check_gdrive_credentials()
results["dropbox"] = self.check_dropbox_credentials()
return results
def check_youtube_credentials(self) -> CredentialCheckResult:
"""Check if YouTube credentials are valid and can be refreshed."""
return self._check_google_credentials(
secret_name=self.YOUTUBE_SECRET,
service_name="youtube",
scopes=self.YOUTUBE_SCOPES,
test_api_call=self._test_youtube_api
)
def check_gdrive_credentials(self) -> CredentialCheckResult:
"""Check if Google Drive credentials are valid and can be refreshed."""
return self._check_google_credentials(
secret_name=self.GDRIVE_SECRET,
service_name="gdrive",
scopes=self.GDRIVE_SCOPES,
test_api_call=self._test_gdrive_api
)
def check_dropbox_credentials(self) -> CredentialCheckResult:
"""Check if Dropbox credentials are valid."""
try:
creds_json = self.settings.get_secret(self.DROPBOX_SECRET)
if not creds_json:
return CredentialCheckResult(
service="dropbox",
status=CredentialStatus.NOT_CONFIGURED,
message="Dropbox credentials not configured",
last_checked=datetime.utcnow()
)
creds = json.loads(creds_json)
if not creds.get("access_token"):
return CredentialCheckResult(
service="dropbox",
status=CredentialStatus.INVALID,
message="Dropbox credentials missing access_token",
last_checked=datetime.utcnow()
)
# Test the credentials with a simple API call
if self._test_dropbox_api(creds):
return CredentialCheckResult(
service="dropbox",
status=CredentialStatus.VALID,
message="Dropbox credentials are valid",
last_checked=datetime.utcnow()
)
else:
# Try to refresh if we have refresh token
if creds.get("refresh_token") and creds.get("app_key") and creds.get("app_secret"):
if self._refresh_dropbox_token(creds):
return CredentialCheckResult(
service="dropbox",
status=CredentialStatus.VALID,
message="Dropbox credentials refreshed successfully",
last_checked=datetime.utcnow()
)
return CredentialCheckResult(
service="dropbox",
status=CredentialStatus.INVALID,
message="Dropbox credentials invalid and refresh failed",
last_checked=datetime.utcnow()
)
except Exception as e:
logger.error(f"Error checking Dropbox credentials: {e}")
return CredentialCheckResult(
service="dropbox",
status=CredentialStatus.ERROR,
message=f"Error checking credentials: {str(e)}",
last_checked=datetime.utcnow()
)
def _check_google_credentials(
self,
secret_name: str,
service_name: str,
scopes: list,
test_api_call: callable
) -> CredentialCheckResult:
"""Generic Google OAuth credential check."""
try:
creds_json = self.settings.get_secret(secret_name)
if not creds_json:
return CredentialCheckResult(
service=service_name,
status=CredentialStatus.NOT_CONFIGURED,
message=f"{service_name} credentials not configured",
last_checked=datetime.utcnow()
)
creds_data = json.loads(creds_json)
# Check for required fields
required = ["refresh_token", "client_id", "client_secret"]
missing = [f for f in required if not creds_data.get(f)]
if missing:
return CredentialCheckResult(
service=service_name,
status=CredentialStatus.INVALID,
message=f"Missing required fields: {missing}",
last_checked=datetime.utcnow()
)
# Create credentials object
credentials = GoogleCredentials(
token=creds_data.get("token"),
refresh_token=creds_data.get("refresh_token"),
token_uri=creds_data.get("token_uri", self.GOOGLE_TOKEN_URI),
client_id=creds_data.get("client_id"),
client_secret=creds_data.get("client_secret"),
scopes=creds_data.get("scopes", scopes)
)
# Try to refresh if expired
if credentials.expired or not credentials.token:
try:
credentials.refresh(GoogleAuthRequest())
# Update stored credentials with new token
self._update_google_credentials(secret_name, creds_data, credentials)
except Exception as e:
logger.error(f"Failed to refresh {service_name} credentials: {e}")
return CredentialCheckResult(
service=service_name,
status=CredentialStatus.INVALID,
message=f"Token refresh failed: {str(e)}",
last_checked=datetime.utcnow()
)
# Test with API call
if test_api_call(credentials):
return CredentialCheckResult(
service=service_name,
status=CredentialStatus.VALID,
message=f"{service_name} credentials are valid",
last_checked=datetime.utcnow(),
expires_at=credentials.expiry
)
else:
return CredentialCheckResult(
service=service_name,
status=CredentialStatus.INVALID,
message=f"{service_name} API test failed",
last_checked=datetime.utcnow()
)
except json.JSONDecodeError as e:
return CredentialCheckResult(
service=service_name,
status=CredentialStatus.INVALID,
message=f"Invalid JSON in credentials: {str(e)}",
last_checked=datetime.utcnow()
)
except Exception as e:
logger.error(f"Error checking {service_name} credentials: {e}")
return CredentialCheckResult(
service=service_name,
status=CredentialStatus.ERROR,
message=f"Error: {str(e)}",
last_checked=datetime.utcnow()
)
def _update_google_credentials(
self,
secret_name: str,
original_data: dict,
credentials: GoogleCredentials
) -> bool:
"""Update stored Google credentials with refreshed token."""
try:
from google.cloud import secretmanager
updated_data = original_data.copy()
updated_data["token"] = credentials.token
if credentials.expiry:
updated_data["expiry"] = credentials.expiry.isoformat()
# Add new secret version
client = secretmanager.SecretManagerServiceClient()
project_id = self.settings.google_cloud_project
secret_path = f"projects/{project_id}/secrets/{secret_name}"
client.add_secret_version(
parent=secret_path,
payload={"data": json.dumps(updated_data).encode("utf-8")}
)
logger.info(f"Updated {secret_name} with refreshed token")
return True
except Exception as e:
logger.error(f"Failed to update {secret_name}: {e}")
return False
# =========================================================================
# API Test Methods
# =========================================================================
def _test_youtube_api(self, credentials: GoogleCredentials) -> bool:
"""Test YouTube credentials with a simple API call."""
try:
from googleapiclient.discovery import build
youtube = build("youtube", "v3", credentials=credentials)
# Just get channel info - minimal API call
request = youtube.channels().list(part="id", mine=True)
response = request.execute()
return "items" in response
except Exception as e:
logger.error(f"YouTube API test failed: {e}")
return False
def _test_gdrive_api(self, credentials: GoogleCredentials) -> bool:
"""Test Google Drive credentials with a simple API call."""
try:
from googleapiclient.discovery import build
drive = build("drive", "v3", credentials=credentials)
# Just get about info - minimal API call
request = drive.about().get(fields="user")
response = request.execute()
return "user" in response
except Exception as e:
logger.error(f"Google Drive API test failed: {e}")
return False
def _test_dropbox_api(self, creds: dict) -> bool:
"""Test Dropbox credentials with a simple API call."""
try:
import dropbox
dbx = dropbox.Dropbox(creds["access_token"])
# Just get account info - minimal API call
dbx.users_get_current_account()
return True
except Exception as e:
logger.error(f"Dropbox API test failed: {e}")
return False
def _refresh_dropbox_token(self, creds: dict) -> bool:
"""Try to refresh Dropbox token."""
try:
import requests
response = requests.post(
self.DROPBOX_TOKEN_URI,
data={
"grant_type": "refresh_token",
"refresh_token": creds["refresh_token"],
"client_id": creds["app_key"],
"client_secret": creds["app_secret"],
},
timeout=30
)
if response.status_code == 200:
token_data = response.json()
creds["access_token"] = token_data["access_token"]
# Update in Secret Manager
self._update_dropbox_credentials(creds)
return True
return False
except Exception as e:
logger.error(f"Dropbox token refresh failed: {e}")
return False
def _update_dropbox_credentials(self, creds: dict) -> bool:
"""Update stored Dropbox credentials."""
try:
from google.cloud import secretmanager
client = secretmanager.SecretManagerServiceClient()
project_id = self.settings.google_cloud_project
secret_path = f"projects/{project_id}/secrets/{self.DROPBOX_SECRET}"
client.add_secret_version(
parent=secret_path,
payload={"data": json.dumps(creds).encode("utf-8")}
)
logger.info("Updated Dropbox credentials with refreshed token")
return True
except Exception as e:
logger.error(f"Failed to update Dropbox credentials: {e}")
return False
# =========================================================================
# Device Authorization Flow
# =========================================================================
def start_youtube_device_auth(
self,
client_id: Optional[str] = None,
client_secret: Optional[str] = None
) -> DeviceAuthInfo:
"""
Start YouTube device authorization flow.
Args:
client_id: Google OAuth client ID (optional, reads from Secret Manager if not provided)
client_secret: Google OAuth client secret (optional, reads from Secret Manager if not provided)
Returns:
DeviceAuthInfo with user code and verification URL
"""
# Load from Secret Manager if not provided
if not client_id or not client_secret:
stored_creds = self.get_youtube_client_credentials()
if not stored_creds:
raise Exception(
"YouTube client credentials not found. Either pass client_id/client_secret "
"or create the 'youtube-client-credentials' secret in Secret Manager."
)
client_id = stored_creds["client_id"]
client_secret = stored_creds["client_secret"]
return self._start_google_device_auth(
client_id=client_id,
client_secret=client_secret,
scopes=self.YOUTUBE_SCOPES,
service_name="youtube"
)
def start_gdrive_device_auth(
self,
client_id: Optional[str] = None,
client_secret: Optional[str] = None
) -> DeviceAuthInfo:
"""
Start Google Drive device authorization flow.
Args:
client_id: Google OAuth client ID (optional, reads from Secret Manager if not provided)
client_secret: Google OAuth client secret (optional, reads from Secret Manager if not provided)
Returns:
DeviceAuthInfo with user code and verification URL
"""
# Load from Secret Manager if not provided
if not client_id or not client_secret:
stored_creds = self.get_gdrive_client_credentials()
if not stored_creds:
raise Exception(
"Google Drive client credentials not found. Either pass client_id/client_secret "
"or create the 'gdrive-client-credentials' secret in Secret Manager."
)
client_id = stored_creds["client_id"]
client_secret = stored_creds["client_secret"]
return self._start_google_device_auth(
client_id=client_id,
client_secret=client_secret,
scopes=self.GDRIVE_SCOPES,
service_name="gdrive"
)
def _start_google_device_auth(
self,
client_id: str,
client_secret: str,
scopes: list,
service_name: str
) -> DeviceAuthInfo:
"""Start Google device authorization flow."""
import requests
logger.info(f"[{service_name}] Starting device auth flow with scopes: {scopes}")
response = requests.post(
self.GOOGLE_DEVICE_AUTH_URI,
data={
"client_id": client_id,
"scope": " ".join(scopes),
},
timeout=30
)
if response.status_code != 200:
logger.error(f"[{service_name}] Device auth request failed: {response.status_code} - {response.text}")
raise Exception(f"Device auth request failed: {response.text}")
logger.info(f"[{service_name}] Device auth initiated successfully")
data = response.json()
# Google uses 'verification_uri' but some docs show 'verification_url'
verification_url = data.get("verification_uri") or data.get("verification_url")
device_info = DeviceAuthInfo(
device_code=data["device_code"],
user_code=data["user_code"],
verification_url=verification_url,
expires_in=data["expires_in"],
interval=data.get("interval", 5),
started_at=datetime.utcnow()
)
# Store for polling, include client secret for token exchange
self._pending_device_auths[f"{service_name}:{data['device_code']}"] = {
"info": device_info,
"client_id": client_id,
"client_secret": client_secret,
"scopes": scopes,
"service_name": service_name
}
return device_info
def poll_device_auth(self, service_name: str, device_code: str) -> Tuple[str, Optional[dict]]:
"""
Poll for device authorization completion.
This method is STATELESS - it fetches client credentials from Secret Manager
and polls Google directly. This works correctly in serverless environments
like Cloud Run where in-memory state is not preserved between requests.
Args:
service_name: "youtube" or "gdrive"
device_code: The device code from start_*_device_auth
Returns:
Tuple of (status, token_data)
status: "pending", "complete", "expired", "error"
token_data: Token data if complete, None otherwise
"""
import requests
logger.info(f"[{service_name}] Polling device auth for code: {device_code[:20]}...")
# Get client credentials from Secret Manager (stateless approach)
if service_name == "youtube":
client_creds = self.get_youtube_client_credentials()
scopes = self.YOUTUBE_SCOPES
secret_name = self.YOUTUBE_SECRET
elif service_name == "gdrive":
client_creds = self.get_gdrive_client_credentials()
scopes = self.GDRIVE_SCOPES
secret_name = self.GDRIVE_SECRET
else:
logger.error(f"[{service_name}] Unknown service")
return ("error", {"message": f"Unknown service: {service_name}"})
if not client_creds:
logger.error(f"[{service_name}] Client credentials not found in Secret Manager")
return ("error", {"message": f"Client credentials not found in Secret Manager for {service_name}"})
client_id = client_creds.get("client_id")
client_secret = client_creds.get("client_secret")
if not client_id or not client_secret:
logger.error(f"[{service_name}] Invalid client credentials (missing client_id or client_secret)")
return ("error", {"message": f"Invalid client credentials for {service_name}"})
logger.info(f"[{service_name}] Got client credentials, polling Google token endpoint...")
# Poll Google token endpoint
response = requests.post(
self.GOOGLE_TOKEN_URI,
data={
"client_id": client_id,
"client_secret": client_secret,
"device_code": device_code,
"grant_type": "urn:ietf:params:oauth:grant-type:device_code",
},
timeout=30
)
data = response.json()
logger.info(f"[{service_name}] Token endpoint response: status={response.status_code}")
if response.status_code == 200:
logger.info(f"[{service_name}] Token exchange successful! Got access token and refresh_token={bool(data.get('refresh_token'))}")
# Success! Store the credentials
token_data = {
"token": data["access_token"],
"refresh_token": data.get("refresh_token"),
"token_uri": self.GOOGLE_TOKEN_URI,
"client_id": client_id,
"client_secret": client_secret,
"scopes": scopes,
}
# Save to Secret Manager
logger.info(f"[{service_name}] Saving credentials to secret: {secret_name}")
saved = self._save_credentials_to_secret(secret_name, token_data)
if not saved:
logger.error(f"[{service_name}] Failed to save credentials to Secret Manager!")
return ("error", {"message": "Token exchange succeeded but failed to save credentials to Secret Manager"})
logger.info(f"[{service_name}] Device auth flow COMPLETE - credentials saved successfully")
return ("complete", token_data)
elif "error" in data:
error = data["error"]
logger.info(f"[{service_name}] Token endpoint returned error: {error}")
if error == "authorization_pending":
return ("pending", {"message": "Waiting for user to authorize. Please visit the verification URL and enter the code."})
elif error == "slow_down":
return ("pending", {"message": "Please wait a few more seconds before polling again."})
elif error == "expired_token":
logger.warning(f"[{service_name}] Device code expired")
return ("expired", {"message": "Device code expired. Please start a new device auth flow."})
elif error == "access_denied":
logger.warning(f"[{service_name}] User denied access")
return ("error", {"message": "User denied access"})
else:
logger.error(f"[{service_name}] Unexpected error: {data}")
return ("error", {"message": data.get("error_description", error)})
logger.error(f"[{service_name}] Unknown response from token endpoint: {data}")
return ("error", {"message": "Unknown response from token endpoint"})
def _save_credentials_to_secret(self, secret_name: str, token_data: dict) -> bool:
"""Save credentials to Secret Manager, creating the secret if needed."""
logger.info(f"[SecretManager] Attempting to save credentials to: {secret_name}")
try:
from google.cloud import secretmanager
from google.api_core import exceptions as gcp_exceptions
client = secretmanager.SecretManagerServiceClient()
project_id = self.settings.google_cloud_project
if not project_id:
logger.error(f"[SecretManager] No GCP project configured!")
return False
parent = f"projects/{project_id}"
secret_path = f"{parent}/secrets/{secret_name}"
logger.info(f"[SecretManager] Using project: {project_id}, secret path: {secret_path}")
# Try to add a version to existing secret
try:
client.add_secret_version(
parent=secret_path,
payload={"data": json.dumps(token_data).encode("utf-8")}
)
logger.info(f"[SecretManager] SUCCESS - Added new version to existing secret: {secret_name}")
return True
except gcp_exceptions.NotFound:
# Secret doesn't exist, create it first
logger.info(f"[SecretManager] Secret {secret_name} not found, creating new secret...")
client.create_secret(
parent=parent,
secret_id=secret_name,
secret={"replication": {"automatic": {}}}
)
logger.info(f"[SecretManager] Created secret: {secret_name}")
# Now add the version
client.add_secret_version(
parent=secret_path,
payload={"data": json.dumps(token_data).encode("utf-8")}
)
logger.info(f"[SecretManager] SUCCESS - Created secret and saved credentials to: {secret_name}")
return True
except Exception as e:
logger.error(f"[SecretManager] FAILED to save credentials to {secret_name}: {type(e).__name__}: {e}")
import traceback
logger.error(f"[SecretManager] Traceback: {traceback.format_exc()}")
return False
# =========================================================================
# Alerts
# =========================================================================
def send_credential_alert(
self,
invalid_services: list[CredentialCheckResult],
discord_webhook_url: Optional[str] = None
) -> bool:
"""
Send alert about invalid credentials.
Args:
invalid_services: List of services with invalid credentials
discord_webhook_url: Discord webhook URL for notifications
Returns:
True if alert was sent successfully
"""
if not discord_webhook_url:
logger.warning("No Discord webhook URL configured for alerts")
return False
try:
import requests
services_list = "\n".join([
f"• **{r.service}**: {r.message}"
for r in invalid_services
])
# Get the API base URL for re-auth links
api_url = self.settings.api_base_url if hasattr(self.settings, 'api_base_url') else "https://your-api-url"
message = {
"embeds": [{
"title": "⚠️ OAuth Credentials Need Attention",
"description": f"The following service credentials need re-authorization:\n\n{services_list}",
"color": 16744256, # Orange
"fields": [
{
"name": "Re-authorize",
"value": f"Visit `{api_url}/api/auth/status` to start re-authorization flow",
"inline": False
}
],
"footer": {
"text": "karaoke-gen backend"
},
"timestamp": datetime.utcnow().isoformat()
}]
}
response = requests.post(discord_webhook_url, json=message, timeout=30)
return response.status_code in (200, 204)
except Exception as e:
logger.error(f"Failed to send credential alert: {e}")
return False
# Singleton instance
_credential_manager: Optional[CredentialManager] = None
def get_credential_manager() -> CredentialManager:
"""Get the singleton credential manager instance."""
global _credential_manager
if _credential_manager is None:
_credential_manager = CredentialManager()
return _credential_manager
-262
View File
@@ -1,262 +0,0 @@
"""Native Dropbox API service for cloud backend.
This service provides Dropbox operations using the native Python SDK,
replacing rclone for server-side operations. It handles:
- Folder listing for brand code calculation
- File and folder uploads
- Shared link generation
Credentials are loaded from Google Cloud Secret Manager.
"""
import json
import logging
import os
import re
from typing import Optional
from google.cloud import secretmanager
logger = logging.getLogger(__name__)
class DropboxService:
"""Dropbox operations using native Python SDK."""
def __init__(self):
self._client = None
self._is_configured = False
def _load_credentials(self) -> Optional[dict]:
"""Load OAuth credentials from Secret Manager."""
try:
client = secretmanager.SecretManagerServiceClient()
# Try to get the project ID from environment or use default
project_id = os.environ.get("GOOGLE_CLOUD_PROJECT", "karaoke-gen")
name = f"projects/{project_id}/secrets/dropbox-oauth-credentials/versions/latest"
response = client.access_secret_version(request={"name": name})
return json.loads(response.payload.data.decode("UTF-8"))
except Exception as e:
logger.warning(f"Failed to load Dropbox credentials from Secret Manager: {e}")
return None
@property
def is_configured(self) -> bool:
"""Check if Dropbox credentials are available."""
if self._is_configured:
return True
creds = self._load_credentials()
self._is_configured = creds is not None and "access_token" in creds
return self._is_configured
@property
def client(self):
"""Get or create Dropbox client."""
if self._client is None:
# Import here to avoid import errors if dropbox is not installed
try:
from dropbox import Dropbox
except ImportError:
raise ImportError(
"dropbox package is not installed. "
"Install it with: pip install dropbox"
)
creds = self._load_credentials()
if not creds:
raise RuntimeError("Dropbox credentials not configured in Secret Manager")
self._client = Dropbox(
oauth2_access_token=creds["access_token"],
oauth2_refresh_token=creds.get("refresh_token"),
app_key=creds.get("app_key"),
app_secret=creds.get("app_secret"),
)
return self._client
def list_folders(self, path: str) -> list[str]:
"""
List folder names at path for brand code calculation.
Args:
path: Dropbox path to list (e.g., "/Karaoke/Tracks-Organized")
Returns:
List of folder names in the path
"""
from dropbox.files import FolderMetadata
# Ensure path starts with /
if not path.startswith("/"):
path = f"/{path}"
logger.info(f"Listing folders at Dropbox path: {path}")
result = self.client.files_list_folder(path)
folders = []
# Get all entries (handling pagination)
while True:
for entry in result.entries:
if isinstance(entry, FolderMetadata):
folders.append(entry.name)
if not result.has_more:
break
result = self.client.files_list_folder_continue(result.cursor)
logger.info(f"Found {len(folders)} folders in {path}")
return folders
def get_next_brand_code(self, path: str, brand_prefix: str) -> str:
"""
Calculate next brand code from existing folders.
Scans the folder for existing brand codes like "NOMAD-0001" and
returns the next sequential code.
Args:
path: Dropbox path containing organized folders
brand_prefix: Brand prefix (e.g., "NOMAD")
Returns:
Next brand code (e.g., "NOMAD-1164")
"""
folders = self.list_folders(path)
pattern = re.compile(rf"^{re.escape(brand_prefix)}-(\d{{4}})")
max_num = 0
for folder in folders:
match = pattern.match(folder)
if match:
num = int(match.group(1))
max_num = max(max_num, num)
next_code = f"{brand_prefix}-{max_num + 1:04d}"
logger.info(f"Next brand code for {brand_prefix}: {next_code} (max existing: {max_num})")
return next_code
def upload_file(self, local_path: str, remote_path: str) -> None:
"""
Upload a single file to Dropbox.
Args:
local_path: Local file path
remote_path: Dropbox destination path (must start with /)
"""
from dropbox.files import WriteMode
# Ensure remote path starts with /
if not remote_path.startswith("/"):
remote_path = f"/{remote_path}"
file_size = os.path.getsize(local_path)
logger.info(f"Uploading {local_path} ({file_size / 1024 / 1024:.1f} MB) to {remote_path}")
# For large files (>150MB), use upload sessions
CHUNK_SIZE = 150 * 1024 * 1024 # 150 MB
with open(local_path, "rb") as f:
if file_size <= CHUNK_SIZE:
# Simple upload for smaller files
self.client.files_upload(f.read(), remote_path, mode=WriteMode.overwrite)
else:
# Chunked upload for large files
self._upload_large_file(f, remote_path, file_size)
logger.info(f"Successfully uploaded to {remote_path}")
def _upload_large_file(self, file_obj, remote_path: str, file_size: int) -> None:
"""Upload a large file using upload sessions."""
from dropbox.files import CommitInfo, UploadSessionCursor, WriteMode
CHUNK_SIZE = 150 * 1024 * 1024 # 150 MB
# Start upload session
session_start = self.client.files_upload_session_start(file_obj.read(CHUNK_SIZE))
cursor = UploadSessionCursor(
session_id=session_start.session_id,
offset=file_obj.tell(),
)
commit = CommitInfo(path=remote_path, mode=WriteMode.overwrite)
# Upload remaining chunks
while file_obj.tell() < file_size:
remaining = file_size - file_obj.tell()
if remaining <= CHUNK_SIZE:
# Final chunk
self.client.files_upload_session_finish(
file_obj.read(CHUNK_SIZE),
cursor,
commit,
)
else:
# Intermediate chunk
self.client.files_upload_session_append_v2(
file_obj.read(CHUNK_SIZE),
cursor,
)
cursor.offset = file_obj.tell()
def upload_folder(self, local_dir: str, remote_path: str) -> None:
"""
Upload all files in a directory to Dropbox folder.
Args:
local_dir: Local directory to upload
remote_path: Dropbox destination folder path
"""
# Ensure remote path starts with /
if not remote_path.startswith("/"):
remote_path = f"/{remote_path}"
logger.info(f"Uploading folder {local_dir} to {remote_path}")
uploaded_count = 0
for filename in os.listdir(local_dir):
local_file = os.path.join(local_dir, filename)
if os.path.isfile(local_file):
remote_file = f"{remote_path}/{filename}"
self.upload_file(local_file, remote_file)
uploaded_count += 1
logger.info(f"Uploaded {uploaded_count} files to {remote_path}")
def create_shared_link(self, path: str) -> str:
"""
Create and return a shared link for the path.
Args:
path: Dropbox path to share
Returns:
Shared link URL
"""
from dropbox.exceptions import ApiError
from dropbox.sharing import SharedLinkSettings
# Ensure path starts with /
if not path.startswith("/"):
path = f"/{path}"
logger.info(f"Creating shared link for: {path}")
try:
# Try to create a new shared link
settings = SharedLinkSettings(requested_visibility=None)
link = self.client.sharing_create_shared_link_with_settings(path, settings)
logger.info(f"Created new shared link: {link.url}")
return link.url
except ApiError as e:
# Link may already exist - try to get existing link
if e.error.is_shared_link_already_exists():
logger.info("Shared link already exists, retrieving existing link")
links = self.client.sharing_list_shared_links(path=path, direct_only=True)
if links.links:
logger.info(f"Found existing shared link: {links.links[0].url}")
return links.links[0].url
raise
def get_dropbox_service() -> DropboxService:
"""Factory function to get a DropboxService instance."""
return DropboxService()
-343
View File
@@ -1,343 +0,0 @@
"""
Firestore database operations for job management.
"""
import logging
from typing import Optional, Dict, Any, List
from datetime import datetime
from google.cloud import firestore
from google.cloud.firestore_v1 import FieldFilter
from backend.config import settings
from backend.models.job import Job, JobStatus, TimelineEvent
logger = logging.getLogger(__name__)
class FirestoreService:
"""Service for Firestore database operations."""
def __init__(self):
"""Initialize Firestore client."""
self.db = firestore.Client(project=settings.google_cloud_project)
self.collection = settings.firestore_collection
self.tokens_collection = "auth_tokens" # Collection for access tokens
def create_job(self, job: Job) -> None:
"""Create a new job in Firestore."""
try:
doc_ref = self.db.collection(self.collection).document(job.job_id)
doc_ref.set(job.model_dump(mode='json'))
logger.info(f"Created job {job.job_id} in Firestore")
except Exception as e:
logger.error(f"Error creating job {job.job_id}: {e}")
raise
def get_job(self, job_id: str) -> Optional[Job]:
"""Get a job by ID."""
try:
doc_ref = self.db.collection(self.collection).document(job_id)
doc = doc_ref.get()
logger.debug(f"Fetching job {job_id} from collection {self.collection}")
logger.debug(f"Document exists: {doc.exists}")
if not doc.exists:
return None
data = doc.to_dict()
logger.debug(f"Document data keys: {list(data.keys()) if data else 'None'}")
logger.debug(f"Document data: {data}")
return Job(**data)
except Exception as e:
logger.error(f"Error getting job {job_id}: {e}")
raise
def update_job(self, job_id: str, updates: Dict[str, Any]) -> None:
"""Update a job with partial data."""
try:
doc_ref = self.db.collection(self.collection).document(job_id)
# Add updated_at timestamp
updates['updated_at'] = datetime.utcnow()
doc_ref.update(updates)
logger.info(f"Updated job {job_id} in Firestore")
except Exception as e:
logger.error(f"Error updating job {job_id}: {e}")
raise
def update_job_status(
self,
job_id: str,
status: JobStatus,
progress: Optional[int] = None,
message: Optional[str] = None,
**additional_fields
) -> None:
"""Update job status and add timeline event."""
try:
doc_ref = self.db.collection(self.collection).document(job_id)
# Create timeline event
timeline_event = TimelineEvent(
status=status.value,
timestamp=datetime.utcnow().isoformat(),
progress=progress,
message=message
)
# Prepare updates
updates = {
'status': status.value,
'updated_at': datetime.utcnow(),
'timeline': firestore.ArrayUnion([timeline_event.model_dump(mode='json')])
}
if progress is not None:
updates['progress'] = progress
# Add any additional fields
updates.update(additional_fields)
doc_ref.update(updates)
logger.info(f"Updated job {job_id} status to {status.value}")
except Exception as e:
logger.error(f"Error updating job status {job_id}: {e}")
raise
def list_jobs(
self,
status: Optional[JobStatus] = None,
environment: Optional[str] = None,
client_id: Optional[str] = None,
created_after: Optional[datetime] = None,
created_before: Optional[datetime] = None,
limit: int = 100
) -> List[Job]:
"""
List jobs with optional filters.
Args:
status: Filter by job status
environment: Filter by request_metadata.environment (test/production/development)
client_id: Filter by request_metadata.client_id
created_after: Filter jobs created after this datetime
created_before: Filter jobs created before this datetime
limit: Maximum number of jobs to return
Returns:
List of Job objects matching filters, ordered by created_at descending
"""
try:
query = self.db.collection(self.collection)
if status:
query = query.where(filter=FieldFilter('status', '==', status.value))
# Filter by request_metadata fields using dot notation
if environment:
query = query.where(filter=FieldFilter('request_metadata.environment', '==', environment))
if client_id:
query = query.where(filter=FieldFilter('request_metadata.client_id', '==', client_id))
# Date range filters
if created_after:
query = query.where(filter=FieldFilter('created_at', '>=', created_after))
if created_before:
query = query.where(filter=FieldFilter('created_at', '<=', created_before))
query = query.order_by('created_at', direction=firestore.Query.DESCENDING).limit(limit)
docs = query.stream()
jobs = [Job(**doc.to_dict()) for doc in docs]
return jobs
except Exception as e:
logger.error(f"Error listing jobs: {e}")
raise
def delete_jobs_by_filter(
self,
environment: Optional[str] = None,
client_id: Optional[str] = None,
status: Optional[JobStatus] = None,
created_before: Optional[datetime] = None,
) -> int:
"""
Delete multiple jobs matching filter criteria.
CAUTION: This is a destructive operation. Use carefully.
Args:
environment: Delete jobs with this environment (e.g., "test")
client_id: Delete jobs from this client
status: Delete jobs with this status
created_before: Delete jobs created before this datetime
Returns:
Number of jobs deleted
"""
try:
query = self.db.collection(self.collection)
if environment:
query = query.where(filter=FieldFilter('request_metadata.environment', '==', environment))
if client_id:
query = query.where(filter=FieldFilter('request_metadata.client_id', '==', client_id))
if status:
query = query.where(filter=FieldFilter('status', '==', status.value))
if created_before:
query = query.where(filter=FieldFilter('created_at', '<=', created_before))
# Get matching documents
docs = list(query.stream())
deleted_count = 0
# Delete in batches
batch = self.db.batch()
batch_count = 0
for doc in docs:
batch.delete(doc.reference)
batch_count += 1
deleted_count += 1
# Firestore batch limit is 500
if batch_count >= 500:
batch.commit()
batch = self.db.batch()
batch_count = 0
# Commit any remaining deletes
if batch_count > 0:
batch.commit()
logger.info(f"Deleted {deleted_count} jobs matching filter criteria")
return deleted_count
except Exception as e:
logger.error(f"Error deleting jobs by filter: {e}")
raise
def delete_job(self, job_id: str) -> None:
"""Delete a job from Firestore."""
try:
doc_ref = self.db.collection(self.collection).document(job_id)
doc_ref.delete()
logger.info(f"Deleted job {job_id} from Firestore")
except Exception as e:
logger.error(f"Error deleting job {job_id}: {e}")
raise
def append_worker_log(self, job_id: str, log_entry: Dict[str, Any]) -> None:
"""
Atomically append a log entry to worker_logs using ArrayUnion.
This avoids the race condition of read-modify-write when multiple
workers are logging concurrently.
Args:
job_id: Job ID
log_entry: Log entry dict with timestamp, level, worker, message
"""
try:
doc_ref = self.db.collection(self.collection).document(job_id)
doc_ref.update({
'worker_logs': firestore.ArrayUnion([log_entry]),
'updated_at': datetime.utcnow()
})
# Don't log every append - too spammy
except Exception as e:
# Log but don't raise - logging shouldn't break workers
logger.debug(f"Error appending worker log for job {job_id}: {e}")
# ============================================
# Token Management Methods
# ============================================
def create_token(self, token: str, token_data: Dict[str, Any]) -> None:
"""Create a new access token in Firestore."""
try:
doc_ref = self.db.collection(self.tokens_collection).document(token)
doc_ref.set(token_data)
logger.info(f"Created token in Firestore")
except Exception as e:
logger.error(f"Error creating token: {e}")
raise
def get_token(self, token: str) -> Optional[Dict[str, Any]]:
"""Get token data by token string."""
try:
doc_ref = self.db.collection(self.tokens_collection).document(token)
doc = doc_ref.get()
if not doc.exists:
return None
return doc.to_dict()
except Exception as e:
logger.error(f"Error getting token: {e}")
return None
def update_token(self, token: str, updates: Dict[str, Any]) -> None:
"""Update token data."""
try:
doc_ref = self.db.collection(self.tokens_collection).document(token)
doc_ref.update(updates)
logger.info(f"Updated token in Firestore")
except Exception as e:
logger.error(f"Error updating token: {e}")
raise
def increment_token_usage(self, token: str, job_id: str) -> None:
"""Increment token usage count and add job to history."""
try:
doc_ref = self.db.collection(self.tokens_collection).document(token)
# Use Firestore transaction to ensure atomic increment
@firestore.transactional
def update_in_transaction(transaction, doc_ref):
snapshot = doc_ref.get(transaction=transaction)
if not snapshot.exists:
raise ValueError("Token not found")
data = snapshot.to_dict()
current_usage = data.get("usage_count", 0)
jobs = data.get("jobs", [])
# Increment usage
transaction.update(doc_ref, {
"usage_count": current_usage + 1,
"last_used": datetime.utcnow(),
"jobs": firestore.ArrayUnion([{
"job_id": job_id,
"created_at": datetime.utcnow()
}])
})
transaction = self.db.transaction()
update_in_transaction(transaction, doc_ref)
logger.info(f"Incremented token usage for job {job_id}")
except Exception as e:
logger.error(f"Error incrementing token usage: {e}")
raise
def list_tokens(self) -> List[Dict[str, Any]]:
"""List all tokens (admin only)."""
try:
docs = self.db.collection(self.tokens_collection).stream()
tokens = [doc.to_dict() for doc in docs]
logger.info(f"Retrieved {len(tokens)} tokens from Firestore")
return tokens
except Exception as e:
logger.error(f"Error listing tokens: {e}")
return []
-473
View File
@@ -1,473 +0,0 @@
"""
Client for the remote flacfetch HTTP API service.
This client communicates with a dedicated flacfetch VM that handles:
- BitTorrent downloads from private trackers (RED, OPS)
- YouTube downloads
- GCS uploads of downloaded files
The flacfetch service provides:
- Full peer connectivity for torrents (not possible in Cloud Run)
- Indefinite seeding of completed torrents
- Automatic disk cleanup
Usage:
client = get_flacfetch_client()
if client:
# Search for audio
search_result = await client.search("ABBA", "Waterloo")
# Download the best result
download_id = await client.download(
search_id=search_result["search_id"],
result_index=0,
gcs_path="uploads/job123/audio/",
)
# Wait for completion
result = await client.wait_for_download(download_id)
print(f"Downloaded to: {result['gcs_path']}")
"""
import asyncio
import logging
from typing import Any, Dict, List, Optional
import httpx
logger = logging.getLogger(__name__)
class FlacfetchServiceError(Exception):
"""Error communicating with or returned from flacfetch service."""
pass
class FlacfetchClient:
"""
Client for remote flacfetch HTTP API.
All endpoints require authentication via X-API-Key header.
"""
def __init__(
self,
base_url: str,
api_key: str,
timeout: int = 60,
):
"""
Initialize flacfetch client.
Args:
base_url: Base URL of flacfetch service (e.g., http://10.0.0.5:8080)
api_key: API key for authentication
timeout: Default request timeout in seconds
"""
self.base_url = base_url.rstrip('/')
self.api_key = api_key
self.timeout = timeout
logger.info(f"FlacfetchClient initialized with base_url={self.base_url}")
def _headers(self) -> Dict[str, str]:
"""Get request headers with authentication."""
return {
"X-API-Key": self.api_key,
"Content-Type": "application/json",
}
async def health_check(self) -> Dict[str, Any]:
"""
Check if flacfetch service is healthy.
Returns:
Health status dict with transmission, disk, and provider info
Raises:
FlacfetchServiceError: If service is unhealthy or unreachable
"""
try:
async with httpx.AsyncClient() as client:
resp = await client.get(
f"{self.base_url}/health",
headers=self._headers(),
timeout=10,
)
resp.raise_for_status()
data = resp.json()
if data.get("status") not in ["healthy", "degraded"]:
raise FlacfetchServiceError(f"Service unhealthy: {data}")
return data
except httpx.RequestError as e:
raise FlacfetchServiceError(f"Cannot reach flacfetch service: {e}")
except httpx.HTTPStatusError as e:
raise FlacfetchServiceError(f"Health check failed: {e.response.status_code}")
async def search(
self,
artist: str,
title: str,
) -> Dict[str, Any]:
"""
Search for audio matching artist and title.
Args:
artist: Artist name
title: Track title
Returns:
Search response dict with search_id and results list
Raises:
FlacfetchServiceError: On search failure
"""
try:
async with httpx.AsyncClient() as client:
resp = await client.post(
f"{self.base_url}/search",
headers=self._headers(),
json={"artist": artist, "title": title},
timeout=self.timeout,
)
if resp.status_code == 404:
# No results found - return empty results
return {
"search_id": None,
"artist": artist,
"title": title,
"results": [],
"results_count": 0,
}
resp.raise_for_status()
return resp.json()
except httpx.RequestError as e:
raise FlacfetchServiceError(f"Search request failed: {e}")
except httpx.HTTPStatusError as e:
raise FlacfetchServiceError(f"Search failed: {e.response.status_code} - {e.response.text}")
async def download(
self,
search_id: str,
result_index: int,
output_filename: Optional[str] = None,
gcs_path: Optional[str] = None,
) -> str:
"""
Start downloading an audio file.
Args:
search_id: Search ID from previous search
result_index: Index of result to download
output_filename: Optional custom filename (without extension)
gcs_path: GCS path for upload (e.g., "uploads/job123/audio/")
Returns:
Download ID for tracking progress
Raises:
FlacfetchServiceError: On download start failure
"""
try:
async with httpx.AsyncClient() as client:
payload = {
"search_id": search_id,
"result_index": result_index,
}
if output_filename:
payload["output_filename"] = output_filename
if gcs_path:
payload["upload_to_gcs"] = True
payload["gcs_path"] = gcs_path
resp = await client.post(
f"{self.base_url}/download",
headers=self._headers(),
json=payload,
timeout=self.timeout,
)
resp.raise_for_status()
data = resp.json()
return data["download_id"]
except httpx.RequestError as e:
raise FlacfetchServiceError(f"Download request failed: {e}")
except httpx.HTTPStatusError as e:
raise FlacfetchServiceError(f"Download start failed: {e.response.status_code} - {e.response.text}")
async def download_by_id(
self,
source_name: str,
source_id: str,
output_filename: Optional[str] = None,
target_file: Optional[str] = None,
download_url: Optional[str] = None,
gcs_path: Optional[str] = None,
) -> str:
"""
Start downloading directly by source ID (no prior search required).
This is useful when you have stored the source_id from a previous search
and want to download later without re-searching.
Args:
source_name: Provider name (RED, OPS, YouTube, Spotify)
source_id: Source-specific ID (torrent ID, video ID, track ID)
output_filename: Optional custom filename (without extension)
target_file: For torrents, specific file to extract
download_url: For YouTube/Spotify, direct URL (optional)
gcs_path: GCS path for upload (e.g., "uploads/job123/audio/")
Returns:
Download ID for tracking progress
Raises:
FlacfetchServiceError: On download start failure
"""
try:
async with httpx.AsyncClient() as client:
payload = {
"source_name": source_name,
"source_id": source_id,
}
if output_filename:
payload["output_filename"] = output_filename
if target_file:
payload["target_file"] = target_file
if download_url:
payload["download_url"] = download_url
if gcs_path:
payload["upload_to_gcs"] = True
payload["gcs_path"] = gcs_path
resp = await client.post(
f"{self.base_url}/download-by-id",
headers=self._headers(),
json=payload,
timeout=self.timeout,
)
resp.raise_for_status()
data = resp.json()
return data["download_id"]
except httpx.RequestError as e:
raise FlacfetchServiceError(f"Download by ID request failed: {e}")
except httpx.HTTPStatusError as e:
raise FlacfetchServiceError(f"Download by ID failed: {e.response.status_code} - {e.response.text}")
async def get_download_status(self, download_id: str) -> Dict[str, Any]:
"""
Get the current status of a download.
Args:
download_id: Download ID from start_download
Returns:
Status dict with progress, speed, path info
Raises:
FlacfetchServiceError: On status check failure
"""
try:
async with httpx.AsyncClient() as client:
resp = await client.get(
f"{self.base_url}/download/{download_id}/status",
headers=self._headers(),
timeout=10,
)
resp.raise_for_status()
return resp.json()
except httpx.RequestError as e:
raise FlacfetchServiceError(f"Status request failed: {e}")
except httpx.HTTPStatusError as e:
if e.response.status_code == 404:
raise FlacfetchServiceError(f"Download not found: {download_id}")
raise FlacfetchServiceError(f"Status check failed: {e.response.status_code}")
async def wait_for_download(
self,
download_id: str,
timeout: int = 600,
poll_interval: float = 2.0,
progress_callback: Optional[callable] = None,
) -> Dict[str, Any]:
"""
Wait for a download to complete.
Args:
download_id: Download ID to wait for
timeout: Maximum wait time in seconds
poll_interval: Time between status checks
progress_callback: Optional callback(status_dict) for progress updates
Returns:
Final status dict with output_path or gcs_path
Raises:
FlacfetchServiceError: On download failure or timeout
"""
elapsed = 0.0
while elapsed < timeout:
status = await self.get_download_status(download_id)
if progress_callback:
try:
progress_callback(status)
except Exception as e:
logger.warning(f"Progress callback error: {e}")
download_status = status.get("status")
if download_status in ["complete", "seeding"]:
logger.info(f"Download {download_id} complete: {status.get('gcs_path') or status.get('output_path')}")
return status
elif download_status == "failed":
error = status.get("error", "Unknown error")
raise FlacfetchServiceError(f"Download failed: {error}")
elif download_status == "cancelled":
raise FlacfetchServiceError("Download was cancelled")
# Log progress
progress = status.get("progress", 0)
speed = status.get("download_speed_kbps", 0)
peers = status.get("peers", 0)
logger.debug(
f"Download {download_id}: {progress:.1f}% "
f"({speed:.1f} KB/s, {peers} peers)"
)
await asyncio.sleep(poll_interval)
elapsed += poll_interval
raise FlacfetchServiceError(f"Download timed out after {timeout}s")
async def list_torrents(self) -> Dict[str, Any]:
"""
List all torrents in Transmission.
Returns:
Torrent list response with torrent info and totals
"""
try:
async with httpx.AsyncClient() as client:
resp = await client.get(
f"{self.base_url}/torrents",
headers=self._headers(),
timeout=30,
)
resp.raise_for_status()
return resp.json()
except httpx.RequestError as e:
raise FlacfetchServiceError(f"List torrents failed: {e}")
except httpx.HTTPStatusError as e:
raise FlacfetchServiceError(f"List torrents failed: {e.response.status_code}")
async def delete_torrent(
self,
torrent_id: int,
delete_data: bool = True,
) -> Dict[str, Any]:
"""
Delete a torrent from Transmission.
Args:
torrent_id: Transmission torrent ID
delete_data: Also delete downloaded files
Returns:
Delete response
"""
try:
async with httpx.AsyncClient() as client:
resp = await client.delete(
f"{self.base_url}/torrents/{torrent_id}",
headers=self._headers(),
params={"delete_data": str(delete_data).lower()},
timeout=30,
)
resp.raise_for_status()
return resp.json()
except httpx.RequestError as e:
raise FlacfetchServiceError(f"Delete torrent failed: {e}")
except httpx.HTTPStatusError as e:
raise FlacfetchServiceError(f"Delete torrent failed: {e.response.status_code}")
async def cleanup_torrents(
self,
strategy: str = "oldest",
target_free_gb: float = 10.0,
) -> Dict[str, Any]:
"""
Trigger disk cleanup.
Args:
strategy: Cleanup strategy (oldest, largest, lowest_ratio)
target_free_gb: Target free space to achieve
Returns:
Cleanup response with removed count and freed space
"""
try:
async with httpx.AsyncClient() as client:
resp = await client.post(
f"{self.base_url}/torrents/cleanup",
headers=self._headers(),
json={
"strategy": strategy,
"target_free_gb": target_free_gb,
},
timeout=60,
)
resp.raise_for_status()
return resp.json()
except httpx.RequestError as e:
raise FlacfetchServiceError(f"Cleanup failed: {e}")
except httpx.HTTPStatusError as e:
raise FlacfetchServiceError(f"Cleanup failed: {e.response.status_code}")
# Singleton client instance
_client: Optional[FlacfetchClient] = None
def get_flacfetch_client() -> Optional[FlacfetchClient]:
"""
Get the flacfetch client if configured.
Returns:
FlacfetchClient instance if FLACFETCH_API_URL and FLACFETCH_API_KEY are set,
None otherwise (indicating local-only mode).
"""
global _client
if _client is None:
from backend.config import get_settings
settings = get_settings()
if settings.flacfetch_api_url and settings.flacfetch_api_key:
_client = FlacfetchClient(
base_url=settings.flacfetch_api_url,
api_key=settings.flacfetch_api_key,
)
logger.info("FlacfetchClient initialized for remote service")
else:
logger.debug("FlacfetchClient not configured (missing URL or API key)")
return _client
def reset_flacfetch_client():
"""Reset the singleton client (for testing)."""
global _client
_client = None
-309
View File
@@ -1,309 +0,0 @@
"""Native Google Drive API service for cloud backend.
This service provides Google Drive operations using the native API,
for uploading files to public share folders. It handles:
- Folder creation and lookup
- File uploads with resumable upload support
- Uploading to organized folder structure (MP4/, MP4-720p/, CDG/)
Credentials are loaded from Google Cloud Secret Manager and can be
shared with YouTube credentials if scopes include drive.file.
"""
import json
import logging
import os
from typing import Any, Dict, Optional
from backend.config import get_settings
logger = logging.getLogger(__name__)
class GoogleDriveService:
"""Google Drive operations using native API."""
# Secret Manager secret name for Google Drive credentials
# Can be same as YouTube if scopes include drive.file
GDRIVE_CREDENTIALS_SECRET = "gdrive-oauth-credentials"
def __init__(self):
self.settings = get_settings()
self._service = None
self._credentials_data: Optional[Dict[str, Any]] = None
self._loaded = False
def _load_credentials(self) -> Optional[Dict[str, Any]]:
"""Load OAuth credentials from Secret Manager."""
if self._loaded:
return self._credentials_data
try:
creds_json = self.settings.get_secret(self.GDRIVE_CREDENTIALS_SECRET)
if not creds_json:
# Try falling back to YouTube credentials (may have drive scope)
logger.info(
"Google Drive credentials not found, trying YouTube credentials"
)
creds_json = self.settings.get_secret("youtube-oauth-credentials")
if not creds_json:
logger.warning("Google Drive credentials not found in Secret Manager")
self._loaded = True
return None
self._credentials_data = json.loads(creds_json)
# Validate required fields
required_fields = ["refresh_token", "client_id", "client_secret"]
missing = [f for f in required_fields if not self._credentials_data.get(f)]
if missing:
logger.error(f"Google Drive credentials missing required fields: {missing}")
self._credentials_data = None
self._loaded = True
return None
logger.info("Google Drive credentials loaded successfully from Secret Manager")
self._loaded = True
return self._credentials_data
except json.JSONDecodeError as e:
logger.error(f"Failed to parse Google Drive credentials JSON: {e}")
self._loaded = True
return None
except Exception as e:
logger.error(f"Failed to load Google Drive credentials: {e}")
self._loaded = True
return None
@property
def is_configured(self) -> bool:
"""Check if Google Drive credentials are available."""
creds = self._load_credentials()
return creds is not None
@property
def service(self):
"""Get or create Google Drive service."""
if self._service is None:
# Import here to avoid import errors if google packages not installed
try:
from google.oauth2.credentials import Credentials
from googleapiclient.discovery import build
except ImportError:
raise ImportError(
"google-api-python-client and google-auth packages are required. "
"Install them with: pip install google-api-python-client google-auth"
)
creds_data = self._load_credentials()
if not creds_data:
raise RuntimeError(
"Google Drive credentials not configured in Secret Manager"
)
# Create credentials object
credentials = Credentials(
token=creds_data.get("token"),
refresh_token=creds_data.get("refresh_token"),
token_uri=creds_data.get(
"token_uri", "https://oauth2.googleapis.com/token"
),
client_id=creds_data.get("client_id"),
client_secret=creds_data.get("client_secret"),
scopes=creds_data.get(
"scopes", ["https://www.googleapis.com/auth/drive.file"]
),
)
self._service = build("drive", "v3", credentials=credentials)
logger.info("Google Drive service initialized successfully")
return self._service
def get_or_create_folder(self, parent_id: str, folder_name: str) -> str:
"""
Get existing folder or create new one, return folder ID.
Args:
parent_id: Parent folder ID
folder_name: Name of folder to find or create
Returns:
Folder ID
"""
logger.info(f"Looking for folder '{folder_name}' in parent {parent_id}")
# Search for existing folder
query = (
f"name='{folder_name}' and '{parent_id}' in parents "
f"and mimeType='application/vnd.google-apps.folder' and trashed=false"
)
results = self.service.files().list(q=query, fields="files(id, name)").execute()
if results.get("files"):
folder_id = results["files"][0]["id"]
logger.info(f"Found existing folder '{folder_name}': {folder_id}")
return folder_id
# Create folder
logger.info(f"Creating new folder '{folder_name}'")
metadata = {
"name": folder_name,
"mimeType": "application/vnd.google-apps.folder",
"parents": [parent_id],
}
folder = self.service.files().create(body=metadata, fields="id").execute()
folder_id = folder["id"]
logger.info(f"Created folder '{folder_name}': {folder_id}")
return folder_id
def upload_file(
self,
local_path: str,
parent_id: str,
filename: str,
replace_existing: bool = True,
) -> str:
"""
Upload a file to a specific Drive folder.
Args:
local_path: Local file path
parent_id: Parent folder ID in Google Drive
filename: Name for the file in Drive
replace_existing: If True, delete existing file with same name first
Returns:
File ID of uploaded file
"""
from googleapiclient.http import MediaFileUpload
file_size = os.path.getsize(local_path)
logger.info(
f"Uploading {local_path} ({file_size / 1024 / 1024:.1f} MB) "
f"as '{filename}' to folder {parent_id}"
)
# Determine MIME type
ext = os.path.splitext(local_path)[1].lower()
mime_types = {
".mp4": "video/mp4",
".mkv": "video/x-matroska",
".zip": "application/zip",
".flac": "audio/flac",
".mp3": "audio/mpeg",
".wav": "audio/wav",
".png": "image/png",
".jpg": "image/jpeg",
".jpeg": "image/jpeg",
}
mime_type = mime_types.get(ext, "application/octet-stream")
# Check for existing file with same name
if replace_existing:
query = (
f"name='{filename}' and '{parent_id}' in parents and trashed=false"
)
results = self.service.files().list(q=query, fields="files(id)").execute()
for existing_file in results.get("files", []):
logger.info(f"Deleting existing file: {existing_file['id']}")
self.service.files().delete(fileId=existing_file["id"]).execute()
# Upload file
metadata = {"name": filename, "parents": [parent_id]}
media = MediaFileUpload(local_path, mimetype=mime_type, resumable=True)
file_result = (
self.service.files().create(body=metadata, media_body=media, fields="id").execute()
)
file_id = file_result["id"]
logger.info(f"Successfully uploaded '{filename}': {file_id}")
return file_id
def upload_to_public_share(
self,
root_folder_id: str,
brand_code: str,
base_name: str,
output_files: dict,
) -> Dict[str, str]:
"""
Upload final files to public share folder structure.
Creates/uses subfolders:
- MP4/{brand_code} - {base_name}.mp4 (lossy 4k)
- MP4-720p/{brand_code} - {base_name}.mp4
- CDG/{brand_code} - {base_name}.zip
Args:
root_folder_id: Google Drive folder ID for public share root
brand_code: Brand code (e.g., "NOMAD-1163")
base_name: Base filename (e.g., "Artist - Title")
output_files: Dictionary with output file paths:
- final_karaoke_lossy_mp4: 4K lossy MP4
- final_karaoke_lossy_720p_mp4: 720p lossy MP4
- final_karaoke_cdg_zip: CDG package ZIP
Returns:
Dictionary mapping category to uploaded file ID
"""
filename_base = f"{brand_code} - {base_name}"
uploaded_files = {}
logger.info(
f"Uploading public share files to Google Drive folder {root_folder_id}"
)
logger.info(f"Filename base: {filename_base}")
# Upload lossy 4k to MP4/
lossy_mp4_path = output_files.get("final_karaoke_lossy_mp4")
if lossy_mp4_path and os.path.exists(lossy_mp4_path):
mp4_folder_id = self.get_or_create_folder(root_folder_id, "MP4")
file_id = self.upload_file(
lossy_mp4_path,
mp4_folder_id,
f"{filename_base}.mp4",
)
uploaded_files["mp4"] = file_id
logger.info(f"Uploaded 4K MP4 to MP4/ folder")
# Upload 720p to MP4-720p/
mp4_720p_path = output_files.get("final_karaoke_lossy_720p_mp4")
if mp4_720p_path and os.path.exists(mp4_720p_path):
mp4_720_folder_id = self.get_or_create_folder(root_folder_id, "MP4-720p")
file_id = self.upload_file(
mp4_720p_path,
mp4_720_folder_id,
f"{filename_base}.mp4",
)
uploaded_files["mp4_720p"] = file_id
logger.info(f"Uploaded 720p MP4 to MP4-720p/ folder")
# Upload CDG ZIP to CDG/
cdg_zip_path = output_files.get("final_karaoke_cdg_zip")
if cdg_zip_path and os.path.exists(cdg_zip_path):
cdg_folder_id = self.get_or_create_folder(root_folder_id, "CDG")
file_id = self.upload_file(
cdg_zip_path,
cdg_folder_id,
f"{filename_base}.zip",
)
uploaded_files["cdg"] = file_id
logger.info(f"Uploaded CDG ZIP to CDG/ folder")
logger.info(f"Public share upload complete: {len(uploaded_files)} files uploaded")
return uploaded_files
# Singleton instance
_gdrive_service: Optional[GoogleDriveService] = None
def get_gdrive_service() -> GoogleDriveService:
"""Get the singleton Google Drive service instance."""
global _gdrive_service
if _gdrive_service is None:
_gdrive_service = GoogleDriveService()
return _gdrive_service
-258
View File
@@ -1,258 +0,0 @@
"""
Job-aware logging utilities.
This module provides a context manager and logging handler that captures
Python logging output and forwards it to Firestore for real-time streaming
to the CLI.
Usage:
with JobLogContext(job_id, worker="review") as log_context:
# Any logging within this block will be captured and sent to Firestore
logger.info("Processing...") # This will be stored and streamed to CLI
"""
import logging
import threading
from typing import Optional
from contextlib import contextmanager
from datetime import datetime
from backend.services.firestore_service import FirestoreService
class JobLogHandler(logging.Handler):
"""
A logging handler that forwards log records to Firestore for job tracking.
This enables real-time log streaming to the CLI during long-running operations
like lyrics correction, preview generation, etc.
"""
def __init__(
self,
job_id: str,
worker: str = "review",
firestore: Optional[FirestoreService] = None,
min_level: int = logging.INFO,
batch_size: int = 1, # Send immediately for real-time experience
):
"""
Initialize the job log handler.
Args:
job_id: The job ID to associate logs with
worker: Worker name (e.g., "review", "add-lyrics", "preview")
firestore: FirestoreService instance (creates one if not provided)
min_level: Minimum log level to capture (default INFO)
batch_size: Number of logs to batch before sending (1 = immediate)
"""
super().__init__()
self.job_id = job_id
self.worker = worker
self.firestore = firestore or FirestoreService()
self.min_level = min_level
self.batch_size = batch_size
self._log_buffer = []
self._lock = threading.Lock()
# Set the handler's level
self.setLevel(min_level)
# Create a formatter that extracts the useful parts
self.setFormatter(logging.Formatter('%(message)s'))
def emit(self, record: logging.LogRecord) -> None:
"""
Process a log record and send it to Firestore.
This method is called by the logging framework for each log message.
"""
try:
# Skip if below minimum level
if record.levelno < self.min_level:
return
# Format the message
message = self.format(record)
# Truncate very long messages
if len(message) > 1000:
message = message[:997] + "..."
# Create log entry
log_entry = {
'timestamp': datetime.utcnow().isoformat() + 'Z',
'level': record.levelname,
'worker': self.worker,
'message': message,
}
# Add to buffer
with self._lock:
self._log_buffer.append(log_entry)
# Flush if batch size reached
if len(self._log_buffer) >= self.batch_size:
self._flush_buffer()
except Exception:
# Never let logging errors propagate
self.handleError(record)
def _flush_buffer(self) -> None:
"""Flush buffered logs to Firestore."""
if not self._log_buffer:
return
try:
for log_entry in self._log_buffer:
self.firestore.append_worker_log(self.job_id, log_entry)
self._log_buffer = []
except Exception:
# If Firestore fails, just clear buffer and continue
self._log_buffer = []
def flush(self) -> None:
"""Flush any remaining buffered logs."""
with self._lock:
self._flush_buffer()
def close(self) -> None:
"""Clean up the handler."""
self.flush()
super().close()
@contextmanager
def job_log_context(
job_id: str,
worker: str = "review",
logger_names: Optional[list] = None,
min_level: int = logging.INFO,
):
"""
Context manager that captures logs and forwards them to Firestore.
This allows the CLI to see real-time logs from operations like
add-lyrics, preview generation, etc.
Args:
job_id: Job ID to associate logs with
worker: Worker name for log categorization
logger_names: List of logger names to capture (None = capture common loggers)
min_level: Minimum log level to capture
Usage:
with job_log_context(job_id, worker="add-lyrics"):
# All logging in this block will be captured
correction_operations.add_lyrics_source(...)
"""
# Default loggers to capture - these cover most of our processing
# Note: We intentionally DON'T include the root logger ('') to avoid
# duplicate log entries since logs propagate up the logger hierarchy.
# Only capture at the top-level package loggers.
if logger_names is None:
logger_names = [
'backend.api.routes.review',
'lyrics_transcriber',
'karaoke_gen',
]
# Create the handler
handler = JobLogHandler(
job_id=job_id,
worker=worker,
min_level=min_level,
)
# Add handler to all specified loggers
loggers = []
for name in logger_names:
log = logging.getLogger(name) if name else logging.getLogger()
log.addHandler(handler)
loggers.append(log)
try:
# Log start
handler.emit(logging.LogRecord(
name='job_log_context',
level=logging.INFO,
pathname='',
lineno=0,
msg=f"🚀 Starting {worker} operation for job {job_id}",
args=(),
exc_info=None,
))
yield handler
finally:
# Log completion
handler.emit(logging.LogRecord(
name='job_log_context',
level=logging.INFO,
pathname='',
lineno=0,
msg=f"✅ Completed {worker} operation for job {job_id}",
args=(),
exc_info=None,
))
# Flush remaining logs
handler.flush()
# Remove handler from all loggers
for log in loggers:
log.removeHandler(handler)
# Close handler
handler.close()
class JobLogger:
"""
A convenience class for direct logging to a job's log stream.
Use this when you want to log specific messages without capturing
all logging output.
"""
def __init__(self, job_id: str, worker: str = "review"):
"""
Initialize job logger.
Args:
job_id: Job ID to log to
worker: Worker name for log categorization
"""
self.job_id = job_id
self.worker = worker
self.firestore = FirestoreService()
def _log(self, level: str, message: str) -> None:
"""Internal logging method."""
log_entry = {
'timestamp': datetime.utcnow().isoformat() + 'Z',
'level': level,
'worker': self.worker,
'message': message[:1000],
}
try:
self.firestore.append_worker_log(self.job_id, log_entry)
except Exception:
pass # Don't let logging errors propagate
def info(self, message: str) -> None:
"""Log an INFO message."""
self._log('INFO', message)
def warning(self, message: str) -> None:
"""Log a WARNING message."""
self._log('WARNING', message)
def error(self, message: str) -> None:
"""Log an ERROR message."""
self._log('ERROR', message)
def debug(self, message: str) -> None:
"""Log a DEBUG message."""
self._log('DEBUG', message)
-639
View File
@@ -1,639 +0,0 @@
"""
Job management and queue operations.
This module handles the complete job lifecycle including:
- Job creation and initialization
- State transitions and validation
- Worker coordination (parallel audio + lyrics processing)
- Progress tracking and timeline events
- Error handling and retries
"""
import logging
import uuid
from datetime import datetime
from typing import Optional, Dict, Any, List
from backend.models.job import Job, JobStatus, JobCreate, STATE_TRANSITIONS
from backend.services.firestore_service import FirestoreService
from backend.services.storage_service import StorageService
logger = logging.getLogger(__name__)
class JobManager:
"""Manager for job lifecycle and state."""
def __init__(self):
"""Initialize job manager with required services."""
self.firestore = FirestoreService()
self.storage = StorageService()
def create_job(self, job_create: JobCreate) -> Job:
"""
Create a new job with initial state PENDING.
Jobs start in PENDING state and transition to DOWNLOADING
when a worker picks them up.
"""
job_id = str(uuid.uuid4())[:8]
now = datetime.utcnow()
job = Job(
job_id=job_id,
status=JobStatus.PENDING, # New state machine starts with PENDING
progress=0,
created_at=now,
updated_at=now,
url=str(job_create.url) if job_create.url else None,
artist=job_create.artist,
title=job_create.title,
# User preferences
enable_cdg=job_create.enable_cdg,
enable_txt=job_create.enable_txt,
enable_youtube_upload=job_create.enable_youtube_upload,
youtube_description=job_create.youtube_description,
youtube_description_template=job_create.youtube_description_template, # video_worker reads this
webhook_url=job_create.webhook_url,
user_email=job_create.user_email,
# Distribution settings
brand_prefix=job_create.brand_prefix,
discord_webhook_url=job_create.discord_webhook_url,
dropbox_path=job_create.dropbox_path,
gdrive_folder_id=job_create.gdrive_folder_id,
# Theme configuration
theme_id=job_create.theme_id,
color_overrides=job_create.color_overrides,
style_params_gcs_path=job_create.style_params_gcs_path,
style_assets=job_create.style_assets,
# Two-phase workflow (Batch 6)
prep_only=job_create.prep_only,
finalise_only=job_create.finalise_only,
keep_brand_code=job_create.keep_brand_code,
# Request metadata (for tracking and filtering)
request_metadata=job_create.request_metadata,
)
self.firestore.create_job(job)
logger.info(f"Created new job {job_id} with status PENDING")
return job
def get_job(self, job_id: str) -> Optional[Job]:
"""Get a job by ID."""
return self.firestore.get_job(job_id)
def update_job_status(
self,
job_id: str,
status: JobStatus,
progress: Optional[int] = None,
message: Optional[str] = None,
**kwargs
) -> None:
"""Update job status with timeline tracking."""
self.firestore.update_job_status(
job_id=job_id,
status=status,
progress=progress,
message=message,
**kwargs
)
def update_job(self, job_id: str, updates: Dict[str, Any]) -> None:
"""Update job with arbitrary fields."""
self.firestore.update_job(job_id, updates)
def list_jobs(
self,
status: Optional[JobStatus] = None,
environment: Optional[str] = None,
client_id: Optional[str] = None,
created_after: Optional[datetime] = None,
created_before: Optional[datetime] = None,
limit: int = 100
) -> List[Job]:
"""
List jobs with optional filtering.
Args:
status: Filter by job status
environment: Filter by request_metadata.environment (test/production/development)
client_id: Filter by request_metadata.client_id (customer identifier)
created_after: Filter jobs created after this datetime
created_before: Filter jobs created before this datetime
limit: Maximum number of jobs to return
Returns:
List of Job objects matching filters
"""
return self.firestore.list_jobs(
status=status,
environment=environment,
client_id=client_id,
created_after=created_after,
created_before=created_before,
limit=limit
)
def delete_jobs_by_filter(
self,
environment: Optional[str] = None,
client_id: Optional[str] = None,
status: Optional[JobStatus] = None,
created_before: Optional[datetime] = None,
delete_files: bool = True
) -> Dict[str, Any]:
"""
Delete multiple jobs matching filter criteria.
CAUTION: This is a destructive operation. Use carefully.
Args:
environment: Delete jobs with this environment (e.g., "test")
client_id: Delete jobs from this client
status: Delete jobs with this status
created_before: Delete jobs created before this datetime
delete_files: Also delete GCS files (default True)
Returns:
Dict with deletion statistics
"""
# First get matching jobs to delete their files
if delete_files:
jobs = self.firestore.list_jobs(
status=status,
environment=environment,
client_id=client_id,
created_before=created_before,
limit=10000 # High limit for deletion
)
files_deleted = 0
for job in jobs:
# Delete files from various locations
try:
# Delete uploads folder
self.storage.delete_folder(f"uploads/{job.job_id}/")
# Delete jobs folder
self.storage.delete_folder(f"jobs/{job.job_id}/")
files_deleted += 1
except Exception as e:
logger.warning(f"Error deleting files for job {job.job_id}: {e}")
# Delete the jobs from Firestore
deleted_count = self.firestore.delete_jobs_by_filter(
environment=environment,
client_id=client_id,
status=status,
created_before=created_before
)
return {
'jobs_deleted': deleted_count,
'files_deleted': files_deleted if delete_files else 0
}
def mark_job_error(self, job_id: str, error_message: str) -> None:
"""Mark a job as errored."""
self.firestore.update_job_status(
job_id=job_id,
status=JobStatus.ERROR,
progress=0,
message=error_message,
error_message=error_message
)
logger.error(f"Job {job_id} marked as error: {error_message}")
def get_output_urls(self, job_id: str) -> Dict[str, str]:
"""Generate signed URLs for job output files."""
job = self.get_job(job_id)
if not job or not job.output_files:
return {}
urls = {}
for file_type, gcs_path in job.output_files.items():
try:
urls[file_type] = self.storage.generate_signed_url(gcs_path, expiration_minutes=120)
except Exception as e:
logger.error(f"Error generating URL for {file_type}: {e}")
return urls
def delete_job(self, job_id: str, delete_files: bool = True) -> None:
"""Delete a job and optionally its files."""
if delete_files:
job = self.get_job(job_id)
if job and job.output_files:
for gcs_path in job.output_files.values():
try:
self.storage.delete_file(gcs_path)
except Exception as e:
logger.error(f"Error deleting file {gcs_path}: {e}")
self.firestore.delete_job(job_id)
logger.info(f"Deleted job {job_id}")
def validate_state_transition(self, job_id: str, new_status: JobStatus) -> bool:
"""
Validate that a state transition is legal.
Returns:
True if transition is valid, False otherwise
"""
job = self.get_job(job_id)
if not job:
logger.error(f"Job {job_id} not found")
return False
current_status = job.status
valid_transitions = STATE_TRANSITIONS.get(current_status, [])
if new_status not in valid_transitions:
logger.error(
f"Invalid state transition for job {job_id}: "
f"{current_status} -> {new_status}. "
f"Valid transitions: {valid_transitions}"
)
return False
return True
def transition_to_state(
self,
job_id: str,
new_status: JobStatus,
progress: Optional[int] = None,
message: Optional[str] = None,
state_data_updates: Optional[Dict[str, Any]] = None
) -> bool:
"""
Transition job to new state with validation.
Args:
job_id: Job ID
new_status: Target state
progress: Progress percentage (0-100)
message: Timeline message
state_data_updates: Updates to state_data field
Returns:
True if transition succeeded, False otherwise
"""
if not self.validate_state_transition(job_id, new_status):
return False
updates = {
'status': new_status,
'updated_at': datetime.utcnow()
}
if progress is not None:
updates['progress'] = progress
# Generate review token when entering AWAITING_REVIEW state
if new_status == JobStatus.AWAITING_REVIEW:
from backend.api.dependencies import generate_review_token, get_review_token_expiry
review_token = generate_review_token()
review_token_expires = get_review_token_expiry(hours=48) # 48 hour expiry
updates['review_token'] = review_token
updates['review_token_expires_at'] = review_token_expires
logger.info(f"Generated review token for job {job_id}, expires in 48 hours")
# Generate instrumental token when entering AWAITING_INSTRUMENTAL_SELECTION state
if new_status == JobStatus.AWAITING_INSTRUMENTAL_SELECTION:
from backend.api.dependencies import generate_review_token, get_review_token_expiry
instrumental_token = generate_review_token() # Reuse same token generator
instrumental_token_expires = get_review_token_expiry(hours=48) # 48 hour expiry
updates['instrumental_token'] = instrumental_token
updates['instrumental_token_expires_at'] = instrumental_token_expires
logger.info(f"Generated instrumental token for job {job_id}, expires in 48 hours")
# If we have state_data_updates, merge them with existing state_data
merged_state_data = None
if state_data_updates:
job = self.get_job(job_id)
if job:
merged_state_data = {**job.state_data, **state_data_updates}
# Update job status (includes timeline event), passing state_data if present
if merged_state_data is not None:
self.update_job_status(
job_id=job_id,
status=new_status,
progress=progress,
message=message,
state_data=merged_state_data
)
else:
self.update_job_status(
job_id=job_id,
status=new_status,
progress=progress,
message=message
)
# Apply review token update separately if generated
if new_status == JobStatus.AWAITING_REVIEW and 'review_token' in updates:
self.firestore.update_job(job_id, {
'review_token': updates['review_token'],
'review_token_expires_at': updates['review_token_expires_at']
})
# Apply instrumental token update separately if generated
if new_status == JobStatus.AWAITING_INSTRUMENTAL_SELECTION and 'instrumental_token' in updates:
self.firestore.update_job(job_id, {
'instrumental_token': updates['instrumental_token'],
'instrumental_token_expires_at': updates['instrumental_token_expires_at']
})
logger.info(f"Job {job_id} transitioned to {new_status}")
return True
def update_state_data(self, job_id: str, key: str, value: Any) -> None:
"""
Update a specific key in the job's state_data field.
This is used by workers to store stage-specific metadata.
"""
job = self.get_job(job_id)
if not job:
logger.error(f"Job {job_id} not found")
return
state_data = job.state_data.copy()
state_data[key] = value
self.update_job(job_id, {'state_data': state_data})
logger.debug(f"Job {job_id} state_data updated: {key} = {value}")
def fail_job(self, job_id: str, error_message: str, error_details: Optional[Dict[str, Any]] = None) -> bool:
"""
Mark a job as failed with error information.
Args:
job_id: Job ID
error_message: Human-readable error message
error_details: Optional structured error details
Returns:
True if successful
"""
try:
# Update error fields
self.update_job(job_id, {
'error_message': error_message,
'error_details': error_details or {}
})
# Use update_job_status which handles timeline
self.update_job_status(
job_id=job_id,
status=JobStatus.FAILED,
message=error_message
)
logger.error(f"Job {job_id} failed: {error_message}")
return True
except Exception as e:
logger.error(f"Error marking job {job_id} as failed: {e}")
return False
def update_file_url(self, job_id: str, category: str, file_type: str, url: str) -> None:
"""
Update a file URL in the job's file_urls structure.
Args:
job_id: Job ID
category: Category (e.g., "stems", "lyrics", "finals")
file_type: File type within category (e.g., "clean", "lrc", "lossless_4k_mp4")
url: GCS URL
"""
job = self.get_job(job_id)
if not job:
logger.error(f"Job {job_id} not found")
return
file_urls = job.file_urls.copy()
if category not in file_urls:
file_urls[category] = {}
file_urls[category][file_type] = url
self.update_job(job_id, {'file_urls': file_urls})
logger.debug(f"Job {job_id} file URL updated: {category}.{file_type}")
def check_parallel_processing_complete(self, job_id: str) -> bool:
"""
Check if both parallel tracks (audio + lyrics) are complete.
This is called after audio_complete or lyrics_complete to determine
if we can proceed to screen generation.
Returns:
True if both tracks complete, False otherwise
"""
job = self.get_job(job_id)
if not job:
return False
audio_complete = job.state_data.get('audio_complete', False)
lyrics_complete = job.state_data.get('lyrics_complete', False)
return audio_complete and lyrics_complete
def mark_audio_complete(self, job_id: str) -> None:
"""
Mark audio processing as complete and check if can proceed.
If lyrics are also complete, automatically triggers screens worker.
"""
self.update_state_data(job_id, 'audio_complete', True)
if self.check_parallel_processing_complete(job_id):
logger.info(f"Job {job_id}: Both audio and lyrics complete, triggering screens worker")
# Transition happens in screens worker
# We just trigger it here
self._trigger_screens_worker(job_id)
def mark_lyrics_complete(self, job_id: str) -> None:
"""
Mark lyrics processing as complete and check if can proceed.
If audio is also complete, automatically triggers screens worker.
"""
self.update_state_data(job_id, 'lyrics_complete', True)
if self.check_parallel_processing_complete(job_id):
logger.info(f"Job {job_id}: Both audio and lyrics complete, triggering screens worker")
# Transition happens in screens worker
# We just trigger it here
self._trigger_screens_worker(job_id)
def _trigger_screens_worker(self, job_id: str) -> None:
"""
Trigger screens generation worker.
Uses WorkerService to make HTTP call to internal API.
This must be async, so we use asyncio to create a task.
"""
import asyncio
from backend.services.worker_service import get_worker_service
logger.info(f"Job {job_id}: Triggering screens worker")
# Create async task to trigger worker
# This allows us to call async code from sync context
async def _trigger():
worker_service = get_worker_service()
await worker_service.trigger_screens_worker(job_id)
# Create task in event loop
try:
loop = asyncio.get_event_loop()
if loop.is_running():
# If loop is already running, create task
asyncio.create_task(_trigger())
else:
# If no loop, run directly
asyncio.run(_trigger())
except RuntimeError:
# Fallback: just log
logger.warning(f"Job {job_id}: Could not trigger screens worker (no event loop)")
# TODO: In production, use message queue instead
def cancel_job(self, job_id: str, reason: Optional[str] = None) -> bool:
"""
Cancel a job.
Only jobs in non-terminal states can be cancelled.
"""
job = self.get_job(job_id)
if not job:
return False
terminal_states = [JobStatus.COMPLETE, JobStatus.FAILED, JobStatus.CANCELLED]
if job.status in terminal_states:
logger.warning(f"Cannot cancel job {job_id} in terminal state {job.status}")
return False
message = f"Job cancelled{f': {reason}' if reason else ''}"
self.update_job_status(
job_id=job_id,
status=JobStatus.CANCELLED,
message=message
)
logger.info(f"Job {job_id} cancelled")
return True
def mark_job_failed(
self,
job_id: str,
error_message: str,
error_details: Optional[Dict[str, Any]] = None
) -> None:
"""
Mark a job as failed with error details.
This replaces mark_job_error() with better error tracking.
"""
updates = {
'error_message': error_message
}
if error_details:
updates['error_details'] = error_details
self.firestore.update_job_status(
job_id=job_id,
status=JobStatus.FAILED,
progress=0,
message=f"Failed: {error_message}",
**updates
)
logger.error(f"Job {job_id} failed: {error_message}")
def increment_retry_count(self, job_id: str) -> int:
"""
Increment retry count for a job.
Returns:
New retry count
"""
job = self.get_job(job_id)
if not job:
return 0
new_count = job.retry_count + 1
self.update_job(job_id, {'retry_count': new_count})
return new_count
def append_worker_log(
self,
job_id: str,
worker: str,
level: str,
message: str,
max_logs: int = 500
) -> None:
"""
Append a log entry to the job's worker_logs atomically.
Uses Firestore ArrayUnion for atomic append to avoid race conditions
when multiple workers (audio + lyrics) are logging concurrently.
Note: max_logs is not enforced per-append (would require transactions).
Logs may grow beyond max_logs; trim periodically if needed.
Args:
job_id: Job ID
worker: Worker name (audio, lyrics, screens, video, render)
level: Log level (DEBUG, INFO, WARNING, ERROR)
message: Log message
max_logs: Not used (kept for API compatibility)
"""
log_entry = {
'timestamp': datetime.utcnow().isoformat() + 'Z',
'level': level,
'worker': worker,
'message': message[:1000] # Truncate long messages
}
# Use atomic ArrayUnion to avoid race conditions
self.firestore.append_worker_log(job_id, log_entry)
def get_worker_logs(
self,
job_id: str,
since_index: int = 0,
worker: Optional[str] = None
) -> List[Dict[str, Any]]:
"""
Get worker logs for a job, optionally filtered by worker and index.
Args:
job_id: Job ID
since_index: Return only logs after this index (for pagination)
worker: Filter by worker name (optional)
Returns:
List of log entries as dicts
"""
job = self.get_job(job_id)
if not job or not job.worker_logs:
return []
logs = [log.dict() if hasattr(log, 'dict') else log for log in job.worker_logs]
# Filter by index
if since_index > 0:
logs = logs[since_index:]
# Filter by worker
if worker:
logs = [log for log in logs if log.get('worker') == worker]
return logs
-413
View File
@@ -1,413 +0,0 @@
"""
Custom metrics for Cloud Monitoring.
This module provides application-level metrics for tracking job processing,
worker performance, and external API usage. Metrics can be viewed in:
1. Cloud Logging (via structured log entries)
2. Cloud Monitoring (via log-based metrics or OpenTelemetry)
The metrics service uses a pragmatic approach:
- Always emits metrics as structured log entries (works immediately)
- Uses the same JSON format as Cloud Logging
- Can be enhanced with OpenTelemetry metrics exporters when available
Usage:
from backend.services.metrics import metrics
# Record a job completion
metrics.record_job_completed("abc123", source="upload")
# Record worker duration
metrics.record_worker_duration("audio", 45.2, success=True)
# Record external API call
with metrics.time_external_api("modal"):
response = await modal_client.separate_audio(...)
"""
import logging
import time
from contextlib import contextmanager
from dataclasses import dataclass, field
from typing import Any, Dict, Optional
from backend.services.tracing import get_current_trace_id, get_current_span_id
logger = logging.getLogger("metrics")
@dataclass
class MetricLabels:
"""Common metric labels."""
job_id: Optional[str] = None
worker: Optional[str] = None
status: Optional[str] = None
source: Optional[str] = None
api: Optional[str] = None
operation: Optional[str] = None
bucket: Optional[str] = None
def to_dict(self) -> Dict[str, Any]:
"""Convert to dict, excluding None values."""
return {k: v for k, v in self.__dict__.items() if v is not None}
class MetricsService:
"""
Application metrics service.
Emits metrics as structured log entries that can be:
1. Queried directly in Cloud Logging
2. Converted to Cloud Monitoring metrics via log-based metrics
3. Exported via OpenTelemetry (when configured)
"""
def __init__(self):
"""Initialize the metrics service."""
self._logger = logging.getLogger("metrics")
# Ensure metrics logger outputs at INFO level
self._logger.setLevel(logging.INFO)
def _emit_metric(self, metric_name: str, metric_type: str, value: float, labels: Dict[str, Any]) -> None:
"""
Emit a metric as a structured log entry.
The log format is designed to be easily parsed by Cloud Logging
and converted to log-based metrics.
Args:
metric_name: Name of the metric (e.g., "jobs_total")
metric_type: Type of metric (counter, histogram, gauge)
value: Metric value
labels: Metric labels/dimensions
"""
# Add trace context if available
trace_id = get_current_trace_id()
span_id = get_current_span_id()
# Build metric entry
metric_entry = {
"metric_name": metric_name,
"metric_type": metric_type,
"metric_value": value,
**labels,
}
if trace_id:
metric_entry["trace_id"] = trace_id
if span_id:
metric_entry["span_id"] = span_id
# Emit as structured log entry
# Use INFO level so metrics always show up
self._logger.info(
f"METRIC {metric_name}={value}",
extra=metric_entry
)
# =========================================
# Job Metrics
# =========================================
def record_job_created(self, job_id: str, source: str = "unknown") -> None:
"""
Record a new job creation.
Args:
job_id: Job ID
source: Job source (upload, url, search)
"""
self._emit_metric(
metric_name="jobs_total",
metric_type="counter",
value=1,
labels={"job_id": job_id, "status": "created", "source": source}
)
def record_job_completed(self, job_id: str, source: str = "unknown") -> None:
"""
Record a job completion.
Args:
job_id: Job ID
source: Job source (upload, url, search)
"""
self._emit_metric(
metric_name="jobs_total",
metric_type="counter",
value=1,
labels={"job_id": job_id, "status": "completed", "source": source}
)
def record_job_failed(self, job_id: str, source: str = "unknown", error: Optional[str] = None) -> None:
"""
Record a job failure.
Args:
job_id: Job ID
source: Job source (upload, url, search)
error: Optional error message
"""
labels = {"job_id": job_id, "status": "failed", "source": source}
if error:
labels["error"] = error[:200] # Truncate long errors
self._emit_metric(
metric_name="jobs_total",
metric_type="counter",
value=1,
labels=labels
)
def record_job_duration(self, job_id: str, duration_seconds: float, source: str = "unknown") -> None:
"""
Record total job processing duration.
Args:
job_id: Job ID
duration_seconds: Total processing time in seconds
source: Job source (upload, url, search)
"""
self._emit_metric(
metric_name="job_duration_seconds",
metric_type="histogram",
value=duration_seconds,
labels={"job_id": job_id, "source": source}
)
# =========================================
# Worker Metrics
# =========================================
def record_worker_started(self, worker: str, job_id: str) -> None:
"""
Record a worker invocation start.
Args:
worker: Worker name (audio, lyrics, screens, video, render_video)
job_id: Job ID
"""
self._emit_metric(
metric_name="worker_invocations_total",
metric_type="counter",
value=1,
labels={"worker": worker, "job_id": job_id, "status": "started"}
)
def record_worker_duration(self, worker: str, duration_seconds: float, success: bool, job_id: Optional[str] = None) -> None:
"""
Record worker processing duration.
Args:
worker: Worker name
duration_seconds: Processing time in seconds
success: Whether worker completed successfully
job_id: Optional job ID
"""
labels = {
"worker": worker,
"success": str(success).lower(),
}
if job_id:
labels["job_id"] = job_id
self._emit_metric(
metric_name="job_stage_duration_seconds",
metric_type="histogram",
value=duration_seconds,
labels=labels
)
# Also emit a counter for success/failure tracking
self._emit_metric(
metric_name="worker_invocations_total",
metric_type="counter",
value=1,
labels={"worker": worker, "success": str(success).lower(), "job_id": job_id or "unknown"}
)
# =========================================
# GCS Metrics
# =========================================
def record_gcs_operation(
self,
operation: str,
bucket: str,
success: bool,
size_bytes: Optional[int] = None,
duration_seconds: Optional[float] = None,
job_id: Optional[str] = None,
) -> None:
"""
Record a GCS operation.
Args:
operation: Operation type (upload, download, delete)
bucket: GCS bucket name
success: Whether operation succeeded
size_bytes: Optional file size in bytes
duration_seconds: Optional operation duration
job_id: Optional job ID
"""
labels = {
"operation": operation,
"bucket": bucket,
"success": str(success).lower(),
}
if job_id:
labels["job_id"] = job_id
self._emit_metric(
metric_name="gcs_operations_total",
metric_type="counter",
value=1,
labels=labels
)
if size_bytes is not None:
self._emit_metric(
metric_name="gcs_operation_bytes",
metric_type="histogram",
value=size_bytes,
labels={**labels, "operation": operation}
)
if duration_seconds is not None:
self._emit_metric(
metric_name="gcs_operation_duration_seconds",
metric_type="histogram",
value=duration_seconds,
labels={**labels, "operation": operation}
)
# =========================================
# External API Metrics
# =========================================
def record_external_api_call(
self,
api: str,
success: bool,
duration_seconds: float,
job_id: Optional[str] = None,
error: Optional[str] = None,
) -> None:
"""
Record an external API call.
Args:
api: API name (modal, audioshake, genius, spotify)
success: Whether call succeeded
duration_seconds: API call duration
job_id: Optional job ID
error: Optional error message
"""
labels = {
"api": api,
"success": str(success).lower(),
}
if job_id:
labels["job_id"] = job_id
if error:
labels["error"] = error[:100] # Truncate
self._emit_metric(
metric_name="external_api_calls_total",
metric_type="counter",
value=1,
labels=labels
)
self._emit_metric(
metric_name="external_api_duration_seconds",
metric_type="histogram",
value=duration_seconds,
labels={"api": api, "success": str(success).lower()}
)
@contextmanager
def time_external_api(self, api: str, job_id: Optional[str] = None):
"""
Context manager to time an external API call.
Usage:
with metrics.time_external_api("modal", job_id) as timer:
response = await client.call_api()
timer.set_success(True)
Args:
api: API name
job_id: Optional job ID
Yields:
Timer object with set_success() method
"""
timer = _ApiTimer()
start_time = time.time()
try:
yield timer
except Exception as e:
timer.set_success(False)
timer.error = str(e)
raise
finally:
duration = time.time() - start_time
self.record_external_api_call(
api=api,
success=timer.success,
duration_seconds=duration,
job_id=job_id,
error=timer.error,
)
@contextmanager
def time_worker(self, worker: str, job_id: str):
"""
Context manager to time a worker execution.
Usage:
with metrics.time_worker("audio", job_id) as timer:
await process_audio()
timer.set_success(True)
Args:
worker: Worker name
job_id: Job ID
Yields:
Timer object with set_success() method
"""
timer = _ApiTimer()
self.record_worker_started(worker, job_id)
start_time = time.time()
try:
yield timer
except Exception as e:
timer.set_success(False)
raise
finally:
duration = time.time() - start_time
self.record_worker_duration(
worker=worker,
duration_seconds=duration,
success=timer.success,
job_id=job_id,
)
class _ApiTimer:
"""Helper class for tracking API call success state."""
def __init__(self):
self.success = True # Assume success unless set otherwise
self.error: Optional[str] = None
def set_success(self, success: bool) -> None:
self.success = success
# Global metrics instance
metrics = MetricsService()
-106
View File
@@ -1,106 +0,0 @@
"""
Rclone configuration service for cloud storage integration.
This service manages the rclone configuration needed for Dropbox
and other cloud storage uploads from the backend workers.
"""
import logging
import os
import tempfile
from typing import Optional
from backend.config import get_settings
logger = logging.getLogger(__name__)
class RcloneService:
"""Service for managing rclone configuration."""
# Secret Manager secret name for rclone config
RCLONE_CONFIG_SECRET = "rclone-config"
def __init__(self):
self.settings = get_settings()
self._config_file: Optional[str] = None
self._config_loaded = False
def setup_rclone_config(self) -> bool:
"""
Load rclone config from Secret Manager and set up environment.
Writes the config to a temp file and sets RCLONE_CONFIG env var.
Returns:
True if successful, False otherwise
"""
if self._config_loaded:
logger.debug("Rclone config already loaded")
return True
try:
# Get rclone config from Secret Manager
config_content = self.settings.get_secret(self.RCLONE_CONFIG_SECRET)
if not config_content:
logger.warning("Rclone config not found in Secret Manager")
return False
# Write to a temp file
fd, config_path = tempfile.mkstemp(prefix="rclone_", suffix=".conf")
try:
with os.fdopen(fd, 'w') as f:
f.write(config_content)
self._config_file = config_path
# Set environment variable for rclone to find the config
os.environ["RCLONE_CONFIG"] = config_path
logger.info(f"Rclone config loaded and written to {config_path}")
self._config_loaded = True
return True
except Exception:
# Clean up the temp file on error
# Note: os.fdopen() takes ownership of fd, so it's already closed
# We only need to remove the temp file if it exists
if os.path.exists(config_path):
os.unlink(config_path)
raise
except Exception as e:
logger.error(f"Failed to setup rclone config: {e}")
return False
def cleanup(self) -> None:
"""Remove the temporary config file."""
if self._config_file and os.path.exists(self._config_file):
try:
os.unlink(self._config_file)
logger.debug(f"Cleaned up rclone config file: {self._config_file}")
except Exception as e:
logger.warning(f"Failed to cleanup rclone config: {e}")
# Always reset internal state and environment, even if the file was missing
if self._config_file is not None:
os.environ.pop("RCLONE_CONFIG", None)
self._config_file = None
self._config_loaded = False
@property
def is_configured(self) -> bool:
"""Check if rclone is configured and ready to use."""
return self._config_loaded and self._config_file is not None
# Singleton instance
_rclone_service: Optional[RcloneService] = None
def get_rclone_service() -> RcloneService:
"""Get the singleton rclone service instance."""
global _rclone_service
if _rclone_service is None:
_rclone_service = RcloneService()
return _rclone_service
-209
View File
@@ -1,209 +0,0 @@
"""
Google Cloud Storage operations for file management.
"""
import logging
import os
import json
from typing import Optional, BinaryIO, Any, Dict
from pathlib import Path
from google.cloud import storage
from datetime import timedelta
from backend.config import settings
logger = logging.getLogger(__name__)
class StorageService:
"""Service for Google Cloud Storage operations."""
def __init__(self):
"""Initialize GCS client."""
self.client = storage.Client(project=settings.google_cloud_project)
self.bucket = self.client.bucket(settings.gcs_bucket_name)
def upload_file(self, local_path: str, destination_path: str) -> str:
"""Upload a file to GCS."""
try:
blob = self.bucket.blob(destination_path)
blob.upload_from_filename(local_path)
logger.info(f"Uploaded {local_path} to gs://{settings.gcs_bucket_name}/{destination_path}")
return destination_path
except Exception as e:
logger.error(f"Error uploading file {local_path}: {e}")
raise
def upload_fileobj(self, file_obj: BinaryIO, destination_path: str, content_type: Optional[str] = None) -> str:
"""Upload a file object to GCS."""
try:
blob = self.bucket.blob(destination_path)
if content_type:
blob.content_type = content_type
blob.upload_from_file(file_obj, rewind=True)
logger.info(f"Uploaded file object to gs://{settings.gcs_bucket_name}/{destination_path}")
return destination_path
except Exception as e:
logger.error(f"Error uploading file object: {e}")
raise
def download_file(self, source_path: str, destination_path: str) -> str:
"""Download a file from GCS."""
try:
blob = self.bucket.blob(source_path)
blob.download_to_filename(destination_path)
logger.info(f"Downloaded gs://{settings.gcs_bucket_name}/{source_path} to {destination_path}")
return destination_path
except Exception as e:
logger.error(f"Error downloading file {source_path}: {e}")
raise
def generate_signed_url(self, blob_path: str, expiration_minutes: int = 60) -> str:
"""Generate a signed URL for downloading a file.
In Cloud Run, this uses the IAM signBlob API since we don't have
a private key available. Requires the service account to have
roles/iam.serviceAccountTokenCreator on itself.
"""
return self._generate_signed_url_internal(blob_path, "GET", expiration_minutes)
def generate_signed_upload_url(self, blob_path: str, content_type: str = "application/octet-stream", expiration_minutes: int = 60) -> str:
"""Generate a signed URL for uploading a file directly to GCS.
This allows clients to upload files directly to GCS without going through
the backend, bypassing any request body size limits.
Args:
blob_path: The destination path in GCS
content_type: The expected content type of the upload
expiration_minutes: How long the URL is valid for
Returns:
A signed URL that accepts PUT requests with the file content
"""
return self._generate_signed_url_internal(blob_path, "PUT", expiration_minutes, content_type)
def _generate_signed_url_internal(self, blob_path: str, method: str, expiration_minutes: int = 60, content_type: Optional[str] = None) -> str:
"""Internal method to generate signed URLs for GET or PUT operations."""
import google.auth
from google.auth.transport import requests
try:
blob = self.bucket.blob(blob_path)
# Get default credentials and refresh to ensure we have a valid token
credentials, project = google.auth.default()
# Common kwargs for signed URL generation
kwargs = {
"version": "v4",
"expiration": timedelta(minutes=expiration_minutes),
"method": method,
}
# For PUT requests, we need to specify the content type in headers
if method == "PUT" and content_type:
kwargs["headers"] = {"Content-Type": content_type}
# Check if we're using compute credentials (Cloud Run/GCE)
# These need to use IAM signBlob via service_account_email + access_token
if hasattr(credentials, 'service_account_email'):
# Refresh credentials to get a valid access token
auth_request = requests.Request()
credentials.refresh(auth_request)
kwargs["service_account_email"] = credentials.service_account_email
kwargs["access_token"] = credentials.token
url = blob.generate_signed_url(**kwargs)
logger.info(f"Generated signed {method} URL for {blob_path}")
return url
except Exception as e:
logger.error(f"Error generating signed {method} URL for {blob_path}: {e}")
raise
def delete_file(self, blob_path: str) -> None:
"""Delete a file from GCS."""
try:
blob = self.bucket.blob(blob_path)
blob.delete()
logger.info(f"Deleted gs://{settings.gcs_bucket_name}/{blob_path}")
except Exception as e:
logger.error(f"Error deleting file {blob_path}: {e}")
raise
def delete_folder(self, prefix: str) -> int:
"""
Delete all files in GCS with a given prefix (folder).
Args:
prefix: The folder prefix to delete (e.g., "uploads/abc123/")
Returns:
Number of files deleted
"""
try:
blobs = list(self.bucket.list_blobs(prefix=prefix))
deleted_count = 0
for blob in blobs:
try:
blob.delete()
deleted_count += 1
except Exception as e:
logger.warning(f"Error deleting blob {blob.name}: {e}")
if deleted_count > 0:
logger.info(f"Deleted {deleted_count} files from gs://{settings.gcs_bucket_name}/{prefix}")
return deleted_count
except Exception as e:
logger.error(f"Error deleting folder {prefix}: {e}")
return 0 # Don't raise - folder deletion shouldn't break operations
def list_files(self, prefix: str) -> list:
"""List files in GCS with a given prefix."""
try:
blobs = self.bucket.list_blobs(prefix=prefix)
return [blob.name for blob in blobs]
except Exception as e:
logger.error(f"Error listing files with prefix {prefix}: {e}")
raise
def file_exists(self, blob_path: str) -> bool:
"""Check if a file exists in GCS."""
try:
blob = self.bucket.blob(blob_path)
return blob.exists()
except Exception as e:
logger.error(f"Error checking file existence {blob_path}: {e}")
raise
def upload_json(self, destination_path: str, data: Dict[str, Any]) -> str:
"""Upload a JSON object to GCS."""
try:
blob = self.bucket.blob(destination_path)
blob.content_type = "application/json"
blob.upload_from_string(
json.dumps(data, indent=2, ensure_ascii=False),
content_type="application/json"
)
logger.info(f"Uploaded JSON to gs://{settings.gcs_bucket_name}/{destination_path}")
return destination_path
except Exception as e:
logger.error(f"Error uploading JSON to {destination_path}: {e}")
raise
def download_json(self, source_path: str) -> Dict[str, Any]:
"""Download and parse a JSON file from GCS."""
try:
blob = self.bucket.blob(source_path)
content = blob.download_as_text()
data = json.loads(content)
logger.info(f"Downloaded JSON from gs://{settings.gcs_bucket_name}/{source_path}")
return data
except Exception as e:
logger.error(f"Error downloading JSON from {source_path}: {e}")
raise
-246
View File
@@ -1,246 +0,0 @@
"""
Structured logging with trace correlation for Cloud Logging.
This module provides JSON-formatted logging that integrates with Google Cloud Logging
and correlates logs with OpenTelemetry traces in Cloud Trace.
When running in Cloud Run:
- Logs are output as JSON for Cloud Logging to parse
- Each log entry includes trace ID and span ID for correlation
- Custom fields (job_id, worker) are preserved in the log structure
Usage:
# In main.py, before any logging:
from backend.services.structured_logging import setup_structured_logging
setup_structured_logging()
# Then use standard logging:
logger = logging.getLogger(__name__)
logger.info("Processing started", extra={"job_id": "abc123"})
"""
import json
import logging
import os
import sys
from datetime import datetime
from typing import Any, Dict, Optional
# Cloud Logging severity mapping (compatible with GCP)
# https://cloud.google.com/logging/docs/reference/v2/rest/v2/LogEntry#LogSeverity
SEVERITY_MAP = {
"DEBUG": "DEBUG",
"INFO": "INFO",
"WARNING": "WARNING",
"ERROR": "ERROR",
"CRITICAL": "CRITICAL",
}
class StructuredFormatter(logging.Formatter):
"""
JSON formatter with trace correlation for Google Cloud Logging.
Produces log entries in the format expected by Cloud Logging, including:
- logging.googleapis.com/trace: Links to Cloud Trace
- logging.googleapis.com/spanId: Current span ID
- severity: Cloud Logging severity level
- Custom fields passed via 'extra' dict
"""
def __init__(self, project_id: Optional[str] = None):
"""
Initialize the structured formatter.
Args:
project_id: GCP project ID for trace URLs (auto-detected if not provided)
"""
super().__init__()
self.project_id = project_id or os.environ.get("GOOGLE_CLOUD_PROJECT") or os.environ.get("GCP_PROJECT")
def format(self, record: logging.LogRecord) -> str:
"""
Format a log record as JSON.
Args:
record: The log record to format
Returns:
JSON string for Cloud Logging
"""
from backend.services.tracing import get_current_trace_id, get_current_span_id
# Build base log entry
log_entry: Dict[str, Any] = {
"timestamp": datetime.utcfromtimestamp(record.created).isoformat() + "Z",
"severity": SEVERITY_MAP.get(record.levelname, record.levelname),
"message": record.getMessage(),
"logger": record.name,
}
# Add trace correlation if available
trace_id = get_current_trace_id()
span_id = get_current_span_id()
if trace_id and self.project_id:
# Format for Cloud Logging trace correlation
log_entry["logging.googleapis.com/trace"] = f"projects/{self.project_id}/traces/{trace_id}"
if span_id:
log_entry["logging.googleapis.com/spanId"] = span_id
# Add custom fields from 'extra' dict
# Common fields we want to extract from log records
custom_fields = ["job_id", "worker", "operation", "duration", "status", "error"]
for field in custom_fields:
value = getattr(record, field, None)
if value is not None:
log_entry[field] = value
# Add source location for debugging
if record.pathname and record.lineno:
log_entry["logging.googleapis.com/sourceLocation"] = {
"file": record.pathname,
"line": record.lineno,
"function": record.funcName,
}
# Add exception info if present
if record.exc_info:
log_entry["exception"] = self.formatException(record.exc_info)
# Remove None values for cleaner output
return json.dumps({k: v for k, v in log_entry.items() if v is not None})
class HumanReadableFormatter(logging.Formatter):
"""
Human-readable formatter for local development.
Includes trace context when available but outputs in traditional format.
"""
def format(self, record: logging.LogRecord) -> str:
"""Format a log record for human readability."""
from backend.services.tracing import get_current_trace_id
# Build base message
timestamp = datetime.utcfromtimestamp(record.created).strftime("%Y-%m-%d %H:%M:%S")
# Add job_id prefix if available
job_id = getattr(record, "job_id", None)
job_prefix = f"[job:{job_id[:8]}] " if job_id else ""
# Add trace ID suffix if available
trace_id = get_current_trace_id()
trace_suffix = f" [trace:{trace_id[:8]}]" if trace_id else ""
message = f"{timestamp} - {record.name} - {record.levelname} - {job_prefix}{record.getMessage()}{trace_suffix}"
# Add exception info if present
if record.exc_info:
message += "\n" + self.formatException(record.exc_info)
return message
def is_cloud_run() -> bool:
"""Check if we're running in Cloud Run."""
return os.environ.get("K_SERVICE") is not None
def setup_structured_logging(force_json: bool = False, log_level: Optional[str] = None) -> None:
"""
Configure structured logging for the application.
In Cloud Run: Uses JSON format with trace correlation for Cloud Logging
In development: Uses human-readable format with optional trace context
Args:
force_json: Force JSON output even in development
log_level: Override log level (default from settings or INFO)
"""
from backend.config import settings
# Determine log level
level = log_level or getattr(settings, "log_level", "INFO")
level = getattr(logging, level.upper(), logging.INFO)
# Choose formatter based on environment
if is_cloud_run() or force_json:
formatter = StructuredFormatter()
else:
formatter = HumanReadableFormatter()
# Create handler
handler = logging.StreamHandler(sys.stdout)
handler.setFormatter(formatter)
handler.setLevel(level)
# Configure root logger
root_logger = logging.getLogger()
root_logger.setLevel(level)
# Remove existing handlers and add our handler
root_logger.handlers = [handler]
# Also configure uvicorn loggers to use our format
for logger_name in ["uvicorn", "uvicorn.access", "uvicorn.error"]:
logger = logging.getLogger(logger_name)
logger.handlers = [handler]
logger.propagate = False
class JobLogAdapter(logging.LoggerAdapter):
"""
Logger adapter that automatically adds job context to log records.
Usage:
logger = logging.getLogger(__name__)
job_logger = JobLogAdapter(logger, job_id="abc123", worker="audio")
job_logger.info("Processing started") # Automatically includes job_id and worker
"""
def __init__(self, logger: logging.Logger, job_id: str, worker: Optional[str] = None, **extra):
"""
Initialize the adapter.
Args:
logger: Base logger instance
job_id: Job ID to include in all log records
worker: Optional worker name
**extra: Additional fields to include in all log records
"""
super().__init__(logger, {"job_id": job_id, "worker": worker, **extra})
def process(self, msg: str, kwargs: Dict[str, Any]) -> tuple:
"""Process a logging call to add extra context."""
# Merge extra dict from adapter with any extra passed to the log call
extra = kwargs.get("extra", {})
extra.update(self.extra)
kwargs["extra"] = extra
return msg, kwargs
def get_job_logger(job_id: str, worker: Optional[str] = None, name: Optional[str] = None) -> JobLogAdapter:
"""
Get a logger adapter configured for a specific job.
This is a convenience function for creating job-specific loggers.
Args:
job_id: Job ID to include in all log records
worker: Optional worker name
name: Logger name (defaults to "job")
Returns:
JobLogAdapter configured for the job
Usage:
logger = get_job_logger("abc123", "audio")
logger.info("Starting audio separation")
logger.error("Failed to process", extra={"error": str(e)})
"""
base_logger = logging.getLogger(name or "job")
return JobLogAdapter(base_logger, job_id=job_id, worker=worker)
-469
View File
@@ -1,469 +0,0 @@
"""
Theme service for managing pre-created style themes.
Themes are stored in GCS at themes/{theme_id}/ with:
- style_params.json: Complete style configuration
- preview.png: Preview image for theme selection UI
- youtube_description.txt: Optional YouTube description template
- assets/: Fonts, backgrounds, and other assets
The service provides:
- Theme listing with signed preview URLs
- Theme detail retrieval
- Color override application
- Job style preparation (copying theme to job folder)
"""
import copy
import json
import logging
import os
import tempfile
import threading
from datetime import datetime, timedelta
from typing import Any, Dict, List, Optional, Tuple
from backend.models.theme import (
ColorOverrides,
ThemeDetail,
ThemeMetadata,
ThemeRegistry,
ThemeSummary,
hex_to_rgba,
)
from backend.services.storage_service import StorageService
logger = logging.getLogger(__name__)
# GCS paths for themes
THEMES_PREFIX = "themes"
METADATA_FILE = f"{THEMES_PREFIX}/_metadata.json"
class ThemeService:
"""Service for managing themes from GCS."""
def __init__(self, storage: Optional[StorageService] = None):
"""
Initialize the theme service.
Args:
storage: StorageService instance (creates new one if not provided)
"""
self.storage = storage or StorageService()
self._metadata_cache: Optional[ThemeRegistry] = None
self._cache_time: Optional[datetime] = None
self.CACHE_TTL_SECONDS = 300 # 5 minute cache for theme metadata
def _get_theme_path(self, theme_id: str, filename: str = "") -> str:
"""Get the GCS path for a theme file."""
if filename:
return f"{THEMES_PREFIX}/{theme_id}/{filename}"
return f"{THEMES_PREFIX}/{theme_id}"
def _is_cache_valid(self) -> bool:
"""Check if the metadata cache is still valid."""
if self._metadata_cache is None or self._cache_time is None:
return False
age = datetime.now() - self._cache_time
return age.total_seconds() < self.CACHE_TTL_SECONDS
def _load_metadata(self, force_refresh: bool = False) -> ThemeRegistry:
"""
Load theme metadata from GCS with caching.
Args:
force_refresh: Force reload from GCS even if cache is valid
Returns:
ThemeRegistry containing all theme metadata
"""
if not force_refresh and self._is_cache_valid():
return self._metadata_cache # type: ignore
try:
data = self.storage.download_json(METADATA_FILE)
self._metadata_cache = ThemeRegistry(**data)
self._cache_time = datetime.now()
logger.info(f"Loaded theme metadata: {len(self._metadata_cache.themes)} themes")
return self._metadata_cache
except Exception as e:
logger.error(f"Failed to load theme metadata: {e}")
# Return empty registry on error
return ThemeRegistry(version=1, themes=[])
def list_themes(self) -> List[ThemeSummary]:
"""
List all available themes with signed preview URLs.
Returns:
List of ThemeSummary objects with preview URLs
"""
metadata = self._load_metadata()
summaries = []
for theme in metadata.themes:
try:
# Generate signed URLs for preview images
preview_path = self._get_theme_path(theme.id, "preview.png")
thumbnail_path = self._get_theme_path(theme.id, "preview_thumbnail.png")
preview_url = None
thumbnail_url = None
if self.storage.file_exists(preview_path):
preview_url = self.storage.generate_signed_url(preview_path, expiration_minutes=60)
# Try thumbnail, fall back to main preview
if self.storage.file_exists(thumbnail_path):
thumbnail_url = self.storage.generate_signed_url(thumbnail_path, expiration_minutes=60)
elif preview_url:
thumbnail_url = preview_url
summaries.append(
ThemeSummary(
id=theme.id,
name=theme.name,
description=theme.description,
preview_url=preview_url,
thumbnail_url=thumbnail_url,
is_default=theme.is_default,
)
)
except Exception as e:
logger.warning(f"Error processing theme {theme.id}: {e}")
# Still include theme but without preview URLs
summaries.append(
ThemeSummary(
id=theme.id,
name=theme.name,
description=theme.description,
is_default=theme.is_default,
)
)
return summaries
def get_theme(self, theme_id: str) -> Optional[ThemeDetail]:
"""
Get full theme details including style parameters.
Args:
theme_id: The theme identifier
Returns:
ThemeDetail if found, None otherwise
"""
metadata = self._load_metadata()
# Find theme in metadata
theme_meta = next((t for t in metadata.themes if t.id == theme_id), None)
if not theme_meta:
logger.warning(f"Theme not found: {theme_id}")
return None
try:
# Load style_params.json
style_params_path = self._get_theme_path(theme_id, "style_params.json")
style_params = self.storage.download_json(style_params_path)
# Check for YouTube description
youtube_desc_path = self._get_theme_path(theme_id, "youtube_description.txt")
has_youtube_desc = self.storage.file_exists(youtube_desc_path)
# Generate preview URL
preview_path = self._get_theme_path(theme_id, "preview.png")
preview_url = None
if self.storage.file_exists(preview_path):
preview_url = self.storage.generate_signed_url(preview_path, expiration_minutes=60)
return ThemeDetail(
id=theme_id,
name=theme_meta.name,
description=theme_meta.description,
preview_url=preview_url,
is_default=theme_meta.is_default,
style_params=style_params,
has_youtube_description=has_youtube_desc,
)
except Exception as e:
logger.error(f"Error loading theme {theme_id}: {e}")
return None
def get_theme_style_params(self, theme_id: str) -> Optional[Dict[str, Any]]:
"""
Get just the style_params.json for a theme.
Args:
theme_id: The theme identifier
Returns:
Style parameters dict if found, None otherwise
"""
try:
style_params_path = self._get_theme_path(theme_id, "style_params.json")
return self.storage.download_json(style_params_path)
except Exception as e:
logger.error(f"Error loading style params for theme {theme_id}: {e}")
return None
def get_youtube_description(self, theme_id: str) -> Optional[str]:
"""
Get the YouTube description template for a theme.
Args:
theme_id: The theme identifier
Returns:
YouTube description text if found, None otherwise
"""
try:
youtube_desc_path = self._get_theme_path(theme_id, "youtube_description.txt")
if not self.storage.file_exists(youtube_desc_path):
return None
# Download to temp file and read
with tempfile.NamedTemporaryFile(mode="w", delete=False, suffix=".txt") as tmp:
tmp_path = tmp.name
try:
self.storage.download_file(youtube_desc_path, tmp_path)
with open(tmp_path, "r") as f:
return f.read()
finally:
if os.path.exists(tmp_path):
os.remove(tmp_path)
except Exception as e:
logger.error(f"Error loading YouTube description for theme {theme_id}: {e}")
return None
def apply_color_overrides(
self, style_params: Dict[str, Any], overrides: ColorOverrides
) -> Dict[str, Any]:
"""
Apply user color overrides to theme style parameters.
Args:
style_params: Base style parameters from theme
overrides: User color overrides
Returns:
Modified style parameters with overrides applied
"""
if not overrides.has_overrides():
return style_params
# Deep copy to avoid modifying original
result = copy.deepcopy(style_params)
# Apply artist_color to intro, end, and CDG sections
if overrides.artist_color:
if "intro" in result:
result["intro"]["artist_color"] = overrides.artist_color
if "end" in result:
result["end"]["artist_color"] = overrides.artist_color
if "cdg" in result:
result["cdg"]["artist_color"] = overrides.artist_color
# Apply title_color to intro, end, and CDG sections
if overrides.title_color:
if "intro" in result:
result["intro"]["title_color"] = overrides.title_color
if "end" in result:
result["end"]["title_color"] = overrides.title_color
if "cdg" in result:
result["cdg"]["title_color"] = overrides.title_color
# Apply sung_lyrics_color (karaoke uses RGBA, CDG uses hex)
if overrides.sung_lyrics_color:
if "karaoke" in result:
result["karaoke"]["primary_color"] = hex_to_rgba(overrides.sung_lyrics_color)
if "cdg" in result:
result["cdg"]["active_fill"] = overrides.sung_lyrics_color
# Apply unsung_lyrics_color (karaoke uses RGBA, CDG uses hex)
if overrides.unsung_lyrics_color:
if "karaoke" in result:
result["karaoke"]["secondary_color"] = hex_to_rgba(overrides.unsung_lyrics_color)
if "cdg" in result:
result["cdg"]["inactive_fill"] = overrides.unsung_lyrics_color
return result
def prepare_job_style(
self,
job_id: str,
theme_id: str,
color_overrides: Optional[ColorOverrides] = None,
) -> Tuple[str, Dict[str, str]]:
"""
Prepare style files for a job by copying theme to job folder.
This method:
1. Loads the theme's style_params.json
2. Applies any color overrides
3. Updates asset paths to point to theme assets (shared, not copied)
4. Uploads modified style_params.json to job folder
5. Returns the GCS path and style_assets mapping
Args:
job_id: The job ID
theme_id: The theme to use
color_overrides: Optional color overrides
Returns:
Tuple of (style_params_gcs_path, style_assets dict)
Raises:
ValueError: If theme not found
"""
# Load theme style params
style_params = self.get_theme_style_params(theme_id)
if style_params is None:
raise ValueError(f"Theme not found: {theme_id}")
# Apply color overrides if provided
if color_overrides:
style_params = self.apply_color_overrides(style_params, color_overrides)
# Update asset paths to point to theme's shared assets
# Theme assets stay in themes/{theme_id}/assets/ - they're shared
style_assets = self._build_style_assets_mapping(theme_id, style_params)
# Update style_params to use theme asset paths (not job-specific)
style_params = self._update_asset_paths_in_style(theme_id, style_params)
# Upload modified style_params.json to job's style folder
job_style_path = f"uploads/{job_id}/style/style_params.json"
self.storage.upload_json(job_style_path, style_params)
logger.info(f"Prepared job {job_id} style from theme {theme_id}")
return job_style_path, style_assets
def _build_style_assets_mapping(
self, theme_id: str, style_params: Dict[str, Any]
) -> Dict[str, str]:
"""
Build the style_assets mapping for a theme.
Maps asset keys to their GCS paths in the theme folder.
"""
style_assets = {}
theme_assets_prefix = f"{THEMES_PREFIX}/{theme_id}/assets"
# Check each potential asset type
asset_checks = [
("intro_background", "intro", "background_image"),
("karaoke_background", "karaoke", "background_image"),
("end_background", "end", "background_image"),
("font", "intro", "font"), # Font is typically shared across sections
("cdg_instrumental_background", "cdg", "instrumental_background"),
("cdg_title_background", "cdg", "title_screen_background"),
("cdg_outro_background", "cdg", "outro_background"),
]
for asset_key, section, field in asset_checks:
if section in style_params and field in style_params[section]:
value = style_params[section][field]
if value and isinstance(value, str):
# Check if it's a theme asset path or already has the full path
if value.startswith(f"{THEMES_PREFIX}/"):
# Already a full theme path
style_assets[asset_key] = value
elif not value.startswith("/") and not value.startswith("gs://"):
# Relative path - prepend theme assets prefix
asset_path = f"{theme_assets_prefix}/{os.path.basename(value)}"
if self.storage.file_exists(asset_path):
style_assets[asset_key] = asset_path
return style_assets
def _update_asset_paths_in_style(
self, theme_id: str, style_params: Dict[str, Any]
) -> Dict[str, Any]:
"""
Update asset paths in style_params to use full GCS theme paths.
This ensures the style_loader can find assets when processing.
"""
result = copy.deepcopy(style_params)
theme_assets_prefix = f"{THEMES_PREFIX}/{theme_id}/assets"
# Fields that contain asset paths
path_fields = {
"intro": ["background_image", "font"],
"end": ["background_image", "font"],
"karaoke": ["background_image", "font_path"],
"cdg": [
"font_path",
"instrumental_background",
"title_screen_background",
"outro_background",
],
}
for section, fields in path_fields.items():
if section not in result:
continue
for field in fields:
if field not in result[section]:
continue
value = result[section][field]
if value and isinstance(value, str):
# Skip if already a full path or URL
if value.startswith(f"{THEMES_PREFIX}/") or value.startswith("gs://"):
continue
# Skip absolute local paths (shouldn't happen but be safe)
if value.startswith("/"):
continue
# Update to full theme asset path
result[section][field] = f"{theme_assets_prefix}/{os.path.basename(value)}"
return result
def theme_exists(self, theme_id: str) -> bool:
"""
Check if a theme exists.
Args:
theme_id: The theme identifier
Returns:
True if theme exists, False otherwise
"""
metadata = self._load_metadata()
return any(t.id == theme_id for t in metadata.themes)
def get_default_theme_id(self) -> Optional[str]:
"""
Get the ID of the default theme.
Returns:
Default theme ID if one exists, None otherwise
"""
metadata = self._load_metadata()
default_theme = next((t for t in metadata.themes if t.is_default), None)
return default_theme.id if default_theme else None
def invalidate_cache(self) -> None:
"""Force invalidation of the metadata cache."""
self._metadata_cache = None
self._cache_time = None
logger.info("Theme metadata cache invalidated")
# Singleton instance with thread-safe initialization
_theme_service: Optional[ThemeService] = None
_theme_service_lock = threading.Lock()
def get_theme_service() -> ThemeService:
"""Get or create the singleton ThemeService instance (thread-safe)."""
global _theme_service
if _theme_service is None:
with _theme_service_lock:
# Double-check after acquiring lock
if _theme_service is None:
_theme_service = ThemeService()
return _theme_service
-543
View File
@@ -1,543 +0,0 @@
"""
OpenTelemetry tracing configuration for Google Cloud Trace.
This module sets up distributed tracing that exports to Google Cloud Trace,
providing visibility into request flows, latency, and errors across the
entire backend.
Usage:
# In main.py or app initialization
from backend.services.tracing import setup_tracing, instrument_app
setup_tracing(service_name="karaoke-backend")
app = FastAPI()
instrument_app(app)
# For custom spans in your code
from backend.services.tracing import tracer, create_span
with tracer.start_as_current_span("my-operation") as span:
span.set_attribute("job_id", job_id)
# ... do work ...
# For job-specific spans with automatic job_id attribute
from backend.services.tracing import job_span
with job_span("audio-worker", job_id) as span:
# job_id is automatically set as an attribute
span.set_attribute("stage", "separation")
# ... do work ...
# For propagating trace context through Cloud Tasks
from backend.services.tracing import inject_trace_context, extract_trace_context
# In worker_service.py (when creating task):
headers = inject_trace_context({})
# In internal.py (when receiving task):
context = extract_trace_context(request.headers)
with tracer.start_as_current_span("worker", context=context):
...
"""
import os
import logging
from typing import Optional, Any, Dict
from contextlib import contextmanager
from functools import wraps
# OpenTelemetry imports
from opentelemetry import trace
from opentelemetry.sdk.trace import TracerProvider
from opentelemetry.sdk.trace.export import BatchSpanProcessor, SimpleSpanProcessor
from opentelemetry.sdk.resources import Resource, SERVICE_NAME, SERVICE_VERSION
from opentelemetry.trace import Status, StatusCode, Span
from opentelemetry.trace.propagation.tracecontext import TraceContextTextMapPropagator
# Google Cloud Trace exporter
try:
from opentelemetry.exporter.cloud_trace import CloudTraceSpanExporter
from opentelemetry.resourcedetector.gcp_resource_detector import GoogleCloudResourceDetector
GCP_AVAILABLE = True
except ImportError:
GCP_AVAILABLE = False
CloudTraceSpanExporter = None
GoogleCloudResourceDetector = None
# FastAPI instrumentation
try:
from opentelemetry.instrumentation.fastapi import FastAPIInstrumentor
FASTAPI_INSTRUMENTATION_AVAILABLE = True
except ImportError:
FASTAPI_INSTRUMENTATION_AVAILABLE = False
FastAPIInstrumentor = None
# HTTPX instrumentation (for outgoing HTTP calls)
try:
from opentelemetry.instrumentation.httpx import HTTPXClientInstrumentation
HTTPX_INSTRUMENTATION_AVAILABLE = True
except ImportError:
HTTPX_INSTRUMENTATION_AVAILABLE = False
HTTPXClientInstrumentation = None
# Logging instrumentation
try:
from opentelemetry.instrumentation.logging import LoggingInstrumentor
LOGGING_INSTRUMENTATION_AVAILABLE = True
except ImportError:
LOGGING_INSTRUMENTATION_AVAILABLE = False
LoggingInstrumentor = None
logger = logging.getLogger(__name__)
# Global tracer instance
_tracer: Optional[trace.Tracer] = None
_initialized = False
def setup_tracing(
service_name: str = "karaoke-backend",
service_version: str = "0.71.7",
enable_in_dev: bool = False,
) -> bool:
"""
Initialize OpenTelemetry tracing with Google Cloud Trace exporter.
Args:
service_name: Name of the service (appears in Cloud Trace)
service_version: Version of the service
enable_in_dev: Whether to enable tracing in development (default False)
Returns:
True if tracing was initialized, False if skipped
"""
global _tracer, _initialized
if _initialized:
logger.debug("Tracing already initialized")
return True
# Check if we should enable tracing
# In Cloud Run, GOOGLE_CLOUD_PROJECT is always set
is_cloud_run = os.environ.get("K_SERVICE") is not None
gcp_project = os.environ.get("GOOGLE_CLOUD_PROJECT") or os.environ.get("GCP_PROJECT")
if not is_cloud_run and not enable_in_dev:
logger.info("Tracing disabled in development (set enable_in_dev=True to enable)")
_tracer = trace.get_tracer(service_name)
return False
if not GCP_AVAILABLE:
logger.warning("Google Cloud Trace exporter not available, tracing disabled")
_tracer = trace.get_tracer(service_name)
return False
logger.info(f"Initializing OpenTelemetry tracing for {service_name} v{service_version}")
try:
# Create resource with service info and GCP resource detection
resource_attributes = {
SERVICE_NAME: service_name,
SERVICE_VERSION: service_version,
}
# Detect GCP resources (project, region, instance, etc.)
if GoogleCloudResourceDetector and is_cloud_run:
try:
gcp_resource = GoogleCloudResourceDetector().detect()
resource = Resource.create(resource_attributes).merge(gcp_resource)
logger.info("Detected GCP resources for tracing")
except Exception as e:
logger.warning(f"Could not detect GCP resources: {e}")
resource = Resource.create(resource_attributes)
else:
resource = Resource.create(resource_attributes)
# Create tracer provider
provider = TracerProvider(resource=resource)
# Add Cloud Trace exporter
if CloudTraceSpanExporter and gcp_project:
try:
exporter = CloudTraceSpanExporter(project_id=gcp_project)
# Use BatchSpanProcessor for production (better performance)
processor = BatchSpanProcessor(exporter)
provider.add_span_processor(processor)
logger.info(f"Cloud Trace exporter configured for project: {gcp_project}")
except Exception as e:
logger.warning(f"Could not configure Cloud Trace exporter: {e}")
# Set the global tracer provider
trace.set_tracer_provider(provider)
# Get tracer instance
_tracer = trace.get_tracer(service_name, service_version)
# Instrument logging to include trace context
if LOGGING_INSTRUMENTATION_AVAILABLE and LoggingInstrumentor:
try:
LoggingInstrumentor().instrument(set_logging_format=True)
logger.info("Logging instrumentation enabled")
except Exception as e:
logger.warning(f"Could not instrument logging: {e}")
# Instrument HTTPX for outgoing requests
if HTTPX_INSTRUMENTATION_AVAILABLE and HTTPXClientInstrumentation:
try:
HTTPXClientInstrumentation().instrument()
logger.info("HTTPX instrumentation enabled")
except Exception as e:
logger.warning(f"Could not instrument HTTPX: {e}")
_initialized = True
logger.info("OpenTelemetry tracing initialized successfully")
return True
except Exception as e:
logger.error(f"Failed to initialize tracing: {e}")
_tracer = trace.get_tracer(service_name)
return False
def instrument_app(app) -> None:
"""
Instrument a FastAPI application with automatic tracing.
This adds spans for all incoming requests, including:
- Request method and path
- Response status code
- Request duration
- Error information
Args:
app: FastAPI application instance
"""
if not FASTAPI_INSTRUMENTATION_AVAILABLE or not FastAPIInstrumentor:
logger.warning("FastAPI instrumentation not available")
return
try:
FastAPIInstrumentor.instrument_app(
app,
excluded_urls="health,healthz,ready,readiness,ping",
)
logger.info("FastAPI instrumentation enabled")
except Exception as e:
logger.warning(f"Could not instrument FastAPI: {e}")
def get_tracer() -> trace.Tracer:
"""
Get the global tracer instance.
Returns:
Tracer instance (creates a no-op tracer if not initialized)
"""
global _tracer
if _tracer is None:
_tracer = trace.get_tracer("karaoke-backend")
return _tracer
# Convenience alias - lazily forwards attribute access to get_tracer()
class _TracerProxy:
"""Proxy that lazily returns the global tracer."""
def __getattr__(self, name):
return getattr(get_tracer(), name)
tracer = _TracerProxy()
@contextmanager
def create_span(
name: str,
attributes: Optional[Dict[str, Any]] = None,
kind: trace.SpanKind = trace.SpanKind.INTERNAL,
):
"""
Create a traced span as a context manager.
Usage:
with create_span("process-lyrics", {"job_id": job_id}) as span:
# ... do work ...
span.set_attribute("words_processed", 500)
Args:
name: Span name (appears in Cloud Trace)
attributes: Initial span attributes
kind: Span kind (INTERNAL, SERVER, CLIENT, PRODUCER, CONSUMER)
Yields:
The active span
"""
tracer = get_tracer()
with tracer.start_as_current_span(name, kind=kind) as span:
if attributes:
for key, value in attributes.items():
span.set_attribute(key, value)
try:
yield span
except Exception as e:
span.set_status(Status(StatusCode.ERROR, str(e)))
span.record_exception(e)
raise
def traced(
name: Optional[str] = None,
attributes: Optional[Dict[str, Any]] = None,
):
"""
Decorator to trace a function.
Usage:
@traced("process-correction")
def process_correction(job_id: str):
...
@traced(attributes={"operation": "add-lyrics"})
async def add_lyrics(job_id: str, source: str):
...
Args:
name: Span name (defaults to function name)
attributes: Static attributes to add to span
"""
def decorator(func):
span_name = name or func.__name__
@wraps(func)
async def async_wrapper(*args, **kwargs):
with create_span(span_name, attributes) as span:
# Add function arguments as attributes
span.set_attribute("function", func.__name__)
return await func(*args, **kwargs)
@wraps(func)
def sync_wrapper(*args, **kwargs):
with create_span(span_name, attributes) as span:
span.set_attribute("function", func.__name__)
return func(*args, **kwargs)
# Return appropriate wrapper based on function type
import asyncio
if asyncio.iscoroutinefunction(func):
return async_wrapper
return sync_wrapper
return decorator
def add_span_attribute(key: str, value: Any) -> None:
"""
Add an attribute to the current span.
Args:
key: Attribute name
value: Attribute value (must be string, int, float, bool, or list thereof)
"""
span = trace.get_current_span()
if span and span.is_recording():
span.set_attribute(key, value)
def add_span_event(name: str, attributes: Optional[Dict[str, Any]] = None) -> None:
"""
Add an event to the current span.
Events are timestamped annotations that appear in the trace timeline.
Args:
name: Event name
attributes: Event attributes
"""
span = trace.get_current_span()
if span and span.is_recording():
span.add_event(name, attributes=attributes or {})
def set_span_error(error: Exception) -> None:
"""
Mark the current span as errored.
Args:
error: The exception that occurred
"""
span = trace.get_current_span()
if span and span.is_recording():
span.set_status(Status(StatusCode.ERROR, str(error)))
span.record_exception(error)
def get_current_trace_id() -> Optional[str]:
"""
Get the current trace ID for correlation.
This can be used to link logs to traces.
Returns:
Trace ID as hex string, or None if no active trace
"""
span = trace.get_current_span()
if span:
ctx = span.get_span_context()
if ctx.is_valid:
return format(ctx.trace_id, '032x')
return None
def get_current_span_id() -> Optional[str]:
"""
Get the current span ID.
Returns:
Span ID as hex string, or None if no active span
"""
span = trace.get_current_span()
if span:
ctx = span.get_span_context()
if ctx.is_valid:
return format(ctx.span_id, '016x')
return None
# =====================================================
# Job-Aware Tracing Utilities
# =====================================================
@contextmanager
def job_span(
name: str,
job_id: str,
attributes: Optional[Dict[str, Any]] = None,
kind: trace.SpanKind = trace.SpanKind.INTERNAL,
):
"""
Create a traced span with job_id automatically set as an attribute.
This is the preferred way to create spans in job processing workers.
The job_id attribute enables filtering spans by job in Cloud Trace.
Usage:
with job_span("audio-separation", job_id) as span:
span.set_attribute("stage", "clean_instrumental")
# ... do work ...
Args:
name: Span name (appears in Cloud Trace)
job_id: Job ID to attach to the span
attributes: Additional span attributes
kind: Span kind (INTERNAL, SERVER, CLIENT, PRODUCER, CONSUMER)
Yields:
The active span
"""
all_attributes = {
"job_id": job_id,
"service.operation": name,
}
if attributes:
all_attributes.update(attributes)
with create_span(name, all_attributes, kind) as span:
yield span
def inject_trace_context(headers: Dict[str, str]) -> Dict[str, str]:
"""
Inject current trace context into HTTP headers.
Use this when creating Cloud Tasks or making HTTP calls to propagate
the trace context so child spans link to the parent trace.
Usage:
headers = inject_trace_context({
"Content-Type": "application/json",
"Authorization": "Bearer token",
})
# Use headers in HTTP request or Cloud Task
Args:
headers: Existing headers dict to add trace context to
Returns:
Headers dict with traceparent/tracestate headers added
"""
propagator = TraceContextTextMapPropagator()
carrier = dict(headers) # Make a copy
propagator.inject(carrier)
return carrier
def extract_trace_context(headers: Dict[str, str]) -> Optional[trace.Context]:
"""
Extract trace context from HTTP headers.
Use this when receiving Cloud Tasks or HTTP requests to continue
the trace from the parent span.
Usage:
context = extract_trace_context(request.headers)
with tracer.start_as_current_span("worker", context=context):
# ... processing ...
Args:
headers: HTTP headers containing traceparent/tracestate
Returns:
Context object for use with start_as_current_span, or None
"""
propagator = TraceContextTextMapPropagator()
return propagator.extract(headers)
def start_span_with_context(
name: str,
context: Optional[trace.Context],
job_id: Optional[str] = None,
attributes: Optional[Dict[str, Any]] = None,
):
"""
Start a span as child of the given context (or current context if None).
This is useful for starting worker spans that should link to the
trace from the original API request.
Usage:
context = extract_trace_context(request.headers)
with start_span_with_context("audio-worker", context, job_id) as span:
# ... processing ...
Args:
name: Span name
context: Parent context (from extract_trace_context) or None
job_id: Optional job ID to set as attribute
attributes: Additional span attributes
Returns:
Context manager yielding the span
"""
tracer = get_tracer()
all_attributes = attributes or {}
if job_id:
all_attributes["job_id"] = job_id
all_attributes["service.operation"] = name
@contextmanager
def _span():
kwargs = {"kind": trace.SpanKind.INTERNAL}
if context is not None:
kwargs["context"] = context
with tracer.start_as_current_span(name, **kwargs) as span:
for key, value in all_attributes.items():
span.set_attribute(key, value)
try:
yield span
except Exception as e:
span.set_status(Status(StatusCode.ERROR, str(e)))
span.record_exception(e)
raise
return _span()
-439
View File
@@ -1,439 +0,0 @@
"""
Worker coordination service.
Handles triggering and coordinating background workers.
Supports two modes:
- Cloud Tasks (production): Guaranteed delivery, automatic retries, horizontal scaling
- Direct HTTP (development): Faster iteration, simpler debugging
SOLID Principles:
- Single Responsibility: Only handles worker coordination
- Dependency Inversion: Depends on HTTP abstraction, not implementation
- Open/Closed: Can add new workers without modifying existing code
Observability:
- Propagates trace context through Cloud Tasks for distributed tracing
- All worker invocations create spans linked to original request trace
"""
import logging
import os
import json
from typing import Optional
import httpx
from backend.config import get_settings
from backend.services.tracing import inject_trace_context
logger = logging.getLogger(__name__)
# Mapping of worker types to their Cloud Tasks queue names
# These queues are created by Pulumi in infrastructure/__main__.py
WORKER_QUEUES = {
"audio": "audio-worker-queue",
"lyrics": "lyrics-worker-queue",
"screens": "screens-worker-queue",
"render-video": "render-worker-queue",
"video": "video-worker-queue",
}
class WorkerService:
"""
Service for coordinating background workers.
Supports two execution modes controlled by ENABLE_CLOUD_TASKS env var:
1. Cloud Tasks mode (production, ENABLE_CLOUD_TASKS=true):
- Tasks enqueued to Cloud Tasks for guaranteed delivery
- Automatic retries on failure
- Each task runs in dedicated Cloud Run instance
- Rate limiting to protect external APIs (Modal, AudioShake)
2. Direct HTTP mode (development, ENABLE_CLOUD_TASKS=false):
- Direct HTTP POST to internal worker endpoints
- Worker runs as FastAPI BackgroundTask
- Faster iteration for local development
- No retry guarantees
"""
def __init__(self):
"""Initialize worker service."""
self.settings = get_settings()
self._base_url = self._get_base_url()
self._admin_token = self._get_admin_token()
self._use_cloud_tasks = self._should_use_cloud_tasks()
self._tasks_client = None
if self._use_cloud_tasks:
logger.info("WorkerService initialized with Cloud Tasks mode")
else:
logger.info("WorkerService initialized with direct HTTP mode")
def _should_use_cloud_tasks(self) -> bool:
"""
Check if Cloud Tasks should be used for worker coordination.
Controlled by ENABLE_CLOUD_TASKS setting (from environment variable).
Default is false (direct HTTP mode) for backward compatibility.
Returns:
True if Cloud Tasks should be used, False for direct HTTP
"""
return self.settings.enable_cloud_tasks
@property
def tasks_client(self):
"""
Lazy-initialize Cloud Tasks client.
Only created when needed to avoid import overhead in development mode.
"""
if self._tasks_client is None and self._use_cloud_tasks:
try:
from google.cloud import tasks_v2
self._tasks_client = tasks_v2.CloudTasksClient()
except ImportError:
logger.error(
"google-cloud-tasks not installed. "
"Install with: pip install google-cloud-tasks"
)
raise
return self._tasks_client
def _get_admin_token(self) -> Optional[str]:
"""
Get admin token for internal API authentication.
Returns the first admin token from ADMIN_TOKENS env var.
"""
admin_tokens_str = self.settings.admin_tokens or ""
tokens = [t.strip() for t in admin_tokens_str.split(",") if t.strip()]
if tokens:
return tokens[0]
return None
def _get_base_url(self) -> str:
"""
Get base URL for internal API calls.
Priority:
1. TEST_SERVER_URL env var (for tests)
2. CLOUD_RUN_SERVICE_URL env var (for Cloud Tasks to call back)
3. localhost with PORT env var (for development)
Returns:
Base URL for API calls
"""
# Check for test environment override
test_url = os.getenv("TEST_SERVER_URL")
if test_url:
return test_url
# Production: Cloud Run service URL
# This must be the publicly accessible URL for Cloud Tasks to call
service_url = os.getenv("CLOUD_RUN_SERVICE_URL")
if service_url:
return service_url
# Development mode: use localhost with PORT env var
port = os.getenv("PORT", "8000")
return f"http://localhost:{port}"
async def trigger_worker(
self,
worker_type: str,
job_id: str,
timeout_seconds: int = 30
) -> bool:
"""
Trigger a background worker for a job.
In production (ENABLE_CLOUD_TASKS=true):
- Enqueues task to Cloud Tasks queue
- Cloud Tasks delivers HTTP request to internal endpoint
- Automatic retries on failure with exponential backoff
- Rate limiting protects external APIs
In development (ENABLE_CLOUD_TASKS=false):
- Direct HTTP POST to internal endpoint
- Faster iteration, but no retry guarantees
- Worker runs in same container as API
Args:
worker_type: Worker type ("audio", "lyrics", "screens", "render-video", "video")
job_id: Job ID to process
timeout_seconds: Request timeout (for direct HTTP mode)
Returns:
True if trigger successful (or task enqueued), False otherwise
"""
if self._use_cloud_tasks:
return await self._enqueue_cloud_task(worker_type, job_id)
else:
return await self._trigger_http(worker_type, job_id, timeout_seconds)
async def _enqueue_cloud_task(self, worker_type: str, job_id: str) -> bool:
"""
Enqueue task to Cloud Tasks for guaranteed delivery.
The task will be delivered as an HTTP POST to the internal worker endpoint.
Cloud Tasks handles:
- Retry on failure (with exponential backoff)
- Rate limiting (max dispatches per second)
- Deduplication (via task name if needed)
- OIDC authentication for Cloud Run
Observability:
- Injects trace context headers so worker spans link to parent trace
Args:
worker_type: Worker type
job_id: Job ID to process
Returns:
True if task enqueued successfully, False otherwise
"""
try:
from google.cloud import tasks_v2
queue_name = WORKER_QUEUES.get(worker_type)
if not queue_name:
logger.error(f"Unknown worker type: {worker_type}")
return False
project = self.settings.google_cloud_project
if not project:
logger.error("GOOGLE_CLOUD_PROJECT not set, cannot enqueue Cloud Task")
return False
location = self.settings.gcp_region # Must match queue location
# Build queue path
parent = self.tasks_client.queue_path(project, location, queue_name)
# Build base headers
headers = {
"Content-Type": "application/json",
}
# Add admin auth via custom header
# NOTE: We use X-Admin-Token instead of Authorization because Cloud Tasks
# OIDC token overwrites the Authorization header when oidc_token is specified.
# The OIDC token handles Cloud Run authentication (allows Cloud Tasks to invoke
# the service), while X-Admin-Token handles application-level authentication.
if self._admin_token:
headers["X-Admin-Token"] = self._admin_token
logger.debug(
f"[job:{job_id}] Using admin token for Cloud Task auth via X-Admin-Token, "
f"token prefix: {self._admin_token[:8]}..., len={len(self._admin_token)}"
)
# Inject trace context for distributed tracing
# This allows worker spans to link back to the original request trace
headers = inject_trace_context(headers)
# Build task payload
task = {
"http_request": {
"http_method": tasks_v2.HttpMethod.POST,
"url": f"{self._base_url}/api/internal/workers/{worker_type}",
"headers": headers,
"body": json.dumps({"job_id": job_id}).encode(),
# Use OIDC token for Cloud Run authentication
# This allows Cloud Tasks to invoke the Cloud Run service
"oidc_token": {
"service_account_email": f"karaoke-backend@{project}.iam.gserviceaccount.com",
},
},
}
# Create task
# Note: We don't set a task name, allowing Cloud Tasks to generate one
# This prevents duplicate task errors on retries
response = self.tasks_client.create_task(parent=parent, task=task)
logger.info(
f"[job:{job_id}] Created Cloud Task for {worker_type} worker: {response.name}"
)
return True
except Exception as e:
logger.error(
f"[job:{job_id}] Failed to enqueue Cloud Task for {worker_type}: {e}",
exc_info=True
)
return False
async def _trigger_http(
self,
worker_type: str,
job_id: str,
timeout_seconds: int = 30
) -> bool:
"""
Trigger worker via direct HTTP call (development mode).
This is the original implementation - direct HTTP POST to internal endpoint.
The endpoint adds the worker function to FastAPI BackgroundTasks.
Observability:
- Injects trace context headers so worker spans link to parent trace
Args:
worker_type: Worker type
job_id: Job ID to process
timeout_seconds: Request timeout
Returns:
True if trigger successful, False otherwise
"""
try:
# Build headers with admin auth token
headers = {}
if self._admin_token:
headers["Authorization"] = f"Bearer {self._admin_token}"
# Inject trace context for distributed tracing
headers = inject_trace_context(headers)
async with httpx.AsyncClient(timeout=timeout_seconds) as client:
url = f"{self._base_url}/api/internal/workers/{worker_type}"
response = await client.post(
url,
json={"job_id": job_id},
headers=headers
)
if response.status_code == 200:
logger.info(f"[job:{job_id}] Successfully triggered {worker_type} worker")
return True
else:
logger.error(
f"[job:{job_id}] Failed to trigger {worker_type} worker: "
f"HTTP {response.status_code} - {response.text}"
)
return False
except httpx.TimeoutException:
logger.error(f"[job:{job_id}] Timeout triggering {worker_type} worker")
return False
except Exception as e:
logger.error(
f"[job:{job_id}] Error triggering {worker_type} worker: {e}",
exc_info=True
)
return False
# Convenience methods for specific workers
# These provide a cleaner API and better IDE autocomplete
async def trigger_audio_worker(self, job_id: str) -> bool:
"""Trigger audio separation worker."""
return await self.trigger_worker("audio", job_id)
async def trigger_lyrics_worker(self, job_id: str) -> bool:
"""Trigger lyrics transcription worker."""
return await self.trigger_worker("lyrics", job_id)
async def trigger_screens_worker(self, job_id: str) -> bool:
"""Trigger screen generation worker."""
return await self.trigger_worker("screens", job_id)
async def trigger_video_worker(self, job_id: str) -> bool:
"""
Trigger video generation worker.
When USE_CLOUD_RUN_JOBS_FOR_VIDEO=true and ENABLE_CLOUD_TASKS=true,
uses Cloud Run Jobs for execution (supports >30 min encoding).
Otherwise, uses Cloud Tasks or direct HTTP.
"""
if self._use_cloud_tasks and self.settings.use_cloud_run_jobs_for_video:
return await self._trigger_cloud_run_job(job_id)
return await self.trigger_worker("video", job_id)
async def _trigger_cloud_run_job(self, job_id: str) -> bool:
"""
Trigger video encoding as a Cloud Run Job.
Cloud Run Jobs support up to 24 hours of execution time,
making them suitable for very long video encoding tasks.
Args:
job_id: Job ID to process
Returns:
True if job was triggered successfully, False otherwise
"""
try:
from google.cloud import run_v2
project = self.settings.google_cloud_project
if not project:
logger.error("GOOGLE_CLOUD_PROJECT not set, cannot trigger Cloud Run Job")
return False
location = self.settings.gcp_region
job_name = f"projects/{project}/locations/{location}/jobs/video-encoding-job"
# Create Cloud Run Jobs client
client = run_v2.JobsClient()
# Run the job with overrides for the specific job_id
request = run_v2.RunJobRequest(
name=job_name,
overrides=run_v2.RunJobRequest.Overrides(
container_overrides=[
run_v2.RunJobRequest.Overrides.ContainerOverride(
args=["python", "-m", "backend.workers.video_worker", "--job-id", job_id],
)
]
)
)
# Run the job (async operation)
operation = client.run_job(request=request)
logger.info(f"Started Cloud Run Job for video encoding, job {job_id}: {operation.metadata}")
return True
except Exception as e:
logger.error(
f"Failed to trigger Cloud Run Job for video/{job_id}: {e}",
exc_info=True
)
return False
async def trigger_render_video_worker(self, job_id: str) -> bool:
"""Trigger render video worker (post-review)."""
return await self.trigger_worker("render-video", job_id)
# Global worker service instance
_worker_service: Optional[WorkerService] = None
def get_worker_service() -> WorkerService:
"""
Get global worker service instance.
Singleton pattern to reuse HTTP client pool.
Returns:
WorkerService instance
"""
global _worker_service
if _worker_service is None:
_worker_service = WorkerService()
return _worker_service
def reset_worker_service() -> None:
"""
Reset the global worker service instance.
Used in tests to ensure clean state between test cases.
"""
global _worker_service
_worker_service = None
-112
View File
@@ -1,112 +0,0 @@
"""
YouTube credential service for server-side video uploads.
This service manages YouTube OAuth credentials stored in Secret Manager
for non-interactive uploads from backend workers.
"""
import json
import logging
from typing import Optional, Dict, Any
from backend.config import get_settings
logger = logging.getLogger(__name__)
class YouTubeService:
"""Service for managing YouTube OAuth credentials."""
# Secret Manager secret name for YouTube credentials
YOUTUBE_CREDENTIALS_SECRET = "youtube-oauth-credentials"
def __init__(self):
self.settings = get_settings()
self._credentials: Optional[Dict[str, Any]] = None
self._loaded = False
def load_credentials(self) -> bool:
"""
Load YouTube OAuth credentials from Secret Manager.
The secret should contain a JSON object with:
- token: Current access token (may be expired)
- refresh_token: Refresh token for getting new access tokens
- token_uri: Token endpoint URL
- client_id: OAuth client ID
- client_secret: OAuth client secret
- scopes: List of OAuth scopes
Returns:
True if credentials were loaded successfully, False otherwise
"""
if self._loaded:
return self._credentials is not None
try:
# Get credentials from Secret Manager
creds_json = self.settings.get_secret(self.YOUTUBE_CREDENTIALS_SECRET)
if not creds_json:
logger.warning("YouTube credentials not found in Secret Manager")
self._loaded = True
return False
# Parse the credentials JSON
self._credentials = json.loads(creds_json)
# Validate required fields
required_fields = ['refresh_token', 'token_uri', 'client_id', 'client_secret']
missing = [f for f in required_fields if not self._credentials.get(f)]
if missing:
logger.error(f"YouTube credentials missing required fields: {missing}")
self._credentials = None
self._loaded = True
return False
logger.info("YouTube credentials loaded successfully from Secret Manager")
self._loaded = True
return True
except json.JSONDecodeError as e:
logger.error(f"Failed to parse YouTube credentials JSON: {e}")
self._loaded = True
return False
except Exception as e:
logger.error(f"Failed to load YouTube credentials: {e}")
self._loaded = True
return False
def get_credentials_dict(self) -> Optional[Dict[str, Any]]:
"""
Get the YouTube credentials as a dictionary.
This format is compatible with KaraokeFinalise's user_youtube_credentials
parameter, which creates a google.oauth2.credentials.Credentials object.
Returns:
Dictionary with credential data, or None if not available
"""
if not self._loaded:
self.load_credentials()
return self._credentials
@property
def is_configured(self) -> bool:
"""Check if YouTube credentials are configured and ready to use."""
if not self._loaded:
self.load_credentials()
return self._credentials is not None
# Singleton instance
_youtube_service: Optional[YouTubeService] = None
def get_youtube_service() -> YouTubeService:
"""Get the singleton YouTube service instance."""
global _youtube_service
if _youtube_service is None:
_youtube_service = YouTubeService()
return _youtube_service
-4
View File
@@ -1,4 +0,0 @@
"""
Backend integration tests package.
"""
-218
View File
@@ -1,218 +0,0 @@
"""
Shared pytest fixtures for backend tests.
Provides common mocks and test utilities across all test modules.
NOTE: Module-level mocks are NOT applied here to allow emulator tests to work.
Individual tests must mock dependencies as needed.
"""
import os
import pytest
from unittest.mock import Mock, MagicMock, AsyncMock, patch
from datetime import datetime, UTC
from fastapi.testclient import TestClient
# Mock google.auth.default AND firestore for unit tests if not using emulator
# This prevents DefaultCredentialsError and FirestoreClient initialization during imports
if 'FIRESTORE_EMULATOR_HOST' not in os.environ:
from unittest.mock import MagicMock
# Set up test environment variables BEFORE importing any backend modules
os.environ.setdefault('ADMIN_TOKENS', 'test-admin-token')
os.environ.setdefault('GOOGLE_CLOUD_PROJECT', 'test-project')
os.environ.setdefault('GCS_BUCKET_NAME', 'test-bucket')
os.environ.setdefault('FIRESTORE_COLLECTION', 'jobs')
os.environ.setdefault('ENVIRONMENT', 'test')
# Mock google.auth.default - prevents credential errors
try:
import google.auth
mock_credentials = MagicMock()
mock_credentials.token = 'fake-token'
mock_credentials.valid = True
mock_credentials.universe_domain = 'googleapis.com' # Required by google-cloud-storage
mock_credentials.project_id = 'test-project'
google.auth.default = MagicMock(return_value=(mock_credentials, 'test-project'))
except ImportError:
# If google.auth not installed, that's ok for some tests
pass
# Mock google.cloud.firestore.Client - prevents Firestore initialization
# This is CRITICAL: when AuthService is instantiated, it creates FirestoreService
# which tries to create a real Firestore client. The client returns MagicMocks
# which fail when used as enum values (e.g., UserType).
try:
import google.cloud.firestore as firestore_module
# Create a mock client that returns proper values for get_token
mock_firestore_client = MagicMock()
mock_collection = MagicMock()
mock_doc = MagicMock()
mock_snapshot = MagicMock()
# For auth token lookups, return None (token not found in Firestore)
# This forces validation to use admin tokens from environment
mock_snapshot.exists = False
mock_snapshot.to_dict.return_value = None
mock_doc.get.return_value = mock_snapshot
mock_collection.document.return_value = mock_doc
mock_firestore_client.collection.return_value = mock_collection
# Replace the Client class
original_client = firestore_module.Client
firestore_module.Client = MagicMock(return_value=mock_firestore_client)
except ImportError:
pass
# Mock google.cloud.storage.Client - prevents GCS initialization
try:
import google.cloud.storage as storage_module
mock_storage_client = MagicMock()
mock_bucket = MagicMock()
mock_storage_client.bucket.return_value = mock_bucket
storage_module.Client = MagicMock(return_value=mock_storage_client)
except ImportError:
pass
from backend.models.job import Job, JobStatus, JobCreate
@pytest.fixture(autouse=True)
def mock_auth_dependency(request):
"""Mock the require_auth dependency for all tests using FastAPI's dependency override system."""
# Skip for emulator tests and integration tests which use real auth
test_path = str(request.fspath)
if 'emulator' in test_path or 'integration' in test_path:
yield
return
# Skip if FIRESTORE_EMULATOR_HOST is set (running in emulator environment)
import os
if os.environ.get('FIRESTORE_EMULATOR_HOST'):
yield
return
from backend.services.auth_service import UserType
from backend.api.dependencies import require_auth, require_admin, require_review_auth, require_instrumental_auth
from backend.main import app
# Create mock auth functions
async def mock_require_auth():
"""Mock require_auth to always return valid admin credentials."""
return ("test-admin-token", UserType.ADMIN, 999)
async def mock_require_admin():
"""Mock require_admin to always return valid admin credentials."""
return ("test-admin-token", UserType.ADMIN, 999)
async def mock_require_review_auth(job_id: str = "test123"):
"""Mock require_review_auth to always return valid review access."""
return (job_id, "full")
async def mock_require_instrumental_auth(job_id: str = "test123"):
"""Mock require_instrumental_auth to always return valid instrumental review access."""
return (job_id, "full")
# Use FastAPI's dependency override system
app.dependency_overrides[require_auth] = mock_require_auth
app.dependency_overrides[require_admin] = mock_require_admin
app.dependency_overrides[require_review_auth] = mock_require_review_auth
app.dependency_overrides[require_instrumental_auth] = mock_require_instrumental_auth
yield
# Clean up after test
app.dependency_overrides.clear()
@pytest.fixture
def auth_headers():
"""Get authentication headers for test requests."""
return {
"Authorization": "Bearer test-admin-token",
"Content-Type": "application/json"
}
@pytest.fixture
def mock_firestore():
"""Mock Firestore client for unit tests."""
# Mock at module level for unit tests
with patch('google.cloud.firestore.Client') as mock_cls:
client = MagicMock()
mock_cls.return_value = client
yield client
@pytest.fixture
def mock_storage_client():
"""Mock GCS Storage client for unit tests."""
# Mock at module level for unit tests
with patch('google.cloud.storage.Client') as mock_cls:
client = MagicMock()
mock_cls.return_value = client
yield client
@pytest.fixture
def mock_httpx_client():
"""Mock httpx AsyncClient for worker service."""
with patch('backend.services.worker_service.httpx.AsyncClient') as mock:
client = AsyncMock()
mock.return_value.__aenter__.return_value = client
yield client
@pytest.fixture
def sample_job():
"""Create a sample Job instance for testing."""
return Job(
job_id="test123",
status=JobStatus.PENDING,
progress=0,
created_at=datetime.now(UTC),
updated_at=datetime.now(UTC),
artist="Test Artist",
title="Test Song",
url="https://youtube.com/watch?v=test"
)
@pytest.fixture
def sample_job_create():
"""Create a sample JobCreate instance for testing."""
return JobCreate(
url="https://youtube.com/watch?v=test",
artist="Test Artist",
title="Test Song"
)
@pytest.fixture
def test_client():
"""Create a FastAPI TestClient."""
# Import here to avoid circular dependency
from backend.main import app
return TestClient(app)
def create_mock_job(**kwargs):
"""
Factory function to create mock Job instances with custom fields.
Args:
**kwargs: Job fields to override defaults
Returns:
Job instance with specified fields
"""
defaults = {
"job_id": "test123",
"status": JobStatus.PENDING,
"progress": 0,
"created_at": datetime.now(UTC),
"updated_at": datetime.now(UTC)
}
defaults.update(kwargs)
return Job(**defaults)
-7
View File
@@ -1,7 +0,0 @@
"""
Emulator integration tests.
These tests run against local GCP emulators (Firestore, GCS)
to provide true integration testing without cloud costs.
"""
-88
View File
@@ -1,88 +0,0 @@
"""
Fixtures for emulator integration tests.
These tests use REAL Firestore and GCS emulators, so we DON'T mock google.cloud modules.
Run with: scripts/run-emulator-tests.sh
"""
import pytest
import os
import requests
from unittest.mock import AsyncMock, patch
def emulators_running() -> bool:
"""Check if GCP emulators are running."""
try:
# Check Firestore emulator
requests.get("http://127.0.0.1:8080", timeout=1)
# Check GCS emulator
requests.get("http://127.0.0.1:4443", timeout=1)
return True
except (requests.exceptions.ConnectionError, requests.exceptions.Timeout):
return False
# Skip all tests in this module if emulators aren't running
pytestmark = pytest.mark.skipif(
not emulators_running(),
reason="GCP emulators not running. Start with: scripts/start-emulators.sh"
)
# Set emulator environment variables (use 127.0.0.1 to force IPv4)
os.environ["FIRESTORE_EMULATOR_HOST"] = "127.0.0.1:8080"
os.environ["STORAGE_EMULATOR_HOST"] = "http://127.0.0.1:4443"
os.environ["GOOGLE_CLOUD_PROJECT"] = "test-project"
os.environ["GCS_BUCKET_NAME"] = "test-bucket"
os.environ["FIRESTORE_COLLECTION"] = "test-jobs"
os.environ["ENVIRONMENT"] = "test"
os.environ["ADMIN_TOKENS"] = "test-admin-token"
# Only import app if emulators are running
if emulators_running():
from fastapi.testclient import TestClient
from backend.main import app
else:
TestClient = None
app = None
@pytest.fixture(scope="session", autouse=True)
def setup_gcs_bucket():
"""Create GCS bucket in emulator before all tests."""
try:
response = requests.post(
"http://127.0.0.1:4443/storage/v1/b",
json={"name": "test-bucket"},
params={"project": "test-project"},
timeout=5
)
if response.status_code in [200, 409]:
print("✅ GCS bucket 'test-bucket' ready in emulator")
except Exception as e:
print(f"⚠️ GCS bucket setup failed: {e}")
yield
@pytest.fixture(scope="session")
def mock_worker_service():
"""Mock worker service to prevent background tasks."""
with patch("backend.api.routes.jobs.worker_service") as mock:
mock.trigger_audio_worker = AsyncMock(return_value=True)
mock.trigger_lyrics_worker = AsyncMock(return_value=True)
mock.trigger_screens_worker = AsyncMock(return_value=True)
mock.trigger_video_worker = AsyncMock(return_value=True)
yield mock
@pytest.fixture(scope="session")
def client(mock_worker_service):
"""Create FastAPI test client with mocked workers."""
with patch("backend.api.routes.file_upload.worker_service", mock_worker_service):
return TestClient(app)
@pytest.fixture
def auth_headers():
"""Auth headers for testing."""
return {"Authorization": "Bearer test-admin-token"}
File diff suppressed because it is too large. Load diff
@@ -1,356 +0,0 @@
"""
Integration tests using local GCP emulators.
These tests use real Firestore and GCS emulators running locally,
providing true integration testing without cloud resources or costs.
Run with: ./scripts/run-emulator-tests.sh
NOTE: These tests mock background workers to avoid race conditions.
The fixtures are defined in conftest.py in this directory.
"""
import pytest
import time
from .conftest import emulators_running
# Skip all tests in this module if emulators aren't running
pytestmark = pytest.mark.skipif(
not emulators_running(),
reason="GCP emulators not running. Start with: scripts/start-emulators.sh"
)
# Fixtures are loaded from conftest.py in this directory
class TestEmulatorBasics:
"""Basic emulator connectivity tests."""
def test_health_endpoint(self, client, auth_headers):
"""Test health endpoint works."""
response = client.get("/api/health", )
assert response.status_code == 200
assert response.json()["status"] == "healthy"
def test_root_endpoint(self, client, auth_headers):
"""Test root endpoint works."""
response = client.get("/", )
assert response.status_code == 200
assert response.json()["service"] == "karaoke-gen-backend"
class TestJobCreation:
"""Test job creation with Firestore emulator."""
def test_create_job_simple(self, client, auth_headers):
"""Test creating a simple job."""
response = client.post(
"/api/jobs",
headers=auth_headers,
json={"url": "https://youtube.com/watch?v=test123"}
)
assert response.status_code == 200
data = response.json()
assert data["status"] == "success"
assert "job_id" in data
def test_create_job_with_metadata(self, client, auth_headers):
"""Test creating a job with artist/title."""
response = client.post(
"/api/jobs",
headers=auth_headers,
json={
"url": "https://youtube.com/watch?v=test",
"artist": "Test Artist",
"title": "Test Song"
}
)
assert response.status_code == 200
data = response.json()
assert "job_id" in data
class TestJobRetrieval:
"""Test job retrieval from Firestore emulator."""
def test_create_and_get_job(self, client, auth_headers):
"""Test creating and then retrieving a job."""
# Create
create_resp = client.post(
"/api/jobs",
headers=auth_headers,
json={
"url": "https://youtube.com/watch?v=abc123",
"artist": "Test Artist",
"title": "Test Song"
}
)
assert create_resp.status_code == 200
job_id = create_resp.json()["job_id"]
# Small delay for emulator consistency
time.sleep(0.2)
# Retrieve
get_resp = client.get(f"/api/jobs/{job_id}", headers=auth_headers)
if get_resp.status_code != 200:
print(f"GET failed: {get_resp.status_code} - {get_resp.text}")
assert get_resp.status_code == 200
job = get_resp.json()
assert job["job_id"] == job_id
assert job["status"] == "pending"
assert job["artist"] == "Test Artist"
assert job["title"] == "Test Song"
def test_get_nonexistent_job(self, client, auth_headers):
"""Test fetching a job that doesn't exist."""
response = client.get("/api/jobs/nonexistent-id", headers=auth_headers)
assert response.status_code == 404
class TestJobList:
"""Test listing jobs from Firestore."""
def test_list_jobs(self, client, auth_headers):
"""Test listing jobs."""
# Create a few jobs
for i in range(3):
client.post(
"/api/jobs",
headers=auth_headers,
json={"url": f"https://youtube.com/watch?v=list{i}"}
)
time.sleep(0.2)
# List
response = client.get("/api/jobs", headers=auth_headers)
assert response.status_code == 200
jobs = response.json()
assert isinstance(jobs, list)
# We should have at least the 3 we just created
assert len(jobs) >= 3
class TestJobDeletion:
"""Test job deletion."""
def test_delete_job(self, client, auth_headers):
"""Test deleting a job."""
# Create
create_resp = client.post(
"/api/jobs",
headers=auth_headers,
json={"url": "https://youtube.com/watch?v=delete-me"}
)
job_id = create_resp.json()["job_id"]
time.sleep(0.2)
# Delete
del_resp = client.delete(f"/api/jobs/{job_id}", headers=auth_headers)
assert del_resp.status_code == 200
# Verify deleted
get_resp = client.get(f"/api/jobs/{job_id}", headers=auth_headers)
assert get_resp.status_code == 404
class TestJobUpdates:
"""Test job status updates."""
def test_cancel_job(self, client, auth_headers):
"""Test cancelling a job."""
# Create
create_resp = client.post(
"/api/jobs",
headers=auth_headers,
json={"url": "https://youtube.com/watch?v=cancel-me"}
)
job_id = create_resp.json()["job_id"]
time.sleep(0.2)
# Cancel
cancel_resp = client.post(
f"/api/jobs/{job_id}/cancel",
headers=auth_headers,
json={"reason": "test cancellation"}
)
assert cancel_resp.status_code == 200
# Verify cancelled
get_resp = client.get(f"/api/jobs/{job_id}", headers=auth_headers)
assert get_resp.status_code == 200
job = get_resp.json()
assert job["status"] == "cancelled"
class TestFileUpload:
"""Test file upload with GCS emulator."""
def test_upload_file(self, client, auth_headers):
"""Test uploading a file."""
test_file_content = b"fake audio data for testing"
response = client.post(
"/api/jobs/upload",
headers={"Authorization": auth_headers["Authorization"]},
files={"file": ("test.flac", test_file_content, "audio/flac")},
data={"artist": "Upload Artist", "title": "Upload Song"}
)
assert response.status_code == 200
data = response.json()
assert data["status"] == "success"
assert "job_id" in data
job_id = data["job_id"]
time.sleep(0.2)
# Verify job created with upload data
get_resp = client.get(f"/api/jobs/{job_id}", headers=auth_headers)
assert get_resp.status_code == 200
job = get_resp.json()
assert job["artist"] == "Upload Artist"
assert job["title"] == "Upload Song"
assert "input_media_gcs_path" in job
class TestInternalEndpoints:
"""Test internal worker endpoints."""
def test_internal_workers_exist(self, client, auth_headers):
"""Test that internal worker endpoints exist and respond."""
# Create a job first
create_resp = client.post(
"/api/jobs",
headers=auth_headers,
json={"url": "https://youtube.com/watch?v=worker-test"}
)
job_id = create_resp.json()["job_id"]
time.sleep(0.2)
# Test audio worker endpoint exists
response = client.post(
"/api/internal/workers/audio",
headers=auth_headers,
json={"job_id": job_id}
)
assert response.status_code == 200
assert response.json()["status"] == "started"
class TestReviewEndpoints:
"""Test lyrics review API endpoints."""
def test_review_ping(self, client, auth_headers):
"""Test review ping endpoint."""
# Create a job first
create_resp = client.post(
"/api/jobs",
headers=auth_headers,
json={"url": "https://youtube.com/watch?v=review-test"}
)
job_id = create_resp.json()["job_id"]
# Ping should work for any job
response = client.get(f"/api/review/{job_id}/ping")
assert response.status_code == 200
assert response.json()["status"] == "ok"
def test_review_correction_data_wrong_status(self, client, auth_headers):
"""Test that correction-data returns error for non-review jobs."""
# Create a job (starts in pending status)
create_resp = client.post(
"/api/jobs",
headers=auth_headers,
json={"url": "https://youtube.com/watch?v=review-status-test"}
)
job_id = create_resp.json()["job_id"]
time.sleep(0.2)
# Should fail - job not in AWAITING_REVIEW status
response = client.get(f"/api/review/{job_id}/correction-data", headers=auth_headers)
assert response.status_code == 400
assert "not ready for review" in response.json()["detail"].lower()
def test_review_audio_no_job(self, client, auth_headers):
"""Test audio endpoint returns 404 for nonexistent job."""
response = client.get("/api/review/nonexistent-job/audio/", headers=auth_headers)
assert response.status_code == 404
def test_review_preview_video_stub(self, client, auth_headers):
"""Test preview video endpoint exists."""
# Create a job
create_resp = client.post(
"/api/jobs",
headers=auth_headers,
json={"url": "https://youtube.com/watch?v=preview-test"}
)
job_id = create_resp.json()["job_id"]
# Preview video should return error since job not ready
response = client.post(
f"/api/review/{job_id}/preview-video",
headers=auth_headers,
json={"corrections": [], "corrected_segments": []}
)
# Should return 400 (job not in AWAITING_REVIEW state)
assert response.status_code == 400
def test_review_annotations_stub(self, client, auth_headers):
"""Test annotations endpoint (stub)."""
create_resp = client.post(
"/api/jobs",
headers=auth_headers,
json={"url": "https://youtube.com/watch?v=annotation-test"}
)
job_id = create_resp.json()["job_id"]
# Annotations endpoint should accept data (even if it just logs it)
response = client.post(
f"/api/review/{job_id}/v1/annotations",
headers=auth_headers,
json={"type": "test", "data": "test annotation"}
)
assert response.status_code == 200
assert response.json()["status"] == "success"
class TestRenderVideoWorker:
"""Test render video worker endpoint."""
def test_render_video_worker_endpoint_exists(self, client, auth_headers):
"""Test that render-video worker endpoint exists."""
# Create a job
create_resp = client.post(
"/api/jobs",
headers=auth_headers,
json={"url": "https://youtube.com/watch?v=render-test"}
)
job_id = create_resp.json()["job_id"]
time.sleep(0.2)
# Test render-video worker endpoint exists
response = client.post(
"/api/internal/workers/render-video",
headers=auth_headers,
json={"job_id": job_id}
)
# Should return 200 (endpoint exists and starts)
assert response.status_code == 200
print("✅ Emulator integration tests ready to run")
@@ -1,436 +0,0 @@
"""
Direct tests for render_video_worker style loading.
Tests the fix for styles being loaded from job.style_params_gcs_path
instead of the incorrect job.state_data['styles_gcs_path'].
Run with:
./scripts/start-emulators.sh
pytest backend/tests/emulator/test_style_loading_direct.py -v
"""
import pytest
import json
import os
import tempfile
import requests
from dataclasses import dataclass
from typing import Dict, Optional
def emulators_running() -> bool:
"""Check if GCP emulators are running."""
try:
requests.get("http://127.0.0.1:8080", timeout=1)
requests.get("http://127.0.0.1:4443", timeout=1)
return True
except (requests.exceptions.ConnectionError, requests.exceptions.Timeout):
return False
# Skip all tests in this module if emulators aren't running
pytestmark = pytest.mark.skipif(
not emulators_running(),
reason="GCP emulators not running. Start with: scripts/start-emulators.sh"
)
# Set up environment for emulators
os.environ["FIRESTORE_EMULATOR_HOST"] = "127.0.0.1:8080"
os.environ["STORAGE_EMULATOR_HOST"] = "http://127.0.0.1:4443"
os.environ["GOOGLE_CLOUD_PROJECT"] = "test-project"
os.environ["GCS_BUCKET_NAME"] = "test-bucket"
@dataclass
class MockJob:
"""Mock job object for testing style loading."""
job_id: str
style_params_gcs_path: Optional[str] = None
style_assets: Dict[str, str] = None
state_data: Dict = None
def __post_init__(self):
if self.style_assets is None:
self.style_assets = {}
if self.state_data is None:
self.state_data = {}
class MockStorageService:
"""Mock storage service that uses GCS emulator."""
def __init__(self):
from google.cloud import storage
self.client = storage.Client()
self.bucket_name = "test-bucket"
self._ensure_bucket_exists()
def _ensure_bucket_exists(self):
"""Create bucket in emulator if it doesn't exist."""
try:
self.client.create_bucket(self.bucket_name)
except Exception:
pass # Bucket already exists
def upload_string(self, content: str, gcs_path: str, content_type: str = "application/json"):
"""Upload string content to GCS."""
bucket = self.client.bucket(self.bucket_name)
blob = bucket.blob(gcs_path)
blob.upload_from_string(content, content_type=content_type)
def upload_bytes(self, content: bytes, gcs_path: str, content_type: str = "application/octet-stream"):
"""Upload bytes to GCS."""
bucket = self.client.bucket(self.bucket_name)
blob = bucket.blob(gcs_path)
blob.upload_from_string(content, content_type=content_type)
def download_file(self, gcs_path: str, local_path: str):
"""Download file from GCS to local path."""
bucket = self.client.bucket(self.bucket_name)
blob = bucket.blob(gcs_path)
blob.download_to_filename(local_path)
def file_exists(self, gcs_path: str) -> bool:
"""Check if file exists in GCS."""
bucket = self.client.bucket(self.bucket_name)
blob = bucket.blob(gcs_path)
return blob.exists()
class TestStyleLoadingFix:
"""
Tests for the render_video_worker style loading fix.
The bug was: render_video_worker looked for styles at
job.state_data.get('styles_gcs_path') but styles are actually stored at
job.style_params_gcs_path and job.style_assets.
"""
@pytest.fixture(autouse=True)
def setup(self):
"""Set up storage service and temp directory for each test."""
self.storage = MockStorageService()
self.temp_dir = tempfile.mkdtemp()
yield
# Cleanup
import shutil
shutil.rmtree(self.temp_dir, ignore_errors=True)
def _import_get_or_create_styles(self):
"""
Import _get_or_create_styles without importing the full worker module.
We copy the function here to avoid import issues with lyrics_transcriber.
"""
# We'll test using a copy of the function logic
# This avoids importing the full worker which has lyrics_transcriber dependency
pass
def test_style_loading_from_correct_location(self):
"""
Test that styles are loaded from job.style_params_gcs_path,
NOT from job.state_data['styles_gcs_path'].
"""
import time
job_id = f"test-style-{int(time.time() * 1000)}"
# Create style JSON with placeholder paths
style_json = {
"intro": {
"background_image": "/original/path/intro_bg.png",
"font": "/original/path/font.ttf"
},
"karaoke": {
"background_image": "/original/path/karaoke_bg.png",
"font_path": "/original/path/font.ttf",
"font_size": 100
},
"end": {
"background_image": "/original/path/end_bg.png",
"font": "/original/path/font.ttf"
}
}
# Upload style JSON to GCS (the CORRECT location)
style_gcs_path = f"uploads/{job_id}/style/style_params.json"
self.storage.upload_string(json.dumps(style_json), style_gcs_path)
# Upload mock assets
assets = {
"intro_background": f"uploads/{job_id}/style/intro_background.png",
"karaoke_background": f"uploads/{job_id}/style/karaoke_background.png",
"end_background": f"uploads/{job_id}/style/end_background.png",
"font": f"uploads/{job_id}/style/font.ttf",
}
for asset_key, gcs_path in assets.items():
# Upload fake file content
self.storage.upload_bytes(b"fake image/font data", gcs_path)
# Create mock job with styles in the CORRECT location
job = MockJob(
job_id=job_id,
style_params_gcs_path=style_gcs_path, # CORRECT
style_assets=assets, # CORRECT
state_data={} # state_data is EMPTY (no styles_gcs_path)
)
# Verify files exist in GCS
assert self.storage.file_exists(style_gcs_path), "Style JSON should exist in GCS"
for gcs_path in assets.values():
assert self.storage.file_exists(gcs_path), f"Asset {gcs_path} should exist in GCS"
# Now test the style loading logic (inline version of _get_or_create_styles)
style_dir = os.path.join(self.temp_dir, "style")
os.makedirs(style_dir, exist_ok=True)
styles_path = os.path.join(style_dir, "styles.json")
# This is the FIX: check job.style_params_gcs_path, not state_data
if job.style_params_gcs_path:
# Download style JSON
self.storage.download_file(job.style_params_gcs_path, styles_path)
# Load and update paths
with open(styles_path, 'r') as f:
style_data = json.load(f)
# Download assets and update paths
local_assets = {}
for asset_key, gcs_path in job.style_assets.items():
ext = os.path.splitext(gcs_path)[1] or '.png'
local_path = os.path.join(style_dir, f"{asset_key}{ext}")
self.storage.download_file(gcs_path, local_path)
local_assets[asset_key] = local_path
# Update style_data with local paths
asset_mapping = {
'intro_background': ('intro', 'background_image'),
'karaoke_background': ('karaoke', 'background_image'),
'end_background': ('end', 'background_image'),
'font': [('intro', 'font'), ('karaoke', 'font_path'), ('end', 'font')],
}
for asset_key, local_path in local_assets.items():
if asset_key in asset_mapping:
mappings = asset_mapping[asset_key]
if isinstance(mappings[0], str):
mappings = [mappings]
for section, field in mappings:
if section in style_data:
style_data[section][field] = local_path
# Save updated styles
with open(styles_path, 'w') as f:
json.dump(style_data, f, indent=2)
# Verify the result
with open(styles_path, 'r') as f:
result = json.load(f)
# Check that paths were updated to local paths
assert style_dir in result['karaoke']['background_image'], \
f"karaoke.background_image should be local path, got: {result['karaoke']['background_image']}"
assert style_dir in result['karaoke']['font_path'], \
f"karaoke.font_path should be local path, got: {result['karaoke']['font_path']}"
assert style_dir in result['intro']['background_image'], \
f"intro.background_image should be local path"
# Verify files actually exist locally
assert os.path.exists(result['karaoke']['background_image']), \
"Downloaded karaoke background should exist"
assert os.path.exists(result['karaoke']['font_path']), \
"Downloaded font should exist"
print(f"\n✅ Style loading from job.style_params_gcs_path works!")
print(f" karaoke.background_image: {result['karaoke']['background_image']}")
print(f" karaoke.font_path: {result['karaoke']['font_path']}")
def test_old_bug_state_data_lookup_fails(self):
"""
Demonstrate that the OLD bug would fail to find styles.
If we look at job.state_data['styles_gcs_path'] (the bug),
we won't find anything because styles are stored at
job.style_params_gcs_path.
"""
import time
job_id = f"test-bug-{int(time.time() * 1000)}"
# Upload style to CORRECT location
style_gcs_path = f"uploads/{job_id}/style/style_params.json"
self.storage.upload_string('{"karaoke": {"font_size": 100}}', style_gcs_path)
# Create job with styles in CORRECT location
job = MockJob(
job_id=job_id,
style_params_gcs_path=style_gcs_path, # CORRECT location
style_assets={},
state_data={} # No styles_gcs_path here!
)
# THE BUG: looking in state_data instead of style_params_gcs_path
wrong_path = job.state_data.get('styles_gcs_path')
correct_path = job.style_params_gcs_path
assert wrong_path is None, "state_data['styles_gcs_path'] should be None (the bug location)"
assert correct_path is not None, "style_params_gcs_path should have the correct path"
assert self.storage.file_exists(correct_path), "Style should exist at correct path"
print(f"\n✅ Demonstrated the bug: state_data lookup returns None")
print(f" state_data['styles_gcs_path'] = {wrong_path}")
print(f" job.style_params_gcs_path = {correct_path}")
def test_default_styles_when_no_custom_styles(self):
"""Test that default styles are used when no custom styles provided."""
import time
job_id = f"test-default-{int(time.time() * 1000)}"
# Job with NO custom styles
job = MockJob(
job_id=job_id,
style_params_gcs_path=None,
style_assets={},
state_data={}
)
# Simulate the logic
style_dir = os.path.join(self.temp_dir, "style")
os.makedirs(style_dir, exist_ok=True)
styles_path = os.path.join(style_dir, "styles.json")
if job.style_params_gcs_path:
# Would load custom styles
pass
else:
# Use default styles
default_styles = {
"karaoke": {
"background_color": "#000000",
"font": "Arial",
"font_path": "",
"font_size": 100
}
}
with open(styles_path, 'w') as f:
json.dump(default_styles, f, indent=2)
# Verify defaults were used
with open(styles_path, 'r') as f:
result = json.load(f)
assert result['karaoke']['background_color'] == "#000000"
assert result['karaoke']['font'] == "Arial"
print(f"\n✅ Default styles used when no custom styles provided")
class TestParallelWorkerExecution:
"""
Tests for the parallel worker execution fix.
The bug was: FastAPI's BackgroundTasks runs async tasks sequentially,
causing audio worker to complete before lyrics worker starts.
The fix uses asyncio.gather() to run both workers in parallel.
"""
def test_asyncio_gather_runs_parallel(self):
"""Test that asyncio.gather actually runs tasks in parallel."""
import asyncio
import time
start_times = {}
end_times = {}
async def task1():
start_times['task1'] = time.time()
await asyncio.sleep(0.1)
end_times['task1'] = time.time()
return "task1 done"
async def task2():
start_times['task2'] = time.time()
await asyncio.sleep(0.1)
end_times['task2'] = time.time()
return "task2 done"
async def run_parallel():
return await asyncio.gather(task1(), task2())
overall_start = time.time()
results = asyncio.run(run_parallel())
overall_duration = time.time() - overall_start
# Both tasks should have started at nearly the same time
start_diff = abs(start_times['task1'] - start_times['task2'])
# If running in parallel:
# - Both start at ~same time (diff < 0.05s)
# - Total duration is ~0.1s (not 0.2s for sequential)
assert start_diff < 0.05, \
f"Tasks should start together, diff was {start_diff:.3f}s"
assert overall_duration < 0.15, \
f"Parallel execution should take ~0.1s, took {overall_duration:.3f}s"
print(f"\n✅ asyncio.gather runs tasks in parallel!")
print(f" Task start time difference: {start_diff:.3f}s")
print(f" Total duration: {overall_duration:.3f}s (sequential would be ~0.2s)")
def test_sequential_background_tasks_is_slow(self):
"""
Demonstrate that sequential execution (the bug) is slower.
"""
import asyncio
import time
execution_order = []
async def task1():
execution_order.append(('task1', 'start'))
await asyncio.sleep(0.05)
execution_order.append(('task1', 'end'))
async def task2():
execution_order.append(('task2', 'start'))
await asyncio.sleep(0.05)
execution_order.append(('task2', 'end'))
# Sequential (the bug)
async def run_sequential():
await task1()
await task2()
execution_order.clear()
start = time.time()
asyncio.run(run_sequential())
sequential_duration = time.time() - start
sequential_order = execution_order.copy()
# Parallel (the fix)
async def run_parallel():
await asyncio.gather(task1(), task2())
execution_order.clear()
start = time.time()
asyncio.run(run_parallel())
parallel_duration = time.time() - start
parallel_order = execution_order.copy()
# Sequential: task1 starts, task1 ends, task2 starts, task2 ends
assert sequential_order[0] == ('task1', 'start')
assert sequential_order[1] == ('task1', 'end')
assert sequential_order[2] == ('task2', 'start')
# Parallel: task1 starts, task2 starts (interleaved)
assert parallel_order[0][1] == 'start'
assert parallel_order[1][1] == 'start'
# Parallel should be ~2x faster
assert parallel_duration < sequential_duration * 0.8, \
f"Parallel ({parallel_duration:.3f}s) should be faster than sequential ({sequential_duration:.3f}s)"
print(f"\n✅ Demonstrated sequential vs parallel execution")
print(f" Sequential: {sequential_duration:.3f}s - {sequential_order}")
print(f" Parallel: {parallel_duration:.3f}s - {parallel_order}")
print("✅ Style loading and parallel execution tests ready")
@@ -1,229 +0,0 @@
"""
Direct tests for worker logging Firestore operations.
These tests bypass the full app/worker imports and test the Firestore
operations directly, avoiding dependency issues.
Run with: ./scripts/run-emulator-tests.sh
"""
import pytest
import time
import requests
import os
import threading
from concurrent.futures import ThreadPoolExecutor, as_completed
from datetime import datetime
def emulators_running() -> bool:
"""Check if GCP emulators are running."""
try:
requests.get("http://127.0.0.1:8080", timeout=1)
return True
except (requests.exceptions.ConnectionError, requests.exceptions.Timeout):
return False
# Skip all tests in this module if emulators aren't running
pytestmark = pytest.mark.skipif(
not emulators_running(),
reason="GCP emulators not running. Start with: scripts/start-emulators.sh"
)
# Set up environment for emulator
os.environ["FIRESTORE_EMULATOR_HOST"] = "127.0.0.1:8080"
os.environ["GOOGLE_CLOUD_PROJECT"] = "test-project"
class TestWorkerLogsFirestoreDirect:
"""Direct Firestore tests for worker logs - no app imports needed."""
@pytest.fixture(autouse=True)
def setup_firestore(self):
"""Set up Firestore client for each test."""
from google.cloud import firestore
self.db = firestore.Client(project="test-project")
self.collection = "test-worker-logs"
yield
def _create_test_job(self):
"""Create a test job document."""
job_id = f"test-{int(time.time() * 1000)}"
doc_ref = self.db.collection(self.collection).document(job_id)
doc_ref.set({
"job_id": job_id,
"status": "pending",
"created_at": datetime.utcnow(),
"worker_logs": []
})
return job_id
def _append_log_read_modify_write(self, job_id: str, worker: str, message: str):
"""
OLD METHOD: Read-modify-write (has race condition).
This is what we were doing before.
"""
doc_ref = self.db.collection(self.collection).document(job_id)
doc = doc_ref.get()
if not doc.exists:
return
data = doc.to_dict()
logs = data.get("worker_logs", [])
logs.append({
"timestamp": datetime.utcnow().isoformat() + "Z",
"level": "INFO",
"worker": worker,
"message": message
})
doc_ref.update({"worker_logs": logs})
def _append_log_array_union(self, job_id: str, worker: str, message: str):
"""
NEW METHOD: ArrayUnion (atomic, no race condition).
This is what we're doing now.
"""
from google.cloud import firestore
doc_ref = self.db.collection(self.collection).document(job_id)
doc_ref.update({
"worker_logs": firestore.ArrayUnion([{
"timestamp": datetime.utcnow().isoformat() + "Z",
"level": "INFO",
"worker": worker,
"message": message
}])
})
def _get_logs(self, job_id: str):
"""Get logs from job document."""
doc_ref = self.db.collection(self.collection).document(job_id)
doc = doc_ref.get()
if not doc.exists:
return []
return doc.to_dict().get("worker_logs", [])
def test_array_union_single_write(self):
"""Test ArrayUnion works for single write."""
job_id = self._create_test_job()
self._append_log_array_union(job_id, "test", "Single log message")
time.sleep(0.1)
logs = self._get_logs(job_id)
assert len(logs) == 1
assert logs[0]["message"] == "Single log message"
assert logs[0]["worker"] == "test"
def test_array_union_sequential_writes(self):
"""Test ArrayUnion preserves all sequential writes."""
job_id = self._create_test_job()
for i in range(10):
self._append_log_array_union(job_id, "test", f"Log {i}")
time.sleep(0.2)
logs = self._get_logs(job_id)
assert len(logs) == 10, f"Expected 10 logs, got {len(logs)}"
def test_read_modify_write_race_condition(self):
"""
Demonstrate the race condition with read-modify-write.
This test shows WHY we needed ArrayUnion.
"""
job_id = self._create_test_job()
num_writes = 20
def write_log(index):
self._append_log_read_modify_write(job_id, "worker", f"RMW Log {index}")
# Write concurrently - should lose some logs due to race condition
with ThreadPoolExecutor(max_workers=5) as executor:
futures = [executor.submit(write_log, i) for i in range(num_writes)]
for future in as_completed(futures):
try:
future.result()
except Exception:
pass # Ignore errors for this test
time.sleep(0.5)
logs = self._get_logs(job_id)
# With read-modify-write, we likely lose some logs
# This is expected - the test documents the problem
rmw_count = len([l for l in logs if "RMW Log" in l.get("message", "")])
print(f"\nRead-modify-write: {rmw_count}/{num_writes} logs preserved")
# We expect to lose some logs (this is the bug we're fixing)
# If all 20 are there, the race condition didn't trigger (which is fine)
# The important thing is that ArrayUnion test below ALWAYS preserves all
def test_array_union_no_race_condition(self):
"""
Verify ArrayUnion preserves ALL concurrent writes.
This is the critical test.
"""
job_id = self._create_test_job()
num_writes = 20
def write_log(index):
self._append_log_array_union(job_id, "worker", f"ArrayUnion Log {index}")
# Write concurrently
with ThreadPoolExecutor(max_workers=5) as executor:
futures = [executor.submit(write_log, i) for i in range(num_writes)]
for future in as_completed(futures):
future.result() # Raise any exceptions
time.sleep(0.5)
logs = self._get_logs(job_id)
# With ArrayUnion, ALL logs should be preserved
au_count = len([l for l in logs if "ArrayUnion Log" in l.get("message", "")])
print(f"\nArrayUnion: {au_count}/{num_writes} logs preserved")
assert au_count == num_writes, \
f"ArrayUnion should preserve all {num_writes} logs, got {au_count}"
def test_concurrent_workers_array_union(self):
"""
Test simulating audio and lyrics workers writing concurrently.
"""
job_id = self._create_test_job()
def audio_worker():
for i in range(10):
self._append_log_array_union(job_id, "audio", f"Audio log {i}")
time.sleep(0.01)
def lyrics_worker():
for i in range(10):
self._append_log_array_union(job_id, "lyrics", f"Lyrics log {i}")
time.sleep(0.01)
# Start both workers
audio_thread = threading.Thread(target=audio_worker)
lyrics_thread = threading.Thread(target=lyrics_worker)
audio_thread.start()
lyrics_thread.start()
audio_thread.join()
lyrics_thread.join()
time.sleep(0.5)
logs = self._get_logs(job_id)
audio_logs = [l for l in logs if l.get("worker") == "audio"]
lyrics_logs = [l for l in logs if l.get("worker") == "lyrics"]
print(f"\nConcurrent workers: {len(audio_logs)} audio + {len(lyrics_logs)} lyrics")
assert len(audio_logs) == 10, f"Expected 10 audio logs, got {len(audio_logs)}"
assert len(lyrics_logs) == 10, f"Expected 10 lyrics logs, got {len(lyrics_logs)}"
print("✅ All logs from both workers preserved!")
print("✅ Direct Firestore worker logs tests ready")
-10
View File
@@ -1,10 +0,0 @@
# Testing dependencies
pytest>=7.4.0
pytest-asyncio>=0.21.0
pytest-cov>=4.1.0
pytest-mock>=3.11.1
httpx>=0.25.0 # For async testing
# Parent requirements (for import)
-r ../requirements.txt
-6
View File
@@ -1,6 +0,0 @@
pytest>=8.0.0
pytest-asyncio>=0.23.0
pytest-timeout>=2.2.0
pytest-cov>=4.1.0
requests>=2.31.0
-460
View File
@@ -1,460 +0,0 @@
"""
Integration tests for the karaoke generation backend API.
These tests verify the backend works end-to-end with real Cloud Run deployment.
These tests require a deployed backend and are marked to skip unless explicitly enabled.
Run with: pytest backend/tests/test_api_integration.py -m integration
"""
import pytest
import requests
import subprocess
import time
import json
from pathlib import Path
from typing import Optional, Dict, Any
import os
# Skip all tests unless explicitly running integration tests
pytestmark = pytest.mark.skipif(
os.environ.get("RUN_INTEGRATION_TESTS") != "true",
reason="Integration tests require deployed backend. Set RUN_INTEGRATION_TESTS=true to run."
)
# Configuration
SERVICE_URL = "https://karaoke-backend-718638054799.us-central1.run.app"
TEST_YOUTUBE_URL = "https://www.youtube.com/watch?v=dQw4w9WgXcQ" # Rick Astley - Never Gonna Give You Up
DEFAULT_TIMEOUT = 30 # seconds
def api_get(url: str, headers: Optional[Dict[str, str]] = None, **kwargs) -> requests.Response:
"""Make a GET request with default timeout."""
return requests.get(url, headers=headers, timeout=DEFAULT_TIMEOUT, **kwargs)
def api_post(url: str, headers: Optional[Dict[str, str]] = None, **kwargs) -> requests.Response:
"""Make a POST request with default timeout."""
return requests.post(url, headers=headers, timeout=DEFAULT_TIMEOUT, **kwargs)
def api_delete(url: str, headers: Optional[Dict[str, str]] = None, **kwargs) -> requests.Response:
"""Make a DELETE request with default timeout."""
return requests.delete(url, headers=headers, timeout=DEFAULT_TIMEOUT, **kwargs)
def get_auth_token() -> str:
"""Get authentication token for Cloud Run."""
result = subprocess.run(
["gcloud", "auth", "print-identity-token"],
capture_output=True,
text=True,
check=True
)
return result.stdout.strip()
@pytest.fixture
def auth_headers():
"""Get authentication headers for requests."""
token = get_auth_token()
return {
"Authorization": f"Bearer {token}",
"Content-Type": "application/json"
}
class TestHealthEndpoint:
"""Test health check endpoint."""
def test_health_check(self, auth_headers):
"""Test that health endpoint returns 200 OK."""
response = api_get(f"{SERVICE_URL}/api/health", headers=auth_headers)
assert response.status_code == 200
data = response.json()
assert data["status"] == "healthy"
assert data["service"] == "karaoke-gen-backend"
def test_health_check_without_auth(self):
"""Test that health endpoint requires authentication."""
response = api_get(f"{SERVICE_URL}/api/health")
assert response.status_code == 403
class TestRootEndpoint:
"""Test root endpoint."""
def test_root_endpoint(self, auth_headers):
"""Test root endpoint returns service info."""
response = api_get(SERVICE_URL, headers=auth_headers)
assert response.status_code == 200
data = response.json()
assert data["service"] == "karaoke-gen-backend"
assert data["status"] == "running"
assert "version" in data
class TestJobSubmission:
"""Test job submission workflows."""
def test_submit_job_with_youtube_url(self, auth_headers):
"""Test submitting a job with a YouTube URL."""
payload = {"url": TEST_YOUTUBE_URL}
response = api_post(
f"{SERVICE_URL}/api/jobs",
headers=auth_headers,
json=payload
)
assert response.status_code == 200
data = response.json()
assert data["status"] == "success"
assert "job_id" in data
assert len(data["job_id"]) > 0
assert "message" in data
# Store job_id for cleanup
return data["job_id"]
def test_submit_job_with_invalid_url(self, auth_headers):
"""Test that invalid URLs are rejected."""
payload = {"url": "not-a-url"}
response = api_post(
f"{SERVICE_URL}/api/jobs",
headers=auth_headers,
json=payload
)
assert response.status_code == 422 # Validation error
def test_submit_job_without_url(self, auth_headers):
"""Test that missing URL is rejected."""
payload = {}
response = api_post(
f"{SERVICE_URL}/api/jobs",
headers=auth_headers,
json=payload
)
assert response.status_code == 422 # Validation error
class TestJobRetrieval:
"""Test job retrieval and status checking."""
@pytest.fixture
def test_job_id(self, auth_headers):
"""Create a test job for retrieval tests."""
payload = {"url": TEST_YOUTUBE_URL}
response = api_post(
f"{SERVICE_URL}/api/jobs",
headers=auth_headers,
json=payload
)
assert response.status_code == 200
job_id = response.json()["job_id"]
yield job_id
# Cleanup
api_delete(
f"{SERVICE_URL}/api/jobs/{job_id}",
headers=auth_headers
)
def test_get_job_status(self, auth_headers, test_job_id):
"""Test retrieving job status."""
response = api_get(
f"{SERVICE_URL}/api/jobs/{test_job_id}",
headers=auth_headers
)
assert response.status_code == 200
data = response.json()
assert data["job_id"] == test_job_id
assert "status" in data
# Updated to use current JobStatus enum values
assert data["status"] in ["pending", "downloading", "separating_stage1",
"separating_stage2", "audio_complete",
"transcribing", "correcting", "lyrics_complete",
"generating_screens", "applying_padding",
"awaiting_review", "in_review", "review_complete",
"rendering_video", "awaiting_instrumental_selection",
"instrumental_selected", "generating_video",
"encoding", "packaging", "uploading", "notifying",
"complete", "failed", "cancelled"]
assert "progress" in data
assert "created_at" in data
assert "updated_at" in data
assert "timeline" in data
def test_get_nonexistent_job(self, auth_headers):
"""Test that requesting nonexistent job returns 404."""
fake_job_id = "nonexistent-job-id"
response = api_get(
f"{SERVICE_URL}/api/jobs/{fake_job_id}",
headers=auth_headers
)
assert response.status_code == 404
def test_list_jobs(self, auth_headers, test_job_id):
"""Test listing all jobs."""
response = api_get(
f"{SERVICE_URL}/api/jobs",
headers=auth_headers
)
assert response.status_code == 200
data = response.json()
assert isinstance(data, list)
# Our test job should be in the list
job_ids = [job["job_id"] for job in data]
assert test_job_id in job_ids
def test_list_jobs_with_status_filter(self, auth_headers):
"""Test filtering jobs by status."""
response = api_get(
f"{SERVICE_URL}/api/jobs?status=pending",
headers=auth_headers
)
assert response.status_code == 200
data = response.json()
assert isinstance(data, list)
# All returned jobs should be pending
for job in data:
assert job["status"] == "pending"
def test_list_jobs_with_limit(self, auth_headers):
"""Test limiting number of returned jobs."""
response = api_get(
f"{SERVICE_URL}/api/jobs?limit=5",
headers=auth_headers
)
assert response.status_code == 200
data = response.json()
assert isinstance(data, list)
assert len(data) <= 5
class TestJobDeletion:
"""Test job deletion."""
def test_delete_job(self, auth_headers):
"""Test deleting a job."""
# Create a job
payload = {"url": TEST_YOUTUBE_URL}
response = api_post(
f"{SERVICE_URL}/api/jobs",
headers=auth_headers,
json=payload
)
assert response.status_code == 200
job_id = response.json()["job_id"]
# Delete the job
response = api_delete(
f"{SERVICE_URL}/api/jobs/{job_id}",
headers=auth_headers
)
assert response.status_code == 200
data = response.json()
assert data["status"] == "success"
# Verify job is deleted
response = api_get(
f"{SERVICE_URL}/api/jobs/{job_id}",
headers=auth_headers
)
assert response.status_code == 404
def test_delete_job_without_files(self, auth_headers):
"""Test deleting a job without deleting files."""
# Create a job
payload = {"url": TEST_YOUTUBE_URL}
response = api_post(
f"{SERVICE_URL}/api/jobs",
headers=auth_headers,
json=payload
)
assert response.status_code == 200
job_id = response.json()["job_id"]
# Delete job but keep files
response = api_delete(
f"{SERVICE_URL}/api/jobs/{job_id}?delete_files=false",
headers=auth_headers
)
assert response.status_code == 200
data = response.json()
assert data["status"] == "success"
def test_delete_nonexistent_job(self, auth_headers):
"""Test deleting nonexistent job returns 404."""
fake_job_id = "nonexistent-job-id"
response = api_delete(
f"{SERVICE_URL}/api/jobs/{fake_job_id}",
headers=auth_headers
)
assert response.status_code == 404
class TestFileUpload:
"""Test file upload endpoint."""
def test_upload_audio_file(self, auth_headers, tmp_path):
"""Test uploading an audio file."""
# Create a small test file
test_file = tmp_path / "test_audio.mp3"
test_file.write_bytes(b"fake audio content for testing")
# Upload file
with open(test_file, "rb") as f:
files = {"file": ("test_audio.mp3", f, "audio/mpeg")}
data = {
"artist": "Test Artist",
"title": "Test Song"
}
# Remove Content-Type header for multipart/form-data
headers = {
"Authorization": auth_headers["Authorization"]
}
response = api_post(
f"{SERVICE_URL}/api/jobs/upload",
headers=headers,
files=files,
data=data
)
assert response.status_code == 200
result = response.json()
assert result["status"] == "success"
assert "job_id" in result
# Cleanup
api_delete(
f"{SERVICE_URL}/api/jobs/{result['job_id']}",
headers=auth_headers
)
def test_upload_invalid_file_type(self, auth_headers, tmp_path):
"""Test that invalid file types are rejected."""
# Create a test file with invalid extension
test_file = tmp_path / "test.txt"
test_file.write_bytes(b"not an audio file")
with open(test_file, "rb") as f:
files = {"file": ("test.txt", f, "text/plain")}
data = {
"artist": "Test Artist",
"title": "Test Song"
}
headers = {
"Authorization": auth_headers["Authorization"]
}
response = api_post(
f"{SERVICE_URL}/api/jobs/upload",
headers=headers,
files=files,
data=data
)
assert response.status_code == 400
def test_upload_without_metadata(self, auth_headers, tmp_path):
"""Test that uploads without artist/title are rejected."""
test_file = tmp_path / "test_audio.mp3"
test_file.write_bytes(b"fake audio content")
with open(test_file, "rb") as f:
files = {"file": ("test_audio.mp3", f, "audio/mpeg")}
headers = {
"Authorization": auth_headers["Authorization"]
}
response = api_post(
f"{SERVICE_URL}/api/jobs/upload",
headers=headers,
files=files
)
assert response.status_code == 422 # Validation error
class TestJobProcessing:
"""Test end-to-end job processing (long-running)."""
@pytest.mark.slow
@pytest.mark.integration
def test_complete_job_workflow(self, auth_headers):
"""
Test complete job workflow from submission to completion.
This test is slow and requires actual processing.
Run with: pytest -m slow
"""
# Submit job
payload = {"url": TEST_YOUTUBE_URL}
response = api_post(
f"{SERVICE_URL}/api/jobs",
headers=auth_headers,
json=payload
)
assert response.status_code == 200
job_id = response.json()["job_id"]
# Poll for completion (with timeout)
timeout = 600 # 10 minutes
start_time = time.time()
while time.time() - start_time < timeout:
response = api_get(
f"{SERVICE_URL}/api/jobs/{job_id}",
headers=auth_headers
)
assert response.status_code == 200
job = response.json()
status = job["status"]
if status == "complete":
# Verify outputs exist
assert "download_urls" in job
assert len(job["download_urls"]) > 0
break
elif status == "failed":
pytest.fail(f"Job failed: {job.get('error_message')}")
# Wait before next poll
time.sleep(10)
else:
pytest.fail(f"Job did not complete within {timeout} seconds")
# Cleanup
api_delete(
f"{SERVICE_URL}/api/jobs/{job_id}",
headers=auth_headers
)
if __name__ == "__main__":
pytest.main([__file__, "-v", "--tb=short"])
-93
View File
@@ -1,93 +0,0 @@
"""
Unit tests for API routes.
These tests use FastAPI TestClient with mocked services to test
route logic without hitting real cloud services.
"""
import pytest
import json
from datetime import datetime, UTC
from unittest.mock import MagicMock, AsyncMock, patch
from fastapi.testclient import TestClient
from io import BytesIO
from backend.models.job import Job, JobStatus
class TestHealthRoutes:
"""Tests for health.py routes."""
@pytest.fixture
def client(self):
"""Create test client with mocked dependencies."""
mock_creds = MagicMock()
mock_creds.universe_domain = 'googleapis.com'
with patch('backend.services.firestore_service.firestore'), \
patch('backend.services.storage_service.storage'), \
patch('google.auth.default', return_value=(mock_creds, 'test-project')):
from backend.main import app
return TestClient(app)
def test_health_endpoint_returns_200(self, client, auth_headers):
"""Test /api/health returns 200 OK."""
response = client.get("/api/health", )
assert response.status_code == 200
def test_health_endpoint_returns_healthy_status(self, client, auth_headers):
"""Test health endpoint returns healthy status."""
response = client.get("/api/health", )
data = response.json()
assert data["status"] == "healthy"
def test_root_endpoint_returns_200(self, client, auth_headers):
"""Test root endpoint returns 200."""
response = client.get("/", )
assert response.status_code == 200
class TestJobRoutes:
"""Tests for jobs.py routes.
Note: These tests verify the route module structure.
Full integration tests are in test_api_integration.py.
"""
def test_jobs_router_exists(self):
"""Test jobs router can be imported."""
from backend.api.routes import jobs
assert hasattr(jobs, 'router')
def test_jobs_router_has_expected_endpoints(self):
"""Test jobs router defines expected endpoints."""
from backend.api.routes.jobs import router
routes = [route.path for route in router.routes]
assert '/jobs' in routes or any('/jobs' in r for r in routes)
class TestInternalRoutes:
"""Tests for internal.py routes.
Note: These tests are minimal because the internal routes
trigger actual worker processing which requires complex mocking.
The actual worker logic is tested in test_workers.py.
"""
def test_internal_endpoint_structure(self):
"""Test internal route module has expected endpoints."""
from backend.api.routes import internal
# Just verify the module can be imported
assert hasattr(internal, 'router')
class TestFileUploadRoutes:
"""Tests for file_upload.py routes.
Note: These tests verify the route module structure.
Full integration tests are in test_api_integration.py.
"""
def test_file_upload_router_exists(self):
"""Test file upload router can be imported."""
from backend.api.routes import file_upload
assert hasattr(file_upload, 'router')
@@ -1,294 +0,0 @@
"""
Tests for audio_analysis_service.py - Audio analysis for backing vocals.
These tests mock the storage service and shared karaoke_gen modules to verify:
- GCS file download/upload operations
- Audio analysis delegation to shared AudioAnalyzer
- Waveform generation delegation to shared WaveformGenerator
"""
import pytest
from unittest.mock import Mock, MagicMock, patch, ANY
import tempfile
import os
class TestAudioAnalysisServiceInit:
"""Test AudioAnalysisService initialization."""
@patch("backend.services.audio_analysis_service.StorageService")
@patch("backend.services.audio_analysis_service.AudioAnalyzer")
@patch("backend.services.audio_analysis_service.WaveformGenerator")
def test_init_creates_dependencies(
self, mock_waveform_class, mock_analyzer_class, mock_storage_class
):
"""Test initialization creates storage service and analyzers."""
from backend.services.audio_analysis_service import AudioAnalysisService
service = AudioAnalysisService()
mock_storage_class.assert_called_once()
mock_analyzer_class.assert_called_once_with(
silence_threshold_db=-40.0,
min_segment_duration_ms=100,
)
mock_waveform_class.assert_called_once()
@patch("backend.services.audio_analysis_service.StorageService")
@patch("backend.services.audio_analysis_service.AudioAnalyzer")
@patch("backend.services.audio_analysis_service.WaveformGenerator")
def test_init_with_custom_params(
self, mock_waveform_class, mock_analyzer_class, mock_storage_class
):
"""Test initialization with custom threshold and duration."""
from backend.services.audio_analysis_service import AudioAnalysisService
service = AudioAnalysisService(
silence_threshold_db=-50.0,
min_segment_duration_ms=200,
)
mock_analyzer_class.assert_called_once_with(
silence_threshold_db=-50.0,
min_segment_duration_ms=200,
)
@patch("backend.services.audio_analysis_service.AudioAnalyzer")
@patch("backend.services.audio_analysis_service.WaveformGenerator")
def test_init_with_provided_storage_service(
self, mock_waveform_class, mock_analyzer_class
):
"""Test initialization with externally provided storage service."""
from backend.services.audio_analysis_service import AudioAnalysisService
mock_storage = Mock()
service = AudioAnalysisService(storage_service=mock_storage)
assert service.storage_service is mock_storage
class TestAnalyzeBackingVocals:
"""Test analyze_backing_vocals method."""
@patch("backend.services.audio_analysis_service.StorageService")
@patch("backend.services.audio_analysis_service.AudioAnalyzer")
@patch("backend.services.audio_analysis_service.WaveformGenerator")
def test_analyze_downloads_and_analyzes(
self, mock_waveform_class, mock_analyzer_class, mock_storage_class
):
"""Test analyze_backing_vocals downloads file and runs analysis."""
from backend.services.audio_analysis_service import AudioAnalysisService
# Setup mocks
mock_storage = Mock()
mock_storage_class.return_value = mock_storage
mock_result = Mock()
mock_result.has_audible_content = True
mock_result.segment_count = 5
mock_result.recommended_selection = Mock(value="with_backing")
mock_analyzer = Mock()
mock_analyzer.analyze.return_value = mock_result
mock_analyzer_class.return_value = mock_analyzer
service = AudioAnalysisService()
# Call method
result = service.analyze_backing_vocals(
gcs_audio_path="uploads/job123/backing_vocals.flac",
job_id="job123",
)
# Verify storage download was called
mock_storage.download_file.assert_called_once()
call_args = mock_storage.download_file.call_args
assert call_args[0][0] == "uploads/job123/backing_vocals.flac"
# Verify analyzer was called
mock_analyzer.analyze.assert_called_once()
# Verify result
assert result is mock_result
assert result.has_audible_content is True
@patch("backend.services.audio_analysis_service.StorageService")
@patch("backend.services.audio_analysis_service.AudioAnalyzer")
@patch("backend.services.audio_analysis_service.WaveformGenerator")
def test_analyze_cleans_up_temp_files(
self, mock_waveform_class, mock_analyzer_class, mock_storage_class
):
"""Test that temp files are cleaned up after analysis."""
from backend.services.audio_analysis_service import AudioAnalysisService
mock_storage = Mock()
mock_storage_class.return_value = mock_storage
mock_result = Mock()
mock_result.has_audible_content = False
mock_result.segment_count = 0
mock_result.recommended_selection = Mock(value="clean")
mock_analyzer = Mock()
mock_analyzer.analyze.return_value = mock_result
mock_analyzer_class.return_value = mock_analyzer
service = AudioAnalysisService()
# Call method
service.analyze_backing_vocals(
gcs_audio_path="uploads/job123/backing.flac",
job_id="job123",
)
# Temp directory should be cleaned up (no assertion needed,
# TemporaryDirectory context manager handles this)
# Just verify the method completes without error
class TestAnalyzeAndGenerateWaveform:
"""Test analyze_and_generate_waveform method."""
@patch("backend.services.audio_analysis_service.StorageService")
@patch("backend.services.audio_analysis_service.AudioAnalyzer")
@patch("backend.services.audio_analysis_service.WaveformGenerator")
def test_analyze_and_generate_waveform(
self, mock_waveform_class, mock_analyzer_class, mock_storage_class
):
"""Test analyze_and_generate_waveform does both operations."""
from backend.services.audio_analysis_service import AudioAnalysisService
mock_storage = Mock()
mock_storage_class.return_value = mock_storage
mock_result = Mock()
mock_result.has_audible_content = True
mock_result.audible_segments = [Mock(), Mock()]
mock_analyzer = Mock()
mock_analyzer.analyze.return_value = mock_result
mock_analyzer.silence_threshold_db = -40.0
mock_analyzer_class.return_value = mock_analyzer
mock_waveform = Mock()
mock_waveform_class.return_value = mock_waveform
service = AudioAnalysisService()
# Call method
result, waveform_path = service.analyze_and_generate_waveform(
gcs_audio_path="uploads/job123/backing.flac",
job_id="job123",
gcs_waveform_destination="uploads/job123/waveform.png",
)
# Verify analysis was performed
mock_analyzer.analyze.assert_called_once()
# Verify waveform was generated
mock_waveform.generate.assert_called_once()
call_kwargs = mock_waveform.generate.call_args.kwargs
assert call_kwargs["segments"] == mock_result.audible_segments
assert call_kwargs["show_time_axis"] is True
# Verify waveform was uploaded
assert mock_storage.upload_file.call_count == 1
# Verify return values
assert result is mock_result
assert waveform_path == "uploads/job123/waveform.png"
class TestGetWaveformData:
"""Test get_waveform_data method."""
@patch("backend.services.audio_analysis_service.StorageService")
@patch("backend.services.audio_analysis_service.AudioAnalyzer")
@patch("backend.services.audio_analysis_service.WaveformGenerator")
def test_get_waveform_data(
self, mock_waveform_class, mock_analyzer_class, mock_storage_class
):
"""Test get_waveform_data returns amplitude data."""
from backend.services.audio_analysis_service import AudioAnalysisService
mock_storage = Mock()
mock_storage_class.return_value = mock_storage
mock_waveform = Mock()
mock_waveform.generate_data_only.return_value = (
[0.1, 0.3, 0.5, 0.3, 0.1], # amplitudes
180.5, # duration
)
mock_waveform_class.return_value = mock_waveform
service = AudioAnalysisService()
amplitudes, duration = service.get_waveform_data(
gcs_audio_path="uploads/job123/backing.flac",
job_id="job123",
num_points=500,
)
# Verify storage download
mock_storage.download_file.assert_called_once()
# Verify waveform generator called
mock_waveform.generate_data_only.assert_called_once()
call_kwargs = mock_waveform.generate_data_only.call_args.kwargs
assert call_kwargs["num_points"] == 500
# Verify return values
assert amplitudes == [0.1, 0.3, 0.5, 0.3, 0.1]
assert duration == 180.5
class TestGenerateWaveformWithMutes:
"""Test generate_waveform_with_mutes method."""
@patch("backend.services.audio_analysis_service.StorageService")
@patch("backend.services.audio_analysis_service.AudioAnalyzer")
@patch("backend.services.audio_analysis_service.WaveformGenerator")
def test_generate_waveform_with_mutes(
self, mock_waveform_class, mock_analyzer_class, mock_storage_class
):
"""Test generate_waveform_with_mutes highlights mute regions."""
from backend.services.audio_analysis_service import AudioAnalysisService
mock_storage = Mock()
mock_storage_class.return_value = mock_storage
mock_result = Mock()
mock_result.audible_segments = [Mock(), Mock()]
mock_analyzer = Mock()
mock_analyzer.analyze.return_value = mock_result
mock_analyzer_class.return_value = mock_analyzer
mock_waveform = Mock()
mock_waveform_class.return_value = mock_waveform
# Create mock mute regions
mock_mute1 = Mock()
mock_mute2 = Mock()
mute_regions = [mock_mute1, mock_mute2]
service = AudioAnalysisService()
result_path = service.generate_waveform_with_mutes(
gcs_audio_path="uploads/job123/backing.flac",
job_id="job123",
gcs_waveform_destination="uploads/job123/waveform_muted.png",
mute_regions=mute_regions,
)
# Verify waveform generator was called with mute_regions
mock_waveform.generate.assert_called_once()
call_kwargs = mock_waveform.generate.call_args.kwargs
assert call_kwargs["mute_regions"] == mute_regions
assert call_kwargs["segments"] == mock_result.audible_segments
# Verify upload
mock_storage.upload_file.assert_called_once()
# Verify return
assert result_path == "uploads/job123/waveform_muted.png"
-386
View File
@@ -1,386 +0,0 @@
"""
Tests for audio_editing_service.py - Custom instrumental creation.
These tests mock the storage service and shared karaoke_gen modules to verify:
- GCS file download/upload operations
- Audio editing delegation to shared AudioEditor
- Mute region validation
"""
import pytest
from unittest.mock import Mock, MagicMock, patch, ANY
class TestAudioEditingServiceInit:
"""Test AudioEditingService initialization."""
@patch("backend.services.audio_editing_service.StorageService")
@patch("backend.services.audio_editing_service.AudioEditor")
def test_init_creates_dependencies(self, mock_editor_class, mock_storage_class):
"""Test initialization creates storage service and editor."""
from backend.services.audio_editing_service import AudioEditingService
service = AudioEditingService()
mock_storage_class.assert_called_once()
mock_editor_class.assert_called_once_with(output_format="flac")
@patch("backend.services.audio_editing_service.StorageService")
@patch("backend.services.audio_editing_service.AudioEditor")
def test_init_with_custom_format(self, mock_editor_class, mock_storage_class):
"""Test initialization with custom output format."""
from backend.services.audio_editing_service import AudioEditingService
service = AudioEditingService(output_format="wav")
mock_editor_class.assert_called_once_with(output_format="wav")
@patch("backend.services.audio_editing_service.AudioEditor")
def test_init_with_provided_storage_service(self, mock_editor_class):
"""Test initialization with externally provided storage service."""
from backend.services.audio_editing_service import AudioEditingService
mock_storage = Mock()
service = AudioEditingService(storage_service=mock_storage)
assert service.storage_service is mock_storage
class TestCreateCustomInstrumental:
"""Test create_custom_instrumental method."""
@patch("backend.services.audio_editing_service.StorageService")
@patch("backend.services.audio_editing_service.AudioEditor")
def test_create_custom_instrumental_success(
self, mock_editor_class, mock_storage_class
):
"""Test creating a custom instrumental successfully."""
from backend.services.audio_editing_service import AudioEditingService
mock_storage = Mock()
mock_storage_class.return_value = mock_storage
mock_result = Mock()
mock_result.total_muted_duration_seconds = 15.5
mock_result.mute_regions_applied = [Mock(), Mock(), Mock()]
mock_result.output_path = None # Will be set by service
mock_editor = Mock()
mock_editor.create_custom_instrumental.return_value = mock_result
mock_editor_class.return_value = mock_editor
service = AudioEditingService()
# Create mock mute regions
mock_mute1 = Mock()
mock_mute2 = Mock()
mute_regions = [mock_mute1, mock_mute2]
result = service.create_custom_instrumental(
gcs_clean_instrumental_path="uploads/job123/clean.flac",
gcs_backing_vocals_path="uploads/job123/backing.flac",
mute_regions=mute_regions,
gcs_output_path="uploads/job123/custom.flac",
job_id="job123",
)
# Verify downloads
assert mock_storage.download_file.call_count == 2
# Verify editor was called
mock_editor.create_custom_instrumental.assert_called_once()
call_kwargs = mock_editor.create_custom_instrumental.call_args.kwargs
assert call_kwargs["mute_regions"] == mute_regions
# Verify upload
mock_storage.upload_file.assert_called_once()
# Verify result path was updated
assert result.output_path == "uploads/job123/custom.flac"
@patch("backend.services.audio_editing_service.StorageService")
@patch("backend.services.audio_editing_service.AudioEditor")
def test_create_custom_instrumental_downloads_both_files(
self, mock_editor_class, mock_storage_class
):
"""Test that both input files are downloaded."""
from backend.services.audio_editing_service import AudioEditingService
mock_storage = Mock()
mock_storage_class.return_value = mock_storage
mock_result = Mock()
mock_result.total_muted_duration_seconds = 0
mock_result.mute_regions_applied = []
mock_editor = Mock()
mock_editor.create_custom_instrumental.return_value = mock_result
mock_editor_class.return_value = mock_editor
service = AudioEditingService()
service.create_custom_instrumental(
gcs_clean_instrumental_path="uploads/job123/clean.flac",
gcs_backing_vocals_path="uploads/job123/backing.flac",
mute_regions=[],
gcs_output_path="uploads/job123/output.flac",
job_id="job123",
)
# Verify both files were downloaded
download_calls = mock_storage.download_file.call_args_list
assert len(download_calls) == 2
# First call should be clean instrumental
assert download_calls[0][0][0] == "uploads/job123/clean.flac"
# Second call should be backing vocals
assert download_calls[1][0][0] == "uploads/job123/backing.flac"
class TestCreatePreview:
"""Test create_preview method."""
@patch("pydub.AudioSegment")
@patch("backend.services.audio_editing_service.StorageService")
@patch("backend.services.audio_editing_service.AudioEditor")
def test_create_preview_success(
self, mock_editor_class, mock_storage_class, mock_audio_segment
):
"""Test creating a preview successfully."""
from backend.services.audio_editing_service import AudioEditingService
mock_storage = Mock()
mock_storage_class.return_value = mock_storage
mock_preview = Mock()
mock_preview.export = Mock()
mock_preview.__getitem__ = Mock(return_value=mock_preview)
mock_editor = Mock()
mock_editor.preview_with_mutes.return_value = mock_preview
mock_editor_class.return_value = mock_editor
service = AudioEditingService()
result = service.create_preview(
gcs_clean_instrumental_path="uploads/job123/clean.flac",
gcs_backing_vocals_path="uploads/job123/backing.flac",
mute_regions=[Mock(), Mock()],
gcs_preview_path="uploads/job123/preview.flac",
job_id="job123",
)
# Verify editor was called
mock_editor.preview_with_mutes.assert_called_once()
# Verify result
assert result == "uploads/job123/preview.flac"
@patch("pydub.AudioSegment")
@patch("backend.services.audio_editing_service.StorageService")
@patch("backend.services.audio_editing_service.AudioEditor")
def test_create_preview_with_duration_limit(
self, mock_editor_class, mock_storage_class, mock_audio_segment
):
"""Test creating a preview with duration limit."""
from backend.services.audio_editing_service import AudioEditingService
mock_storage = Mock()
mock_storage_class.return_value = mock_storage
mock_preview = Mock()
mock_truncated = Mock()
mock_preview.__getitem__ = Mock(return_value=mock_truncated)
mock_truncated.export = Mock()
mock_editor = Mock()
mock_editor.preview_with_mutes.return_value = mock_preview
mock_editor_class.return_value = mock_editor
service = AudioEditingService()
service.create_preview(
gcs_clean_instrumental_path="uploads/job123/clean.flac",
gcs_backing_vocals_path="uploads/job123/backing.flac",
mute_regions=[],
gcs_preview_path="uploads/job123/preview.flac",
job_id="job123",
preview_duration_seconds=30.0,
)
# Verify preview was truncated (30 seconds = 30000ms)
mock_preview.__getitem__.assert_called_once_with(slice(None, 30000))
class TestMuteBackingVocalsOnly:
"""Test mute_backing_vocals_only method."""
@patch("backend.services.audio_editing_service.StorageService")
@patch("backend.services.audio_editing_service.AudioEditor")
def test_mute_backing_vocals_only(self, mock_editor_class, mock_storage_class):
"""Test muting backing vocals without combining."""
from backend.services.audio_editing_service import AudioEditingService
mock_storage = Mock()
mock_storage_class.return_value = mock_storage
mock_editor = Mock()
mock_editor_class.return_value = mock_editor
service = AudioEditingService()
mute_regions = [Mock(), Mock()]
result = service.mute_backing_vocals_only(
gcs_backing_vocals_path="uploads/job123/backing.flac",
mute_regions=mute_regions,
gcs_output_path="uploads/job123/muted_backing.flac",
job_id="job123",
)
# Verify only one file was downloaded
mock_storage.download_file.assert_called_once()
assert mock_storage.download_file.call_args[0][0] == "uploads/job123/backing.flac"
# Verify editor method was called
mock_editor.apply_mute_to_single_track.assert_called_once()
call_kwargs = mock_editor.apply_mute_to_single_track.call_args.kwargs
assert call_kwargs["mute_regions"] == mute_regions
# Verify upload
mock_storage.upload_file.assert_called_once()
# Verify result
assert result == "uploads/job123/muted_backing.flac"
class TestValidateMuteRegions:
"""Test validate_mute_regions method.
Note: MuteRegion uses pydantic validation that rejects invalid values at
creation time. These tests use Mock objects to test the validation logic
in the service, simulating edge cases that would normally be rejected.
"""
@patch("backend.services.audio_editing_service.StorageService")
@patch("backend.services.audio_editing_service.AudioEditor")
def test_valid_mute_regions(self, mock_editor_class, mock_storage_class):
"""Test validation passes for valid mute regions."""
from backend.services.audio_editing_service import AudioEditingService
from karaoke_gen.instrumental_review import MuteRegion
service = AudioEditingService()
mute_regions = [
MuteRegion(start_seconds=10.0, end_seconds=15.0),
MuteRegion(start_seconds=30.0, end_seconds=45.0),
]
errors = service.validate_mute_regions(mute_regions, total_duration_seconds=180.0)
assert errors == []
@patch("backend.services.audio_editing_service.StorageService")
@patch("backend.services.audio_editing_service.AudioEditor")
def test_negative_start_seconds(self, mock_editor_class, mock_storage_class):
"""Test validation catches negative start time.
Uses Mock since MuteRegion pydantic model rejects negative values.
"""
from backend.services.audio_editing_service import AudioEditingService
service = AudioEditingService()
# Use mock to simulate invalid region (pydantic would reject this)
mock_region = Mock()
mock_region.start_seconds = -5.0
mock_region.end_seconds = 10.0
errors = service.validate_mute_regions([mock_region], total_duration_seconds=180.0)
assert len(errors) == 1
assert "cannot be negative" in errors[0]
@patch("backend.services.audio_editing_service.StorageService")
@patch("backend.services.audio_editing_service.AudioEditor")
def test_end_before_start(self, mock_editor_class, mock_storage_class):
"""Test validation catches end time before start time."""
from backend.services.audio_editing_service import AudioEditingService
service = AudioEditingService()
# Use mock to simulate invalid region
mock_region = Mock()
mock_region.start_seconds = 30.0
mock_region.end_seconds = 20.0
errors = service.validate_mute_regions([mock_region], total_duration_seconds=180.0)
assert len(errors) == 1
assert "must be after" in errors[0]
@patch("backend.services.audio_editing_service.StorageService")
@patch("backend.services.audio_editing_service.AudioEditor")
def test_start_exceeds_duration(self, mock_editor_class, mock_storage_class):
"""Test validation catches start time exceeding audio duration."""
from backend.services.audio_editing_service import AudioEditingService
from karaoke_gen.instrumental_review import MuteRegion
service = AudioEditingService()
mute_regions = [
MuteRegion(start_seconds=200.0, end_seconds=210.0),
]
errors = service.validate_mute_regions(mute_regions, total_duration_seconds=180.0)
assert len(errors) == 1
assert "exceeds audio duration" in errors[0]
@patch("backend.services.audio_editing_service.StorageService")
@patch("backend.services.audio_editing_service.AudioEditor")
def test_end_exceeds_duration_is_warning_not_error(
self, mock_editor_class, mock_storage_class
):
"""Test that end time exceeding duration is logged as warning, not error."""
from backend.services.audio_editing_service import AudioEditingService
from karaoke_gen.instrumental_review import MuteRegion
service = AudioEditingService()
mute_regions = [
MuteRegion(start_seconds=170.0, end_seconds=200.0), # End exceeds 180s
]
errors = service.validate_mute_regions(mute_regions, total_duration_seconds=180.0)
# Should not be an error (will be clamped)
assert len(errors) == 0
@patch("backend.services.audio_editing_service.StorageService")
@patch("backend.services.audio_editing_service.AudioEditor")
def test_multiple_errors(self, mock_editor_class, mock_storage_class):
"""Test validation reports multiple errors."""
from backend.services.audio_editing_service import AudioEditingService
from karaoke_gen.instrumental_review import MuteRegion
service = AudioEditingService()
# Use mocks for invalid regions
mock_region1 = Mock()
mock_region1.start_seconds = -5.0
mock_region1.end_seconds = 10.0
mock_region2 = Mock()
mock_region2.start_seconds = 30.0
mock_region2.end_seconds = 20.0 # End before start
mock_region3 = Mock()
mock_region3.start_seconds = 200.0 # Exceeds duration
mock_region3.end_seconds = 210.0
mute_regions = [mock_region1, mock_region2, mock_region3]
errors = service.validate_mute_regions(mute_regions, total_duration_seconds=180.0)
assert len(errors) == 3
File diff suppressed because it is too large. Load diff
-378
View File
@@ -1,378 +0,0 @@
"""
Unit tests for audio analysis and editing services.
These tests verify the backend service wrappers that use the shared
karaoke_gen.instrumental_review module with GCS integration.
NOTE: These tests require ffmpeg to be installed. They will be skipped
if ffmpeg is not available (e.g., in CI environments without ffmpeg).
"""
import os
import shutil
import tempfile
import pytest
from unittest.mock import MagicMock, patch, ANY
# Check if ffmpeg is available
def _ffmpeg_available():
"""Check if ffmpeg is available on the system."""
return shutil.which('ffmpeg') is not None
# Skip all tests in this module if ffmpeg is not available
pytestmark = pytest.mark.skipif(
not _ffmpeg_available(),
reason="ffmpeg not available - required for audio processing tests"
)
from pydub import AudioSegment
from pydub.generators import Sine
from karaoke_gen.instrumental_review import MuteRegion
class TestAudioAnalysisService:
"""Tests for AudioAnalysisService."""
@pytest.fixture
def mock_storage_service(self):
"""Create a mock storage service."""
mock = MagicMock()
return mock
@pytest.fixture
def temp_audio_file(self, tmp_path):
"""Create a temporary audio file for testing."""
audio_path = tmp_path / "test_audio.flac"
# Create a simple sine wave audio
tone = Sine(440).to_audio_segment(duration=5000) - 15
tone.export(str(audio_path), format="flac")
return str(audio_path)
@pytest.fixture
def silent_audio_file(self, tmp_path):
"""Create a silent audio file for testing."""
audio_path = tmp_path / "silent_audio.flac"
audio = AudioSegment.silent(duration=5000, frame_rate=44100)
audio.export(str(audio_path), format="flac")
return str(audio_path)
def test_service_initialization(self, mock_storage_service):
"""Test service initializes with default parameters."""
from backend.services.audio_analysis_service import AudioAnalysisService
service = AudioAnalysisService(storage_service=mock_storage_service)
assert service.storage_service == mock_storage_service
assert service.analyzer.silence_threshold_db == -40.0
def test_service_custom_threshold(self, mock_storage_service):
"""Test service accepts custom silence threshold."""
from backend.services.audio_analysis_service import AudioAnalysisService
service = AudioAnalysisService(
storage_service=mock_storage_service,
silence_threshold_db=-35.0,
)
assert service.analyzer.silence_threshold_db == -35.0
def test_analyze_backing_vocals_downloads_file(
self, mock_storage_service, temp_audio_file
):
"""Test that analyze_backing_vocals downloads file from GCS."""
from backend.services.audio_analysis_service import AudioAnalysisService
# Set up mock to copy the temp file to the download location
def mock_download(gcs_path, local_path):
import shutil
shutil.copy(temp_audio_file, local_path)
return local_path
mock_storage_service.download_file.side_effect = mock_download
service = AudioAnalysisService(storage_service=mock_storage_service)
result = service.analyze_backing_vocals(
gcs_audio_path="jobs/test/stems/backing_vocals.flac",
job_id="test-job",
)
# Verify download was called
mock_storage_service.download_file.assert_called_once()
# Verify result is valid
assert result is not None
assert hasattr(result, 'has_audible_content')
def test_analyze_silent_audio_returns_no_audible(
self, mock_storage_service, silent_audio_file
):
"""Test that silent audio returns has_audible_content=False."""
from backend.services.audio_analysis_service import AudioAnalysisService
def mock_download(gcs_path, local_path):
import shutil
shutil.copy(silent_audio_file, local_path)
return local_path
mock_storage_service.download_file.side_effect = mock_download
service = AudioAnalysisService(storage_service=mock_storage_service)
result = service.analyze_backing_vocals(
gcs_audio_path="jobs/test/stems/backing_vocals.flac",
job_id="test-job",
)
assert result.has_audible_content is False
assert len(result.audible_segments) == 0
def test_analyze_and_generate_waveform_uploads_image(
self, mock_storage_service, temp_audio_file
):
"""Test that analyze_and_generate_waveform uploads waveform to GCS."""
from backend.services.audio_analysis_service import AudioAnalysisService
def mock_download(gcs_path, local_path):
import shutil
shutil.copy(temp_audio_file, local_path)
return local_path
mock_storage_service.download_file.side_effect = mock_download
mock_storage_service.upload_file.return_value = "jobs/test/analysis/waveform.png"
service = AudioAnalysisService(storage_service=mock_storage_service)
result, waveform_path = service.analyze_and_generate_waveform(
gcs_audio_path="jobs/test/stems/backing_vocals.flac",
job_id="test-job",
gcs_waveform_destination="jobs/test/analysis/waveform.png",
)
# Verify upload was called
mock_storage_service.upload_file.assert_called_once()
# Verify waveform path is returned
assert waveform_path == "jobs/test/analysis/waveform.png"
def test_get_waveform_data_returns_amplitudes(
self, mock_storage_service, temp_audio_file
):
"""Test that get_waveform_data returns amplitude data."""
from backend.services.audio_analysis_service import AudioAnalysisService
def mock_download(gcs_path, local_path):
import shutil
shutil.copy(temp_audio_file, local_path)
return local_path
mock_storage_service.download_file.side_effect = mock_download
service = AudioAnalysisService(storage_service=mock_storage_service)
amplitudes, duration = service.get_waveform_data(
gcs_audio_path="jobs/test/stems/backing_vocals.flac",
job_id="test-job",
num_points=100,
)
# Verify we get amplitude data
assert isinstance(amplitudes, list)
assert len(amplitudes) > 0
assert all(isinstance(a, float) for a in amplitudes)
# Verify duration is reasonable
assert duration > 0
assert duration <= 10 # Our test file is 5 seconds
class TestAudioEditingService:
"""Tests for AudioEditingService."""
@pytest.fixture
def mock_storage_service(self):
"""Create a mock storage service."""
mock = MagicMock()
return mock
@pytest.fixture
def temp_audio_files(self, tmp_path):
"""Create temporary audio files for testing."""
clean_path = tmp_path / "clean_instrumental.flac"
backing_path = tmp_path / "backing_vocals.flac"
# Create clean instrumental (chord)
tone1 = Sine(220).to_audio_segment(duration=5000) - 15
tone2 = Sine(330).to_audio_segment(duration=5000) - 15
clean = tone1.overlay(tone2)
clean.export(str(clean_path), format="flac")
# Create backing vocals
backing = Sine(440).to_audio_segment(duration=5000) - 20
backing.export(str(backing_path), format="flac")
return str(clean_path), str(backing_path)
def test_service_initialization(self, mock_storage_service):
"""Test service initializes correctly."""
from backend.services.audio_editing_service import AudioEditingService
service = AudioEditingService(storage_service=mock_storage_service)
assert service.storage_service == mock_storage_service
assert service.editor.output_format == "flac"
def test_create_custom_instrumental_downloads_files(
self, mock_storage_service, temp_audio_files
):
"""Test that create_custom_instrumental downloads input files."""
from backend.services.audio_editing_service import AudioEditingService
clean_path, backing_path = temp_audio_files
# Track which files were downloaded
downloaded_files = []
def mock_download(gcs_path, local_path):
downloaded_files.append(gcs_path)
# Copy the appropriate test file
import shutil
if 'clean' in gcs_path:
shutil.copy(clean_path, local_path)
else:
shutil.copy(backing_path, local_path)
return local_path
mock_storage_service.download_file.side_effect = mock_download
mock_storage_service.upload_file.return_value = "jobs/test/stems/custom.flac"
service = AudioEditingService(storage_service=mock_storage_service)
mute_regions = [MuteRegion(start_seconds=1.0, end_seconds=2.0)]
result = service.create_custom_instrumental(
gcs_clean_instrumental_path="jobs/test/stems/clean.flac",
gcs_backing_vocals_path="jobs/test/stems/backing.flac",
mute_regions=mute_regions,
gcs_output_path="jobs/test/stems/custom.flac",
job_id="test-job",
)
# Verify both files were downloaded
assert len(downloaded_files) == 2
assert mock_storage_service.upload_file.called
def test_create_custom_instrumental_uploads_result(
self, mock_storage_service, temp_audio_files
):
"""Test that create_custom_instrumental uploads the result."""
from backend.services.audio_editing_service import AudioEditingService
clean_path, backing_path = temp_audio_files
def mock_download(gcs_path, local_path):
import shutil
if 'clean' in gcs_path:
shutil.copy(clean_path, local_path)
else:
shutil.copy(backing_path, local_path)
return local_path
mock_storage_service.download_file.side_effect = mock_download
mock_storage_service.upload_file.return_value = "jobs/test/stems/custom.flac"
service = AudioEditingService(storage_service=mock_storage_service)
mute_regions = [MuteRegion(start_seconds=1.0, end_seconds=2.0)]
result = service.create_custom_instrumental(
gcs_clean_instrumental_path="jobs/test/stems/clean.flac",
gcs_backing_vocals_path="jobs/test/stems/backing.flac",
mute_regions=mute_regions,
gcs_output_path="jobs/test/stems/custom.flac",
job_id="test-job",
)
# Verify upload was called with correct destination
mock_storage_service.upload_file.assert_called_once()
call_args = mock_storage_service.upload_file.call_args
assert call_args[0][1] == "jobs/test/stems/custom.flac"
# Verify result has correct path
assert result.output_path == "jobs/test/stems/custom.flac"
def test_create_custom_instrumental_returns_statistics(
self, mock_storage_service, temp_audio_files
):
"""Test that create_custom_instrumental returns correct statistics."""
from backend.services.audio_editing_service import AudioEditingService
clean_path, backing_path = temp_audio_files
def mock_download(gcs_path, local_path):
import shutil
if 'clean' in gcs_path:
shutil.copy(clean_path, local_path)
else:
shutil.copy(backing_path, local_path)
return local_path
mock_storage_service.download_file.side_effect = mock_download
mock_storage_service.upload_file.return_value = "jobs/test/stems/custom.flac"
service = AudioEditingService(storage_service=mock_storage_service)
mute_regions = [
MuteRegion(start_seconds=1.0, end_seconds=2.0),
MuteRegion(start_seconds=3.0, end_seconds=4.0),
]
result = service.create_custom_instrumental(
gcs_clean_instrumental_path="jobs/test/stems/clean.flac",
gcs_backing_vocals_path="jobs/test/stems/backing.flac",
mute_regions=mute_regions,
gcs_output_path="jobs/test/stems/custom.flac",
job_id="test-job",
)
# Verify statistics
assert len(result.mute_regions_applied) == 2
assert result.total_muted_duration_seconds == 2.0 # 1s + 1s
assert result.output_duration_seconds > 0
def test_validate_mute_regions_returns_errors_for_invalid(
self, mock_storage_service
):
"""Test that validate_mute_regions catches invalid regions."""
from backend.services.audio_editing_service import AudioEditingService
service = AudioEditingService(storage_service=mock_storage_service)
mute_regions = [
MuteRegion(start_seconds=0.0, end_seconds=1.0), # Valid
MuteRegion(start_seconds=100.0, end_seconds=110.0), # Beyond duration
]
errors = service.validate_mute_regions(mute_regions, total_duration_seconds=60.0)
# Should have error for region exceeding duration
assert len(errors) == 1
assert "exceeds" in errors[0].lower()
def test_validate_mute_regions_returns_empty_for_valid(
self, mock_storage_service
):
"""Test that validate_mute_regions returns empty for valid regions."""
from backend.services.audio_editing_service import AudioEditingService
service = AudioEditingService(storage_service=mock_storage_service)
mute_regions = [
MuteRegion(start_seconds=0.0, end_seconds=1.0),
MuteRegion(start_seconds=10.0, end_seconds=15.0),
]
errors = service.validate_mute_regions(mute_regions, total_duration_seconds=60.0)
assert len(errors) == 0
-143
View File
@@ -1,143 +0,0 @@
"""
Tests for AuthService and FirestoreService.
These tests focus on the service initialization and basic operations.
"""
import pytest
from datetime import datetime, UTC
from unittest.mock import MagicMock, patch
class TestAuthServiceBasics:
"""Basic tests for AuthService."""
def test_auth_service_can_import(self):
"""Test AuthService can be imported."""
from backend.services.auth_service import AuthService
assert AuthService is not None
def test_auth_service_initialization(self):
"""Test AuthService initializes with mocked dependencies."""
mock_firestore = MagicMock()
with patch('backend.services.auth_service.FirestoreService', return_value=mock_firestore), \
patch.dict('os.environ', {'ADMIN_TOKENS': 'test-token'}):
from backend.services.auth_service import AuthService
service = AuthService()
assert service is not None
def test_auth_service_has_validate_method(self):
"""Test AuthService has token validation method."""
from backend.services.auth_service import AuthService
assert hasattr(AuthService, 'validate_token') or hasattr(AuthService, 'verify_token')
class TestFirestoreServiceBasics:
"""Basic tests for FirestoreService."""
def test_firestore_service_can_import(self):
"""Test FirestoreService can be imported."""
from backend.services.firestore_service import FirestoreService
assert FirestoreService is not None
def test_firestore_service_initialization(self):
"""Test FirestoreService initializes with mocked client."""
mock_client = MagicMock()
with patch('backend.services.firestore_service.firestore.Client', return_value=mock_client):
from backend.services.firestore_service import FirestoreService
service = FirestoreService()
assert service is not None
def test_firestore_service_has_crud_methods(self):
"""Test FirestoreService has CRUD methods."""
from backend.services.firestore_service import FirestoreService
assert hasattr(FirestoreService, 'create_job')
assert hasattr(FirestoreService, 'get_job')
assert hasattr(FirestoreService, 'update_job')
assert hasattr(FirestoreService, 'delete_job')
class TestStorageServiceBasics:
"""Basic tests for StorageService."""
def test_storage_service_can_import(self):
"""Test StorageService can be imported."""
from backend.services.storage_service import StorageService
assert StorageService is not None
def test_storage_service_initialization(self):
"""Test StorageService initializes with mocked client."""
mock_client = MagicMock()
mock_bucket = MagicMock()
mock_client.bucket.return_value = mock_bucket
with patch('backend.services.storage_service.storage.Client', return_value=mock_client):
from backend.services.storage_service import StorageService
service = StorageService()
assert service is not None
def test_storage_service_has_file_methods(self):
"""Test StorageService has file operation methods."""
from backend.services.storage_service import StorageService
assert hasattr(StorageService, 'upload_file')
assert hasattr(StorageService, 'download_file')
class TestWorkerServiceBasics:
"""Basic tests for WorkerService."""
def test_worker_service_can_import(self):
"""Test WorkerService can be imported."""
from backend.services.worker_service import WorkerService
assert WorkerService is not None
def test_worker_service_initialization(self):
"""Test WorkerService initializes correctly."""
from backend.services.worker_service import WorkerService
service = WorkerService()
assert service is not None
def test_worker_service_has_trigger_methods(self):
"""Test WorkerService has trigger methods."""
from backend.services.worker_service import WorkerService
assert hasattr(WorkerService, 'trigger_audio_worker')
assert hasattr(WorkerService, 'trigger_lyrics_worker')
assert hasattr(WorkerService, 'trigger_screens_worker')
assert hasattr(WorkerService, 'trigger_video_worker')
def test_worker_service_get_base_url(self):
"""Test WorkerService constructs base URL."""
from backend.services.worker_service import WorkerService
service = WorkerService()
url = service._get_base_url()
assert 'http' in url
class TestJobManagerBasics:
"""Basic tests for JobManager."""
def test_job_manager_can_import(self):
"""Test JobManager can be imported."""
from backend.services.job_manager import JobManager
assert JobManager is not None
def test_job_manager_initialization(self):
"""Test JobManager initializes with mocked services."""
mock_firestore = MagicMock()
mock_storage = MagicMock()
with patch('backend.services.job_manager.FirestoreService', return_value=mock_firestore), \
patch('backend.services.job_manager.StorageService', return_value=mock_storage):
from backend.services.job_manager import JobManager
manager = JobManager()
assert manager is not None
def test_job_manager_has_crud_methods(self):
"""Test JobManager has CRUD methods."""
from backend.services.job_manager import JobManager
assert hasattr(JobManager, 'create_job')
assert hasattr(JobManager, 'get_job')
assert hasattr(JobManager, 'delete_job')
assert hasattr(JobManager, 'list_jobs')
-68
View File
@@ -1,68 +0,0 @@
"""
Extended tests for config.py to improve coverage.
"""
import pytest
import os
from unittest.mock import patch
class TestSettingsConfiguration:
"""Tests for Settings configuration."""
def test_settings_class_exists(self):
"""Test Settings class exists."""
from backend.config import Settings
assert Settings is not None
def test_settings_loads_defaults(self):
"""Test Settings loads default values."""
from backend.config import Settings
settings = Settings()
assert settings is not None
def test_gcs_bucket_name_from_env(self):
"""Test GCS bucket name from environment."""
with patch.dict(os.environ, {'GCS_BUCKET_NAME': 'test-bucket'}):
from backend.config import Settings
settings = Settings()
assert settings.gcs_bucket_name == 'test-bucket'
def test_firestore_collection_from_env(self):
"""Test Firestore collection from environment."""
with patch.dict(os.environ, {'FIRESTORE_COLLECTION': 'test-jobs'}):
from backend.config import Settings
settings = Settings()
assert settings.firestore_collection == 'test-jobs'
def test_environment_setting(self):
"""Test environment setting."""
from backend.config import Settings
settings = Settings()
# Accept 'test' as well since pytest may set ENVIRONMENT=test
assert settings.environment in ['development', 'production', 'testing', 'test']
def test_google_cloud_project(self):
"""Test Google Cloud project setting."""
with patch.dict(os.environ, {'GOOGLE_CLOUD_PROJECT': 'test-project'}):
from backend.config import Settings
settings = Settings()
# Should either use the env var or have a default
class TestGetSettings:
"""Tests for get_settings function."""
def test_get_settings_returns_settings(self):
"""Test get_settings returns Settings instance."""
from backend.config import get_settings
settings = get_settings()
assert settings is not None
def test_get_settings_returns_same_instance(self):
"""Test get_settings returns cached instance."""
from backend.config import get_settings
settings1 = get_settings()
settings2 = get_settings()
# Should be same cached instance
assert settings1 is settings2
-377
View File
@@ -1,377 +0,0 @@
"""
Tests for credential manager and auth endpoints.
"""
import pytest
import json
from unittest.mock import MagicMock, patch
from datetime import datetime
from backend.services.credential_manager import (
CredentialManager,
CredentialStatus,
CredentialCheckResult,
DeviceAuthInfo,
get_credential_manager,
)
class TestCredentialStatus:
"""Tests for CredentialStatus enum."""
def test_status_values(self):
"""Test all status values exist."""
assert CredentialStatus.VALID == "valid"
assert CredentialStatus.EXPIRED == "expired"
assert CredentialStatus.INVALID == "invalid"
assert CredentialStatus.NOT_CONFIGURED == "not_configured"
assert CredentialStatus.ERROR == "error"
class TestCredentialCheckResult:
"""Tests for CredentialCheckResult dataclass."""
def test_create_result(self):
"""Test creating a check result."""
result = CredentialCheckResult(
service="youtube",
status=CredentialStatus.VALID,
message="Credentials are valid",
last_checked=datetime.utcnow()
)
assert result.service == "youtube"
assert result.status == CredentialStatus.VALID
assert result.message == "Credentials are valid"
assert result.expires_at is None
def test_create_result_with_expiry(self):
"""Test creating a check result with expiry."""
expiry = datetime.utcnow()
result = CredentialCheckResult(
service="gdrive",
status=CredentialStatus.VALID,
message="Valid",
last_checked=datetime.utcnow(),
expires_at=expiry
)
assert result.expires_at == expiry
class TestCredentialManager:
"""Tests for CredentialManager class."""
def test_init(self):
"""Test manager initialization."""
manager = CredentialManager()
assert manager._pending_device_auths == {}
def test_check_youtube_not_configured(self):
"""Test YouTube check when not configured."""
with patch('backend.services.credential_manager.get_settings') as mock_get_settings:
mock_settings = MagicMock()
mock_settings.get_secret.return_value = None
mock_get_settings.return_value = mock_settings
manager = CredentialManager()
result = manager.check_youtube_credentials()
assert result.status == CredentialStatus.NOT_CONFIGURED
assert "not configured" in result.message.lower()
def test_check_youtube_invalid_json(self):
"""Test YouTube check with invalid JSON."""
with patch('backend.services.credential_manager.get_settings') as mock_get_settings:
mock_settings = MagicMock()
mock_settings.get_secret.return_value = "not valid json"
mock_get_settings.return_value = mock_settings
manager = CredentialManager()
result = manager.check_youtube_credentials()
assert result.status == CredentialStatus.INVALID
assert "json" in result.message.lower()
def test_check_youtube_missing_fields(self):
"""Test YouTube check with missing required fields."""
# Missing refresh_token and client_secret
creds = json.dumps({"token": "test", "client_id": "test"})
with patch('backend.services.credential_manager.get_settings') as mock_get_settings:
mock_settings = MagicMock()
mock_settings.get_secret.return_value = creds
mock_get_settings.return_value = mock_settings
manager = CredentialManager()
result = manager.check_youtube_credentials()
assert result.status == CredentialStatus.INVALID
assert "missing" in result.message.lower()
def test_check_gdrive_not_configured(self):
"""Test Google Drive check when not configured."""
with patch('backend.services.credential_manager.get_settings') as mock_get_settings:
mock_settings = MagicMock()
mock_settings.get_secret.return_value = None
mock_get_settings.return_value = mock_settings
manager = CredentialManager()
result = manager.check_gdrive_credentials()
assert result.status == CredentialStatus.NOT_CONFIGURED
def test_check_dropbox_not_configured(self):
"""Test Dropbox check when not configured."""
with patch('backend.services.credential_manager.get_settings') as mock_get_settings:
mock_settings = MagicMock()
mock_settings.get_secret.return_value = None
mock_get_settings.return_value = mock_settings
manager = CredentialManager()
result = manager.check_dropbox_credentials()
assert result.status == CredentialStatus.NOT_CONFIGURED
def test_check_dropbox_missing_access_token(self):
"""Test Dropbox check with missing access token."""
creds = json.dumps({"refresh_token": "test"}) # No access_token
with patch('backend.services.credential_manager.get_settings') as mock_get_settings:
mock_settings = MagicMock()
mock_settings.get_secret.return_value = creds
mock_get_settings.return_value = mock_settings
manager = CredentialManager()
result = manager.check_dropbox_credentials()
assert result.status == CredentialStatus.INVALID
assert "access_token" in result.message.lower()
def test_check_all_credentials(self):
"""Test checking all credentials."""
with patch('backend.services.credential_manager.get_settings') as mock_get_settings:
mock_settings = MagicMock()
mock_settings.get_secret.return_value = None
mock_get_settings.return_value = mock_settings
manager = CredentialManager()
results = manager.check_all_credentials()
assert "youtube" in results
assert "gdrive" in results
assert "dropbox" in results
assert all(r.status == CredentialStatus.NOT_CONFIGURED for r in results.values())
def test_test_youtube_api_failure(self):
"""Test YouTube API test failure."""
manager = CredentialManager()
mock_creds = MagicMock()
with patch('googleapiclient.discovery.build', side_effect=Exception("API Error")):
result = manager._test_youtube_api(mock_creds)
assert result is False
def test_test_gdrive_api_failure(self):
"""Test Google Drive API test failure."""
manager = CredentialManager()
mock_creds = MagicMock()
with patch('googleapiclient.discovery.build', side_effect=Exception("API Error")):
result = manager._test_gdrive_api(mock_creds)
assert result is False
def test_test_dropbox_api_failure(self):
"""Test Dropbox API test failure."""
manager = CredentialManager()
with patch('dropbox.Dropbox', side_effect=Exception("API Error")):
result = manager._test_dropbox_api({"access_token": "test"})
assert result is False
def test_send_credential_alert_no_webhook(self):
"""Test alert without webhook URL."""
manager = CredentialManager()
result = manager.send_credential_alert([], discord_webhook_url=None)
assert result is False
def test_send_credential_alert_success(self):
"""Test successful alert sending."""
manager = CredentialManager()
mock_response = MagicMock()
mock_response.status_code = 200
invalid_services = [
CredentialCheckResult(
service="youtube",
status=CredentialStatus.INVALID,
message="Test message",
last_checked=datetime.utcnow()
)
]
with patch('requests.post', return_value=mock_response) as mock_post:
result = manager.send_credential_alert(
invalid_services,
discord_webhook_url="https://discord.com/webhook/test"
)
assert result is True
mock_post.assert_called_once()
def test_start_youtube_device_auth(self):
"""Test starting YouTube device auth flow."""
manager = CredentialManager()
mock_response = MagicMock()
mock_response.status_code = 200
mock_response.json.return_value = {
"device_code": "test_device_code",
"user_code": "TEST-CODE",
"verification_uri": "https://google.com/device",
"expires_in": 1800,
"interval": 5
}
with patch('requests.post', return_value=mock_response):
device_info = manager.start_youtube_device_auth(
client_id="test_client",
client_secret="test_secret"
)
assert device_info.device_code == "test_device_code"
assert device_info.user_code == "TEST-CODE"
assert device_info.verification_url == "https://google.com/device"
# Should be stored for polling
key = f"youtube:{device_info.device_code}"
assert key in manager._pending_device_auths
def test_poll_device_auth_no_client_creds(self):
"""Test polling when client credentials are not in Secret Manager."""
manager = CredentialManager()
# Mock get_youtube_client_credentials to return None (no creds)
with patch.object(manager, 'get_youtube_client_credentials', return_value=None):
status, data = manager.poll_device_auth("youtube", "some_code")
assert status == "error"
assert "client credentials not found" in data["message"].lower()
def test_poll_device_auth_expired(self):
"""Test polling with expired device code."""
manager = CredentialManager()
mock_response = MagicMock()
mock_response.status_code = 400
mock_response.json.return_value = {"error": "expired_token"}
with patch.object(manager, 'get_youtube_client_credentials',
return_value={"client_id": "test", "client_secret": "test"}):
with patch('requests.post', return_value=mock_response):
status, data = manager.poll_device_auth("youtube", "expired_code")
assert status == "expired"
assert "expired" in data["message"].lower()
def test_poll_device_auth_pending(self):
"""Test polling when authorization is pending."""
manager = CredentialManager()
mock_response = MagicMock()
mock_response.status_code = 400
mock_response.json.return_value = {"error": "authorization_pending"}
with patch.object(manager, 'get_youtube_client_credentials',
return_value={"client_id": "test", "client_secret": "test"}):
with patch('requests.post', return_value=mock_response):
status, data = manager.poll_device_auth("youtube", "pending_code")
assert status == "pending"
assert "waiting" in data["message"].lower()
def test_poll_device_auth_success(self):
"""Test polling when authorization completes successfully."""
manager = CredentialManager()
mock_response = MagicMock()
mock_response.status_code = 200
mock_response.json.return_value = {
"access_token": "new_access_token",
"refresh_token": "new_refresh_token"
}
with patch.object(manager, 'get_youtube_client_credentials',
return_value={"client_id": "test_id", "client_secret": "test_secret"}):
with patch('requests.post', return_value=mock_response):
with patch.object(manager, '_save_credentials_to_secret', return_value=True):
status, data = manager.poll_device_auth("youtube", "completed_code")
assert status == "complete"
assert data["token"] == "new_access_token"
assert data["refresh_token"] == "new_refresh_token"
assert data["client_id"] == "test_id"
class TestGetCredentialManager:
"""Tests for singleton factory function."""
def test_returns_singleton(self):
"""Test that factory returns same instance."""
# Reset singleton
import backend.services.credential_manager as module
module._credential_manager = None
manager1 = get_credential_manager()
manager2 = get_credential_manager()
assert manager1 is manager2
class TestAuthRoutes:
"""Tests for auth API routes."""
def test_routes_import(self):
"""Test that routes can be imported."""
from backend.api.routes.auth import router
assert router is not None
def test_status_endpoint_exists(self):
"""Test that status endpoint is defined."""
from backend.api.routes.auth import get_credentials_status
assert get_credentials_status is not None
def test_validate_endpoint_exists(self):
"""Test that validate endpoint is defined."""
from backend.api.routes.auth import validate_credentials
assert validate_credentials is not None
def test_device_auth_endpoint_exists(self):
"""Test that device auth endpoints are defined."""
from backend.api.routes.auth import (
start_youtube_device_auth,
poll_youtube_device_auth,
start_gdrive_device_auth,
poll_gdrive_device_auth,
)
assert start_youtube_device_auth is not None
assert poll_youtube_device_auth is not None
assert start_gdrive_device_auth is not None
assert poll_gdrive_device_auth is not None
class TestFileUploadCredentialValidation:
"""Tests for credential validation in file upload."""
def test_credential_manager_imported(self):
"""Test that credential manager is imported in file upload."""
from backend.api.routes.file_upload import get_credential_manager, CredentialStatus
assert get_credential_manager is not None
assert CredentialStatus is not None
-54
View File
@@ -1,54 +0,0 @@
"""
Tests for API dependencies (authentication, authorization).
"""
import pytest
from unittest.mock import MagicMock, patch
class TestAuthDependencies:
"""Tests for authentication dependencies module structure."""
def test_get_token_from_request_function_exists(self):
"""Test get_token_from_request function exists."""
from backend.api.dependencies import get_token_from_request
assert get_token_from_request is not None
def test_require_auth_function_exists(self):
"""Test require_auth function exists."""
from backend.api.dependencies import require_auth
assert require_auth is not None
def test_require_admin_function_exists(self):
"""Test require_admin function exists."""
from backend.api.dependencies import require_admin
assert require_admin is not None
def test_optional_auth_function_exists(self):
"""Test optional_auth function exists."""
from backend.api.dependencies import optional_auth
assert optional_auth is not None
def test_security_scheme_defined(self):
"""Test HTTP Bearer security scheme is defined."""
from backend.api.dependencies import security
assert security is not None
def test_logger_configured(self):
"""Test logger is configured."""
from backend.api.dependencies import logger
assert logger is not None
class TestAuthServiceAccess:
"""Tests for auth service access."""
def test_get_auth_service_function_exists(self):
"""Test get_auth_service function exists."""
from backend.services.auth_service import get_auth_service
assert get_auth_service is not None
def test_user_type_enum_defined(self):
"""Test UserType enum is defined."""
from backend.services.auth_service import UserType
assert UserType is not None
-820
View File
@@ -1,820 +0,0 @@
"""
Tests for native distribution services (Dropbox and Google Drive).
These tests verify the service interfaces without requiring actual
cloud credentials, using mocks to simulate API responses.
"""
import json
import os
import pytest
from unittest.mock import MagicMock, patch, mock_open
class TestDropboxService:
"""Tests for the DropboxService class."""
def test_module_imports(self):
"""Test that the module imports correctly."""
from backend.services.dropbox_service import DropboxService, get_dropbox_service
assert DropboxService is not None
assert get_dropbox_service is not None
def test_init_creates_instance(self):
"""Test that we can create a DropboxService instance."""
from backend.services.dropbox_service import DropboxService
service = DropboxService()
assert service is not None
assert service._client is None # Client not initialized until first use
assert service._is_configured is False
def test_is_configured_returns_false_without_credentials(self):
"""Test that is_configured returns False when credentials are missing."""
from backend.services.dropbox_service import DropboxService
with patch.object(DropboxService, '_load_credentials', return_value=None):
service = DropboxService()
assert service.is_configured is False
def test_is_configured_returns_true_with_credentials(self):
"""Test that is_configured returns True when credentials are present."""
from backend.services.dropbox_service import DropboxService
mock_creds = {"access_token": "test_token"}
with patch.object(DropboxService, '_load_credentials', return_value=mock_creds):
service = DropboxService()
assert service.is_configured is True
def test_is_configured_caches_result(self):
"""Test that is_configured caches the result."""
from backend.services.dropbox_service import DropboxService
service = DropboxService()
service._is_configured = True
# Should return True without calling _load_credentials
assert service.is_configured is True
def test_is_configured_returns_false_without_access_token(self):
"""Test that is_configured returns False when access_token is missing."""
from backend.services.dropbox_service import DropboxService
mock_creds = {"refresh_token": "test"} # No access_token
with patch.object(DropboxService, '_load_credentials', return_value=mock_creds):
service = DropboxService()
assert service.is_configured is False
def test_get_next_brand_code_first_track(self):
"""Test brand code calculation when no existing tracks."""
from backend.services.dropbox_service import DropboxService
service = DropboxService()
with patch.object(service, 'list_folders', return_value=[]):
brand_code = service.get_next_brand_code("/test/path", "NOMAD")
assert brand_code == "NOMAD-0001"
def test_get_next_brand_code_sequential(self):
"""Test brand code calculation with existing tracks."""
from backend.services.dropbox_service import DropboxService
service = DropboxService()
existing_folders = [
"NOMAD-0001 - Artist1 - Song1",
"NOMAD-0002 - Artist2 - Song2",
"NOMAD-0005 - Artist3 - Song3", # Gap in sequence
"OTHER-0001 - Different Brand", # Different brand
]
with patch.object(service, 'list_folders', return_value=existing_folders):
brand_code = service.get_next_brand_code("/test/path", "NOMAD")
assert brand_code == "NOMAD-0006"
def test_get_next_brand_code_different_prefix(self):
"""Test brand code calculation with different brand prefix."""
from backend.services.dropbox_service import DropboxService
service = DropboxService()
existing_folders = [
"NOMAD-0001 - Artist1 - Song1",
"TEST-0010 - Artist2 - Song2",
]
with patch.object(service, 'list_folders', return_value=existing_folders):
brand_code = service.get_next_brand_code("/test/path", "TEST")
assert brand_code == "TEST-0011"
def test_factory_function_returns_instance(self):
"""Test that get_dropbox_service returns a service instance."""
from backend.services.dropbox_service import get_dropbox_service
service = get_dropbox_service()
assert service is not None
def test_load_credentials_handles_exception(self):
"""Test that _load_credentials handles exceptions gracefully."""
from backend.services.dropbox_service import DropboxService
with patch('backend.services.dropbox_service.secretmanager.SecretManagerServiceClient') as mock_client:
mock_client.return_value.access_secret_version.side_effect = Exception("API Error")
service = DropboxService()
result = service._load_credentials()
assert result is None
def test_load_credentials_parses_json(self):
"""Test that _load_credentials correctly parses JSON credentials."""
from backend.services.dropbox_service import DropboxService
mock_creds = {"access_token": "test_token", "refresh_token": "test_refresh"}
mock_response = MagicMock()
mock_response.payload.data.decode.return_value = json.dumps(mock_creds)
with patch('backend.services.dropbox_service.secretmanager.SecretManagerServiceClient') as mock_client:
mock_client.return_value.access_secret_version.return_value = mock_response
service = DropboxService()
result = service._load_credentials()
assert result == mock_creds
def test_client_property_raises_without_credentials(self):
"""Test that client property raises error without credentials."""
from backend.services.dropbox_service import DropboxService
with patch.object(DropboxService, '_load_credentials', return_value=None):
service = DropboxService()
with pytest.raises(RuntimeError, match="credentials not configured"):
_ = service.client
def test_client_property_raises_import_error(self):
"""Test that client property raises helpful error if dropbox not installed."""
from backend.services.dropbox_service import DropboxService
import sys
mock_creds = {"access_token": "test_token"}
with patch.object(DropboxService, '_load_credentials', return_value=mock_creds):
with patch.dict(sys.modules, {'dropbox': None}):
service = DropboxService()
# Force reimport to fail
with patch('builtins.__import__', side_effect=ImportError("No module named 'dropbox'")):
with pytest.raises(ImportError, match="dropbox package"):
_ = service.client
def test_list_folders_adds_leading_slash(self):
"""Test that list_folders adds leading slash if missing."""
from backend.services.dropbox_service import DropboxService
service = DropboxService()
mock_client = MagicMock()
mock_result = MagicMock()
mock_result.entries = []
mock_result.has_more = False
mock_client.files_list_folder.return_value = mock_result
service._client = mock_client
service.list_folders("test/path") # No leading slash
# Verify it was called with leading slash
mock_client.files_list_folder.assert_called_with("/test/path")
def test_upload_file_adds_leading_slash(self):
"""Test that upload_file adds leading slash to remote path."""
from backend.services.dropbox_service import DropboxService
import tempfile
service = DropboxService()
mock_client = MagicMock()
service._client = mock_client
# Create a small temp file
with tempfile.NamedTemporaryFile(delete=False, mode='wb') as f:
f.write(b"test content")
temp_path = f.name
try:
service.upload_file(temp_path, "remote/path.txt") # No leading slash
# Verify upload was called with leading slash
mock_client.files_upload.assert_called_once()
call_args = mock_client.files_upload.call_args
assert call_args[0][1] == "/remote/path.txt"
finally:
os.unlink(temp_path)
def test_upload_folder_uploads_all_files(self):
"""Test that upload_folder uploads all files in directory."""
from backend.services.dropbox_service import DropboxService
import tempfile
service = DropboxService()
with tempfile.TemporaryDirectory() as tmpdir:
# Create some test files
(open(os.path.join(tmpdir, "file1.txt"), "w")).write("content1")
(open(os.path.join(tmpdir, "file2.txt"), "w")).write("content2")
with patch.object(service, 'upload_file') as mock_upload:
service.upload_folder(tmpdir, "/remote/folder")
assert mock_upload.call_count == 2
def test_create_shared_link_success(self):
"""Test successful shared link creation."""
from backend.services.dropbox_service import DropboxService
service = DropboxService()
mock_client = MagicMock()
mock_link = MagicMock()
mock_link.url = "https://dropbox.com/shared/test"
mock_client.sharing_create_shared_link_with_settings.return_value = mock_link
service._client = mock_client
result = service.create_shared_link("/test/path")
assert result == "https://dropbox.com/shared/test"
def test_sharing_list_shared_links_mock_setup(self):
"""Test that mock setup for sharing_list_shared_links works correctly.
Note: Properly mocking Dropbox's ApiError is complex because it
requires specific exception class structure. This test verifies
the mock configuration is correct for the success path, which is
a prerequisite for more complex error-handling tests.
"""
from backend.services.dropbox_service import DropboxService
service = DropboxService()
mock_client = MagicMock()
# Mock the existing link retrieval (success case)
mock_existing_link = MagicMock()
mock_existing_link.url = "https://dropbox.com/existing/link"
mock_links_result = MagicMock()
mock_links_result.links = [mock_existing_link]
mock_client.sharing_list_shared_links.return_value = mock_links_result
# Assign the mock client to the service
service._client = mock_client
# Verify the mock returns the expected link structure
result = mock_client.sharing_list_shared_links(path="/test/path")
assert result.links[0].url == "https://dropbox.com/existing/link"
mock_client.sharing_list_shared_links.assert_called_once_with(path="/test/path")
class TestGoogleDriveService:
"""Tests for the GoogleDriveService class."""
def test_module_imports(self):
"""Test that the module imports correctly."""
from backend.services.gdrive_service import GoogleDriveService, get_gdrive_service
assert GoogleDriveService is not None
assert get_gdrive_service is not None
def test_init_creates_instance(self):
"""Test that we can create a GoogleDriveService instance."""
from backend.services.gdrive_service import GoogleDriveService
service = GoogleDriveService()
assert service is not None
assert service._service is None # Service not initialized until first use
assert service._loaded is False
def test_is_configured_returns_false_without_credentials(self):
"""Test that is_configured returns False when credentials are missing."""
from backend.services.gdrive_service import GoogleDriveService
with patch.object(GoogleDriveService, '_load_credentials', return_value=None):
service = GoogleDriveService()
assert service.is_configured is False
def test_is_configured_returns_true_with_credentials(self):
"""Test that is_configured returns True when credentials are present."""
from backend.services.gdrive_service import GoogleDriveService
mock_creds = {
"token": "test_token",
"refresh_token": "test_refresh",
"client_id": "test_client_id",
"client_secret": "test_client_secret",
}
with patch.object(GoogleDriveService, '_load_credentials', return_value=mock_creds):
service = GoogleDriveService()
assert service.is_configured is True
def test_factory_function_returns_singleton(self):
"""Test that get_gdrive_service returns a singleton."""
from backend.services.gdrive_service import get_gdrive_service, _gdrive_service
# Reset singleton
import backend.services.gdrive_service as gdrive_module
gdrive_module._gdrive_service = None
service1 = get_gdrive_service()
service2 = get_gdrive_service()
assert service1 is service2
def test_load_credentials_caches_result(self):
"""Test that _load_credentials caches the result."""
from backend.services.gdrive_service import GoogleDriveService
service = GoogleDriveService()
service._loaded = True
service._credentials_data = {"cached": "data"}
# Should return cached data without calling settings
result = service._load_credentials()
assert result == {"cached": "data"}
def test_load_credentials_falls_back_to_youtube(self):
"""Test that _load_credentials falls back to YouTube credentials."""
from backend.services.gdrive_service import GoogleDriveService
mock_settings = MagicMock()
mock_settings.get_secret.side_effect = [
None, # gdrive-oauth-credentials not found
'{"refresh_token": "yt_refresh", "client_id": "yt_client", "client_secret": "yt_secret"}', # youtube-oauth-credentials
]
service = GoogleDriveService()
service.settings = mock_settings
result = service._load_credentials()
assert result is not None
assert result["refresh_token"] == "yt_refresh"
def test_load_credentials_returns_none_when_both_fail(self):
"""Test that _load_credentials returns None when no credentials available."""
from backend.services.gdrive_service import GoogleDriveService
mock_settings = MagicMock()
mock_settings.get_secret.return_value = None
service = GoogleDriveService()
service.settings = mock_settings
result = service._load_credentials()
assert result is None
assert service._loaded is True
def test_load_credentials_validates_required_fields(self):
"""Test that _load_credentials validates required fields."""
from backend.services.gdrive_service import GoogleDriveService
mock_settings = MagicMock()
# Missing client_secret
mock_settings.get_secret.return_value = '{"refresh_token": "test", "client_id": "test"}'
service = GoogleDriveService()
service.settings = mock_settings
result = service._load_credentials()
assert result is None # Should fail validation
def test_load_credentials_handles_json_error(self):
"""Test that _load_credentials handles JSON parse errors."""
from backend.services.gdrive_service import GoogleDriveService
mock_settings = MagicMock()
mock_settings.get_secret.return_value = "not valid json"
service = GoogleDriveService()
service.settings = mock_settings
result = service._load_credentials()
assert result is None
def test_service_property_raises_without_credentials(self):
"""Test that service property raises error without credentials."""
from backend.services.gdrive_service import GoogleDriveService
with patch.object(GoogleDriveService, '_load_credentials', return_value=None):
service = GoogleDriveService()
with pytest.raises(RuntimeError, match="credentials not configured"):
_ = service.service
def test_get_or_create_folder_finds_existing(self):
"""Test that get_or_create_folder finds existing folders."""
from backend.services.gdrive_service import GoogleDriveService
service = GoogleDriveService()
mock_service = MagicMock()
mock_files = MagicMock()
mock_list = MagicMock()
mock_list.execute.return_value = {"files": [{"id": "existing_folder_id", "name": "TestFolder"}]}
mock_files.list.return_value = mock_list
mock_service.files.return_value = mock_files
service._service = mock_service
result = service.get_or_create_folder("parent_id", "TestFolder")
assert result == "existing_folder_id"
def test_get_or_create_folder_creates_new(self):
"""Test that get_or_create_folder creates new folders when not found."""
from backend.services.gdrive_service import GoogleDriveService
service = GoogleDriveService()
mock_service = MagicMock()
mock_files = MagicMock()
# First call - folder doesn't exist
mock_list = MagicMock()
mock_list.execute.return_value = {"files": []}
mock_files.list.return_value = mock_list
# Second call - create folder
mock_create = MagicMock()
mock_create.execute.return_value = {"id": "new_folder_id"}
mock_files.create.return_value = mock_create
mock_service.files.return_value = mock_files
service._service = mock_service
result = service.get_or_create_folder("parent_id", "NewFolder")
assert result == "new_folder_id"
def test_upload_file_determines_mime_type(self):
"""Test that upload_file determines correct MIME type."""
from backend.services.gdrive_service import GoogleDriveService
import tempfile
service = GoogleDriveService()
mock_service = MagicMock()
mock_files = MagicMock()
# Mock list (for replace_existing check)
mock_list = MagicMock()
mock_list.execute.return_value = {"files": []}
mock_files.list.return_value = mock_list
# Mock create
mock_create = MagicMock()
mock_create.execute.return_value = {"id": "uploaded_file_id"}
mock_files.create.return_value = mock_create
mock_service.files.return_value = mock_files
service._service = mock_service
# Create a temp file with .mp4 extension
with tempfile.NamedTemporaryFile(suffix='.mp4', delete=False) as f:
f.write(b"fake video content")
temp_path = f.name
try:
result = service.upload_file(temp_path, "parent_id", "test.mp4")
assert result == "uploaded_file_id"
finally:
os.unlink(temp_path)
def test_upload_file_replaces_existing(self):
"""Test that upload_file can replace existing files."""
from backend.services.gdrive_service import GoogleDriveService
import tempfile
service = GoogleDriveService()
mock_service = MagicMock()
mock_files = MagicMock()
# Mock list - file exists
mock_list = MagicMock()
mock_list.execute.return_value = {"files": [{"id": "existing_id"}]}
mock_files.list.return_value = mock_list
# Mock delete
mock_delete = MagicMock()
mock_files.delete.return_value = mock_delete
# Mock create
mock_create = MagicMock()
mock_create.execute.return_value = {"id": "new_file_id"}
mock_files.create.return_value = mock_create
mock_service.files.return_value = mock_files
service._service = mock_service
with tempfile.NamedTemporaryFile(suffix='.txt', delete=False) as f:
f.write(b"content")
temp_path = f.name
try:
result = service.upload_file(temp_path, "parent_id", "test.txt", replace_existing=True)
assert result == "new_file_id"
mock_files.delete.assert_called_once()
finally:
os.unlink(temp_path)
def test_upload_to_public_share_creates_structure(self):
"""Test that upload_to_public_share creates folder structure."""
from backend.services.gdrive_service import GoogleDriveService
import tempfile
service = GoogleDriveService()
with tempfile.TemporaryDirectory() as tmpdir:
# Create test files
mp4_path = os.path.join(tmpdir, "test.mp4")
mp4_720_path = os.path.join(tmpdir, "test_720p.mp4")
cdg_path = os.path.join(tmpdir, "test.zip")
open(mp4_path, 'wb').write(b"mp4 content")
open(mp4_720_path, 'wb').write(b"720p content")
open(cdg_path, 'wb').write(b"cdg content")
with patch.object(service, 'get_or_create_folder', return_value="folder_id"):
with patch.object(service, 'upload_file', return_value="file_id"):
result = service.upload_to_public_share(
root_folder_id="root_id",
brand_code="TEST-0001",
base_name="Artist - Title",
output_files={
"final_karaoke_lossy_mp4": mp4_path,
"final_karaoke_lossy_720p_mp4": mp4_720_path,
"final_karaoke_cdg_zip": cdg_path,
}
)
assert "mp4" in result
assert "mp4_720p" in result
assert "cdg" in result
def test_upload_to_public_share_handles_missing_files(self):
"""Test that upload_to_public_share handles missing files gracefully."""
from backend.services.gdrive_service import GoogleDriveService
service = GoogleDriveService()
with patch.object(service, 'get_or_create_folder', return_value="folder_id"):
with patch.object(service, 'upload_file', return_value="file_id"):
result = service.upload_to_public_share(
root_folder_id="root_id",
brand_code="TEST-0001",
base_name="Artist - Title",
output_files={
"final_karaoke_lossy_mp4": "/nonexistent/path.mp4",
}
)
# Should return empty since file doesn't exist
assert result == {}
class TestJobModelDistributionFields:
"""Tests for distribution fields in Job model."""
def test_job_model_has_dropbox_path_field(self):
"""Test that Job model has dropbox_path field."""
from backend.models.job import Job, JobStatus
from datetime import datetime
job = Job(
job_id="test-123",
status=JobStatus.PENDING,
created_at=datetime.now(),
updated_at=datetime.now(),
dropbox_path="/Karaoke/Tracks-Organized",
)
assert job.dropbox_path == "/Karaoke/Tracks-Organized"
def test_job_model_has_gdrive_folder_id_field(self):
"""Test that Job model has gdrive_folder_id field."""
from backend.models.job import Job, JobStatus
from datetime import datetime
job = Job(
job_id="test-123",
status=JobStatus.PENDING,
created_at=datetime.now(),
updated_at=datetime.now(),
gdrive_folder_id="1abc123xyz",
)
assert job.gdrive_folder_id == "1abc123xyz"
def test_job_model_distribution_fields_optional(self):
"""Test that distribution fields are optional."""
from backend.models.job import Job, JobStatus
from datetime import datetime
job = Job(
job_id="test-123",
status=JobStatus.PENDING,
created_at=datetime.now(),
updated_at=datetime.now(),
)
assert job.dropbox_path is None
assert job.gdrive_folder_id is None
class TestJobCreateDistributionFields:
"""Tests for distribution fields in JobCreate model."""
def test_job_create_has_dropbox_path_field(self):
"""Test that JobCreate model has dropbox_path field."""
from backend.models.job import JobCreate
job_create = JobCreate(
artist="Test Artist",
title="Test Title",
dropbox_path="/Karaoke/Tracks-Organized",
)
assert job_create.dropbox_path == "/Karaoke/Tracks-Organized"
def test_job_create_has_gdrive_folder_id_field(self):
"""Test that JobCreate model has gdrive_folder_id field."""
from backend.models.job import JobCreate
job_create = JobCreate(
artist="Test Artist",
title="Test Title",
gdrive_folder_id="1abc123xyz",
)
assert job_create.gdrive_folder_id == "1abc123xyz"
def test_job_create_distribution_fields_optional(self):
"""Test that distribution fields are optional in JobCreate."""
from backend.models.job import JobCreate
job_create = JobCreate(
artist="Test Artist",
title="Test Title",
)
assert job_create.dropbox_path is None
assert job_create.gdrive_folder_id is None
class TestFileUploadDistributionParams:
"""Tests for distribution parameters in file upload endpoint."""
def test_endpoint_has_dropbox_path_parameter(self):
"""Test that upload endpoint signature includes dropbox_path parameter."""
import inspect
from backend.api.routes.file_upload import upload_and_create_job
sig = inspect.signature(upload_and_create_job)
param_names = list(sig.parameters.keys())
assert "dropbox_path" in param_names
def test_endpoint_has_gdrive_folder_id_parameter(self):
"""Test that upload endpoint signature includes gdrive_folder_id parameter."""
import inspect
from backend.api.routes.file_upload import upload_and_create_job
sig = inspect.signature(upload_and_create_job)
param_names = list(sig.parameters.keys())
assert "gdrive_folder_id" in param_names
def test_endpoint_dropbox_path_is_optional(self):
"""Test that dropbox_path parameter has a default value (Form(None))."""
import inspect
from backend.api.routes.file_upload import upload_and_create_job
sig = inspect.signature(upload_and_create_job)
param = sig.parameters.get("dropbox_path")
assert param is not None
# Default is Form(None), so check that it's not required
assert param.default is not inspect.Parameter.empty
def test_endpoint_gdrive_folder_id_is_optional(self):
"""Test that gdrive_folder_id parameter has a default value (Form(None))."""
import inspect
from backend.api.routes.file_upload import upload_and_create_job
sig = inspect.signature(upload_and_create_job)
param = sig.parameters.get("gdrive_folder_id")
assert param is not None
# Default is Form(None), so check that it's not required
assert param.default is not inspect.Parameter.empty
class TestVideoWorkerDistribution:
"""Tests for distribution handling in video worker."""
def test_handle_native_distribution_function_exists(self):
"""Test that _handle_native_distribution function exists."""
from backend.workers import video_worker
assert hasattr(video_worker, '_handle_native_distribution')
def test_video_worker_imports_distribution_services(self):
"""Test that video worker can import distribution services."""
# The services should be importable (even if credentials aren't available)
try:
from backend.services.dropbox_service import DropboxService
from backend.services.gdrive_service import GoogleDriveService
assert True
except ImportError as e:
pytest.fail(f"Failed to import distribution services: {e}")
@pytest.mark.asyncio
async def test_handle_native_distribution_skips_without_config(self):
"""Test that _handle_native_distribution skips when not configured."""
from backend.workers.video_worker import _handle_native_distribution
mock_job = MagicMock()
mock_job.dropbox_path = None
mock_job.gdrive_folder_id = None
mock_job.brand_prefix = None
mock_job.artist = "Test"
mock_job.title = "Song"
mock_job_log = MagicMock()
mock_job_manager = MagicMock()
# Should complete without error
await _handle_native_distribution(
job_id="test-123",
job=mock_job,
job_log=mock_job_log,
job_manager=mock_job_manager,
temp_dir="/tmp/test",
result={},
)
# No errors should have been logged
mock_job_log.error.assert_not_called()
@pytest.mark.asyncio
async def test_handle_native_distribution_dropbox_not_configured(self):
"""Test Dropbox upload skipped when service not configured."""
from backend.workers.video_worker import _handle_native_distribution
mock_job = MagicMock()
mock_job.dropbox_path = "/test/path"
mock_job.brand_prefix = "TEST"
mock_job.gdrive_folder_id = None
mock_job.artist = "Test"
mock_job.title = "Song"
mock_job.state_data = {}
mock_job_log = MagicMock()
mock_job_manager = MagicMock()
mock_dropbox = MagicMock()
mock_dropbox.is_configured = False
with patch('backend.services.dropbox_service.get_dropbox_service', return_value=mock_dropbox):
await _handle_native_distribution(
job_id="test-123",
job=mock_job,
job_log=mock_job_log,
job_manager=mock_job_manager,
temp_dir="/tmp/test",
result={},
)
# Should log warning about not configured
mock_job_log.warning.assert_called()
@pytest.mark.asyncio
async def test_handle_native_distribution_gdrive_not_configured(self):
"""Test Google Drive upload skipped when service not configured."""
from backend.workers.video_worker import _handle_native_distribution
mock_job = MagicMock()
mock_job.dropbox_path = None
mock_job.brand_prefix = None
mock_job.gdrive_folder_id = "test_folder_id"
mock_job.artist = "Test"
mock_job.title = "Song"
mock_job.state_data = {}
mock_job_log = MagicMock()
mock_job_manager = MagicMock()
mock_gdrive = MagicMock()
mock_gdrive.is_configured = False
with patch('backend.services.gdrive_service.get_gdrive_service', return_value=mock_gdrive):
await _handle_native_distribution(
job_id="test-123",
job=mock_job,
job_log=mock_job_log,
job_manager=mock_job_manager,
temp_dir="/tmp/test",
result={},
)
# Should log warning about not configured
mock_job_log.warning.assert_called()
@pytest.mark.asyncio
async def test_handle_native_distribution_handles_import_error(self):
"""Test that import errors for services are handled gracefully."""
from backend.workers.video_worker import _handle_native_distribution
mock_job = MagicMock()
mock_job.dropbox_path = "/test/path"
mock_job.brand_prefix = "TEST"
mock_job.gdrive_folder_id = None
mock_job.artist = "Test"
mock_job.title = "Song"
mock_job.state_data = {}
mock_job_log = MagicMock()
mock_job_manager = MagicMock()
# Simulate import error
with patch('backend.services.dropbox_service.get_dropbox_service', side_effect=ImportError("No module")):
await _handle_native_distribution(
job_id="test-123",
job=mock_job,
job_log=mock_job_log,
job_manager=mock_job_manager,
temp_dir="/tmp/test",
result={},
)
# Should log warning about import error
mock_job_log.warning.assert_called()
-408
View File
@@ -1,408 +0,0 @@
"""
Tests for dropbox_service.py - Dropbox file operations.
These tests mock the Dropbox SDK and Secret Manager to verify:
- Credential loading from Secret Manager
- Folder listing and brand code calculation
- File and folder uploads
- Shared link creation
"""
import json
import os
import pytest
from unittest.mock import Mock, MagicMock, patch
class TestDropboxServiceInit:
"""Test DropboxService initialization."""
def test_init_creates_service(self):
"""Test initialization creates service with no client."""
from backend.services.dropbox_service import DropboxService
service = DropboxService()
assert service._client is None
assert service._is_configured is False
class TestLoadCredentials:
"""Test _load_credentials method."""
@patch("backend.services.dropbox_service.secretmanager.SecretManagerServiceClient")
def test_load_credentials_success(self, mock_sm_client_class):
"""Test successful credential loading from Secret Manager."""
from backend.services.dropbox_service import DropboxService
mock_sm_client = Mock()
mock_response = Mock()
mock_response.payload.data = json.dumps({
"access_token": "access-token-123",
"refresh_token": "refresh-token-456",
"app_key": "app-key",
"app_secret": "app-secret",
}).encode("UTF-8")
mock_sm_client.access_secret_version.return_value = mock_response
mock_sm_client_class.return_value = mock_sm_client
service = DropboxService()
creds = service._load_credentials()
assert creds is not None
assert creds["access_token"] == "access-token-123"
@patch("backend.services.dropbox_service.secretmanager.SecretManagerServiceClient")
def test_load_credentials_failure(self, mock_sm_client_class):
"""Test handling when Secret Manager fails."""
from backend.services.dropbox_service import DropboxService
mock_sm_client = Mock()
mock_sm_client.access_secret_version.side_effect = Exception("Access denied")
mock_sm_client_class.return_value = mock_sm_client
service = DropboxService()
creds = service._load_credentials()
assert creds is None
class TestIsConfigured:
"""Test is_configured property."""
@patch("backend.services.dropbox_service.secretmanager.SecretManagerServiceClient")
def test_is_configured_true(self, mock_sm_client_class):
"""Test is_configured returns True when credentials available."""
from backend.services.dropbox_service import DropboxService
mock_sm_client = Mock()
mock_response = Mock()
mock_response.payload.data = json.dumps({
"access_token": "token"
}).encode("UTF-8")
mock_sm_client.access_secret_version.return_value = mock_response
mock_sm_client_class.return_value = mock_sm_client
service = DropboxService()
assert service.is_configured is True
@patch("backend.services.dropbox_service.secretmanager.SecretManagerServiceClient")
def test_is_configured_false_no_token(self, mock_sm_client_class):
"""Test is_configured returns False when no access_token."""
from backend.services.dropbox_service import DropboxService
mock_sm_client = Mock()
mock_response = Mock()
mock_response.payload.data = json.dumps({
"refresh_token": "refresh" # Missing access_token
}).encode("UTF-8")
mock_sm_client.access_secret_version.return_value = mock_response
mock_sm_client_class.return_value = mock_sm_client
service = DropboxService()
assert service.is_configured is False
class TestDropboxClient:
"""Test client property."""
@patch("dropbox.Dropbox")
@patch("backend.services.dropbox_service.secretmanager.SecretManagerServiceClient")
def test_client_creates_dropbox_instance(self, mock_sm_client_class, mock_dropbox_class):
"""Test client property creates Dropbox SDK client."""
from backend.services.dropbox_service import DropboxService
mock_sm_client = Mock()
mock_response = Mock()
mock_response.payload.data = json.dumps({
"access_token": "token",
"refresh_token": "refresh",
"app_key": "key",
"app_secret": "secret",
}).encode("UTF-8")
mock_sm_client.access_secret_version.return_value = mock_response
mock_sm_client_class.return_value = mock_sm_client
mock_dropbox = Mock()
mock_dropbox_class.return_value = mock_dropbox
service = DropboxService()
client = service.client
mock_dropbox_class.assert_called_once_with(
oauth2_access_token="token",
oauth2_refresh_token="refresh",
app_key="key",
app_secret="secret",
)
assert client == mock_dropbox
@patch("backend.services.dropbox_service.secretmanager.SecretManagerServiceClient")
def test_client_raises_on_missing_credentials(self, mock_sm_client_class):
"""Test client raises RuntimeError when credentials missing."""
from backend.services.dropbox_service import DropboxService
mock_sm_client = Mock()
mock_sm_client.access_secret_version.side_effect = Exception("Not found")
mock_sm_client_class.return_value = mock_sm_client
service = DropboxService()
with pytest.raises(RuntimeError) as exc_info:
_ = service.client
assert "not configured" in str(exc_info.value)
class TestListFolders:
"""Test list_folders method."""
@patch("backend.services.dropbox_service.secretmanager.SecretManagerServiceClient")
def test_list_folders(self, mock_sm_client_class):
"""Test listing folders at a path."""
from backend.services.dropbox_service import DropboxService
from dropbox.files import FolderMetadata
mock_sm_client = Mock()
mock_response = Mock()
mock_response.payload.data = json.dumps({"access_token": "token"}).encode()
mock_sm_client.access_secret_version.return_value = mock_response
mock_sm_client_class.return_value = mock_sm_client
# Create mock folder entries
mock_folder1 = Mock(spec=FolderMetadata)
mock_folder1.name = "NOMAD-0001"
mock_folder2 = Mock(spec=FolderMetadata)
mock_folder2.name = "NOMAD-0002"
mock_file = Mock() # Not a FolderMetadata
mock_result = Mock()
mock_result.entries = [mock_folder1, mock_folder2, mock_file]
mock_result.has_more = False
service = DropboxService()
# Directly set the client to avoid needing to mock the whole init chain
mock_dropbox = Mock()
mock_dropbox.files_list_folder.return_value = mock_result
service._client = mock_dropbox
folders = service.list_folders("/Karaoke/Tracks")
mock_dropbox.files_list_folder.assert_called_once_with("/Karaoke/Tracks")
assert folders == ["NOMAD-0001", "NOMAD-0002"]
@patch("backend.services.dropbox_service.secretmanager.SecretManagerServiceClient")
def test_list_folders_adds_leading_slash(self, mock_sm_client_class):
"""Test that path without leading slash gets one added."""
from backend.services.dropbox_service import DropboxService
mock_sm_client = Mock()
mock_response = Mock()
mock_response.payload.data = json.dumps({"access_token": "token"}).encode()
mock_sm_client.access_secret_version.return_value = mock_response
mock_sm_client_class.return_value = mock_sm_client
mock_result = Mock()
mock_result.entries = []
mock_result.has_more = False
service = DropboxService()
mock_dropbox = Mock()
mock_dropbox.files_list_folder.return_value = mock_result
service._client = mock_dropbox
service.list_folders("path/without/slash")
mock_dropbox.files_list_folder.assert_called_once_with("/path/without/slash")
class TestGetNextBrandCode:
"""Test get_next_brand_code method."""
@patch("backend.services.dropbox_service.secretmanager.SecretManagerServiceClient")
def test_get_next_brand_code(self, mock_sm_client_class):
"""Test calculating next brand code from existing folders."""
from backend.services.dropbox_service import DropboxService
mock_sm_client = Mock()
mock_response = Mock()
mock_response.payload.data = json.dumps({"access_token": "token"}).encode()
mock_sm_client.access_secret_version.return_value = mock_response
mock_sm_client_class.return_value = mock_sm_client
service = DropboxService()
# Mock list_folders to return existing codes
with patch.object(service, "list_folders") as mock_list:
mock_list.return_value = [
"NOMAD-1161",
"NOMAD-1162",
"NOMAD-1163",
"Other Folder",
"NOMAD-0001",
]
next_code = service.get_next_brand_code("/path", "NOMAD")
assert next_code == "NOMAD-1164"
@patch("backend.services.dropbox_service.secretmanager.SecretManagerServiceClient")
def test_get_next_brand_code_empty_folder(self, mock_sm_client_class):
"""Test brand code calculation with no existing codes."""
from backend.services.dropbox_service import DropboxService
mock_sm_client = Mock()
mock_response = Mock()
mock_response.payload.data = json.dumps({"access_token": "token"}).encode()
mock_sm_client.access_secret_version.return_value = mock_response
mock_sm_client_class.return_value = mock_sm_client
service = DropboxService()
with patch.object(service, "list_folders") as mock_list:
mock_list.return_value = []
next_code = service.get_next_brand_code("/path", "BRAND")
assert next_code == "BRAND-0001"
class TestUploadFile:
"""Test upload_file method."""
@patch("backend.services.dropbox_service.secretmanager.SecretManagerServiceClient")
def test_upload_small_file(self, mock_sm_client_class, tmp_path):
"""Test uploading a small file directly."""
from backend.services.dropbox_service import DropboxService
mock_sm_client = Mock()
mock_response = Mock()
mock_response.payload.data = json.dumps({"access_token": "token"}).encode()
mock_sm_client.access_secret_version.return_value = mock_response
mock_sm_client_class.return_value = mock_sm_client
# Create test file
test_file = tmp_path / "test.txt"
test_file.write_text("Small file content")
service = DropboxService()
mock_dropbox = Mock()
service._client = mock_dropbox
service.upload_file(str(test_file), "/Uploads/test.txt")
mock_dropbox.files_upload.assert_called_once()
@patch("backend.services.dropbox_service.secretmanager.SecretManagerServiceClient")
def test_upload_file_adds_leading_slash(self, mock_sm_client_class, tmp_path):
"""Test upload adds leading slash to remote path."""
from backend.services.dropbox_service import DropboxService
mock_sm_client = Mock()
mock_response = Mock()
mock_response.payload.data = json.dumps({"access_token": "token"}).encode()
mock_sm_client.access_secret_version.return_value = mock_response
mock_sm_client_class.return_value = mock_sm_client
test_file = tmp_path / "test.txt"
test_file.write_text("content")
service = DropboxService()
mock_dropbox = Mock()
service._client = mock_dropbox
service.upload_file(str(test_file), "uploads/test.txt")
# Check that the path has leading slash
call_args = mock_dropbox.files_upload.call_args
assert call_args[0][1] == "/uploads/test.txt"
class TestUploadFolder:
"""Test upload_folder method."""
@patch("backend.services.dropbox_service.secretmanager.SecretManagerServiceClient")
def test_upload_folder(self, mock_sm_client_class, tmp_path):
"""Test uploading a folder with multiple files."""
from backend.services.dropbox_service import DropboxService
mock_sm_client = Mock()
mock_response = Mock()
mock_response.payload.data = json.dumps({"access_token": "token"}).encode()
mock_sm_client.access_secret_version.return_value = mock_response
mock_sm_client_class.return_value = mock_sm_client
# Create test folder with files
(tmp_path / "file1.txt").write_text("content1")
(tmp_path / "file2.txt").write_text("content2")
(tmp_path / "subdir").mkdir() # Should be skipped
service = DropboxService()
with patch.object(service, "upload_file") as mock_upload:
service.upload_folder(str(tmp_path), "/Uploads/folder")
# Should upload 2 files (not subdir)
assert mock_upload.call_count == 2
class TestCreateSharedLink:
"""Test create_shared_link method."""
@patch("backend.services.dropbox_service.secretmanager.SecretManagerServiceClient")
def test_create_shared_link_new(self, mock_sm_client_class):
"""Test creating a new shared link."""
from backend.services.dropbox_service import DropboxService
mock_sm_client = Mock()
mock_response = Mock()
mock_response.payload.data = json.dumps({"access_token": "token"}).encode()
mock_sm_client.access_secret_version.return_value = mock_response
mock_sm_client_class.return_value = mock_sm_client
mock_link = Mock()
mock_link.url = "https://dropbox.com/s/abc123/file.mp4"
service = DropboxService()
mock_dropbox = Mock()
mock_dropbox.sharing_create_shared_link_with_settings.return_value = mock_link
service._client = mock_dropbox
url = service.create_shared_link("/Uploads/file.mp4")
assert url == "https://dropbox.com/s/abc123/file.mp4"
@patch("backend.services.dropbox_service.secretmanager.SecretManagerServiceClient")
def test_create_shared_link_existing(self, mock_sm_client_class):
"""Test getting existing shared link when one already exists."""
from backend.services.dropbox_service import DropboxService
from dropbox.exceptions import ApiError
mock_sm_client = Mock()
mock_response = Mock()
mock_response.payload.data = json.dumps({"access_token": "token"}).encode()
mock_sm_client.access_secret_version.return_value = mock_response
mock_sm_client_class.return_value = mock_sm_client
# Simulate "link already exists" error
mock_error = Mock()
mock_error.is_shared_link_already_exists.return_value = True
mock_existing_link = Mock()
mock_existing_link.url = "https://dropbox.com/s/existing/file.mp4"
mock_links_result = Mock()
mock_links_result.links = [mock_existing_link]
service = DropboxService()
mock_dropbox = Mock()
mock_dropbox.sharing_create_shared_link_with_settings.side_effect = \
ApiError("req_id", mock_error, "message", "headers")
mock_dropbox.sharing_list_shared_links.return_value = mock_links_result
service._client = mock_dropbox
url = service.create_shared_link("/Uploads/file.mp4")
assert url == "https://dropbox.com/s/existing/file.mp4"
-322
View File
@@ -1,322 +0,0 @@
"""
Integration tests using local GCP emulators.
These tests use real Firestore and GCS emulators running locally,
providing true integration testing without cloud resources or costs.
Run with: ./scripts/run-emulator-tests.sh
NOTE: These are simplified integration tests that focus on the most critical
paths. They mock background workers to avoid race conditions and network issues.
"""
import pytest
import os
import time
from unittest.mock import AsyncMock, patch
from datetime import datetime, UTC
import requests
def emulators_running() -> bool:
"""Check if GCP emulators are running."""
try:
requests.get("http://127.0.0.1:8080", timeout=1)
requests.get("http://127.0.0.1:4443", timeout=1)
return True
except (requests.exceptions.ConnectionError, requests.exceptions.Timeout):
return False
# Skip all tests in this module if emulators aren't running
pytestmark = pytest.mark.skipif(
not emulators_running(),
reason="GCP emulators not running. Start with: scripts/start-emulators.sh"
)
# Set emulator environment variables before any imports (use 127.0.0.1 for IPv4)
os.environ["FIRESTORE_EMULATOR_HOST"] = "127.0.0.1:8080"
os.environ["STORAGE_EMULATOR_HOST"] = "http://127.0.0.1:4443"
os.environ["GOOGLE_CLOUD_PROJECT"] = "test-project"
os.environ["GCS_BUCKET_NAME"] = "test-bucket"
os.environ["FIRESTORE_COLLECTION"] = "test-jobs" # Use separate collection for tests
os.environ["ENVIRONMENT"] = "test"
os.environ["ADMIN_TOKENS"] = "test-admin-token"
# Only import if emulators are running
if emulators_running():
from fastapi.testclient import TestClient
from backend.main import app
from backend.models.job import JobStatus
else:
TestClient = None
app = None
JobStatus = None
@pytest.fixture(scope="module", autouse=True)
def setup_gcs_bucket():
"""Create GCS bucket in emulator before tests."""
try:
response = requests.post(
"http://localhost:4443/storage/v1/b",
json={"name": "test-bucket"},
params={"project": "test-project"}
)
if response.status_code in [200, 409]:
print(f"✅ GCS bucket 'test-bucket' ready")
except Exception as e:
print(f"⚠️ GCS bucket setup failed: {e}")
yield
@pytest.fixture(scope="module")
def mock_worker_service():
"""Mock the worker service to prevent background tasks."""
with patch("backend.api.routes.jobs.worker_service") as mock:
mock.trigger_audio_worker = AsyncMock(return_value=True)
mock.trigger_lyrics_worker = AsyncMock(return_value=True)
mock.trigger_screens_worker = AsyncMock(return_value=True)
mock.trigger_video_worker = AsyncMock(return_value=True)
yield mock
@pytest.fixture(scope="module")
def client(mock_worker_service):
"""Create FastAPI test client with mocked workers."""
with patch("backend.api.routes.file_upload.worker_service", mock_worker_service):
return TestClient(app)
@pytest.fixture
def auth_headers():
"""Auth headers for testing."""
return {"Authorization": "Bearer test-admin-token"}
class TestEmulatorBasics:
"""Basic emulator connectivity tests."""
def test_health_endpoint(self, client, auth_headers):
"""Test health endpoint works."""
response = client.get("/api/health", )
assert response.status_code == 200
assert response.json()["status"] == "healthy"
def test_root_endpoint(self, client, auth_headers):
"""Test root endpoint works."""
response = client.get("/", )
assert response.status_code == 200
assert response.json()["service"] == "karaoke-gen-backend"
class TestJobCreation:
"""Test job creation with Firestore emulator."""
def test_create_job_simple(self, client, auth_headers):
"""Test creating a simple job."""
response = client.post(
"/api/jobs",
headers=auth_headers,
json={"url": "https://youtube.com/watch?v=test123"}
)
assert response.status_code == 200
data = response.json()
assert data["status"] == "success"
assert "job_id" in data
def test_create_job_with_metadata(self, client, auth_headers):
"""Test creating a job with artist/title."""
response = client.post(
"/api/jobs",
headers=auth_headers,
json={
"url": "https://youtube.com/watch?v=test",
"artist": "Test Artist",
"title": "Test Song"
}
)
assert response.status_code == 200
data = response.json()
assert "job_id" in data
class TestJobRetrieval:
"""Test job retrieval from Firestore emulator."""
def test_create_and_get_job(self, client, auth_headers):
"""Test creating and then retrieving a job."""
# Create
create_resp = client.post(
"/api/jobs",
headers=auth_headers,
json={
"url": "https://youtube.com/watch?v=abc123",
"artist": "Test Artist",
"title": "Test Song"
}
)
assert create_resp.status_code == 200
job_id = create_resp.json()["job_id"]
# Small delay for emulator consistency
time.sleep(0.2)
# Retrieve
get_resp = client.get(f"/api/jobs/{job_id}", headers=auth_headers)
if get_resp.status_code != 200:
print(f"GET failed: {get_resp.status_code} - {get_resp.text}")
assert get_resp.status_code == 200
job = get_resp.json()
assert job["job_id"] == job_id
assert job["status"] == "pending"
assert job["artist"] == "Test Artist"
assert job["title"] == "Test Song"
def test_get_nonexistent_job(self, client, auth_headers):
"""Test fetching a job that doesn't exist."""
response = client.get("/api/jobs/nonexistent-id", headers=auth_headers)
assert response.status_code == 404
class TestJobList:
"""Test listing jobs from Firestore."""
def test_list_jobs(self, client, auth_headers):
"""Test listing jobs."""
# Create a few jobs
for i in range(3):
client.post(
"/api/jobs",
headers=auth_headers,
json={"url": f"https://youtube.com/watch?v=list{i}"}
)
time.sleep(0.2)
# List
response = client.get("/api/jobs", headers=auth_headers)
assert response.status_code == 200
jobs = response.json()
assert isinstance(jobs, list)
# We should have at least the 3 we just created
assert len(jobs) >= 3
class TestJobDeletion:
"""Test job deletion."""
def test_delete_job(self, client, auth_headers):
"""Test deleting a job."""
# Create
create_resp = client.post(
"/api/jobs",
headers=auth_headers,
json={"url": "https://youtube.com/watch?v=delete-me"}
)
job_id = create_resp.json()["job_id"]
time.sleep(0.2)
# Delete
del_resp = client.delete(f"/api/jobs/{job_id}", headers=auth_headers)
assert del_resp.status_code == 200
# Verify deleted
get_resp = client.get(f"/api/jobs/{job_id}", headers=auth_headers)
assert get_resp.status_code == 404
class TestJobUpdates:
"""Test job status updates."""
def test_cancel_job(self, client, auth_headers):
"""Test cancelling a job."""
# Create
create_resp = client.post(
"/api/jobs",
headers=auth_headers,
json={"url": "https://youtube.com/watch?v=cancel-me"}
)
job_id = create_resp.json()["job_id"]
time.sleep(0.2)
# Cancel
cancel_resp = client.post(
f"/api/jobs/{job_id}/cancel",
headers=auth_headers,
json={"reason": "test cancellation"}
)
assert cancel_resp.status_code == 200
# Verify cancelled
get_resp = client.get(f"/api/jobs/{job_id}", headers=auth_headers)
assert get_resp.status_code == 200
job = get_resp.json()
assert job["status"] == "cancelled"
class TestFileUpload:
"""Test file upload with GCS emulator."""
def test_upload_file(self, client, auth_headers):
"""Test uploading a file."""
test_file_content = b"fake audio data for testing"
response = client.post(
"/api/jobs/upload",
headers={"Authorization": auth_headers["Authorization"]},
files={"file": ("test.flac", test_file_content, "audio/flac")},
data={"artist": "Upload Artist", "title": "Upload Song"}
)
assert response.status_code == 200
data = response.json()
assert data["status"] == "success"
assert "job_id" in data
job_id = data["job_id"]
time.sleep(0.2)
# Verify job created with upload data
get_resp = client.get(f"/api/jobs/{job_id}", headers=auth_headers)
assert get_resp.status_code == 200
job = get_resp.json()
assert job["artist"] == "Upload Artist"
assert job["title"] == "Upload Song"
assert "input_media_gcs_path" in job
class TestInternalEndpoints:
"""Test internal worker endpoints."""
def test_internal_workers_exist(self, client, auth_headers):
"""Test that internal worker endpoints exist and respond."""
# Create a job first
create_resp = client.post(
"/api/jobs",
headers=auth_headers,
json={"url": "https://youtube.com/watch?v=worker-test"}
)
job_id = create_resp.json()["job_id"]
time.sleep(0.2)
# Test audio worker endpoint exists
response = client.post(
"/api/internal/workers/audio",
headers=auth_headers,
json={"job_id": job_id}
)
assert response.status_code == 200
assert response.json()["status"] == "started"
print("✅ Emulator integration tests ready to run")
File diff suppressed because it is too large. Load diff
-632
View File
@@ -1,632 +0,0 @@
"""
Tests for the flacfetch remote HTTP API client.
This module tests FlacfetchClient which handles communication with the
dedicated flacfetch VM for torrent/audio downloads.
These tests use mocked HTTP responses to verify:
- Request formatting (headers, payloads)
- Response parsing
- Error handling
- State management (singleton pattern)
"""
import pytest
from unittest.mock import AsyncMock, Mock, patch, MagicMock
import asyncio
import httpx
from backend.services.flacfetch_client import (
FlacfetchClient,
FlacfetchServiceError,
get_flacfetch_client,
reset_flacfetch_client,
)
class TestFlacfetchClientInit:
"""Test FlacfetchClient initialization."""
def test_init_with_required_params(self):
"""Test client initializes with base_url and api_key."""
client = FlacfetchClient(
base_url="http://localhost:8080",
api_key="test-key-123",
)
assert client.base_url == "http://localhost:8080"
assert client.api_key == "test-key-123"
assert client.timeout == 60 # Default timeout
def test_init_strips_trailing_slash(self):
"""Test base_url trailing slash is stripped."""
client = FlacfetchClient(
base_url="http://localhost:8080/",
api_key="test-key",
)
assert client.base_url == "http://localhost:8080"
def test_init_with_custom_timeout(self):
"""Test custom timeout is set."""
client = FlacfetchClient(
base_url="http://localhost:8080",
api_key="test-key",
timeout=120,
)
assert client.timeout == 120
def test_headers_include_api_key(self):
"""Test _headers() includes X-API-Key."""
client = FlacfetchClient(
base_url="http://localhost:8080",
api_key="secret-key-456",
)
headers = client._headers()
assert headers["X-API-Key"] == "secret-key-456"
assert headers["Content-Type"] == "application/json"
class TestFlacfetchClientHealthCheck:
"""Test health_check() method."""
@pytest.mark.asyncio
async def test_health_check_success(self):
"""Test successful health check."""
client = FlacfetchClient(
base_url="http://localhost:8080",
api_key="test-key",
)
mock_response = Mock()
mock_response.json.return_value = {
"status": "healthy",
"transmission": {"available": True, "version": "4.0.5"},
"disk": {"free_gb": 15.5},
}
mock_response.raise_for_status = Mock()
with patch("httpx.AsyncClient") as mock_client_class:
mock_client = AsyncMock()
mock_client.get = AsyncMock(return_value=mock_response)
mock_client.__aenter__ = AsyncMock(return_value=mock_client)
mock_client.__aexit__ = AsyncMock(return_value=None)
mock_client_class.return_value = mock_client
result = await client.health_check()
assert result["status"] == "healthy"
assert result["transmission"]["available"] is True
@pytest.mark.asyncio
async def test_health_check_degraded_status(self):
"""Test health check with degraded status still succeeds."""
client = FlacfetchClient(
base_url="http://localhost:8080",
api_key="test-key",
)
mock_response = Mock()
mock_response.json.return_value = {"status": "degraded"}
mock_response.raise_for_status = Mock()
with patch("httpx.AsyncClient") as mock_client_class:
mock_client = AsyncMock()
mock_client.get = AsyncMock(return_value=mock_response)
mock_client.__aenter__ = AsyncMock(return_value=mock_client)
mock_client.__aexit__ = AsyncMock(return_value=None)
mock_client_class.return_value = mock_client
result = await client.health_check()
assert result["status"] == "degraded"
@pytest.mark.asyncio
async def test_health_check_unhealthy_raises_error(self):
"""Test health check with unhealthy status raises error."""
client = FlacfetchClient(
base_url="http://localhost:8080",
api_key="test-key",
)
mock_response = Mock()
mock_response.json.return_value = {"status": "unhealthy", "error": "Transmission down"}
mock_response.raise_for_status = Mock()
with patch("httpx.AsyncClient") as mock_client_class:
mock_client = AsyncMock()
mock_client.get = AsyncMock(return_value=mock_response)
mock_client.__aenter__ = AsyncMock(return_value=mock_client)
mock_client.__aexit__ = AsyncMock(return_value=None)
mock_client_class.return_value = mock_client
with pytest.raises(FlacfetchServiceError) as exc_info:
await client.health_check()
assert "unhealthy" in str(exc_info.value)
@pytest.mark.asyncio
async def test_health_check_network_error(self):
"""Test health check handles network errors."""
client = FlacfetchClient(
base_url="http://localhost:8080",
api_key="test-key",
)
with patch("httpx.AsyncClient") as mock_client_class:
mock_client = AsyncMock()
mock_client.get = AsyncMock(side_effect=httpx.RequestError("Connection refused"))
mock_client.__aenter__ = AsyncMock(return_value=mock_client)
mock_client.__aexit__ = AsyncMock(return_value=None)
mock_client_class.return_value = mock_client
with pytest.raises(FlacfetchServiceError) as exc_info:
await client.health_check()
assert "Cannot reach" in str(exc_info.value)
class TestFlacfetchClientSearch:
"""Test search() method."""
@pytest.mark.asyncio
async def test_search_success(self):
"""Test successful search returns results."""
client = FlacfetchClient(
base_url="http://localhost:8080",
api_key="test-key",
)
mock_response = Mock()
mock_response.status_code = 200
mock_response.json.return_value = {
"search_id": "search_abc123",
"results": [
{
"title": "Waterloo",
"artist": "ABBA",
"provider": "RED",
"quality": "FLAC 16bit CD",
"seeders": 50,
"index": 0,
}
],
"results_count": 1,
}
mock_response.raise_for_status = Mock()
with patch("httpx.AsyncClient") as mock_client_class:
mock_client = AsyncMock()
mock_client.post = AsyncMock(return_value=mock_response)
mock_client.__aenter__ = AsyncMock(return_value=mock_client)
mock_client.__aexit__ = AsyncMock(return_value=None)
mock_client_class.return_value = mock_client
result = await client.search("ABBA", "Waterloo")
assert result["search_id"] == "search_abc123"
assert len(result["results"]) == 1
assert result["results"][0]["title"] == "Waterloo"
# Verify request was made correctly
mock_client.post.assert_called_once()
call_args = mock_client.post.call_args
assert call_args.kwargs["json"] == {"artist": "ABBA", "title": "Waterloo"}
@pytest.mark.asyncio
async def test_search_no_results_returns_empty(self):
"""Test search with no results returns empty list."""
client = FlacfetchClient(
base_url="http://localhost:8080",
api_key="test-key",
)
mock_response = Mock()
mock_response.status_code = 404
with patch("httpx.AsyncClient") as mock_client_class:
mock_client = AsyncMock()
mock_client.post = AsyncMock(return_value=mock_response)
mock_client.__aenter__ = AsyncMock(return_value=mock_client)
mock_client.__aexit__ = AsyncMock(return_value=None)
mock_client_class.return_value = mock_client
result = await client.search("Unknown", "Artist")
assert result["results"] == []
assert result["search_id"] is None
class TestFlacfetchClientDownload:
"""Test download() method."""
@pytest.mark.asyncio
async def test_download_starts_successfully(self):
"""Test download() returns download_id."""
client = FlacfetchClient(
base_url="http://localhost:8080",
api_key="test-key",
)
mock_response = Mock()
mock_response.json.return_value = {
"download_id": "dl_xyz789",
"status": "queued",
}
mock_response.raise_for_status = Mock()
with patch("httpx.AsyncClient") as mock_client_class:
mock_client = AsyncMock()
mock_client.post = AsyncMock(return_value=mock_response)
mock_client.__aenter__ = AsyncMock(return_value=mock_client)
mock_client.__aexit__ = AsyncMock(return_value=None)
mock_client_class.return_value = mock_client
download_id = await client.download(
search_id="search_abc123",
result_index=0,
)
assert download_id == "dl_xyz789"
@pytest.mark.asyncio
async def test_download_with_gcs_path(self):
"""Test download() includes GCS upload parameters."""
client = FlacfetchClient(
base_url="http://localhost:8080",
api_key="test-key",
)
mock_response = Mock()
mock_response.json.return_value = {"download_id": "dl_xyz789"}
mock_response.raise_for_status = Mock()
with patch("httpx.AsyncClient") as mock_client_class:
mock_client = AsyncMock()
mock_client.post = AsyncMock(return_value=mock_response)
mock_client.__aenter__ = AsyncMock(return_value=mock_client)
mock_client.__aexit__ = AsyncMock(return_value=None)
mock_client_class.return_value = mock_client
await client.download(
search_id="search_abc123",
result_index=0,
gcs_path="uploads/job123/audio/",
)
# Verify GCS params were included
call_args = mock_client.post.call_args
payload = call_args.kwargs["json"]
assert payload["upload_to_gcs"] is True
assert payload["gcs_path"] == "uploads/job123/audio/"
@pytest.mark.asyncio
async def test_download_with_custom_filename(self):
"""Test download() with custom output filename."""
client = FlacfetchClient(
base_url="http://localhost:8080",
api_key="test-key",
)
mock_response = Mock()
mock_response.json.return_value = {"download_id": "dl_xyz789"}
mock_response.raise_for_status = Mock()
with patch("httpx.AsyncClient") as mock_client_class:
mock_client = AsyncMock()
mock_client.post = AsyncMock(return_value=mock_response)
mock_client.__aenter__ = AsyncMock(return_value=mock_client)
mock_client.__aexit__ = AsyncMock(return_value=None)
mock_client_class.return_value = mock_client
await client.download(
search_id="search_abc123",
result_index=0,
output_filename="ABBA - Waterloo",
)
call_args = mock_client.post.call_args
payload = call_args.kwargs["json"]
assert payload["output_filename"] == "ABBA - Waterloo"
class TestFlacfetchClientGetDownloadStatus:
"""Test get_download_status() method."""
@pytest.mark.asyncio
async def test_get_download_status_success(self):
"""Test getting download status."""
client = FlacfetchClient(
base_url="http://localhost:8080",
api_key="test-key",
)
mock_response = Mock()
mock_response.json.return_value = {
"download_id": "dl_xyz789",
"status": "downloading",
"progress": 45.5,
"download_speed_kbps": 1250.0,
"peers": 5,
}
mock_response.raise_for_status = Mock()
with patch("httpx.AsyncClient") as mock_client_class:
mock_client = AsyncMock()
mock_client.get = AsyncMock(return_value=mock_response)
mock_client.__aenter__ = AsyncMock(return_value=mock_client)
mock_client.__aexit__ = AsyncMock(return_value=None)
mock_client_class.return_value = mock_client
status = await client.get_download_status("dl_xyz789")
assert status["status"] == "downloading"
assert status["progress"] == 45.5
@pytest.mark.asyncio
async def test_get_download_status_not_found(self):
"""Test getting status for non-existent download."""
client = FlacfetchClient(
base_url="http://localhost:8080",
api_key="test-key",
)
mock_response = Mock()
mock_response.status_code = 404
mock_response.raise_for_status = Mock(
side_effect=httpx.HTTPStatusError("Not found", request=Mock(), response=mock_response)
)
with patch("httpx.AsyncClient") as mock_client_class:
mock_client = AsyncMock()
mock_client.get = AsyncMock(return_value=mock_response)
mock_client.__aenter__ = AsyncMock(return_value=mock_client)
mock_client.__aexit__ = AsyncMock(return_value=None)
mock_client_class.return_value = mock_client
with pytest.raises(FlacfetchServiceError) as exc_info:
await client.get_download_status("nonexistent")
assert "not found" in str(exc_info.value).lower()
class TestFlacfetchClientWaitForDownload:
"""Test wait_for_download() method."""
@pytest.mark.asyncio
async def test_wait_for_download_completes(self):
"""Test waiting for download to complete."""
client = FlacfetchClient(
base_url="http://localhost:8080",
api_key="test-key",
)
# Mock get_download_status to return complete status
async def mock_get_status(download_id):
return {
"download_id": download_id,
"status": "complete",
"gcs_path": "gs://bucket/uploads/test.flac",
}
client.get_download_status = mock_get_status
result = await client.wait_for_download("dl_xyz789", timeout=10)
assert result["status"] == "complete"
assert result["gcs_path"] == "gs://bucket/uploads/test.flac"
@pytest.mark.asyncio
async def test_wait_for_download_seeding_status_completes(self):
"""Test that 'seeding' status is treated as complete."""
client = FlacfetchClient(
base_url="http://localhost:8080",
api_key="test-key",
)
async def mock_get_status(download_id):
return {
"download_id": download_id,
"status": "seeding",
"output_path": "/downloads/test.flac",
}
client.get_download_status = mock_get_status
result = await client.wait_for_download("dl_xyz789", timeout=10)
assert result["status"] == "seeding"
@pytest.mark.asyncio
async def test_wait_for_download_failed(self):
"""Test wait raises error when download fails."""
client = FlacfetchClient(
base_url="http://localhost:8080",
api_key="test-key",
)
async def mock_get_status(download_id):
return {
"download_id": download_id,
"status": "failed",
"error": "Torrent download failed: no peers",
}
client.get_download_status = mock_get_status
with pytest.raises(FlacfetchServiceError) as exc_info:
await client.wait_for_download("dl_xyz789", timeout=10)
assert "failed" in str(exc_info.value).lower()
assert "no peers" in str(exc_info.value)
@pytest.mark.asyncio
async def test_wait_for_download_cancelled(self):
"""Test wait raises error when download is cancelled."""
client = FlacfetchClient(
base_url="http://localhost:8080",
api_key="test-key",
)
async def mock_get_status(download_id):
return {
"download_id": download_id,
"status": "cancelled",
}
client.get_download_status = mock_get_status
with pytest.raises(FlacfetchServiceError) as exc_info:
await client.wait_for_download("dl_xyz789", timeout=10)
assert "cancelled" in str(exc_info.value).lower()
@pytest.mark.asyncio
async def test_wait_for_download_timeout(self):
"""Test wait raises error on timeout."""
client = FlacfetchClient(
base_url="http://localhost:8080",
api_key="test-key",
)
call_count = 0
async def mock_get_status(download_id):
nonlocal call_count
call_count += 1
return {
"download_id": download_id,
"status": "downloading",
"progress": 10.0 * call_count,
}
client.get_download_status = mock_get_status
with pytest.raises(FlacfetchServiceError) as exc_info:
# Very short timeout with short poll interval
await client.wait_for_download("dl_xyz789", timeout=0.2, poll_interval=0.1)
assert "timed out" in str(exc_info.value).lower()
@pytest.mark.asyncio
async def test_wait_for_download_calls_progress_callback(self):
"""Test progress callback is called during wait."""
client = FlacfetchClient(
base_url="http://localhost:8080",
api_key="test-key",
)
call_count = 0
async def mock_get_status(download_id):
nonlocal call_count
call_count += 1
if call_count >= 3:
return {"status": "complete", "gcs_path": "gs://test"}
return {"status": "downloading", "progress": 33.0 * call_count}
client.get_download_status = mock_get_status
progress_updates = []
def progress_callback(status):
progress_updates.append(status)
await client.wait_for_download(
"dl_xyz789",
timeout=10,
poll_interval=0.01,
progress_callback=progress_callback,
)
# Should have received progress updates
assert len(progress_updates) >= 2
class TestGetFlacfetchClient:
"""Test get_flacfetch_client() singleton factory."""
def test_returns_none_when_not_configured(self):
"""Test returns None when FLACFETCH_API_URL not set."""
reset_flacfetch_client()
with patch("backend.config.get_settings") as mock_settings:
mock_settings.return_value = Mock(
flacfetch_api_url=None,
flacfetch_api_key=None,
)
client = get_flacfetch_client()
assert client is None
def test_returns_none_when_key_missing(self):
"""Test returns None when API key not set."""
reset_flacfetch_client()
with patch("backend.config.get_settings") as mock_settings:
mock_settings.return_value = Mock(
flacfetch_api_url="http://localhost:8080",
flacfetch_api_key=None,
)
client = get_flacfetch_client()
assert client is None
def test_returns_client_when_configured(self):
"""Test returns FlacfetchClient when both URL and key are set."""
reset_flacfetch_client()
with patch("backend.config.get_settings") as mock_settings:
mock_settings.return_value = Mock(
flacfetch_api_url="http://10.0.0.5:8080",
flacfetch_api_key="secret-key",
)
client = get_flacfetch_client()
assert client is not None
assert isinstance(client, FlacfetchClient)
assert client.base_url == "http://10.0.0.5:8080"
assert client.api_key == "secret-key"
def test_returns_same_instance(self):
"""Test singleton returns same instance."""
reset_flacfetch_client()
with patch("backend.config.get_settings") as mock_settings:
mock_settings.return_value = Mock(
flacfetch_api_url="http://localhost:8080",
flacfetch_api_key="test-key",
)
client1 = get_flacfetch_client()
client2 = get_flacfetch_client()
assert client1 is client2
def test_reset_clears_singleton(self):
"""Test reset_flacfetch_client() clears the singleton."""
reset_flacfetch_client()
with patch("backend.config.get_settings") as mock_settings:
mock_settings.return_value = Mock(
flacfetch_api_url="http://localhost:8080",
flacfetch_api_key="test-key",
)
client1 = get_flacfetch_client()
reset_flacfetch_client()
# Change the settings
mock_settings.return_value = Mock(
flacfetch_api_url="http://other:8080",
flacfetch_api_key="other-key",
)
client2 = get_flacfetch_client()
assert client1 is not client2
assert client2.base_url == "http://other:8080"
-524
View File
@@ -1,524 +0,0 @@
"""
Tests for gdrive_service.py - Google Drive file operations.
These tests mock the Google API client and Secret Manager to verify:
- Credential loading from Secret Manager
- Folder creation and lookup
- File uploads with proper MIME types
- Public share folder structure uploads
"""
import json
import os
import pytest
from unittest.mock import Mock, MagicMock, patch
class TestGoogleDriveServiceInit:
"""Test GoogleDriveService initialization."""
@patch("backend.services.gdrive_service.get_settings")
def test_init_creates_service(self, mock_get_settings):
"""Test initialization creates service with settings."""
from backend.services.gdrive_service import GoogleDriveService
mock_settings = Mock()
mock_get_settings.return_value = mock_settings
service = GoogleDriveService()
assert service.settings == mock_settings
assert service._service is None
assert service._credentials_data is None
assert service._loaded is False
class TestLoadCredentials:
"""Test _load_credentials method."""
@patch("backend.services.gdrive_service.get_settings")
def test_load_credentials_success(self, mock_get_settings):
"""Test successful credential loading from Secret Manager."""
from backend.services.gdrive_service import GoogleDriveService
mock_settings = Mock()
mock_settings.get_secret.return_value = json.dumps({
"token": "access-token",
"refresh_token": "refresh-token",
"client_id": "client-id",
"client_secret": "client-secret",
})
mock_get_settings.return_value = mock_settings
service = GoogleDriveService()
creds = service._load_credentials()
assert creds is not None
assert creds["refresh_token"] == "refresh-token"
mock_settings.get_secret.assert_called_once_with("gdrive-oauth-credentials")
@patch("backend.services.gdrive_service.get_settings")
def test_load_credentials_fallback_to_youtube(self, mock_get_settings):
"""Test fallback to YouTube credentials when Drive creds not found."""
from backend.services.gdrive_service import GoogleDriveService
mock_settings = Mock()
# First call returns None (Drive creds not found)
# Second call returns YouTube creds
mock_settings.get_secret.side_effect = [
None,
json.dumps({
"refresh_token": "youtube-token",
"client_id": "youtube-id",
"client_secret": "youtube-secret",
})
]
mock_get_settings.return_value = mock_settings
service = GoogleDriveService()
creds = service._load_credentials()
assert creds is not None
assert creds["refresh_token"] == "youtube-token"
assert mock_settings.get_secret.call_count == 2
@patch("backend.services.gdrive_service.get_settings")
def test_load_credentials_not_found(self, mock_get_settings):
"""Test handling when no credentials found."""
from backend.services.gdrive_service import GoogleDriveService
mock_settings = Mock()
mock_settings.get_secret.return_value = None
mock_get_settings.return_value = mock_settings
service = GoogleDriveService()
creds = service._load_credentials()
assert creds is None
assert service._loaded is True
@patch("backend.services.gdrive_service.get_settings")
def test_load_credentials_missing_required_fields(self, mock_get_settings):
"""Test handling when credentials missing required fields."""
from backend.services.gdrive_service import GoogleDriveService
mock_settings = Mock()
mock_settings.get_secret.return_value = json.dumps({
"token": "access-token",
# Missing: refresh_token, client_id, client_secret
})
mock_get_settings.return_value = mock_settings
service = GoogleDriveService()
creds = service._load_credentials()
assert creds is None
@patch("backend.services.gdrive_service.get_settings")
def test_load_credentials_cached(self, mock_get_settings):
"""Test credentials are cached after first load."""
from backend.services.gdrive_service import GoogleDriveService
mock_settings = Mock()
mock_settings.get_secret.return_value = json.dumps({
"refresh_token": "token",
"client_id": "id",
"client_secret": "secret",
})
mock_get_settings.return_value = mock_settings
service = GoogleDriveService()
creds1 = service._load_credentials()
creds2 = service._load_credentials()
assert creds1 == creds2
assert mock_settings.get_secret.call_count == 1
class TestIsConfigured:
"""Test is_configured property."""
@patch("backend.services.gdrive_service.get_settings")
def test_is_configured_true(self, mock_get_settings):
"""Test is_configured returns True when credentials exist."""
from backend.services.gdrive_service import GoogleDriveService
mock_settings = Mock()
mock_settings.get_secret.return_value = json.dumps({
"refresh_token": "token",
"client_id": "id",
"client_secret": "secret",
})
mock_get_settings.return_value = mock_settings
service = GoogleDriveService()
assert service.is_configured is True
@patch("backend.services.gdrive_service.get_settings")
def test_is_configured_false(self, mock_get_settings):
"""Test is_configured returns False when no credentials."""
from backend.services.gdrive_service import GoogleDriveService
mock_settings = Mock()
mock_settings.get_secret.return_value = None
mock_get_settings.return_value = mock_settings
service = GoogleDriveService()
assert service.is_configured is False
class TestDriveService:
"""Test service property."""
@patch("googleapiclient.discovery.build")
@patch("google.oauth2.credentials.Credentials")
@patch("backend.services.gdrive_service.get_settings")
def test_service_creates_drive_client(
self, mock_get_settings, mock_creds_class, mock_build
):
"""Test service property creates Google Drive API client."""
from backend.services.gdrive_service import GoogleDriveService
mock_settings = Mock()
mock_settings.get_secret.return_value = json.dumps({
"token": "access-token",
"refresh_token": "refresh-token",
"client_id": "client-id",
"client_secret": "client-secret",
})
mock_get_settings.return_value = mock_settings
mock_creds = Mock()
mock_creds_class.return_value = mock_creds
mock_drive_service = Mock()
mock_build.return_value = mock_drive_service
service = GoogleDriveService()
result = service.service
mock_build.assert_called_once_with("drive", "v3", credentials=mock_creds)
assert result == mock_drive_service
@patch("backend.services.gdrive_service.get_settings")
def test_service_raises_on_missing_credentials(self, mock_get_settings):
"""Test service raises RuntimeError when credentials missing."""
from backend.services.gdrive_service import GoogleDriveService
mock_settings = Mock()
mock_settings.get_secret.return_value = None
mock_get_settings.return_value = mock_settings
service = GoogleDriveService()
with pytest.raises(RuntimeError) as exc_info:
_ = service.service
assert "not configured" in str(exc_info.value)
class TestGetOrCreateFolder:
"""Test get_or_create_folder method."""
@patch("backend.services.gdrive_service.get_settings")
def test_get_existing_folder(self, mock_get_settings):
"""Test finding an existing folder."""
from backend.services.gdrive_service import GoogleDriveService
mock_settings = Mock()
mock_settings.get_secret.return_value = json.dumps({
"refresh_token": "token",
"client_id": "id",
"client_secret": "secret",
})
mock_get_settings.return_value = mock_settings
service = GoogleDriveService()
# Mock the Drive API
mock_files_api = Mock()
mock_list_result = Mock()
mock_list_result.execute.return_value = {
"files": [{"id": "folder-id-123", "name": "MP4"}]
}
mock_files_api.list.return_value = mock_list_result
mock_drive = Mock()
mock_drive.files.return_value = mock_files_api
service._service = mock_drive
service._loaded = True
service._credentials_data = {"refresh_token": "token", "client_id": "id", "client_secret": "secret"}
folder_id = service.get_or_create_folder("parent-123", "MP4")
assert folder_id == "folder-id-123"
@patch("backend.services.gdrive_service.get_settings")
def test_create_new_folder(self, mock_get_settings):
"""Test creating a new folder when it doesn't exist."""
from backend.services.gdrive_service import GoogleDriveService
mock_settings = Mock()
mock_settings.get_secret.return_value = json.dumps({
"refresh_token": "token",
"client_id": "id",
"client_secret": "secret",
})
mock_get_settings.return_value = mock_settings
service = GoogleDriveService()
# Mock the Drive API
mock_files_api = Mock()
# list returns empty (folder doesn't exist)
mock_list_result = Mock()
mock_list_result.execute.return_value = {"files": []}
mock_files_api.list.return_value = mock_list_result
# create returns new folder
mock_create_result = Mock()
mock_create_result.execute.return_value = {"id": "new-folder-id"}
mock_files_api.create.return_value = mock_create_result
mock_drive = Mock()
mock_drive.files.return_value = mock_files_api
service._service = mock_drive
service._loaded = True
service._credentials_data = {"refresh_token": "token", "client_id": "id", "client_secret": "secret"}
folder_id = service.get_or_create_folder("parent-123", "NewFolder")
assert folder_id == "new-folder-id"
mock_files_api.create.assert_called_once()
class TestUploadFile:
"""Test upload_file method."""
@patch("googleapiclient.http.MediaFileUpload")
@patch("backend.services.gdrive_service.get_settings")
def test_upload_file_success(self, mock_get_settings, mock_media, tmp_path):
"""Test uploading a file to Drive."""
from backend.services.gdrive_service import GoogleDriveService
mock_settings = Mock()
mock_settings.get_secret.return_value = json.dumps({
"refresh_token": "token",
"client_id": "id",
"client_secret": "secret",
})
mock_get_settings.return_value = mock_settings
# Create test file
test_file = tmp_path / "test.mp4"
test_file.write_bytes(b"video content")
service = GoogleDriveService()
mock_files_api = Mock()
# list returns empty (no existing file)
mock_list_result = Mock()
mock_list_result.execute.return_value = {"files": []}
mock_files_api.list.return_value = mock_list_result
# create returns new file
mock_create_result = Mock()
mock_create_result.execute.return_value = {"id": "file-id-123"}
mock_files_api.create.return_value = mock_create_result
mock_drive = Mock()
mock_drive.files.return_value = mock_files_api
service._service = mock_drive
service._loaded = True
service._credentials_data = {"refresh_token": "token", "client_id": "id", "client_secret": "secret"}
file_id = service.upload_file(str(test_file), "parent-123", "video.mp4")
assert file_id == "file-id-123"
mock_media.assert_called_once()
# Verify MIME type was set correctly
call_kwargs = mock_media.call_args.kwargs
assert call_kwargs["mimetype"] == "video/mp4"
@patch("googleapiclient.http.MediaFileUpload")
@patch("backend.services.gdrive_service.get_settings")
def test_upload_file_replaces_existing(self, mock_get_settings, mock_media, tmp_path):
"""Test upload deletes existing file before upload."""
from backend.services.gdrive_service import GoogleDriveService
mock_settings = Mock()
mock_settings.get_secret.return_value = json.dumps({
"refresh_token": "token",
"client_id": "id",
"client_secret": "secret",
})
mock_get_settings.return_value = mock_settings
test_file = tmp_path / "test.mp4"
test_file.write_bytes(b"content")
service = GoogleDriveService()
mock_files_api = Mock()
# list returns existing file
mock_list_result = Mock()
mock_list_result.execute.return_value = {
"files": [{"id": "existing-file-id"}]
}
mock_files_api.list.return_value = mock_list_result
mock_delete_result = Mock()
mock_delete_result.execute.return_value = {}
mock_files_api.delete.return_value = mock_delete_result
mock_create_result = Mock()
mock_create_result.execute.return_value = {"id": "new-file-id"}
mock_files_api.create.return_value = mock_create_result
mock_drive = Mock()
mock_drive.files.return_value = mock_files_api
service._service = mock_drive
service._loaded = True
service._credentials_data = {"refresh_token": "token", "client_id": "id", "client_secret": "secret"}
file_id = service.upload_file(str(test_file), "parent-123", "video.mp4")
# Should have deleted existing file
mock_files_api.delete.assert_called_once_with(fileId="existing-file-id")
assert file_id == "new-file-id"
class TestUploadToPublicShare:
"""Test upload_to_public_share method."""
@patch("backend.services.gdrive_service.get_settings")
def test_upload_to_public_share(self, mock_get_settings, tmp_path):
"""Test uploading files to public share folder structure."""
from backend.services.gdrive_service import GoogleDriveService
mock_settings = Mock()
mock_settings.get_secret.return_value = json.dumps({
"refresh_token": "token",
"client_id": "id",
"client_secret": "secret",
})
mock_get_settings.return_value = mock_settings
# Create test files
mp4_file = tmp_path / "output.mp4"
mp4_file.write_bytes(b"4k video")
mp4_720_file = tmp_path / "output_720p.mp4"
mp4_720_file.write_bytes(b"720p video")
cdg_file = tmp_path / "output.zip"
cdg_file.write_bytes(b"cdg package")
service = GoogleDriveService()
# Mock methods
with patch.object(service, "get_or_create_folder") as mock_get_folder:
with patch.object(service, "upload_file") as mock_upload:
mock_get_folder.side_effect = [
"mp4-folder-id",
"mp4-720-folder-id",
"cdg-folder-id",
]
mock_upload.side_effect = [
"mp4-file-id",
"720p-file-id",
"cdg-file-id",
]
output_files = {
"final_karaoke_lossy_mp4": str(mp4_file),
"final_karaoke_lossy_720p_mp4": str(mp4_720_file),
"final_karaoke_cdg_zip": str(cdg_file),
}
result = service.upload_to_public_share(
root_folder_id="root-123",
brand_code="NOMAD-1163",
base_name="Artist - Title",
output_files=output_files,
)
# Should have created/found 3 folders
assert mock_get_folder.call_count == 3
# Should have uploaded 3 files
assert mock_upload.call_count == 3
# Check result
assert result["mp4"] == "mp4-file-id"
assert result["mp4_720p"] == "720p-file-id"
assert result["cdg"] == "cdg-file-id"
@patch("backend.services.gdrive_service.get_settings")
def test_upload_to_public_share_skips_missing_files(
self, mock_get_settings, tmp_path
):
"""Test upload skips files that don't exist."""
from backend.services.gdrive_service import GoogleDriveService
mock_settings = Mock()
mock_settings.get_secret.return_value = json.dumps({
"refresh_token": "token",
"client_id": "id",
"client_secret": "secret",
})
mock_get_settings.return_value = mock_settings
# Only create one file
mp4_file = tmp_path / "output.mp4"
mp4_file.write_bytes(b"video")
service = GoogleDriveService()
with patch.object(service, "get_or_create_folder") as mock_get_folder:
with patch.object(service, "upload_file") as mock_upload:
mock_get_folder.return_value = "folder-id"
mock_upload.return_value = "file-id"
output_files = {
"final_karaoke_lossy_mp4": str(mp4_file),
"final_karaoke_lossy_720p_mp4": "/nonexistent/file.mp4",
"final_karaoke_cdg_zip": None,
}
result = service.upload_to_public_share(
root_folder_id="root-123",
brand_code="CODE",
base_name="Name",
output_files=output_files,
)
# Should only upload the one file that exists
assert mock_upload.call_count == 1
assert len(result) == 1
class TestGetGdriveService:
"""Test get_gdrive_service singleton."""
@patch("backend.services.gdrive_service.get_settings")
def test_get_gdrive_service_singleton(self, mock_get_settings):
"""Test get_gdrive_service returns singleton instance."""
from backend.services.gdrive_service import get_gdrive_service
import backend.services.gdrive_service as gdrive_module
# Reset singleton
gdrive_module._gdrive_service = None
mock_settings = Mock()
mock_settings.get_secret.return_value = None
mock_get_settings.return_value = mock_settings
service1 = get_gdrive_service()
service2 = get_gdrive_service()
assert service1 is service2
-431
View File
@@ -1,431 +0,0 @@
"""
API endpoint tests for instrumental review functionality.
Tests for:
- GET /api/jobs/{job_id}/instrumental-analysis
- GET /api/jobs/{job_id}/audio-stream/{stem_type}
- POST /api/jobs/{job_id}/create-custom-instrumental
- GET /api/jobs/{job_id}/waveform-data
"""
import pytest
from datetime import datetime
from unittest.mock import MagicMock, patch, AsyncMock
from backend.models.job import Job, JobStatus
@pytest.fixture
def mock_job_manager():
"""Create a mock job manager."""
with patch('backend.api.routes.jobs.job_manager') as mock:
yield mock
@pytest.fixture
def mock_storage_service():
"""Create a mock storage service."""
with patch('backend.api.routes.jobs.StorageService') as mock:
mock_instance = MagicMock()
mock_instance.generate_signed_url.return_value = "https://storage.googleapis.com/signed-url"
mock.return_value = mock_instance
yield mock_instance
@pytest.fixture
def sample_job():
"""Create a sample job for testing."""
return Job(
job_id="test-job-123",
artist="Test Artist",
title="Test Song",
status=JobStatus.AWAITING_INSTRUMENTAL_SELECTION,
created_at=datetime.now(),
updated_at=datetime.now(),
file_urls={
"stems": {
"instrumental_clean": "jobs/test/stems/instrumental_clean.flac",
"backing_vocals": "jobs/test/stems/backing_vocals.flac",
"instrumental_with_backing": "jobs/test/stems/instrumental_with_backing.flac",
},
"analysis": {
"backing_vocals_waveform": "jobs/test/analysis/waveform.png",
},
},
state_data={
"backing_vocals_analysis": {
"has_audible_content": True,
"total_duration_seconds": 180.0,
"audible_segments": [
{
"start_seconds": 10.0,
"end_seconds": 20.0,
"duration_seconds": 10.0,
"avg_amplitude_db": -25.0,
},
{
"start_seconds": 60.0,
"end_seconds": 80.0,
"duration_seconds": 20.0,
"avg_amplitude_db": -30.0,
},
],
"recommended_selection": "review_needed",
"total_audible_duration_seconds": 30.0,
"audible_percentage": 16.67,
"silence_threshold_db": -40.0,
}
},
)
@pytest.fixture
def silent_job():
"""Create a job with silent backing vocals."""
return Job(
job_id="test-job-silent",
artist="Test Artist",
title="Test Song",
status=JobStatus.AWAITING_INSTRUMENTAL_SELECTION,
created_at=datetime.now(),
updated_at=datetime.now(),
file_urls={
"stems": {
"instrumental_clean": "jobs/test/stems/instrumental_clean.flac",
"backing_vocals": "jobs/test/stems/backing_vocals.flac",
"instrumental_with_backing": "jobs/test/stems/instrumental_with_backing.flac",
},
"analysis": {
"backing_vocals_waveform": "jobs/test/analysis/waveform.png",
},
},
state_data={
"backing_vocals_analysis": {
"has_audible_content": False,
"total_duration_seconds": 180.0,
"audible_segments": [],
"recommended_selection": "clean",
"total_audible_duration_seconds": 0.0,
"audible_percentage": 0.0,
"silence_threshold_db": -40.0,
}
},
)
class TestGetInstrumentalAnalysis:
"""Tests for GET /api/jobs/{job_id}/instrumental-analysis."""
@pytest.mark.asyncio
async def test_get_analysis_returns_data(self, mock_job_manager, mock_storage_service, sample_job):
"""GET /instrumental-analysis should return analysis data."""
mock_job_manager.get_job.return_value = sample_job
from backend.api.routes.jobs import get_instrumental_analysis
result = await get_instrumental_analysis("test-job-123")
assert result["job_id"] == "test-job-123"
assert result["artist"] == "Test Artist"
assert result["title"] == "Test Song"
assert result["analysis"]["has_audible_content"] is True
assert len(result["analysis"]["audible_segments"]) == 2
@pytest.mark.asyncio
async def test_get_analysis_includes_audio_urls(self, mock_job_manager, mock_storage_service, sample_job):
"""GET /instrumental-analysis should include audio URLs."""
mock_job_manager.get_job.return_value = sample_job
from backend.api.routes.jobs import get_instrumental_analysis
result = await get_instrumental_analysis("test-job-123")
assert "audio_urls" in result
assert "clean_instrumental" in result["audio_urls"]
assert "backing_vocals" in result["audio_urls"]
assert "with_backing" in result["audio_urls"]
@pytest.mark.asyncio
async def test_get_analysis_includes_waveform_url(self, mock_job_manager, sample_job):
"""GET /instrumental-analysis should include waveform URL."""
mock_job_manager.get_job.return_value = sample_job
# Need to mock StorageService at module level for the signed URL
with patch('backend.api.routes.jobs.StorageService') as mock_storage_cls:
mock_storage = MagicMock()
mock_storage.generate_signed_url.return_value = "https://storage.googleapis.com/signed-url"
mock_storage_cls.return_value = mock_storage
from backend.api.routes.jobs import get_instrumental_analysis
result = await get_instrumental_analysis("test-job-123")
assert "waveform_url" in result
# Waveform URL should be generated from the file_urls
assert result["waveform_url"] == "https://storage.googleapis.com/signed-url"
@pytest.mark.asyncio
async def test_get_analysis_silent_audio(self, mock_job_manager, mock_storage_service, silent_job):
"""GET /instrumental-analysis should correctly report silent audio."""
mock_job_manager.get_job.return_value = silent_job
from backend.api.routes.jobs import get_instrumental_analysis
result = await get_instrumental_analysis("test-job-silent")
assert result["analysis"]["has_audible_content"] is False
assert result["analysis"]["recommended_selection"] == "clean"
assert len(result["analysis"]["audible_segments"]) == 0
@pytest.mark.asyncio
async def test_get_analysis_job_not_found(self, mock_job_manager):
"""GET /instrumental-analysis should return 404 for non-existent job."""
mock_job_manager.get_job.return_value = None
from backend.api.routes.jobs import get_instrumental_analysis
from fastapi import HTTPException
with pytest.raises(HTTPException) as exc_info:
await get_instrumental_analysis("non-existent")
assert exc_info.value.status_code == 404
@pytest.mark.asyncio
async def test_get_analysis_wrong_status(self, mock_job_manager, sample_job):
"""GET /instrumental-analysis should return 400 for wrong job status."""
sample_job.status = JobStatus.PENDING
mock_job_manager.get_job.return_value = sample_job
from backend.api.routes.jobs import get_instrumental_analysis
from fastapi import HTTPException
with pytest.raises(HTTPException) as exc_info:
await get_instrumental_analysis("test-job-123")
assert exc_info.value.status_code == 400
class TestAudioStream:
"""Tests for GET /api/jobs/{job_id}/audio-stream/{stem_type}."""
def test_stream_backing_vocals_valid(self, mock_job_manager, sample_job):
"""Should return redirect for backing vocals stream."""
mock_job_manager.get_job.return_value = sample_job
from backend.api.routes.jobs import stream_audio
# This will try to stream from GCS which we can't test easily
# Instead, verify the function exists and has correct signature
assert callable(stream_audio)
@pytest.mark.asyncio
async def test_stream_invalid_stem_type(self, mock_job_manager, sample_job):
"""Should return 400 for invalid stem type."""
mock_job_manager.get_job.return_value = sample_job
from backend.api.routes.jobs import stream_audio
from fastapi import HTTPException
with pytest.raises(HTTPException) as exc_info:
await stream_audio("test-job-123", "invalid_stem")
assert exc_info.value.status_code == 400
@pytest.mark.asyncio
async def test_stream_job_not_found(self, mock_job_manager):
"""Should return 404 for non-existent job."""
mock_job_manager.get_job.return_value = None
from backend.api.routes.jobs import stream_audio
from fastapi import HTTPException
with pytest.raises(HTTPException) as exc_info:
await stream_audio("non-existent", "backing_vocals")
assert exc_info.value.status_code == 404
class TestCreateCustomInstrumental:
"""Tests for POST /api/jobs/{job_id}/create-custom-instrumental."""
@pytest.fixture
def mock_audio_editing_service(self):
"""Mock the audio editing service."""
with patch('backend.api.routes.jobs.AudioEditingService') as mock:
mock_instance = MagicMock()
mock_instance.create_custom_instrumental.return_value = MagicMock(
output_path="jobs/test/stems/custom_instrumental.flac",
mute_regions_applied=[
MagicMock(start_seconds=10.0, end_seconds=20.0),
],
total_muted_duration_seconds=10.0,
output_duration_seconds=180.0,
)
mock.return_value = mock_instance
yield mock_instance
def test_create_custom_validates_mute_regions(self):
"""Should validate mute regions at model level."""
from backend.models.requests import CreateCustomInstrumentalRequest, MuteRegionRequest
from pydantic import ValidationError
# Empty mute regions should fail at model validation
with pytest.raises(ValidationError) as exc_info:
CreateCustomInstrumentalRequest(mute_regions=[])
assert "At least one mute region is required" in str(exc_info.value)
@pytest.mark.asyncio
async def test_create_custom_job_not_found(self, mock_job_manager):
"""Should return 404 for non-existent job."""
mock_job_manager.get_job.return_value = None
from backend.api.routes.jobs import create_custom_instrumental
from backend.models.requests import CreateCustomInstrumentalRequest, MuteRegionRequest
from fastapi import HTTPException
request = CreateCustomInstrumentalRequest(
mute_regions=[MuteRegionRequest(start_seconds=10.0, end_seconds=20.0)]
)
with pytest.raises(HTTPException) as exc_info:
await create_custom_instrumental("non-existent", request)
assert exc_info.value.status_code == 404
@pytest.mark.asyncio
async def test_create_custom_wrong_status(self, mock_job_manager, sample_job):
"""Should return 400 for wrong job status."""
sample_job.status = JobStatus.PENDING
mock_job_manager.get_job.return_value = sample_job
from backend.api.routes.jobs import create_custom_instrumental
from backend.models.requests import CreateCustomInstrumentalRequest, MuteRegionRequest
from fastapi import HTTPException
request = CreateCustomInstrumentalRequest(
mute_regions=[MuteRegionRequest(start_seconds=10.0, end_seconds=20.0)]
)
with pytest.raises(HTTPException) as exc_info:
await create_custom_instrumental("test-job-123", request)
assert exc_info.value.status_code == 400
class TestGetWaveformData:
"""Tests for GET /api/jobs/{job_id}/waveform-data."""
@pytest.mark.asyncio
async def test_get_waveform_data_job_not_found(self, mock_job_manager):
"""Should return 404 for non-existent job."""
mock_job_manager.get_job.return_value = None
from backend.api.routes.jobs import get_waveform_data
from fastapi import HTTPException
with pytest.raises(HTTPException) as exc_info:
await get_waveform_data("non-existent")
assert exc_info.value.status_code == 404
@pytest.mark.asyncio
async def test_get_waveform_data_wrong_status(self, mock_job_manager, sample_job):
"""Should return 400 for wrong job status."""
sample_job.status = JobStatus.PENDING
mock_job_manager.get_job.return_value = sample_job
from backend.api.routes.jobs import get_waveform_data
from fastapi import HTTPException
with pytest.raises(HTTPException) as exc_info:
await get_waveform_data("test-job-123")
assert exc_info.value.status_code == 400
def test_waveform_data_endpoint_exists(self):
"""Verify the endpoint function exists and is callable."""
from backend.api.routes.jobs import get_waveform_data
assert callable(get_waveform_data)
# Verify it's an async function
import inspect
assert inspect.iscoroutinefunction(get_waveform_data)
class TestRequestModels:
"""Tests for API request models."""
def test_mute_region_request_valid(self):
"""MuteRegionRequest should accept valid values."""
from backend.models.requests import MuteRegionRequest
region = MuteRegionRequest(start_seconds=10.0, end_seconds=20.0)
assert region.start_seconds == 10.0
assert region.end_seconds == 20.0
def test_mute_region_request_invalid_start(self):
"""MuteRegionRequest should reject negative start."""
from backend.models.requests import MuteRegionRequest
from pydantic import ValidationError
with pytest.raises(ValidationError):
MuteRegionRequest(start_seconds=-1.0, end_seconds=20.0)
def test_mute_region_request_invalid_order(self):
"""MuteRegionRequest should reject end before start."""
from backend.models.requests import MuteRegionRequest
from pydantic import ValidationError
with pytest.raises(ValidationError):
MuteRegionRequest(start_seconds=20.0, end_seconds=10.0)
def test_create_custom_instrumental_request_valid(self):
"""CreateCustomInstrumentalRequest should accept valid mute regions."""
from backend.models.requests import (
CreateCustomInstrumentalRequest,
MuteRegionRequest,
)
request = CreateCustomInstrumentalRequest(
mute_regions=[
MuteRegionRequest(start_seconds=10.0, end_seconds=20.0),
MuteRegionRequest(start_seconds=60.0, end_seconds=80.0),
]
)
assert len(request.mute_regions) == 2
class TestInstrumentalSelectionExtension:
"""Tests for extended instrumental selection (including 'custom')."""
def test_instrumental_selection_accepts_custom(self):
"""InstrumentalSelection should accept 'custom' as valid selection."""
from backend.models.requests import InstrumentalSelection
selection = InstrumentalSelection(selection="custom")
assert selection.selection == "custom"
def test_instrumental_selection_accepts_clean(self):
"""InstrumentalSelection should accept 'clean' as valid selection."""
from backend.models.requests import InstrumentalSelection
selection = InstrumentalSelection(selection="clean")
assert selection.selection == "clean"
def test_instrumental_selection_accepts_with_backing(self):
"""InstrumentalSelection should accept 'with_backing' as valid selection."""
from backend.models.requests import InstrumentalSelection
selection = InstrumentalSelection(selection="with_backing")
assert selection.selection == "with_backing"
def test_instrumental_selection_rejects_invalid(self):
"""InstrumentalSelection should reject invalid values."""
from backend.models.requests import InstrumentalSelection
from pydantic import ValidationError
with pytest.raises(ValidationError):
InstrumentalSelection(selection="invalid_option")
-343
View File
@@ -1,343 +0,0 @@
"""
Unit tests for internal.py API routes using FastAPI TestClient.
Internal routes are used for worker communication within Cloud Run.
These endpoints require admin authentication.
With Cloud Tasks integration, these endpoints include idempotency checks
to prevent duplicate processing on retries.
"""
import os
import pytest
from datetime import datetime, UTC
from unittest.mock import MagicMock, AsyncMock, patch
from fastapi.testclient import TestClient
from backend.models.job import Job, JobStatus
# Set test admin token for auth
os.environ.setdefault('ADMIN_TOKENS', 'test-admin-token')
def _create_mock_job(job_id: str = "test123") -> Job:
"""Create a mock Job object for testing."""
return Job(
job_id=job_id,
status=JobStatus.PROCESSING,
created_at=datetime.now(UTC),
updated_at=datetime.now(UTC),
state_data={}, # Empty state_data - no worker progress yet
)
@pytest.fixture
def client():
"""Create TestClient with mocked workers and job manager."""
mock_creds = MagicMock()
mock_creds.universe_domain = 'googleapis.com'
# Mock the worker functions at the internal routes module level
mock_audio = AsyncMock()
mock_lyrics = AsyncMock()
mock_screens = AsyncMock()
mock_video = AsyncMock()
mock_render = AsyncMock()
# Mock JobManager for idempotency checks
mock_job_manager = MagicMock()
mock_job_manager.get_job.return_value = _create_mock_job()
mock_job_manager.update_state_data.return_value = None
with patch('backend.api.routes.internal.process_audio_separation', mock_audio), \
patch('backend.api.routes.internal.process_lyrics_transcription', mock_lyrics), \
patch('backend.api.routes.internal.generate_screens', mock_screens), \
patch('backend.api.routes.internal.generate_video', mock_video), \
patch('backend.api.routes.internal.process_render_video', mock_render), \
patch('backend.api.routes.internal.JobManager', return_value=mock_job_manager), \
patch('backend.services.firestore_service.firestore'), \
patch('backend.services.storage_service.storage'), \
patch('google.auth.default', return_value=(mock_creds, 'test-project')):
from backend.main import app
yield TestClient(app)
@pytest.fixture
def auth_headers():
"""Auth headers for internal API testing."""
return {"Authorization": "Bearer test-admin-token"}
class TestAudioWorkerEndpoint:
"""Tests for POST /api/internal/workers/audio."""
def test_audio_worker_returns_200(self, client, auth_headers):
"""Test audio worker endpoint returns 200 with auth."""
response = client.post(
"/api/internal/workers/audio",
headers=auth_headers,
json={"job_id": "test123"}
)
assert response.status_code == 200
def test_audio_worker_accepts_job_id(self, client, auth_headers):
"""Test audio worker accepts job_id in body."""
response = client.post(
"/api/internal/workers/audio",
headers=auth_headers,
json={"job_id": "any-job-id"}
)
assert response.status_code == 200
def test_audio_worker_requires_job_id(self, client, auth_headers):
"""Test audio worker requires job_id."""
response = client.post(
"/api/internal/workers/audio",
headers=auth_headers,
json={}
)
assert response.status_code == 422
def test_audio_worker_requires_auth(self, client, auth_headers):
"""Test audio worker requires authentication."""
response = client.post(
"/api/internal/workers/audio",
json={"job_id": "test123"}
)
assert response.status_code == 401
class TestLyricsWorkerEndpoint:
"""Tests for POST /api/internal/workers/lyrics."""
def test_lyrics_worker_returns_200(self, client, auth_headers):
"""Test lyrics worker endpoint returns 200 with auth."""
response = client.post(
"/api/internal/workers/lyrics",
headers=auth_headers,
json={"job_id": "test123"}
)
assert response.status_code == 200
def test_lyrics_worker_requires_job_id(self, client, auth_headers):
"""Test lyrics worker requires job_id."""
response = client.post(
"/api/internal/workers/lyrics",
headers=auth_headers,
json={}
)
assert response.status_code == 422
def test_lyrics_worker_requires_auth(self, client, auth_headers):
"""Test lyrics worker requires authentication."""
response = client.post(
"/api/internal/workers/lyrics",
json={"job_id": "test123"}
)
assert response.status_code == 401
class TestScreensWorkerEndpoint:
"""Tests for POST /api/internal/workers/screens."""
def test_screens_worker_returns_200(self, client, auth_headers):
"""Test screens worker endpoint returns 200 with auth."""
response = client.post(
"/api/internal/workers/screens",
headers=auth_headers,
json={"job_id": "test123"}
)
assert response.status_code == 200
def test_screens_worker_requires_job_id(self, client, auth_headers):
"""Test screens worker requires job_id."""
response = client.post(
"/api/internal/workers/screens",
headers=auth_headers,
json={}
)
assert response.status_code == 422
def test_screens_worker_requires_auth(self, client, auth_headers):
"""Test screens worker requires authentication."""
response = client.post(
"/api/internal/workers/screens",
json={"job_id": "test123"}
)
assert response.status_code == 401
class TestVideoWorkerEndpoint:
"""Tests for POST /api/internal/workers/video."""
def test_video_worker_returns_200(self, client, auth_headers):
"""Test video worker endpoint returns 200 with auth."""
response = client.post(
"/api/internal/workers/video",
headers=auth_headers,
json={"job_id": "test123"}
)
assert response.status_code == 200
def test_video_worker_requires_job_id(self, client, auth_headers):
"""Test video worker requires job_id."""
response = client.post(
"/api/internal/workers/video",
headers=auth_headers,
json={}
)
assert response.status_code == 422
def test_video_worker_requires_auth(self, client, auth_headers):
"""Test video worker requires authentication."""
response = client.post(
"/api/internal/workers/video",
json={"job_id": "test123"}
)
assert response.status_code == 401
class TestRenderVideoWorkerEndpoint:
"""Tests for POST /api/internal/workers/render-video."""
def test_render_video_worker_returns_200(self, client, auth_headers):
"""Test render-video worker endpoint returns 200 with auth."""
response = client.post(
"/api/internal/workers/render-video",
headers=auth_headers,
json={"job_id": "test123"}
)
assert response.status_code == 200
def test_render_video_worker_requires_job_id(self, client, auth_headers):
"""Test render-video worker requires job_id."""
response = client.post(
"/api/internal/workers/render-video",
headers=auth_headers,
json={}
)
assert response.status_code == 422
def test_render_video_worker_requires_auth(self, client, auth_headers):
"""Test render-video worker requires authentication."""
response = client.post(
"/api/internal/workers/render-video",
json={"job_id": "test123"}
)
assert response.status_code == 401
class TestWorkerResponseFormat:
"""Tests for worker response format."""
def test_audio_worker_response_contains_status(self, client, auth_headers):
"""Test audio worker response contains status."""
response = client.post(
"/api/internal/workers/audio",
headers=auth_headers,
json={"job_id": "test123"}
)
data = response.json()
assert "status" in data or "message" in data
def test_lyrics_worker_response_contains_status(self, client, auth_headers):
"""Test lyrics worker response contains status."""
response = client.post(
"/api/internal/workers/lyrics",
headers=auth_headers,
json={"job_id": "test123"}
)
data = response.json()
assert "status" in data or "message" in data
class TestWorkerIdempotency:
"""Tests for worker idempotency checks (Cloud Tasks retry support)."""
def test_audio_worker_returns_already_running_when_in_progress(self, auth_headers):
"""Test audio worker returns already_running if worker is already running."""
mock_creds = MagicMock()
mock_creds.universe_domain = 'googleapis.com'
# Create job with audio already running
mock_job = _create_mock_job()
mock_job.state_data = {'audio_progress': {'stage': 'running'}}
mock_job_manager = MagicMock()
mock_job_manager.get_job.return_value = mock_job
with patch('backend.api.routes.internal.process_audio_separation', AsyncMock()), \
patch('backend.api.routes.internal.JobManager', return_value=mock_job_manager), \
patch('backend.services.firestore_service.firestore'), \
patch('backend.services.storage_service.storage'), \
patch('google.auth.default', return_value=(mock_creds, 'test-project')):
from backend.main import app
client = TestClient(app)
response = client.post(
"/api/internal/workers/audio",
headers=auth_headers,
json={"job_id": "test123"}
)
assert response.status_code == 200
data = response.json()
assert data["status"] == "already_running"
def test_audio_worker_returns_already_complete_when_finished(self, auth_headers):
"""Test audio worker returns already_complete if worker already finished."""
mock_creds = MagicMock()
mock_creds.universe_domain = 'googleapis.com'
# Create job with audio already complete
mock_job = _create_mock_job()
mock_job.state_data = {'audio_progress': {'stage': 'complete'}}
mock_job_manager = MagicMock()
mock_job_manager.get_job.return_value = mock_job
with patch('backend.api.routes.internal.process_audio_separation', AsyncMock()), \
patch('backend.api.routes.internal.JobManager', return_value=mock_job_manager), \
patch('backend.services.firestore_service.firestore'), \
patch('backend.services.storage_service.storage'), \
patch('google.auth.default', return_value=(mock_creds, 'test-project')):
from backend.main import app
client = TestClient(app)
response = client.post(
"/api/internal/workers/audio",
headers=auth_headers,
json={"job_id": "test123"}
)
assert response.status_code == 200
data = response.json()
assert data["status"] == "already_complete"
def test_audio_worker_returns_not_found_when_job_missing(self, auth_headers):
"""Test audio worker returns not_found if job doesn't exist."""
mock_creds = MagicMock()
mock_creds.universe_domain = 'googleapis.com'
# Return None for job not found
mock_job_manager = MagicMock()
mock_job_manager.get_job.return_value = None
with patch('backend.api.routes.internal.process_audio_separation', AsyncMock()), \
patch('backend.api.routes.internal.JobManager', return_value=mock_job_manager), \
patch('backend.services.firestore_service.firestore'), \
patch('backend.services.storage_service.storage'), \
patch('google.auth.default', return_value=(mock_creds, 'test-project')):
from backend.main import app
client = TestClient(app)
response = client.post(
"/api/internal/workers/audio",
headers=auth_headers,
json={"job_id": "nonexistent"}
)
assert response.status_code == 200
data = response.json()
assert data["status"] == "not_found"
-339
View File
@@ -1,339 +0,0 @@
"""
Unit tests for Job Manager service.
Tests the job lifecycle management without requiring actual Firestore connection.
Uses mocking to isolate the business logic.
"""
import pytest
from unittest.mock import Mock, MagicMock, patch
from datetime import datetime, UTC
# Mock Firestore before importing JobManager
import sys
sys.modules['google.cloud.firestore'] = MagicMock()
from backend.services.job_manager import JobManager
from backend.models.job import Job, JobCreate, JobStatus
@pytest.fixture
def mock_firestore_service():
"""Mock FirestoreService."""
with patch('backend.services.job_manager.FirestoreService') as mock:
service = Mock()
mock.return_value = service
yield service
@pytest.fixture
def job_manager(mock_firestore_service):
"""Create JobManager with mocked dependencies."""
return JobManager()
class TestJobCreation:
"""Test job creation logic."""
def test_create_job_with_url(self, job_manager, mock_firestore_service):
"""Test creating a job with YouTube URL."""
job_create = JobCreate(
url="https://youtube.com/watch?v=test",
artist="Test Artist",
title="Test Song"
)
# The actual create_job method creates the job and returns it
job = job_manager.create_job(job_create)
assert job.job_id is not None
assert job.status == JobStatus.PENDING
assert job.url == "https://youtube.com/watch?v=test"
assert job.artist == "Test Artist"
assert job.title == "Test Song"
assert job.progress == 0
# Verify Firestore was called
mock_firestore_service.create_job.assert_called_once()
def test_create_job_without_url(self, job_manager, mock_firestore_service):
"""Test creating a job without URL (for file upload)."""
job_create = JobCreate(
artist="Test Artist",
title="Test Song"
)
job = job_manager.create_job(job_create)
assert job.artist == "Test Artist"
assert job.title == "Test Song"
assert job.url is None
def test_create_job_generates_unique_id(self, job_manager, mock_firestore_service):
"""Test that each job gets a unique ID."""
job_create = JobCreate()
# Create multiple jobs
ids = []
for i in range(5):
mock_firestore_service.create_job.return_value = Job(
job_id=f"test{i}",
status=JobStatus.PENDING,
created_at=datetime.now(UTC),
updated_at=datetime.now(UTC)
)
job = job_manager.create_job(job_create)
ids.append(job.job_id)
# All IDs should be unique
assert len(ids) == len(set(ids))
def test_create_job_sets_initial_status(self, job_manager, mock_firestore_service):
"""Test that new jobs start with PENDING status."""
job_create = JobCreate()
mock_firestore_service.create_job.return_value = Job(
job_id="test123",
status=JobStatus.PENDING,
created_at=datetime.now(UTC),
updated_at=datetime.now(UTC)
)
job = job_manager.create_job(job_create)
assert job.status == JobStatus.PENDING
assert job.progress == 0
def test_create_job_with_distribution_settings(self, job_manager, mock_firestore_service):
"""Test that distribution settings are passed from JobCreate to Job.
This was a bug where brand_prefix, dropbox_path, gdrive_folder_id, and
discord_webhook_url were NOT being passed to the Job constructor.
"""
job_create = JobCreate(
artist="Test Artist",
title="Test Song",
brand_prefix="NOMAD",
discord_webhook_url="https://discord.com/webhook/test",
dropbox_path="/Karaoke/Tracks-Organized",
gdrive_folder_id="1abc123xyz",
enable_youtube_upload=True,
)
job = job_manager.create_job(job_create)
# Verify distribution settings are passed through
assert job.brand_prefix == "NOMAD"
assert job.discord_webhook_url == "https://discord.com/webhook/test"
assert job.dropbox_path == "/Karaoke/Tracks-Organized"
assert job.gdrive_folder_id == "1abc123xyz"
assert job.enable_youtube_upload is True
# Verify Firestore was called with job containing these fields
mock_firestore_service.create_job.assert_called_once()
created_job = mock_firestore_service.create_job.call_args[0][0]
assert created_job.brand_prefix == "NOMAD"
assert created_job.dropbox_path == "/Karaoke/Tracks-Organized"
class TestJobRetrieval:
"""Test job retrieval logic."""
def test_get_existing_job(self, job_manager, mock_firestore_service):
"""Test retrieving an existing job."""
expected_job = Job(
job_id="test123",
status=JobStatus.SEPARATING_STAGE1,
progress=25,
created_at=datetime.now(UTC),
updated_at=datetime.now(UTC)
)
mock_firestore_service.get_job.return_value = expected_job
job = job_manager.get_job("test123")
assert job.job_id == "test123"
assert job.status == JobStatus.SEPARATING_STAGE1
assert job.progress == 25
mock_firestore_service.get_job.assert_called_once_with("test123")
def test_get_nonexistent_job(self, job_manager, mock_firestore_service):
"""Test retrieving a nonexistent job returns None."""
mock_firestore_service.get_job.return_value = None
job = job_manager.get_job("nonexistent")
assert job is None
class TestJobUpdate:
"""Test job update logic."""
def test_update_job_status(self, job_manager, mock_firestore_service):
"""Test updating job status."""
# Setup: job exists
existing_job = Job(
job_id="test123",
status=JobStatus.PENDING,
progress=0,
created_at=datetime.now(UTC),
updated_at=datetime.now(UTC)
)
mock_firestore_service.get_job.return_value = existing_job
# Update status
updates = {"status": JobStatus.SEPARATING_STAGE1, "progress": 25}
job_manager.update_job("test123", updates)
# Verify update was called
mock_firestore_service.update_job.assert_called_once()
call_args = mock_firestore_service.update_job.call_args
assert call_args[0][0] == "test123" # job_id
assert "status" in call_args[0][1] # updates dict
def test_update_job_multiple_fields(self, job_manager, mock_firestore_service):
"""Test updating multiple fields at once."""
existing_job = Job(
job_id="test123",
status=JobStatus.PENDING,
progress=0,
created_at=datetime.now(UTC),
updated_at=datetime.now(UTC),
timeline=[]
)
mock_firestore_service.get_job.return_value = existing_job
updates = {"status": JobStatus.SEPARATING_STAGE1, "progress": 25}
job_manager.update_job("test123", updates)
# Verify update was called
call_args = mock_firestore_service.update_job.call_args
updates_dict = call_args[0][1]
assert "status" in updates_dict
assert "progress" in updates_dict
def test_update_input_media_gcs_path(self, job_manager, mock_firestore_service):
"""Test updating input_media_gcs_path field."""
existing_job = Job(
job_id="test123",
status=JobStatus.PENDING,
progress=0,
created_at=datetime.now(UTC),
updated_at=datetime.now(UTC)
)
mock_firestore_service.get_job.return_value = existing_job
updates = {"input_media_gcs_path": "uploads/test123/file.flac"}
job_manager.update_job("test123", updates)
# Verify update was called with input_media_gcs_path
call_args = mock_firestore_service.update_job.call_args
updates_dict = call_args[0][1]
assert "input_media_gcs_path" in updates_dict
assert updates_dict["input_media_gcs_path"] == "uploads/test123/file.flac"
class TestJobStatusTransitions:
"""Test valid job status transitions."""
def test_pending_to_downloading(self, job_manager, mock_firestore_service):
"""Test transition from PENDING to DOWNLOADING."""
existing_job = Job(
job_id="test123",
status=JobStatus.PENDING,
progress=0,
created_at=datetime.now(UTC),
updated_at=datetime.now(UTC)
)
mock_firestore_service.get_job.return_value = existing_job
job_manager.update_job("test123", {"status": JobStatus.DOWNLOADING})
# Should succeed (valid transition)
mock_firestore_service.update_job.assert_called_once()
def test_downloading_to_separating(self, job_manager, mock_firestore_service):
"""Test transition from DOWNLOADING to SEPARATING_STAGE1."""
existing_job = Job(
job_id="test123",
status=JobStatus.DOWNLOADING,
progress=0,
created_at=datetime.now(UTC),
updated_at=datetime.now(UTC)
)
mock_firestore_service.get_job.return_value = existing_job
job_manager.update_job("test123", {"status": JobStatus.SEPARATING_STAGE1})
mock_firestore_service.update_job.assert_called_once()
class TestJobFailure:
"""Test job failure handling."""
def test_mark_job_as_failed(self, job_manager, mock_firestore_service):
"""Test marking a job as failed."""
error_message = "Audio separation failed"
job_manager.mark_job_failed("test123", error_message)
# mark_job_failed calls firestore.update_job_status
mock_firestore_service.update_job_status.assert_called_once()
call_args = mock_firestore_service.update_job_status.call_args
assert call_args[1]['job_id'] == "test123"
assert call_args[1]['status'] == JobStatus.FAILED
assert 'error_message' in call_args[1]
def test_mark_job_error(self, job_manager, mock_firestore_service):
"""Test marking a job with an error."""
error_message = "Test error"
job_manager.mark_job_error("test123", error_message)
# mark_job_error calls firestore.update_job_status
mock_firestore_service.update_job_status.assert_called_once()
call_args = mock_firestore_service.update_job_status.call_args
assert call_args[1]['job_id'] == "test123"
assert call_args[1]['error_message'] == error_message
class TestJobDeletion:
"""Test job deletion logic."""
def test_delete_job(self, job_manager, mock_firestore_service):
"""Test deleting a job."""
existing_job = Job(
job_id="test123",
status=JobStatus.COMPLETE,
progress=100,
created_at=datetime.now(UTC),
updated_at=datetime.now(UTC),
output_files={} # Empty dict so iteration works
)
mock_firestore_service.get_job.return_value = existing_job
job_manager.delete_job("test123")
# Verify delete was called
mock_firestore_service.delete_job.assert_called_once_with("test123")
def test_delete_job_with_files(self, job_manager, mock_firestore_service):
"""Test deleting a job and its files."""
existing_job = Job(
job_id="test123",
status=JobStatus.COMPLETE,
progress=100,
created_at=datetime.now(UTC),
updated_at=datetime.now(UTC),
output_files={} # Empty dict so iteration works
)
mock_firestore_service.get_job.return_value = existing_job
job_manager.delete_job("test123", delete_files=True)
# Verify delete was called
mock_firestore_service.delete_job.assert_called_once_with("test123")
if __name__ == "__main__":
pytest.main([__file__, "-v"])
-258
View File
@@ -1,258 +0,0 @@
"""
Unit tests for jobs.py API routes using FastAPI TestClient.
These tests mock the underlying services and test the route logic directly.
"""
import pytest
from datetime import datetime, UTC
from unittest.mock import MagicMock, AsyncMock, patch
from fastapi.testclient import TestClient
from backend.models.job import Job, JobStatus
@pytest.fixture
def mock_job():
"""Create a standard mock job for testing."""
return Job(
job_id="test123",
status=JobStatus.PENDING,
created_at=datetime.now(UTC),
updated_at=datetime.now(UTC),
artist="Test Artist",
title="Test Song"
)
@pytest.fixture
def mock_job_manager(mock_job):
"""Create a mock JobManager with common methods."""
manager = MagicMock()
manager.get_job.return_value = mock_job
manager.list_jobs.return_value = [mock_job]
manager.create_job.return_value = mock_job
manager.delete_job.return_value = True
manager.cancel_job.return_value = mock_job
manager.update_job_status.return_value = None
manager.update_state_data.return_value = None
return manager
@pytest.fixture
def mock_worker_service():
"""Create a mock WorkerService."""
service = MagicMock()
service.trigger_audio_worker = AsyncMock(return_value=True)
service.trigger_lyrics_worker = AsyncMock(return_value=True)
service.trigger_screens_worker = AsyncMock(return_value=True)
service.trigger_video_worker = AsyncMock(return_value=True)
return service
@pytest.fixture
def client(mock_job_manager, mock_worker_service):
"""Create TestClient with mocked dependencies."""
mock_creds = MagicMock()
mock_creds.universe_domain = 'googleapis.com'
# Patch at the module level where jobs.py imports them
with patch('backend.api.routes.jobs.job_manager', mock_job_manager), \
patch('backend.api.routes.jobs.worker_service', mock_worker_service), \
patch('backend.services.firestore_service.firestore'), \
patch('backend.services.storage_service.storage'), \
patch('google.auth.default', return_value=(mock_creds, 'test-project')):
from backend.main import app
yield TestClient(app)
class TestGetJob:
"""Tests for GET /api/jobs/{job_id}."""
def test_get_job_returns_200(self, client, mock_job_manager, mock_job, auth_headers):
"""Test getting an existing job returns 200."""
response = client.get("/api/jobs/test123", headers=auth_headers)
assert response.status_code == 200
def test_get_job_returns_job_data(self, client, mock_job_manager, mock_job, auth_headers):
"""Test response contains job data."""
response = client.get("/api/jobs/test123", headers=auth_headers)
data = response.json()
assert data["job_id"] == "test123"
assert data["status"] == "pending"
assert data["artist"] == "Test Artist"
def test_get_nonexistent_job_returns_404(self, mock_worker_service, auth_headers):
"""Test getting non-existent job returns 404."""
mock_job_manager = MagicMock()
mock_job_manager.get_job.return_value = None
mock_creds = MagicMock()
mock_creds.universe_domain = 'googleapis.com'
with patch('backend.api.routes.jobs.job_manager', mock_job_manager), \
patch('backend.api.routes.jobs.worker_service', mock_worker_service), \
patch('backend.services.firestore_service.firestore'), \
patch('backend.services.storage_service.storage'), \
patch('google.auth.default', return_value=(mock_creds, 'test-project')):
from backend.main import app
client = TestClient(app)
response = client.get("/api/jobs/nonexistent", headers=auth_headers)
assert response.status_code == 404
class TestListJobs:
"""Tests for GET /api/jobs."""
def test_list_jobs_returns_200(self, client, auth_headers):
"""Test listing jobs returns 200."""
response = client.get("/api/jobs", headers=auth_headers)
assert response.status_code == 200
def test_list_jobs_returns_array(self, client, mock_job_manager, auth_headers):
"""Test response is an array of jobs."""
response = client.get("/api/jobs", headers=auth_headers)
data = response.json()
assert isinstance(data, list)
assert len(data) == 1
assert data[0]["job_id"] == "test123"
def test_list_jobs_with_status_filter(self, client, mock_job_manager, auth_headers):
"""Test listing jobs with status filter."""
response = client.get("/api/jobs?status=pending", headers=auth_headers)
assert response.status_code == 200
def test_list_jobs_with_limit(self, client, mock_job_manager, auth_headers):
"""Test listing jobs with limit."""
response = client.get("/api/jobs?limit=10", headers=auth_headers)
assert response.status_code == 200
class TestCreateJob:
"""Tests for POST /api/jobs."""
def test_create_job_with_url_returns_200(self, client, mock_job_manager, auth_headers):
"""Test creating job with URL returns 200."""
response = client.post(
"/api/jobs",
json={"url": "https://youtube.com/watch?v=test123"},
headers=auth_headers
)
assert response.status_code == 200
def test_create_job_returns_job_id(self, client, mock_job_manager, auth_headers):
"""Test create response contains job_id."""
response = client.post(
"/api/jobs",
json={"url": "https://youtube.com/watch?v=test"},
headers=auth_headers
)
data = response.json()
assert "job_id" in data
def test_create_job_with_artist_title(self, client, mock_job_manager, auth_headers):
"""Test creating job with artist and title."""
response = client.post(
"/api/jobs",
json={
"url": "https://youtube.com/watch?v=test",
"artist": "Test Artist",
"title": "Test Song"
},
headers=auth_headers
)
assert response.status_code == 200
class TestDeleteJob:
"""Tests for DELETE /api/jobs/{job_id}."""
def test_delete_job_returns_200(self, client, mock_job_manager, auth_headers):
"""Test deleting job returns 200."""
response = client.delete("/api/jobs/test123", headers=auth_headers)
assert response.status_code == 200
def test_delete_nonexistent_job(self, mock_worker_service, auth_headers):
"""Test deleting non-existent job."""
mock_job_manager = MagicMock()
mock_job_manager.delete_job.return_value = False
mock_creds = MagicMock()
mock_creds.universe_domain = 'googleapis.com'
with patch('backend.api.routes.jobs.job_manager', mock_job_manager), \
patch('backend.api.routes.jobs.worker_service', mock_worker_service), \
patch('backend.services.firestore_service.firestore'), \
patch('backend.services.storage_service.storage'), \
patch('google.auth.default', return_value=(mock_creds, 'test-project')):
from backend.main import app
client = TestClient(app)
response = client.delete("/api/jobs/nonexistent", headers=auth_headers)
# Either 200 or 404 depending on implementation
assert response.status_code in [200, 404]
class TestCancelJob:
"""Tests for POST /api/jobs/{job_id}/cancel."""
def test_cancel_job_returns_200(self, client, mock_job_manager, mock_job, auth_headers):
"""Test cancelling job returns 200."""
mock_job_manager.cancel_job.return_value = mock_job
response = client.post(
"/api/jobs/test123/cancel",
json={},
headers=auth_headers
)
assert response.status_code == 200
class TestSubmitCorrections:
"""Tests for POST /api/jobs/{job_id}/corrections."""
def test_submit_corrections_returns_200(self, client, mock_job_manager, mock_job, auth_headers):
"""Test submitting corrections returns 200."""
# Job needs to be in AWAITING_REVIEW or IN_REVIEW status
review_job = Job(
job_id="test123",
status=JobStatus.IN_REVIEW,
created_at=datetime.now(UTC),
updated_at=datetime.now(UTC),
artist="Test",
title="Test"
)
mock_job_manager.get_job.return_value = review_job
response = client.post(
"/api/jobs/test123/corrections",
json={
"corrections": {
"lines": [],
"metadata": {"source": "test"}
}
},
headers=auth_headers
)
# Should be 200 or validation error
assert response.status_code in [200, 400, 422]
class TestSelectInstrumental:
"""Tests for POST /api/jobs/{job_id}/select-instrumental."""
def test_select_instrumental_requires_selection_field(self, client, mock_job_manager, mock_job, auth_headers):
"""Test instrumental selection requires selection field."""
response = client.post(
"/api/jobs/test123/select-instrumental",
json={}, # Missing selection field
headers=auth_headers
)
# Missing field should cause validation error
assert response.status_code == 422
class TestStartReview:
"""Tests for POST /api/jobs/{job_id}/start-review."""
def test_start_review_endpoint_exists(self, client, mock_job_manager, mock_job, auth_headers):
"""Test start review endpoint exists."""
response = client.post("/api/jobs/test123/start-review", headers=auth_headers)
# Should not be 404 or 405
assert response.status_code not in [404, 405]
-87
View File
@@ -1,87 +0,0 @@
"""
Unit tests for backend main.py and config.py.
These tests cover the FastAPI app initialization and configuration.
"""
import pytest
import os
from unittest.mock import MagicMock, patch
class TestConfig:
"""Tests for config.py."""
def test_settings_from_environment(self):
"""Test settings can be loaded from environment variables."""
with patch.dict(os.environ, {
'GCS_BUCKET_NAME': 'test-bucket',
'FIRESTORE_COLLECTION': 'test-jobs',
'ADMIN_TOKENS': 'token1,token2',
'ENVIRONMENT': 'testing'
}):
from backend.config import Settings
settings = Settings()
assert settings.gcs_bucket_name == 'test-bucket'
assert settings.firestore_collection == 'test-jobs'
# admin_tokens is a string, comma-separated
assert 'token1' in settings.admin_tokens
assert 'token2' in settings.admin_tokens
def test_admin_tokens_is_comma_separated_string(self):
"""Test admin tokens stored as comma-separated string."""
with patch.dict(os.environ, {
'ADMIN_TOKENS': 'token1,token2,token3'
}):
from backend.config import Settings
settings = Settings()
# admin_tokens is stored as a string
assert settings.admin_tokens == 'token1,token2,token3'
def test_default_environment(self):
"""Test default environment is development."""
with patch.dict(os.environ, {}, clear=True):
from backend.config import Settings
settings = Settings()
# Accept 'test' as well since pytest may set ENVIRONMENT=test
assert settings.environment in ['development', 'production', 'testing', 'test']
class TestMain:
"""Tests for main.py FastAPI app."""
@pytest.fixture
def app(self):
"""Create FastAPI app with mocked services."""
mock_creds = MagicMock()
mock_creds.universe_domain = 'googleapis.com'
with patch('backend.services.firestore_service.firestore'), \
patch('backend.services.storage_service.storage'), \
patch('google.auth.default', return_value=(mock_creds, 'test-project')):
from backend.main import app
return app
def test_app_has_api_routes(self, app):
"""Test app includes API routes."""
routes = [route.path for route in app.routes]
assert any('/api' in route or '/jobs' in route for route in routes)
def test_app_has_cors_middleware(self, app):
"""Test app has CORS middleware configured."""
# Check that middleware is configured (not necessarily CORS specific)
assert len(app.user_middleware) >= 0 # App initializes middleware
def test_app_title(self, app):
"""Test app has expected title."""
assert 'karaoke' in app.title.lower() or app.title
class TestDependencies:
"""Tests for dependencies.py."""
def test_dependencies_module_imports(self):
"""Test dependencies module can be imported and has auth functions."""
from backend.api import dependencies
assert hasattr(dependencies, 'require_auth')
assert hasattr(dependencies, 'require_admin')
assert hasattr(dependencies, 'optional_auth')
-918
View File
@@ -1,918 +0,0 @@
"""
Unit tests for Pydantic models.
Tests validate that our data models work correctly, including:
- Basic field validation
- Nested dictionary structures (file_urls, state_data)
- Serialization/deserialization round-trips (simulating Firestore)
- Real-world data structures from workers
These tests catch type mismatches that would cause runtime errors when
reading data back from Firestore.
"""
import pytest
import json
from datetime import datetime, UTC
from pydantic import ValidationError
from backend.models.job import (
Job, JobCreate, JobStatus, TimelineEvent
)
class TestJobModel:
"""Test Job Pydantic model - the bug we just fixed!"""
def test_input_media_gcs_path_field_exists(self):
"""
Test that Job model has input_media_gcs_path field.
This test would have caught the bug where we tried to use
job.input_media_gcs_path but it wasn't defined in the model!
"""
job = Job(
job_id="test123",
status=JobStatus.PENDING,
created_at=datetime.now(UTC),
updated_at=datetime.now(UTC),
input_media_gcs_path="uploads/test123/file.flac"
)
# This should NOT raise AttributeError
assert hasattr(job, 'input_media_gcs_path')
assert job.input_media_gcs_path == "uploads/test123/file.flac"
def test_input_media_gcs_path_optional(self):
"""Test that input_media_gcs_path is optional."""
job = Job(
job_id="test123",
status=JobStatus.PENDING,
created_at=datetime.now(UTC),
updated_at=datetime.now(UTC)
)
assert job.input_media_gcs_path is None
def test_pydantic_includes_input_media_gcs_path_in_dict(self):
"""
Test that Pydantic includes input_media_gcs_path in serialization.
This ensures Pydantic doesn't silently ignore the field.
"""
job = Job(
job_id="test123",
status=JobStatus.PENDING,
created_at=datetime.now(UTC),
updated_at=datetime.now(UTC),
input_media_gcs_path="uploads/test123/file.flac"
)
job_dict = job.model_dump()
# Pydantic should include it
assert "input_media_gcs_path" in job_dict
assert job_dict["input_media_gcs_path"] == "uploads/test123/file.flac"
def test_create_minimal_job(self):
"""Test creating a job with minimal required fields."""
job = Job(
job_id="test123",
status=JobStatus.PENDING,
created_at=datetime.now(UTC),
updated_at=datetime.now(UTC)
)
assert job.job_id == "test123"
assert job.status == JobStatus.PENDING
assert job.progress == 0
assert job.url is None
assert job.input_media_gcs_path is None
def test_create_job_with_url(self):
"""Test creating a job with YouTube URL."""
job = Job(
job_id="test123",
status=JobStatus.PENDING,
created_at=datetime.now(UTC),
updated_at=datetime.now(UTC),
url="https://youtube.com/watch?v=test",
artist="Test Artist",
title="Test Song"
)
assert job.url == "https://youtube.com/watch?v=test"
assert job.artist == "Test Artist"
assert job.title == "Test Song"
def test_create_job_with_uploaded_file(self):
"""Test creating a job with uploaded file."""
job = Job(
job_id="test123",
status=JobStatus.PENDING,
created_at=datetime.now(UTC),
updated_at=datetime.now(UTC),
input_media_gcs_path="uploads/test123/file.flac",
artist="Test Artist",
title="Test Song"
)
assert job.input_media_gcs_path == "uploads/test123/file.flac"
assert job.url is None
class TestJobCreate:
"""Test JobCreate validation model."""
def test_create_with_url(self):
"""Test creating job with YouTube URL."""
job_create = JobCreate(
url="https://youtube.com/watch?v=test",
artist="Test Artist",
title="Test Song"
)
assert job_create.url == "https://youtube.com/watch?v=test"
assert job_create.artist == "Test Artist"
assert job_create.title == "Test Song"
def test_create_minimal(self):
"""Test creating job with minimal fields."""
job_create = JobCreate()
assert job_create.url is None
assert job_create.artist is None
assert job_create.title is None
class TestJobStatus:
"""Test JobStatus enum."""
def test_critical_statuses_defined(self):
"""Test that critical statuses exist."""
critical_statuses = [
"pending", "downloading",
"separating_stage1", "separating_stage2", "audio_complete",
"transcribing", "correcting", "lyrics_complete",
"generating_screens", "applying_padding",
"awaiting_review", "in_review", "review_complete",
"awaiting_instrumental_selection", "instrumental_selected",
"generating_video", "encoding", "packaging",
"complete", "failed"
]
actual_statuses = [status.value for status in JobStatus]
for status in critical_statuses:
assert status in actual_statuses, f"Missing critical status: {status}"
class TestTimelineEvent:
"""Test TimelineEvent model."""
def test_create_timeline_event(self):
"""Test creating a timeline event."""
event = TimelineEvent(
status="pending",
timestamp="2025-12-01T08:00:00Z",
progress=0,
message="Job created"
)
assert event.status == "pending"
assert event.timestamp == "2025-12-01T08:00:00Z"
assert event.progress == 0
assert event.message == "Job created"
def test_timeline_event_optional_fields(self):
"""Test that progress and message are optional."""
event = TimelineEvent(
status="pending",
timestamp="2025-12-01T08:00:00Z"
)
assert event.progress is None
assert event.message is None
class TestModelValidation:
"""Test model validation rules."""
def test_invalid_job_status(self):
"""Test that invalid status is rejected."""
with pytest.raises((ValidationError, ValueError)):
Job(
job_id="test123",
status="invalid_status", # Not a valid JobStatus
created_at=datetime.now(UTC),
updated_at=datetime.now(UTC)
)
def test_missing_required_fields(self):
"""Test that missing required fields are rejected."""
with pytest.raises(ValidationError):
Job(
job_id="test123"
# Missing required fields: status, created_at, updated_at
)
class TestFileUrlsNestedStructure:
"""
Test that file_urls field correctly handles nested dictionaries.
THIS IS THE TEST THAT WOULD HAVE CAUGHT THE Dict[str, str] vs Dict[str, Any] BUG!
Workers store nested structures like:
{
"stems": {"instrumental_clean": "gs://...", "vocals": "gs://..."},
"lyrics": {"corrections": "gs://...", "lrc": "gs://..."},
...
}
If file_urls is typed as Dict[str, str], this will fail on deserialization.
"""
def test_file_urls_with_nested_stems(self):
"""Test that file_urls accepts nested stems dictionary."""
job = Job(
job_id="test123",
status=JobStatus.AUDIO_COMPLETE,
created_at=datetime.now(UTC),
updated_at=datetime.now(UTC),
file_urls={
"stems": {
"instrumental_clean": "gs://bucket/jobs/test123/stems/instrumental_clean.flac",
"instrumental_with_backing": "gs://bucket/jobs/test123/stems/instrumental_with_backing.flac",
"vocals_clean": "gs://bucket/jobs/test123/stems/vocals_clean.flac",
"lead_vocals": "gs://bucket/jobs/test123/stems/lead_vocals.flac",
"backing_vocals": "gs://bucket/jobs/test123/stems/backing_vocals.flac",
"bass": "gs://bucket/jobs/test123/stems/bass.flac",
"drums": "gs://bucket/jobs/test123/stems/drums.flac",
}
}
)
assert job.file_urls["stems"]["instrumental_clean"] == "gs://bucket/jobs/test123/stems/instrumental_clean.flac"
assert len(job.file_urls["stems"]) == 7
def test_file_urls_with_nested_lyrics(self):
"""Test that file_urls accepts nested lyrics dictionary."""
job = Job(
job_id="test123",
status=JobStatus.LYRICS_COMPLETE,
created_at=datetime.now(UTC),
updated_at=datetime.now(UTC),
file_urls={
"lyrics": {
"corrections": "gs://bucket/jobs/test123/lyrics/corrections.json",
"audio": "gs://bucket/jobs/test123/lyrics/audio.flac",
"lrc": "gs://bucket/jobs/test123/lyrics/karaoke.lrc",
"uncorrected": "gs://bucket/jobs/test123/lyrics/uncorrected.txt",
}
}
)
assert job.file_urls["lyrics"]["corrections"] == "gs://bucket/jobs/test123/lyrics/corrections.json"
def test_file_urls_full_worker_structure(self):
"""
Test with the full structure that workers actually create.
This simulates what actually gets stored in Firestore after
all workers have run.
"""
full_file_urls = {
"input": {
"audio": "gs://bucket/jobs/test123/input/waterloo.flac"
},
"stems": {
"instrumental_clean": "gs://bucket/jobs/test123/stems/instrumental_clean.flac",
"instrumental_with_backing": "gs://bucket/jobs/test123/stems/instrumental_with_backing.flac",
"vocals_clean": "gs://bucket/jobs/test123/stems/vocals_clean.flac",
"lead_vocals": "gs://bucket/jobs/test123/stems/lead_vocals.flac",
"backing_vocals": "gs://bucket/jobs/test123/stems/backing_vocals.flac",
},
"lyrics": {
"corrections": "gs://bucket/jobs/test123/lyrics/corrections.json",
"lrc": "gs://bucket/jobs/test123/lyrics/karaoke.lrc",
},
"screens": {
"title": "gs://bucket/jobs/test123/screens/title.mov",
"end": "gs://bucket/jobs/test123/screens/end.mov",
},
"videos": {
"with_vocals": "gs://bucket/jobs/test123/videos/with_vocals.mkv",
},
"finals": {
"lossless_4k_mp4": "gs://bucket/jobs/test123/finals/lossless_4k.mp4",
"lossy_720p_mp4": "gs://bucket/jobs/test123/finals/lossy_720p.mp4",
}
}
job = Job(
job_id="test123",
status=JobStatus.COMPLETE,
created_at=datetime.now(UTC),
updated_at=datetime.now(UTC),
file_urls=full_file_urls
)
# Verify nested access works
assert job.file_urls["stems"]["instrumental_clean"].endswith("instrumental_clean.flac")
assert job.file_urls["lyrics"]["corrections"].endswith("corrections.json")
assert job.file_urls["finals"]["lossy_720p_mp4"].endswith("lossy_720p.mp4")
def test_file_urls_roundtrip_serialization(self):
"""
Test that file_urls survives serialization/deserialization.
This simulates what happens when we:
1. Create a Job with nested file_urls
2. Serialize to dict (for Firestore)
3. Deserialize back to Job (reading from Firestore)
THIS IS THE KEY TEST - it would catch the Dict[str, str] bug!
"""
original_file_urls = {
"stems": {
"instrumental_clean": "gs://bucket/stems/clean.flac",
"vocals": "gs://bucket/stems/vocals.flac",
},
"lyrics": {
"corrections": "gs://bucket/lyrics/corrections.json",
}
}
# Create job
job = Job(
job_id="test123",
status=JobStatus.PENDING,
created_at=datetime.now(UTC),
updated_at=datetime.now(UTC),
file_urls=original_file_urls
)
# Serialize to dict (simulates writing to Firestore)
job_dict = job.model_dump()
# Convert to JSON and back (simulates Firestore storage)
json_str = json.dumps(job_dict, default=str)
restored_dict = json.loads(json_str)
# Deserialize back to Job (simulates reading from Firestore)
restored_job = Job(**restored_dict)
# Verify nested structure survived
assert restored_job.file_urls["stems"]["instrumental_clean"] == "gs://bucket/stems/clean.flac"
assert restored_job.file_urls["stems"]["vocals"] == "gs://bucket/stems/vocals.flac"
assert restored_job.file_urls["lyrics"]["corrections"] == "gs://bucket/lyrics/corrections.json"
class TestStateDataNestedStructure:
"""Test that state_data field correctly handles nested dictionaries."""
def test_state_data_with_instrumental_options(self):
"""Test state_data with instrumental options structure."""
job = Job(
job_id="test123",
status=JobStatus.AWAITING_INSTRUMENTAL_SELECTION,
created_at=datetime.now(UTC),
updated_at=datetime.now(UTC),
state_data={
"instrumental_options": {
"clean": "gs://bucket/jobs/test123/stems/instrumental_clean.flac",
"with_backing": "gs://bucket/jobs/test123/stems/instrumental_with_backing.flac",
},
"audio_complete": True,
"lyrics_complete": True,
}
)
assert job.state_data["instrumental_options"]["clean"].endswith("clean.flac")
assert job.state_data["audio_complete"] is True
def test_state_data_with_lyrics_metadata(self):
"""Test state_data with lyrics metadata structure."""
job = Job(
job_id="test123",
status=JobStatus.LYRICS_COMPLETE,
created_at=datetime.now(UTC),
updated_at=datetime.now(UTC),
state_data={
"lyrics_metadata": {
"line_count": 42,
"has_corrections": True,
"ready_for_review": True,
},
"corrected_lyrics": {
"lines": [
{"text": "Hello world", "start_time": 1.0, "end_time": 2.0}
],
"metadata": {"song": "Test Song"}
}
}
)
assert job.state_data["lyrics_metadata"]["line_count"] == 42
assert len(job.state_data["corrected_lyrics"]["lines"]) == 1
def test_state_data_roundtrip_serialization(self):
"""Test that state_data survives serialization/deserialization."""
original_state_data = {
"instrumental_selection": "clean",
"review_notes": "Looks good!",
"complex_nested": {
"level1": {
"level2": {
"value": 123
}
}
}
}
job = Job(
job_id="test123",
status=JobStatus.PENDING,
created_at=datetime.now(UTC),
updated_at=datetime.now(UTC),
state_data=original_state_data
)
# Serialize and deserialize
job_dict = job.model_dump()
json_str = json.dumps(job_dict, default=str)
restored_dict = json.loads(json_str)
restored_job = Job(**restored_dict)
# Verify nested structure survived
assert restored_job.state_data["complex_nested"]["level1"]["level2"]["value"] == 123
class TestWorkerIdsNestedStructure:
"""Test that worker_ids field correctly handles nested dictionaries."""
def test_worker_ids_with_all_workers(self):
"""Test worker_ids with all worker types."""
job = Job(
job_id="test123",
status=JobStatus.GENERATING_VIDEO,
created_at=datetime.now(UTC),
updated_at=datetime.now(UTC),
worker_ids={
"audio_worker": "cloud-run-request-12345",
"lyrics_worker": "cloud-run-request-67890",
"screens_worker": "cloud-run-request-11111",
"video_worker": "cloud-run-request-22222",
}
)
assert job.worker_ids["audio_worker"] == "cloud-run-request-12345"
assert len(job.worker_ids) == 4
class TestJobModelIntegration:
"""
Integration tests that simulate real Firestore interactions.
These tests verify that the Job model works correctly when used
the same way as our actual services.
"""
def test_simulate_job_manager_update_file_url(self):
"""
Simulate what JobManager.update_file_url does.
This is the pattern that caused the bug - updating nested file_urls.
"""
# Start with a fresh job
job = Job(
job_id="test123",
status=JobStatus.PENDING,
created_at=datetime.now(UTC),
updated_at=datetime.now(UTC),
file_urls={}
)
# Simulate update_file_url('test123', 'stems', 'instrumental_clean', 'gs://...')
# This creates nested structure: {"stems": {"instrumental_clean": "gs://..."}}
if "stems" not in job.file_urls:
job.file_urls["stems"] = {}
job.file_urls["stems"]["instrumental_clean"] = "gs://bucket/stems/clean.flac"
# Simulate Firestore roundtrip
job_dict = job.model_dump()
restored_job = Job(**job_dict)
# This is what was failing with Dict[str, str]!
assert restored_job.file_urls["stems"]["instrumental_clean"] == "gs://bucket/stems/clean.flac"
def test_simulate_audio_worker_completion(self):
"""Simulate the data structure created when audio worker completes."""
# This is the actual structure created by audio_worker.py
file_urls = {
"stems": {
"instrumental_clean": "gs://bucket/jobs/abc123/stems/instrumental_clean.flac",
"vocals_clean": "gs://bucket/jobs/abc123/stems/vocals_clean.flac",
"lead_vocals": "gs://bucket/jobs/abc123/stems/lead_vocals.flac",
"backing_vocals": "gs://bucket/jobs/abc123/stems/backing_vocals.flac",
"instrumental_with_backing": "gs://bucket/jobs/abc123/stems/instrumental_with_backing.flac",
}
}
state_data = {
"instrumental_options": {
"clean": "jobs/abc123/stems/instrumental_clean.flac",
"with_backing": "jobs/abc123/stems/instrumental_with_backing.flac",
}
}
job = Job(
job_id="abc123",
status=JobStatus.AUDIO_COMPLETE,
created_at=datetime.now(UTC),
updated_at=datetime.now(UTC),
artist="ABBA",
title="Waterloo",
file_urls=file_urls,
state_data=state_data
)
# Roundtrip
restored_job = Job(**job.model_dump())
assert len(restored_job.file_urls["stems"]) == 5
assert "instrumental_options" in restored_job.state_data
def test_simulate_lyrics_worker_completion(self):
"""Simulate the data structure created when lyrics worker completes."""
file_urls = {
"lyrics": {
"corrections": "gs://bucket/jobs/abc123/lyrics/corrections.json",
"lrc": "gs://bucket/jobs/abc123/lyrics/karaoke.lrc",
"uncorrected": "gs://bucket/jobs/abc123/lyrics/uncorrected.txt",
}
}
state_data = {
"lyrics_metadata": {
"line_count": 50,
"has_corrections": True,
"ready_for_review": True,
}
}
job = Job(
job_id="abc123",
status=JobStatus.LYRICS_COMPLETE,
created_at=datetime.now(UTC),
updated_at=datetime.now(UTC),
file_urls=file_urls,
state_data=state_data
)
# Roundtrip
restored_job = Job(**job.model_dump())
assert restored_job.file_urls["lyrics"]["corrections"].endswith("corrections.json")
assert restored_job.state_data["lyrics_metadata"]["line_count"] == 50
class TestExistingInstrumentalModelFields:
"""Test existing instrumental field in Job and JobCreate models (Batch 3)."""
def test_job_has_existing_instrumental_gcs_path(self):
"""Test that Job model has existing_instrumental_gcs_path field."""
job = Job(
job_id="test123",
status=JobStatus.PENDING,
created_at=datetime.now(UTC),
updated_at=datetime.now(UTC),
existing_instrumental_gcs_path="uploads/test123/audio/existing_instrumental.flac"
)
assert hasattr(job, 'existing_instrumental_gcs_path')
assert job.existing_instrumental_gcs_path == "uploads/test123/audio/existing_instrumental.flac"
def test_job_existing_instrumental_defaults_to_none(self):
"""Test that existing_instrumental_gcs_path defaults to None."""
job = Job(
job_id="test123",
status=JobStatus.PENDING,
created_at=datetime.now(UTC),
updated_at=datetime.now(UTC)
)
assert job.existing_instrumental_gcs_path is None
def test_job_create_has_existing_instrumental_gcs_path(self):
"""Test that JobCreate model has existing_instrumental_gcs_path field."""
from backend.models.job import JobCreate
job_create = JobCreate(
artist="Artist",
title="Title",
existing_instrumental_gcs_path="uploads/test/audio/existing_instrumental.mp3"
)
assert job_create.existing_instrumental_gcs_path == "uploads/test/audio/existing_instrumental.mp3"
def test_job_create_existing_instrumental_optional(self):
"""Test that existing_instrumental_gcs_path is optional in JobCreate."""
from backend.models.job import JobCreate
job_create = JobCreate(
artist="Artist",
title="Title"
)
assert job_create.existing_instrumental_gcs_path is None
def test_existing_instrumental_roundtrip_serialization(self):
"""Test that existing_instrumental_gcs_path survives serialization/deserialization."""
job = Job(
job_id="test123",
status=JobStatus.PENDING,
created_at=datetime.now(UTC),
updated_at=datetime.now(UTC),
existing_instrumental_gcs_path="uploads/test123/audio/existing_instrumental.wav"
)
# Serialize to dict (simulates writing to Firestore)
job_dict = job.model_dump()
# Convert to JSON and back (simulates Firestore storage)
json_str = json.dumps(job_dict, default=str)
restored_dict = json.loads(json_str)
# Deserialize back to Job (simulates reading from Firestore)
restored_job = Job(**restored_dict)
assert restored_job.existing_instrumental_gcs_path == "uploads/test123/audio/existing_instrumental.wav"
def test_job_with_full_audio_config(self):
"""Test Job with complete audio configuration including existing instrumental."""
job = Job(
job_id="test123",
status=JobStatus.AUDIO_COMPLETE,
created_at=datetime.now(UTC),
updated_at=datetime.now(UTC),
artist="Test Artist",
title="Test Song",
# Audio model configuration
clean_instrumental_model="model_bs_roformer_ep_317_sdr_12.9755.ckpt",
backing_vocals_models=["mel_band_roformer_karaoke.ckpt"],
other_stems_models=["htdemucs_6s.yaml"],
# Existing instrumental
existing_instrumental_gcs_path="uploads/test123/audio/my_instrumental.flac",
)
assert job.clean_instrumental_model == "model_bs_roformer_ep_317_sdr_12.9755.ckpt"
assert job.existing_instrumental_gcs_path == "uploads/test123/audio/my_instrumental.flac"
class TestTwoPhaseWorkflowFields:
"""Test two-phase workflow fields in Job and JobCreate models (Batch 6)."""
def test_job_has_prep_only_field(self):
"""Test that Job model has prep_only field."""
job = Job(
job_id="test123",
status=JobStatus.PENDING,
created_at=datetime.now(UTC),
updated_at=datetime.now(UTC),
prep_only=True
)
assert hasattr(job, 'prep_only')
assert job.prep_only is True
def test_job_prep_only_defaults_to_false(self):
"""Test that prep_only defaults to False."""
job = Job(
job_id="test123",
status=JobStatus.PENDING,
created_at=datetime.now(UTC),
updated_at=datetime.now(UTC)
)
assert job.prep_only is False
def test_job_has_finalise_only_field(self):
"""Test that Job model has finalise_only field."""
job = Job(
job_id="test123",
status=JobStatus.PENDING,
created_at=datetime.now(UTC),
updated_at=datetime.now(UTC),
finalise_only=True
)
assert hasattr(job, 'finalise_only')
assert job.finalise_only is True
def test_job_finalise_only_defaults_to_false(self):
"""Test that finalise_only defaults to False."""
job = Job(
job_id="test123",
status=JobStatus.PENDING,
created_at=datetime.now(UTC),
updated_at=datetime.now(UTC)
)
assert job.finalise_only is False
def test_job_has_keep_brand_code_field(self):
"""Test that Job model has keep_brand_code field."""
job = Job(
job_id="test123",
status=JobStatus.PENDING,
created_at=datetime.now(UTC),
updated_at=datetime.now(UTC),
keep_brand_code="NOMAD-1234"
)
assert hasattr(job, 'keep_brand_code')
assert job.keep_brand_code == "NOMAD-1234"
def test_job_keep_brand_code_defaults_to_none(self):
"""Test that keep_brand_code defaults to None."""
job = Job(
job_id="test123",
status=JobStatus.PENDING,
created_at=datetime.now(UTC),
updated_at=datetime.now(UTC)
)
assert job.keep_brand_code is None
def test_job_create_has_two_phase_workflow_fields(self):
"""Test that JobCreate model has two-phase workflow fields."""
job_create = JobCreate(
artist="Artist",
title="Title",
prep_only=True,
finalise_only=False,
keep_brand_code="BRAND-0001"
)
assert job_create.prep_only is True
assert job_create.finalise_only is False
assert job_create.keep_brand_code == "BRAND-0001"
def test_two_phase_workflow_roundtrip_serialization(self):
"""Test that two-phase workflow fields survive serialization/deserialization."""
job = Job(
job_id="test123",
status=JobStatus.PENDING,
created_at=datetime.now(UTC),
updated_at=datetime.now(UTC),
prep_only=True,
finalise_only=False,
keep_brand_code="NOMAD-5678"
)
# Serialize to dict (simulates writing to Firestore)
job_dict = job.model_dump()
# Convert to JSON and back (simulates Firestore storage)
json_str = json.dumps(job_dict, default=str)
restored_dict = json.loads(json_str)
# Deserialize back to Job (simulates reading from Firestore)
restored_job = Job(**restored_dict)
assert restored_job.prep_only is True
assert restored_job.finalise_only is False
assert restored_job.keep_brand_code == "NOMAD-5678"
class TestPrepCompleteStatus:
"""Test PREP_COMPLETE status and state transitions (Batch 6)."""
def test_prep_complete_status_exists(self):
"""Test that PREP_COMPLETE status exists in JobStatus enum."""
assert hasattr(JobStatus, 'PREP_COMPLETE')
assert JobStatus.PREP_COMPLETE.value == "prep_complete"
def test_job_can_have_prep_complete_status(self):
"""Test that Job can be created with PREP_COMPLETE status."""
job = Job(
job_id="test123",
status=JobStatus.PREP_COMPLETE,
created_at=datetime.now(UTC),
updated_at=datetime.now(UTC)
)
assert job.status == JobStatus.PREP_COMPLETE
def test_state_transitions_review_complete_to_prep_complete(self):
"""Test that REVIEW_COMPLETE can transition to PREP_COMPLETE."""
from backend.models.job import STATE_TRANSITIONS
valid_transitions = STATE_TRANSITIONS.get(JobStatus.REVIEW_COMPLETE, [])
assert JobStatus.PREP_COMPLETE in valid_transitions
def test_state_transitions_prep_complete_to_awaiting_instrumental(self):
"""Test that PREP_COMPLETE can transition to AWAITING_INSTRUMENTAL_SELECTION."""
from backend.models.job import STATE_TRANSITIONS
valid_transitions = STATE_TRANSITIONS.get(JobStatus.PREP_COMPLETE, [])
assert JobStatus.AWAITING_INSTRUMENTAL_SELECTION in valid_transitions
def test_state_transitions_pending_to_awaiting_instrumental(self):
"""Test that PENDING can transition to AWAITING_INSTRUMENTAL_SELECTION (for finalise-only)."""
from backend.models.job import STATE_TRANSITIONS
valid_transitions = STATE_TRANSITIONS.get(JobStatus.PENDING, [])
assert JobStatus.AWAITING_INSTRUMENTAL_SELECTION in valid_transitions
def test_critical_statuses_include_prep_complete(self):
"""Test that PREP_COMPLETE is in the list of job statuses."""
all_statuses = [status.value for status in JobStatus]
assert "prep_complete" in all_statuses
class TestJobWithFullTwoPhaseConfig:
"""Integration tests for two-phase workflow job configuration."""
def test_prep_only_job_full_config(self):
"""Test creating a prep-only job with full configuration."""
job = Job(
job_id="prep123",
status=JobStatus.PENDING,
created_at=datetime.now(UTC),
updated_at=datetime.now(UTC),
artist="Test Artist",
title="Test Song",
prep_only=True,
enable_cdg=True,
enable_txt=True,
brand_prefix="NOMAD",
)
assert job.prep_only is True
assert job.finalise_only is False
assert job.brand_prefix == "NOMAD"
def test_finalise_only_job_full_config(self):
"""Test creating a finalise-only job with full configuration."""
job = Job(
job_id="final123",
status=JobStatus.AWAITING_INSTRUMENTAL_SELECTION,
created_at=datetime.now(UTC),
updated_at=datetime.now(UTC),
artist="Test Artist",
title="Test Song",
finalise_only=True,
keep_brand_code="NOMAD-1234",
enable_youtube_upload=True,
dropbox_path="/Karaoke/Tracks",
state_data={
"audio_complete": True,
"lyrics_complete": True,
"finalise_only": True,
}
)
assert job.finalise_only is True
assert job.keep_brand_code == "NOMAD-1234"
assert job.state_data["finalise_only"] is True
def test_prep_complete_job_with_file_urls(self):
"""Test a job that has reached PREP_COMPLETE status with all prep outputs."""
job = Job(
job_id="prepcomp123",
status=JobStatus.PREP_COMPLETE,
created_at=datetime.now(UTC),
updated_at=datetime.now(UTC),
artist="ABBA",
title="Waterloo",
prep_only=True,
file_urls={
"stems": {
"instrumental_clean": "gs://bucket/jobs/prepcomp123/stems/instrumental_clean.flac",
"instrumental_with_backing": "gs://bucket/jobs/prepcomp123/stems/instrumental_with_backing.flac",
},
"videos": {
"with_vocals": "gs://bucket/jobs/prepcomp123/videos/with_vocals.mkv",
},
"screens": {
"title": "gs://bucket/jobs/prepcomp123/screens/title.mov",
"end": "gs://bucket/jobs/prepcomp123/screens/end.mov",
},
"lyrics": {
"lrc": "gs://bucket/jobs/prepcomp123/lyrics/karaoke.lrc",
"ass": "gs://bucket/jobs/prepcomp123/lyrics/karaoke.ass",
},
}
)
assert job.status == JobStatus.PREP_COMPLETE
assert job.prep_only is True
assert "with_vocals" in job.file_urls["videos"]
assert "title" in job.file_urls["screens"]
if __name__ == "__main__":
pytest.main([__file__, "-v"])
-200
View File
@@ -1,200 +0,0 @@
"""
Unit tests for request models.
These tests validate Pydantic request model validation and serialization.
"""
import pytest
from pydantic import ValidationError
from backend.models.requests import (
URLSubmissionRequest,
UploadSubmissionRequest,
CorrectionsSubmission,
InstrumentalSelection,
StartReviewRequest,
CancelJobRequest,
RetryJobRequest
)
class TestURLSubmissionRequest:
"""Tests for URLSubmissionRequest model."""
def test_valid_youtube_url(self):
"""Test valid YouTube URL is accepted."""
request = URLSubmissionRequest(url="https://youtube.com/watch?v=test123")
assert str(request.url) == "https://youtube.com/watch?v=test123"
def test_valid_youtube_short_url(self):
"""Test valid YouTube short URL is accepted."""
request = URLSubmissionRequest(url="https://youtu.be/test123")
assert str(request.url) == "https://youtu.be/test123"
def test_optional_artist_title(self):
"""Test artist and title are optional."""
request = URLSubmissionRequest(url="https://youtube.com/watch?v=test")
assert request.artist is None
assert request.title is None
def test_with_artist_title(self):
"""Test request with artist and title."""
request = URLSubmissionRequest(
url="https://youtube.com/watch?v=test",
artist="Test Artist",
title="Test Song"
)
assert request.artist == "Test Artist"
assert request.title == "Test Song"
def test_url_required(self):
"""Test URL is required."""
with pytest.raises(ValidationError):
URLSubmissionRequest()
def test_empty_url_rejected(self):
"""Test empty URL is rejected."""
with pytest.raises(ValidationError):
URLSubmissionRequest(url="")
class TestCorrectionsSubmission:
"""Tests for CorrectionsSubmission model."""
def test_valid_corrections(self):
"""Test valid corrections submission."""
corrections = CorrectionsSubmission(
corrections={
"lines": [
{"text": "Hello world", "start": 0.0, "end": 1.0}
],
"metadata": {"source": "test"}
}
)
assert "lines" in corrections.corrections
assert "metadata" in corrections.corrections
def test_missing_lines_rejected(self):
"""Test corrections without 'lines' field is rejected."""
with pytest.raises(ValidationError):
CorrectionsSubmission(corrections={"metadata": {}})
def test_missing_metadata_rejected(self):
"""Test corrections without 'metadata' field is rejected."""
with pytest.raises(ValidationError):
CorrectionsSubmission(corrections={"lines": []})
def test_corrections_required(self):
"""Test corrections field is required."""
with pytest.raises(ValidationError):
CorrectionsSubmission()
def test_with_user_notes(self):
"""Test corrections with user notes."""
corrections = CorrectionsSubmission(
corrections={"lines": [], "metadata": {}},
user_notes="Fixed typo in line 3"
)
assert corrections.user_notes == "Fixed typo in line 3"
class TestInstrumentalSelection:
"""Tests for InstrumentalSelection model."""
def test_valid_clean_selection(self):
"""Test selecting clean instrumental."""
selection = InstrumentalSelection(selection="clean")
assert selection.selection == "clean"
def test_valid_with_backing_selection(self):
"""Test selecting instrumental with backing vocals."""
selection = InstrumentalSelection(selection="with_backing")
assert selection.selection == "with_backing"
def test_selection_required(self):
"""Test selection is required."""
with pytest.raises(ValidationError):
InstrumentalSelection()
def test_invalid_selection_rejected(self):
"""Test invalid selection values are rejected."""
with pytest.raises(ValidationError):
InstrumentalSelection(selection="invalid_option")
class TestUploadSubmissionRequest:
"""Tests for UploadSubmissionRequest model."""
def test_valid_upload_request(self):
"""Test valid upload submission."""
request = UploadSubmissionRequest(
artist="Test Artist",
title="Test Song"
)
assert request.artist == "Test Artist"
assert request.title == "Test Song"
def test_artist_required(self):
"""Test artist is required."""
with pytest.raises(ValidationError):
UploadSubmissionRequest(title="Test Song")
def test_title_required(self):
"""Test title is required."""
with pytest.raises(ValidationError):
UploadSubmissionRequest(artist="Test Artist")
def test_default_options(self):
"""Test default option values."""
request = UploadSubmissionRequest(artist="Test", title="Test")
# CDG/TXT disabled by default (requires style config)
assert request.enable_cdg is False
assert request.enable_txt is False
assert request.enable_youtube_upload is False
class TestStartReviewRequest:
"""Tests for StartReviewRequest model."""
def test_valid_request(self):
"""Test valid start review request."""
request = StartReviewRequest()
assert request is not None
class TestCancelJobRequest:
"""Tests for CancelJobRequest model."""
def test_valid_request(self):
"""Test valid cancel request."""
request = CancelJobRequest()
assert request is not None
def test_with_reason(self):
"""Test cancel with reason."""
request = CancelJobRequest(reason="User requested")
assert request.reason == "User requested"
def test_reason_optional(self):
"""Test reason is optional."""
request = CancelJobRequest()
assert request.reason is None
class TestRetryJobRequest:
"""Tests for RetryJobRequest model."""
def test_valid_request(self):
"""Test valid retry request."""
request = RetryJobRequest()
assert request is not None
def test_with_from_stage(self):
"""Test retry from specific stage."""
request = RetryJobRequest(from_stage="transcription")
assert request.from_stage == "transcription"
def test_from_stage_optional(self):
"""Test from_stage is optional."""
request = RetryJobRequest()
assert request.from_stage is None
-282
View File
@@ -1,282 +0,0 @@
"""
Unit tests for jobs.py routes.
These tests exercise the route logic with mocked services.
"""
import pytest
from datetime import datetime, UTC
from unittest.mock import MagicMock, AsyncMock, patch
from backend.models.job import Job, JobStatus
class TestJobsRouteHelpers:
"""Tests for helper functions in jobs.py routes."""
def test_jobs_router_structure(self):
"""Test jobs router has expected structure."""
from backend.api.routes.jobs import router
assert router is not None
# Check that common route patterns exist
route_paths = [route.path for route in router.routes]
assert any('/jobs' in p or 'jobs' in str(p) for p in route_paths)
class TestJobStatusTransitions:
"""Tests for job status transition validation.
These test the Job model's state machine which is critical for
preventing invalid operations.
"""
def test_valid_pending_to_downloading(self):
"""Test PENDING -> DOWNLOADING is valid."""
from backend.models.job import STATE_TRANSITIONS
valid_transitions = STATE_TRANSITIONS.get(JobStatus.PENDING, [])
assert JobStatus.DOWNLOADING in valid_transitions
def test_valid_downloading_to_separating(self):
"""Test DOWNLOADING -> SEPARATING_STAGE1 is valid."""
from backend.models.job import STATE_TRANSITIONS
valid_transitions = STATE_TRANSITIONS.get(JobStatus.DOWNLOADING, [])
assert JobStatus.SEPARATING_STAGE1 in valid_transitions
def test_invalid_pending_to_complete(self):
"""Test PENDING -> COMPLETE is invalid."""
from backend.models.job import STATE_TRANSITIONS
valid_transitions = STATE_TRANSITIONS.get(JobStatus.PENDING, [])
assert JobStatus.COMPLETE not in valid_transitions
def test_any_active_status_can_fail(self):
"""Test any active (non-terminal, non-legacy) status can transition to FAILED."""
from backend.models.job import STATE_TRANSITIONS
# Terminal states and legacy states are excluded
excluded = [
JobStatus.COMPLETE, JobStatus.FAILED, JobStatus.CANCELLED,
# Legacy states that don't include FAILED
JobStatus.QUEUED, JobStatus.PROCESSING,
JobStatus.READY_FOR_FINALIZATION, JobStatus.FINALIZING, JobStatus.ERROR
]
for status in JobStatus:
if status not in excluded:
valid_transitions = STATE_TRANSITIONS.get(status, [])
assert JobStatus.FAILED in valid_transitions, f"{status} should be able to fail"
def test_failed_can_transition_for_retry(self):
"""Test FAILED status can transition to retry checkpoint states."""
from backend.models.job import STATE_TRANSITIONS
valid_transitions = STATE_TRANSITIONS.get(JobStatus.FAILED, [])
# FAILED should allow retry transitions
assert JobStatus.INSTRUMENTAL_SELECTED in valid_transitions, "FAILED should allow retry to INSTRUMENTAL_SELECTED"
assert JobStatus.REVIEW_COMPLETE in valid_transitions, "FAILED should allow retry to REVIEW_COMPLETE"
assert JobStatus.LYRICS_COMPLETE in valid_transitions, "FAILED should allow retry to LYRICS_COMPLETE"
def test_cancelled_can_transition_for_retry(self):
"""Test CANCELLED status can transition to retry checkpoint states."""
from backend.models.job import STATE_TRANSITIONS
valid_transitions = STATE_TRANSITIONS.get(JobStatus.CANCELLED, [])
# CANCELLED should allow same retry transitions as FAILED
assert JobStatus.INSTRUMENTAL_SELECTED in valid_transitions, "CANCELLED should allow retry to INSTRUMENTAL_SELECTED"
assert JobStatus.REVIEW_COMPLETE in valid_transitions, "CANCELLED should allow retry to REVIEW_COMPLETE"
assert JobStatus.LYRICS_COMPLETE in valid_transitions, "CANCELLED should allow retry to LYRICS_COMPLETE"
assert JobStatus.DOWNLOADING in valid_transitions, "CANCELLED should allow retry to DOWNLOADING (restart)"
def test_failed_can_restart_from_beginning(self):
"""Test FAILED status can transition to DOWNLOADING for restart from beginning."""
from backend.models.job import STATE_TRANSITIONS
valid_transitions = STATE_TRANSITIONS.get(JobStatus.FAILED, [])
# FAILED should allow restart from beginning
assert JobStatus.DOWNLOADING in valid_transitions, "FAILED should allow restart to DOWNLOADING"
class TestJobModelSerialization:
"""Tests for Job model serialization to/from API."""
def test_job_to_dict(self):
"""Test Job converts to dict correctly."""
job = Job(
job_id="test123",
status=JobStatus.PENDING,
created_at=datetime.now(UTC),
updated_at=datetime.now(UTC),
artist="Test Artist",
title="Test Song"
)
data = job.model_dump()
assert data["job_id"] == "test123"
assert data["status"] == "pending"
assert data["artist"] == "Test Artist"
def test_job_from_dict(self):
"""Test Job can be created from dict."""
data = {
"job_id": "test123",
"status": "pending",
"created_at": datetime.now(UTC).isoformat(),
"updated_at": datetime.now(UTC).isoformat(),
"artist": "Test",
"title": "Song"
}
job = Job.model_validate(data)
assert job.job_id == "test123"
assert job.status == JobStatus.PENDING
def test_job_handles_null_optional_fields(self):
"""Test Job handles null optional fields."""
job = Job(
job_id="test",
status=JobStatus.PENDING,
created_at=datetime.now(UTC),
updated_at=datetime.now(UTC)
)
assert job.artist is None
assert job.title is None
assert job.url is None
assert job.file_urls == {}
class TestRetryEndpoint:
"""Tests for the retry job endpoint."""
@pytest.fixture
def mock_job_manager(self):
"""Create mock job manager."""
return MagicMock()
@pytest.fixture
def mock_worker_service(self):
"""Create mock worker service."""
return MagicMock()
def test_retry_rejects_pending_job(self, mock_job_manager):
"""Test retry endpoint rejects jobs that are not failed or cancelled."""
# Create a pending job
job = Job(
job_id="test123",
status=JobStatus.PENDING,
created_at=datetime.now(UTC),
updated_at=datetime.now(UTC),
artist="Test",
title="Song"
)
mock_job_manager.get_job.return_value = job
# The endpoint should reject this with a 400 error
# (status check happens before any logic)
assert job.status not in [JobStatus.FAILED, JobStatus.CANCELLED]
def test_retry_accepts_failed_job(self, mock_job_manager):
"""Test retry endpoint accepts failed jobs."""
job = Job(
job_id="test123",
status=JobStatus.FAILED,
created_at=datetime.now(UTC),
updated_at=datetime.now(UTC),
artist="Test",
title="Song",
input_media_gcs_path="jobs/test123/input/audio.flac"
)
mock_job_manager.get_job.return_value = job
# Status should be accepted
assert job.status in [JobStatus.FAILED, JobStatus.CANCELLED]
def test_retry_accepts_cancelled_job(self, mock_job_manager):
"""Test retry endpoint accepts cancelled jobs."""
job = Job(
job_id="test123",
status=JobStatus.CANCELLED,
created_at=datetime.now(UTC),
updated_at=datetime.now(UTC),
artist="Test",
title="Song",
input_media_gcs_path="jobs/test123/input/audio.flac"
)
mock_job_manager.get_job.return_value = job
# Status should be accepted
assert job.status in [JobStatus.FAILED, JobStatus.CANCELLED]
def test_retry_checkpoint_detection_video_generation(self):
"""Test retry detects video generation checkpoint."""
job = Job(
job_id="test123",
status=JobStatus.FAILED,
created_at=datetime.now(UTC),
updated_at=datetime.now(UTC),
artist="Test",
title="Song",
file_urls={
'videos': {'with_vocals': 'gs://bucket/path/video.mkv'}
},
state_data={
'instrumental_selection': 'clean'
}
)
# This job has video and instrumental selection - should retry from video generation
has_video = job.file_urls.get('videos', {}).get('with_vocals')
has_instrumental_selection = (job.state_data or {}).get('instrumental_selection')
assert has_video and has_instrumental_selection
def test_retry_checkpoint_detection_render_stage(self):
"""Test retry detects render stage checkpoint."""
job = Job(
job_id="test123",
status=JobStatus.FAILED,
created_at=datetime.now(UTC),
updated_at=datetime.now(UTC),
artist="Test",
title="Song",
file_urls={
'lyrics': {'corrections': 'gs://bucket/path/corrections.json'},
'screens': {'title': 'gs://bucket/path/title.mov'}
}
)
# This job has corrections and screens - should retry from render
has_corrections = job.file_urls.get('lyrics', {}).get('corrections')
has_screens = job.file_urls.get('screens', {}).get('title')
has_video = job.file_urls.get('videos', {}).get('with_vocals')
assert has_corrections and has_screens and not has_video
def test_retry_checkpoint_detection_from_beginning(self):
"""Test retry detects need to restart from beginning."""
job = Job(
job_id="test123",
status=JobStatus.CANCELLED,
created_at=datetime.now(UTC),
updated_at=datetime.now(UTC),
artist="Test",
title="Song",
input_media_gcs_path="jobs/test123/input/audio.flac",
file_urls={} # No progress yet
)
# This job has input audio but no other files - should restart from beginning
has_input = job.input_media_gcs_path
has_stems = job.file_urls.get('stems', {}).get('instrumental_clean')
has_corrections = job.file_urls.get('lyrics', {}).get('corrections')
assert has_input and not has_stems and not has_corrections
def test_retry_no_input_audio(self):
"""Test retry fails when no input audio available."""
job = Job(
job_id="test123",
status=JobStatus.CANCELLED,
created_at=datetime.now(UTC),
updated_at=datetime.now(UTC),
artist="Test",
title="Song",
# No input_media_gcs_path and no url
file_urls={}
)
# This job has no input audio - should not be retryable
has_input = job.input_media_gcs_path or job.url
assert not has_input
-337
View File
@@ -1,337 +0,0 @@
"""
Unit tests for review.py routes and related components.
These tests verify the review-related state transitions and data structures.
Tests that require full backend imports are in the emulator integration tests.
"""
import pytest
import json
class TestJobStatusTransitionsForReview:
"""Tests for review-related state transitions.
These tests verify the Job model's state machine handles review flow correctly.
"""
def test_awaiting_review_can_transition_to_in_review(self):
"""Test AWAITING_REVIEW -> IN_REVIEW is valid."""
from backend.models.job import STATE_TRANSITIONS, JobStatus
valid_transitions = STATE_TRANSITIONS.get(JobStatus.AWAITING_REVIEW, [])
assert JobStatus.IN_REVIEW in valid_transitions
def test_awaiting_review_can_transition_to_review_complete(self):
"""Test AWAITING_REVIEW -> REVIEW_COMPLETE is valid (skip in_review)."""
from backend.models.job import STATE_TRANSITIONS, JobStatus
valid_transitions = STATE_TRANSITIONS.get(JobStatus.AWAITING_REVIEW, [])
assert JobStatus.REVIEW_COMPLETE in valid_transitions
def test_in_review_can_transition_to_review_complete(self):
"""Test IN_REVIEW -> REVIEW_COMPLETE is valid."""
from backend.models.job import STATE_TRANSITIONS, JobStatus
valid_transitions = STATE_TRANSITIONS.get(JobStatus.IN_REVIEW, [])
assert JobStatus.REVIEW_COMPLETE in valid_transitions
def test_review_complete_transitions_to_rendering_video(self):
"""Test REVIEW_COMPLETE -> RENDERING_VIDEO is valid."""
from backend.models.job import STATE_TRANSITIONS, JobStatus
valid_transitions = STATE_TRANSITIONS.get(JobStatus.REVIEW_COMPLETE, [])
assert JobStatus.RENDERING_VIDEO in valid_transitions
def test_rendering_video_status_exists(self):
"""Test RENDERING_VIDEO status exists in JobStatus enum."""
from backend.models.job import JobStatus
assert hasattr(JobStatus, 'RENDERING_VIDEO')
assert JobStatus.RENDERING_VIDEO.value == "rendering_video"
def test_rendering_video_transitions_to_instrumental(self):
"""Test RENDERING_VIDEO -> AWAITING_INSTRUMENTAL_SELECTION is valid."""
from backend.models.job import STATE_TRANSITIONS, JobStatus
valid_transitions = STATE_TRANSITIONS.get(JobStatus.RENDERING_VIDEO, [])
assert JobStatus.AWAITING_INSTRUMENTAL_SELECTION in valid_transitions
class TestStylesConfigRequirements:
"""Tests documenting the required fields for ASS subtitle generation.
These tests verify that the styles config format is documented correctly.
The actual styles creation is tested in integration tests.
"""
def test_required_karaoke_fields_documented(self):
"""Document required karaoke style fields for ASS generation."""
# These fields are required by the ASS subtitle generator
# Missing any of them will cause video generation to fail
required_fields = [
"font", # Font name for ASS
"font_path", # Path to font file (can be empty string, NOT None)
"ass_name", # Style name in ASS file
"primary_color", # Format: "R, G, B, A"
"secondary_color", # Format: "R, G, B, A"
"outline_color", # Format: "R, G, B, A"
"back_color", # Format: "R, G, B, A"
"bold", # Boolean
"italic", # Boolean
"underline", # Boolean
"strike_out", # Boolean
"scale_x", # Integer
"scale_y", # Integer
"spacing", # Integer
"angle", # Float
"border_style", # Integer
"outline", # Integer
"shadow", # Integer
"margin_l", # Integer
"margin_r", # Integer
"margin_v", # Integer
"encoding", # Integer
]
# This test documents the required fields
# Actual validation happens in integration tests
assert len(required_fields) == 22
assert "ass_name" in required_fields # This was missing initially
assert "font_path" in required_fields # Must be string, not None
def test_minimal_styles_structure(self):
"""Document the minimal styles JSON structure."""
minimal_styles = {
"karaoke": {
"background_color": "#000000",
"font_path": "", # MUST be string, NOT None
"font": "Arial",
"ass_name": "Default", # REQUIRED
"primary_color": "112, 112, 247, 255",
"secondary_color": "255, 255, 255, 255",
"outline_color": "26, 58, 235, 255",
"back_color": "0, 0, 0, 0",
"bold": False,
"italic": False,
"underline": False,
"strike_out": False,
"scale_x": 100,
"scale_y": 100,
"spacing": 0,
"angle": 0.0,
"border_style": 1,
"outline": 1,
"shadow": 0,
"margin_l": 0,
"margin_r": 0,
"margin_v": 0,
"encoding": 0
}
}
# Verify structure is valid JSON
json_str = json.dumps(minimal_styles)
parsed = json.loads(json_str)
assert "karaoke" in parsed
assert parsed["karaoke"]["font_path"] == "" # Not None
assert parsed["karaoke"]["ass_name"] == "Default"
class TestCorrectionDataMerging:
"""Tests documenting correction data merging requirements.
The LyricsTranscriber React UI sends only partial correction data:
- corrections
- corrected_segments
The backend must merge this with the original corrections.json to
reconstruct a full CorrectionResult.
"""
def test_frontend_sends_partial_data(self):
"""Document what the frontend sends."""
# The frontend sends only these fields
frontend_payload = {
"corrections": [], # List of corrections made
"corrected_segments": [] # Updated segment data
}
# This is NOT a full CorrectionResult
assert "original_segments" not in frontend_payload
assert "metadata" not in frontend_payload
def test_merging_strategy(self):
"""Document the merging strategy for corrections."""
# Original data has full structure
original_data = {
"original_segments": [{"id": 1, "text": "test"}],
"corrected_segments": [{"id": 1, "text": "test"}],
"corrections": [],
"metadata": {"artist": "Test"}
}
# Frontend sends partial update
frontend_update = {
"corrections": [{"id": 1, "type": "edit"}],
"corrected_segments": [{"id": 1, "text": "updated"}]
}
# Merging strategy: update only the fields sent by frontend
if 'corrections' in frontend_update:
original_data['corrections'] = frontend_update['corrections']
if 'corrected_segments' in frontend_update:
original_data['corrected_segments'] = frontend_update['corrected_segments']
# Result preserves original_segments and metadata
assert original_data['original_segments'] == [{"id": 1, "text": "test"}]
assert original_data['metadata'] == {"artist": "Test"}
# But has updated corrections and corrected_segments
assert original_data['corrections'] == [{"id": 1, "type": "edit"}]
assert original_data['corrected_segments'] == [{"id": 1, "text": "updated"}]
class TestAddLyricsEndpoint:
"""Tests for the add-lyrics endpoint."""
def test_add_lyrics_requires_source_and_lyrics(self):
"""Document that add_lyrics expects source and lyrics fields."""
# The frontend sends this payload
valid_payload = {
"source": "custom",
"lyrics": "Line 1\nLine 2\nLine 3"
}
# Both fields are required
assert "source" in valid_payload
assert "lyrics" in valid_payload
assert len(valid_payload["source"].strip()) > 0
assert len(valid_payload["lyrics"].strip()) > 0
def test_add_lyrics_uses_correction_operations(self):
"""Verify CorrectionOperations.add_lyrics_source is available."""
from lyrics_transcriber.correction.operations import CorrectionOperations
# The method should exist
assert hasattr(CorrectionOperations, 'add_lyrics_source')
# It should be a static method
import inspect
# Get the method and check it's callable
method = getattr(CorrectionOperations, 'add_lyrics_source')
assert callable(method)
class TestPreviewStyleLoading:
"""Tests for the unified style loader used in preview video generation.
When a job has custom styles (uploaded via --style_params_json), these
must be loaded and applied to preview videos, not just the final render.
This was a bug: preview videos were using minimal styles (black background)
even when the job had custom backgrounds and fonts configured.
The style loading logic is now consolidated in karaoke_gen.style_loader
to avoid duplication between workers and API routes.
"""
def test_load_styles_from_gcs_with_custom_styles(self, tmp_path):
"""Test that custom styles are downloaded and applied for preview."""
import os
from karaoke_gen.style_loader import load_styles_from_gcs
# Create source style params file
source_style_params = tmp_path / "source_styles.json"
style_data = {
"karaoke": {
"background_image": "/original/path/background.png",
"font_path": "/original/path/font.ttf",
"background_color": "#000000",
"font": "Arial",
"ass_name": "Default",
}
}
source_style_params.write_text(json.dumps(style_data))
# Create source asset files
source_background = tmp_path / "background.png"
source_background.write_bytes(b"PNG image data")
source_font = tmp_path / "font.ttf"
source_font.write_bytes(b"TTF font data")
# Create mock download function that simulates GCS download
def mock_download(gcs_path, local_path):
if "style_params.json" in gcs_path:
with open(local_path, 'w') as f:
f.write(source_style_params.read_text())
elif "karaoke_background" in gcs_path:
with open(local_path, 'wb') as f:
f.write(source_background.read_bytes())
elif "font.ttf" in gcs_path:
with open(local_path, 'wb') as f:
f.write(source_font.read_bytes())
# Call the unified style loader function
style_assets = {
"style_params": "uploads/test123/style/style_params.json",
"karaoke_background": "uploads/test123/style/karaoke_background.png",
"font": "uploads/test123/style/font.ttf",
}
styles_path, result_styles = load_styles_from_gcs(
style_params_gcs_path="uploads/test123/style/style_params.json",
style_assets=style_assets,
temp_dir=str(tmp_path / "workdir"),
download_func=mock_download,
)
# Verify styles file was created
assert os.path.exists(styles_path)
# The paths should now point to the local downloaded files, not the original paths
assert "karaoke" in result_styles
assert result_styles["karaoke"]["background_image"] != "/original/path/background.png"
assert "karaoke_background.png" in result_styles["karaoke"]["background_image"]
assert result_styles["karaoke"]["font_path"] != "/original/path/font.ttf"
assert "font.ttf" in result_styles["karaoke"]["font_path"]
def test_load_styles_from_gcs_falls_back_to_minimal(self, tmp_path):
"""Test that minimal styles are used when job has no custom styles."""
import os
from karaoke_gen.style_loader import load_styles_from_gcs
# Call with no custom styles
styles_path, result_styles = load_styles_from_gcs(
style_params_gcs_path=None, # No custom styles
style_assets={},
temp_dir=str(tmp_path),
download_func=lambda gcs_path, local_path: None, # Won't be called
)
# Verify styles file was created
assert os.path.exists(styles_path)
# Should have karaoke section with minimal/default values
assert "karaoke" in result_styles
assert result_styles["karaoke"]["background_color"] == "#000000"
assert result_styles["karaoke"]["font"] == "Arial"
# Minimal styles have background_image as None (default)
assert result_styles["karaoke"].get("background_image") is None
def test_asset_mapping_is_complete(self):
"""Verify all required asset mappings are defined in the unified style loader."""
from karaoke_gen.style_loader import ASSET_KEY_MAPPINGS
# These mappings must be present for styles to work correctly
required_keys = [
"karaoke_background",
"intro_background",
"end_background",
"font",
]
for key in required_keys:
assert key in ASSET_KEY_MAPPINGS, f"Asset mapping '{key}' not found in ASSET_KEY_MAPPINGS"
# Verify karaoke_background maps to the correct path
karaoke_mapping = ASSET_KEY_MAPPINGS["karaoke_background"]
assert karaoke_mapping == ("karaoke", "background_image")
# Verify font maps to multiple sections
font_mappings = ASSET_KEY_MAPPINGS["font"]
assert isinstance(font_mappings, list)
assert ("karaoke", "font_path") in font_mappings
-556
View File
@@ -1,556 +0,0 @@
"""
Unit tests for service layer.
Tests AuthService, StorageService, WorkerService, and FirestoreService
without requiring actual cloud resources.
"""
import pytest
from unittest.mock import Mock, MagicMock, AsyncMock, patch, call
from datetime import datetime, UTC
# Mock Google Cloud before imports
import sys
sys.modules['google.cloud.firestore'] = MagicMock()
sys.modules['google.cloud.storage'] = MagicMock()
sys.modules['google.cloud.tasks_v2'] = MagicMock()
from backend.services.auth_service import AuthService, UserType
from backend.services.storage_service import StorageService
from backend.services.worker_service import WorkerService, get_worker_service
from backend.services.firestore_service import FirestoreService
from backend.models.job import Job, JobStatus
class TestAuthService:
"""Test AuthService token validation and management."""
def test_validate_admin_token(self):
"""Test validating hardcoded admin tokens."""
with patch('backend.services.auth_service.get_settings') as mock_settings:
mock_settings.return_value.admin_tokens = "admin123,secret456"
auth_service = AuthService()
# Valid admin token - returns (is_valid, user_type, usage_count, token)
is_valid, user_type, usage_count, token = auth_service.validate_token('admin123')
assert is_valid is True
assert user_type == UserType.ADMIN
# Another valid admin token
is_valid, user_type, _, _ = auth_service.validate_token('secret456')
assert is_valid is True
assert user_type == UserType.ADMIN
def test_validate_invalid_token(self):
"""Test rejecting invalid tokens."""
with patch('backend.services.auth_service.FirestoreService') as mock_fs:
mock_fs_instance = Mock()
mock_fs.return_value = mock_fs_instance
mock_fs_instance.get_token.return_value = None # Token not in DB
with patch('backend.services.auth_service.get_settings') as mock_settings:
mock_settings.return_value.admin_tokens = "admin123"
auth_service = AuthService()
is_valid, _, _, _ = auth_service.validate_token('invalid_token')
assert is_valid is False
def test_validate_token_from_firestore(self):
"""Test validating tokens stored in Firestore."""
with patch('backend.services.auth_service.FirestoreService') as mock_fs:
mock_fs_instance = Mock()
mock_fs.return_value = mock_fs_instance
# Mock Firestore returning a token
mock_fs_instance.get_token.return_value = {
'token': 'db_token123',
'type': 'unlimited',
'valid': True,
'usage_count': 5,
'created_at': datetime.now(UTC).isoformat()
}
with patch('backend.services.auth_service.get_settings') as mock_settings:
mock_settings.return_value.admin_tokens = ""
auth_service = AuthService()
is_valid, user_type, usage_count, token = auth_service.validate_token('db_token123')
assert is_valid is True
assert user_type == UserType.UNLIMITED
# usage_count can be -1 for unlimited tokens
assert isinstance(usage_count, int)
class TestStorageService:
"""Test StorageService GCS operations."""
def test_upload_file(self):
"""Test uploading a file to GCS."""
with patch('backend.services.storage_service.storage') as mock_storage:
mock_client = MagicMock()
mock_storage.Client.return_value = mock_client
mock_bucket = MagicMock()
mock_client.bucket.return_value = mock_bucket
mock_blob = MagicMock()
mock_bucket.blob.return_value = mock_blob
storage_service = StorageService()
# Upload file - correct parameter names
result = storage_service.upload_file(
local_path="/tmp/test.flac",
destination_path="uploads/test123/test.flac"
)
# Verify blob was created and uploaded
mock_bucket.blob.assert_called_once_with("uploads/test123/test.flac")
mock_blob.upload_from_filename.assert_called_once_with("/tmp/test.flac")
# Result should be the destination path
assert result == "uploads/test123/test.flac"
def test_download_file(self):
"""Test downloading a file from GCS."""
with patch('backend.services.storage_service.storage') as mock_storage:
mock_client = MagicMock()
mock_storage.Client.return_value = mock_client
mock_bucket = MagicMock()
mock_client.bucket.return_value = mock_bucket
mock_blob = MagicMock()
mock_bucket.blob.return_value = mock_blob
storage_service = StorageService()
# Download file - correct parameter names
storage_service.download_file(
source_path="uploads/test123/test.flac",
destination_path="/tmp/downloaded.flac"
)
# Verify download was called
mock_blob.download_to_filename.assert_called_once_with("/tmp/downloaded.flac")
def test_delete_file(self):
"""Test deleting a file from GCS."""
with patch('backend.services.storage_service.storage') as mock_storage:
mock_client = MagicMock()
mock_storage.Client.return_value = mock_client
mock_bucket = MagicMock()
mock_client.bucket.return_value = mock_bucket
mock_blob = MagicMock()
mock_bucket.blob.return_value = mock_blob
storage_service = StorageService()
# Delete file
storage_service.delete_file("uploads/test123/test.flac")
# Verify delete was called
mock_blob.delete.assert_called_once()
class TestWorkerService:
"""Test WorkerService internal HTTP calls."""
@pytest.mark.asyncio
async def test_trigger_audio_worker(self):
"""Test triggering audio worker via internal HTTP."""
with patch('backend.services.worker_service.httpx.AsyncClient') as mock_client_cls:
mock_client = AsyncMock()
mock_client_cls.return_value.__aenter__.return_value = mock_client
mock_response = Mock()
mock_response.status_code = 200
mock_client.post.return_value = mock_response
worker_service = get_worker_service()
await worker_service.trigger_audio_worker("test123")
# Verify HTTP POST was made
mock_client.post.assert_called_once()
call_args = mock_client.post.call_args
assert "/api/internal/workers/audio" in str(call_args)
@pytest.mark.asyncio
async def test_trigger_lyrics_worker(self):
"""Test triggering lyrics worker via internal HTTP."""
with patch('backend.services.worker_service.httpx.AsyncClient') as mock_client_cls:
mock_client = AsyncMock()
mock_client_cls.return_value.__aenter__.return_value = mock_client
mock_response = Mock()
mock_response.status_code = 200
mock_client.post.return_value = mock_response
worker_service = get_worker_service()
await worker_service.trigger_lyrics_worker("test123")
# Verify HTTP POST was made
mock_client.post.assert_called_once()
call_args = mock_client.post.call_args
assert "/api/internal/workers/lyrics" in str(call_args)
@pytest.mark.asyncio
async def test_trigger_screens_worker(self):
"""Test triggering screens worker via internal HTTP."""
with patch('backend.services.worker_service.httpx.AsyncClient') as mock_client_cls:
mock_client = AsyncMock()
mock_client_cls.return_value.__aenter__.return_value = mock_client
mock_response = Mock()
mock_response.status_code = 200
mock_client.post.return_value = mock_response
worker_service = get_worker_service()
await worker_service.trigger_screens_worker("test123")
# Verify HTTP POST was made
mock_client.post.assert_called_once()
call_args = mock_client.post.call_args
assert "/api/internal/workers/screens" in str(call_args)
@pytest.mark.asyncio
async def test_trigger_video_worker(self):
"""Test triggering video worker via internal HTTP."""
with patch('backend.services.worker_service.httpx.AsyncClient') as mock_client_cls:
mock_client = AsyncMock()
mock_client_cls.return_value.__aenter__.return_value = mock_client
mock_response = Mock()
mock_response.status_code = 200
mock_client.post.return_value = mock_response
worker_service = get_worker_service()
await worker_service.trigger_video_worker("test123")
# Verify HTTP POST was made
mock_client.post.assert_called_once()
call_args = mock_client.post.call_args
assert "/api/internal/workers/video" in str(call_args)
class TestWorkerServiceCloudTasks:
"""Test WorkerService Cloud Tasks integration."""
def test_should_use_cloud_tasks_default_false(self):
"""Test that Cloud Tasks is disabled by default."""
from backend.services.worker_service import WorkerService, reset_worker_service
reset_worker_service()
with patch('backend.services.worker_service.os.getenv') as mock_getenv:
mock_getenv.side_effect = lambda k, d=None: {
'PORT': '8000',
}.get(k, d)
with patch('backend.services.worker_service.get_settings') as mock_settings:
mock_settings.return_value.admin_tokens = None
mock_settings.return_value.google_cloud_project = "test-project"
mock_settings.return_value.enable_cloud_tasks = False
mock_settings.return_value.gcp_region = "us-central1"
mock_settings.return_value.use_cloud_run_jobs_for_video = False
service = WorkerService()
assert service._use_cloud_tasks is False
def test_should_use_cloud_tasks_enabled(self):
"""Test that Cloud Tasks can be enabled via settings."""
from backend.services.worker_service import WorkerService, reset_worker_service
reset_worker_service()
with patch('backend.services.worker_service.os.getenv') as mock_getenv:
mock_getenv.side_effect = lambda k, d=None: {
'PORT': '8000',
}.get(k, d)
with patch('backend.services.worker_service.get_settings') as mock_settings:
mock_settings.return_value.admin_tokens = None
mock_settings.return_value.google_cloud_project = "test-project"
mock_settings.return_value.enable_cloud_tasks = True
mock_settings.return_value.gcp_region = "us-central1"
mock_settings.return_value.use_cloud_run_jobs_for_video = False
service = WorkerService()
assert service._use_cloud_tasks is True
def test_worker_queues_mapping(self):
"""Test that all worker types have queue mappings."""
from backend.services.worker_service import WORKER_QUEUES
# Verify all expected workers have queues
assert "audio" in WORKER_QUEUES
assert "lyrics" in WORKER_QUEUES
assert "screens" in WORKER_QUEUES
assert "render-video" in WORKER_QUEUES
assert "video" in WORKER_QUEUES
# Verify queue names are correct
assert WORKER_QUEUES["audio"] == "audio-worker-queue"
assert WORKER_QUEUES["lyrics"] == "lyrics-worker-queue"
assert WORKER_QUEUES["screens"] == "screens-worker-queue"
assert WORKER_QUEUES["render-video"] == "render-worker-queue"
assert WORKER_QUEUES["video"] == "video-worker-queue"
@pytest.mark.asyncio
async def test_trigger_worker_uses_http_when_cloud_tasks_disabled(self):
"""Test that trigger_worker uses HTTP when Cloud Tasks is disabled."""
from backend.services.worker_service import WorkerService, reset_worker_service
reset_worker_service()
with patch('backend.services.worker_service.os.getenv') as mock_getenv:
mock_getenv.side_effect = lambda k, d=None: {
'PORT': '8000',
}.get(k, d)
with patch('backend.services.worker_service.get_settings') as mock_settings:
mock_settings.return_value.admin_tokens = "test-token"
mock_settings.return_value.google_cloud_project = "test-project"
mock_settings.return_value.enable_cloud_tasks = False
mock_settings.return_value.gcp_region = "us-central1"
mock_settings.return_value.use_cloud_run_jobs_for_video = False
with patch('backend.services.worker_service.httpx.AsyncClient') as mock_client_cls:
mock_client = AsyncMock()
mock_client_cls.return_value.__aenter__.return_value = mock_client
mock_response = Mock()
mock_response.status_code = 200
mock_client.post.return_value = mock_response
service = WorkerService()
assert service._use_cloud_tasks is False
result = await service.trigger_worker("audio", "test-job-123")
# Verify HTTP was used
assert result is True
mock_client.post.assert_called_once()
@pytest.mark.asyncio
async def test_trigger_worker_uses_cloud_tasks_when_enabled(self):
"""Test that trigger_worker uses Cloud Tasks when enabled."""
from backend.services.worker_service import WorkerService, reset_worker_service
reset_worker_service()
with patch('backend.services.worker_service.os.getenv') as mock_getenv:
mock_getenv.side_effect = lambda k, d=None: {
'CLOUD_RUN_SERVICE_URL': 'https://api.example.com',
}.get(k, d)
with patch('backend.services.worker_service.get_settings') as mock_settings:
mock_settings.return_value.admin_tokens = "test-token"
mock_settings.return_value.google_cloud_project = "test-project"
mock_settings.return_value.enable_cloud_tasks = True
mock_settings.return_value.gcp_region = "us-central1"
mock_settings.return_value.use_cloud_run_jobs_for_video = False
# Mock Cloud Tasks client
mock_tasks_client = MagicMock()
mock_tasks_client.queue_path.return_value = "projects/test-project/locations/us-central1/queues/audio-worker-queue"
mock_task_response = MagicMock()
mock_task_response.name = "projects/test-project/locations/us-central1/queues/audio-worker-queue/tasks/abc123"
mock_tasks_client.create_task.return_value = mock_task_response
# Mock the google.cloud.tasks_v2 module which is imported inside the method
mock_tasks_module = MagicMock()
mock_tasks_module.CloudTasksClient.return_value = mock_tasks_client
mock_tasks_module.HttpMethod.POST = "POST"
with patch.dict('sys.modules', {'google.cloud.tasks_v2': mock_tasks_module}):
with patch.dict(sys.modules, {'google.cloud': MagicMock()}):
service = WorkerService()
service._tasks_client = mock_tasks_client # Inject mocked client
result = await service.trigger_worker("audio", "test-job-123")
# Verify Cloud Tasks was used
assert result is True
mock_tasks_client.create_task.assert_called_once()
# Verify task payload
call_args = mock_tasks_client.create_task.call_args
assert call_args is not None
@pytest.mark.asyncio
async def test_trigger_worker_returns_false_for_unknown_worker_type(self):
"""Test that trigger_worker returns False for unknown worker types."""
from backend.services.worker_service import WorkerService, reset_worker_service
reset_worker_service()
with patch('backend.services.worker_service.os.getenv') as mock_getenv:
mock_getenv.side_effect = lambda k, d=None: {
'CLOUD_RUN_SERVICE_URL': 'https://api.example.com',
}.get(k, d)
with patch('backend.services.worker_service.get_settings') as mock_settings:
mock_settings.return_value.admin_tokens = "test-token"
mock_settings.return_value.google_cloud_project = "test-project"
mock_settings.return_value.enable_cloud_tasks = True
mock_settings.return_value.gcp_region = "us-central1"
mock_settings.return_value.use_cloud_run_jobs_for_video = False
service = WorkerService()
result = await service.trigger_worker("unknown-worker", "test-job-123")
assert result is False
def test_reset_worker_service(self):
"""Test that reset_worker_service resets the singleton."""
from backend.services.worker_service import get_worker_service, reset_worker_service
with patch('backend.services.worker_service.get_settings') as mock_settings:
mock_settings.return_value.admin_tokens = None
mock_settings.return_value.google_cloud_project = "test-project"
mock_settings.return_value.enable_cloud_tasks = False
mock_settings.return_value.gcp_region = "us-central1"
mock_settings.return_value.use_cloud_run_jobs_for_video = False
service1 = get_worker_service()
reset_worker_service()
service2 = get_worker_service()
# After reset, should be different instances
assert service1 is not service2
class TestFirestoreService:
"""Test FirestoreService database operations."""
def test_create_job(self):
"""Test creating a job document in Firestore."""
with patch('backend.services.firestore_service.firestore') as mock_firestore:
mock_client = MagicMock()
mock_firestore.Client.return_value = mock_client
mock_collection = MagicMock()
mock_client.collection.return_value = mock_collection
mock_doc_ref = MagicMock()
mock_collection.document.return_value = mock_doc_ref
firestore_service = FirestoreService()
job = Job(
job_id="test123",
status=JobStatus.PENDING,
created_at=datetime.now(UTC),
updated_at=datetime.now(UTC)
)
firestore_service.create_job(job) # Returns None
# Verify document was set
mock_doc_ref.set.assert_called_once()
def test_get_job(self):
"""Test fetching a job from Firestore."""
with patch('backend.services.firestore_service.firestore') as mock_firestore:
mock_client = MagicMock()
mock_firestore.Client.return_value = mock_client
mock_collection = MagicMock()
mock_client.collection.return_value = mock_collection
mock_doc_ref = MagicMock()
mock_collection.document.return_value = mock_doc_ref
mock_doc = MagicMock()
mock_doc.exists = True
mock_doc.to_dict.return_value = {
'job_id': 'test123',
'status': 'pending',
'progress': 0,
'created_at': datetime.now(UTC).isoformat(),
'updated_at': datetime.now(UTC).isoformat()
}
mock_doc_ref.get.return_value = mock_doc
firestore_service = FirestoreService()
job = firestore_service.get_job("test123")
# Verify document was fetched
mock_doc_ref.get.assert_called_once()
assert job is not None
assert job.job_id == "test123"
def test_get_nonexistent_job(self):
"""Test fetching a job that doesn't exist."""
with patch('backend.services.firestore_service.firestore') as mock_firestore:
mock_client = MagicMock()
mock_firestore.Client.return_value = mock_client
mock_collection = MagicMock()
mock_client.collection.return_value = mock_collection
mock_doc_ref = MagicMock()
mock_collection.document.return_value = mock_doc_ref
mock_doc = MagicMock()
mock_doc.exists = False
mock_doc_ref.get.return_value = mock_doc
firestore_service = FirestoreService()
job = firestore_service.get_job("nonexistent")
assert job is None
def test_update_job(self):
"""Test updating a job in Firestore."""
with patch('backend.services.firestore_service.firestore') as mock_firestore:
mock_client = MagicMock()
mock_firestore.Client.return_value = mock_client
mock_collection = MagicMock()
mock_client.collection.return_value = mock_collection
mock_doc_ref = MagicMock()
mock_collection.document.return_value = mock_doc_ref
firestore_service = FirestoreService()
updates = {
'status': JobStatus.SEPARATING_STAGE1,
'progress': 25
}
firestore_service.update_job("test123", updates)
# Verify update was called
mock_doc_ref.update.assert_called_once()
def test_delete_job(self):
"""Test deleting a job from Firestore."""
with patch('backend.services.firestore_service.firestore') as mock_firestore:
mock_client = MagicMock()
mock_firestore.Client.return_value = mock_client
mock_collection = MagicMock()
mock_client.collection.return_value = mock_collection
mock_doc_ref = MagicMock()
mock_collection.document.return_value = mock_doc_ref
firestore_service = FirestoreService()
firestore_service.delete_job("test123")
# Verify delete was called
mock_doc_ref.delete.assert_called_once()
if __name__ == "__main__":
pytest.main([__file__, "-v"])
-112
View File
@@ -1,112 +0,0 @@
"""
Extended unit tests for services.
These tests provide additional coverage for service layer code.
"""
import pytest
from datetime import datetime, UTC
from unittest.mock import MagicMock, AsyncMock, patch
from backend.models.job import Job, JobStatus
class TestJobManagerExtended:
"""Extended tests for JobManager.
Note: These tests verify the module structure and basic behavior.
Full JobManager testing is in test_job_manager.py.
"""
def test_job_manager_module_imports(self):
"""Test JobManager module can be imported."""
from backend.services import job_manager
assert hasattr(job_manager, 'JobManager')
def test_state_transitions_defined(self):
"""Test STATE_TRANSITIONS dict is defined."""
from backend.models.job import STATE_TRANSITIONS
assert STATE_TRANSITIONS is not None
assert len(STATE_TRANSITIONS) > 0
class TestStorageServiceExtended:
"""Extended tests for StorageService."""
def test_storage_service_initialization(self):
"""Test StorageService can be initialized."""
with patch('backend.services.storage_service.storage'):
from backend.services.storage_service import StorageService
service = StorageService()
assert service is not None
def test_storage_service_bucket_name(self):
"""Test StorageService uses configured bucket."""
mock_client = MagicMock()
mock_bucket = MagicMock()
mock_client.bucket.return_value = mock_bucket
with patch('backend.services.storage_service.storage.Client', return_value=mock_client), \
patch.dict('os.environ', {'GCS_BUCKET_NAME': 'test-bucket'}):
from backend.services.storage_service import StorageService
service = StorageService()
# Service should use the configured bucket
class TestAuthServiceExtended:
"""Extended tests for AuthService.
Note: Full auth service testing is in test_services.py.
"""
def test_auth_service_module_imports(self):
"""Test AuthService module can be imported."""
from backend.services import auth_service
assert hasattr(auth_service, 'AuthService')
class TestWorkerServiceExtended:
"""Extended tests for WorkerService."""
def test_worker_service_initialization(self):
"""Test WorkerService can be initialized."""
from backend.services.worker_service import WorkerService
service = WorkerService()
assert service is not None
def test_worker_service_get_base_url(self):
"""Test WorkerService constructs correct base URL."""
from backend.services.worker_service import WorkerService
service = WorkerService()
url = service._get_base_url()
assert 'http' in url
@pytest.mark.asyncio
async def test_worker_service_trigger_with_mock(self):
"""Test triggering a worker with mocked HTTP."""
from backend.services.worker_service import WorkerService
service = WorkerService()
# Mock the HTTP client
with patch('backend.services.worker_service.httpx.AsyncClient') as mock_client_cls:
mock_client = MagicMock()
mock_response = MagicMock()
mock_response.status_code = 200
mock_client.post = AsyncMock(return_value=mock_response)
mock_client_cls.return_value.__aenter__ = AsyncMock(return_value=mock_client)
mock_client_cls.return_value.__aexit__ = AsyncMock(return_value=None)
# The trigger methods should work with mocked HTTP
class TestFirestoreServiceExtended:
"""Extended tests for FirestoreService."""
def test_firestore_service_initialization(self):
"""Test FirestoreService can be initialized."""
with patch('backend.services.firestore_service.firestore'):
from backend.services.firestore_service import FirestoreService
service = FirestoreService()
assert service is not None
-448
View File
@@ -1,448 +0,0 @@
"""
Tests for storage_service.py - Google Cloud Storage operations.
These tests mock the GCS client to verify:
- Upload operations (file, fileobj, JSON)
- Download operations (file, JSON)
- Signed URL generation
- File operations (delete, list, exists)
"""
import json
import pytest
from unittest.mock import Mock, MagicMock, patch
from io import BytesIO
from backend.services.storage_service import StorageService
class TestStorageServiceInit:
"""Test StorageService initialization."""
@patch("backend.services.storage_service.storage.Client")
@patch("backend.services.storage_service.settings")
def test_init_creates_client_and_bucket(self, mock_settings, mock_client_class):
"""Test initialization creates GCS client and gets bucket."""
mock_settings.google_cloud_project = "test-project"
mock_settings.gcs_bucket_name = "test-bucket"
mock_client = Mock()
mock_bucket = Mock()
mock_client.bucket.return_value = mock_bucket
mock_client_class.return_value = mock_client
service = StorageService()
mock_client_class.assert_called_once_with(project="test-project")
mock_client.bucket.assert_called_once_with("test-bucket")
assert service.client == mock_client
assert service.bucket == mock_bucket
class TestStorageServiceUpload:
"""Test upload operations."""
@patch("backend.services.storage_service.storage.Client")
@patch("backend.services.storage_service.settings")
def test_upload_file(self, mock_settings, mock_client_class):
"""Test uploading a file to GCS."""
mock_settings.google_cloud_project = "test-project"
mock_settings.gcs_bucket_name = "test-bucket"
mock_blob = Mock()
mock_bucket = Mock()
mock_bucket.blob.return_value = mock_blob
mock_client = Mock()
mock_client.bucket.return_value = mock_bucket
mock_client_class.return_value = mock_client
service = StorageService()
result = service.upload_file("/local/path/file.flac", "uploads/job123/file.flac")
mock_bucket.blob.assert_called_once_with("uploads/job123/file.flac")
mock_blob.upload_from_filename.assert_called_once_with("/local/path/file.flac")
assert result == "uploads/job123/file.flac"
@patch("backend.services.storage_service.storage.Client")
@patch("backend.services.storage_service.settings")
def test_upload_file_raises_on_error(self, mock_settings, mock_client_class):
"""Test upload_file raises exception on GCS error."""
mock_settings.google_cloud_project = "test-project"
mock_settings.gcs_bucket_name = "test-bucket"
mock_blob = Mock()
mock_blob.upload_from_filename.side_effect = Exception("GCS error")
mock_bucket = Mock()
mock_bucket.blob.return_value = mock_blob
mock_client = Mock()
mock_client.bucket.return_value = mock_bucket
mock_client_class.return_value = mock_client
service = StorageService()
with pytest.raises(Exception) as exc_info:
service.upload_file("/local/path/file.flac", "dest/file.flac")
assert "GCS error" in str(exc_info.value)
@patch("backend.services.storage_service.storage.Client")
@patch("backend.services.storage_service.settings")
def test_upload_fileobj(self, mock_settings, mock_client_class):
"""Test uploading a file object to GCS."""
mock_settings.google_cloud_project = "test-project"
mock_settings.gcs_bucket_name = "test-bucket"
mock_blob = Mock()
mock_bucket = Mock()
mock_bucket.blob.return_value = mock_blob
mock_client = Mock()
mock_client.bucket.return_value = mock_bucket
mock_client_class.return_value = mock_client
service = StorageService()
file_obj = BytesIO(b"test content")
result = service.upload_fileobj(file_obj, "uploads/test.txt", content_type="text/plain")
mock_bucket.blob.assert_called_once_with("uploads/test.txt")
assert mock_blob.content_type == "text/plain"
mock_blob.upload_from_file.assert_called_once_with(file_obj, rewind=True)
assert result == "uploads/test.txt"
@patch("backend.services.storage_service.storage.Client")
@patch("backend.services.storage_service.settings")
def test_upload_json(self, mock_settings, mock_client_class):
"""Test uploading JSON data to GCS."""
mock_settings.google_cloud_project = "test-project"
mock_settings.gcs_bucket_name = "test-bucket"
mock_blob = Mock()
mock_bucket = Mock()
mock_bucket.blob.return_value = mock_blob
mock_client = Mock()
mock_client.bucket.return_value = mock_bucket
mock_client_class.return_value = mock_client
service = StorageService()
data = {"key": "value", "nested": {"data": True}}
result = service.upload_json("data/config.json", data)
mock_bucket.blob.assert_called_once_with("data/config.json")
assert mock_blob.content_type == "application/json"
# Check the uploaded content
call_args = mock_blob.upload_from_string.call_args
uploaded_content = call_args[0][0]
assert json.loads(uploaded_content) == data
assert result == "data/config.json"
class TestStorageServiceDownload:
"""Test download operations."""
@patch("backend.services.storage_service.storage.Client")
@patch("backend.services.storage_service.settings")
def test_download_file(self, mock_settings, mock_client_class):
"""Test downloading a file from GCS."""
mock_settings.google_cloud_project = "test-project"
mock_settings.gcs_bucket_name = "test-bucket"
mock_blob = Mock()
mock_bucket = Mock()
mock_bucket.blob.return_value = mock_blob
mock_client = Mock()
mock_client.bucket.return_value = mock_bucket
mock_client_class.return_value = mock_client
service = StorageService()
result = service.download_file("uploads/job123/file.flac", "/local/path/file.flac")
mock_bucket.blob.assert_called_once_with("uploads/job123/file.flac")
mock_blob.download_to_filename.assert_called_once_with("/local/path/file.flac")
assert result == "/local/path/file.flac"
@patch("backend.services.storage_service.storage.Client")
@patch("backend.services.storage_service.settings")
def test_download_file_raises_on_error(self, mock_settings, mock_client_class):
"""Test download_file raises exception on GCS error."""
mock_settings.google_cloud_project = "test-project"
mock_settings.gcs_bucket_name = "test-bucket"
mock_blob = Mock()
mock_blob.download_to_filename.side_effect = Exception("File not found")
mock_bucket = Mock()
mock_bucket.blob.return_value = mock_blob
mock_client = Mock()
mock_client.bucket.return_value = mock_bucket
mock_client_class.return_value = mock_client
service = StorageService()
with pytest.raises(Exception) as exc_info:
service.download_file("missing/file.flac", "/local/path")
assert "File not found" in str(exc_info.value)
@patch("backend.services.storage_service.storage.Client")
@patch("backend.services.storage_service.settings")
def test_download_json(self, mock_settings, mock_client_class):
"""Test downloading and parsing JSON from GCS."""
mock_settings.google_cloud_project = "test-project"
mock_settings.gcs_bucket_name = "test-bucket"
mock_blob = Mock()
mock_blob.download_as_text.return_value = '{"key": "value", "count": 42}'
mock_bucket = Mock()
mock_bucket.blob.return_value = mock_blob
mock_client = Mock()
mock_client.bucket.return_value = mock_bucket
mock_client_class.return_value = mock_client
service = StorageService()
result = service.download_json("data/config.json")
assert result == {"key": "value", "count": 42}
class TestStorageServiceDelete:
"""Test delete operations."""
@patch("backend.services.storage_service.storage.Client")
@patch("backend.services.storage_service.settings")
def test_delete_file(self, mock_settings, mock_client_class):
"""Test deleting a single file from GCS."""
mock_settings.google_cloud_project = "test-project"
mock_settings.gcs_bucket_name = "test-bucket"
mock_blob = Mock()
mock_bucket = Mock()
mock_bucket.blob.return_value = mock_blob
mock_client = Mock()
mock_client.bucket.return_value = mock_bucket
mock_client_class.return_value = mock_client
service = StorageService()
service.delete_file("uploads/job123/file.flac")
mock_bucket.blob.assert_called_once_with("uploads/job123/file.flac")
mock_blob.delete.assert_called_once()
@patch("backend.services.storage_service.storage.Client")
@patch("backend.services.storage_service.settings")
def test_delete_folder(self, mock_settings, mock_client_class):
"""Test deleting all files with a prefix (folder)."""
mock_settings.google_cloud_project = "test-project"
mock_settings.gcs_bucket_name = "test-bucket"
# Create mock blobs
mock_blob1 = Mock()
mock_blob1.name = "uploads/job123/file1.flac"
mock_blob2 = Mock()
mock_blob2.name = "uploads/job123/file2.flac"
mock_blob3 = Mock()
mock_blob3.name = "uploads/job123/subdir/file3.flac"
mock_bucket = Mock()
mock_bucket.list_blobs.return_value = [mock_blob1, mock_blob2, mock_blob3]
mock_client = Mock()
mock_client.bucket.return_value = mock_bucket
mock_client_class.return_value = mock_client
service = StorageService()
count = service.delete_folder("uploads/job123/")
mock_bucket.list_blobs.assert_called_once_with(prefix="uploads/job123/")
mock_blob1.delete.assert_called_once()
mock_blob2.delete.assert_called_once()
mock_blob3.delete.assert_called_once()
assert count == 3
@patch("backend.services.storage_service.storage.Client")
@patch("backend.services.storage_service.settings")
def test_delete_folder_handles_errors_gracefully(self, mock_settings, mock_client_class):
"""Test delete_folder continues even if some deletes fail."""
mock_settings.google_cloud_project = "test-project"
mock_settings.gcs_bucket_name = "test-bucket"
mock_blob1 = Mock()
mock_blob1.name = "file1.flac"
mock_blob2 = Mock()
mock_blob2.name = "file2.flac"
mock_blob2.delete.side_effect = Exception("Permission denied")
mock_blob3 = Mock()
mock_blob3.name = "file3.flac"
mock_bucket = Mock()
mock_bucket.list_blobs.return_value = [mock_blob1, mock_blob2, mock_blob3]
mock_client = Mock()
mock_client.bucket.return_value = mock_bucket
mock_client_class.return_value = mock_client
service = StorageService()
count = service.delete_folder("uploads/")
# Should have deleted 2 files (blob2 failed)
assert count == 2
class TestStorageServiceFileOperations:
"""Test file listing and existence checks."""
@patch("backend.services.storage_service.storage.Client")
@patch("backend.services.storage_service.settings")
def test_list_files(self, mock_settings, mock_client_class):
"""Test listing files with a prefix."""
mock_settings.google_cloud_project = "test-project"
mock_settings.gcs_bucket_name = "test-bucket"
mock_blob1 = Mock()
mock_blob1.name = "uploads/job123/audio.flac"
mock_blob2 = Mock()
mock_blob2.name = "uploads/job123/lyrics.txt"
mock_bucket = Mock()
mock_bucket.list_blobs.return_value = [mock_blob1, mock_blob2]
mock_client = Mock()
mock_client.bucket.return_value = mock_bucket
mock_client_class.return_value = mock_client
service = StorageService()
files = service.list_files("uploads/job123/")
assert files == ["uploads/job123/audio.flac", "uploads/job123/lyrics.txt"]
@patch("backend.services.storage_service.storage.Client")
@patch("backend.services.storage_service.settings")
def test_file_exists_returns_true(self, mock_settings, mock_client_class):
"""Test file_exists returns True for existing file."""
mock_settings.google_cloud_project = "test-project"
mock_settings.gcs_bucket_name = "test-bucket"
mock_blob = Mock()
mock_blob.exists.return_value = True
mock_bucket = Mock()
mock_bucket.blob.return_value = mock_blob
mock_client = Mock()
mock_client.bucket.return_value = mock_bucket
mock_client_class.return_value = mock_client
service = StorageService()
result = service.file_exists("uploads/job123/file.flac")
assert result is True
@patch("backend.services.storage_service.storage.Client")
@patch("backend.services.storage_service.settings")
def test_file_exists_returns_false(self, mock_settings, mock_client_class):
"""Test file_exists returns False for missing file."""
mock_settings.google_cloud_project = "test-project"
mock_settings.gcs_bucket_name = "test-bucket"
mock_blob = Mock()
mock_blob.exists.return_value = False
mock_bucket = Mock()
mock_bucket.blob.return_value = mock_blob
mock_client = Mock()
mock_client.bucket.return_value = mock_bucket
mock_client_class.return_value = mock_client
service = StorageService()
result = service.file_exists("missing/file.flac")
assert result is False
class TestStorageServiceSignedUrls:
"""Test signed URL generation."""
@patch("backend.services.storage_service.storage.Client")
@patch("backend.services.storage_service.settings")
def test_generate_signed_url(self, mock_settings, mock_client_class):
"""Test generating a signed download URL."""
mock_settings.google_cloud_project = "test-project"
mock_settings.gcs_bucket_name = "test-bucket"
mock_blob = Mock()
mock_blob.generate_signed_url.return_value = "https://storage.googleapis.com/signed-url"
mock_bucket = Mock()
mock_bucket.blob.return_value = mock_blob
mock_client = Mock()
mock_client.bucket.return_value = mock_bucket
mock_client_class.return_value = mock_client
# Mock google.auth.default to return credentials without service_account_email
with patch("google.auth.default") as mock_auth_default:
mock_credentials = Mock(spec=[]) # No service_account_email attr
mock_auth_default.return_value = (mock_credentials, "test-project")
service = StorageService()
url = service.generate_signed_url("uploads/file.flac", expiration_minutes=30)
assert url == "https://storage.googleapis.com/signed-url"
mock_blob.generate_signed_url.assert_called_once()
call_kwargs = mock_blob.generate_signed_url.call_args.kwargs
assert call_kwargs["method"] == "GET"
assert call_kwargs["version"] == "v4"
@patch("backend.services.storage_service.storage.Client")
@patch("backend.services.storage_service.settings")
def test_generate_signed_upload_url(self, mock_settings, mock_client_class):
"""Test generating a signed upload URL."""
mock_settings.google_cloud_project = "test-project"
mock_settings.gcs_bucket_name = "test-bucket"
mock_blob = Mock()
mock_blob.generate_signed_url.return_value = "https://storage.googleapis.com/signed-upload-url"
mock_bucket = Mock()
mock_bucket.blob.return_value = mock_blob
mock_client = Mock()
mock_client.bucket.return_value = mock_bucket
mock_client_class.return_value = mock_client
with patch("google.auth.default") as mock_auth_default:
mock_credentials = Mock(spec=[])
mock_auth_default.return_value = (mock_credentials, "test-project")
service = StorageService()
url = service.generate_signed_upload_url(
"uploads/file.flac",
content_type="audio/flac",
expiration_minutes=15
)
assert url == "https://storage.googleapis.com/signed-upload-url"
call_kwargs = mock_blob.generate_signed_url.call_args.kwargs
assert call_kwargs["method"] == "PUT"
assert call_kwargs["headers"] == {"Content-Type": "audio/flac"}
@patch("backend.services.storage_service.storage.Client")
@patch("backend.services.storage_service.settings")
def test_signed_url_with_service_account(self, mock_settings, mock_client_class):
"""Test signed URL generation with service account credentials."""
mock_settings.google_cloud_project = "test-project"
mock_settings.gcs_bucket_name = "test-bucket"
mock_blob = Mock()
mock_blob.generate_signed_url.return_value = "https://storage.googleapis.com/signed-url"
mock_bucket = Mock()
mock_bucket.blob.return_value = mock_blob
mock_client = Mock()
mock_client.bucket.return_value = mock_bucket
mock_client_class.return_value = mock_client
with patch("google.auth.default") as mock_auth_default:
with patch("google.auth.transport.requests.Request") as mock_request:
# Mock credentials with service_account_email
mock_credentials = Mock()
mock_credentials.service_account_email = "sa@project.iam.gserviceaccount.com"
mock_credentials.token = "access-token-123"
mock_auth_default.return_value = (mock_credentials, "test-project")
service = StorageService()
url = service.generate_signed_url("uploads/file.flac")
# Should have refreshed credentials and used IAM signing
mock_credentials.refresh.assert_called_once()
call_kwargs = mock_blob.generate_signed_url.call_args.kwargs
assert call_kwargs["service_account_email"] == "sa@project.iam.gserviceaccount.com"
assert call_kwargs["access_token"] == "access-token-123"
Loaded 100 of 871 files, more files were not shown because too many files have changed in this diff. Show more