mirror of
https://github.com/nomadkaraoke/karaoke-gen.git
synced 2026-11-21 21:38:07 +09:00
Compare commits
1
Commits
| Author | SHA1 | Date | |
|---|---|---|---|
|
|
3f483f67cc |
No files matched your search
@@ -1,74 +0,0 @@
|
||||
---
|
||||
description: Periodic documentation maintenance and organization check
|
||||
allowed-tools: Read, Glob, Grep, Edit, Write, Bash
|
||||
---
|
||||
|
||||
# Documentation Maintenance Check
|
||||
|
||||
You are performing periodic maintenance on the project documentation.
|
||||
|
||||
## Checks to Perform
|
||||
|
||||
### 1. Structure Compliance
|
||||
Verify docs/ follows the correct structure:
|
||||
```
|
||||
docs/
|
||||
├── README.md # Current status + navigation
|
||||
├── ARCHITECTURE.md # System design
|
||||
├── DEVELOPMENT.md # Dev setup, testing, deployment
|
||||
├── API.md # Backend API reference
|
||||
├── LESSONS-LEARNED.md # Accumulated wisdom
|
||||
└── archive/ # Historical docs (YYYY-MM-DD-topic.md)
|
||||
```
|
||||
|
||||
Report any files that don't belong or are misplaced.
|
||||
|
||||
### 2. Freshness Check
|
||||
- Read `docs/README.md` - is the status section current?
|
||||
- Check recent git commits - do they suggest docs should be updated?
|
||||
- Look for stale information that contradicts current code
|
||||
|
||||
### 3. Archive Organization
|
||||
- Verify all archive files have YYYY-MM-DD prefix
|
||||
- Check for any docs that should be archived (completed work, old plans)
|
||||
- Remove truly obsolete content (empty files, duplicate info)
|
||||
|
||||
### 4. Cross-Reference Check
|
||||
- Verify CLAUDE.md points to current doc locations
|
||||
- Check that docs reference each other correctly
|
||||
- Ensure no broken internal links
|
||||
|
||||
### 5. Content Quality
|
||||
- Flag any docs over 500 lines (may need splitting)
|
||||
- Identify duplicate information across docs
|
||||
- Note sections that seem outdated
|
||||
|
||||
## Actions to Take
|
||||
|
||||
1. **Fix structural issues** - Move/rename misplaced files
|
||||
2. **Update stale content** - Refresh outdated information
|
||||
3. **Archive completed work** - Move old docs to archive/
|
||||
4. **Report findings** - Summarize what was found and fixed
|
||||
|
||||
## Output Format
|
||||
|
||||
Provide a maintenance report:
|
||||
```
|
||||
## Documentation Maintenance Report - YYYY-MM-DD
|
||||
|
||||
### Issues Found
|
||||
- [list of issues]
|
||||
|
||||
### Actions Taken
|
||||
- [list of fixes made]
|
||||
|
||||
### Recommendations
|
||||
- [suggestions for future attention]
|
||||
|
||||
### Status
|
||||
- [ ] Structure: OK/Fixed
|
||||
- [ ] Freshness: OK/Updated
|
||||
- [ ] Archive: OK/Organized
|
||||
- [ ] Cross-refs: OK/Fixed
|
||||
- [ ] Quality: OK/Flagged
|
||||
```
|
||||
@@ -1,43 +0,0 @@
|
||||
---
|
||||
description: Review current session for documentation updates before merging PR
|
||||
allowed-tools: Read, Glob, Grep, Edit, Write
|
||||
---
|
||||
|
||||
# Pre-Merge Documentation Review
|
||||
|
||||
You are reviewing this chat session to identify documentation updates needed before merging the current PR.
|
||||
|
||||
## Your Task
|
||||
|
||||
1. **Analyze the current session** - What work was done? What was learned? What changed?
|
||||
|
||||
2. **Check if docs need updates:**
|
||||
- Does `docs/README.md` status section need updating?
|
||||
- Should anything be added to `docs/LESSONS-LEARNED.md`?
|
||||
- Does `docs/ARCHITECTURE.md` need changes (if architecture changed)?
|
||||
- Does `docs/DEVELOPMENT.md` need changes (if dev workflow changed)?
|
||||
- Does `docs/API.md` need changes (if API changed)?
|
||||
|
||||
3. **For significant completed work:**
|
||||
- Create `docs/archive/YYYY-MM-DD-topic.md` summarizing the work
|
||||
- Use today's date and a descriptive topic name
|
||||
|
||||
4. **Update docs/README.md** with:
|
||||
- Current status if it changed
|
||||
- Any new known issues discovered
|
||||
- Link to new archive doc if created
|
||||
|
||||
## Guidelines
|
||||
|
||||
- Keep updates concise and factual
|
||||
- Focus on information useful for future AI agents
|
||||
- Don't duplicate information that's in the code
|
||||
- Archive docs should capture "what was done and why" not implementation details
|
||||
|
||||
## Output
|
||||
|
||||
After reviewing, either:
|
||||
- Make the necessary documentation updates
|
||||
- Report "No documentation updates needed" with brief explanation why
|
||||
|
||||
Remember: Good documentation helps the next agent continue where you left off.
|
||||
@@ -0,0 +1,21 @@
|
||||
---
|
||||
description: Git branching and PR workflow
|
||||
alwaysApply: true
|
||||
---
|
||||
|
||||
# Branching Workflow
|
||||
|
||||
**Never commit directly to `main`.** Always work on a feature branch.
|
||||
|
||||
Create descriptive branch names like `feature/add-discord-notifications`, `fix/audio-sync-issue`, or `refactor/cleanup-api-routes`.
|
||||
|
||||
# Creating a Pull Request
|
||||
|
||||
Once all changes are complete, tests pass, and commits are pushed, create a PR with:
|
||||
|
||||
1. **Summary** - Brief overview of the changes
|
||||
2. **Changes Made** - List of specific modifications
|
||||
3. **Testing** - How changes were tested (unit tests, manual testing)
|
||||
4. **Related Issues** - Reference any GitHub issues (e.g., "Fixes #123")
|
||||
|
||||
Only merge after review and CI checks pass.
|
||||
@@ -0,0 +1,31 @@
|
||||
---
|
||||
description: Infrastructure as Code requirements using Pulumi
|
||||
alwaysApply: true
|
||||
---
|
||||
|
||||
# Infrastructure Changes
|
||||
|
||||
All Google Cloud infrastructure (IAM, Cloud Storage, Cloud Run, Secret Manager, Artifact Registry, etc.) must be managed via **Pulumi IaC** in the `infrastructure/` folder.
|
||||
|
||||
## Rules
|
||||
|
||||
- **Reading/querying** with `gcloud` CLI is fine for investigation and debugging
|
||||
- **Temporary changes** via `gcloud` CLI or Console are acceptable for urgent fixes or experimentation, but **must be cleaned up** and properly codified in the IaC afterward
|
||||
- **Permanent changes** must go through Pulumi — never leave `gcloud` or Console changes as the final state
|
||||
|
||||
## Workflow
|
||||
|
||||
1. Make changes in `infrastructure/`
|
||||
2. Run `pulumi preview` and verify the output
|
||||
3. Run `pulumi up` to apply
|
||||
|
||||
## Authentication Issues
|
||||
|
||||
If you encounter any authentication issues with `gcloud` or `pulumi` (e.g., expired tokens, permission errors), **stop and notify the user** so they can run the appropriate auth commands in their external terminal:
|
||||
|
||||
```bash
|
||||
gcloud auth login
|
||||
gcloud auth application-default login
|
||||
```
|
||||
|
||||
Do not attempt to work around auth issues or make assumptions about credentials.
|
||||
@@ -0,0 +1,27 @@
|
||||
---
|
||||
description: Version bumping rule for commits
|
||||
alwaysApply: true
|
||||
---
|
||||
|
||||
# Version Management
|
||||
|
||||
When committing and pushing changes, always bump the patch version in `pyproject.toml`.
|
||||
|
||||
The version is located at `tool.poetry.version` in `pyproject.toml`. For example, if the current version is `0.70.3`, bump it to `0.70.4`.
|
||||
|
||||
This ensures every commit pushed to the repository has a unique version number for tracking and deployment purposes.
|
||||
|
||||
# Testing Requirements
|
||||
|
||||
Before committing and pushing changes, ensure all tests pass:
|
||||
|
||||
```bash
|
||||
python -m pytest tests/ backend/tests/ -v -x --ignore=tests/integration
|
||||
```
|
||||
|
||||
Ensure all code touched is thoroughly covered by unit tests, we have a 70% coverage minimum in CI:
|
||||
```bash
|
||||
poetry run pytest tests/unit/ -v --cov=karaoke_gen --cov-report=xml --cov-report=term-missing --cov-fail-under=70
|
||||
```
|
||||
|
||||
Do not commit or push code that fails tests. Fix any test failures before proceeding.
|
||||
@@ -1,83 +0,0 @@
|
||||
# .dockerignore - Reduce Docker build context size
|
||||
# This is critical for fast builds as it reduces upload time and cache invalidation
|
||||
|
||||
# Git
|
||||
.git/
|
||||
.gitignore
|
||||
.gitmodules
|
||||
|
||||
# Node.js / Frontend (not needed for backend image)
|
||||
node_modules/
|
||||
frontend/
|
||||
frontend-react/
|
||||
*.log
|
||||
npm-debug.log*
|
||||
yarn-debug.log*
|
||||
yarn-error.log*
|
||||
|
||||
# Python virtual environments (use Docker's Python instead)
|
||||
__pycache__/
|
||||
*.py[cod]
|
||||
*$py.class
|
||||
*.so
|
||||
.Python
|
||||
.venv/
|
||||
venv/
|
||||
backend/venv/
|
||||
ENV/
|
||||
env/
|
||||
*.egg-info/
|
||||
.eggs/
|
||||
dist/
|
||||
build/
|
||||
*.egg
|
||||
|
||||
# IDE
|
||||
.idea/
|
||||
.vscode/
|
||||
*.swp
|
||||
*.swo
|
||||
*~
|
||||
.DS_Store
|
||||
|
||||
# Testing (not needed in production image)
|
||||
.pytest_cache/
|
||||
.coverage
|
||||
htmlcov/
|
||||
.tox/
|
||||
.nox/
|
||||
coverage.xml
|
||||
*.cover
|
||||
tests/
|
||||
backend/tests/
|
||||
|
||||
# Documentation (not needed in Docker image)
|
||||
docs/
|
||||
*.md
|
||||
!README.md
|
||||
|
||||
# Infrastructure (not needed for Docker build)
|
||||
infrastructure/
|
||||
|
||||
# CI/CD configs
|
||||
.github/
|
||||
|
||||
# Cloud Build configs (keep for reference but not in image)
|
||||
cloudbuild*.yaml
|
||||
|
||||
# Local development files
|
||||
lyrics_transcriber_local/
|
||||
*.local
|
||||
.env
|
||||
.env.*
|
||||
*.bak
|
||||
*.tmp
|
||||
Thumbs.db
|
||||
|
||||
# Large test/sample files
|
||||
tests/data/
|
||||
*.mp3
|
||||
*.wav
|
||||
*.mp4
|
||||
*.mkv
|
||||
*.flac
|
||||
+146
-358
@@ -17,120 +17,12 @@ concurrency:
|
||||
|
||||
jobs:
|
||||
# ============================================
|
||||
# PATH FILTERING - Determine what changed
|
||||
# Only needed for PRs (main branch skips tests)
|
||||
# ============================================
|
||||
|
||||
changes:
|
||||
name: "Detect Changes"
|
||||
runs-on: ubuntu-latest
|
||||
if: github.ref != 'refs/heads/main'
|
||||
outputs:
|
||||
frontend: ${{ steps.filter.outputs.frontend }}
|
||||
backend: ${{ steps.filter.outputs.backend }}
|
||||
package: ${{ steps.filter.outputs.package }}
|
||||
infra: ${{ steps.filter.outputs.infra }}
|
||||
deps: ${{ steps.filter.outputs.deps }}
|
||||
steps:
|
||||
- uses: actions/checkout@v4
|
||||
|
||||
- uses: dorny/paths-filter@v3
|
||||
id: filter
|
||||
with:
|
||||
filters: |
|
||||
frontend:
|
||||
- 'frontend/**'
|
||||
backend:
|
||||
- 'backend/**'
|
||||
package:
|
||||
- 'karaoke_gen/**'
|
||||
- 'lyrics_transcriber_temp/**'
|
||||
- 'tests/**'
|
||||
infra:
|
||||
- 'infrastructure/**'
|
||||
deps:
|
||||
- 'poetry.lock'
|
||||
|
||||
- name: Summary
|
||||
run: |
|
||||
echo "### Change Detection Results" >> $GITHUB_STEP_SUMMARY
|
||||
echo "| Area | Changed |" >> $GITHUB_STEP_SUMMARY
|
||||
echo "|------|---------|" >> $GITHUB_STEP_SUMMARY
|
||||
echo "| Frontend | ${{ steps.filter.outputs.frontend }} |" >> $GITHUB_STEP_SUMMARY
|
||||
echo "| Backend | ${{ steps.filter.outputs.backend }} |" >> $GITHUB_STEP_SUMMARY
|
||||
echo "| Package | ${{ steps.filter.outputs.package }} |" >> $GITHUB_STEP_SUMMARY
|
||||
echo "| Infrastructure | ${{ steps.filter.outputs.infra }} |" >> $GITHUB_STEP_SUMMARY
|
||||
echo "| Dependencies | ${{ steps.filter.outputs.deps }} |" >> $GITHUB_STEP_SUMMARY
|
||||
|
||||
# ============================================
|
||||
# FRONTEND TESTS (PRs only - main skips tests)
|
||||
# ============================================
|
||||
|
||||
frontend-unit-tests:
|
||||
name: "Frontend - Unit Tests"
|
||||
runs-on: ubuntu-latest
|
||||
needs: changes
|
||||
if: github.ref != 'refs/heads/main' && needs.changes.outputs.frontend == 'true'
|
||||
steps:
|
||||
- uses: actions/checkout@v4
|
||||
|
||||
- name: Setup Node.js with caching
|
||||
uses: actions/setup-node@v4
|
||||
with:
|
||||
node-version: '20'
|
||||
cache: 'npm'
|
||||
cache-dependency-path: frontend/package-lock.json
|
||||
|
||||
- name: Install dependencies
|
||||
working-directory: frontend
|
||||
run: npm ci --legacy-peer-deps
|
||||
|
||||
- name: Run unit tests with coverage
|
||||
working-directory: frontend
|
||||
run: npm run test:ci
|
||||
|
||||
- name: Upload coverage report
|
||||
if: always()
|
||||
uses: actions/upload-artifact@v4
|
||||
with:
|
||||
name: frontend-coverage
|
||||
path: frontend/coverage/
|
||||
retention-days: 7
|
||||
|
||||
frontend-build:
|
||||
name: "Frontend - Build"
|
||||
runs-on: ubuntu-latest
|
||||
needs: changes
|
||||
if: github.ref != 'refs/heads/main' && needs.changes.outputs.frontend == 'true'
|
||||
steps:
|
||||
- uses: actions/checkout@v4
|
||||
|
||||
- name: Setup Node.js with caching
|
||||
uses: actions/setup-node@v4
|
||||
with:
|
||||
node-version: '20'
|
||||
cache: 'npm'
|
||||
cache-dependency-path: frontend/package-lock.json
|
||||
|
||||
- name: Install dependencies
|
||||
working-directory: frontend
|
||||
run: npm ci --legacy-peer-deps
|
||||
|
||||
- name: Build frontend
|
||||
working-directory: frontend
|
||||
env:
|
||||
NEXT_PUBLIC_API_URL: https://api.nomadkaraoke.com
|
||||
run: npm run build
|
||||
|
||||
# ============================================
|
||||
# PACKAGE TESTS (PRs only - main skips tests)
|
||||
# PACKAGE TESTS (karaoke-gen Python CLI)
|
||||
# ============================================
|
||||
|
||||
package-unit-tests:
|
||||
name: "Package - Unit Tests (70% coverage)"
|
||||
runs-on: ubuntu-latest
|
||||
needs: changes
|
||||
if: github.ref != 'refs/heads/main' && (needs.changes.outputs.package == 'true' || needs.changes.outputs.deps == 'true')
|
||||
steps:
|
||||
- uses: actions/checkout@v4
|
||||
with:
|
||||
@@ -147,14 +39,6 @@ jobs:
|
||||
virtualenvs-create: true
|
||||
virtualenvs-in-project: true
|
||||
|
||||
- name: Cache Poetry dependencies
|
||||
uses: actions/cache@v4
|
||||
with:
|
||||
path: .venv
|
||||
key: venv-package-${{ runner.os }}-py3.13-${{ hashFiles('poetry.lock') }}
|
||||
restore-keys: |
|
||||
venv-package-${{ runner.os }}-py3.13-
|
||||
|
||||
- name: Install dependencies
|
||||
run: poetry install --with dev
|
||||
|
||||
@@ -163,18 +47,12 @@ jobs:
|
||||
sudo apt-get update
|
||||
sudo apt-get install -y ffmpeg
|
||||
|
||||
- name: Cache spaCy model
|
||||
uses: actions/cache@v4
|
||||
with:
|
||||
path: ~/.cache/spacy
|
||||
key: spacy-en_core_web_sm-${{ runner.os }}
|
||||
|
||||
- name: Install spaCy language models
|
||||
run: poetry run python -m spacy download en_core_web_sm
|
||||
|
||||
- name: Run unit tests
|
||||
run: |
|
||||
poetry run pytest tests/unit/ -v --cov=karaoke_gen --cov-report=xml --cov-report=term-missing --cov-fail-under=69
|
||||
poetry run pytest tests/unit/ -v --cov=karaoke_gen --cov-report=xml --cov-report=term-missing --cov-fail-under=70
|
||||
|
||||
- name: Upload coverage reports
|
||||
uses: codecov/codecov-action@v5
|
||||
@@ -187,8 +65,6 @@ jobs:
|
||||
package-integration-tests:
|
||||
name: "Package - Integration Tests"
|
||||
runs-on: ubuntu-latest
|
||||
needs: changes
|
||||
if: github.ref != 'refs/heads/main' && (needs.changes.outputs.package == 'true' || needs.changes.outputs.deps == 'true')
|
||||
steps:
|
||||
- uses: actions/checkout@v4
|
||||
with:
|
||||
@@ -205,14 +81,6 @@ jobs:
|
||||
virtualenvs-create: true
|
||||
virtualenvs-in-project: true
|
||||
|
||||
- name: Cache Poetry dependencies
|
||||
uses: actions/cache@v4
|
||||
with:
|
||||
path: .venv
|
||||
key: venv-package-${{ runner.os }}-py3.13-${{ hashFiles('poetry.lock') }}
|
||||
restore-keys: |
|
||||
venv-package-${{ runner.os }}-py3.13-
|
||||
|
||||
- name: Install dependencies
|
||||
run: poetry install --with dev
|
||||
|
||||
@@ -221,27 +89,16 @@ jobs:
|
||||
sudo apt-get update
|
||||
sudo apt-get install -y ffmpeg
|
||||
|
||||
- name: Cache spaCy model
|
||||
uses: actions/cache@v4
|
||||
with:
|
||||
path: ~/.cache/spacy
|
||||
key: spacy-en_core_web_sm-${{ runner.os }}
|
||||
|
||||
- name: Install spaCy language models
|
||||
run: poetry run python -m spacy download en_core_web_sm
|
||||
|
||||
- name: Run integration tests
|
||||
run: poetry run pytest tests/integration/ -v
|
||||
|
||||
package-build-test:
|
||||
name: "Package - Build & Install (Test)"
|
||||
runs-on: [self-hosted, linux, gcp]
|
||||
needs: [changes, package-unit-tests, package-integration-tests]
|
||||
if: >-
|
||||
github.ref != 'refs/heads/main' && always() &&
|
||||
(needs.package-unit-tests.result == 'success' || needs.package-unit-tests.result == 'skipped') &&
|
||||
(needs.package-integration-tests.result == 'success' || needs.package-integration-tests.result == 'skipped') &&
|
||||
(needs.changes.outputs.package == 'true' || needs.changes.outputs.deps == 'true')
|
||||
package-build:
|
||||
name: "Package - Build & Install"
|
||||
runs-on: ubuntu-latest
|
||||
needs: [package-unit-tests, package-integration-tests]
|
||||
steps:
|
||||
- uses: actions/checkout@v4
|
||||
with:
|
||||
@@ -261,20 +118,36 @@ jobs:
|
||||
- name: Build package
|
||||
run: poetry build
|
||||
|
||||
- name: Free disk space
|
||||
uses: jlumbroso/free-disk-space@main
|
||||
with:
|
||||
tool-cache: false
|
||||
android: true
|
||||
haskell: true
|
||||
dotnet: false
|
||||
large-packages: false
|
||||
docker-images: false
|
||||
swap-storage: false
|
||||
|
||||
- name: Test package installation
|
||||
run: |
|
||||
pip install dist/*.whl
|
||||
python -m karaoke_gen.utils.gen_cli --help
|
||||
|
||||
- name: Upload build artifacts
|
||||
uses: actions/upload-artifact@v4
|
||||
with:
|
||||
name: package-dist
|
||||
path: dist/
|
||||
retention-days: 1
|
||||
|
||||
# ============================================
|
||||
# BACKEND TESTS (PRs only - main skips tests)
|
||||
# BACKEND TESTS (Cloud Run API service)
|
||||
# ============================================
|
||||
|
||||
backend-unit-tests:
|
||||
name: "Backend - Unit Tests"
|
||||
runs-on: [self-hosted, linux, gcp]
|
||||
needs: changes
|
||||
if: github.ref != 'refs/heads/main' && (needs.changes.outputs.backend == 'true' || needs.changes.outputs.package == 'true' || needs.changes.outputs.deps == 'true')
|
||||
runs-on: ubuntu-latest
|
||||
steps:
|
||||
- name: Checkout code
|
||||
uses: actions/checkout@v4
|
||||
@@ -295,7 +168,7 @@ jobs:
|
||||
- name: Install dependencies
|
||||
run: poetry install
|
||||
|
||||
- name: Run all backend unit tests
|
||||
- name: Run unit tests
|
||||
env:
|
||||
GOOGLE_CLOUD_PROJECT: test-project
|
||||
GCS_BUCKET_NAME: test-bucket
|
||||
@@ -303,16 +176,23 @@ jobs:
|
||||
ENVIRONMENT: test
|
||||
run: |
|
||||
poetry run pytest \
|
||||
backend/tests/ \
|
||||
--ignore=backend/tests/emulator \
|
||||
backend/tests/test_models.py \
|
||||
backend/tests/test_job_manager.py \
|
||||
backend/tests/test_file_upload.py \
|
||||
backend/tests/test_services.py \
|
||||
-v --tb=short
|
||||
|
||||
backend-emulator-tests:
|
||||
name: "Backend - Emulator Integration Tests"
|
||||
runs-on: [self-hosted, linux, gcp]
|
||||
needs: changes
|
||||
if: github.ref != 'refs/heads/main' && (needs.changes.outputs.backend == 'true' || needs.changes.outputs.package == 'true' || needs.changes.outputs.deps == 'true')
|
||||
runs-on: ubuntu-latest
|
||||
steps:
|
||||
- name: Free disk space
|
||||
run: |
|
||||
sudo rm -rf /usr/share/dotnet
|
||||
sudo rm -rf /opt/ghc
|
||||
sudo rm -rf /usr/local/share/boost
|
||||
df -h
|
||||
|
||||
- name: Checkout code
|
||||
uses: actions/checkout@v4
|
||||
with:
|
||||
@@ -335,6 +215,14 @@ jobs:
|
||||
distribution: "temurin"
|
||||
java-version: "21"
|
||||
|
||||
- name: Set up Cloud SDK
|
||||
uses: google-github-actions/setup-gcloud@v2
|
||||
with:
|
||||
version: "latest"
|
||||
|
||||
- name: Install Firestore emulator
|
||||
run: gcloud components install beta cloud-firestore-emulator --quiet
|
||||
|
||||
- name: Install dependencies
|
||||
run: poetry install
|
||||
|
||||
@@ -357,8 +245,6 @@ jobs:
|
||||
backend-lint:
|
||||
name: "Backend - Code Quality"
|
||||
runs-on: ubuntu-latest
|
||||
needs: changes
|
||||
if: github.ref != 'refs/heads/main' && (needs.changes.outputs.backend == 'true' || needs.changes.outputs.package == 'true')
|
||||
steps:
|
||||
- name: Checkout code
|
||||
uses: actions/checkout@v4
|
||||
@@ -376,14 +262,6 @@ jobs:
|
||||
virtualenvs-create: true
|
||||
virtualenvs-in-project: true
|
||||
|
||||
- name: Cache Poetry dependencies
|
||||
uses: actions/cache@v4
|
||||
with:
|
||||
path: .venv
|
||||
key: venv-lint-${{ runner.os }}-py3.13-${{ hashFiles('poetry.lock') }}
|
||||
restore-keys: |
|
||||
venv-lint-${{ runner.os }}-py3.13-
|
||||
|
||||
- name: Install dependencies
|
||||
run: poetry install
|
||||
|
||||
@@ -392,79 +270,10 @@ jobs:
|
||||
poetry run python -m py_compile backend/main.py backend/config.py
|
||||
echo "✅ Basic syntax validation passed"
|
||||
|
||||
# ============================================
|
||||
# CI GATE - Required status check for PRs
|
||||
# Only runs on PRs, not on main
|
||||
# ============================================
|
||||
|
||||
ci-gate:
|
||||
name: "CI Gate"
|
||||
runs-on: ubuntu-latest
|
||||
needs: [frontend-unit-tests, frontend-build, package-unit-tests, package-integration-tests, backend-unit-tests, backend-emulator-tests, backend-lint]
|
||||
if: github.ref != 'refs/heads/main' && always()
|
||||
steps:
|
||||
- name: Check results
|
||||
run: |
|
||||
results=(
|
||||
"${{ needs.frontend-unit-tests.result }}"
|
||||
"${{ needs.frontend-build.result }}"
|
||||
"${{ needs.package-unit-tests.result }}"
|
||||
"${{ needs.package-integration-tests.result }}"
|
||||
"${{ needs.backend-unit-tests.result }}"
|
||||
"${{ needs.backend-emulator-tests.result }}"
|
||||
"${{ needs.backend-lint.result }}"
|
||||
)
|
||||
for result in "${results[@]}"; do
|
||||
if [[ "$result" == "failure" || "$result" == "cancelled" ]]; then
|
||||
echo "❌ CI failed"
|
||||
exit 1
|
||||
fi
|
||||
done
|
||||
echo "✅ CI passed"
|
||||
|
||||
# ============================================
|
||||
# DEPLOYMENTS (main branch only)
|
||||
# Tests already passed on PR, so we skip them here
|
||||
# ============================================
|
||||
|
||||
package-build:
|
||||
name: "Package - Build"
|
||||
runs-on: ubuntu-latest
|
||||
if: github.ref == 'refs/heads/main' && github.event_name == 'push'
|
||||
steps:
|
||||
- uses: actions/checkout@v4
|
||||
with:
|
||||
submodules: true
|
||||
|
||||
- name: Set up Python 3.13
|
||||
uses: actions/setup-python@v5
|
||||
with:
|
||||
python-version: "3.13"
|
||||
|
||||
- name: Install Poetry
|
||||
uses: snok/install-poetry@v1
|
||||
with:
|
||||
virtualenvs-create: true
|
||||
virtualenvs-in-project: true
|
||||
|
||||
- name: Cache Poetry dependencies
|
||||
uses: actions/cache@v4
|
||||
with:
|
||||
path: .venv
|
||||
key: venv-package-${{ runner.os }}-py3.13-${{ hashFiles('poetry.lock') }}
|
||||
restore-keys: |
|
||||
venv-package-${{ runner.os }}-py3.13-
|
||||
|
||||
- name: Build package
|
||||
run: poetry build
|
||||
|
||||
- name: Upload build artifacts
|
||||
uses: actions/upload-artifact@v4
|
||||
with:
|
||||
name: package-dist
|
||||
path: dist/
|
||||
retention-days: 1
|
||||
|
||||
package-publish:
|
||||
name: "Deploy - Publish to PyPI"
|
||||
runs-on: ubuntu-latest
|
||||
@@ -556,16 +365,17 @@ jobs:
|
||||
name: "Deploy - Frontend (GitHub Pages)"
|
||||
runs-on: ubuntu-latest
|
||||
if: github.ref == 'refs/heads/main' && github.event_name == 'push'
|
||||
environment:
|
||||
name: github-pages
|
||||
url: ${{ steps.deployment.outputs.page_url }}
|
||||
steps:
|
||||
- name: Checkout
|
||||
uses: actions/checkout@v4
|
||||
|
||||
- name: Setup Node.js with caching
|
||||
- name: Setup Node.js
|
||||
uses: actions/setup-node@v4
|
||||
with:
|
||||
node-version: '20'
|
||||
cache: 'npm'
|
||||
cache-dependency-path: frontend/package-lock.json
|
||||
|
||||
- name: Enable Yarn (for lyrics frontend)
|
||||
run: corepack enable
|
||||
@@ -582,14 +392,6 @@ jobs:
|
||||
run: npm run build
|
||||
|
||||
# ===== Build Lyrics Review Frontend (Vite) =====
|
||||
- name: Cache Yarn dependencies (lyrics frontend)
|
||||
uses: actions/cache@v4
|
||||
with:
|
||||
path: lyrics_transcriber_temp/lyrics_transcriber/frontend/node_modules
|
||||
key: yarn-lyrics-${{ runner.os }}-${{ hashFiles('lyrics_transcriber_temp/lyrics_transcriber/frontend/yarn.lock') }}
|
||||
restore-keys: |
|
||||
yarn-lyrics-${{ runner.os }}-
|
||||
|
||||
- name: Install lyrics frontend dependencies
|
||||
working-directory: lyrics_transcriber_temp/lyrics_transcriber/frontend
|
||||
run: yarn install --frozen-lockfile
|
||||
@@ -626,28 +428,20 @@ jobs:
|
||||
|
||||
deploy-backend:
|
||||
name: "Deploy - Backend (Cloud Run)"
|
||||
runs-on: [self-hosted, linux, gcp]
|
||||
runs-on: ubuntu-latest
|
||||
needs: [backend-unit-tests, backend-emulator-tests, backend-lint]
|
||||
if: github.ref == 'refs/heads/main' && github.event_name == 'push'
|
||||
permissions:
|
||||
contents: read
|
||||
id-token: write
|
||||
env:
|
||||
REGISTRY: us-central1-docker.pkg.dev
|
||||
PROJECT_ID: nomadkaraoke
|
||||
REPOSITORY: karaoke-repo
|
||||
BASE_IMAGE_NAME: karaoke-backend-base
|
||||
APP_IMAGE_NAME: karaoke-backend
|
||||
steps:
|
||||
# ============================================
|
||||
# Check if deployment is needed FIRST
|
||||
# ============================================
|
||||
- name: Checkout code
|
||||
uses: actions/checkout@v4
|
||||
with:
|
||||
submodules: false
|
||||
fetch-depth: 0 # Need full history for tag check
|
||||
|
||||
- name: Set up Python 3.13
|
||||
- name: Set up Python (for version extraction)
|
||||
uses: actions/setup-python@v5
|
||||
with:
|
||||
python-version: "3.13"
|
||||
@@ -661,7 +455,7 @@ jobs:
|
||||
VERSION=$(poetry version --short)
|
||||
echo "version=$VERSION" >> $GITHUB_OUTPUT
|
||||
echo "📋 Current version: $VERSION"
|
||||
|
||||
|
||||
if git ls-remote --tags origin | grep -q "refs/tags/v$VERSION$"; then
|
||||
echo "needs_deploy=false" >> $GITHUB_OUTPUT
|
||||
echo "⏭️ Tag v$VERSION already exists - skipping backend deployment"
|
||||
@@ -671,8 +465,6 @@ jobs:
|
||||
echo "✨ Tag v$VERSION does not exist - proceeding with deployment"
|
||||
fi
|
||||
|
||||
# No disk cleanup needed - self-hosted runner has 200GB SSD
|
||||
|
||||
- name: Authenticate to Google Cloud (Workload Identity)
|
||||
if: steps.version_check.outputs.needs_deploy == 'true'
|
||||
id: auth
|
||||
@@ -687,37 +479,21 @@ jobs:
|
||||
with:
|
||||
version: "latest"
|
||||
|
||||
- name: Configure Docker for Artifact Registry
|
||||
if: steps.version_check.outputs.needs_deploy == 'true'
|
||||
run: gcloud auth configure-docker ${{ env.REGISTRY }} --quiet
|
||||
|
||||
# ============================================
|
||||
# Docker Build with BuildKit + Registry Cache
|
||||
# This replaces expensive Cloud Build with free GitHub Actions
|
||||
# Registry cache persists between builds for fast rebuilds
|
||||
# ============================================
|
||||
|
||||
- name: Set up Docker Buildx
|
||||
if: steps.version_check.outputs.needs_deploy == 'true'
|
||||
uses: docker/setup-buildx-action@v3
|
||||
|
||||
- name: Calculate base image hash
|
||||
if: steps.version_check.outputs.needs_deploy == 'true'
|
||||
id: base-hash
|
||||
run: |
|
||||
# Hash both poetry.lock AND Dockerfile.base - rebuild if either changes
|
||||
CURRENT_HASH=$(cat poetry.lock backend/Dockerfile.base | sha256sum | cut -d' ' -f1)
|
||||
echo "hash=$CURRENT_HASH" >> $GITHUB_OUTPUT
|
||||
echo "📋 Base content hash: $CURRENT_HASH"
|
||||
|
||||
- name: Check if base image exists and is current
|
||||
- name: Check if base image needs rebuild
|
||||
if: steps.version_check.outputs.needs_deploy == 'true'
|
||||
id: check-base
|
||||
run: |
|
||||
BASE_IMAGE="${{ env.REGISTRY }}/${{ env.PROJECT_ID }}/${{ env.REPOSITORY }}/${{ env.BASE_IMAGE_NAME }}:latest"
|
||||
echo "🔍 Checking if base image needs rebuild..."
|
||||
|
||||
# Try to get the existing base image's hash label
|
||||
# Hash both poetry.lock AND Dockerfile.base - rebuild if either changes
|
||||
CURRENT_HASH=$(cat poetry.lock backend/Dockerfile.base | sha256sum | cut -d' ' -f1)
|
||||
echo "Current combined hash (poetry.lock + Dockerfile.base): $CURRENT_HASH"
|
||||
|
||||
gcloud auth configure-docker us-central1-docker.pkg.dev --quiet
|
||||
|
||||
BASE_IMAGE="us-central1-docker.pkg.dev/nomadkaraoke/karaoke-repo/karaoke-backend-base:latest"
|
||||
EXISTING_HASH=""
|
||||
|
||||
if docker pull $BASE_IMAGE 2>/dev/null; then
|
||||
EXISTING_HASH=$(docker inspect $BASE_IMAGE --format='{{index .Config.Labels "base.content.hash"}}' 2>/dev/null || echo "")
|
||||
# Fallback to old label name for backwards compatibility
|
||||
@@ -726,10 +502,10 @@ jobs:
|
||||
fi
|
||||
echo "Existing base image hash: $EXISTING_HASH"
|
||||
else
|
||||
echo "⚠️ No existing base image found - will build"
|
||||
echo "⚠️ No existing base image found"
|
||||
fi
|
||||
|
||||
if [ "${{ steps.base-hash.outputs.hash }}" != "$EXISTING_HASH" ]; then
|
||||
if [ "$CURRENT_HASH" != "$EXISTING_HASH" ]; then
|
||||
echo "🔄 Base image needs rebuild (poetry.lock or Dockerfile.base changed)"
|
||||
echo "needs_rebuild=true" >> $GITHUB_OUTPUT
|
||||
else
|
||||
@@ -737,73 +513,88 @@ jobs:
|
||||
echo "needs_rebuild=false" >> $GITHUB_OUTPUT
|
||||
fi
|
||||
|
||||
- name: Build and push base image (if needed)
|
||||
- name: Rebuild base image (if needed)
|
||||
if: steps.version_check.outputs.needs_deploy == 'true' && steps.check-base.outputs.needs_rebuild == 'true'
|
||||
uses: docker/build-push-action@v6
|
||||
with:
|
||||
context: .
|
||||
file: backend/Dockerfile.base
|
||||
push: true
|
||||
tags: |
|
||||
${{ env.REGISTRY }}/${{ env.PROJECT_ID }}/${{ env.REPOSITORY }}/${{ env.BASE_IMAGE_NAME }}:latest
|
||||
${{ env.REGISTRY }}/${{ env.PROJECT_ID }}/${{ env.REPOSITORY }}/${{ env.BASE_IMAGE_NAME }}:${{ github.sha }}
|
||||
# Registry cache for ALL layers (mode=max) - persists between builds
|
||||
# ignore-error=true allows build to continue if cache doesn't exist yet
|
||||
cache-from: type=registry,ref=${{ env.REGISTRY }}/${{ env.PROJECT_ID }}/${{ env.REPOSITORY }}/${{ env.BASE_IMAGE_NAME }}:cache,ignore-error=true
|
||||
cache-to: type=registry,ref=${{ env.REGISTRY }}/${{ env.PROJECT_ID }}/${{ env.REPOSITORY }}/${{ env.BASE_IMAGE_NAME }}:cache,mode=max,ignore-error=true
|
||||
build-args: |
|
||||
BUILD_DATE=${{ github.run_id }}
|
||||
BUILD_ID=${{ github.sha }}
|
||||
BASE_CONTENT_HASH=${{ steps.base-hash.outputs.hash }}
|
||||
# Provenance disabled for smaller image and faster push
|
||||
provenance: false
|
||||
run: |
|
||||
echo "🏗️ Rebuilding base image with updated dependencies..."
|
||||
|
||||
- name: Build and push app image
|
||||
if: steps.version_check.outputs.needs_deploy == 'true'
|
||||
uses: docker/build-push-action@v6
|
||||
with:
|
||||
context: .
|
||||
file: backend/Dockerfile
|
||||
push: true
|
||||
tags: |
|
||||
${{ env.REGISTRY }}/${{ env.PROJECT_ID }}/${{ env.REPOSITORY }}/${{ env.APP_IMAGE_NAME }}:latest
|
||||
${{ env.REGISTRY }}/${{ env.PROJECT_ID }}/${{ env.REPOSITORY }}/${{ env.APP_IMAGE_NAME }}:${{ github.sha }}
|
||||
${{ env.REGISTRY }}/${{ env.PROJECT_ID }}/${{ env.REPOSITORY }}/${{ env.APP_IMAGE_NAME }}:v${{ steps.version_check.outputs.version }}
|
||||
# Registry cache for app layers
|
||||
# ignore-error=true allows build to continue if cache doesn't exist yet
|
||||
cache-from: type=registry,ref=${{ env.REGISTRY }}/${{ env.PROJECT_ID }}/${{ env.REPOSITORY }}/${{ env.APP_IMAGE_NAME }}:cache,ignore-error=true
|
||||
cache-to: type=registry,ref=${{ env.REGISTRY }}/${{ env.PROJECT_ID }}/${{ env.REPOSITORY }}/${{ env.APP_IMAGE_NAME }}:cache,mode=max,ignore-error=true
|
||||
provenance: false
|
||||
BASE_BUILD_ID=$(gcloud builds submit \
|
||||
--config=cloudbuild-base.yaml \
|
||||
--project=nomadkaraoke \
|
||||
--format='value(id)' \
|
||||
--async \
|
||||
.)
|
||||
|
||||
# ============================================
|
||||
# Deploy to Cloud Run
|
||||
# ============================================
|
||||
echo "Base Build ID: $BASE_BUILD_ID"
|
||||
echo "Build URL: https://console.cloud.google.com/cloud-build/builds/$BASE_BUILD_ID?project=nomadkaraoke"
|
||||
|
||||
- name: Deploy to Cloud Run
|
||||
echo "Waiting for base image build to complete..."
|
||||
while true; do
|
||||
BUILD_STATUS=$(gcloud builds describe $BASE_BUILD_ID --project=nomadkaraoke --format='value(status)')
|
||||
echo "Base build status: $BUILD_STATUS"
|
||||
|
||||
if [ "$BUILD_STATUS" = "SUCCESS" ]; then
|
||||
echo "✅ Base image build completed successfully"
|
||||
break
|
||||
elif [ "$BUILD_STATUS" = "FAILURE" ] || [ "$BUILD_STATUS" = "CANCELLED" ] || [ "$BUILD_STATUS" = "TIMEOUT" ]; then
|
||||
echo "❌ Base image build failed with status: $BUILD_STATUS"
|
||||
echo ""
|
||||
echo "==================== BUILD LOGS ===================="
|
||||
gcloud logging read "resource.type=build AND resource.labels.build_id=$BASE_BUILD_ID" \
|
||||
--project=nomadkaraoke \
|
||||
--format="value(textPayload)" \
|
||||
--limit=200 \
|
||||
--order=asc || echo "Failed to retrieve logs from Cloud Logging"
|
||||
echo "===================================================="
|
||||
gcloud builds log $BASE_BUILD_ID --project=nomadkaraoke 2>&1 || true
|
||||
exit 1
|
||||
fi
|
||||
|
||||
sleep 30
|
||||
done
|
||||
|
||||
- name: Trigger Cloud Build
|
||||
if: steps.version_check.outputs.needs_deploy == 'true'
|
||||
run: |
|
||||
VERSION="${{ steps.version_check.outputs.version }}"
|
||||
IMAGE="${{ env.REGISTRY }}/${{ env.PROJECT_ID }}/${{ env.REPOSITORY }}/${{ env.APP_IMAGE_NAME }}:${{ github.sha }}"
|
||||
echo "🚀 Triggering Cloud Build for deployment..."
|
||||
echo "Commit: ${{ github.sha }}"
|
||||
echo "Version: ${{ steps.version_check.outputs.version }}"
|
||||
|
||||
echo "🚀 Deploying to Cloud Run..."
|
||||
echo "Image: $IMAGE"
|
||||
echo "Version: $VERSION"
|
||||
BUILD_ID=$(gcloud builds submit \
|
||||
--config=cloudbuild.yaml \
|
||||
--project=nomadkaraoke \
|
||||
--format='value(id)' \
|
||||
--async \
|
||||
.)
|
||||
|
||||
gcloud run deploy karaoke-backend \
|
||||
--image $IMAGE \
|
||||
--region us-central1 \
|
||||
--platform managed \
|
||||
--allow-unauthenticated \
|
||||
--memory 16Gi \
|
||||
--cpu 8 \
|
||||
--cpu-boost \
|
||||
--execution-environment gen2 \
|
||||
--timeout 1800 \
|
||||
--concurrency 1 \
|
||||
--max-instances 50 \
|
||||
--min-instances 0 \
|
||||
--set-env-vars "GOOGLE_CLOUD_PROJECT=${{ env.PROJECT_ID }},GCS_BUCKET_NAME=karaoke-gen-storage-${{ env.PROJECT_ID }},FIRESTORE_COLLECTION=jobs,ENVIRONMENT=production,AUDIO_SEPARATOR_API_URL=https://nomadkaraoke--audio-separator-api.modal.run,KARAOKE_GEN_VERSION=$VERSION,DEFAULT_DROPBOX_PATH=/MediaUnsynced/Karaoke/Tracks-Organized,DEFAULT_GDRIVE_FOLDER_ID=1laRKAyxo0v817SstfM5XkpbWiNKNAMSX,ENABLE_CLOUD_TASKS=true,CLOUD_RUN_SERVICE_URL=https://api.nomadkaraoke.com,GCP_REGION=us-central1,DEFAULT_ENABLE_YOUTUBE_UPLOAD=true,DEFAULT_BRAND_PREFIX=NOMAD" \
|
||||
--set-secrets "AUDIOSHAKE_API_TOKEN=audioshake-api-key:latest,GENIUS_API_TOKEN=genius-api-key:latest,SPOTIFY_COOKIE_SP_DC=spotify-cookie:latest,RAPIDAPI_KEY=rapidapi-key:latest,DEFAULT_DISCORD_WEBHOOK_URL=discord-releases-webhook:latest,ADMIN_TOKENS=admin-tokens:latest,RED_API_KEY=red-api-key:latest,RED_API_URL=red-api-url:latest,OPS_API_KEY=ops-api-key:latest,OPS_API_URL=ops-api-url:latest,FLACFETCH_API_URL=flacfetch-api-url:latest,FLACFETCH_API_KEY=flacfetch-api-key:latest"
|
||||
echo "Build ID: $BUILD_ID"
|
||||
echo "Build URL: https://console.cloud.google.com/cloud-build/builds/$BUILD_ID?project=nomadkaraoke"
|
||||
|
||||
echo "Waiting for build to complete..."
|
||||
while true; do
|
||||
BUILD_STATUS=$(gcloud builds describe $BUILD_ID --project=nomadkaraoke --format='value(status)')
|
||||
echo "Build status: $BUILD_STATUS"
|
||||
|
||||
if [ "$BUILD_STATUS" = "SUCCESS" ]; then
|
||||
echo "✅ Build completed successfully"
|
||||
break
|
||||
elif [ "$BUILD_STATUS" = "FAILURE" ] || [ "$BUILD_STATUS" = "CANCELLED" ] || [ "$BUILD_STATUS" = "TIMEOUT" ]; then
|
||||
echo "❌ Build failed with status: $BUILD_STATUS"
|
||||
echo ""
|
||||
echo "==================== BUILD LOGS ===================="
|
||||
gcloud logging read "resource.type=build AND resource.labels.build_id=$BUILD_ID" \
|
||||
--project=nomadkaraoke \
|
||||
--format="value(textPayload)" \
|
||||
--limit=200 \
|
||||
--order=asc || echo "Failed to retrieve logs from Cloud Logging"
|
||||
echo "===================================================="
|
||||
echo ""
|
||||
gcloud builds log $BUILD_ID --project=nomadkaraoke 2>&1 || true
|
||||
exit 1
|
||||
fi
|
||||
|
||||
sleep 30
|
||||
done
|
||||
|
||||
- name: Verify deployment and version
|
||||
if: steps.version_check.outputs.needs_deploy == 'true'
|
||||
@@ -815,22 +606,22 @@ jobs:
|
||||
|
||||
echo "Service URL: $SERVICE_URL"
|
||||
echo "Expected version: $EXPECTED_VERSION"
|
||||
|
||||
|
||||
# Wait for new revision to be ready
|
||||
echo "Waiting for new revision to be ready..."
|
||||
sleep 15
|
||||
|
||||
|
||||
# Basic health check
|
||||
curl -f "$SERVICE_URL/api/health" || exit 1
|
||||
echo "✅ Service is healthy"
|
||||
|
||||
|
||||
# Version verification with retries (Cloud Run may take time to route to new revision)
|
||||
MAX_RETRIES=6
|
||||
RETRY_DELAY=10
|
||||
for i in $(seq 1 $MAX_RETRIES); do
|
||||
echo "Checking deployed version (attempt $i/$MAX_RETRIES)..."
|
||||
DEPLOYED_VERSION=$(curl -sf "$SERVICE_URL/api/health/detailed" | jq -r '.version // empty')
|
||||
|
||||
|
||||
if [ "$DEPLOYED_VERSION" = "$EXPECTED_VERSION" ]; then
|
||||
echo "✅ Version verified: $DEPLOYED_VERSION"
|
||||
break
|
||||
@@ -854,10 +645,7 @@ jobs:
|
||||
echo "Commit: ${{ github.sha }}"
|
||||
echo "Branch: ${{ github.ref_name }}"
|
||||
echo "Run: ${{ github.run_number }}"
|
||||
echo ""
|
||||
echo "📊 Built on self-hosted GCP runner (200GB SSD, no disk space issues)"
|
||||
echo " Runner: gcp-runner with 4 vCPU, 16GB RAM"
|
||||
|
||||
|
||||
- name: Skipped deployment summary
|
||||
if: steps.version_check.outputs.needs_deploy == 'false'
|
||||
run: |
|
||||
|
||||
-10
@@ -12,9 +12,6 @@ tests/outputs
|
||||
/input
|
||||
/output/
|
||||
|
||||
# Cloud Infra
|
||||
.emulator-logs
|
||||
|
||||
# Next.js / React Frontend (./frontend)
|
||||
node_modules/
|
||||
frontend/node_modules/
|
||||
@@ -44,9 +41,6 @@ frontend/.pnpm-debug.log*
|
||||
|
||||
# Testing
|
||||
frontend/coverage/
|
||||
frontend/test-results/
|
||||
frontend/playwright-report/
|
||||
frontend/playwright/.cache/
|
||||
|
||||
# TypeScript
|
||||
frontend/*.tsbuildinfo
|
||||
@@ -226,7 +220,3 @@ cython_debug/
|
||||
# and can be added to the global gitignore or merged into this file. For a more nuclear
|
||||
# option (not recommended) you can uncomment the following to ignore the entire idea folder.
|
||||
#.idea/
|
||||
flacfetch
|
||||
|
||||
# Test output directories
|
||||
Rick Astley*/
|
||||
@@ -1,97 +0,0 @@
|
||||
# Karaoke-Gen AI Assistant Guidelines
|
||||
|
||||
## Project Overview
|
||||
|
||||
**Karaoke video generation platform** - CLI tool and web service that creates professional karaoke videos with synchronized lyrics.
|
||||
|
||||
- **Production**: <https://gen.nomadkaraoke.com> (frontend), <https://api.nomadkaraoke.com> (backend)
|
||||
- **CLI**: `karaoke-gen` (local), `karaoke-gen-remote` (cloud)
|
||||
- **Repo**: <https://github.com/nomadkaraoke/karaoke-gen>
|
||||
|
||||
## Quick Reference
|
||||
|
||||
| What | Where |
|
||||
|------|-------|
|
||||
| Current status | `docs/README.md` |
|
||||
| Architecture | `docs/ARCHITECTURE.md` |
|
||||
| Dev setup & testing | `docs/DEVELOPMENT.md` |
|
||||
| API reference | `docs/API.md` |
|
||||
| Past learnings | `docs/LESSONS-LEARNED.md` |
|
||||
|
||||
## Tech Stack
|
||||
|
||||
- **Backend**: FastAPI on Cloud Run, Firestore, GCS, Secret Manager
|
||||
- **Frontend**: Next.js on Cloudflare Pages
|
||||
- **Processing**: Modal (GPU audio separation), AudioShake/Whisper (transcription)
|
||||
- **Infra**: Pulumi IaC, GitHub Actions CI/CD
|
||||
|
||||
## Essential Rules
|
||||
|
||||
### Git Workflow
|
||||
- **Never commit directly to `main`** - always use feature branches
|
||||
- **Use git worktrees** for all changes:
|
||||
```bash
|
||||
git worktree add -b feature/xyz ../karaoke-gen-feat-xyz main
|
||||
```
|
||||
- Create PR with summary, changes, testing info
|
||||
- Merge only after CI passes
|
||||
|
||||
### Testing (Required Before Commit)
|
||||
```bash
|
||||
# Backend (10 min timeout, limit output)
|
||||
make test 2>&1 | tail -n 500
|
||||
|
||||
# Frontend
|
||||
cd frontend && npm run test:all 2>&1 | tail -n 200
|
||||
```
|
||||
|
||||
### Version Bumping
|
||||
- Bump `tool.poetry.version` in `pyproject.toml` for code changes
|
||||
- Skip for docs-only changes
|
||||
|
||||
### Infrastructure
|
||||
- All GCP changes via Pulumi in `infrastructure/`
|
||||
- `gcloud` CLI for reading/debugging only
|
||||
- Stop and notify user on auth issues
|
||||
|
||||
## Documentation Maintenance
|
||||
|
||||
### Structure
|
||||
```text
|
||||
docs/
|
||||
├── README.md # Current status + navigation (UPDATE THIS)
|
||||
├── ARCHITECTURE.md # System design
|
||||
├── DEVELOPMENT.md # Dev setup, testing, deployment
|
||||
├── API.md # Backend API reference
|
||||
├── LESSONS-LEARNED.md # Key insights for future agents
|
||||
└── archive/ # Historical docs (YYYY-MM-DD-topic.md)
|
||||
```
|
||||
|
||||
### Rules
|
||||
1. **Before merging PRs**: Run `/docs-review` to check if docs need updating
|
||||
2. **Periodically**: Run `/docs-maintain` to verify docs are organized
|
||||
3. **For significant work**: Create `docs/archive/YYYY-MM-DD-topic.md`
|
||||
4. **Add learnings**: Update `docs/LESSONS-LEARNED.md` with insights
|
||||
5. **Keep status current**: Update `docs/README.md` when project state changes
|
||||
|
||||
### What Goes Where
|
||||
- **README.md**: Current status, known issues, quick links
|
||||
- **ARCHITECTURE.md**: System design, data flow, tech decisions
|
||||
- **DEVELOPMENT.md**: Setup, testing, deployment, CI/CD
|
||||
- **API.md**: Endpoints, request/response formats
|
||||
- **LESSONS-LEARNED.md**: What worked, what didn't, gotchas
|
||||
- **archive/**: Completed work, session summaries, old plans
|
||||
|
||||
## Related Projects
|
||||
|
||||
### flacfetch
|
||||
Location: `/Users/andrew/Projects/flacfetch`
|
||||
|
||||
Workflow: Make changes on `main`, bump version, push, wait for CI, then `poetry update flacfetch` in karaoke-gen.
|
||||
|
||||
## Pre-Commit Checklist
|
||||
|
||||
- [ ] Tests pass (`make test` / `npm run test:all`)
|
||||
- [ ] Version bumped (if code changed)
|
||||
- [ ] `/docs-review` run (check for doc updates)
|
||||
- [ ] PR created with clear description
|
||||
@@ -1,21 +0,0 @@
|
||||
MIT License
|
||||
|
||||
Copyright (c) 2024 Nomad Karaoke LLC
|
||||
|
||||
Permission is hereby granted, free of charge, to any person obtaining a copy
|
||||
of this software and associated documentation files (the "Software"), to deal
|
||||
in the Software without restriction, including without limitation the rights
|
||||
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
|
||||
copies of the Software, and to permit persons to whom the Software is
|
||||
furnished to do so, subject to the following conditions:
|
||||
|
||||
The above copyright notice and this permission notice shall be included in all
|
||||
copies or substantial portions of the Software.
|
||||
|
||||
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
|
||||
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
|
||||
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
|
||||
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
|
||||
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
|
||||
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
|
||||
SOFTWARE.
|
||||
@@ -1,77 +0,0 @@
|
||||
# Makefile for karaoke-gen project
|
||||
# Run `make help` to see available commands
|
||||
|
||||
.PHONY: help test test-unit test-backend test-integration test-e2e test-all lint clean emulators-start emulators-stop
|
||||
|
||||
# Default target
|
||||
help:
|
||||
@echo "Available commands:"
|
||||
@echo " make test - Run all tests (unit + backend + emulator-based)"
|
||||
@echo " make test-unit - Run unit tests only (karaoke_gen)"
|
||||
@echo " make test-backend - Run backend unit tests only"
|
||||
@echo " make test-e2e - Run E2E tests with emulators (starts/stops automatically)"
|
||||
@echo " make test-all - Alias for 'make test'"
|
||||
@echo " make lint - Run linter checks"
|
||||
@echo " make emulators-start - Start GCP emulators for local development"
|
||||
@echo " make emulators-stop - Stop GCP emulators"
|
||||
@echo ""
|
||||
@echo "Before committing, run: make test"
|
||||
|
||||
# Run unit tests for karaoke_gen package
|
||||
test-unit:
|
||||
@echo "=== Running karaoke_gen unit tests ==="
|
||||
poetry run pytest tests/unit/ -v --cov=karaoke_gen --cov-report=term-missing --cov-fail-under=69
|
||||
|
||||
# Run backend unit tests
|
||||
test-backend:
|
||||
@echo "=== Running backend unit tests ==="
|
||||
poetry run pytest backend/tests/ --ignore=backend/tests/emulator -v
|
||||
|
||||
# Run E2E integration tests with emulators
|
||||
test-e2e:
|
||||
@echo "=== Running E2E integration tests with emulators ==="
|
||||
@./scripts/start-emulators.sh || (echo "Failed to start emulators" && exit 1)
|
||||
@export FIRESTORE_EMULATOR_HOST=127.0.0.1:8080 && \
|
||||
export STORAGE_EMULATOR_HOST=http://127.0.0.1:4443 && \
|
||||
export GOOGLE_CLOUD_PROJECT=test-project && \
|
||||
export GCS_BUCKET_NAME=test-bucket && \
|
||||
export ADMIN_TOKENS=test-admin-token && \
|
||||
export ENVIRONMENT=test && \
|
||||
poetry run pytest backend/tests/emulator/ -v; \
|
||||
TEST_RESULT=$$?; \
|
||||
./scripts/stop-emulators.sh; \
|
||||
exit $$TEST_RESULT
|
||||
|
||||
# Run all tests (use this before committing!)
|
||||
test: test-unit test-backend test-e2e
|
||||
@echo ""
|
||||
@echo "✅ All tests passed! Ready to commit."
|
||||
|
||||
# Alias for test
|
||||
test-all: test
|
||||
|
||||
# Lint checks
|
||||
lint:
|
||||
@echo "=== Running lint checks ==="
|
||||
poetry run ruff check karaoke_gen/ backend/
|
||||
poetry run ruff format --check karaoke_gen/ backend/
|
||||
|
||||
# Start emulators for local development
|
||||
emulators-start:
|
||||
@echo "=== Starting GCP emulators ==="
|
||||
./scripts/start-emulators.sh
|
||||
|
||||
# Stop emulators
|
||||
emulators-stop:
|
||||
@echo "=== Stopping GCP emulators ==="
|
||||
./scripts/stop-emulators.sh
|
||||
|
||||
# Clean up temporary files
|
||||
clean:
|
||||
find . -type d -name "__pycache__" -exec rm -rf {} + 2>/dev/null || true
|
||||
find . -type d -name ".pytest_cache" -exec rm -rf {} + 2>/dev/null || true
|
||||
find . -type d -name ".coverage" -exec rm -rf {} + 2>/dev/null || true
|
||||
find . -type f -name "*.pyc" -delete 2>/dev/null || true
|
||||
rm -rf htmlcov/ .coverage coverage.xml 2>/dev/null || true
|
||||
@echo "Cleaned up temporary files"
|
||||
|
||||
@@ -1,693 +0,0 @@
|
||||
# Karaoke Generator 🎶 🎥 🚀
|
||||
|
||||

|
||||

|
||||

|
||||

|
||||
|
||||
Generate professional karaoke videos with instrumental audio and synchronized lyrics. Available as a **local CLI** (`karaoke-gen`) or **cloud-based CLI** (`karaoke-gen-remote`) that offloads processing to Google Cloud.
|
||||
|
||||
## ✨ Two Ways to Generate Karaoke
|
||||
|
||||
### 1. Local CLI (`karaoke-gen`)
|
||||
Run all processing locally on your machine. Requires GPU for optimal audio separation performance.
|
||||
|
||||
```bash
|
||||
karaoke-gen "ABBA" "Waterloo"
|
||||
```
|
||||
|
||||
### 2. Remote CLI (`karaoke-gen-remote`)
|
||||
Offload all processing to a cloud backend. No GPU required - just authenticate and submit jobs.
|
||||
|
||||
```bash
|
||||
karaoke-gen-remote ./song.flac "ABBA" "Waterloo"
|
||||
```
|
||||
|
||||
Both CLIs produce identical outputs: 4K karaoke videos, CDG+MP3 packages, audio stems, and more.
|
||||
|
||||
---
|
||||
|
||||
## 🎯 Features
|
||||
|
||||
### Core Pipeline
|
||||
- **Audio Separation**: AI-powered vocal/instrumental separation using MDX and Demucs models
|
||||
- **Lyrics Transcription**: Word-level timestamps via AudioShake API
|
||||
- **Lyrics Correction**: Match transcription against online lyrics (Genius, Spotify, Musixmatch)
|
||||
- **Human Review**: Interactive UI for correcting lyrics before final render
|
||||
- **Video Rendering**: High-quality 4K karaoke videos with customizable styles
|
||||
- **Multiple Outputs**: MP4 (4K lossless/lossy, 720p), MKV, CDG+MP3, TXT+MP3
|
||||
|
||||
### Distribution Features
|
||||
- **YouTube Upload**: Automatic upload to your YouTube channel
|
||||
- **Dropbox Integration**: Organize output in brand-coded folders
|
||||
- **Google Drive**: Upload to public share folders
|
||||
- **Discord Notifications**: Webhook notifications on completion
|
||||
|
||||
---
|
||||
|
||||
## 📦 Installation
|
||||
|
||||
```bash
|
||||
pip install karaoke-gen
|
||||
```
|
||||
|
||||
This installs both `karaoke-gen` (local) and `karaoke-gen-remote` (cloud) CLIs.
|
||||
|
||||
### Requirements
|
||||
- Python 3.10-3.13
|
||||
- FFmpeg
|
||||
- For local processing: CUDA-capable GPU or Apple Silicon CPU recommended
|
||||
|
||||
### Transcription Provider Setup
|
||||
|
||||
**Transcription is required** for creating karaoke videos with synchronized lyrics. The system needs word-level timing data to display lyrics in sync with the music.
|
||||
|
||||
#### Option 1: AudioShake (Recommended)
|
||||
Commercial service with high-quality transcription. Best for production use.
|
||||
|
||||
```bash
|
||||
export AUDIOSHAKE_API_TOKEN="your_audioshake_token"
|
||||
```
|
||||
|
||||
Get an API key at [https://www.audioshake.ai/](https://www.audioshake.ai/) - business only, at time of writing this.
|
||||
|
||||
#### Option 2: Local Whisper (No Cloud Required)
|
||||
Run Whisper directly on your local machine using whisper-timestamped. Works on CPU, NVIDIA GPU (CUDA), or Apple Silicon.
|
||||
|
||||
```bash
|
||||
# Install with local Whisper support
|
||||
pip install "karaoke-gen[local-whisper]"
|
||||
|
||||
# Optional: Configure model size (tiny, base, small, medium, large)
|
||||
export WHISPER_MODEL_SIZE="medium"
|
||||
|
||||
# Optional: Force specific device (cpu, cuda, mps)
|
||||
export WHISPER_DEVICE="cpu"
|
||||
```
|
||||
|
||||
**Model Size Guide:**
|
||||
| Model | VRAM | Speed | Quality |
|
||||
|-------|------|-------|---------|
|
||||
| tiny | ~1GB | Fast | Lower |
|
||||
| base | ~1GB | Fast | Basic |
|
||||
| small | ~2GB | Medium | Good |
|
||||
| medium | ~5GB | Slower | Better |
|
||||
| large | ~10GB | Slowest | Best |
|
||||
|
||||
**CPU-Only Installation** (no GPU required):
|
||||
```bash
|
||||
# Pre-install CPU-only PyTorch first
|
||||
pip install torch torchaudio --index-url https://download.pytorch.org/whl/cpu
|
||||
pip install "karaoke-gen[local-whisper]"
|
||||
```
|
||||
|
||||
Local Whisper runs automatically as a fallback when no cloud transcription services are configured.
|
||||
|
||||
#### Option 3: Whisper via RunPod
|
||||
Cloud-based alternative using OpenAI's Whisper model on RunPod infrastructure.
|
||||
|
||||
```bash
|
||||
export RUNPOD_API_KEY="your_runpod_key"
|
||||
export WHISPER_RUNPOD_ID="your_whisper_endpoint_id"
|
||||
```
|
||||
|
||||
Set up a Whisper endpoint at [https://www.runpod.io/](https://www.runpod.io/)
|
||||
|
||||
#### Without Transcription (Instrumental Only)
|
||||
If you don't need synchronized lyrics, use the `--skip-lyrics` flag:
|
||||
|
||||
```bash
|
||||
karaoke-gen --skip-lyrics "Artist" "Title"
|
||||
```
|
||||
|
||||
This creates an instrumental-only karaoke video without lyrics overlay.
|
||||
|
||||
> **Note:** See `lyrics_transcriber_temp/README.md` for detailed transcription provider configuration options.
|
||||
|
||||
---
|
||||
|
||||
## 🖥️ Local CLI (`karaoke-gen`)
|
||||
|
||||
### Basic Usage
|
||||
|
||||
```bash
|
||||
# Generate from local audio file
|
||||
karaoke-gen ./song.mp3 "Artist Name" "Song Title"
|
||||
|
||||
# Search and download audio automatically
|
||||
karaoke-gen "Rick Astley" "Never Gonna Give You Up"
|
||||
|
||||
# Process from YouTube URL
|
||||
karaoke-gen "https://www.youtube.com/watch?v=dQw4w9WgXcQ" "Rick Astley" "Never Gonna Give You Up"
|
||||
```
|
||||
|
||||
### Remote Audio Separation (Optional)
|
||||
|
||||
Offload just the GPU-intensive audio separation to Modal.com while keeping other processing local:
|
||||
|
||||
```bash
|
||||
export AUDIO_SEPARATOR_API_URL="https://USERNAME--audio-separator-api.modal.run"
|
||||
karaoke-gen "Artist" "Title"
|
||||
```
|
||||
|
||||
### Key Options
|
||||
|
||||
```bash
|
||||
# Custom styling
|
||||
karaoke-gen --style_params_json="./styles.json" "Artist" "Title"
|
||||
|
||||
# Generate CDG and TXT packages
|
||||
karaoke-gen --enable_cdg --enable_txt "Artist" "Title"
|
||||
|
||||
# YouTube upload
|
||||
karaoke-gen --enable_youtube_upload --youtube_description_file="./desc.txt" "Artist" "Title"
|
||||
|
||||
# Full production run
|
||||
karaoke-gen \
|
||||
--style_params_json="./branding.json" \
|
||||
--enable_cdg \
|
||||
--enable_txt \
|
||||
--brand_prefix="BRAND" \
|
||||
--enable_youtube_upload \
|
||||
--youtube_description_file="./description.txt" \
|
||||
"Artist" "Title"
|
||||
```
|
||||
|
||||
### Full Options Reference
|
||||
|
||||
```bash
|
||||
karaoke-gen --help
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## ☁️ Remote CLI (`karaoke-gen-remote`)
|
||||
|
||||
The remote CLI submits jobs to a Google Cloud backend that handles all processing. You don't need a GPU or any audio processing libraries installed locally.
|
||||
|
||||
### Setup
|
||||
|
||||
1. **Set the backend URL:**
|
||||
```bash
|
||||
export KARAOKE_GEN_URL="https://api.nomadkaraoke.com" # Or your own backend
|
||||
```
|
||||
|
||||
2. **Authenticate with Google Cloud:**
|
||||
```bash
|
||||
gcloud auth login
|
||||
```
|
||||
|
||||
### Basic Usage
|
||||
|
||||
```bash
|
||||
# Submit a job
|
||||
karaoke-gen-remote ./song.flac "ABBA" "Waterloo"
|
||||
|
||||
# The CLI will:
|
||||
# 1. Upload your audio file
|
||||
# 2. Monitor processing progress
|
||||
# 3. Open lyrics review UI when ready
|
||||
# 4. Prompt for instrumental selection
|
||||
# 5. Download all outputs when complete
|
||||
```
|
||||
|
||||
### Job Management
|
||||
|
||||
```bash
|
||||
# List all jobs
|
||||
karaoke-gen-remote --list
|
||||
|
||||
# Resume monitoring an existing job
|
||||
karaoke-gen-remote --resume abc12345
|
||||
|
||||
# Cancel a running job
|
||||
karaoke-gen-remote --cancel abc12345
|
||||
|
||||
# Delete a job and its files
|
||||
karaoke-gen-remote --delete abc12345
|
||||
```
|
||||
|
||||
### Full Production Run
|
||||
|
||||
```bash
|
||||
karaoke-gen-remote \
|
||||
--style_params_json="./karaoke-styles.json" \
|
||||
--enable_cdg \
|
||||
--enable_txt \
|
||||
--brand_prefix=NOMAD \
|
||||
--enable_youtube_upload \
|
||||
--youtube_description_file="./youtube-description.txt" \
|
||||
./song.flac "Artist" "Title"
|
||||
```
|
||||
|
||||
### Environment Variables
|
||||
|
||||
| Variable | Description | Default |
|
||||
|----------|-------------|---------|
|
||||
| `KARAOKE_GEN_URL` | Backend service URL | Required |
|
||||
| `KARAOKE_GEN_AUTH_TOKEN` | Admin auth token (for protected endpoints) | Optional |
|
||||
| `REVIEW_UI_URL` | Lyrics review UI URL | `https://gen.nomadkaraoke.com/lyrics/` |
|
||||
| `POLL_INTERVAL` | Seconds between status polls | `5` |
|
||||
|
||||
**Note:** The `REVIEW_UI_URL` defaults to the hosted lyrics review UI. For local development, set it to `http://localhost:5173` if you're running the frontend dev server.
|
||||
|
||||
### Authentication
|
||||
|
||||
The backend uses token-based authentication for admin operations (bulk delete, internal worker triggers). For basic job submission and monitoring, authentication is optional.
|
||||
|
||||
**For admin access:**
|
||||
```bash
|
||||
export KARAOKE_GEN_AUTH_TOKEN="your-admin-token"
|
||||
```
|
||||
|
||||
The token must match one of the tokens configured in the backend's `ADMIN_TOKENS` environment variable.
|
||||
|
||||
### Non-Interactive Mode
|
||||
|
||||
For automated/CI usage:
|
||||
|
||||
```bash
|
||||
karaoke-gen-remote -y ./song.flac "Artist" "Title"
|
||||
```
|
||||
|
||||
The `-y` flag auto-accepts default corrections and selects clean instrumental.
|
||||
|
||||
---
|
||||
|
||||
## 🎨 Style Configuration
|
||||
|
||||
Create a `styles.json` file to customize the karaoke video appearance:
|
||||
|
||||
```json
|
||||
{
|
||||
"intro": {
|
||||
"video_duration": 5,
|
||||
"background_image": "/path/to/title-background.png",
|
||||
"font": "/path/to/Font.ttf",
|
||||
"artist_color": "#ffdf6b",
|
||||
"title_color": "#ffffff"
|
||||
},
|
||||
"karaoke": {
|
||||
"background_image": "/path/to/karaoke-background.png",
|
||||
"font_path": "/path/to/Font.ttf"
|
||||
},
|
||||
"end": {
|
||||
"background_image": "/path/to/end-background.png"
|
||||
},
|
||||
"cdg": {
|
||||
"font_path": "/path/to/Font.ttf",
|
||||
"instrumental_background": "/path/to/cdg-background.png"
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
When using `karaoke-gen-remote`, all referenced files are automatically uploaded with your job.
|
||||
|
||||
---
|
||||
|
||||
## 📤 Output Files
|
||||
|
||||
A completed job produces:
|
||||
|
||||
```
|
||||
BRAND-1234 - Artist - Title/
|
||||
├── Artist - Title (Final Karaoke Lossless 4k).mp4 # ProRes 4K
|
||||
├── Artist - Title (Final Karaoke Lossless 4k).mkv # FLAC audio 4K
|
||||
├── Artist - Title (Final Karaoke Lossy 4k).mp4 # H.264 4K
|
||||
├── Artist - Title (Final Karaoke Lossy 720p).mp4 # H.264 720p
|
||||
├── Artist - Title (Final Karaoke CDG).zip # CDG+MP3 package
|
||||
├── Artist - Title (Final Karaoke TXT).zip # TXT+MP3 package
|
||||
├── Artist - Title (Karaoke).cdg # Individual CDG
|
||||
├── Artist - Title (Karaoke).mp3 # Karaoke audio
|
||||
├── Artist - Title (Karaoke).lrc # LRC lyrics
|
||||
├── Artist - Title (Karaoke).ass # ASS subtitles
|
||||
├── Artist - Title (Title).mov # Title screen video
|
||||
├── Artist - Title (End).mov # End screen video
|
||||
├── Artist - Title (Instrumental...).flac # Clean instrumental
|
||||
├── Artist - Title (Instrumental +BV...).flac # With backing vocals
|
||||
└── stems/ # All audio stems
|
||||
├── ...Vocals....flac
|
||||
├── ...Bass....flac
|
||||
├── ...Drums....flac
|
||||
└── ...
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 🏗️ Deploy Your Own Backend
|
||||
|
||||
The cloud backend runs on Google Cloud Platform using:
|
||||
- **Cloud Run**: Serverless API hosting
|
||||
- **Firestore**: Job state management
|
||||
- **Cloud Storage**: File uploads and outputs
|
||||
- **Modal.com**: GPU-accelerated audio separation
|
||||
- **AudioShake**: Lyrics transcription API
|
||||
|
||||
### Prerequisites
|
||||
|
||||
- Google Cloud account with billing enabled
|
||||
- [Pulumi CLI](https://www.pulumi.com/docs/install/)
|
||||
- Modal.com account (for audio separation)
|
||||
- AudioShake API key
|
||||
|
||||
### Infrastructure Setup
|
||||
|
||||
```bash
|
||||
cd infrastructure
|
||||
|
||||
# Install dependencies
|
||||
pip install -r requirements.txt
|
||||
|
||||
# Login to Pulumi
|
||||
pulumi login
|
||||
|
||||
# Create a stack
|
||||
pulumi stack init prod
|
||||
|
||||
# Configure GCP project
|
||||
pulumi config set gcp:project your-project-id
|
||||
pulumi config set gcp:region us-central1
|
||||
|
||||
# Deploy infrastructure
|
||||
pulumi up
|
||||
```
|
||||
|
||||
This creates:
|
||||
- Firestore database
|
||||
- Cloud Storage bucket
|
||||
- Artifact Registry
|
||||
- Service account with IAM roles
|
||||
- Secret Manager secrets (you add values)
|
||||
|
||||
### Add Secret Values
|
||||
|
||||
```bash
|
||||
# AudioShake API key
|
||||
echo -n "your-audioshake-key" | gcloud secrets versions add audioshake-api-key --data-file=-
|
||||
|
||||
# Genius API key
|
||||
echo -n "your-genius-key" | gcloud secrets versions add genius-api-key --data-file=-
|
||||
|
||||
# Modal API URL
|
||||
echo -n "https://your-modal-url" | gcloud secrets versions add audio-separator-api-url --data-file=-
|
||||
|
||||
# YouTube OAuth credentials (JSON)
|
||||
gcloud secrets versions add youtube-oauth-credentials --data-file=./youtube-creds.json
|
||||
|
||||
# Dropbox OAuth credentials (JSON)
|
||||
gcloud secrets versions add dropbox-oauth-credentials --data-file=./dropbox-creds.json
|
||||
|
||||
# Google Drive service account (JSON)
|
||||
gcloud secrets versions add gdrive-service-account --data-file=./gdrive-sa.json
|
||||
```
|
||||
|
||||
### Deploy Cloud Run
|
||||
|
||||
```bash
|
||||
# Build and deploy
|
||||
gcloud builds submit --config=cloudbuild.yaml
|
||||
|
||||
# Get outputs from Pulumi
|
||||
SA_EMAIL=$(pulumi stack output service_account_email)
|
||||
BUCKET_NAME=$(pulumi stack output bucket_name)
|
||||
|
||||
# Deploy Cloud Run service
|
||||
gcloud run deploy karaoke-backend \
|
||||
--image us-central1-docker.pkg.dev/YOUR-PROJECT/karaoke-repo/karaoke-backend:latest \
|
||||
--platform managed \
|
||||
--region us-central1 \
|
||||
--allow-unauthenticated \
|
||||
--service-account $SA_EMAIL \
|
||||
--memory 2Gi \
|
||||
--cpu 2 \
|
||||
--timeout 600 \
|
||||
--set-env-vars="GOOGLE_CLOUD_PROJECT=YOUR-PROJECT,GCS_BUCKET_NAME=$BUCKET_NAME"
|
||||
```
|
||||
|
||||
### Point CLI to Your Backend
|
||||
|
||||
```bash
|
||||
export KARAOKE_GEN_URL="https://your-backend.run.app"
|
||||
karaoke-gen-remote ./song.flac "Artist" "Title"
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 🔌 Backend API Reference
|
||||
|
||||
The backend exposes a REST API for job management.
|
||||
|
||||
### Job Submission
|
||||
|
||||
**POST** `/api/jobs/upload`
|
||||
|
||||
Submit a new karaoke generation job with audio file and options.
|
||||
|
||||
```bash
|
||||
curl -X POST "https://api.example.com/api/jobs/upload" \
|
||||
-F "file=@song.flac" \
|
||||
-F "artist=ABBA" \
|
||||
-F "title=Waterloo" \
|
||||
-F "enable_cdg=true" \
|
||||
-F "enable_txt=true" \
|
||||
-F "brand_prefix=NOMAD" \
|
||||
-F "style_params=@styles.json" \
|
||||
-F "style_karaoke_background=@background.png"
|
||||
```
|
||||
|
||||
### Job Status
|
||||
|
||||
**GET** `/api/jobs/{job_id}`
|
||||
|
||||
Get job status and details.
|
||||
|
||||
```bash
|
||||
curl "https://api.example.com/api/jobs/abc12345"
|
||||
```
|
||||
|
||||
### List Jobs
|
||||
|
||||
**GET** `/api/jobs`
|
||||
|
||||
List all jobs with optional status filter.
|
||||
|
||||
```bash
|
||||
curl "https://api.example.com/api/jobs?status=complete&limit=10"
|
||||
```
|
||||
|
||||
### Cancel Job
|
||||
|
||||
**POST** `/api/jobs/{job_id}/cancel`
|
||||
|
||||
Cancel a running job.
|
||||
|
||||
```bash
|
||||
curl -X POST "https://api.example.com/api/jobs/abc12345/cancel" \
|
||||
-H "Content-Type: application/json" \
|
||||
-d '{"reason": "User cancelled"}'
|
||||
```
|
||||
|
||||
### Delete Job
|
||||
|
||||
**DELETE** `/api/jobs/{job_id}`
|
||||
|
||||
Delete a job and its files.
|
||||
|
||||
```bash
|
||||
curl -X DELETE "https://api.example.com/api/jobs/abc12345?delete_files=true"
|
||||
```
|
||||
|
||||
### Lyrics Review
|
||||
|
||||
**GET** `/api/review/{job_id}/correction-data`
|
||||
|
||||
Get correction data for lyrics review.
|
||||
|
||||
**POST** `/api/review/{job_id}/complete`
|
||||
|
||||
Submit corrected lyrics and trigger video rendering.
|
||||
|
||||
### Instrumental Selection
|
||||
|
||||
**GET** `/api/jobs/{job_id}/instrumental-options`
|
||||
|
||||
Get available instrumental options.
|
||||
|
||||
**POST** `/api/jobs/{job_id}/select-instrumental`
|
||||
|
||||
Submit instrumental selection (clean or with_backing).
|
||||
|
||||
```bash
|
||||
curl -X POST "https://api.example.com/api/jobs/abc12345/select-instrumental" \
|
||||
-H "Content-Type: application/json" \
|
||||
-d '{"selection": "clean"}'
|
||||
```
|
||||
|
||||
### Download Files
|
||||
|
||||
**GET** `/api/jobs/{job_id}/download-urls`
|
||||
|
||||
Get download URLs for all output files.
|
||||
|
||||
**GET** `/api/jobs/{job_id}/download/{category}/{file_key}`
|
||||
|
||||
Stream download a specific file.
|
||||
|
||||
### Health Check
|
||||
|
||||
**GET** `/api/health`
|
||||
|
||||
Check backend health status.
|
||||
|
||||
---
|
||||
|
||||
## 🔧 Troubleshooting
|
||||
|
||||
### "No suitable files found for processing"
|
||||
|
||||
This error occurs during the finalisation step when the `(With Vocals).mkv` file is missing. This file is created during lyrics transcription.
|
||||
|
||||
**Most common cause:** No transcription provider configured.
|
||||
|
||||
**Quick fix:**
|
||||
1. Check if transcription providers are configured:
|
||||
```bash
|
||||
echo $AUDIOSHAKE_API_TOKEN
|
||||
echo $RUNPOD_API_KEY
|
||||
```
|
||||
|
||||
2. If both are empty, set up a provider (see [Transcription Provider Setup](#transcription-provider-setup))
|
||||
|
||||
3. Or use `--skip-lyrics` for instrumental-only karaoke:
|
||||
```bash
|
||||
karaoke-gen --skip-lyrics "Artist" "Title"
|
||||
```
|
||||
|
||||
**Other causes:**
|
||||
- Invalid API credentials - verify your tokens are correct and active
|
||||
- API service unavailable - check service status pages
|
||||
- Network connectivity issues - ensure you can reach the API endpoints
|
||||
- Transcription timeout - try again or use a different provider
|
||||
|
||||
### Transcription Fails Silently
|
||||
|
||||
If karaoke-gen runs without errors but produces no synchronized lyrics:
|
||||
|
||||
1. **Check logs** - Run with `--log_level debug` for detailed output:
|
||||
```bash
|
||||
karaoke-gen --log_level debug "Artist" "Title"
|
||||
```
|
||||
|
||||
2. **Verify environment variables** - Ensure API tokens are exported in your shell:
|
||||
```bash
|
||||
# Check if set
|
||||
printenv | grep -E "(AUDIOSHAKE|RUNPOD|WHISPER)"
|
||||
|
||||
# Set in current session
|
||||
export AUDIOSHAKE_API_TOKEN="your_token"
|
||||
```
|
||||
|
||||
3. **Test API connectivity** - Verify you can reach the transcription service
|
||||
|
||||
### "No lyrics found from any source"
|
||||
|
||||
This warning means no reference lyrics were fetched from online sources (Genius, Spotify, Musixmatch). The transcription will still work, but auto-correction may be less accurate.
|
||||
|
||||
**To fix:**
|
||||
- Set `GENIUS_API_TOKEN` for Genius lyrics
|
||||
- Set `SPOTIFY_COOKIE_SP_DC` for Spotify lyrics
|
||||
- Set `RAPIDAPI_KEY` for Musixmatch lyrics
|
||||
- Or provide lyrics manually with `--lyrics_file /path/to/lyrics.txt`
|
||||
|
||||
### Video Quality Issues
|
||||
|
||||
If the output video has quality problems:
|
||||
- Ensure FFmpeg is properly installed: `ffmpeg -version`
|
||||
- Check available codecs: `ffmpeg -codecs`
|
||||
- For 4K output, ensure sufficient disk space (10GB+ per track)
|
||||
|
||||
### Local Whisper Issues
|
||||
|
||||
#### GPU Out of Memory
|
||||
If you get CUDA out of memory errors:
|
||||
```bash
|
||||
# Use a smaller model
|
||||
export WHISPER_MODEL_SIZE="small" # or "tiny"
|
||||
|
||||
# Or force CPU mode
|
||||
export WHISPER_DEVICE="cpu"
|
||||
```
|
||||
|
||||
#### Slow Transcription on CPU
|
||||
CPU transcription is significantly slower than GPU. For faster processing:
|
||||
- Use a smaller model (`tiny` or `base`)
|
||||
- Consider using cloud transcription (AudioShake or RunPod)
|
||||
- On Apple Silicon, the `small` model offers good speed/quality balance
|
||||
|
||||
#### Model Download Issues
|
||||
Whisper models are downloaded on first use (~1-3GB depending on size). If downloads fail:
|
||||
- Check your internet connection
|
||||
- Set a custom cache directory: `export WHISPER_CACHE_DIR="/path/with/space"`
|
||||
- Models are cached in `~/.cache/whisper/` by default
|
||||
|
||||
#### whisper-timestamped Not Found
|
||||
If you get "whisper-timestamped is not installed":
|
||||
```bash
|
||||
pip install "karaoke-gen[local-whisper]"
|
||||
# Or install directly:
|
||||
pip install whisper-timestamped
|
||||
```
|
||||
|
||||
#### Disabling Local Whisper
|
||||
If you want to disable local Whisper (e.g., to force cloud transcription):
|
||||
```bash
|
||||
export ENABLE_LOCAL_WHISPER="false"
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 🧪 Development
|
||||
|
||||
### Running Tests
|
||||
|
||||
```bash
|
||||
# Run all tests
|
||||
pytest tests/ backend/tests/ -v
|
||||
|
||||
# Run only unit tests
|
||||
pytest tests/unit/ -v
|
||||
|
||||
# Run with coverage
|
||||
pytest tests/unit/ -v --cov=karaoke_gen --cov-report=term-missing
|
||||
```
|
||||
|
||||
### Project Structure
|
||||
|
||||
```
|
||||
karaoke-gen/
|
||||
├── karaoke_gen/ # Core CLI package
|
||||
│ ├── utils/
|
||||
│ │ ├── gen_cli.py # Local CLI (karaoke-gen)
|
||||
│ │ └── remote_cli.py # Remote CLI (karaoke-gen-remote)
|
||||
│ ├── karaoke_finalise/ # Video encoding, packaging, distribution
|
||||
│ └── style_loader.py # Unified style configuration
|
||||
├── backend/ # Cloud backend (FastAPI)
|
||||
│ ├── api/routes/ # API endpoints
|
||||
│ ├── workers/ # Background processing workers
|
||||
│ └── services/ # Business logic services
|
||||
├── infrastructure/ # Pulumi IaC for GCP
|
||||
├── docs/ # Documentation
|
||||
└── tests/ # Test suite
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 📄 License
|
||||
|
||||
MIT
|
||||
|
||||
---
|
||||
|
||||
## 🤝 Contributing
|
||||
|
||||
Contributions are welcome! Please see our contributing guidelines.
|
||||
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
@@ -1,43 +0,0 @@
|
||||
# Fast application image - uses pre-built base with all dependencies
|
||||
#
|
||||
# The base image (karaoke-backend-base) contains:
|
||||
# - Python 3.11, ffmpeg, sox, system deps
|
||||
# - All pip dependencies from requirements-frozen.txt (NO code)
|
||||
#
|
||||
# This Dockerfile adds the actual application code on top.
|
||||
# Build time: ~1-2 minutes (deps already cached in base)
|
||||
#
|
||||
# If the base image doesn't exist yet, build it first:
|
||||
# gcloud builds submit --config=cloudbuild-base.yaml --project=nomadkaraoke
|
||||
|
||||
# Use pre-built base image with all dependencies
|
||||
# Override with --build-arg BASE_IMAGE=karaoke-backend-base:local for local testing
|
||||
ARG BASE_IMAGE=us-central1-docker.pkg.dev/nomadkaraoke/karaoke-repo/karaoke-backend-base:latest
|
||||
FROM ${BASE_IMAGE}
|
||||
|
||||
# Set working directory (should already exist from base)
|
||||
WORKDIR /app
|
||||
|
||||
# Copy backend code (this is the only thing that changes frequently)
|
||||
COPY backend /app/backend
|
||||
|
||||
# Copy and reinstall karaoke_gen from local source
|
||||
# This ensures new modules (like style_loader.py) are included without rebuilding base image
|
||||
# Note: lyrics_transcriber is vendored in lyrics_transcriber_temp/ and bundled via pyproject.toml
|
||||
COPY karaoke_gen /app/karaoke_gen
|
||||
COPY lyrics_transcriber_temp /app/lyrics_transcriber_temp
|
||||
COPY pyproject.toml README.md LICENSE /app/
|
||||
RUN pip install --no-cache-dir -e .
|
||||
|
||||
# Set environment variables
|
||||
ENV PYTHONUNBUFFERED=1
|
||||
ENV PORT=8080
|
||||
|
||||
# Expose port
|
||||
EXPOSE 8080
|
||||
|
||||
# Run uvicorn directly
|
||||
# Note: Transmission daemon removed - torrent downloads now handled by
|
||||
# dedicated flacfetch VM service (see docs/00-current-plan/TORRENT-SERVICE-PLAN.md)
|
||||
# Using --loop asyncio (instead of uvloop) to support nest_asyncio for remote flacfetch calls
|
||||
CMD ["uvicorn", "backend.main:app", "--host", "0.0.0.0", "--port", "8080", "--loop", "asyncio"]
|
||||
@@ -1,45 +0,0 @@
|
||||
# Base image with all Python dependencies pre-installed
|
||||
# This image is rebuilt only when poetry.lock changes
|
||||
#
|
||||
# Build with: gcloud builds submit --config=cloudbuild-base.yaml
|
||||
# Or manually: docker build -f backend/Dockerfile.base -t karaoke-backend-base .
|
||||
|
||||
FROM python:3.11-slim
|
||||
|
||||
# Install system dependencies required by karaoke-gen
|
||||
# Note: Transmission daemon removed - torrent downloads now handled by
|
||||
# dedicated flacfetch VM service (see docs/00-current-plan/TORRENT-SERVICE-PLAN.md)
|
||||
RUN apt-get update && apt-get install -y \
|
||||
ffmpeg \
|
||||
libsndfile1 \
|
||||
libsox-dev \
|
||||
sox \
|
||||
build-essential \
|
||||
curl \
|
||||
&& rm -rf /var/lib/apt/lists/*
|
||||
|
||||
# Set working directory
|
||||
WORKDIR /app
|
||||
|
||||
# Copy dependency files and source code needed for pip install
|
||||
COPY pyproject.toml README.md LICENSE /app/
|
||||
COPY karaoke_gen /app/karaoke_gen
|
||||
COPY lyrics_transcriber_temp /app/lyrics_transcriber_temp
|
||||
|
||||
# Install all Python dependencies (the slow part - cached in base image)
|
||||
RUN pip install --no-cache-dir --upgrade pip && \
|
||||
pip install --no-cache-dir -e .
|
||||
|
||||
# Download spaCy language model for lyrics transcription
|
||||
# This must be done at build time, not runtime (Cloud Run network restrictions)
|
||||
RUN python -m spacy download en_core_web_sm
|
||||
|
||||
# Label with build info
|
||||
ARG BUILD_DATE
|
||||
ARG BUILD_ID
|
||||
ARG BASE_CONTENT_HASH
|
||||
LABEL org.opencontainers.image.created="${BUILD_DATE}"
|
||||
LABEL org.opencontainers.image.revision="${BUILD_ID}"
|
||||
LABEL org.opencontainers.image.title="karaoke-backend-base"
|
||||
LABEL org.opencontainers.image.description="Base image with Python dependencies and system tools for karaoke-backend"
|
||||
LABEL base.content.hash="${BASE_CONTENT_HASH}"
|
||||
@@ -1,242 +0,0 @@
|
||||
# Backend Development Setup
|
||||
|
||||
## One-Time Setup
|
||||
|
||||
### 1. Create Virtual Environment
|
||||
|
||||
```bash
|
||||
cd /Users/andrew/Projects/karaoke-gen
|
||||
python3 -m venv backend/venv
|
||||
```
|
||||
|
||||
### 2. Activate Virtual Environment
|
||||
|
||||
```bash
|
||||
source backend/venv/bin/activate
|
||||
```
|
||||
|
||||
### 3. Install Dependencies
|
||||
|
||||
**IMPORTANT:** Use Python 3.12 (not 3.13/3.14 - pydantic has compatibility issues)
|
||||
|
||||
```bash
|
||||
# Backend dependencies
|
||||
pip install -r backend/requirements.txt
|
||||
|
||||
# Install karaoke_gen in development mode
|
||||
pip install -e .
|
||||
|
||||
# Install lyrics_transcriber submodule
|
||||
pip install -e lyrics_transcriber_local/
|
||||
|
||||
# Install Google API client (for YouTube upload)
|
||||
pip install google-api-python-client google-auth-httplib2 google-auth-oauthlib
|
||||
```
|
||||
|
||||
### 4. Add Shell Alias (Optional)
|
||||
|
||||
Add to `~/.zshrc`:
|
||||
|
||||
```bash
|
||||
alias backend='cd /Users/andrew/Projects/karaoke-gen && source backend/venv/bin/activate'
|
||||
```
|
||||
|
||||
Then you can just run:
|
||||
```bash
|
||||
backend
|
||||
# Now you're in the project directory with venv activated!
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Daily Workflow
|
||||
|
||||
### Before Making Changes
|
||||
|
||||
```bash
|
||||
# Activate venv
|
||||
source backend/venv/bin/activate
|
||||
|
||||
# Or if you set up the alias:
|
||||
backend
|
||||
```
|
||||
|
||||
### After Making Changes
|
||||
|
||||
```bash
|
||||
# 1. Run full validation (catches import errors!)
|
||||
python3 backend/validate.py
|
||||
|
||||
# 2. If validation passes, deploy
|
||||
./scripts/deploy.sh
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Validation Options
|
||||
|
||||
### Full Validation (Recommended - Requires venv)
|
||||
|
||||
```bash
|
||||
source backend/venv/bin/activate
|
||||
python3 backend/validate.py
|
||||
```
|
||||
|
||||
**Catches:**
|
||||
- ✅ Syntax errors
|
||||
- ✅ **Import errors** (like the processing_service issue!)
|
||||
- ✅ Configuration issues
|
||||
- ✅ FastAPI app creation problems
|
||||
- ✅ Missing dependencies
|
||||
|
||||
**This would have caught the ModuleNotFoundError!**
|
||||
|
||||
### Quick Check (No venv needed)
|
||||
|
||||
```bash
|
||||
./backend/quick-check.sh
|
||||
```
|
||||
|
||||
**Catches:**
|
||||
- ✅ Syntax errors
|
||||
- ✅ Missing files
|
||||
- ✅ Some obvious import issues
|
||||
|
||||
**Good for quick checks, but won't catch all import errors**
|
||||
|
||||
---
|
||||
|
||||
## Example: Complete Development Session
|
||||
|
||||
```bash
|
||||
# 1. Activate environment
|
||||
backend # or: source backend/venv/bin/activate
|
||||
|
||||
# 2. Make your changes
|
||||
vim backend/api/routes/jobs.py
|
||||
|
||||
# 3. Validate (catches import errors!)
|
||||
python3 backend/validate.py
|
||||
# ✅ All validations passed!
|
||||
|
||||
# 4. Deploy
|
||||
./scripts/deploy.sh
|
||||
|
||||
# 5. Test
|
||||
export BACKEND_URL="https://karaoke-backend-ipzqd2k4yq-uc.a.run.app"
|
||||
export AUTH_TOKEN=$(gcloud auth print-identity-token)
|
||||
curl -H "Authorization: Bearer $AUTH_TOKEN" $BACKEND_URL/api/health
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## What About karaoke_gen Imports?
|
||||
|
||||
The backend imports from `karaoke_gen` (the CLI package):
|
||||
|
||||
```python
|
||||
from karaoke_gen.karaoke_gen import KaraokePrep
|
||||
from karaoke_gen.audio_processor import AudioProcessor
|
||||
```
|
||||
|
||||
**Options:**
|
||||
|
||||
### Option 1: Install in development mode (Recommended)
|
||||
|
||||
```bash
|
||||
source backend/venv/bin/activate
|
||||
pip install -e .
|
||||
```
|
||||
|
||||
This makes `karaoke_gen` available in the venv without copying files.
|
||||
|
||||
### Option 2: Use system Python
|
||||
|
||||
Your system Python already has `karaoke_gen` installed, so validation will work even if the venv doesn't have it (Python will fall back to system packages).
|
||||
|
||||
---
|
||||
|
||||
## Running the Backend Locally
|
||||
|
||||
```bash
|
||||
# Activate venv
|
||||
source backend/venv/bin/activate
|
||||
|
||||
# Set required environment variables
|
||||
export GOOGLE_CLOUD_PROJECT="nomadkaraoke"
|
||||
export GCS_BUCKET_NAME="karaoke-gen-storage-nomadkaraoke"
|
||||
|
||||
# Optional: Point to local credentials
|
||||
export GOOGLE_APPLICATION_CREDENTIALS="/path/to/service-account-key.json"
|
||||
|
||||
# Run the server
|
||||
cd backend
|
||||
uvicorn main:app --reload --port 8080
|
||||
```
|
||||
|
||||
Visit: http://localhost:8080/api/health
|
||||
|
||||
---
|
||||
|
||||
## Troubleshooting
|
||||
|
||||
### "No module named 'fastapi'"
|
||||
|
||||
**Fix:**
|
||||
```bash
|
||||
source backend/venv/bin/activate
|
||||
pip install -r backend/requirements.txt
|
||||
```
|
||||
|
||||
### "No module named 'karaoke_gen'"
|
||||
|
||||
**Fix:**
|
||||
```bash
|
||||
source backend/venv/bin/activate
|
||||
cd /Users/andrew/Projects/karaoke-gen
|
||||
pip install -e .
|
||||
```
|
||||
|
||||
### Validation passes locally but fails in Cloud Run
|
||||
|
||||
**Check:**
|
||||
1. Is `requirements.txt` up to date?
|
||||
2. Did you add a new dependency without adding it to `requirements.txt`?
|
||||
3. Run: `pip freeze | grep <package-name>`
|
||||
|
||||
---
|
||||
|
||||
## Updating Dependencies
|
||||
|
||||
When you add new packages:
|
||||
|
||||
```bash
|
||||
# 1. Install in venv
|
||||
source backend/venv/bin/activate
|
||||
pip install new-package
|
||||
|
||||
# 2. Update requirements.txt
|
||||
pip freeze > backend/requirements.txt
|
||||
|
||||
# 3. Or manually add just the new package:
|
||||
echo "new-package==1.2.3" >> backend/requirements.txt
|
||||
|
||||
# 4. Validate
|
||||
python3 backend/validate.py
|
||||
|
||||
# 5. Deploy
|
||||
./scripts/deploy.sh
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Summary
|
||||
|
||||
**Best practice workflow:**
|
||||
|
||||
1. **One-time setup:** Create venv and install dependencies
|
||||
2. **Daily:** Activate venv when working on backend
|
||||
3. **Before deploy:** Run `python3 backend/validate.py` (catches import errors!)
|
||||
4. **Deploy:** Use `./scripts/deploy.sh`
|
||||
|
||||
**The import error would have been caught if you ran validation in a venv!** ✅
|
||||
Whitespace-only changes.
Whitespace-only changes.
@@ -1,345 +0,0 @@
|
||||
"""
|
||||
FastAPI dependencies for authentication and authorization.
|
||||
"""
|
||||
import logging
|
||||
import secrets
|
||||
from datetime import datetime, timedelta, UTC
|
||||
from typing import Optional, Tuple, Callable
|
||||
from fastapi import Depends, HTTPException, Header, Query, Request, Path
|
||||
from fastapi.security import HTTPBearer, HTTPAuthorizationCredentials
|
||||
|
||||
from backend.services.auth_service import get_auth_service, UserType, AuthService
|
||||
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
|
||||
def generate_review_token() -> str:
|
||||
"""Generate a cryptographically secure review token."""
|
||||
return secrets.token_urlsafe(32)
|
||||
|
||||
|
||||
def get_review_token_expiry(hours: int = 24) -> datetime:
|
||||
"""Get expiry time for a review token."""
|
||||
return datetime.now(UTC) + timedelta(hours=hours)
|
||||
|
||||
|
||||
# HTTP Bearer security scheme (Authorization: Bearer <token>)
|
||||
security = HTTPBearer(auto_error=False)
|
||||
|
||||
|
||||
async def get_token_from_request(
|
||||
request: Request,
|
||||
credentials: Optional[HTTPAuthorizationCredentials] = Depends(security),
|
||||
token: Optional[str] = Query(None, description="Access token (alternative to Bearer header)")
|
||||
) -> Optional[str]:
|
||||
"""
|
||||
Extract access token from request.
|
||||
|
||||
Supports three methods (in priority order):
|
||||
1. X-Admin-Token header: Used by Cloud Tasks (since OIDC overwrites Authorization)
|
||||
2. Authorization header: Authorization: Bearer <token>
|
||||
3. Query parameter: ?token=<token> (for download links)
|
||||
|
||||
Returns:
|
||||
Token string or None
|
||||
"""
|
||||
# Check X-Admin-Token header first (used by Cloud Tasks)
|
||||
# Cloud Tasks OIDC token overwrites Authorization header, so we use a custom header
|
||||
admin_token = request.headers.get("X-Admin-Token")
|
||||
if admin_token:
|
||||
return admin_token
|
||||
|
||||
# Try Authorization header
|
||||
if credentials:
|
||||
return credentials.credentials
|
||||
|
||||
# Try query parameter
|
||||
if token:
|
||||
return token
|
||||
|
||||
return None
|
||||
|
||||
|
||||
async def require_auth(
|
||||
request: Request,
|
||||
auth_service: AuthService = Depends(get_auth_service),
|
||||
credentials: Optional[HTTPAuthorizationCredentials] = Depends(security),
|
||||
token: Optional[str] = Query(None)
|
||||
) -> Tuple[str, UserType, int]:
|
||||
"""
|
||||
Require authentication for an endpoint.
|
||||
|
||||
Returns:
|
||||
(token, user_type, remaining_uses)
|
||||
|
||||
Raises:
|
||||
HTTPException: 401 if authentication fails
|
||||
"""
|
||||
# Get token from request
|
||||
token_str = await get_token_from_request(request, credentials, token)
|
||||
|
||||
if not token_str:
|
||||
raise HTTPException(
|
||||
status_code=401,
|
||||
detail="Authentication required. Provide token via Authorization header or ?token= parameter"
|
||||
)
|
||||
|
||||
# Validate token
|
||||
is_valid, user_type, remaining_uses, message = auth_service.validate_token(token_str)
|
||||
|
||||
if not is_valid:
|
||||
# Log more details for debugging token issues
|
||||
auth_header = request.headers.get("Authorization", "")
|
||||
logger.warning(
|
||||
f"Authentication failed: {message}. "
|
||||
f"Token provided: {bool(token_str)}, "
|
||||
f"Token length: {len(token_str) if token_str else 0}, "
|
||||
f"Auth header present: {bool(auth_header)}, "
|
||||
f"Header prefix: {auth_header[:20] if auth_header else 'none'}..."
|
||||
)
|
||||
raise HTTPException(
|
||||
status_code=401,
|
||||
detail=f"Authentication failed: {message}"
|
||||
)
|
||||
|
||||
logger.info(f"Authenticated as {user_type} (remaining: {remaining_uses})")
|
||||
|
||||
return token_str, user_type, remaining_uses
|
||||
|
||||
|
||||
async def require_admin(
|
||||
auth_data: Tuple[str, UserType, int] = Depends(require_auth)
|
||||
) -> Tuple[str, UserType, int]:
|
||||
"""
|
||||
Require admin access for an endpoint.
|
||||
|
||||
Returns:
|
||||
(token, user_type, remaining_uses)
|
||||
|
||||
Raises:
|
||||
HTTPException: 403 if user is not admin
|
||||
"""
|
||||
token, user_type, remaining_uses = auth_data
|
||||
|
||||
if user_type != UserType.ADMIN:
|
||||
logger.warning(f"Admin access denied for {user_type} user")
|
||||
raise HTTPException(
|
||||
status_code=403,
|
||||
detail="Admin access required"
|
||||
)
|
||||
|
||||
return token, user_type, remaining_uses
|
||||
|
||||
|
||||
def optional_auth(
|
||||
auth_service: AuthService = Depends(get_auth_service),
|
||||
credentials: Optional[HTTPAuthorizationCredentials] = Depends(security),
|
||||
token: Optional[str] = Query(None)
|
||||
) -> Optional[Tuple[str, UserType, int]]:
|
||||
"""
|
||||
Optional authentication - doesn't fail if no token provided.
|
||||
|
||||
Useful for endpoints that have different behavior for authenticated users.
|
||||
|
||||
Returns:
|
||||
(token, user_type, remaining_uses) if authenticated, None otherwise
|
||||
"""
|
||||
# Get token
|
||||
token_str = None
|
||||
if credentials:
|
||||
token_str = credentials.credentials
|
||||
elif token:
|
||||
token_str = token
|
||||
|
||||
if not token_str:
|
||||
return None
|
||||
|
||||
# Validate token
|
||||
is_valid, user_type, remaining_uses, message = auth_service.validate_token(token_str)
|
||||
|
||||
if not is_valid:
|
||||
logger.debug(f"Optional auth failed: {message}")
|
||||
return None
|
||||
|
||||
return token_str, user_type, remaining_uses
|
||||
|
||||
|
||||
def require_review_auth_factory(job_id_param: str = "job_id"):
|
||||
"""
|
||||
Factory to create a review authentication dependency.
|
||||
|
||||
Accepts either:
|
||||
1. Full user authentication (admin/user token)
|
||||
2. Job-specific review token (only valid for the specific job)
|
||||
|
||||
Args:
|
||||
job_id_param: Name of the path parameter containing the job ID
|
||||
|
||||
Returns:
|
||||
Dependency function that validates review access
|
||||
"""
|
||||
async def require_review_auth(
|
||||
request: Request,
|
||||
job_id: str = Path(...),
|
||||
review_token: Optional[str] = Query(None, description="Job-specific review token"),
|
||||
auth_service: AuthService = Depends(get_auth_service),
|
||||
credentials: Optional[HTTPAuthorizationCredentials] = Depends(security),
|
||||
token: Optional[str] = Query(None, alias="token", description="Full access token")
|
||||
) -> Tuple[str, str]:
|
||||
"""
|
||||
Validate review access for a job.
|
||||
|
||||
Returns:
|
||||
(job_id, auth_type) where auth_type is "full" or "review_token"
|
||||
|
||||
Raises:
|
||||
HTTPException: 401 if authentication fails
|
||||
"""
|
||||
# Import here to avoid circular dependency
|
||||
from backend.services.job_manager import JobManager
|
||||
|
||||
# Try full authentication first
|
||||
full_token = None
|
||||
if credentials:
|
||||
full_token = credentials.credentials
|
||||
elif token:
|
||||
full_token = token
|
||||
|
||||
if full_token:
|
||||
is_valid, user_type, remaining_uses, message = auth_service.validate_token(full_token)
|
||||
if is_valid:
|
||||
logger.info(f"Review access granted via full auth ({user_type}) for job {job_id}")
|
||||
return job_id, "full"
|
||||
|
||||
# Try review token
|
||||
if review_token:
|
||||
job_manager = JobManager()
|
||||
job = job_manager.get_job(job_id)
|
||||
|
||||
if not job:
|
||||
raise HTTPException(status_code=404, detail=f"Job {job_id} not found")
|
||||
|
||||
# Validate review token matches
|
||||
if job.review_token and secrets.compare_digest(job.review_token, review_token):
|
||||
# Check expiry if set
|
||||
if job.review_token_expires_at:
|
||||
now = datetime.now(UTC)
|
||||
# Handle timezone-naive datetimes from Firestore
|
||||
expiry = job.review_token_expires_at
|
||||
if expiry.tzinfo is None:
|
||||
expiry = expiry.replace(tzinfo=UTC)
|
||||
|
||||
if now > expiry:
|
||||
logger.warning(f"Review token expired for job {job_id}")
|
||||
raise HTTPException(
|
||||
status_code=401,
|
||||
detail="Review token has expired. Please request a new review link."
|
||||
)
|
||||
|
||||
logger.info(f"Review access granted via review_token for job {job_id}")
|
||||
return job_id, "review_token"
|
||||
else:
|
||||
logger.warning(f"Invalid review token for job {job_id}")
|
||||
|
||||
# No valid authentication
|
||||
raise HTTPException(
|
||||
status_code=401,
|
||||
detail="Authentication required. Provide either a full access token or a valid review_token for this job."
|
||||
)
|
||||
|
||||
return require_review_auth
|
||||
|
||||
|
||||
# Default instance for most review endpoints
|
||||
require_review_auth = require_review_auth_factory()
|
||||
|
||||
|
||||
def require_instrumental_auth_factory(job_id_param: str = "job_id"):
|
||||
"""
|
||||
Factory to create an instrumental review authentication dependency.
|
||||
|
||||
Accepts either:
|
||||
1. Full user authentication (admin/user token)
|
||||
2. Job-specific instrumental token (only valid for the specific job)
|
||||
|
||||
Args:
|
||||
job_id_param: Name of the path parameter containing the job ID
|
||||
|
||||
Returns:
|
||||
Dependency function that validates instrumental review access
|
||||
"""
|
||||
async def require_instrumental_auth(
|
||||
request: Request,
|
||||
job_id: str = Path(...),
|
||||
instrumental_token: Optional[str] = Query(None, description="Job-specific instrumental review token"),
|
||||
auth_service: AuthService = Depends(get_auth_service),
|
||||
credentials: Optional[HTTPAuthorizationCredentials] = Depends(security),
|
||||
token: Optional[str] = Query(None, alias="token", description="Full access token")
|
||||
) -> Tuple[str, str]:
|
||||
"""
|
||||
Validate instrumental review access for a job.
|
||||
|
||||
Returns:
|
||||
(job_id, auth_type) where auth_type is "full" or "instrumental_token"
|
||||
|
||||
Raises:
|
||||
HTTPException: 401 if authentication fails
|
||||
"""
|
||||
# Import here to avoid circular dependency
|
||||
from backend.services.job_manager import JobManager
|
||||
|
||||
# Try full authentication first
|
||||
full_token = None
|
||||
if credentials:
|
||||
full_token = credentials.credentials
|
||||
elif token:
|
||||
full_token = token
|
||||
|
||||
if full_token:
|
||||
is_valid, user_type, remaining_uses, message = auth_service.validate_token(full_token)
|
||||
if is_valid:
|
||||
logger.info(f"Instrumental access granted via full auth ({user_type}) for job {job_id}")
|
||||
return job_id, "full"
|
||||
|
||||
# Try instrumental token
|
||||
if instrumental_token:
|
||||
job_manager = JobManager()
|
||||
job = job_manager.get_job(job_id)
|
||||
|
||||
if not job:
|
||||
raise HTTPException(status_code=404, detail=f"Job {job_id} not found")
|
||||
|
||||
# Validate instrumental token matches
|
||||
if job.instrumental_token and secrets.compare_digest(job.instrumental_token, instrumental_token):
|
||||
# Check expiry if set
|
||||
if job.instrumental_token_expires_at:
|
||||
now = datetime.now(UTC)
|
||||
# Handle timezone-naive datetimes from Firestore
|
||||
expiry = job.instrumental_token_expires_at
|
||||
if expiry.tzinfo is None:
|
||||
expiry = expiry.replace(tzinfo=UTC)
|
||||
|
||||
if now > expiry:
|
||||
logger.warning(f"Instrumental token expired for job {job_id}")
|
||||
raise HTTPException(
|
||||
status_code=401,
|
||||
detail="Instrumental review token has expired. Please request a new review link."
|
||||
)
|
||||
|
||||
logger.info(f"Instrumental access granted via instrumental_token for job {job_id}")
|
||||
return job_id, "instrumental_token"
|
||||
else:
|
||||
logger.warning(f"Invalid instrumental token for job {job_id}")
|
||||
|
||||
# No valid authentication
|
||||
raise HTTPException(
|
||||
status_code=401,
|
||||
detail="Authentication required. Provide either a full access token or a valid instrumental_token for this job."
|
||||
)
|
||||
|
||||
return require_instrumental_auth
|
||||
|
||||
|
||||
# Default instance for most instrumental review endpoints
|
||||
require_instrumental_auth = require_instrumental_auth_factory()
|
||||
Whitespace-only changes.
@@ -1,853 +0,0 @@
|
||||
"""
|
||||
Audio search API routes for artist+title search mode.
|
||||
|
||||
This module provides endpoints for:
|
||||
1. Creating a job with audio search (artist+title without file)
|
||||
2. Getting search results for a job
|
||||
3. Selecting an audio source to download
|
||||
|
||||
The flow is:
|
||||
1. POST /api/audio-search/search - Create job and search for audio
|
||||
2. GET /api/audio-search/{job_id}/results - Get search results
|
||||
3. POST /api/audio-search/{job_id}/select - Select audio source
|
||||
"""
|
||||
import asyncio
|
||||
import logging
|
||||
import os
|
||||
import tempfile
|
||||
from typing import Optional, List, Dict, Any, Tuple
|
||||
|
||||
from fastapi import APIRouter, HTTPException, BackgroundTasks, Request, Depends
|
||||
from pydantic import BaseModel, Field
|
||||
|
||||
from backend.models.job import JobCreate, JobStatus
|
||||
from backend.services.job_manager import JobManager
|
||||
from backend.services.storage_service import StorageService
|
||||
from backend.services.worker_service import get_worker_service
|
||||
from backend.services.credential_manager import get_credential_manager, CredentialStatus
|
||||
from backend.services.audio_search_service import (
|
||||
get_audio_search_service,
|
||||
AudioSearchResult,
|
||||
AudioSearchError,
|
||||
NoResultsError,
|
||||
DownloadError,
|
||||
)
|
||||
from backend.config import get_settings
|
||||
from backend.version import VERSION
|
||||
from backend.api.dependencies import require_auth
|
||||
from backend.services.auth_service import UserType
|
||||
from pathlib import Path
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
# Valid style file types (from file_upload.py)
|
||||
STYLE_FILE_TYPES = {
|
||||
'style_params': {'.json'},
|
||||
'style_intro_background': {'.jpg', '.jpeg', '.png'},
|
||||
'style_karaoke_background': {'.jpg', '.jpeg', '.png'},
|
||||
'style_end_background': {'.jpg', '.jpeg', '.png'},
|
||||
'style_font': {'.ttf', '.otf'},
|
||||
'style_cdg_instrumental_background': {'.jpg', '.jpeg', '.png'},
|
||||
'style_cdg_title_background': {'.jpg', '.jpeg', '.png'},
|
||||
'style_cdg_outro_background': {'.jpg', '.jpeg', '.png'},
|
||||
}
|
||||
router = APIRouter(tags=["audio-search"])
|
||||
|
||||
# Initialize services
|
||||
job_manager = JobManager()
|
||||
storage_service = StorageService()
|
||||
worker_service = get_worker_service()
|
||||
|
||||
|
||||
# ============================================================================
|
||||
# Pydantic models
|
||||
# ============================================================================
|
||||
|
||||
class StyleFileRequest(BaseModel):
|
||||
"""Information about a style file to be uploaded."""
|
||||
filename: str = Field(..., description="Original filename with extension")
|
||||
content_type: str = Field(..., description="MIME type of the file")
|
||||
file_type: str = Field(..., description="Type of file: 'style_params', 'style_intro_background', 'style_karaoke_background', 'style_end_background', 'style_font', 'style_cdg_instrumental_background', 'style_cdg_title_background', 'style_cdg_outro_background'")
|
||||
|
||||
|
||||
class StyleUploadUrl(BaseModel):
|
||||
"""Signed URL for uploading a style file."""
|
||||
file_type: str = Field(..., description="Type of file being uploaded")
|
||||
gcs_path: str = Field(..., description="Destination path in GCS")
|
||||
upload_url: str = Field(..., description="Signed URL to PUT the file to")
|
||||
|
||||
|
||||
class AudioSearchRequest(BaseModel):
|
||||
"""Request to search for audio by artist and title."""
|
||||
artist: str = Field(..., description="Artist name to search for")
|
||||
title: str = Field(..., description="Song title to search for")
|
||||
|
||||
# Auto-download mode
|
||||
auto_download: bool = Field(False, description="Automatically select best result and download")
|
||||
|
||||
# Theme configuration
|
||||
theme_id: Optional[str] = Field(None, description="Theme ID to use (e.g., 'nomad', 'default'). If set, CDG/TXT are enabled by default.")
|
||||
color_overrides: Optional[Dict[str, str]] = Field(None, description="Color overrides: artist_color, title_color, sung_lyrics_color, unsung_lyrics_color (hex #RRGGBB)")
|
||||
|
||||
# Processing options (CDG/TXT require style config or theme, disabled by default unless theme is set)
|
||||
enable_cdg: Optional[bool] = Field(None, description="Generate CDG+MP3 package. Default: True if theme_id set, False otherwise")
|
||||
enable_txt: Optional[bool] = Field(None, description="Generate TXT+MP3 package. Default: True if theme_id set, False otherwise")
|
||||
|
||||
# Finalisation options
|
||||
brand_prefix: Optional[str] = Field(None, description="Brand code prefix (e.g., NOMAD)")
|
||||
enable_youtube_upload: bool = Field(False, description="Upload to YouTube")
|
||||
youtube_description: Optional[str] = Field(None, description="YouTube video description text")
|
||||
discord_webhook_url: Optional[str] = Field(None, description="Discord webhook URL for notifications")
|
||||
|
||||
# Distribution options
|
||||
dropbox_path: Optional[str] = Field(None, description="Dropbox folder path for organized output")
|
||||
gdrive_folder_id: Optional[str] = Field(None, description="Google Drive folder ID for public share uploads")
|
||||
|
||||
# Lyrics configuration
|
||||
lyrics_artist: Optional[str] = Field(None, description="Override artist name for lyrics search")
|
||||
lyrics_title: Optional[str] = Field(None, description="Override title for lyrics search")
|
||||
subtitle_offset_ms: int = Field(0, description="Subtitle timing offset in milliseconds")
|
||||
|
||||
# Audio separation model configuration
|
||||
clean_instrumental_model: Optional[str] = Field(None, description="Model for clean instrumental separation")
|
||||
backing_vocals_models: Optional[List[str]] = Field(None, description="Models for backing vocals separation")
|
||||
other_stems_models: Optional[List[str]] = Field(None, description="Models for other stems")
|
||||
|
||||
# Style file uploads (optional)
|
||||
# Style params JSON and related assets (background images, fonts)
|
||||
style_files: Optional[List[StyleFileRequest]] = Field(None, description="Style files to upload (style_params JSON, backgrounds, fonts)")
|
||||
|
||||
|
||||
class AudioSearchResultResponse(BaseModel):
|
||||
"""A single audio search result.
|
||||
|
||||
Contains all fields needed for rich display using flacfetch's
|
||||
shared formatting functions.
|
||||
"""
|
||||
index: int
|
||||
title: str
|
||||
artist: str
|
||||
provider: str # Maps to source_name in Release
|
||||
url: Optional[str] = None # Maps to download_url in Release (may be None for remote)
|
||||
duration: Optional[int] = None # Maps to duration_seconds in Release
|
||||
quality: Optional[str] = None # Stringified quality
|
||||
source_id: Optional[str] = None # Maps to info_hash in Release
|
||||
seeders: Optional[int] = None
|
||||
target_file: Optional[str] = None
|
||||
# Additional fields for rich display (from Release.to_dict())
|
||||
year: Optional[int] = None
|
||||
label: Optional[str] = None
|
||||
edition_info: Optional[str] = None
|
||||
release_type: Optional[str] = None
|
||||
channel: Optional[str] = None # For YouTube
|
||||
view_count: Optional[int] = None # For YouTube
|
||||
size_bytes: Optional[int] = None
|
||||
target_file_size: Optional[int] = None
|
||||
track_pattern: Optional[str] = None
|
||||
match_score: Optional[float] = None
|
||||
# Pre-computed display fields
|
||||
formatted_size: Optional[str] = None
|
||||
formatted_duration: Optional[str] = None
|
||||
formatted_views: Optional[str] = None
|
||||
is_lossless: Optional[bool] = None
|
||||
quality_str: Optional[str] = None
|
||||
# Full quality object for Release.from_dict()
|
||||
quality_data: Optional[Dict[str, Any]] = None
|
||||
|
||||
|
||||
class AudioSearchResponse(BaseModel):
|
||||
"""Response from audio search."""
|
||||
status: str
|
||||
job_id: str
|
||||
message: str
|
||||
results: Optional[List[AudioSearchResultResponse]] = None
|
||||
results_count: int = 0
|
||||
auto_download: bool = False
|
||||
server_version: str
|
||||
# Style file upload URLs (when style_files are specified in request)
|
||||
style_upload_urls: Optional[List[StyleUploadUrl]] = None
|
||||
|
||||
|
||||
class AudioSelectRequest(BaseModel):
|
||||
"""Request to select an audio source."""
|
||||
selection_index: int = Field(..., description="Index of the selected audio source from search results")
|
||||
|
||||
|
||||
class AudioSelectResponse(BaseModel):
|
||||
"""Response from audio source selection."""
|
||||
status: str
|
||||
job_id: str
|
||||
message: str
|
||||
selected_index: int
|
||||
selected_title: str
|
||||
selected_artist: str
|
||||
selected_provider: str
|
||||
|
||||
|
||||
def _resolve_cdg_txt_defaults(
|
||||
theme_id: Optional[str],
|
||||
enable_cdg: Optional[bool],
|
||||
enable_txt: Optional[bool]
|
||||
) -> Tuple[bool, bool]:
|
||||
"""
|
||||
Resolve CDG/TXT settings based on theme and explicit settings.
|
||||
|
||||
When a theme is selected, CDG and TXT are enabled by default.
|
||||
Explicit True/False values always override the default.
|
||||
|
||||
Args:
|
||||
theme_id: Theme identifier (if any)
|
||||
enable_cdg: Explicit CDG setting (None means use default)
|
||||
enable_txt: Explicit TXT setting (None means use default)
|
||||
|
||||
Returns:
|
||||
Tuple of (resolved_enable_cdg, resolved_enable_txt)
|
||||
"""
|
||||
# Default based on whether theme is set
|
||||
default_enabled = theme_id is not None
|
||||
|
||||
# Explicit values override defaults, None uses default
|
||||
resolved_cdg = enable_cdg if enable_cdg is not None else default_enabled
|
||||
resolved_txt = enable_txt if enable_txt is not None else default_enabled
|
||||
|
||||
return resolved_cdg, resolved_txt
|
||||
|
||||
|
||||
def extract_request_metadata(request: Request, created_from: str = "audio_search") -> Dict[str, Any]:
|
||||
"""Extract metadata from request for job tracking."""
|
||||
headers = dict(request.headers)
|
||||
|
||||
client_ip = headers.get('x-forwarded-for', '').split(',')[0].strip()
|
||||
if not client_ip and request.client:
|
||||
client_ip = request.client.host
|
||||
|
||||
user_agent = headers.get('user-agent', '')
|
||||
environment = headers.get('x-environment', '')
|
||||
client_id = headers.get('x-client-id', '')
|
||||
|
||||
custom_headers = {}
|
||||
for key, value in headers.items():
|
||||
if key.lower().startswith('x-') and key.lower() not in ('x-forwarded-for', 'x-forwarded-proto', 'x-forwarded-host'):
|
||||
custom_headers[key] = value
|
||||
|
||||
return {
|
||||
'client_ip': client_ip,
|
||||
'user_agent': user_agent,
|
||||
'environment': environment,
|
||||
'client_id': client_id,
|
||||
'server_version': VERSION,
|
||||
'created_from': created_from,
|
||||
'custom_headers': custom_headers,
|
||||
}
|
||||
|
||||
|
||||
async def _trigger_workers_parallel(job_id: str) -> None:
|
||||
"""Trigger both audio and lyrics workers in parallel."""
|
||||
await asyncio.gather(
|
||||
worker_service.trigger_audio_worker(job_id),
|
||||
worker_service.trigger_lyrics_worker(job_id)
|
||||
)
|
||||
|
||||
|
||||
async def _download_and_start_processing(
|
||||
job_id: str,
|
||||
selection_index: int,
|
||||
audio_search_service,
|
||||
background_tasks: BackgroundTasks,
|
||||
) -> Dict[str, Any]:
|
||||
"""
|
||||
Download selected audio and start job processing.
|
||||
|
||||
This is called either:
|
||||
- Immediately after search if auto_download=True
|
||||
- When user calls the select endpoint
|
||||
|
||||
For remote flacfetch downloads (torrent sources), the file is downloaded
|
||||
on the flacfetch VM and uploaded directly to GCS. For local downloads
|
||||
(YouTube), the file is downloaded locally and then uploaded to GCS.
|
||||
"""
|
||||
job = job_manager.get_job(job_id)
|
||||
if not job:
|
||||
raise HTTPException(status_code=404, detail=f"Job {job_id} not found")
|
||||
|
||||
# Get search results from state_data
|
||||
search_results = job.state_data.get('audio_search_results', [])
|
||||
if not search_results:
|
||||
raise HTTPException(status_code=400, detail="No search results available")
|
||||
|
||||
if selection_index < 0 or selection_index >= len(search_results):
|
||||
raise HTTPException(
|
||||
status_code=400,
|
||||
detail=f"Invalid selection index {selection_index}. Valid range: 0-{len(search_results)-1}"
|
||||
)
|
||||
|
||||
selected = search_results[selection_index]
|
||||
|
||||
# Get remote_search_id from state_data (stored during initial search)
|
||||
remote_search_id = job.state_data.get('remote_search_id')
|
||||
|
||||
# Transition to downloading state
|
||||
job_manager.transition_to_state(
|
||||
job_id=job_id,
|
||||
new_status=JobStatus.DOWNLOADING_AUDIO,
|
||||
progress=10,
|
||||
message=f"Downloading from {selected['provider']}: {selected['artist']} - {selected['title']}",
|
||||
state_data_updates={
|
||||
'selected_audio_index': selection_index,
|
||||
'selected_audio_provider': selected['provider'],
|
||||
}
|
||||
)
|
||||
|
||||
try:
|
||||
# Determine if this is a remote torrent download
|
||||
is_torrent_source = selected.get('provider') in ['RED', 'OPS']
|
||||
is_remote_enabled = audio_search_service.is_remote_enabled()
|
||||
|
||||
# Check if we can use download_by_id (preferred - avoids re-searching)
|
||||
source_id = selected.get('source_id')
|
||||
source_name = selected.get('provider')
|
||||
target_file = selected.get('target_file')
|
||||
download_url = selected.get('url')
|
||||
|
||||
# For remote torrent downloads, have flacfetch VM upload directly to GCS
|
||||
if is_torrent_source and is_remote_enabled:
|
||||
# Generate GCS path for remote upload
|
||||
gcs_destination = f"uploads/{job_id}/audio/"
|
||||
|
||||
# Use download_by_id if we have source_id (preferred - no re-search needed)
|
||||
if source_id and source_name:
|
||||
logger.info(f"Using download_by_id for {source_name} ID={source_id} with GCS upload to: {gcs_destination}")
|
||||
|
||||
result = audio_search_service.download_by_id(
|
||||
source_name=source_name,
|
||||
source_id=source_id,
|
||||
output_dir="", # Not used for remote
|
||||
target_file=target_file,
|
||||
download_url=download_url,
|
||||
gcs_path=gcs_destination,
|
||||
)
|
||||
else:
|
||||
# Fallback to search-based download (requires re-search)
|
||||
logger.info(f"No source_id available, falling back to search-based download to: {gcs_destination}")
|
||||
|
||||
result = audio_search_service.download(
|
||||
result_index=selection_index,
|
||||
output_dir="", # Not used for remote
|
||||
gcs_path=gcs_destination,
|
||||
remote_search_id=remote_search_id,
|
||||
)
|
||||
|
||||
# For remote downloads, filepath is already the GCS path
|
||||
if result.filepath.startswith("gs://"):
|
||||
# Extract the path portion after the bucket name
|
||||
# Format: gs://bucket/uploads/job_id/audio/filename.flac
|
||||
parts = result.filepath.replace("gs://", "").split("/", 1)
|
||||
if len(parts) == 2:
|
||||
audio_gcs_path = parts[1]
|
||||
else:
|
||||
audio_gcs_path = result.filepath
|
||||
filename = os.path.basename(result.filepath)
|
||||
else:
|
||||
# Fallback: treat as local path (shouldn't happen for remote)
|
||||
filename = os.path.basename(result.filepath)
|
||||
audio_gcs_path = f"uploads/{job_id}/audio/{filename}"
|
||||
|
||||
logger.warning(f"Remote download returned local path: {result.filepath}, uploading manually")
|
||||
with open(result.filepath, 'rb') as f:
|
||||
storage_service.upload_fileobj(f, audio_gcs_path, content_type='audio/flac')
|
||||
|
||||
logger.info(f"Remote download complete, GCS path: {audio_gcs_path}")
|
||||
else:
|
||||
# Local download (YouTube or fallback)
|
||||
temp_dir = tempfile.mkdtemp(prefix=f"audio_download_{job_id}_")
|
||||
|
||||
# Use download_by_id if we have source_id and remote is enabled
|
||||
if source_id and source_name and is_remote_enabled:
|
||||
logger.info(f"Using download_by_id for local download: {source_name} ID={source_id}")
|
||||
|
||||
result = audio_search_service.download_by_id(
|
||||
source_name=source_name,
|
||||
source_id=source_id,
|
||||
output_dir=temp_dir,
|
||||
target_file=target_file,
|
||||
download_url=download_url,
|
||||
)
|
||||
else:
|
||||
# Fallback to search-based download
|
||||
result = audio_search_service.download(
|
||||
result_index=selection_index,
|
||||
output_dir=temp_dir,
|
||||
remote_search_id=remote_search_id, # Pass for potential fallback scenarios
|
||||
)
|
||||
|
||||
# Upload to GCS
|
||||
filename = os.path.basename(result.filepath)
|
||||
audio_gcs_path = f"uploads/{job_id}/audio/{filename}"
|
||||
|
||||
with open(result.filepath, 'rb') as f:
|
||||
storage_service.upload_fileobj(
|
||||
f,
|
||||
audio_gcs_path,
|
||||
content_type='audio/flac' # flacfetch typically returns FLAC
|
||||
)
|
||||
|
||||
logger.info(f"Uploaded audio to GCS: {audio_gcs_path}")
|
||||
|
||||
# Clean up temp file
|
||||
try:
|
||||
os.remove(result.filepath)
|
||||
os.rmdir(temp_dir)
|
||||
except Exception as e:
|
||||
logger.warning(f"Failed to clean up temp files: {e}")
|
||||
|
||||
# Update job with GCS path and transition to DOWNLOADING
|
||||
job_manager.update_job(job_id, {
|
||||
'input_media_gcs_path': audio_gcs_path,
|
||||
'filename': filename,
|
||||
})
|
||||
|
||||
job_manager.transition_to_state(
|
||||
job_id=job_id,
|
||||
new_status=JobStatus.DOWNLOADING,
|
||||
progress=15,
|
||||
message="Audio downloaded, starting processing"
|
||||
)
|
||||
|
||||
# Trigger workers
|
||||
background_tasks.add_task(_trigger_workers_parallel, job_id)
|
||||
|
||||
return {
|
||||
'selected_index': selection_index,
|
||||
'selected_title': selected['title'],
|
||||
'selected_artist': selected['artist'],
|
||||
'selected_provider': selected['provider'],
|
||||
}
|
||||
|
||||
except DownloadError as e:
|
||||
job_manager.fail_job(job_id, f"Audio download failed: {e}")
|
||||
raise HTTPException(status_code=500, detail=f"Download failed: {e}")
|
||||
except Exception as e:
|
||||
job_manager.fail_job(job_id, f"Audio download failed: {e}")
|
||||
raise HTTPException(status_code=500, detail=f"Download failed: {e}")
|
||||
|
||||
|
||||
@router.post("/audio-search/search", response_model=AudioSearchResponse)
|
||||
async def search_audio(
|
||||
request: Request,
|
||||
background_tasks: BackgroundTasks,
|
||||
body: AudioSearchRequest,
|
||||
auth_data: Tuple[str, UserType, int] = Depends(require_auth)
|
||||
):
|
||||
"""
|
||||
Search for audio by artist and title, creating a new job.
|
||||
|
||||
This endpoint:
|
||||
1. Creates a job in PENDING state
|
||||
2. Searches for audio using flacfetch
|
||||
3. Either returns search results for user selection, or
|
||||
4. If auto_download=True, automatically selects best and starts processing
|
||||
|
||||
Use cases:
|
||||
- Interactive mode (default): Returns results, user calls /select endpoint
|
||||
- Auto mode (auto_download=True): Automatically selects and downloads best
|
||||
"""
|
||||
try:
|
||||
# Apply default distribution settings
|
||||
settings = get_settings()
|
||||
effective_dropbox_path = body.dropbox_path or settings.default_dropbox_path
|
||||
effective_gdrive_folder_id = body.gdrive_folder_id or settings.default_gdrive_folder_id
|
||||
effective_discord_webhook_url = body.discord_webhook_url or settings.default_discord_webhook_url
|
||||
|
||||
# Apply defaults for YouTube/Dropbox distribution (for web service)
|
||||
effective_enable_youtube_upload = body.enable_youtube_upload or settings.default_enable_youtube_upload
|
||||
effective_brand_prefix = body.brand_prefix or settings.default_brand_prefix
|
||||
effective_youtube_description = body.youtube_description or settings.default_youtube_description
|
||||
|
||||
# Validate credentials if distribution services are requested
|
||||
invalid_services = []
|
||||
credential_manager = get_credential_manager()
|
||||
|
||||
if effective_enable_youtube_upload:
|
||||
result = credential_manager.check_youtube_credentials()
|
||||
if result.status != CredentialStatus.VALID:
|
||||
invalid_services.append(f"youtube ({result.message})")
|
||||
|
||||
if effective_dropbox_path:
|
||||
result = credential_manager.check_dropbox_credentials()
|
||||
if result.status != CredentialStatus.VALID:
|
||||
invalid_services.append(f"dropbox ({result.message})")
|
||||
|
||||
if effective_gdrive_folder_id:
|
||||
result = credential_manager.check_gdrive_credentials()
|
||||
if result.status != CredentialStatus.VALID:
|
||||
invalid_services.append(f"gdrive ({result.message})")
|
||||
|
||||
if invalid_services:
|
||||
raise HTTPException(
|
||||
status_code=400,
|
||||
detail={
|
||||
"error": "credentials_invalid",
|
||||
"message": f"Distribution services need re-authorization: {', '.join(invalid_services)}",
|
||||
"invalid_services": invalid_services,
|
||||
}
|
||||
)
|
||||
|
||||
# Extract request metadata
|
||||
request_metadata = extract_request_metadata(request, created_from="audio_search")
|
||||
|
||||
# Resolve CDG/TXT defaults based on theme
|
||||
resolved_cdg, resolved_txt = _resolve_cdg_txt_defaults(
|
||||
body.theme_id, body.enable_cdg, body.enable_txt
|
||||
)
|
||||
|
||||
# Create job
|
||||
job_create = JobCreate(
|
||||
artist=body.artist,
|
||||
title=body.title,
|
||||
theme_id=body.theme_id,
|
||||
color_overrides=body.color_overrides or {},
|
||||
enable_cdg=resolved_cdg,
|
||||
enable_txt=resolved_txt,
|
||||
brand_prefix=effective_brand_prefix,
|
||||
enable_youtube_upload=effective_enable_youtube_upload,
|
||||
youtube_description=effective_youtube_description,
|
||||
youtube_description_template=effective_youtube_description, # video_worker reads this field
|
||||
discord_webhook_url=effective_discord_webhook_url,
|
||||
dropbox_path=effective_dropbox_path,
|
||||
gdrive_folder_id=effective_gdrive_folder_id,
|
||||
lyrics_artist=body.lyrics_artist,
|
||||
lyrics_title=body.lyrics_title,
|
||||
subtitle_offset_ms=body.subtitle_offset_ms,
|
||||
clean_instrumental_model=body.clean_instrumental_model,
|
||||
backing_vocals_models=body.backing_vocals_models,
|
||||
other_stems_models=body.other_stems_models,
|
||||
audio_search_artist=body.artist,
|
||||
audio_search_title=body.title,
|
||||
auto_download=body.auto_download,
|
||||
request_metadata=request_metadata,
|
||||
)
|
||||
job = job_manager.create_job(job_create)
|
||||
job_id = job.job_id
|
||||
|
||||
logger.info(f"Created job {job_id} for audio search: {body.artist} - {body.title}")
|
||||
|
||||
# Update job with audio search fields
|
||||
job_manager.update_job(job_id, {
|
||||
'audio_search_artist': body.artist,
|
||||
'audio_search_title': body.title,
|
||||
'auto_download': body.auto_download,
|
||||
})
|
||||
|
||||
# If theme is set and no custom style files are being uploaded, prepare theme style now
|
||||
# This copies the theme's style_params.json to the job folder so LyricsTranscriber
|
||||
# can access the style configuration for preview videos
|
||||
if body.theme_id and not body.style_files:
|
||||
from backend.api.routes.file_upload import _prepare_theme_for_job
|
||||
try:
|
||||
style_params_path, theme_style_assets, youtube_desc = _prepare_theme_for_job(
|
||||
job_id, body.theme_id, body.color_overrides
|
||||
)
|
||||
theme_update = {
|
||||
'style_params_gcs_path': style_params_path,
|
||||
'style_assets': theme_style_assets,
|
||||
}
|
||||
if youtube_desc and not effective_youtube_description:
|
||||
theme_update['youtube_description_template'] = youtube_desc
|
||||
job_manager.update_job(job_id, theme_update)
|
||||
logger.info(f"Applied theme '{body.theme_id}' to job {job_id}")
|
||||
except Exception as e:
|
||||
logger.warning(f"Failed to prepare theme '{body.theme_id}' for job {job_id}: {e}")
|
||||
# Continue without theme - job can still be processed with defaults
|
||||
|
||||
# Handle style file uploads if provided
|
||||
style_upload_urls: List[StyleUploadUrl] = []
|
||||
style_assets = {}
|
||||
|
||||
if body.style_files:
|
||||
logger.info(f"Processing {len(body.style_files)} style file uploads for job {job_id}")
|
||||
|
||||
for file_info in body.style_files:
|
||||
# Validate file type
|
||||
if file_info.file_type not in STYLE_FILE_TYPES:
|
||||
raise HTTPException(
|
||||
status_code=400,
|
||||
detail=f"Invalid file type '{file_info.file_type}'. Must be one of: {', '.join(STYLE_FILE_TYPES.keys())}"
|
||||
)
|
||||
|
||||
# Validate extension
|
||||
ext = Path(file_info.filename).suffix.lower()
|
||||
allowed_exts = STYLE_FILE_TYPES[file_info.file_type]
|
||||
if ext not in allowed_exts:
|
||||
raise HTTPException(
|
||||
status_code=400,
|
||||
detail=f"Invalid extension '{ext}' for {file_info.file_type}. Allowed: {', '.join(allowed_exts)}"
|
||||
)
|
||||
|
||||
# Generate GCS path
|
||||
if file_info.file_type == 'style_params':
|
||||
gcs_path = f"uploads/{job_id}/style/style_params.json"
|
||||
else:
|
||||
# style_intro_background -> intro_background, etc.
|
||||
asset_key = file_info.file_type.replace('style_', '')
|
||||
gcs_path = f"uploads/{job_id}/style/{asset_key}{ext}"
|
||||
|
||||
# Generate signed upload URL
|
||||
signed_url = storage_service.generate_signed_upload_url(
|
||||
gcs_path,
|
||||
content_type=file_info.content_type,
|
||||
expiration_minutes=60
|
||||
)
|
||||
|
||||
style_upload_urls.append(StyleUploadUrl(
|
||||
file_type=file_info.file_type,
|
||||
gcs_path=gcs_path,
|
||||
upload_url=signed_url
|
||||
))
|
||||
|
||||
# Track the expected asset paths
|
||||
if file_info.file_type == 'style_params':
|
||||
style_assets['style_params'] = gcs_path
|
||||
else:
|
||||
asset_key = file_info.file_type.replace('style_', '')
|
||||
style_assets[asset_key] = gcs_path
|
||||
|
||||
# Update job with style asset expectations
|
||||
style_update = {'style_assets': style_assets}
|
||||
if 'style_params' in style_assets:
|
||||
style_update['style_params_gcs_path'] = style_assets['style_params']
|
||||
job_manager.update_job(job_id, style_update)
|
||||
|
||||
logger.info(f"Generated {len(style_upload_urls)} style upload URLs for job {job_id}")
|
||||
|
||||
# Transition to searching state
|
||||
job_manager.transition_to_state(
|
||||
job_id=job_id,
|
||||
new_status=JobStatus.SEARCHING_AUDIO,
|
||||
progress=5,
|
||||
message=f"Searching for: {body.artist} - {body.title}"
|
||||
)
|
||||
|
||||
# Perform search
|
||||
audio_search_service = get_audio_search_service()
|
||||
|
||||
try:
|
||||
search_results = audio_search_service.search(body.artist, body.title)
|
||||
except NoResultsError as e:
|
||||
job_manager.fail_job(job_id, f"No audio sources found for: {body.artist} - {body.title}")
|
||||
raise HTTPException(
|
||||
status_code=404,
|
||||
detail={
|
||||
"error": "no_results",
|
||||
"message": str(e),
|
||||
"job_id": job_id,
|
||||
}
|
||||
)
|
||||
except AudioSearchError as e:
|
||||
job_manager.fail_job(job_id, f"Audio search failed: {e}")
|
||||
raise HTTPException(status_code=500, detail=f"Search failed: {e}")
|
||||
|
||||
# Store results in job state_data, including remote_search_id if available
|
||||
results_dicts = [r.to_dict() for r in search_results]
|
||||
state_data_update = {
|
||||
'audio_search_results': results_dicts,
|
||||
'audio_search_count': len(results_dicts),
|
||||
}
|
||||
# Store remote_search_id for use during download (important for concurrent requests)
|
||||
if audio_search_service.last_remote_search_id:
|
||||
state_data_update['remote_search_id'] = audio_search_service.last_remote_search_id
|
||||
job_manager.update_job(job_id, {
|
||||
'state_data': state_data_update
|
||||
})
|
||||
|
||||
# If auto_download, select best and start processing
|
||||
if body.auto_download:
|
||||
best_index = audio_search_service.select_best(search_results)
|
||||
|
||||
logger.info(f"Auto-download enabled, selecting result {best_index}")
|
||||
|
||||
selection_info = await _download_and_start_processing(
|
||||
job_id=job_id,
|
||||
selection_index=best_index,
|
||||
audio_search_service=audio_search_service,
|
||||
background_tasks=background_tasks,
|
||||
)
|
||||
|
||||
return AudioSearchResponse(
|
||||
status="success",
|
||||
job_id=job_id,
|
||||
message=f"Audio found and download started: {selection_info['selected_artist']} - {selection_info['selected_title']} ({selection_info['selected_provider']})",
|
||||
results=None, # Don't return results in auto mode
|
||||
results_count=len(search_results),
|
||||
auto_download=True,
|
||||
server_version=VERSION,
|
||||
style_upload_urls=style_upload_urls if style_upload_urls else None,
|
||||
)
|
||||
|
||||
# Interactive mode: return results for user selection
|
||||
job_manager.transition_to_state(
|
||||
job_id=job_id,
|
||||
new_status=JobStatus.AWAITING_AUDIO_SELECTION,
|
||||
progress=10,
|
||||
message=f"Found {len(search_results)} audio sources. Waiting for selection."
|
||||
)
|
||||
|
||||
# Convert to response format with full Release data for rich display
|
||||
result_responses = []
|
||||
for r in search_results:
|
||||
# Get full serialized data from raw_result if available
|
||||
# raw_result can be either:
|
||||
# - A dict (from remote flacfetch API)
|
||||
# - A Release object (from local flacfetch)
|
||||
raw_dict = {}
|
||||
if r.raw_result:
|
||||
if isinstance(r.raw_result, dict):
|
||||
# Remote flacfetch API returns dicts directly
|
||||
raw_dict = r.raw_result
|
||||
else:
|
||||
# Local flacfetch returns Release objects
|
||||
try:
|
||||
raw_dict = r.raw_result.to_dict()
|
||||
except AttributeError:
|
||||
pass # Not a Release object
|
||||
|
||||
result_responses.append(
|
||||
AudioSearchResultResponse(
|
||||
index=r.index,
|
||||
title=r.title,
|
||||
artist=r.artist,
|
||||
provider=r.provider,
|
||||
url=r.url,
|
||||
duration=r.duration,
|
||||
quality=r.quality,
|
||||
source_id=r.source_id,
|
||||
seeders=raw_dict.get('seeders') or r.seeders,
|
||||
target_file=raw_dict.get('target_file') or r.target_file,
|
||||
# Additional fields for rich display
|
||||
year=raw_dict.get('year'),
|
||||
label=raw_dict.get('label'),
|
||||
edition_info=raw_dict.get('edition_info'),
|
||||
release_type=raw_dict.get('release_type'),
|
||||
channel=raw_dict.get('channel'),
|
||||
view_count=raw_dict.get('view_count'),
|
||||
size_bytes=raw_dict.get('size_bytes'),
|
||||
target_file_size=raw_dict.get('target_file_size'),
|
||||
track_pattern=raw_dict.get('track_pattern'),
|
||||
match_score=raw_dict.get('match_score'),
|
||||
# Pre-computed display fields
|
||||
formatted_size=raw_dict.get('formatted_size'),
|
||||
formatted_duration=raw_dict.get('formatted_duration'),
|
||||
formatted_views=raw_dict.get('formatted_views'),
|
||||
is_lossless=raw_dict.get('is_lossless'),
|
||||
quality_str=raw_dict.get('quality_str'),
|
||||
# Full quality object for Release.from_dict()
|
||||
# Remote API uses 'quality_data', local uses 'quality' for the dict
|
||||
quality_data=raw_dict.get('quality_data') or raw_dict.get('quality'),
|
||||
)
|
||||
)
|
||||
|
||||
return AudioSearchResponse(
|
||||
status="awaiting_selection",
|
||||
job_id=job_id,
|
||||
message=f"Found {len(search_results)} audio sources. Call /api/audio-search/{job_id}/select to choose one.",
|
||||
results=result_responses,
|
||||
results_count=len(search_results),
|
||||
auto_download=False,
|
||||
server_version=VERSION,
|
||||
style_upload_urls=style_upload_urls if style_upload_urls else None,
|
||||
)
|
||||
|
||||
except HTTPException:
|
||||
raise
|
||||
except Exception as e:
|
||||
logger.error(f"Error in audio search: {e}", exc_info=True)
|
||||
raise HTTPException(status_code=500, detail=str(e))
|
||||
|
||||
|
||||
@router.get("/audio-search/{job_id}/results")
|
||||
async def get_audio_search_results(
|
||||
job_id: str,
|
||||
auth_data: Tuple[str, UserType, int] = Depends(require_auth)
|
||||
):
|
||||
"""
|
||||
Get audio search results for a job.
|
||||
|
||||
Returns the cached search results so user can select one.
|
||||
"""
|
||||
job = job_manager.get_job(job_id)
|
||||
if not job:
|
||||
raise HTTPException(status_code=404, detail=f"Job {job_id} not found")
|
||||
|
||||
search_results = job.state_data.get('audio_search_results', [])
|
||||
|
||||
if not search_results:
|
||||
raise HTTPException(
|
||||
status_code=400,
|
||||
detail="No search results available for this job"
|
||||
)
|
||||
|
||||
return {
|
||||
"status": "success",
|
||||
"job_id": job_id,
|
||||
"job_status": job.status,
|
||||
"artist": job.audio_search_artist or job.artist,
|
||||
"title": job.audio_search_title or job.title,
|
||||
"results": search_results,
|
||||
"results_count": len(search_results),
|
||||
}
|
||||
|
||||
|
||||
@router.post("/audio-search/{job_id}/select", response_model=AudioSelectResponse)
|
||||
async def select_audio_source(
|
||||
job_id: str,
|
||||
background_tasks: BackgroundTasks,
|
||||
body: AudioSelectRequest,
|
||||
auth_data: Tuple[str, UserType, int] = Depends(require_auth)
|
||||
):
|
||||
"""
|
||||
Select an audio source and start job processing.
|
||||
|
||||
This endpoint:
|
||||
1. Validates the job is awaiting selection
|
||||
2. Downloads the selected audio
|
||||
3. Uploads to GCS
|
||||
4. Triggers audio and lyrics workers
|
||||
"""
|
||||
job = job_manager.get_job(job_id)
|
||||
if not job:
|
||||
raise HTTPException(status_code=404, detail=f"Job {job_id} not found")
|
||||
|
||||
# Verify job is awaiting selection
|
||||
if job.status != JobStatus.AWAITING_AUDIO_SELECTION:
|
||||
raise HTTPException(
|
||||
status_code=400,
|
||||
detail=f"Job is not awaiting audio selection (status: {job.status})"
|
||||
)
|
||||
|
||||
# Get search service instance
|
||||
# Note: With download_by_id, we no longer need to re-search to populate the cache.
|
||||
# The source_id stored in job.state_data['audio_search_results'] is sufficient.
|
||||
audio_search_service = get_audio_search_service()
|
||||
|
||||
# Validate search results exist in job state_data
|
||||
search_results = job.state_data.get('audio_search_results', [])
|
||||
if not search_results:
|
||||
raise HTTPException(status_code=400, detail="No search results cached for this job")
|
||||
|
||||
selection_info = await _download_and_start_processing(
|
||||
job_id=job_id,
|
||||
selection_index=body.selection_index,
|
||||
audio_search_service=audio_search_service,
|
||||
background_tasks=background_tasks,
|
||||
)
|
||||
|
||||
return AudioSelectResponse(
|
||||
status="success",
|
||||
job_id=job_id,
|
||||
message="Audio selected and download started",
|
||||
selected_index=selection_info['selected_index'],
|
||||
selected_title=selection_info['selected_title'],
|
||||
selected_artist=selection_info['selected_artist'],
|
||||
selected_provider=selection_info['selected_provider'],
|
||||
)
|
||||
|
||||
|
||||
@@ -1,348 +0,0 @@
|
||||
"""
|
||||
OAuth credential management API routes.
|
||||
|
||||
Provides endpoints for:
|
||||
1. Checking credential status
|
||||
2. Device authorization flow for re-authentication
|
||||
3. Credential validation before job submission
|
||||
"""
|
||||
import logging
|
||||
from typing import Optional
|
||||
from datetime import datetime
|
||||
from fastapi import APIRouter, HTTPException, BackgroundTasks
|
||||
from pydantic import BaseModel
|
||||
|
||||
from backend.services.credential_manager import (
|
||||
get_credential_manager,
|
||||
CredentialStatus,
|
||||
CredentialCheckResult,
|
||||
)
|
||||
from backend.config import get_settings
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
router = APIRouter(prefix="/auth", tags=["auth"])
|
||||
|
||||
|
||||
# ===========================================================================
|
||||
# Request/Response Models
|
||||
# ===========================================================================
|
||||
|
||||
class CredentialStatusResponse(BaseModel):
|
||||
"""Response model for credential status."""
|
||||
service: str
|
||||
status: str
|
||||
message: str
|
||||
last_checked: datetime
|
||||
expires_at: Optional[datetime] = None
|
||||
|
||||
class Config:
|
||||
from_attributes = True
|
||||
|
||||
|
||||
class AllCredentialsStatusResponse(BaseModel):
|
||||
"""Response model for all credentials status."""
|
||||
youtube: CredentialStatusResponse
|
||||
gdrive: CredentialStatusResponse
|
||||
dropbox: CredentialStatusResponse
|
||||
all_valid: bool
|
||||
services_needing_auth: list[str]
|
||||
|
||||
|
||||
class DeviceAuthStartRequest(BaseModel):
|
||||
"""Request to start device authorization flow."""
|
||||
client_id: Optional[str] = None # Optional - reads from Secret Manager if not provided
|
||||
client_secret: Optional[str] = None # Optional - reads from Secret Manager if not provided
|
||||
|
||||
|
||||
class DeviceAuthStartResponse(BaseModel):
|
||||
"""Response with device authorization info."""
|
||||
device_code: str
|
||||
user_code: str
|
||||
verification_url: str
|
||||
expires_in: int
|
||||
interval: int
|
||||
instructions: str
|
||||
|
||||
|
||||
class DeviceAuthPollResponse(BaseModel):
|
||||
"""Response from polling device authorization."""
|
||||
status: str # "pending", "complete", "expired", "error"
|
||||
message: Optional[str] = None
|
||||
credentials_saved: bool = False
|
||||
|
||||
|
||||
class CredentialValidationRequest(BaseModel):
|
||||
"""Request to validate credentials for specific services."""
|
||||
youtube: bool = False
|
||||
gdrive: bool = False
|
||||
dropbox: bool = False
|
||||
|
||||
|
||||
class CredentialValidationResponse(BaseModel):
|
||||
"""Response from credential validation."""
|
||||
valid: bool
|
||||
invalid_services: list[str]
|
||||
message: str
|
||||
|
||||
|
||||
# ===========================================================================
|
||||
# Endpoints
|
||||
# ===========================================================================
|
||||
|
||||
@router.get("/status", response_model=AllCredentialsStatusResponse)
|
||||
async def get_credentials_status():
|
||||
"""
|
||||
Get the status of all OAuth credentials.
|
||||
|
||||
Returns the validation status for YouTube, Google Drive, and Dropbox
|
||||
credentials, including whether they are valid, expired, or need
|
||||
re-authorization.
|
||||
"""
|
||||
manager = get_credential_manager()
|
||||
results = manager.check_all_credentials()
|
||||
|
||||
def to_response(result: CredentialCheckResult) -> CredentialStatusResponse:
|
||||
return CredentialStatusResponse(
|
||||
service=result.service,
|
||||
status=result.status.value,
|
||||
message=result.message,
|
||||
last_checked=result.last_checked,
|
||||
expires_at=result.expires_at
|
||||
)
|
||||
|
||||
services_needing_auth = [
|
||||
name for name, result in results.items()
|
||||
if result.status in (CredentialStatus.INVALID, CredentialStatus.EXPIRED, CredentialStatus.NOT_CONFIGURED)
|
||||
]
|
||||
|
||||
return AllCredentialsStatusResponse(
|
||||
youtube=to_response(results["youtube"]),
|
||||
gdrive=to_response(results["gdrive"]),
|
||||
dropbox=to_response(results["dropbox"]),
|
||||
all_valid=len(services_needing_auth) == 0,
|
||||
services_needing_auth=services_needing_auth
|
||||
)
|
||||
|
||||
|
||||
@router.get("/status/{service}", response_model=CredentialStatusResponse)
|
||||
async def get_service_credential_status(service: str):
|
||||
"""
|
||||
Get the status of a specific service's OAuth credentials.
|
||||
|
||||
Args:
|
||||
service: One of "youtube", "gdrive", "dropbox"
|
||||
"""
|
||||
manager = get_credential_manager()
|
||||
|
||||
if service == "youtube":
|
||||
result = manager.check_youtube_credentials()
|
||||
elif service == "gdrive":
|
||||
result = manager.check_gdrive_credentials()
|
||||
elif service == "dropbox":
|
||||
result = manager.check_dropbox_credentials()
|
||||
else:
|
||||
raise HTTPException(status_code=400, detail=f"Unknown service: {service}")
|
||||
|
||||
return CredentialStatusResponse(
|
||||
service=result.service,
|
||||
status=result.status.value,
|
||||
message=result.message,
|
||||
last_checked=result.last_checked,
|
||||
expires_at=result.expires_at
|
||||
)
|
||||
|
||||
|
||||
@router.post("/validate", response_model=CredentialValidationResponse)
|
||||
async def validate_credentials(request: CredentialValidationRequest):
|
||||
"""
|
||||
Validate that credentials are available for requested services.
|
||||
|
||||
Use this before submitting a job to ensure all required
|
||||
credentials are valid.
|
||||
"""
|
||||
manager = get_credential_manager()
|
||||
invalid_services = []
|
||||
|
||||
if request.youtube:
|
||||
result = manager.check_youtube_credentials()
|
||||
if result.status != CredentialStatus.VALID:
|
||||
invalid_services.append("youtube")
|
||||
|
||||
if request.gdrive:
|
||||
result = manager.check_gdrive_credentials()
|
||||
if result.status != CredentialStatus.VALID:
|
||||
invalid_services.append("gdrive")
|
||||
|
||||
if request.dropbox:
|
||||
result = manager.check_dropbox_credentials()
|
||||
if result.status != CredentialStatus.VALID:
|
||||
invalid_services.append("dropbox")
|
||||
|
||||
if invalid_services:
|
||||
return CredentialValidationResponse(
|
||||
valid=False,
|
||||
invalid_services=invalid_services,
|
||||
message=f"The following services need re-authorization: {', '.join(invalid_services)}"
|
||||
)
|
||||
|
||||
return CredentialValidationResponse(
|
||||
valid=True,
|
||||
invalid_services=[],
|
||||
message="All requested credentials are valid"
|
||||
)
|
||||
|
||||
|
||||
# ===========================================================================
|
||||
# Device Authorization Flow Endpoints
|
||||
# ===========================================================================
|
||||
|
||||
@router.post("/youtube/device", response_model=DeviceAuthStartResponse)
|
||||
async def start_youtube_device_auth(request: Optional[DeviceAuthStartRequest] = None):
|
||||
"""
|
||||
Start YouTube device authorization flow.
|
||||
|
||||
This initiates a device auth flow that allows authorization
|
||||
from any device. The user must visit the verification URL
|
||||
and enter the user code.
|
||||
|
||||
Client credentials are loaded from Secret Manager ('youtube-client-credentials')
|
||||
unless explicitly provided in the request body.
|
||||
|
||||
After starting, poll /auth/youtube/device/{device_code} to
|
||||
check for completion.
|
||||
"""
|
||||
manager = get_credential_manager()
|
||||
|
||||
try:
|
||||
device_info = manager.start_youtube_device_auth(
|
||||
client_id=request.client_id if request else None,
|
||||
client_secret=request.client_secret if request else None
|
||||
)
|
||||
|
||||
return DeviceAuthStartResponse(
|
||||
device_code=device_info.device_code,
|
||||
user_code=device_info.user_code,
|
||||
verification_url=device_info.verification_url,
|
||||
expires_in=device_info.expires_in,
|
||||
interval=device_info.interval,
|
||||
instructions=f"Visit {device_info.verification_url} and enter code: {device_info.user_code}"
|
||||
)
|
||||
|
||||
except Exception as e:
|
||||
logger.error(f"Failed to start YouTube device auth: {e}")
|
||||
raise HTTPException(status_code=500, detail=str(e))
|
||||
|
||||
|
||||
@router.get("/youtube/device/{device_code}", response_model=DeviceAuthPollResponse)
|
||||
async def poll_youtube_device_auth(device_code: str):
|
||||
"""
|
||||
Poll for YouTube device authorization completion.
|
||||
|
||||
Call this endpoint at the interval specified in the device auth
|
||||
start response. Keep polling while status is "pending".
|
||||
|
||||
Status values:
|
||||
- pending: User has not yet authorized
|
||||
- complete: Authorization successful, credentials saved
|
||||
- expired: Device code expired, start new flow
|
||||
- error: An error occurred
|
||||
"""
|
||||
manager = get_credential_manager()
|
||||
|
||||
status, data = manager.poll_device_auth("youtube", device_code)
|
||||
|
||||
return DeviceAuthPollResponse(
|
||||
status=status,
|
||||
message=data.get("message") if data else None,
|
||||
credentials_saved=(status == "complete")
|
||||
)
|
||||
|
||||
|
||||
@router.post("/gdrive/device", response_model=DeviceAuthStartResponse)
|
||||
async def start_gdrive_device_auth(request: Optional[DeviceAuthStartRequest] = None):
|
||||
"""
|
||||
Start Google Drive device authorization flow.
|
||||
|
||||
Client credentials are loaded from Secret Manager ('gdrive-client-credentials')
|
||||
unless explicitly provided in the request body.
|
||||
|
||||
After starting, poll /auth/gdrive/device/{device_code} to check for completion.
|
||||
"""
|
||||
manager = get_credential_manager()
|
||||
|
||||
try:
|
||||
device_info = manager.start_gdrive_device_auth(
|
||||
client_id=request.client_id if request else None,
|
||||
client_secret=request.client_secret if request else None
|
||||
)
|
||||
|
||||
return DeviceAuthStartResponse(
|
||||
device_code=device_info.device_code,
|
||||
user_code=device_info.user_code,
|
||||
verification_url=device_info.verification_url,
|
||||
expires_in=device_info.expires_in,
|
||||
interval=device_info.interval,
|
||||
instructions=f"Visit {device_info.verification_url} and enter code: {device_info.user_code}"
|
||||
)
|
||||
|
||||
except Exception as e:
|
||||
logger.error(f"Failed to start Google Drive device auth: {e}")
|
||||
raise HTTPException(status_code=500, detail=str(e))
|
||||
|
||||
|
||||
@router.get("/gdrive/device/{device_code}", response_model=DeviceAuthPollResponse)
|
||||
async def poll_gdrive_device_auth(device_code: str):
|
||||
"""
|
||||
Poll for Google Drive device authorization completion.
|
||||
"""
|
||||
manager = get_credential_manager()
|
||||
|
||||
status, data = manager.poll_device_auth("gdrive", device_code)
|
||||
|
||||
return DeviceAuthPollResponse(
|
||||
status=status,
|
||||
message=data.get("message") if data else None,
|
||||
credentials_saved=(status == "complete")
|
||||
)
|
||||
|
||||
|
||||
# ===========================================================================
|
||||
# Alert Testing
|
||||
# ===========================================================================
|
||||
|
||||
@router.post("/test-alert")
|
||||
async def test_credential_alert(background_tasks: BackgroundTasks):
|
||||
"""
|
||||
Test the credential alert mechanism.
|
||||
|
||||
Sends a test alert to Discord if configured.
|
||||
"""
|
||||
settings = get_settings()
|
||||
discord_url = settings.get_secret("discord-alert-webhook")
|
||||
|
||||
if not discord_url:
|
||||
raise HTTPException(
|
||||
status_code=400,
|
||||
detail="Discord alert webhook not configured"
|
||||
)
|
||||
|
||||
manager = get_credential_manager()
|
||||
|
||||
# Create a fake invalid result for testing
|
||||
from backend.services.credential_manager import CredentialCheckResult
|
||||
test_results = [
|
||||
CredentialCheckResult(
|
||||
service="test",
|
||||
status=CredentialStatus.INVALID,
|
||||
message="This is a test alert",
|
||||
last_checked=datetime.utcnow()
|
||||
)
|
||||
]
|
||||
|
||||
background_tasks.add_task(
|
||||
manager.send_credential_alert,
|
||||
test_results,
|
||||
discord_url
|
||||
)
|
||||
|
||||
return {"message": "Test alert queued"}
|
||||
File diff suppressed because it is too large.
Load diff
@@ -1,141 +0,0 @@
|
||||
"""
|
||||
Health check routes.
|
||||
"""
|
||||
import os
|
||||
import logging
|
||||
from fastapi import APIRouter
|
||||
from typing import Dict, Any, Optional
|
||||
|
||||
from backend.version import VERSION
|
||||
from backend.services.flacfetch_client import get_flacfetch_client
|
||||
|
||||
router = APIRouter()
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
|
||||
def check_transmission_status() -> Dict[str, Any]:
|
||||
"""Check if Transmission daemon is available and responsive."""
|
||||
try:
|
||||
import transmission_rpc
|
||||
host = os.environ.get("TRANSMISSION_HOST", "localhost")
|
||||
port = int(os.environ.get("TRANSMISSION_PORT", "9091"))
|
||||
|
||||
client = transmission_rpc.Client(host=host, port=port, timeout=5)
|
||||
stats = client.session_stats()
|
||||
|
||||
# Get detailed torrent info
|
||||
torrents = client.get_torrents()
|
||||
torrent_details = []
|
||||
for t in torrents:
|
||||
torrent_info = {
|
||||
"name": t.name,
|
||||
"progress": round(t.progress, 1),
|
||||
"status": str(t.status),
|
||||
"peers": t.peers_connected if hasattr(t, 'peers_connected') else 0,
|
||||
"download_speed": round(t.rate_download / 1024, 1) if hasattr(t, 'rate_download') else 0, # KB/s
|
||||
}
|
||||
# Check if stalled (downloading but no progress and no peers)
|
||||
if t.status.downloading and t.progress < 100 and torrent_info['peers'] == 0:
|
||||
torrent_info['stalled'] = True
|
||||
torrent_details.append(torrent_info)
|
||||
|
||||
return {
|
||||
"available": True,
|
||||
"host": host,
|
||||
"port": port,
|
||||
"download_dir": stats.download_dir if hasattr(stats, 'download_dir') else None,
|
||||
"active_torrent_count": stats.active_torrent_count if hasattr(stats, 'active_torrent_count') else 0,
|
||||
"torrents": torrent_details,
|
||||
}
|
||||
except ImportError as e:
|
||||
return {
|
||||
"available": False,
|
||||
"error": f"transmission_rpc not installed: {e}",
|
||||
}
|
||||
except Exception as e:
|
||||
return {
|
||||
"available": False,
|
||||
"host": os.environ.get("TRANSMISSION_HOST", "localhost"),
|
||||
"port": int(os.environ.get("TRANSMISSION_PORT", "9091")),
|
||||
"error": str(e),
|
||||
}
|
||||
|
||||
|
||||
@router.get("/health")
|
||||
async def health_check() -> Dict[str, str]:
|
||||
"""Health check endpoint."""
|
||||
return {
|
||||
"status": "healthy",
|
||||
"service": "karaoke-gen-backend"
|
||||
}
|
||||
|
||||
|
||||
async def check_flacfetch_service_status() -> Dict[str, Any]:
|
||||
"""Check if remote flacfetch service is available and healthy."""
|
||||
client = get_flacfetch_client()
|
||||
|
||||
if not client:
|
||||
return {
|
||||
"configured": False,
|
||||
"message": "Remote flacfetch service not configured (FLACFETCH_API_URL not set)",
|
||||
}
|
||||
|
||||
try:
|
||||
health = await client.health_check()
|
||||
return {
|
||||
"configured": True,
|
||||
"available": True,
|
||||
"status": health.get("status"),
|
||||
"version": health.get("version"),
|
||||
"transmission": health.get("transmission", {}),
|
||||
"disk": health.get("disk", {}),
|
||||
"providers": health.get("providers", {}),
|
||||
}
|
||||
except Exception as e:
|
||||
return {
|
||||
"configured": True,
|
||||
"available": False,
|
||||
"error": str(e),
|
||||
}
|
||||
|
||||
|
||||
@router.get("/health/detailed")
|
||||
async def detailed_health_check() -> Dict[str, Any]:
|
||||
"""
|
||||
Detailed health check including dependencies.
|
||||
|
||||
Use this to debug issues with Transmission, flacfetch service, etc.
|
||||
"""
|
||||
transmission_status = check_transmission_status()
|
||||
flacfetch_status = await check_flacfetch_service_status()
|
||||
|
||||
# Log for debugging
|
||||
if not transmission_status.get("available"):
|
||||
logger.warning(f"Local Transmission not available: {transmission_status.get('error')}")
|
||||
else:
|
||||
logger.info(f"Local Transmission available at {transmission_status.get('host')}:{transmission_status.get('port')}")
|
||||
|
||||
if flacfetch_status.get("configured") and not flacfetch_status.get("available"):
|
||||
logger.warning(f"Remote flacfetch service not available: {flacfetch_status.get('error')}")
|
||||
elif flacfetch_status.get("available"):
|
||||
logger.info(f"Remote flacfetch service healthy: {flacfetch_status.get('status')}")
|
||||
|
||||
return {
|
||||
"status": "healthy",
|
||||
"service": "karaoke-gen-backend",
|
||||
"version": VERSION,
|
||||
"dependencies": {
|
||||
"transmission_local": transmission_status,
|
||||
"flacfetch_remote": flacfetch_status,
|
||||
}
|
||||
}
|
||||
|
||||
|
||||
@router.get("/readiness")
|
||||
async def readiness_check() -> Dict[str, str]:
|
||||
"""Readiness check endpoint for Cloud Run."""
|
||||
return {
|
||||
"status": "ready",
|
||||
"service": "karaoke-gen-backend"
|
||||
}
|
||||
|
||||
@@ -1,330 +0,0 @@
|
||||
"""
|
||||
Internal API routes for worker coordination.
|
||||
|
||||
These endpoints are for internal use only (backend → workers).
|
||||
They are protected by admin authentication.
|
||||
|
||||
With Cloud Tasks integration, these endpoints may be called multiple times
|
||||
(retry on failure). Idempotency checks prevent duplicate processing.
|
||||
|
||||
Observability:
|
||||
- Extracts trace context from incoming requests (propagated via Cloud Tasks)
|
||||
- Creates worker spans linked to the original request trace
|
||||
- All logs include job_id for easy filtering in Cloud Logging
|
||||
"""
|
||||
import logging
|
||||
import asyncio
|
||||
import time
|
||||
from typing import Tuple, Optional
|
||||
from fastapi import APIRouter, HTTPException, BackgroundTasks, Depends, Request
|
||||
from pydantic import BaseModel
|
||||
|
||||
from backend.workers.audio_worker import process_audio_separation
|
||||
from backend.workers.lyrics_worker import process_lyrics_transcription
|
||||
from backend.workers.screens_worker import generate_screens
|
||||
from backend.workers.video_worker import generate_video
|
||||
from backend.workers.render_video_worker import process_render_video
|
||||
from backend.api.dependencies import require_admin
|
||||
from backend.services.auth_service import UserType
|
||||
from backend.services.job_manager import JobManager
|
||||
from backend.services.tracing import (
|
||||
extract_trace_context,
|
||||
start_span_with_context,
|
||||
add_span_attribute,
|
||||
add_span_event,
|
||||
)
|
||||
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
router = APIRouter(prefix="/internal", tags=["internal"])
|
||||
|
||||
|
||||
class WorkerRequest(BaseModel):
|
||||
"""Request to trigger a worker."""
|
||||
job_id: str
|
||||
|
||||
|
||||
class WorkerResponse(BaseModel):
|
||||
"""Response from worker trigger."""
|
||||
status: str
|
||||
job_id: str
|
||||
message: str
|
||||
|
||||
|
||||
def _check_worker_idempotency(job_id: str, worker_name: str) -> Optional[WorkerResponse]:
|
||||
"""
|
||||
Check if a worker is already running or completed for this job.
|
||||
|
||||
This provides idempotency for Cloud Tasks retries - if a task is retried
|
||||
but the worker is already running or has completed, we skip processing.
|
||||
|
||||
Args:
|
||||
job_id: Job ID to check
|
||||
worker_name: Worker name (audio, lyrics, screens, render, video)
|
||||
|
||||
Returns:
|
||||
WorkerResponse if should skip (already running/complete), None to proceed
|
||||
"""
|
||||
job_manager = JobManager()
|
||||
job = job_manager.get_job(job_id)
|
||||
|
||||
if not job:
|
||||
logger.warning(f"[job:{job_id}] Job not found for {worker_name} worker")
|
||||
return WorkerResponse(
|
||||
status="not_found",
|
||||
job_id=job_id,
|
||||
message=f"Job {job_id} not found"
|
||||
)
|
||||
|
||||
# Check worker-specific progress in state_data
|
||||
progress_key = f"{worker_name}_progress"
|
||||
worker_progress = job.state_data.get(progress_key, {})
|
||||
stage = worker_progress.get('stage')
|
||||
|
||||
if stage == 'running':
|
||||
logger.info(f"[job:{job_id}] {worker_name.capitalize()} worker already running, skipping")
|
||||
return WorkerResponse(
|
||||
status="already_running",
|
||||
job_id=job_id,
|
||||
message=f"{worker_name.capitalize()} worker already in progress"
|
||||
)
|
||||
|
||||
if stage == 'complete':
|
||||
logger.info(f"[job:{job_id}] {worker_name.capitalize()} worker already complete, skipping")
|
||||
return WorkerResponse(
|
||||
status="already_complete",
|
||||
job_id=job_id,
|
||||
message=f"{worker_name.capitalize()} worker already completed"
|
||||
)
|
||||
|
||||
# Mark as running before starting (for idempotency on next retry)
|
||||
job_manager.update_state_data(job_id, progress_key, {'stage': 'running'})
|
||||
return None
|
||||
|
||||
|
||||
@router.post("/workers/audio", response_model=WorkerResponse)
|
||||
async def trigger_audio_worker(
|
||||
request: WorkerRequest,
|
||||
http_request: Request,
|
||||
background_tasks: BackgroundTasks,
|
||||
auth_data: Tuple[str, UserType, int] = Depends(require_admin)
|
||||
):
|
||||
"""
|
||||
Trigger audio separation worker for a job.
|
||||
|
||||
This endpoint is called internally after job creation to start
|
||||
the audio processing track (parallel with lyrics processing).
|
||||
|
||||
Idempotency: If worker is already running or complete, returns early.
|
||||
|
||||
The worker runs in the background and updates job state as it progresses.
|
||||
"""
|
||||
job_id = request.job_id
|
||||
|
||||
# Extract trace context from incoming request (propagated via Cloud Tasks)
|
||||
trace_context = extract_trace_context(dict(http_request.headers))
|
||||
|
||||
logger.info(f"[job:{job_id}] WORKER_TRIGGER worker=audio")
|
||||
add_span_attribute("job_id", job_id)
|
||||
add_span_attribute("worker", "audio")
|
||||
|
||||
# Idempotency check
|
||||
skip_response = _check_worker_idempotency(job_id, "audio")
|
||||
if skip_response:
|
||||
add_span_event("worker_skipped", {"reason": skip_response.status})
|
||||
return skip_response
|
||||
|
||||
# Add task to background tasks
|
||||
# This allows the HTTP response to return immediately
|
||||
# while the worker continues processing
|
||||
background_tasks.add_task(process_audio_separation, job_id)
|
||||
|
||||
add_span_event("worker_started")
|
||||
return WorkerResponse(
|
||||
status="started",
|
||||
job_id=job_id,
|
||||
message="Audio separation worker started"
|
||||
)
|
||||
|
||||
|
||||
@router.post("/workers/lyrics", response_model=WorkerResponse)
|
||||
async def trigger_lyrics_worker(
|
||||
request: WorkerRequest,
|
||||
http_request: Request,
|
||||
background_tasks: BackgroundTasks,
|
||||
auth_data: Tuple[str, UserType, int] = Depends(require_admin)
|
||||
):
|
||||
"""
|
||||
Trigger lyrics transcription worker for a job.
|
||||
|
||||
This endpoint is called internally after job creation to start
|
||||
the lyrics processing track (parallel with audio processing).
|
||||
|
||||
Idempotency: If worker is already running or complete, returns early.
|
||||
|
||||
The worker runs in the background and updates job state as it progresses.
|
||||
"""
|
||||
job_id = request.job_id
|
||||
|
||||
# Extract trace context from incoming request
|
||||
trace_context = extract_trace_context(dict(http_request.headers))
|
||||
|
||||
logger.info(f"[job:{job_id}] WORKER_TRIGGER worker=lyrics")
|
||||
add_span_attribute("job_id", job_id)
|
||||
add_span_attribute("worker", "lyrics")
|
||||
|
||||
# Idempotency check
|
||||
skip_response = _check_worker_idempotency(job_id, "lyrics")
|
||||
if skip_response:
|
||||
add_span_event("worker_skipped", {"reason": skip_response.status})
|
||||
return skip_response
|
||||
|
||||
# Add task to background tasks
|
||||
background_tasks.add_task(process_lyrics_transcription, job_id)
|
||||
|
||||
add_span_event("worker_started")
|
||||
return WorkerResponse(
|
||||
status="started",
|
||||
job_id=job_id,
|
||||
message="Lyrics transcription worker started"
|
||||
)
|
||||
|
||||
|
||||
@router.post("/workers/screens", response_model=WorkerResponse)
|
||||
async def trigger_screens_worker(
|
||||
request: WorkerRequest,
|
||||
http_request: Request,
|
||||
background_tasks: BackgroundTasks,
|
||||
auth_data: Tuple[str, UserType, int] = Depends(require_admin)
|
||||
):
|
||||
"""
|
||||
Trigger title/end screen generation worker.
|
||||
|
||||
This is called automatically when both audio and lyrics are complete.
|
||||
|
||||
Idempotency: If worker is already running or complete, returns early.
|
||||
"""
|
||||
job_id = request.job_id
|
||||
|
||||
# Extract trace context from incoming request
|
||||
trace_context = extract_trace_context(dict(http_request.headers))
|
||||
|
||||
logger.info(f"[job:{job_id}] WORKER_TRIGGER worker=screens")
|
||||
add_span_attribute("job_id", job_id)
|
||||
add_span_attribute("worker", "screens")
|
||||
|
||||
# Idempotency check
|
||||
skip_response = _check_worker_idempotency(job_id, "screens")
|
||||
if skip_response:
|
||||
add_span_event("worker_skipped", {"reason": skip_response.status})
|
||||
return skip_response
|
||||
|
||||
# Add task to background tasks
|
||||
background_tasks.add_task(generate_screens, job_id)
|
||||
|
||||
add_span_event("worker_started")
|
||||
return WorkerResponse(
|
||||
status="started",
|
||||
job_id=job_id,
|
||||
message="Screens generation worker started"
|
||||
)
|
||||
|
||||
|
||||
@router.post("/workers/video", response_model=WorkerResponse)
|
||||
async def trigger_video_worker(
|
||||
request: WorkerRequest,
|
||||
http_request: Request,
|
||||
background_tasks: BackgroundTasks,
|
||||
auth_data: Tuple[str, UserType, int] = Depends(require_admin)
|
||||
):
|
||||
"""
|
||||
Trigger final video generation and encoding worker.
|
||||
|
||||
This is called after user selects their preferred instrumental.
|
||||
This is the longest-running stage (15-20 minutes).
|
||||
|
||||
Idempotency: If worker is already running or complete, returns early.
|
||||
"""
|
||||
job_id = request.job_id
|
||||
|
||||
# Extract trace context from incoming request
|
||||
trace_context = extract_trace_context(dict(http_request.headers))
|
||||
|
||||
logger.info(f"[job:{job_id}] WORKER_TRIGGER worker=video")
|
||||
add_span_attribute("job_id", job_id)
|
||||
add_span_attribute("worker", "video")
|
||||
|
||||
# Idempotency check
|
||||
skip_response = _check_worker_idempotency(job_id, "video")
|
||||
if skip_response:
|
||||
add_span_event("worker_skipped", {"reason": skip_response.status})
|
||||
return skip_response
|
||||
|
||||
# Add task to background tasks
|
||||
background_tasks.add_task(generate_video, job_id)
|
||||
|
||||
add_span_event("worker_started")
|
||||
return WorkerResponse(
|
||||
status="started",
|
||||
job_id=job_id,
|
||||
message="Video generation worker started"
|
||||
)
|
||||
|
||||
|
||||
@router.post("/workers/render-video", response_model=WorkerResponse)
|
||||
async def trigger_render_video_worker(
|
||||
request: WorkerRequest,
|
||||
http_request: Request,
|
||||
background_tasks: BackgroundTasks,
|
||||
auth_data: Tuple[str, UserType, int] = Depends(require_admin)
|
||||
):
|
||||
"""
|
||||
Trigger render video worker (post-review).
|
||||
|
||||
This is called after human review is complete.
|
||||
Uses OutputGenerator from LyricsTranscriber to generate the karaoke video
|
||||
with the corrected lyrics.
|
||||
|
||||
Idempotency: If worker is already running or complete, returns early.
|
||||
|
||||
Output: with_vocals.mkv in GCS
|
||||
Next state: AWAITING_INSTRUMENTAL_SELECTION
|
||||
"""
|
||||
job_id = request.job_id
|
||||
|
||||
# Extract trace context from incoming request
|
||||
trace_context = extract_trace_context(dict(http_request.headers))
|
||||
|
||||
logger.info(f"[job:{job_id}] WORKER_TRIGGER worker=render-video")
|
||||
add_span_attribute("job_id", job_id)
|
||||
add_span_attribute("worker", "render-video")
|
||||
|
||||
# Idempotency check
|
||||
skip_response = _check_worker_idempotency(job_id, "render")
|
||||
if skip_response:
|
||||
add_span_event("worker_skipped", {"reason": skip_response.status})
|
||||
return skip_response
|
||||
|
||||
# Add task to background tasks
|
||||
background_tasks.add_task(process_render_video, job_id)
|
||||
|
||||
add_span_event("worker_started")
|
||||
return WorkerResponse(
|
||||
status="started",
|
||||
job_id=job_id,
|
||||
message="Render video worker started (post-review)"
|
||||
)
|
||||
|
||||
|
||||
@router.get("/health")
|
||||
async def internal_health(
|
||||
auth_data: Tuple[str, UserType, int] = Depends(require_admin)
|
||||
):
|
||||
"""
|
||||
Internal health check endpoint.
|
||||
|
||||
Used to verify the internal API is responsive.
|
||||
Requires admin authentication.
|
||||
"""
|
||||
return {"status": "healthy", "service": "karaoke-backend-internal"}
|
||||
|
||||
File diff suppressed because it is too large.
Load diff
@@ -1,570 +0,0 @@
|
||||
"""
|
||||
Review API routes - Compatible with LyricsTranscriber frontend.
|
||||
|
||||
These endpoints match the API that the LyricsTranscriber review frontend expects,
|
||||
allowing us to use the existing React review UI with our cloud backend.
|
||||
|
||||
Usage:
|
||||
Frontend URL: http://localhost:5173/?baseApiUrl=http://localhost:8000/api/review/{job_id}
|
||||
|
||||
The baseApiUrl includes the job_id, and all endpoints are relative to that.
|
||||
"""
|
||||
import asyncio
|
||||
import logging
|
||||
import hashlib
|
||||
import json
|
||||
import os
|
||||
import tempfile
|
||||
from pathlib import Path
|
||||
from typing import Dict, Any, Set, Tuple
|
||||
|
||||
from fastapi import APIRouter, HTTPException, Request, Depends
|
||||
from fastapi.responses import FileResponse, StreamingResponse
|
||||
from starlette.background import BackgroundTask
|
||||
|
||||
from backend.models.job import JobStatus
|
||||
from backend.services.job_manager import JobManager
|
||||
from backend.services.storage_service import StorageService
|
||||
from backend.services.job_logging import job_log_context, JobLogger
|
||||
from backend.services.tracing import create_span, add_span_attribute, add_span_event
|
||||
from backend.api.dependencies import require_auth, require_review_auth
|
||||
from backend.services.auth_service import UserType
|
||||
|
||||
# LyricsTranscriber imports for preview generation
|
||||
from lyrics_transcriber.types import CorrectionResult
|
||||
from lyrics_transcriber.core.config import OutputConfig
|
||||
from lyrics_transcriber.correction.operations import CorrectionOperations
|
||||
|
||||
# Import from the unified style loader
|
||||
from karaoke_gen.style_loader import load_styles_from_gcs
|
||||
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
router = APIRouter(prefix="/review", tags=["review"])
|
||||
|
||||
# Store job context for the session
|
||||
# In production, this would be handled differently (e.g., session tokens)
|
||||
_job_contexts: Dict[str, Dict[str, Any]] = {}
|
||||
|
||||
# Store preview video paths for serving
|
||||
_preview_videos: Dict[str, Dict[str, str]] = {}
|
||||
|
||||
# Keep references to background tasks to prevent garbage collection
|
||||
_background_tasks: Set[asyncio.Task] = set()
|
||||
|
||||
|
||||
def _get_audio_hash(job_id: str) -> str:
|
||||
"""Generate a consistent audio hash for a job."""
|
||||
return hashlib.md5(job_id.encode()).hexdigest()
|
||||
|
||||
|
||||
@router.get("/{job_id}/ping")
|
||||
async def ping(job_id: str):
|
||||
"""Health check endpoint expected by frontend."""
|
||||
return {"status": "ok"}
|
||||
|
||||
|
||||
@router.get("/{job_id}/correction-data")
|
||||
async def get_correction_data(
|
||||
job_id: str,
|
||||
auth_info: Tuple[str, str] = Depends(require_review_auth)
|
||||
):
|
||||
"""
|
||||
Get correction data for the review interface.
|
||||
|
||||
Returns the CorrectionResult data that the frontend needs to render
|
||||
the lyrics review UI.
|
||||
"""
|
||||
job_manager = JobManager()
|
||||
storage = StorageService()
|
||||
|
||||
job = job_manager.get_job(job_id)
|
||||
if not job:
|
||||
raise HTTPException(status_code=404, detail="Job not found")
|
||||
|
||||
if job.status not in [JobStatus.AWAITING_REVIEW, JobStatus.IN_REVIEW]:
|
||||
raise HTTPException(
|
||||
status_code=400,
|
||||
detail=f"Job not ready for review (current status: {job.status})"
|
||||
)
|
||||
|
||||
# Get corrections URL from file_urls
|
||||
corrections_gcs = job.file_urls.get('lyrics', {}).get('corrections')
|
||||
if not corrections_gcs:
|
||||
# Try direct path
|
||||
corrections_gcs = f"jobs/{job_id}/lyrics/corrections.json"
|
||||
if not storage.file_exists(corrections_gcs):
|
||||
raise HTTPException(
|
||||
status_code=404,
|
||||
detail="Corrections data not found. Lyrics processing may not be complete."
|
||||
)
|
||||
|
||||
# Download and return corrections data
|
||||
try:
|
||||
corrections_data = storage.download_json(corrections_gcs)
|
||||
|
||||
# Add audio hash for the frontend
|
||||
audio_hash = _get_audio_hash(job_id)
|
||||
if 'metadata' not in corrections_data:
|
||||
corrections_data['metadata'] = {}
|
||||
corrections_data['metadata']['audio_hash'] = audio_hash
|
||||
corrections_data['metadata']['artist'] = job.artist
|
||||
corrections_data['metadata']['title'] = job.title
|
||||
|
||||
# Store context for audio serving
|
||||
_job_contexts[job_id] = {
|
||||
'audio_hash': audio_hash,
|
||||
'audio_gcs_path': job.input_media_gcs_path
|
||||
}
|
||||
|
||||
# Transition to IN_REVIEW if not already
|
||||
if job.status == JobStatus.AWAITING_REVIEW:
|
||||
job_manager.transition_to_state(
|
||||
job_id=job_id,
|
||||
new_status=JobStatus.IN_REVIEW,
|
||||
message="User opened review interface"
|
||||
)
|
||||
|
||||
logger.info(f"Job {job_id}: Serving correction data for review")
|
||||
return corrections_data
|
||||
|
||||
except Exception as e:
|
||||
logger.error(f"Job {job_id}: Error loading corrections: {e}", exc_info=True)
|
||||
raise HTTPException(status_code=500, detail=f"Error loading corrections: {str(e)}")
|
||||
|
||||
|
||||
@router.get("/{job_id}/audio/{audio_hash}")
|
||||
async def get_audio_with_hash(
|
||||
job_id: str,
|
||||
audio_hash: str,
|
||||
auth_info: Tuple[str, str] = Depends(require_review_auth)
|
||||
):
|
||||
"""Stream the audio file for playback (with hash parameter)."""
|
||||
return await _stream_audio(job_id)
|
||||
|
||||
|
||||
@router.get("/{job_id}/audio/")
|
||||
@router.get("/{job_id}/audio")
|
||||
async def get_audio_no_hash(
|
||||
job_id: str,
|
||||
auth_info: Tuple[str, str] = Depends(require_review_auth)
|
||||
):
|
||||
"""Stream the audio file for playback (without hash parameter)."""
|
||||
return await _stream_audio(job_id)
|
||||
|
||||
|
||||
async def _stream_audio(job_id: str):
|
||||
"""
|
||||
Stream the audio file for playback in the review interface.
|
||||
"""
|
||||
job_manager = JobManager()
|
||||
storage = StorageService()
|
||||
|
||||
job = job_manager.get_job(job_id)
|
||||
if not job:
|
||||
raise HTTPException(status_code=404, detail="Job not found")
|
||||
|
||||
audio_gcs_path = job.input_media_gcs_path
|
||||
if not audio_gcs_path:
|
||||
raise HTTPException(status_code=404, detail="Audio file not found")
|
||||
|
||||
# Download to temp file and stream
|
||||
try:
|
||||
# Create temp file
|
||||
with tempfile.NamedTemporaryFile(delete=False, suffix=".flac") as tmp:
|
||||
tmp_path = tmp.name
|
||||
|
||||
storage.download_file(audio_gcs_path, tmp_path)
|
||||
|
||||
# Determine content type
|
||||
if audio_gcs_path.endswith('.flac'):
|
||||
media_type = "audio/flac"
|
||||
elif audio_gcs_path.endswith('.wav'):
|
||||
media_type = "audio/wav"
|
||||
elif audio_gcs_path.endswith('.mp3'):
|
||||
media_type = "audio/mpeg"
|
||||
else:
|
||||
media_type = "audio/mpeg"
|
||||
|
||||
logger.info(f"Job {job_id}: Streaming audio for review")
|
||||
|
||||
return FileResponse(
|
||||
tmp_path,
|
||||
media_type=media_type,
|
||||
filename=os.path.basename(audio_gcs_path),
|
||||
background=BackgroundTask(os.unlink, tmp_path),
|
||||
)
|
||||
|
||||
except Exception as e:
|
||||
logger.error(f"Job {job_id}: Error streaming audio: {e}", exc_info=True)
|
||||
raise HTTPException(status_code=500, detail=f"Error streaming audio: {str(e)}")
|
||||
|
||||
|
||||
@router.post("/{job_id}/complete")
|
||||
async def complete_review(
|
||||
job_id: str,
|
||||
updated_data: Dict[str, Any],
|
||||
auth_info: Tuple[str, str] = Depends(require_review_auth)
|
||||
):
|
||||
"""
|
||||
Complete the review and save corrected lyrics.
|
||||
|
||||
This endpoint receives the updated correction data from the frontend
|
||||
and saves it, then triggers the render video worker.
|
||||
"""
|
||||
job_manager = JobManager()
|
||||
storage = StorageService()
|
||||
|
||||
job = job_manager.get_job(job_id)
|
||||
if not job:
|
||||
raise HTTPException(status_code=404, detail="Job not found")
|
||||
|
||||
if job.status not in [JobStatus.AWAITING_REVIEW, JobStatus.IN_REVIEW]:
|
||||
raise HTTPException(
|
||||
status_code=400,
|
||||
detail=f"Job not in review state (current status: {job.status})"
|
||||
)
|
||||
|
||||
try:
|
||||
# Save updated corrections to GCS
|
||||
corrections_gcs_path = f"jobs/{job_id}/lyrics/corrections_updated.json"
|
||||
storage.upload_json(corrections_gcs_path, updated_data)
|
||||
job_manager.update_file_url(job_id, 'lyrics', 'corrections_updated', corrections_gcs_path)
|
||||
|
||||
logger.info(f"Job {job_id}: Saved updated corrections")
|
||||
|
||||
# Transition to REVIEW_COMPLETE
|
||||
job_manager.transition_to_state(
|
||||
job_id=job_id,
|
||||
new_status=JobStatus.REVIEW_COMPLETE,
|
||||
progress=70,
|
||||
message="Review complete, rendering video with corrected lyrics"
|
||||
)
|
||||
|
||||
# Trigger render video worker
|
||||
from backend.services.worker_service import get_worker_service
|
||||
worker_service = get_worker_service()
|
||||
|
||||
# Run in background, keep reference to prevent garbage collection
|
||||
task = asyncio.create_task(worker_service.trigger_render_video_worker(job_id))
|
||||
_background_tasks.add(task)
|
||||
task.add_done_callback(_background_tasks.discard)
|
||||
|
||||
logger.info(f"Job {job_id}: Review complete, triggered render video worker")
|
||||
|
||||
return {"status": "success"}
|
||||
|
||||
except Exception as e:
|
||||
logger.error(f"Job {job_id}: Error completing review: {e}", exc_info=True)
|
||||
raise HTTPException(status_code=500, detail=f"Error completing review: {str(e)}")
|
||||
|
||||
|
||||
@router.post("/{job_id}/handlers")
|
||||
async def update_handlers(
|
||||
job_id: str,
|
||||
enabled_handlers: list,
|
||||
auth_info: Tuple[str, str] = Depends(require_review_auth)
|
||||
):
|
||||
"""
|
||||
Update enabled correction handlers (optional feature).
|
||||
|
||||
For now, just acknowledge the request - full handler support
|
||||
would require re-running correction.
|
||||
"""
|
||||
logger.info(f"Job {job_id}: Handler update requested (not implemented)")
|
||||
return {"status": "success", "message": "Handler updates not yet implemented"}
|
||||
|
||||
|
||||
@router.post("/{job_id}/add-lyrics")
|
||||
async def add_lyrics(
|
||||
job_id: str,
|
||||
data: Dict[str, str],
|
||||
auth_info: Tuple[str, str] = Depends(require_review_auth)
|
||||
):
|
||||
"""
|
||||
Add custom lyrics source and rerun correction.
|
||||
|
||||
Uses the LyricsTranscriber's CorrectionOperations to add a new lyrics source
|
||||
and regenerate corrections with the new source included.
|
||||
"""
|
||||
job_manager = JobManager()
|
||||
storage = StorageService()
|
||||
|
||||
job = job_manager.get_job(job_id)
|
||||
if not job:
|
||||
raise HTTPException(status_code=404, detail="Job not found")
|
||||
|
||||
# Job must be in review state to add lyrics
|
||||
if job.status not in [JobStatus.AWAITING_REVIEW, JobStatus.IN_REVIEW]:
|
||||
raise HTTPException(
|
||||
status_code=400,
|
||||
detail=f"Job not in review state (current status: {job.status})"
|
||||
)
|
||||
|
||||
source = data.get("source", "").strip()
|
||||
lyrics_text = data.get("lyrics", "").strip()
|
||||
|
||||
logger.info(f"Job {job_id}: Adding lyrics source '{source}' with {len(lyrics_text)} characters")
|
||||
|
||||
# Use tracing and job_log_context for full observability
|
||||
with create_span("add-lyrics", {"job_id": job_id, "source": source, "lyrics_length": len(lyrics_text)}) as span:
|
||||
with job_log_context(job_id, worker="add-lyrics"):
|
||||
try:
|
||||
# Create temp directory for this operation
|
||||
with tempfile.TemporaryDirectory() as temp_dir:
|
||||
# Download current corrections.json
|
||||
with create_span("download-corrections") as download_span:
|
||||
corrections_gcs = f"jobs/{job_id}/lyrics/corrections.json"
|
||||
corrections_path = os.path.join(temp_dir, "corrections.json")
|
||||
storage.download_file(corrections_gcs, corrections_path)
|
||||
download_span.set_attribute("gcs_path", corrections_gcs)
|
||||
|
||||
with open(corrections_path, 'r', encoding='utf-8') as f:
|
||||
original_data = json.load(f)
|
||||
|
||||
# Load as CorrectionResult
|
||||
correction_result = CorrectionResult.from_dict(original_data)
|
||||
add_span_event("corrections_loaded", {
|
||||
"segments": len(correction_result.corrected_segments) if correction_result.corrected_segments else 0,
|
||||
"reference_sources": len(correction_result.reference_lyrics) if correction_result.reference_lyrics else 0,
|
||||
})
|
||||
|
||||
# Set up cache directory
|
||||
cache_dir = os.path.join(temp_dir, "cache")
|
||||
os.makedirs(cache_dir, exist_ok=True)
|
||||
|
||||
# Add lyrics source using CorrectionOperations (this is the heavy operation)
|
||||
with create_span("correction-operations-add-lyrics") as correction_span:
|
||||
correction_span.set_attribute("source_name", source)
|
||||
updated_result = CorrectionOperations.add_lyrics_source(
|
||||
correction_result=correction_result,
|
||||
source=source,
|
||||
lyrics_text=lyrics_text,
|
||||
cache_dir=cache_dir,
|
||||
logger=logger
|
||||
)
|
||||
add_span_event("correction_complete", {
|
||||
"new_segments": len(updated_result.corrected_segments) if updated_result.corrected_segments else 0,
|
||||
})
|
||||
|
||||
# Add audio hash for the frontend
|
||||
audio_hash = _get_audio_hash(job_id)
|
||||
if not updated_result.metadata:
|
||||
updated_result.metadata = {}
|
||||
updated_result.metadata['audio_hash'] = audio_hash
|
||||
updated_result.metadata['artist'] = job.artist
|
||||
updated_result.metadata['title'] = job.title
|
||||
|
||||
# Upload updated corrections back to GCS
|
||||
with create_span("upload-corrections") as upload_span:
|
||||
updated_data = updated_result.to_dict()
|
||||
storage.upload_json(corrections_gcs, updated_data)
|
||||
upload_span.set_attribute("gcs_path", corrections_gcs)
|
||||
|
||||
logger.info(f"Job {job_id}: Successfully added lyrics source '{source}'")
|
||||
span.set_attribute("success", True)
|
||||
|
||||
return {"status": "success", "data": updated_data}
|
||||
|
||||
except ValueError as e:
|
||||
# ValueError from CorrectionOperations (e.g., duplicate source name)
|
||||
logger.warning(f"Job {job_id}: Invalid add lyrics request: {e}")
|
||||
span.set_attribute("error", str(e))
|
||||
raise HTTPException(status_code=400, detail=str(e))
|
||||
except Exception as e:
|
||||
logger.error(f"Job {job_id}: Failed to add lyrics: {e}", exc_info=True)
|
||||
span.set_attribute("error", str(e))
|
||||
raise HTTPException(status_code=500, detail=f"Failed to add lyrics: {str(e)}")
|
||||
|
||||
|
||||
@router.post("/{job_id}/preview-video")
|
||||
async def generate_preview_video(
|
||||
job_id: str,
|
||||
updated_data: Dict[str, Any],
|
||||
auth_info: Tuple[str, str] = Depends(require_review_auth)
|
||||
):
|
||||
"""
|
||||
Generate a preview video with the current corrections.
|
||||
|
||||
Uses the LyricsTranscriber's CorrectionOperations to generate a 360p preview
|
||||
video with the user's current corrections applied.
|
||||
"""
|
||||
job_manager = JobManager()
|
||||
storage = StorageService()
|
||||
|
||||
job = job_manager.get_job(job_id)
|
||||
if not job:
|
||||
raise HTTPException(status_code=404, detail="Job not found")
|
||||
|
||||
# Job must be in review state to generate preview
|
||||
if job.status not in [JobStatus.AWAITING_REVIEW, JobStatus.IN_REVIEW]:
|
||||
raise HTTPException(
|
||||
status_code=400,
|
||||
detail=f"Job not in review state (current status: {job.status})"
|
||||
)
|
||||
|
||||
logger.info(f"Job {job_id}: Generating preview video")
|
||||
|
||||
# Use tracing and job_log_context for full observability
|
||||
with create_span("generate-preview-video", {"job_id": job_id}) as span:
|
||||
with job_log_context(job_id, worker="preview"):
|
||||
try:
|
||||
# Create temp directory for this preview operation
|
||||
with tempfile.TemporaryDirectory() as temp_dir:
|
||||
# 1. Download original corrections.json (has full structure)
|
||||
with create_span("download-corrections-and-audio") as download_span:
|
||||
corrections_gcs = f"jobs/{job_id}/lyrics/corrections.json"
|
||||
corrections_path = os.path.join(temp_dir, "corrections.json")
|
||||
storage.download_file(corrections_gcs, corrections_path)
|
||||
|
||||
with open(corrections_path, 'r', encoding='utf-8') as f:
|
||||
original_data = json.load(f)
|
||||
|
||||
# 2. Download input audio
|
||||
audio_path = os.path.join(temp_dir, "audio.flac")
|
||||
storage.download_file(job.input_media_gcs_path, audio_path)
|
||||
download_span.set_attribute("audio_gcs_path", job.input_media_gcs_path)
|
||||
|
||||
# 3. Load original as CorrectionResult
|
||||
correction_result = CorrectionResult.from_dict(original_data)
|
||||
add_span_event("corrections_loaded")
|
||||
|
||||
# 4. Get or create styles file for preview using unified style loader
|
||||
with create_span("load-styles") as styles_span:
|
||||
styles_path, _ = load_styles_from_gcs(
|
||||
style_params_gcs_path=job.style_params_gcs_path,
|
||||
style_assets=job.style_assets,
|
||||
temp_dir=temp_dir,
|
||||
download_func=storage.download_file,
|
||||
logger=logger,
|
||||
)
|
||||
styles_span.set_attribute("styles_path", styles_path)
|
||||
|
||||
# 5. Set up output config for preview
|
||||
output_dir = os.path.join(temp_dir, "output")
|
||||
cache_dir = os.path.join(temp_dir, "cache")
|
||||
os.makedirs(output_dir, exist_ok=True)
|
||||
os.makedirs(cache_dir, exist_ok=True)
|
||||
|
||||
output_config = OutputConfig(
|
||||
output_styles_json=styles_path,
|
||||
output_dir=output_dir,
|
||||
cache_dir=cache_dir,
|
||||
video_resolution="360p",
|
||||
)
|
||||
|
||||
# 6. Generate preview video using CorrectionOperations (heavy operation)
|
||||
with create_span("render-preview-video") as render_span:
|
||||
render_span.set_attribute("resolution", "360p")
|
||||
result = CorrectionOperations.generate_preview_video(
|
||||
correction_result=correction_result,
|
||||
updated_data=updated_data,
|
||||
output_config=output_config,
|
||||
audio_filepath=audio_path,
|
||||
artist=job.artist,
|
||||
title=job.title,
|
||||
logger=logger
|
||||
)
|
||||
add_span_event("render_complete")
|
||||
|
||||
# 7. Upload preview video to GCS for serving
|
||||
preview_hash = result["preview_hash"]
|
||||
video_path = result["video_path"]
|
||||
|
||||
with create_span("upload-preview-video") as upload_span:
|
||||
preview_gcs_path = f"jobs/{job_id}/previews/{preview_hash}.mp4"
|
||||
storage.upload_file(video_path, preview_gcs_path)
|
||||
upload_span.set_attribute("gcs_path", preview_gcs_path)
|
||||
|
||||
# Store the GCS path for serving
|
||||
if job_id not in _preview_videos:
|
||||
_preview_videos[job_id] = {}
|
||||
_preview_videos[job_id][preview_hash] = preview_gcs_path
|
||||
|
||||
logger.info(f"Job {job_id}: Preview video generated: {preview_hash}")
|
||||
span.set_attribute("preview_hash", preview_hash)
|
||||
span.set_attribute("success", True)
|
||||
|
||||
return {"status": "success", "preview_hash": preview_hash}
|
||||
|
||||
except Exception as e:
|
||||
logger.error(f"Job {job_id}: Failed to generate preview video: {e}", exc_info=True)
|
||||
span.set_attribute("error", str(e))
|
||||
raise HTTPException(
|
||||
status_code=500,
|
||||
detail=f"Failed to generate preview video: {e}"
|
||||
) from e
|
||||
|
||||
|
||||
@router.get("/{job_id}/preview-video/{preview_hash}")
|
||||
async def get_preview_video(
|
||||
job_id: str,
|
||||
preview_hash: str,
|
||||
auth_info: Tuple[str, str] = Depends(require_review_auth)
|
||||
):
|
||||
"""Stream the generated preview video."""
|
||||
storage = StorageService()
|
||||
|
||||
# Check in-memory cache first
|
||||
preview_gcs_path = None
|
||||
if job_id in _preview_videos and preview_hash in _preview_videos[job_id]:
|
||||
preview_gcs_path = _preview_videos[job_id][preview_hash]
|
||||
else:
|
||||
# Try standard path
|
||||
preview_gcs_path = f"jobs/{job_id}/previews/{preview_hash}.mp4"
|
||||
if not storage.file_exists(preview_gcs_path):
|
||||
raise HTTPException(status_code=404, detail="Preview video not found")
|
||||
|
||||
try:
|
||||
# Download to temp file and stream
|
||||
with tempfile.NamedTemporaryFile(delete=False, suffix=".mp4") as tmp:
|
||||
tmp_path = tmp.name
|
||||
|
||||
storage.download_file(preview_gcs_path, tmp_path)
|
||||
|
||||
logger.info(f"Job {job_id}: Streaming preview video {preview_hash}")
|
||||
|
||||
return FileResponse(
|
||||
tmp_path,
|
||||
media_type="video/mp4",
|
||||
filename=f"preview_{preview_hash}.mp4",
|
||||
headers={
|
||||
"Accept-Ranges": "bytes",
|
||||
"Content-Disposition": "inline",
|
||||
"Cache-Control": "no-cache",
|
||||
},
|
||||
background=BackgroundTask(os.unlink, tmp_path),
|
||||
)
|
||||
|
||||
except Exception as e:
|
||||
logger.error(f"Job {job_id}: Error streaming preview video: {e}", exc_info=True)
|
||||
raise HTTPException(status_code=500, detail=f"Error streaming preview video: {str(e)}")
|
||||
|
||||
|
||||
@router.post("/{job_id}/v1/annotations")
|
||||
async def submit_annotation(
|
||||
job_id: str,
|
||||
annotation: Dict[str, Any],
|
||||
auth_info: Tuple[str, str] = Depends(require_review_auth)
|
||||
):
|
||||
"""
|
||||
Submit a correction annotation for ML training data.
|
||||
|
||||
For now, just log and acknowledge - full annotation support
|
||||
would require a database.
|
||||
"""
|
||||
logger.info(f"Job {job_id}: Annotation submitted (logged but not stored)")
|
||||
return {"status": "success", "annotation_id": "stub"}
|
||||
|
||||
|
||||
@router.get("/{job_id}/v1/annotations/stats")
|
||||
async def get_annotation_stats(
|
||||
job_id: str,
|
||||
auth_info: Tuple[str, str] = Depends(require_review_auth)
|
||||
):
|
||||
"""Get annotation statistics."""
|
||||
return {
|
||||
"total_annotations": 0,
|
||||
"by_type": {},
|
||||
"message": "Annotation stats not yet implemented"
|
||||
}
|
||||
@@ -1,162 +0,0 @@
|
||||
"""
|
||||
Theme API routes for listing and retrieving theme information.
|
||||
|
||||
Themes are pre-created style configurations that users can select when creating jobs.
|
||||
These endpoints are public (no authentication required) to allow theme selection
|
||||
before job creation.
|
||||
"""
|
||||
|
||||
import logging
|
||||
from typing import Optional
|
||||
|
||||
from fastapi import APIRouter, HTTPException
|
||||
|
||||
from backend.models.theme import ThemeDetailResponse, ThemesListResponse
|
||||
from backend.services.theme_service import get_theme_service
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
router = APIRouter(prefix="/themes", tags=["themes"])
|
||||
|
||||
|
||||
@router.get("", response_model=ThemesListResponse)
|
||||
async def list_themes() -> ThemesListResponse:
|
||||
"""
|
||||
List all available themes.
|
||||
|
||||
Returns a list of themes with their metadata and preview image URLs.
|
||||
Preview URLs are signed and valid for 60 minutes.
|
||||
|
||||
This endpoint is public and does not require authentication.
|
||||
"""
|
||||
try:
|
||||
theme_service = get_theme_service()
|
||||
themes = theme_service.list_themes()
|
||||
return ThemesListResponse(themes=themes)
|
||||
except Exception as e:
|
||||
logger.error(f"Error listing themes: {e}")
|
||||
raise HTTPException(
|
||||
status_code=500,
|
||||
detail="Failed to load themes"
|
||||
)
|
||||
|
||||
|
||||
@router.get("/{theme_id}", response_model=ThemeDetailResponse)
|
||||
async def get_theme(theme_id: str) -> ThemeDetailResponse:
|
||||
"""
|
||||
Get detailed information about a specific theme.
|
||||
|
||||
Returns the full theme details including style parameters.
|
||||
This can be used to preview theme settings or inspect configuration.
|
||||
|
||||
Args:
|
||||
theme_id: The unique identifier of the theme
|
||||
|
||||
Returns:
|
||||
ThemeDetailResponse with full theme details
|
||||
|
||||
Raises:
|
||||
404: Theme not found
|
||||
"""
|
||||
try:
|
||||
theme_service = get_theme_service()
|
||||
theme = theme_service.get_theme(theme_id)
|
||||
|
||||
if theme is None:
|
||||
raise HTTPException(
|
||||
status_code=404,
|
||||
detail=f"Theme not found: {theme_id}"
|
||||
)
|
||||
|
||||
return ThemeDetailResponse(theme=theme)
|
||||
except HTTPException:
|
||||
raise
|
||||
except Exception as e:
|
||||
logger.error(f"Error getting theme {theme_id}: {e}")
|
||||
raise HTTPException(
|
||||
status_code=500,
|
||||
detail="Failed to load theme"
|
||||
)
|
||||
|
||||
|
||||
@router.get("/{theme_id}/preview")
|
||||
async def get_theme_preview(theme_id: str) -> dict:
|
||||
"""
|
||||
Get a signed URL for the theme's preview image.
|
||||
|
||||
This endpoint is useful when you need just the preview URL
|
||||
without loading the full theme details.
|
||||
|
||||
Args:
|
||||
theme_id: The unique identifier of the theme
|
||||
|
||||
Returns:
|
||||
Object with preview_url field
|
||||
|
||||
Raises:
|
||||
404: Theme not found or no preview available
|
||||
"""
|
||||
try:
|
||||
theme_service = get_theme_service()
|
||||
|
||||
if not theme_service.theme_exists(theme_id):
|
||||
raise HTTPException(
|
||||
status_code=404,
|
||||
detail=f"Theme not found: {theme_id}"
|
||||
)
|
||||
|
||||
theme = theme_service.get_theme(theme_id)
|
||||
if theme is None or theme.preview_url is None:
|
||||
raise HTTPException(
|
||||
status_code=404,
|
||||
detail=f"No preview available for theme: {theme_id}"
|
||||
)
|
||||
|
||||
return {"preview_url": theme.preview_url}
|
||||
except HTTPException:
|
||||
raise
|
||||
except Exception as e:
|
||||
logger.error(f"Error getting preview for theme {theme_id}: {e}")
|
||||
raise HTTPException(
|
||||
status_code=500,
|
||||
detail="Failed to load theme preview"
|
||||
)
|
||||
|
||||
|
||||
@router.get("/{theme_id}/youtube-description")
|
||||
async def get_theme_youtube_description(theme_id: str) -> dict:
|
||||
"""
|
||||
Get the YouTube description template for a theme.
|
||||
|
||||
Some themes include a default YouTube video description template
|
||||
that can be used when uploading videos.
|
||||
|
||||
Args:
|
||||
theme_id: The unique identifier of the theme
|
||||
|
||||
Returns:
|
||||
Object with description field containing the template text,
|
||||
or null if no template is available
|
||||
|
||||
Raises:
|
||||
404: Theme not found
|
||||
"""
|
||||
try:
|
||||
theme_service = get_theme_service()
|
||||
|
||||
if not theme_service.theme_exists(theme_id):
|
||||
raise HTTPException(
|
||||
status_code=404,
|
||||
detail=f"Theme not found: {theme_id}"
|
||||
)
|
||||
|
||||
description = theme_service.get_youtube_description(theme_id)
|
||||
return {"description": description}
|
||||
except HTTPException:
|
||||
raise
|
||||
except Exception as e:
|
||||
logger.error(f"Error getting YouTube description for theme {theme_id}: {e}")
|
||||
raise HTTPException(
|
||||
status_code=500,
|
||||
detail="Failed to load YouTube description"
|
||||
)
|
||||
@@ -1,144 +0,0 @@
|
||||
"""
|
||||
Configuration management for the karaoke generation backend.
|
||||
"""
|
||||
import os
|
||||
import logging
|
||||
from typing import Optional, Dict
|
||||
from pydantic_settings import BaseSettings
|
||||
from google.cloud import secretmanager
|
||||
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
|
||||
class Settings(BaseSettings):
|
||||
"""Application settings."""
|
||||
|
||||
# Google Cloud
|
||||
google_cloud_project: str = os.getenv("GOOGLE_CLOUD_PROJECT", "")
|
||||
gcs_bucket_name: str = os.getenv("GCS_BUCKET_NAME", "karaoke-gen-storage")
|
||||
gcs_temp_bucket: str = os.getenv("GCS_TEMP_BUCKET", "karaoke-gen-temp")
|
||||
gcs_output_bucket: str = os.getenv("GCS_OUTPUT_BUCKET", "karaoke-gen-outputs")
|
||||
firestore_collection: str = os.getenv("FIRESTORE_COLLECTION", "jobs")
|
||||
|
||||
# Audio Separator API (for GPU processing)
|
||||
audio_separator_api_url: Optional[str] = os.getenv("AUDIO_SEPARATOR_API_URL")
|
||||
|
||||
# External APIs (can be set via env or Secret Manager)
|
||||
audioshake_api_key: Optional[str] = os.getenv("AUDIOSHAKE_API_KEY")
|
||||
genius_api_key: Optional[str] = os.getenv("GENIUS_API_KEY")
|
||||
spotify_cookie: Optional[str] = os.getenv("SPOTIFY_COOKIE_SP_DC")
|
||||
rapidapi_key: Optional[str] = os.getenv("RAPIDAPI_KEY")
|
||||
|
||||
# Authentication
|
||||
admin_tokens: Optional[str] = os.getenv("ADMIN_TOKENS") # Comma-separated list
|
||||
|
||||
# Application
|
||||
environment: str = os.getenv("ENVIRONMENT", "development")
|
||||
log_level: str = os.getenv("LOG_LEVEL", "INFO")
|
||||
|
||||
# Processing
|
||||
max_concurrent_jobs: int = int(os.getenv("MAX_CONCURRENT_JOBS", "5"))
|
||||
job_timeout_seconds: int = int(os.getenv("JOB_TIMEOUT_SECONDS", "3600"))
|
||||
|
||||
# Cloud Tasks (for scalable worker coordination)
|
||||
# When enabled, workers are triggered via Cloud Tasks for guaranteed delivery
|
||||
# When disabled (default), workers are triggered via direct HTTP (for development)
|
||||
enable_cloud_tasks: bool = os.getenv("ENABLE_CLOUD_TASKS", "false").lower() in ("true", "1", "yes")
|
||||
gcp_region: str = os.getenv("GCP_REGION", "us-central1")
|
||||
|
||||
# Cloud Run Jobs (for long-running video encoding)
|
||||
# When enabled AND enable_cloud_tasks is true, video worker uses Cloud Run Jobs
|
||||
# instead of Cloud Tasks. This supports encoding times >30 minutes (up to 24 hours).
|
||||
# Default is false - Cloud Tasks is sufficient for most videos (15-20 min).
|
||||
use_cloud_run_jobs_for_video: bool = os.getenv("USE_CLOUD_RUN_JOBS_FOR_VIDEO", "false").lower() in ("true", "1", "yes")
|
||||
|
||||
# Storage paths
|
||||
temp_dir: str = os.getenv("TEMP_DIR", "/tmp/karaoke-gen")
|
||||
|
||||
# Flacfetch remote service (for torrent downloads)
|
||||
# When configured, audio search uses the remote flacfetch HTTP API instead of local flacfetch.
|
||||
# This is required for torrent downloads since Cloud Run doesn't support BitTorrent.
|
||||
flacfetch_api_url: Optional[str] = os.getenv("FLACFETCH_API_URL") # e.g., http://10.0.0.5:8080
|
||||
flacfetch_api_key: Optional[str] = os.getenv("FLACFETCH_API_KEY")
|
||||
|
||||
# Default distribution settings (can be overridden per-request)
|
||||
default_dropbox_path: Optional[str] = os.getenv("DEFAULT_DROPBOX_PATH")
|
||||
default_gdrive_folder_id: Optional[str] = os.getenv("DEFAULT_GDRIVE_FOLDER_ID")
|
||||
# Strip whitespace/newlines from webhook URL - common issue when env vars are set with trailing newlines
|
||||
default_discord_webhook_url: Optional[str] = (
|
||||
os.getenv("DEFAULT_DISCORD_WEBHOOK_URL", "").strip() or None
|
||||
)
|
||||
|
||||
# Default values for web service jobs (YouTube/Dropbox distribution)
|
||||
default_enable_youtube_upload: bool = os.getenv("DEFAULT_ENABLE_YOUTUBE_UPLOAD", "false").lower() in ("true", "1", "yes")
|
||||
default_brand_prefix: Optional[str] = os.getenv("DEFAULT_BRAND_PREFIX")
|
||||
default_youtube_description: str = os.getenv(
|
||||
"DEFAULT_YOUTUBE_DESCRIPTION",
|
||||
"Karaoke video created with Nomad Karaoke (https://nomadkaraoke.com)\n\n"
|
||||
"AI-powered vocal separation and synchronized lyrics.\n\n"
|
||||
"#karaoke #music #singing #instrumental #lyrics"
|
||||
)
|
||||
|
||||
# Secret Manager cache
|
||||
_secret_cache: Dict[str, str] = {}
|
||||
|
||||
class Config:
|
||||
env_file = ".env"
|
||||
case_sensitive = False
|
||||
|
||||
def get_secret(self, secret_id: str) -> Optional[str]:
|
||||
"""
|
||||
Get a secret from Google Secret Manager.
|
||||
|
||||
Caches secrets in memory to avoid repeated API calls.
|
||||
Falls back to environment variables if Secret Manager unavailable.
|
||||
|
||||
Args:
|
||||
secret_id: Secret name (e.g., "audioshake-api-key")
|
||||
|
||||
Returns:
|
||||
Secret value or None if not found
|
||||
"""
|
||||
# Check cache first
|
||||
if secret_id in self._secret_cache:
|
||||
return self._secret_cache[secret_id]
|
||||
|
||||
# Check environment variable (development mode)
|
||||
env_var = secret_id.upper().replace('-', '_')
|
||||
env_value = os.getenv(env_var)
|
||||
if env_value:
|
||||
logger.debug(f"Using {secret_id} from environment variable")
|
||||
self._secret_cache[secret_id] = env_value
|
||||
return env_value
|
||||
|
||||
# Try Secret Manager (production mode)
|
||||
if not self.google_cloud_project:
|
||||
logger.warning(f"Cannot fetch secret {secret_id}: No GCP project configured")
|
||||
return None
|
||||
|
||||
try:
|
||||
client = secretmanager.SecretManagerServiceClient()
|
||||
name = f"projects/{self.google_cloud_project}/secrets/{secret_id}/versions/latest"
|
||||
response = client.access_secret_version(request={"name": name})
|
||||
# Strip whitespace/newlines - common issue when secrets are created with trailing newlines
|
||||
secret_value = response.payload.data.decode('UTF-8').strip()
|
||||
|
||||
# Cache it
|
||||
self._secret_cache[secret_id] = secret_value
|
||||
logger.info(f"Loaded secret {secret_id} from Secret Manager")
|
||||
return secret_value
|
||||
|
||||
except Exception as e:
|
||||
logger.error(f"Failed to load secret {secret_id} from Secret Manager: {e}")
|
||||
return None
|
||||
|
||||
|
||||
# Global settings instance
|
||||
settings = Settings()
|
||||
|
||||
|
||||
def get_settings() -> Settings:
|
||||
"""Get the global settings instance."""
|
||||
return settings
|
||||
|
||||
-127
@@ -1,127 +0,0 @@
|
||||
"""
|
||||
FastAPI application entry point for karaoke generation backend.
|
||||
"""
|
||||
import logging
|
||||
from contextlib import asynccontextmanager
|
||||
from fastapi import FastAPI
|
||||
from fastapi.middleware.cors import CORSMiddleware
|
||||
|
||||
from backend.config import settings
|
||||
from backend.api.routes import health, jobs, internal, file_upload, review, auth, audio_search, themes
|
||||
from backend.services.tracing import setup_tracing, instrument_app, get_current_trace_id
|
||||
from backend.services.structured_logging import setup_structured_logging
|
||||
|
||||
|
||||
from backend.version import VERSION
|
||||
|
||||
|
||||
# Configure structured logging (JSON in Cloud Run, human-readable locally)
|
||||
# This must happen before any logging calls
|
||||
setup_structured_logging()
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
# Initialize OpenTelemetry tracing (must happen before app creation)
|
||||
tracing_enabled = setup_tracing(
|
||||
service_name="karaoke-backend",
|
||||
service_version=VERSION,
|
||||
enable_in_dev=False, # Set to True to enable tracing locally
|
||||
)
|
||||
|
||||
|
||||
async def validate_credentials_on_startup():
|
||||
"""Validate OAuth credentials on startup and send alerts if needed."""
|
||||
try:
|
||||
from backend.services.credential_manager import get_credential_manager, CredentialStatus
|
||||
|
||||
manager = get_credential_manager()
|
||||
results = manager.check_all_credentials()
|
||||
|
||||
invalid_services = [
|
||||
result for result in results.values()
|
||||
if result.status in (CredentialStatus.INVALID, CredentialStatus.EXPIRED)
|
||||
]
|
||||
|
||||
if invalid_services:
|
||||
logger.warning(f"Some OAuth credentials need attention:")
|
||||
for result in invalid_services:
|
||||
logger.warning(f" - {result.service}: {result.message}")
|
||||
|
||||
# Try to send Discord alert
|
||||
discord_url = settings.get_secret("discord-alert-webhook") if hasattr(settings, 'get_secret') else None
|
||||
if discord_url:
|
||||
manager.send_credential_alert(invalid_services, discord_url)
|
||||
logger.info("Sent credential alert to Discord")
|
||||
else:
|
||||
logger.info("All OAuth credentials validated successfully")
|
||||
|
||||
except Exception as e:
|
||||
logger.error(f"Failed to validate credentials on startup: {e}")
|
||||
|
||||
|
||||
@asynccontextmanager
|
||||
async def lifespan(app: FastAPI):
|
||||
"""Lifespan event handler for startup and shutdown."""
|
||||
# Startup
|
||||
logger.info("Starting karaoke generation backend")
|
||||
logger.info(f"Environment: {settings.environment}")
|
||||
logger.info(f"GCS Bucket: {settings.gcs_bucket_name}")
|
||||
logger.info(f"Tracing enabled: {tracing_enabled}")
|
||||
|
||||
# Validate OAuth credentials (non-blocking)
|
||||
try:
|
||||
await validate_credentials_on_startup()
|
||||
except Exception as e:
|
||||
logger.error(f"Credential validation failed: {e}")
|
||||
|
||||
yield
|
||||
|
||||
# Shutdown
|
||||
logger.info("Shutting down karaoke generation backend")
|
||||
|
||||
|
||||
# Create FastAPI app
|
||||
app = FastAPI(
|
||||
title="Karaoke Generator API",
|
||||
description="Backend API for web-based karaoke video generation",
|
||||
version=VERSION,
|
||||
lifespan=lifespan
|
||||
)
|
||||
|
||||
# Instrument FastAPI with OpenTelemetry (adds automatic spans for all requests)
|
||||
if tracing_enabled:
|
||||
instrument_app(app)
|
||||
|
||||
# Configure CORS
|
||||
app.add_middleware(
|
||||
CORSMiddleware,
|
||||
allow_origins=["*"], # Configure this properly for production
|
||||
allow_credentials=True,
|
||||
allow_methods=["*"],
|
||||
allow_headers=["*"],
|
||||
)
|
||||
|
||||
# Include routers
|
||||
app.include_router(health.router, prefix="/api")
|
||||
app.include_router(jobs.router, prefix="/api")
|
||||
app.include_router(file_upload.router, prefix="/api") # File upload endpoint
|
||||
app.include_router(internal.router, prefix="/api") # Internal worker endpoints
|
||||
app.include_router(review.router, prefix="/api") # Review UI compatibility endpoints
|
||||
app.include_router(auth.router, prefix="/api") # OAuth credential management
|
||||
app.include_router(audio_search.router, prefix="/api") # Audio search (artist+title mode)
|
||||
app.include_router(themes.router, prefix="/api") # Theme selection for styles
|
||||
|
||||
|
||||
@app.get("/")
|
||||
async def root():
|
||||
"""Root endpoint."""
|
||||
return {
|
||||
"service": "karaoke-gen-backend",
|
||||
"version": VERSION,
|
||||
"status": "running"
|
||||
}
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
import uvicorn
|
||||
uvicorn.run(app, host="0.0.0.0", port=8080)
|
||||
|
||||
Whitespace-only changes.
@@ -1,499 +0,0 @@
|
||||
"""
|
||||
Job data models for karaoke generation.
|
||||
|
||||
This module defines the complete state machine for karaoke generation jobs,
|
||||
mirroring the 8-stage CLI workflow with human-in-the-loop interaction points.
|
||||
"""
|
||||
from typing import Optional, Dict, Any, List
|
||||
from datetime import datetime
|
||||
from enum import Enum
|
||||
from pydantic import BaseModel, Field, validator
|
||||
|
||||
|
||||
class JobStatus(str, Enum):
|
||||
"""
|
||||
Job status enumeration - Complete state machine.
|
||||
|
||||
The workflow has 8 main stages with 3 human interaction points:
|
||||
1. Input & Setup (may include audio source selection)
|
||||
2. Parallel Processing (audio + lyrics)
|
||||
3. Title/End Screen Generation
|
||||
4. Countdown Padding Synchronization
|
||||
5. Human Review (BLOCKING)
|
||||
6. Instrumental Selection (BLOCKING)
|
||||
7. Video Finalization
|
||||
8. Distribution
|
||||
"""
|
||||
# Initial states
|
||||
PENDING = "pending" # Job created, queued for processing
|
||||
|
||||
# Audio search states (for artist+title search mode)
|
||||
SEARCHING_AUDIO = "searching_audio" # Searching for audio sources via flacfetch
|
||||
AWAITING_AUDIO_SELECTION = "awaiting_audio_selection" # ⚠️ WAITING FOR USER - select audio source
|
||||
DOWNLOADING_AUDIO = "downloading_audio" # Downloading selected audio from source
|
||||
|
||||
DOWNLOADING = "downloading" # Downloading from URL or processing upload
|
||||
|
||||
# Stage 2a: Audio separation (parallel track 1)
|
||||
SEPARATING_STAGE1 = "separating_stage1" # Clean instrumental separation (Modal API)
|
||||
SEPARATING_STAGE2 = "separating_stage2" # Backing vocals separation (Modal API)
|
||||
AUDIO_COMPLETE = "audio_complete" # All audio stems ready
|
||||
|
||||
# Stage 2b: Lyrics processing (parallel track 2)
|
||||
TRANSCRIBING = "transcribing" # AudioShake API transcription
|
||||
CORRECTING = "correcting" # Automatic lyrics correction
|
||||
LYRICS_COMPLETE = "lyrics_complete" # Corrections JSON ready
|
||||
|
||||
# Stage 3: Title/End screens
|
||||
GENERATING_SCREENS = "generating_screens" # Creating title and end screen videos
|
||||
|
||||
# Stage 4: Countdown padding (automatic)
|
||||
APPLYING_PADDING = "applying_padding" # Synchronizing countdown padding
|
||||
|
||||
# Stage 5: Human review (BLOCKING)
|
||||
AWAITING_REVIEW = "awaiting_review" # ⚠️ WAITING FOR USER - lyrics review needed
|
||||
IN_REVIEW = "in_review" # User is actively reviewing lyrics
|
||||
REVIEW_COMPLETE = "review_complete" # User submitted corrected lyrics
|
||||
|
||||
# Stage 5.5: Render video with corrected lyrics (post-review)
|
||||
RENDERING_VIDEO = "rendering_video" # Using OutputGenerator to create with_vocals.mkv
|
||||
|
||||
# Stage 6: Instrumental selection (BLOCKING)
|
||||
AWAITING_INSTRUMENTAL_SELECTION = "awaiting_instrumental_selection" # ⚠️ WAITING FOR USER
|
||||
INSTRUMENTAL_SELECTED = "instrumental_selected" # User made selection
|
||||
|
||||
# Stage 7: Video generation and finalization
|
||||
GENERATING_VIDEO = "generating_video" # Creating initial karaoke video
|
||||
ENCODING = "encoding" # Multi-format video encoding (Cloud Build)
|
||||
PACKAGING = "packaging" # CDG/TXT ZIP generation
|
||||
|
||||
# Stage 8: Distribution (optional)
|
||||
UPLOADING = "uploading" # YouTube/Dropbox upload
|
||||
NOTIFYING = "notifying" # Discord/Email notifications
|
||||
|
||||
# Terminal states
|
||||
COMPLETE = "complete" # All processing finished successfully
|
||||
PREP_COMPLETE = "prep_complete" # Prep-only job completed (stops after review)
|
||||
FAILED = "failed" # Unrecoverable error occurred
|
||||
CANCELLED = "cancelled" # User cancelled the job
|
||||
|
||||
# Legacy compatibility (will be removed)
|
||||
QUEUED = "queued" # Deprecated: use PENDING
|
||||
PROCESSING = "processing" # Deprecated: use specific states
|
||||
READY_FOR_FINALIZATION = "ready_for_finalization" # Deprecated
|
||||
FINALIZING = "finalizing" # Deprecated: use ENCODING/PACKAGING
|
||||
ERROR = "error" # Deprecated: use FAILED
|
||||
|
||||
|
||||
# Valid state transitions
|
||||
STATE_TRANSITIONS = {
|
||||
# PENDING can go to DOWNLOADING (file upload), SEARCHING_AUDIO (artist+title search), or AWAITING_INSTRUMENTAL_SELECTION (finalise-only)
|
||||
JobStatus.PENDING: [JobStatus.DOWNLOADING, JobStatus.SEARCHING_AUDIO, JobStatus.AWAITING_INSTRUMENTAL_SELECTION, JobStatus.FAILED, JobStatus.CANCELLED],
|
||||
|
||||
# Audio search flow (for artist+title search mode)
|
||||
JobStatus.SEARCHING_AUDIO: [JobStatus.AWAITING_AUDIO_SELECTION, JobStatus.DOWNLOADING_AUDIO, JobStatus.FAILED],
|
||||
JobStatus.AWAITING_AUDIO_SELECTION: [JobStatus.DOWNLOADING_AUDIO, JobStatus.FAILED, JobStatus.CANCELLED],
|
||||
JobStatus.DOWNLOADING_AUDIO: [JobStatus.DOWNLOADING, JobStatus.FAILED],
|
||||
|
||||
# DOWNLOADING allows parallel processing (audio + lyrics) and then screens when both complete
|
||||
JobStatus.DOWNLOADING: [JobStatus.SEPARATING_STAGE1, JobStatus.TRANSCRIBING, JobStatus.GENERATING_SCREENS, JobStatus.FAILED],
|
||||
|
||||
# Audio separation flow
|
||||
JobStatus.SEPARATING_STAGE1: [JobStatus.SEPARATING_STAGE2, JobStatus.FAILED],
|
||||
JobStatus.SEPARATING_STAGE2: [JobStatus.AUDIO_COMPLETE, JobStatus.FAILED],
|
||||
JobStatus.AUDIO_COMPLETE: [JobStatus.GENERATING_SCREENS, JobStatus.FAILED],
|
||||
|
||||
# Lyrics flow
|
||||
JobStatus.TRANSCRIBING: [JobStatus.CORRECTING, JobStatus.FAILED],
|
||||
JobStatus.CORRECTING: [JobStatus.LYRICS_COMPLETE, JobStatus.FAILED],
|
||||
JobStatus.LYRICS_COMPLETE: [JobStatus.GENERATING_SCREENS, JobStatus.FAILED],
|
||||
|
||||
# Post-parallel processing
|
||||
JobStatus.GENERATING_SCREENS: [JobStatus.APPLYING_PADDING, JobStatus.AWAITING_REVIEW, JobStatus.AWAITING_INSTRUMENTAL_SELECTION, JobStatus.FAILED],
|
||||
JobStatus.APPLYING_PADDING: [JobStatus.AWAITING_REVIEW, JobStatus.FAILED],
|
||||
|
||||
# Human review flow
|
||||
# AWAITING_REVIEW can go directly to REVIEW_COMPLETE (quick review) or to IN_REVIEW (editing)
|
||||
JobStatus.AWAITING_REVIEW: [JobStatus.IN_REVIEW, JobStatus.REVIEW_COMPLETE, JobStatus.FAILED, JobStatus.CANCELLED],
|
||||
JobStatus.IN_REVIEW: [JobStatus.REVIEW_COMPLETE, JobStatus.AWAITING_REVIEW, JobStatus.FAILED],
|
||||
JobStatus.REVIEW_COMPLETE: [JobStatus.RENDERING_VIDEO, JobStatus.PREP_COMPLETE, JobStatus.FAILED], # PREP_COMPLETE for prep-only jobs
|
||||
|
||||
# Video rendering (post-review)
|
||||
JobStatus.RENDERING_VIDEO: [JobStatus.AWAITING_INSTRUMENTAL_SELECTION, JobStatus.PREP_COMPLETE, JobStatus.FAILED],
|
||||
|
||||
# Instrumental selection flow
|
||||
JobStatus.AWAITING_INSTRUMENTAL_SELECTION: [JobStatus.INSTRUMENTAL_SELECTED, JobStatus.FAILED, JobStatus.CANCELLED],
|
||||
JobStatus.INSTRUMENTAL_SELECTED: [JobStatus.GENERATING_VIDEO, JobStatus.FAILED],
|
||||
|
||||
# Video generation flow
|
||||
JobStatus.GENERATING_VIDEO: [JobStatus.ENCODING, JobStatus.FAILED],
|
||||
JobStatus.ENCODING: [JobStatus.PACKAGING, JobStatus.COMPLETE, JobStatus.FAILED],
|
||||
JobStatus.PACKAGING: [JobStatus.UPLOADING, JobStatus.COMPLETE, JobStatus.FAILED],
|
||||
|
||||
# Distribution flow
|
||||
JobStatus.UPLOADING: [JobStatus.NOTIFYING, JobStatus.COMPLETE, JobStatus.FAILED],
|
||||
JobStatus.NOTIFYING: [JobStatus.COMPLETE, JobStatus.FAILED],
|
||||
|
||||
# Terminal states - COMPLETE, PREP_COMPLETE have no transitions
|
||||
# FAILED and CANCELLED allow retry transitions to resume from checkpoints
|
||||
# PREP_COMPLETE allows finalise-only continuation
|
||||
JobStatus.COMPLETE: [],
|
||||
JobStatus.PREP_COMPLETE: [JobStatus.AWAITING_INSTRUMENTAL_SELECTION, JobStatus.FAILED], # Finalise-only continues from here
|
||||
JobStatus.FAILED: [
|
||||
JobStatus.DOWNLOADING, # Retry from beginning (if input audio exists)
|
||||
JobStatus.INSTRUMENTAL_SELECTED, # Retry from video generation
|
||||
JobStatus.REVIEW_COMPLETE, # Retry from render stage
|
||||
JobStatus.LYRICS_COMPLETE, # Retry from screens generation
|
||||
],
|
||||
JobStatus.CANCELLED: [
|
||||
JobStatus.DOWNLOADING, # Retry from beginning (if input audio exists)
|
||||
JobStatus.INSTRUMENTAL_SELECTED, # Retry from video generation
|
||||
JobStatus.REVIEW_COMPLETE, # Retry from render stage
|
||||
JobStatus.LYRICS_COMPLETE, # Retry from screens generation
|
||||
],
|
||||
|
||||
# Legacy states (for backward compatibility)
|
||||
JobStatus.QUEUED: [JobStatus.PENDING],
|
||||
JobStatus.PROCESSING: [JobStatus.SEPARATING_STAGE1, JobStatus.TRANSCRIBING],
|
||||
JobStatus.READY_FOR_FINALIZATION: [JobStatus.GENERATING_VIDEO],
|
||||
JobStatus.FINALIZING: [JobStatus.ENCODING],
|
||||
JobStatus.ERROR: [JobStatus.FAILED],
|
||||
}
|
||||
|
||||
|
||||
class TimelineEvent(BaseModel):
|
||||
"""Timeline event for job progress tracking."""
|
||||
status: str
|
||||
timestamp: str
|
||||
progress: Optional[int] = None
|
||||
message: Optional[str] = None
|
||||
|
||||
|
||||
class LogEntry(BaseModel):
|
||||
"""Worker log entry for debugging and monitoring."""
|
||||
timestamp: str
|
||||
level: str # DEBUG, INFO, WARNING, ERROR
|
||||
worker: str # audio, lyrics, screens, video, render
|
||||
message: str
|
||||
|
||||
|
||||
class Job(BaseModel):
|
||||
"""
|
||||
Complete job data model.
|
||||
|
||||
Tracks the full lifecycle of a karaoke generation job from submission
|
||||
through all 8 stages to completion or failure.
|
||||
"""
|
||||
job_id: str
|
||||
status: JobStatus
|
||||
progress: int = 0 # 0-100 percentage for UI display
|
||||
created_at: datetime
|
||||
updated_at: datetime
|
||||
|
||||
# Input data
|
||||
url: Optional[str] = None # YouTube URL (if provided)
|
||||
artist: Optional[str] = None
|
||||
title: Optional[str] = None
|
||||
filename: Optional[str] = None # Original uploaded filename
|
||||
input_media_gcs_path: Optional[str] = None # GCS path to uploaded file
|
||||
|
||||
# User preferences
|
||||
enable_cdg: bool = False # Generate CDG+MP3 package (requires style config)
|
||||
enable_txt: bool = False # Generate TXT+MP3 package (requires style config)
|
||||
enable_youtube_upload: bool = False # Upload to YouTube
|
||||
youtube_description: Optional[str] = None # YouTube video description
|
||||
webhook_url: Optional[str] = None # Webhook for notifications
|
||||
user_email: Optional[str] = None # Email for notifications
|
||||
|
||||
# Theme configuration (pre-made themes from GCS)
|
||||
theme_id: Optional[str] = None # Theme identifier (e.g., "nomad", "default")
|
||||
color_overrides: Dict[str, str] = Field(default_factory=dict)
|
||||
"""
|
||||
User color overrides applied on top of theme. Keys:
|
||||
- artist_color: Hex color for artist name (#RRGGBB)
|
||||
- title_color: Hex color for song title
|
||||
- sung_lyrics_color: Hex color for highlighted lyrics
|
||||
- unsung_lyrics_color: Hex color for unhighlighted lyrics
|
||||
"""
|
||||
|
||||
# Style configuration (uploaded files - used when theme_id is not set)
|
||||
style_params_gcs_path: Optional[str] = None # GCS path to style_params.json
|
||||
style_assets: Dict[str, str] = Field(default_factory=dict)
|
||||
"""
|
||||
GCS paths to style asset files:
|
||||
{
|
||||
"intro_background": "gs://bucket/jobs/{job_id}/style/intro_bg.png",
|
||||
"karaoke_background": "gs://bucket/jobs/{job_id}/style/karaoke_bg.png",
|
||||
"end_background": "gs://bucket/jobs/{job_id}/style/end_bg.png",
|
||||
"font": "gs://bucket/jobs/{job_id}/style/font.ttf",
|
||||
"cdg_instrumental_background": "gs://bucket/jobs/{job_id}/style/cdg_instr.png",
|
||||
"cdg_title_background": "gs://bucket/jobs/{job_id}/style/cdg_title.png",
|
||||
"cdg_outro_background": "gs://bucket/jobs/{job_id}/style/cdg_outro.png"
|
||||
}
|
||||
"""
|
||||
|
||||
# Finalisation configuration
|
||||
brand_prefix: Optional[str] = None # Brand code prefix (e.g., "NOMAD")
|
||||
discord_webhook_url: Optional[str] = None # Discord notification webhook
|
||||
youtube_description_template: Optional[str] = None # YouTube description template text
|
||||
|
||||
# Distribution configuration (native API - for remote CLI)
|
||||
dropbox_path: Optional[str] = None # Dropbox folder path for organized output (e.g., "/Karaoke/Tracks-Organized")
|
||||
gdrive_folder_id: Optional[str] = None # Google Drive folder ID for public share uploads
|
||||
|
||||
# Legacy distribution configuration (rclone - for local CLI backward compat)
|
||||
organised_dir_rclone_root: Optional[str] = None # Deprecated: use dropbox_path instead
|
||||
|
||||
# Lyrics configuration (overrides for search/transcription)
|
||||
lyrics_artist: Optional[str] = None # Override artist name for lyrics search
|
||||
lyrics_title: Optional[str] = None # Override title for lyrics search
|
||||
lyrics_file_gcs_path: Optional[str] = None # GCS path to user-provided lyrics file
|
||||
subtitle_offset_ms: int = 0 # Offset for subtitle timing (positive = delay)
|
||||
|
||||
# Audio separation model configuration
|
||||
clean_instrumental_model: Optional[str] = None # Model for clean instrumental separation
|
||||
backing_vocals_models: Optional[List[str]] = None # Models for backing vocals separation
|
||||
other_stems_models: Optional[List[str]] = None # Models for other stems (bass, drums, etc.)
|
||||
|
||||
# Existing instrumental configuration (Batch 3)
|
||||
existing_instrumental_gcs_path: Optional[str] = None # GCS path to user-provided instrumental file
|
||||
|
||||
# Audio search configuration (Batch 5 - artist+title search mode)
|
||||
audio_search_artist: Optional[str] = None # Artist name used for audio search
|
||||
audio_search_title: Optional[str] = None # Title used for audio search
|
||||
auto_download: bool = False # Auto-select best audio source (skip selection)
|
||||
|
||||
# Two-phase workflow configuration (Batch 6)
|
||||
prep_only: bool = False # Stop after review, don't run finalisation
|
||||
finalise_only: bool = False # Skip prep, run only finalisation (requires uploaded prep outputs)
|
||||
keep_brand_code: Optional[str] = None # Preserve existing brand code instead of generating new one
|
||||
|
||||
# Review authentication (Batch 7)
|
||||
review_token: Optional[str] = None # Job-scoped token for lyrics review UI access (generated when entering AWAITING_REVIEW)
|
||||
review_token_expires_at: Optional[datetime] = None # Token expiry time (optional, for extra security)
|
||||
instrumental_token: Optional[str] = None # Job-scoped token for instrumental review UI access (generated when entering AWAITING_INSTRUMENTAL_SELECTION)
|
||||
instrumental_token_expires_at: Optional[datetime] = None # Token expiry time
|
||||
|
||||
# Processing state
|
||||
track_output_dir: Optional[str] = None # Local output directory (temp)
|
||||
audio_hash: Optional[str] = None # Hash for deduplication
|
||||
|
||||
# State-specific data (JSON field for stage-specific metadata)
|
||||
state_data: Dict[str, Any] = Field(default_factory=dict)
|
||||
"""
|
||||
Stage-specific metadata. Examples:
|
||||
- audio_complete: {"stems": {"clean": "gs://...", "backing": "gs://..."}}
|
||||
- lyrics_complete: {"corrections_url": "gs://...", "audio_url": "gs://..."}
|
||||
- review_complete: {"corrected_lyrics": {...}}
|
||||
- instrumental_selected: {"selection": "clean" | "with_backing"}
|
||||
- encoding: {"build_id": "...", "progress": 45}
|
||||
"""
|
||||
|
||||
# Timeline tracking
|
||||
timeline: List[TimelineEvent] = Field(default_factory=list)
|
||||
|
||||
# Worker logs for debugging (limited to last N entries to avoid document size issues)
|
||||
worker_logs: List[LogEntry] = Field(default_factory=list)
|
||||
|
||||
# File URLs (GCS storage)
|
||||
file_urls: Dict[str, Any] = Field(default_factory=dict)
|
||||
"""
|
||||
File storage URLs. Structure:
|
||||
{
|
||||
"input": "gs://bucket/jobs/{job_id}/input.flac",
|
||||
"stems": {
|
||||
"instrumental_clean": "gs://...",
|
||||
"instrumental_with_backing": "gs://...",
|
||||
"vocals": "gs://...",
|
||||
"backing_vocals": "gs://...",
|
||||
"lead_vocals": "gs://...",
|
||||
"bass": "gs://...",
|
||||
"drums": "gs://...",
|
||||
"guitar": "gs://...",
|
||||
"piano": "gs://...",
|
||||
"other": "gs://..."
|
||||
},
|
||||
"lyrics": {
|
||||
"corrections": "gs://bucket/jobs/{job_id}/lyrics/corrections.json",
|
||||
"audio": "gs://bucket/jobs/{job_id}/lyrics/audio.flac",
|
||||
"lrc": "gs://...",
|
||||
"ass": "gs://..."
|
||||
},
|
||||
"screens": {
|
||||
"title": "gs://bucket/jobs/{job_id}/screens/title.mov",
|
||||
"end": "gs://bucket/jobs/{job_id}/screens/end.mov"
|
||||
},
|
||||
"videos": {
|
||||
"with_vocals": "gs://bucket/jobs/{job_id}/videos/with_vocals.mkv"
|
||||
},
|
||||
"finals": {
|
||||
"lossless_4k_mp4": "gs://...",
|
||||
"lossless_4k_mkv": "gs://...",
|
||||
"lossy_4k_mp4": "gs://...",
|
||||
"lossy_720p_mp4": "gs://..."
|
||||
},
|
||||
"packages": {
|
||||
"cdg_zip": "gs://...",
|
||||
"txt_zip": "gs://..."
|
||||
},
|
||||
"youtube": {
|
||||
"url": "https://youtube.com/watch?v=...",
|
||||
"video_id": "..."
|
||||
}
|
||||
}
|
||||
"""
|
||||
|
||||
# Results (for backward compatibility, will be deprecated)
|
||||
output_files: Dict[str, str] = Field(default_factory=dict)
|
||||
download_urls: Dict[str, str] = Field(default_factory=dict)
|
||||
|
||||
# Error handling
|
||||
error_message: Optional[str] = None
|
||||
error_details: Optional[Dict[str, Any]] = None # Structured error information
|
||||
retry_count: int = 0 # Number of retry attempts
|
||||
|
||||
# Worker tracking
|
||||
worker_ids: Dict[str, str] = Field(default_factory=dict)
|
||||
"""
|
||||
IDs of background workers/jobs:
|
||||
{
|
||||
"audio_worker": "cloud-run-request-id",
|
||||
"lyrics_worker": "cloud-run-request-id",
|
||||
"video_encoder": "cloud-build-id",
|
||||
"distribution": "cloud-run-request-id"
|
||||
}
|
||||
"""
|
||||
|
||||
# Request metadata (captured at job creation for tracking and filtering)
|
||||
request_metadata: Dict[str, Any] = Field(default_factory=dict)
|
||||
"""
|
||||
Metadata captured from the original API request.
|
||||
Used for tracking, filtering, and operational management.
|
||||
|
||||
Standard fields:
|
||||
{
|
||||
"client_ip": "192.168.1.1", # IP address of the client
|
||||
"user_agent": "karaoke-gen-remote/0.71.0", # User-Agent header
|
||||
"environment": "test", # From X-Environment header (test/production/development)
|
||||
"client_id": "cli-user-123", # From X-Client-ID header (customer/user identifier)
|
||||
"server_version": "0.71.0", # Server version at job creation
|
||||
"created_from": "upload", # "upload" (file) or "url" (YouTube URL)
|
||||
"custom_headers": { # All X-* headers for extensibility
|
||||
"X-Environment": "test",
|
||||
"X-Client-ID": "cli-user-123",
|
||||
"X-Request-ID": "abc-123"
|
||||
}
|
||||
}
|
||||
|
||||
Use cases:
|
||||
- Filter test vs production jobs
|
||||
- Track jobs by customer/client
|
||||
- Debug issues with specific clients
|
||||
- Bulk cleanup of test jobs
|
||||
"""
|
||||
|
||||
# Note: Status transition validation is handled by JobManager.validate_state_transition()
|
||||
# which is called before status updates. The Job model does not validate transitions
|
||||
# because Firestore updates happen directly without reconstructing the model.
|
||||
|
||||
class Config:
|
||||
use_enum_values = True
|
||||
|
||||
|
||||
class JobCreate(BaseModel):
|
||||
"""
|
||||
Job creation request.
|
||||
|
||||
Either `url` OR file upload is required (file upload handled separately).
|
||||
Artist and title are optional - will be auto-detected from YouTube if not provided.
|
||||
"""
|
||||
url: Optional[str] = None
|
||||
artist: Optional[str] = None
|
||||
title: Optional[str] = None
|
||||
filename: Optional[str] = None # Original uploaded filename
|
||||
|
||||
# Optional preferences
|
||||
enable_cdg: bool = False # Requires style config
|
||||
enable_txt: bool = False # Requires style config
|
||||
enable_youtube_upload: bool = False
|
||||
youtube_description: Optional[str] = None
|
||||
webhook_url: Optional[str] = None
|
||||
user_email: Optional[str] = None
|
||||
|
||||
# Theme configuration (pre-made themes from GCS)
|
||||
theme_id: Optional[str] = None # Theme identifier (e.g., "nomad", "default")
|
||||
color_overrides: Dict[str, str] = Field(default_factory=dict)
|
||||
"""
|
||||
User color overrides applied on top of theme. Keys:
|
||||
- artist_color: Hex color for artist name (#RRGGBB)
|
||||
- title_color: Hex color for song title
|
||||
- sung_lyrics_color: Hex color for highlighted lyrics
|
||||
- unsung_lyrics_color: Hex color for unhighlighted lyrics
|
||||
"""
|
||||
|
||||
# Style configuration (will be populated after file upload, or from theme)
|
||||
style_params_gcs_path: Optional[str] = None
|
||||
style_assets: Dict[str, str] = Field(default_factory=dict)
|
||||
|
||||
# Finalisation configuration
|
||||
brand_prefix: Optional[str] = None
|
||||
discord_webhook_url: Optional[str] = None
|
||||
youtube_description_template: Optional[str] = None
|
||||
|
||||
# Distribution configuration (native API - for remote CLI)
|
||||
dropbox_path: Optional[str] = None # Dropbox folder path for organized output
|
||||
gdrive_folder_id: Optional[str] = None # Google Drive folder ID for public share uploads
|
||||
|
||||
# Legacy (rclone - deprecated, use dropbox_path instead)
|
||||
organised_dir_rclone_root: Optional[str] = None
|
||||
|
||||
# Lyrics configuration (overrides for search/transcription)
|
||||
lyrics_artist: Optional[str] = None # Override artist name for lyrics search
|
||||
lyrics_title: Optional[str] = None # Override title for lyrics search
|
||||
lyrics_file_gcs_path: Optional[str] = None # GCS path to user-provided lyrics file
|
||||
subtitle_offset_ms: int = 0 # Offset for subtitle timing (positive = delay)
|
||||
|
||||
# Audio separation model configuration
|
||||
clean_instrumental_model: Optional[str] = None # Model for clean instrumental separation
|
||||
backing_vocals_models: Optional[List[str]] = None # Models for backing vocals separation
|
||||
other_stems_models: Optional[List[str]] = None # Models for other stems (bass, drums, etc.)
|
||||
|
||||
# Existing instrumental configuration (Batch 3)
|
||||
existing_instrumental_gcs_path: Optional[str] = None # GCS path to user-provided instrumental file
|
||||
|
||||
# Audio search configuration (Batch 5 - artist+title search mode)
|
||||
audio_search_artist: Optional[str] = None # Artist name used for audio search
|
||||
audio_search_title: Optional[str] = None # Title used for audio search
|
||||
auto_download: bool = False # Auto-select best audio source (skip selection)
|
||||
|
||||
# Two-phase workflow configuration (Batch 6)
|
||||
prep_only: bool = False # Stop after review, don't run finalisation
|
||||
finalise_only: bool = False # Skip prep, run only finalisation
|
||||
keep_brand_code: Optional[str] = None # Preserve existing brand code instead of generating new one
|
||||
|
||||
# Request metadata (set by API endpoint from request headers)
|
||||
request_metadata: Dict[str, Any] = Field(default_factory=dict)
|
||||
"""
|
||||
Populated by the API endpoint with request context:
|
||||
- client_ip: Client IP address
|
||||
- user_agent: User-Agent header
|
||||
- environment: From X-Environment header (test/production/development)
|
||||
- client_id: From X-Client-ID header
|
||||
- server_version: Current server version
|
||||
- custom_headers: All X-* headers
|
||||
"""
|
||||
|
||||
@validator('url', 'artist', 'title')
|
||||
def validate_inputs(cls, v):
|
||||
"""Validate string inputs are not empty."""
|
||||
if v is not None and isinstance(v, str) and not v.strip():
|
||||
raise ValueError("Field cannot be empty string")
|
||||
return v.strip() if isinstance(v, str) else v
|
||||
|
||||
|
||||
class JobResponse(BaseModel):
|
||||
"""Job response model."""
|
||||
status: str
|
||||
job_id: str
|
||||
message: str
|
||||
|
||||
@@ -1,123 +0,0 @@
|
||||
"""
|
||||
API request models for karaoke generation endpoints.
|
||||
"""
|
||||
from typing import Optional, Dict, Any, List
|
||||
from pydantic import BaseModel, HttpUrl, validator
|
||||
|
||||
|
||||
class URLSubmissionRequest(BaseModel):
|
||||
"""Request to submit a job from a URL (YouTube, etc.)."""
|
||||
url: HttpUrl
|
||||
artist: Optional[str] = None # Auto-detected if not provided
|
||||
title: Optional[str] = None # Auto-detected if not provided
|
||||
|
||||
# Optional preferences
|
||||
enable_cdg: bool = False # Requires style config
|
||||
enable_txt: bool = False # Requires style config
|
||||
enable_youtube_upload: bool = False
|
||||
youtube_description: Optional[str] = None
|
||||
webhook_url: Optional[str] = None
|
||||
user_email: Optional[str] = None
|
||||
|
||||
|
||||
class UploadSubmissionRequest(BaseModel):
|
||||
"""Request to submit a job from an uploaded file."""
|
||||
artist: str
|
||||
title: str
|
||||
|
||||
# Optional preferences
|
||||
enable_cdg: bool = False # Requires style config
|
||||
enable_txt: bool = False # Requires style config
|
||||
enable_youtube_upload: bool = False
|
||||
youtube_description: Optional[str] = None
|
||||
webhook_url: Optional[str] = None
|
||||
user_email: Optional[str] = None
|
||||
|
||||
|
||||
class CorrectionsSubmission(BaseModel):
|
||||
"""
|
||||
Request to submit corrected lyrics after human review.
|
||||
|
||||
This is the critical human-in-the-loop interaction point.
|
||||
The corrections data comes from the lyrics-transcriber review interface.
|
||||
"""
|
||||
corrections: Dict[str, Any] # Full corrections JSON from frontend
|
||||
user_notes: Optional[str] = None # Optional notes from reviewer
|
||||
|
||||
@validator('corrections')
|
||||
def validate_corrections_format(cls, v):
|
||||
"""Validate corrections has required fields."""
|
||||
required_fields = ['lines', 'metadata']
|
||||
for field in required_fields:
|
||||
if field not in v:
|
||||
raise ValueError(f"Corrections must include '{field}' field")
|
||||
return v
|
||||
|
||||
|
||||
class InstrumentalSelection(BaseModel):
|
||||
"""
|
||||
Request to select instrumental audio option.
|
||||
|
||||
This is the second critical human-in-the-loop interaction point.
|
||||
User chooses between clean instrumental, instrumental with backing vocals,
|
||||
or a custom instrumental (created via create-custom-instrumental endpoint).
|
||||
"""
|
||||
selection: str # "clean", "with_backing", or "custom"
|
||||
|
||||
@validator('selection')
|
||||
def validate_selection(cls, v):
|
||||
"""Validate selection is a valid option."""
|
||||
valid_options = ['clean', 'with_backing', 'custom']
|
||||
if v not in valid_options:
|
||||
raise ValueError(f"Selection must be one of: {valid_options}")
|
||||
return v
|
||||
|
||||
|
||||
class MuteRegionRequest(BaseModel):
|
||||
"""A region to mute in the backing vocals."""
|
||||
start_seconds: float
|
||||
end_seconds: float
|
||||
|
||||
@validator('start_seconds')
|
||||
def validate_start(cls, v):
|
||||
if v < 0:
|
||||
raise ValueError("start_seconds must be non-negative")
|
||||
return v
|
||||
|
||||
@validator('end_seconds')
|
||||
def validate_end(cls, v, values):
|
||||
if 'start_seconds' in values and v <= values['start_seconds']:
|
||||
raise ValueError("end_seconds must be greater than start_seconds")
|
||||
return v
|
||||
|
||||
|
||||
class CreateCustomInstrumentalRequest(BaseModel):
|
||||
"""
|
||||
Request to create a custom instrumental with muted backing vocal regions.
|
||||
|
||||
The mute_regions specify time ranges in the backing vocals track that
|
||||
should be silenced before mixing with the clean instrumental.
|
||||
"""
|
||||
mute_regions: List[MuteRegionRequest]
|
||||
|
||||
@validator('mute_regions')
|
||||
def validate_regions(cls, v):
|
||||
if not v:
|
||||
raise ValueError("At least one mute region is required")
|
||||
return v
|
||||
|
||||
|
||||
class StartReviewRequest(BaseModel):
|
||||
"""Request to mark job as in-review (user opened interface)."""
|
||||
pass # No body needed, just triggers state transition
|
||||
|
||||
|
||||
class CancelJobRequest(BaseModel):
|
||||
"""Request to cancel a job."""
|
||||
reason: Optional[str] = None
|
||||
|
||||
|
||||
class RetryJobRequest(BaseModel):
|
||||
"""Request to retry a failed job."""
|
||||
from_stage: Optional[str] = None # Optional: restart from specific stage
|
||||
|
||||
@@ -1,153 +0,0 @@
|
||||
"""
|
||||
Theme data models for karaoke-gen.
|
||||
|
||||
Themes are pre-created style configurations stored in GCS that users can select
|
||||
when creating karaoke jobs. Each theme includes:
|
||||
- Style parameters (fonts, colors, backgrounds, layouts)
|
||||
- Asset files (images, fonts)
|
||||
- Preview images
|
||||
- Optional YouTube description template
|
||||
"""
|
||||
|
||||
from datetime import datetime
|
||||
from typing import Any, Dict, List, Optional
|
||||
|
||||
from pydantic import BaseModel, Field
|
||||
|
||||
|
||||
class ThemeSummary(BaseModel):
|
||||
"""Summary of a theme for listing purposes."""
|
||||
|
||||
id: str = Field(..., description="Unique theme identifier (e.g., 'nomad', 'default')")
|
||||
name: str = Field(..., description="Human-readable theme name")
|
||||
description: str = Field(..., description="Brief description of the theme style")
|
||||
preview_url: Optional[str] = Field(None, description="Signed URL for preview image")
|
||||
thumbnail_url: Optional[str] = Field(None, description="Signed URL for smaller thumbnail")
|
||||
is_default: bool = Field(False, description="Whether this is the default theme")
|
||||
|
||||
|
||||
class ThemeMetadata(BaseModel):
|
||||
"""Metadata for a single theme in the registry."""
|
||||
|
||||
id: str
|
||||
name: str
|
||||
description: str
|
||||
is_default: bool = False
|
||||
created_at: Optional[datetime] = None
|
||||
|
||||
|
||||
class ThemeRegistry(BaseModel):
|
||||
"""Registry of all available themes, stored as _metadata.json in GCS."""
|
||||
|
||||
version: int = 1
|
||||
themes: List[ThemeMetadata] = Field(default_factory=list)
|
||||
|
||||
|
||||
class ThemeDetail(BaseModel):
|
||||
"""Full theme details including style parameters."""
|
||||
|
||||
id: str
|
||||
name: str
|
||||
description: str
|
||||
preview_url: Optional[str] = None
|
||||
is_default: bool = False
|
||||
style_params: Dict[str, Any] = Field(default_factory=dict)
|
||||
has_youtube_description: bool = False
|
||||
|
||||
|
||||
class ColorOverrides(BaseModel):
|
||||
"""
|
||||
User-customizable color overrides that can be applied on top of a theme.
|
||||
|
||||
All colors are hex format (#RRGGBB). When applied:
|
||||
- artist_color: Applied to intro.artist_color, end.artist_color, cdg.artist_color
|
||||
- title_color: Applied to intro.title_color, end.title_color, cdg.title_color
|
||||
- sung_lyrics_color: Applied to karaoke.primary_color (converted to RGBA), cdg.active_fill
|
||||
- unsung_lyrics_color: Applied to karaoke.secondary_color (converted to RGBA), cdg.inactive_fill
|
||||
"""
|
||||
|
||||
artist_color: Optional[str] = Field(
|
||||
None,
|
||||
pattern=r"^#[0-9A-Fa-f]{6}$",
|
||||
description="Hex color for artist name on intro/end screens",
|
||||
)
|
||||
title_color: Optional[str] = Field(
|
||||
None,
|
||||
pattern=r"^#[0-9A-Fa-f]{6}$",
|
||||
description="Hex color for song title on intro/end screens",
|
||||
)
|
||||
sung_lyrics_color: Optional[str] = Field(
|
||||
None,
|
||||
pattern=r"^#[0-9A-Fa-f]{6}$",
|
||||
description="Hex color for lyrics being sung (highlighted)",
|
||||
)
|
||||
unsung_lyrics_color: Optional[str] = Field(
|
||||
None,
|
||||
pattern=r"^#[0-9A-Fa-f]{6}$",
|
||||
description="Hex color for lyrics not yet sung",
|
||||
)
|
||||
|
||||
def has_overrides(self) -> bool:
|
||||
"""Check if any color overrides are set."""
|
||||
return any(
|
||||
[
|
||||
self.artist_color,
|
||||
self.title_color,
|
||||
self.sung_lyrics_color,
|
||||
self.unsung_lyrics_color,
|
||||
]
|
||||
)
|
||||
|
||||
def to_dict(self) -> Dict[str, str]:
|
||||
"""Convert to dict, excluding None values."""
|
||||
return {k: v for k, v in self.model_dump().items() if v is not None}
|
||||
|
||||
|
||||
class ThemesListResponse(BaseModel):
|
||||
"""Response from GET /api/themes endpoint."""
|
||||
|
||||
themes: List[ThemeSummary]
|
||||
|
||||
|
||||
class ThemeDetailResponse(BaseModel):
|
||||
"""Response from GET /api/themes/{theme_id} endpoint."""
|
||||
|
||||
theme: ThemeDetail
|
||||
|
||||
|
||||
# Utility functions for color conversion
|
||||
|
||||
|
||||
def hex_to_rgba(hex_color: str, alpha: int = 255) -> str:
|
||||
"""
|
||||
Convert hex color (#RRGGBB) to RGBA format for ASS subtitles.
|
||||
|
||||
ASS subtitle format uses "R, G, B, A" string format.
|
||||
|
||||
Args:
|
||||
hex_color: Hex color string like "#7070F7"
|
||||
alpha: Alpha value 0-255 (default 255 = opaque)
|
||||
|
||||
Returns:
|
||||
RGBA string like "112, 112, 247, 255"
|
||||
"""
|
||||
hex_color = hex_color.lstrip("#")
|
||||
r = int(hex_color[0:2], 16)
|
||||
g = int(hex_color[2:4], 16)
|
||||
b = int(hex_color[4:6], 16)
|
||||
return f"{r}, {g}, {b}, {alpha}"
|
||||
|
||||
|
||||
def rgba_to_hex(rgba_str: str) -> str:
|
||||
"""
|
||||
Convert RGBA format to hex color.
|
||||
|
||||
Args:
|
||||
rgba_str: RGBA string like "112, 112, 247, 255"
|
||||
|
||||
Returns:
|
||||
Hex color string like "#7070F7"
|
||||
"""
|
||||
parts = [int(x.strip()) for x in rgba_str.split(",")]
|
||||
r, g, b = parts[0], parts[1], parts[2]
|
||||
return f"#{r:02x}{g:02x}{b:02x}"
|
||||
@@ -1,29 +0,0 @@
|
||||
[tool.pytest.ini_options]
|
||||
testpaths = ["tests"]
|
||||
python_files = ["test_*.py"]
|
||||
python_classes = ["Test*"]
|
||||
python_functions = ["test_*"]
|
||||
addopts = [
|
||||
"-v",
|
||||
"--tb=short",
|
||||
"--strict-markers",
|
||||
"-ra",
|
||||
]
|
||||
|
||||
markers = [
|
||||
"slow: marks tests as slow (deselect with '-m \"not slow\"')",
|
||||
"integration: marks tests as integration tests requiring live service",
|
||||
]
|
||||
|
||||
# Timeout for slow tests (in seconds)
|
||||
timeout = 300
|
||||
|
||||
[tool.coverage.run]
|
||||
source = ["backend"]
|
||||
omit = ["*/tests/*", "*/test_*"]
|
||||
|
||||
[tool.coverage.report]
|
||||
precision = 2
|
||||
show_missing = true
|
||||
skip_covered = false
|
||||
|
||||
@@ -1,93 +0,0 @@
|
||||
#!/bin/bash
|
||||
# Quick syntax and structure check (no dependencies required)
|
||||
# Run this before every deployment!
|
||||
|
||||
set -e
|
||||
|
||||
echo "🔍 Quick Backend Check (no dependencies required)"
|
||||
echo "=================================================="
|
||||
echo ""
|
||||
|
||||
# Change to backend directory
|
||||
cd "$(dirname "$0")"
|
||||
|
||||
# Colors
|
||||
RED='\033[0;31m'
|
||||
GREEN='\033[0;32m'
|
||||
NC='\033[0m'
|
||||
|
||||
ERRORS=0
|
||||
|
||||
# Check 1: Python syntax
|
||||
echo "1. Checking Python syntax..."
|
||||
for file in $(find . -name "*.py" -not -path "./__pycache__/*" -not -path "./venv/*"); do
|
||||
if ! python3 -m py_compile "$file" 2>/dev/null; then
|
||||
echo -e " ${RED}❌ Syntax error in $file${NC}"
|
||||
python3 -m py_compile "$file"
|
||||
ERRORS=$((ERRORS+1))
|
||||
fi
|
||||
done
|
||||
|
||||
if [ $ERRORS -eq 0 ]; then
|
||||
echo -e " ${GREEN}✅ All Python files have valid syntax${NC}"
|
||||
fi
|
||||
|
||||
# Check 2: No missing imports (basic check)
|
||||
echo ""
|
||||
echo "2. Checking for obvious import issues..."
|
||||
|
||||
# Check that we don't import deleted modules
|
||||
if grep -r "from backend.services.processing_service import" --include="*.py" . 2>/dev/null; then
|
||||
echo -e " ${RED}❌ Found import of deleted processing_service${NC}"
|
||||
ERRORS=$((ERRORS+1))
|
||||
else
|
||||
echo -e " ${GREEN}✅ No imports of deleted modules${NC}"
|
||||
fi
|
||||
|
||||
# Check 3: Required files exist
|
||||
echo ""
|
||||
echo "3. Checking required files exist..."
|
||||
REQUIRED_FILES=(
|
||||
"main.py"
|
||||
"config.py"
|
||||
"requirements.txt"
|
||||
"Dockerfile"
|
||||
"api/routes/health.py"
|
||||
"api/routes/jobs.py"
|
||||
"api/routes/internal.py"
|
||||
"api/routes/file_upload.py"
|
||||
"services/job_manager.py"
|
||||
"services/firestore_service.py"
|
||||
"services/storage_service.py"
|
||||
"services/worker_service.py"
|
||||
"services/auth_service.py"
|
||||
"workers/audio_worker.py"
|
||||
"workers/lyrics_worker.py"
|
||||
"workers/screens_worker.py"
|
||||
"workers/video_worker.py"
|
||||
"models/job.py"
|
||||
"models/requests.py"
|
||||
)
|
||||
|
||||
for file in "${REQUIRED_FILES[@]}"; do
|
||||
if [ ! -f "$file" ]; then
|
||||
echo -e " ${RED}❌ Missing required file: $file${NC}"
|
||||
ERRORS=$((ERRORS+1))
|
||||
fi
|
||||
done
|
||||
|
||||
if [ $ERRORS -eq 0 ]; then
|
||||
echo -e " ${GREEN}✅ All required files exist${NC}"
|
||||
fi
|
||||
|
||||
# Summary
|
||||
echo ""
|
||||
echo "=================================================="
|
||||
if [ $ERRORS -eq 0 ]; then
|
||||
echo -e "${GREEN}✅ All checks passed! Safe to deploy.${NC}"
|
||||
exit 0
|
||||
else
|
||||
echo -e "${RED}❌ $ERRORS error(s) found. Fix before deploying!${NC}"
|
||||
exit 1
|
||||
fi
|
||||
|
||||
@@ -1,29 +0,0 @@
|
||||
# DEPRECATED: This file is no longer used for Docker builds or CI.
|
||||
# Docker and CI both use poetry/pyproject.toml directly.
|
||||
# This file is kept for reference only.
|
||||
|
||||
fastapi>=0.104.0,<1.0.0
|
||||
uvicorn[standard]>=0.24.0,<1.0.0
|
||||
pydantic>=2.5.0,<3.0.0
|
||||
pydantic-settings>=2.1.0,<3.0.0
|
||||
python-multipart>=0.0.6,<1.0.0
|
||||
httpx>=0.25.0,<1.0.0
|
||||
google-cloud-firestore>=2.14.0,<3.0.0
|
||||
google-cloud-storage>=2.14.0,<3.0.0
|
||||
google-cloud-secret-manager>=2.18.0,<3.0.0
|
||||
google-cloud-tasks>=2.14.0,<3.0.0
|
||||
google-cloud-run>=0.10.0,<1.0.0
|
||||
python-dotenv>=1.0.0,<2.0.0
|
||||
|
||||
# Audio processing (reuses karaoke_gen library)
|
||||
audio-separator[remote]>=0.18.0 # Remote Modal API client
|
||||
pydub>=0.25.1 # Audio manipulation (used by karaoke_gen)
|
||||
yt-dlp>=2024.0.0 # YouTube downloads (future use)
|
||||
|
||||
# Cloud storage native APIs
|
||||
dropbox>=11.0.0 # Dropbox Python SDK for native API access
|
||||
# google-api-python-client already installed via karaoke_gen for YouTube
|
||||
|
||||
# karaoke-gen dependencies (will be installed from parent dir in Dockerfile)
|
||||
# Using explicit dependencies instead of -e ../
|
||||
|
||||
@@ -1,60 +0,0 @@
|
||||
#!/bin/bash
|
||||
# Backend Test Runner
|
||||
# Runs integration tests against the deployed Cloud Run service
|
||||
|
||||
set -e
|
||||
|
||||
echo "🧪 Backend Integration Test Suite"
|
||||
echo "=================================="
|
||||
echo ""
|
||||
|
||||
# Check prerequisites
|
||||
echo "Checking prerequisites..."
|
||||
if ! command -v gcloud &> /dev/null; then
|
||||
echo "❌ gcloud CLI not found. Please install Google Cloud SDK."
|
||||
exit 1
|
||||
fi
|
||||
|
||||
if ! command -v pytest &> /dev/null; then
|
||||
echo "❌ pytest not found. Installing test dependencies..."
|
||||
pip install -r tests/requirements.txt
|
||||
fi
|
||||
|
||||
# Verify authentication
|
||||
echo "Verifying authentication..."
|
||||
if ! gcloud auth print-identity-token &> /dev/null; then
|
||||
echo "❌ Not authenticated with gcloud. Run: gcloud auth login"
|
||||
exit 1
|
||||
fi
|
||||
|
||||
echo "✅ Prerequisites met"
|
||||
echo ""
|
||||
|
||||
# Run tests
|
||||
echo "Running integration tests..."
|
||||
echo ""
|
||||
|
||||
# Run fast tests (no slow marker)
|
||||
echo "📝 Running fast tests..."
|
||||
pytest tests/test_api_integration.py -v -m "not slow" --tb=short
|
||||
|
||||
echo ""
|
||||
echo "✅ Fast tests complete!"
|
||||
echo ""
|
||||
|
||||
# Ask if user wants to run slow tests
|
||||
read -p "Run slow/integration tests? These test actual job processing (5-10 min). [y/N] " -n 1 -r
|
||||
echo
|
||||
if [[ $REPLY =~ ^[Yy]$ ]]; then
|
||||
echo "🐌 Running slow integration tests..."
|
||||
pytest tests/test_api_integration.py -v -m "slow" --tb=short
|
||||
echo ""
|
||||
echo "✅ All tests complete!"
|
||||
else
|
||||
echo "⏭️ Skipping slow tests"
|
||||
fi
|
||||
|
||||
echo ""
|
||||
echo "=================================="
|
||||
echo "Test run complete! 🎉"
|
||||
|
||||
Whitespace-only changes.
@@ -1,243 +0,0 @@
|
||||
"""
|
||||
Audio analysis service for analyzing backing vocals.
|
||||
|
||||
This service wraps the shared karaoke_gen.instrumental_review module
|
||||
to provide GCS-integrated audio analysis capabilities.
|
||||
"""
|
||||
|
||||
import logging
|
||||
import os
|
||||
import tempfile
|
||||
from typing import Optional
|
||||
|
||||
from backend.services.storage_service import StorageService
|
||||
from karaoke_gen.instrumental_review import (
|
||||
AudioAnalyzer,
|
||||
AnalysisResult,
|
||||
WaveformGenerator,
|
||||
)
|
||||
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
|
||||
class AudioAnalysisService:
|
||||
"""
|
||||
Service for analyzing backing vocals audio files stored in GCS.
|
||||
|
||||
This service acts as a thin wrapper around the shared AudioAnalyzer
|
||||
and WaveformGenerator classes, handling GCS download/upload operations.
|
||||
|
||||
The actual analysis logic is in the shared karaoke_gen.instrumental_review
|
||||
module, ensuring feature parity between local and remote workflows.
|
||||
"""
|
||||
|
||||
def __init__(
|
||||
self,
|
||||
storage_service: Optional[StorageService] = None,
|
||||
silence_threshold_db: float = -40.0,
|
||||
min_segment_duration_ms: int = 100,
|
||||
):
|
||||
"""
|
||||
Initialize the audio analysis service.
|
||||
|
||||
Args:
|
||||
storage_service: GCS storage service. If not provided, a new
|
||||
instance will be created.
|
||||
silence_threshold_db: Threshold for considering audio as silent.
|
||||
Default is -40.0 dB.
|
||||
min_segment_duration_ms: Minimum duration for audible segments.
|
||||
Default is 100ms.
|
||||
"""
|
||||
self.storage_service = storage_service or StorageService()
|
||||
self.analyzer = AudioAnalyzer(
|
||||
silence_threshold_db=silence_threshold_db,
|
||||
min_segment_duration_ms=min_segment_duration_ms,
|
||||
)
|
||||
self.waveform_generator = WaveformGenerator()
|
||||
|
||||
def analyze_backing_vocals(
|
||||
self,
|
||||
gcs_audio_path: str,
|
||||
job_id: str,
|
||||
) -> AnalysisResult:
|
||||
"""
|
||||
Analyze a backing vocals audio file from GCS.
|
||||
|
||||
This method:
|
||||
1. Downloads the audio file from GCS to a temp file
|
||||
2. Runs the analysis using the shared AudioAnalyzer
|
||||
3. Returns the analysis result
|
||||
|
||||
Args:
|
||||
gcs_audio_path: Path to the audio file in GCS
|
||||
job_id: Job ID for logging
|
||||
|
||||
Returns:
|
||||
AnalysisResult containing analysis data
|
||||
"""
|
||||
logger.info(f"[{job_id}] Analyzing backing vocals: {gcs_audio_path}")
|
||||
|
||||
with tempfile.TemporaryDirectory() as temp_dir:
|
||||
# Download audio file
|
||||
local_audio_path = os.path.join(temp_dir, "backing_vocals.flac")
|
||||
self.storage_service.download_file(gcs_audio_path, local_audio_path)
|
||||
|
||||
# Run analysis
|
||||
result = self.analyzer.analyze(local_audio_path)
|
||||
|
||||
logger.info(
|
||||
f"[{job_id}] Analysis complete: "
|
||||
f"has_audible={result.has_audible_content}, "
|
||||
f"segments={result.segment_count}, "
|
||||
f"recommendation={result.recommended_selection.value}"
|
||||
)
|
||||
|
||||
return result
|
||||
|
||||
def analyze_and_generate_waveform(
|
||||
self,
|
||||
gcs_audio_path: str,
|
||||
job_id: str,
|
||||
gcs_waveform_destination: str,
|
||||
) -> tuple[AnalysisResult, str]:
|
||||
"""
|
||||
Analyze backing vocals and generate a waveform image.
|
||||
|
||||
This method:
|
||||
1. Downloads the audio file from GCS
|
||||
2. Runs analysis using AudioAnalyzer
|
||||
3. Generates waveform image using WaveformGenerator
|
||||
4. Uploads the waveform image to GCS
|
||||
5. Returns analysis result and waveform GCS path
|
||||
|
||||
Args:
|
||||
gcs_audio_path: Path to the audio file in GCS
|
||||
job_id: Job ID for logging
|
||||
gcs_waveform_destination: Where to upload the waveform image in GCS
|
||||
|
||||
Returns:
|
||||
Tuple of (AnalysisResult, waveform_gcs_path)
|
||||
"""
|
||||
logger.info(f"[{job_id}] Analyzing and generating waveform: {gcs_audio_path}")
|
||||
|
||||
with tempfile.TemporaryDirectory() as temp_dir:
|
||||
# Download audio file
|
||||
local_audio_path = os.path.join(temp_dir, "backing_vocals.flac")
|
||||
self.storage_service.download_file(gcs_audio_path, local_audio_path)
|
||||
|
||||
# Run analysis
|
||||
result = self.analyzer.analyze(local_audio_path)
|
||||
|
||||
# Generate waveform
|
||||
local_waveform_path = os.path.join(temp_dir, "waveform.png")
|
||||
self.waveform_generator.generate(
|
||||
audio_path=local_audio_path,
|
||||
output_path=local_waveform_path,
|
||||
segments=result.audible_segments,
|
||||
show_time_axis=True,
|
||||
silence_threshold_db=self.analyzer.silence_threshold_db,
|
||||
)
|
||||
|
||||
# Upload waveform to GCS
|
||||
self.storage_service.upload_file(
|
||||
local_waveform_path,
|
||||
gcs_waveform_destination
|
||||
)
|
||||
|
||||
logger.info(
|
||||
f"[{job_id}] Analysis and waveform generation complete. "
|
||||
f"Waveform uploaded to: {gcs_waveform_destination}"
|
||||
)
|
||||
|
||||
return result, gcs_waveform_destination
|
||||
|
||||
def get_waveform_data(
|
||||
self,
|
||||
gcs_audio_path: str,
|
||||
job_id: str,
|
||||
num_points: int = 500,
|
||||
) -> tuple[list[float], float]:
|
||||
"""
|
||||
Get waveform data (amplitude envelope) for client-side rendering.
|
||||
|
||||
This is useful when the frontend wants to render the waveform
|
||||
itself using Canvas or SVG, rather than displaying a pre-generated
|
||||
image.
|
||||
|
||||
Args:
|
||||
gcs_audio_path: Path to the audio file in GCS
|
||||
job_id: Job ID for logging
|
||||
num_points: Number of data points to return
|
||||
|
||||
Returns:
|
||||
Tuple of (amplitude_values, duration_seconds)
|
||||
"""
|
||||
logger.info(f"[{job_id}] Getting waveform data: {gcs_audio_path}")
|
||||
|
||||
with tempfile.TemporaryDirectory() as temp_dir:
|
||||
# Download audio file
|
||||
local_audio_path = os.path.join(temp_dir, "backing_vocals.flac")
|
||||
self.storage_service.download_file(gcs_audio_path, local_audio_path)
|
||||
|
||||
# Generate waveform data
|
||||
amplitudes, duration = self.waveform_generator.generate_data_only(
|
||||
audio_path=local_audio_path,
|
||||
num_points=num_points,
|
||||
)
|
||||
|
||||
return amplitudes, duration
|
||||
|
||||
def generate_waveform_with_mutes(
|
||||
self,
|
||||
gcs_audio_path: str,
|
||||
job_id: str,
|
||||
gcs_waveform_destination: str,
|
||||
mute_regions: list,
|
||||
) -> str:
|
||||
"""
|
||||
Generate a waveform image with mute regions highlighted.
|
||||
|
||||
This is useful for showing the user which regions will be muted
|
||||
in the custom instrumental.
|
||||
|
||||
Args:
|
||||
gcs_audio_path: Path to the audio file in GCS
|
||||
job_id: Job ID for logging
|
||||
gcs_waveform_destination: Where to upload the waveform image
|
||||
mute_regions: List of MuteRegion objects to highlight
|
||||
|
||||
Returns:
|
||||
GCS path to the uploaded waveform image
|
||||
"""
|
||||
from karaoke_gen.instrumental_review import MuteRegion
|
||||
|
||||
logger.info(
|
||||
f"[{job_id}] Generating waveform with {len(mute_regions)} mute regions"
|
||||
)
|
||||
|
||||
with tempfile.TemporaryDirectory() as temp_dir:
|
||||
# Download audio file
|
||||
local_audio_path = os.path.join(temp_dir, "backing_vocals.flac")
|
||||
self.storage_service.download_file(gcs_audio_path, local_audio_path)
|
||||
|
||||
# First run analysis to get segments
|
||||
result = self.analyzer.analyze(local_audio_path)
|
||||
|
||||
# Generate waveform with mute regions
|
||||
local_waveform_path = os.path.join(temp_dir, "waveform_with_mutes.png")
|
||||
self.waveform_generator.generate(
|
||||
audio_path=local_audio_path,
|
||||
output_path=local_waveform_path,
|
||||
segments=result.audible_segments,
|
||||
mute_regions=mute_regions,
|
||||
show_time_axis=True,
|
||||
)
|
||||
|
||||
# Upload to GCS
|
||||
self.storage_service.upload_file(
|
||||
local_waveform_path,
|
||||
gcs_waveform_destination
|
||||
)
|
||||
|
||||
return gcs_waveform_destination
|
||||
@@ -1,278 +0,0 @@
|
||||
"""
|
||||
Audio editing service for creating custom instrumentals.
|
||||
|
||||
This service wraps the shared karaoke_gen.instrumental_review module
|
||||
to provide GCS-integrated audio editing capabilities.
|
||||
"""
|
||||
|
||||
import logging
|
||||
import os
|
||||
import tempfile
|
||||
from typing import List, Optional
|
||||
|
||||
from backend.services.storage_service import StorageService
|
||||
from karaoke_gen.instrumental_review import (
|
||||
AudioEditor,
|
||||
MuteRegion,
|
||||
)
|
||||
from karaoke_gen.instrumental_review.models import CustomInstrumentalResult
|
||||
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
|
||||
class AudioEditingService:
|
||||
"""
|
||||
Service for creating custom instrumental tracks with GCS integration.
|
||||
|
||||
This service acts as a thin wrapper around the shared AudioEditor
|
||||
class, handling GCS download/upload operations.
|
||||
|
||||
The actual editing logic is in the shared karaoke_gen.instrumental_review
|
||||
module, ensuring feature parity between local and remote workflows.
|
||||
"""
|
||||
|
||||
def __init__(
|
||||
self,
|
||||
storage_service: Optional[StorageService] = None,
|
||||
output_format: str = "flac",
|
||||
):
|
||||
"""
|
||||
Initialize the audio editing service.
|
||||
|
||||
Args:
|
||||
storage_service: GCS storage service. If not provided, a new
|
||||
instance will be created.
|
||||
output_format: Output audio format. Default is "flac".
|
||||
"""
|
||||
self.storage_service = storage_service or StorageService()
|
||||
self.editor = AudioEditor(output_format=output_format)
|
||||
|
||||
def create_custom_instrumental(
|
||||
self,
|
||||
gcs_clean_instrumental_path: str,
|
||||
gcs_backing_vocals_path: str,
|
||||
mute_regions: List[MuteRegion],
|
||||
gcs_output_path: str,
|
||||
job_id: str,
|
||||
) -> CustomInstrumentalResult:
|
||||
"""
|
||||
Create a custom instrumental by muting regions of backing vocals.
|
||||
|
||||
This method:
|
||||
1. Downloads clean instrumental and backing vocals from GCS
|
||||
2. Applies mute regions to backing vocals
|
||||
3. Combines with clean instrumental
|
||||
4. Uploads result to GCS
|
||||
|
||||
Args:
|
||||
gcs_clean_instrumental_path: Path to clean instrumental in GCS
|
||||
gcs_backing_vocals_path: Path to backing vocals in GCS
|
||||
mute_regions: List of regions to mute in backing vocals
|
||||
gcs_output_path: Where to upload the custom instrumental in GCS
|
||||
job_id: Job ID for logging
|
||||
|
||||
Returns:
|
||||
CustomInstrumentalResult with details about the created file
|
||||
"""
|
||||
logger.info(
|
||||
f"[{job_id}] Creating custom instrumental with "
|
||||
f"{len(mute_regions)} mute regions"
|
||||
)
|
||||
|
||||
with tempfile.TemporaryDirectory() as temp_dir:
|
||||
# Download input files
|
||||
local_clean_path = os.path.join(temp_dir, "clean_instrumental.flac")
|
||||
local_backing_path = os.path.join(temp_dir, "backing_vocals.flac")
|
||||
local_output_path = os.path.join(temp_dir, "custom_instrumental.flac")
|
||||
|
||||
logger.debug(f"[{job_id}] Downloading clean instrumental")
|
||||
self.storage_service.download_file(
|
||||
gcs_clean_instrumental_path,
|
||||
local_clean_path
|
||||
)
|
||||
|
||||
logger.debug(f"[{job_id}] Downloading backing vocals")
|
||||
self.storage_service.download_file(
|
||||
gcs_backing_vocals_path,
|
||||
local_backing_path
|
||||
)
|
||||
|
||||
# Create custom instrumental using shared editor
|
||||
result = self.editor.create_custom_instrumental(
|
||||
clean_instrumental_path=local_clean_path,
|
||||
backing_vocals_path=local_backing_path,
|
||||
mute_regions=mute_regions,
|
||||
output_path=local_output_path,
|
||||
)
|
||||
|
||||
# Upload result to GCS
|
||||
logger.debug(f"[{job_id}] Uploading custom instrumental to GCS")
|
||||
self.storage_service.upload_file(local_output_path, gcs_output_path)
|
||||
|
||||
# Update result with GCS path
|
||||
result.output_path = gcs_output_path
|
||||
|
||||
logger.info(
|
||||
f"[{job_id}] Custom instrumental created: {gcs_output_path}, "
|
||||
f"muted {result.total_muted_duration_seconds:.1f}s across "
|
||||
f"{len(result.mute_regions_applied)} regions"
|
||||
)
|
||||
|
||||
return result
|
||||
|
||||
def create_preview(
|
||||
self,
|
||||
gcs_clean_instrumental_path: str,
|
||||
gcs_backing_vocals_path: str,
|
||||
mute_regions: List[MuteRegion],
|
||||
gcs_preview_path: str,
|
||||
job_id: str,
|
||||
preview_duration_seconds: Optional[float] = None,
|
||||
) -> str:
|
||||
"""
|
||||
Create a preview of the custom instrumental.
|
||||
|
||||
This is similar to create_custom_instrumental but optimized for
|
||||
quick preview generation (optionally truncated duration).
|
||||
|
||||
Args:
|
||||
gcs_clean_instrumental_path: Path to clean instrumental in GCS
|
||||
gcs_backing_vocals_path: Path to backing vocals in GCS
|
||||
mute_regions: List of regions to mute
|
||||
gcs_preview_path: Where to upload the preview in GCS
|
||||
job_id: Job ID for logging
|
||||
preview_duration_seconds: Optional max duration for preview
|
||||
|
||||
Returns:
|
||||
GCS path to the uploaded preview
|
||||
"""
|
||||
logger.info(f"[{job_id}] Creating preview with {len(mute_regions)} mute regions")
|
||||
|
||||
with tempfile.TemporaryDirectory() as temp_dir:
|
||||
# Download input files
|
||||
local_clean_path = os.path.join(temp_dir, "clean_instrumental.flac")
|
||||
local_backing_path = os.path.join(temp_dir, "backing_vocals.flac")
|
||||
local_preview_path = os.path.join(temp_dir, "preview.flac")
|
||||
|
||||
self.storage_service.download_file(
|
||||
gcs_clean_instrumental_path,
|
||||
local_clean_path
|
||||
)
|
||||
self.storage_service.download_file(
|
||||
gcs_backing_vocals_path,
|
||||
local_backing_path
|
||||
)
|
||||
|
||||
# Generate preview
|
||||
from pydub import AudioSegment
|
||||
|
||||
preview = self.editor.preview_with_mutes(
|
||||
clean_instrumental_path=local_clean_path,
|
||||
backing_vocals_path=local_backing_path,
|
||||
mute_regions=mute_regions,
|
||||
)
|
||||
|
||||
# Optionally truncate
|
||||
if preview_duration_seconds:
|
||||
max_ms = int(preview_duration_seconds * 1000)
|
||||
preview = preview[:max_ms]
|
||||
|
||||
# Export and upload
|
||||
preview.export(local_preview_path, format="flac")
|
||||
self.storage_service.upload_file(local_preview_path, gcs_preview_path)
|
||||
|
||||
logger.info(f"[{job_id}] Preview created: {gcs_preview_path}")
|
||||
|
||||
return gcs_preview_path
|
||||
|
||||
def mute_backing_vocals_only(
|
||||
self,
|
||||
gcs_backing_vocals_path: str,
|
||||
mute_regions: List[MuteRegion],
|
||||
gcs_output_path: str,
|
||||
job_id: str,
|
||||
) -> str:
|
||||
"""
|
||||
Apply mute regions to backing vocals without combining with instrumental.
|
||||
|
||||
This is useful for creating an edited backing vocals track that
|
||||
can be combined with the instrumental later.
|
||||
|
||||
Args:
|
||||
gcs_backing_vocals_path: Path to backing vocals in GCS
|
||||
mute_regions: List of regions to mute
|
||||
gcs_output_path: Where to upload the edited backing vocals
|
||||
job_id: Job ID for logging
|
||||
|
||||
Returns:
|
||||
GCS path to the uploaded edited backing vocals
|
||||
"""
|
||||
logger.info(
|
||||
f"[{job_id}] Muting backing vocals with {len(mute_regions)} regions"
|
||||
)
|
||||
|
||||
with tempfile.TemporaryDirectory() as temp_dir:
|
||||
local_backing_path = os.path.join(temp_dir, "backing_vocals.flac")
|
||||
local_output_path = os.path.join(temp_dir, "muted_backing.flac")
|
||||
|
||||
self.storage_service.download_file(
|
||||
gcs_backing_vocals_path,
|
||||
local_backing_path
|
||||
)
|
||||
|
||||
self.editor.apply_mute_to_single_track(
|
||||
audio_path=local_backing_path,
|
||||
mute_regions=mute_regions,
|
||||
output_path=local_output_path,
|
||||
)
|
||||
|
||||
self.storage_service.upload_file(local_output_path, gcs_output_path)
|
||||
|
||||
logger.info(f"[{job_id}] Muted backing vocals: {gcs_output_path}")
|
||||
|
||||
return gcs_output_path
|
||||
|
||||
def validate_mute_regions(
|
||||
self,
|
||||
mute_regions: List[MuteRegion],
|
||||
total_duration_seconds: float,
|
||||
) -> List[str]:
|
||||
"""
|
||||
Validate mute regions for consistency.
|
||||
|
||||
Args:
|
||||
mute_regions: List of mute regions to validate
|
||||
total_duration_seconds: Total duration of the audio
|
||||
|
||||
Returns:
|
||||
List of validation error messages (empty if valid)
|
||||
"""
|
||||
errors = []
|
||||
|
||||
for i, region in enumerate(mute_regions):
|
||||
if region.start_seconds < 0:
|
||||
errors.append(
|
||||
f"Region {i}: start_seconds ({region.start_seconds}) cannot be negative"
|
||||
)
|
||||
|
||||
if region.end_seconds <= region.start_seconds:
|
||||
errors.append(
|
||||
f"Region {i}: end_seconds ({region.end_seconds}) must be after "
|
||||
f"start_seconds ({region.start_seconds})"
|
||||
)
|
||||
|
||||
if region.start_seconds > total_duration_seconds:
|
||||
errors.append(
|
||||
f"Region {i}: start_seconds ({region.start_seconds}) exceeds "
|
||||
f"audio duration ({total_duration_seconds})"
|
||||
)
|
||||
|
||||
if region.end_seconds > total_duration_seconds:
|
||||
# Not an error, but log a warning - the region will be clamped
|
||||
logger.warning(
|
||||
f"Region {i}: end_seconds ({region.end_seconds}) exceeds "
|
||||
f"audio duration ({total_duration_seconds}), will be clamped"
|
||||
)
|
||||
|
||||
return errors
|
||||
@@ -1,681 +0,0 @@
|
||||
"""
|
||||
Audio search service for finding and downloading audio files.
|
||||
|
||||
This service integrates with karaoke_gen.audio_fetcher (which wraps flacfetch)
|
||||
to search for audio from various sources (YouTube, music trackers, etc.)
|
||||
and download the selected audio file.
|
||||
|
||||
This is a thin wrapper that adds backend-specific functionality:
|
||||
- Caching raw results for API-based selection flow
|
||||
- Firestore-compatible serialization
|
||||
- Singleton pattern for service lifecycle
|
||||
- Remote flacfetch service integration for torrent downloads
|
||||
|
||||
When FLACFETCH_API_URL is configured, this service uses the remote flacfetch
|
||||
HTTP API for search and download operations. This is required for torrent
|
||||
downloads since Cloud Run doesn't support BitTorrent peer connections.
|
||||
"""
|
||||
import asyncio
|
||||
import logging
|
||||
import os
|
||||
from typing import List, Optional
|
||||
|
||||
import nest_asyncio
|
||||
|
||||
# Import shared classes from karaoke_gen.audio_fetcher - single source of truth
|
||||
# Note: Import directly from audio_fetcher module, not karaoke_gen.__init__
|
||||
# to avoid pulling in heavier dependencies like lyrics_transcriber
|
||||
from karaoke_gen.audio_fetcher import (
|
||||
FlacFetcher,
|
||||
AudioSearchResult,
|
||||
AudioFetchResult,
|
||||
AudioFetcherError,
|
||||
NoResultsError,
|
||||
DownloadError,
|
||||
)
|
||||
|
||||
from .flacfetch_client import get_flacfetch_client, FlacfetchServiceError
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
|
||||
# Re-export exception under backend's naming convention for compatibility
|
||||
AudioSearchError = AudioFetcherError
|
||||
|
||||
|
||||
# Also alias AudioFetchResult as AudioDownloadResult for backwards compatibility
|
||||
AudioDownloadResult = AudioFetchResult
|
||||
|
||||
|
||||
class AudioSearchService:
|
||||
"""
|
||||
Service for searching and downloading audio files.
|
||||
|
||||
This is a thin wrapper around karaoke_gen.audio_fetcher.FlacFetcher that adds
|
||||
backend-specific functionality:
|
||||
- Caches raw results for API-based selection (client picks by index)
|
||||
- Provides Firestore-compatible serialization
|
||||
- Manages service lifecycle as singleton
|
||||
- Remote flacfetch service integration for torrent downloads
|
||||
|
||||
When FLACFETCH_API_URL is configured, this service uses the remote flacfetch
|
||||
HTTP API for search and download operations. This enables torrent downloads
|
||||
which are not possible in Cloud Run due to network restrictions.
|
||||
|
||||
The actual flacfetch integration is in karaoke_gen.audio_fetcher.FlacFetcher,
|
||||
which is shared between local CLI and cloud backend.
|
||||
"""
|
||||
|
||||
# Sentinel value to indicate "use environment variable"
|
||||
_USE_ENV = object()
|
||||
|
||||
def __init__(
|
||||
self,
|
||||
red_api_key: Optional[str] = _USE_ENV,
|
||||
red_api_url: Optional[str] = _USE_ENV,
|
||||
ops_api_key: Optional[str] = _USE_ENV,
|
||||
ops_api_url: Optional[str] = _USE_ENV,
|
||||
):
|
||||
"""
|
||||
Initialize the audio search service.
|
||||
|
||||
Args:
|
||||
red_api_key: API key for RED tracker (optional, uses env if not provided)
|
||||
red_api_url: Base URL for RED tracker API (optional, uses env if not provided)
|
||||
ops_api_key: API key for OPS tracker (optional, uses env if not provided)
|
||||
ops_api_url: Base URL for OPS tracker API (optional, uses env if not provided)
|
||||
"""
|
||||
# Use environment variables if not explicitly provided
|
||||
if red_api_key is self._USE_ENV:
|
||||
red_api_key = os.environ.get("RED_API_KEY")
|
||||
if red_api_url is self._USE_ENV:
|
||||
red_api_url = os.environ.get("RED_API_URL")
|
||||
|
||||
if ops_api_key is self._USE_ENV:
|
||||
ops_api_key = os.environ.get("OPS_API_KEY")
|
||||
if ops_api_url is self._USE_ENV:
|
||||
ops_api_url = os.environ.get("OPS_API_URL")
|
||||
|
||||
# Check for remote flacfetch client
|
||||
self._remote_client = get_flacfetch_client()
|
||||
|
||||
# Log which mode we're using
|
||||
if self._remote_client:
|
||||
logger.info("AudioSearchService using REMOTE flacfetch service (torrent downloads enabled)")
|
||||
else:
|
||||
logger.info("AudioSearchService using LOCAL flacfetch (YouTube only in Cloud Run)")
|
||||
|
||||
# Delegate to shared FlacFetcher implementation (for local mode or fallback)
|
||||
self._fetcher = FlacFetcher(
|
||||
red_api_key=red_api_key,
|
||||
red_api_url=red_api_url,
|
||||
ops_api_key=ops_api_key,
|
||||
ops_api_url=ops_api_url,
|
||||
)
|
||||
|
||||
# Cache search results for API-based selection flow
|
||||
# Key: index, Value: AudioSearchResult (with raw_result)
|
||||
self._cached_results: List[AudioSearchResult] = []
|
||||
|
||||
# Cache for remote search results (search_id -> results)
|
||||
self._remote_search_id: Optional[str] = None
|
||||
|
||||
def search(self, artist: str, title: str) -> List[AudioSearchResult]:
|
||||
"""
|
||||
Search for audio matching the given artist and title.
|
||||
|
||||
Results are cached internally for later download via download().
|
||||
|
||||
If a remote flacfetch service is configured (FLACFETCH_API_URL), uses that
|
||||
for better torrent download support. Otherwise falls back to local flacfetch.
|
||||
|
||||
Args:
|
||||
artist: The artist name to search for
|
||||
title: The track title to search for
|
||||
|
||||
Returns:
|
||||
List of AudioSearchResult objects
|
||||
|
||||
Raises:
|
||||
NoResultsError: If no results are found
|
||||
AudioSearchError: For other errors (e.g., flacfetch not installed)
|
||||
"""
|
||||
try:
|
||||
# Try remote flacfetch service first if configured
|
||||
if self._remote_client:
|
||||
return self._search_remote(artist, title)
|
||||
|
||||
# Fallback to local flacfetch
|
||||
results = self._fetcher.search(artist, title)
|
||||
|
||||
# Cache results for later download
|
||||
self._cached_results = results
|
||||
self._remote_search_id = None
|
||||
|
||||
logger.info(f"Found {len(results)} results for: {artist} - {title}")
|
||||
return results
|
||||
|
||||
except (NoResultsError, AudioFetcherError):
|
||||
# Re-raise these as-is
|
||||
raise
|
||||
except Exception as e:
|
||||
logger.error(f"Search failed: {e}")
|
||||
raise AudioSearchError(f"Search failed: {e}") from e
|
||||
|
||||
def _search_remote(self, artist: str, title: str) -> List[AudioSearchResult]:
|
||||
"""
|
||||
Search using the remote flacfetch service.
|
||||
|
||||
Runs async code in a sync context for compatibility with existing API.
|
||||
"""
|
||||
try:
|
||||
# Enable nested event loops (needed when called from FastAPI async context)
|
||||
# Apply to the current loop if one exists, otherwise it's a no-op
|
||||
try:
|
||||
loop = asyncio.get_running_loop()
|
||||
nest_asyncio.apply(loop)
|
||||
except RuntimeError:
|
||||
pass # No running loop, nest_asyncio not needed
|
||||
|
||||
# Run async search in sync context
|
||||
loop = asyncio.new_event_loop()
|
||||
asyncio.set_event_loop(loop)
|
||||
try:
|
||||
response = loop.run_until_complete(
|
||||
self._remote_client.search(artist, title)
|
||||
)
|
||||
finally:
|
||||
loop.close()
|
||||
|
||||
# Check for empty results
|
||||
if not response.get("results"):
|
||||
raise NoResultsError(f"No results found for: {artist} - {title}")
|
||||
|
||||
# Store search_id for download
|
||||
self._remote_search_id = response.get("search_id")
|
||||
|
||||
# Convert remote results to AudioSearchResult objects
|
||||
results = []
|
||||
for item in response.get("results", []):
|
||||
result = AudioSearchResult(
|
||||
title=item.get("title", "Unknown"),
|
||||
artist=item.get("artist", "Unknown"),
|
||||
url=item.get("download_url", ""), # May be empty for remote
|
||||
provider=item.get("provider", "Unknown"),
|
||||
duration=item.get("duration_seconds"),
|
||||
quality=item.get("quality", ""),
|
||||
source_id=item.get("info_hash"),
|
||||
index=item.get("index", 0),
|
||||
seeders=item.get("seeders"),
|
||||
target_file=item.get("target_file"),
|
||||
# Store full remote data in raw_result for rich display
|
||||
raw_result=item,
|
||||
)
|
||||
results.append(result)
|
||||
|
||||
# Cache results
|
||||
self._cached_results = results
|
||||
|
||||
logger.info(f"Found {len(results)} results from remote flacfetch for: {artist} - {title}")
|
||||
return results
|
||||
|
||||
except FlacfetchServiceError as e:
|
||||
logger.warning(f"Remote flacfetch search failed, falling back to local: {e}")
|
||||
# Fallback to local search
|
||||
self._remote_search_id = None
|
||||
results = self._fetcher.search(artist, title)
|
||||
self._cached_results = results
|
||||
return results
|
||||
|
||||
def select_best(self, results: List[AudioSearchResult]) -> int:
|
||||
"""
|
||||
Select the best result from a list of search results.
|
||||
|
||||
Uses flacfetch's built-in quality ranking to select the best source.
|
||||
|
||||
Args:
|
||||
results: List of search results
|
||||
|
||||
Returns:
|
||||
Index of the best result
|
||||
"""
|
||||
return self._fetcher.select_best(results)
|
||||
|
||||
def download(
|
||||
self,
|
||||
result_index: int,
|
||||
output_dir: str,
|
||||
output_filename: Optional[str] = None,
|
||||
gcs_path: Optional[str] = None,
|
||||
remote_search_id: Optional[str] = None,
|
||||
) -> AudioDownloadResult:
|
||||
"""
|
||||
Download audio from a cached search result.
|
||||
|
||||
This method uses the cached results from the last search() call.
|
||||
The API flow is:
|
||||
1. Client calls search() -> gets list of results
|
||||
2. Client picks an index
|
||||
3. Client calls download(index) -> gets downloaded file
|
||||
|
||||
If a remote flacfetch service is configured and the search was performed
|
||||
remotely (for torrent sources), uses the remote service for download.
|
||||
|
||||
Args:
|
||||
result_index: Index of the result to download (from search results)
|
||||
output_dir: Directory to save the downloaded file
|
||||
output_filename: Optional filename (without extension)
|
||||
gcs_path: Optional GCS path for remote uploads (e.g., "uploads/job123/audio/")
|
||||
remote_search_id: Optional search_id for remote downloads. If provided,
|
||||
uses this instead of the cached _remote_search_id. This is important
|
||||
for concurrent requests where the service-level cache may be stale.
|
||||
|
||||
Returns:
|
||||
AudioDownloadResult with the downloaded file path
|
||||
|
||||
Raises:
|
||||
DownloadError: If download fails or no cached result for index
|
||||
"""
|
||||
if result_index < 0 or result_index >= len(self._cached_results):
|
||||
raise DownloadError(
|
||||
f"No cached result for index {result_index}. "
|
||||
f"Available indices: 0-{len(self._cached_results) - 1}. "
|
||||
"Run search() first."
|
||||
)
|
||||
|
||||
result = self._cached_results[result_index]
|
||||
|
||||
logger.info(f"Downloading: {result.artist} - {result.title} from {result.provider}")
|
||||
|
||||
# Use provided search_id or fall back to service-level cache
|
||||
effective_search_id = remote_search_id or self._remote_search_id
|
||||
|
||||
# Check if this was a remote search (torrent sources need remote download)
|
||||
if effective_search_id and self._remote_client:
|
||||
# Check if this is a torrent source that needs remote handling
|
||||
is_torrent = result.provider in ["RED", "OPS"]
|
||||
|
||||
if is_torrent:
|
||||
return self._download_remote(result_index, output_dir, output_filename, gcs_path, effective_search_id)
|
||||
|
||||
# Delegate to shared FlacFetcher (local download)
|
||||
fetch_result = self._fetcher.download(result, output_dir, output_filename)
|
||||
|
||||
logger.info(f"Downloaded to: {fetch_result.filepath}")
|
||||
|
||||
return fetch_result
|
||||
|
||||
def download_by_id(
|
||||
self,
|
||||
source_name: str,
|
||||
source_id: str,
|
||||
output_dir: str,
|
||||
output_filename: Optional[str] = None,
|
||||
target_file: Optional[str] = None,
|
||||
download_url: Optional[str] = None,
|
||||
gcs_path: Optional[str] = None,
|
||||
) -> AudioDownloadResult:
|
||||
"""
|
||||
Download audio directly by source ID (no prior search required).
|
||||
|
||||
This is the preferred method when you have stored the source_id from a
|
||||
previous search and want to download later without re-searching. This
|
||||
avoids unnecessary API calls to private trackers.
|
||||
|
||||
Args:
|
||||
source_name: Provider name (RED, OPS, YouTube, Spotify)
|
||||
source_id: Source-specific ID (torrent ID, video ID, track ID)
|
||||
output_dir: Directory to save the downloaded file
|
||||
output_filename: Optional filename (without extension)
|
||||
target_file: For torrents, specific file to extract from the torrent
|
||||
download_url: For YouTube/Spotify, direct URL (optional)
|
||||
gcs_path: Optional GCS path for remote uploads
|
||||
|
||||
Returns:
|
||||
AudioDownloadResult with the downloaded file path
|
||||
|
||||
Raises:
|
||||
DownloadError: If download fails
|
||||
"""
|
||||
logger.info(f"Download by ID: {source_name} ID={source_id}")
|
||||
|
||||
# For torrent sources, must use remote client
|
||||
if source_name in ["RED", "OPS"]:
|
||||
if not self._remote_client:
|
||||
raise DownloadError(
|
||||
f"Cannot download from {source_name} without remote flacfetch service"
|
||||
)
|
||||
return self._download_by_id_remote(
|
||||
source_name=source_name,
|
||||
source_id=source_id,
|
||||
output_dir=output_dir,
|
||||
output_filename=output_filename,
|
||||
target_file=target_file,
|
||||
download_url=download_url,
|
||||
gcs_path=gcs_path,
|
||||
)
|
||||
|
||||
# For YouTube/Spotify, can use local flacfetch via FetchManager.download_by_id
|
||||
# But we currently only have remote support, so use remote if available
|
||||
if self._remote_client:
|
||||
return self._download_by_id_remote(
|
||||
source_name=source_name,
|
||||
source_id=source_id,
|
||||
output_dir=output_dir,
|
||||
output_filename=output_filename,
|
||||
target_file=target_file,
|
||||
download_url=download_url,
|
||||
gcs_path=gcs_path,
|
||||
)
|
||||
|
||||
# Local download for YouTube/Spotify (fallback)
|
||||
# Use the local fetcher's download_by_id if available
|
||||
raise DownloadError(
|
||||
f"Local download_by_id not yet implemented for {source_name}. "
|
||||
"Configure FLACFETCH_API_URL for remote downloads."
|
||||
)
|
||||
|
||||
def _download_by_id_remote(
|
||||
self,
|
||||
source_name: str,
|
||||
source_id: str,
|
||||
output_dir: str, # Not used for remote downloads (files go to GCS or remote server)
|
||||
output_filename: Optional[str] = None,
|
||||
target_file: Optional[str] = None,
|
||||
download_url: Optional[str] = None,
|
||||
gcs_path: Optional[str] = None,
|
||||
) -> AudioDownloadResult:
|
||||
"""
|
||||
Download by ID using the remote flacfetch service.
|
||||
|
||||
Note: output_dir is accepted for API compatibility but ignored for remote
|
||||
downloads. Remote downloads either go to GCS (if gcs_path is set) or to
|
||||
the remote server's download directory.
|
||||
"""
|
||||
# output_dir is intentionally unused - remote downloads don't use local paths
|
||||
_ = output_dir
|
||||
try:
|
||||
# Enable nested event loops
|
||||
try:
|
||||
running_loop = asyncio.get_running_loop()
|
||||
nest_asyncio.apply(running_loop)
|
||||
except RuntimeError:
|
||||
pass
|
||||
|
||||
loop = asyncio.new_event_loop()
|
||||
asyncio.set_event_loop(loop)
|
||||
try:
|
||||
# Start download
|
||||
download_id = loop.run_until_complete(
|
||||
self._remote_client.download_by_id(
|
||||
source_name=source_name,
|
||||
source_id=source_id,
|
||||
output_filename=output_filename,
|
||||
target_file=target_file,
|
||||
download_url=download_url,
|
||||
gcs_path=gcs_path,
|
||||
)
|
||||
)
|
||||
|
||||
logger.info(f"Remote download by ID started: {download_id}")
|
||||
|
||||
# Wait for completion
|
||||
def log_progress(status):
|
||||
progress = status.get("progress", 0)
|
||||
speed = status.get("download_speed_kbps", 0)
|
||||
logger.debug(f"Download progress: {progress:.1f}% ({speed:.1f} KB/s)")
|
||||
|
||||
final_status = loop.run_until_complete(
|
||||
self._remote_client.wait_for_download(
|
||||
download_id,
|
||||
timeout=600,
|
||||
progress_callback=log_progress,
|
||||
)
|
||||
)
|
||||
finally:
|
||||
loop.close()
|
||||
|
||||
# Determine file path
|
||||
filepath = final_status.get("gcs_path") or final_status.get("output_path")
|
||||
|
||||
if not filepath:
|
||||
raise DownloadError("Remote download completed but no file path returned")
|
||||
|
||||
logger.info(f"Remote download by ID complete: {filepath}")
|
||||
|
||||
return AudioDownloadResult(
|
||||
filepath=filepath,
|
||||
artist="", # Not available without search
|
||||
title="", # Not available without search
|
||||
provider=source_name,
|
||||
quality="", # Not available without search
|
||||
)
|
||||
|
||||
except FlacfetchServiceError as e:
|
||||
raise DownloadError(f"Remote download by ID failed: {e}") from e
|
||||
except Exception as e:
|
||||
logger.error(f"Remote download by ID error: {e}")
|
||||
raise DownloadError(f"Remote download by ID failed: {e}") from e
|
||||
|
||||
def _download_remote(
|
||||
self,
|
||||
result_index: int,
|
||||
output_dir: str,
|
||||
output_filename: Optional[str] = None,
|
||||
gcs_path: Optional[str] = None,
|
||||
search_id: Optional[str] = None,
|
||||
) -> AudioDownloadResult:
|
||||
"""
|
||||
Download using the remote flacfetch service.
|
||||
|
||||
The remote service downloads via torrent and optionally uploads to GCS.
|
||||
|
||||
Args:
|
||||
result_index: Index of the result to download
|
||||
output_dir: Directory to save downloaded file
|
||||
output_filename: Optional filename
|
||||
gcs_path: Optional GCS path for remote uploads
|
||||
search_id: Remote search ID to use (overrides self._remote_search_id)
|
||||
"""
|
||||
effective_search_id = search_id or self._remote_search_id
|
||||
if not effective_search_id:
|
||||
raise DownloadError("No remote search ID - run search() first")
|
||||
|
||||
result = self._cached_results[result_index]
|
||||
|
||||
try:
|
||||
# Enable nested event loops (needed when called from FastAPI async context)
|
||||
try:
|
||||
running_loop = asyncio.get_running_loop()
|
||||
nest_asyncio.apply(running_loop)
|
||||
except RuntimeError:
|
||||
pass # No running loop, nest_asyncio not needed
|
||||
|
||||
loop = asyncio.new_event_loop()
|
||||
asyncio.set_event_loop(loop)
|
||||
try:
|
||||
# Start download
|
||||
download_id = loop.run_until_complete(
|
||||
self._remote_client.download(
|
||||
search_id=effective_search_id,
|
||||
result_index=result_index,
|
||||
output_filename=output_filename,
|
||||
gcs_path=gcs_path,
|
||||
)
|
||||
)
|
||||
|
||||
logger.info(f"Remote download started: {download_id}")
|
||||
|
||||
# Wait for completion with progress logging
|
||||
def log_progress(status):
|
||||
progress = status.get("progress", 0)
|
||||
speed = status.get("download_speed_kbps", 0)
|
||||
logger.debug(f"Download progress: {progress:.1f}% ({speed:.1f} KB/s)")
|
||||
|
||||
final_status = loop.run_until_complete(
|
||||
self._remote_client.wait_for_download(
|
||||
download_id,
|
||||
timeout=600, # 10 minute timeout
|
||||
progress_callback=log_progress,
|
||||
)
|
||||
)
|
||||
finally:
|
||||
loop.close()
|
||||
|
||||
# Determine file path
|
||||
filepath = final_status.get("gcs_path") or final_status.get("output_path")
|
||||
|
||||
if not filepath:
|
||||
raise DownloadError("Remote download completed but no file path returned")
|
||||
|
||||
logger.info(f"Remote download complete: {filepath}")
|
||||
|
||||
return AudioDownloadResult(
|
||||
filepath=filepath,
|
||||
artist=result.artist,
|
||||
title=result.title,
|
||||
provider=result.provider,
|
||||
quality=result.quality,
|
||||
)
|
||||
|
||||
except FlacfetchServiceError as e:
|
||||
raise DownloadError(f"Remote download failed: {e}") from e
|
||||
except Exception as e:
|
||||
logger.error(f"Remote download error: {e}")
|
||||
raise DownloadError(f"Remote download failed: {e}") from e
|
||||
|
||||
def search_and_download_auto(
|
||||
self,
|
||||
artist: str,
|
||||
title: str,
|
||||
output_dir: str,
|
||||
output_filename: Optional[str] = None,
|
||||
gcs_path: Optional[str] = None,
|
||||
) -> AudioDownloadResult:
|
||||
"""
|
||||
Search for audio and automatically download the best result.
|
||||
|
||||
This is a convenience method that combines search(), select_best(),
|
||||
and download() for automated/non-interactive usage.
|
||||
|
||||
Args:
|
||||
artist: The artist name to search for
|
||||
title: The track title to search for
|
||||
output_dir: Directory to save the downloaded file
|
||||
output_filename: Optional filename (without extension)
|
||||
gcs_path: Optional GCS path for remote uploads
|
||||
|
||||
Returns:
|
||||
AudioDownloadResult with the downloaded file path
|
||||
|
||||
Raises:
|
||||
NoResultsError: If no results are found
|
||||
DownloadError: If download fails
|
||||
"""
|
||||
# Search
|
||||
results = self.search(artist, title)
|
||||
|
||||
# Select best
|
||||
best_index = self.select_best(results)
|
||||
best_result = results[best_index]
|
||||
logger.info(
|
||||
f"Auto-selected result {best_index}: "
|
||||
f"{best_result.artist} - {best_result.title} ({best_result.provider})"
|
||||
)
|
||||
|
||||
# Download
|
||||
return self.download(best_index, output_dir, output_filename, gcs_path)
|
||||
|
||||
async def search_async(self, artist: str, title: str) -> List[AudioSearchResult]:
|
||||
"""
|
||||
Async version of search for use in async routes.
|
||||
|
||||
Note: Currently wraps sync search in executor. Future optimization
|
||||
could make this fully async.
|
||||
"""
|
||||
loop = asyncio.get_event_loop()
|
||||
return await loop.run_in_executor(None, self.search, artist, title)
|
||||
|
||||
async def download_async(
|
||||
self,
|
||||
result_index: int,
|
||||
output_dir: str,
|
||||
output_filename: Optional[str] = None,
|
||||
gcs_path: Optional[str] = None,
|
||||
remote_search_id: Optional[str] = None,
|
||||
) -> AudioDownloadResult:
|
||||
"""
|
||||
Async version of download for use in async routes.
|
||||
|
||||
Note: Currently wraps sync download in executor. Future optimization
|
||||
could make this fully async.
|
||||
"""
|
||||
loop = asyncio.get_event_loop()
|
||||
return await loop.run_in_executor(
|
||||
None,
|
||||
lambda: self.download(result_index, output_dir, output_filename, gcs_path, remote_search_id)
|
||||
)
|
||||
|
||||
async def download_by_id_async(
|
||||
self,
|
||||
source_name: str,
|
||||
source_id: str,
|
||||
output_dir: str,
|
||||
output_filename: Optional[str] = None,
|
||||
target_file: Optional[str] = None,
|
||||
download_url: Optional[str] = None,
|
||||
gcs_path: Optional[str] = None,
|
||||
) -> AudioDownloadResult:
|
||||
"""
|
||||
Async version of download_by_id for use in async routes.
|
||||
"""
|
||||
loop = asyncio.get_event_loop()
|
||||
return await loop.run_in_executor(
|
||||
None,
|
||||
lambda: self.download_by_id(
|
||||
source_name, source_id, output_dir, output_filename,
|
||||
target_file, download_url, gcs_path
|
||||
)
|
||||
)
|
||||
|
||||
def is_remote_enabled(self) -> bool:
|
||||
"""Check if remote flacfetch service is configured."""
|
||||
return self._remote_client is not None
|
||||
|
||||
@property
|
||||
def last_remote_search_id(self) -> Optional[str]:
|
||||
"""Get the search_id from the last remote search.
|
||||
|
||||
This should be stored in job state_data after search and passed
|
||||
back to download() to ensure correct remote download handling.
|
||||
"""
|
||||
return self._remote_search_id
|
||||
|
||||
async def check_remote_health(self) -> Optional[dict]:
|
||||
"""
|
||||
Check health of remote flacfetch service.
|
||||
|
||||
Returns:
|
||||
Health status dict if remote service is configured and healthy,
|
||||
None if not configured or unhealthy.
|
||||
"""
|
||||
if not self._remote_client:
|
||||
return None
|
||||
|
||||
try:
|
||||
return await self._remote_client.health_check()
|
||||
except Exception as e:
|
||||
logger.warning(f"Remote flacfetch health check failed: {e}")
|
||||
return None
|
||||
|
||||
|
||||
# Singleton instance
|
||||
_audio_search_service: Optional[AudioSearchService] = None
|
||||
|
||||
|
||||
def get_audio_search_service() -> AudioSearchService:
|
||||
"""Get the singleton AudioSearchService instance."""
|
||||
global _audio_search_service
|
||||
if _audio_search_service is None:
|
||||
_audio_search_service = AudioSearchService()
|
||||
return _audio_search_service
|
||||
@@ -1,295 +0,0 @@
|
||||
"""
|
||||
Authentication service for karaoke backend.
|
||||
|
||||
Provides token-based authentication with support for:
|
||||
- Admin tokens (hardcoded in environment)
|
||||
- User tokens (stored in Firestore)
|
||||
- Usage tracking and limits
|
||||
- Token management API
|
||||
|
||||
This is designed to be extensible for future Stripe integration
|
||||
and admin dashboard functionality.
|
||||
"""
|
||||
import logging
|
||||
import hashlib
|
||||
import time
|
||||
from enum import Enum
|
||||
from typing import Optional, Tuple, Dict, Any
|
||||
from datetime import datetime
|
||||
|
||||
from backend.services.firestore_service import FirestoreService
|
||||
from backend.config import get_settings
|
||||
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
|
||||
class UserType(str, Enum):
|
||||
"""User access levels."""
|
||||
ADMIN = "admin" # Unlimited access, can manage tokens
|
||||
UNLIMITED = "unlimited" # Unlimited karaoke generation
|
||||
LIMITED = "limited" # Limited number of uses
|
||||
STRIPE = "stripe" # Paid access via Stripe (future)
|
||||
|
||||
|
||||
class AuthService:
|
||||
"""Service for authentication and token management."""
|
||||
|
||||
def __init__(self):
|
||||
"""Initialize auth service."""
|
||||
self.firestore = FirestoreService()
|
||||
self.settings = get_settings()
|
||||
self._init_admin_tokens()
|
||||
|
||||
def _init_admin_tokens(self):
|
||||
"""Load admin tokens from environment variables."""
|
||||
# Get admin tokens from secret manager or env vars
|
||||
admin_tokens_str = self.settings.admin_tokens or ""
|
||||
self.admin_tokens = [t.strip() for t in admin_tokens_str.split(",") if t.strip()]
|
||||
|
||||
if self.admin_tokens:
|
||||
logger.info(f"Loaded {len(self.admin_tokens)} admin token(s)")
|
||||
else:
|
||||
logger.warning("No admin tokens configured! Set ADMIN_TOKENS environment variable.")
|
||||
|
||||
def validate_token(self, token: str) -> Tuple[bool, UserType, int, str]:
|
||||
"""
|
||||
Validate an access token.
|
||||
|
||||
Returns:
|
||||
(is_valid, user_type, remaining_uses, message)
|
||||
remaining_uses: -1 = unlimited, 0 = exhausted, >0 = remaining count
|
||||
"""
|
||||
if not token:
|
||||
return False, UserType.LIMITED, 0, "No token provided"
|
||||
|
||||
# Check for admin tokens (highest priority)
|
||||
if token in self.admin_tokens:
|
||||
logger.info("Admin token validated")
|
||||
return True, UserType.ADMIN, -1, "Admin access granted"
|
||||
|
||||
# Debug: Log token comparison info (only first/last few chars for security)
|
||||
if self.admin_tokens:
|
||||
expected_prefix = self.admin_tokens[0][:8] if len(self.admin_tokens[0]) >= 8 else self.admin_tokens[0]
|
||||
provided_prefix = token[:8] if len(token) >= 8 else token
|
||||
logger.debug(
|
||||
f"Token mismatch: expected prefix '{expected_prefix}...', "
|
||||
f"got '{provided_prefix}...', "
|
||||
f"expected len={len(self.admin_tokens[0])}, got len={len(token)}"
|
||||
)
|
||||
|
||||
# Check stored tokens in Firestore
|
||||
token_data = self.firestore.get_token(token)
|
||||
|
||||
if not token_data:
|
||||
return False, UserType.LIMITED, 0, "Invalid token"
|
||||
|
||||
# Check if token is active
|
||||
if not token_data.get("active", True):
|
||||
return False, UserType(token_data["type"]), 0, "Token has been revoked"
|
||||
|
||||
token_type = UserType(token_data["type"])
|
||||
max_uses = token_data.get("max_uses", -1)
|
||||
|
||||
# UNLIMITED tokens: no usage limits
|
||||
if token_type == UserType.UNLIMITED:
|
||||
return True, token_type, -1, "Unlimited access granted"
|
||||
|
||||
# LIMITED tokens: check usage count
|
||||
if token_type == UserType.LIMITED:
|
||||
if max_uses <= 0: # -1 means unlimited
|
||||
return True, token_type, -1, "Limited token with unlimited uses"
|
||||
|
||||
current_uses = token_data.get("usage_count", 0)
|
||||
remaining = max_uses - current_uses
|
||||
|
||||
if remaining <= 0:
|
||||
return False, token_type, 0, "Token usage limit exceeded"
|
||||
|
||||
return True, token_type, remaining, f"Limited token: {remaining} uses remaining"
|
||||
|
||||
# STRIPE tokens: check expiration and usage
|
||||
if token_type == UserType.STRIPE:
|
||||
expires_at = token_data.get("expires_at")
|
||||
|
||||
if expires_at:
|
||||
# expires_at is stored as timestamp
|
||||
if time.time() > expires_at:
|
||||
return False, token_type, 0, "Token has expired"
|
||||
|
||||
if max_uses > 0:
|
||||
current_uses = token_data.get("usage_count", 0)
|
||||
remaining = max_uses - current_uses
|
||||
|
||||
if remaining <= 0:
|
||||
return False, token_type, 0, "Token usage limit exceeded"
|
||||
|
||||
return True, token_type, remaining, f"Stripe token: {remaining} uses remaining"
|
||||
|
||||
return True, token_type, -1, "Stripe access granted"
|
||||
|
||||
return False, UserType.LIMITED, 0, "Unknown token type"
|
||||
|
||||
def increment_token_usage(self, token: str, job_id: str) -> bool:
|
||||
"""
|
||||
Increment usage count for a token and track the job.
|
||||
|
||||
Args:
|
||||
token: The access token
|
||||
job_id: The job ID being created
|
||||
|
||||
Returns:
|
||||
True if usage was tracked, False otherwise
|
||||
"""
|
||||
# Validate token first
|
||||
is_valid, user_type, remaining_uses, message = self.validate_token(token)
|
||||
|
||||
if not is_valid:
|
||||
logger.warning(f"Cannot increment usage for invalid token: {message}")
|
||||
return False
|
||||
|
||||
# Don't track usage for admin or unlimited tokens
|
||||
if user_type in [UserType.ADMIN, UserType.UNLIMITED]:
|
||||
logger.debug(f"Skipping usage tracking for {user_type} token")
|
||||
return True
|
||||
|
||||
# For admin tokens (not in Firestore), no tracking needed
|
||||
if token in self.admin_tokens:
|
||||
return True
|
||||
|
||||
# Increment usage in Firestore
|
||||
try:
|
||||
self.firestore.increment_token_usage(token, job_id)
|
||||
logger.info(f"Incremented usage for token (remaining: {remaining_uses - 1})")
|
||||
return True
|
||||
except Exception as e:
|
||||
logger.error(f"Failed to increment token usage: {e}")
|
||||
return False
|
||||
|
||||
def create_token(
|
||||
self,
|
||||
token_value: str,
|
||||
token_type: UserType,
|
||||
max_uses: int = -1,
|
||||
expires_at: Optional[float] = None,
|
||||
created_by: Optional[str] = None
|
||||
) -> Dict[str, Any]:
|
||||
"""
|
||||
Create a new access token.
|
||||
|
||||
Args:
|
||||
token_value: The actual token string (should be secure random string)
|
||||
token_type: Type of access (admin, unlimited, limited, stripe)
|
||||
max_uses: Maximum number of uses (-1 = unlimited)
|
||||
expires_at: Expiration timestamp (None = no expiration)
|
||||
created_by: Admin token that created this token
|
||||
|
||||
Returns:
|
||||
Token data dictionary
|
||||
"""
|
||||
token_data = {
|
||||
"token": token_value,
|
||||
"type": token_type.value,
|
||||
"max_uses": max_uses,
|
||||
"usage_count": 0,
|
||||
"active": True,
|
||||
"created_at": time.time(),
|
||||
"created_by": created_by,
|
||||
"expires_at": expires_at,
|
||||
"jobs": []
|
||||
}
|
||||
|
||||
self.firestore.create_token(token_value, token_data)
|
||||
logger.info(f"Created new {token_type} token with max_uses={max_uses}")
|
||||
|
||||
return token_data
|
||||
|
||||
def revoke_token(self, token: str, revoked_by: Optional[str] = None) -> bool:
|
||||
"""
|
||||
Revoke an access token.
|
||||
|
||||
Args:
|
||||
token: The token to revoke
|
||||
revoked_by: Admin token that revoked this token
|
||||
|
||||
Returns:
|
||||
True if revoked, False if token not found
|
||||
"""
|
||||
try:
|
||||
self.firestore.update_token(token, {
|
||||
"active": False,
|
||||
"revoked_at": time.time(),
|
||||
"revoked_by": revoked_by
|
||||
})
|
||||
logger.info(f"Revoked token")
|
||||
return True
|
||||
except Exception as e:
|
||||
logger.error(f"Failed to revoke token: {e}")
|
||||
return False
|
||||
|
||||
def list_tokens(self, include_inactive: bool = False) -> list:
|
||||
"""
|
||||
List all tokens (admin only).
|
||||
|
||||
Args:
|
||||
include_inactive: Whether to include revoked tokens
|
||||
|
||||
Returns:
|
||||
List of token data dictionaries (with sensitive data masked)
|
||||
"""
|
||||
try:
|
||||
tokens = self.firestore.list_tokens()
|
||||
|
||||
# Filter inactive if requested
|
||||
if not include_inactive:
|
||||
tokens = [t for t in tokens if t.get("active", True)]
|
||||
|
||||
# Mask token values for security (show only first 8 chars)
|
||||
for token in tokens:
|
||||
if "token" in token and len(token["token"]) > 8:
|
||||
token["token"] = token["token"][:8] + "..."
|
||||
|
||||
return tokens
|
||||
except Exception as e:
|
||||
logger.error(f"Failed to list tokens: {e}")
|
||||
return []
|
||||
|
||||
def get_token_info(self, token: str) -> Optional[Dict[str, Any]]:
|
||||
"""
|
||||
Get information about a token (for admin dashboard).
|
||||
|
||||
Returns:
|
||||
Token data or None if not found
|
||||
"""
|
||||
# Check if it's an admin token (not stored in Firestore)
|
||||
if token in self.admin_tokens:
|
||||
return {
|
||||
"token": token[:8] + "...",
|
||||
"type": UserType.ADMIN.value,
|
||||
"max_uses": -1,
|
||||
"usage_count": 0,
|
||||
"active": True,
|
||||
"source": "environment"
|
||||
}
|
||||
|
||||
# Get from Firestore
|
||||
token_data = self.firestore.get_token(token)
|
||||
|
||||
if token_data and "token" in token_data:
|
||||
# Mask the full token value
|
||||
token_data["token"] = token_data["token"][:8] + "..."
|
||||
|
||||
return token_data
|
||||
|
||||
|
||||
# Global instance
|
||||
_auth_service = None
|
||||
|
||||
|
||||
def get_auth_service() -> AuthService:
|
||||
"""Get the global auth service instance."""
|
||||
global _auth_service
|
||||
if _auth_service is None:
|
||||
_auth_service = AuthService()
|
||||
return _auth_service
|
||||
|
||||
@@ -1,792 +0,0 @@
|
||||
"""
|
||||
OAuth Credential Manager for validating and refreshing credentials.
|
||||
|
||||
This service provides:
|
||||
1. Credential validation for YouTube, Dropbox, and Google Drive
|
||||
2. Device Authorization Flow for re-authentication
|
||||
3. Proactive monitoring with alerts when credentials expire
|
||||
"""
|
||||
import json
|
||||
import logging
|
||||
import time
|
||||
from typing import Optional, Dict, Any, Tuple
|
||||
from dataclasses import dataclass
|
||||
from enum import Enum
|
||||
from datetime import datetime
|
||||
|
||||
from google.oauth2.credentials import Credentials as GoogleCredentials
|
||||
from google.auth.transport.requests import Request as GoogleAuthRequest
|
||||
|
||||
from backend.config import get_settings
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
|
||||
class CredentialStatus(str, Enum):
|
||||
"""Status of OAuth credentials."""
|
||||
VALID = "valid"
|
||||
EXPIRED = "expired" # Token expired but refresh may work
|
||||
INVALID = "invalid" # Refresh failed, re-auth needed
|
||||
NOT_CONFIGURED = "not_configured" # No credentials stored
|
||||
ERROR = "error" # Unknown error during validation
|
||||
|
||||
|
||||
@dataclass
|
||||
class CredentialCheckResult:
|
||||
"""Result of a credential validation check."""
|
||||
service: str
|
||||
status: CredentialStatus
|
||||
message: str
|
||||
last_checked: datetime
|
||||
expires_at: Optional[datetime] = None
|
||||
|
||||
|
||||
@dataclass
|
||||
class DeviceAuthInfo:
|
||||
"""Information for device authorization flow."""
|
||||
device_code: str
|
||||
user_code: str
|
||||
verification_url: str
|
||||
expires_in: int
|
||||
interval: int
|
||||
started_at: datetime
|
||||
|
||||
|
||||
class CredentialManager:
|
||||
"""
|
||||
Manages OAuth credentials for all external services.
|
||||
|
||||
Provides validation, refresh, and device authorization flow for:
|
||||
- YouTube (Google OAuth)
|
||||
- Google Drive (Google OAuth)
|
||||
- Dropbox
|
||||
"""
|
||||
|
||||
# Secret names in Secret Manager
|
||||
YOUTUBE_SECRET = "youtube-oauth-credentials"
|
||||
GDRIVE_SECRET = "gdrive-oauth-credentials"
|
||||
DROPBOX_SECRET = "dropbox-oauth-credentials"
|
||||
|
||||
# OAuth client credentials (for device auth flow)
|
||||
YOUTUBE_CLIENT_SECRET = "youtube-client-credentials"
|
||||
GDRIVE_CLIENT_SECRET = "gdrive-client-credentials"
|
||||
|
||||
# Google OAuth endpoints
|
||||
GOOGLE_TOKEN_URI = "https://oauth2.googleapis.com/token"
|
||||
GOOGLE_DEVICE_AUTH_URI = "https://oauth2.googleapis.com/device/code"
|
||||
|
||||
# Dropbox OAuth endpoints
|
||||
DROPBOX_TOKEN_URI = "https://api.dropboxapi.com/oauth2/token"
|
||||
|
||||
# Scopes for each service
|
||||
YOUTUBE_SCOPES = ["https://www.googleapis.com/auth/youtube"]
|
||||
GDRIVE_SCOPES = ["https://www.googleapis.com/auth/drive.file"]
|
||||
|
||||
def __init__(self):
|
||||
self.settings = get_settings()
|
||||
self._pending_device_auths: Dict[str, DeviceAuthInfo] = {}
|
||||
|
||||
def _get_client_credentials(self, secret_name: str) -> Optional[Dict[str, str]]:
|
||||
"""
|
||||
Load OAuth client credentials from Secret Manager.
|
||||
|
||||
The secret should contain:
|
||||
{
|
||||
"client_id": "...",
|
||||
"client_secret": "..."
|
||||
}
|
||||
"""
|
||||
try:
|
||||
creds_json = self.settings.get_secret(secret_name)
|
||||
if not creds_json:
|
||||
return None
|
||||
|
||||
creds = json.loads(creds_json)
|
||||
if creds.get("client_id") and creds.get("client_secret"):
|
||||
return creds
|
||||
|
||||
logger.warning(f"{secret_name} missing client_id or client_secret")
|
||||
return None
|
||||
|
||||
except Exception as e:
|
||||
logger.error(f"Failed to load {secret_name}: {e}")
|
||||
return None
|
||||
|
||||
def get_youtube_client_credentials(self) -> Optional[Dict[str, str]]:
|
||||
"""Get YouTube OAuth client credentials from Secret Manager."""
|
||||
return self._get_client_credentials(self.YOUTUBE_CLIENT_SECRET)
|
||||
|
||||
def get_gdrive_client_credentials(self) -> Optional[Dict[str, str]]:
|
||||
"""Get Google Drive OAuth client credentials from Secret Manager."""
|
||||
return self._get_client_credentials(self.GDRIVE_CLIENT_SECRET)
|
||||
|
||||
# =========================================================================
|
||||
# Credential Validation
|
||||
# =========================================================================
|
||||
|
||||
def check_all_credentials(self) -> Dict[str, CredentialCheckResult]:
|
||||
"""
|
||||
Check validity of all OAuth credentials.
|
||||
|
||||
Returns:
|
||||
Dictionary mapping service name to check result
|
||||
"""
|
||||
results = {}
|
||||
|
||||
results["youtube"] = self.check_youtube_credentials()
|
||||
results["gdrive"] = self.check_gdrive_credentials()
|
||||
results["dropbox"] = self.check_dropbox_credentials()
|
||||
|
||||
return results
|
||||
|
||||
def check_youtube_credentials(self) -> CredentialCheckResult:
|
||||
"""Check if YouTube credentials are valid and can be refreshed."""
|
||||
return self._check_google_credentials(
|
||||
secret_name=self.YOUTUBE_SECRET,
|
||||
service_name="youtube",
|
||||
scopes=self.YOUTUBE_SCOPES,
|
||||
test_api_call=self._test_youtube_api
|
||||
)
|
||||
|
||||
def check_gdrive_credentials(self) -> CredentialCheckResult:
|
||||
"""Check if Google Drive credentials are valid and can be refreshed."""
|
||||
return self._check_google_credentials(
|
||||
secret_name=self.GDRIVE_SECRET,
|
||||
service_name="gdrive",
|
||||
scopes=self.GDRIVE_SCOPES,
|
||||
test_api_call=self._test_gdrive_api
|
||||
)
|
||||
|
||||
def check_dropbox_credentials(self) -> CredentialCheckResult:
|
||||
"""Check if Dropbox credentials are valid."""
|
||||
try:
|
||||
creds_json = self.settings.get_secret(self.DROPBOX_SECRET)
|
||||
|
||||
if not creds_json:
|
||||
return CredentialCheckResult(
|
||||
service="dropbox",
|
||||
status=CredentialStatus.NOT_CONFIGURED,
|
||||
message="Dropbox credentials not configured",
|
||||
last_checked=datetime.utcnow()
|
||||
)
|
||||
|
||||
creds = json.loads(creds_json)
|
||||
|
||||
if not creds.get("access_token"):
|
||||
return CredentialCheckResult(
|
||||
service="dropbox",
|
||||
status=CredentialStatus.INVALID,
|
||||
message="Dropbox credentials missing access_token",
|
||||
last_checked=datetime.utcnow()
|
||||
)
|
||||
|
||||
# Test the credentials with a simple API call
|
||||
if self._test_dropbox_api(creds):
|
||||
return CredentialCheckResult(
|
||||
service="dropbox",
|
||||
status=CredentialStatus.VALID,
|
||||
message="Dropbox credentials are valid",
|
||||
last_checked=datetime.utcnow()
|
||||
)
|
||||
else:
|
||||
# Try to refresh if we have refresh token
|
||||
if creds.get("refresh_token") and creds.get("app_key") and creds.get("app_secret"):
|
||||
if self._refresh_dropbox_token(creds):
|
||||
return CredentialCheckResult(
|
||||
service="dropbox",
|
||||
status=CredentialStatus.VALID,
|
||||
message="Dropbox credentials refreshed successfully",
|
||||
last_checked=datetime.utcnow()
|
||||
)
|
||||
|
||||
return CredentialCheckResult(
|
||||
service="dropbox",
|
||||
status=CredentialStatus.INVALID,
|
||||
message="Dropbox credentials invalid and refresh failed",
|
||||
last_checked=datetime.utcnow()
|
||||
)
|
||||
|
||||
except Exception as e:
|
||||
logger.error(f"Error checking Dropbox credentials: {e}")
|
||||
return CredentialCheckResult(
|
||||
service="dropbox",
|
||||
status=CredentialStatus.ERROR,
|
||||
message=f"Error checking credentials: {str(e)}",
|
||||
last_checked=datetime.utcnow()
|
||||
)
|
||||
|
||||
def _check_google_credentials(
|
||||
self,
|
||||
secret_name: str,
|
||||
service_name: str,
|
||||
scopes: list,
|
||||
test_api_call: callable
|
||||
) -> CredentialCheckResult:
|
||||
"""Generic Google OAuth credential check."""
|
||||
try:
|
||||
creds_json = self.settings.get_secret(secret_name)
|
||||
|
||||
if not creds_json:
|
||||
return CredentialCheckResult(
|
||||
service=service_name,
|
||||
status=CredentialStatus.NOT_CONFIGURED,
|
||||
message=f"{service_name} credentials not configured",
|
||||
last_checked=datetime.utcnow()
|
||||
)
|
||||
|
||||
creds_data = json.loads(creds_json)
|
||||
|
||||
# Check for required fields
|
||||
required = ["refresh_token", "client_id", "client_secret"]
|
||||
missing = [f for f in required if not creds_data.get(f)]
|
||||
if missing:
|
||||
return CredentialCheckResult(
|
||||
service=service_name,
|
||||
status=CredentialStatus.INVALID,
|
||||
message=f"Missing required fields: {missing}",
|
||||
last_checked=datetime.utcnow()
|
||||
)
|
||||
|
||||
# Create credentials object
|
||||
credentials = GoogleCredentials(
|
||||
token=creds_data.get("token"),
|
||||
refresh_token=creds_data.get("refresh_token"),
|
||||
token_uri=creds_data.get("token_uri", self.GOOGLE_TOKEN_URI),
|
||||
client_id=creds_data.get("client_id"),
|
||||
client_secret=creds_data.get("client_secret"),
|
||||
scopes=creds_data.get("scopes", scopes)
|
||||
)
|
||||
|
||||
# Try to refresh if expired
|
||||
if credentials.expired or not credentials.token:
|
||||
try:
|
||||
credentials.refresh(GoogleAuthRequest())
|
||||
# Update stored credentials with new token
|
||||
self._update_google_credentials(secret_name, creds_data, credentials)
|
||||
except Exception as e:
|
||||
logger.error(f"Failed to refresh {service_name} credentials: {e}")
|
||||
return CredentialCheckResult(
|
||||
service=service_name,
|
||||
status=CredentialStatus.INVALID,
|
||||
message=f"Token refresh failed: {str(e)}",
|
||||
last_checked=datetime.utcnow()
|
||||
)
|
||||
|
||||
# Test with API call
|
||||
if test_api_call(credentials):
|
||||
return CredentialCheckResult(
|
||||
service=service_name,
|
||||
status=CredentialStatus.VALID,
|
||||
message=f"{service_name} credentials are valid",
|
||||
last_checked=datetime.utcnow(),
|
||||
expires_at=credentials.expiry
|
||||
)
|
||||
else:
|
||||
return CredentialCheckResult(
|
||||
service=service_name,
|
||||
status=CredentialStatus.INVALID,
|
||||
message=f"{service_name} API test failed",
|
||||
last_checked=datetime.utcnow()
|
||||
)
|
||||
|
||||
except json.JSONDecodeError as e:
|
||||
return CredentialCheckResult(
|
||||
service=service_name,
|
||||
status=CredentialStatus.INVALID,
|
||||
message=f"Invalid JSON in credentials: {str(e)}",
|
||||
last_checked=datetime.utcnow()
|
||||
)
|
||||
except Exception as e:
|
||||
logger.error(f"Error checking {service_name} credentials: {e}")
|
||||
return CredentialCheckResult(
|
||||
service=service_name,
|
||||
status=CredentialStatus.ERROR,
|
||||
message=f"Error: {str(e)}",
|
||||
last_checked=datetime.utcnow()
|
||||
)
|
||||
|
||||
def _update_google_credentials(
|
||||
self,
|
||||
secret_name: str,
|
||||
original_data: dict,
|
||||
credentials: GoogleCredentials
|
||||
) -> bool:
|
||||
"""Update stored Google credentials with refreshed token."""
|
||||
try:
|
||||
from google.cloud import secretmanager
|
||||
|
||||
updated_data = original_data.copy()
|
||||
updated_data["token"] = credentials.token
|
||||
if credentials.expiry:
|
||||
updated_data["expiry"] = credentials.expiry.isoformat()
|
||||
|
||||
# Add new secret version
|
||||
client = secretmanager.SecretManagerServiceClient()
|
||||
project_id = self.settings.google_cloud_project
|
||||
secret_path = f"projects/{project_id}/secrets/{secret_name}"
|
||||
|
||||
client.add_secret_version(
|
||||
parent=secret_path,
|
||||
payload={"data": json.dumps(updated_data).encode("utf-8")}
|
||||
)
|
||||
|
||||
logger.info(f"Updated {secret_name} with refreshed token")
|
||||
return True
|
||||
|
||||
except Exception as e:
|
||||
logger.error(f"Failed to update {secret_name}: {e}")
|
||||
return False
|
||||
|
||||
# =========================================================================
|
||||
# API Test Methods
|
||||
# =========================================================================
|
||||
|
||||
def _test_youtube_api(self, credentials: GoogleCredentials) -> bool:
|
||||
"""Test YouTube credentials with a simple API call."""
|
||||
try:
|
||||
from googleapiclient.discovery import build
|
||||
|
||||
youtube = build("youtube", "v3", credentials=credentials)
|
||||
# Just get channel info - minimal API call
|
||||
request = youtube.channels().list(part="id", mine=True)
|
||||
response = request.execute()
|
||||
|
||||
return "items" in response
|
||||
|
||||
except Exception as e:
|
||||
logger.error(f"YouTube API test failed: {e}")
|
||||
return False
|
||||
|
||||
def _test_gdrive_api(self, credentials: GoogleCredentials) -> bool:
|
||||
"""Test Google Drive credentials with a simple API call."""
|
||||
try:
|
||||
from googleapiclient.discovery import build
|
||||
|
||||
drive = build("drive", "v3", credentials=credentials)
|
||||
# Just get about info - minimal API call
|
||||
request = drive.about().get(fields="user")
|
||||
response = request.execute()
|
||||
|
||||
return "user" in response
|
||||
|
||||
except Exception as e:
|
||||
logger.error(f"Google Drive API test failed: {e}")
|
||||
return False
|
||||
|
||||
def _test_dropbox_api(self, creds: dict) -> bool:
|
||||
"""Test Dropbox credentials with a simple API call."""
|
||||
try:
|
||||
import dropbox
|
||||
|
||||
dbx = dropbox.Dropbox(creds["access_token"])
|
||||
# Just get account info - minimal API call
|
||||
dbx.users_get_current_account()
|
||||
return True
|
||||
|
||||
except Exception as e:
|
||||
logger.error(f"Dropbox API test failed: {e}")
|
||||
return False
|
||||
|
||||
def _refresh_dropbox_token(self, creds: dict) -> bool:
|
||||
"""Try to refresh Dropbox token."""
|
||||
try:
|
||||
import requests
|
||||
|
||||
response = requests.post(
|
||||
self.DROPBOX_TOKEN_URI,
|
||||
data={
|
||||
"grant_type": "refresh_token",
|
||||
"refresh_token": creds["refresh_token"],
|
||||
"client_id": creds["app_key"],
|
||||
"client_secret": creds["app_secret"],
|
||||
},
|
||||
timeout=30
|
||||
)
|
||||
|
||||
if response.status_code == 200:
|
||||
token_data = response.json()
|
||||
creds["access_token"] = token_data["access_token"]
|
||||
|
||||
# Update in Secret Manager
|
||||
self._update_dropbox_credentials(creds)
|
||||
return True
|
||||
|
||||
return False
|
||||
|
||||
except Exception as e:
|
||||
logger.error(f"Dropbox token refresh failed: {e}")
|
||||
return False
|
||||
|
||||
def _update_dropbox_credentials(self, creds: dict) -> bool:
|
||||
"""Update stored Dropbox credentials."""
|
||||
try:
|
||||
from google.cloud import secretmanager
|
||||
|
||||
client = secretmanager.SecretManagerServiceClient()
|
||||
project_id = self.settings.google_cloud_project
|
||||
secret_path = f"projects/{project_id}/secrets/{self.DROPBOX_SECRET}"
|
||||
|
||||
client.add_secret_version(
|
||||
parent=secret_path,
|
||||
payload={"data": json.dumps(creds).encode("utf-8")}
|
||||
)
|
||||
|
||||
logger.info("Updated Dropbox credentials with refreshed token")
|
||||
return True
|
||||
|
||||
except Exception as e:
|
||||
logger.error(f"Failed to update Dropbox credentials: {e}")
|
||||
return False
|
||||
|
||||
# =========================================================================
|
||||
# Device Authorization Flow
|
||||
# =========================================================================
|
||||
|
||||
def start_youtube_device_auth(
|
||||
self,
|
||||
client_id: Optional[str] = None,
|
||||
client_secret: Optional[str] = None
|
||||
) -> DeviceAuthInfo:
|
||||
"""
|
||||
Start YouTube device authorization flow.
|
||||
|
||||
Args:
|
||||
client_id: Google OAuth client ID (optional, reads from Secret Manager if not provided)
|
||||
client_secret: Google OAuth client secret (optional, reads from Secret Manager if not provided)
|
||||
|
||||
Returns:
|
||||
DeviceAuthInfo with user code and verification URL
|
||||
"""
|
||||
# Load from Secret Manager if not provided
|
||||
if not client_id or not client_secret:
|
||||
stored_creds = self.get_youtube_client_credentials()
|
||||
if not stored_creds:
|
||||
raise Exception(
|
||||
"YouTube client credentials not found. Either pass client_id/client_secret "
|
||||
"or create the 'youtube-client-credentials' secret in Secret Manager."
|
||||
)
|
||||
client_id = stored_creds["client_id"]
|
||||
client_secret = stored_creds["client_secret"]
|
||||
|
||||
return self._start_google_device_auth(
|
||||
client_id=client_id,
|
||||
client_secret=client_secret,
|
||||
scopes=self.YOUTUBE_SCOPES,
|
||||
service_name="youtube"
|
||||
)
|
||||
|
||||
def start_gdrive_device_auth(
|
||||
self,
|
||||
client_id: Optional[str] = None,
|
||||
client_secret: Optional[str] = None
|
||||
) -> DeviceAuthInfo:
|
||||
"""
|
||||
Start Google Drive device authorization flow.
|
||||
|
||||
Args:
|
||||
client_id: Google OAuth client ID (optional, reads from Secret Manager if not provided)
|
||||
client_secret: Google OAuth client secret (optional, reads from Secret Manager if not provided)
|
||||
|
||||
Returns:
|
||||
DeviceAuthInfo with user code and verification URL
|
||||
"""
|
||||
# Load from Secret Manager if not provided
|
||||
if not client_id or not client_secret:
|
||||
stored_creds = self.get_gdrive_client_credentials()
|
||||
if not stored_creds:
|
||||
raise Exception(
|
||||
"Google Drive client credentials not found. Either pass client_id/client_secret "
|
||||
"or create the 'gdrive-client-credentials' secret in Secret Manager."
|
||||
)
|
||||
client_id = stored_creds["client_id"]
|
||||
client_secret = stored_creds["client_secret"]
|
||||
|
||||
return self._start_google_device_auth(
|
||||
client_id=client_id,
|
||||
client_secret=client_secret,
|
||||
scopes=self.GDRIVE_SCOPES,
|
||||
service_name="gdrive"
|
||||
)
|
||||
|
||||
def _start_google_device_auth(
|
||||
self,
|
||||
client_id: str,
|
||||
client_secret: str,
|
||||
scopes: list,
|
||||
service_name: str
|
||||
) -> DeviceAuthInfo:
|
||||
"""Start Google device authorization flow."""
|
||||
import requests
|
||||
|
||||
logger.info(f"[{service_name}] Starting device auth flow with scopes: {scopes}")
|
||||
|
||||
response = requests.post(
|
||||
self.GOOGLE_DEVICE_AUTH_URI,
|
||||
data={
|
||||
"client_id": client_id,
|
||||
"scope": " ".join(scopes),
|
||||
},
|
||||
timeout=30
|
||||
)
|
||||
|
||||
if response.status_code != 200:
|
||||
logger.error(f"[{service_name}] Device auth request failed: {response.status_code} - {response.text}")
|
||||
raise Exception(f"Device auth request failed: {response.text}")
|
||||
|
||||
logger.info(f"[{service_name}] Device auth initiated successfully")
|
||||
|
||||
data = response.json()
|
||||
|
||||
# Google uses 'verification_uri' but some docs show 'verification_url'
|
||||
verification_url = data.get("verification_uri") or data.get("verification_url")
|
||||
|
||||
device_info = DeviceAuthInfo(
|
||||
device_code=data["device_code"],
|
||||
user_code=data["user_code"],
|
||||
verification_url=verification_url,
|
||||
expires_in=data["expires_in"],
|
||||
interval=data.get("interval", 5),
|
||||
started_at=datetime.utcnow()
|
||||
)
|
||||
|
||||
# Store for polling, include client secret for token exchange
|
||||
self._pending_device_auths[f"{service_name}:{data['device_code']}"] = {
|
||||
"info": device_info,
|
||||
"client_id": client_id,
|
||||
"client_secret": client_secret,
|
||||
"scopes": scopes,
|
||||
"service_name": service_name
|
||||
}
|
||||
|
||||
return device_info
|
||||
|
||||
def poll_device_auth(self, service_name: str, device_code: str) -> Tuple[str, Optional[dict]]:
|
||||
"""
|
||||
Poll for device authorization completion.
|
||||
|
||||
This method is STATELESS - it fetches client credentials from Secret Manager
|
||||
and polls Google directly. This works correctly in serverless environments
|
||||
like Cloud Run where in-memory state is not preserved between requests.
|
||||
|
||||
Args:
|
||||
service_name: "youtube" or "gdrive"
|
||||
device_code: The device code from start_*_device_auth
|
||||
|
||||
Returns:
|
||||
Tuple of (status, token_data)
|
||||
status: "pending", "complete", "expired", "error"
|
||||
token_data: Token data if complete, None otherwise
|
||||
"""
|
||||
import requests
|
||||
|
||||
logger.info(f"[{service_name}] Polling device auth for code: {device_code[:20]}...")
|
||||
|
||||
# Get client credentials from Secret Manager (stateless approach)
|
||||
if service_name == "youtube":
|
||||
client_creds = self.get_youtube_client_credentials()
|
||||
scopes = self.YOUTUBE_SCOPES
|
||||
secret_name = self.YOUTUBE_SECRET
|
||||
elif service_name == "gdrive":
|
||||
client_creds = self.get_gdrive_client_credentials()
|
||||
scopes = self.GDRIVE_SCOPES
|
||||
secret_name = self.GDRIVE_SECRET
|
||||
else:
|
||||
logger.error(f"[{service_name}] Unknown service")
|
||||
return ("error", {"message": f"Unknown service: {service_name}"})
|
||||
|
||||
if not client_creds:
|
||||
logger.error(f"[{service_name}] Client credentials not found in Secret Manager")
|
||||
return ("error", {"message": f"Client credentials not found in Secret Manager for {service_name}"})
|
||||
|
||||
client_id = client_creds.get("client_id")
|
||||
client_secret = client_creds.get("client_secret")
|
||||
|
||||
if not client_id or not client_secret:
|
||||
logger.error(f"[{service_name}] Invalid client credentials (missing client_id or client_secret)")
|
||||
return ("error", {"message": f"Invalid client credentials for {service_name}"})
|
||||
|
||||
logger.info(f"[{service_name}] Got client credentials, polling Google token endpoint...")
|
||||
|
||||
# Poll Google token endpoint
|
||||
response = requests.post(
|
||||
self.GOOGLE_TOKEN_URI,
|
||||
data={
|
||||
"client_id": client_id,
|
||||
"client_secret": client_secret,
|
||||
"device_code": device_code,
|
||||
"grant_type": "urn:ietf:params:oauth:grant-type:device_code",
|
||||
},
|
||||
timeout=30
|
||||
)
|
||||
|
||||
data = response.json()
|
||||
logger.info(f"[{service_name}] Token endpoint response: status={response.status_code}")
|
||||
|
||||
if response.status_code == 200:
|
||||
logger.info(f"[{service_name}] Token exchange successful! Got access token and refresh_token={bool(data.get('refresh_token'))}")
|
||||
|
||||
# Success! Store the credentials
|
||||
token_data = {
|
||||
"token": data["access_token"],
|
||||
"refresh_token": data.get("refresh_token"),
|
||||
"token_uri": self.GOOGLE_TOKEN_URI,
|
||||
"client_id": client_id,
|
||||
"client_secret": client_secret,
|
||||
"scopes": scopes,
|
||||
}
|
||||
|
||||
# Save to Secret Manager
|
||||
logger.info(f"[{service_name}] Saving credentials to secret: {secret_name}")
|
||||
saved = self._save_credentials_to_secret(secret_name, token_data)
|
||||
if not saved:
|
||||
logger.error(f"[{service_name}] Failed to save credentials to Secret Manager!")
|
||||
return ("error", {"message": "Token exchange succeeded but failed to save credentials to Secret Manager"})
|
||||
|
||||
logger.info(f"[{service_name}] Device auth flow COMPLETE - credentials saved successfully")
|
||||
return ("complete", token_data)
|
||||
|
||||
elif "error" in data:
|
||||
error = data["error"]
|
||||
logger.info(f"[{service_name}] Token endpoint returned error: {error}")
|
||||
|
||||
if error == "authorization_pending":
|
||||
return ("pending", {"message": "Waiting for user to authorize. Please visit the verification URL and enter the code."})
|
||||
elif error == "slow_down":
|
||||
return ("pending", {"message": "Please wait a few more seconds before polling again."})
|
||||
elif error == "expired_token":
|
||||
logger.warning(f"[{service_name}] Device code expired")
|
||||
return ("expired", {"message": "Device code expired. Please start a new device auth flow."})
|
||||
elif error == "access_denied":
|
||||
logger.warning(f"[{service_name}] User denied access")
|
||||
return ("error", {"message": "User denied access"})
|
||||
else:
|
||||
logger.error(f"[{service_name}] Unexpected error: {data}")
|
||||
return ("error", {"message": data.get("error_description", error)})
|
||||
|
||||
logger.error(f"[{service_name}] Unknown response from token endpoint: {data}")
|
||||
return ("error", {"message": "Unknown response from token endpoint"})
|
||||
|
||||
def _save_credentials_to_secret(self, secret_name: str, token_data: dict) -> bool:
|
||||
"""Save credentials to Secret Manager, creating the secret if needed."""
|
||||
logger.info(f"[SecretManager] Attempting to save credentials to: {secret_name}")
|
||||
|
||||
try:
|
||||
from google.cloud import secretmanager
|
||||
from google.api_core import exceptions as gcp_exceptions
|
||||
|
||||
client = secretmanager.SecretManagerServiceClient()
|
||||
project_id = self.settings.google_cloud_project
|
||||
|
||||
if not project_id:
|
||||
logger.error(f"[SecretManager] No GCP project configured!")
|
||||
return False
|
||||
|
||||
parent = f"projects/{project_id}"
|
||||
secret_path = f"{parent}/secrets/{secret_name}"
|
||||
|
||||
logger.info(f"[SecretManager] Using project: {project_id}, secret path: {secret_path}")
|
||||
|
||||
# Try to add a version to existing secret
|
||||
try:
|
||||
client.add_secret_version(
|
||||
parent=secret_path,
|
||||
payload={"data": json.dumps(token_data).encode("utf-8")}
|
||||
)
|
||||
logger.info(f"[SecretManager] SUCCESS - Added new version to existing secret: {secret_name}")
|
||||
return True
|
||||
|
||||
except gcp_exceptions.NotFound:
|
||||
# Secret doesn't exist, create it first
|
||||
logger.info(f"[SecretManager] Secret {secret_name} not found, creating new secret...")
|
||||
client.create_secret(
|
||||
parent=parent,
|
||||
secret_id=secret_name,
|
||||
secret={"replication": {"automatic": {}}}
|
||||
)
|
||||
logger.info(f"[SecretManager] Created secret: {secret_name}")
|
||||
|
||||
# Now add the version
|
||||
client.add_secret_version(
|
||||
parent=secret_path,
|
||||
payload={"data": json.dumps(token_data).encode("utf-8")}
|
||||
)
|
||||
logger.info(f"[SecretManager] SUCCESS - Created secret and saved credentials to: {secret_name}")
|
||||
return True
|
||||
|
||||
except Exception as e:
|
||||
logger.error(f"[SecretManager] FAILED to save credentials to {secret_name}: {type(e).__name__}: {e}")
|
||||
import traceback
|
||||
logger.error(f"[SecretManager] Traceback: {traceback.format_exc()}")
|
||||
return False
|
||||
|
||||
# =========================================================================
|
||||
# Alerts
|
||||
# =========================================================================
|
||||
|
||||
def send_credential_alert(
|
||||
self,
|
||||
invalid_services: list[CredentialCheckResult],
|
||||
discord_webhook_url: Optional[str] = None
|
||||
) -> bool:
|
||||
"""
|
||||
Send alert about invalid credentials.
|
||||
|
||||
Args:
|
||||
invalid_services: List of services with invalid credentials
|
||||
discord_webhook_url: Discord webhook URL for notifications
|
||||
|
||||
Returns:
|
||||
True if alert was sent successfully
|
||||
"""
|
||||
if not discord_webhook_url:
|
||||
logger.warning("No Discord webhook URL configured for alerts")
|
||||
return False
|
||||
|
||||
try:
|
||||
import requests
|
||||
|
||||
services_list = "\n".join([
|
||||
f"• **{r.service}**: {r.message}"
|
||||
for r in invalid_services
|
||||
])
|
||||
|
||||
# Get the API base URL for re-auth links
|
||||
api_url = self.settings.api_base_url if hasattr(self.settings, 'api_base_url') else "https://your-api-url"
|
||||
|
||||
message = {
|
||||
"embeds": [{
|
||||
"title": "⚠️ OAuth Credentials Need Attention",
|
||||
"description": f"The following service credentials need re-authorization:\n\n{services_list}",
|
||||
"color": 16744256, # Orange
|
||||
"fields": [
|
||||
{
|
||||
"name": "Re-authorize",
|
||||
"value": f"Visit `{api_url}/api/auth/status` to start re-authorization flow",
|
||||
"inline": False
|
||||
}
|
||||
],
|
||||
"footer": {
|
||||
"text": "karaoke-gen backend"
|
||||
},
|
||||
"timestamp": datetime.utcnow().isoformat()
|
||||
}]
|
||||
}
|
||||
|
||||
response = requests.post(discord_webhook_url, json=message, timeout=30)
|
||||
return response.status_code in (200, 204)
|
||||
|
||||
except Exception as e:
|
||||
logger.error(f"Failed to send credential alert: {e}")
|
||||
return False
|
||||
|
||||
|
||||
# Singleton instance
|
||||
_credential_manager: Optional[CredentialManager] = None
|
||||
|
||||
|
||||
def get_credential_manager() -> CredentialManager:
|
||||
"""Get the singleton credential manager instance."""
|
||||
global _credential_manager
|
||||
if _credential_manager is None:
|
||||
_credential_manager = CredentialManager()
|
||||
return _credential_manager
|
||||
@@ -1,262 +0,0 @@
|
||||
"""Native Dropbox API service for cloud backend.
|
||||
|
||||
This service provides Dropbox operations using the native Python SDK,
|
||||
replacing rclone for server-side operations. It handles:
|
||||
- Folder listing for brand code calculation
|
||||
- File and folder uploads
|
||||
- Shared link generation
|
||||
|
||||
Credentials are loaded from Google Cloud Secret Manager.
|
||||
"""
|
||||
import json
|
||||
import logging
|
||||
import os
|
||||
import re
|
||||
from typing import Optional
|
||||
|
||||
from google.cloud import secretmanager
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
|
||||
class DropboxService:
|
||||
"""Dropbox operations using native Python SDK."""
|
||||
|
||||
def __init__(self):
|
||||
self._client = None
|
||||
self._is_configured = False
|
||||
|
||||
def _load_credentials(self) -> Optional[dict]:
|
||||
"""Load OAuth credentials from Secret Manager."""
|
||||
try:
|
||||
client = secretmanager.SecretManagerServiceClient()
|
||||
# Try to get the project ID from environment or use default
|
||||
project_id = os.environ.get("GOOGLE_CLOUD_PROJECT", "karaoke-gen")
|
||||
name = f"projects/{project_id}/secrets/dropbox-oauth-credentials/versions/latest"
|
||||
response = client.access_secret_version(request={"name": name})
|
||||
return json.loads(response.payload.data.decode("UTF-8"))
|
||||
except Exception as e:
|
||||
logger.warning(f"Failed to load Dropbox credentials from Secret Manager: {e}")
|
||||
return None
|
||||
|
||||
@property
|
||||
def is_configured(self) -> bool:
|
||||
"""Check if Dropbox credentials are available."""
|
||||
if self._is_configured:
|
||||
return True
|
||||
creds = self._load_credentials()
|
||||
self._is_configured = creds is not None and "access_token" in creds
|
||||
return self._is_configured
|
||||
|
||||
@property
|
||||
def client(self):
|
||||
"""Get or create Dropbox client."""
|
||||
if self._client is None:
|
||||
# Import here to avoid import errors if dropbox is not installed
|
||||
try:
|
||||
from dropbox import Dropbox
|
||||
except ImportError:
|
||||
raise ImportError(
|
||||
"dropbox package is not installed. "
|
||||
"Install it with: pip install dropbox"
|
||||
)
|
||||
|
||||
creds = self._load_credentials()
|
||||
if not creds:
|
||||
raise RuntimeError("Dropbox credentials not configured in Secret Manager")
|
||||
|
||||
self._client = Dropbox(
|
||||
oauth2_access_token=creds["access_token"],
|
||||
oauth2_refresh_token=creds.get("refresh_token"),
|
||||
app_key=creds.get("app_key"),
|
||||
app_secret=creds.get("app_secret"),
|
||||
)
|
||||
return self._client
|
||||
|
||||
def list_folders(self, path: str) -> list[str]:
|
||||
"""
|
||||
List folder names at path for brand code calculation.
|
||||
|
||||
Args:
|
||||
path: Dropbox path to list (e.g., "/Karaoke/Tracks-Organized")
|
||||
|
||||
Returns:
|
||||
List of folder names in the path
|
||||
"""
|
||||
from dropbox.files import FolderMetadata
|
||||
|
||||
# Ensure path starts with /
|
||||
if not path.startswith("/"):
|
||||
path = f"/{path}"
|
||||
|
||||
logger.info(f"Listing folders at Dropbox path: {path}")
|
||||
|
||||
result = self.client.files_list_folder(path)
|
||||
folders = []
|
||||
|
||||
# Get all entries (handling pagination)
|
||||
while True:
|
||||
for entry in result.entries:
|
||||
if isinstance(entry, FolderMetadata):
|
||||
folders.append(entry.name)
|
||||
|
||||
if not result.has_more:
|
||||
break
|
||||
result = self.client.files_list_folder_continue(result.cursor)
|
||||
|
||||
logger.info(f"Found {len(folders)} folders in {path}")
|
||||
return folders
|
||||
|
||||
def get_next_brand_code(self, path: str, brand_prefix: str) -> str:
|
||||
"""
|
||||
Calculate next brand code from existing folders.
|
||||
|
||||
Scans the folder for existing brand codes like "NOMAD-0001" and
|
||||
returns the next sequential code.
|
||||
|
||||
Args:
|
||||
path: Dropbox path containing organized folders
|
||||
brand_prefix: Brand prefix (e.g., "NOMAD")
|
||||
|
||||
Returns:
|
||||
Next brand code (e.g., "NOMAD-1164")
|
||||
"""
|
||||
folders = self.list_folders(path)
|
||||
pattern = re.compile(rf"^{re.escape(brand_prefix)}-(\d{{4}})")
|
||||
|
||||
max_num = 0
|
||||
for folder in folders:
|
||||
match = pattern.match(folder)
|
||||
if match:
|
||||
num = int(match.group(1))
|
||||
max_num = max(max_num, num)
|
||||
|
||||
next_code = f"{brand_prefix}-{max_num + 1:04d}"
|
||||
logger.info(f"Next brand code for {brand_prefix}: {next_code} (max existing: {max_num})")
|
||||
return next_code
|
||||
|
||||
def upload_file(self, local_path: str, remote_path: str) -> None:
|
||||
"""
|
||||
Upload a single file to Dropbox.
|
||||
|
||||
Args:
|
||||
local_path: Local file path
|
||||
remote_path: Dropbox destination path (must start with /)
|
||||
"""
|
||||
from dropbox.files import WriteMode
|
||||
|
||||
# Ensure remote path starts with /
|
||||
if not remote_path.startswith("/"):
|
||||
remote_path = f"/{remote_path}"
|
||||
|
||||
file_size = os.path.getsize(local_path)
|
||||
logger.info(f"Uploading {local_path} ({file_size / 1024 / 1024:.1f} MB) to {remote_path}")
|
||||
|
||||
# For large files (>150MB), use upload sessions
|
||||
CHUNK_SIZE = 150 * 1024 * 1024 # 150 MB
|
||||
|
||||
with open(local_path, "rb") as f:
|
||||
if file_size <= CHUNK_SIZE:
|
||||
# Simple upload for smaller files
|
||||
self.client.files_upload(f.read(), remote_path, mode=WriteMode.overwrite)
|
||||
else:
|
||||
# Chunked upload for large files
|
||||
self._upload_large_file(f, remote_path, file_size)
|
||||
|
||||
logger.info(f"Successfully uploaded to {remote_path}")
|
||||
|
||||
def _upload_large_file(self, file_obj, remote_path: str, file_size: int) -> None:
|
||||
"""Upload a large file using upload sessions."""
|
||||
from dropbox.files import CommitInfo, UploadSessionCursor, WriteMode
|
||||
|
||||
CHUNK_SIZE = 150 * 1024 * 1024 # 150 MB
|
||||
|
||||
# Start upload session
|
||||
session_start = self.client.files_upload_session_start(file_obj.read(CHUNK_SIZE))
|
||||
cursor = UploadSessionCursor(
|
||||
session_id=session_start.session_id,
|
||||
offset=file_obj.tell(),
|
||||
)
|
||||
commit = CommitInfo(path=remote_path, mode=WriteMode.overwrite)
|
||||
|
||||
# Upload remaining chunks
|
||||
while file_obj.tell() < file_size:
|
||||
remaining = file_size - file_obj.tell()
|
||||
if remaining <= CHUNK_SIZE:
|
||||
# Final chunk
|
||||
self.client.files_upload_session_finish(
|
||||
file_obj.read(CHUNK_SIZE),
|
||||
cursor,
|
||||
commit,
|
||||
)
|
||||
else:
|
||||
# Intermediate chunk
|
||||
self.client.files_upload_session_append_v2(
|
||||
file_obj.read(CHUNK_SIZE),
|
||||
cursor,
|
||||
)
|
||||
cursor.offset = file_obj.tell()
|
||||
|
||||
def upload_folder(self, local_dir: str, remote_path: str) -> None:
|
||||
"""
|
||||
Upload all files in a directory to Dropbox folder.
|
||||
|
||||
Args:
|
||||
local_dir: Local directory to upload
|
||||
remote_path: Dropbox destination folder path
|
||||
"""
|
||||
# Ensure remote path starts with /
|
||||
if not remote_path.startswith("/"):
|
||||
remote_path = f"/{remote_path}"
|
||||
|
||||
logger.info(f"Uploading folder {local_dir} to {remote_path}")
|
||||
|
||||
uploaded_count = 0
|
||||
for filename in os.listdir(local_dir):
|
||||
local_file = os.path.join(local_dir, filename)
|
||||
if os.path.isfile(local_file):
|
||||
remote_file = f"{remote_path}/{filename}"
|
||||
self.upload_file(local_file, remote_file)
|
||||
uploaded_count += 1
|
||||
|
||||
logger.info(f"Uploaded {uploaded_count} files to {remote_path}")
|
||||
|
||||
def create_shared_link(self, path: str) -> str:
|
||||
"""
|
||||
Create and return a shared link for the path.
|
||||
|
||||
Args:
|
||||
path: Dropbox path to share
|
||||
|
||||
Returns:
|
||||
Shared link URL
|
||||
"""
|
||||
from dropbox.exceptions import ApiError
|
||||
from dropbox.sharing import SharedLinkSettings
|
||||
|
||||
# Ensure path starts with /
|
||||
if not path.startswith("/"):
|
||||
path = f"/{path}"
|
||||
|
||||
logger.info(f"Creating shared link for: {path}")
|
||||
|
||||
try:
|
||||
# Try to create a new shared link
|
||||
settings = SharedLinkSettings(requested_visibility=None)
|
||||
link = self.client.sharing_create_shared_link_with_settings(path, settings)
|
||||
logger.info(f"Created new shared link: {link.url}")
|
||||
return link.url
|
||||
except ApiError as e:
|
||||
# Link may already exist - try to get existing link
|
||||
if e.error.is_shared_link_already_exists():
|
||||
logger.info("Shared link already exists, retrieving existing link")
|
||||
links = self.client.sharing_list_shared_links(path=path, direct_only=True)
|
||||
if links.links:
|
||||
logger.info(f"Found existing shared link: {links.links[0].url}")
|
||||
return links.links[0].url
|
||||
raise
|
||||
|
||||
|
||||
def get_dropbox_service() -> DropboxService:
|
||||
"""Factory function to get a DropboxService instance."""
|
||||
return DropboxService()
|
||||
@@ -1,343 +0,0 @@
|
||||
"""
|
||||
Firestore database operations for job management.
|
||||
"""
|
||||
import logging
|
||||
from typing import Optional, Dict, Any, List
|
||||
from datetime import datetime
|
||||
from google.cloud import firestore
|
||||
from google.cloud.firestore_v1 import FieldFilter
|
||||
|
||||
from backend.config import settings
|
||||
from backend.models.job import Job, JobStatus, TimelineEvent
|
||||
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
|
||||
class FirestoreService:
|
||||
"""Service for Firestore database operations."""
|
||||
|
||||
def __init__(self):
|
||||
"""Initialize Firestore client."""
|
||||
self.db = firestore.Client(project=settings.google_cloud_project)
|
||||
self.collection = settings.firestore_collection
|
||||
self.tokens_collection = "auth_tokens" # Collection for access tokens
|
||||
|
||||
def create_job(self, job: Job) -> None:
|
||||
"""Create a new job in Firestore."""
|
||||
try:
|
||||
doc_ref = self.db.collection(self.collection).document(job.job_id)
|
||||
doc_ref.set(job.model_dump(mode='json'))
|
||||
logger.info(f"Created job {job.job_id} in Firestore")
|
||||
except Exception as e:
|
||||
logger.error(f"Error creating job {job.job_id}: {e}")
|
||||
raise
|
||||
|
||||
def get_job(self, job_id: str) -> Optional[Job]:
|
||||
"""Get a job by ID."""
|
||||
try:
|
||||
doc_ref = self.db.collection(self.collection).document(job_id)
|
||||
doc = doc_ref.get()
|
||||
|
||||
logger.debug(f"Fetching job {job_id} from collection {self.collection}")
|
||||
logger.debug(f"Document exists: {doc.exists}")
|
||||
|
||||
if not doc.exists:
|
||||
return None
|
||||
|
||||
data = doc.to_dict()
|
||||
logger.debug(f"Document data keys: {list(data.keys()) if data else 'None'}")
|
||||
logger.debug(f"Document data: {data}")
|
||||
|
||||
return Job(**data)
|
||||
except Exception as e:
|
||||
logger.error(f"Error getting job {job_id}: {e}")
|
||||
raise
|
||||
|
||||
def update_job(self, job_id: str, updates: Dict[str, Any]) -> None:
|
||||
"""Update a job with partial data."""
|
||||
try:
|
||||
doc_ref = self.db.collection(self.collection).document(job_id)
|
||||
|
||||
# Add updated_at timestamp
|
||||
updates['updated_at'] = datetime.utcnow()
|
||||
|
||||
doc_ref.update(updates)
|
||||
logger.info(f"Updated job {job_id} in Firestore")
|
||||
except Exception as e:
|
||||
logger.error(f"Error updating job {job_id}: {e}")
|
||||
raise
|
||||
|
||||
def update_job_status(
|
||||
self,
|
||||
job_id: str,
|
||||
status: JobStatus,
|
||||
progress: Optional[int] = None,
|
||||
message: Optional[str] = None,
|
||||
**additional_fields
|
||||
) -> None:
|
||||
"""Update job status and add timeline event."""
|
||||
try:
|
||||
doc_ref = self.db.collection(self.collection).document(job_id)
|
||||
|
||||
# Create timeline event
|
||||
timeline_event = TimelineEvent(
|
||||
status=status.value,
|
||||
timestamp=datetime.utcnow().isoformat(),
|
||||
progress=progress,
|
||||
message=message
|
||||
)
|
||||
|
||||
# Prepare updates
|
||||
updates = {
|
||||
'status': status.value,
|
||||
'updated_at': datetime.utcnow(),
|
||||
'timeline': firestore.ArrayUnion([timeline_event.model_dump(mode='json')])
|
||||
}
|
||||
|
||||
if progress is not None:
|
||||
updates['progress'] = progress
|
||||
|
||||
# Add any additional fields
|
||||
updates.update(additional_fields)
|
||||
|
||||
doc_ref.update(updates)
|
||||
logger.info(f"Updated job {job_id} status to {status.value}")
|
||||
except Exception as e:
|
||||
logger.error(f"Error updating job status {job_id}: {e}")
|
||||
raise
|
||||
|
||||
def list_jobs(
|
||||
self,
|
||||
status: Optional[JobStatus] = None,
|
||||
environment: Optional[str] = None,
|
||||
client_id: Optional[str] = None,
|
||||
created_after: Optional[datetime] = None,
|
||||
created_before: Optional[datetime] = None,
|
||||
limit: int = 100
|
||||
) -> List[Job]:
|
||||
"""
|
||||
List jobs with optional filters.
|
||||
|
||||
Args:
|
||||
status: Filter by job status
|
||||
environment: Filter by request_metadata.environment (test/production/development)
|
||||
client_id: Filter by request_metadata.client_id
|
||||
created_after: Filter jobs created after this datetime
|
||||
created_before: Filter jobs created before this datetime
|
||||
limit: Maximum number of jobs to return
|
||||
|
||||
Returns:
|
||||
List of Job objects matching filters, ordered by created_at descending
|
||||
"""
|
||||
try:
|
||||
query = self.db.collection(self.collection)
|
||||
|
||||
if status:
|
||||
query = query.where(filter=FieldFilter('status', '==', status.value))
|
||||
|
||||
# Filter by request_metadata fields using dot notation
|
||||
if environment:
|
||||
query = query.where(filter=FieldFilter('request_metadata.environment', '==', environment))
|
||||
|
||||
if client_id:
|
||||
query = query.where(filter=FieldFilter('request_metadata.client_id', '==', client_id))
|
||||
|
||||
# Date range filters
|
||||
if created_after:
|
||||
query = query.where(filter=FieldFilter('created_at', '>=', created_after))
|
||||
|
||||
if created_before:
|
||||
query = query.where(filter=FieldFilter('created_at', '<=', created_before))
|
||||
|
||||
query = query.order_by('created_at', direction=firestore.Query.DESCENDING).limit(limit)
|
||||
|
||||
docs = query.stream()
|
||||
jobs = [Job(**doc.to_dict()) for doc in docs]
|
||||
|
||||
return jobs
|
||||
except Exception as e:
|
||||
logger.error(f"Error listing jobs: {e}")
|
||||
raise
|
||||
|
||||
def delete_jobs_by_filter(
|
||||
self,
|
||||
environment: Optional[str] = None,
|
||||
client_id: Optional[str] = None,
|
||||
status: Optional[JobStatus] = None,
|
||||
created_before: Optional[datetime] = None,
|
||||
) -> int:
|
||||
"""
|
||||
Delete multiple jobs matching filter criteria.
|
||||
|
||||
CAUTION: This is a destructive operation. Use carefully.
|
||||
|
||||
Args:
|
||||
environment: Delete jobs with this environment (e.g., "test")
|
||||
client_id: Delete jobs from this client
|
||||
status: Delete jobs with this status
|
||||
created_before: Delete jobs created before this datetime
|
||||
|
||||
Returns:
|
||||
Number of jobs deleted
|
||||
"""
|
||||
try:
|
||||
query = self.db.collection(self.collection)
|
||||
|
||||
if environment:
|
||||
query = query.where(filter=FieldFilter('request_metadata.environment', '==', environment))
|
||||
|
||||
if client_id:
|
||||
query = query.where(filter=FieldFilter('request_metadata.client_id', '==', client_id))
|
||||
|
||||
if status:
|
||||
query = query.where(filter=FieldFilter('status', '==', status.value))
|
||||
|
||||
if created_before:
|
||||
query = query.where(filter=FieldFilter('created_at', '<=', created_before))
|
||||
|
||||
# Get matching documents
|
||||
docs = list(query.stream())
|
||||
deleted_count = 0
|
||||
|
||||
# Delete in batches
|
||||
batch = self.db.batch()
|
||||
batch_count = 0
|
||||
|
||||
for doc in docs:
|
||||
batch.delete(doc.reference)
|
||||
batch_count += 1
|
||||
deleted_count += 1
|
||||
|
||||
# Firestore batch limit is 500
|
||||
if batch_count >= 500:
|
||||
batch.commit()
|
||||
batch = self.db.batch()
|
||||
batch_count = 0
|
||||
|
||||
# Commit any remaining deletes
|
||||
if batch_count > 0:
|
||||
batch.commit()
|
||||
|
||||
logger.info(f"Deleted {deleted_count} jobs matching filter criteria")
|
||||
return deleted_count
|
||||
|
||||
except Exception as e:
|
||||
logger.error(f"Error deleting jobs by filter: {e}")
|
||||
raise
|
||||
|
||||
def delete_job(self, job_id: str) -> None:
|
||||
"""Delete a job from Firestore."""
|
||||
try:
|
||||
doc_ref = self.db.collection(self.collection).document(job_id)
|
||||
doc_ref.delete()
|
||||
logger.info(f"Deleted job {job_id} from Firestore")
|
||||
except Exception as e:
|
||||
logger.error(f"Error deleting job {job_id}: {e}")
|
||||
raise
|
||||
|
||||
def append_worker_log(self, job_id: str, log_entry: Dict[str, Any]) -> None:
|
||||
"""
|
||||
Atomically append a log entry to worker_logs using ArrayUnion.
|
||||
|
||||
This avoids the race condition of read-modify-write when multiple
|
||||
workers are logging concurrently.
|
||||
|
||||
Args:
|
||||
job_id: Job ID
|
||||
log_entry: Log entry dict with timestamp, level, worker, message
|
||||
"""
|
||||
try:
|
||||
doc_ref = self.db.collection(self.collection).document(job_id)
|
||||
doc_ref.update({
|
||||
'worker_logs': firestore.ArrayUnion([log_entry]),
|
||||
'updated_at': datetime.utcnow()
|
||||
})
|
||||
# Don't log every append - too spammy
|
||||
except Exception as e:
|
||||
# Log but don't raise - logging shouldn't break workers
|
||||
logger.debug(f"Error appending worker log for job {job_id}: {e}")
|
||||
|
||||
# ============================================
|
||||
# Token Management Methods
|
||||
# ============================================
|
||||
|
||||
def create_token(self, token: str, token_data: Dict[str, Any]) -> None:
|
||||
"""Create a new access token in Firestore."""
|
||||
try:
|
||||
doc_ref = self.db.collection(self.tokens_collection).document(token)
|
||||
doc_ref.set(token_data)
|
||||
logger.info(f"Created token in Firestore")
|
||||
except Exception as e:
|
||||
logger.error(f"Error creating token: {e}")
|
||||
raise
|
||||
|
||||
def get_token(self, token: str) -> Optional[Dict[str, Any]]:
|
||||
"""Get token data by token string."""
|
||||
try:
|
||||
doc_ref = self.db.collection(self.tokens_collection).document(token)
|
||||
doc = doc_ref.get()
|
||||
|
||||
if not doc.exists:
|
||||
return None
|
||||
|
||||
return doc.to_dict()
|
||||
except Exception as e:
|
||||
logger.error(f"Error getting token: {e}")
|
||||
return None
|
||||
|
||||
def update_token(self, token: str, updates: Dict[str, Any]) -> None:
|
||||
"""Update token data."""
|
||||
try:
|
||||
doc_ref = self.db.collection(self.tokens_collection).document(token)
|
||||
doc_ref.update(updates)
|
||||
logger.info(f"Updated token in Firestore")
|
||||
except Exception as e:
|
||||
logger.error(f"Error updating token: {e}")
|
||||
raise
|
||||
|
||||
def increment_token_usage(self, token: str, job_id: str) -> None:
|
||||
"""Increment token usage count and add job to history."""
|
||||
try:
|
||||
doc_ref = self.db.collection(self.tokens_collection).document(token)
|
||||
|
||||
# Use Firestore transaction to ensure atomic increment
|
||||
@firestore.transactional
|
||||
def update_in_transaction(transaction, doc_ref):
|
||||
snapshot = doc_ref.get(transaction=transaction)
|
||||
if not snapshot.exists:
|
||||
raise ValueError("Token not found")
|
||||
|
||||
data = snapshot.to_dict()
|
||||
current_usage = data.get("usage_count", 0)
|
||||
jobs = data.get("jobs", [])
|
||||
|
||||
# Increment usage
|
||||
transaction.update(doc_ref, {
|
||||
"usage_count": current_usage + 1,
|
||||
"last_used": datetime.utcnow(),
|
||||
"jobs": firestore.ArrayUnion([{
|
||||
"job_id": job_id,
|
||||
"created_at": datetime.utcnow()
|
||||
}])
|
||||
})
|
||||
|
||||
transaction = self.db.transaction()
|
||||
update_in_transaction(transaction, doc_ref)
|
||||
|
||||
logger.info(f"Incremented token usage for job {job_id}")
|
||||
except Exception as e:
|
||||
logger.error(f"Error incrementing token usage: {e}")
|
||||
raise
|
||||
|
||||
def list_tokens(self) -> List[Dict[str, Any]]:
|
||||
"""List all tokens (admin only)."""
|
||||
try:
|
||||
docs = self.db.collection(self.tokens_collection).stream()
|
||||
tokens = [doc.to_dict() for doc in docs]
|
||||
logger.info(f"Retrieved {len(tokens)} tokens from Firestore")
|
||||
return tokens
|
||||
except Exception as e:
|
||||
logger.error(f"Error listing tokens: {e}")
|
||||
return []
|
||||
|
||||
@@ -1,473 +0,0 @@
|
||||
"""
|
||||
Client for the remote flacfetch HTTP API service.
|
||||
|
||||
This client communicates with a dedicated flacfetch VM that handles:
|
||||
- BitTorrent downloads from private trackers (RED, OPS)
|
||||
- YouTube downloads
|
||||
- GCS uploads of downloaded files
|
||||
|
||||
The flacfetch service provides:
|
||||
- Full peer connectivity for torrents (not possible in Cloud Run)
|
||||
- Indefinite seeding of completed torrents
|
||||
- Automatic disk cleanup
|
||||
|
||||
Usage:
|
||||
client = get_flacfetch_client()
|
||||
if client:
|
||||
# Search for audio
|
||||
search_result = await client.search("ABBA", "Waterloo")
|
||||
|
||||
# Download the best result
|
||||
download_id = await client.download(
|
||||
search_id=search_result["search_id"],
|
||||
result_index=0,
|
||||
gcs_path="uploads/job123/audio/",
|
||||
)
|
||||
|
||||
# Wait for completion
|
||||
result = await client.wait_for_download(download_id)
|
||||
print(f"Downloaded to: {result['gcs_path']}")
|
||||
"""
|
||||
import asyncio
|
||||
import logging
|
||||
from typing import Any, Dict, List, Optional
|
||||
|
||||
import httpx
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
|
||||
class FlacfetchServiceError(Exception):
|
||||
"""Error communicating with or returned from flacfetch service."""
|
||||
pass
|
||||
|
||||
|
||||
class FlacfetchClient:
|
||||
"""
|
||||
Client for remote flacfetch HTTP API.
|
||||
|
||||
All endpoints require authentication via X-API-Key header.
|
||||
"""
|
||||
|
||||
def __init__(
|
||||
self,
|
||||
base_url: str,
|
||||
api_key: str,
|
||||
timeout: int = 60,
|
||||
):
|
||||
"""
|
||||
Initialize flacfetch client.
|
||||
|
||||
Args:
|
||||
base_url: Base URL of flacfetch service (e.g., http://10.0.0.5:8080)
|
||||
api_key: API key for authentication
|
||||
timeout: Default request timeout in seconds
|
||||
"""
|
||||
self.base_url = base_url.rstrip('/')
|
||||
self.api_key = api_key
|
||||
self.timeout = timeout
|
||||
logger.info(f"FlacfetchClient initialized with base_url={self.base_url}")
|
||||
|
||||
def _headers(self) -> Dict[str, str]:
|
||||
"""Get request headers with authentication."""
|
||||
return {
|
||||
"X-API-Key": self.api_key,
|
||||
"Content-Type": "application/json",
|
||||
}
|
||||
|
||||
async def health_check(self) -> Dict[str, Any]:
|
||||
"""
|
||||
Check if flacfetch service is healthy.
|
||||
|
||||
Returns:
|
||||
Health status dict with transmission, disk, and provider info
|
||||
|
||||
Raises:
|
||||
FlacfetchServiceError: If service is unhealthy or unreachable
|
||||
"""
|
||||
try:
|
||||
async with httpx.AsyncClient() as client:
|
||||
resp = await client.get(
|
||||
f"{self.base_url}/health",
|
||||
headers=self._headers(),
|
||||
timeout=10,
|
||||
)
|
||||
resp.raise_for_status()
|
||||
data = resp.json()
|
||||
|
||||
if data.get("status") not in ["healthy", "degraded"]:
|
||||
raise FlacfetchServiceError(f"Service unhealthy: {data}")
|
||||
|
||||
return data
|
||||
except httpx.RequestError as e:
|
||||
raise FlacfetchServiceError(f"Cannot reach flacfetch service: {e}")
|
||||
except httpx.HTTPStatusError as e:
|
||||
raise FlacfetchServiceError(f"Health check failed: {e.response.status_code}")
|
||||
|
||||
async def search(
|
||||
self,
|
||||
artist: str,
|
||||
title: str,
|
||||
) -> Dict[str, Any]:
|
||||
"""
|
||||
Search for audio matching artist and title.
|
||||
|
||||
Args:
|
||||
artist: Artist name
|
||||
title: Track title
|
||||
|
||||
Returns:
|
||||
Search response dict with search_id and results list
|
||||
|
||||
Raises:
|
||||
FlacfetchServiceError: On search failure
|
||||
"""
|
||||
try:
|
||||
async with httpx.AsyncClient() as client:
|
||||
resp = await client.post(
|
||||
f"{self.base_url}/search",
|
||||
headers=self._headers(),
|
||||
json={"artist": artist, "title": title},
|
||||
timeout=self.timeout,
|
||||
)
|
||||
|
||||
if resp.status_code == 404:
|
||||
# No results found - return empty results
|
||||
return {
|
||||
"search_id": None,
|
||||
"artist": artist,
|
||||
"title": title,
|
||||
"results": [],
|
||||
"results_count": 0,
|
||||
}
|
||||
|
||||
resp.raise_for_status()
|
||||
return resp.json()
|
||||
|
||||
except httpx.RequestError as e:
|
||||
raise FlacfetchServiceError(f"Search request failed: {e}")
|
||||
except httpx.HTTPStatusError as e:
|
||||
raise FlacfetchServiceError(f"Search failed: {e.response.status_code} - {e.response.text}")
|
||||
|
||||
async def download(
|
||||
self,
|
||||
search_id: str,
|
||||
result_index: int,
|
||||
output_filename: Optional[str] = None,
|
||||
gcs_path: Optional[str] = None,
|
||||
) -> str:
|
||||
"""
|
||||
Start downloading an audio file.
|
||||
|
||||
Args:
|
||||
search_id: Search ID from previous search
|
||||
result_index: Index of result to download
|
||||
output_filename: Optional custom filename (without extension)
|
||||
gcs_path: GCS path for upload (e.g., "uploads/job123/audio/")
|
||||
|
||||
Returns:
|
||||
Download ID for tracking progress
|
||||
|
||||
Raises:
|
||||
FlacfetchServiceError: On download start failure
|
||||
"""
|
||||
try:
|
||||
async with httpx.AsyncClient() as client:
|
||||
payload = {
|
||||
"search_id": search_id,
|
||||
"result_index": result_index,
|
||||
}
|
||||
if output_filename:
|
||||
payload["output_filename"] = output_filename
|
||||
if gcs_path:
|
||||
payload["upload_to_gcs"] = True
|
||||
payload["gcs_path"] = gcs_path
|
||||
|
||||
resp = await client.post(
|
||||
f"{self.base_url}/download",
|
||||
headers=self._headers(),
|
||||
json=payload,
|
||||
timeout=self.timeout,
|
||||
)
|
||||
resp.raise_for_status()
|
||||
|
||||
data = resp.json()
|
||||
return data["download_id"]
|
||||
|
||||
except httpx.RequestError as e:
|
||||
raise FlacfetchServiceError(f"Download request failed: {e}")
|
||||
except httpx.HTTPStatusError as e:
|
||||
raise FlacfetchServiceError(f"Download start failed: {e.response.status_code} - {e.response.text}")
|
||||
|
||||
async def download_by_id(
|
||||
self,
|
||||
source_name: str,
|
||||
source_id: str,
|
||||
output_filename: Optional[str] = None,
|
||||
target_file: Optional[str] = None,
|
||||
download_url: Optional[str] = None,
|
||||
gcs_path: Optional[str] = None,
|
||||
) -> str:
|
||||
"""
|
||||
Start downloading directly by source ID (no prior search required).
|
||||
|
||||
This is useful when you have stored the source_id from a previous search
|
||||
and want to download later without re-searching.
|
||||
|
||||
Args:
|
||||
source_name: Provider name (RED, OPS, YouTube, Spotify)
|
||||
source_id: Source-specific ID (torrent ID, video ID, track ID)
|
||||
output_filename: Optional custom filename (without extension)
|
||||
target_file: For torrents, specific file to extract
|
||||
download_url: For YouTube/Spotify, direct URL (optional)
|
||||
gcs_path: GCS path for upload (e.g., "uploads/job123/audio/")
|
||||
|
||||
Returns:
|
||||
Download ID for tracking progress
|
||||
|
||||
Raises:
|
||||
FlacfetchServiceError: On download start failure
|
||||
"""
|
||||
try:
|
||||
async with httpx.AsyncClient() as client:
|
||||
payload = {
|
||||
"source_name": source_name,
|
||||
"source_id": source_id,
|
||||
}
|
||||
if output_filename:
|
||||
payload["output_filename"] = output_filename
|
||||
if target_file:
|
||||
payload["target_file"] = target_file
|
||||
if download_url:
|
||||
payload["download_url"] = download_url
|
||||
if gcs_path:
|
||||
payload["upload_to_gcs"] = True
|
||||
payload["gcs_path"] = gcs_path
|
||||
|
||||
resp = await client.post(
|
||||
f"{self.base_url}/download-by-id",
|
||||
headers=self._headers(),
|
||||
json=payload,
|
||||
timeout=self.timeout,
|
||||
)
|
||||
resp.raise_for_status()
|
||||
|
||||
data = resp.json()
|
||||
return data["download_id"]
|
||||
|
||||
except httpx.RequestError as e:
|
||||
raise FlacfetchServiceError(f"Download by ID request failed: {e}")
|
||||
except httpx.HTTPStatusError as e:
|
||||
raise FlacfetchServiceError(f"Download by ID failed: {e.response.status_code} - {e.response.text}")
|
||||
|
||||
async def get_download_status(self, download_id: str) -> Dict[str, Any]:
|
||||
"""
|
||||
Get the current status of a download.
|
||||
|
||||
Args:
|
||||
download_id: Download ID from start_download
|
||||
|
||||
Returns:
|
||||
Status dict with progress, speed, path info
|
||||
|
||||
Raises:
|
||||
FlacfetchServiceError: On status check failure
|
||||
"""
|
||||
try:
|
||||
async with httpx.AsyncClient() as client:
|
||||
resp = await client.get(
|
||||
f"{self.base_url}/download/{download_id}/status",
|
||||
headers=self._headers(),
|
||||
timeout=10,
|
||||
)
|
||||
resp.raise_for_status()
|
||||
return resp.json()
|
||||
|
||||
except httpx.RequestError as e:
|
||||
raise FlacfetchServiceError(f"Status request failed: {e}")
|
||||
except httpx.HTTPStatusError as e:
|
||||
if e.response.status_code == 404:
|
||||
raise FlacfetchServiceError(f"Download not found: {download_id}")
|
||||
raise FlacfetchServiceError(f"Status check failed: {e.response.status_code}")
|
||||
|
||||
async def wait_for_download(
|
||||
self,
|
||||
download_id: str,
|
||||
timeout: int = 600,
|
||||
poll_interval: float = 2.0,
|
||||
progress_callback: Optional[callable] = None,
|
||||
) -> Dict[str, Any]:
|
||||
"""
|
||||
Wait for a download to complete.
|
||||
|
||||
Args:
|
||||
download_id: Download ID to wait for
|
||||
timeout: Maximum wait time in seconds
|
||||
poll_interval: Time between status checks
|
||||
progress_callback: Optional callback(status_dict) for progress updates
|
||||
|
||||
Returns:
|
||||
Final status dict with output_path or gcs_path
|
||||
|
||||
Raises:
|
||||
FlacfetchServiceError: On download failure or timeout
|
||||
"""
|
||||
elapsed = 0.0
|
||||
|
||||
while elapsed < timeout:
|
||||
status = await self.get_download_status(download_id)
|
||||
|
||||
if progress_callback:
|
||||
try:
|
||||
progress_callback(status)
|
||||
except Exception as e:
|
||||
logger.warning(f"Progress callback error: {e}")
|
||||
|
||||
download_status = status.get("status")
|
||||
|
||||
if download_status in ["complete", "seeding"]:
|
||||
logger.info(f"Download {download_id} complete: {status.get('gcs_path') or status.get('output_path')}")
|
||||
return status
|
||||
elif download_status == "failed":
|
||||
error = status.get("error", "Unknown error")
|
||||
raise FlacfetchServiceError(f"Download failed: {error}")
|
||||
elif download_status == "cancelled":
|
||||
raise FlacfetchServiceError("Download was cancelled")
|
||||
|
||||
# Log progress
|
||||
progress = status.get("progress", 0)
|
||||
speed = status.get("download_speed_kbps", 0)
|
||||
peers = status.get("peers", 0)
|
||||
logger.debug(
|
||||
f"Download {download_id}: {progress:.1f}% "
|
||||
f"({speed:.1f} KB/s, {peers} peers)"
|
||||
)
|
||||
|
||||
await asyncio.sleep(poll_interval)
|
||||
elapsed += poll_interval
|
||||
|
||||
raise FlacfetchServiceError(f"Download timed out after {timeout}s")
|
||||
|
||||
async def list_torrents(self) -> Dict[str, Any]:
|
||||
"""
|
||||
List all torrents in Transmission.
|
||||
|
||||
Returns:
|
||||
Torrent list response with torrent info and totals
|
||||
"""
|
||||
try:
|
||||
async with httpx.AsyncClient() as client:
|
||||
resp = await client.get(
|
||||
f"{self.base_url}/torrents",
|
||||
headers=self._headers(),
|
||||
timeout=30,
|
||||
)
|
||||
resp.raise_for_status()
|
||||
return resp.json()
|
||||
|
||||
except httpx.RequestError as e:
|
||||
raise FlacfetchServiceError(f"List torrents failed: {e}")
|
||||
except httpx.HTTPStatusError as e:
|
||||
raise FlacfetchServiceError(f"List torrents failed: {e.response.status_code}")
|
||||
|
||||
async def delete_torrent(
|
||||
self,
|
||||
torrent_id: int,
|
||||
delete_data: bool = True,
|
||||
) -> Dict[str, Any]:
|
||||
"""
|
||||
Delete a torrent from Transmission.
|
||||
|
||||
Args:
|
||||
torrent_id: Transmission torrent ID
|
||||
delete_data: Also delete downloaded files
|
||||
|
||||
Returns:
|
||||
Delete response
|
||||
"""
|
||||
try:
|
||||
async with httpx.AsyncClient() as client:
|
||||
resp = await client.delete(
|
||||
f"{self.base_url}/torrents/{torrent_id}",
|
||||
headers=self._headers(),
|
||||
params={"delete_data": str(delete_data).lower()},
|
||||
timeout=30,
|
||||
)
|
||||
resp.raise_for_status()
|
||||
return resp.json()
|
||||
|
||||
except httpx.RequestError as e:
|
||||
raise FlacfetchServiceError(f"Delete torrent failed: {e}")
|
||||
except httpx.HTTPStatusError as e:
|
||||
raise FlacfetchServiceError(f"Delete torrent failed: {e.response.status_code}")
|
||||
|
||||
async def cleanup_torrents(
|
||||
self,
|
||||
strategy: str = "oldest",
|
||||
target_free_gb: float = 10.0,
|
||||
) -> Dict[str, Any]:
|
||||
"""
|
||||
Trigger disk cleanup.
|
||||
|
||||
Args:
|
||||
strategy: Cleanup strategy (oldest, largest, lowest_ratio)
|
||||
target_free_gb: Target free space to achieve
|
||||
|
||||
Returns:
|
||||
Cleanup response with removed count and freed space
|
||||
"""
|
||||
try:
|
||||
async with httpx.AsyncClient() as client:
|
||||
resp = await client.post(
|
||||
f"{self.base_url}/torrents/cleanup",
|
||||
headers=self._headers(),
|
||||
json={
|
||||
"strategy": strategy,
|
||||
"target_free_gb": target_free_gb,
|
||||
},
|
||||
timeout=60,
|
||||
)
|
||||
resp.raise_for_status()
|
||||
return resp.json()
|
||||
|
||||
except httpx.RequestError as e:
|
||||
raise FlacfetchServiceError(f"Cleanup failed: {e}")
|
||||
except httpx.HTTPStatusError as e:
|
||||
raise FlacfetchServiceError(f"Cleanup failed: {e.response.status_code}")
|
||||
|
||||
|
||||
# Singleton client instance
|
||||
_client: Optional[FlacfetchClient] = None
|
||||
|
||||
|
||||
def get_flacfetch_client() -> Optional[FlacfetchClient]:
|
||||
"""
|
||||
Get the flacfetch client if configured.
|
||||
|
||||
Returns:
|
||||
FlacfetchClient instance if FLACFETCH_API_URL and FLACFETCH_API_KEY are set,
|
||||
None otherwise (indicating local-only mode).
|
||||
"""
|
||||
global _client
|
||||
|
||||
if _client is None:
|
||||
from backend.config import get_settings
|
||||
settings = get_settings()
|
||||
|
||||
if settings.flacfetch_api_url and settings.flacfetch_api_key:
|
||||
_client = FlacfetchClient(
|
||||
base_url=settings.flacfetch_api_url,
|
||||
api_key=settings.flacfetch_api_key,
|
||||
)
|
||||
logger.info("FlacfetchClient initialized for remote service")
|
||||
else:
|
||||
logger.debug("FlacfetchClient not configured (missing URL or API key)")
|
||||
|
||||
return _client
|
||||
|
||||
|
||||
def reset_flacfetch_client():
|
||||
"""Reset the singleton client (for testing)."""
|
||||
global _client
|
||||
_client = None
|
||||
|
||||
@@ -1,309 +0,0 @@
|
||||
"""Native Google Drive API service for cloud backend.
|
||||
|
||||
This service provides Google Drive operations using the native API,
|
||||
for uploading files to public share folders. It handles:
|
||||
- Folder creation and lookup
|
||||
- File uploads with resumable upload support
|
||||
- Uploading to organized folder structure (MP4/, MP4-720p/, CDG/)
|
||||
|
||||
Credentials are loaded from Google Cloud Secret Manager and can be
|
||||
shared with YouTube credentials if scopes include drive.file.
|
||||
"""
|
||||
import json
|
||||
import logging
|
||||
import os
|
||||
from typing import Any, Dict, Optional
|
||||
|
||||
from backend.config import get_settings
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
|
||||
class GoogleDriveService:
|
||||
"""Google Drive operations using native API."""
|
||||
|
||||
# Secret Manager secret name for Google Drive credentials
|
||||
# Can be same as YouTube if scopes include drive.file
|
||||
GDRIVE_CREDENTIALS_SECRET = "gdrive-oauth-credentials"
|
||||
|
||||
def __init__(self):
|
||||
self.settings = get_settings()
|
||||
self._service = None
|
||||
self._credentials_data: Optional[Dict[str, Any]] = None
|
||||
self._loaded = False
|
||||
|
||||
def _load_credentials(self) -> Optional[Dict[str, Any]]:
|
||||
"""Load OAuth credentials from Secret Manager."""
|
||||
if self._loaded:
|
||||
return self._credentials_data
|
||||
|
||||
try:
|
||||
creds_json = self.settings.get_secret(self.GDRIVE_CREDENTIALS_SECRET)
|
||||
|
||||
if not creds_json:
|
||||
# Try falling back to YouTube credentials (may have drive scope)
|
||||
logger.info(
|
||||
"Google Drive credentials not found, trying YouTube credentials"
|
||||
)
|
||||
creds_json = self.settings.get_secret("youtube-oauth-credentials")
|
||||
|
||||
if not creds_json:
|
||||
logger.warning("Google Drive credentials not found in Secret Manager")
|
||||
self._loaded = True
|
||||
return None
|
||||
|
||||
self._credentials_data = json.loads(creds_json)
|
||||
|
||||
# Validate required fields
|
||||
required_fields = ["refresh_token", "client_id", "client_secret"]
|
||||
missing = [f for f in required_fields if not self._credentials_data.get(f)]
|
||||
|
||||
if missing:
|
||||
logger.error(f"Google Drive credentials missing required fields: {missing}")
|
||||
self._credentials_data = None
|
||||
self._loaded = True
|
||||
return None
|
||||
|
||||
logger.info("Google Drive credentials loaded successfully from Secret Manager")
|
||||
self._loaded = True
|
||||
return self._credentials_data
|
||||
|
||||
except json.JSONDecodeError as e:
|
||||
logger.error(f"Failed to parse Google Drive credentials JSON: {e}")
|
||||
self._loaded = True
|
||||
return None
|
||||
except Exception as e:
|
||||
logger.error(f"Failed to load Google Drive credentials: {e}")
|
||||
self._loaded = True
|
||||
return None
|
||||
|
||||
@property
|
||||
def is_configured(self) -> bool:
|
||||
"""Check if Google Drive credentials are available."""
|
||||
creds = self._load_credentials()
|
||||
return creds is not None
|
||||
|
||||
@property
|
||||
def service(self):
|
||||
"""Get or create Google Drive service."""
|
||||
if self._service is None:
|
||||
# Import here to avoid import errors if google packages not installed
|
||||
try:
|
||||
from google.oauth2.credentials import Credentials
|
||||
from googleapiclient.discovery import build
|
||||
except ImportError:
|
||||
raise ImportError(
|
||||
"google-api-python-client and google-auth packages are required. "
|
||||
"Install them with: pip install google-api-python-client google-auth"
|
||||
)
|
||||
|
||||
creds_data = self._load_credentials()
|
||||
if not creds_data:
|
||||
raise RuntimeError(
|
||||
"Google Drive credentials not configured in Secret Manager"
|
||||
)
|
||||
|
||||
# Create credentials object
|
||||
credentials = Credentials(
|
||||
token=creds_data.get("token"),
|
||||
refresh_token=creds_data.get("refresh_token"),
|
||||
token_uri=creds_data.get(
|
||||
"token_uri", "https://oauth2.googleapis.com/token"
|
||||
),
|
||||
client_id=creds_data.get("client_id"),
|
||||
client_secret=creds_data.get("client_secret"),
|
||||
scopes=creds_data.get(
|
||||
"scopes", ["https://www.googleapis.com/auth/drive.file"]
|
||||
),
|
||||
)
|
||||
|
||||
self._service = build("drive", "v3", credentials=credentials)
|
||||
logger.info("Google Drive service initialized successfully")
|
||||
|
||||
return self._service
|
||||
|
||||
def get_or_create_folder(self, parent_id: str, folder_name: str) -> str:
|
||||
"""
|
||||
Get existing folder or create new one, return folder ID.
|
||||
|
||||
Args:
|
||||
parent_id: Parent folder ID
|
||||
folder_name: Name of folder to find or create
|
||||
|
||||
Returns:
|
||||
Folder ID
|
||||
"""
|
||||
logger.info(f"Looking for folder '{folder_name}' in parent {parent_id}")
|
||||
|
||||
# Search for existing folder
|
||||
query = (
|
||||
f"name='{folder_name}' and '{parent_id}' in parents "
|
||||
f"and mimeType='application/vnd.google-apps.folder' and trashed=false"
|
||||
)
|
||||
results = self.service.files().list(q=query, fields="files(id, name)").execute()
|
||||
|
||||
if results.get("files"):
|
||||
folder_id = results["files"][0]["id"]
|
||||
logger.info(f"Found existing folder '{folder_name}': {folder_id}")
|
||||
return folder_id
|
||||
|
||||
# Create folder
|
||||
logger.info(f"Creating new folder '{folder_name}'")
|
||||
metadata = {
|
||||
"name": folder_name,
|
||||
"mimeType": "application/vnd.google-apps.folder",
|
||||
"parents": [parent_id],
|
||||
}
|
||||
folder = self.service.files().create(body=metadata, fields="id").execute()
|
||||
folder_id = folder["id"]
|
||||
logger.info(f"Created folder '{folder_name}': {folder_id}")
|
||||
return folder_id
|
||||
|
||||
def upload_file(
|
||||
self,
|
||||
local_path: str,
|
||||
parent_id: str,
|
||||
filename: str,
|
||||
replace_existing: bool = True,
|
||||
) -> str:
|
||||
"""
|
||||
Upload a file to a specific Drive folder.
|
||||
|
||||
Args:
|
||||
local_path: Local file path
|
||||
parent_id: Parent folder ID in Google Drive
|
||||
filename: Name for the file in Drive
|
||||
replace_existing: If True, delete existing file with same name first
|
||||
|
||||
Returns:
|
||||
File ID of uploaded file
|
||||
"""
|
||||
from googleapiclient.http import MediaFileUpload
|
||||
|
||||
file_size = os.path.getsize(local_path)
|
||||
logger.info(
|
||||
f"Uploading {local_path} ({file_size / 1024 / 1024:.1f} MB) "
|
||||
f"as '{filename}' to folder {parent_id}"
|
||||
)
|
||||
|
||||
# Determine MIME type
|
||||
ext = os.path.splitext(local_path)[1].lower()
|
||||
mime_types = {
|
||||
".mp4": "video/mp4",
|
||||
".mkv": "video/x-matroska",
|
||||
".zip": "application/zip",
|
||||
".flac": "audio/flac",
|
||||
".mp3": "audio/mpeg",
|
||||
".wav": "audio/wav",
|
||||
".png": "image/png",
|
||||
".jpg": "image/jpeg",
|
||||
".jpeg": "image/jpeg",
|
||||
}
|
||||
mime_type = mime_types.get(ext, "application/octet-stream")
|
||||
|
||||
# Check for existing file with same name
|
||||
if replace_existing:
|
||||
query = (
|
||||
f"name='{filename}' and '{parent_id}' in parents and trashed=false"
|
||||
)
|
||||
results = self.service.files().list(q=query, fields="files(id)").execute()
|
||||
for existing_file in results.get("files", []):
|
||||
logger.info(f"Deleting existing file: {existing_file['id']}")
|
||||
self.service.files().delete(fileId=existing_file["id"]).execute()
|
||||
|
||||
# Upload file
|
||||
metadata = {"name": filename, "parents": [parent_id]}
|
||||
media = MediaFileUpload(local_path, mimetype=mime_type, resumable=True)
|
||||
|
||||
file_result = (
|
||||
self.service.files().create(body=metadata, media_body=media, fields="id").execute()
|
||||
)
|
||||
file_id = file_result["id"]
|
||||
logger.info(f"Successfully uploaded '{filename}': {file_id}")
|
||||
return file_id
|
||||
|
||||
def upload_to_public_share(
|
||||
self,
|
||||
root_folder_id: str,
|
||||
brand_code: str,
|
||||
base_name: str,
|
||||
output_files: dict,
|
||||
) -> Dict[str, str]:
|
||||
"""
|
||||
Upload final files to public share folder structure.
|
||||
|
||||
Creates/uses subfolders:
|
||||
- MP4/{brand_code} - {base_name}.mp4 (lossy 4k)
|
||||
- MP4-720p/{brand_code} - {base_name}.mp4
|
||||
- CDG/{brand_code} - {base_name}.zip
|
||||
|
||||
Args:
|
||||
root_folder_id: Google Drive folder ID for public share root
|
||||
brand_code: Brand code (e.g., "NOMAD-1163")
|
||||
base_name: Base filename (e.g., "Artist - Title")
|
||||
output_files: Dictionary with output file paths:
|
||||
- final_karaoke_lossy_mp4: 4K lossy MP4
|
||||
- final_karaoke_lossy_720p_mp4: 720p lossy MP4
|
||||
- final_karaoke_cdg_zip: CDG package ZIP
|
||||
|
||||
Returns:
|
||||
Dictionary mapping category to uploaded file ID
|
||||
"""
|
||||
filename_base = f"{brand_code} - {base_name}"
|
||||
uploaded_files = {}
|
||||
|
||||
logger.info(
|
||||
f"Uploading public share files to Google Drive folder {root_folder_id}"
|
||||
)
|
||||
logger.info(f"Filename base: {filename_base}")
|
||||
|
||||
# Upload lossy 4k to MP4/
|
||||
lossy_mp4_path = output_files.get("final_karaoke_lossy_mp4")
|
||||
if lossy_mp4_path and os.path.exists(lossy_mp4_path):
|
||||
mp4_folder_id = self.get_or_create_folder(root_folder_id, "MP4")
|
||||
file_id = self.upload_file(
|
||||
lossy_mp4_path,
|
||||
mp4_folder_id,
|
||||
f"{filename_base}.mp4",
|
||||
)
|
||||
uploaded_files["mp4"] = file_id
|
||||
logger.info(f"Uploaded 4K MP4 to MP4/ folder")
|
||||
|
||||
# Upload 720p to MP4-720p/
|
||||
mp4_720p_path = output_files.get("final_karaoke_lossy_720p_mp4")
|
||||
if mp4_720p_path and os.path.exists(mp4_720p_path):
|
||||
mp4_720_folder_id = self.get_or_create_folder(root_folder_id, "MP4-720p")
|
||||
file_id = self.upload_file(
|
||||
mp4_720p_path,
|
||||
mp4_720_folder_id,
|
||||
f"{filename_base}.mp4",
|
||||
)
|
||||
uploaded_files["mp4_720p"] = file_id
|
||||
logger.info(f"Uploaded 720p MP4 to MP4-720p/ folder")
|
||||
|
||||
# Upload CDG ZIP to CDG/
|
||||
cdg_zip_path = output_files.get("final_karaoke_cdg_zip")
|
||||
if cdg_zip_path and os.path.exists(cdg_zip_path):
|
||||
cdg_folder_id = self.get_or_create_folder(root_folder_id, "CDG")
|
||||
file_id = self.upload_file(
|
||||
cdg_zip_path,
|
||||
cdg_folder_id,
|
||||
f"{filename_base}.zip",
|
||||
)
|
||||
uploaded_files["cdg"] = file_id
|
||||
logger.info(f"Uploaded CDG ZIP to CDG/ folder")
|
||||
|
||||
logger.info(f"Public share upload complete: {len(uploaded_files)} files uploaded")
|
||||
return uploaded_files
|
||||
|
||||
|
||||
# Singleton instance
|
||||
_gdrive_service: Optional[GoogleDriveService] = None
|
||||
|
||||
|
||||
def get_gdrive_service() -> GoogleDriveService:
|
||||
"""Get the singleton Google Drive service instance."""
|
||||
global _gdrive_service
|
||||
if _gdrive_service is None:
|
||||
_gdrive_service = GoogleDriveService()
|
||||
return _gdrive_service
|
||||
@@ -1,258 +0,0 @@
|
||||
"""
|
||||
Job-aware logging utilities.
|
||||
|
||||
This module provides a context manager and logging handler that captures
|
||||
Python logging output and forwards it to Firestore for real-time streaming
|
||||
to the CLI.
|
||||
|
||||
Usage:
|
||||
with JobLogContext(job_id, worker="review") as log_context:
|
||||
# Any logging within this block will be captured and sent to Firestore
|
||||
logger.info("Processing...") # This will be stored and streamed to CLI
|
||||
"""
|
||||
import logging
|
||||
import threading
|
||||
from typing import Optional
|
||||
from contextlib import contextmanager
|
||||
from datetime import datetime
|
||||
|
||||
from backend.services.firestore_service import FirestoreService
|
||||
|
||||
|
||||
class JobLogHandler(logging.Handler):
|
||||
"""
|
||||
A logging handler that forwards log records to Firestore for job tracking.
|
||||
|
||||
This enables real-time log streaming to the CLI during long-running operations
|
||||
like lyrics correction, preview generation, etc.
|
||||
"""
|
||||
|
||||
def __init__(
|
||||
self,
|
||||
job_id: str,
|
||||
worker: str = "review",
|
||||
firestore: Optional[FirestoreService] = None,
|
||||
min_level: int = logging.INFO,
|
||||
batch_size: int = 1, # Send immediately for real-time experience
|
||||
):
|
||||
"""
|
||||
Initialize the job log handler.
|
||||
|
||||
Args:
|
||||
job_id: The job ID to associate logs with
|
||||
worker: Worker name (e.g., "review", "add-lyrics", "preview")
|
||||
firestore: FirestoreService instance (creates one if not provided)
|
||||
min_level: Minimum log level to capture (default INFO)
|
||||
batch_size: Number of logs to batch before sending (1 = immediate)
|
||||
"""
|
||||
super().__init__()
|
||||
self.job_id = job_id
|
||||
self.worker = worker
|
||||
self.firestore = firestore or FirestoreService()
|
||||
self.min_level = min_level
|
||||
self.batch_size = batch_size
|
||||
self._log_buffer = []
|
||||
self._lock = threading.Lock()
|
||||
|
||||
# Set the handler's level
|
||||
self.setLevel(min_level)
|
||||
|
||||
# Create a formatter that extracts the useful parts
|
||||
self.setFormatter(logging.Formatter('%(message)s'))
|
||||
|
||||
def emit(self, record: logging.LogRecord) -> None:
|
||||
"""
|
||||
Process a log record and send it to Firestore.
|
||||
|
||||
This method is called by the logging framework for each log message.
|
||||
"""
|
||||
try:
|
||||
# Skip if below minimum level
|
||||
if record.levelno < self.min_level:
|
||||
return
|
||||
|
||||
# Format the message
|
||||
message = self.format(record)
|
||||
|
||||
# Truncate very long messages
|
||||
if len(message) > 1000:
|
||||
message = message[:997] + "..."
|
||||
|
||||
# Create log entry
|
||||
log_entry = {
|
||||
'timestamp': datetime.utcnow().isoformat() + 'Z',
|
||||
'level': record.levelname,
|
||||
'worker': self.worker,
|
||||
'message': message,
|
||||
}
|
||||
|
||||
# Add to buffer
|
||||
with self._lock:
|
||||
self._log_buffer.append(log_entry)
|
||||
|
||||
# Flush if batch size reached
|
||||
if len(self._log_buffer) >= self.batch_size:
|
||||
self._flush_buffer()
|
||||
|
||||
except Exception:
|
||||
# Never let logging errors propagate
|
||||
self.handleError(record)
|
||||
|
||||
def _flush_buffer(self) -> None:
|
||||
"""Flush buffered logs to Firestore."""
|
||||
if not self._log_buffer:
|
||||
return
|
||||
|
||||
try:
|
||||
for log_entry in self._log_buffer:
|
||||
self.firestore.append_worker_log(self.job_id, log_entry)
|
||||
self._log_buffer = []
|
||||
except Exception:
|
||||
# If Firestore fails, just clear buffer and continue
|
||||
self._log_buffer = []
|
||||
|
||||
def flush(self) -> None:
|
||||
"""Flush any remaining buffered logs."""
|
||||
with self._lock:
|
||||
self._flush_buffer()
|
||||
|
||||
def close(self) -> None:
|
||||
"""Clean up the handler."""
|
||||
self.flush()
|
||||
super().close()
|
||||
|
||||
|
||||
@contextmanager
|
||||
def job_log_context(
|
||||
job_id: str,
|
||||
worker: str = "review",
|
||||
logger_names: Optional[list] = None,
|
||||
min_level: int = logging.INFO,
|
||||
):
|
||||
"""
|
||||
Context manager that captures logs and forwards them to Firestore.
|
||||
|
||||
This allows the CLI to see real-time logs from operations like
|
||||
add-lyrics, preview generation, etc.
|
||||
|
||||
Args:
|
||||
job_id: Job ID to associate logs with
|
||||
worker: Worker name for log categorization
|
||||
logger_names: List of logger names to capture (None = capture common loggers)
|
||||
min_level: Minimum log level to capture
|
||||
|
||||
Usage:
|
||||
with job_log_context(job_id, worker="add-lyrics"):
|
||||
# All logging in this block will be captured
|
||||
correction_operations.add_lyrics_source(...)
|
||||
"""
|
||||
# Default loggers to capture - these cover most of our processing
|
||||
# Note: We intentionally DON'T include the root logger ('') to avoid
|
||||
# duplicate log entries since logs propagate up the logger hierarchy.
|
||||
# Only capture at the top-level package loggers.
|
||||
if logger_names is None:
|
||||
logger_names = [
|
||||
'backend.api.routes.review',
|
||||
'lyrics_transcriber',
|
||||
'karaoke_gen',
|
||||
]
|
||||
|
||||
# Create the handler
|
||||
handler = JobLogHandler(
|
||||
job_id=job_id,
|
||||
worker=worker,
|
||||
min_level=min_level,
|
||||
)
|
||||
|
||||
# Add handler to all specified loggers
|
||||
loggers = []
|
||||
for name in logger_names:
|
||||
log = logging.getLogger(name) if name else logging.getLogger()
|
||||
log.addHandler(handler)
|
||||
loggers.append(log)
|
||||
|
||||
try:
|
||||
# Log start
|
||||
handler.emit(logging.LogRecord(
|
||||
name='job_log_context',
|
||||
level=logging.INFO,
|
||||
pathname='',
|
||||
lineno=0,
|
||||
msg=f"🚀 Starting {worker} operation for job {job_id}",
|
||||
args=(),
|
||||
exc_info=None,
|
||||
))
|
||||
|
||||
yield handler
|
||||
|
||||
finally:
|
||||
# Log completion
|
||||
handler.emit(logging.LogRecord(
|
||||
name='job_log_context',
|
||||
level=logging.INFO,
|
||||
pathname='',
|
||||
lineno=0,
|
||||
msg=f"✅ Completed {worker} operation for job {job_id}",
|
||||
args=(),
|
||||
exc_info=None,
|
||||
))
|
||||
|
||||
# Flush remaining logs
|
||||
handler.flush()
|
||||
|
||||
# Remove handler from all loggers
|
||||
for log in loggers:
|
||||
log.removeHandler(handler)
|
||||
|
||||
# Close handler
|
||||
handler.close()
|
||||
|
||||
|
||||
class JobLogger:
|
||||
"""
|
||||
A convenience class for direct logging to a job's log stream.
|
||||
|
||||
Use this when you want to log specific messages without capturing
|
||||
all logging output.
|
||||
"""
|
||||
|
||||
def __init__(self, job_id: str, worker: str = "review"):
|
||||
"""
|
||||
Initialize job logger.
|
||||
|
||||
Args:
|
||||
job_id: Job ID to log to
|
||||
worker: Worker name for log categorization
|
||||
"""
|
||||
self.job_id = job_id
|
||||
self.worker = worker
|
||||
self.firestore = FirestoreService()
|
||||
|
||||
def _log(self, level: str, message: str) -> None:
|
||||
"""Internal logging method."""
|
||||
log_entry = {
|
||||
'timestamp': datetime.utcnow().isoformat() + 'Z',
|
||||
'level': level,
|
||||
'worker': self.worker,
|
||||
'message': message[:1000],
|
||||
}
|
||||
try:
|
||||
self.firestore.append_worker_log(self.job_id, log_entry)
|
||||
except Exception:
|
||||
pass # Don't let logging errors propagate
|
||||
|
||||
def info(self, message: str) -> None:
|
||||
"""Log an INFO message."""
|
||||
self._log('INFO', message)
|
||||
|
||||
def warning(self, message: str) -> None:
|
||||
"""Log a WARNING message."""
|
||||
self._log('WARNING', message)
|
||||
|
||||
def error(self, message: str) -> None:
|
||||
"""Log an ERROR message."""
|
||||
self._log('ERROR', message)
|
||||
|
||||
def debug(self, message: str) -> None:
|
||||
"""Log a DEBUG message."""
|
||||
self._log('DEBUG', message)
|
||||
@@ -1,639 +0,0 @@
|
||||
"""
|
||||
Job management and queue operations.
|
||||
|
||||
This module handles the complete job lifecycle including:
|
||||
- Job creation and initialization
|
||||
- State transitions and validation
|
||||
- Worker coordination (parallel audio + lyrics processing)
|
||||
- Progress tracking and timeline events
|
||||
- Error handling and retries
|
||||
"""
|
||||
import logging
|
||||
import uuid
|
||||
from datetime import datetime
|
||||
from typing import Optional, Dict, Any, List
|
||||
|
||||
from backend.models.job import Job, JobStatus, JobCreate, STATE_TRANSITIONS
|
||||
from backend.services.firestore_service import FirestoreService
|
||||
from backend.services.storage_service import StorageService
|
||||
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
|
||||
class JobManager:
|
||||
"""Manager for job lifecycle and state."""
|
||||
|
||||
def __init__(self):
|
||||
"""Initialize job manager with required services."""
|
||||
self.firestore = FirestoreService()
|
||||
self.storage = StorageService()
|
||||
|
||||
def create_job(self, job_create: JobCreate) -> Job:
|
||||
"""
|
||||
Create a new job with initial state PENDING.
|
||||
|
||||
Jobs start in PENDING state and transition to DOWNLOADING
|
||||
when a worker picks them up.
|
||||
"""
|
||||
job_id = str(uuid.uuid4())[:8]
|
||||
|
||||
now = datetime.utcnow()
|
||||
job = Job(
|
||||
job_id=job_id,
|
||||
status=JobStatus.PENDING, # New state machine starts with PENDING
|
||||
progress=0,
|
||||
created_at=now,
|
||||
updated_at=now,
|
||||
url=str(job_create.url) if job_create.url else None,
|
||||
artist=job_create.artist,
|
||||
title=job_create.title,
|
||||
# User preferences
|
||||
enable_cdg=job_create.enable_cdg,
|
||||
enable_txt=job_create.enable_txt,
|
||||
enable_youtube_upload=job_create.enable_youtube_upload,
|
||||
youtube_description=job_create.youtube_description,
|
||||
youtube_description_template=job_create.youtube_description_template, # video_worker reads this
|
||||
webhook_url=job_create.webhook_url,
|
||||
user_email=job_create.user_email,
|
||||
# Distribution settings
|
||||
brand_prefix=job_create.brand_prefix,
|
||||
discord_webhook_url=job_create.discord_webhook_url,
|
||||
dropbox_path=job_create.dropbox_path,
|
||||
gdrive_folder_id=job_create.gdrive_folder_id,
|
||||
# Theme configuration
|
||||
theme_id=job_create.theme_id,
|
||||
color_overrides=job_create.color_overrides,
|
||||
style_params_gcs_path=job_create.style_params_gcs_path,
|
||||
style_assets=job_create.style_assets,
|
||||
# Two-phase workflow (Batch 6)
|
||||
prep_only=job_create.prep_only,
|
||||
finalise_only=job_create.finalise_only,
|
||||
keep_brand_code=job_create.keep_brand_code,
|
||||
# Request metadata (for tracking and filtering)
|
||||
request_metadata=job_create.request_metadata,
|
||||
)
|
||||
|
||||
self.firestore.create_job(job)
|
||||
logger.info(f"Created new job {job_id} with status PENDING")
|
||||
|
||||
return job
|
||||
|
||||
def get_job(self, job_id: str) -> Optional[Job]:
|
||||
"""Get a job by ID."""
|
||||
return self.firestore.get_job(job_id)
|
||||
|
||||
def update_job_status(
|
||||
self,
|
||||
job_id: str,
|
||||
status: JobStatus,
|
||||
progress: Optional[int] = None,
|
||||
message: Optional[str] = None,
|
||||
**kwargs
|
||||
) -> None:
|
||||
"""Update job status with timeline tracking."""
|
||||
self.firestore.update_job_status(
|
||||
job_id=job_id,
|
||||
status=status,
|
||||
progress=progress,
|
||||
message=message,
|
||||
**kwargs
|
||||
)
|
||||
|
||||
def update_job(self, job_id: str, updates: Dict[str, Any]) -> None:
|
||||
"""Update job with arbitrary fields."""
|
||||
self.firestore.update_job(job_id, updates)
|
||||
|
||||
def list_jobs(
|
||||
self,
|
||||
status: Optional[JobStatus] = None,
|
||||
environment: Optional[str] = None,
|
||||
client_id: Optional[str] = None,
|
||||
created_after: Optional[datetime] = None,
|
||||
created_before: Optional[datetime] = None,
|
||||
limit: int = 100
|
||||
) -> List[Job]:
|
||||
"""
|
||||
List jobs with optional filtering.
|
||||
|
||||
Args:
|
||||
status: Filter by job status
|
||||
environment: Filter by request_metadata.environment (test/production/development)
|
||||
client_id: Filter by request_metadata.client_id (customer identifier)
|
||||
created_after: Filter jobs created after this datetime
|
||||
created_before: Filter jobs created before this datetime
|
||||
limit: Maximum number of jobs to return
|
||||
|
||||
Returns:
|
||||
List of Job objects matching filters
|
||||
"""
|
||||
return self.firestore.list_jobs(
|
||||
status=status,
|
||||
environment=environment,
|
||||
client_id=client_id,
|
||||
created_after=created_after,
|
||||
created_before=created_before,
|
||||
limit=limit
|
||||
)
|
||||
|
||||
def delete_jobs_by_filter(
|
||||
self,
|
||||
environment: Optional[str] = None,
|
||||
client_id: Optional[str] = None,
|
||||
status: Optional[JobStatus] = None,
|
||||
created_before: Optional[datetime] = None,
|
||||
delete_files: bool = True
|
||||
) -> Dict[str, Any]:
|
||||
"""
|
||||
Delete multiple jobs matching filter criteria.
|
||||
|
||||
CAUTION: This is a destructive operation. Use carefully.
|
||||
|
||||
Args:
|
||||
environment: Delete jobs with this environment (e.g., "test")
|
||||
client_id: Delete jobs from this client
|
||||
status: Delete jobs with this status
|
||||
created_before: Delete jobs created before this datetime
|
||||
delete_files: Also delete GCS files (default True)
|
||||
|
||||
Returns:
|
||||
Dict with deletion statistics
|
||||
"""
|
||||
# First get matching jobs to delete their files
|
||||
if delete_files:
|
||||
jobs = self.firestore.list_jobs(
|
||||
status=status,
|
||||
environment=environment,
|
||||
client_id=client_id,
|
||||
created_before=created_before,
|
||||
limit=10000 # High limit for deletion
|
||||
)
|
||||
|
||||
files_deleted = 0
|
||||
for job in jobs:
|
||||
# Delete files from various locations
|
||||
try:
|
||||
# Delete uploads folder
|
||||
self.storage.delete_folder(f"uploads/{job.job_id}/")
|
||||
# Delete jobs folder
|
||||
self.storage.delete_folder(f"jobs/{job.job_id}/")
|
||||
files_deleted += 1
|
||||
except Exception as e:
|
||||
logger.warning(f"Error deleting files for job {job.job_id}: {e}")
|
||||
|
||||
# Delete the jobs from Firestore
|
||||
deleted_count = self.firestore.delete_jobs_by_filter(
|
||||
environment=environment,
|
||||
client_id=client_id,
|
||||
status=status,
|
||||
created_before=created_before
|
||||
)
|
||||
|
||||
return {
|
||||
'jobs_deleted': deleted_count,
|
||||
'files_deleted': files_deleted if delete_files else 0
|
||||
}
|
||||
|
||||
def mark_job_error(self, job_id: str, error_message: str) -> None:
|
||||
"""Mark a job as errored."""
|
||||
self.firestore.update_job_status(
|
||||
job_id=job_id,
|
||||
status=JobStatus.ERROR,
|
||||
progress=0,
|
||||
message=error_message,
|
||||
error_message=error_message
|
||||
)
|
||||
logger.error(f"Job {job_id} marked as error: {error_message}")
|
||||
|
||||
def get_output_urls(self, job_id: str) -> Dict[str, str]:
|
||||
"""Generate signed URLs for job output files."""
|
||||
job = self.get_job(job_id)
|
||||
if not job or not job.output_files:
|
||||
return {}
|
||||
|
||||
urls = {}
|
||||
for file_type, gcs_path in job.output_files.items():
|
||||
try:
|
||||
urls[file_type] = self.storage.generate_signed_url(gcs_path, expiration_minutes=120)
|
||||
except Exception as e:
|
||||
logger.error(f"Error generating URL for {file_type}: {e}")
|
||||
|
||||
return urls
|
||||
|
||||
def delete_job(self, job_id: str, delete_files: bool = True) -> None:
|
||||
"""Delete a job and optionally its files."""
|
||||
if delete_files:
|
||||
job = self.get_job(job_id)
|
||||
if job and job.output_files:
|
||||
for gcs_path in job.output_files.values():
|
||||
try:
|
||||
self.storage.delete_file(gcs_path)
|
||||
except Exception as e:
|
||||
logger.error(f"Error deleting file {gcs_path}: {e}")
|
||||
|
||||
self.firestore.delete_job(job_id)
|
||||
logger.info(f"Deleted job {job_id}")
|
||||
|
||||
def validate_state_transition(self, job_id: str, new_status: JobStatus) -> bool:
|
||||
"""
|
||||
Validate that a state transition is legal.
|
||||
|
||||
Returns:
|
||||
True if transition is valid, False otherwise
|
||||
"""
|
||||
job = self.get_job(job_id)
|
||||
if not job:
|
||||
logger.error(f"Job {job_id} not found")
|
||||
return False
|
||||
|
||||
current_status = job.status
|
||||
valid_transitions = STATE_TRANSITIONS.get(current_status, [])
|
||||
|
||||
if new_status not in valid_transitions:
|
||||
logger.error(
|
||||
f"Invalid state transition for job {job_id}: "
|
||||
f"{current_status} -> {new_status}. "
|
||||
f"Valid transitions: {valid_transitions}"
|
||||
)
|
||||
return False
|
||||
|
||||
return True
|
||||
|
||||
def transition_to_state(
|
||||
self,
|
||||
job_id: str,
|
||||
new_status: JobStatus,
|
||||
progress: Optional[int] = None,
|
||||
message: Optional[str] = None,
|
||||
state_data_updates: Optional[Dict[str, Any]] = None
|
||||
) -> bool:
|
||||
"""
|
||||
Transition job to new state with validation.
|
||||
|
||||
Args:
|
||||
job_id: Job ID
|
||||
new_status: Target state
|
||||
progress: Progress percentage (0-100)
|
||||
message: Timeline message
|
||||
state_data_updates: Updates to state_data field
|
||||
|
||||
Returns:
|
||||
True if transition succeeded, False otherwise
|
||||
"""
|
||||
if not self.validate_state_transition(job_id, new_status):
|
||||
return False
|
||||
|
||||
updates = {
|
||||
'status': new_status,
|
||||
'updated_at': datetime.utcnow()
|
||||
}
|
||||
|
||||
if progress is not None:
|
||||
updates['progress'] = progress
|
||||
|
||||
# Generate review token when entering AWAITING_REVIEW state
|
||||
if new_status == JobStatus.AWAITING_REVIEW:
|
||||
from backend.api.dependencies import generate_review_token, get_review_token_expiry
|
||||
review_token = generate_review_token()
|
||||
review_token_expires = get_review_token_expiry(hours=48) # 48 hour expiry
|
||||
updates['review_token'] = review_token
|
||||
updates['review_token_expires_at'] = review_token_expires
|
||||
logger.info(f"Generated review token for job {job_id}, expires in 48 hours")
|
||||
|
||||
# Generate instrumental token when entering AWAITING_INSTRUMENTAL_SELECTION state
|
||||
if new_status == JobStatus.AWAITING_INSTRUMENTAL_SELECTION:
|
||||
from backend.api.dependencies import generate_review_token, get_review_token_expiry
|
||||
instrumental_token = generate_review_token() # Reuse same token generator
|
||||
instrumental_token_expires = get_review_token_expiry(hours=48) # 48 hour expiry
|
||||
updates['instrumental_token'] = instrumental_token
|
||||
updates['instrumental_token_expires_at'] = instrumental_token_expires
|
||||
logger.info(f"Generated instrumental token for job {job_id}, expires in 48 hours")
|
||||
|
||||
# If we have state_data_updates, merge them with existing state_data
|
||||
merged_state_data = None
|
||||
if state_data_updates:
|
||||
job = self.get_job(job_id)
|
||||
if job:
|
||||
merged_state_data = {**job.state_data, **state_data_updates}
|
||||
|
||||
# Update job status (includes timeline event), passing state_data if present
|
||||
if merged_state_data is not None:
|
||||
self.update_job_status(
|
||||
job_id=job_id,
|
||||
status=new_status,
|
||||
progress=progress,
|
||||
message=message,
|
||||
state_data=merged_state_data
|
||||
)
|
||||
else:
|
||||
self.update_job_status(
|
||||
job_id=job_id,
|
||||
status=new_status,
|
||||
progress=progress,
|
||||
message=message
|
||||
)
|
||||
|
||||
# Apply review token update separately if generated
|
||||
if new_status == JobStatus.AWAITING_REVIEW and 'review_token' in updates:
|
||||
self.firestore.update_job(job_id, {
|
||||
'review_token': updates['review_token'],
|
||||
'review_token_expires_at': updates['review_token_expires_at']
|
||||
})
|
||||
|
||||
# Apply instrumental token update separately if generated
|
||||
if new_status == JobStatus.AWAITING_INSTRUMENTAL_SELECTION and 'instrumental_token' in updates:
|
||||
self.firestore.update_job(job_id, {
|
||||
'instrumental_token': updates['instrumental_token'],
|
||||
'instrumental_token_expires_at': updates['instrumental_token_expires_at']
|
||||
})
|
||||
|
||||
logger.info(f"Job {job_id} transitioned to {new_status}")
|
||||
return True
|
||||
|
||||
def update_state_data(self, job_id: str, key: str, value: Any) -> None:
|
||||
"""
|
||||
Update a specific key in the job's state_data field.
|
||||
|
||||
This is used by workers to store stage-specific metadata.
|
||||
"""
|
||||
job = self.get_job(job_id)
|
||||
if not job:
|
||||
logger.error(f"Job {job_id} not found")
|
||||
return
|
||||
|
||||
state_data = job.state_data.copy()
|
||||
state_data[key] = value
|
||||
|
||||
self.update_job(job_id, {'state_data': state_data})
|
||||
logger.debug(f"Job {job_id} state_data updated: {key} = {value}")
|
||||
|
||||
def fail_job(self, job_id: str, error_message: str, error_details: Optional[Dict[str, Any]] = None) -> bool:
|
||||
"""
|
||||
Mark a job as failed with error information.
|
||||
|
||||
Args:
|
||||
job_id: Job ID
|
||||
error_message: Human-readable error message
|
||||
error_details: Optional structured error details
|
||||
|
||||
Returns:
|
||||
True if successful
|
||||
"""
|
||||
try:
|
||||
# Update error fields
|
||||
self.update_job(job_id, {
|
||||
'error_message': error_message,
|
||||
'error_details': error_details or {}
|
||||
})
|
||||
|
||||
# Use update_job_status which handles timeline
|
||||
self.update_job_status(
|
||||
job_id=job_id,
|
||||
status=JobStatus.FAILED,
|
||||
message=error_message
|
||||
)
|
||||
|
||||
logger.error(f"Job {job_id} failed: {error_message}")
|
||||
return True
|
||||
except Exception as e:
|
||||
logger.error(f"Error marking job {job_id} as failed: {e}")
|
||||
return False
|
||||
|
||||
def update_file_url(self, job_id: str, category: str, file_type: str, url: str) -> None:
|
||||
"""
|
||||
Update a file URL in the job's file_urls structure.
|
||||
|
||||
Args:
|
||||
job_id: Job ID
|
||||
category: Category (e.g., "stems", "lyrics", "finals")
|
||||
file_type: File type within category (e.g., "clean", "lrc", "lossless_4k_mp4")
|
||||
url: GCS URL
|
||||
"""
|
||||
job = self.get_job(job_id)
|
||||
if not job:
|
||||
logger.error(f"Job {job_id} not found")
|
||||
return
|
||||
|
||||
file_urls = job.file_urls.copy()
|
||||
if category not in file_urls:
|
||||
file_urls[category] = {}
|
||||
|
||||
file_urls[category][file_type] = url
|
||||
|
||||
self.update_job(job_id, {'file_urls': file_urls})
|
||||
logger.debug(f"Job {job_id} file URL updated: {category}.{file_type}")
|
||||
|
||||
def check_parallel_processing_complete(self, job_id: str) -> bool:
|
||||
"""
|
||||
Check if both parallel tracks (audio + lyrics) are complete.
|
||||
|
||||
This is called after audio_complete or lyrics_complete to determine
|
||||
if we can proceed to screen generation.
|
||||
|
||||
Returns:
|
||||
True if both tracks complete, False otherwise
|
||||
"""
|
||||
job = self.get_job(job_id)
|
||||
if not job:
|
||||
return False
|
||||
|
||||
audio_complete = job.state_data.get('audio_complete', False)
|
||||
lyrics_complete = job.state_data.get('lyrics_complete', False)
|
||||
|
||||
return audio_complete and lyrics_complete
|
||||
|
||||
def mark_audio_complete(self, job_id: str) -> None:
|
||||
"""
|
||||
Mark audio processing as complete and check if can proceed.
|
||||
|
||||
If lyrics are also complete, automatically triggers screens worker.
|
||||
"""
|
||||
self.update_state_data(job_id, 'audio_complete', True)
|
||||
|
||||
if self.check_parallel_processing_complete(job_id):
|
||||
logger.info(f"Job {job_id}: Both audio and lyrics complete, triggering screens worker")
|
||||
# Transition happens in screens worker
|
||||
# We just trigger it here
|
||||
self._trigger_screens_worker(job_id)
|
||||
|
||||
def mark_lyrics_complete(self, job_id: str) -> None:
|
||||
"""
|
||||
Mark lyrics processing as complete and check if can proceed.
|
||||
|
||||
If audio is also complete, automatically triggers screens worker.
|
||||
"""
|
||||
self.update_state_data(job_id, 'lyrics_complete', True)
|
||||
|
||||
if self.check_parallel_processing_complete(job_id):
|
||||
logger.info(f"Job {job_id}: Both audio and lyrics complete, triggering screens worker")
|
||||
# Transition happens in screens worker
|
||||
# We just trigger it here
|
||||
self._trigger_screens_worker(job_id)
|
||||
|
||||
def _trigger_screens_worker(self, job_id: str) -> None:
|
||||
"""
|
||||
Trigger screens generation worker.
|
||||
|
||||
Uses WorkerService to make HTTP call to internal API.
|
||||
This must be async, so we use asyncio to create a task.
|
||||
"""
|
||||
import asyncio
|
||||
from backend.services.worker_service import get_worker_service
|
||||
|
||||
logger.info(f"Job {job_id}: Triggering screens worker")
|
||||
|
||||
# Create async task to trigger worker
|
||||
# This allows us to call async code from sync context
|
||||
async def _trigger():
|
||||
worker_service = get_worker_service()
|
||||
await worker_service.trigger_screens_worker(job_id)
|
||||
|
||||
# Create task in event loop
|
||||
try:
|
||||
loop = asyncio.get_event_loop()
|
||||
if loop.is_running():
|
||||
# If loop is already running, create task
|
||||
asyncio.create_task(_trigger())
|
||||
else:
|
||||
# If no loop, run directly
|
||||
asyncio.run(_trigger())
|
||||
except RuntimeError:
|
||||
# Fallback: just log
|
||||
logger.warning(f"Job {job_id}: Could not trigger screens worker (no event loop)")
|
||||
# TODO: In production, use message queue instead
|
||||
|
||||
def cancel_job(self, job_id: str, reason: Optional[str] = None) -> bool:
|
||||
"""
|
||||
Cancel a job.
|
||||
|
||||
Only jobs in non-terminal states can be cancelled.
|
||||
"""
|
||||
job = self.get_job(job_id)
|
||||
if not job:
|
||||
return False
|
||||
|
||||
terminal_states = [JobStatus.COMPLETE, JobStatus.FAILED, JobStatus.CANCELLED]
|
||||
if job.status in terminal_states:
|
||||
logger.warning(f"Cannot cancel job {job_id} in terminal state {job.status}")
|
||||
return False
|
||||
|
||||
message = f"Job cancelled{f': {reason}' if reason else ''}"
|
||||
|
||||
self.update_job_status(
|
||||
job_id=job_id,
|
||||
status=JobStatus.CANCELLED,
|
||||
message=message
|
||||
)
|
||||
|
||||
logger.info(f"Job {job_id} cancelled")
|
||||
return True
|
||||
|
||||
def mark_job_failed(
|
||||
self,
|
||||
job_id: str,
|
||||
error_message: str,
|
||||
error_details: Optional[Dict[str, Any]] = None
|
||||
) -> None:
|
||||
"""
|
||||
Mark a job as failed with error details.
|
||||
|
||||
This replaces mark_job_error() with better error tracking.
|
||||
"""
|
||||
updates = {
|
||||
'error_message': error_message
|
||||
}
|
||||
|
||||
if error_details:
|
||||
updates['error_details'] = error_details
|
||||
|
||||
self.firestore.update_job_status(
|
||||
job_id=job_id,
|
||||
status=JobStatus.FAILED,
|
||||
progress=0,
|
||||
message=f"Failed: {error_message}",
|
||||
**updates
|
||||
)
|
||||
logger.error(f"Job {job_id} failed: {error_message}")
|
||||
|
||||
def increment_retry_count(self, job_id: str) -> int:
|
||||
"""
|
||||
Increment retry count for a job.
|
||||
|
||||
Returns:
|
||||
New retry count
|
||||
"""
|
||||
job = self.get_job(job_id)
|
||||
if not job:
|
||||
return 0
|
||||
|
||||
new_count = job.retry_count + 1
|
||||
self.update_job(job_id, {'retry_count': new_count})
|
||||
return new_count
|
||||
|
||||
def append_worker_log(
|
||||
self,
|
||||
job_id: str,
|
||||
worker: str,
|
||||
level: str,
|
||||
message: str,
|
||||
max_logs: int = 500
|
||||
) -> None:
|
||||
"""
|
||||
Append a log entry to the job's worker_logs atomically.
|
||||
|
||||
Uses Firestore ArrayUnion for atomic append to avoid race conditions
|
||||
when multiple workers (audio + lyrics) are logging concurrently.
|
||||
|
||||
Note: max_logs is not enforced per-append (would require transactions).
|
||||
Logs may grow beyond max_logs; trim periodically if needed.
|
||||
|
||||
Args:
|
||||
job_id: Job ID
|
||||
worker: Worker name (audio, lyrics, screens, video, render)
|
||||
level: Log level (DEBUG, INFO, WARNING, ERROR)
|
||||
message: Log message
|
||||
max_logs: Not used (kept for API compatibility)
|
||||
"""
|
||||
log_entry = {
|
||||
'timestamp': datetime.utcnow().isoformat() + 'Z',
|
||||
'level': level,
|
||||
'worker': worker,
|
||||
'message': message[:1000] # Truncate long messages
|
||||
}
|
||||
|
||||
# Use atomic ArrayUnion to avoid race conditions
|
||||
self.firestore.append_worker_log(job_id, log_entry)
|
||||
|
||||
def get_worker_logs(
|
||||
self,
|
||||
job_id: str,
|
||||
since_index: int = 0,
|
||||
worker: Optional[str] = None
|
||||
) -> List[Dict[str, Any]]:
|
||||
"""
|
||||
Get worker logs for a job, optionally filtered by worker and index.
|
||||
|
||||
Args:
|
||||
job_id: Job ID
|
||||
since_index: Return only logs after this index (for pagination)
|
||||
worker: Filter by worker name (optional)
|
||||
|
||||
Returns:
|
||||
List of log entries as dicts
|
||||
"""
|
||||
job = self.get_job(job_id)
|
||||
if not job or not job.worker_logs:
|
||||
return []
|
||||
|
||||
logs = [log.dict() if hasattr(log, 'dict') else log for log in job.worker_logs]
|
||||
|
||||
# Filter by index
|
||||
if since_index > 0:
|
||||
logs = logs[since_index:]
|
||||
|
||||
# Filter by worker
|
||||
if worker:
|
||||
logs = [log for log in logs if log.get('worker') == worker]
|
||||
|
||||
return logs
|
||||
|
||||
@@ -1,413 +0,0 @@
|
||||
"""
|
||||
Custom metrics for Cloud Monitoring.
|
||||
|
||||
This module provides application-level metrics for tracking job processing,
|
||||
worker performance, and external API usage. Metrics can be viewed in:
|
||||
1. Cloud Logging (via structured log entries)
|
||||
2. Cloud Monitoring (via log-based metrics or OpenTelemetry)
|
||||
|
||||
The metrics service uses a pragmatic approach:
|
||||
- Always emits metrics as structured log entries (works immediately)
|
||||
- Uses the same JSON format as Cloud Logging
|
||||
- Can be enhanced with OpenTelemetry metrics exporters when available
|
||||
|
||||
Usage:
|
||||
from backend.services.metrics import metrics
|
||||
|
||||
# Record a job completion
|
||||
metrics.record_job_completed("abc123", source="upload")
|
||||
|
||||
# Record worker duration
|
||||
metrics.record_worker_duration("audio", 45.2, success=True)
|
||||
|
||||
# Record external API call
|
||||
with metrics.time_external_api("modal"):
|
||||
response = await modal_client.separate_audio(...)
|
||||
"""
|
||||
import logging
|
||||
import time
|
||||
from contextlib import contextmanager
|
||||
from dataclasses import dataclass, field
|
||||
from typing import Any, Dict, Optional
|
||||
|
||||
from backend.services.tracing import get_current_trace_id, get_current_span_id
|
||||
|
||||
|
||||
logger = logging.getLogger("metrics")
|
||||
|
||||
|
||||
@dataclass
|
||||
class MetricLabels:
|
||||
"""Common metric labels."""
|
||||
job_id: Optional[str] = None
|
||||
worker: Optional[str] = None
|
||||
status: Optional[str] = None
|
||||
source: Optional[str] = None
|
||||
api: Optional[str] = None
|
||||
operation: Optional[str] = None
|
||||
bucket: Optional[str] = None
|
||||
|
||||
def to_dict(self) -> Dict[str, Any]:
|
||||
"""Convert to dict, excluding None values."""
|
||||
return {k: v for k, v in self.__dict__.items() if v is not None}
|
||||
|
||||
|
||||
class MetricsService:
|
||||
"""
|
||||
Application metrics service.
|
||||
|
||||
Emits metrics as structured log entries that can be:
|
||||
1. Queried directly in Cloud Logging
|
||||
2. Converted to Cloud Monitoring metrics via log-based metrics
|
||||
3. Exported via OpenTelemetry (when configured)
|
||||
"""
|
||||
|
||||
def __init__(self):
|
||||
"""Initialize the metrics service."""
|
||||
self._logger = logging.getLogger("metrics")
|
||||
# Ensure metrics logger outputs at INFO level
|
||||
self._logger.setLevel(logging.INFO)
|
||||
|
||||
def _emit_metric(self, metric_name: str, metric_type: str, value: float, labels: Dict[str, Any]) -> None:
|
||||
"""
|
||||
Emit a metric as a structured log entry.
|
||||
|
||||
The log format is designed to be easily parsed by Cloud Logging
|
||||
and converted to log-based metrics.
|
||||
|
||||
Args:
|
||||
metric_name: Name of the metric (e.g., "jobs_total")
|
||||
metric_type: Type of metric (counter, histogram, gauge)
|
||||
value: Metric value
|
||||
labels: Metric labels/dimensions
|
||||
"""
|
||||
# Add trace context if available
|
||||
trace_id = get_current_trace_id()
|
||||
span_id = get_current_span_id()
|
||||
|
||||
# Build metric entry
|
||||
metric_entry = {
|
||||
"metric_name": metric_name,
|
||||
"metric_type": metric_type,
|
||||
"metric_value": value,
|
||||
**labels,
|
||||
}
|
||||
|
||||
if trace_id:
|
||||
metric_entry["trace_id"] = trace_id
|
||||
if span_id:
|
||||
metric_entry["span_id"] = span_id
|
||||
|
||||
# Emit as structured log entry
|
||||
# Use INFO level so metrics always show up
|
||||
self._logger.info(
|
||||
f"METRIC {metric_name}={value}",
|
||||
extra=metric_entry
|
||||
)
|
||||
|
||||
# =========================================
|
||||
# Job Metrics
|
||||
# =========================================
|
||||
|
||||
def record_job_created(self, job_id: str, source: str = "unknown") -> None:
|
||||
"""
|
||||
Record a new job creation.
|
||||
|
||||
Args:
|
||||
job_id: Job ID
|
||||
source: Job source (upload, url, search)
|
||||
"""
|
||||
self._emit_metric(
|
||||
metric_name="jobs_total",
|
||||
metric_type="counter",
|
||||
value=1,
|
||||
labels={"job_id": job_id, "status": "created", "source": source}
|
||||
)
|
||||
|
||||
def record_job_completed(self, job_id: str, source: str = "unknown") -> None:
|
||||
"""
|
||||
Record a job completion.
|
||||
|
||||
Args:
|
||||
job_id: Job ID
|
||||
source: Job source (upload, url, search)
|
||||
"""
|
||||
self._emit_metric(
|
||||
metric_name="jobs_total",
|
||||
metric_type="counter",
|
||||
value=1,
|
||||
labels={"job_id": job_id, "status": "completed", "source": source}
|
||||
)
|
||||
|
||||
def record_job_failed(self, job_id: str, source: str = "unknown", error: Optional[str] = None) -> None:
|
||||
"""
|
||||
Record a job failure.
|
||||
|
||||
Args:
|
||||
job_id: Job ID
|
||||
source: Job source (upload, url, search)
|
||||
error: Optional error message
|
||||
"""
|
||||
labels = {"job_id": job_id, "status": "failed", "source": source}
|
||||
if error:
|
||||
labels["error"] = error[:200] # Truncate long errors
|
||||
self._emit_metric(
|
||||
metric_name="jobs_total",
|
||||
metric_type="counter",
|
||||
value=1,
|
||||
labels=labels
|
||||
)
|
||||
|
||||
def record_job_duration(self, job_id: str, duration_seconds: float, source: str = "unknown") -> None:
|
||||
"""
|
||||
Record total job processing duration.
|
||||
|
||||
Args:
|
||||
job_id: Job ID
|
||||
duration_seconds: Total processing time in seconds
|
||||
source: Job source (upload, url, search)
|
||||
"""
|
||||
self._emit_metric(
|
||||
metric_name="job_duration_seconds",
|
||||
metric_type="histogram",
|
||||
value=duration_seconds,
|
||||
labels={"job_id": job_id, "source": source}
|
||||
)
|
||||
|
||||
# =========================================
|
||||
# Worker Metrics
|
||||
# =========================================
|
||||
|
||||
def record_worker_started(self, worker: str, job_id: str) -> None:
|
||||
"""
|
||||
Record a worker invocation start.
|
||||
|
||||
Args:
|
||||
worker: Worker name (audio, lyrics, screens, video, render_video)
|
||||
job_id: Job ID
|
||||
"""
|
||||
self._emit_metric(
|
||||
metric_name="worker_invocations_total",
|
||||
metric_type="counter",
|
||||
value=1,
|
||||
labels={"worker": worker, "job_id": job_id, "status": "started"}
|
||||
)
|
||||
|
||||
def record_worker_duration(self, worker: str, duration_seconds: float, success: bool, job_id: Optional[str] = None) -> None:
|
||||
"""
|
||||
Record worker processing duration.
|
||||
|
||||
Args:
|
||||
worker: Worker name
|
||||
duration_seconds: Processing time in seconds
|
||||
success: Whether worker completed successfully
|
||||
job_id: Optional job ID
|
||||
"""
|
||||
labels = {
|
||||
"worker": worker,
|
||||
"success": str(success).lower(),
|
||||
}
|
||||
if job_id:
|
||||
labels["job_id"] = job_id
|
||||
|
||||
self._emit_metric(
|
||||
metric_name="job_stage_duration_seconds",
|
||||
metric_type="histogram",
|
||||
value=duration_seconds,
|
||||
labels=labels
|
||||
)
|
||||
|
||||
# Also emit a counter for success/failure tracking
|
||||
self._emit_metric(
|
||||
metric_name="worker_invocations_total",
|
||||
metric_type="counter",
|
||||
value=1,
|
||||
labels={"worker": worker, "success": str(success).lower(), "job_id": job_id or "unknown"}
|
||||
)
|
||||
|
||||
# =========================================
|
||||
# GCS Metrics
|
||||
# =========================================
|
||||
|
||||
def record_gcs_operation(
|
||||
self,
|
||||
operation: str,
|
||||
bucket: str,
|
||||
success: bool,
|
||||
size_bytes: Optional[int] = None,
|
||||
duration_seconds: Optional[float] = None,
|
||||
job_id: Optional[str] = None,
|
||||
) -> None:
|
||||
"""
|
||||
Record a GCS operation.
|
||||
|
||||
Args:
|
||||
operation: Operation type (upload, download, delete)
|
||||
bucket: GCS bucket name
|
||||
success: Whether operation succeeded
|
||||
size_bytes: Optional file size in bytes
|
||||
duration_seconds: Optional operation duration
|
||||
job_id: Optional job ID
|
||||
"""
|
||||
labels = {
|
||||
"operation": operation,
|
||||
"bucket": bucket,
|
||||
"success": str(success).lower(),
|
||||
}
|
||||
if job_id:
|
||||
labels["job_id"] = job_id
|
||||
|
||||
self._emit_metric(
|
||||
metric_name="gcs_operations_total",
|
||||
metric_type="counter",
|
||||
value=1,
|
||||
labels=labels
|
||||
)
|
||||
|
||||
if size_bytes is not None:
|
||||
self._emit_metric(
|
||||
metric_name="gcs_operation_bytes",
|
||||
metric_type="histogram",
|
||||
value=size_bytes,
|
||||
labels={**labels, "operation": operation}
|
||||
)
|
||||
|
||||
if duration_seconds is not None:
|
||||
self._emit_metric(
|
||||
metric_name="gcs_operation_duration_seconds",
|
||||
metric_type="histogram",
|
||||
value=duration_seconds,
|
||||
labels={**labels, "operation": operation}
|
||||
)
|
||||
|
||||
# =========================================
|
||||
# External API Metrics
|
||||
# =========================================
|
||||
|
||||
def record_external_api_call(
|
||||
self,
|
||||
api: str,
|
||||
success: bool,
|
||||
duration_seconds: float,
|
||||
job_id: Optional[str] = None,
|
||||
error: Optional[str] = None,
|
||||
) -> None:
|
||||
"""
|
||||
Record an external API call.
|
||||
|
||||
Args:
|
||||
api: API name (modal, audioshake, genius, spotify)
|
||||
success: Whether call succeeded
|
||||
duration_seconds: API call duration
|
||||
job_id: Optional job ID
|
||||
error: Optional error message
|
||||
"""
|
||||
labels = {
|
||||
"api": api,
|
||||
"success": str(success).lower(),
|
||||
}
|
||||
if job_id:
|
||||
labels["job_id"] = job_id
|
||||
if error:
|
||||
labels["error"] = error[:100] # Truncate
|
||||
|
||||
self._emit_metric(
|
||||
metric_name="external_api_calls_total",
|
||||
metric_type="counter",
|
||||
value=1,
|
||||
labels=labels
|
||||
)
|
||||
|
||||
self._emit_metric(
|
||||
metric_name="external_api_duration_seconds",
|
||||
metric_type="histogram",
|
||||
value=duration_seconds,
|
||||
labels={"api": api, "success": str(success).lower()}
|
||||
)
|
||||
|
||||
@contextmanager
|
||||
def time_external_api(self, api: str, job_id: Optional[str] = None):
|
||||
"""
|
||||
Context manager to time an external API call.
|
||||
|
||||
Usage:
|
||||
with metrics.time_external_api("modal", job_id) as timer:
|
||||
response = await client.call_api()
|
||||
timer.set_success(True)
|
||||
|
||||
Args:
|
||||
api: API name
|
||||
job_id: Optional job ID
|
||||
|
||||
Yields:
|
||||
Timer object with set_success() method
|
||||
"""
|
||||
timer = _ApiTimer()
|
||||
start_time = time.time()
|
||||
|
||||
try:
|
||||
yield timer
|
||||
except Exception as e:
|
||||
timer.set_success(False)
|
||||
timer.error = str(e)
|
||||
raise
|
||||
finally:
|
||||
duration = time.time() - start_time
|
||||
self.record_external_api_call(
|
||||
api=api,
|
||||
success=timer.success,
|
||||
duration_seconds=duration,
|
||||
job_id=job_id,
|
||||
error=timer.error,
|
||||
)
|
||||
|
||||
@contextmanager
|
||||
def time_worker(self, worker: str, job_id: str):
|
||||
"""
|
||||
Context manager to time a worker execution.
|
||||
|
||||
Usage:
|
||||
with metrics.time_worker("audio", job_id) as timer:
|
||||
await process_audio()
|
||||
timer.set_success(True)
|
||||
|
||||
Args:
|
||||
worker: Worker name
|
||||
job_id: Job ID
|
||||
|
||||
Yields:
|
||||
Timer object with set_success() method
|
||||
"""
|
||||
timer = _ApiTimer()
|
||||
self.record_worker_started(worker, job_id)
|
||||
start_time = time.time()
|
||||
|
||||
try:
|
||||
yield timer
|
||||
except Exception as e:
|
||||
timer.set_success(False)
|
||||
raise
|
||||
finally:
|
||||
duration = time.time() - start_time
|
||||
self.record_worker_duration(
|
||||
worker=worker,
|
||||
duration_seconds=duration,
|
||||
success=timer.success,
|
||||
job_id=job_id,
|
||||
)
|
||||
|
||||
|
||||
class _ApiTimer:
|
||||
"""Helper class for tracking API call success state."""
|
||||
|
||||
def __init__(self):
|
||||
self.success = True # Assume success unless set otherwise
|
||||
self.error: Optional[str] = None
|
||||
|
||||
def set_success(self, success: bool) -> None:
|
||||
self.success = success
|
||||
|
||||
|
||||
# Global metrics instance
|
||||
metrics = MetricsService()
|
||||
|
||||
@@ -1,106 +0,0 @@
|
||||
"""
|
||||
Rclone configuration service for cloud storage integration.
|
||||
|
||||
This service manages the rclone configuration needed for Dropbox
|
||||
and other cloud storage uploads from the backend workers.
|
||||
"""
|
||||
import logging
|
||||
import os
|
||||
import tempfile
|
||||
from typing import Optional
|
||||
|
||||
from backend.config import get_settings
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
|
||||
class RcloneService:
|
||||
"""Service for managing rclone configuration."""
|
||||
|
||||
# Secret Manager secret name for rclone config
|
||||
RCLONE_CONFIG_SECRET = "rclone-config"
|
||||
|
||||
def __init__(self):
|
||||
self.settings = get_settings()
|
||||
self._config_file: Optional[str] = None
|
||||
self._config_loaded = False
|
||||
|
||||
def setup_rclone_config(self) -> bool:
|
||||
"""
|
||||
Load rclone config from Secret Manager and set up environment.
|
||||
|
||||
Writes the config to a temp file and sets RCLONE_CONFIG env var.
|
||||
|
||||
Returns:
|
||||
True if successful, False otherwise
|
||||
"""
|
||||
if self._config_loaded:
|
||||
logger.debug("Rclone config already loaded")
|
||||
return True
|
||||
|
||||
try:
|
||||
# Get rclone config from Secret Manager
|
||||
config_content = self.settings.get_secret(self.RCLONE_CONFIG_SECRET)
|
||||
|
||||
if not config_content:
|
||||
logger.warning("Rclone config not found in Secret Manager")
|
||||
return False
|
||||
|
||||
# Write to a temp file
|
||||
fd, config_path = tempfile.mkstemp(prefix="rclone_", suffix=".conf")
|
||||
try:
|
||||
with os.fdopen(fd, 'w') as f:
|
||||
f.write(config_content)
|
||||
|
||||
self._config_file = config_path
|
||||
|
||||
# Set environment variable for rclone to find the config
|
||||
os.environ["RCLONE_CONFIG"] = config_path
|
||||
|
||||
logger.info(f"Rclone config loaded and written to {config_path}")
|
||||
self._config_loaded = True
|
||||
return True
|
||||
|
||||
except Exception:
|
||||
# Clean up the temp file on error
|
||||
# Note: os.fdopen() takes ownership of fd, so it's already closed
|
||||
# We only need to remove the temp file if it exists
|
||||
if os.path.exists(config_path):
|
||||
os.unlink(config_path)
|
||||
raise
|
||||
|
||||
except Exception as e:
|
||||
logger.error(f"Failed to setup rclone config: {e}")
|
||||
return False
|
||||
|
||||
def cleanup(self) -> None:
|
||||
"""Remove the temporary config file."""
|
||||
if self._config_file and os.path.exists(self._config_file):
|
||||
try:
|
||||
os.unlink(self._config_file)
|
||||
logger.debug(f"Cleaned up rclone config file: {self._config_file}")
|
||||
except Exception as e:
|
||||
logger.warning(f"Failed to cleanup rclone config: {e}")
|
||||
|
||||
# Always reset internal state and environment, even if the file was missing
|
||||
if self._config_file is not None:
|
||||
os.environ.pop("RCLONE_CONFIG", None)
|
||||
self._config_file = None
|
||||
self._config_loaded = False
|
||||
|
||||
@property
|
||||
def is_configured(self) -> bool:
|
||||
"""Check if rclone is configured and ready to use."""
|
||||
return self._config_loaded and self._config_file is not None
|
||||
|
||||
|
||||
# Singleton instance
|
||||
_rclone_service: Optional[RcloneService] = None
|
||||
|
||||
|
||||
def get_rclone_service() -> RcloneService:
|
||||
"""Get the singleton rclone service instance."""
|
||||
global _rclone_service
|
||||
if _rclone_service is None:
|
||||
_rclone_service = RcloneService()
|
||||
return _rclone_service
|
||||
@@ -1,209 +0,0 @@
|
||||
"""
|
||||
Google Cloud Storage operations for file management.
|
||||
"""
|
||||
import logging
|
||||
import os
|
||||
import json
|
||||
from typing import Optional, BinaryIO, Any, Dict
|
||||
from pathlib import Path
|
||||
from google.cloud import storage
|
||||
from datetime import timedelta
|
||||
|
||||
from backend.config import settings
|
||||
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
|
||||
class StorageService:
|
||||
"""Service for Google Cloud Storage operations."""
|
||||
|
||||
def __init__(self):
|
||||
"""Initialize GCS client."""
|
||||
self.client = storage.Client(project=settings.google_cloud_project)
|
||||
self.bucket = self.client.bucket(settings.gcs_bucket_name)
|
||||
|
||||
def upload_file(self, local_path: str, destination_path: str) -> str:
|
||||
"""Upload a file to GCS."""
|
||||
try:
|
||||
blob = self.bucket.blob(destination_path)
|
||||
blob.upload_from_filename(local_path)
|
||||
logger.info(f"Uploaded {local_path} to gs://{settings.gcs_bucket_name}/{destination_path}")
|
||||
return destination_path
|
||||
except Exception as e:
|
||||
logger.error(f"Error uploading file {local_path}: {e}")
|
||||
raise
|
||||
|
||||
def upload_fileobj(self, file_obj: BinaryIO, destination_path: str, content_type: Optional[str] = None) -> str:
|
||||
"""Upload a file object to GCS."""
|
||||
try:
|
||||
blob = self.bucket.blob(destination_path)
|
||||
if content_type:
|
||||
blob.content_type = content_type
|
||||
blob.upload_from_file(file_obj, rewind=True)
|
||||
logger.info(f"Uploaded file object to gs://{settings.gcs_bucket_name}/{destination_path}")
|
||||
return destination_path
|
||||
except Exception as e:
|
||||
logger.error(f"Error uploading file object: {e}")
|
||||
raise
|
||||
|
||||
def download_file(self, source_path: str, destination_path: str) -> str:
|
||||
"""Download a file from GCS."""
|
||||
try:
|
||||
blob = self.bucket.blob(source_path)
|
||||
blob.download_to_filename(destination_path)
|
||||
logger.info(f"Downloaded gs://{settings.gcs_bucket_name}/{source_path} to {destination_path}")
|
||||
return destination_path
|
||||
except Exception as e:
|
||||
logger.error(f"Error downloading file {source_path}: {e}")
|
||||
raise
|
||||
|
||||
def generate_signed_url(self, blob_path: str, expiration_minutes: int = 60) -> str:
|
||||
"""Generate a signed URL for downloading a file.
|
||||
|
||||
In Cloud Run, this uses the IAM signBlob API since we don't have
|
||||
a private key available. Requires the service account to have
|
||||
roles/iam.serviceAccountTokenCreator on itself.
|
||||
"""
|
||||
return self._generate_signed_url_internal(blob_path, "GET", expiration_minutes)
|
||||
|
||||
def generate_signed_upload_url(self, blob_path: str, content_type: str = "application/octet-stream", expiration_minutes: int = 60) -> str:
|
||||
"""Generate a signed URL for uploading a file directly to GCS.
|
||||
|
||||
This allows clients to upload files directly to GCS without going through
|
||||
the backend, bypassing any request body size limits.
|
||||
|
||||
Args:
|
||||
blob_path: The destination path in GCS
|
||||
content_type: The expected content type of the upload
|
||||
expiration_minutes: How long the URL is valid for
|
||||
|
||||
Returns:
|
||||
A signed URL that accepts PUT requests with the file content
|
||||
"""
|
||||
return self._generate_signed_url_internal(blob_path, "PUT", expiration_minutes, content_type)
|
||||
|
||||
def _generate_signed_url_internal(self, blob_path: str, method: str, expiration_minutes: int = 60, content_type: Optional[str] = None) -> str:
|
||||
"""Internal method to generate signed URLs for GET or PUT operations."""
|
||||
import google.auth
|
||||
from google.auth.transport import requests
|
||||
|
||||
try:
|
||||
blob = self.bucket.blob(blob_path)
|
||||
|
||||
# Get default credentials and refresh to ensure we have a valid token
|
||||
credentials, project = google.auth.default()
|
||||
|
||||
# Common kwargs for signed URL generation
|
||||
kwargs = {
|
||||
"version": "v4",
|
||||
"expiration": timedelta(minutes=expiration_minutes),
|
||||
"method": method,
|
||||
}
|
||||
|
||||
# For PUT requests, we need to specify the content type in headers
|
||||
if method == "PUT" and content_type:
|
||||
kwargs["headers"] = {"Content-Type": content_type}
|
||||
|
||||
# Check if we're using compute credentials (Cloud Run/GCE)
|
||||
# These need to use IAM signBlob via service_account_email + access_token
|
||||
if hasattr(credentials, 'service_account_email'):
|
||||
# Refresh credentials to get a valid access token
|
||||
auth_request = requests.Request()
|
||||
credentials.refresh(auth_request)
|
||||
|
||||
kwargs["service_account_email"] = credentials.service_account_email
|
||||
kwargs["access_token"] = credentials.token
|
||||
|
||||
url = blob.generate_signed_url(**kwargs)
|
||||
|
||||
logger.info(f"Generated signed {method} URL for {blob_path}")
|
||||
return url
|
||||
except Exception as e:
|
||||
logger.error(f"Error generating signed {method} URL for {blob_path}: {e}")
|
||||
raise
|
||||
|
||||
def delete_file(self, blob_path: str) -> None:
|
||||
"""Delete a file from GCS."""
|
||||
try:
|
||||
blob = self.bucket.blob(blob_path)
|
||||
blob.delete()
|
||||
logger.info(f"Deleted gs://{settings.gcs_bucket_name}/{blob_path}")
|
||||
except Exception as e:
|
||||
logger.error(f"Error deleting file {blob_path}: {e}")
|
||||
raise
|
||||
|
||||
def delete_folder(self, prefix: str) -> int:
|
||||
"""
|
||||
Delete all files in GCS with a given prefix (folder).
|
||||
|
||||
Args:
|
||||
prefix: The folder prefix to delete (e.g., "uploads/abc123/")
|
||||
|
||||
Returns:
|
||||
Number of files deleted
|
||||
"""
|
||||
try:
|
||||
blobs = list(self.bucket.list_blobs(prefix=prefix))
|
||||
deleted_count = 0
|
||||
|
||||
for blob in blobs:
|
||||
try:
|
||||
blob.delete()
|
||||
deleted_count += 1
|
||||
except Exception as e:
|
||||
logger.warning(f"Error deleting blob {blob.name}: {e}")
|
||||
|
||||
if deleted_count > 0:
|
||||
logger.info(f"Deleted {deleted_count} files from gs://{settings.gcs_bucket_name}/{prefix}")
|
||||
|
||||
return deleted_count
|
||||
except Exception as e:
|
||||
logger.error(f"Error deleting folder {prefix}: {e}")
|
||||
return 0 # Don't raise - folder deletion shouldn't break operations
|
||||
|
||||
def list_files(self, prefix: str) -> list:
|
||||
"""List files in GCS with a given prefix."""
|
||||
try:
|
||||
blobs = self.bucket.list_blobs(prefix=prefix)
|
||||
return [blob.name for blob in blobs]
|
||||
except Exception as e:
|
||||
logger.error(f"Error listing files with prefix {prefix}: {e}")
|
||||
raise
|
||||
|
||||
def file_exists(self, blob_path: str) -> bool:
|
||||
"""Check if a file exists in GCS."""
|
||||
try:
|
||||
blob = self.bucket.blob(blob_path)
|
||||
return blob.exists()
|
||||
except Exception as e:
|
||||
logger.error(f"Error checking file existence {blob_path}: {e}")
|
||||
raise
|
||||
|
||||
def upload_json(self, destination_path: str, data: Dict[str, Any]) -> str:
|
||||
"""Upload a JSON object to GCS."""
|
||||
try:
|
||||
blob = self.bucket.blob(destination_path)
|
||||
blob.content_type = "application/json"
|
||||
blob.upload_from_string(
|
||||
json.dumps(data, indent=2, ensure_ascii=False),
|
||||
content_type="application/json"
|
||||
)
|
||||
logger.info(f"Uploaded JSON to gs://{settings.gcs_bucket_name}/{destination_path}")
|
||||
return destination_path
|
||||
except Exception as e:
|
||||
logger.error(f"Error uploading JSON to {destination_path}: {e}")
|
||||
raise
|
||||
|
||||
def download_json(self, source_path: str) -> Dict[str, Any]:
|
||||
"""Download and parse a JSON file from GCS."""
|
||||
try:
|
||||
blob = self.bucket.blob(source_path)
|
||||
content = blob.download_as_text()
|
||||
data = json.loads(content)
|
||||
logger.info(f"Downloaded JSON from gs://{settings.gcs_bucket_name}/{source_path}")
|
||||
return data
|
||||
except Exception as e:
|
||||
logger.error(f"Error downloading JSON from {source_path}: {e}")
|
||||
raise
|
||||
|
||||
@@ -1,246 +0,0 @@
|
||||
"""
|
||||
Structured logging with trace correlation for Cloud Logging.
|
||||
|
||||
This module provides JSON-formatted logging that integrates with Google Cloud Logging
|
||||
and correlates logs with OpenTelemetry traces in Cloud Trace.
|
||||
|
||||
When running in Cloud Run:
|
||||
- Logs are output as JSON for Cloud Logging to parse
|
||||
- Each log entry includes trace ID and span ID for correlation
|
||||
- Custom fields (job_id, worker) are preserved in the log structure
|
||||
|
||||
Usage:
|
||||
# In main.py, before any logging:
|
||||
from backend.services.structured_logging import setup_structured_logging
|
||||
setup_structured_logging()
|
||||
|
||||
# Then use standard logging:
|
||||
logger = logging.getLogger(__name__)
|
||||
logger.info("Processing started", extra={"job_id": "abc123"})
|
||||
"""
|
||||
import json
|
||||
import logging
|
||||
import os
|
||||
import sys
|
||||
from datetime import datetime
|
||||
from typing import Any, Dict, Optional
|
||||
|
||||
|
||||
# Cloud Logging severity mapping (compatible with GCP)
|
||||
# https://cloud.google.com/logging/docs/reference/v2/rest/v2/LogEntry#LogSeverity
|
||||
SEVERITY_MAP = {
|
||||
"DEBUG": "DEBUG",
|
||||
"INFO": "INFO",
|
||||
"WARNING": "WARNING",
|
||||
"ERROR": "ERROR",
|
||||
"CRITICAL": "CRITICAL",
|
||||
}
|
||||
|
||||
|
||||
class StructuredFormatter(logging.Formatter):
|
||||
"""
|
||||
JSON formatter with trace correlation for Google Cloud Logging.
|
||||
|
||||
Produces log entries in the format expected by Cloud Logging, including:
|
||||
- logging.googleapis.com/trace: Links to Cloud Trace
|
||||
- logging.googleapis.com/spanId: Current span ID
|
||||
- severity: Cloud Logging severity level
|
||||
- Custom fields passed via 'extra' dict
|
||||
"""
|
||||
|
||||
def __init__(self, project_id: Optional[str] = None):
|
||||
"""
|
||||
Initialize the structured formatter.
|
||||
|
||||
Args:
|
||||
project_id: GCP project ID for trace URLs (auto-detected if not provided)
|
||||
"""
|
||||
super().__init__()
|
||||
self.project_id = project_id or os.environ.get("GOOGLE_CLOUD_PROJECT") or os.environ.get("GCP_PROJECT")
|
||||
|
||||
def format(self, record: logging.LogRecord) -> str:
|
||||
"""
|
||||
Format a log record as JSON.
|
||||
|
||||
Args:
|
||||
record: The log record to format
|
||||
|
||||
Returns:
|
||||
JSON string for Cloud Logging
|
||||
"""
|
||||
from backend.services.tracing import get_current_trace_id, get_current_span_id
|
||||
|
||||
# Build base log entry
|
||||
log_entry: Dict[str, Any] = {
|
||||
"timestamp": datetime.utcfromtimestamp(record.created).isoformat() + "Z",
|
||||
"severity": SEVERITY_MAP.get(record.levelname, record.levelname),
|
||||
"message": record.getMessage(),
|
||||
"logger": record.name,
|
||||
}
|
||||
|
||||
# Add trace correlation if available
|
||||
trace_id = get_current_trace_id()
|
||||
span_id = get_current_span_id()
|
||||
|
||||
if trace_id and self.project_id:
|
||||
# Format for Cloud Logging trace correlation
|
||||
log_entry["logging.googleapis.com/trace"] = f"projects/{self.project_id}/traces/{trace_id}"
|
||||
|
||||
if span_id:
|
||||
log_entry["logging.googleapis.com/spanId"] = span_id
|
||||
|
||||
# Add custom fields from 'extra' dict
|
||||
# Common fields we want to extract from log records
|
||||
custom_fields = ["job_id", "worker", "operation", "duration", "status", "error"]
|
||||
for field in custom_fields:
|
||||
value = getattr(record, field, None)
|
||||
if value is not None:
|
||||
log_entry[field] = value
|
||||
|
||||
# Add source location for debugging
|
||||
if record.pathname and record.lineno:
|
||||
log_entry["logging.googleapis.com/sourceLocation"] = {
|
||||
"file": record.pathname,
|
||||
"line": record.lineno,
|
||||
"function": record.funcName,
|
||||
}
|
||||
|
||||
# Add exception info if present
|
||||
if record.exc_info:
|
||||
log_entry["exception"] = self.formatException(record.exc_info)
|
||||
|
||||
# Remove None values for cleaner output
|
||||
return json.dumps({k: v for k, v in log_entry.items() if v is not None})
|
||||
|
||||
|
||||
class HumanReadableFormatter(logging.Formatter):
|
||||
"""
|
||||
Human-readable formatter for local development.
|
||||
|
||||
Includes trace context when available but outputs in traditional format.
|
||||
"""
|
||||
|
||||
def format(self, record: logging.LogRecord) -> str:
|
||||
"""Format a log record for human readability."""
|
||||
from backend.services.tracing import get_current_trace_id
|
||||
|
||||
# Build base message
|
||||
timestamp = datetime.utcfromtimestamp(record.created).strftime("%Y-%m-%d %H:%M:%S")
|
||||
|
||||
# Add job_id prefix if available
|
||||
job_id = getattr(record, "job_id", None)
|
||||
job_prefix = f"[job:{job_id[:8]}] " if job_id else ""
|
||||
|
||||
# Add trace ID suffix if available
|
||||
trace_id = get_current_trace_id()
|
||||
trace_suffix = f" [trace:{trace_id[:8]}]" if trace_id else ""
|
||||
|
||||
message = f"{timestamp} - {record.name} - {record.levelname} - {job_prefix}{record.getMessage()}{trace_suffix}"
|
||||
|
||||
# Add exception info if present
|
||||
if record.exc_info:
|
||||
message += "\n" + self.formatException(record.exc_info)
|
||||
|
||||
return message
|
||||
|
||||
|
||||
def is_cloud_run() -> bool:
|
||||
"""Check if we're running in Cloud Run."""
|
||||
return os.environ.get("K_SERVICE") is not None
|
||||
|
||||
|
||||
def setup_structured_logging(force_json: bool = False, log_level: Optional[str] = None) -> None:
|
||||
"""
|
||||
Configure structured logging for the application.
|
||||
|
||||
In Cloud Run: Uses JSON format with trace correlation for Cloud Logging
|
||||
In development: Uses human-readable format with optional trace context
|
||||
|
||||
Args:
|
||||
force_json: Force JSON output even in development
|
||||
log_level: Override log level (default from settings or INFO)
|
||||
"""
|
||||
from backend.config import settings
|
||||
|
||||
# Determine log level
|
||||
level = log_level or getattr(settings, "log_level", "INFO")
|
||||
level = getattr(logging, level.upper(), logging.INFO)
|
||||
|
||||
# Choose formatter based on environment
|
||||
if is_cloud_run() or force_json:
|
||||
formatter = StructuredFormatter()
|
||||
else:
|
||||
formatter = HumanReadableFormatter()
|
||||
|
||||
# Create handler
|
||||
handler = logging.StreamHandler(sys.stdout)
|
||||
handler.setFormatter(formatter)
|
||||
handler.setLevel(level)
|
||||
|
||||
# Configure root logger
|
||||
root_logger = logging.getLogger()
|
||||
root_logger.setLevel(level)
|
||||
|
||||
# Remove existing handlers and add our handler
|
||||
root_logger.handlers = [handler]
|
||||
|
||||
# Also configure uvicorn loggers to use our format
|
||||
for logger_name in ["uvicorn", "uvicorn.access", "uvicorn.error"]:
|
||||
logger = logging.getLogger(logger_name)
|
||||
logger.handlers = [handler]
|
||||
logger.propagate = False
|
||||
|
||||
|
||||
class JobLogAdapter(logging.LoggerAdapter):
|
||||
"""
|
||||
Logger adapter that automatically adds job context to log records.
|
||||
|
||||
Usage:
|
||||
logger = logging.getLogger(__name__)
|
||||
job_logger = JobLogAdapter(logger, job_id="abc123", worker="audio")
|
||||
job_logger.info("Processing started") # Automatically includes job_id and worker
|
||||
"""
|
||||
|
||||
def __init__(self, logger: logging.Logger, job_id: str, worker: Optional[str] = None, **extra):
|
||||
"""
|
||||
Initialize the adapter.
|
||||
|
||||
Args:
|
||||
logger: Base logger instance
|
||||
job_id: Job ID to include in all log records
|
||||
worker: Optional worker name
|
||||
**extra: Additional fields to include in all log records
|
||||
"""
|
||||
super().__init__(logger, {"job_id": job_id, "worker": worker, **extra})
|
||||
|
||||
def process(self, msg: str, kwargs: Dict[str, Any]) -> tuple:
|
||||
"""Process a logging call to add extra context."""
|
||||
# Merge extra dict from adapter with any extra passed to the log call
|
||||
extra = kwargs.get("extra", {})
|
||||
extra.update(self.extra)
|
||||
kwargs["extra"] = extra
|
||||
return msg, kwargs
|
||||
|
||||
|
||||
def get_job_logger(job_id: str, worker: Optional[str] = None, name: Optional[str] = None) -> JobLogAdapter:
|
||||
"""
|
||||
Get a logger adapter configured for a specific job.
|
||||
|
||||
This is a convenience function for creating job-specific loggers.
|
||||
|
||||
Args:
|
||||
job_id: Job ID to include in all log records
|
||||
worker: Optional worker name
|
||||
name: Logger name (defaults to "job")
|
||||
|
||||
Returns:
|
||||
JobLogAdapter configured for the job
|
||||
|
||||
Usage:
|
||||
logger = get_job_logger("abc123", "audio")
|
||||
logger.info("Starting audio separation")
|
||||
logger.error("Failed to process", extra={"error": str(e)})
|
||||
"""
|
||||
base_logger = logging.getLogger(name or "job")
|
||||
return JobLogAdapter(base_logger, job_id=job_id, worker=worker)
|
||||
|
||||
@@ -1,469 +0,0 @@
|
||||
"""
|
||||
Theme service for managing pre-created style themes.
|
||||
|
||||
Themes are stored in GCS at themes/{theme_id}/ with:
|
||||
- style_params.json: Complete style configuration
|
||||
- preview.png: Preview image for theme selection UI
|
||||
- youtube_description.txt: Optional YouTube description template
|
||||
- assets/: Fonts, backgrounds, and other assets
|
||||
|
||||
The service provides:
|
||||
- Theme listing with signed preview URLs
|
||||
- Theme detail retrieval
|
||||
- Color override application
|
||||
- Job style preparation (copying theme to job folder)
|
||||
"""
|
||||
|
||||
import copy
|
||||
import json
|
||||
import logging
|
||||
import os
|
||||
import tempfile
|
||||
import threading
|
||||
from datetime import datetime, timedelta
|
||||
from typing import Any, Dict, List, Optional, Tuple
|
||||
|
||||
from backend.models.theme import (
|
||||
ColorOverrides,
|
||||
ThemeDetail,
|
||||
ThemeMetadata,
|
||||
ThemeRegistry,
|
||||
ThemeSummary,
|
||||
hex_to_rgba,
|
||||
)
|
||||
from backend.services.storage_service import StorageService
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
# GCS paths for themes
|
||||
THEMES_PREFIX = "themes"
|
||||
METADATA_FILE = f"{THEMES_PREFIX}/_metadata.json"
|
||||
|
||||
|
||||
class ThemeService:
|
||||
"""Service for managing themes from GCS."""
|
||||
|
||||
def __init__(self, storage: Optional[StorageService] = None):
|
||||
"""
|
||||
Initialize the theme service.
|
||||
|
||||
Args:
|
||||
storage: StorageService instance (creates new one if not provided)
|
||||
"""
|
||||
self.storage = storage or StorageService()
|
||||
self._metadata_cache: Optional[ThemeRegistry] = None
|
||||
self._cache_time: Optional[datetime] = None
|
||||
self.CACHE_TTL_SECONDS = 300 # 5 minute cache for theme metadata
|
||||
|
||||
def _get_theme_path(self, theme_id: str, filename: str = "") -> str:
|
||||
"""Get the GCS path for a theme file."""
|
||||
if filename:
|
||||
return f"{THEMES_PREFIX}/{theme_id}/{filename}"
|
||||
return f"{THEMES_PREFIX}/{theme_id}"
|
||||
|
||||
def _is_cache_valid(self) -> bool:
|
||||
"""Check if the metadata cache is still valid."""
|
||||
if self._metadata_cache is None or self._cache_time is None:
|
||||
return False
|
||||
age = datetime.now() - self._cache_time
|
||||
return age.total_seconds() < self.CACHE_TTL_SECONDS
|
||||
|
||||
def _load_metadata(self, force_refresh: bool = False) -> ThemeRegistry:
|
||||
"""
|
||||
Load theme metadata from GCS with caching.
|
||||
|
||||
Args:
|
||||
force_refresh: Force reload from GCS even if cache is valid
|
||||
|
||||
Returns:
|
||||
ThemeRegistry containing all theme metadata
|
||||
"""
|
||||
if not force_refresh and self._is_cache_valid():
|
||||
return self._metadata_cache # type: ignore
|
||||
|
||||
try:
|
||||
data = self.storage.download_json(METADATA_FILE)
|
||||
self._metadata_cache = ThemeRegistry(**data)
|
||||
self._cache_time = datetime.now()
|
||||
logger.info(f"Loaded theme metadata: {len(self._metadata_cache.themes)} themes")
|
||||
return self._metadata_cache
|
||||
except Exception as e:
|
||||
logger.error(f"Failed to load theme metadata: {e}")
|
||||
# Return empty registry on error
|
||||
return ThemeRegistry(version=1, themes=[])
|
||||
|
||||
def list_themes(self) -> List[ThemeSummary]:
|
||||
"""
|
||||
List all available themes with signed preview URLs.
|
||||
|
||||
Returns:
|
||||
List of ThemeSummary objects with preview URLs
|
||||
"""
|
||||
metadata = self._load_metadata()
|
||||
summaries = []
|
||||
|
||||
for theme in metadata.themes:
|
||||
try:
|
||||
# Generate signed URLs for preview images
|
||||
preview_path = self._get_theme_path(theme.id, "preview.png")
|
||||
thumbnail_path = self._get_theme_path(theme.id, "preview_thumbnail.png")
|
||||
|
||||
preview_url = None
|
||||
thumbnail_url = None
|
||||
|
||||
if self.storage.file_exists(preview_path):
|
||||
preview_url = self.storage.generate_signed_url(preview_path, expiration_minutes=60)
|
||||
|
||||
# Try thumbnail, fall back to main preview
|
||||
if self.storage.file_exists(thumbnail_path):
|
||||
thumbnail_url = self.storage.generate_signed_url(thumbnail_path, expiration_minutes=60)
|
||||
elif preview_url:
|
||||
thumbnail_url = preview_url
|
||||
|
||||
summaries.append(
|
||||
ThemeSummary(
|
||||
id=theme.id,
|
||||
name=theme.name,
|
||||
description=theme.description,
|
||||
preview_url=preview_url,
|
||||
thumbnail_url=thumbnail_url,
|
||||
is_default=theme.is_default,
|
||||
)
|
||||
)
|
||||
except Exception as e:
|
||||
logger.warning(f"Error processing theme {theme.id}: {e}")
|
||||
# Still include theme but without preview URLs
|
||||
summaries.append(
|
||||
ThemeSummary(
|
||||
id=theme.id,
|
||||
name=theme.name,
|
||||
description=theme.description,
|
||||
is_default=theme.is_default,
|
||||
)
|
||||
)
|
||||
|
||||
return summaries
|
||||
|
||||
def get_theme(self, theme_id: str) -> Optional[ThemeDetail]:
|
||||
"""
|
||||
Get full theme details including style parameters.
|
||||
|
||||
Args:
|
||||
theme_id: The theme identifier
|
||||
|
||||
Returns:
|
||||
ThemeDetail if found, None otherwise
|
||||
"""
|
||||
metadata = self._load_metadata()
|
||||
|
||||
# Find theme in metadata
|
||||
theme_meta = next((t for t in metadata.themes if t.id == theme_id), None)
|
||||
if not theme_meta:
|
||||
logger.warning(f"Theme not found: {theme_id}")
|
||||
return None
|
||||
|
||||
try:
|
||||
# Load style_params.json
|
||||
style_params_path = self._get_theme_path(theme_id, "style_params.json")
|
||||
style_params = self.storage.download_json(style_params_path)
|
||||
|
||||
# Check for YouTube description
|
||||
youtube_desc_path = self._get_theme_path(theme_id, "youtube_description.txt")
|
||||
has_youtube_desc = self.storage.file_exists(youtube_desc_path)
|
||||
|
||||
# Generate preview URL
|
||||
preview_path = self._get_theme_path(theme_id, "preview.png")
|
||||
preview_url = None
|
||||
if self.storage.file_exists(preview_path):
|
||||
preview_url = self.storage.generate_signed_url(preview_path, expiration_minutes=60)
|
||||
|
||||
return ThemeDetail(
|
||||
id=theme_id,
|
||||
name=theme_meta.name,
|
||||
description=theme_meta.description,
|
||||
preview_url=preview_url,
|
||||
is_default=theme_meta.is_default,
|
||||
style_params=style_params,
|
||||
has_youtube_description=has_youtube_desc,
|
||||
)
|
||||
except Exception as e:
|
||||
logger.error(f"Error loading theme {theme_id}: {e}")
|
||||
return None
|
||||
|
||||
def get_theme_style_params(self, theme_id: str) -> Optional[Dict[str, Any]]:
|
||||
"""
|
||||
Get just the style_params.json for a theme.
|
||||
|
||||
Args:
|
||||
theme_id: The theme identifier
|
||||
|
||||
Returns:
|
||||
Style parameters dict if found, None otherwise
|
||||
"""
|
||||
try:
|
||||
style_params_path = self._get_theme_path(theme_id, "style_params.json")
|
||||
return self.storage.download_json(style_params_path)
|
||||
except Exception as e:
|
||||
logger.error(f"Error loading style params for theme {theme_id}: {e}")
|
||||
return None
|
||||
|
||||
def get_youtube_description(self, theme_id: str) -> Optional[str]:
|
||||
"""
|
||||
Get the YouTube description template for a theme.
|
||||
|
||||
Args:
|
||||
theme_id: The theme identifier
|
||||
|
||||
Returns:
|
||||
YouTube description text if found, None otherwise
|
||||
"""
|
||||
try:
|
||||
youtube_desc_path = self._get_theme_path(theme_id, "youtube_description.txt")
|
||||
if not self.storage.file_exists(youtube_desc_path):
|
||||
return None
|
||||
|
||||
# Download to temp file and read
|
||||
with tempfile.NamedTemporaryFile(mode="w", delete=False, suffix=".txt") as tmp:
|
||||
tmp_path = tmp.name
|
||||
|
||||
try:
|
||||
self.storage.download_file(youtube_desc_path, tmp_path)
|
||||
with open(tmp_path, "r") as f:
|
||||
return f.read()
|
||||
finally:
|
||||
if os.path.exists(tmp_path):
|
||||
os.remove(tmp_path)
|
||||
except Exception as e:
|
||||
logger.error(f"Error loading YouTube description for theme {theme_id}: {e}")
|
||||
return None
|
||||
|
||||
def apply_color_overrides(
|
||||
self, style_params: Dict[str, Any], overrides: ColorOverrides
|
||||
) -> Dict[str, Any]:
|
||||
"""
|
||||
Apply user color overrides to theme style parameters.
|
||||
|
||||
Args:
|
||||
style_params: Base style parameters from theme
|
||||
overrides: User color overrides
|
||||
|
||||
Returns:
|
||||
Modified style parameters with overrides applied
|
||||
"""
|
||||
if not overrides.has_overrides():
|
||||
return style_params
|
||||
|
||||
# Deep copy to avoid modifying original
|
||||
result = copy.deepcopy(style_params)
|
||||
|
||||
# Apply artist_color to intro, end, and CDG sections
|
||||
if overrides.artist_color:
|
||||
if "intro" in result:
|
||||
result["intro"]["artist_color"] = overrides.artist_color
|
||||
if "end" in result:
|
||||
result["end"]["artist_color"] = overrides.artist_color
|
||||
if "cdg" in result:
|
||||
result["cdg"]["artist_color"] = overrides.artist_color
|
||||
|
||||
# Apply title_color to intro, end, and CDG sections
|
||||
if overrides.title_color:
|
||||
if "intro" in result:
|
||||
result["intro"]["title_color"] = overrides.title_color
|
||||
if "end" in result:
|
||||
result["end"]["title_color"] = overrides.title_color
|
||||
if "cdg" in result:
|
||||
result["cdg"]["title_color"] = overrides.title_color
|
||||
|
||||
# Apply sung_lyrics_color (karaoke uses RGBA, CDG uses hex)
|
||||
if overrides.sung_lyrics_color:
|
||||
if "karaoke" in result:
|
||||
result["karaoke"]["primary_color"] = hex_to_rgba(overrides.sung_lyrics_color)
|
||||
if "cdg" in result:
|
||||
result["cdg"]["active_fill"] = overrides.sung_lyrics_color
|
||||
|
||||
# Apply unsung_lyrics_color (karaoke uses RGBA, CDG uses hex)
|
||||
if overrides.unsung_lyrics_color:
|
||||
if "karaoke" in result:
|
||||
result["karaoke"]["secondary_color"] = hex_to_rgba(overrides.unsung_lyrics_color)
|
||||
if "cdg" in result:
|
||||
result["cdg"]["inactive_fill"] = overrides.unsung_lyrics_color
|
||||
|
||||
return result
|
||||
|
||||
def prepare_job_style(
|
||||
self,
|
||||
job_id: str,
|
||||
theme_id: str,
|
||||
color_overrides: Optional[ColorOverrides] = None,
|
||||
) -> Tuple[str, Dict[str, str]]:
|
||||
"""
|
||||
Prepare style files for a job by copying theme to job folder.
|
||||
|
||||
This method:
|
||||
1. Loads the theme's style_params.json
|
||||
2. Applies any color overrides
|
||||
3. Updates asset paths to point to theme assets (shared, not copied)
|
||||
4. Uploads modified style_params.json to job folder
|
||||
5. Returns the GCS path and style_assets mapping
|
||||
|
||||
Args:
|
||||
job_id: The job ID
|
||||
theme_id: The theme to use
|
||||
color_overrides: Optional color overrides
|
||||
|
||||
Returns:
|
||||
Tuple of (style_params_gcs_path, style_assets dict)
|
||||
|
||||
Raises:
|
||||
ValueError: If theme not found
|
||||
"""
|
||||
# Load theme style params
|
||||
style_params = self.get_theme_style_params(theme_id)
|
||||
if style_params is None:
|
||||
raise ValueError(f"Theme not found: {theme_id}")
|
||||
|
||||
# Apply color overrides if provided
|
||||
if color_overrides:
|
||||
style_params = self.apply_color_overrides(style_params, color_overrides)
|
||||
|
||||
# Update asset paths to point to theme's shared assets
|
||||
# Theme assets stay in themes/{theme_id}/assets/ - they're shared
|
||||
style_assets = self._build_style_assets_mapping(theme_id, style_params)
|
||||
|
||||
# Update style_params to use theme asset paths (not job-specific)
|
||||
style_params = self._update_asset_paths_in_style(theme_id, style_params)
|
||||
|
||||
# Upload modified style_params.json to job's style folder
|
||||
job_style_path = f"uploads/{job_id}/style/style_params.json"
|
||||
self.storage.upload_json(job_style_path, style_params)
|
||||
|
||||
logger.info(f"Prepared job {job_id} style from theme {theme_id}")
|
||||
|
||||
return job_style_path, style_assets
|
||||
|
||||
def _build_style_assets_mapping(
|
||||
self, theme_id: str, style_params: Dict[str, Any]
|
||||
) -> Dict[str, str]:
|
||||
"""
|
||||
Build the style_assets mapping for a theme.
|
||||
|
||||
Maps asset keys to their GCS paths in the theme folder.
|
||||
"""
|
||||
style_assets = {}
|
||||
theme_assets_prefix = f"{THEMES_PREFIX}/{theme_id}/assets"
|
||||
|
||||
# Check each potential asset type
|
||||
asset_checks = [
|
||||
("intro_background", "intro", "background_image"),
|
||||
("karaoke_background", "karaoke", "background_image"),
|
||||
("end_background", "end", "background_image"),
|
||||
("font", "intro", "font"), # Font is typically shared across sections
|
||||
("cdg_instrumental_background", "cdg", "instrumental_background"),
|
||||
("cdg_title_background", "cdg", "title_screen_background"),
|
||||
("cdg_outro_background", "cdg", "outro_background"),
|
||||
]
|
||||
|
||||
for asset_key, section, field in asset_checks:
|
||||
if section in style_params and field in style_params[section]:
|
||||
value = style_params[section][field]
|
||||
if value and isinstance(value, str):
|
||||
# Check if it's a theme asset path or already has the full path
|
||||
if value.startswith(f"{THEMES_PREFIX}/"):
|
||||
# Already a full theme path
|
||||
style_assets[asset_key] = value
|
||||
elif not value.startswith("/") and not value.startswith("gs://"):
|
||||
# Relative path - prepend theme assets prefix
|
||||
asset_path = f"{theme_assets_prefix}/{os.path.basename(value)}"
|
||||
if self.storage.file_exists(asset_path):
|
||||
style_assets[asset_key] = asset_path
|
||||
|
||||
return style_assets
|
||||
|
||||
def _update_asset_paths_in_style(
|
||||
self, theme_id: str, style_params: Dict[str, Any]
|
||||
) -> Dict[str, Any]:
|
||||
"""
|
||||
Update asset paths in style_params to use full GCS theme paths.
|
||||
|
||||
This ensures the style_loader can find assets when processing.
|
||||
"""
|
||||
result = copy.deepcopy(style_params)
|
||||
theme_assets_prefix = f"{THEMES_PREFIX}/{theme_id}/assets"
|
||||
|
||||
# Fields that contain asset paths
|
||||
path_fields = {
|
||||
"intro": ["background_image", "font"],
|
||||
"end": ["background_image", "font"],
|
||||
"karaoke": ["background_image", "font_path"],
|
||||
"cdg": [
|
||||
"font_path",
|
||||
"instrumental_background",
|
||||
"title_screen_background",
|
||||
"outro_background",
|
||||
],
|
||||
}
|
||||
|
||||
for section, fields in path_fields.items():
|
||||
if section not in result:
|
||||
continue
|
||||
for field in fields:
|
||||
if field not in result[section]:
|
||||
continue
|
||||
value = result[section][field]
|
||||
if value and isinstance(value, str):
|
||||
# Skip if already a full path or URL
|
||||
if value.startswith(f"{THEMES_PREFIX}/") or value.startswith("gs://"):
|
||||
continue
|
||||
# Skip absolute local paths (shouldn't happen but be safe)
|
||||
if value.startswith("/"):
|
||||
continue
|
||||
# Update to full theme asset path
|
||||
result[section][field] = f"{theme_assets_prefix}/{os.path.basename(value)}"
|
||||
|
||||
return result
|
||||
|
||||
def theme_exists(self, theme_id: str) -> bool:
|
||||
"""
|
||||
Check if a theme exists.
|
||||
|
||||
Args:
|
||||
theme_id: The theme identifier
|
||||
|
||||
Returns:
|
||||
True if theme exists, False otherwise
|
||||
"""
|
||||
metadata = self._load_metadata()
|
||||
return any(t.id == theme_id for t in metadata.themes)
|
||||
|
||||
def get_default_theme_id(self) -> Optional[str]:
|
||||
"""
|
||||
Get the ID of the default theme.
|
||||
|
||||
Returns:
|
||||
Default theme ID if one exists, None otherwise
|
||||
"""
|
||||
metadata = self._load_metadata()
|
||||
default_theme = next((t for t in metadata.themes if t.is_default), None)
|
||||
return default_theme.id if default_theme else None
|
||||
|
||||
def invalidate_cache(self) -> None:
|
||||
"""Force invalidation of the metadata cache."""
|
||||
self._metadata_cache = None
|
||||
self._cache_time = None
|
||||
logger.info("Theme metadata cache invalidated")
|
||||
|
||||
|
||||
# Singleton instance with thread-safe initialization
|
||||
_theme_service: Optional[ThemeService] = None
|
||||
_theme_service_lock = threading.Lock()
|
||||
|
||||
|
||||
def get_theme_service() -> ThemeService:
|
||||
"""Get or create the singleton ThemeService instance (thread-safe)."""
|
||||
global _theme_service
|
||||
if _theme_service is None:
|
||||
with _theme_service_lock:
|
||||
# Double-check after acquiring lock
|
||||
if _theme_service is None:
|
||||
_theme_service = ThemeService()
|
||||
return _theme_service
|
||||
@@ -1,543 +0,0 @@
|
||||
"""
|
||||
OpenTelemetry tracing configuration for Google Cloud Trace.
|
||||
|
||||
This module sets up distributed tracing that exports to Google Cloud Trace,
|
||||
providing visibility into request flows, latency, and errors across the
|
||||
entire backend.
|
||||
|
||||
Usage:
|
||||
# In main.py or app initialization
|
||||
from backend.services.tracing import setup_tracing, instrument_app
|
||||
|
||||
setup_tracing(service_name="karaoke-backend")
|
||||
app = FastAPI()
|
||||
instrument_app(app)
|
||||
|
||||
# For custom spans in your code
|
||||
from backend.services.tracing import tracer, create_span
|
||||
|
||||
with tracer.start_as_current_span("my-operation") as span:
|
||||
span.set_attribute("job_id", job_id)
|
||||
# ... do work ...
|
||||
|
||||
# For job-specific spans with automatic job_id attribute
|
||||
from backend.services.tracing import job_span
|
||||
|
||||
with job_span("audio-worker", job_id) as span:
|
||||
# job_id is automatically set as an attribute
|
||||
span.set_attribute("stage", "separation")
|
||||
# ... do work ...
|
||||
|
||||
# For propagating trace context through Cloud Tasks
|
||||
from backend.services.tracing import inject_trace_context, extract_trace_context
|
||||
|
||||
# In worker_service.py (when creating task):
|
||||
headers = inject_trace_context({})
|
||||
|
||||
# In internal.py (when receiving task):
|
||||
context = extract_trace_context(request.headers)
|
||||
with tracer.start_as_current_span("worker", context=context):
|
||||
...
|
||||
"""
|
||||
import os
|
||||
import logging
|
||||
from typing import Optional, Any, Dict
|
||||
from contextlib import contextmanager
|
||||
from functools import wraps
|
||||
|
||||
# OpenTelemetry imports
|
||||
from opentelemetry import trace
|
||||
from opentelemetry.sdk.trace import TracerProvider
|
||||
from opentelemetry.sdk.trace.export import BatchSpanProcessor, SimpleSpanProcessor
|
||||
from opentelemetry.sdk.resources import Resource, SERVICE_NAME, SERVICE_VERSION
|
||||
from opentelemetry.trace import Status, StatusCode, Span
|
||||
from opentelemetry.trace.propagation.tracecontext import TraceContextTextMapPropagator
|
||||
|
||||
# Google Cloud Trace exporter
|
||||
try:
|
||||
from opentelemetry.exporter.cloud_trace import CloudTraceSpanExporter
|
||||
from opentelemetry.resourcedetector.gcp_resource_detector import GoogleCloudResourceDetector
|
||||
GCP_AVAILABLE = True
|
||||
except ImportError:
|
||||
GCP_AVAILABLE = False
|
||||
CloudTraceSpanExporter = None
|
||||
GoogleCloudResourceDetector = None
|
||||
|
||||
# FastAPI instrumentation
|
||||
try:
|
||||
from opentelemetry.instrumentation.fastapi import FastAPIInstrumentor
|
||||
FASTAPI_INSTRUMENTATION_AVAILABLE = True
|
||||
except ImportError:
|
||||
FASTAPI_INSTRUMENTATION_AVAILABLE = False
|
||||
FastAPIInstrumentor = None
|
||||
|
||||
# HTTPX instrumentation (for outgoing HTTP calls)
|
||||
try:
|
||||
from opentelemetry.instrumentation.httpx import HTTPXClientInstrumentation
|
||||
HTTPX_INSTRUMENTATION_AVAILABLE = True
|
||||
except ImportError:
|
||||
HTTPX_INSTRUMENTATION_AVAILABLE = False
|
||||
HTTPXClientInstrumentation = None
|
||||
|
||||
# Logging instrumentation
|
||||
try:
|
||||
from opentelemetry.instrumentation.logging import LoggingInstrumentor
|
||||
LOGGING_INSTRUMENTATION_AVAILABLE = True
|
||||
except ImportError:
|
||||
LOGGING_INSTRUMENTATION_AVAILABLE = False
|
||||
LoggingInstrumentor = None
|
||||
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
# Global tracer instance
|
||||
_tracer: Optional[trace.Tracer] = None
|
||||
_initialized = False
|
||||
|
||||
|
||||
def setup_tracing(
|
||||
service_name: str = "karaoke-backend",
|
||||
service_version: str = "0.71.7",
|
||||
enable_in_dev: bool = False,
|
||||
) -> bool:
|
||||
"""
|
||||
Initialize OpenTelemetry tracing with Google Cloud Trace exporter.
|
||||
|
||||
Args:
|
||||
service_name: Name of the service (appears in Cloud Trace)
|
||||
service_version: Version of the service
|
||||
enable_in_dev: Whether to enable tracing in development (default False)
|
||||
|
||||
Returns:
|
||||
True if tracing was initialized, False if skipped
|
||||
"""
|
||||
global _tracer, _initialized
|
||||
|
||||
if _initialized:
|
||||
logger.debug("Tracing already initialized")
|
||||
return True
|
||||
|
||||
# Check if we should enable tracing
|
||||
# In Cloud Run, GOOGLE_CLOUD_PROJECT is always set
|
||||
is_cloud_run = os.environ.get("K_SERVICE") is not None
|
||||
gcp_project = os.environ.get("GOOGLE_CLOUD_PROJECT") or os.environ.get("GCP_PROJECT")
|
||||
|
||||
if not is_cloud_run and not enable_in_dev:
|
||||
logger.info("Tracing disabled in development (set enable_in_dev=True to enable)")
|
||||
_tracer = trace.get_tracer(service_name)
|
||||
return False
|
||||
|
||||
if not GCP_AVAILABLE:
|
||||
logger.warning("Google Cloud Trace exporter not available, tracing disabled")
|
||||
_tracer = trace.get_tracer(service_name)
|
||||
return False
|
||||
|
||||
logger.info(f"Initializing OpenTelemetry tracing for {service_name} v{service_version}")
|
||||
|
||||
try:
|
||||
# Create resource with service info and GCP resource detection
|
||||
resource_attributes = {
|
||||
SERVICE_NAME: service_name,
|
||||
SERVICE_VERSION: service_version,
|
||||
}
|
||||
|
||||
# Detect GCP resources (project, region, instance, etc.)
|
||||
if GoogleCloudResourceDetector and is_cloud_run:
|
||||
try:
|
||||
gcp_resource = GoogleCloudResourceDetector().detect()
|
||||
resource = Resource.create(resource_attributes).merge(gcp_resource)
|
||||
logger.info("Detected GCP resources for tracing")
|
||||
except Exception as e:
|
||||
logger.warning(f"Could not detect GCP resources: {e}")
|
||||
resource = Resource.create(resource_attributes)
|
||||
else:
|
||||
resource = Resource.create(resource_attributes)
|
||||
|
||||
# Create tracer provider
|
||||
provider = TracerProvider(resource=resource)
|
||||
|
||||
# Add Cloud Trace exporter
|
||||
if CloudTraceSpanExporter and gcp_project:
|
||||
try:
|
||||
exporter = CloudTraceSpanExporter(project_id=gcp_project)
|
||||
# Use BatchSpanProcessor for production (better performance)
|
||||
processor = BatchSpanProcessor(exporter)
|
||||
provider.add_span_processor(processor)
|
||||
logger.info(f"Cloud Trace exporter configured for project: {gcp_project}")
|
||||
except Exception as e:
|
||||
logger.warning(f"Could not configure Cloud Trace exporter: {e}")
|
||||
|
||||
# Set the global tracer provider
|
||||
trace.set_tracer_provider(provider)
|
||||
|
||||
# Get tracer instance
|
||||
_tracer = trace.get_tracer(service_name, service_version)
|
||||
|
||||
# Instrument logging to include trace context
|
||||
if LOGGING_INSTRUMENTATION_AVAILABLE and LoggingInstrumentor:
|
||||
try:
|
||||
LoggingInstrumentor().instrument(set_logging_format=True)
|
||||
logger.info("Logging instrumentation enabled")
|
||||
except Exception as e:
|
||||
logger.warning(f"Could not instrument logging: {e}")
|
||||
|
||||
# Instrument HTTPX for outgoing requests
|
||||
if HTTPX_INSTRUMENTATION_AVAILABLE and HTTPXClientInstrumentation:
|
||||
try:
|
||||
HTTPXClientInstrumentation().instrument()
|
||||
logger.info("HTTPX instrumentation enabled")
|
||||
except Exception as e:
|
||||
logger.warning(f"Could not instrument HTTPX: {e}")
|
||||
|
||||
_initialized = True
|
||||
logger.info("OpenTelemetry tracing initialized successfully")
|
||||
return True
|
||||
|
||||
except Exception as e:
|
||||
logger.error(f"Failed to initialize tracing: {e}")
|
||||
_tracer = trace.get_tracer(service_name)
|
||||
return False
|
||||
|
||||
|
||||
def instrument_app(app) -> None:
|
||||
"""
|
||||
Instrument a FastAPI application with automatic tracing.
|
||||
|
||||
This adds spans for all incoming requests, including:
|
||||
- Request method and path
|
||||
- Response status code
|
||||
- Request duration
|
||||
- Error information
|
||||
|
||||
Args:
|
||||
app: FastAPI application instance
|
||||
"""
|
||||
if not FASTAPI_INSTRUMENTATION_AVAILABLE or not FastAPIInstrumentor:
|
||||
logger.warning("FastAPI instrumentation not available")
|
||||
return
|
||||
|
||||
try:
|
||||
FastAPIInstrumentor.instrument_app(
|
||||
app,
|
||||
excluded_urls="health,healthz,ready,readiness,ping",
|
||||
)
|
||||
logger.info("FastAPI instrumentation enabled")
|
||||
except Exception as e:
|
||||
logger.warning(f"Could not instrument FastAPI: {e}")
|
||||
|
||||
|
||||
def get_tracer() -> trace.Tracer:
|
||||
"""
|
||||
Get the global tracer instance.
|
||||
|
||||
Returns:
|
||||
Tracer instance (creates a no-op tracer if not initialized)
|
||||
"""
|
||||
global _tracer
|
||||
if _tracer is None:
|
||||
_tracer = trace.get_tracer("karaoke-backend")
|
||||
return _tracer
|
||||
|
||||
|
||||
# Convenience alias - lazily forwards attribute access to get_tracer()
|
||||
class _TracerProxy:
|
||||
"""Proxy that lazily returns the global tracer."""
|
||||
def __getattr__(self, name):
|
||||
return getattr(get_tracer(), name)
|
||||
|
||||
tracer = _TracerProxy()
|
||||
|
||||
|
||||
@contextmanager
|
||||
def create_span(
|
||||
name: str,
|
||||
attributes: Optional[Dict[str, Any]] = None,
|
||||
kind: trace.SpanKind = trace.SpanKind.INTERNAL,
|
||||
):
|
||||
"""
|
||||
Create a traced span as a context manager.
|
||||
|
||||
Usage:
|
||||
with create_span("process-lyrics", {"job_id": job_id}) as span:
|
||||
# ... do work ...
|
||||
span.set_attribute("words_processed", 500)
|
||||
|
||||
Args:
|
||||
name: Span name (appears in Cloud Trace)
|
||||
attributes: Initial span attributes
|
||||
kind: Span kind (INTERNAL, SERVER, CLIENT, PRODUCER, CONSUMER)
|
||||
|
||||
Yields:
|
||||
The active span
|
||||
"""
|
||||
tracer = get_tracer()
|
||||
with tracer.start_as_current_span(name, kind=kind) as span:
|
||||
if attributes:
|
||||
for key, value in attributes.items():
|
||||
span.set_attribute(key, value)
|
||||
try:
|
||||
yield span
|
||||
except Exception as e:
|
||||
span.set_status(Status(StatusCode.ERROR, str(e)))
|
||||
span.record_exception(e)
|
||||
raise
|
||||
|
||||
|
||||
def traced(
|
||||
name: Optional[str] = None,
|
||||
attributes: Optional[Dict[str, Any]] = None,
|
||||
):
|
||||
"""
|
||||
Decorator to trace a function.
|
||||
|
||||
Usage:
|
||||
@traced("process-correction")
|
||||
def process_correction(job_id: str):
|
||||
...
|
||||
|
||||
@traced(attributes={"operation": "add-lyrics"})
|
||||
async def add_lyrics(job_id: str, source: str):
|
||||
...
|
||||
|
||||
Args:
|
||||
name: Span name (defaults to function name)
|
||||
attributes: Static attributes to add to span
|
||||
"""
|
||||
def decorator(func):
|
||||
span_name = name or func.__name__
|
||||
|
||||
@wraps(func)
|
||||
async def async_wrapper(*args, **kwargs):
|
||||
with create_span(span_name, attributes) as span:
|
||||
# Add function arguments as attributes
|
||||
span.set_attribute("function", func.__name__)
|
||||
return await func(*args, **kwargs)
|
||||
|
||||
@wraps(func)
|
||||
def sync_wrapper(*args, **kwargs):
|
||||
with create_span(span_name, attributes) as span:
|
||||
span.set_attribute("function", func.__name__)
|
||||
return func(*args, **kwargs)
|
||||
|
||||
# Return appropriate wrapper based on function type
|
||||
import asyncio
|
||||
if asyncio.iscoroutinefunction(func):
|
||||
return async_wrapper
|
||||
return sync_wrapper
|
||||
|
||||
return decorator
|
||||
|
||||
|
||||
def add_span_attribute(key: str, value: Any) -> None:
|
||||
"""
|
||||
Add an attribute to the current span.
|
||||
|
||||
Args:
|
||||
key: Attribute name
|
||||
value: Attribute value (must be string, int, float, bool, or list thereof)
|
||||
"""
|
||||
span = trace.get_current_span()
|
||||
if span and span.is_recording():
|
||||
span.set_attribute(key, value)
|
||||
|
||||
|
||||
def add_span_event(name: str, attributes: Optional[Dict[str, Any]] = None) -> None:
|
||||
"""
|
||||
Add an event to the current span.
|
||||
|
||||
Events are timestamped annotations that appear in the trace timeline.
|
||||
|
||||
Args:
|
||||
name: Event name
|
||||
attributes: Event attributes
|
||||
"""
|
||||
span = trace.get_current_span()
|
||||
if span and span.is_recording():
|
||||
span.add_event(name, attributes=attributes or {})
|
||||
|
||||
|
||||
def set_span_error(error: Exception) -> None:
|
||||
"""
|
||||
Mark the current span as errored.
|
||||
|
||||
Args:
|
||||
error: The exception that occurred
|
||||
"""
|
||||
span = trace.get_current_span()
|
||||
if span and span.is_recording():
|
||||
span.set_status(Status(StatusCode.ERROR, str(error)))
|
||||
span.record_exception(error)
|
||||
|
||||
|
||||
def get_current_trace_id() -> Optional[str]:
|
||||
"""
|
||||
Get the current trace ID for correlation.
|
||||
|
||||
This can be used to link logs to traces.
|
||||
|
||||
Returns:
|
||||
Trace ID as hex string, or None if no active trace
|
||||
"""
|
||||
span = trace.get_current_span()
|
||||
if span:
|
||||
ctx = span.get_span_context()
|
||||
if ctx.is_valid:
|
||||
return format(ctx.trace_id, '032x')
|
||||
return None
|
||||
|
||||
|
||||
def get_current_span_id() -> Optional[str]:
|
||||
"""
|
||||
Get the current span ID.
|
||||
|
||||
Returns:
|
||||
Span ID as hex string, or None if no active span
|
||||
"""
|
||||
span = trace.get_current_span()
|
||||
if span:
|
||||
ctx = span.get_span_context()
|
||||
if ctx.is_valid:
|
||||
return format(ctx.span_id, '016x')
|
||||
return None
|
||||
|
||||
|
||||
# =====================================================
|
||||
# Job-Aware Tracing Utilities
|
||||
# =====================================================
|
||||
|
||||
@contextmanager
|
||||
def job_span(
|
||||
name: str,
|
||||
job_id: str,
|
||||
attributes: Optional[Dict[str, Any]] = None,
|
||||
kind: trace.SpanKind = trace.SpanKind.INTERNAL,
|
||||
):
|
||||
"""
|
||||
Create a traced span with job_id automatically set as an attribute.
|
||||
|
||||
This is the preferred way to create spans in job processing workers.
|
||||
The job_id attribute enables filtering spans by job in Cloud Trace.
|
||||
|
||||
Usage:
|
||||
with job_span("audio-separation", job_id) as span:
|
||||
span.set_attribute("stage", "clean_instrumental")
|
||||
# ... do work ...
|
||||
|
||||
Args:
|
||||
name: Span name (appears in Cloud Trace)
|
||||
job_id: Job ID to attach to the span
|
||||
attributes: Additional span attributes
|
||||
kind: Span kind (INTERNAL, SERVER, CLIENT, PRODUCER, CONSUMER)
|
||||
|
||||
Yields:
|
||||
The active span
|
||||
"""
|
||||
all_attributes = {
|
||||
"job_id": job_id,
|
||||
"service.operation": name,
|
||||
}
|
||||
if attributes:
|
||||
all_attributes.update(attributes)
|
||||
|
||||
with create_span(name, all_attributes, kind) as span:
|
||||
yield span
|
||||
|
||||
|
||||
def inject_trace_context(headers: Dict[str, str]) -> Dict[str, str]:
|
||||
"""
|
||||
Inject current trace context into HTTP headers.
|
||||
|
||||
Use this when creating Cloud Tasks or making HTTP calls to propagate
|
||||
the trace context so child spans link to the parent trace.
|
||||
|
||||
Usage:
|
||||
headers = inject_trace_context({
|
||||
"Content-Type": "application/json",
|
||||
"Authorization": "Bearer token",
|
||||
})
|
||||
# Use headers in HTTP request or Cloud Task
|
||||
|
||||
Args:
|
||||
headers: Existing headers dict to add trace context to
|
||||
|
||||
Returns:
|
||||
Headers dict with traceparent/tracestate headers added
|
||||
"""
|
||||
propagator = TraceContextTextMapPropagator()
|
||||
carrier = dict(headers) # Make a copy
|
||||
propagator.inject(carrier)
|
||||
return carrier
|
||||
|
||||
|
||||
def extract_trace_context(headers: Dict[str, str]) -> Optional[trace.Context]:
|
||||
"""
|
||||
Extract trace context from HTTP headers.
|
||||
|
||||
Use this when receiving Cloud Tasks or HTTP requests to continue
|
||||
the trace from the parent span.
|
||||
|
||||
Usage:
|
||||
context = extract_trace_context(request.headers)
|
||||
with tracer.start_as_current_span("worker", context=context):
|
||||
# ... processing ...
|
||||
|
||||
Args:
|
||||
headers: HTTP headers containing traceparent/tracestate
|
||||
|
||||
Returns:
|
||||
Context object for use with start_as_current_span, or None
|
||||
"""
|
||||
propagator = TraceContextTextMapPropagator()
|
||||
return propagator.extract(headers)
|
||||
|
||||
|
||||
def start_span_with_context(
|
||||
name: str,
|
||||
context: Optional[trace.Context],
|
||||
job_id: Optional[str] = None,
|
||||
attributes: Optional[Dict[str, Any]] = None,
|
||||
):
|
||||
"""
|
||||
Start a span as child of the given context (or current context if None).
|
||||
|
||||
This is useful for starting worker spans that should link to the
|
||||
trace from the original API request.
|
||||
|
||||
Usage:
|
||||
context = extract_trace_context(request.headers)
|
||||
with start_span_with_context("audio-worker", context, job_id) as span:
|
||||
# ... processing ...
|
||||
|
||||
Args:
|
||||
name: Span name
|
||||
context: Parent context (from extract_trace_context) or None
|
||||
job_id: Optional job ID to set as attribute
|
||||
attributes: Additional span attributes
|
||||
|
||||
Returns:
|
||||
Context manager yielding the span
|
||||
"""
|
||||
tracer = get_tracer()
|
||||
|
||||
all_attributes = attributes or {}
|
||||
if job_id:
|
||||
all_attributes["job_id"] = job_id
|
||||
all_attributes["service.operation"] = name
|
||||
|
||||
@contextmanager
|
||||
def _span():
|
||||
kwargs = {"kind": trace.SpanKind.INTERNAL}
|
||||
if context is not None:
|
||||
kwargs["context"] = context
|
||||
|
||||
with tracer.start_as_current_span(name, **kwargs) as span:
|
||||
for key, value in all_attributes.items():
|
||||
span.set_attribute(key, value)
|
||||
try:
|
||||
yield span
|
||||
except Exception as e:
|
||||
span.set_status(Status(StatusCode.ERROR, str(e)))
|
||||
span.record_exception(e)
|
||||
raise
|
||||
|
||||
return _span()
|
||||
@@ -1,439 +0,0 @@
|
||||
"""
|
||||
Worker coordination service.
|
||||
|
||||
Handles triggering and coordinating background workers.
|
||||
Supports two modes:
|
||||
- Cloud Tasks (production): Guaranteed delivery, automatic retries, horizontal scaling
|
||||
- Direct HTTP (development): Faster iteration, simpler debugging
|
||||
|
||||
SOLID Principles:
|
||||
- Single Responsibility: Only handles worker coordination
|
||||
- Dependency Inversion: Depends on HTTP abstraction, not implementation
|
||||
- Open/Closed: Can add new workers without modifying existing code
|
||||
|
||||
Observability:
|
||||
- Propagates trace context through Cloud Tasks for distributed tracing
|
||||
- All worker invocations create spans linked to original request trace
|
||||
"""
|
||||
import logging
|
||||
import os
|
||||
import json
|
||||
from typing import Optional
|
||||
import httpx
|
||||
|
||||
from backend.config import get_settings
|
||||
from backend.services.tracing import inject_trace_context
|
||||
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
|
||||
# Mapping of worker types to their Cloud Tasks queue names
|
||||
# These queues are created by Pulumi in infrastructure/__main__.py
|
||||
WORKER_QUEUES = {
|
||||
"audio": "audio-worker-queue",
|
||||
"lyrics": "lyrics-worker-queue",
|
||||
"screens": "screens-worker-queue",
|
||||
"render-video": "render-worker-queue",
|
||||
"video": "video-worker-queue",
|
||||
}
|
||||
|
||||
|
||||
class WorkerService:
|
||||
"""
|
||||
Service for coordinating background workers.
|
||||
|
||||
Supports two execution modes controlled by ENABLE_CLOUD_TASKS env var:
|
||||
|
||||
1. Cloud Tasks mode (production, ENABLE_CLOUD_TASKS=true):
|
||||
- Tasks enqueued to Cloud Tasks for guaranteed delivery
|
||||
- Automatic retries on failure
|
||||
- Each task runs in dedicated Cloud Run instance
|
||||
- Rate limiting to protect external APIs (Modal, AudioShake)
|
||||
|
||||
2. Direct HTTP mode (development, ENABLE_CLOUD_TASKS=false):
|
||||
- Direct HTTP POST to internal worker endpoints
|
||||
- Worker runs as FastAPI BackgroundTask
|
||||
- Faster iteration for local development
|
||||
- No retry guarantees
|
||||
"""
|
||||
|
||||
def __init__(self):
|
||||
"""Initialize worker service."""
|
||||
self.settings = get_settings()
|
||||
self._base_url = self._get_base_url()
|
||||
self._admin_token = self._get_admin_token()
|
||||
self._use_cloud_tasks = self._should_use_cloud_tasks()
|
||||
self._tasks_client = None
|
||||
|
||||
if self._use_cloud_tasks:
|
||||
logger.info("WorkerService initialized with Cloud Tasks mode")
|
||||
else:
|
||||
logger.info("WorkerService initialized with direct HTTP mode")
|
||||
|
||||
def _should_use_cloud_tasks(self) -> bool:
|
||||
"""
|
||||
Check if Cloud Tasks should be used for worker coordination.
|
||||
|
||||
Controlled by ENABLE_CLOUD_TASKS setting (from environment variable).
|
||||
Default is false (direct HTTP mode) for backward compatibility.
|
||||
|
||||
Returns:
|
||||
True if Cloud Tasks should be used, False for direct HTTP
|
||||
"""
|
||||
return self.settings.enable_cloud_tasks
|
||||
|
||||
@property
|
||||
def tasks_client(self):
|
||||
"""
|
||||
Lazy-initialize Cloud Tasks client.
|
||||
|
||||
Only created when needed to avoid import overhead in development mode.
|
||||
"""
|
||||
if self._tasks_client is None and self._use_cloud_tasks:
|
||||
try:
|
||||
from google.cloud import tasks_v2
|
||||
self._tasks_client = tasks_v2.CloudTasksClient()
|
||||
except ImportError:
|
||||
logger.error(
|
||||
"google-cloud-tasks not installed. "
|
||||
"Install with: pip install google-cloud-tasks"
|
||||
)
|
||||
raise
|
||||
return self._tasks_client
|
||||
|
||||
def _get_admin_token(self) -> Optional[str]:
|
||||
"""
|
||||
Get admin token for internal API authentication.
|
||||
|
||||
Returns the first admin token from ADMIN_TOKENS env var.
|
||||
"""
|
||||
admin_tokens_str = self.settings.admin_tokens or ""
|
||||
tokens = [t.strip() for t in admin_tokens_str.split(",") if t.strip()]
|
||||
if tokens:
|
||||
return tokens[0]
|
||||
return None
|
||||
|
||||
def _get_base_url(self) -> str:
|
||||
"""
|
||||
Get base URL for internal API calls.
|
||||
|
||||
Priority:
|
||||
1. TEST_SERVER_URL env var (for tests)
|
||||
2. CLOUD_RUN_SERVICE_URL env var (for Cloud Tasks to call back)
|
||||
3. localhost with PORT env var (for development)
|
||||
|
||||
Returns:
|
||||
Base URL for API calls
|
||||
"""
|
||||
# Check for test environment override
|
||||
test_url = os.getenv("TEST_SERVER_URL")
|
||||
if test_url:
|
||||
return test_url
|
||||
|
||||
# Production: Cloud Run service URL
|
||||
# This must be the publicly accessible URL for Cloud Tasks to call
|
||||
service_url = os.getenv("CLOUD_RUN_SERVICE_URL")
|
||||
if service_url:
|
||||
return service_url
|
||||
|
||||
# Development mode: use localhost with PORT env var
|
||||
port = os.getenv("PORT", "8000")
|
||||
return f"http://localhost:{port}"
|
||||
|
||||
async def trigger_worker(
|
||||
self,
|
||||
worker_type: str,
|
||||
job_id: str,
|
||||
timeout_seconds: int = 30
|
||||
) -> bool:
|
||||
"""
|
||||
Trigger a background worker for a job.
|
||||
|
||||
In production (ENABLE_CLOUD_TASKS=true):
|
||||
- Enqueues task to Cloud Tasks queue
|
||||
- Cloud Tasks delivers HTTP request to internal endpoint
|
||||
- Automatic retries on failure with exponential backoff
|
||||
- Rate limiting protects external APIs
|
||||
|
||||
In development (ENABLE_CLOUD_TASKS=false):
|
||||
- Direct HTTP POST to internal endpoint
|
||||
- Faster iteration, but no retry guarantees
|
||||
- Worker runs in same container as API
|
||||
|
||||
Args:
|
||||
worker_type: Worker type ("audio", "lyrics", "screens", "render-video", "video")
|
||||
job_id: Job ID to process
|
||||
timeout_seconds: Request timeout (for direct HTTP mode)
|
||||
|
||||
Returns:
|
||||
True if trigger successful (or task enqueued), False otherwise
|
||||
"""
|
||||
if self._use_cloud_tasks:
|
||||
return await self._enqueue_cloud_task(worker_type, job_id)
|
||||
else:
|
||||
return await self._trigger_http(worker_type, job_id, timeout_seconds)
|
||||
|
||||
async def _enqueue_cloud_task(self, worker_type: str, job_id: str) -> bool:
|
||||
"""
|
||||
Enqueue task to Cloud Tasks for guaranteed delivery.
|
||||
|
||||
The task will be delivered as an HTTP POST to the internal worker endpoint.
|
||||
Cloud Tasks handles:
|
||||
- Retry on failure (with exponential backoff)
|
||||
- Rate limiting (max dispatches per second)
|
||||
- Deduplication (via task name if needed)
|
||||
- OIDC authentication for Cloud Run
|
||||
|
||||
Observability:
|
||||
- Injects trace context headers so worker spans link to parent trace
|
||||
|
||||
Args:
|
||||
worker_type: Worker type
|
||||
job_id: Job ID to process
|
||||
|
||||
Returns:
|
||||
True if task enqueued successfully, False otherwise
|
||||
"""
|
||||
try:
|
||||
from google.cloud import tasks_v2
|
||||
|
||||
queue_name = WORKER_QUEUES.get(worker_type)
|
||||
if not queue_name:
|
||||
logger.error(f"Unknown worker type: {worker_type}")
|
||||
return False
|
||||
|
||||
project = self.settings.google_cloud_project
|
||||
if not project:
|
||||
logger.error("GOOGLE_CLOUD_PROJECT not set, cannot enqueue Cloud Task")
|
||||
return False
|
||||
|
||||
location = self.settings.gcp_region # Must match queue location
|
||||
|
||||
# Build queue path
|
||||
parent = self.tasks_client.queue_path(project, location, queue_name)
|
||||
|
||||
# Build base headers
|
||||
headers = {
|
||||
"Content-Type": "application/json",
|
||||
}
|
||||
|
||||
# Add admin auth via custom header
|
||||
# NOTE: We use X-Admin-Token instead of Authorization because Cloud Tasks
|
||||
# OIDC token overwrites the Authorization header when oidc_token is specified.
|
||||
# The OIDC token handles Cloud Run authentication (allows Cloud Tasks to invoke
|
||||
# the service), while X-Admin-Token handles application-level authentication.
|
||||
if self._admin_token:
|
||||
headers["X-Admin-Token"] = self._admin_token
|
||||
logger.debug(
|
||||
f"[job:{job_id}] Using admin token for Cloud Task auth via X-Admin-Token, "
|
||||
f"token prefix: {self._admin_token[:8]}..., len={len(self._admin_token)}"
|
||||
)
|
||||
|
||||
# Inject trace context for distributed tracing
|
||||
# This allows worker spans to link back to the original request trace
|
||||
headers = inject_trace_context(headers)
|
||||
|
||||
# Build task payload
|
||||
task = {
|
||||
"http_request": {
|
||||
"http_method": tasks_v2.HttpMethod.POST,
|
||||
"url": f"{self._base_url}/api/internal/workers/{worker_type}",
|
||||
"headers": headers,
|
||||
"body": json.dumps({"job_id": job_id}).encode(),
|
||||
# Use OIDC token for Cloud Run authentication
|
||||
# This allows Cloud Tasks to invoke the Cloud Run service
|
||||
"oidc_token": {
|
||||
"service_account_email": f"karaoke-backend@{project}.iam.gserviceaccount.com",
|
||||
},
|
||||
},
|
||||
}
|
||||
|
||||
# Create task
|
||||
# Note: We don't set a task name, allowing Cloud Tasks to generate one
|
||||
# This prevents duplicate task errors on retries
|
||||
response = self.tasks_client.create_task(parent=parent, task=task)
|
||||
logger.info(
|
||||
f"[job:{job_id}] Created Cloud Task for {worker_type} worker: {response.name}"
|
||||
)
|
||||
return True
|
||||
|
||||
except Exception as e:
|
||||
logger.error(
|
||||
f"[job:{job_id}] Failed to enqueue Cloud Task for {worker_type}: {e}",
|
||||
exc_info=True
|
||||
)
|
||||
return False
|
||||
|
||||
async def _trigger_http(
|
||||
self,
|
||||
worker_type: str,
|
||||
job_id: str,
|
||||
timeout_seconds: int = 30
|
||||
) -> bool:
|
||||
"""
|
||||
Trigger worker via direct HTTP call (development mode).
|
||||
|
||||
This is the original implementation - direct HTTP POST to internal endpoint.
|
||||
The endpoint adds the worker function to FastAPI BackgroundTasks.
|
||||
|
||||
Observability:
|
||||
- Injects trace context headers so worker spans link to parent trace
|
||||
|
||||
Args:
|
||||
worker_type: Worker type
|
||||
job_id: Job ID to process
|
||||
timeout_seconds: Request timeout
|
||||
|
||||
Returns:
|
||||
True if trigger successful, False otherwise
|
||||
"""
|
||||
try:
|
||||
# Build headers with admin auth token
|
||||
headers = {}
|
||||
if self._admin_token:
|
||||
headers["Authorization"] = f"Bearer {self._admin_token}"
|
||||
|
||||
# Inject trace context for distributed tracing
|
||||
headers = inject_trace_context(headers)
|
||||
|
||||
async with httpx.AsyncClient(timeout=timeout_seconds) as client:
|
||||
url = f"{self._base_url}/api/internal/workers/{worker_type}"
|
||||
|
||||
response = await client.post(
|
||||
url,
|
||||
json={"job_id": job_id},
|
||||
headers=headers
|
||||
)
|
||||
|
||||
if response.status_code == 200:
|
||||
logger.info(f"[job:{job_id}] Successfully triggered {worker_type} worker")
|
||||
return True
|
||||
else:
|
||||
logger.error(
|
||||
f"[job:{job_id}] Failed to trigger {worker_type} worker: "
|
||||
f"HTTP {response.status_code} - {response.text}"
|
||||
)
|
||||
return False
|
||||
|
||||
except httpx.TimeoutException:
|
||||
logger.error(f"[job:{job_id}] Timeout triggering {worker_type} worker")
|
||||
return False
|
||||
|
||||
except Exception as e:
|
||||
logger.error(
|
||||
f"[job:{job_id}] Error triggering {worker_type} worker: {e}",
|
||||
exc_info=True
|
||||
)
|
||||
return False
|
||||
|
||||
# Convenience methods for specific workers
|
||||
# These provide a cleaner API and better IDE autocomplete
|
||||
|
||||
async def trigger_audio_worker(self, job_id: str) -> bool:
|
||||
"""Trigger audio separation worker."""
|
||||
return await self.trigger_worker("audio", job_id)
|
||||
|
||||
async def trigger_lyrics_worker(self, job_id: str) -> bool:
|
||||
"""Trigger lyrics transcription worker."""
|
||||
return await self.trigger_worker("lyrics", job_id)
|
||||
|
||||
async def trigger_screens_worker(self, job_id: str) -> bool:
|
||||
"""Trigger screen generation worker."""
|
||||
return await self.trigger_worker("screens", job_id)
|
||||
|
||||
async def trigger_video_worker(self, job_id: str) -> bool:
|
||||
"""
|
||||
Trigger video generation worker.
|
||||
|
||||
When USE_CLOUD_RUN_JOBS_FOR_VIDEO=true and ENABLE_CLOUD_TASKS=true,
|
||||
uses Cloud Run Jobs for execution (supports >30 min encoding).
|
||||
Otherwise, uses Cloud Tasks or direct HTTP.
|
||||
"""
|
||||
if self._use_cloud_tasks and self.settings.use_cloud_run_jobs_for_video:
|
||||
return await self._trigger_cloud_run_job(job_id)
|
||||
return await self.trigger_worker("video", job_id)
|
||||
|
||||
async def _trigger_cloud_run_job(self, job_id: str) -> bool:
|
||||
"""
|
||||
Trigger video encoding as a Cloud Run Job.
|
||||
|
||||
Cloud Run Jobs support up to 24 hours of execution time,
|
||||
making them suitable for very long video encoding tasks.
|
||||
|
||||
Args:
|
||||
job_id: Job ID to process
|
||||
|
||||
Returns:
|
||||
True if job was triggered successfully, False otherwise
|
||||
"""
|
||||
try:
|
||||
from google.cloud import run_v2
|
||||
|
||||
project = self.settings.google_cloud_project
|
||||
if not project:
|
||||
logger.error("GOOGLE_CLOUD_PROJECT not set, cannot trigger Cloud Run Job")
|
||||
return False
|
||||
|
||||
location = self.settings.gcp_region
|
||||
job_name = f"projects/{project}/locations/{location}/jobs/video-encoding-job"
|
||||
|
||||
# Create Cloud Run Jobs client
|
||||
client = run_v2.JobsClient()
|
||||
|
||||
# Run the job with overrides for the specific job_id
|
||||
request = run_v2.RunJobRequest(
|
||||
name=job_name,
|
||||
overrides=run_v2.RunJobRequest.Overrides(
|
||||
container_overrides=[
|
||||
run_v2.RunJobRequest.Overrides.ContainerOverride(
|
||||
args=["python", "-m", "backend.workers.video_worker", "--job-id", job_id],
|
||||
)
|
||||
]
|
||||
)
|
||||
)
|
||||
|
||||
# Run the job (async operation)
|
||||
operation = client.run_job(request=request)
|
||||
logger.info(f"Started Cloud Run Job for video encoding, job {job_id}: {operation.metadata}")
|
||||
return True
|
||||
|
||||
except Exception as e:
|
||||
logger.error(
|
||||
f"Failed to trigger Cloud Run Job for video/{job_id}: {e}",
|
||||
exc_info=True
|
||||
)
|
||||
return False
|
||||
|
||||
async def trigger_render_video_worker(self, job_id: str) -> bool:
|
||||
"""Trigger render video worker (post-review)."""
|
||||
return await self.trigger_worker("render-video", job_id)
|
||||
|
||||
|
||||
# Global worker service instance
|
||||
_worker_service: Optional[WorkerService] = None
|
||||
|
||||
|
||||
def get_worker_service() -> WorkerService:
|
||||
"""
|
||||
Get global worker service instance.
|
||||
|
||||
Singleton pattern to reuse HTTP client pool.
|
||||
|
||||
Returns:
|
||||
WorkerService instance
|
||||
"""
|
||||
global _worker_service
|
||||
if _worker_service is None:
|
||||
_worker_service = WorkerService()
|
||||
return _worker_service
|
||||
|
||||
|
||||
def reset_worker_service() -> None:
|
||||
"""
|
||||
Reset the global worker service instance.
|
||||
|
||||
Used in tests to ensure clean state between test cases.
|
||||
"""
|
||||
global _worker_service
|
||||
_worker_service = None
|
||||
@@ -1,112 +0,0 @@
|
||||
"""
|
||||
YouTube credential service for server-side video uploads.
|
||||
|
||||
This service manages YouTube OAuth credentials stored in Secret Manager
|
||||
for non-interactive uploads from backend workers.
|
||||
"""
|
||||
import json
|
||||
import logging
|
||||
from typing import Optional, Dict, Any
|
||||
|
||||
from backend.config import get_settings
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
|
||||
class YouTubeService:
|
||||
"""Service for managing YouTube OAuth credentials."""
|
||||
|
||||
# Secret Manager secret name for YouTube credentials
|
||||
YOUTUBE_CREDENTIALS_SECRET = "youtube-oauth-credentials"
|
||||
|
||||
def __init__(self):
|
||||
self.settings = get_settings()
|
||||
self._credentials: Optional[Dict[str, Any]] = None
|
||||
self._loaded = False
|
||||
|
||||
def load_credentials(self) -> bool:
|
||||
"""
|
||||
Load YouTube OAuth credentials from Secret Manager.
|
||||
|
||||
The secret should contain a JSON object with:
|
||||
- token: Current access token (may be expired)
|
||||
- refresh_token: Refresh token for getting new access tokens
|
||||
- token_uri: Token endpoint URL
|
||||
- client_id: OAuth client ID
|
||||
- client_secret: OAuth client secret
|
||||
- scopes: List of OAuth scopes
|
||||
|
||||
Returns:
|
||||
True if credentials were loaded successfully, False otherwise
|
||||
"""
|
||||
if self._loaded:
|
||||
return self._credentials is not None
|
||||
|
||||
try:
|
||||
# Get credentials from Secret Manager
|
||||
creds_json = self.settings.get_secret(self.YOUTUBE_CREDENTIALS_SECRET)
|
||||
|
||||
if not creds_json:
|
||||
logger.warning("YouTube credentials not found in Secret Manager")
|
||||
self._loaded = True
|
||||
return False
|
||||
|
||||
# Parse the credentials JSON
|
||||
self._credentials = json.loads(creds_json)
|
||||
|
||||
# Validate required fields
|
||||
required_fields = ['refresh_token', 'token_uri', 'client_id', 'client_secret']
|
||||
missing = [f for f in required_fields if not self._credentials.get(f)]
|
||||
|
||||
if missing:
|
||||
logger.error(f"YouTube credentials missing required fields: {missing}")
|
||||
self._credentials = None
|
||||
self._loaded = True
|
||||
return False
|
||||
|
||||
logger.info("YouTube credentials loaded successfully from Secret Manager")
|
||||
self._loaded = True
|
||||
return True
|
||||
|
||||
except json.JSONDecodeError as e:
|
||||
logger.error(f"Failed to parse YouTube credentials JSON: {e}")
|
||||
self._loaded = True
|
||||
return False
|
||||
except Exception as e:
|
||||
logger.error(f"Failed to load YouTube credentials: {e}")
|
||||
self._loaded = True
|
||||
return False
|
||||
|
||||
def get_credentials_dict(self) -> Optional[Dict[str, Any]]:
|
||||
"""
|
||||
Get the YouTube credentials as a dictionary.
|
||||
|
||||
This format is compatible with KaraokeFinalise's user_youtube_credentials
|
||||
parameter, which creates a google.oauth2.credentials.Credentials object.
|
||||
|
||||
Returns:
|
||||
Dictionary with credential data, or None if not available
|
||||
"""
|
||||
if not self._loaded:
|
||||
self.load_credentials()
|
||||
|
||||
return self._credentials
|
||||
|
||||
@property
|
||||
def is_configured(self) -> bool:
|
||||
"""Check if YouTube credentials are configured and ready to use."""
|
||||
if not self._loaded:
|
||||
self.load_credentials()
|
||||
return self._credentials is not None
|
||||
|
||||
|
||||
# Singleton instance
|
||||
_youtube_service: Optional[YouTubeService] = None
|
||||
|
||||
|
||||
def get_youtube_service() -> YouTubeService:
|
||||
"""Get the singleton YouTube service instance."""
|
||||
global _youtube_service
|
||||
if _youtube_service is None:
|
||||
_youtube_service = YouTubeService()
|
||||
return _youtube_service
|
||||
@@ -1,4 +0,0 @@
|
||||
"""
|
||||
Backend integration tests package.
|
||||
"""
|
||||
|
||||
@@ -1,218 +0,0 @@
|
||||
"""
|
||||
Shared pytest fixtures for backend tests.
|
||||
|
||||
Provides common mocks and test utilities across all test modules.
|
||||
|
||||
NOTE: Module-level mocks are NOT applied here to allow emulator tests to work.
|
||||
Individual tests must mock dependencies as needed.
|
||||
"""
|
||||
import os
|
||||
import pytest
|
||||
from unittest.mock import Mock, MagicMock, AsyncMock, patch
|
||||
from datetime import datetime, UTC
|
||||
from fastapi.testclient import TestClient
|
||||
|
||||
# Mock google.auth.default AND firestore for unit tests if not using emulator
|
||||
# This prevents DefaultCredentialsError and FirestoreClient initialization during imports
|
||||
if 'FIRESTORE_EMULATOR_HOST' not in os.environ:
|
||||
from unittest.mock import MagicMock
|
||||
|
||||
# Set up test environment variables BEFORE importing any backend modules
|
||||
os.environ.setdefault('ADMIN_TOKENS', 'test-admin-token')
|
||||
os.environ.setdefault('GOOGLE_CLOUD_PROJECT', 'test-project')
|
||||
os.environ.setdefault('GCS_BUCKET_NAME', 'test-bucket')
|
||||
os.environ.setdefault('FIRESTORE_COLLECTION', 'jobs')
|
||||
os.environ.setdefault('ENVIRONMENT', 'test')
|
||||
|
||||
# Mock google.auth.default - prevents credential errors
|
||||
try:
|
||||
import google.auth
|
||||
mock_credentials = MagicMock()
|
||||
mock_credentials.token = 'fake-token'
|
||||
mock_credentials.valid = True
|
||||
mock_credentials.universe_domain = 'googleapis.com' # Required by google-cloud-storage
|
||||
mock_credentials.project_id = 'test-project'
|
||||
google.auth.default = MagicMock(return_value=(mock_credentials, 'test-project'))
|
||||
except ImportError:
|
||||
# If google.auth not installed, that's ok for some tests
|
||||
pass
|
||||
|
||||
# Mock google.cloud.firestore.Client - prevents Firestore initialization
|
||||
# This is CRITICAL: when AuthService is instantiated, it creates FirestoreService
|
||||
# which tries to create a real Firestore client. The client returns MagicMocks
|
||||
# which fail when used as enum values (e.g., UserType).
|
||||
try:
|
||||
import google.cloud.firestore as firestore_module
|
||||
|
||||
# Create a mock client that returns proper values for get_token
|
||||
mock_firestore_client = MagicMock()
|
||||
mock_collection = MagicMock()
|
||||
mock_doc = MagicMock()
|
||||
mock_snapshot = MagicMock()
|
||||
|
||||
# For auth token lookups, return None (token not found in Firestore)
|
||||
# This forces validation to use admin tokens from environment
|
||||
mock_snapshot.exists = False
|
||||
mock_snapshot.to_dict.return_value = None
|
||||
mock_doc.get.return_value = mock_snapshot
|
||||
mock_collection.document.return_value = mock_doc
|
||||
mock_firestore_client.collection.return_value = mock_collection
|
||||
|
||||
# Replace the Client class
|
||||
original_client = firestore_module.Client
|
||||
firestore_module.Client = MagicMock(return_value=mock_firestore_client)
|
||||
except ImportError:
|
||||
pass
|
||||
|
||||
# Mock google.cloud.storage.Client - prevents GCS initialization
|
||||
try:
|
||||
import google.cloud.storage as storage_module
|
||||
mock_storage_client = MagicMock()
|
||||
mock_bucket = MagicMock()
|
||||
mock_storage_client.bucket.return_value = mock_bucket
|
||||
storage_module.Client = MagicMock(return_value=mock_storage_client)
|
||||
except ImportError:
|
||||
pass
|
||||
|
||||
from backend.models.job import Job, JobStatus, JobCreate
|
||||
|
||||
|
||||
@pytest.fixture(autouse=True)
|
||||
def mock_auth_dependency(request):
|
||||
"""Mock the require_auth dependency for all tests using FastAPI's dependency override system."""
|
||||
# Skip for emulator tests and integration tests which use real auth
|
||||
test_path = str(request.fspath)
|
||||
if 'emulator' in test_path or 'integration' in test_path:
|
||||
yield
|
||||
return
|
||||
|
||||
# Skip if FIRESTORE_EMULATOR_HOST is set (running in emulator environment)
|
||||
import os
|
||||
if os.environ.get('FIRESTORE_EMULATOR_HOST'):
|
||||
yield
|
||||
return
|
||||
|
||||
from backend.services.auth_service import UserType
|
||||
from backend.api.dependencies import require_auth, require_admin, require_review_auth, require_instrumental_auth
|
||||
from backend.main import app
|
||||
|
||||
# Create mock auth functions
|
||||
async def mock_require_auth():
|
||||
"""Mock require_auth to always return valid admin credentials."""
|
||||
return ("test-admin-token", UserType.ADMIN, 999)
|
||||
|
||||
async def mock_require_admin():
|
||||
"""Mock require_admin to always return valid admin credentials."""
|
||||
return ("test-admin-token", UserType.ADMIN, 999)
|
||||
|
||||
async def mock_require_review_auth(job_id: str = "test123"):
|
||||
"""Mock require_review_auth to always return valid review access."""
|
||||
return (job_id, "full")
|
||||
|
||||
async def mock_require_instrumental_auth(job_id: str = "test123"):
|
||||
"""Mock require_instrumental_auth to always return valid instrumental review access."""
|
||||
return (job_id, "full")
|
||||
|
||||
# Use FastAPI's dependency override system
|
||||
app.dependency_overrides[require_auth] = mock_require_auth
|
||||
app.dependency_overrides[require_admin] = mock_require_admin
|
||||
app.dependency_overrides[require_review_auth] = mock_require_review_auth
|
||||
app.dependency_overrides[require_instrumental_auth] = mock_require_instrumental_auth
|
||||
|
||||
yield
|
||||
|
||||
# Clean up after test
|
||||
app.dependency_overrides.clear()
|
||||
|
||||
|
||||
@pytest.fixture
|
||||
def auth_headers():
|
||||
"""Get authentication headers for test requests."""
|
||||
return {
|
||||
"Authorization": "Bearer test-admin-token",
|
||||
"Content-Type": "application/json"
|
||||
}
|
||||
|
||||
|
||||
@pytest.fixture
|
||||
def mock_firestore():
|
||||
"""Mock Firestore client for unit tests."""
|
||||
# Mock at module level for unit tests
|
||||
with patch('google.cloud.firestore.Client') as mock_cls:
|
||||
client = MagicMock()
|
||||
mock_cls.return_value = client
|
||||
yield client
|
||||
|
||||
|
||||
@pytest.fixture
|
||||
def mock_storage_client():
|
||||
"""Mock GCS Storage client for unit tests."""
|
||||
# Mock at module level for unit tests
|
||||
with patch('google.cloud.storage.Client') as mock_cls:
|
||||
client = MagicMock()
|
||||
mock_cls.return_value = client
|
||||
yield client
|
||||
|
||||
|
||||
@pytest.fixture
|
||||
def mock_httpx_client():
|
||||
"""Mock httpx AsyncClient for worker service."""
|
||||
with patch('backend.services.worker_service.httpx.AsyncClient') as mock:
|
||||
client = AsyncMock()
|
||||
mock.return_value.__aenter__.return_value = client
|
||||
yield client
|
||||
|
||||
|
||||
@pytest.fixture
|
||||
def sample_job():
|
||||
"""Create a sample Job instance for testing."""
|
||||
return Job(
|
||||
job_id="test123",
|
||||
status=JobStatus.PENDING,
|
||||
progress=0,
|
||||
created_at=datetime.now(UTC),
|
||||
updated_at=datetime.now(UTC),
|
||||
artist="Test Artist",
|
||||
title="Test Song",
|
||||
url="https://youtube.com/watch?v=test"
|
||||
)
|
||||
|
||||
|
||||
@pytest.fixture
|
||||
def sample_job_create():
|
||||
"""Create a sample JobCreate instance for testing."""
|
||||
return JobCreate(
|
||||
url="https://youtube.com/watch?v=test",
|
||||
artist="Test Artist",
|
||||
title="Test Song"
|
||||
)
|
||||
|
||||
|
||||
@pytest.fixture
|
||||
def test_client():
|
||||
"""Create a FastAPI TestClient."""
|
||||
# Import here to avoid circular dependency
|
||||
from backend.main import app
|
||||
return TestClient(app)
|
||||
|
||||
|
||||
def create_mock_job(**kwargs):
|
||||
"""
|
||||
Factory function to create mock Job instances with custom fields.
|
||||
|
||||
Args:
|
||||
**kwargs: Job fields to override defaults
|
||||
|
||||
Returns:
|
||||
Job instance with specified fields
|
||||
"""
|
||||
defaults = {
|
||||
"job_id": "test123",
|
||||
"status": JobStatus.PENDING,
|
||||
"progress": 0,
|
||||
"created_at": datetime.now(UTC),
|
||||
"updated_at": datetime.now(UTC)
|
||||
}
|
||||
defaults.update(kwargs)
|
||||
return Job(**defaults)
|
||||
|
||||
@@ -1,7 +0,0 @@
|
||||
"""
|
||||
Emulator integration tests.
|
||||
|
||||
These tests run against local GCP emulators (Firestore, GCS)
|
||||
to provide true integration testing without cloud costs.
|
||||
"""
|
||||
|
||||
@@ -1,88 +0,0 @@
|
||||
"""
|
||||
Fixtures for emulator integration tests.
|
||||
|
||||
These tests use REAL Firestore and GCS emulators, so we DON'T mock google.cloud modules.
|
||||
Run with: scripts/run-emulator-tests.sh
|
||||
"""
|
||||
import pytest
|
||||
import os
|
||||
import requests
|
||||
from unittest.mock import AsyncMock, patch
|
||||
|
||||
|
||||
def emulators_running() -> bool:
|
||||
"""Check if GCP emulators are running."""
|
||||
try:
|
||||
# Check Firestore emulator
|
||||
requests.get("http://127.0.0.1:8080", timeout=1)
|
||||
# Check GCS emulator
|
||||
requests.get("http://127.0.0.1:4443", timeout=1)
|
||||
return True
|
||||
except (requests.exceptions.ConnectionError, requests.exceptions.Timeout):
|
||||
return False
|
||||
|
||||
|
||||
# Skip all tests in this module if emulators aren't running
|
||||
pytestmark = pytest.mark.skipif(
|
||||
not emulators_running(),
|
||||
reason="GCP emulators not running. Start with: scripts/start-emulators.sh"
|
||||
)
|
||||
|
||||
# Set emulator environment variables (use 127.0.0.1 to force IPv4)
|
||||
os.environ["FIRESTORE_EMULATOR_HOST"] = "127.0.0.1:8080"
|
||||
os.environ["STORAGE_EMULATOR_HOST"] = "http://127.0.0.1:4443"
|
||||
os.environ["GOOGLE_CLOUD_PROJECT"] = "test-project"
|
||||
os.environ["GCS_BUCKET_NAME"] = "test-bucket"
|
||||
os.environ["FIRESTORE_COLLECTION"] = "test-jobs"
|
||||
os.environ["ENVIRONMENT"] = "test"
|
||||
os.environ["ADMIN_TOKENS"] = "test-admin-token"
|
||||
|
||||
# Only import app if emulators are running
|
||||
if emulators_running():
|
||||
from fastapi.testclient import TestClient
|
||||
from backend.main import app
|
||||
else:
|
||||
TestClient = None
|
||||
app = None
|
||||
|
||||
|
||||
@pytest.fixture(scope="session", autouse=True)
|
||||
def setup_gcs_bucket():
|
||||
"""Create GCS bucket in emulator before all tests."""
|
||||
try:
|
||||
response = requests.post(
|
||||
"http://127.0.0.1:4443/storage/v1/b",
|
||||
json={"name": "test-bucket"},
|
||||
params={"project": "test-project"},
|
||||
timeout=5
|
||||
)
|
||||
if response.status_code in [200, 409]:
|
||||
print("✅ GCS bucket 'test-bucket' ready in emulator")
|
||||
except Exception as e:
|
||||
print(f"⚠️ GCS bucket setup failed: {e}")
|
||||
yield
|
||||
|
||||
|
||||
@pytest.fixture(scope="session")
|
||||
def mock_worker_service():
|
||||
"""Mock worker service to prevent background tasks."""
|
||||
with patch("backend.api.routes.jobs.worker_service") as mock:
|
||||
mock.trigger_audio_worker = AsyncMock(return_value=True)
|
||||
mock.trigger_lyrics_worker = AsyncMock(return_value=True)
|
||||
mock.trigger_screens_worker = AsyncMock(return_value=True)
|
||||
mock.trigger_video_worker = AsyncMock(return_value=True)
|
||||
yield mock
|
||||
|
||||
|
||||
@pytest.fixture(scope="session")
|
||||
def client(mock_worker_service):
|
||||
"""Create FastAPI test client with mocked workers."""
|
||||
with patch("backend.api.routes.file_upload.worker_service", mock_worker_service):
|
||||
return TestClient(app)
|
||||
|
||||
|
||||
@pytest.fixture
|
||||
def auth_headers():
|
||||
"""Auth headers for testing."""
|
||||
return {"Authorization": "Bearer test-admin-token"}
|
||||
|
||||
File diff suppressed because it is too large.
Load diff
@@ -1,356 +0,0 @@
|
||||
"""
|
||||
Integration tests using local GCP emulators.
|
||||
|
||||
These tests use real Firestore and GCS emulators running locally,
|
||||
providing true integration testing without cloud resources or costs.
|
||||
|
||||
Run with: ./scripts/run-emulator-tests.sh
|
||||
|
||||
NOTE: These tests mock background workers to avoid race conditions.
|
||||
The fixtures are defined in conftest.py in this directory.
|
||||
"""
|
||||
import pytest
|
||||
import time
|
||||
|
||||
from .conftest import emulators_running
|
||||
|
||||
|
||||
# Skip all tests in this module if emulators aren't running
|
||||
pytestmark = pytest.mark.skipif(
|
||||
not emulators_running(),
|
||||
reason="GCP emulators not running. Start with: scripts/start-emulators.sh"
|
||||
)
|
||||
|
||||
# Fixtures are loaded from conftest.py in this directory
|
||||
|
||||
|
||||
class TestEmulatorBasics:
|
||||
"""Basic emulator connectivity tests."""
|
||||
|
||||
def test_health_endpoint(self, client, auth_headers):
|
||||
"""Test health endpoint works."""
|
||||
response = client.get("/api/health", )
|
||||
assert response.status_code == 200
|
||||
assert response.json()["status"] == "healthy"
|
||||
|
||||
def test_root_endpoint(self, client, auth_headers):
|
||||
"""Test root endpoint works."""
|
||||
response = client.get("/", )
|
||||
assert response.status_code == 200
|
||||
assert response.json()["service"] == "karaoke-gen-backend"
|
||||
|
||||
|
||||
class TestJobCreation:
|
||||
"""Test job creation with Firestore emulator."""
|
||||
|
||||
def test_create_job_simple(self, client, auth_headers):
|
||||
"""Test creating a simple job."""
|
||||
response = client.post(
|
||||
"/api/jobs",
|
||||
headers=auth_headers,
|
||||
json={"url": "https://youtube.com/watch?v=test123"}
|
||||
)
|
||||
|
||||
assert response.status_code == 200
|
||||
data = response.json()
|
||||
assert data["status"] == "success"
|
||||
assert "job_id" in data
|
||||
|
||||
def test_create_job_with_metadata(self, client, auth_headers):
|
||||
"""Test creating a job with artist/title."""
|
||||
response = client.post(
|
||||
"/api/jobs",
|
||||
headers=auth_headers,
|
||||
json={
|
||||
"url": "https://youtube.com/watch?v=test",
|
||||
"artist": "Test Artist",
|
||||
"title": "Test Song"
|
||||
}
|
||||
)
|
||||
|
||||
assert response.status_code == 200
|
||||
data = response.json()
|
||||
assert "job_id" in data
|
||||
|
||||
|
||||
class TestJobRetrieval:
|
||||
"""Test job retrieval from Firestore emulator."""
|
||||
|
||||
def test_create_and_get_job(self, client, auth_headers):
|
||||
"""Test creating and then retrieving a job."""
|
||||
# Create
|
||||
create_resp = client.post(
|
||||
"/api/jobs",
|
||||
headers=auth_headers,
|
||||
json={
|
||||
"url": "https://youtube.com/watch?v=abc123",
|
||||
"artist": "Test Artist",
|
||||
"title": "Test Song"
|
||||
}
|
||||
)
|
||||
assert create_resp.status_code == 200
|
||||
job_id = create_resp.json()["job_id"]
|
||||
|
||||
# Small delay for emulator consistency
|
||||
time.sleep(0.2)
|
||||
|
||||
# Retrieve
|
||||
get_resp = client.get(f"/api/jobs/{job_id}", headers=auth_headers)
|
||||
|
||||
if get_resp.status_code != 200:
|
||||
print(f"GET failed: {get_resp.status_code} - {get_resp.text}")
|
||||
|
||||
assert get_resp.status_code == 200
|
||||
|
||||
job = get_resp.json()
|
||||
assert job["job_id"] == job_id
|
||||
assert job["status"] == "pending"
|
||||
assert job["artist"] == "Test Artist"
|
||||
assert job["title"] == "Test Song"
|
||||
|
||||
def test_get_nonexistent_job(self, client, auth_headers):
|
||||
"""Test fetching a job that doesn't exist."""
|
||||
response = client.get("/api/jobs/nonexistent-id", headers=auth_headers)
|
||||
assert response.status_code == 404
|
||||
|
||||
|
||||
class TestJobList:
|
||||
"""Test listing jobs from Firestore."""
|
||||
|
||||
def test_list_jobs(self, client, auth_headers):
|
||||
"""Test listing jobs."""
|
||||
# Create a few jobs
|
||||
for i in range(3):
|
||||
client.post(
|
||||
"/api/jobs",
|
||||
headers=auth_headers,
|
||||
json={"url": f"https://youtube.com/watch?v=list{i}"}
|
||||
)
|
||||
|
||||
time.sleep(0.2)
|
||||
|
||||
# List
|
||||
response = client.get("/api/jobs", headers=auth_headers)
|
||||
assert response.status_code == 200
|
||||
|
||||
jobs = response.json()
|
||||
assert isinstance(jobs, list)
|
||||
# We should have at least the 3 we just created
|
||||
assert len(jobs) >= 3
|
||||
|
||||
|
||||
class TestJobDeletion:
|
||||
"""Test job deletion."""
|
||||
|
||||
def test_delete_job(self, client, auth_headers):
|
||||
"""Test deleting a job."""
|
||||
# Create
|
||||
create_resp = client.post(
|
||||
"/api/jobs",
|
||||
headers=auth_headers,
|
||||
json={"url": "https://youtube.com/watch?v=delete-me"}
|
||||
)
|
||||
job_id = create_resp.json()["job_id"]
|
||||
|
||||
time.sleep(0.2)
|
||||
|
||||
# Delete
|
||||
del_resp = client.delete(f"/api/jobs/{job_id}", headers=auth_headers)
|
||||
assert del_resp.status_code == 200
|
||||
|
||||
# Verify deleted
|
||||
get_resp = client.get(f"/api/jobs/{job_id}", headers=auth_headers)
|
||||
assert get_resp.status_code == 404
|
||||
|
||||
|
||||
class TestJobUpdates:
|
||||
"""Test job status updates."""
|
||||
|
||||
def test_cancel_job(self, client, auth_headers):
|
||||
"""Test cancelling a job."""
|
||||
# Create
|
||||
create_resp = client.post(
|
||||
"/api/jobs",
|
||||
headers=auth_headers,
|
||||
json={"url": "https://youtube.com/watch?v=cancel-me"}
|
||||
)
|
||||
job_id = create_resp.json()["job_id"]
|
||||
|
||||
time.sleep(0.2)
|
||||
|
||||
# Cancel
|
||||
cancel_resp = client.post(
|
||||
f"/api/jobs/{job_id}/cancel",
|
||||
headers=auth_headers,
|
||||
json={"reason": "test cancellation"}
|
||||
)
|
||||
assert cancel_resp.status_code == 200
|
||||
|
||||
# Verify cancelled
|
||||
get_resp = client.get(f"/api/jobs/{job_id}", headers=auth_headers)
|
||||
assert get_resp.status_code == 200
|
||||
job = get_resp.json()
|
||||
assert job["status"] == "cancelled"
|
||||
|
||||
|
||||
class TestFileUpload:
|
||||
"""Test file upload with GCS emulator."""
|
||||
|
||||
def test_upload_file(self, client, auth_headers):
|
||||
"""Test uploading a file."""
|
||||
test_file_content = b"fake audio data for testing"
|
||||
|
||||
response = client.post(
|
||||
"/api/jobs/upload",
|
||||
headers={"Authorization": auth_headers["Authorization"]},
|
||||
files={"file": ("test.flac", test_file_content, "audio/flac")},
|
||||
data={"artist": "Upload Artist", "title": "Upload Song"}
|
||||
)
|
||||
|
||||
assert response.status_code == 200
|
||||
data = response.json()
|
||||
assert data["status"] == "success"
|
||||
assert "job_id" in data
|
||||
|
||||
job_id = data["job_id"]
|
||||
|
||||
time.sleep(0.2)
|
||||
|
||||
# Verify job created with upload data
|
||||
get_resp = client.get(f"/api/jobs/{job_id}", headers=auth_headers)
|
||||
assert get_resp.status_code == 200
|
||||
job = get_resp.json()
|
||||
assert job["artist"] == "Upload Artist"
|
||||
assert job["title"] == "Upload Song"
|
||||
assert "input_media_gcs_path" in job
|
||||
|
||||
|
||||
class TestInternalEndpoints:
|
||||
"""Test internal worker endpoints."""
|
||||
|
||||
def test_internal_workers_exist(self, client, auth_headers):
|
||||
"""Test that internal worker endpoints exist and respond."""
|
||||
# Create a job first
|
||||
create_resp = client.post(
|
||||
"/api/jobs",
|
||||
headers=auth_headers,
|
||||
json={"url": "https://youtube.com/watch?v=worker-test"}
|
||||
)
|
||||
job_id = create_resp.json()["job_id"]
|
||||
|
||||
time.sleep(0.2)
|
||||
|
||||
# Test audio worker endpoint exists
|
||||
response = client.post(
|
||||
"/api/internal/workers/audio",
|
||||
headers=auth_headers,
|
||||
json={"job_id": job_id}
|
||||
)
|
||||
assert response.status_code == 200
|
||||
assert response.json()["status"] == "started"
|
||||
|
||||
|
||||
class TestReviewEndpoints:
|
||||
"""Test lyrics review API endpoints."""
|
||||
|
||||
def test_review_ping(self, client, auth_headers):
|
||||
"""Test review ping endpoint."""
|
||||
# Create a job first
|
||||
create_resp = client.post(
|
||||
"/api/jobs",
|
||||
headers=auth_headers,
|
||||
json={"url": "https://youtube.com/watch?v=review-test"}
|
||||
)
|
||||
job_id = create_resp.json()["job_id"]
|
||||
|
||||
# Ping should work for any job
|
||||
response = client.get(f"/api/review/{job_id}/ping")
|
||||
assert response.status_code == 200
|
||||
assert response.json()["status"] == "ok"
|
||||
|
||||
def test_review_correction_data_wrong_status(self, client, auth_headers):
|
||||
"""Test that correction-data returns error for non-review jobs."""
|
||||
# Create a job (starts in pending status)
|
||||
create_resp = client.post(
|
||||
"/api/jobs",
|
||||
headers=auth_headers,
|
||||
json={"url": "https://youtube.com/watch?v=review-status-test"}
|
||||
)
|
||||
job_id = create_resp.json()["job_id"]
|
||||
|
||||
time.sleep(0.2)
|
||||
|
||||
# Should fail - job not in AWAITING_REVIEW status
|
||||
response = client.get(f"/api/review/{job_id}/correction-data", headers=auth_headers)
|
||||
assert response.status_code == 400
|
||||
assert "not ready for review" in response.json()["detail"].lower()
|
||||
|
||||
def test_review_audio_no_job(self, client, auth_headers):
|
||||
"""Test audio endpoint returns 404 for nonexistent job."""
|
||||
response = client.get("/api/review/nonexistent-job/audio/", headers=auth_headers)
|
||||
assert response.status_code == 404
|
||||
|
||||
def test_review_preview_video_stub(self, client, auth_headers):
|
||||
"""Test preview video endpoint exists."""
|
||||
# Create a job
|
||||
create_resp = client.post(
|
||||
"/api/jobs",
|
||||
headers=auth_headers,
|
||||
json={"url": "https://youtube.com/watch?v=preview-test"}
|
||||
)
|
||||
job_id = create_resp.json()["job_id"]
|
||||
|
||||
# Preview video should return error since job not ready
|
||||
response = client.post(
|
||||
f"/api/review/{job_id}/preview-video",
|
||||
headers=auth_headers,
|
||||
json={"corrections": [], "corrected_segments": []}
|
||||
)
|
||||
# Should return 400 (job not in AWAITING_REVIEW state)
|
||||
assert response.status_code == 400
|
||||
|
||||
def test_review_annotations_stub(self, client, auth_headers):
|
||||
"""Test annotations endpoint (stub)."""
|
||||
create_resp = client.post(
|
||||
"/api/jobs",
|
||||
headers=auth_headers,
|
||||
json={"url": "https://youtube.com/watch?v=annotation-test"}
|
||||
)
|
||||
job_id = create_resp.json()["job_id"]
|
||||
|
||||
# Annotations endpoint should accept data (even if it just logs it)
|
||||
response = client.post(
|
||||
f"/api/review/{job_id}/v1/annotations",
|
||||
headers=auth_headers,
|
||||
json={"type": "test", "data": "test annotation"}
|
||||
)
|
||||
assert response.status_code == 200
|
||||
assert response.json()["status"] == "success"
|
||||
|
||||
|
||||
class TestRenderVideoWorker:
|
||||
"""Test render video worker endpoint."""
|
||||
|
||||
def test_render_video_worker_endpoint_exists(self, client, auth_headers):
|
||||
"""Test that render-video worker endpoint exists."""
|
||||
# Create a job
|
||||
create_resp = client.post(
|
||||
"/api/jobs",
|
||||
headers=auth_headers,
|
||||
json={"url": "https://youtube.com/watch?v=render-test"}
|
||||
)
|
||||
job_id = create_resp.json()["job_id"]
|
||||
|
||||
time.sleep(0.2)
|
||||
|
||||
# Test render-video worker endpoint exists
|
||||
response = client.post(
|
||||
"/api/internal/workers/render-video",
|
||||
headers=auth_headers,
|
||||
json={"job_id": job_id}
|
||||
)
|
||||
# Should return 200 (endpoint exists and starts)
|
||||
assert response.status_code == 200
|
||||
|
||||
|
||||
print("✅ Emulator integration tests ready to run")
|
||||
@@ -1,436 +0,0 @@
|
||||
"""
|
||||
Direct tests for render_video_worker style loading.
|
||||
|
||||
Tests the fix for styles being loaded from job.style_params_gcs_path
|
||||
instead of the incorrect job.state_data['styles_gcs_path'].
|
||||
|
||||
Run with:
|
||||
./scripts/start-emulators.sh
|
||||
pytest backend/tests/emulator/test_style_loading_direct.py -v
|
||||
"""
|
||||
import pytest
|
||||
import json
|
||||
import os
|
||||
import tempfile
|
||||
import requests
|
||||
from dataclasses import dataclass
|
||||
from typing import Dict, Optional
|
||||
|
||||
|
||||
def emulators_running() -> bool:
|
||||
"""Check if GCP emulators are running."""
|
||||
try:
|
||||
requests.get("http://127.0.0.1:8080", timeout=1)
|
||||
requests.get("http://127.0.0.1:4443", timeout=1)
|
||||
return True
|
||||
except (requests.exceptions.ConnectionError, requests.exceptions.Timeout):
|
||||
return False
|
||||
|
||||
|
||||
# Skip all tests in this module if emulators aren't running
|
||||
pytestmark = pytest.mark.skipif(
|
||||
not emulators_running(),
|
||||
reason="GCP emulators not running. Start with: scripts/start-emulators.sh"
|
||||
)
|
||||
|
||||
# Set up environment for emulators
|
||||
os.environ["FIRESTORE_EMULATOR_HOST"] = "127.0.0.1:8080"
|
||||
os.environ["STORAGE_EMULATOR_HOST"] = "http://127.0.0.1:4443"
|
||||
os.environ["GOOGLE_CLOUD_PROJECT"] = "test-project"
|
||||
os.environ["GCS_BUCKET_NAME"] = "test-bucket"
|
||||
|
||||
|
||||
@dataclass
|
||||
class MockJob:
|
||||
"""Mock job object for testing style loading."""
|
||||
job_id: str
|
||||
style_params_gcs_path: Optional[str] = None
|
||||
style_assets: Dict[str, str] = None
|
||||
state_data: Dict = None
|
||||
|
||||
def __post_init__(self):
|
||||
if self.style_assets is None:
|
||||
self.style_assets = {}
|
||||
if self.state_data is None:
|
||||
self.state_data = {}
|
||||
|
||||
|
||||
class MockStorageService:
|
||||
"""Mock storage service that uses GCS emulator."""
|
||||
|
||||
def __init__(self):
|
||||
from google.cloud import storage
|
||||
self.client = storage.Client()
|
||||
self.bucket_name = "test-bucket"
|
||||
self._ensure_bucket_exists()
|
||||
|
||||
def _ensure_bucket_exists(self):
|
||||
"""Create bucket in emulator if it doesn't exist."""
|
||||
try:
|
||||
self.client.create_bucket(self.bucket_name)
|
||||
except Exception:
|
||||
pass # Bucket already exists
|
||||
|
||||
def upload_string(self, content: str, gcs_path: str, content_type: str = "application/json"):
|
||||
"""Upload string content to GCS."""
|
||||
bucket = self.client.bucket(self.bucket_name)
|
||||
blob = bucket.blob(gcs_path)
|
||||
blob.upload_from_string(content, content_type=content_type)
|
||||
|
||||
def upload_bytes(self, content: bytes, gcs_path: str, content_type: str = "application/octet-stream"):
|
||||
"""Upload bytes to GCS."""
|
||||
bucket = self.client.bucket(self.bucket_name)
|
||||
blob = bucket.blob(gcs_path)
|
||||
blob.upload_from_string(content, content_type=content_type)
|
||||
|
||||
def download_file(self, gcs_path: str, local_path: str):
|
||||
"""Download file from GCS to local path."""
|
||||
bucket = self.client.bucket(self.bucket_name)
|
||||
blob = bucket.blob(gcs_path)
|
||||
blob.download_to_filename(local_path)
|
||||
|
||||
def file_exists(self, gcs_path: str) -> bool:
|
||||
"""Check if file exists in GCS."""
|
||||
bucket = self.client.bucket(self.bucket_name)
|
||||
blob = bucket.blob(gcs_path)
|
||||
return blob.exists()
|
||||
|
||||
|
||||
class TestStyleLoadingFix:
|
||||
"""
|
||||
Tests for the render_video_worker style loading fix.
|
||||
|
||||
The bug was: render_video_worker looked for styles at
|
||||
job.state_data.get('styles_gcs_path') but styles are actually stored at
|
||||
job.style_params_gcs_path and job.style_assets.
|
||||
"""
|
||||
|
||||
@pytest.fixture(autouse=True)
|
||||
def setup(self):
|
||||
"""Set up storage service and temp directory for each test."""
|
||||
self.storage = MockStorageService()
|
||||
self.temp_dir = tempfile.mkdtemp()
|
||||
yield
|
||||
# Cleanup
|
||||
import shutil
|
||||
shutil.rmtree(self.temp_dir, ignore_errors=True)
|
||||
|
||||
def _import_get_or_create_styles(self):
|
||||
"""
|
||||
Import _get_or_create_styles without importing the full worker module.
|
||||
We copy the function here to avoid import issues with lyrics_transcriber.
|
||||
"""
|
||||
# We'll test using a copy of the function logic
|
||||
# This avoids importing the full worker which has lyrics_transcriber dependency
|
||||
pass
|
||||
|
||||
def test_style_loading_from_correct_location(self):
|
||||
"""
|
||||
Test that styles are loaded from job.style_params_gcs_path,
|
||||
NOT from job.state_data['styles_gcs_path'].
|
||||
"""
|
||||
import time
|
||||
job_id = f"test-style-{int(time.time() * 1000)}"
|
||||
|
||||
# Create style JSON with placeholder paths
|
||||
style_json = {
|
||||
"intro": {
|
||||
"background_image": "/original/path/intro_bg.png",
|
||||
"font": "/original/path/font.ttf"
|
||||
},
|
||||
"karaoke": {
|
||||
"background_image": "/original/path/karaoke_bg.png",
|
||||
"font_path": "/original/path/font.ttf",
|
||||
"font_size": 100
|
||||
},
|
||||
"end": {
|
||||
"background_image": "/original/path/end_bg.png",
|
||||
"font": "/original/path/font.ttf"
|
||||
}
|
||||
}
|
||||
|
||||
# Upload style JSON to GCS (the CORRECT location)
|
||||
style_gcs_path = f"uploads/{job_id}/style/style_params.json"
|
||||
self.storage.upload_string(json.dumps(style_json), style_gcs_path)
|
||||
|
||||
# Upload mock assets
|
||||
assets = {
|
||||
"intro_background": f"uploads/{job_id}/style/intro_background.png",
|
||||
"karaoke_background": f"uploads/{job_id}/style/karaoke_background.png",
|
||||
"end_background": f"uploads/{job_id}/style/end_background.png",
|
||||
"font": f"uploads/{job_id}/style/font.ttf",
|
||||
}
|
||||
|
||||
for asset_key, gcs_path in assets.items():
|
||||
# Upload fake file content
|
||||
self.storage.upload_bytes(b"fake image/font data", gcs_path)
|
||||
|
||||
# Create mock job with styles in the CORRECT location
|
||||
job = MockJob(
|
||||
job_id=job_id,
|
||||
style_params_gcs_path=style_gcs_path, # CORRECT
|
||||
style_assets=assets, # CORRECT
|
||||
state_data={} # state_data is EMPTY (no styles_gcs_path)
|
||||
)
|
||||
|
||||
# Verify files exist in GCS
|
||||
assert self.storage.file_exists(style_gcs_path), "Style JSON should exist in GCS"
|
||||
for gcs_path in assets.values():
|
||||
assert self.storage.file_exists(gcs_path), f"Asset {gcs_path} should exist in GCS"
|
||||
|
||||
# Now test the style loading logic (inline version of _get_or_create_styles)
|
||||
style_dir = os.path.join(self.temp_dir, "style")
|
||||
os.makedirs(style_dir, exist_ok=True)
|
||||
styles_path = os.path.join(style_dir, "styles.json")
|
||||
|
||||
# This is the FIX: check job.style_params_gcs_path, not state_data
|
||||
if job.style_params_gcs_path:
|
||||
# Download style JSON
|
||||
self.storage.download_file(job.style_params_gcs_path, styles_path)
|
||||
|
||||
# Load and update paths
|
||||
with open(styles_path, 'r') as f:
|
||||
style_data = json.load(f)
|
||||
|
||||
# Download assets and update paths
|
||||
local_assets = {}
|
||||
for asset_key, gcs_path in job.style_assets.items():
|
||||
ext = os.path.splitext(gcs_path)[1] or '.png'
|
||||
local_path = os.path.join(style_dir, f"{asset_key}{ext}")
|
||||
self.storage.download_file(gcs_path, local_path)
|
||||
local_assets[asset_key] = local_path
|
||||
|
||||
# Update style_data with local paths
|
||||
asset_mapping = {
|
||||
'intro_background': ('intro', 'background_image'),
|
||||
'karaoke_background': ('karaoke', 'background_image'),
|
||||
'end_background': ('end', 'background_image'),
|
||||
'font': [('intro', 'font'), ('karaoke', 'font_path'), ('end', 'font')],
|
||||
}
|
||||
|
||||
for asset_key, local_path in local_assets.items():
|
||||
if asset_key in asset_mapping:
|
||||
mappings = asset_mapping[asset_key]
|
||||
if isinstance(mappings[0], str):
|
||||
mappings = [mappings]
|
||||
for section, field in mappings:
|
||||
if section in style_data:
|
||||
style_data[section][field] = local_path
|
||||
|
||||
# Save updated styles
|
||||
with open(styles_path, 'w') as f:
|
||||
json.dump(style_data, f, indent=2)
|
||||
|
||||
# Verify the result
|
||||
with open(styles_path, 'r') as f:
|
||||
result = json.load(f)
|
||||
|
||||
# Check that paths were updated to local paths
|
||||
assert style_dir in result['karaoke']['background_image'], \
|
||||
f"karaoke.background_image should be local path, got: {result['karaoke']['background_image']}"
|
||||
assert style_dir in result['karaoke']['font_path'], \
|
||||
f"karaoke.font_path should be local path, got: {result['karaoke']['font_path']}"
|
||||
assert style_dir in result['intro']['background_image'], \
|
||||
f"intro.background_image should be local path"
|
||||
|
||||
# Verify files actually exist locally
|
||||
assert os.path.exists(result['karaoke']['background_image']), \
|
||||
"Downloaded karaoke background should exist"
|
||||
assert os.path.exists(result['karaoke']['font_path']), \
|
||||
"Downloaded font should exist"
|
||||
|
||||
print(f"\n✅ Style loading from job.style_params_gcs_path works!")
|
||||
print(f" karaoke.background_image: {result['karaoke']['background_image']}")
|
||||
print(f" karaoke.font_path: {result['karaoke']['font_path']}")
|
||||
|
||||
def test_old_bug_state_data_lookup_fails(self):
|
||||
"""
|
||||
Demonstrate that the OLD bug would fail to find styles.
|
||||
|
||||
If we look at job.state_data['styles_gcs_path'] (the bug),
|
||||
we won't find anything because styles are stored at
|
||||
job.style_params_gcs_path.
|
||||
"""
|
||||
import time
|
||||
job_id = f"test-bug-{int(time.time() * 1000)}"
|
||||
|
||||
# Upload style to CORRECT location
|
||||
style_gcs_path = f"uploads/{job_id}/style/style_params.json"
|
||||
self.storage.upload_string('{"karaoke": {"font_size": 100}}', style_gcs_path)
|
||||
|
||||
# Create job with styles in CORRECT location
|
||||
job = MockJob(
|
||||
job_id=job_id,
|
||||
style_params_gcs_path=style_gcs_path, # CORRECT location
|
||||
style_assets={},
|
||||
state_data={} # No styles_gcs_path here!
|
||||
)
|
||||
|
||||
# THE BUG: looking in state_data instead of style_params_gcs_path
|
||||
wrong_path = job.state_data.get('styles_gcs_path')
|
||||
correct_path = job.style_params_gcs_path
|
||||
|
||||
assert wrong_path is None, "state_data['styles_gcs_path'] should be None (the bug location)"
|
||||
assert correct_path is not None, "style_params_gcs_path should have the correct path"
|
||||
assert self.storage.file_exists(correct_path), "Style should exist at correct path"
|
||||
|
||||
print(f"\n✅ Demonstrated the bug: state_data lookup returns None")
|
||||
print(f" state_data['styles_gcs_path'] = {wrong_path}")
|
||||
print(f" job.style_params_gcs_path = {correct_path}")
|
||||
|
||||
def test_default_styles_when_no_custom_styles(self):
|
||||
"""Test that default styles are used when no custom styles provided."""
|
||||
import time
|
||||
job_id = f"test-default-{int(time.time() * 1000)}"
|
||||
|
||||
# Job with NO custom styles
|
||||
job = MockJob(
|
||||
job_id=job_id,
|
||||
style_params_gcs_path=None,
|
||||
style_assets={},
|
||||
state_data={}
|
||||
)
|
||||
|
||||
# Simulate the logic
|
||||
style_dir = os.path.join(self.temp_dir, "style")
|
||||
os.makedirs(style_dir, exist_ok=True)
|
||||
styles_path = os.path.join(style_dir, "styles.json")
|
||||
|
||||
if job.style_params_gcs_path:
|
||||
# Would load custom styles
|
||||
pass
|
||||
else:
|
||||
# Use default styles
|
||||
default_styles = {
|
||||
"karaoke": {
|
||||
"background_color": "#000000",
|
||||
"font": "Arial",
|
||||
"font_path": "",
|
||||
"font_size": 100
|
||||
}
|
||||
}
|
||||
with open(styles_path, 'w') as f:
|
||||
json.dump(default_styles, f, indent=2)
|
||||
|
||||
# Verify defaults were used
|
||||
with open(styles_path, 'r') as f:
|
||||
result = json.load(f)
|
||||
|
||||
assert result['karaoke']['background_color'] == "#000000"
|
||||
assert result['karaoke']['font'] == "Arial"
|
||||
|
||||
print(f"\n✅ Default styles used when no custom styles provided")
|
||||
|
||||
|
||||
class TestParallelWorkerExecution:
|
||||
"""
|
||||
Tests for the parallel worker execution fix.
|
||||
|
||||
The bug was: FastAPI's BackgroundTasks runs async tasks sequentially,
|
||||
causing audio worker to complete before lyrics worker starts.
|
||||
|
||||
The fix uses asyncio.gather() to run both workers in parallel.
|
||||
"""
|
||||
|
||||
def test_asyncio_gather_runs_parallel(self):
|
||||
"""Test that asyncio.gather actually runs tasks in parallel."""
|
||||
import asyncio
|
||||
import time
|
||||
|
||||
start_times = {}
|
||||
end_times = {}
|
||||
|
||||
async def task1():
|
||||
start_times['task1'] = time.time()
|
||||
await asyncio.sleep(0.1)
|
||||
end_times['task1'] = time.time()
|
||||
return "task1 done"
|
||||
|
||||
async def task2():
|
||||
start_times['task2'] = time.time()
|
||||
await asyncio.sleep(0.1)
|
||||
end_times['task2'] = time.time()
|
||||
return "task2 done"
|
||||
|
||||
async def run_parallel():
|
||||
return await asyncio.gather(task1(), task2())
|
||||
|
||||
overall_start = time.time()
|
||||
results = asyncio.run(run_parallel())
|
||||
overall_duration = time.time() - overall_start
|
||||
|
||||
# Both tasks should have started at nearly the same time
|
||||
start_diff = abs(start_times['task1'] - start_times['task2'])
|
||||
|
||||
# If running in parallel:
|
||||
# - Both start at ~same time (diff < 0.05s)
|
||||
# - Total duration is ~0.1s (not 0.2s for sequential)
|
||||
|
||||
assert start_diff < 0.05, \
|
||||
f"Tasks should start together, diff was {start_diff:.3f}s"
|
||||
assert overall_duration < 0.15, \
|
||||
f"Parallel execution should take ~0.1s, took {overall_duration:.3f}s"
|
||||
|
||||
print(f"\n✅ asyncio.gather runs tasks in parallel!")
|
||||
print(f" Task start time difference: {start_diff:.3f}s")
|
||||
print(f" Total duration: {overall_duration:.3f}s (sequential would be ~0.2s)")
|
||||
|
||||
def test_sequential_background_tasks_is_slow(self):
|
||||
"""
|
||||
Demonstrate that sequential execution (the bug) is slower.
|
||||
"""
|
||||
import asyncio
|
||||
import time
|
||||
|
||||
execution_order = []
|
||||
|
||||
async def task1():
|
||||
execution_order.append(('task1', 'start'))
|
||||
await asyncio.sleep(0.05)
|
||||
execution_order.append(('task1', 'end'))
|
||||
|
||||
async def task2():
|
||||
execution_order.append(('task2', 'start'))
|
||||
await asyncio.sleep(0.05)
|
||||
execution_order.append(('task2', 'end'))
|
||||
|
||||
# Sequential (the bug)
|
||||
async def run_sequential():
|
||||
await task1()
|
||||
await task2()
|
||||
|
||||
execution_order.clear()
|
||||
start = time.time()
|
||||
asyncio.run(run_sequential())
|
||||
sequential_duration = time.time() - start
|
||||
sequential_order = execution_order.copy()
|
||||
|
||||
# Parallel (the fix)
|
||||
async def run_parallel():
|
||||
await asyncio.gather(task1(), task2())
|
||||
|
||||
execution_order.clear()
|
||||
start = time.time()
|
||||
asyncio.run(run_parallel())
|
||||
parallel_duration = time.time() - start
|
||||
parallel_order = execution_order.copy()
|
||||
|
||||
# Sequential: task1 starts, task1 ends, task2 starts, task2 ends
|
||||
assert sequential_order[0] == ('task1', 'start')
|
||||
assert sequential_order[1] == ('task1', 'end')
|
||||
assert sequential_order[2] == ('task2', 'start')
|
||||
|
||||
# Parallel: task1 starts, task2 starts (interleaved)
|
||||
assert parallel_order[0][1] == 'start'
|
||||
assert parallel_order[1][1] == 'start'
|
||||
|
||||
# Parallel should be ~2x faster
|
||||
assert parallel_duration < sequential_duration * 0.8, \
|
||||
f"Parallel ({parallel_duration:.3f}s) should be faster than sequential ({sequential_duration:.3f}s)"
|
||||
|
||||
print(f"\n✅ Demonstrated sequential vs parallel execution")
|
||||
print(f" Sequential: {sequential_duration:.3f}s - {sequential_order}")
|
||||
print(f" Parallel: {parallel_duration:.3f}s - {parallel_order}")
|
||||
|
||||
|
||||
print("✅ Style loading and parallel execution tests ready")
|
||||
@@ -1,229 +0,0 @@
|
||||
"""
|
||||
Direct tests for worker logging Firestore operations.
|
||||
|
||||
These tests bypass the full app/worker imports and test the Firestore
|
||||
operations directly, avoiding dependency issues.
|
||||
|
||||
Run with: ./scripts/run-emulator-tests.sh
|
||||
"""
|
||||
import pytest
|
||||
import time
|
||||
import requests
|
||||
import os
|
||||
import threading
|
||||
from concurrent.futures import ThreadPoolExecutor, as_completed
|
||||
from datetime import datetime
|
||||
|
||||
|
||||
def emulators_running() -> bool:
|
||||
"""Check if GCP emulators are running."""
|
||||
try:
|
||||
requests.get("http://127.0.0.1:8080", timeout=1)
|
||||
return True
|
||||
except (requests.exceptions.ConnectionError, requests.exceptions.Timeout):
|
||||
return False
|
||||
|
||||
|
||||
# Skip all tests in this module if emulators aren't running
|
||||
pytestmark = pytest.mark.skipif(
|
||||
not emulators_running(),
|
||||
reason="GCP emulators not running. Start with: scripts/start-emulators.sh"
|
||||
)
|
||||
|
||||
# Set up environment for emulator
|
||||
os.environ["FIRESTORE_EMULATOR_HOST"] = "127.0.0.1:8080"
|
||||
os.environ["GOOGLE_CLOUD_PROJECT"] = "test-project"
|
||||
|
||||
|
||||
class TestWorkerLogsFirestoreDirect:
|
||||
"""Direct Firestore tests for worker logs - no app imports needed."""
|
||||
|
||||
@pytest.fixture(autouse=True)
|
||||
def setup_firestore(self):
|
||||
"""Set up Firestore client for each test."""
|
||||
from google.cloud import firestore
|
||||
self.db = firestore.Client(project="test-project")
|
||||
self.collection = "test-worker-logs"
|
||||
yield
|
||||
|
||||
def _create_test_job(self):
|
||||
"""Create a test job document."""
|
||||
job_id = f"test-{int(time.time() * 1000)}"
|
||||
doc_ref = self.db.collection(self.collection).document(job_id)
|
||||
doc_ref.set({
|
||||
"job_id": job_id,
|
||||
"status": "pending",
|
||||
"created_at": datetime.utcnow(),
|
||||
"worker_logs": []
|
||||
})
|
||||
return job_id
|
||||
|
||||
def _append_log_read_modify_write(self, job_id: str, worker: str, message: str):
|
||||
"""
|
||||
OLD METHOD: Read-modify-write (has race condition).
|
||||
This is what we were doing before.
|
||||
"""
|
||||
doc_ref = self.db.collection(self.collection).document(job_id)
|
||||
doc = doc_ref.get()
|
||||
if not doc.exists:
|
||||
return
|
||||
|
||||
data = doc.to_dict()
|
||||
logs = data.get("worker_logs", [])
|
||||
logs.append({
|
||||
"timestamp": datetime.utcnow().isoformat() + "Z",
|
||||
"level": "INFO",
|
||||
"worker": worker,
|
||||
"message": message
|
||||
})
|
||||
doc_ref.update({"worker_logs": logs})
|
||||
|
||||
def _append_log_array_union(self, job_id: str, worker: str, message: str):
|
||||
"""
|
||||
NEW METHOD: ArrayUnion (atomic, no race condition).
|
||||
This is what we're doing now.
|
||||
"""
|
||||
from google.cloud import firestore
|
||||
doc_ref = self.db.collection(self.collection).document(job_id)
|
||||
doc_ref.update({
|
||||
"worker_logs": firestore.ArrayUnion([{
|
||||
"timestamp": datetime.utcnow().isoformat() + "Z",
|
||||
"level": "INFO",
|
||||
"worker": worker,
|
||||
"message": message
|
||||
}])
|
||||
})
|
||||
|
||||
def _get_logs(self, job_id: str):
|
||||
"""Get logs from job document."""
|
||||
doc_ref = self.db.collection(self.collection).document(job_id)
|
||||
doc = doc_ref.get()
|
||||
if not doc.exists:
|
||||
return []
|
||||
return doc.to_dict().get("worker_logs", [])
|
||||
|
||||
def test_array_union_single_write(self):
|
||||
"""Test ArrayUnion works for single write."""
|
||||
job_id = self._create_test_job()
|
||||
|
||||
self._append_log_array_union(job_id, "test", "Single log message")
|
||||
|
||||
time.sleep(0.1)
|
||||
logs = self._get_logs(job_id)
|
||||
|
||||
assert len(logs) == 1
|
||||
assert logs[0]["message"] == "Single log message"
|
||||
assert logs[0]["worker"] == "test"
|
||||
|
||||
def test_array_union_sequential_writes(self):
|
||||
"""Test ArrayUnion preserves all sequential writes."""
|
||||
job_id = self._create_test_job()
|
||||
|
||||
for i in range(10):
|
||||
self._append_log_array_union(job_id, "test", f"Log {i}")
|
||||
|
||||
time.sleep(0.2)
|
||||
logs = self._get_logs(job_id)
|
||||
|
||||
assert len(logs) == 10, f"Expected 10 logs, got {len(logs)}"
|
||||
|
||||
def test_read_modify_write_race_condition(self):
|
||||
"""
|
||||
Demonstrate the race condition with read-modify-write.
|
||||
This test shows WHY we needed ArrayUnion.
|
||||
"""
|
||||
job_id = self._create_test_job()
|
||||
num_writes = 20
|
||||
|
||||
def write_log(index):
|
||||
self._append_log_read_modify_write(job_id, "worker", f"RMW Log {index}")
|
||||
|
||||
# Write concurrently - should lose some logs due to race condition
|
||||
with ThreadPoolExecutor(max_workers=5) as executor:
|
||||
futures = [executor.submit(write_log, i) for i in range(num_writes)]
|
||||
for future in as_completed(futures):
|
||||
try:
|
||||
future.result()
|
||||
except Exception:
|
||||
pass # Ignore errors for this test
|
||||
|
||||
time.sleep(0.5)
|
||||
logs = self._get_logs(job_id)
|
||||
|
||||
# With read-modify-write, we likely lose some logs
|
||||
# This is expected - the test documents the problem
|
||||
rmw_count = len([l for l in logs if "RMW Log" in l.get("message", "")])
|
||||
print(f"\nRead-modify-write: {rmw_count}/{num_writes} logs preserved")
|
||||
|
||||
# We expect to lose some logs (this is the bug we're fixing)
|
||||
# If all 20 are there, the race condition didn't trigger (which is fine)
|
||||
# The important thing is that ArrayUnion test below ALWAYS preserves all
|
||||
|
||||
def test_array_union_no_race_condition(self):
|
||||
"""
|
||||
Verify ArrayUnion preserves ALL concurrent writes.
|
||||
This is the critical test.
|
||||
"""
|
||||
job_id = self._create_test_job()
|
||||
num_writes = 20
|
||||
|
||||
def write_log(index):
|
||||
self._append_log_array_union(job_id, "worker", f"ArrayUnion Log {index}")
|
||||
|
||||
# Write concurrently
|
||||
with ThreadPoolExecutor(max_workers=5) as executor:
|
||||
futures = [executor.submit(write_log, i) for i in range(num_writes)]
|
||||
for future in as_completed(futures):
|
||||
future.result() # Raise any exceptions
|
||||
|
||||
time.sleep(0.5)
|
||||
logs = self._get_logs(job_id)
|
||||
|
||||
# With ArrayUnion, ALL logs should be preserved
|
||||
au_count = len([l for l in logs if "ArrayUnion Log" in l.get("message", "")])
|
||||
print(f"\nArrayUnion: {au_count}/{num_writes} logs preserved")
|
||||
|
||||
assert au_count == num_writes, \
|
||||
f"ArrayUnion should preserve all {num_writes} logs, got {au_count}"
|
||||
|
||||
def test_concurrent_workers_array_union(self):
|
||||
"""
|
||||
Test simulating audio and lyrics workers writing concurrently.
|
||||
"""
|
||||
job_id = self._create_test_job()
|
||||
|
||||
def audio_worker():
|
||||
for i in range(10):
|
||||
self._append_log_array_union(job_id, "audio", f"Audio log {i}")
|
||||
time.sleep(0.01)
|
||||
|
||||
def lyrics_worker():
|
||||
for i in range(10):
|
||||
self._append_log_array_union(job_id, "lyrics", f"Lyrics log {i}")
|
||||
time.sleep(0.01)
|
||||
|
||||
# Start both workers
|
||||
audio_thread = threading.Thread(target=audio_worker)
|
||||
lyrics_thread = threading.Thread(target=lyrics_worker)
|
||||
|
||||
audio_thread.start()
|
||||
lyrics_thread.start()
|
||||
|
||||
audio_thread.join()
|
||||
lyrics_thread.join()
|
||||
|
||||
time.sleep(0.5)
|
||||
logs = self._get_logs(job_id)
|
||||
|
||||
audio_logs = [l for l in logs if l.get("worker") == "audio"]
|
||||
lyrics_logs = [l for l in logs if l.get("worker") == "lyrics"]
|
||||
|
||||
print(f"\nConcurrent workers: {len(audio_logs)} audio + {len(lyrics_logs)} lyrics")
|
||||
|
||||
assert len(audio_logs) == 10, f"Expected 10 audio logs, got {len(audio_logs)}"
|
||||
assert len(lyrics_logs) == 10, f"Expected 10 lyrics logs, got {len(lyrics_logs)}"
|
||||
|
||||
print("✅ All logs from both workers preserved!")
|
||||
|
||||
|
||||
print("✅ Direct Firestore worker logs tests ready")
|
||||
@@ -1,10 +0,0 @@
|
||||
# Testing dependencies
|
||||
pytest>=7.4.0
|
||||
pytest-asyncio>=0.21.0
|
||||
pytest-cov>=4.1.0
|
||||
pytest-mock>=3.11.1
|
||||
httpx>=0.25.0 # For async testing
|
||||
|
||||
# Parent requirements (for import)
|
||||
-r ../requirements.txt
|
||||
|
||||
@@ -1,6 +0,0 @@
|
||||
pytest>=8.0.0
|
||||
pytest-asyncio>=0.23.0
|
||||
pytest-timeout>=2.2.0
|
||||
pytest-cov>=4.1.0
|
||||
requests>=2.31.0
|
||||
|
||||
@@ -1,460 +0,0 @@
|
||||
"""
|
||||
Integration tests for the karaoke generation backend API.
|
||||
|
||||
These tests verify the backend works end-to-end with real Cloud Run deployment.
|
||||
These tests require a deployed backend and are marked to skip unless explicitly enabled.
|
||||
|
||||
Run with: pytest backend/tests/test_api_integration.py -m integration
|
||||
"""
|
||||
import pytest
|
||||
import requests
|
||||
import subprocess
|
||||
import time
|
||||
import json
|
||||
from pathlib import Path
|
||||
from typing import Optional, Dict, Any
|
||||
import os
|
||||
|
||||
|
||||
# Skip all tests unless explicitly running integration tests
|
||||
pytestmark = pytest.mark.skipif(
|
||||
os.environ.get("RUN_INTEGRATION_TESTS") != "true",
|
||||
reason="Integration tests require deployed backend. Set RUN_INTEGRATION_TESTS=true to run."
|
||||
)
|
||||
|
||||
# Configuration
|
||||
SERVICE_URL = "https://karaoke-backend-718638054799.us-central1.run.app"
|
||||
TEST_YOUTUBE_URL = "https://www.youtube.com/watch?v=dQw4w9WgXcQ" # Rick Astley - Never Gonna Give You Up
|
||||
DEFAULT_TIMEOUT = 30 # seconds
|
||||
|
||||
|
||||
def api_get(url: str, headers: Optional[Dict[str, str]] = None, **kwargs) -> requests.Response:
|
||||
"""Make a GET request with default timeout."""
|
||||
return requests.get(url, headers=headers, timeout=DEFAULT_TIMEOUT, **kwargs)
|
||||
|
||||
|
||||
def api_post(url: str, headers: Optional[Dict[str, str]] = None, **kwargs) -> requests.Response:
|
||||
"""Make a POST request with default timeout."""
|
||||
return requests.post(url, headers=headers, timeout=DEFAULT_TIMEOUT, **kwargs)
|
||||
|
||||
|
||||
def api_delete(url: str, headers: Optional[Dict[str, str]] = None, **kwargs) -> requests.Response:
|
||||
"""Make a DELETE request with default timeout."""
|
||||
return requests.delete(url, headers=headers, timeout=DEFAULT_TIMEOUT, **kwargs)
|
||||
|
||||
|
||||
def get_auth_token() -> str:
|
||||
"""Get authentication token for Cloud Run."""
|
||||
result = subprocess.run(
|
||||
["gcloud", "auth", "print-identity-token"],
|
||||
capture_output=True,
|
||||
text=True,
|
||||
check=True
|
||||
)
|
||||
return result.stdout.strip()
|
||||
|
||||
|
||||
@pytest.fixture
|
||||
def auth_headers():
|
||||
"""Get authentication headers for requests."""
|
||||
token = get_auth_token()
|
||||
return {
|
||||
"Authorization": f"Bearer {token}",
|
||||
"Content-Type": "application/json"
|
||||
}
|
||||
|
||||
|
||||
class TestHealthEndpoint:
|
||||
"""Test health check endpoint."""
|
||||
|
||||
def test_health_check(self, auth_headers):
|
||||
"""Test that health endpoint returns 200 OK."""
|
||||
response = api_get(f"{SERVICE_URL}/api/health", headers=auth_headers)
|
||||
assert response.status_code == 200
|
||||
|
||||
data = response.json()
|
||||
assert data["status"] == "healthy"
|
||||
assert data["service"] == "karaoke-gen-backend"
|
||||
|
||||
def test_health_check_without_auth(self):
|
||||
"""Test that health endpoint requires authentication."""
|
||||
response = api_get(f"{SERVICE_URL}/api/health")
|
||||
assert response.status_code == 403
|
||||
|
||||
|
||||
class TestRootEndpoint:
|
||||
"""Test root endpoint."""
|
||||
|
||||
def test_root_endpoint(self, auth_headers):
|
||||
"""Test root endpoint returns service info."""
|
||||
response = api_get(SERVICE_URL, headers=auth_headers)
|
||||
assert response.status_code == 200
|
||||
|
||||
data = response.json()
|
||||
assert data["service"] == "karaoke-gen-backend"
|
||||
assert data["status"] == "running"
|
||||
assert "version" in data
|
||||
|
||||
|
||||
class TestJobSubmission:
|
||||
"""Test job submission workflows."""
|
||||
|
||||
def test_submit_job_with_youtube_url(self, auth_headers):
|
||||
"""Test submitting a job with a YouTube URL."""
|
||||
payload = {"url": TEST_YOUTUBE_URL}
|
||||
response = api_post(
|
||||
f"{SERVICE_URL}/api/jobs",
|
||||
headers=auth_headers,
|
||||
json=payload
|
||||
)
|
||||
|
||||
assert response.status_code == 200
|
||||
data = response.json()
|
||||
|
||||
assert data["status"] == "success"
|
||||
assert "job_id" in data
|
||||
assert len(data["job_id"]) > 0
|
||||
assert "message" in data
|
||||
|
||||
# Store job_id for cleanup
|
||||
return data["job_id"]
|
||||
|
||||
def test_submit_job_with_invalid_url(self, auth_headers):
|
||||
"""Test that invalid URLs are rejected."""
|
||||
payload = {"url": "not-a-url"}
|
||||
response = api_post(
|
||||
f"{SERVICE_URL}/api/jobs",
|
||||
headers=auth_headers,
|
||||
json=payload
|
||||
)
|
||||
|
||||
assert response.status_code == 422 # Validation error
|
||||
|
||||
def test_submit_job_without_url(self, auth_headers):
|
||||
"""Test that missing URL is rejected."""
|
||||
payload = {}
|
||||
response = api_post(
|
||||
f"{SERVICE_URL}/api/jobs",
|
||||
headers=auth_headers,
|
||||
json=payload
|
||||
)
|
||||
|
||||
assert response.status_code == 422 # Validation error
|
||||
|
||||
|
||||
class TestJobRetrieval:
|
||||
"""Test job retrieval and status checking."""
|
||||
|
||||
@pytest.fixture
|
||||
def test_job_id(self, auth_headers):
|
||||
"""Create a test job for retrieval tests."""
|
||||
payload = {"url": TEST_YOUTUBE_URL}
|
||||
response = api_post(
|
||||
f"{SERVICE_URL}/api/jobs",
|
||||
headers=auth_headers,
|
||||
json=payload
|
||||
)
|
||||
assert response.status_code == 200
|
||||
job_id = response.json()["job_id"]
|
||||
|
||||
yield job_id
|
||||
|
||||
# Cleanup
|
||||
api_delete(
|
||||
f"{SERVICE_URL}/api/jobs/{job_id}",
|
||||
headers=auth_headers
|
||||
)
|
||||
|
||||
def test_get_job_status(self, auth_headers, test_job_id):
|
||||
"""Test retrieving job status."""
|
||||
response = api_get(
|
||||
f"{SERVICE_URL}/api/jobs/{test_job_id}",
|
||||
headers=auth_headers
|
||||
)
|
||||
|
||||
assert response.status_code == 200
|
||||
data = response.json()
|
||||
|
||||
assert data["job_id"] == test_job_id
|
||||
assert "status" in data
|
||||
# Updated to use current JobStatus enum values
|
||||
assert data["status"] in ["pending", "downloading", "separating_stage1",
|
||||
"separating_stage2", "audio_complete",
|
||||
"transcribing", "correcting", "lyrics_complete",
|
||||
"generating_screens", "applying_padding",
|
||||
"awaiting_review", "in_review", "review_complete",
|
||||
"rendering_video", "awaiting_instrumental_selection",
|
||||
"instrumental_selected", "generating_video",
|
||||
"encoding", "packaging", "uploading", "notifying",
|
||||
"complete", "failed", "cancelled"]
|
||||
assert "progress" in data
|
||||
assert "created_at" in data
|
||||
assert "updated_at" in data
|
||||
assert "timeline" in data
|
||||
|
||||
def test_get_nonexistent_job(self, auth_headers):
|
||||
"""Test that requesting nonexistent job returns 404."""
|
||||
fake_job_id = "nonexistent-job-id"
|
||||
response = api_get(
|
||||
f"{SERVICE_URL}/api/jobs/{fake_job_id}",
|
||||
headers=auth_headers
|
||||
)
|
||||
|
||||
assert response.status_code == 404
|
||||
|
||||
def test_list_jobs(self, auth_headers, test_job_id):
|
||||
"""Test listing all jobs."""
|
||||
response = api_get(
|
||||
f"{SERVICE_URL}/api/jobs",
|
||||
headers=auth_headers
|
||||
)
|
||||
|
||||
assert response.status_code == 200
|
||||
data = response.json()
|
||||
|
||||
assert isinstance(data, list)
|
||||
# Our test job should be in the list
|
||||
job_ids = [job["job_id"] for job in data]
|
||||
assert test_job_id in job_ids
|
||||
|
||||
def test_list_jobs_with_status_filter(self, auth_headers):
|
||||
"""Test filtering jobs by status."""
|
||||
response = api_get(
|
||||
f"{SERVICE_URL}/api/jobs?status=pending",
|
||||
headers=auth_headers
|
||||
)
|
||||
|
||||
assert response.status_code == 200
|
||||
data = response.json()
|
||||
|
||||
assert isinstance(data, list)
|
||||
# All returned jobs should be pending
|
||||
for job in data:
|
||||
assert job["status"] == "pending"
|
||||
|
||||
def test_list_jobs_with_limit(self, auth_headers):
|
||||
"""Test limiting number of returned jobs."""
|
||||
response = api_get(
|
||||
f"{SERVICE_URL}/api/jobs?limit=5",
|
||||
headers=auth_headers
|
||||
)
|
||||
|
||||
assert response.status_code == 200
|
||||
data = response.json()
|
||||
|
||||
assert isinstance(data, list)
|
||||
assert len(data) <= 5
|
||||
|
||||
|
||||
class TestJobDeletion:
|
||||
"""Test job deletion."""
|
||||
|
||||
def test_delete_job(self, auth_headers):
|
||||
"""Test deleting a job."""
|
||||
# Create a job
|
||||
payload = {"url": TEST_YOUTUBE_URL}
|
||||
response = api_post(
|
||||
f"{SERVICE_URL}/api/jobs",
|
||||
headers=auth_headers,
|
||||
json=payload
|
||||
)
|
||||
assert response.status_code == 200
|
||||
job_id = response.json()["job_id"]
|
||||
|
||||
# Delete the job
|
||||
response = api_delete(
|
||||
f"{SERVICE_URL}/api/jobs/{job_id}",
|
||||
headers=auth_headers
|
||||
)
|
||||
|
||||
assert response.status_code == 200
|
||||
data = response.json()
|
||||
assert data["status"] == "success"
|
||||
|
||||
# Verify job is deleted
|
||||
response = api_get(
|
||||
f"{SERVICE_URL}/api/jobs/{job_id}",
|
||||
headers=auth_headers
|
||||
)
|
||||
assert response.status_code == 404
|
||||
|
||||
def test_delete_job_without_files(self, auth_headers):
|
||||
"""Test deleting a job without deleting files."""
|
||||
# Create a job
|
||||
payload = {"url": TEST_YOUTUBE_URL}
|
||||
response = api_post(
|
||||
f"{SERVICE_URL}/api/jobs",
|
||||
headers=auth_headers,
|
||||
json=payload
|
||||
)
|
||||
assert response.status_code == 200
|
||||
job_id = response.json()["job_id"]
|
||||
|
||||
# Delete job but keep files
|
||||
response = api_delete(
|
||||
f"{SERVICE_URL}/api/jobs/{job_id}?delete_files=false",
|
||||
headers=auth_headers
|
||||
)
|
||||
|
||||
assert response.status_code == 200
|
||||
data = response.json()
|
||||
assert data["status"] == "success"
|
||||
|
||||
def test_delete_nonexistent_job(self, auth_headers):
|
||||
"""Test deleting nonexistent job returns 404."""
|
||||
fake_job_id = "nonexistent-job-id"
|
||||
response = api_delete(
|
||||
f"{SERVICE_URL}/api/jobs/{fake_job_id}",
|
||||
headers=auth_headers
|
||||
)
|
||||
|
||||
assert response.status_code == 404
|
||||
|
||||
|
||||
class TestFileUpload:
|
||||
"""Test file upload endpoint."""
|
||||
|
||||
def test_upload_audio_file(self, auth_headers, tmp_path):
|
||||
"""Test uploading an audio file."""
|
||||
# Create a small test file
|
||||
test_file = tmp_path / "test_audio.mp3"
|
||||
test_file.write_bytes(b"fake audio content for testing")
|
||||
|
||||
# Upload file
|
||||
with open(test_file, "rb") as f:
|
||||
files = {"file": ("test_audio.mp3", f, "audio/mpeg")}
|
||||
data = {
|
||||
"artist": "Test Artist",
|
||||
"title": "Test Song"
|
||||
}
|
||||
|
||||
# Remove Content-Type header for multipart/form-data
|
||||
headers = {
|
||||
"Authorization": auth_headers["Authorization"]
|
||||
}
|
||||
|
||||
response = api_post(
|
||||
f"{SERVICE_URL}/api/jobs/upload",
|
||||
headers=headers,
|
||||
files=files,
|
||||
data=data
|
||||
)
|
||||
|
||||
assert response.status_code == 200
|
||||
result = response.json()
|
||||
|
||||
assert result["status"] == "success"
|
||||
assert "job_id" in result
|
||||
|
||||
# Cleanup
|
||||
api_delete(
|
||||
f"{SERVICE_URL}/api/jobs/{result['job_id']}",
|
||||
headers=auth_headers
|
||||
)
|
||||
|
||||
def test_upload_invalid_file_type(self, auth_headers, tmp_path):
|
||||
"""Test that invalid file types are rejected."""
|
||||
# Create a test file with invalid extension
|
||||
test_file = tmp_path / "test.txt"
|
||||
test_file.write_bytes(b"not an audio file")
|
||||
|
||||
with open(test_file, "rb") as f:
|
||||
files = {"file": ("test.txt", f, "text/plain")}
|
||||
data = {
|
||||
"artist": "Test Artist",
|
||||
"title": "Test Song"
|
||||
}
|
||||
|
||||
headers = {
|
||||
"Authorization": auth_headers["Authorization"]
|
||||
}
|
||||
|
||||
response = api_post(
|
||||
f"{SERVICE_URL}/api/jobs/upload",
|
||||
headers=headers,
|
||||
files=files,
|
||||
data=data
|
||||
)
|
||||
|
||||
assert response.status_code == 400
|
||||
|
||||
def test_upload_without_metadata(self, auth_headers, tmp_path):
|
||||
"""Test that uploads without artist/title are rejected."""
|
||||
test_file = tmp_path / "test_audio.mp3"
|
||||
test_file.write_bytes(b"fake audio content")
|
||||
|
||||
with open(test_file, "rb") as f:
|
||||
files = {"file": ("test_audio.mp3", f, "audio/mpeg")}
|
||||
|
||||
headers = {
|
||||
"Authorization": auth_headers["Authorization"]
|
||||
}
|
||||
|
||||
response = api_post(
|
||||
f"{SERVICE_URL}/api/jobs/upload",
|
||||
headers=headers,
|
||||
files=files
|
||||
)
|
||||
|
||||
assert response.status_code == 422 # Validation error
|
||||
|
||||
|
||||
class TestJobProcessing:
|
||||
"""Test end-to-end job processing (long-running)."""
|
||||
|
||||
@pytest.mark.slow
|
||||
@pytest.mark.integration
|
||||
def test_complete_job_workflow(self, auth_headers):
|
||||
"""
|
||||
Test complete job workflow from submission to completion.
|
||||
|
||||
This test is slow and requires actual processing.
|
||||
Run with: pytest -m slow
|
||||
"""
|
||||
# Submit job
|
||||
payload = {"url": TEST_YOUTUBE_URL}
|
||||
response = api_post(
|
||||
f"{SERVICE_URL}/api/jobs",
|
||||
headers=auth_headers,
|
||||
json=payload
|
||||
)
|
||||
assert response.status_code == 200
|
||||
job_id = response.json()["job_id"]
|
||||
|
||||
# Poll for completion (with timeout)
|
||||
timeout = 600 # 10 minutes
|
||||
start_time = time.time()
|
||||
|
||||
while time.time() - start_time < timeout:
|
||||
response = api_get(
|
||||
f"{SERVICE_URL}/api/jobs/{job_id}",
|
||||
headers=auth_headers
|
||||
)
|
||||
assert response.status_code == 200
|
||||
|
||||
job = response.json()
|
||||
status = job["status"]
|
||||
|
||||
if status == "complete":
|
||||
# Verify outputs exist
|
||||
assert "download_urls" in job
|
||||
assert len(job["download_urls"]) > 0
|
||||
break
|
||||
elif status == "failed":
|
||||
pytest.fail(f"Job failed: {job.get('error_message')}")
|
||||
|
||||
# Wait before next poll
|
||||
time.sleep(10)
|
||||
else:
|
||||
pytest.fail(f"Job did not complete within {timeout} seconds")
|
||||
|
||||
# Cleanup
|
||||
api_delete(
|
||||
f"{SERVICE_URL}/api/jobs/{job_id}",
|
||||
headers=auth_headers
|
||||
)
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
pytest.main([__file__, "-v", "--tb=short"])
|
||||
|
||||
@@ -1,93 +0,0 @@
|
||||
"""
|
||||
Unit tests for API routes.
|
||||
|
||||
These tests use FastAPI TestClient with mocked services to test
|
||||
route logic without hitting real cloud services.
|
||||
"""
|
||||
import pytest
|
||||
import json
|
||||
from datetime import datetime, UTC
|
||||
from unittest.mock import MagicMock, AsyncMock, patch
|
||||
from fastapi.testclient import TestClient
|
||||
from io import BytesIO
|
||||
|
||||
from backend.models.job import Job, JobStatus
|
||||
|
||||
|
||||
class TestHealthRoutes:
|
||||
"""Tests for health.py routes."""
|
||||
|
||||
@pytest.fixture
|
||||
def client(self):
|
||||
"""Create test client with mocked dependencies."""
|
||||
mock_creds = MagicMock()
|
||||
mock_creds.universe_domain = 'googleapis.com'
|
||||
with patch('backend.services.firestore_service.firestore'), \
|
||||
patch('backend.services.storage_service.storage'), \
|
||||
patch('google.auth.default', return_value=(mock_creds, 'test-project')):
|
||||
from backend.main import app
|
||||
return TestClient(app)
|
||||
|
||||
def test_health_endpoint_returns_200(self, client, auth_headers):
|
||||
"""Test /api/health returns 200 OK."""
|
||||
response = client.get("/api/health", )
|
||||
assert response.status_code == 200
|
||||
|
||||
def test_health_endpoint_returns_healthy_status(self, client, auth_headers):
|
||||
"""Test health endpoint returns healthy status."""
|
||||
response = client.get("/api/health", )
|
||||
data = response.json()
|
||||
assert data["status"] == "healthy"
|
||||
|
||||
def test_root_endpoint_returns_200(self, client, auth_headers):
|
||||
"""Test root endpoint returns 200."""
|
||||
response = client.get("/", )
|
||||
assert response.status_code == 200
|
||||
|
||||
|
||||
class TestJobRoutes:
|
||||
"""Tests for jobs.py routes.
|
||||
|
||||
Note: These tests verify the route module structure.
|
||||
Full integration tests are in test_api_integration.py.
|
||||
"""
|
||||
|
||||
def test_jobs_router_exists(self):
|
||||
"""Test jobs router can be imported."""
|
||||
from backend.api.routes import jobs
|
||||
assert hasattr(jobs, 'router')
|
||||
|
||||
def test_jobs_router_has_expected_endpoints(self):
|
||||
"""Test jobs router defines expected endpoints."""
|
||||
from backend.api.routes.jobs import router
|
||||
routes = [route.path for route in router.routes]
|
||||
assert '/jobs' in routes or any('/jobs' in r for r in routes)
|
||||
|
||||
|
||||
class TestInternalRoutes:
|
||||
"""Tests for internal.py routes.
|
||||
|
||||
Note: These tests are minimal because the internal routes
|
||||
trigger actual worker processing which requires complex mocking.
|
||||
The actual worker logic is tested in test_workers.py.
|
||||
"""
|
||||
|
||||
def test_internal_endpoint_structure(self):
|
||||
"""Test internal route module has expected endpoints."""
|
||||
from backend.api.routes import internal
|
||||
# Just verify the module can be imported
|
||||
assert hasattr(internal, 'router')
|
||||
|
||||
|
||||
class TestFileUploadRoutes:
|
||||
"""Tests for file_upload.py routes.
|
||||
|
||||
Note: These tests verify the route module structure.
|
||||
Full integration tests are in test_api_integration.py.
|
||||
"""
|
||||
|
||||
def test_file_upload_router_exists(self):
|
||||
"""Test file upload router can be imported."""
|
||||
from backend.api.routes import file_upload
|
||||
assert hasattr(file_upload, 'router')
|
||||
|
||||
@@ -1,294 +0,0 @@
|
||||
"""
|
||||
Tests for audio_analysis_service.py - Audio analysis for backing vocals.
|
||||
|
||||
These tests mock the storage service and shared karaoke_gen modules to verify:
|
||||
- GCS file download/upload operations
|
||||
- Audio analysis delegation to shared AudioAnalyzer
|
||||
- Waveform generation delegation to shared WaveformGenerator
|
||||
"""
|
||||
import pytest
|
||||
from unittest.mock import Mock, MagicMock, patch, ANY
|
||||
import tempfile
|
||||
import os
|
||||
|
||||
|
||||
class TestAudioAnalysisServiceInit:
|
||||
"""Test AudioAnalysisService initialization."""
|
||||
|
||||
@patch("backend.services.audio_analysis_service.StorageService")
|
||||
@patch("backend.services.audio_analysis_service.AudioAnalyzer")
|
||||
@patch("backend.services.audio_analysis_service.WaveformGenerator")
|
||||
def test_init_creates_dependencies(
|
||||
self, mock_waveform_class, mock_analyzer_class, mock_storage_class
|
||||
):
|
||||
"""Test initialization creates storage service and analyzers."""
|
||||
from backend.services.audio_analysis_service import AudioAnalysisService
|
||||
|
||||
service = AudioAnalysisService()
|
||||
|
||||
mock_storage_class.assert_called_once()
|
||||
mock_analyzer_class.assert_called_once_with(
|
||||
silence_threshold_db=-40.0,
|
||||
min_segment_duration_ms=100,
|
||||
)
|
||||
mock_waveform_class.assert_called_once()
|
||||
|
||||
@patch("backend.services.audio_analysis_service.StorageService")
|
||||
@patch("backend.services.audio_analysis_service.AudioAnalyzer")
|
||||
@patch("backend.services.audio_analysis_service.WaveformGenerator")
|
||||
def test_init_with_custom_params(
|
||||
self, mock_waveform_class, mock_analyzer_class, mock_storage_class
|
||||
):
|
||||
"""Test initialization with custom threshold and duration."""
|
||||
from backend.services.audio_analysis_service import AudioAnalysisService
|
||||
|
||||
service = AudioAnalysisService(
|
||||
silence_threshold_db=-50.0,
|
||||
min_segment_duration_ms=200,
|
||||
)
|
||||
|
||||
mock_analyzer_class.assert_called_once_with(
|
||||
silence_threshold_db=-50.0,
|
||||
min_segment_duration_ms=200,
|
||||
)
|
||||
|
||||
@patch("backend.services.audio_analysis_service.AudioAnalyzer")
|
||||
@patch("backend.services.audio_analysis_service.WaveformGenerator")
|
||||
def test_init_with_provided_storage_service(
|
||||
self, mock_waveform_class, mock_analyzer_class
|
||||
):
|
||||
"""Test initialization with externally provided storage service."""
|
||||
from backend.services.audio_analysis_service import AudioAnalysisService
|
||||
|
||||
mock_storage = Mock()
|
||||
service = AudioAnalysisService(storage_service=mock_storage)
|
||||
|
||||
assert service.storage_service is mock_storage
|
||||
|
||||
|
||||
class TestAnalyzeBackingVocals:
|
||||
"""Test analyze_backing_vocals method."""
|
||||
|
||||
@patch("backend.services.audio_analysis_service.StorageService")
|
||||
@patch("backend.services.audio_analysis_service.AudioAnalyzer")
|
||||
@patch("backend.services.audio_analysis_service.WaveformGenerator")
|
||||
def test_analyze_downloads_and_analyzes(
|
||||
self, mock_waveform_class, mock_analyzer_class, mock_storage_class
|
||||
):
|
||||
"""Test analyze_backing_vocals downloads file and runs analysis."""
|
||||
from backend.services.audio_analysis_service import AudioAnalysisService
|
||||
|
||||
# Setup mocks
|
||||
mock_storage = Mock()
|
||||
mock_storage_class.return_value = mock_storage
|
||||
|
||||
mock_result = Mock()
|
||||
mock_result.has_audible_content = True
|
||||
mock_result.segment_count = 5
|
||||
mock_result.recommended_selection = Mock(value="with_backing")
|
||||
|
||||
mock_analyzer = Mock()
|
||||
mock_analyzer.analyze.return_value = mock_result
|
||||
mock_analyzer_class.return_value = mock_analyzer
|
||||
|
||||
service = AudioAnalysisService()
|
||||
|
||||
# Call method
|
||||
result = service.analyze_backing_vocals(
|
||||
gcs_audio_path="uploads/job123/backing_vocals.flac",
|
||||
job_id="job123",
|
||||
)
|
||||
|
||||
# Verify storage download was called
|
||||
mock_storage.download_file.assert_called_once()
|
||||
call_args = mock_storage.download_file.call_args
|
||||
assert call_args[0][0] == "uploads/job123/backing_vocals.flac"
|
||||
|
||||
# Verify analyzer was called
|
||||
mock_analyzer.analyze.assert_called_once()
|
||||
|
||||
# Verify result
|
||||
assert result is mock_result
|
||||
assert result.has_audible_content is True
|
||||
|
||||
@patch("backend.services.audio_analysis_service.StorageService")
|
||||
@patch("backend.services.audio_analysis_service.AudioAnalyzer")
|
||||
@patch("backend.services.audio_analysis_service.WaveformGenerator")
|
||||
def test_analyze_cleans_up_temp_files(
|
||||
self, mock_waveform_class, mock_analyzer_class, mock_storage_class
|
||||
):
|
||||
"""Test that temp files are cleaned up after analysis."""
|
||||
from backend.services.audio_analysis_service import AudioAnalysisService
|
||||
|
||||
mock_storage = Mock()
|
||||
mock_storage_class.return_value = mock_storage
|
||||
|
||||
mock_result = Mock()
|
||||
mock_result.has_audible_content = False
|
||||
mock_result.segment_count = 0
|
||||
mock_result.recommended_selection = Mock(value="clean")
|
||||
|
||||
mock_analyzer = Mock()
|
||||
mock_analyzer.analyze.return_value = mock_result
|
||||
mock_analyzer_class.return_value = mock_analyzer
|
||||
|
||||
service = AudioAnalysisService()
|
||||
|
||||
# Call method
|
||||
service.analyze_backing_vocals(
|
||||
gcs_audio_path="uploads/job123/backing.flac",
|
||||
job_id="job123",
|
||||
)
|
||||
|
||||
# Temp directory should be cleaned up (no assertion needed,
|
||||
# TemporaryDirectory context manager handles this)
|
||||
# Just verify the method completes without error
|
||||
|
||||
|
||||
class TestAnalyzeAndGenerateWaveform:
|
||||
"""Test analyze_and_generate_waveform method."""
|
||||
|
||||
@patch("backend.services.audio_analysis_service.StorageService")
|
||||
@patch("backend.services.audio_analysis_service.AudioAnalyzer")
|
||||
@patch("backend.services.audio_analysis_service.WaveformGenerator")
|
||||
def test_analyze_and_generate_waveform(
|
||||
self, mock_waveform_class, mock_analyzer_class, mock_storage_class
|
||||
):
|
||||
"""Test analyze_and_generate_waveform does both operations."""
|
||||
from backend.services.audio_analysis_service import AudioAnalysisService
|
||||
|
||||
mock_storage = Mock()
|
||||
mock_storage_class.return_value = mock_storage
|
||||
|
||||
mock_result = Mock()
|
||||
mock_result.has_audible_content = True
|
||||
mock_result.audible_segments = [Mock(), Mock()]
|
||||
|
||||
mock_analyzer = Mock()
|
||||
mock_analyzer.analyze.return_value = mock_result
|
||||
mock_analyzer.silence_threshold_db = -40.0
|
||||
mock_analyzer_class.return_value = mock_analyzer
|
||||
|
||||
mock_waveform = Mock()
|
||||
mock_waveform_class.return_value = mock_waveform
|
||||
|
||||
service = AudioAnalysisService()
|
||||
|
||||
# Call method
|
||||
result, waveform_path = service.analyze_and_generate_waveform(
|
||||
gcs_audio_path="uploads/job123/backing.flac",
|
||||
job_id="job123",
|
||||
gcs_waveform_destination="uploads/job123/waveform.png",
|
||||
)
|
||||
|
||||
# Verify analysis was performed
|
||||
mock_analyzer.analyze.assert_called_once()
|
||||
|
||||
# Verify waveform was generated
|
||||
mock_waveform.generate.assert_called_once()
|
||||
call_kwargs = mock_waveform.generate.call_args.kwargs
|
||||
assert call_kwargs["segments"] == mock_result.audible_segments
|
||||
assert call_kwargs["show_time_axis"] is True
|
||||
|
||||
# Verify waveform was uploaded
|
||||
assert mock_storage.upload_file.call_count == 1
|
||||
|
||||
# Verify return values
|
||||
assert result is mock_result
|
||||
assert waveform_path == "uploads/job123/waveform.png"
|
||||
|
||||
|
||||
class TestGetWaveformData:
|
||||
"""Test get_waveform_data method."""
|
||||
|
||||
@patch("backend.services.audio_analysis_service.StorageService")
|
||||
@patch("backend.services.audio_analysis_service.AudioAnalyzer")
|
||||
@patch("backend.services.audio_analysis_service.WaveformGenerator")
|
||||
def test_get_waveform_data(
|
||||
self, mock_waveform_class, mock_analyzer_class, mock_storage_class
|
||||
):
|
||||
"""Test get_waveform_data returns amplitude data."""
|
||||
from backend.services.audio_analysis_service import AudioAnalysisService
|
||||
|
||||
mock_storage = Mock()
|
||||
mock_storage_class.return_value = mock_storage
|
||||
|
||||
mock_waveform = Mock()
|
||||
mock_waveform.generate_data_only.return_value = (
|
||||
[0.1, 0.3, 0.5, 0.3, 0.1], # amplitudes
|
||||
180.5, # duration
|
||||
)
|
||||
mock_waveform_class.return_value = mock_waveform
|
||||
|
||||
service = AudioAnalysisService()
|
||||
|
||||
amplitudes, duration = service.get_waveform_data(
|
||||
gcs_audio_path="uploads/job123/backing.flac",
|
||||
job_id="job123",
|
||||
num_points=500,
|
||||
)
|
||||
|
||||
# Verify storage download
|
||||
mock_storage.download_file.assert_called_once()
|
||||
|
||||
# Verify waveform generator called
|
||||
mock_waveform.generate_data_only.assert_called_once()
|
||||
call_kwargs = mock_waveform.generate_data_only.call_args.kwargs
|
||||
assert call_kwargs["num_points"] == 500
|
||||
|
||||
# Verify return values
|
||||
assert amplitudes == [0.1, 0.3, 0.5, 0.3, 0.1]
|
||||
assert duration == 180.5
|
||||
|
||||
|
||||
class TestGenerateWaveformWithMutes:
|
||||
"""Test generate_waveform_with_mutes method."""
|
||||
|
||||
@patch("backend.services.audio_analysis_service.StorageService")
|
||||
@patch("backend.services.audio_analysis_service.AudioAnalyzer")
|
||||
@patch("backend.services.audio_analysis_service.WaveformGenerator")
|
||||
def test_generate_waveform_with_mutes(
|
||||
self, mock_waveform_class, mock_analyzer_class, mock_storage_class
|
||||
):
|
||||
"""Test generate_waveform_with_mutes highlights mute regions."""
|
||||
from backend.services.audio_analysis_service import AudioAnalysisService
|
||||
|
||||
mock_storage = Mock()
|
||||
mock_storage_class.return_value = mock_storage
|
||||
|
||||
mock_result = Mock()
|
||||
mock_result.audible_segments = [Mock(), Mock()]
|
||||
|
||||
mock_analyzer = Mock()
|
||||
mock_analyzer.analyze.return_value = mock_result
|
||||
mock_analyzer_class.return_value = mock_analyzer
|
||||
|
||||
mock_waveform = Mock()
|
||||
mock_waveform_class.return_value = mock_waveform
|
||||
|
||||
# Create mock mute regions
|
||||
mock_mute1 = Mock()
|
||||
mock_mute2 = Mock()
|
||||
mute_regions = [mock_mute1, mock_mute2]
|
||||
|
||||
service = AudioAnalysisService()
|
||||
|
||||
result_path = service.generate_waveform_with_mutes(
|
||||
gcs_audio_path="uploads/job123/backing.flac",
|
||||
job_id="job123",
|
||||
gcs_waveform_destination="uploads/job123/waveform_muted.png",
|
||||
mute_regions=mute_regions,
|
||||
)
|
||||
|
||||
# Verify waveform generator was called with mute_regions
|
||||
mock_waveform.generate.assert_called_once()
|
||||
call_kwargs = mock_waveform.generate.call_args.kwargs
|
||||
assert call_kwargs["mute_regions"] == mute_regions
|
||||
assert call_kwargs["segments"] == mock_result.audible_segments
|
||||
|
||||
# Verify upload
|
||||
mock_storage.upload_file.assert_called_once()
|
||||
|
||||
# Verify return
|
||||
assert result_path == "uploads/job123/waveform_muted.png"
|
||||
|
||||
@@ -1,386 +0,0 @@
|
||||
"""
|
||||
Tests for audio_editing_service.py - Custom instrumental creation.
|
||||
|
||||
These tests mock the storage service and shared karaoke_gen modules to verify:
|
||||
- GCS file download/upload operations
|
||||
- Audio editing delegation to shared AudioEditor
|
||||
- Mute region validation
|
||||
"""
|
||||
import pytest
|
||||
from unittest.mock import Mock, MagicMock, patch, ANY
|
||||
|
||||
|
||||
class TestAudioEditingServiceInit:
|
||||
"""Test AudioEditingService initialization."""
|
||||
|
||||
@patch("backend.services.audio_editing_service.StorageService")
|
||||
@patch("backend.services.audio_editing_service.AudioEditor")
|
||||
def test_init_creates_dependencies(self, mock_editor_class, mock_storage_class):
|
||||
"""Test initialization creates storage service and editor."""
|
||||
from backend.services.audio_editing_service import AudioEditingService
|
||||
|
||||
service = AudioEditingService()
|
||||
|
||||
mock_storage_class.assert_called_once()
|
||||
mock_editor_class.assert_called_once_with(output_format="flac")
|
||||
|
||||
@patch("backend.services.audio_editing_service.StorageService")
|
||||
@patch("backend.services.audio_editing_service.AudioEditor")
|
||||
def test_init_with_custom_format(self, mock_editor_class, mock_storage_class):
|
||||
"""Test initialization with custom output format."""
|
||||
from backend.services.audio_editing_service import AudioEditingService
|
||||
|
||||
service = AudioEditingService(output_format="wav")
|
||||
|
||||
mock_editor_class.assert_called_once_with(output_format="wav")
|
||||
|
||||
@patch("backend.services.audio_editing_service.AudioEditor")
|
||||
def test_init_with_provided_storage_service(self, mock_editor_class):
|
||||
"""Test initialization with externally provided storage service."""
|
||||
from backend.services.audio_editing_service import AudioEditingService
|
||||
|
||||
mock_storage = Mock()
|
||||
service = AudioEditingService(storage_service=mock_storage)
|
||||
|
||||
assert service.storage_service is mock_storage
|
||||
|
||||
|
||||
class TestCreateCustomInstrumental:
|
||||
"""Test create_custom_instrumental method."""
|
||||
|
||||
@patch("backend.services.audio_editing_service.StorageService")
|
||||
@patch("backend.services.audio_editing_service.AudioEditor")
|
||||
def test_create_custom_instrumental_success(
|
||||
self, mock_editor_class, mock_storage_class
|
||||
):
|
||||
"""Test creating a custom instrumental successfully."""
|
||||
from backend.services.audio_editing_service import AudioEditingService
|
||||
|
||||
mock_storage = Mock()
|
||||
mock_storage_class.return_value = mock_storage
|
||||
|
||||
mock_result = Mock()
|
||||
mock_result.total_muted_duration_seconds = 15.5
|
||||
mock_result.mute_regions_applied = [Mock(), Mock(), Mock()]
|
||||
mock_result.output_path = None # Will be set by service
|
||||
|
||||
mock_editor = Mock()
|
||||
mock_editor.create_custom_instrumental.return_value = mock_result
|
||||
mock_editor_class.return_value = mock_editor
|
||||
|
||||
service = AudioEditingService()
|
||||
|
||||
# Create mock mute regions
|
||||
mock_mute1 = Mock()
|
||||
mock_mute2 = Mock()
|
||||
mute_regions = [mock_mute1, mock_mute2]
|
||||
|
||||
result = service.create_custom_instrumental(
|
||||
gcs_clean_instrumental_path="uploads/job123/clean.flac",
|
||||
gcs_backing_vocals_path="uploads/job123/backing.flac",
|
||||
mute_regions=mute_regions,
|
||||
gcs_output_path="uploads/job123/custom.flac",
|
||||
job_id="job123",
|
||||
)
|
||||
|
||||
# Verify downloads
|
||||
assert mock_storage.download_file.call_count == 2
|
||||
|
||||
# Verify editor was called
|
||||
mock_editor.create_custom_instrumental.assert_called_once()
|
||||
call_kwargs = mock_editor.create_custom_instrumental.call_args.kwargs
|
||||
assert call_kwargs["mute_regions"] == mute_regions
|
||||
|
||||
# Verify upload
|
||||
mock_storage.upload_file.assert_called_once()
|
||||
|
||||
# Verify result path was updated
|
||||
assert result.output_path == "uploads/job123/custom.flac"
|
||||
|
||||
@patch("backend.services.audio_editing_service.StorageService")
|
||||
@patch("backend.services.audio_editing_service.AudioEditor")
|
||||
def test_create_custom_instrumental_downloads_both_files(
|
||||
self, mock_editor_class, mock_storage_class
|
||||
):
|
||||
"""Test that both input files are downloaded."""
|
||||
from backend.services.audio_editing_service import AudioEditingService
|
||||
|
||||
mock_storage = Mock()
|
||||
mock_storage_class.return_value = mock_storage
|
||||
|
||||
mock_result = Mock()
|
||||
mock_result.total_muted_duration_seconds = 0
|
||||
mock_result.mute_regions_applied = []
|
||||
|
||||
mock_editor = Mock()
|
||||
mock_editor.create_custom_instrumental.return_value = mock_result
|
||||
mock_editor_class.return_value = mock_editor
|
||||
|
||||
service = AudioEditingService()
|
||||
|
||||
service.create_custom_instrumental(
|
||||
gcs_clean_instrumental_path="uploads/job123/clean.flac",
|
||||
gcs_backing_vocals_path="uploads/job123/backing.flac",
|
||||
mute_regions=[],
|
||||
gcs_output_path="uploads/job123/output.flac",
|
||||
job_id="job123",
|
||||
)
|
||||
|
||||
# Verify both files were downloaded
|
||||
download_calls = mock_storage.download_file.call_args_list
|
||||
assert len(download_calls) == 2
|
||||
|
||||
# First call should be clean instrumental
|
||||
assert download_calls[0][0][0] == "uploads/job123/clean.flac"
|
||||
# Second call should be backing vocals
|
||||
assert download_calls[1][0][0] == "uploads/job123/backing.flac"
|
||||
|
||||
|
||||
class TestCreatePreview:
|
||||
"""Test create_preview method."""
|
||||
|
||||
@patch("pydub.AudioSegment")
|
||||
@patch("backend.services.audio_editing_service.StorageService")
|
||||
@patch("backend.services.audio_editing_service.AudioEditor")
|
||||
def test_create_preview_success(
|
||||
self, mock_editor_class, mock_storage_class, mock_audio_segment
|
||||
):
|
||||
"""Test creating a preview successfully."""
|
||||
from backend.services.audio_editing_service import AudioEditingService
|
||||
|
||||
mock_storage = Mock()
|
||||
mock_storage_class.return_value = mock_storage
|
||||
|
||||
mock_preview = Mock()
|
||||
mock_preview.export = Mock()
|
||||
mock_preview.__getitem__ = Mock(return_value=mock_preview)
|
||||
|
||||
mock_editor = Mock()
|
||||
mock_editor.preview_with_mutes.return_value = mock_preview
|
||||
mock_editor_class.return_value = mock_editor
|
||||
|
||||
service = AudioEditingService()
|
||||
|
||||
result = service.create_preview(
|
||||
gcs_clean_instrumental_path="uploads/job123/clean.flac",
|
||||
gcs_backing_vocals_path="uploads/job123/backing.flac",
|
||||
mute_regions=[Mock(), Mock()],
|
||||
gcs_preview_path="uploads/job123/preview.flac",
|
||||
job_id="job123",
|
||||
)
|
||||
|
||||
# Verify editor was called
|
||||
mock_editor.preview_with_mutes.assert_called_once()
|
||||
|
||||
# Verify result
|
||||
assert result == "uploads/job123/preview.flac"
|
||||
|
||||
@patch("pydub.AudioSegment")
|
||||
@patch("backend.services.audio_editing_service.StorageService")
|
||||
@patch("backend.services.audio_editing_service.AudioEditor")
|
||||
def test_create_preview_with_duration_limit(
|
||||
self, mock_editor_class, mock_storage_class, mock_audio_segment
|
||||
):
|
||||
"""Test creating a preview with duration limit."""
|
||||
from backend.services.audio_editing_service import AudioEditingService
|
||||
|
||||
mock_storage = Mock()
|
||||
mock_storage_class.return_value = mock_storage
|
||||
|
||||
mock_preview = Mock()
|
||||
mock_truncated = Mock()
|
||||
mock_preview.__getitem__ = Mock(return_value=mock_truncated)
|
||||
mock_truncated.export = Mock()
|
||||
|
||||
mock_editor = Mock()
|
||||
mock_editor.preview_with_mutes.return_value = mock_preview
|
||||
mock_editor_class.return_value = mock_editor
|
||||
|
||||
service = AudioEditingService()
|
||||
|
||||
service.create_preview(
|
||||
gcs_clean_instrumental_path="uploads/job123/clean.flac",
|
||||
gcs_backing_vocals_path="uploads/job123/backing.flac",
|
||||
mute_regions=[],
|
||||
gcs_preview_path="uploads/job123/preview.flac",
|
||||
job_id="job123",
|
||||
preview_duration_seconds=30.0,
|
||||
)
|
||||
|
||||
# Verify preview was truncated (30 seconds = 30000ms)
|
||||
mock_preview.__getitem__.assert_called_once_with(slice(None, 30000))
|
||||
|
||||
|
||||
class TestMuteBackingVocalsOnly:
|
||||
"""Test mute_backing_vocals_only method."""
|
||||
|
||||
@patch("backend.services.audio_editing_service.StorageService")
|
||||
@patch("backend.services.audio_editing_service.AudioEditor")
|
||||
def test_mute_backing_vocals_only(self, mock_editor_class, mock_storage_class):
|
||||
"""Test muting backing vocals without combining."""
|
||||
from backend.services.audio_editing_service import AudioEditingService
|
||||
|
||||
mock_storage = Mock()
|
||||
mock_storage_class.return_value = mock_storage
|
||||
|
||||
mock_editor = Mock()
|
||||
mock_editor_class.return_value = mock_editor
|
||||
|
||||
service = AudioEditingService()
|
||||
|
||||
mute_regions = [Mock(), Mock()]
|
||||
|
||||
result = service.mute_backing_vocals_only(
|
||||
gcs_backing_vocals_path="uploads/job123/backing.flac",
|
||||
mute_regions=mute_regions,
|
||||
gcs_output_path="uploads/job123/muted_backing.flac",
|
||||
job_id="job123",
|
||||
)
|
||||
|
||||
# Verify only one file was downloaded
|
||||
mock_storage.download_file.assert_called_once()
|
||||
assert mock_storage.download_file.call_args[0][0] == "uploads/job123/backing.flac"
|
||||
|
||||
# Verify editor method was called
|
||||
mock_editor.apply_mute_to_single_track.assert_called_once()
|
||||
call_kwargs = mock_editor.apply_mute_to_single_track.call_args.kwargs
|
||||
assert call_kwargs["mute_regions"] == mute_regions
|
||||
|
||||
# Verify upload
|
||||
mock_storage.upload_file.assert_called_once()
|
||||
|
||||
# Verify result
|
||||
assert result == "uploads/job123/muted_backing.flac"
|
||||
|
||||
|
||||
class TestValidateMuteRegions:
|
||||
"""Test validate_mute_regions method.
|
||||
|
||||
Note: MuteRegion uses pydantic validation that rejects invalid values at
|
||||
creation time. These tests use Mock objects to test the validation logic
|
||||
in the service, simulating edge cases that would normally be rejected.
|
||||
"""
|
||||
|
||||
@patch("backend.services.audio_editing_service.StorageService")
|
||||
@patch("backend.services.audio_editing_service.AudioEditor")
|
||||
def test_valid_mute_regions(self, mock_editor_class, mock_storage_class):
|
||||
"""Test validation passes for valid mute regions."""
|
||||
from backend.services.audio_editing_service import AudioEditingService
|
||||
from karaoke_gen.instrumental_review import MuteRegion
|
||||
|
||||
service = AudioEditingService()
|
||||
|
||||
mute_regions = [
|
||||
MuteRegion(start_seconds=10.0, end_seconds=15.0),
|
||||
MuteRegion(start_seconds=30.0, end_seconds=45.0),
|
||||
]
|
||||
|
||||
errors = service.validate_mute_regions(mute_regions, total_duration_seconds=180.0)
|
||||
|
||||
assert errors == []
|
||||
|
||||
@patch("backend.services.audio_editing_service.StorageService")
|
||||
@patch("backend.services.audio_editing_service.AudioEditor")
|
||||
def test_negative_start_seconds(self, mock_editor_class, mock_storage_class):
|
||||
"""Test validation catches negative start time.
|
||||
|
||||
Uses Mock since MuteRegion pydantic model rejects negative values.
|
||||
"""
|
||||
from backend.services.audio_editing_service import AudioEditingService
|
||||
|
||||
service = AudioEditingService()
|
||||
|
||||
# Use mock to simulate invalid region (pydantic would reject this)
|
||||
mock_region = Mock()
|
||||
mock_region.start_seconds = -5.0
|
||||
mock_region.end_seconds = 10.0
|
||||
|
||||
errors = service.validate_mute_regions([mock_region], total_duration_seconds=180.0)
|
||||
|
||||
assert len(errors) == 1
|
||||
assert "cannot be negative" in errors[0]
|
||||
|
||||
@patch("backend.services.audio_editing_service.StorageService")
|
||||
@patch("backend.services.audio_editing_service.AudioEditor")
|
||||
def test_end_before_start(self, mock_editor_class, mock_storage_class):
|
||||
"""Test validation catches end time before start time."""
|
||||
from backend.services.audio_editing_service import AudioEditingService
|
||||
|
||||
service = AudioEditingService()
|
||||
|
||||
# Use mock to simulate invalid region
|
||||
mock_region = Mock()
|
||||
mock_region.start_seconds = 30.0
|
||||
mock_region.end_seconds = 20.0
|
||||
|
||||
errors = service.validate_mute_regions([mock_region], total_duration_seconds=180.0)
|
||||
|
||||
assert len(errors) == 1
|
||||
assert "must be after" in errors[0]
|
||||
|
||||
@patch("backend.services.audio_editing_service.StorageService")
|
||||
@patch("backend.services.audio_editing_service.AudioEditor")
|
||||
def test_start_exceeds_duration(self, mock_editor_class, mock_storage_class):
|
||||
"""Test validation catches start time exceeding audio duration."""
|
||||
from backend.services.audio_editing_service import AudioEditingService
|
||||
from karaoke_gen.instrumental_review import MuteRegion
|
||||
|
||||
service = AudioEditingService()
|
||||
|
||||
mute_regions = [
|
||||
MuteRegion(start_seconds=200.0, end_seconds=210.0),
|
||||
]
|
||||
|
||||
errors = service.validate_mute_regions(mute_regions, total_duration_seconds=180.0)
|
||||
|
||||
assert len(errors) == 1
|
||||
assert "exceeds audio duration" in errors[0]
|
||||
|
||||
@patch("backend.services.audio_editing_service.StorageService")
|
||||
@patch("backend.services.audio_editing_service.AudioEditor")
|
||||
def test_end_exceeds_duration_is_warning_not_error(
|
||||
self, mock_editor_class, mock_storage_class
|
||||
):
|
||||
"""Test that end time exceeding duration is logged as warning, not error."""
|
||||
from backend.services.audio_editing_service import AudioEditingService
|
||||
from karaoke_gen.instrumental_review import MuteRegion
|
||||
|
||||
service = AudioEditingService()
|
||||
|
||||
mute_regions = [
|
||||
MuteRegion(start_seconds=170.0, end_seconds=200.0), # End exceeds 180s
|
||||
]
|
||||
|
||||
errors = service.validate_mute_regions(mute_regions, total_duration_seconds=180.0)
|
||||
|
||||
# Should not be an error (will be clamped)
|
||||
assert len(errors) == 0
|
||||
|
||||
@patch("backend.services.audio_editing_service.StorageService")
|
||||
@patch("backend.services.audio_editing_service.AudioEditor")
|
||||
def test_multiple_errors(self, mock_editor_class, mock_storage_class):
|
||||
"""Test validation reports multiple errors."""
|
||||
from backend.services.audio_editing_service import AudioEditingService
|
||||
from karaoke_gen.instrumental_review import MuteRegion
|
||||
|
||||
service = AudioEditingService()
|
||||
|
||||
# Use mocks for invalid regions
|
||||
mock_region1 = Mock()
|
||||
mock_region1.start_seconds = -5.0
|
||||
mock_region1.end_seconds = 10.0
|
||||
|
||||
mock_region2 = Mock()
|
||||
mock_region2.start_seconds = 30.0
|
||||
mock_region2.end_seconds = 20.0 # End before start
|
||||
|
||||
mock_region3 = Mock()
|
||||
mock_region3.start_seconds = 200.0 # Exceeds duration
|
||||
mock_region3.end_seconds = 210.0
|
||||
|
||||
mute_regions = [mock_region1, mock_region2, mock_region3]
|
||||
|
||||
errors = service.validate_mute_regions(mute_regions, total_duration_seconds=180.0)
|
||||
|
||||
assert len(errors) == 3
|
||||
|
||||
File diff suppressed because it is too large.
Load diff
@@ -1,378 +0,0 @@
|
||||
"""
|
||||
Unit tests for audio analysis and editing services.
|
||||
|
||||
These tests verify the backend service wrappers that use the shared
|
||||
karaoke_gen.instrumental_review module with GCS integration.
|
||||
|
||||
NOTE: These tests require ffmpeg to be installed. They will be skipped
|
||||
if ffmpeg is not available (e.g., in CI environments without ffmpeg).
|
||||
"""
|
||||
|
||||
import os
|
||||
import shutil
|
||||
import tempfile
|
||||
import pytest
|
||||
from unittest.mock import MagicMock, patch, ANY
|
||||
|
||||
# Check if ffmpeg is available
|
||||
def _ffmpeg_available():
|
||||
"""Check if ffmpeg is available on the system."""
|
||||
return shutil.which('ffmpeg') is not None
|
||||
|
||||
# Skip all tests in this module if ffmpeg is not available
|
||||
pytestmark = pytest.mark.skipif(
|
||||
not _ffmpeg_available(),
|
||||
reason="ffmpeg not available - required for audio processing tests"
|
||||
)
|
||||
|
||||
from pydub import AudioSegment
|
||||
from pydub.generators import Sine
|
||||
|
||||
from karaoke_gen.instrumental_review import MuteRegion
|
||||
|
||||
|
||||
class TestAudioAnalysisService:
|
||||
"""Tests for AudioAnalysisService."""
|
||||
|
||||
@pytest.fixture
|
||||
def mock_storage_service(self):
|
||||
"""Create a mock storage service."""
|
||||
mock = MagicMock()
|
||||
return mock
|
||||
|
||||
@pytest.fixture
|
||||
def temp_audio_file(self, tmp_path):
|
||||
"""Create a temporary audio file for testing."""
|
||||
audio_path = tmp_path / "test_audio.flac"
|
||||
|
||||
# Create a simple sine wave audio
|
||||
tone = Sine(440).to_audio_segment(duration=5000) - 15
|
||||
tone.export(str(audio_path), format="flac")
|
||||
|
||||
return str(audio_path)
|
||||
|
||||
@pytest.fixture
|
||||
def silent_audio_file(self, tmp_path):
|
||||
"""Create a silent audio file for testing."""
|
||||
audio_path = tmp_path / "silent_audio.flac"
|
||||
|
||||
audio = AudioSegment.silent(duration=5000, frame_rate=44100)
|
||||
audio.export(str(audio_path), format="flac")
|
||||
|
||||
return str(audio_path)
|
||||
|
||||
def test_service_initialization(self, mock_storage_service):
|
||||
"""Test service initializes with default parameters."""
|
||||
from backend.services.audio_analysis_service import AudioAnalysisService
|
||||
|
||||
service = AudioAnalysisService(storage_service=mock_storage_service)
|
||||
|
||||
assert service.storage_service == mock_storage_service
|
||||
assert service.analyzer.silence_threshold_db == -40.0
|
||||
|
||||
def test_service_custom_threshold(self, mock_storage_service):
|
||||
"""Test service accepts custom silence threshold."""
|
||||
from backend.services.audio_analysis_service import AudioAnalysisService
|
||||
|
||||
service = AudioAnalysisService(
|
||||
storage_service=mock_storage_service,
|
||||
silence_threshold_db=-35.0,
|
||||
)
|
||||
|
||||
assert service.analyzer.silence_threshold_db == -35.0
|
||||
|
||||
def test_analyze_backing_vocals_downloads_file(
|
||||
self, mock_storage_service, temp_audio_file
|
||||
):
|
||||
"""Test that analyze_backing_vocals downloads file from GCS."""
|
||||
from backend.services.audio_analysis_service import AudioAnalysisService
|
||||
|
||||
# Set up mock to copy the temp file to the download location
|
||||
def mock_download(gcs_path, local_path):
|
||||
import shutil
|
||||
shutil.copy(temp_audio_file, local_path)
|
||||
return local_path
|
||||
|
||||
mock_storage_service.download_file.side_effect = mock_download
|
||||
|
||||
service = AudioAnalysisService(storage_service=mock_storage_service)
|
||||
result = service.analyze_backing_vocals(
|
||||
gcs_audio_path="jobs/test/stems/backing_vocals.flac",
|
||||
job_id="test-job",
|
||||
)
|
||||
|
||||
# Verify download was called
|
||||
mock_storage_service.download_file.assert_called_once()
|
||||
|
||||
# Verify result is valid
|
||||
assert result is not None
|
||||
assert hasattr(result, 'has_audible_content')
|
||||
|
||||
def test_analyze_silent_audio_returns_no_audible(
|
||||
self, mock_storage_service, silent_audio_file
|
||||
):
|
||||
"""Test that silent audio returns has_audible_content=False."""
|
||||
from backend.services.audio_analysis_service import AudioAnalysisService
|
||||
|
||||
def mock_download(gcs_path, local_path):
|
||||
import shutil
|
||||
shutil.copy(silent_audio_file, local_path)
|
||||
return local_path
|
||||
|
||||
mock_storage_service.download_file.side_effect = mock_download
|
||||
|
||||
service = AudioAnalysisService(storage_service=mock_storage_service)
|
||||
result = service.analyze_backing_vocals(
|
||||
gcs_audio_path="jobs/test/stems/backing_vocals.flac",
|
||||
job_id="test-job",
|
||||
)
|
||||
|
||||
assert result.has_audible_content is False
|
||||
assert len(result.audible_segments) == 0
|
||||
|
||||
def test_analyze_and_generate_waveform_uploads_image(
|
||||
self, mock_storage_service, temp_audio_file
|
||||
):
|
||||
"""Test that analyze_and_generate_waveform uploads waveform to GCS."""
|
||||
from backend.services.audio_analysis_service import AudioAnalysisService
|
||||
|
||||
def mock_download(gcs_path, local_path):
|
||||
import shutil
|
||||
shutil.copy(temp_audio_file, local_path)
|
||||
return local_path
|
||||
|
||||
mock_storage_service.download_file.side_effect = mock_download
|
||||
mock_storage_service.upload_file.return_value = "jobs/test/analysis/waveform.png"
|
||||
|
||||
service = AudioAnalysisService(storage_service=mock_storage_service)
|
||||
result, waveform_path = service.analyze_and_generate_waveform(
|
||||
gcs_audio_path="jobs/test/stems/backing_vocals.flac",
|
||||
job_id="test-job",
|
||||
gcs_waveform_destination="jobs/test/analysis/waveform.png",
|
||||
)
|
||||
|
||||
# Verify upload was called
|
||||
mock_storage_service.upload_file.assert_called_once()
|
||||
|
||||
# Verify waveform path is returned
|
||||
assert waveform_path == "jobs/test/analysis/waveform.png"
|
||||
|
||||
def test_get_waveform_data_returns_amplitudes(
|
||||
self, mock_storage_service, temp_audio_file
|
||||
):
|
||||
"""Test that get_waveform_data returns amplitude data."""
|
||||
from backend.services.audio_analysis_service import AudioAnalysisService
|
||||
|
||||
def mock_download(gcs_path, local_path):
|
||||
import shutil
|
||||
shutil.copy(temp_audio_file, local_path)
|
||||
return local_path
|
||||
|
||||
mock_storage_service.download_file.side_effect = mock_download
|
||||
|
||||
service = AudioAnalysisService(storage_service=mock_storage_service)
|
||||
amplitudes, duration = service.get_waveform_data(
|
||||
gcs_audio_path="jobs/test/stems/backing_vocals.flac",
|
||||
job_id="test-job",
|
||||
num_points=100,
|
||||
)
|
||||
|
||||
# Verify we get amplitude data
|
||||
assert isinstance(amplitudes, list)
|
||||
assert len(amplitudes) > 0
|
||||
assert all(isinstance(a, float) for a in amplitudes)
|
||||
|
||||
# Verify duration is reasonable
|
||||
assert duration > 0
|
||||
assert duration <= 10 # Our test file is 5 seconds
|
||||
|
||||
|
||||
class TestAudioEditingService:
|
||||
"""Tests for AudioEditingService."""
|
||||
|
||||
@pytest.fixture
|
||||
def mock_storage_service(self):
|
||||
"""Create a mock storage service."""
|
||||
mock = MagicMock()
|
||||
return mock
|
||||
|
||||
@pytest.fixture
|
||||
def temp_audio_files(self, tmp_path):
|
||||
"""Create temporary audio files for testing."""
|
||||
clean_path = tmp_path / "clean_instrumental.flac"
|
||||
backing_path = tmp_path / "backing_vocals.flac"
|
||||
|
||||
# Create clean instrumental (chord)
|
||||
tone1 = Sine(220).to_audio_segment(duration=5000) - 15
|
||||
tone2 = Sine(330).to_audio_segment(duration=5000) - 15
|
||||
clean = tone1.overlay(tone2)
|
||||
clean.export(str(clean_path), format="flac")
|
||||
|
||||
# Create backing vocals
|
||||
backing = Sine(440).to_audio_segment(duration=5000) - 20
|
||||
backing.export(str(backing_path), format="flac")
|
||||
|
||||
return str(clean_path), str(backing_path)
|
||||
|
||||
def test_service_initialization(self, mock_storage_service):
|
||||
"""Test service initializes correctly."""
|
||||
from backend.services.audio_editing_service import AudioEditingService
|
||||
|
||||
service = AudioEditingService(storage_service=mock_storage_service)
|
||||
|
||||
assert service.storage_service == mock_storage_service
|
||||
assert service.editor.output_format == "flac"
|
||||
|
||||
def test_create_custom_instrumental_downloads_files(
|
||||
self, mock_storage_service, temp_audio_files
|
||||
):
|
||||
"""Test that create_custom_instrumental downloads input files."""
|
||||
from backend.services.audio_editing_service import AudioEditingService
|
||||
|
||||
clean_path, backing_path = temp_audio_files
|
||||
|
||||
# Track which files were downloaded
|
||||
downloaded_files = []
|
||||
|
||||
def mock_download(gcs_path, local_path):
|
||||
downloaded_files.append(gcs_path)
|
||||
# Copy the appropriate test file
|
||||
import shutil
|
||||
if 'clean' in gcs_path:
|
||||
shutil.copy(clean_path, local_path)
|
||||
else:
|
||||
shutil.copy(backing_path, local_path)
|
||||
return local_path
|
||||
|
||||
mock_storage_service.download_file.side_effect = mock_download
|
||||
mock_storage_service.upload_file.return_value = "jobs/test/stems/custom.flac"
|
||||
|
||||
service = AudioEditingService(storage_service=mock_storage_service)
|
||||
|
||||
mute_regions = [MuteRegion(start_seconds=1.0, end_seconds=2.0)]
|
||||
|
||||
result = service.create_custom_instrumental(
|
||||
gcs_clean_instrumental_path="jobs/test/stems/clean.flac",
|
||||
gcs_backing_vocals_path="jobs/test/stems/backing.flac",
|
||||
mute_regions=mute_regions,
|
||||
gcs_output_path="jobs/test/stems/custom.flac",
|
||||
job_id="test-job",
|
||||
)
|
||||
|
||||
# Verify both files were downloaded
|
||||
assert len(downloaded_files) == 2
|
||||
assert mock_storage_service.upload_file.called
|
||||
|
||||
def test_create_custom_instrumental_uploads_result(
|
||||
self, mock_storage_service, temp_audio_files
|
||||
):
|
||||
"""Test that create_custom_instrumental uploads the result."""
|
||||
from backend.services.audio_editing_service import AudioEditingService
|
||||
|
||||
clean_path, backing_path = temp_audio_files
|
||||
|
||||
def mock_download(gcs_path, local_path):
|
||||
import shutil
|
||||
if 'clean' in gcs_path:
|
||||
shutil.copy(clean_path, local_path)
|
||||
else:
|
||||
shutil.copy(backing_path, local_path)
|
||||
return local_path
|
||||
|
||||
mock_storage_service.download_file.side_effect = mock_download
|
||||
mock_storage_service.upload_file.return_value = "jobs/test/stems/custom.flac"
|
||||
|
||||
service = AudioEditingService(storage_service=mock_storage_service)
|
||||
|
||||
mute_regions = [MuteRegion(start_seconds=1.0, end_seconds=2.0)]
|
||||
|
||||
result = service.create_custom_instrumental(
|
||||
gcs_clean_instrumental_path="jobs/test/stems/clean.flac",
|
||||
gcs_backing_vocals_path="jobs/test/stems/backing.flac",
|
||||
mute_regions=mute_regions,
|
||||
gcs_output_path="jobs/test/stems/custom.flac",
|
||||
job_id="test-job",
|
||||
)
|
||||
|
||||
# Verify upload was called with correct destination
|
||||
mock_storage_service.upload_file.assert_called_once()
|
||||
call_args = mock_storage_service.upload_file.call_args
|
||||
assert call_args[0][1] == "jobs/test/stems/custom.flac"
|
||||
|
||||
# Verify result has correct path
|
||||
assert result.output_path == "jobs/test/stems/custom.flac"
|
||||
|
||||
def test_create_custom_instrumental_returns_statistics(
|
||||
self, mock_storage_service, temp_audio_files
|
||||
):
|
||||
"""Test that create_custom_instrumental returns correct statistics."""
|
||||
from backend.services.audio_editing_service import AudioEditingService
|
||||
|
||||
clean_path, backing_path = temp_audio_files
|
||||
|
||||
def mock_download(gcs_path, local_path):
|
||||
import shutil
|
||||
if 'clean' in gcs_path:
|
||||
shutil.copy(clean_path, local_path)
|
||||
else:
|
||||
shutil.copy(backing_path, local_path)
|
||||
return local_path
|
||||
|
||||
mock_storage_service.download_file.side_effect = mock_download
|
||||
mock_storage_service.upload_file.return_value = "jobs/test/stems/custom.flac"
|
||||
|
||||
service = AudioEditingService(storage_service=mock_storage_service)
|
||||
|
||||
mute_regions = [
|
||||
MuteRegion(start_seconds=1.0, end_seconds=2.0),
|
||||
MuteRegion(start_seconds=3.0, end_seconds=4.0),
|
||||
]
|
||||
|
||||
result = service.create_custom_instrumental(
|
||||
gcs_clean_instrumental_path="jobs/test/stems/clean.flac",
|
||||
gcs_backing_vocals_path="jobs/test/stems/backing.flac",
|
||||
mute_regions=mute_regions,
|
||||
gcs_output_path="jobs/test/stems/custom.flac",
|
||||
job_id="test-job",
|
||||
)
|
||||
|
||||
# Verify statistics
|
||||
assert len(result.mute_regions_applied) == 2
|
||||
assert result.total_muted_duration_seconds == 2.0 # 1s + 1s
|
||||
assert result.output_duration_seconds > 0
|
||||
|
||||
def test_validate_mute_regions_returns_errors_for_invalid(
|
||||
self, mock_storage_service
|
||||
):
|
||||
"""Test that validate_mute_regions catches invalid regions."""
|
||||
from backend.services.audio_editing_service import AudioEditingService
|
||||
|
||||
service = AudioEditingService(storage_service=mock_storage_service)
|
||||
|
||||
mute_regions = [
|
||||
MuteRegion(start_seconds=0.0, end_seconds=1.0), # Valid
|
||||
MuteRegion(start_seconds=100.0, end_seconds=110.0), # Beyond duration
|
||||
]
|
||||
|
||||
errors = service.validate_mute_regions(mute_regions, total_duration_seconds=60.0)
|
||||
|
||||
# Should have error for region exceeding duration
|
||||
assert len(errors) == 1
|
||||
assert "exceeds" in errors[0].lower()
|
||||
|
||||
def test_validate_mute_regions_returns_empty_for_valid(
|
||||
self, mock_storage_service
|
||||
):
|
||||
"""Test that validate_mute_regions returns empty for valid regions."""
|
||||
from backend.services.audio_editing_service import AudioEditingService
|
||||
|
||||
service = AudioEditingService(storage_service=mock_storage_service)
|
||||
|
||||
mute_regions = [
|
||||
MuteRegion(start_seconds=0.0, end_seconds=1.0),
|
||||
MuteRegion(start_seconds=10.0, end_seconds=15.0),
|
||||
]
|
||||
|
||||
errors = service.validate_mute_regions(mute_regions, total_duration_seconds=60.0)
|
||||
|
||||
assert len(errors) == 0
|
||||
@@ -1,143 +0,0 @@
|
||||
"""
|
||||
Tests for AuthService and FirestoreService.
|
||||
|
||||
These tests focus on the service initialization and basic operations.
|
||||
"""
|
||||
import pytest
|
||||
from datetime import datetime, UTC
|
||||
from unittest.mock import MagicMock, patch
|
||||
|
||||
|
||||
class TestAuthServiceBasics:
|
||||
"""Basic tests for AuthService."""
|
||||
|
||||
def test_auth_service_can_import(self):
|
||||
"""Test AuthService can be imported."""
|
||||
from backend.services.auth_service import AuthService
|
||||
assert AuthService is not None
|
||||
|
||||
def test_auth_service_initialization(self):
|
||||
"""Test AuthService initializes with mocked dependencies."""
|
||||
mock_firestore = MagicMock()
|
||||
|
||||
with patch('backend.services.auth_service.FirestoreService', return_value=mock_firestore), \
|
||||
patch.dict('os.environ', {'ADMIN_TOKENS': 'test-token'}):
|
||||
from backend.services.auth_service import AuthService
|
||||
service = AuthService()
|
||||
assert service is not None
|
||||
|
||||
def test_auth_service_has_validate_method(self):
|
||||
"""Test AuthService has token validation method."""
|
||||
from backend.services.auth_service import AuthService
|
||||
assert hasattr(AuthService, 'validate_token') or hasattr(AuthService, 'verify_token')
|
||||
|
||||
|
||||
class TestFirestoreServiceBasics:
|
||||
"""Basic tests for FirestoreService."""
|
||||
|
||||
def test_firestore_service_can_import(self):
|
||||
"""Test FirestoreService can be imported."""
|
||||
from backend.services.firestore_service import FirestoreService
|
||||
assert FirestoreService is not None
|
||||
|
||||
def test_firestore_service_initialization(self):
|
||||
"""Test FirestoreService initializes with mocked client."""
|
||||
mock_client = MagicMock()
|
||||
|
||||
with patch('backend.services.firestore_service.firestore.Client', return_value=mock_client):
|
||||
from backend.services.firestore_service import FirestoreService
|
||||
service = FirestoreService()
|
||||
assert service is not None
|
||||
|
||||
def test_firestore_service_has_crud_methods(self):
|
||||
"""Test FirestoreService has CRUD methods."""
|
||||
from backend.services.firestore_service import FirestoreService
|
||||
assert hasattr(FirestoreService, 'create_job')
|
||||
assert hasattr(FirestoreService, 'get_job')
|
||||
assert hasattr(FirestoreService, 'update_job')
|
||||
assert hasattr(FirestoreService, 'delete_job')
|
||||
|
||||
|
||||
class TestStorageServiceBasics:
|
||||
"""Basic tests for StorageService."""
|
||||
|
||||
def test_storage_service_can_import(self):
|
||||
"""Test StorageService can be imported."""
|
||||
from backend.services.storage_service import StorageService
|
||||
assert StorageService is not None
|
||||
|
||||
def test_storage_service_initialization(self):
|
||||
"""Test StorageService initializes with mocked client."""
|
||||
mock_client = MagicMock()
|
||||
mock_bucket = MagicMock()
|
||||
mock_client.bucket.return_value = mock_bucket
|
||||
|
||||
with patch('backend.services.storage_service.storage.Client', return_value=mock_client):
|
||||
from backend.services.storage_service import StorageService
|
||||
service = StorageService()
|
||||
assert service is not None
|
||||
|
||||
def test_storage_service_has_file_methods(self):
|
||||
"""Test StorageService has file operation methods."""
|
||||
from backend.services.storage_service import StorageService
|
||||
assert hasattr(StorageService, 'upload_file')
|
||||
assert hasattr(StorageService, 'download_file')
|
||||
|
||||
|
||||
class TestWorkerServiceBasics:
|
||||
"""Basic tests for WorkerService."""
|
||||
|
||||
def test_worker_service_can_import(self):
|
||||
"""Test WorkerService can be imported."""
|
||||
from backend.services.worker_service import WorkerService
|
||||
assert WorkerService is not None
|
||||
|
||||
def test_worker_service_initialization(self):
|
||||
"""Test WorkerService initializes correctly."""
|
||||
from backend.services.worker_service import WorkerService
|
||||
service = WorkerService()
|
||||
assert service is not None
|
||||
|
||||
def test_worker_service_has_trigger_methods(self):
|
||||
"""Test WorkerService has trigger methods."""
|
||||
from backend.services.worker_service import WorkerService
|
||||
assert hasattr(WorkerService, 'trigger_audio_worker')
|
||||
assert hasattr(WorkerService, 'trigger_lyrics_worker')
|
||||
assert hasattr(WorkerService, 'trigger_screens_worker')
|
||||
assert hasattr(WorkerService, 'trigger_video_worker')
|
||||
|
||||
def test_worker_service_get_base_url(self):
|
||||
"""Test WorkerService constructs base URL."""
|
||||
from backend.services.worker_service import WorkerService
|
||||
service = WorkerService()
|
||||
url = service._get_base_url()
|
||||
assert 'http' in url
|
||||
|
||||
|
||||
class TestJobManagerBasics:
|
||||
"""Basic tests for JobManager."""
|
||||
|
||||
def test_job_manager_can_import(self):
|
||||
"""Test JobManager can be imported."""
|
||||
from backend.services.job_manager import JobManager
|
||||
assert JobManager is not None
|
||||
|
||||
def test_job_manager_initialization(self):
|
||||
"""Test JobManager initializes with mocked services."""
|
||||
mock_firestore = MagicMock()
|
||||
mock_storage = MagicMock()
|
||||
|
||||
with patch('backend.services.job_manager.FirestoreService', return_value=mock_firestore), \
|
||||
patch('backend.services.job_manager.StorageService', return_value=mock_storage):
|
||||
from backend.services.job_manager import JobManager
|
||||
manager = JobManager()
|
||||
assert manager is not None
|
||||
|
||||
def test_job_manager_has_crud_methods(self):
|
||||
"""Test JobManager has CRUD methods."""
|
||||
from backend.services.job_manager import JobManager
|
||||
assert hasattr(JobManager, 'create_job')
|
||||
assert hasattr(JobManager, 'get_job')
|
||||
assert hasattr(JobManager, 'delete_job')
|
||||
assert hasattr(JobManager, 'list_jobs')
|
||||
|
||||
@@ -1,68 +0,0 @@
|
||||
"""
|
||||
Extended tests for config.py to improve coverage.
|
||||
"""
|
||||
import pytest
|
||||
import os
|
||||
from unittest.mock import patch
|
||||
|
||||
|
||||
class TestSettingsConfiguration:
|
||||
"""Tests for Settings configuration."""
|
||||
|
||||
def test_settings_class_exists(self):
|
||||
"""Test Settings class exists."""
|
||||
from backend.config import Settings
|
||||
assert Settings is not None
|
||||
|
||||
def test_settings_loads_defaults(self):
|
||||
"""Test Settings loads default values."""
|
||||
from backend.config import Settings
|
||||
settings = Settings()
|
||||
assert settings is not None
|
||||
|
||||
def test_gcs_bucket_name_from_env(self):
|
||||
"""Test GCS bucket name from environment."""
|
||||
with patch.dict(os.environ, {'GCS_BUCKET_NAME': 'test-bucket'}):
|
||||
from backend.config import Settings
|
||||
settings = Settings()
|
||||
assert settings.gcs_bucket_name == 'test-bucket'
|
||||
|
||||
def test_firestore_collection_from_env(self):
|
||||
"""Test Firestore collection from environment."""
|
||||
with patch.dict(os.environ, {'FIRESTORE_COLLECTION': 'test-jobs'}):
|
||||
from backend.config import Settings
|
||||
settings = Settings()
|
||||
assert settings.firestore_collection == 'test-jobs'
|
||||
|
||||
def test_environment_setting(self):
|
||||
"""Test environment setting."""
|
||||
from backend.config import Settings
|
||||
settings = Settings()
|
||||
# Accept 'test' as well since pytest may set ENVIRONMENT=test
|
||||
assert settings.environment in ['development', 'production', 'testing', 'test']
|
||||
|
||||
def test_google_cloud_project(self):
|
||||
"""Test Google Cloud project setting."""
|
||||
with patch.dict(os.environ, {'GOOGLE_CLOUD_PROJECT': 'test-project'}):
|
||||
from backend.config import Settings
|
||||
settings = Settings()
|
||||
# Should either use the env var or have a default
|
||||
|
||||
|
||||
class TestGetSettings:
|
||||
"""Tests for get_settings function."""
|
||||
|
||||
def test_get_settings_returns_settings(self):
|
||||
"""Test get_settings returns Settings instance."""
|
||||
from backend.config import get_settings
|
||||
settings = get_settings()
|
||||
assert settings is not None
|
||||
|
||||
def test_get_settings_returns_same_instance(self):
|
||||
"""Test get_settings returns cached instance."""
|
||||
from backend.config import get_settings
|
||||
settings1 = get_settings()
|
||||
settings2 = get_settings()
|
||||
# Should be same cached instance
|
||||
assert settings1 is settings2
|
||||
|
||||
@@ -1,377 +0,0 @@
|
||||
"""
|
||||
Tests for credential manager and auth endpoints.
|
||||
"""
|
||||
import pytest
|
||||
import json
|
||||
from unittest.mock import MagicMock, patch
|
||||
from datetime import datetime
|
||||
|
||||
from backend.services.credential_manager import (
|
||||
CredentialManager,
|
||||
CredentialStatus,
|
||||
CredentialCheckResult,
|
||||
DeviceAuthInfo,
|
||||
get_credential_manager,
|
||||
)
|
||||
|
||||
|
||||
class TestCredentialStatus:
|
||||
"""Tests for CredentialStatus enum."""
|
||||
|
||||
def test_status_values(self):
|
||||
"""Test all status values exist."""
|
||||
assert CredentialStatus.VALID == "valid"
|
||||
assert CredentialStatus.EXPIRED == "expired"
|
||||
assert CredentialStatus.INVALID == "invalid"
|
||||
assert CredentialStatus.NOT_CONFIGURED == "not_configured"
|
||||
assert CredentialStatus.ERROR == "error"
|
||||
|
||||
|
||||
class TestCredentialCheckResult:
|
||||
"""Tests for CredentialCheckResult dataclass."""
|
||||
|
||||
def test_create_result(self):
|
||||
"""Test creating a check result."""
|
||||
result = CredentialCheckResult(
|
||||
service="youtube",
|
||||
status=CredentialStatus.VALID,
|
||||
message="Credentials are valid",
|
||||
last_checked=datetime.utcnow()
|
||||
)
|
||||
|
||||
assert result.service == "youtube"
|
||||
assert result.status == CredentialStatus.VALID
|
||||
assert result.message == "Credentials are valid"
|
||||
assert result.expires_at is None
|
||||
|
||||
def test_create_result_with_expiry(self):
|
||||
"""Test creating a check result with expiry."""
|
||||
expiry = datetime.utcnow()
|
||||
result = CredentialCheckResult(
|
||||
service="gdrive",
|
||||
status=CredentialStatus.VALID,
|
||||
message="Valid",
|
||||
last_checked=datetime.utcnow(),
|
||||
expires_at=expiry
|
||||
)
|
||||
|
||||
assert result.expires_at == expiry
|
||||
|
||||
|
||||
class TestCredentialManager:
|
||||
"""Tests for CredentialManager class."""
|
||||
|
||||
def test_init(self):
|
||||
"""Test manager initialization."""
|
||||
manager = CredentialManager()
|
||||
assert manager._pending_device_auths == {}
|
||||
|
||||
def test_check_youtube_not_configured(self):
|
||||
"""Test YouTube check when not configured."""
|
||||
with patch('backend.services.credential_manager.get_settings') as mock_get_settings:
|
||||
mock_settings = MagicMock()
|
||||
mock_settings.get_secret.return_value = None
|
||||
mock_get_settings.return_value = mock_settings
|
||||
|
||||
manager = CredentialManager()
|
||||
result = manager.check_youtube_credentials()
|
||||
|
||||
assert result.status == CredentialStatus.NOT_CONFIGURED
|
||||
assert "not configured" in result.message.lower()
|
||||
|
||||
def test_check_youtube_invalid_json(self):
|
||||
"""Test YouTube check with invalid JSON."""
|
||||
with patch('backend.services.credential_manager.get_settings') as mock_get_settings:
|
||||
mock_settings = MagicMock()
|
||||
mock_settings.get_secret.return_value = "not valid json"
|
||||
mock_get_settings.return_value = mock_settings
|
||||
|
||||
manager = CredentialManager()
|
||||
result = manager.check_youtube_credentials()
|
||||
|
||||
assert result.status == CredentialStatus.INVALID
|
||||
assert "json" in result.message.lower()
|
||||
|
||||
def test_check_youtube_missing_fields(self):
|
||||
"""Test YouTube check with missing required fields."""
|
||||
# Missing refresh_token and client_secret
|
||||
creds = json.dumps({"token": "test", "client_id": "test"})
|
||||
|
||||
with patch('backend.services.credential_manager.get_settings') as mock_get_settings:
|
||||
mock_settings = MagicMock()
|
||||
mock_settings.get_secret.return_value = creds
|
||||
mock_get_settings.return_value = mock_settings
|
||||
|
||||
manager = CredentialManager()
|
||||
result = manager.check_youtube_credentials()
|
||||
|
||||
assert result.status == CredentialStatus.INVALID
|
||||
assert "missing" in result.message.lower()
|
||||
|
||||
def test_check_gdrive_not_configured(self):
|
||||
"""Test Google Drive check when not configured."""
|
||||
with patch('backend.services.credential_manager.get_settings') as mock_get_settings:
|
||||
mock_settings = MagicMock()
|
||||
mock_settings.get_secret.return_value = None
|
||||
mock_get_settings.return_value = mock_settings
|
||||
|
||||
manager = CredentialManager()
|
||||
result = manager.check_gdrive_credentials()
|
||||
|
||||
assert result.status == CredentialStatus.NOT_CONFIGURED
|
||||
|
||||
def test_check_dropbox_not_configured(self):
|
||||
"""Test Dropbox check when not configured."""
|
||||
with patch('backend.services.credential_manager.get_settings') as mock_get_settings:
|
||||
mock_settings = MagicMock()
|
||||
mock_settings.get_secret.return_value = None
|
||||
mock_get_settings.return_value = mock_settings
|
||||
|
||||
manager = CredentialManager()
|
||||
result = manager.check_dropbox_credentials()
|
||||
|
||||
assert result.status == CredentialStatus.NOT_CONFIGURED
|
||||
|
||||
def test_check_dropbox_missing_access_token(self):
|
||||
"""Test Dropbox check with missing access token."""
|
||||
creds = json.dumps({"refresh_token": "test"}) # No access_token
|
||||
|
||||
with patch('backend.services.credential_manager.get_settings') as mock_get_settings:
|
||||
mock_settings = MagicMock()
|
||||
mock_settings.get_secret.return_value = creds
|
||||
mock_get_settings.return_value = mock_settings
|
||||
|
||||
manager = CredentialManager()
|
||||
result = manager.check_dropbox_credentials()
|
||||
|
||||
assert result.status == CredentialStatus.INVALID
|
||||
assert "access_token" in result.message.lower()
|
||||
|
||||
def test_check_all_credentials(self):
|
||||
"""Test checking all credentials."""
|
||||
with patch('backend.services.credential_manager.get_settings') as mock_get_settings:
|
||||
mock_settings = MagicMock()
|
||||
mock_settings.get_secret.return_value = None
|
||||
mock_get_settings.return_value = mock_settings
|
||||
|
||||
manager = CredentialManager()
|
||||
results = manager.check_all_credentials()
|
||||
|
||||
assert "youtube" in results
|
||||
assert "gdrive" in results
|
||||
assert "dropbox" in results
|
||||
assert all(r.status == CredentialStatus.NOT_CONFIGURED for r in results.values())
|
||||
|
||||
def test_test_youtube_api_failure(self):
|
||||
"""Test YouTube API test failure."""
|
||||
manager = CredentialManager()
|
||||
|
||||
mock_creds = MagicMock()
|
||||
|
||||
with patch('googleapiclient.discovery.build', side_effect=Exception("API Error")):
|
||||
result = manager._test_youtube_api(mock_creds)
|
||||
|
||||
assert result is False
|
||||
|
||||
def test_test_gdrive_api_failure(self):
|
||||
"""Test Google Drive API test failure."""
|
||||
manager = CredentialManager()
|
||||
|
||||
mock_creds = MagicMock()
|
||||
|
||||
with patch('googleapiclient.discovery.build', side_effect=Exception("API Error")):
|
||||
result = manager._test_gdrive_api(mock_creds)
|
||||
|
||||
assert result is False
|
||||
|
||||
def test_test_dropbox_api_failure(self):
|
||||
"""Test Dropbox API test failure."""
|
||||
manager = CredentialManager()
|
||||
|
||||
with patch('dropbox.Dropbox', side_effect=Exception("API Error")):
|
||||
result = manager._test_dropbox_api({"access_token": "test"})
|
||||
|
||||
assert result is False
|
||||
|
||||
def test_send_credential_alert_no_webhook(self):
|
||||
"""Test alert without webhook URL."""
|
||||
manager = CredentialManager()
|
||||
|
||||
result = manager.send_credential_alert([], discord_webhook_url=None)
|
||||
|
||||
assert result is False
|
||||
|
||||
def test_send_credential_alert_success(self):
|
||||
"""Test successful alert sending."""
|
||||
manager = CredentialManager()
|
||||
|
||||
mock_response = MagicMock()
|
||||
mock_response.status_code = 200
|
||||
|
||||
invalid_services = [
|
||||
CredentialCheckResult(
|
||||
service="youtube",
|
||||
status=CredentialStatus.INVALID,
|
||||
message="Test message",
|
||||
last_checked=datetime.utcnow()
|
||||
)
|
||||
]
|
||||
|
||||
with patch('requests.post', return_value=mock_response) as mock_post:
|
||||
result = manager.send_credential_alert(
|
||||
invalid_services,
|
||||
discord_webhook_url="https://discord.com/webhook/test"
|
||||
)
|
||||
|
||||
assert result is True
|
||||
mock_post.assert_called_once()
|
||||
|
||||
def test_start_youtube_device_auth(self):
|
||||
"""Test starting YouTube device auth flow."""
|
||||
manager = CredentialManager()
|
||||
|
||||
mock_response = MagicMock()
|
||||
mock_response.status_code = 200
|
||||
mock_response.json.return_value = {
|
||||
"device_code": "test_device_code",
|
||||
"user_code": "TEST-CODE",
|
||||
"verification_uri": "https://google.com/device",
|
||||
"expires_in": 1800,
|
||||
"interval": 5
|
||||
}
|
||||
|
||||
with patch('requests.post', return_value=mock_response):
|
||||
device_info = manager.start_youtube_device_auth(
|
||||
client_id="test_client",
|
||||
client_secret="test_secret"
|
||||
)
|
||||
|
||||
assert device_info.device_code == "test_device_code"
|
||||
assert device_info.user_code == "TEST-CODE"
|
||||
assert device_info.verification_url == "https://google.com/device"
|
||||
|
||||
# Should be stored for polling
|
||||
key = f"youtube:{device_info.device_code}"
|
||||
assert key in manager._pending_device_auths
|
||||
|
||||
def test_poll_device_auth_no_client_creds(self):
|
||||
"""Test polling when client credentials are not in Secret Manager."""
|
||||
manager = CredentialManager()
|
||||
|
||||
# Mock get_youtube_client_credentials to return None (no creds)
|
||||
with patch.object(manager, 'get_youtube_client_credentials', return_value=None):
|
||||
status, data = manager.poll_device_auth("youtube", "some_code")
|
||||
|
||||
assert status == "error"
|
||||
assert "client credentials not found" in data["message"].lower()
|
||||
|
||||
def test_poll_device_auth_expired(self):
|
||||
"""Test polling with expired device code."""
|
||||
manager = CredentialManager()
|
||||
|
||||
mock_response = MagicMock()
|
||||
mock_response.status_code = 400
|
||||
mock_response.json.return_value = {"error": "expired_token"}
|
||||
|
||||
with patch.object(manager, 'get_youtube_client_credentials',
|
||||
return_value={"client_id": "test", "client_secret": "test"}):
|
||||
with patch('requests.post', return_value=mock_response):
|
||||
status, data = manager.poll_device_auth("youtube", "expired_code")
|
||||
|
||||
assert status == "expired"
|
||||
assert "expired" in data["message"].lower()
|
||||
|
||||
def test_poll_device_auth_pending(self):
|
||||
"""Test polling when authorization is pending."""
|
||||
manager = CredentialManager()
|
||||
|
||||
mock_response = MagicMock()
|
||||
mock_response.status_code = 400
|
||||
mock_response.json.return_value = {"error": "authorization_pending"}
|
||||
|
||||
with patch.object(manager, 'get_youtube_client_credentials',
|
||||
return_value={"client_id": "test", "client_secret": "test"}):
|
||||
with patch('requests.post', return_value=mock_response):
|
||||
status, data = manager.poll_device_auth("youtube", "pending_code")
|
||||
|
||||
assert status == "pending"
|
||||
assert "waiting" in data["message"].lower()
|
||||
|
||||
def test_poll_device_auth_success(self):
|
||||
"""Test polling when authorization completes successfully."""
|
||||
manager = CredentialManager()
|
||||
|
||||
mock_response = MagicMock()
|
||||
mock_response.status_code = 200
|
||||
mock_response.json.return_value = {
|
||||
"access_token": "new_access_token",
|
||||
"refresh_token": "new_refresh_token"
|
||||
}
|
||||
|
||||
with patch.object(manager, 'get_youtube_client_credentials',
|
||||
return_value={"client_id": "test_id", "client_secret": "test_secret"}):
|
||||
with patch('requests.post', return_value=mock_response):
|
||||
with patch.object(manager, '_save_credentials_to_secret', return_value=True):
|
||||
status, data = manager.poll_device_auth("youtube", "completed_code")
|
||||
|
||||
assert status == "complete"
|
||||
assert data["token"] == "new_access_token"
|
||||
assert data["refresh_token"] == "new_refresh_token"
|
||||
assert data["client_id"] == "test_id"
|
||||
|
||||
|
||||
class TestGetCredentialManager:
|
||||
"""Tests for singleton factory function."""
|
||||
|
||||
def test_returns_singleton(self):
|
||||
"""Test that factory returns same instance."""
|
||||
# Reset singleton
|
||||
import backend.services.credential_manager as module
|
||||
module._credential_manager = None
|
||||
|
||||
manager1 = get_credential_manager()
|
||||
manager2 = get_credential_manager()
|
||||
|
||||
assert manager1 is manager2
|
||||
|
||||
|
||||
class TestAuthRoutes:
|
||||
"""Tests for auth API routes."""
|
||||
|
||||
def test_routes_import(self):
|
||||
"""Test that routes can be imported."""
|
||||
from backend.api.routes.auth import router
|
||||
assert router is not None
|
||||
|
||||
def test_status_endpoint_exists(self):
|
||||
"""Test that status endpoint is defined."""
|
||||
from backend.api.routes.auth import get_credentials_status
|
||||
assert get_credentials_status is not None
|
||||
|
||||
def test_validate_endpoint_exists(self):
|
||||
"""Test that validate endpoint is defined."""
|
||||
from backend.api.routes.auth import validate_credentials
|
||||
assert validate_credentials is not None
|
||||
|
||||
def test_device_auth_endpoint_exists(self):
|
||||
"""Test that device auth endpoints are defined."""
|
||||
from backend.api.routes.auth import (
|
||||
start_youtube_device_auth,
|
||||
poll_youtube_device_auth,
|
||||
start_gdrive_device_auth,
|
||||
poll_gdrive_device_auth,
|
||||
)
|
||||
assert start_youtube_device_auth is not None
|
||||
assert poll_youtube_device_auth is not None
|
||||
assert start_gdrive_device_auth is not None
|
||||
assert poll_gdrive_device_auth is not None
|
||||
|
||||
|
||||
class TestFileUploadCredentialValidation:
|
||||
"""Tests for credential validation in file upload."""
|
||||
|
||||
def test_credential_manager_imported(self):
|
||||
"""Test that credential manager is imported in file upload."""
|
||||
from backend.api.routes.file_upload import get_credential_manager, CredentialStatus
|
||||
assert get_credential_manager is not None
|
||||
assert CredentialStatus is not None
|
||||
@@ -1,54 +0,0 @@
|
||||
"""
|
||||
Tests for API dependencies (authentication, authorization).
|
||||
"""
|
||||
import pytest
|
||||
from unittest.mock import MagicMock, patch
|
||||
|
||||
|
||||
class TestAuthDependencies:
|
||||
"""Tests for authentication dependencies module structure."""
|
||||
|
||||
def test_get_token_from_request_function_exists(self):
|
||||
"""Test get_token_from_request function exists."""
|
||||
from backend.api.dependencies import get_token_from_request
|
||||
assert get_token_from_request is not None
|
||||
|
||||
def test_require_auth_function_exists(self):
|
||||
"""Test require_auth function exists."""
|
||||
from backend.api.dependencies import require_auth
|
||||
assert require_auth is not None
|
||||
|
||||
def test_require_admin_function_exists(self):
|
||||
"""Test require_admin function exists."""
|
||||
from backend.api.dependencies import require_admin
|
||||
assert require_admin is not None
|
||||
|
||||
def test_optional_auth_function_exists(self):
|
||||
"""Test optional_auth function exists."""
|
||||
from backend.api.dependencies import optional_auth
|
||||
assert optional_auth is not None
|
||||
|
||||
def test_security_scheme_defined(self):
|
||||
"""Test HTTP Bearer security scheme is defined."""
|
||||
from backend.api.dependencies import security
|
||||
assert security is not None
|
||||
|
||||
def test_logger_configured(self):
|
||||
"""Test logger is configured."""
|
||||
from backend.api.dependencies import logger
|
||||
assert logger is not None
|
||||
|
||||
|
||||
class TestAuthServiceAccess:
|
||||
"""Tests for auth service access."""
|
||||
|
||||
def test_get_auth_service_function_exists(self):
|
||||
"""Test get_auth_service function exists."""
|
||||
from backend.services.auth_service import get_auth_service
|
||||
assert get_auth_service is not None
|
||||
|
||||
def test_user_type_enum_defined(self):
|
||||
"""Test UserType enum is defined."""
|
||||
from backend.services.auth_service import UserType
|
||||
assert UserType is not None
|
||||
|
||||
@@ -1,820 +0,0 @@
|
||||
"""
|
||||
Tests for native distribution services (Dropbox and Google Drive).
|
||||
|
||||
These tests verify the service interfaces without requiring actual
|
||||
cloud credentials, using mocks to simulate API responses.
|
||||
"""
|
||||
import json
|
||||
import os
|
||||
import pytest
|
||||
from unittest.mock import MagicMock, patch, mock_open
|
||||
|
||||
|
||||
class TestDropboxService:
|
||||
"""Tests for the DropboxService class."""
|
||||
|
||||
def test_module_imports(self):
|
||||
"""Test that the module imports correctly."""
|
||||
from backend.services.dropbox_service import DropboxService, get_dropbox_service
|
||||
assert DropboxService is not None
|
||||
assert get_dropbox_service is not None
|
||||
|
||||
def test_init_creates_instance(self):
|
||||
"""Test that we can create a DropboxService instance."""
|
||||
from backend.services.dropbox_service import DropboxService
|
||||
service = DropboxService()
|
||||
assert service is not None
|
||||
assert service._client is None # Client not initialized until first use
|
||||
assert service._is_configured is False
|
||||
|
||||
def test_is_configured_returns_false_without_credentials(self):
|
||||
"""Test that is_configured returns False when credentials are missing."""
|
||||
from backend.services.dropbox_service import DropboxService
|
||||
|
||||
with patch.object(DropboxService, '_load_credentials', return_value=None):
|
||||
service = DropboxService()
|
||||
assert service.is_configured is False
|
||||
|
||||
def test_is_configured_returns_true_with_credentials(self):
|
||||
"""Test that is_configured returns True when credentials are present."""
|
||||
from backend.services.dropbox_service import DropboxService
|
||||
|
||||
mock_creds = {"access_token": "test_token"}
|
||||
with patch.object(DropboxService, '_load_credentials', return_value=mock_creds):
|
||||
service = DropboxService()
|
||||
assert service.is_configured is True
|
||||
|
||||
def test_is_configured_caches_result(self):
|
||||
"""Test that is_configured caches the result."""
|
||||
from backend.services.dropbox_service import DropboxService
|
||||
|
||||
service = DropboxService()
|
||||
service._is_configured = True
|
||||
# Should return True without calling _load_credentials
|
||||
assert service.is_configured is True
|
||||
|
||||
def test_is_configured_returns_false_without_access_token(self):
|
||||
"""Test that is_configured returns False when access_token is missing."""
|
||||
from backend.services.dropbox_service import DropboxService
|
||||
|
||||
mock_creds = {"refresh_token": "test"} # No access_token
|
||||
with patch.object(DropboxService, '_load_credentials', return_value=mock_creds):
|
||||
service = DropboxService()
|
||||
assert service.is_configured is False
|
||||
|
||||
def test_get_next_brand_code_first_track(self):
|
||||
"""Test brand code calculation when no existing tracks."""
|
||||
from backend.services.dropbox_service import DropboxService
|
||||
|
||||
service = DropboxService()
|
||||
with patch.object(service, 'list_folders', return_value=[]):
|
||||
brand_code = service.get_next_brand_code("/test/path", "NOMAD")
|
||||
assert brand_code == "NOMAD-0001"
|
||||
|
||||
def test_get_next_brand_code_sequential(self):
|
||||
"""Test brand code calculation with existing tracks."""
|
||||
from backend.services.dropbox_service import DropboxService
|
||||
|
||||
service = DropboxService()
|
||||
existing_folders = [
|
||||
"NOMAD-0001 - Artist1 - Song1",
|
||||
"NOMAD-0002 - Artist2 - Song2",
|
||||
"NOMAD-0005 - Artist3 - Song3", # Gap in sequence
|
||||
"OTHER-0001 - Different Brand", # Different brand
|
||||
]
|
||||
with patch.object(service, 'list_folders', return_value=existing_folders):
|
||||
brand_code = service.get_next_brand_code("/test/path", "NOMAD")
|
||||
assert brand_code == "NOMAD-0006"
|
||||
|
||||
def test_get_next_brand_code_different_prefix(self):
|
||||
"""Test brand code calculation with different brand prefix."""
|
||||
from backend.services.dropbox_service import DropboxService
|
||||
|
||||
service = DropboxService()
|
||||
existing_folders = [
|
||||
"NOMAD-0001 - Artist1 - Song1",
|
||||
"TEST-0010 - Artist2 - Song2",
|
||||
]
|
||||
with patch.object(service, 'list_folders', return_value=existing_folders):
|
||||
brand_code = service.get_next_brand_code("/test/path", "TEST")
|
||||
assert brand_code == "TEST-0011"
|
||||
|
||||
def test_factory_function_returns_instance(self):
|
||||
"""Test that get_dropbox_service returns a service instance."""
|
||||
from backend.services.dropbox_service import get_dropbox_service
|
||||
|
||||
service = get_dropbox_service()
|
||||
assert service is not None
|
||||
|
||||
def test_load_credentials_handles_exception(self):
|
||||
"""Test that _load_credentials handles exceptions gracefully."""
|
||||
from backend.services.dropbox_service import DropboxService
|
||||
|
||||
with patch('backend.services.dropbox_service.secretmanager.SecretManagerServiceClient') as mock_client:
|
||||
mock_client.return_value.access_secret_version.side_effect = Exception("API Error")
|
||||
service = DropboxService()
|
||||
result = service._load_credentials()
|
||||
assert result is None
|
||||
|
||||
def test_load_credentials_parses_json(self):
|
||||
"""Test that _load_credentials correctly parses JSON credentials."""
|
||||
from backend.services.dropbox_service import DropboxService
|
||||
|
||||
mock_creds = {"access_token": "test_token", "refresh_token": "test_refresh"}
|
||||
mock_response = MagicMock()
|
||||
mock_response.payload.data.decode.return_value = json.dumps(mock_creds)
|
||||
|
||||
with patch('backend.services.dropbox_service.secretmanager.SecretManagerServiceClient') as mock_client:
|
||||
mock_client.return_value.access_secret_version.return_value = mock_response
|
||||
service = DropboxService()
|
||||
result = service._load_credentials()
|
||||
assert result == mock_creds
|
||||
|
||||
def test_client_property_raises_without_credentials(self):
|
||||
"""Test that client property raises error without credentials."""
|
||||
from backend.services.dropbox_service import DropboxService
|
||||
|
||||
with patch.object(DropboxService, '_load_credentials', return_value=None):
|
||||
service = DropboxService()
|
||||
with pytest.raises(RuntimeError, match="credentials not configured"):
|
||||
_ = service.client
|
||||
|
||||
def test_client_property_raises_import_error(self):
|
||||
"""Test that client property raises helpful error if dropbox not installed."""
|
||||
from backend.services.dropbox_service import DropboxService
|
||||
import sys
|
||||
|
||||
mock_creds = {"access_token": "test_token"}
|
||||
with patch.object(DropboxService, '_load_credentials', return_value=mock_creds):
|
||||
with patch.dict(sys.modules, {'dropbox': None}):
|
||||
service = DropboxService()
|
||||
# Force reimport to fail
|
||||
with patch('builtins.__import__', side_effect=ImportError("No module named 'dropbox'")):
|
||||
with pytest.raises(ImportError, match="dropbox package"):
|
||||
_ = service.client
|
||||
|
||||
def test_list_folders_adds_leading_slash(self):
|
||||
"""Test that list_folders adds leading slash if missing."""
|
||||
from backend.services.dropbox_service import DropboxService
|
||||
|
||||
service = DropboxService()
|
||||
mock_client = MagicMock()
|
||||
mock_result = MagicMock()
|
||||
mock_result.entries = []
|
||||
mock_result.has_more = False
|
||||
mock_client.files_list_folder.return_value = mock_result
|
||||
service._client = mock_client
|
||||
|
||||
service.list_folders("test/path") # No leading slash
|
||||
|
||||
# Verify it was called with leading slash
|
||||
mock_client.files_list_folder.assert_called_with("/test/path")
|
||||
|
||||
def test_upload_file_adds_leading_slash(self):
|
||||
"""Test that upload_file adds leading slash to remote path."""
|
||||
from backend.services.dropbox_service import DropboxService
|
||||
import tempfile
|
||||
|
||||
service = DropboxService()
|
||||
mock_client = MagicMock()
|
||||
service._client = mock_client
|
||||
|
||||
# Create a small temp file
|
||||
with tempfile.NamedTemporaryFile(delete=False, mode='wb') as f:
|
||||
f.write(b"test content")
|
||||
temp_path = f.name
|
||||
|
||||
try:
|
||||
service.upload_file(temp_path, "remote/path.txt") # No leading slash
|
||||
# Verify upload was called with leading slash
|
||||
mock_client.files_upload.assert_called_once()
|
||||
call_args = mock_client.files_upload.call_args
|
||||
assert call_args[0][1] == "/remote/path.txt"
|
||||
finally:
|
||||
os.unlink(temp_path)
|
||||
|
||||
def test_upload_folder_uploads_all_files(self):
|
||||
"""Test that upload_folder uploads all files in directory."""
|
||||
from backend.services.dropbox_service import DropboxService
|
||||
import tempfile
|
||||
|
||||
service = DropboxService()
|
||||
|
||||
with tempfile.TemporaryDirectory() as tmpdir:
|
||||
# Create some test files
|
||||
(open(os.path.join(tmpdir, "file1.txt"), "w")).write("content1")
|
||||
(open(os.path.join(tmpdir, "file2.txt"), "w")).write("content2")
|
||||
|
||||
with patch.object(service, 'upload_file') as mock_upload:
|
||||
service.upload_folder(tmpdir, "/remote/folder")
|
||||
|
||||
assert mock_upload.call_count == 2
|
||||
|
||||
def test_create_shared_link_success(self):
|
||||
"""Test successful shared link creation."""
|
||||
from backend.services.dropbox_service import DropboxService
|
||||
|
||||
service = DropboxService()
|
||||
mock_client = MagicMock()
|
||||
mock_link = MagicMock()
|
||||
mock_link.url = "https://dropbox.com/shared/test"
|
||||
mock_client.sharing_create_shared_link_with_settings.return_value = mock_link
|
||||
service._client = mock_client
|
||||
|
||||
result = service.create_shared_link("/test/path")
|
||||
assert result == "https://dropbox.com/shared/test"
|
||||
|
||||
def test_sharing_list_shared_links_mock_setup(self):
|
||||
"""Test that mock setup for sharing_list_shared_links works correctly.
|
||||
|
||||
Note: Properly mocking Dropbox's ApiError is complex because it
|
||||
requires specific exception class structure. This test verifies
|
||||
the mock configuration is correct for the success path, which is
|
||||
a prerequisite for more complex error-handling tests.
|
||||
"""
|
||||
from backend.services.dropbox_service import DropboxService
|
||||
|
||||
service = DropboxService()
|
||||
mock_client = MagicMock()
|
||||
|
||||
# Mock the existing link retrieval (success case)
|
||||
mock_existing_link = MagicMock()
|
||||
mock_existing_link.url = "https://dropbox.com/existing/link"
|
||||
mock_links_result = MagicMock()
|
||||
mock_links_result.links = [mock_existing_link]
|
||||
mock_client.sharing_list_shared_links.return_value = mock_links_result
|
||||
|
||||
# Assign the mock client to the service
|
||||
service._client = mock_client
|
||||
|
||||
# Verify the mock returns the expected link structure
|
||||
result = mock_client.sharing_list_shared_links(path="/test/path")
|
||||
assert result.links[0].url == "https://dropbox.com/existing/link"
|
||||
mock_client.sharing_list_shared_links.assert_called_once_with(path="/test/path")
|
||||
|
||||
|
||||
class TestGoogleDriveService:
|
||||
"""Tests for the GoogleDriveService class."""
|
||||
|
||||
def test_module_imports(self):
|
||||
"""Test that the module imports correctly."""
|
||||
from backend.services.gdrive_service import GoogleDriveService, get_gdrive_service
|
||||
assert GoogleDriveService is not None
|
||||
assert get_gdrive_service is not None
|
||||
|
||||
def test_init_creates_instance(self):
|
||||
"""Test that we can create a GoogleDriveService instance."""
|
||||
from backend.services.gdrive_service import GoogleDriveService
|
||||
service = GoogleDriveService()
|
||||
assert service is not None
|
||||
assert service._service is None # Service not initialized until first use
|
||||
assert service._loaded is False
|
||||
|
||||
def test_is_configured_returns_false_without_credentials(self):
|
||||
"""Test that is_configured returns False when credentials are missing."""
|
||||
from backend.services.gdrive_service import GoogleDriveService
|
||||
|
||||
with patch.object(GoogleDriveService, '_load_credentials', return_value=None):
|
||||
service = GoogleDriveService()
|
||||
assert service.is_configured is False
|
||||
|
||||
def test_is_configured_returns_true_with_credentials(self):
|
||||
"""Test that is_configured returns True when credentials are present."""
|
||||
from backend.services.gdrive_service import GoogleDriveService
|
||||
|
||||
mock_creds = {
|
||||
"token": "test_token",
|
||||
"refresh_token": "test_refresh",
|
||||
"client_id": "test_client_id",
|
||||
"client_secret": "test_client_secret",
|
||||
}
|
||||
with patch.object(GoogleDriveService, '_load_credentials', return_value=mock_creds):
|
||||
service = GoogleDriveService()
|
||||
assert service.is_configured is True
|
||||
|
||||
def test_factory_function_returns_singleton(self):
|
||||
"""Test that get_gdrive_service returns a singleton."""
|
||||
from backend.services.gdrive_service import get_gdrive_service, _gdrive_service
|
||||
|
||||
# Reset singleton
|
||||
import backend.services.gdrive_service as gdrive_module
|
||||
gdrive_module._gdrive_service = None
|
||||
|
||||
service1 = get_gdrive_service()
|
||||
service2 = get_gdrive_service()
|
||||
assert service1 is service2
|
||||
|
||||
def test_load_credentials_caches_result(self):
|
||||
"""Test that _load_credentials caches the result."""
|
||||
from backend.services.gdrive_service import GoogleDriveService
|
||||
|
||||
service = GoogleDriveService()
|
||||
service._loaded = True
|
||||
service._credentials_data = {"cached": "data"}
|
||||
|
||||
# Should return cached data without calling settings
|
||||
result = service._load_credentials()
|
||||
assert result == {"cached": "data"}
|
||||
|
||||
def test_load_credentials_falls_back_to_youtube(self):
|
||||
"""Test that _load_credentials falls back to YouTube credentials."""
|
||||
from backend.services.gdrive_service import GoogleDriveService
|
||||
|
||||
mock_settings = MagicMock()
|
||||
mock_settings.get_secret.side_effect = [
|
||||
None, # gdrive-oauth-credentials not found
|
||||
'{"refresh_token": "yt_refresh", "client_id": "yt_client", "client_secret": "yt_secret"}', # youtube-oauth-credentials
|
||||
]
|
||||
|
||||
service = GoogleDriveService()
|
||||
service.settings = mock_settings
|
||||
|
||||
result = service._load_credentials()
|
||||
assert result is not None
|
||||
assert result["refresh_token"] == "yt_refresh"
|
||||
|
||||
def test_load_credentials_returns_none_when_both_fail(self):
|
||||
"""Test that _load_credentials returns None when no credentials available."""
|
||||
from backend.services.gdrive_service import GoogleDriveService
|
||||
|
||||
mock_settings = MagicMock()
|
||||
mock_settings.get_secret.return_value = None
|
||||
|
||||
service = GoogleDriveService()
|
||||
service.settings = mock_settings
|
||||
|
||||
result = service._load_credentials()
|
||||
assert result is None
|
||||
assert service._loaded is True
|
||||
|
||||
def test_load_credentials_validates_required_fields(self):
|
||||
"""Test that _load_credentials validates required fields."""
|
||||
from backend.services.gdrive_service import GoogleDriveService
|
||||
|
||||
mock_settings = MagicMock()
|
||||
# Missing client_secret
|
||||
mock_settings.get_secret.return_value = '{"refresh_token": "test", "client_id": "test"}'
|
||||
|
||||
service = GoogleDriveService()
|
||||
service.settings = mock_settings
|
||||
|
||||
result = service._load_credentials()
|
||||
assert result is None # Should fail validation
|
||||
|
||||
def test_load_credentials_handles_json_error(self):
|
||||
"""Test that _load_credentials handles JSON parse errors."""
|
||||
from backend.services.gdrive_service import GoogleDriveService
|
||||
|
||||
mock_settings = MagicMock()
|
||||
mock_settings.get_secret.return_value = "not valid json"
|
||||
|
||||
service = GoogleDriveService()
|
||||
service.settings = mock_settings
|
||||
|
||||
result = service._load_credentials()
|
||||
assert result is None
|
||||
|
||||
def test_service_property_raises_without_credentials(self):
|
||||
"""Test that service property raises error without credentials."""
|
||||
from backend.services.gdrive_service import GoogleDriveService
|
||||
|
||||
with patch.object(GoogleDriveService, '_load_credentials', return_value=None):
|
||||
service = GoogleDriveService()
|
||||
with pytest.raises(RuntimeError, match="credentials not configured"):
|
||||
_ = service.service
|
||||
|
||||
def test_get_or_create_folder_finds_existing(self):
|
||||
"""Test that get_or_create_folder finds existing folders."""
|
||||
from backend.services.gdrive_service import GoogleDriveService
|
||||
|
||||
service = GoogleDriveService()
|
||||
mock_service = MagicMock()
|
||||
mock_files = MagicMock()
|
||||
mock_list = MagicMock()
|
||||
mock_list.execute.return_value = {"files": [{"id": "existing_folder_id", "name": "TestFolder"}]}
|
||||
mock_files.list.return_value = mock_list
|
||||
mock_service.files.return_value = mock_files
|
||||
service._service = mock_service
|
||||
|
||||
result = service.get_or_create_folder("parent_id", "TestFolder")
|
||||
assert result == "existing_folder_id"
|
||||
|
||||
def test_get_or_create_folder_creates_new(self):
|
||||
"""Test that get_or_create_folder creates new folders when not found."""
|
||||
from backend.services.gdrive_service import GoogleDriveService
|
||||
|
||||
service = GoogleDriveService()
|
||||
mock_service = MagicMock()
|
||||
mock_files = MagicMock()
|
||||
|
||||
# First call - folder doesn't exist
|
||||
mock_list = MagicMock()
|
||||
mock_list.execute.return_value = {"files": []}
|
||||
mock_files.list.return_value = mock_list
|
||||
|
||||
# Second call - create folder
|
||||
mock_create = MagicMock()
|
||||
mock_create.execute.return_value = {"id": "new_folder_id"}
|
||||
mock_files.create.return_value = mock_create
|
||||
|
||||
mock_service.files.return_value = mock_files
|
||||
service._service = mock_service
|
||||
|
||||
result = service.get_or_create_folder("parent_id", "NewFolder")
|
||||
assert result == "new_folder_id"
|
||||
|
||||
def test_upload_file_determines_mime_type(self):
|
||||
"""Test that upload_file determines correct MIME type."""
|
||||
from backend.services.gdrive_service import GoogleDriveService
|
||||
import tempfile
|
||||
|
||||
service = GoogleDriveService()
|
||||
mock_service = MagicMock()
|
||||
mock_files = MagicMock()
|
||||
|
||||
# Mock list (for replace_existing check)
|
||||
mock_list = MagicMock()
|
||||
mock_list.execute.return_value = {"files": []}
|
||||
mock_files.list.return_value = mock_list
|
||||
|
||||
# Mock create
|
||||
mock_create = MagicMock()
|
||||
mock_create.execute.return_value = {"id": "uploaded_file_id"}
|
||||
mock_files.create.return_value = mock_create
|
||||
|
||||
mock_service.files.return_value = mock_files
|
||||
service._service = mock_service
|
||||
|
||||
# Create a temp file with .mp4 extension
|
||||
with tempfile.NamedTemporaryFile(suffix='.mp4', delete=False) as f:
|
||||
f.write(b"fake video content")
|
||||
temp_path = f.name
|
||||
|
||||
try:
|
||||
result = service.upload_file(temp_path, "parent_id", "test.mp4")
|
||||
assert result == "uploaded_file_id"
|
||||
finally:
|
||||
os.unlink(temp_path)
|
||||
|
||||
def test_upload_file_replaces_existing(self):
|
||||
"""Test that upload_file can replace existing files."""
|
||||
from backend.services.gdrive_service import GoogleDriveService
|
||||
import tempfile
|
||||
|
||||
service = GoogleDriveService()
|
||||
mock_service = MagicMock()
|
||||
mock_files = MagicMock()
|
||||
|
||||
# Mock list - file exists
|
||||
mock_list = MagicMock()
|
||||
mock_list.execute.return_value = {"files": [{"id": "existing_id"}]}
|
||||
mock_files.list.return_value = mock_list
|
||||
|
||||
# Mock delete
|
||||
mock_delete = MagicMock()
|
||||
mock_files.delete.return_value = mock_delete
|
||||
|
||||
# Mock create
|
||||
mock_create = MagicMock()
|
||||
mock_create.execute.return_value = {"id": "new_file_id"}
|
||||
mock_files.create.return_value = mock_create
|
||||
|
||||
mock_service.files.return_value = mock_files
|
||||
service._service = mock_service
|
||||
|
||||
with tempfile.NamedTemporaryFile(suffix='.txt', delete=False) as f:
|
||||
f.write(b"content")
|
||||
temp_path = f.name
|
||||
|
||||
try:
|
||||
result = service.upload_file(temp_path, "parent_id", "test.txt", replace_existing=True)
|
||||
assert result == "new_file_id"
|
||||
mock_files.delete.assert_called_once()
|
||||
finally:
|
||||
os.unlink(temp_path)
|
||||
|
||||
def test_upload_to_public_share_creates_structure(self):
|
||||
"""Test that upload_to_public_share creates folder structure."""
|
||||
from backend.services.gdrive_service import GoogleDriveService
|
||||
import tempfile
|
||||
|
||||
service = GoogleDriveService()
|
||||
|
||||
with tempfile.TemporaryDirectory() as tmpdir:
|
||||
# Create test files
|
||||
mp4_path = os.path.join(tmpdir, "test.mp4")
|
||||
mp4_720_path = os.path.join(tmpdir, "test_720p.mp4")
|
||||
cdg_path = os.path.join(tmpdir, "test.zip")
|
||||
|
||||
open(mp4_path, 'wb').write(b"mp4 content")
|
||||
open(mp4_720_path, 'wb').write(b"720p content")
|
||||
open(cdg_path, 'wb').write(b"cdg content")
|
||||
|
||||
with patch.object(service, 'get_or_create_folder', return_value="folder_id"):
|
||||
with patch.object(service, 'upload_file', return_value="file_id"):
|
||||
result = service.upload_to_public_share(
|
||||
root_folder_id="root_id",
|
||||
brand_code="TEST-0001",
|
||||
base_name="Artist - Title",
|
||||
output_files={
|
||||
"final_karaoke_lossy_mp4": mp4_path,
|
||||
"final_karaoke_lossy_720p_mp4": mp4_720_path,
|
||||
"final_karaoke_cdg_zip": cdg_path,
|
||||
}
|
||||
)
|
||||
|
||||
assert "mp4" in result
|
||||
assert "mp4_720p" in result
|
||||
assert "cdg" in result
|
||||
|
||||
def test_upload_to_public_share_handles_missing_files(self):
|
||||
"""Test that upload_to_public_share handles missing files gracefully."""
|
||||
from backend.services.gdrive_service import GoogleDriveService
|
||||
|
||||
service = GoogleDriveService()
|
||||
|
||||
with patch.object(service, 'get_or_create_folder', return_value="folder_id"):
|
||||
with patch.object(service, 'upload_file', return_value="file_id"):
|
||||
result = service.upload_to_public_share(
|
||||
root_folder_id="root_id",
|
||||
brand_code="TEST-0001",
|
||||
base_name="Artist - Title",
|
||||
output_files={
|
||||
"final_karaoke_lossy_mp4": "/nonexistent/path.mp4",
|
||||
}
|
||||
)
|
||||
|
||||
# Should return empty since file doesn't exist
|
||||
assert result == {}
|
||||
|
||||
|
||||
class TestJobModelDistributionFields:
|
||||
"""Tests for distribution fields in Job model."""
|
||||
|
||||
def test_job_model_has_dropbox_path_field(self):
|
||||
"""Test that Job model has dropbox_path field."""
|
||||
from backend.models.job import Job, JobStatus
|
||||
from datetime import datetime
|
||||
|
||||
job = Job(
|
||||
job_id="test-123",
|
||||
status=JobStatus.PENDING,
|
||||
created_at=datetime.now(),
|
||||
updated_at=datetime.now(),
|
||||
dropbox_path="/Karaoke/Tracks-Organized",
|
||||
)
|
||||
assert job.dropbox_path == "/Karaoke/Tracks-Organized"
|
||||
|
||||
def test_job_model_has_gdrive_folder_id_field(self):
|
||||
"""Test that Job model has gdrive_folder_id field."""
|
||||
from backend.models.job import Job, JobStatus
|
||||
from datetime import datetime
|
||||
|
||||
job = Job(
|
||||
job_id="test-123",
|
||||
status=JobStatus.PENDING,
|
||||
created_at=datetime.now(),
|
||||
updated_at=datetime.now(),
|
||||
gdrive_folder_id="1abc123xyz",
|
||||
)
|
||||
assert job.gdrive_folder_id == "1abc123xyz"
|
||||
|
||||
def test_job_model_distribution_fields_optional(self):
|
||||
"""Test that distribution fields are optional."""
|
||||
from backend.models.job import Job, JobStatus
|
||||
from datetime import datetime
|
||||
|
||||
job = Job(
|
||||
job_id="test-123",
|
||||
status=JobStatus.PENDING,
|
||||
created_at=datetime.now(),
|
||||
updated_at=datetime.now(),
|
||||
)
|
||||
assert job.dropbox_path is None
|
||||
assert job.gdrive_folder_id is None
|
||||
|
||||
|
||||
class TestJobCreateDistributionFields:
|
||||
"""Tests for distribution fields in JobCreate model."""
|
||||
|
||||
def test_job_create_has_dropbox_path_field(self):
|
||||
"""Test that JobCreate model has dropbox_path field."""
|
||||
from backend.models.job import JobCreate
|
||||
|
||||
job_create = JobCreate(
|
||||
artist="Test Artist",
|
||||
title="Test Title",
|
||||
dropbox_path="/Karaoke/Tracks-Organized",
|
||||
)
|
||||
assert job_create.dropbox_path == "/Karaoke/Tracks-Organized"
|
||||
|
||||
def test_job_create_has_gdrive_folder_id_field(self):
|
||||
"""Test that JobCreate model has gdrive_folder_id field."""
|
||||
from backend.models.job import JobCreate
|
||||
|
||||
job_create = JobCreate(
|
||||
artist="Test Artist",
|
||||
title="Test Title",
|
||||
gdrive_folder_id="1abc123xyz",
|
||||
)
|
||||
assert job_create.gdrive_folder_id == "1abc123xyz"
|
||||
|
||||
def test_job_create_distribution_fields_optional(self):
|
||||
"""Test that distribution fields are optional in JobCreate."""
|
||||
from backend.models.job import JobCreate
|
||||
|
||||
job_create = JobCreate(
|
||||
artist="Test Artist",
|
||||
title="Test Title",
|
||||
)
|
||||
assert job_create.dropbox_path is None
|
||||
assert job_create.gdrive_folder_id is None
|
||||
|
||||
|
||||
class TestFileUploadDistributionParams:
|
||||
"""Tests for distribution parameters in file upload endpoint."""
|
||||
|
||||
def test_endpoint_has_dropbox_path_parameter(self):
|
||||
"""Test that upload endpoint signature includes dropbox_path parameter."""
|
||||
import inspect
|
||||
from backend.api.routes.file_upload import upload_and_create_job
|
||||
|
||||
sig = inspect.signature(upload_and_create_job)
|
||||
param_names = list(sig.parameters.keys())
|
||||
|
||||
assert "dropbox_path" in param_names
|
||||
|
||||
def test_endpoint_has_gdrive_folder_id_parameter(self):
|
||||
"""Test that upload endpoint signature includes gdrive_folder_id parameter."""
|
||||
import inspect
|
||||
from backend.api.routes.file_upload import upload_and_create_job
|
||||
|
||||
sig = inspect.signature(upload_and_create_job)
|
||||
param_names = list(sig.parameters.keys())
|
||||
|
||||
assert "gdrive_folder_id" in param_names
|
||||
|
||||
def test_endpoint_dropbox_path_is_optional(self):
|
||||
"""Test that dropbox_path parameter has a default value (Form(None))."""
|
||||
import inspect
|
||||
from backend.api.routes.file_upload import upload_and_create_job
|
||||
|
||||
sig = inspect.signature(upload_and_create_job)
|
||||
param = sig.parameters.get("dropbox_path")
|
||||
|
||||
assert param is not None
|
||||
# Default is Form(None), so check that it's not required
|
||||
assert param.default is not inspect.Parameter.empty
|
||||
|
||||
def test_endpoint_gdrive_folder_id_is_optional(self):
|
||||
"""Test that gdrive_folder_id parameter has a default value (Form(None))."""
|
||||
import inspect
|
||||
from backend.api.routes.file_upload import upload_and_create_job
|
||||
|
||||
sig = inspect.signature(upload_and_create_job)
|
||||
param = sig.parameters.get("gdrive_folder_id")
|
||||
|
||||
assert param is not None
|
||||
# Default is Form(None), so check that it's not required
|
||||
assert param.default is not inspect.Parameter.empty
|
||||
|
||||
|
||||
class TestVideoWorkerDistribution:
|
||||
"""Tests for distribution handling in video worker."""
|
||||
|
||||
def test_handle_native_distribution_function_exists(self):
|
||||
"""Test that _handle_native_distribution function exists."""
|
||||
from backend.workers import video_worker
|
||||
assert hasattr(video_worker, '_handle_native_distribution')
|
||||
|
||||
def test_video_worker_imports_distribution_services(self):
|
||||
"""Test that video worker can import distribution services."""
|
||||
# The services should be importable (even if credentials aren't available)
|
||||
try:
|
||||
from backend.services.dropbox_service import DropboxService
|
||||
from backend.services.gdrive_service import GoogleDriveService
|
||||
assert True
|
||||
except ImportError as e:
|
||||
pytest.fail(f"Failed to import distribution services: {e}")
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_handle_native_distribution_skips_without_config(self):
|
||||
"""Test that _handle_native_distribution skips when not configured."""
|
||||
from backend.workers.video_worker import _handle_native_distribution
|
||||
|
||||
mock_job = MagicMock()
|
||||
mock_job.dropbox_path = None
|
||||
mock_job.gdrive_folder_id = None
|
||||
mock_job.brand_prefix = None
|
||||
mock_job.artist = "Test"
|
||||
mock_job.title = "Song"
|
||||
|
||||
mock_job_log = MagicMock()
|
||||
mock_job_manager = MagicMock()
|
||||
|
||||
# Should complete without error
|
||||
await _handle_native_distribution(
|
||||
job_id="test-123",
|
||||
job=mock_job,
|
||||
job_log=mock_job_log,
|
||||
job_manager=mock_job_manager,
|
||||
temp_dir="/tmp/test",
|
||||
result={},
|
||||
)
|
||||
|
||||
# No errors should have been logged
|
||||
mock_job_log.error.assert_not_called()
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_handle_native_distribution_dropbox_not_configured(self):
|
||||
"""Test Dropbox upload skipped when service not configured."""
|
||||
from backend.workers.video_worker import _handle_native_distribution
|
||||
|
||||
mock_job = MagicMock()
|
||||
mock_job.dropbox_path = "/test/path"
|
||||
mock_job.brand_prefix = "TEST"
|
||||
mock_job.gdrive_folder_id = None
|
||||
mock_job.artist = "Test"
|
||||
mock_job.title = "Song"
|
||||
mock_job.state_data = {}
|
||||
|
||||
mock_job_log = MagicMock()
|
||||
mock_job_manager = MagicMock()
|
||||
|
||||
mock_dropbox = MagicMock()
|
||||
mock_dropbox.is_configured = False
|
||||
|
||||
with patch('backend.services.dropbox_service.get_dropbox_service', return_value=mock_dropbox):
|
||||
await _handle_native_distribution(
|
||||
job_id="test-123",
|
||||
job=mock_job,
|
||||
job_log=mock_job_log,
|
||||
job_manager=mock_job_manager,
|
||||
temp_dir="/tmp/test",
|
||||
result={},
|
||||
)
|
||||
|
||||
# Should log warning about not configured
|
||||
mock_job_log.warning.assert_called()
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_handle_native_distribution_gdrive_not_configured(self):
|
||||
"""Test Google Drive upload skipped when service not configured."""
|
||||
from backend.workers.video_worker import _handle_native_distribution
|
||||
|
||||
mock_job = MagicMock()
|
||||
mock_job.dropbox_path = None
|
||||
mock_job.brand_prefix = None
|
||||
mock_job.gdrive_folder_id = "test_folder_id"
|
||||
mock_job.artist = "Test"
|
||||
mock_job.title = "Song"
|
||||
mock_job.state_data = {}
|
||||
|
||||
mock_job_log = MagicMock()
|
||||
mock_job_manager = MagicMock()
|
||||
|
||||
mock_gdrive = MagicMock()
|
||||
mock_gdrive.is_configured = False
|
||||
|
||||
with patch('backend.services.gdrive_service.get_gdrive_service', return_value=mock_gdrive):
|
||||
await _handle_native_distribution(
|
||||
job_id="test-123",
|
||||
job=mock_job,
|
||||
job_log=mock_job_log,
|
||||
job_manager=mock_job_manager,
|
||||
temp_dir="/tmp/test",
|
||||
result={},
|
||||
)
|
||||
|
||||
# Should log warning about not configured
|
||||
mock_job_log.warning.assert_called()
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_handle_native_distribution_handles_import_error(self):
|
||||
"""Test that import errors for services are handled gracefully."""
|
||||
from backend.workers.video_worker import _handle_native_distribution
|
||||
|
||||
mock_job = MagicMock()
|
||||
mock_job.dropbox_path = "/test/path"
|
||||
mock_job.brand_prefix = "TEST"
|
||||
mock_job.gdrive_folder_id = None
|
||||
mock_job.artist = "Test"
|
||||
mock_job.title = "Song"
|
||||
mock_job.state_data = {}
|
||||
|
||||
mock_job_log = MagicMock()
|
||||
mock_job_manager = MagicMock()
|
||||
|
||||
# Simulate import error
|
||||
with patch('backend.services.dropbox_service.get_dropbox_service', side_effect=ImportError("No module")):
|
||||
await _handle_native_distribution(
|
||||
job_id="test-123",
|
||||
job=mock_job,
|
||||
job_log=mock_job_log,
|
||||
job_manager=mock_job_manager,
|
||||
temp_dir="/tmp/test",
|
||||
result={},
|
||||
)
|
||||
|
||||
# Should log warning about import error
|
||||
mock_job_log.warning.assert_called()
|
||||
@@ -1,408 +0,0 @@
|
||||
"""
|
||||
Tests for dropbox_service.py - Dropbox file operations.
|
||||
|
||||
These tests mock the Dropbox SDK and Secret Manager to verify:
|
||||
- Credential loading from Secret Manager
|
||||
- Folder listing and brand code calculation
|
||||
- File and folder uploads
|
||||
- Shared link creation
|
||||
"""
|
||||
import json
|
||||
import os
|
||||
import pytest
|
||||
from unittest.mock import Mock, MagicMock, patch
|
||||
|
||||
|
||||
class TestDropboxServiceInit:
|
||||
"""Test DropboxService initialization."""
|
||||
|
||||
def test_init_creates_service(self):
|
||||
"""Test initialization creates service with no client."""
|
||||
from backend.services.dropbox_service import DropboxService
|
||||
|
||||
service = DropboxService()
|
||||
|
||||
assert service._client is None
|
||||
assert service._is_configured is False
|
||||
|
||||
|
||||
class TestLoadCredentials:
|
||||
"""Test _load_credentials method."""
|
||||
|
||||
@patch("backend.services.dropbox_service.secretmanager.SecretManagerServiceClient")
|
||||
def test_load_credentials_success(self, mock_sm_client_class):
|
||||
"""Test successful credential loading from Secret Manager."""
|
||||
from backend.services.dropbox_service import DropboxService
|
||||
|
||||
mock_sm_client = Mock()
|
||||
mock_response = Mock()
|
||||
mock_response.payload.data = json.dumps({
|
||||
"access_token": "access-token-123",
|
||||
"refresh_token": "refresh-token-456",
|
||||
"app_key": "app-key",
|
||||
"app_secret": "app-secret",
|
||||
}).encode("UTF-8")
|
||||
mock_sm_client.access_secret_version.return_value = mock_response
|
||||
mock_sm_client_class.return_value = mock_sm_client
|
||||
|
||||
service = DropboxService()
|
||||
creds = service._load_credentials()
|
||||
|
||||
assert creds is not None
|
||||
assert creds["access_token"] == "access-token-123"
|
||||
|
||||
@patch("backend.services.dropbox_service.secretmanager.SecretManagerServiceClient")
|
||||
def test_load_credentials_failure(self, mock_sm_client_class):
|
||||
"""Test handling when Secret Manager fails."""
|
||||
from backend.services.dropbox_service import DropboxService
|
||||
|
||||
mock_sm_client = Mock()
|
||||
mock_sm_client.access_secret_version.side_effect = Exception("Access denied")
|
||||
mock_sm_client_class.return_value = mock_sm_client
|
||||
|
||||
service = DropboxService()
|
||||
creds = service._load_credentials()
|
||||
|
||||
assert creds is None
|
||||
|
||||
|
||||
class TestIsConfigured:
|
||||
"""Test is_configured property."""
|
||||
|
||||
@patch("backend.services.dropbox_service.secretmanager.SecretManagerServiceClient")
|
||||
def test_is_configured_true(self, mock_sm_client_class):
|
||||
"""Test is_configured returns True when credentials available."""
|
||||
from backend.services.dropbox_service import DropboxService
|
||||
|
||||
mock_sm_client = Mock()
|
||||
mock_response = Mock()
|
||||
mock_response.payload.data = json.dumps({
|
||||
"access_token": "token"
|
||||
}).encode("UTF-8")
|
||||
mock_sm_client.access_secret_version.return_value = mock_response
|
||||
mock_sm_client_class.return_value = mock_sm_client
|
||||
|
||||
service = DropboxService()
|
||||
|
||||
assert service.is_configured is True
|
||||
|
||||
@patch("backend.services.dropbox_service.secretmanager.SecretManagerServiceClient")
|
||||
def test_is_configured_false_no_token(self, mock_sm_client_class):
|
||||
"""Test is_configured returns False when no access_token."""
|
||||
from backend.services.dropbox_service import DropboxService
|
||||
|
||||
mock_sm_client = Mock()
|
||||
mock_response = Mock()
|
||||
mock_response.payload.data = json.dumps({
|
||||
"refresh_token": "refresh" # Missing access_token
|
||||
}).encode("UTF-8")
|
||||
mock_sm_client.access_secret_version.return_value = mock_response
|
||||
mock_sm_client_class.return_value = mock_sm_client
|
||||
|
||||
service = DropboxService()
|
||||
|
||||
assert service.is_configured is False
|
||||
|
||||
|
||||
class TestDropboxClient:
|
||||
"""Test client property."""
|
||||
|
||||
@patch("dropbox.Dropbox")
|
||||
@patch("backend.services.dropbox_service.secretmanager.SecretManagerServiceClient")
|
||||
def test_client_creates_dropbox_instance(self, mock_sm_client_class, mock_dropbox_class):
|
||||
"""Test client property creates Dropbox SDK client."""
|
||||
from backend.services.dropbox_service import DropboxService
|
||||
|
||||
mock_sm_client = Mock()
|
||||
mock_response = Mock()
|
||||
mock_response.payload.data = json.dumps({
|
||||
"access_token": "token",
|
||||
"refresh_token": "refresh",
|
||||
"app_key": "key",
|
||||
"app_secret": "secret",
|
||||
}).encode("UTF-8")
|
||||
mock_sm_client.access_secret_version.return_value = mock_response
|
||||
mock_sm_client_class.return_value = mock_sm_client
|
||||
|
||||
mock_dropbox = Mock()
|
||||
mock_dropbox_class.return_value = mock_dropbox
|
||||
|
||||
service = DropboxService()
|
||||
client = service.client
|
||||
|
||||
mock_dropbox_class.assert_called_once_with(
|
||||
oauth2_access_token="token",
|
||||
oauth2_refresh_token="refresh",
|
||||
app_key="key",
|
||||
app_secret="secret",
|
||||
)
|
||||
assert client == mock_dropbox
|
||||
|
||||
@patch("backend.services.dropbox_service.secretmanager.SecretManagerServiceClient")
|
||||
def test_client_raises_on_missing_credentials(self, mock_sm_client_class):
|
||||
"""Test client raises RuntimeError when credentials missing."""
|
||||
from backend.services.dropbox_service import DropboxService
|
||||
|
||||
mock_sm_client = Mock()
|
||||
mock_sm_client.access_secret_version.side_effect = Exception("Not found")
|
||||
mock_sm_client_class.return_value = mock_sm_client
|
||||
|
||||
service = DropboxService()
|
||||
|
||||
with pytest.raises(RuntimeError) as exc_info:
|
||||
_ = service.client
|
||||
|
||||
assert "not configured" in str(exc_info.value)
|
||||
|
||||
|
||||
class TestListFolders:
|
||||
"""Test list_folders method."""
|
||||
|
||||
@patch("backend.services.dropbox_service.secretmanager.SecretManagerServiceClient")
|
||||
def test_list_folders(self, mock_sm_client_class):
|
||||
"""Test listing folders at a path."""
|
||||
from backend.services.dropbox_service import DropboxService
|
||||
from dropbox.files import FolderMetadata
|
||||
|
||||
mock_sm_client = Mock()
|
||||
mock_response = Mock()
|
||||
mock_response.payload.data = json.dumps({"access_token": "token"}).encode()
|
||||
mock_sm_client.access_secret_version.return_value = mock_response
|
||||
mock_sm_client_class.return_value = mock_sm_client
|
||||
|
||||
# Create mock folder entries
|
||||
mock_folder1 = Mock(spec=FolderMetadata)
|
||||
mock_folder1.name = "NOMAD-0001"
|
||||
mock_folder2 = Mock(spec=FolderMetadata)
|
||||
mock_folder2.name = "NOMAD-0002"
|
||||
mock_file = Mock() # Not a FolderMetadata
|
||||
|
||||
mock_result = Mock()
|
||||
mock_result.entries = [mock_folder1, mock_folder2, mock_file]
|
||||
mock_result.has_more = False
|
||||
|
||||
service = DropboxService()
|
||||
# Directly set the client to avoid needing to mock the whole init chain
|
||||
mock_dropbox = Mock()
|
||||
mock_dropbox.files_list_folder.return_value = mock_result
|
||||
service._client = mock_dropbox
|
||||
|
||||
folders = service.list_folders("/Karaoke/Tracks")
|
||||
|
||||
mock_dropbox.files_list_folder.assert_called_once_with("/Karaoke/Tracks")
|
||||
assert folders == ["NOMAD-0001", "NOMAD-0002"]
|
||||
|
||||
@patch("backend.services.dropbox_service.secretmanager.SecretManagerServiceClient")
|
||||
def test_list_folders_adds_leading_slash(self, mock_sm_client_class):
|
||||
"""Test that path without leading slash gets one added."""
|
||||
from backend.services.dropbox_service import DropboxService
|
||||
|
||||
mock_sm_client = Mock()
|
||||
mock_response = Mock()
|
||||
mock_response.payload.data = json.dumps({"access_token": "token"}).encode()
|
||||
mock_sm_client.access_secret_version.return_value = mock_response
|
||||
mock_sm_client_class.return_value = mock_sm_client
|
||||
|
||||
mock_result = Mock()
|
||||
mock_result.entries = []
|
||||
mock_result.has_more = False
|
||||
|
||||
service = DropboxService()
|
||||
mock_dropbox = Mock()
|
||||
mock_dropbox.files_list_folder.return_value = mock_result
|
||||
service._client = mock_dropbox
|
||||
|
||||
service.list_folders("path/without/slash")
|
||||
|
||||
mock_dropbox.files_list_folder.assert_called_once_with("/path/without/slash")
|
||||
|
||||
|
||||
class TestGetNextBrandCode:
|
||||
"""Test get_next_brand_code method."""
|
||||
|
||||
@patch("backend.services.dropbox_service.secretmanager.SecretManagerServiceClient")
|
||||
def test_get_next_brand_code(self, mock_sm_client_class):
|
||||
"""Test calculating next brand code from existing folders."""
|
||||
from backend.services.dropbox_service import DropboxService
|
||||
|
||||
mock_sm_client = Mock()
|
||||
mock_response = Mock()
|
||||
mock_response.payload.data = json.dumps({"access_token": "token"}).encode()
|
||||
mock_sm_client.access_secret_version.return_value = mock_response
|
||||
mock_sm_client_class.return_value = mock_sm_client
|
||||
|
||||
service = DropboxService()
|
||||
|
||||
# Mock list_folders to return existing codes
|
||||
with patch.object(service, "list_folders") as mock_list:
|
||||
mock_list.return_value = [
|
||||
"NOMAD-1161",
|
||||
"NOMAD-1162",
|
||||
"NOMAD-1163",
|
||||
"Other Folder",
|
||||
"NOMAD-0001",
|
||||
]
|
||||
|
||||
next_code = service.get_next_brand_code("/path", "NOMAD")
|
||||
|
||||
assert next_code == "NOMAD-1164"
|
||||
|
||||
@patch("backend.services.dropbox_service.secretmanager.SecretManagerServiceClient")
|
||||
def test_get_next_brand_code_empty_folder(self, mock_sm_client_class):
|
||||
"""Test brand code calculation with no existing codes."""
|
||||
from backend.services.dropbox_service import DropboxService
|
||||
|
||||
mock_sm_client = Mock()
|
||||
mock_response = Mock()
|
||||
mock_response.payload.data = json.dumps({"access_token": "token"}).encode()
|
||||
mock_sm_client.access_secret_version.return_value = mock_response
|
||||
mock_sm_client_class.return_value = mock_sm_client
|
||||
|
||||
service = DropboxService()
|
||||
|
||||
with patch.object(service, "list_folders") as mock_list:
|
||||
mock_list.return_value = []
|
||||
|
||||
next_code = service.get_next_brand_code("/path", "BRAND")
|
||||
|
||||
assert next_code == "BRAND-0001"
|
||||
|
||||
|
||||
class TestUploadFile:
|
||||
"""Test upload_file method."""
|
||||
|
||||
@patch("backend.services.dropbox_service.secretmanager.SecretManagerServiceClient")
|
||||
def test_upload_small_file(self, mock_sm_client_class, tmp_path):
|
||||
"""Test uploading a small file directly."""
|
||||
from backend.services.dropbox_service import DropboxService
|
||||
|
||||
mock_sm_client = Mock()
|
||||
mock_response = Mock()
|
||||
mock_response.payload.data = json.dumps({"access_token": "token"}).encode()
|
||||
mock_sm_client.access_secret_version.return_value = mock_response
|
||||
mock_sm_client_class.return_value = mock_sm_client
|
||||
|
||||
# Create test file
|
||||
test_file = tmp_path / "test.txt"
|
||||
test_file.write_text("Small file content")
|
||||
|
||||
service = DropboxService()
|
||||
mock_dropbox = Mock()
|
||||
service._client = mock_dropbox
|
||||
|
||||
service.upload_file(str(test_file), "/Uploads/test.txt")
|
||||
|
||||
mock_dropbox.files_upload.assert_called_once()
|
||||
|
||||
@patch("backend.services.dropbox_service.secretmanager.SecretManagerServiceClient")
|
||||
def test_upload_file_adds_leading_slash(self, mock_sm_client_class, tmp_path):
|
||||
"""Test upload adds leading slash to remote path."""
|
||||
from backend.services.dropbox_service import DropboxService
|
||||
|
||||
mock_sm_client = Mock()
|
||||
mock_response = Mock()
|
||||
mock_response.payload.data = json.dumps({"access_token": "token"}).encode()
|
||||
mock_sm_client.access_secret_version.return_value = mock_response
|
||||
mock_sm_client_class.return_value = mock_sm_client
|
||||
|
||||
test_file = tmp_path / "test.txt"
|
||||
test_file.write_text("content")
|
||||
|
||||
service = DropboxService()
|
||||
mock_dropbox = Mock()
|
||||
service._client = mock_dropbox
|
||||
|
||||
service.upload_file(str(test_file), "uploads/test.txt")
|
||||
|
||||
# Check that the path has leading slash
|
||||
call_args = mock_dropbox.files_upload.call_args
|
||||
assert call_args[0][1] == "/uploads/test.txt"
|
||||
|
||||
|
||||
class TestUploadFolder:
|
||||
"""Test upload_folder method."""
|
||||
|
||||
@patch("backend.services.dropbox_service.secretmanager.SecretManagerServiceClient")
|
||||
def test_upload_folder(self, mock_sm_client_class, tmp_path):
|
||||
"""Test uploading a folder with multiple files."""
|
||||
from backend.services.dropbox_service import DropboxService
|
||||
|
||||
mock_sm_client = Mock()
|
||||
mock_response = Mock()
|
||||
mock_response.payload.data = json.dumps({"access_token": "token"}).encode()
|
||||
mock_sm_client.access_secret_version.return_value = mock_response
|
||||
mock_sm_client_class.return_value = mock_sm_client
|
||||
|
||||
# Create test folder with files
|
||||
(tmp_path / "file1.txt").write_text("content1")
|
||||
(tmp_path / "file2.txt").write_text("content2")
|
||||
(tmp_path / "subdir").mkdir() # Should be skipped
|
||||
|
||||
service = DropboxService()
|
||||
|
||||
with patch.object(service, "upload_file") as mock_upload:
|
||||
service.upload_folder(str(tmp_path), "/Uploads/folder")
|
||||
|
||||
# Should upload 2 files (not subdir)
|
||||
assert mock_upload.call_count == 2
|
||||
|
||||
|
||||
class TestCreateSharedLink:
|
||||
"""Test create_shared_link method."""
|
||||
|
||||
@patch("backend.services.dropbox_service.secretmanager.SecretManagerServiceClient")
|
||||
def test_create_shared_link_new(self, mock_sm_client_class):
|
||||
"""Test creating a new shared link."""
|
||||
from backend.services.dropbox_service import DropboxService
|
||||
|
||||
mock_sm_client = Mock()
|
||||
mock_response = Mock()
|
||||
mock_response.payload.data = json.dumps({"access_token": "token"}).encode()
|
||||
mock_sm_client.access_secret_version.return_value = mock_response
|
||||
mock_sm_client_class.return_value = mock_sm_client
|
||||
|
||||
mock_link = Mock()
|
||||
mock_link.url = "https://dropbox.com/s/abc123/file.mp4"
|
||||
|
||||
service = DropboxService()
|
||||
mock_dropbox = Mock()
|
||||
mock_dropbox.sharing_create_shared_link_with_settings.return_value = mock_link
|
||||
service._client = mock_dropbox
|
||||
|
||||
url = service.create_shared_link("/Uploads/file.mp4")
|
||||
|
||||
assert url == "https://dropbox.com/s/abc123/file.mp4"
|
||||
|
||||
@patch("backend.services.dropbox_service.secretmanager.SecretManagerServiceClient")
|
||||
def test_create_shared_link_existing(self, mock_sm_client_class):
|
||||
"""Test getting existing shared link when one already exists."""
|
||||
from backend.services.dropbox_service import DropboxService
|
||||
from dropbox.exceptions import ApiError
|
||||
|
||||
mock_sm_client = Mock()
|
||||
mock_response = Mock()
|
||||
mock_response.payload.data = json.dumps({"access_token": "token"}).encode()
|
||||
mock_sm_client.access_secret_version.return_value = mock_response
|
||||
mock_sm_client_class.return_value = mock_sm_client
|
||||
|
||||
# Simulate "link already exists" error
|
||||
mock_error = Mock()
|
||||
mock_error.is_shared_link_already_exists.return_value = True
|
||||
|
||||
mock_existing_link = Mock()
|
||||
mock_existing_link.url = "https://dropbox.com/s/existing/file.mp4"
|
||||
|
||||
mock_links_result = Mock()
|
||||
mock_links_result.links = [mock_existing_link]
|
||||
|
||||
service = DropboxService()
|
||||
mock_dropbox = Mock()
|
||||
mock_dropbox.sharing_create_shared_link_with_settings.side_effect = \
|
||||
ApiError("req_id", mock_error, "message", "headers")
|
||||
mock_dropbox.sharing_list_shared_links.return_value = mock_links_result
|
||||
service._client = mock_dropbox
|
||||
|
||||
url = service.create_shared_link("/Uploads/file.mp4")
|
||||
|
||||
assert url == "https://dropbox.com/s/existing/file.mp4"
|
||||
|
||||
@@ -1,322 +0,0 @@
|
||||
"""
|
||||
Integration tests using local GCP emulators.
|
||||
|
||||
These tests use real Firestore and GCS emulators running locally,
|
||||
providing true integration testing without cloud resources or costs.
|
||||
|
||||
Run with: ./scripts/run-emulator-tests.sh
|
||||
|
||||
NOTE: These are simplified integration tests that focus on the most critical
|
||||
paths. They mock background workers to avoid race conditions and network issues.
|
||||
"""
|
||||
import pytest
|
||||
import os
|
||||
import time
|
||||
from unittest.mock import AsyncMock, patch
|
||||
from datetime import datetime, UTC
|
||||
import requests
|
||||
|
||||
|
||||
def emulators_running() -> bool:
|
||||
"""Check if GCP emulators are running."""
|
||||
try:
|
||||
requests.get("http://127.0.0.1:8080", timeout=1)
|
||||
requests.get("http://127.0.0.1:4443", timeout=1)
|
||||
return True
|
||||
except (requests.exceptions.ConnectionError, requests.exceptions.Timeout):
|
||||
return False
|
||||
|
||||
|
||||
# Skip all tests in this module if emulators aren't running
|
||||
pytestmark = pytest.mark.skipif(
|
||||
not emulators_running(),
|
||||
reason="GCP emulators not running. Start with: scripts/start-emulators.sh"
|
||||
)
|
||||
|
||||
# Set emulator environment variables before any imports (use 127.0.0.1 for IPv4)
|
||||
os.environ["FIRESTORE_EMULATOR_HOST"] = "127.0.0.1:8080"
|
||||
os.environ["STORAGE_EMULATOR_HOST"] = "http://127.0.0.1:4443"
|
||||
os.environ["GOOGLE_CLOUD_PROJECT"] = "test-project"
|
||||
os.environ["GCS_BUCKET_NAME"] = "test-bucket"
|
||||
os.environ["FIRESTORE_COLLECTION"] = "test-jobs" # Use separate collection for tests
|
||||
os.environ["ENVIRONMENT"] = "test"
|
||||
os.environ["ADMIN_TOKENS"] = "test-admin-token"
|
||||
|
||||
# Only import if emulators are running
|
||||
if emulators_running():
|
||||
from fastapi.testclient import TestClient
|
||||
from backend.main import app
|
||||
from backend.models.job import JobStatus
|
||||
else:
|
||||
TestClient = None
|
||||
app = None
|
||||
JobStatus = None
|
||||
|
||||
|
||||
@pytest.fixture(scope="module", autouse=True)
|
||||
def setup_gcs_bucket():
|
||||
"""Create GCS bucket in emulator before tests."""
|
||||
try:
|
||||
response = requests.post(
|
||||
"http://localhost:4443/storage/v1/b",
|
||||
json={"name": "test-bucket"},
|
||||
params={"project": "test-project"}
|
||||
)
|
||||
if response.status_code in [200, 409]:
|
||||
print(f"✅ GCS bucket 'test-bucket' ready")
|
||||
except Exception as e:
|
||||
print(f"⚠️ GCS bucket setup failed: {e}")
|
||||
yield
|
||||
|
||||
|
||||
@pytest.fixture(scope="module")
|
||||
def mock_worker_service():
|
||||
"""Mock the worker service to prevent background tasks."""
|
||||
with patch("backend.api.routes.jobs.worker_service") as mock:
|
||||
mock.trigger_audio_worker = AsyncMock(return_value=True)
|
||||
mock.trigger_lyrics_worker = AsyncMock(return_value=True)
|
||||
mock.trigger_screens_worker = AsyncMock(return_value=True)
|
||||
mock.trigger_video_worker = AsyncMock(return_value=True)
|
||||
yield mock
|
||||
|
||||
|
||||
@pytest.fixture(scope="module")
|
||||
def client(mock_worker_service):
|
||||
"""Create FastAPI test client with mocked workers."""
|
||||
with patch("backend.api.routes.file_upload.worker_service", mock_worker_service):
|
||||
return TestClient(app)
|
||||
|
||||
|
||||
@pytest.fixture
|
||||
def auth_headers():
|
||||
"""Auth headers for testing."""
|
||||
return {"Authorization": "Bearer test-admin-token"}
|
||||
|
||||
|
||||
class TestEmulatorBasics:
|
||||
"""Basic emulator connectivity tests."""
|
||||
|
||||
def test_health_endpoint(self, client, auth_headers):
|
||||
"""Test health endpoint works."""
|
||||
response = client.get("/api/health", )
|
||||
assert response.status_code == 200
|
||||
assert response.json()["status"] == "healthy"
|
||||
|
||||
def test_root_endpoint(self, client, auth_headers):
|
||||
"""Test root endpoint works."""
|
||||
response = client.get("/", )
|
||||
assert response.status_code == 200
|
||||
assert response.json()["service"] == "karaoke-gen-backend"
|
||||
|
||||
|
||||
class TestJobCreation:
|
||||
"""Test job creation with Firestore emulator."""
|
||||
|
||||
def test_create_job_simple(self, client, auth_headers):
|
||||
"""Test creating a simple job."""
|
||||
response = client.post(
|
||||
"/api/jobs",
|
||||
headers=auth_headers,
|
||||
json={"url": "https://youtube.com/watch?v=test123"}
|
||||
)
|
||||
|
||||
assert response.status_code == 200
|
||||
data = response.json()
|
||||
assert data["status"] == "success"
|
||||
assert "job_id" in data
|
||||
|
||||
def test_create_job_with_metadata(self, client, auth_headers):
|
||||
"""Test creating a job with artist/title."""
|
||||
response = client.post(
|
||||
"/api/jobs",
|
||||
headers=auth_headers,
|
||||
json={
|
||||
"url": "https://youtube.com/watch?v=test",
|
||||
"artist": "Test Artist",
|
||||
"title": "Test Song"
|
||||
}
|
||||
)
|
||||
|
||||
assert response.status_code == 200
|
||||
data = response.json()
|
||||
assert "job_id" in data
|
||||
|
||||
|
||||
class TestJobRetrieval:
|
||||
"""Test job retrieval from Firestore emulator."""
|
||||
|
||||
def test_create_and_get_job(self, client, auth_headers):
|
||||
"""Test creating and then retrieving a job."""
|
||||
# Create
|
||||
create_resp = client.post(
|
||||
"/api/jobs",
|
||||
headers=auth_headers,
|
||||
json={
|
||||
"url": "https://youtube.com/watch?v=abc123",
|
||||
"artist": "Test Artist",
|
||||
"title": "Test Song"
|
||||
}
|
||||
)
|
||||
assert create_resp.status_code == 200
|
||||
job_id = create_resp.json()["job_id"]
|
||||
|
||||
# Small delay for emulator consistency
|
||||
time.sleep(0.2)
|
||||
|
||||
# Retrieve
|
||||
get_resp = client.get(f"/api/jobs/{job_id}", headers=auth_headers)
|
||||
|
||||
if get_resp.status_code != 200:
|
||||
print(f"GET failed: {get_resp.status_code} - {get_resp.text}")
|
||||
|
||||
assert get_resp.status_code == 200
|
||||
|
||||
job = get_resp.json()
|
||||
assert job["job_id"] == job_id
|
||||
assert job["status"] == "pending"
|
||||
assert job["artist"] == "Test Artist"
|
||||
assert job["title"] == "Test Song"
|
||||
|
||||
def test_get_nonexistent_job(self, client, auth_headers):
|
||||
"""Test fetching a job that doesn't exist."""
|
||||
response = client.get("/api/jobs/nonexistent-id", headers=auth_headers)
|
||||
assert response.status_code == 404
|
||||
|
||||
|
||||
class TestJobList:
|
||||
"""Test listing jobs from Firestore."""
|
||||
|
||||
def test_list_jobs(self, client, auth_headers):
|
||||
"""Test listing jobs."""
|
||||
# Create a few jobs
|
||||
for i in range(3):
|
||||
client.post(
|
||||
"/api/jobs",
|
||||
headers=auth_headers,
|
||||
json={"url": f"https://youtube.com/watch?v=list{i}"}
|
||||
)
|
||||
|
||||
time.sleep(0.2)
|
||||
|
||||
# List
|
||||
response = client.get("/api/jobs", headers=auth_headers)
|
||||
assert response.status_code == 200
|
||||
|
||||
jobs = response.json()
|
||||
assert isinstance(jobs, list)
|
||||
# We should have at least the 3 we just created
|
||||
assert len(jobs) >= 3
|
||||
|
||||
|
||||
class TestJobDeletion:
|
||||
"""Test job deletion."""
|
||||
|
||||
def test_delete_job(self, client, auth_headers):
|
||||
"""Test deleting a job."""
|
||||
# Create
|
||||
create_resp = client.post(
|
||||
"/api/jobs",
|
||||
headers=auth_headers,
|
||||
json={"url": "https://youtube.com/watch?v=delete-me"}
|
||||
)
|
||||
job_id = create_resp.json()["job_id"]
|
||||
|
||||
time.sleep(0.2)
|
||||
|
||||
# Delete
|
||||
del_resp = client.delete(f"/api/jobs/{job_id}", headers=auth_headers)
|
||||
assert del_resp.status_code == 200
|
||||
|
||||
# Verify deleted
|
||||
get_resp = client.get(f"/api/jobs/{job_id}", headers=auth_headers)
|
||||
assert get_resp.status_code == 404
|
||||
|
||||
|
||||
class TestJobUpdates:
|
||||
"""Test job status updates."""
|
||||
|
||||
def test_cancel_job(self, client, auth_headers):
|
||||
"""Test cancelling a job."""
|
||||
# Create
|
||||
create_resp = client.post(
|
||||
"/api/jobs",
|
||||
headers=auth_headers,
|
||||
json={"url": "https://youtube.com/watch?v=cancel-me"}
|
||||
)
|
||||
job_id = create_resp.json()["job_id"]
|
||||
|
||||
time.sleep(0.2)
|
||||
|
||||
# Cancel
|
||||
cancel_resp = client.post(
|
||||
f"/api/jobs/{job_id}/cancel",
|
||||
headers=auth_headers,
|
||||
json={"reason": "test cancellation"}
|
||||
)
|
||||
assert cancel_resp.status_code == 200
|
||||
|
||||
# Verify cancelled
|
||||
get_resp = client.get(f"/api/jobs/{job_id}", headers=auth_headers)
|
||||
assert get_resp.status_code == 200
|
||||
job = get_resp.json()
|
||||
assert job["status"] == "cancelled"
|
||||
|
||||
|
||||
class TestFileUpload:
|
||||
"""Test file upload with GCS emulator."""
|
||||
|
||||
def test_upload_file(self, client, auth_headers):
|
||||
"""Test uploading a file."""
|
||||
test_file_content = b"fake audio data for testing"
|
||||
|
||||
response = client.post(
|
||||
"/api/jobs/upload",
|
||||
headers={"Authorization": auth_headers["Authorization"]},
|
||||
files={"file": ("test.flac", test_file_content, "audio/flac")},
|
||||
data={"artist": "Upload Artist", "title": "Upload Song"}
|
||||
)
|
||||
|
||||
assert response.status_code == 200
|
||||
data = response.json()
|
||||
assert data["status"] == "success"
|
||||
assert "job_id" in data
|
||||
|
||||
job_id = data["job_id"]
|
||||
|
||||
time.sleep(0.2)
|
||||
|
||||
# Verify job created with upload data
|
||||
get_resp = client.get(f"/api/jobs/{job_id}", headers=auth_headers)
|
||||
assert get_resp.status_code == 200
|
||||
job = get_resp.json()
|
||||
assert job["artist"] == "Upload Artist"
|
||||
assert job["title"] == "Upload Song"
|
||||
assert "input_media_gcs_path" in job
|
||||
|
||||
|
||||
class TestInternalEndpoints:
|
||||
"""Test internal worker endpoints."""
|
||||
|
||||
def test_internal_workers_exist(self, client, auth_headers):
|
||||
"""Test that internal worker endpoints exist and respond."""
|
||||
# Create a job first
|
||||
create_resp = client.post(
|
||||
"/api/jobs",
|
||||
headers=auth_headers,
|
||||
json={"url": "https://youtube.com/watch?v=worker-test"}
|
||||
)
|
||||
job_id = create_resp.json()["job_id"]
|
||||
|
||||
time.sleep(0.2)
|
||||
|
||||
# Test audio worker endpoint exists
|
||||
response = client.post(
|
||||
"/api/internal/workers/audio",
|
||||
headers=auth_headers,
|
||||
json={"job_id": job_id}
|
||||
)
|
||||
assert response.status_code == 200
|
||||
assert response.json()["status"] == "started"
|
||||
|
||||
|
||||
print("✅ Emulator integration tests ready to run")
|
||||
File diff suppressed because it is too large.
Load diff
@@ -1,632 +0,0 @@
|
||||
"""
|
||||
Tests for the flacfetch remote HTTP API client.
|
||||
|
||||
This module tests FlacfetchClient which handles communication with the
|
||||
dedicated flacfetch VM for torrent/audio downloads.
|
||||
|
||||
These tests use mocked HTTP responses to verify:
|
||||
- Request formatting (headers, payloads)
|
||||
- Response parsing
|
||||
- Error handling
|
||||
- State management (singleton pattern)
|
||||
"""
|
||||
import pytest
|
||||
from unittest.mock import AsyncMock, Mock, patch, MagicMock
|
||||
import asyncio
|
||||
|
||||
import httpx
|
||||
|
||||
from backend.services.flacfetch_client import (
|
||||
FlacfetchClient,
|
||||
FlacfetchServiceError,
|
||||
get_flacfetch_client,
|
||||
reset_flacfetch_client,
|
||||
)
|
||||
|
||||
|
||||
class TestFlacfetchClientInit:
|
||||
"""Test FlacfetchClient initialization."""
|
||||
|
||||
def test_init_with_required_params(self):
|
||||
"""Test client initializes with base_url and api_key."""
|
||||
client = FlacfetchClient(
|
||||
base_url="http://localhost:8080",
|
||||
api_key="test-key-123",
|
||||
)
|
||||
|
||||
assert client.base_url == "http://localhost:8080"
|
||||
assert client.api_key == "test-key-123"
|
||||
assert client.timeout == 60 # Default timeout
|
||||
|
||||
def test_init_strips_trailing_slash(self):
|
||||
"""Test base_url trailing slash is stripped."""
|
||||
client = FlacfetchClient(
|
||||
base_url="http://localhost:8080/",
|
||||
api_key="test-key",
|
||||
)
|
||||
|
||||
assert client.base_url == "http://localhost:8080"
|
||||
|
||||
def test_init_with_custom_timeout(self):
|
||||
"""Test custom timeout is set."""
|
||||
client = FlacfetchClient(
|
||||
base_url="http://localhost:8080",
|
||||
api_key="test-key",
|
||||
timeout=120,
|
||||
)
|
||||
|
||||
assert client.timeout == 120
|
||||
|
||||
def test_headers_include_api_key(self):
|
||||
"""Test _headers() includes X-API-Key."""
|
||||
client = FlacfetchClient(
|
||||
base_url="http://localhost:8080",
|
||||
api_key="secret-key-456",
|
||||
)
|
||||
|
||||
headers = client._headers()
|
||||
|
||||
assert headers["X-API-Key"] == "secret-key-456"
|
||||
assert headers["Content-Type"] == "application/json"
|
||||
|
||||
|
||||
class TestFlacfetchClientHealthCheck:
|
||||
"""Test health_check() method."""
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_health_check_success(self):
|
||||
"""Test successful health check."""
|
||||
client = FlacfetchClient(
|
||||
base_url="http://localhost:8080",
|
||||
api_key="test-key",
|
||||
)
|
||||
|
||||
mock_response = Mock()
|
||||
mock_response.json.return_value = {
|
||||
"status": "healthy",
|
||||
"transmission": {"available": True, "version": "4.0.5"},
|
||||
"disk": {"free_gb": 15.5},
|
||||
}
|
||||
mock_response.raise_for_status = Mock()
|
||||
|
||||
with patch("httpx.AsyncClient") as mock_client_class:
|
||||
mock_client = AsyncMock()
|
||||
mock_client.get = AsyncMock(return_value=mock_response)
|
||||
mock_client.__aenter__ = AsyncMock(return_value=mock_client)
|
||||
mock_client.__aexit__ = AsyncMock(return_value=None)
|
||||
mock_client_class.return_value = mock_client
|
||||
|
||||
result = await client.health_check()
|
||||
|
||||
assert result["status"] == "healthy"
|
||||
assert result["transmission"]["available"] is True
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_health_check_degraded_status(self):
|
||||
"""Test health check with degraded status still succeeds."""
|
||||
client = FlacfetchClient(
|
||||
base_url="http://localhost:8080",
|
||||
api_key="test-key",
|
||||
)
|
||||
|
||||
mock_response = Mock()
|
||||
mock_response.json.return_value = {"status": "degraded"}
|
||||
mock_response.raise_for_status = Mock()
|
||||
|
||||
with patch("httpx.AsyncClient") as mock_client_class:
|
||||
mock_client = AsyncMock()
|
||||
mock_client.get = AsyncMock(return_value=mock_response)
|
||||
mock_client.__aenter__ = AsyncMock(return_value=mock_client)
|
||||
mock_client.__aexit__ = AsyncMock(return_value=None)
|
||||
mock_client_class.return_value = mock_client
|
||||
|
||||
result = await client.health_check()
|
||||
|
||||
assert result["status"] == "degraded"
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_health_check_unhealthy_raises_error(self):
|
||||
"""Test health check with unhealthy status raises error."""
|
||||
client = FlacfetchClient(
|
||||
base_url="http://localhost:8080",
|
||||
api_key="test-key",
|
||||
)
|
||||
|
||||
mock_response = Mock()
|
||||
mock_response.json.return_value = {"status": "unhealthy", "error": "Transmission down"}
|
||||
mock_response.raise_for_status = Mock()
|
||||
|
||||
with patch("httpx.AsyncClient") as mock_client_class:
|
||||
mock_client = AsyncMock()
|
||||
mock_client.get = AsyncMock(return_value=mock_response)
|
||||
mock_client.__aenter__ = AsyncMock(return_value=mock_client)
|
||||
mock_client.__aexit__ = AsyncMock(return_value=None)
|
||||
mock_client_class.return_value = mock_client
|
||||
|
||||
with pytest.raises(FlacfetchServiceError) as exc_info:
|
||||
await client.health_check()
|
||||
|
||||
assert "unhealthy" in str(exc_info.value)
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_health_check_network_error(self):
|
||||
"""Test health check handles network errors."""
|
||||
client = FlacfetchClient(
|
||||
base_url="http://localhost:8080",
|
||||
api_key="test-key",
|
||||
)
|
||||
|
||||
with patch("httpx.AsyncClient") as mock_client_class:
|
||||
mock_client = AsyncMock()
|
||||
mock_client.get = AsyncMock(side_effect=httpx.RequestError("Connection refused"))
|
||||
mock_client.__aenter__ = AsyncMock(return_value=mock_client)
|
||||
mock_client.__aexit__ = AsyncMock(return_value=None)
|
||||
mock_client_class.return_value = mock_client
|
||||
|
||||
with pytest.raises(FlacfetchServiceError) as exc_info:
|
||||
await client.health_check()
|
||||
|
||||
assert "Cannot reach" in str(exc_info.value)
|
||||
|
||||
|
||||
class TestFlacfetchClientSearch:
|
||||
"""Test search() method."""
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_search_success(self):
|
||||
"""Test successful search returns results."""
|
||||
client = FlacfetchClient(
|
||||
base_url="http://localhost:8080",
|
||||
api_key="test-key",
|
||||
)
|
||||
|
||||
mock_response = Mock()
|
||||
mock_response.status_code = 200
|
||||
mock_response.json.return_value = {
|
||||
"search_id": "search_abc123",
|
||||
"results": [
|
||||
{
|
||||
"title": "Waterloo",
|
||||
"artist": "ABBA",
|
||||
"provider": "RED",
|
||||
"quality": "FLAC 16bit CD",
|
||||
"seeders": 50,
|
||||
"index": 0,
|
||||
}
|
||||
],
|
||||
"results_count": 1,
|
||||
}
|
||||
mock_response.raise_for_status = Mock()
|
||||
|
||||
with patch("httpx.AsyncClient") as mock_client_class:
|
||||
mock_client = AsyncMock()
|
||||
mock_client.post = AsyncMock(return_value=mock_response)
|
||||
mock_client.__aenter__ = AsyncMock(return_value=mock_client)
|
||||
mock_client.__aexit__ = AsyncMock(return_value=None)
|
||||
mock_client_class.return_value = mock_client
|
||||
|
||||
result = await client.search("ABBA", "Waterloo")
|
||||
|
||||
assert result["search_id"] == "search_abc123"
|
||||
assert len(result["results"]) == 1
|
||||
assert result["results"][0]["title"] == "Waterloo"
|
||||
|
||||
# Verify request was made correctly
|
||||
mock_client.post.assert_called_once()
|
||||
call_args = mock_client.post.call_args
|
||||
assert call_args.kwargs["json"] == {"artist": "ABBA", "title": "Waterloo"}
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_search_no_results_returns_empty(self):
|
||||
"""Test search with no results returns empty list."""
|
||||
client = FlacfetchClient(
|
||||
base_url="http://localhost:8080",
|
||||
api_key="test-key",
|
||||
)
|
||||
|
||||
mock_response = Mock()
|
||||
mock_response.status_code = 404
|
||||
|
||||
with patch("httpx.AsyncClient") as mock_client_class:
|
||||
mock_client = AsyncMock()
|
||||
mock_client.post = AsyncMock(return_value=mock_response)
|
||||
mock_client.__aenter__ = AsyncMock(return_value=mock_client)
|
||||
mock_client.__aexit__ = AsyncMock(return_value=None)
|
||||
mock_client_class.return_value = mock_client
|
||||
|
||||
result = await client.search("Unknown", "Artist")
|
||||
|
||||
assert result["results"] == []
|
||||
assert result["search_id"] is None
|
||||
|
||||
|
||||
class TestFlacfetchClientDownload:
|
||||
"""Test download() method."""
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_download_starts_successfully(self):
|
||||
"""Test download() returns download_id."""
|
||||
client = FlacfetchClient(
|
||||
base_url="http://localhost:8080",
|
||||
api_key="test-key",
|
||||
)
|
||||
|
||||
mock_response = Mock()
|
||||
mock_response.json.return_value = {
|
||||
"download_id": "dl_xyz789",
|
||||
"status": "queued",
|
||||
}
|
||||
mock_response.raise_for_status = Mock()
|
||||
|
||||
with patch("httpx.AsyncClient") as mock_client_class:
|
||||
mock_client = AsyncMock()
|
||||
mock_client.post = AsyncMock(return_value=mock_response)
|
||||
mock_client.__aenter__ = AsyncMock(return_value=mock_client)
|
||||
mock_client.__aexit__ = AsyncMock(return_value=None)
|
||||
mock_client_class.return_value = mock_client
|
||||
|
||||
download_id = await client.download(
|
||||
search_id="search_abc123",
|
||||
result_index=0,
|
||||
)
|
||||
|
||||
assert download_id == "dl_xyz789"
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_download_with_gcs_path(self):
|
||||
"""Test download() includes GCS upload parameters."""
|
||||
client = FlacfetchClient(
|
||||
base_url="http://localhost:8080",
|
||||
api_key="test-key",
|
||||
)
|
||||
|
||||
mock_response = Mock()
|
||||
mock_response.json.return_value = {"download_id": "dl_xyz789"}
|
||||
mock_response.raise_for_status = Mock()
|
||||
|
||||
with patch("httpx.AsyncClient") as mock_client_class:
|
||||
mock_client = AsyncMock()
|
||||
mock_client.post = AsyncMock(return_value=mock_response)
|
||||
mock_client.__aenter__ = AsyncMock(return_value=mock_client)
|
||||
mock_client.__aexit__ = AsyncMock(return_value=None)
|
||||
mock_client_class.return_value = mock_client
|
||||
|
||||
await client.download(
|
||||
search_id="search_abc123",
|
||||
result_index=0,
|
||||
gcs_path="uploads/job123/audio/",
|
||||
)
|
||||
|
||||
# Verify GCS params were included
|
||||
call_args = mock_client.post.call_args
|
||||
payload = call_args.kwargs["json"]
|
||||
assert payload["upload_to_gcs"] is True
|
||||
assert payload["gcs_path"] == "uploads/job123/audio/"
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_download_with_custom_filename(self):
|
||||
"""Test download() with custom output filename."""
|
||||
client = FlacfetchClient(
|
||||
base_url="http://localhost:8080",
|
||||
api_key="test-key",
|
||||
)
|
||||
|
||||
mock_response = Mock()
|
||||
mock_response.json.return_value = {"download_id": "dl_xyz789"}
|
||||
mock_response.raise_for_status = Mock()
|
||||
|
||||
with patch("httpx.AsyncClient") as mock_client_class:
|
||||
mock_client = AsyncMock()
|
||||
mock_client.post = AsyncMock(return_value=mock_response)
|
||||
mock_client.__aenter__ = AsyncMock(return_value=mock_client)
|
||||
mock_client.__aexit__ = AsyncMock(return_value=None)
|
||||
mock_client_class.return_value = mock_client
|
||||
|
||||
await client.download(
|
||||
search_id="search_abc123",
|
||||
result_index=0,
|
||||
output_filename="ABBA - Waterloo",
|
||||
)
|
||||
|
||||
call_args = mock_client.post.call_args
|
||||
payload = call_args.kwargs["json"]
|
||||
assert payload["output_filename"] == "ABBA - Waterloo"
|
||||
|
||||
|
||||
class TestFlacfetchClientGetDownloadStatus:
|
||||
"""Test get_download_status() method."""
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_get_download_status_success(self):
|
||||
"""Test getting download status."""
|
||||
client = FlacfetchClient(
|
||||
base_url="http://localhost:8080",
|
||||
api_key="test-key",
|
||||
)
|
||||
|
||||
mock_response = Mock()
|
||||
mock_response.json.return_value = {
|
||||
"download_id": "dl_xyz789",
|
||||
"status": "downloading",
|
||||
"progress": 45.5,
|
||||
"download_speed_kbps": 1250.0,
|
||||
"peers": 5,
|
||||
}
|
||||
mock_response.raise_for_status = Mock()
|
||||
|
||||
with patch("httpx.AsyncClient") as mock_client_class:
|
||||
mock_client = AsyncMock()
|
||||
mock_client.get = AsyncMock(return_value=mock_response)
|
||||
mock_client.__aenter__ = AsyncMock(return_value=mock_client)
|
||||
mock_client.__aexit__ = AsyncMock(return_value=None)
|
||||
mock_client_class.return_value = mock_client
|
||||
|
||||
status = await client.get_download_status("dl_xyz789")
|
||||
|
||||
assert status["status"] == "downloading"
|
||||
assert status["progress"] == 45.5
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_get_download_status_not_found(self):
|
||||
"""Test getting status for non-existent download."""
|
||||
client = FlacfetchClient(
|
||||
base_url="http://localhost:8080",
|
||||
api_key="test-key",
|
||||
)
|
||||
|
||||
mock_response = Mock()
|
||||
mock_response.status_code = 404
|
||||
mock_response.raise_for_status = Mock(
|
||||
side_effect=httpx.HTTPStatusError("Not found", request=Mock(), response=mock_response)
|
||||
)
|
||||
|
||||
with patch("httpx.AsyncClient") as mock_client_class:
|
||||
mock_client = AsyncMock()
|
||||
mock_client.get = AsyncMock(return_value=mock_response)
|
||||
mock_client.__aenter__ = AsyncMock(return_value=mock_client)
|
||||
mock_client.__aexit__ = AsyncMock(return_value=None)
|
||||
mock_client_class.return_value = mock_client
|
||||
|
||||
with pytest.raises(FlacfetchServiceError) as exc_info:
|
||||
await client.get_download_status("nonexistent")
|
||||
|
||||
assert "not found" in str(exc_info.value).lower()
|
||||
|
||||
|
||||
class TestFlacfetchClientWaitForDownload:
|
||||
"""Test wait_for_download() method."""
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_wait_for_download_completes(self):
|
||||
"""Test waiting for download to complete."""
|
||||
client = FlacfetchClient(
|
||||
base_url="http://localhost:8080",
|
||||
api_key="test-key",
|
||||
)
|
||||
|
||||
# Mock get_download_status to return complete status
|
||||
async def mock_get_status(download_id):
|
||||
return {
|
||||
"download_id": download_id,
|
||||
"status": "complete",
|
||||
"gcs_path": "gs://bucket/uploads/test.flac",
|
||||
}
|
||||
|
||||
client.get_download_status = mock_get_status
|
||||
|
||||
result = await client.wait_for_download("dl_xyz789", timeout=10)
|
||||
|
||||
assert result["status"] == "complete"
|
||||
assert result["gcs_path"] == "gs://bucket/uploads/test.flac"
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_wait_for_download_seeding_status_completes(self):
|
||||
"""Test that 'seeding' status is treated as complete."""
|
||||
client = FlacfetchClient(
|
||||
base_url="http://localhost:8080",
|
||||
api_key="test-key",
|
||||
)
|
||||
|
||||
async def mock_get_status(download_id):
|
||||
return {
|
||||
"download_id": download_id,
|
||||
"status": "seeding",
|
||||
"output_path": "/downloads/test.flac",
|
||||
}
|
||||
|
||||
client.get_download_status = mock_get_status
|
||||
|
||||
result = await client.wait_for_download("dl_xyz789", timeout=10)
|
||||
|
||||
assert result["status"] == "seeding"
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_wait_for_download_failed(self):
|
||||
"""Test wait raises error when download fails."""
|
||||
client = FlacfetchClient(
|
||||
base_url="http://localhost:8080",
|
||||
api_key="test-key",
|
||||
)
|
||||
|
||||
async def mock_get_status(download_id):
|
||||
return {
|
||||
"download_id": download_id,
|
||||
"status": "failed",
|
||||
"error": "Torrent download failed: no peers",
|
||||
}
|
||||
|
||||
client.get_download_status = mock_get_status
|
||||
|
||||
with pytest.raises(FlacfetchServiceError) as exc_info:
|
||||
await client.wait_for_download("dl_xyz789", timeout=10)
|
||||
|
||||
assert "failed" in str(exc_info.value).lower()
|
||||
assert "no peers" in str(exc_info.value)
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_wait_for_download_cancelled(self):
|
||||
"""Test wait raises error when download is cancelled."""
|
||||
client = FlacfetchClient(
|
||||
base_url="http://localhost:8080",
|
||||
api_key="test-key",
|
||||
)
|
||||
|
||||
async def mock_get_status(download_id):
|
||||
return {
|
||||
"download_id": download_id,
|
||||
"status": "cancelled",
|
||||
}
|
||||
|
||||
client.get_download_status = mock_get_status
|
||||
|
||||
with pytest.raises(FlacfetchServiceError) as exc_info:
|
||||
await client.wait_for_download("dl_xyz789", timeout=10)
|
||||
|
||||
assert "cancelled" in str(exc_info.value).lower()
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_wait_for_download_timeout(self):
|
||||
"""Test wait raises error on timeout."""
|
||||
client = FlacfetchClient(
|
||||
base_url="http://localhost:8080",
|
||||
api_key="test-key",
|
||||
)
|
||||
|
||||
call_count = 0
|
||||
async def mock_get_status(download_id):
|
||||
nonlocal call_count
|
||||
call_count += 1
|
||||
return {
|
||||
"download_id": download_id,
|
||||
"status": "downloading",
|
||||
"progress": 10.0 * call_count,
|
||||
}
|
||||
|
||||
client.get_download_status = mock_get_status
|
||||
|
||||
with pytest.raises(FlacfetchServiceError) as exc_info:
|
||||
# Very short timeout with short poll interval
|
||||
await client.wait_for_download("dl_xyz789", timeout=0.2, poll_interval=0.1)
|
||||
|
||||
assert "timed out" in str(exc_info.value).lower()
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_wait_for_download_calls_progress_callback(self):
|
||||
"""Test progress callback is called during wait."""
|
||||
client = FlacfetchClient(
|
||||
base_url="http://localhost:8080",
|
||||
api_key="test-key",
|
||||
)
|
||||
|
||||
call_count = 0
|
||||
async def mock_get_status(download_id):
|
||||
nonlocal call_count
|
||||
call_count += 1
|
||||
if call_count >= 3:
|
||||
return {"status": "complete", "gcs_path": "gs://test"}
|
||||
return {"status": "downloading", "progress": 33.0 * call_count}
|
||||
|
||||
client.get_download_status = mock_get_status
|
||||
|
||||
progress_updates = []
|
||||
def progress_callback(status):
|
||||
progress_updates.append(status)
|
||||
|
||||
await client.wait_for_download(
|
||||
"dl_xyz789",
|
||||
timeout=10,
|
||||
poll_interval=0.01,
|
||||
progress_callback=progress_callback,
|
||||
)
|
||||
|
||||
# Should have received progress updates
|
||||
assert len(progress_updates) >= 2
|
||||
|
||||
|
||||
class TestGetFlacfetchClient:
|
||||
"""Test get_flacfetch_client() singleton factory."""
|
||||
|
||||
def test_returns_none_when_not_configured(self):
|
||||
"""Test returns None when FLACFETCH_API_URL not set."""
|
||||
reset_flacfetch_client()
|
||||
|
||||
with patch("backend.config.get_settings") as mock_settings:
|
||||
mock_settings.return_value = Mock(
|
||||
flacfetch_api_url=None,
|
||||
flacfetch_api_key=None,
|
||||
)
|
||||
|
||||
client = get_flacfetch_client()
|
||||
|
||||
assert client is None
|
||||
|
||||
def test_returns_none_when_key_missing(self):
|
||||
"""Test returns None when API key not set."""
|
||||
reset_flacfetch_client()
|
||||
|
||||
with patch("backend.config.get_settings") as mock_settings:
|
||||
mock_settings.return_value = Mock(
|
||||
flacfetch_api_url="http://localhost:8080",
|
||||
flacfetch_api_key=None,
|
||||
)
|
||||
|
||||
client = get_flacfetch_client()
|
||||
|
||||
assert client is None
|
||||
|
||||
def test_returns_client_when_configured(self):
|
||||
"""Test returns FlacfetchClient when both URL and key are set."""
|
||||
reset_flacfetch_client()
|
||||
|
||||
with patch("backend.config.get_settings") as mock_settings:
|
||||
mock_settings.return_value = Mock(
|
||||
flacfetch_api_url="http://10.0.0.5:8080",
|
||||
flacfetch_api_key="secret-key",
|
||||
)
|
||||
|
||||
client = get_flacfetch_client()
|
||||
|
||||
assert client is not None
|
||||
assert isinstance(client, FlacfetchClient)
|
||||
assert client.base_url == "http://10.0.0.5:8080"
|
||||
assert client.api_key == "secret-key"
|
||||
|
||||
def test_returns_same_instance(self):
|
||||
"""Test singleton returns same instance."""
|
||||
reset_flacfetch_client()
|
||||
|
||||
with patch("backend.config.get_settings") as mock_settings:
|
||||
mock_settings.return_value = Mock(
|
||||
flacfetch_api_url="http://localhost:8080",
|
||||
flacfetch_api_key="test-key",
|
||||
)
|
||||
|
||||
client1 = get_flacfetch_client()
|
||||
client2 = get_flacfetch_client()
|
||||
|
||||
assert client1 is client2
|
||||
|
||||
def test_reset_clears_singleton(self):
|
||||
"""Test reset_flacfetch_client() clears the singleton."""
|
||||
reset_flacfetch_client()
|
||||
|
||||
with patch("backend.config.get_settings") as mock_settings:
|
||||
mock_settings.return_value = Mock(
|
||||
flacfetch_api_url="http://localhost:8080",
|
||||
flacfetch_api_key="test-key",
|
||||
)
|
||||
|
||||
client1 = get_flacfetch_client()
|
||||
reset_flacfetch_client()
|
||||
|
||||
# Change the settings
|
||||
mock_settings.return_value = Mock(
|
||||
flacfetch_api_url="http://other:8080",
|
||||
flacfetch_api_key="other-key",
|
||||
)
|
||||
|
||||
client2 = get_flacfetch_client()
|
||||
|
||||
assert client1 is not client2
|
||||
assert client2.base_url == "http://other:8080"
|
||||
|
||||
@@ -1,524 +0,0 @@
|
||||
"""
|
||||
Tests for gdrive_service.py - Google Drive file operations.
|
||||
|
||||
These tests mock the Google API client and Secret Manager to verify:
|
||||
- Credential loading from Secret Manager
|
||||
- Folder creation and lookup
|
||||
- File uploads with proper MIME types
|
||||
- Public share folder structure uploads
|
||||
"""
|
||||
import json
|
||||
import os
|
||||
import pytest
|
||||
from unittest.mock import Mock, MagicMock, patch
|
||||
|
||||
|
||||
class TestGoogleDriveServiceInit:
|
||||
"""Test GoogleDriveService initialization."""
|
||||
|
||||
@patch("backend.services.gdrive_service.get_settings")
|
||||
def test_init_creates_service(self, mock_get_settings):
|
||||
"""Test initialization creates service with settings."""
|
||||
from backend.services.gdrive_service import GoogleDriveService
|
||||
|
||||
mock_settings = Mock()
|
||||
mock_get_settings.return_value = mock_settings
|
||||
|
||||
service = GoogleDriveService()
|
||||
|
||||
assert service.settings == mock_settings
|
||||
assert service._service is None
|
||||
assert service._credentials_data is None
|
||||
assert service._loaded is False
|
||||
|
||||
|
||||
class TestLoadCredentials:
|
||||
"""Test _load_credentials method."""
|
||||
|
||||
@patch("backend.services.gdrive_service.get_settings")
|
||||
def test_load_credentials_success(self, mock_get_settings):
|
||||
"""Test successful credential loading from Secret Manager."""
|
||||
from backend.services.gdrive_service import GoogleDriveService
|
||||
|
||||
mock_settings = Mock()
|
||||
mock_settings.get_secret.return_value = json.dumps({
|
||||
"token": "access-token",
|
||||
"refresh_token": "refresh-token",
|
||||
"client_id": "client-id",
|
||||
"client_secret": "client-secret",
|
||||
})
|
||||
mock_get_settings.return_value = mock_settings
|
||||
|
||||
service = GoogleDriveService()
|
||||
creds = service._load_credentials()
|
||||
|
||||
assert creds is not None
|
||||
assert creds["refresh_token"] == "refresh-token"
|
||||
mock_settings.get_secret.assert_called_once_with("gdrive-oauth-credentials")
|
||||
|
||||
@patch("backend.services.gdrive_service.get_settings")
|
||||
def test_load_credentials_fallback_to_youtube(self, mock_get_settings):
|
||||
"""Test fallback to YouTube credentials when Drive creds not found."""
|
||||
from backend.services.gdrive_service import GoogleDriveService
|
||||
|
||||
mock_settings = Mock()
|
||||
# First call returns None (Drive creds not found)
|
||||
# Second call returns YouTube creds
|
||||
mock_settings.get_secret.side_effect = [
|
||||
None,
|
||||
json.dumps({
|
||||
"refresh_token": "youtube-token",
|
||||
"client_id": "youtube-id",
|
||||
"client_secret": "youtube-secret",
|
||||
})
|
||||
]
|
||||
mock_get_settings.return_value = mock_settings
|
||||
|
||||
service = GoogleDriveService()
|
||||
creds = service._load_credentials()
|
||||
|
||||
assert creds is not None
|
||||
assert creds["refresh_token"] == "youtube-token"
|
||||
assert mock_settings.get_secret.call_count == 2
|
||||
|
||||
@patch("backend.services.gdrive_service.get_settings")
|
||||
def test_load_credentials_not_found(self, mock_get_settings):
|
||||
"""Test handling when no credentials found."""
|
||||
from backend.services.gdrive_service import GoogleDriveService
|
||||
|
||||
mock_settings = Mock()
|
||||
mock_settings.get_secret.return_value = None
|
||||
mock_get_settings.return_value = mock_settings
|
||||
|
||||
service = GoogleDriveService()
|
||||
creds = service._load_credentials()
|
||||
|
||||
assert creds is None
|
||||
assert service._loaded is True
|
||||
|
||||
@patch("backend.services.gdrive_service.get_settings")
|
||||
def test_load_credentials_missing_required_fields(self, mock_get_settings):
|
||||
"""Test handling when credentials missing required fields."""
|
||||
from backend.services.gdrive_service import GoogleDriveService
|
||||
|
||||
mock_settings = Mock()
|
||||
mock_settings.get_secret.return_value = json.dumps({
|
||||
"token": "access-token",
|
||||
# Missing: refresh_token, client_id, client_secret
|
||||
})
|
||||
mock_get_settings.return_value = mock_settings
|
||||
|
||||
service = GoogleDriveService()
|
||||
creds = service._load_credentials()
|
||||
|
||||
assert creds is None
|
||||
|
||||
@patch("backend.services.gdrive_service.get_settings")
|
||||
def test_load_credentials_cached(self, mock_get_settings):
|
||||
"""Test credentials are cached after first load."""
|
||||
from backend.services.gdrive_service import GoogleDriveService
|
||||
|
||||
mock_settings = Mock()
|
||||
mock_settings.get_secret.return_value = json.dumps({
|
||||
"refresh_token": "token",
|
||||
"client_id": "id",
|
||||
"client_secret": "secret",
|
||||
})
|
||||
mock_get_settings.return_value = mock_settings
|
||||
|
||||
service = GoogleDriveService()
|
||||
|
||||
creds1 = service._load_credentials()
|
||||
creds2 = service._load_credentials()
|
||||
|
||||
assert creds1 == creds2
|
||||
assert mock_settings.get_secret.call_count == 1
|
||||
|
||||
|
||||
class TestIsConfigured:
|
||||
"""Test is_configured property."""
|
||||
|
||||
@patch("backend.services.gdrive_service.get_settings")
|
||||
def test_is_configured_true(self, mock_get_settings):
|
||||
"""Test is_configured returns True when credentials exist."""
|
||||
from backend.services.gdrive_service import GoogleDriveService
|
||||
|
||||
mock_settings = Mock()
|
||||
mock_settings.get_secret.return_value = json.dumps({
|
||||
"refresh_token": "token",
|
||||
"client_id": "id",
|
||||
"client_secret": "secret",
|
||||
})
|
||||
mock_get_settings.return_value = mock_settings
|
||||
|
||||
service = GoogleDriveService()
|
||||
|
||||
assert service.is_configured is True
|
||||
|
||||
@patch("backend.services.gdrive_service.get_settings")
|
||||
def test_is_configured_false(self, mock_get_settings):
|
||||
"""Test is_configured returns False when no credentials."""
|
||||
from backend.services.gdrive_service import GoogleDriveService
|
||||
|
||||
mock_settings = Mock()
|
||||
mock_settings.get_secret.return_value = None
|
||||
mock_get_settings.return_value = mock_settings
|
||||
|
||||
service = GoogleDriveService()
|
||||
|
||||
assert service.is_configured is False
|
||||
|
||||
|
||||
class TestDriveService:
|
||||
"""Test service property."""
|
||||
|
||||
@patch("googleapiclient.discovery.build")
|
||||
@patch("google.oauth2.credentials.Credentials")
|
||||
@patch("backend.services.gdrive_service.get_settings")
|
||||
def test_service_creates_drive_client(
|
||||
self, mock_get_settings, mock_creds_class, mock_build
|
||||
):
|
||||
"""Test service property creates Google Drive API client."""
|
||||
from backend.services.gdrive_service import GoogleDriveService
|
||||
|
||||
mock_settings = Mock()
|
||||
mock_settings.get_secret.return_value = json.dumps({
|
||||
"token": "access-token",
|
||||
"refresh_token": "refresh-token",
|
||||
"client_id": "client-id",
|
||||
"client_secret": "client-secret",
|
||||
})
|
||||
mock_get_settings.return_value = mock_settings
|
||||
|
||||
mock_creds = Mock()
|
||||
mock_creds_class.return_value = mock_creds
|
||||
|
||||
mock_drive_service = Mock()
|
||||
mock_build.return_value = mock_drive_service
|
||||
|
||||
service = GoogleDriveService()
|
||||
result = service.service
|
||||
|
||||
mock_build.assert_called_once_with("drive", "v3", credentials=mock_creds)
|
||||
assert result == mock_drive_service
|
||||
|
||||
@patch("backend.services.gdrive_service.get_settings")
|
||||
def test_service_raises_on_missing_credentials(self, mock_get_settings):
|
||||
"""Test service raises RuntimeError when credentials missing."""
|
||||
from backend.services.gdrive_service import GoogleDriveService
|
||||
|
||||
mock_settings = Mock()
|
||||
mock_settings.get_secret.return_value = None
|
||||
mock_get_settings.return_value = mock_settings
|
||||
|
||||
service = GoogleDriveService()
|
||||
|
||||
with pytest.raises(RuntimeError) as exc_info:
|
||||
_ = service.service
|
||||
|
||||
assert "not configured" in str(exc_info.value)
|
||||
|
||||
|
||||
class TestGetOrCreateFolder:
|
||||
"""Test get_or_create_folder method."""
|
||||
|
||||
@patch("backend.services.gdrive_service.get_settings")
|
||||
def test_get_existing_folder(self, mock_get_settings):
|
||||
"""Test finding an existing folder."""
|
||||
from backend.services.gdrive_service import GoogleDriveService
|
||||
|
||||
mock_settings = Mock()
|
||||
mock_settings.get_secret.return_value = json.dumps({
|
||||
"refresh_token": "token",
|
||||
"client_id": "id",
|
||||
"client_secret": "secret",
|
||||
})
|
||||
mock_get_settings.return_value = mock_settings
|
||||
|
||||
service = GoogleDriveService()
|
||||
|
||||
# Mock the Drive API
|
||||
mock_files_api = Mock()
|
||||
mock_list_result = Mock()
|
||||
mock_list_result.execute.return_value = {
|
||||
"files": [{"id": "folder-id-123", "name": "MP4"}]
|
||||
}
|
||||
mock_files_api.list.return_value = mock_list_result
|
||||
|
||||
mock_drive = Mock()
|
||||
mock_drive.files.return_value = mock_files_api
|
||||
service._service = mock_drive
|
||||
service._loaded = True
|
||||
service._credentials_data = {"refresh_token": "token", "client_id": "id", "client_secret": "secret"}
|
||||
|
||||
folder_id = service.get_or_create_folder("parent-123", "MP4")
|
||||
|
||||
assert folder_id == "folder-id-123"
|
||||
|
||||
@patch("backend.services.gdrive_service.get_settings")
|
||||
def test_create_new_folder(self, mock_get_settings):
|
||||
"""Test creating a new folder when it doesn't exist."""
|
||||
from backend.services.gdrive_service import GoogleDriveService
|
||||
|
||||
mock_settings = Mock()
|
||||
mock_settings.get_secret.return_value = json.dumps({
|
||||
"refresh_token": "token",
|
||||
"client_id": "id",
|
||||
"client_secret": "secret",
|
||||
})
|
||||
mock_get_settings.return_value = mock_settings
|
||||
|
||||
service = GoogleDriveService()
|
||||
|
||||
# Mock the Drive API
|
||||
mock_files_api = Mock()
|
||||
|
||||
# list returns empty (folder doesn't exist)
|
||||
mock_list_result = Mock()
|
||||
mock_list_result.execute.return_value = {"files": []}
|
||||
mock_files_api.list.return_value = mock_list_result
|
||||
|
||||
# create returns new folder
|
||||
mock_create_result = Mock()
|
||||
mock_create_result.execute.return_value = {"id": "new-folder-id"}
|
||||
mock_files_api.create.return_value = mock_create_result
|
||||
|
||||
mock_drive = Mock()
|
||||
mock_drive.files.return_value = mock_files_api
|
||||
service._service = mock_drive
|
||||
service._loaded = True
|
||||
service._credentials_data = {"refresh_token": "token", "client_id": "id", "client_secret": "secret"}
|
||||
|
||||
folder_id = service.get_or_create_folder("parent-123", "NewFolder")
|
||||
|
||||
assert folder_id == "new-folder-id"
|
||||
mock_files_api.create.assert_called_once()
|
||||
|
||||
|
||||
class TestUploadFile:
|
||||
"""Test upload_file method."""
|
||||
|
||||
@patch("googleapiclient.http.MediaFileUpload")
|
||||
@patch("backend.services.gdrive_service.get_settings")
|
||||
def test_upload_file_success(self, mock_get_settings, mock_media, tmp_path):
|
||||
"""Test uploading a file to Drive."""
|
||||
from backend.services.gdrive_service import GoogleDriveService
|
||||
|
||||
mock_settings = Mock()
|
||||
mock_settings.get_secret.return_value = json.dumps({
|
||||
"refresh_token": "token",
|
||||
"client_id": "id",
|
||||
"client_secret": "secret",
|
||||
})
|
||||
mock_get_settings.return_value = mock_settings
|
||||
|
||||
# Create test file
|
||||
test_file = tmp_path / "test.mp4"
|
||||
test_file.write_bytes(b"video content")
|
||||
|
||||
service = GoogleDriveService()
|
||||
|
||||
mock_files_api = Mock()
|
||||
|
||||
# list returns empty (no existing file)
|
||||
mock_list_result = Mock()
|
||||
mock_list_result.execute.return_value = {"files": []}
|
||||
mock_files_api.list.return_value = mock_list_result
|
||||
|
||||
# create returns new file
|
||||
mock_create_result = Mock()
|
||||
mock_create_result.execute.return_value = {"id": "file-id-123"}
|
||||
mock_files_api.create.return_value = mock_create_result
|
||||
|
||||
mock_drive = Mock()
|
||||
mock_drive.files.return_value = mock_files_api
|
||||
service._service = mock_drive
|
||||
service._loaded = True
|
||||
service._credentials_data = {"refresh_token": "token", "client_id": "id", "client_secret": "secret"}
|
||||
|
||||
file_id = service.upload_file(str(test_file), "parent-123", "video.mp4")
|
||||
|
||||
assert file_id == "file-id-123"
|
||||
mock_media.assert_called_once()
|
||||
# Verify MIME type was set correctly
|
||||
call_kwargs = mock_media.call_args.kwargs
|
||||
assert call_kwargs["mimetype"] == "video/mp4"
|
||||
|
||||
@patch("googleapiclient.http.MediaFileUpload")
|
||||
@patch("backend.services.gdrive_service.get_settings")
|
||||
def test_upload_file_replaces_existing(self, mock_get_settings, mock_media, tmp_path):
|
||||
"""Test upload deletes existing file before upload."""
|
||||
from backend.services.gdrive_service import GoogleDriveService
|
||||
|
||||
mock_settings = Mock()
|
||||
mock_settings.get_secret.return_value = json.dumps({
|
||||
"refresh_token": "token",
|
||||
"client_id": "id",
|
||||
"client_secret": "secret",
|
||||
})
|
||||
mock_get_settings.return_value = mock_settings
|
||||
|
||||
test_file = tmp_path / "test.mp4"
|
||||
test_file.write_bytes(b"content")
|
||||
|
||||
service = GoogleDriveService()
|
||||
|
||||
mock_files_api = Mock()
|
||||
|
||||
# list returns existing file
|
||||
mock_list_result = Mock()
|
||||
mock_list_result.execute.return_value = {
|
||||
"files": [{"id": "existing-file-id"}]
|
||||
}
|
||||
mock_files_api.list.return_value = mock_list_result
|
||||
|
||||
mock_delete_result = Mock()
|
||||
mock_delete_result.execute.return_value = {}
|
||||
mock_files_api.delete.return_value = mock_delete_result
|
||||
|
||||
mock_create_result = Mock()
|
||||
mock_create_result.execute.return_value = {"id": "new-file-id"}
|
||||
mock_files_api.create.return_value = mock_create_result
|
||||
|
||||
mock_drive = Mock()
|
||||
mock_drive.files.return_value = mock_files_api
|
||||
service._service = mock_drive
|
||||
service._loaded = True
|
||||
service._credentials_data = {"refresh_token": "token", "client_id": "id", "client_secret": "secret"}
|
||||
|
||||
file_id = service.upload_file(str(test_file), "parent-123", "video.mp4")
|
||||
|
||||
# Should have deleted existing file
|
||||
mock_files_api.delete.assert_called_once_with(fileId="existing-file-id")
|
||||
assert file_id == "new-file-id"
|
||||
|
||||
|
||||
class TestUploadToPublicShare:
|
||||
"""Test upload_to_public_share method."""
|
||||
|
||||
@patch("backend.services.gdrive_service.get_settings")
|
||||
def test_upload_to_public_share(self, mock_get_settings, tmp_path):
|
||||
"""Test uploading files to public share folder structure."""
|
||||
from backend.services.gdrive_service import GoogleDriveService
|
||||
|
||||
mock_settings = Mock()
|
||||
mock_settings.get_secret.return_value = json.dumps({
|
||||
"refresh_token": "token",
|
||||
"client_id": "id",
|
||||
"client_secret": "secret",
|
||||
})
|
||||
mock_get_settings.return_value = mock_settings
|
||||
|
||||
# Create test files
|
||||
mp4_file = tmp_path / "output.mp4"
|
||||
mp4_file.write_bytes(b"4k video")
|
||||
mp4_720_file = tmp_path / "output_720p.mp4"
|
||||
mp4_720_file.write_bytes(b"720p video")
|
||||
cdg_file = tmp_path / "output.zip"
|
||||
cdg_file.write_bytes(b"cdg package")
|
||||
|
||||
service = GoogleDriveService()
|
||||
|
||||
# Mock methods
|
||||
with patch.object(service, "get_or_create_folder") as mock_get_folder:
|
||||
with patch.object(service, "upload_file") as mock_upload:
|
||||
mock_get_folder.side_effect = [
|
||||
"mp4-folder-id",
|
||||
"mp4-720-folder-id",
|
||||
"cdg-folder-id",
|
||||
]
|
||||
mock_upload.side_effect = [
|
||||
"mp4-file-id",
|
||||
"720p-file-id",
|
||||
"cdg-file-id",
|
||||
]
|
||||
|
||||
output_files = {
|
||||
"final_karaoke_lossy_mp4": str(mp4_file),
|
||||
"final_karaoke_lossy_720p_mp4": str(mp4_720_file),
|
||||
"final_karaoke_cdg_zip": str(cdg_file),
|
||||
}
|
||||
|
||||
result = service.upload_to_public_share(
|
||||
root_folder_id="root-123",
|
||||
brand_code="NOMAD-1163",
|
||||
base_name="Artist - Title",
|
||||
output_files=output_files,
|
||||
)
|
||||
|
||||
# Should have created/found 3 folders
|
||||
assert mock_get_folder.call_count == 3
|
||||
|
||||
# Should have uploaded 3 files
|
||||
assert mock_upload.call_count == 3
|
||||
|
||||
# Check result
|
||||
assert result["mp4"] == "mp4-file-id"
|
||||
assert result["mp4_720p"] == "720p-file-id"
|
||||
assert result["cdg"] == "cdg-file-id"
|
||||
|
||||
@patch("backend.services.gdrive_service.get_settings")
|
||||
def test_upload_to_public_share_skips_missing_files(
|
||||
self, mock_get_settings, tmp_path
|
||||
):
|
||||
"""Test upload skips files that don't exist."""
|
||||
from backend.services.gdrive_service import GoogleDriveService
|
||||
|
||||
mock_settings = Mock()
|
||||
mock_settings.get_secret.return_value = json.dumps({
|
||||
"refresh_token": "token",
|
||||
"client_id": "id",
|
||||
"client_secret": "secret",
|
||||
})
|
||||
mock_get_settings.return_value = mock_settings
|
||||
|
||||
# Only create one file
|
||||
mp4_file = tmp_path / "output.mp4"
|
||||
mp4_file.write_bytes(b"video")
|
||||
|
||||
service = GoogleDriveService()
|
||||
|
||||
with patch.object(service, "get_or_create_folder") as mock_get_folder:
|
||||
with patch.object(service, "upload_file") as mock_upload:
|
||||
mock_get_folder.return_value = "folder-id"
|
||||
mock_upload.return_value = "file-id"
|
||||
|
||||
output_files = {
|
||||
"final_karaoke_lossy_mp4": str(mp4_file),
|
||||
"final_karaoke_lossy_720p_mp4": "/nonexistent/file.mp4",
|
||||
"final_karaoke_cdg_zip": None,
|
||||
}
|
||||
|
||||
result = service.upload_to_public_share(
|
||||
root_folder_id="root-123",
|
||||
brand_code="CODE",
|
||||
base_name="Name",
|
||||
output_files=output_files,
|
||||
)
|
||||
|
||||
# Should only upload the one file that exists
|
||||
assert mock_upload.call_count == 1
|
||||
assert len(result) == 1
|
||||
|
||||
|
||||
class TestGetGdriveService:
|
||||
"""Test get_gdrive_service singleton."""
|
||||
|
||||
@patch("backend.services.gdrive_service.get_settings")
|
||||
def test_get_gdrive_service_singleton(self, mock_get_settings):
|
||||
"""Test get_gdrive_service returns singleton instance."""
|
||||
from backend.services.gdrive_service import get_gdrive_service
|
||||
import backend.services.gdrive_service as gdrive_module
|
||||
|
||||
# Reset singleton
|
||||
gdrive_module._gdrive_service = None
|
||||
|
||||
mock_settings = Mock()
|
||||
mock_settings.get_secret.return_value = None
|
||||
mock_get_settings.return_value = mock_settings
|
||||
|
||||
service1 = get_gdrive_service()
|
||||
service2 = get_gdrive_service()
|
||||
|
||||
assert service1 is service2
|
||||
|
||||
@@ -1,431 +0,0 @@
|
||||
"""
|
||||
API endpoint tests for instrumental review functionality.
|
||||
|
||||
Tests for:
|
||||
- GET /api/jobs/{job_id}/instrumental-analysis
|
||||
- GET /api/jobs/{job_id}/audio-stream/{stem_type}
|
||||
- POST /api/jobs/{job_id}/create-custom-instrumental
|
||||
- GET /api/jobs/{job_id}/waveform-data
|
||||
"""
|
||||
|
||||
import pytest
|
||||
from datetime import datetime
|
||||
from unittest.mock import MagicMock, patch, AsyncMock
|
||||
|
||||
from backend.models.job import Job, JobStatus
|
||||
|
||||
|
||||
@pytest.fixture
|
||||
def mock_job_manager():
|
||||
"""Create a mock job manager."""
|
||||
with patch('backend.api.routes.jobs.job_manager') as mock:
|
||||
yield mock
|
||||
|
||||
|
||||
@pytest.fixture
|
||||
def mock_storage_service():
|
||||
"""Create a mock storage service."""
|
||||
with patch('backend.api.routes.jobs.StorageService') as mock:
|
||||
mock_instance = MagicMock()
|
||||
mock_instance.generate_signed_url.return_value = "https://storage.googleapis.com/signed-url"
|
||||
mock.return_value = mock_instance
|
||||
yield mock_instance
|
||||
|
||||
|
||||
@pytest.fixture
|
||||
def sample_job():
|
||||
"""Create a sample job for testing."""
|
||||
return Job(
|
||||
job_id="test-job-123",
|
||||
artist="Test Artist",
|
||||
title="Test Song",
|
||||
status=JobStatus.AWAITING_INSTRUMENTAL_SELECTION,
|
||||
created_at=datetime.now(),
|
||||
updated_at=datetime.now(),
|
||||
file_urls={
|
||||
"stems": {
|
||||
"instrumental_clean": "jobs/test/stems/instrumental_clean.flac",
|
||||
"backing_vocals": "jobs/test/stems/backing_vocals.flac",
|
||||
"instrumental_with_backing": "jobs/test/stems/instrumental_with_backing.flac",
|
||||
},
|
||||
"analysis": {
|
||||
"backing_vocals_waveform": "jobs/test/analysis/waveform.png",
|
||||
},
|
||||
},
|
||||
state_data={
|
||||
"backing_vocals_analysis": {
|
||||
"has_audible_content": True,
|
||||
"total_duration_seconds": 180.0,
|
||||
"audible_segments": [
|
||||
{
|
||||
"start_seconds": 10.0,
|
||||
"end_seconds": 20.0,
|
||||
"duration_seconds": 10.0,
|
||||
"avg_amplitude_db": -25.0,
|
||||
},
|
||||
{
|
||||
"start_seconds": 60.0,
|
||||
"end_seconds": 80.0,
|
||||
"duration_seconds": 20.0,
|
||||
"avg_amplitude_db": -30.0,
|
||||
},
|
||||
],
|
||||
"recommended_selection": "review_needed",
|
||||
"total_audible_duration_seconds": 30.0,
|
||||
"audible_percentage": 16.67,
|
||||
"silence_threshold_db": -40.0,
|
||||
}
|
||||
},
|
||||
)
|
||||
|
||||
|
||||
@pytest.fixture
|
||||
def silent_job():
|
||||
"""Create a job with silent backing vocals."""
|
||||
return Job(
|
||||
job_id="test-job-silent",
|
||||
artist="Test Artist",
|
||||
title="Test Song",
|
||||
status=JobStatus.AWAITING_INSTRUMENTAL_SELECTION,
|
||||
created_at=datetime.now(),
|
||||
updated_at=datetime.now(),
|
||||
file_urls={
|
||||
"stems": {
|
||||
"instrumental_clean": "jobs/test/stems/instrumental_clean.flac",
|
||||
"backing_vocals": "jobs/test/stems/backing_vocals.flac",
|
||||
"instrumental_with_backing": "jobs/test/stems/instrumental_with_backing.flac",
|
||||
},
|
||||
"analysis": {
|
||||
"backing_vocals_waveform": "jobs/test/analysis/waveform.png",
|
||||
},
|
||||
},
|
||||
state_data={
|
||||
"backing_vocals_analysis": {
|
||||
"has_audible_content": False,
|
||||
"total_duration_seconds": 180.0,
|
||||
"audible_segments": [],
|
||||
"recommended_selection": "clean",
|
||||
"total_audible_duration_seconds": 0.0,
|
||||
"audible_percentage": 0.0,
|
||||
"silence_threshold_db": -40.0,
|
||||
}
|
||||
},
|
||||
)
|
||||
|
||||
|
||||
class TestGetInstrumentalAnalysis:
|
||||
"""Tests for GET /api/jobs/{job_id}/instrumental-analysis."""
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_get_analysis_returns_data(self, mock_job_manager, mock_storage_service, sample_job):
|
||||
"""GET /instrumental-analysis should return analysis data."""
|
||||
mock_job_manager.get_job.return_value = sample_job
|
||||
|
||||
from backend.api.routes.jobs import get_instrumental_analysis
|
||||
|
||||
result = await get_instrumental_analysis("test-job-123")
|
||||
|
||||
assert result["job_id"] == "test-job-123"
|
||||
assert result["artist"] == "Test Artist"
|
||||
assert result["title"] == "Test Song"
|
||||
assert result["analysis"]["has_audible_content"] is True
|
||||
assert len(result["analysis"]["audible_segments"]) == 2
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_get_analysis_includes_audio_urls(self, mock_job_manager, mock_storage_service, sample_job):
|
||||
"""GET /instrumental-analysis should include audio URLs."""
|
||||
mock_job_manager.get_job.return_value = sample_job
|
||||
|
||||
from backend.api.routes.jobs import get_instrumental_analysis
|
||||
|
||||
result = await get_instrumental_analysis("test-job-123")
|
||||
|
||||
assert "audio_urls" in result
|
||||
assert "clean_instrumental" in result["audio_urls"]
|
||||
assert "backing_vocals" in result["audio_urls"]
|
||||
assert "with_backing" in result["audio_urls"]
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_get_analysis_includes_waveform_url(self, mock_job_manager, sample_job):
|
||||
"""GET /instrumental-analysis should include waveform URL."""
|
||||
mock_job_manager.get_job.return_value = sample_job
|
||||
|
||||
# Need to mock StorageService at module level for the signed URL
|
||||
with patch('backend.api.routes.jobs.StorageService') as mock_storage_cls:
|
||||
mock_storage = MagicMock()
|
||||
mock_storage.generate_signed_url.return_value = "https://storage.googleapis.com/signed-url"
|
||||
mock_storage_cls.return_value = mock_storage
|
||||
|
||||
from backend.api.routes.jobs import get_instrumental_analysis
|
||||
|
||||
result = await get_instrumental_analysis("test-job-123")
|
||||
|
||||
assert "waveform_url" in result
|
||||
# Waveform URL should be generated from the file_urls
|
||||
assert result["waveform_url"] == "https://storage.googleapis.com/signed-url"
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_get_analysis_silent_audio(self, mock_job_manager, mock_storage_service, silent_job):
|
||||
"""GET /instrumental-analysis should correctly report silent audio."""
|
||||
mock_job_manager.get_job.return_value = silent_job
|
||||
|
||||
from backend.api.routes.jobs import get_instrumental_analysis
|
||||
|
||||
result = await get_instrumental_analysis("test-job-silent")
|
||||
|
||||
assert result["analysis"]["has_audible_content"] is False
|
||||
assert result["analysis"]["recommended_selection"] == "clean"
|
||||
assert len(result["analysis"]["audible_segments"]) == 0
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_get_analysis_job_not_found(self, mock_job_manager):
|
||||
"""GET /instrumental-analysis should return 404 for non-existent job."""
|
||||
mock_job_manager.get_job.return_value = None
|
||||
|
||||
from backend.api.routes.jobs import get_instrumental_analysis
|
||||
from fastapi import HTTPException
|
||||
|
||||
with pytest.raises(HTTPException) as exc_info:
|
||||
await get_instrumental_analysis("non-existent")
|
||||
|
||||
assert exc_info.value.status_code == 404
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_get_analysis_wrong_status(self, mock_job_manager, sample_job):
|
||||
"""GET /instrumental-analysis should return 400 for wrong job status."""
|
||||
sample_job.status = JobStatus.PENDING
|
||||
mock_job_manager.get_job.return_value = sample_job
|
||||
|
||||
from backend.api.routes.jobs import get_instrumental_analysis
|
||||
from fastapi import HTTPException
|
||||
|
||||
with pytest.raises(HTTPException) as exc_info:
|
||||
await get_instrumental_analysis("test-job-123")
|
||||
|
||||
assert exc_info.value.status_code == 400
|
||||
|
||||
|
||||
class TestAudioStream:
|
||||
"""Tests for GET /api/jobs/{job_id}/audio-stream/{stem_type}."""
|
||||
|
||||
def test_stream_backing_vocals_valid(self, mock_job_manager, sample_job):
|
||||
"""Should return redirect for backing vocals stream."""
|
||||
mock_job_manager.get_job.return_value = sample_job
|
||||
|
||||
from backend.api.routes.jobs import stream_audio
|
||||
|
||||
# This will try to stream from GCS which we can't test easily
|
||||
# Instead, verify the function exists and has correct signature
|
||||
assert callable(stream_audio)
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_stream_invalid_stem_type(self, mock_job_manager, sample_job):
|
||||
"""Should return 400 for invalid stem type."""
|
||||
mock_job_manager.get_job.return_value = sample_job
|
||||
|
||||
from backend.api.routes.jobs import stream_audio
|
||||
from fastapi import HTTPException
|
||||
|
||||
with pytest.raises(HTTPException) as exc_info:
|
||||
await stream_audio("test-job-123", "invalid_stem")
|
||||
|
||||
assert exc_info.value.status_code == 400
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_stream_job_not_found(self, mock_job_manager):
|
||||
"""Should return 404 for non-existent job."""
|
||||
mock_job_manager.get_job.return_value = None
|
||||
|
||||
from backend.api.routes.jobs import stream_audio
|
||||
from fastapi import HTTPException
|
||||
|
||||
with pytest.raises(HTTPException) as exc_info:
|
||||
await stream_audio("non-existent", "backing_vocals")
|
||||
|
||||
assert exc_info.value.status_code == 404
|
||||
|
||||
|
||||
class TestCreateCustomInstrumental:
|
||||
"""Tests for POST /api/jobs/{job_id}/create-custom-instrumental."""
|
||||
|
||||
@pytest.fixture
|
||||
def mock_audio_editing_service(self):
|
||||
"""Mock the audio editing service."""
|
||||
with patch('backend.api.routes.jobs.AudioEditingService') as mock:
|
||||
mock_instance = MagicMock()
|
||||
mock_instance.create_custom_instrumental.return_value = MagicMock(
|
||||
output_path="jobs/test/stems/custom_instrumental.flac",
|
||||
mute_regions_applied=[
|
||||
MagicMock(start_seconds=10.0, end_seconds=20.0),
|
||||
],
|
||||
total_muted_duration_seconds=10.0,
|
||||
output_duration_seconds=180.0,
|
||||
)
|
||||
mock.return_value = mock_instance
|
||||
yield mock_instance
|
||||
|
||||
def test_create_custom_validates_mute_regions(self):
|
||||
"""Should validate mute regions at model level."""
|
||||
from backend.models.requests import CreateCustomInstrumentalRequest, MuteRegionRequest
|
||||
from pydantic import ValidationError
|
||||
|
||||
# Empty mute regions should fail at model validation
|
||||
with pytest.raises(ValidationError) as exc_info:
|
||||
CreateCustomInstrumentalRequest(mute_regions=[])
|
||||
|
||||
assert "At least one mute region is required" in str(exc_info.value)
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_create_custom_job_not_found(self, mock_job_manager):
|
||||
"""Should return 404 for non-existent job."""
|
||||
mock_job_manager.get_job.return_value = None
|
||||
|
||||
from backend.api.routes.jobs import create_custom_instrumental
|
||||
from backend.models.requests import CreateCustomInstrumentalRequest, MuteRegionRequest
|
||||
from fastapi import HTTPException
|
||||
|
||||
request = CreateCustomInstrumentalRequest(
|
||||
mute_regions=[MuteRegionRequest(start_seconds=10.0, end_seconds=20.0)]
|
||||
)
|
||||
|
||||
with pytest.raises(HTTPException) as exc_info:
|
||||
await create_custom_instrumental("non-existent", request)
|
||||
|
||||
assert exc_info.value.status_code == 404
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_create_custom_wrong_status(self, mock_job_manager, sample_job):
|
||||
"""Should return 400 for wrong job status."""
|
||||
sample_job.status = JobStatus.PENDING
|
||||
mock_job_manager.get_job.return_value = sample_job
|
||||
|
||||
from backend.api.routes.jobs import create_custom_instrumental
|
||||
from backend.models.requests import CreateCustomInstrumentalRequest, MuteRegionRequest
|
||||
from fastapi import HTTPException
|
||||
|
||||
request = CreateCustomInstrumentalRequest(
|
||||
mute_regions=[MuteRegionRequest(start_seconds=10.0, end_seconds=20.0)]
|
||||
)
|
||||
|
||||
with pytest.raises(HTTPException) as exc_info:
|
||||
await create_custom_instrumental("test-job-123", request)
|
||||
|
||||
assert exc_info.value.status_code == 400
|
||||
|
||||
|
||||
class TestGetWaveformData:
|
||||
"""Tests for GET /api/jobs/{job_id}/waveform-data."""
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_get_waveform_data_job_not_found(self, mock_job_manager):
|
||||
"""Should return 404 for non-existent job."""
|
||||
mock_job_manager.get_job.return_value = None
|
||||
|
||||
from backend.api.routes.jobs import get_waveform_data
|
||||
from fastapi import HTTPException
|
||||
|
||||
with pytest.raises(HTTPException) as exc_info:
|
||||
await get_waveform_data("non-existent")
|
||||
|
||||
assert exc_info.value.status_code == 404
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_get_waveform_data_wrong_status(self, mock_job_manager, sample_job):
|
||||
"""Should return 400 for wrong job status."""
|
||||
sample_job.status = JobStatus.PENDING
|
||||
mock_job_manager.get_job.return_value = sample_job
|
||||
|
||||
from backend.api.routes.jobs import get_waveform_data
|
||||
from fastapi import HTTPException
|
||||
|
||||
with pytest.raises(HTTPException) as exc_info:
|
||||
await get_waveform_data("test-job-123")
|
||||
|
||||
assert exc_info.value.status_code == 400
|
||||
|
||||
def test_waveform_data_endpoint_exists(self):
|
||||
"""Verify the endpoint function exists and is callable."""
|
||||
from backend.api.routes.jobs import get_waveform_data
|
||||
|
||||
assert callable(get_waveform_data)
|
||||
# Verify it's an async function
|
||||
import inspect
|
||||
assert inspect.iscoroutinefunction(get_waveform_data)
|
||||
|
||||
|
||||
class TestRequestModels:
|
||||
"""Tests for API request models."""
|
||||
|
||||
def test_mute_region_request_valid(self):
|
||||
"""MuteRegionRequest should accept valid values."""
|
||||
from backend.models.requests import MuteRegionRequest
|
||||
|
||||
region = MuteRegionRequest(start_seconds=10.0, end_seconds=20.0)
|
||||
|
||||
assert region.start_seconds == 10.0
|
||||
assert region.end_seconds == 20.0
|
||||
|
||||
def test_mute_region_request_invalid_start(self):
|
||||
"""MuteRegionRequest should reject negative start."""
|
||||
from backend.models.requests import MuteRegionRequest
|
||||
from pydantic import ValidationError
|
||||
|
||||
with pytest.raises(ValidationError):
|
||||
MuteRegionRequest(start_seconds=-1.0, end_seconds=20.0)
|
||||
|
||||
def test_mute_region_request_invalid_order(self):
|
||||
"""MuteRegionRequest should reject end before start."""
|
||||
from backend.models.requests import MuteRegionRequest
|
||||
from pydantic import ValidationError
|
||||
|
||||
with pytest.raises(ValidationError):
|
||||
MuteRegionRequest(start_seconds=20.0, end_seconds=10.0)
|
||||
|
||||
def test_create_custom_instrumental_request_valid(self):
|
||||
"""CreateCustomInstrumentalRequest should accept valid mute regions."""
|
||||
from backend.models.requests import (
|
||||
CreateCustomInstrumentalRequest,
|
||||
MuteRegionRequest,
|
||||
)
|
||||
|
||||
request = CreateCustomInstrumentalRequest(
|
||||
mute_regions=[
|
||||
MuteRegionRequest(start_seconds=10.0, end_seconds=20.0),
|
||||
MuteRegionRequest(start_seconds=60.0, end_seconds=80.0),
|
||||
]
|
||||
)
|
||||
|
||||
assert len(request.mute_regions) == 2
|
||||
|
||||
|
||||
class TestInstrumentalSelectionExtension:
|
||||
"""Tests for extended instrumental selection (including 'custom')."""
|
||||
|
||||
def test_instrumental_selection_accepts_custom(self):
|
||||
"""InstrumentalSelection should accept 'custom' as valid selection."""
|
||||
from backend.models.requests import InstrumentalSelection
|
||||
|
||||
selection = InstrumentalSelection(selection="custom")
|
||||
assert selection.selection == "custom"
|
||||
|
||||
def test_instrumental_selection_accepts_clean(self):
|
||||
"""InstrumentalSelection should accept 'clean' as valid selection."""
|
||||
from backend.models.requests import InstrumentalSelection
|
||||
|
||||
selection = InstrumentalSelection(selection="clean")
|
||||
assert selection.selection == "clean"
|
||||
|
||||
def test_instrumental_selection_accepts_with_backing(self):
|
||||
"""InstrumentalSelection should accept 'with_backing' as valid selection."""
|
||||
from backend.models.requests import InstrumentalSelection
|
||||
|
||||
selection = InstrumentalSelection(selection="with_backing")
|
||||
assert selection.selection == "with_backing"
|
||||
|
||||
def test_instrumental_selection_rejects_invalid(self):
|
||||
"""InstrumentalSelection should reject invalid values."""
|
||||
from backend.models.requests import InstrumentalSelection
|
||||
from pydantic import ValidationError
|
||||
|
||||
with pytest.raises(ValidationError):
|
||||
InstrumentalSelection(selection="invalid_option")
|
||||
@@ -1,343 +0,0 @@
|
||||
"""
|
||||
Unit tests for internal.py API routes using FastAPI TestClient.
|
||||
|
||||
Internal routes are used for worker communication within Cloud Run.
|
||||
These endpoints require admin authentication.
|
||||
|
||||
With Cloud Tasks integration, these endpoints include idempotency checks
|
||||
to prevent duplicate processing on retries.
|
||||
"""
|
||||
import os
|
||||
import pytest
|
||||
from datetime import datetime, UTC
|
||||
from unittest.mock import MagicMock, AsyncMock, patch
|
||||
from fastapi.testclient import TestClient
|
||||
|
||||
from backend.models.job import Job, JobStatus
|
||||
|
||||
|
||||
# Set test admin token for auth
|
||||
os.environ.setdefault('ADMIN_TOKENS', 'test-admin-token')
|
||||
|
||||
|
||||
def _create_mock_job(job_id: str = "test123") -> Job:
|
||||
"""Create a mock Job object for testing."""
|
||||
return Job(
|
||||
job_id=job_id,
|
||||
status=JobStatus.PROCESSING,
|
||||
created_at=datetime.now(UTC),
|
||||
updated_at=datetime.now(UTC),
|
||||
state_data={}, # Empty state_data - no worker progress yet
|
||||
)
|
||||
|
||||
|
||||
@pytest.fixture
|
||||
def client():
|
||||
"""Create TestClient with mocked workers and job manager."""
|
||||
mock_creds = MagicMock()
|
||||
mock_creds.universe_domain = 'googleapis.com'
|
||||
|
||||
# Mock the worker functions at the internal routes module level
|
||||
mock_audio = AsyncMock()
|
||||
mock_lyrics = AsyncMock()
|
||||
mock_screens = AsyncMock()
|
||||
mock_video = AsyncMock()
|
||||
mock_render = AsyncMock()
|
||||
|
||||
# Mock JobManager for idempotency checks
|
||||
mock_job_manager = MagicMock()
|
||||
mock_job_manager.get_job.return_value = _create_mock_job()
|
||||
mock_job_manager.update_state_data.return_value = None
|
||||
|
||||
with patch('backend.api.routes.internal.process_audio_separation', mock_audio), \
|
||||
patch('backend.api.routes.internal.process_lyrics_transcription', mock_lyrics), \
|
||||
patch('backend.api.routes.internal.generate_screens', mock_screens), \
|
||||
patch('backend.api.routes.internal.generate_video', mock_video), \
|
||||
patch('backend.api.routes.internal.process_render_video', mock_render), \
|
||||
patch('backend.api.routes.internal.JobManager', return_value=mock_job_manager), \
|
||||
patch('backend.services.firestore_service.firestore'), \
|
||||
patch('backend.services.storage_service.storage'), \
|
||||
patch('google.auth.default', return_value=(mock_creds, 'test-project')):
|
||||
from backend.main import app
|
||||
yield TestClient(app)
|
||||
|
||||
|
||||
@pytest.fixture
|
||||
def auth_headers():
|
||||
"""Auth headers for internal API testing."""
|
||||
return {"Authorization": "Bearer test-admin-token"}
|
||||
|
||||
|
||||
class TestAudioWorkerEndpoint:
|
||||
"""Tests for POST /api/internal/workers/audio."""
|
||||
|
||||
def test_audio_worker_returns_200(self, client, auth_headers):
|
||||
"""Test audio worker endpoint returns 200 with auth."""
|
||||
response = client.post(
|
||||
"/api/internal/workers/audio",
|
||||
headers=auth_headers,
|
||||
json={"job_id": "test123"}
|
||||
)
|
||||
assert response.status_code == 200
|
||||
|
||||
def test_audio_worker_accepts_job_id(self, client, auth_headers):
|
||||
"""Test audio worker accepts job_id in body."""
|
||||
response = client.post(
|
||||
"/api/internal/workers/audio",
|
||||
headers=auth_headers,
|
||||
json={"job_id": "any-job-id"}
|
||||
)
|
||||
assert response.status_code == 200
|
||||
|
||||
def test_audio_worker_requires_job_id(self, client, auth_headers):
|
||||
"""Test audio worker requires job_id."""
|
||||
response = client.post(
|
||||
"/api/internal/workers/audio",
|
||||
headers=auth_headers,
|
||||
json={}
|
||||
)
|
||||
assert response.status_code == 422
|
||||
|
||||
def test_audio_worker_requires_auth(self, client, auth_headers):
|
||||
"""Test audio worker requires authentication."""
|
||||
response = client.post(
|
||||
"/api/internal/workers/audio",
|
||||
json={"job_id": "test123"}
|
||||
)
|
||||
assert response.status_code == 401
|
||||
|
||||
|
||||
class TestLyricsWorkerEndpoint:
|
||||
"""Tests for POST /api/internal/workers/lyrics."""
|
||||
|
||||
def test_lyrics_worker_returns_200(self, client, auth_headers):
|
||||
"""Test lyrics worker endpoint returns 200 with auth."""
|
||||
response = client.post(
|
||||
"/api/internal/workers/lyrics",
|
||||
headers=auth_headers,
|
||||
json={"job_id": "test123"}
|
||||
)
|
||||
assert response.status_code == 200
|
||||
|
||||
def test_lyrics_worker_requires_job_id(self, client, auth_headers):
|
||||
"""Test lyrics worker requires job_id."""
|
||||
response = client.post(
|
||||
"/api/internal/workers/lyrics",
|
||||
headers=auth_headers,
|
||||
json={}
|
||||
)
|
||||
assert response.status_code == 422
|
||||
|
||||
def test_lyrics_worker_requires_auth(self, client, auth_headers):
|
||||
"""Test lyrics worker requires authentication."""
|
||||
response = client.post(
|
||||
"/api/internal/workers/lyrics",
|
||||
json={"job_id": "test123"}
|
||||
)
|
||||
assert response.status_code == 401
|
||||
|
||||
|
||||
class TestScreensWorkerEndpoint:
|
||||
"""Tests for POST /api/internal/workers/screens."""
|
||||
|
||||
def test_screens_worker_returns_200(self, client, auth_headers):
|
||||
"""Test screens worker endpoint returns 200 with auth."""
|
||||
response = client.post(
|
||||
"/api/internal/workers/screens",
|
||||
headers=auth_headers,
|
||||
json={"job_id": "test123"}
|
||||
)
|
||||
assert response.status_code == 200
|
||||
|
||||
def test_screens_worker_requires_job_id(self, client, auth_headers):
|
||||
"""Test screens worker requires job_id."""
|
||||
response = client.post(
|
||||
"/api/internal/workers/screens",
|
||||
headers=auth_headers,
|
||||
json={}
|
||||
)
|
||||
assert response.status_code == 422
|
||||
|
||||
def test_screens_worker_requires_auth(self, client, auth_headers):
|
||||
"""Test screens worker requires authentication."""
|
||||
response = client.post(
|
||||
"/api/internal/workers/screens",
|
||||
json={"job_id": "test123"}
|
||||
)
|
||||
assert response.status_code == 401
|
||||
|
||||
|
||||
class TestVideoWorkerEndpoint:
|
||||
"""Tests for POST /api/internal/workers/video."""
|
||||
|
||||
def test_video_worker_returns_200(self, client, auth_headers):
|
||||
"""Test video worker endpoint returns 200 with auth."""
|
||||
response = client.post(
|
||||
"/api/internal/workers/video",
|
||||
headers=auth_headers,
|
||||
json={"job_id": "test123"}
|
||||
)
|
||||
assert response.status_code == 200
|
||||
|
||||
def test_video_worker_requires_job_id(self, client, auth_headers):
|
||||
"""Test video worker requires job_id."""
|
||||
response = client.post(
|
||||
"/api/internal/workers/video",
|
||||
headers=auth_headers,
|
||||
json={}
|
||||
)
|
||||
assert response.status_code == 422
|
||||
|
||||
def test_video_worker_requires_auth(self, client, auth_headers):
|
||||
"""Test video worker requires authentication."""
|
||||
response = client.post(
|
||||
"/api/internal/workers/video",
|
||||
json={"job_id": "test123"}
|
||||
)
|
||||
assert response.status_code == 401
|
||||
|
||||
|
||||
class TestRenderVideoWorkerEndpoint:
|
||||
"""Tests for POST /api/internal/workers/render-video."""
|
||||
|
||||
def test_render_video_worker_returns_200(self, client, auth_headers):
|
||||
"""Test render-video worker endpoint returns 200 with auth."""
|
||||
response = client.post(
|
||||
"/api/internal/workers/render-video",
|
||||
headers=auth_headers,
|
||||
json={"job_id": "test123"}
|
||||
)
|
||||
assert response.status_code == 200
|
||||
|
||||
def test_render_video_worker_requires_job_id(self, client, auth_headers):
|
||||
"""Test render-video worker requires job_id."""
|
||||
response = client.post(
|
||||
"/api/internal/workers/render-video",
|
||||
headers=auth_headers,
|
||||
json={}
|
||||
)
|
||||
assert response.status_code == 422
|
||||
|
||||
def test_render_video_worker_requires_auth(self, client, auth_headers):
|
||||
"""Test render-video worker requires authentication."""
|
||||
response = client.post(
|
||||
"/api/internal/workers/render-video",
|
||||
json={"job_id": "test123"}
|
||||
)
|
||||
assert response.status_code == 401
|
||||
|
||||
|
||||
class TestWorkerResponseFormat:
|
||||
"""Tests for worker response format."""
|
||||
|
||||
def test_audio_worker_response_contains_status(self, client, auth_headers):
|
||||
"""Test audio worker response contains status."""
|
||||
response = client.post(
|
||||
"/api/internal/workers/audio",
|
||||
headers=auth_headers,
|
||||
json={"job_id": "test123"}
|
||||
)
|
||||
data = response.json()
|
||||
assert "status" in data or "message" in data
|
||||
|
||||
def test_lyrics_worker_response_contains_status(self, client, auth_headers):
|
||||
"""Test lyrics worker response contains status."""
|
||||
response = client.post(
|
||||
"/api/internal/workers/lyrics",
|
||||
headers=auth_headers,
|
||||
json={"job_id": "test123"}
|
||||
)
|
||||
data = response.json()
|
||||
assert "status" in data or "message" in data
|
||||
|
||||
|
||||
class TestWorkerIdempotency:
|
||||
"""Tests for worker idempotency checks (Cloud Tasks retry support)."""
|
||||
|
||||
def test_audio_worker_returns_already_running_when_in_progress(self, auth_headers):
|
||||
"""Test audio worker returns already_running if worker is already running."""
|
||||
mock_creds = MagicMock()
|
||||
mock_creds.universe_domain = 'googleapis.com'
|
||||
|
||||
# Create job with audio already running
|
||||
mock_job = _create_mock_job()
|
||||
mock_job.state_data = {'audio_progress': {'stage': 'running'}}
|
||||
|
||||
mock_job_manager = MagicMock()
|
||||
mock_job_manager.get_job.return_value = mock_job
|
||||
|
||||
with patch('backend.api.routes.internal.process_audio_separation', AsyncMock()), \
|
||||
patch('backend.api.routes.internal.JobManager', return_value=mock_job_manager), \
|
||||
patch('backend.services.firestore_service.firestore'), \
|
||||
patch('backend.services.storage_service.storage'), \
|
||||
patch('google.auth.default', return_value=(mock_creds, 'test-project')):
|
||||
from backend.main import app
|
||||
client = TestClient(app)
|
||||
|
||||
response = client.post(
|
||||
"/api/internal/workers/audio",
|
||||
headers=auth_headers,
|
||||
json={"job_id": "test123"}
|
||||
)
|
||||
|
||||
assert response.status_code == 200
|
||||
data = response.json()
|
||||
assert data["status"] == "already_running"
|
||||
|
||||
def test_audio_worker_returns_already_complete_when_finished(self, auth_headers):
|
||||
"""Test audio worker returns already_complete if worker already finished."""
|
||||
mock_creds = MagicMock()
|
||||
mock_creds.universe_domain = 'googleapis.com'
|
||||
|
||||
# Create job with audio already complete
|
||||
mock_job = _create_mock_job()
|
||||
mock_job.state_data = {'audio_progress': {'stage': 'complete'}}
|
||||
|
||||
mock_job_manager = MagicMock()
|
||||
mock_job_manager.get_job.return_value = mock_job
|
||||
|
||||
with patch('backend.api.routes.internal.process_audio_separation', AsyncMock()), \
|
||||
patch('backend.api.routes.internal.JobManager', return_value=mock_job_manager), \
|
||||
patch('backend.services.firestore_service.firestore'), \
|
||||
patch('backend.services.storage_service.storage'), \
|
||||
patch('google.auth.default', return_value=(mock_creds, 'test-project')):
|
||||
from backend.main import app
|
||||
client = TestClient(app)
|
||||
|
||||
response = client.post(
|
||||
"/api/internal/workers/audio",
|
||||
headers=auth_headers,
|
||||
json={"job_id": "test123"}
|
||||
)
|
||||
|
||||
assert response.status_code == 200
|
||||
data = response.json()
|
||||
assert data["status"] == "already_complete"
|
||||
|
||||
def test_audio_worker_returns_not_found_when_job_missing(self, auth_headers):
|
||||
"""Test audio worker returns not_found if job doesn't exist."""
|
||||
mock_creds = MagicMock()
|
||||
mock_creds.universe_domain = 'googleapis.com'
|
||||
|
||||
# Return None for job not found
|
||||
mock_job_manager = MagicMock()
|
||||
mock_job_manager.get_job.return_value = None
|
||||
|
||||
with patch('backend.api.routes.internal.process_audio_separation', AsyncMock()), \
|
||||
patch('backend.api.routes.internal.JobManager', return_value=mock_job_manager), \
|
||||
patch('backend.services.firestore_service.firestore'), \
|
||||
patch('backend.services.storage_service.storage'), \
|
||||
patch('google.auth.default', return_value=(mock_creds, 'test-project')):
|
||||
from backend.main import app
|
||||
client = TestClient(app)
|
||||
|
||||
response = client.post(
|
||||
"/api/internal/workers/audio",
|
||||
headers=auth_headers,
|
||||
json={"job_id": "nonexistent"}
|
||||
)
|
||||
|
||||
assert response.status_code == 200
|
||||
data = response.json()
|
||||
assert data["status"] == "not_found"
|
||||
|
||||
@@ -1,339 +0,0 @@
|
||||
"""
|
||||
Unit tests for Job Manager service.
|
||||
|
||||
Tests the job lifecycle management without requiring actual Firestore connection.
|
||||
Uses mocking to isolate the business logic.
|
||||
"""
|
||||
import pytest
|
||||
from unittest.mock import Mock, MagicMock, patch
|
||||
from datetime import datetime, UTC
|
||||
|
||||
# Mock Firestore before importing JobManager
|
||||
import sys
|
||||
sys.modules['google.cloud.firestore'] = MagicMock()
|
||||
|
||||
from backend.services.job_manager import JobManager
|
||||
from backend.models.job import Job, JobCreate, JobStatus
|
||||
|
||||
|
||||
@pytest.fixture
|
||||
def mock_firestore_service():
|
||||
"""Mock FirestoreService."""
|
||||
with patch('backend.services.job_manager.FirestoreService') as mock:
|
||||
service = Mock()
|
||||
mock.return_value = service
|
||||
yield service
|
||||
|
||||
|
||||
@pytest.fixture
|
||||
def job_manager(mock_firestore_service):
|
||||
"""Create JobManager with mocked dependencies."""
|
||||
return JobManager()
|
||||
|
||||
|
||||
class TestJobCreation:
|
||||
"""Test job creation logic."""
|
||||
|
||||
def test_create_job_with_url(self, job_manager, mock_firestore_service):
|
||||
"""Test creating a job with YouTube URL."""
|
||||
job_create = JobCreate(
|
||||
url="https://youtube.com/watch?v=test",
|
||||
artist="Test Artist",
|
||||
title="Test Song"
|
||||
)
|
||||
|
||||
# The actual create_job method creates the job and returns it
|
||||
job = job_manager.create_job(job_create)
|
||||
|
||||
assert job.job_id is not None
|
||||
assert job.status == JobStatus.PENDING
|
||||
assert job.url == "https://youtube.com/watch?v=test"
|
||||
assert job.artist == "Test Artist"
|
||||
assert job.title == "Test Song"
|
||||
assert job.progress == 0
|
||||
|
||||
# Verify Firestore was called
|
||||
mock_firestore_service.create_job.assert_called_once()
|
||||
|
||||
def test_create_job_without_url(self, job_manager, mock_firestore_service):
|
||||
"""Test creating a job without URL (for file upload)."""
|
||||
job_create = JobCreate(
|
||||
artist="Test Artist",
|
||||
title="Test Song"
|
||||
)
|
||||
|
||||
job = job_manager.create_job(job_create)
|
||||
|
||||
assert job.artist == "Test Artist"
|
||||
assert job.title == "Test Song"
|
||||
assert job.url is None
|
||||
|
||||
def test_create_job_generates_unique_id(self, job_manager, mock_firestore_service):
|
||||
"""Test that each job gets a unique ID."""
|
||||
job_create = JobCreate()
|
||||
|
||||
# Create multiple jobs
|
||||
ids = []
|
||||
for i in range(5):
|
||||
mock_firestore_service.create_job.return_value = Job(
|
||||
job_id=f"test{i}",
|
||||
status=JobStatus.PENDING,
|
||||
created_at=datetime.now(UTC),
|
||||
updated_at=datetime.now(UTC)
|
||||
)
|
||||
job = job_manager.create_job(job_create)
|
||||
ids.append(job.job_id)
|
||||
|
||||
# All IDs should be unique
|
||||
assert len(ids) == len(set(ids))
|
||||
|
||||
def test_create_job_sets_initial_status(self, job_manager, mock_firestore_service):
|
||||
"""Test that new jobs start with PENDING status."""
|
||||
job_create = JobCreate()
|
||||
|
||||
mock_firestore_service.create_job.return_value = Job(
|
||||
job_id="test123",
|
||||
status=JobStatus.PENDING,
|
||||
created_at=datetime.now(UTC),
|
||||
updated_at=datetime.now(UTC)
|
||||
)
|
||||
|
||||
job = job_manager.create_job(job_create)
|
||||
|
||||
assert job.status == JobStatus.PENDING
|
||||
assert job.progress == 0
|
||||
|
||||
def test_create_job_with_distribution_settings(self, job_manager, mock_firestore_service):
|
||||
"""Test that distribution settings are passed from JobCreate to Job.
|
||||
|
||||
This was a bug where brand_prefix, dropbox_path, gdrive_folder_id, and
|
||||
discord_webhook_url were NOT being passed to the Job constructor.
|
||||
"""
|
||||
job_create = JobCreate(
|
||||
artist="Test Artist",
|
||||
title="Test Song",
|
||||
brand_prefix="NOMAD",
|
||||
discord_webhook_url="https://discord.com/webhook/test",
|
||||
dropbox_path="/Karaoke/Tracks-Organized",
|
||||
gdrive_folder_id="1abc123xyz",
|
||||
enable_youtube_upload=True,
|
||||
)
|
||||
|
||||
job = job_manager.create_job(job_create)
|
||||
|
||||
# Verify distribution settings are passed through
|
||||
assert job.brand_prefix == "NOMAD"
|
||||
assert job.discord_webhook_url == "https://discord.com/webhook/test"
|
||||
assert job.dropbox_path == "/Karaoke/Tracks-Organized"
|
||||
assert job.gdrive_folder_id == "1abc123xyz"
|
||||
assert job.enable_youtube_upload is True
|
||||
|
||||
# Verify Firestore was called with job containing these fields
|
||||
mock_firestore_service.create_job.assert_called_once()
|
||||
created_job = mock_firestore_service.create_job.call_args[0][0]
|
||||
assert created_job.brand_prefix == "NOMAD"
|
||||
assert created_job.dropbox_path == "/Karaoke/Tracks-Organized"
|
||||
|
||||
|
||||
class TestJobRetrieval:
|
||||
"""Test job retrieval logic."""
|
||||
|
||||
def test_get_existing_job(self, job_manager, mock_firestore_service):
|
||||
"""Test retrieving an existing job."""
|
||||
expected_job = Job(
|
||||
job_id="test123",
|
||||
status=JobStatus.SEPARATING_STAGE1,
|
||||
progress=25,
|
||||
created_at=datetime.now(UTC),
|
||||
updated_at=datetime.now(UTC)
|
||||
)
|
||||
|
||||
mock_firestore_service.get_job.return_value = expected_job
|
||||
|
||||
job = job_manager.get_job("test123")
|
||||
|
||||
assert job.job_id == "test123"
|
||||
assert job.status == JobStatus.SEPARATING_STAGE1
|
||||
assert job.progress == 25
|
||||
|
||||
mock_firestore_service.get_job.assert_called_once_with("test123")
|
||||
|
||||
def test_get_nonexistent_job(self, job_manager, mock_firestore_service):
|
||||
"""Test retrieving a nonexistent job returns None."""
|
||||
mock_firestore_service.get_job.return_value = None
|
||||
|
||||
job = job_manager.get_job("nonexistent")
|
||||
|
||||
assert job is None
|
||||
|
||||
|
||||
class TestJobUpdate:
|
||||
"""Test job update logic."""
|
||||
|
||||
def test_update_job_status(self, job_manager, mock_firestore_service):
|
||||
"""Test updating job status."""
|
||||
# Setup: job exists
|
||||
existing_job = Job(
|
||||
job_id="test123",
|
||||
status=JobStatus.PENDING,
|
||||
progress=0,
|
||||
created_at=datetime.now(UTC),
|
||||
updated_at=datetime.now(UTC)
|
||||
)
|
||||
mock_firestore_service.get_job.return_value = existing_job
|
||||
|
||||
# Update status
|
||||
updates = {"status": JobStatus.SEPARATING_STAGE1, "progress": 25}
|
||||
job_manager.update_job("test123", updates)
|
||||
|
||||
# Verify update was called
|
||||
mock_firestore_service.update_job.assert_called_once()
|
||||
call_args = mock_firestore_service.update_job.call_args
|
||||
assert call_args[0][0] == "test123" # job_id
|
||||
assert "status" in call_args[0][1] # updates dict
|
||||
|
||||
def test_update_job_multiple_fields(self, job_manager, mock_firestore_service):
|
||||
"""Test updating multiple fields at once."""
|
||||
existing_job = Job(
|
||||
job_id="test123",
|
||||
status=JobStatus.PENDING,
|
||||
progress=0,
|
||||
created_at=datetime.now(UTC),
|
||||
updated_at=datetime.now(UTC),
|
||||
timeline=[]
|
||||
)
|
||||
mock_firestore_service.get_job.return_value = existing_job
|
||||
|
||||
updates = {"status": JobStatus.SEPARATING_STAGE1, "progress": 25}
|
||||
job_manager.update_job("test123", updates)
|
||||
|
||||
# Verify update was called
|
||||
call_args = mock_firestore_service.update_job.call_args
|
||||
updates_dict = call_args[0][1]
|
||||
assert "status" in updates_dict
|
||||
assert "progress" in updates_dict
|
||||
|
||||
def test_update_input_media_gcs_path(self, job_manager, mock_firestore_service):
|
||||
"""Test updating input_media_gcs_path field."""
|
||||
existing_job = Job(
|
||||
job_id="test123",
|
||||
status=JobStatus.PENDING,
|
||||
progress=0,
|
||||
created_at=datetime.now(UTC),
|
||||
updated_at=datetime.now(UTC)
|
||||
)
|
||||
mock_firestore_service.get_job.return_value = existing_job
|
||||
|
||||
updates = {"input_media_gcs_path": "uploads/test123/file.flac"}
|
||||
job_manager.update_job("test123", updates)
|
||||
|
||||
# Verify update was called with input_media_gcs_path
|
||||
call_args = mock_firestore_service.update_job.call_args
|
||||
updates_dict = call_args[0][1]
|
||||
assert "input_media_gcs_path" in updates_dict
|
||||
assert updates_dict["input_media_gcs_path"] == "uploads/test123/file.flac"
|
||||
|
||||
|
||||
class TestJobStatusTransitions:
|
||||
"""Test valid job status transitions."""
|
||||
|
||||
def test_pending_to_downloading(self, job_manager, mock_firestore_service):
|
||||
"""Test transition from PENDING to DOWNLOADING."""
|
||||
existing_job = Job(
|
||||
job_id="test123",
|
||||
status=JobStatus.PENDING,
|
||||
progress=0,
|
||||
created_at=datetime.now(UTC),
|
||||
updated_at=datetime.now(UTC)
|
||||
)
|
||||
mock_firestore_service.get_job.return_value = existing_job
|
||||
|
||||
job_manager.update_job("test123", {"status": JobStatus.DOWNLOADING})
|
||||
|
||||
# Should succeed (valid transition)
|
||||
mock_firestore_service.update_job.assert_called_once()
|
||||
|
||||
def test_downloading_to_separating(self, job_manager, mock_firestore_service):
|
||||
"""Test transition from DOWNLOADING to SEPARATING_STAGE1."""
|
||||
existing_job = Job(
|
||||
job_id="test123",
|
||||
status=JobStatus.DOWNLOADING,
|
||||
progress=0,
|
||||
created_at=datetime.now(UTC),
|
||||
updated_at=datetime.now(UTC)
|
||||
)
|
||||
mock_firestore_service.get_job.return_value = existing_job
|
||||
|
||||
job_manager.update_job("test123", {"status": JobStatus.SEPARATING_STAGE1})
|
||||
|
||||
mock_firestore_service.update_job.assert_called_once()
|
||||
|
||||
|
||||
class TestJobFailure:
|
||||
"""Test job failure handling."""
|
||||
|
||||
def test_mark_job_as_failed(self, job_manager, mock_firestore_service):
|
||||
"""Test marking a job as failed."""
|
||||
error_message = "Audio separation failed"
|
||||
job_manager.mark_job_failed("test123", error_message)
|
||||
|
||||
# mark_job_failed calls firestore.update_job_status
|
||||
mock_firestore_service.update_job_status.assert_called_once()
|
||||
call_args = mock_firestore_service.update_job_status.call_args
|
||||
assert call_args[1]['job_id'] == "test123"
|
||||
assert call_args[1]['status'] == JobStatus.FAILED
|
||||
assert 'error_message' in call_args[1]
|
||||
|
||||
def test_mark_job_error(self, job_manager, mock_firestore_service):
|
||||
"""Test marking a job with an error."""
|
||||
error_message = "Test error"
|
||||
job_manager.mark_job_error("test123", error_message)
|
||||
|
||||
# mark_job_error calls firestore.update_job_status
|
||||
mock_firestore_service.update_job_status.assert_called_once()
|
||||
call_args = mock_firestore_service.update_job_status.call_args
|
||||
assert call_args[1]['job_id'] == "test123"
|
||||
assert call_args[1]['error_message'] == error_message
|
||||
|
||||
|
||||
class TestJobDeletion:
|
||||
"""Test job deletion logic."""
|
||||
|
||||
def test_delete_job(self, job_manager, mock_firestore_service):
|
||||
"""Test deleting a job."""
|
||||
existing_job = Job(
|
||||
job_id="test123",
|
||||
status=JobStatus.COMPLETE,
|
||||
progress=100,
|
||||
created_at=datetime.now(UTC),
|
||||
updated_at=datetime.now(UTC),
|
||||
output_files={} # Empty dict so iteration works
|
||||
)
|
||||
mock_firestore_service.get_job.return_value = existing_job
|
||||
|
||||
job_manager.delete_job("test123")
|
||||
|
||||
# Verify delete was called
|
||||
mock_firestore_service.delete_job.assert_called_once_with("test123")
|
||||
|
||||
def test_delete_job_with_files(self, job_manager, mock_firestore_service):
|
||||
"""Test deleting a job and its files."""
|
||||
existing_job = Job(
|
||||
job_id="test123",
|
||||
status=JobStatus.COMPLETE,
|
||||
progress=100,
|
||||
created_at=datetime.now(UTC),
|
||||
updated_at=datetime.now(UTC),
|
||||
output_files={} # Empty dict so iteration works
|
||||
)
|
||||
mock_firestore_service.get_job.return_value = existing_job
|
||||
|
||||
job_manager.delete_job("test123", delete_files=True)
|
||||
|
||||
# Verify delete was called
|
||||
mock_firestore_service.delete_job.assert_called_once_with("test123")
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
pytest.main([__file__, "-v"])
|
||||
|
||||
@@ -1,258 +0,0 @@
|
||||
"""
|
||||
Unit tests for jobs.py API routes using FastAPI TestClient.
|
||||
|
||||
These tests mock the underlying services and test the route logic directly.
|
||||
"""
|
||||
import pytest
|
||||
from datetime import datetime, UTC
|
||||
from unittest.mock import MagicMock, AsyncMock, patch
|
||||
from fastapi.testclient import TestClient
|
||||
|
||||
from backend.models.job import Job, JobStatus
|
||||
|
||||
|
||||
@pytest.fixture
|
||||
def mock_job():
|
||||
"""Create a standard mock job for testing."""
|
||||
return Job(
|
||||
job_id="test123",
|
||||
status=JobStatus.PENDING,
|
||||
created_at=datetime.now(UTC),
|
||||
updated_at=datetime.now(UTC),
|
||||
artist="Test Artist",
|
||||
title="Test Song"
|
||||
)
|
||||
|
||||
|
||||
@pytest.fixture
|
||||
def mock_job_manager(mock_job):
|
||||
"""Create a mock JobManager with common methods."""
|
||||
manager = MagicMock()
|
||||
manager.get_job.return_value = mock_job
|
||||
manager.list_jobs.return_value = [mock_job]
|
||||
manager.create_job.return_value = mock_job
|
||||
manager.delete_job.return_value = True
|
||||
manager.cancel_job.return_value = mock_job
|
||||
manager.update_job_status.return_value = None
|
||||
manager.update_state_data.return_value = None
|
||||
return manager
|
||||
|
||||
|
||||
@pytest.fixture
|
||||
def mock_worker_service():
|
||||
"""Create a mock WorkerService."""
|
||||
service = MagicMock()
|
||||
service.trigger_audio_worker = AsyncMock(return_value=True)
|
||||
service.trigger_lyrics_worker = AsyncMock(return_value=True)
|
||||
service.trigger_screens_worker = AsyncMock(return_value=True)
|
||||
service.trigger_video_worker = AsyncMock(return_value=True)
|
||||
return service
|
||||
|
||||
|
||||
@pytest.fixture
|
||||
def client(mock_job_manager, mock_worker_service):
|
||||
"""Create TestClient with mocked dependencies."""
|
||||
mock_creds = MagicMock()
|
||||
mock_creds.universe_domain = 'googleapis.com'
|
||||
|
||||
# Patch at the module level where jobs.py imports them
|
||||
with patch('backend.api.routes.jobs.job_manager', mock_job_manager), \
|
||||
patch('backend.api.routes.jobs.worker_service', mock_worker_service), \
|
||||
patch('backend.services.firestore_service.firestore'), \
|
||||
patch('backend.services.storage_service.storage'), \
|
||||
patch('google.auth.default', return_value=(mock_creds, 'test-project')):
|
||||
from backend.main import app
|
||||
yield TestClient(app)
|
||||
|
||||
|
||||
class TestGetJob:
|
||||
"""Tests for GET /api/jobs/{job_id}."""
|
||||
|
||||
def test_get_job_returns_200(self, client, mock_job_manager, mock_job, auth_headers):
|
||||
"""Test getting an existing job returns 200."""
|
||||
response = client.get("/api/jobs/test123", headers=auth_headers)
|
||||
assert response.status_code == 200
|
||||
|
||||
def test_get_job_returns_job_data(self, client, mock_job_manager, mock_job, auth_headers):
|
||||
"""Test response contains job data."""
|
||||
response = client.get("/api/jobs/test123", headers=auth_headers)
|
||||
data = response.json()
|
||||
assert data["job_id"] == "test123"
|
||||
assert data["status"] == "pending"
|
||||
assert data["artist"] == "Test Artist"
|
||||
|
||||
def test_get_nonexistent_job_returns_404(self, mock_worker_service, auth_headers):
|
||||
"""Test getting non-existent job returns 404."""
|
||||
mock_job_manager = MagicMock()
|
||||
mock_job_manager.get_job.return_value = None
|
||||
mock_creds = MagicMock()
|
||||
mock_creds.universe_domain = 'googleapis.com'
|
||||
|
||||
with patch('backend.api.routes.jobs.job_manager', mock_job_manager), \
|
||||
patch('backend.api.routes.jobs.worker_service', mock_worker_service), \
|
||||
patch('backend.services.firestore_service.firestore'), \
|
||||
patch('backend.services.storage_service.storage'), \
|
||||
patch('google.auth.default', return_value=(mock_creds, 'test-project')):
|
||||
from backend.main import app
|
||||
client = TestClient(app)
|
||||
response = client.get("/api/jobs/nonexistent", headers=auth_headers)
|
||||
assert response.status_code == 404
|
||||
|
||||
|
||||
class TestListJobs:
|
||||
"""Tests for GET /api/jobs."""
|
||||
|
||||
def test_list_jobs_returns_200(self, client, auth_headers):
|
||||
"""Test listing jobs returns 200."""
|
||||
response = client.get("/api/jobs", headers=auth_headers)
|
||||
assert response.status_code == 200
|
||||
|
||||
def test_list_jobs_returns_array(self, client, mock_job_manager, auth_headers):
|
||||
"""Test response is an array of jobs."""
|
||||
response = client.get("/api/jobs", headers=auth_headers)
|
||||
data = response.json()
|
||||
assert isinstance(data, list)
|
||||
assert len(data) == 1
|
||||
assert data[0]["job_id"] == "test123"
|
||||
|
||||
def test_list_jobs_with_status_filter(self, client, mock_job_manager, auth_headers):
|
||||
"""Test listing jobs with status filter."""
|
||||
response = client.get("/api/jobs?status=pending", headers=auth_headers)
|
||||
assert response.status_code == 200
|
||||
|
||||
def test_list_jobs_with_limit(self, client, mock_job_manager, auth_headers):
|
||||
"""Test listing jobs with limit."""
|
||||
response = client.get("/api/jobs?limit=10", headers=auth_headers)
|
||||
assert response.status_code == 200
|
||||
|
||||
|
||||
class TestCreateJob:
|
||||
"""Tests for POST /api/jobs."""
|
||||
|
||||
def test_create_job_with_url_returns_200(self, client, mock_job_manager, auth_headers):
|
||||
"""Test creating job with URL returns 200."""
|
||||
response = client.post(
|
||||
"/api/jobs",
|
||||
json={"url": "https://youtube.com/watch?v=test123"},
|
||||
headers=auth_headers
|
||||
)
|
||||
assert response.status_code == 200
|
||||
|
||||
def test_create_job_returns_job_id(self, client, mock_job_manager, auth_headers):
|
||||
"""Test create response contains job_id."""
|
||||
response = client.post(
|
||||
"/api/jobs",
|
||||
json={"url": "https://youtube.com/watch?v=test"},
|
||||
headers=auth_headers
|
||||
)
|
||||
data = response.json()
|
||||
assert "job_id" in data
|
||||
|
||||
def test_create_job_with_artist_title(self, client, mock_job_manager, auth_headers):
|
||||
"""Test creating job with artist and title."""
|
||||
response = client.post(
|
||||
"/api/jobs",
|
||||
json={
|
||||
"url": "https://youtube.com/watch?v=test",
|
||||
"artist": "Test Artist",
|
||||
"title": "Test Song"
|
||||
},
|
||||
headers=auth_headers
|
||||
)
|
||||
assert response.status_code == 200
|
||||
|
||||
|
||||
class TestDeleteJob:
|
||||
"""Tests for DELETE /api/jobs/{job_id}."""
|
||||
|
||||
def test_delete_job_returns_200(self, client, mock_job_manager, auth_headers):
|
||||
"""Test deleting job returns 200."""
|
||||
response = client.delete("/api/jobs/test123", headers=auth_headers)
|
||||
assert response.status_code == 200
|
||||
|
||||
def test_delete_nonexistent_job(self, mock_worker_service, auth_headers):
|
||||
"""Test deleting non-existent job."""
|
||||
mock_job_manager = MagicMock()
|
||||
mock_job_manager.delete_job.return_value = False
|
||||
mock_creds = MagicMock()
|
||||
mock_creds.universe_domain = 'googleapis.com'
|
||||
|
||||
with patch('backend.api.routes.jobs.job_manager', mock_job_manager), \
|
||||
patch('backend.api.routes.jobs.worker_service', mock_worker_service), \
|
||||
patch('backend.services.firestore_service.firestore'), \
|
||||
patch('backend.services.storage_service.storage'), \
|
||||
patch('google.auth.default', return_value=(mock_creds, 'test-project')):
|
||||
from backend.main import app
|
||||
client = TestClient(app)
|
||||
response = client.delete("/api/jobs/nonexistent", headers=auth_headers)
|
||||
# Either 200 or 404 depending on implementation
|
||||
assert response.status_code in [200, 404]
|
||||
|
||||
|
||||
class TestCancelJob:
|
||||
"""Tests for POST /api/jobs/{job_id}/cancel."""
|
||||
|
||||
def test_cancel_job_returns_200(self, client, mock_job_manager, mock_job, auth_headers):
|
||||
"""Test cancelling job returns 200."""
|
||||
mock_job_manager.cancel_job.return_value = mock_job
|
||||
response = client.post(
|
||||
"/api/jobs/test123/cancel",
|
||||
json={},
|
||||
headers=auth_headers
|
||||
)
|
||||
assert response.status_code == 200
|
||||
|
||||
|
||||
class TestSubmitCorrections:
|
||||
"""Tests for POST /api/jobs/{job_id}/corrections."""
|
||||
|
||||
def test_submit_corrections_returns_200(self, client, mock_job_manager, mock_job, auth_headers):
|
||||
"""Test submitting corrections returns 200."""
|
||||
# Job needs to be in AWAITING_REVIEW or IN_REVIEW status
|
||||
review_job = Job(
|
||||
job_id="test123",
|
||||
status=JobStatus.IN_REVIEW,
|
||||
created_at=datetime.now(UTC),
|
||||
updated_at=datetime.now(UTC),
|
||||
artist="Test",
|
||||
title="Test"
|
||||
)
|
||||
mock_job_manager.get_job.return_value = review_job
|
||||
|
||||
response = client.post(
|
||||
"/api/jobs/test123/corrections",
|
||||
json={
|
||||
"corrections": {
|
||||
"lines": [],
|
||||
"metadata": {"source": "test"}
|
||||
}
|
||||
},
|
||||
headers=auth_headers
|
||||
)
|
||||
# Should be 200 or validation error
|
||||
assert response.status_code in [200, 400, 422]
|
||||
|
||||
|
||||
class TestSelectInstrumental:
|
||||
"""Tests for POST /api/jobs/{job_id}/select-instrumental."""
|
||||
|
||||
def test_select_instrumental_requires_selection_field(self, client, mock_job_manager, mock_job, auth_headers):
|
||||
"""Test instrumental selection requires selection field."""
|
||||
response = client.post(
|
||||
"/api/jobs/test123/select-instrumental",
|
||||
json={}, # Missing selection field
|
||||
headers=auth_headers
|
||||
)
|
||||
# Missing field should cause validation error
|
||||
assert response.status_code == 422
|
||||
|
||||
|
||||
class TestStartReview:
|
||||
"""Tests for POST /api/jobs/{job_id}/start-review."""
|
||||
|
||||
def test_start_review_endpoint_exists(self, client, mock_job_manager, mock_job, auth_headers):
|
||||
"""Test start review endpoint exists."""
|
||||
response = client.post("/api/jobs/test123/start-review", headers=auth_headers)
|
||||
# Should not be 404 or 405
|
||||
assert response.status_code not in [404, 405]
|
||||
|
||||
@@ -1,87 +0,0 @@
|
||||
"""
|
||||
Unit tests for backend main.py and config.py.
|
||||
|
||||
These tests cover the FastAPI app initialization and configuration.
|
||||
"""
|
||||
import pytest
|
||||
import os
|
||||
from unittest.mock import MagicMock, patch
|
||||
|
||||
|
||||
class TestConfig:
|
||||
"""Tests for config.py."""
|
||||
|
||||
def test_settings_from_environment(self):
|
||||
"""Test settings can be loaded from environment variables."""
|
||||
with patch.dict(os.environ, {
|
||||
'GCS_BUCKET_NAME': 'test-bucket',
|
||||
'FIRESTORE_COLLECTION': 'test-jobs',
|
||||
'ADMIN_TOKENS': 'token1,token2',
|
||||
'ENVIRONMENT': 'testing'
|
||||
}):
|
||||
from backend.config import Settings
|
||||
settings = Settings()
|
||||
assert settings.gcs_bucket_name == 'test-bucket'
|
||||
assert settings.firestore_collection == 'test-jobs'
|
||||
# admin_tokens is a string, comma-separated
|
||||
assert 'token1' in settings.admin_tokens
|
||||
assert 'token2' in settings.admin_tokens
|
||||
|
||||
def test_admin_tokens_is_comma_separated_string(self):
|
||||
"""Test admin tokens stored as comma-separated string."""
|
||||
with patch.dict(os.environ, {
|
||||
'ADMIN_TOKENS': 'token1,token2,token3'
|
||||
}):
|
||||
from backend.config import Settings
|
||||
settings = Settings()
|
||||
# admin_tokens is stored as a string
|
||||
assert settings.admin_tokens == 'token1,token2,token3'
|
||||
|
||||
def test_default_environment(self):
|
||||
"""Test default environment is development."""
|
||||
with patch.dict(os.environ, {}, clear=True):
|
||||
from backend.config import Settings
|
||||
settings = Settings()
|
||||
# Accept 'test' as well since pytest may set ENVIRONMENT=test
|
||||
assert settings.environment in ['development', 'production', 'testing', 'test']
|
||||
|
||||
|
||||
class TestMain:
|
||||
"""Tests for main.py FastAPI app."""
|
||||
|
||||
@pytest.fixture
|
||||
def app(self):
|
||||
"""Create FastAPI app with mocked services."""
|
||||
mock_creds = MagicMock()
|
||||
mock_creds.universe_domain = 'googleapis.com'
|
||||
with patch('backend.services.firestore_service.firestore'), \
|
||||
patch('backend.services.storage_service.storage'), \
|
||||
patch('google.auth.default', return_value=(mock_creds, 'test-project')):
|
||||
from backend.main import app
|
||||
return app
|
||||
|
||||
def test_app_has_api_routes(self, app):
|
||||
"""Test app includes API routes."""
|
||||
routes = [route.path for route in app.routes]
|
||||
assert any('/api' in route or '/jobs' in route for route in routes)
|
||||
|
||||
def test_app_has_cors_middleware(self, app):
|
||||
"""Test app has CORS middleware configured."""
|
||||
# Check that middleware is configured (not necessarily CORS specific)
|
||||
assert len(app.user_middleware) >= 0 # App initializes middleware
|
||||
|
||||
def test_app_title(self, app):
|
||||
"""Test app has expected title."""
|
||||
assert 'karaoke' in app.title.lower() or app.title
|
||||
|
||||
|
||||
class TestDependencies:
|
||||
"""Tests for dependencies.py."""
|
||||
|
||||
def test_dependencies_module_imports(self):
|
||||
"""Test dependencies module can be imported and has auth functions."""
|
||||
from backend.api import dependencies
|
||||
assert hasattr(dependencies, 'require_auth')
|
||||
assert hasattr(dependencies, 'require_admin')
|
||||
assert hasattr(dependencies, 'optional_auth')
|
||||
|
||||
@@ -1,918 +0,0 @@
|
||||
"""
|
||||
Unit tests for Pydantic models.
|
||||
|
||||
Tests validate that our data models work correctly, including:
|
||||
- Basic field validation
|
||||
- Nested dictionary structures (file_urls, state_data)
|
||||
- Serialization/deserialization round-trips (simulating Firestore)
|
||||
- Real-world data structures from workers
|
||||
|
||||
These tests catch type mismatches that would cause runtime errors when
|
||||
reading data back from Firestore.
|
||||
"""
|
||||
import pytest
|
||||
import json
|
||||
from datetime import datetime, UTC
|
||||
from pydantic import ValidationError
|
||||
|
||||
from backend.models.job import (
|
||||
Job, JobCreate, JobStatus, TimelineEvent
|
||||
)
|
||||
|
||||
|
||||
class TestJobModel:
|
||||
"""Test Job Pydantic model - the bug we just fixed!"""
|
||||
|
||||
def test_input_media_gcs_path_field_exists(self):
|
||||
"""
|
||||
Test that Job model has input_media_gcs_path field.
|
||||
|
||||
This test would have caught the bug where we tried to use
|
||||
job.input_media_gcs_path but it wasn't defined in the model!
|
||||
"""
|
||||
job = Job(
|
||||
job_id="test123",
|
||||
status=JobStatus.PENDING,
|
||||
created_at=datetime.now(UTC),
|
||||
updated_at=datetime.now(UTC),
|
||||
input_media_gcs_path="uploads/test123/file.flac"
|
||||
)
|
||||
|
||||
# This should NOT raise AttributeError
|
||||
assert hasattr(job, 'input_media_gcs_path')
|
||||
assert job.input_media_gcs_path == "uploads/test123/file.flac"
|
||||
|
||||
def test_input_media_gcs_path_optional(self):
|
||||
"""Test that input_media_gcs_path is optional."""
|
||||
job = Job(
|
||||
job_id="test123",
|
||||
status=JobStatus.PENDING,
|
||||
created_at=datetime.now(UTC),
|
||||
updated_at=datetime.now(UTC)
|
||||
)
|
||||
|
||||
assert job.input_media_gcs_path is None
|
||||
|
||||
def test_pydantic_includes_input_media_gcs_path_in_dict(self):
|
||||
"""
|
||||
Test that Pydantic includes input_media_gcs_path in serialization.
|
||||
|
||||
This ensures Pydantic doesn't silently ignore the field.
|
||||
"""
|
||||
job = Job(
|
||||
job_id="test123",
|
||||
status=JobStatus.PENDING,
|
||||
created_at=datetime.now(UTC),
|
||||
updated_at=datetime.now(UTC),
|
||||
input_media_gcs_path="uploads/test123/file.flac"
|
||||
)
|
||||
|
||||
job_dict = job.model_dump()
|
||||
|
||||
# Pydantic should include it
|
||||
assert "input_media_gcs_path" in job_dict
|
||||
assert job_dict["input_media_gcs_path"] == "uploads/test123/file.flac"
|
||||
|
||||
def test_create_minimal_job(self):
|
||||
"""Test creating a job with minimal required fields."""
|
||||
job = Job(
|
||||
job_id="test123",
|
||||
status=JobStatus.PENDING,
|
||||
created_at=datetime.now(UTC),
|
||||
updated_at=datetime.now(UTC)
|
||||
)
|
||||
|
||||
assert job.job_id == "test123"
|
||||
assert job.status == JobStatus.PENDING
|
||||
assert job.progress == 0
|
||||
assert job.url is None
|
||||
assert job.input_media_gcs_path is None
|
||||
|
||||
def test_create_job_with_url(self):
|
||||
"""Test creating a job with YouTube URL."""
|
||||
job = Job(
|
||||
job_id="test123",
|
||||
status=JobStatus.PENDING,
|
||||
created_at=datetime.now(UTC),
|
||||
updated_at=datetime.now(UTC),
|
||||
url="https://youtube.com/watch?v=test",
|
||||
artist="Test Artist",
|
||||
title="Test Song"
|
||||
)
|
||||
|
||||
assert job.url == "https://youtube.com/watch?v=test"
|
||||
assert job.artist == "Test Artist"
|
||||
assert job.title == "Test Song"
|
||||
|
||||
def test_create_job_with_uploaded_file(self):
|
||||
"""Test creating a job with uploaded file."""
|
||||
job = Job(
|
||||
job_id="test123",
|
||||
status=JobStatus.PENDING,
|
||||
created_at=datetime.now(UTC),
|
||||
updated_at=datetime.now(UTC),
|
||||
input_media_gcs_path="uploads/test123/file.flac",
|
||||
artist="Test Artist",
|
||||
title="Test Song"
|
||||
)
|
||||
|
||||
assert job.input_media_gcs_path == "uploads/test123/file.flac"
|
||||
assert job.url is None
|
||||
|
||||
|
||||
class TestJobCreate:
|
||||
"""Test JobCreate validation model."""
|
||||
|
||||
def test_create_with_url(self):
|
||||
"""Test creating job with YouTube URL."""
|
||||
job_create = JobCreate(
|
||||
url="https://youtube.com/watch?v=test",
|
||||
artist="Test Artist",
|
||||
title="Test Song"
|
||||
)
|
||||
|
||||
assert job_create.url == "https://youtube.com/watch?v=test"
|
||||
assert job_create.artist == "Test Artist"
|
||||
assert job_create.title == "Test Song"
|
||||
|
||||
def test_create_minimal(self):
|
||||
"""Test creating job with minimal fields."""
|
||||
job_create = JobCreate()
|
||||
|
||||
assert job_create.url is None
|
||||
assert job_create.artist is None
|
||||
assert job_create.title is None
|
||||
|
||||
|
||||
class TestJobStatus:
|
||||
"""Test JobStatus enum."""
|
||||
|
||||
def test_critical_statuses_defined(self):
|
||||
"""Test that critical statuses exist."""
|
||||
critical_statuses = [
|
||||
"pending", "downloading",
|
||||
"separating_stage1", "separating_stage2", "audio_complete",
|
||||
"transcribing", "correcting", "lyrics_complete",
|
||||
"generating_screens", "applying_padding",
|
||||
"awaiting_review", "in_review", "review_complete",
|
||||
"awaiting_instrumental_selection", "instrumental_selected",
|
||||
"generating_video", "encoding", "packaging",
|
||||
"complete", "failed"
|
||||
]
|
||||
|
||||
actual_statuses = [status.value for status in JobStatus]
|
||||
|
||||
for status in critical_statuses:
|
||||
assert status in actual_statuses, f"Missing critical status: {status}"
|
||||
|
||||
|
||||
class TestTimelineEvent:
|
||||
"""Test TimelineEvent model."""
|
||||
|
||||
def test_create_timeline_event(self):
|
||||
"""Test creating a timeline event."""
|
||||
event = TimelineEvent(
|
||||
status="pending",
|
||||
timestamp="2025-12-01T08:00:00Z",
|
||||
progress=0,
|
||||
message="Job created"
|
||||
)
|
||||
|
||||
assert event.status == "pending"
|
||||
assert event.timestamp == "2025-12-01T08:00:00Z"
|
||||
assert event.progress == 0
|
||||
assert event.message == "Job created"
|
||||
|
||||
def test_timeline_event_optional_fields(self):
|
||||
"""Test that progress and message are optional."""
|
||||
event = TimelineEvent(
|
||||
status="pending",
|
||||
timestamp="2025-12-01T08:00:00Z"
|
||||
)
|
||||
|
||||
assert event.progress is None
|
||||
assert event.message is None
|
||||
|
||||
|
||||
class TestModelValidation:
|
||||
"""Test model validation rules."""
|
||||
|
||||
def test_invalid_job_status(self):
|
||||
"""Test that invalid status is rejected."""
|
||||
with pytest.raises((ValidationError, ValueError)):
|
||||
Job(
|
||||
job_id="test123",
|
||||
status="invalid_status", # Not a valid JobStatus
|
||||
created_at=datetime.now(UTC),
|
||||
updated_at=datetime.now(UTC)
|
||||
)
|
||||
|
||||
def test_missing_required_fields(self):
|
||||
"""Test that missing required fields are rejected."""
|
||||
with pytest.raises(ValidationError):
|
||||
Job(
|
||||
job_id="test123"
|
||||
# Missing required fields: status, created_at, updated_at
|
||||
)
|
||||
|
||||
|
||||
class TestFileUrlsNestedStructure:
|
||||
"""
|
||||
Test that file_urls field correctly handles nested dictionaries.
|
||||
|
||||
THIS IS THE TEST THAT WOULD HAVE CAUGHT THE Dict[str, str] vs Dict[str, Any] BUG!
|
||||
|
||||
Workers store nested structures like:
|
||||
{
|
||||
"stems": {"instrumental_clean": "gs://...", "vocals": "gs://..."},
|
||||
"lyrics": {"corrections": "gs://...", "lrc": "gs://..."},
|
||||
...
|
||||
}
|
||||
|
||||
If file_urls is typed as Dict[str, str], this will fail on deserialization.
|
||||
"""
|
||||
|
||||
def test_file_urls_with_nested_stems(self):
|
||||
"""Test that file_urls accepts nested stems dictionary."""
|
||||
job = Job(
|
||||
job_id="test123",
|
||||
status=JobStatus.AUDIO_COMPLETE,
|
||||
created_at=datetime.now(UTC),
|
||||
updated_at=datetime.now(UTC),
|
||||
file_urls={
|
||||
"stems": {
|
||||
"instrumental_clean": "gs://bucket/jobs/test123/stems/instrumental_clean.flac",
|
||||
"instrumental_with_backing": "gs://bucket/jobs/test123/stems/instrumental_with_backing.flac",
|
||||
"vocals_clean": "gs://bucket/jobs/test123/stems/vocals_clean.flac",
|
||||
"lead_vocals": "gs://bucket/jobs/test123/stems/lead_vocals.flac",
|
||||
"backing_vocals": "gs://bucket/jobs/test123/stems/backing_vocals.flac",
|
||||
"bass": "gs://bucket/jobs/test123/stems/bass.flac",
|
||||
"drums": "gs://bucket/jobs/test123/stems/drums.flac",
|
||||
}
|
||||
}
|
||||
)
|
||||
|
||||
assert job.file_urls["stems"]["instrumental_clean"] == "gs://bucket/jobs/test123/stems/instrumental_clean.flac"
|
||||
assert len(job.file_urls["stems"]) == 7
|
||||
|
||||
def test_file_urls_with_nested_lyrics(self):
|
||||
"""Test that file_urls accepts nested lyrics dictionary."""
|
||||
job = Job(
|
||||
job_id="test123",
|
||||
status=JobStatus.LYRICS_COMPLETE,
|
||||
created_at=datetime.now(UTC),
|
||||
updated_at=datetime.now(UTC),
|
||||
file_urls={
|
||||
"lyrics": {
|
||||
"corrections": "gs://bucket/jobs/test123/lyrics/corrections.json",
|
||||
"audio": "gs://bucket/jobs/test123/lyrics/audio.flac",
|
||||
"lrc": "gs://bucket/jobs/test123/lyrics/karaoke.lrc",
|
||||
"uncorrected": "gs://bucket/jobs/test123/lyrics/uncorrected.txt",
|
||||
}
|
||||
}
|
||||
)
|
||||
|
||||
assert job.file_urls["lyrics"]["corrections"] == "gs://bucket/jobs/test123/lyrics/corrections.json"
|
||||
|
||||
def test_file_urls_full_worker_structure(self):
|
||||
"""
|
||||
Test with the full structure that workers actually create.
|
||||
|
||||
This simulates what actually gets stored in Firestore after
|
||||
all workers have run.
|
||||
"""
|
||||
full_file_urls = {
|
||||
"input": {
|
||||
"audio": "gs://bucket/jobs/test123/input/waterloo.flac"
|
||||
},
|
||||
"stems": {
|
||||
"instrumental_clean": "gs://bucket/jobs/test123/stems/instrumental_clean.flac",
|
||||
"instrumental_with_backing": "gs://bucket/jobs/test123/stems/instrumental_with_backing.flac",
|
||||
"vocals_clean": "gs://bucket/jobs/test123/stems/vocals_clean.flac",
|
||||
"lead_vocals": "gs://bucket/jobs/test123/stems/lead_vocals.flac",
|
||||
"backing_vocals": "gs://bucket/jobs/test123/stems/backing_vocals.flac",
|
||||
},
|
||||
"lyrics": {
|
||||
"corrections": "gs://bucket/jobs/test123/lyrics/corrections.json",
|
||||
"lrc": "gs://bucket/jobs/test123/lyrics/karaoke.lrc",
|
||||
},
|
||||
"screens": {
|
||||
"title": "gs://bucket/jobs/test123/screens/title.mov",
|
||||
"end": "gs://bucket/jobs/test123/screens/end.mov",
|
||||
},
|
||||
"videos": {
|
||||
"with_vocals": "gs://bucket/jobs/test123/videos/with_vocals.mkv",
|
||||
},
|
||||
"finals": {
|
||||
"lossless_4k_mp4": "gs://bucket/jobs/test123/finals/lossless_4k.mp4",
|
||||
"lossy_720p_mp4": "gs://bucket/jobs/test123/finals/lossy_720p.mp4",
|
||||
}
|
||||
}
|
||||
|
||||
job = Job(
|
||||
job_id="test123",
|
||||
status=JobStatus.COMPLETE,
|
||||
created_at=datetime.now(UTC),
|
||||
updated_at=datetime.now(UTC),
|
||||
file_urls=full_file_urls
|
||||
)
|
||||
|
||||
# Verify nested access works
|
||||
assert job.file_urls["stems"]["instrumental_clean"].endswith("instrumental_clean.flac")
|
||||
assert job.file_urls["lyrics"]["corrections"].endswith("corrections.json")
|
||||
assert job.file_urls["finals"]["lossy_720p_mp4"].endswith("lossy_720p.mp4")
|
||||
|
||||
def test_file_urls_roundtrip_serialization(self):
|
||||
"""
|
||||
Test that file_urls survives serialization/deserialization.
|
||||
|
||||
This simulates what happens when we:
|
||||
1. Create a Job with nested file_urls
|
||||
2. Serialize to dict (for Firestore)
|
||||
3. Deserialize back to Job (reading from Firestore)
|
||||
|
||||
THIS IS THE KEY TEST - it would catch the Dict[str, str] bug!
|
||||
"""
|
||||
original_file_urls = {
|
||||
"stems": {
|
||||
"instrumental_clean": "gs://bucket/stems/clean.flac",
|
||||
"vocals": "gs://bucket/stems/vocals.flac",
|
||||
},
|
||||
"lyrics": {
|
||||
"corrections": "gs://bucket/lyrics/corrections.json",
|
||||
}
|
||||
}
|
||||
|
||||
# Create job
|
||||
job = Job(
|
||||
job_id="test123",
|
||||
status=JobStatus.PENDING,
|
||||
created_at=datetime.now(UTC),
|
||||
updated_at=datetime.now(UTC),
|
||||
file_urls=original_file_urls
|
||||
)
|
||||
|
||||
# Serialize to dict (simulates writing to Firestore)
|
||||
job_dict = job.model_dump()
|
||||
|
||||
# Convert to JSON and back (simulates Firestore storage)
|
||||
json_str = json.dumps(job_dict, default=str)
|
||||
restored_dict = json.loads(json_str)
|
||||
|
||||
# Deserialize back to Job (simulates reading from Firestore)
|
||||
restored_job = Job(**restored_dict)
|
||||
|
||||
# Verify nested structure survived
|
||||
assert restored_job.file_urls["stems"]["instrumental_clean"] == "gs://bucket/stems/clean.flac"
|
||||
assert restored_job.file_urls["stems"]["vocals"] == "gs://bucket/stems/vocals.flac"
|
||||
assert restored_job.file_urls["lyrics"]["corrections"] == "gs://bucket/lyrics/corrections.json"
|
||||
|
||||
|
||||
class TestStateDataNestedStructure:
|
||||
"""Test that state_data field correctly handles nested dictionaries."""
|
||||
|
||||
def test_state_data_with_instrumental_options(self):
|
||||
"""Test state_data with instrumental options structure."""
|
||||
job = Job(
|
||||
job_id="test123",
|
||||
status=JobStatus.AWAITING_INSTRUMENTAL_SELECTION,
|
||||
created_at=datetime.now(UTC),
|
||||
updated_at=datetime.now(UTC),
|
||||
state_data={
|
||||
"instrumental_options": {
|
||||
"clean": "gs://bucket/jobs/test123/stems/instrumental_clean.flac",
|
||||
"with_backing": "gs://bucket/jobs/test123/stems/instrumental_with_backing.flac",
|
||||
},
|
||||
"audio_complete": True,
|
||||
"lyrics_complete": True,
|
||||
}
|
||||
)
|
||||
|
||||
assert job.state_data["instrumental_options"]["clean"].endswith("clean.flac")
|
||||
assert job.state_data["audio_complete"] is True
|
||||
|
||||
def test_state_data_with_lyrics_metadata(self):
|
||||
"""Test state_data with lyrics metadata structure."""
|
||||
job = Job(
|
||||
job_id="test123",
|
||||
status=JobStatus.LYRICS_COMPLETE,
|
||||
created_at=datetime.now(UTC),
|
||||
updated_at=datetime.now(UTC),
|
||||
state_data={
|
||||
"lyrics_metadata": {
|
||||
"line_count": 42,
|
||||
"has_corrections": True,
|
||||
"ready_for_review": True,
|
||||
},
|
||||
"corrected_lyrics": {
|
||||
"lines": [
|
||||
{"text": "Hello world", "start_time": 1.0, "end_time": 2.0}
|
||||
],
|
||||
"metadata": {"song": "Test Song"}
|
||||
}
|
||||
}
|
||||
)
|
||||
|
||||
assert job.state_data["lyrics_metadata"]["line_count"] == 42
|
||||
assert len(job.state_data["corrected_lyrics"]["lines"]) == 1
|
||||
|
||||
def test_state_data_roundtrip_serialization(self):
|
||||
"""Test that state_data survives serialization/deserialization."""
|
||||
original_state_data = {
|
||||
"instrumental_selection": "clean",
|
||||
"review_notes": "Looks good!",
|
||||
"complex_nested": {
|
||||
"level1": {
|
||||
"level2": {
|
||||
"value": 123
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
job = Job(
|
||||
job_id="test123",
|
||||
status=JobStatus.PENDING,
|
||||
created_at=datetime.now(UTC),
|
||||
updated_at=datetime.now(UTC),
|
||||
state_data=original_state_data
|
||||
)
|
||||
|
||||
# Serialize and deserialize
|
||||
job_dict = job.model_dump()
|
||||
json_str = json.dumps(job_dict, default=str)
|
||||
restored_dict = json.loads(json_str)
|
||||
restored_job = Job(**restored_dict)
|
||||
|
||||
# Verify nested structure survived
|
||||
assert restored_job.state_data["complex_nested"]["level1"]["level2"]["value"] == 123
|
||||
|
||||
|
||||
class TestWorkerIdsNestedStructure:
|
||||
"""Test that worker_ids field correctly handles nested dictionaries."""
|
||||
|
||||
def test_worker_ids_with_all_workers(self):
|
||||
"""Test worker_ids with all worker types."""
|
||||
job = Job(
|
||||
job_id="test123",
|
||||
status=JobStatus.GENERATING_VIDEO,
|
||||
created_at=datetime.now(UTC),
|
||||
updated_at=datetime.now(UTC),
|
||||
worker_ids={
|
||||
"audio_worker": "cloud-run-request-12345",
|
||||
"lyrics_worker": "cloud-run-request-67890",
|
||||
"screens_worker": "cloud-run-request-11111",
|
||||
"video_worker": "cloud-run-request-22222",
|
||||
}
|
||||
)
|
||||
|
||||
assert job.worker_ids["audio_worker"] == "cloud-run-request-12345"
|
||||
assert len(job.worker_ids) == 4
|
||||
|
||||
|
||||
class TestJobModelIntegration:
|
||||
"""
|
||||
Integration tests that simulate real Firestore interactions.
|
||||
|
||||
These tests verify that the Job model works correctly when used
|
||||
the same way as our actual services.
|
||||
"""
|
||||
|
||||
def test_simulate_job_manager_update_file_url(self):
|
||||
"""
|
||||
Simulate what JobManager.update_file_url does.
|
||||
|
||||
This is the pattern that caused the bug - updating nested file_urls.
|
||||
"""
|
||||
# Start with a fresh job
|
||||
job = Job(
|
||||
job_id="test123",
|
||||
status=JobStatus.PENDING,
|
||||
created_at=datetime.now(UTC),
|
||||
updated_at=datetime.now(UTC),
|
||||
file_urls={}
|
||||
)
|
||||
|
||||
# Simulate update_file_url('test123', 'stems', 'instrumental_clean', 'gs://...')
|
||||
# This creates nested structure: {"stems": {"instrumental_clean": "gs://..."}}
|
||||
if "stems" not in job.file_urls:
|
||||
job.file_urls["stems"] = {}
|
||||
job.file_urls["stems"]["instrumental_clean"] = "gs://bucket/stems/clean.flac"
|
||||
|
||||
# Simulate Firestore roundtrip
|
||||
job_dict = job.model_dump()
|
||||
restored_job = Job(**job_dict)
|
||||
|
||||
# This is what was failing with Dict[str, str]!
|
||||
assert restored_job.file_urls["stems"]["instrumental_clean"] == "gs://bucket/stems/clean.flac"
|
||||
|
||||
def test_simulate_audio_worker_completion(self):
|
||||
"""Simulate the data structure created when audio worker completes."""
|
||||
# This is the actual structure created by audio_worker.py
|
||||
file_urls = {
|
||||
"stems": {
|
||||
"instrumental_clean": "gs://bucket/jobs/abc123/stems/instrumental_clean.flac",
|
||||
"vocals_clean": "gs://bucket/jobs/abc123/stems/vocals_clean.flac",
|
||||
"lead_vocals": "gs://bucket/jobs/abc123/stems/lead_vocals.flac",
|
||||
"backing_vocals": "gs://bucket/jobs/abc123/stems/backing_vocals.flac",
|
||||
"instrumental_with_backing": "gs://bucket/jobs/abc123/stems/instrumental_with_backing.flac",
|
||||
}
|
||||
}
|
||||
|
||||
state_data = {
|
||||
"instrumental_options": {
|
||||
"clean": "jobs/abc123/stems/instrumental_clean.flac",
|
||||
"with_backing": "jobs/abc123/stems/instrumental_with_backing.flac",
|
||||
}
|
||||
}
|
||||
|
||||
job = Job(
|
||||
job_id="abc123",
|
||||
status=JobStatus.AUDIO_COMPLETE,
|
||||
created_at=datetime.now(UTC),
|
||||
updated_at=datetime.now(UTC),
|
||||
artist="ABBA",
|
||||
title="Waterloo",
|
||||
file_urls=file_urls,
|
||||
state_data=state_data
|
||||
)
|
||||
|
||||
# Roundtrip
|
||||
restored_job = Job(**job.model_dump())
|
||||
|
||||
assert len(restored_job.file_urls["stems"]) == 5
|
||||
assert "instrumental_options" in restored_job.state_data
|
||||
|
||||
def test_simulate_lyrics_worker_completion(self):
|
||||
"""Simulate the data structure created when lyrics worker completes."""
|
||||
file_urls = {
|
||||
"lyrics": {
|
||||
"corrections": "gs://bucket/jobs/abc123/lyrics/corrections.json",
|
||||
"lrc": "gs://bucket/jobs/abc123/lyrics/karaoke.lrc",
|
||||
"uncorrected": "gs://bucket/jobs/abc123/lyrics/uncorrected.txt",
|
||||
}
|
||||
}
|
||||
|
||||
state_data = {
|
||||
"lyrics_metadata": {
|
||||
"line_count": 50,
|
||||
"has_corrections": True,
|
||||
"ready_for_review": True,
|
||||
}
|
||||
}
|
||||
|
||||
job = Job(
|
||||
job_id="abc123",
|
||||
status=JobStatus.LYRICS_COMPLETE,
|
||||
created_at=datetime.now(UTC),
|
||||
updated_at=datetime.now(UTC),
|
||||
file_urls=file_urls,
|
||||
state_data=state_data
|
||||
)
|
||||
|
||||
# Roundtrip
|
||||
restored_job = Job(**job.model_dump())
|
||||
|
||||
assert restored_job.file_urls["lyrics"]["corrections"].endswith("corrections.json")
|
||||
assert restored_job.state_data["lyrics_metadata"]["line_count"] == 50
|
||||
|
||||
|
||||
class TestExistingInstrumentalModelFields:
|
||||
"""Test existing instrumental field in Job and JobCreate models (Batch 3)."""
|
||||
|
||||
def test_job_has_existing_instrumental_gcs_path(self):
|
||||
"""Test that Job model has existing_instrumental_gcs_path field."""
|
||||
job = Job(
|
||||
job_id="test123",
|
||||
status=JobStatus.PENDING,
|
||||
created_at=datetime.now(UTC),
|
||||
updated_at=datetime.now(UTC),
|
||||
existing_instrumental_gcs_path="uploads/test123/audio/existing_instrumental.flac"
|
||||
)
|
||||
|
||||
assert hasattr(job, 'existing_instrumental_gcs_path')
|
||||
assert job.existing_instrumental_gcs_path == "uploads/test123/audio/existing_instrumental.flac"
|
||||
|
||||
def test_job_existing_instrumental_defaults_to_none(self):
|
||||
"""Test that existing_instrumental_gcs_path defaults to None."""
|
||||
job = Job(
|
||||
job_id="test123",
|
||||
status=JobStatus.PENDING,
|
||||
created_at=datetime.now(UTC),
|
||||
updated_at=datetime.now(UTC)
|
||||
)
|
||||
|
||||
assert job.existing_instrumental_gcs_path is None
|
||||
|
||||
def test_job_create_has_existing_instrumental_gcs_path(self):
|
||||
"""Test that JobCreate model has existing_instrumental_gcs_path field."""
|
||||
from backend.models.job import JobCreate
|
||||
|
||||
job_create = JobCreate(
|
||||
artist="Artist",
|
||||
title="Title",
|
||||
existing_instrumental_gcs_path="uploads/test/audio/existing_instrumental.mp3"
|
||||
)
|
||||
|
||||
assert job_create.existing_instrumental_gcs_path == "uploads/test/audio/existing_instrumental.mp3"
|
||||
|
||||
def test_job_create_existing_instrumental_optional(self):
|
||||
"""Test that existing_instrumental_gcs_path is optional in JobCreate."""
|
||||
from backend.models.job import JobCreate
|
||||
|
||||
job_create = JobCreate(
|
||||
artist="Artist",
|
||||
title="Title"
|
||||
)
|
||||
|
||||
assert job_create.existing_instrumental_gcs_path is None
|
||||
|
||||
def test_existing_instrumental_roundtrip_serialization(self):
|
||||
"""Test that existing_instrumental_gcs_path survives serialization/deserialization."""
|
||||
job = Job(
|
||||
job_id="test123",
|
||||
status=JobStatus.PENDING,
|
||||
created_at=datetime.now(UTC),
|
||||
updated_at=datetime.now(UTC),
|
||||
existing_instrumental_gcs_path="uploads/test123/audio/existing_instrumental.wav"
|
||||
)
|
||||
|
||||
# Serialize to dict (simulates writing to Firestore)
|
||||
job_dict = job.model_dump()
|
||||
|
||||
# Convert to JSON and back (simulates Firestore storage)
|
||||
json_str = json.dumps(job_dict, default=str)
|
||||
restored_dict = json.loads(json_str)
|
||||
|
||||
# Deserialize back to Job (simulates reading from Firestore)
|
||||
restored_job = Job(**restored_dict)
|
||||
|
||||
assert restored_job.existing_instrumental_gcs_path == "uploads/test123/audio/existing_instrumental.wav"
|
||||
|
||||
def test_job_with_full_audio_config(self):
|
||||
"""Test Job with complete audio configuration including existing instrumental."""
|
||||
job = Job(
|
||||
job_id="test123",
|
||||
status=JobStatus.AUDIO_COMPLETE,
|
||||
created_at=datetime.now(UTC),
|
||||
updated_at=datetime.now(UTC),
|
||||
artist="Test Artist",
|
||||
title="Test Song",
|
||||
# Audio model configuration
|
||||
clean_instrumental_model="model_bs_roformer_ep_317_sdr_12.9755.ckpt",
|
||||
backing_vocals_models=["mel_band_roformer_karaoke.ckpt"],
|
||||
other_stems_models=["htdemucs_6s.yaml"],
|
||||
# Existing instrumental
|
||||
existing_instrumental_gcs_path="uploads/test123/audio/my_instrumental.flac",
|
||||
)
|
||||
|
||||
assert job.clean_instrumental_model == "model_bs_roformer_ep_317_sdr_12.9755.ckpt"
|
||||
assert job.existing_instrumental_gcs_path == "uploads/test123/audio/my_instrumental.flac"
|
||||
|
||||
|
||||
class TestTwoPhaseWorkflowFields:
|
||||
"""Test two-phase workflow fields in Job and JobCreate models (Batch 6)."""
|
||||
|
||||
def test_job_has_prep_only_field(self):
|
||||
"""Test that Job model has prep_only field."""
|
||||
job = Job(
|
||||
job_id="test123",
|
||||
status=JobStatus.PENDING,
|
||||
created_at=datetime.now(UTC),
|
||||
updated_at=datetime.now(UTC),
|
||||
prep_only=True
|
||||
)
|
||||
|
||||
assert hasattr(job, 'prep_only')
|
||||
assert job.prep_only is True
|
||||
|
||||
def test_job_prep_only_defaults_to_false(self):
|
||||
"""Test that prep_only defaults to False."""
|
||||
job = Job(
|
||||
job_id="test123",
|
||||
status=JobStatus.PENDING,
|
||||
created_at=datetime.now(UTC),
|
||||
updated_at=datetime.now(UTC)
|
||||
)
|
||||
|
||||
assert job.prep_only is False
|
||||
|
||||
def test_job_has_finalise_only_field(self):
|
||||
"""Test that Job model has finalise_only field."""
|
||||
job = Job(
|
||||
job_id="test123",
|
||||
status=JobStatus.PENDING,
|
||||
created_at=datetime.now(UTC),
|
||||
updated_at=datetime.now(UTC),
|
||||
finalise_only=True
|
||||
)
|
||||
|
||||
assert hasattr(job, 'finalise_only')
|
||||
assert job.finalise_only is True
|
||||
|
||||
def test_job_finalise_only_defaults_to_false(self):
|
||||
"""Test that finalise_only defaults to False."""
|
||||
job = Job(
|
||||
job_id="test123",
|
||||
status=JobStatus.PENDING,
|
||||
created_at=datetime.now(UTC),
|
||||
updated_at=datetime.now(UTC)
|
||||
)
|
||||
|
||||
assert job.finalise_only is False
|
||||
|
||||
def test_job_has_keep_brand_code_field(self):
|
||||
"""Test that Job model has keep_brand_code field."""
|
||||
job = Job(
|
||||
job_id="test123",
|
||||
status=JobStatus.PENDING,
|
||||
created_at=datetime.now(UTC),
|
||||
updated_at=datetime.now(UTC),
|
||||
keep_brand_code="NOMAD-1234"
|
||||
)
|
||||
|
||||
assert hasattr(job, 'keep_brand_code')
|
||||
assert job.keep_brand_code == "NOMAD-1234"
|
||||
|
||||
def test_job_keep_brand_code_defaults_to_none(self):
|
||||
"""Test that keep_brand_code defaults to None."""
|
||||
job = Job(
|
||||
job_id="test123",
|
||||
status=JobStatus.PENDING,
|
||||
created_at=datetime.now(UTC),
|
||||
updated_at=datetime.now(UTC)
|
||||
)
|
||||
|
||||
assert job.keep_brand_code is None
|
||||
|
||||
def test_job_create_has_two_phase_workflow_fields(self):
|
||||
"""Test that JobCreate model has two-phase workflow fields."""
|
||||
job_create = JobCreate(
|
||||
artist="Artist",
|
||||
title="Title",
|
||||
prep_only=True,
|
||||
finalise_only=False,
|
||||
keep_brand_code="BRAND-0001"
|
||||
)
|
||||
|
||||
assert job_create.prep_only is True
|
||||
assert job_create.finalise_only is False
|
||||
assert job_create.keep_brand_code == "BRAND-0001"
|
||||
|
||||
def test_two_phase_workflow_roundtrip_serialization(self):
|
||||
"""Test that two-phase workflow fields survive serialization/deserialization."""
|
||||
job = Job(
|
||||
job_id="test123",
|
||||
status=JobStatus.PENDING,
|
||||
created_at=datetime.now(UTC),
|
||||
updated_at=datetime.now(UTC),
|
||||
prep_only=True,
|
||||
finalise_only=False,
|
||||
keep_brand_code="NOMAD-5678"
|
||||
)
|
||||
|
||||
# Serialize to dict (simulates writing to Firestore)
|
||||
job_dict = job.model_dump()
|
||||
|
||||
# Convert to JSON and back (simulates Firestore storage)
|
||||
json_str = json.dumps(job_dict, default=str)
|
||||
restored_dict = json.loads(json_str)
|
||||
|
||||
# Deserialize back to Job (simulates reading from Firestore)
|
||||
restored_job = Job(**restored_dict)
|
||||
|
||||
assert restored_job.prep_only is True
|
||||
assert restored_job.finalise_only is False
|
||||
assert restored_job.keep_brand_code == "NOMAD-5678"
|
||||
|
||||
|
||||
class TestPrepCompleteStatus:
|
||||
"""Test PREP_COMPLETE status and state transitions (Batch 6)."""
|
||||
|
||||
def test_prep_complete_status_exists(self):
|
||||
"""Test that PREP_COMPLETE status exists in JobStatus enum."""
|
||||
assert hasattr(JobStatus, 'PREP_COMPLETE')
|
||||
assert JobStatus.PREP_COMPLETE.value == "prep_complete"
|
||||
|
||||
def test_job_can_have_prep_complete_status(self):
|
||||
"""Test that Job can be created with PREP_COMPLETE status."""
|
||||
job = Job(
|
||||
job_id="test123",
|
||||
status=JobStatus.PREP_COMPLETE,
|
||||
created_at=datetime.now(UTC),
|
||||
updated_at=datetime.now(UTC)
|
||||
)
|
||||
|
||||
assert job.status == JobStatus.PREP_COMPLETE
|
||||
|
||||
def test_state_transitions_review_complete_to_prep_complete(self):
|
||||
"""Test that REVIEW_COMPLETE can transition to PREP_COMPLETE."""
|
||||
from backend.models.job import STATE_TRANSITIONS
|
||||
|
||||
valid_transitions = STATE_TRANSITIONS.get(JobStatus.REVIEW_COMPLETE, [])
|
||||
assert JobStatus.PREP_COMPLETE in valid_transitions
|
||||
|
||||
def test_state_transitions_prep_complete_to_awaiting_instrumental(self):
|
||||
"""Test that PREP_COMPLETE can transition to AWAITING_INSTRUMENTAL_SELECTION."""
|
||||
from backend.models.job import STATE_TRANSITIONS
|
||||
|
||||
valid_transitions = STATE_TRANSITIONS.get(JobStatus.PREP_COMPLETE, [])
|
||||
assert JobStatus.AWAITING_INSTRUMENTAL_SELECTION in valid_transitions
|
||||
|
||||
def test_state_transitions_pending_to_awaiting_instrumental(self):
|
||||
"""Test that PENDING can transition to AWAITING_INSTRUMENTAL_SELECTION (for finalise-only)."""
|
||||
from backend.models.job import STATE_TRANSITIONS
|
||||
|
||||
valid_transitions = STATE_TRANSITIONS.get(JobStatus.PENDING, [])
|
||||
assert JobStatus.AWAITING_INSTRUMENTAL_SELECTION in valid_transitions
|
||||
|
||||
def test_critical_statuses_include_prep_complete(self):
|
||||
"""Test that PREP_COMPLETE is in the list of job statuses."""
|
||||
all_statuses = [status.value for status in JobStatus]
|
||||
assert "prep_complete" in all_statuses
|
||||
|
||||
|
||||
class TestJobWithFullTwoPhaseConfig:
|
||||
"""Integration tests for two-phase workflow job configuration."""
|
||||
|
||||
def test_prep_only_job_full_config(self):
|
||||
"""Test creating a prep-only job with full configuration."""
|
||||
job = Job(
|
||||
job_id="prep123",
|
||||
status=JobStatus.PENDING,
|
||||
created_at=datetime.now(UTC),
|
||||
updated_at=datetime.now(UTC),
|
||||
artist="Test Artist",
|
||||
title="Test Song",
|
||||
prep_only=True,
|
||||
enable_cdg=True,
|
||||
enable_txt=True,
|
||||
brand_prefix="NOMAD",
|
||||
)
|
||||
|
||||
assert job.prep_only is True
|
||||
assert job.finalise_only is False
|
||||
assert job.brand_prefix == "NOMAD"
|
||||
|
||||
def test_finalise_only_job_full_config(self):
|
||||
"""Test creating a finalise-only job with full configuration."""
|
||||
job = Job(
|
||||
job_id="final123",
|
||||
status=JobStatus.AWAITING_INSTRUMENTAL_SELECTION,
|
||||
created_at=datetime.now(UTC),
|
||||
updated_at=datetime.now(UTC),
|
||||
artist="Test Artist",
|
||||
title="Test Song",
|
||||
finalise_only=True,
|
||||
keep_brand_code="NOMAD-1234",
|
||||
enable_youtube_upload=True,
|
||||
dropbox_path="/Karaoke/Tracks",
|
||||
state_data={
|
||||
"audio_complete": True,
|
||||
"lyrics_complete": True,
|
||||
"finalise_only": True,
|
||||
}
|
||||
)
|
||||
|
||||
assert job.finalise_only is True
|
||||
assert job.keep_brand_code == "NOMAD-1234"
|
||||
assert job.state_data["finalise_only"] is True
|
||||
|
||||
def test_prep_complete_job_with_file_urls(self):
|
||||
"""Test a job that has reached PREP_COMPLETE status with all prep outputs."""
|
||||
job = Job(
|
||||
job_id="prepcomp123",
|
||||
status=JobStatus.PREP_COMPLETE,
|
||||
created_at=datetime.now(UTC),
|
||||
updated_at=datetime.now(UTC),
|
||||
artist="ABBA",
|
||||
title="Waterloo",
|
||||
prep_only=True,
|
||||
file_urls={
|
||||
"stems": {
|
||||
"instrumental_clean": "gs://bucket/jobs/prepcomp123/stems/instrumental_clean.flac",
|
||||
"instrumental_with_backing": "gs://bucket/jobs/prepcomp123/stems/instrumental_with_backing.flac",
|
||||
},
|
||||
"videos": {
|
||||
"with_vocals": "gs://bucket/jobs/prepcomp123/videos/with_vocals.mkv",
|
||||
},
|
||||
"screens": {
|
||||
"title": "gs://bucket/jobs/prepcomp123/screens/title.mov",
|
||||
"end": "gs://bucket/jobs/prepcomp123/screens/end.mov",
|
||||
},
|
||||
"lyrics": {
|
||||
"lrc": "gs://bucket/jobs/prepcomp123/lyrics/karaoke.lrc",
|
||||
"ass": "gs://bucket/jobs/prepcomp123/lyrics/karaoke.ass",
|
||||
},
|
||||
}
|
||||
)
|
||||
|
||||
assert job.status == JobStatus.PREP_COMPLETE
|
||||
assert job.prep_only is True
|
||||
assert "with_vocals" in job.file_urls["videos"]
|
||||
assert "title" in job.file_urls["screens"]
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
pytest.main([__file__, "-v"])
|
||||
|
||||
@@ -1,200 +0,0 @@
|
||||
"""
|
||||
Unit tests for request models.
|
||||
|
||||
These tests validate Pydantic request model validation and serialization.
|
||||
"""
|
||||
import pytest
|
||||
from pydantic import ValidationError
|
||||
|
||||
from backend.models.requests import (
|
||||
URLSubmissionRequest,
|
||||
UploadSubmissionRequest,
|
||||
CorrectionsSubmission,
|
||||
InstrumentalSelection,
|
||||
StartReviewRequest,
|
||||
CancelJobRequest,
|
||||
RetryJobRequest
|
||||
)
|
||||
|
||||
|
||||
class TestURLSubmissionRequest:
|
||||
"""Tests for URLSubmissionRequest model."""
|
||||
|
||||
def test_valid_youtube_url(self):
|
||||
"""Test valid YouTube URL is accepted."""
|
||||
request = URLSubmissionRequest(url="https://youtube.com/watch?v=test123")
|
||||
assert str(request.url) == "https://youtube.com/watch?v=test123"
|
||||
|
||||
def test_valid_youtube_short_url(self):
|
||||
"""Test valid YouTube short URL is accepted."""
|
||||
request = URLSubmissionRequest(url="https://youtu.be/test123")
|
||||
assert str(request.url) == "https://youtu.be/test123"
|
||||
|
||||
def test_optional_artist_title(self):
|
||||
"""Test artist and title are optional."""
|
||||
request = URLSubmissionRequest(url="https://youtube.com/watch?v=test")
|
||||
assert request.artist is None
|
||||
assert request.title is None
|
||||
|
||||
def test_with_artist_title(self):
|
||||
"""Test request with artist and title."""
|
||||
request = URLSubmissionRequest(
|
||||
url="https://youtube.com/watch?v=test",
|
||||
artist="Test Artist",
|
||||
title="Test Song"
|
||||
)
|
||||
assert request.artist == "Test Artist"
|
||||
assert request.title == "Test Song"
|
||||
|
||||
def test_url_required(self):
|
||||
"""Test URL is required."""
|
||||
with pytest.raises(ValidationError):
|
||||
URLSubmissionRequest()
|
||||
|
||||
def test_empty_url_rejected(self):
|
||||
"""Test empty URL is rejected."""
|
||||
with pytest.raises(ValidationError):
|
||||
URLSubmissionRequest(url="")
|
||||
|
||||
|
||||
class TestCorrectionsSubmission:
|
||||
"""Tests for CorrectionsSubmission model."""
|
||||
|
||||
def test_valid_corrections(self):
|
||||
"""Test valid corrections submission."""
|
||||
corrections = CorrectionsSubmission(
|
||||
corrections={
|
||||
"lines": [
|
||||
{"text": "Hello world", "start": 0.0, "end": 1.0}
|
||||
],
|
||||
"metadata": {"source": "test"}
|
||||
}
|
||||
)
|
||||
assert "lines" in corrections.corrections
|
||||
assert "metadata" in corrections.corrections
|
||||
|
||||
def test_missing_lines_rejected(self):
|
||||
"""Test corrections without 'lines' field is rejected."""
|
||||
with pytest.raises(ValidationError):
|
||||
CorrectionsSubmission(corrections={"metadata": {}})
|
||||
|
||||
def test_missing_metadata_rejected(self):
|
||||
"""Test corrections without 'metadata' field is rejected."""
|
||||
with pytest.raises(ValidationError):
|
||||
CorrectionsSubmission(corrections={"lines": []})
|
||||
|
||||
def test_corrections_required(self):
|
||||
"""Test corrections field is required."""
|
||||
with pytest.raises(ValidationError):
|
||||
CorrectionsSubmission()
|
||||
|
||||
def test_with_user_notes(self):
|
||||
"""Test corrections with user notes."""
|
||||
corrections = CorrectionsSubmission(
|
||||
corrections={"lines": [], "metadata": {}},
|
||||
user_notes="Fixed typo in line 3"
|
||||
)
|
||||
assert corrections.user_notes == "Fixed typo in line 3"
|
||||
|
||||
|
||||
class TestInstrumentalSelection:
|
||||
"""Tests for InstrumentalSelection model."""
|
||||
|
||||
def test_valid_clean_selection(self):
|
||||
"""Test selecting clean instrumental."""
|
||||
selection = InstrumentalSelection(selection="clean")
|
||||
assert selection.selection == "clean"
|
||||
|
||||
def test_valid_with_backing_selection(self):
|
||||
"""Test selecting instrumental with backing vocals."""
|
||||
selection = InstrumentalSelection(selection="with_backing")
|
||||
assert selection.selection == "with_backing"
|
||||
|
||||
def test_selection_required(self):
|
||||
"""Test selection is required."""
|
||||
with pytest.raises(ValidationError):
|
||||
InstrumentalSelection()
|
||||
|
||||
def test_invalid_selection_rejected(self):
|
||||
"""Test invalid selection values are rejected."""
|
||||
with pytest.raises(ValidationError):
|
||||
InstrumentalSelection(selection="invalid_option")
|
||||
|
||||
|
||||
class TestUploadSubmissionRequest:
|
||||
"""Tests for UploadSubmissionRequest model."""
|
||||
|
||||
def test_valid_upload_request(self):
|
||||
"""Test valid upload submission."""
|
||||
request = UploadSubmissionRequest(
|
||||
artist="Test Artist",
|
||||
title="Test Song"
|
||||
)
|
||||
assert request.artist == "Test Artist"
|
||||
assert request.title == "Test Song"
|
||||
|
||||
def test_artist_required(self):
|
||||
"""Test artist is required."""
|
||||
with pytest.raises(ValidationError):
|
||||
UploadSubmissionRequest(title="Test Song")
|
||||
|
||||
def test_title_required(self):
|
||||
"""Test title is required."""
|
||||
with pytest.raises(ValidationError):
|
||||
UploadSubmissionRequest(artist="Test Artist")
|
||||
|
||||
def test_default_options(self):
|
||||
"""Test default option values."""
|
||||
request = UploadSubmissionRequest(artist="Test", title="Test")
|
||||
# CDG/TXT disabled by default (requires style config)
|
||||
assert request.enable_cdg is False
|
||||
assert request.enable_txt is False
|
||||
assert request.enable_youtube_upload is False
|
||||
|
||||
|
||||
class TestStartReviewRequest:
|
||||
"""Tests for StartReviewRequest model."""
|
||||
|
||||
def test_valid_request(self):
|
||||
"""Test valid start review request."""
|
||||
request = StartReviewRequest()
|
||||
assert request is not None
|
||||
|
||||
|
||||
class TestCancelJobRequest:
|
||||
"""Tests for CancelJobRequest model."""
|
||||
|
||||
def test_valid_request(self):
|
||||
"""Test valid cancel request."""
|
||||
request = CancelJobRequest()
|
||||
assert request is not None
|
||||
|
||||
def test_with_reason(self):
|
||||
"""Test cancel with reason."""
|
||||
request = CancelJobRequest(reason="User requested")
|
||||
assert request.reason == "User requested"
|
||||
|
||||
def test_reason_optional(self):
|
||||
"""Test reason is optional."""
|
||||
request = CancelJobRequest()
|
||||
assert request.reason is None
|
||||
|
||||
|
||||
class TestRetryJobRequest:
|
||||
"""Tests for RetryJobRequest model."""
|
||||
|
||||
def test_valid_request(self):
|
||||
"""Test valid retry request."""
|
||||
request = RetryJobRequest()
|
||||
assert request is not None
|
||||
|
||||
def test_with_from_stage(self):
|
||||
"""Test retry from specific stage."""
|
||||
request = RetryJobRequest(from_stage="transcription")
|
||||
assert request.from_stage == "transcription"
|
||||
|
||||
def test_from_stage_optional(self):
|
||||
"""Test from_stage is optional."""
|
||||
request = RetryJobRequest()
|
||||
assert request.from_stage is None
|
||||
|
||||
@@ -1,282 +0,0 @@
|
||||
"""
|
||||
Unit tests for jobs.py routes.
|
||||
|
||||
These tests exercise the route logic with mocked services.
|
||||
"""
|
||||
import pytest
|
||||
from datetime import datetime, UTC
|
||||
from unittest.mock import MagicMock, AsyncMock, patch
|
||||
|
||||
from backend.models.job import Job, JobStatus
|
||||
|
||||
|
||||
class TestJobsRouteHelpers:
|
||||
"""Tests for helper functions in jobs.py routes."""
|
||||
|
||||
def test_jobs_router_structure(self):
|
||||
"""Test jobs router has expected structure."""
|
||||
from backend.api.routes.jobs import router
|
||||
assert router is not None
|
||||
|
||||
# Check that common route patterns exist
|
||||
route_paths = [route.path for route in router.routes]
|
||||
assert any('/jobs' in p or 'jobs' in str(p) for p in route_paths)
|
||||
|
||||
|
||||
class TestJobStatusTransitions:
|
||||
"""Tests for job status transition validation.
|
||||
|
||||
These test the Job model's state machine which is critical for
|
||||
preventing invalid operations.
|
||||
"""
|
||||
|
||||
def test_valid_pending_to_downloading(self):
|
||||
"""Test PENDING -> DOWNLOADING is valid."""
|
||||
from backend.models.job import STATE_TRANSITIONS
|
||||
valid_transitions = STATE_TRANSITIONS.get(JobStatus.PENDING, [])
|
||||
assert JobStatus.DOWNLOADING in valid_transitions
|
||||
|
||||
def test_valid_downloading_to_separating(self):
|
||||
"""Test DOWNLOADING -> SEPARATING_STAGE1 is valid."""
|
||||
from backend.models.job import STATE_TRANSITIONS
|
||||
valid_transitions = STATE_TRANSITIONS.get(JobStatus.DOWNLOADING, [])
|
||||
assert JobStatus.SEPARATING_STAGE1 in valid_transitions
|
||||
|
||||
def test_invalid_pending_to_complete(self):
|
||||
"""Test PENDING -> COMPLETE is invalid."""
|
||||
from backend.models.job import STATE_TRANSITIONS
|
||||
valid_transitions = STATE_TRANSITIONS.get(JobStatus.PENDING, [])
|
||||
assert JobStatus.COMPLETE not in valid_transitions
|
||||
|
||||
def test_any_active_status_can_fail(self):
|
||||
"""Test any active (non-terminal, non-legacy) status can transition to FAILED."""
|
||||
from backend.models.job import STATE_TRANSITIONS
|
||||
# Terminal states and legacy states are excluded
|
||||
excluded = [
|
||||
JobStatus.COMPLETE, JobStatus.FAILED, JobStatus.CANCELLED,
|
||||
# Legacy states that don't include FAILED
|
||||
JobStatus.QUEUED, JobStatus.PROCESSING,
|
||||
JobStatus.READY_FOR_FINALIZATION, JobStatus.FINALIZING, JobStatus.ERROR
|
||||
]
|
||||
for status in JobStatus:
|
||||
if status not in excluded:
|
||||
valid_transitions = STATE_TRANSITIONS.get(status, [])
|
||||
assert JobStatus.FAILED in valid_transitions, f"{status} should be able to fail"
|
||||
|
||||
def test_failed_can_transition_for_retry(self):
|
||||
"""Test FAILED status can transition to retry checkpoint states."""
|
||||
from backend.models.job import STATE_TRANSITIONS
|
||||
valid_transitions = STATE_TRANSITIONS.get(JobStatus.FAILED, [])
|
||||
|
||||
# FAILED should allow retry transitions
|
||||
assert JobStatus.INSTRUMENTAL_SELECTED in valid_transitions, "FAILED should allow retry to INSTRUMENTAL_SELECTED"
|
||||
assert JobStatus.REVIEW_COMPLETE in valid_transitions, "FAILED should allow retry to REVIEW_COMPLETE"
|
||||
assert JobStatus.LYRICS_COMPLETE in valid_transitions, "FAILED should allow retry to LYRICS_COMPLETE"
|
||||
|
||||
def test_cancelled_can_transition_for_retry(self):
|
||||
"""Test CANCELLED status can transition to retry checkpoint states."""
|
||||
from backend.models.job import STATE_TRANSITIONS
|
||||
valid_transitions = STATE_TRANSITIONS.get(JobStatus.CANCELLED, [])
|
||||
|
||||
# CANCELLED should allow same retry transitions as FAILED
|
||||
assert JobStatus.INSTRUMENTAL_SELECTED in valid_transitions, "CANCELLED should allow retry to INSTRUMENTAL_SELECTED"
|
||||
assert JobStatus.REVIEW_COMPLETE in valid_transitions, "CANCELLED should allow retry to REVIEW_COMPLETE"
|
||||
assert JobStatus.LYRICS_COMPLETE in valid_transitions, "CANCELLED should allow retry to LYRICS_COMPLETE"
|
||||
assert JobStatus.DOWNLOADING in valid_transitions, "CANCELLED should allow retry to DOWNLOADING (restart)"
|
||||
|
||||
def test_failed_can_restart_from_beginning(self):
|
||||
"""Test FAILED status can transition to DOWNLOADING for restart from beginning."""
|
||||
from backend.models.job import STATE_TRANSITIONS
|
||||
valid_transitions = STATE_TRANSITIONS.get(JobStatus.FAILED, [])
|
||||
|
||||
# FAILED should allow restart from beginning
|
||||
assert JobStatus.DOWNLOADING in valid_transitions, "FAILED should allow restart to DOWNLOADING"
|
||||
|
||||
|
||||
class TestJobModelSerialization:
|
||||
"""Tests for Job model serialization to/from API."""
|
||||
|
||||
def test_job_to_dict(self):
|
||||
"""Test Job converts to dict correctly."""
|
||||
job = Job(
|
||||
job_id="test123",
|
||||
status=JobStatus.PENDING,
|
||||
created_at=datetime.now(UTC),
|
||||
updated_at=datetime.now(UTC),
|
||||
artist="Test Artist",
|
||||
title="Test Song"
|
||||
)
|
||||
data = job.model_dump()
|
||||
assert data["job_id"] == "test123"
|
||||
assert data["status"] == "pending"
|
||||
assert data["artist"] == "Test Artist"
|
||||
|
||||
def test_job_from_dict(self):
|
||||
"""Test Job can be created from dict."""
|
||||
data = {
|
||||
"job_id": "test123",
|
||||
"status": "pending",
|
||||
"created_at": datetime.now(UTC).isoformat(),
|
||||
"updated_at": datetime.now(UTC).isoformat(),
|
||||
"artist": "Test",
|
||||
"title": "Song"
|
||||
}
|
||||
job = Job.model_validate(data)
|
||||
assert job.job_id == "test123"
|
||||
assert job.status == JobStatus.PENDING
|
||||
|
||||
def test_job_handles_null_optional_fields(self):
|
||||
"""Test Job handles null optional fields."""
|
||||
job = Job(
|
||||
job_id="test",
|
||||
status=JobStatus.PENDING,
|
||||
created_at=datetime.now(UTC),
|
||||
updated_at=datetime.now(UTC)
|
||||
)
|
||||
assert job.artist is None
|
||||
assert job.title is None
|
||||
assert job.url is None
|
||||
assert job.file_urls == {}
|
||||
|
||||
|
||||
class TestRetryEndpoint:
|
||||
"""Tests for the retry job endpoint."""
|
||||
|
||||
@pytest.fixture
|
||||
def mock_job_manager(self):
|
||||
"""Create mock job manager."""
|
||||
return MagicMock()
|
||||
|
||||
@pytest.fixture
|
||||
def mock_worker_service(self):
|
||||
"""Create mock worker service."""
|
||||
return MagicMock()
|
||||
|
||||
def test_retry_rejects_pending_job(self, mock_job_manager):
|
||||
"""Test retry endpoint rejects jobs that are not failed or cancelled."""
|
||||
# Create a pending job
|
||||
job = Job(
|
||||
job_id="test123",
|
||||
status=JobStatus.PENDING,
|
||||
created_at=datetime.now(UTC),
|
||||
updated_at=datetime.now(UTC),
|
||||
artist="Test",
|
||||
title="Song"
|
||||
)
|
||||
mock_job_manager.get_job.return_value = job
|
||||
|
||||
# The endpoint should reject this with a 400 error
|
||||
# (status check happens before any logic)
|
||||
assert job.status not in [JobStatus.FAILED, JobStatus.CANCELLED]
|
||||
|
||||
def test_retry_accepts_failed_job(self, mock_job_manager):
|
||||
"""Test retry endpoint accepts failed jobs."""
|
||||
job = Job(
|
||||
job_id="test123",
|
||||
status=JobStatus.FAILED,
|
||||
created_at=datetime.now(UTC),
|
||||
updated_at=datetime.now(UTC),
|
||||
artist="Test",
|
||||
title="Song",
|
||||
input_media_gcs_path="jobs/test123/input/audio.flac"
|
||||
)
|
||||
mock_job_manager.get_job.return_value = job
|
||||
|
||||
# Status should be accepted
|
||||
assert job.status in [JobStatus.FAILED, JobStatus.CANCELLED]
|
||||
|
||||
def test_retry_accepts_cancelled_job(self, mock_job_manager):
|
||||
"""Test retry endpoint accepts cancelled jobs."""
|
||||
job = Job(
|
||||
job_id="test123",
|
||||
status=JobStatus.CANCELLED,
|
||||
created_at=datetime.now(UTC),
|
||||
updated_at=datetime.now(UTC),
|
||||
artist="Test",
|
||||
title="Song",
|
||||
input_media_gcs_path="jobs/test123/input/audio.flac"
|
||||
)
|
||||
mock_job_manager.get_job.return_value = job
|
||||
|
||||
# Status should be accepted
|
||||
assert job.status in [JobStatus.FAILED, JobStatus.CANCELLED]
|
||||
|
||||
def test_retry_checkpoint_detection_video_generation(self):
|
||||
"""Test retry detects video generation checkpoint."""
|
||||
job = Job(
|
||||
job_id="test123",
|
||||
status=JobStatus.FAILED,
|
||||
created_at=datetime.now(UTC),
|
||||
updated_at=datetime.now(UTC),
|
||||
artist="Test",
|
||||
title="Song",
|
||||
file_urls={
|
||||
'videos': {'with_vocals': 'gs://bucket/path/video.mkv'}
|
||||
},
|
||||
state_data={
|
||||
'instrumental_selection': 'clean'
|
||||
}
|
||||
)
|
||||
|
||||
# This job has video and instrumental selection - should retry from video generation
|
||||
has_video = job.file_urls.get('videos', {}).get('with_vocals')
|
||||
has_instrumental_selection = (job.state_data or {}).get('instrumental_selection')
|
||||
assert has_video and has_instrumental_selection
|
||||
|
||||
def test_retry_checkpoint_detection_render_stage(self):
|
||||
"""Test retry detects render stage checkpoint."""
|
||||
job = Job(
|
||||
job_id="test123",
|
||||
status=JobStatus.FAILED,
|
||||
created_at=datetime.now(UTC),
|
||||
updated_at=datetime.now(UTC),
|
||||
artist="Test",
|
||||
title="Song",
|
||||
file_urls={
|
||||
'lyrics': {'corrections': 'gs://bucket/path/corrections.json'},
|
||||
'screens': {'title': 'gs://bucket/path/title.mov'}
|
||||
}
|
||||
)
|
||||
|
||||
# This job has corrections and screens - should retry from render
|
||||
has_corrections = job.file_urls.get('lyrics', {}).get('corrections')
|
||||
has_screens = job.file_urls.get('screens', {}).get('title')
|
||||
has_video = job.file_urls.get('videos', {}).get('with_vocals')
|
||||
assert has_corrections and has_screens and not has_video
|
||||
|
||||
def test_retry_checkpoint_detection_from_beginning(self):
|
||||
"""Test retry detects need to restart from beginning."""
|
||||
job = Job(
|
||||
job_id="test123",
|
||||
status=JobStatus.CANCELLED,
|
||||
created_at=datetime.now(UTC),
|
||||
updated_at=datetime.now(UTC),
|
||||
artist="Test",
|
||||
title="Song",
|
||||
input_media_gcs_path="jobs/test123/input/audio.flac",
|
||||
file_urls={} # No progress yet
|
||||
)
|
||||
|
||||
# This job has input audio but no other files - should restart from beginning
|
||||
has_input = job.input_media_gcs_path
|
||||
has_stems = job.file_urls.get('stems', {}).get('instrumental_clean')
|
||||
has_corrections = job.file_urls.get('lyrics', {}).get('corrections')
|
||||
assert has_input and not has_stems and not has_corrections
|
||||
|
||||
def test_retry_no_input_audio(self):
|
||||
"""Test retry fails when no input audio available."""
|
||||
job = Job(
|
||||
job_id="test123",
|
||||
status=JobStatus.CANCELLED,
|
||||
created_at=datetime.now(UTC),
|
||||
updated_at=datetime.now(UTC),
|
||||
artist="Test",
|
||||
title="Song",
|
||||
# No input_media_gcs_path and no url
|
||||
file_urls={}
|
||||
)
|
||||
|
||||
# This job has no input audio - should not be retryable
|
||||
has_input = job.input_media_gcs_path or job.url
|
||||
assert not has_input
|
||||
|
||||
@@ -1,337 +0,0 @@
|
||||
"""
|
||||
Unit tests for review.py routes and related components.
|
||||
|
||||
These tests verify the review-related state transitions and data structures.
|
||||
Tests that require full backend imports are in the emulator integration tests.
|
||||
"""
|
||||
import pytest
|
||||
import json
|
||||
|
||||
|
||||
class TestJobStatusTransitionsForReview:
|
||||
"""Tests for review-related state transitions.
|
||||
|
||||
These tests verify the Job model's state machine handles review flow correctly.
|
||||
"""
|
||||
|
||||
def test_awaiting_review_can_transition_to_in_review(self):
|
||||
"""Test AWAITING_REVIEW -> IN_REVIEW is valid."""
|
||||
from backend.models.job import STATE_TRANSITIONS, JobStatus
|
||||
valid_transitions = STATE_TRANSITIONS.get(JobStatus.AWAITING_REVIEW, [])
|
||||
assert JobStatus.IN_REVIEW in valid_transitions
|
||||
|
||||
def test_awaiting_review_can_transition_to_review_complete(self):
|
||||
"""Test AWAITING_REVIEW -> REVIEW_COMPLETE is valid (skip in_review)."""
|
||||
from backend.models.job import STATE_TRANSITIONS, JobStatus
|
||||
valid_transitions = STATE_TRANSITIONS.get(JobStatus.AWAITING_REVIEW, [])
|
||||
assert JobStatus.REVIEW_COMPLETE in valid_transitions
|
||||
|
||||
def test_in_review_can_transition_to_review_complete(self):
|
||||
"""Test IN_REVIEW -> REVIEW_COMPLETE is valid."""
|
||||
from backend.models.job import STATE_TRANSITIONS, JobStatus
|
||||
valid_transitions = STATE_TRANSITIONS.get(JobStatus.IN_REVIEW, [])
|
||||
assert JobStatus.REVIEW_COMPLETE in valid_transitions
|
||||
|
||||
def test_review_complete_transitions_to_rendering_video(self):
|
||||
"""Test REVIEW_COMPLETE -> RENDERING_VIDEO is valid."""
|
||||
from backend.models.job import STATE_TRANSITIONS, JobStatus
|
||||
valid_transitions = STATE_TRANSITIONS.get(JobStatus.REVIEW_COMPLETE, [])
|
||||
assert JobStatus.RENDERING_VIDEO in valid_transitions
|
||||
|
||||
def test_rendering_video_status_exists(self):
|
||||
"""Test RENDERING_VIDEO status exists in JobStatus enum."""
|
||||
from backend.models.job import JobStatus
|
||||
assert hasattr(JobStatus, 'RENDERING_VIDEO')
|
||||
assert JobStatus.RENDERING_VIDEO.value == "rendering_video"
|
||||
|
||||
def test_rendering_video_transitions_to_instrumental(self):
|
||||
"""Test RENDERING_VIDEO -> AWAITING_INSTRUMENTAL_SELECTION is valid."""
|
||||
from backend.models.job import STATE_TRANSITIONS, JobStatus
|
||||
valid_transitions = STATE_TRANSITIONS.get(JobStatus.RENDERING_VIDEO, [])
|
||||
assert JobStatus.AWAITING_INSTRUMENTAL_SELECTION in valid_transitions
|
||||
|
||||
|
||||
class TestStylesConfigRequirements:
|
||||
"""Tests documenting the required fields for ASS subtitle generation.
|
||||
|
||||
These tests verify that the styles config format is documented correctly.
|
||||
The actual styles creation is tested in integration tests.
|
||||
"""
|
||||
|
||||
def test_required_karaoke_fields_documented(self):
|
||||
"""Document required karaoke style fields for ASS generation."""
|
||||
# These fields are required by the ASS subtitle generator
|
||||
# Missing any of them will cause video generation to fail
|
||||
required_fields = [
|
||||
"font", # Font name for ASS
|
||||
"font_path", # Path to font file (can be empty string, NOT None)
|
||||
"ass_name", # Style name in ASS file
|
||||
"primary_color", # Format: "R, G, B, A"
|
||||
"secondary_color", # Format: "R, G, B, A"
|
||||
"outline_color", # Format: "R, G, B, A"
|
||||
"back_color", # Format: "R, G, B, A"
|
||||
"bold", # Boolean
|
||||
"italic", # Boolean
|
||||
"underline", # Boolean
|
||||
"strike_out", # Boolean
|
||||
"scale_x", # Integer
|
||||
"scale_y", # Integer
|
||||
"spacing", # Integer
|
||||
"angle", # Float
|
||||
"border_style", # Integer
|
||||
"outline", # Integer
|
||||
"shadow", # Integer
|
||||
"margin_l", # Integer
|
||||
"margin_r", # Integer
|
||||
"margin_v", # Integer
|
||||
"encoding", # Integer
|
||||
]
|
||||
|
||||
# This test documents the required fields
|
||||
# Actual validation happens in integration tests
|
||||
assert len(required_fields) == 22
|
||||
assert "ass_name" in required_fields # This was missing initially
|
||||
assert "font_path" in required_fields # Must be string, not None
|
||||
|
||||
def test_minimal_styles_structure(self):
|
||||
"""Document the minimal styles JSON structure."""
|
||||
minimal_styles = {
|
||||
"karaoke": {
|
||||
"background_color": "#000000",
|
||||
"font_path": "", # MUST be string, NOT None
|
||||
"font": "Arial",
|
||||
"ass_name": "Default", # REQUIRED
|
||||
"primary_color": "112, 112, 247, 255",
|
||||
"secondary_color": "255, 255, 255, 255",
|
||||
"outline_color": "26, 58, 235, 255",
|
||||
"back_color": "0, 0, 0, 0",
|
||||
"bold": False,
|
||||
"italic": False,
|
||||
"underline": False,
|
||||
"strike_out": False,
|
||||
"scale_x": 100,
|
||||
"scale_y": 100,
|
||||
"spacing": 0,
|
||||
"angle": 0.0,
|
||||
"border_style": 1,
|
||||
"outline": 1,
|
||||
"shadow": 0,
|
||||
"margin_l": 0,
|
||||
"margin_r": 0,
|
||||
"margin_v": 0,
|
||||
"encoding": 0
|
||||
}
|
||||
}
|
||||
|
||||
# Verify structure is valid JSON
|
||||
json_str = json.dumps(minimal_styles)
|
||||
parsed = json.loads(json_str)
|
||||
|
||||
assert "karaoke" in parsed
|
||||
assert parsed["karaoke"]["font_path"] == "" # Not None
|
||||
assert parsed["karaoke"]["ass_name"] == "Default"
|
||||
|
||||
|
||||
class TestCorrectionDataMerging:
|
||||
"""Tests documenting correction data merging requirements.
|
||||
|
||||
The LyricsTranscriber React UI sends only partial correction data:
|
||||
- corrections
|
||||
- corrected_segments
|
||||
|
||||
The backend must merge this with the original corrections.json to
|
||||
reconstruct a full CorrectionResult.
|
||||
"""
|
||||
|
||||
def test_frontend_sends_partial_data(self):
|
||||
"""Document what the frontend sends."""
|
||||
# The frontend sends only these fields
|
||||
frontend_payload = {
|
||||
"corrections": [], # List of corrections made
|
||||
"corrected_segments": [] # Updated segment data
|
||||
}
|
||||
|
||||
# This is NOT a full CorrectionResult
|
||||
assert "original_segments" not in frontend_payload
|
||||
assert "metadata" not in frontend_payload
|
||||
|
||||
def test_merging_strategy(self):
|
||||
"""Document the merging strategy for corrections."""
|
||||
# Original data has full structure
|
||||
original_data = {
|
||||
"original_segments": [{"id": 1, "text": "test"}],
|
||||
"corrected_segments": [{"id": 1, "text": "test"}],
|
||||
"corrections": [],
|
||||
"metadata": {"artist": "Test"}
|
||||
}
|
||||
|
||||
# Frontend sends partial update
|
||||
frontend_update = {
|
||||
"corrections": [{"id": 1, "type": "edit"}],
|
||||
"corrected_segments": [{"id": 1, "text": "updated"}]
|
||||
}
|
||||
|
||||
# Merging strategy: update only the fields sent by frontend
|
||||
if 'corrections' in frontend_update:
|
||||
original_data['corrections'] = frontend_update['corrections']
|
||||
if 'corrected_segments' in frontend_update:
|
||||
original_data['corrected_segments'] = frontend_update['corrected_segments']
|
||||
|
||||
# Result preserves original_segments and metadata
|
||||
assert original_data['original_segments'] == [{"id": 1, "text": "test"}]
|
||||
assert original_data['metadata'] == {"artist": "Test"}
|
||||
# But has updated corrections and corrected_segments
|
||||
assert original_data['corrections'] == [{"id": 1, "type": "edit"}]
|
||||
assert original_data['corrected_segments'] == [{"id": 1, "text": "updated"}]
|
||||
|
||||
|
||||
class TestAddLyricsEndpoint:
|
||||
"""Tests for the add-lyrics endpoint."""
|
||||
|
||||
def test_add_lyrics_requires_source_and_lyrics(self):
|
||||
"""Document that add_lyrics expects source and lyrics fields."""
|
||||
# The frontend sends this payload
|
||||
valid_payload = {
|
||||
"source": "custom",
|
||||
"lyrics": "Line 1\nLine 2\nLine 3"
|
||||
}
|
||||
|
||||
# Both fields are required
|
||||
assert "source" in valid_payload
|
||||
assert "lyrics" in valid_payload
|
||||
assert len(valid_payload["source"].strip()) > 0
|
||||
assert len(valid_payload["lyrics"].strip()) > 0
|
||||
|
||||
def test_add_lyrics_uses_correction_operations(self):
|
||||
"""Verify CorrectionOperations.add_lyrics_source is available."""
|
||||
from lyrics_transcriber.correction.operations import CorrectionOperations
|
||||
|
||||
# The method should exist
|
||||
assert hasattr(CorrectionOperations, 'add_lyrics_source')
|
||||
|
||||
# It should be a static method
|
||||
import inspect
|
||||
# Get the method and check it's callable
|
||||
method = getattr(CorrectionOperations, 'add_lyrics_source')
|
||||
assert callable(method)
|
||||
|
||||
|
||||
class TestPreviewStyleLoading:
|
||||
"""Tests for the unified style loader used in preview video generation.
|
||||
|
||||
When a job has custom styles (uploaded via --style_params_json), these
|
||||
must be loaded and applied to preview videos, not just the final render.
|
||||
|
||||
This was a bug: preview videos were using minimal styles (black background)
|
||||
even when the job had custom backgrounds and fonts configured.
|
||||
|
||||
The style loading logic is now consolidated in karaoke_gen.style_loader
|
||||
to avoid duplication between workers and API routes.
|
||||
"""
|
||||
|
||||
def test_load_styles_from_gcs_with_custom_styles(self, tmp_path):
|
||||
"""Test that custom styles are downloaded and applied for preview."""
|
||||
import os
|
||||
from karaoke_gen.style_loader import load_styles_from_gcs
|
||||
|
||||
# Create source style params file
|
||||
source_style_params = tmp_path / "source_styles.json"
|
||||
style_data = {
|
||||
"karaoke": {
|
||||
"background_image": "/original/path/background.png",
|
||||
"font_path": "/original/path/font.ttf",
|
||||
"background_color": "#000000",
|
||||
"font": "Arial",
|
||||
"ass_name": "Default",
|
||||
}
|
||||
}
|
||||
source_style_params.write_text(json.dumps(style_data))
|
||||
|
||||
# Create source asset files
|
||||
source_background = tmp_path / "background.png"
|
||||
source_background.write_bytes(b"PNG image data")
|
||||
source_font = tmp_path / "font.ttf"
|
||||
source_font.write_bytes(b"TTF font data")
|
||||
|
||||
# Create mock download function that simulates GCS download
|
||||
def mock_download(gcs_path, local_path):
|
||||
if "style_params.json" in gcs_path:
|
||||
with open(local_path, 'w') as f:
|
||||
f.write(source_style_params.read_text())
|
||||
elif "karaoke_background" in gcs_path:
|
||||
with open(local_path, 'wb') as f:
|
||||
f.write(source_background.read_bytes())
|
||||
elif "font.ttf" in gcs_path:
|
||||
with open(local_path, 'wb') as f:
|
||||
f.write(source_font.read_bytes())
|
||||
|
||||
# Call the unified style loader function
|
||||
style_assets = {
|
||||
"style_params": "uploads/test123/style/style_params.json",
|
||||
"karaoke_background": "uploads/test123/style/karaoke_background.png",
|
||||
"font": "uploads/test123/style/font.ttf",
|
||||
}
|
||||
|
||||
styles_path, result_styles = load_styles_from_gcs(
|
||||
style_params_gcs_path="uploads/test123/style/style_params.json",
|
||||
style_assets=style_assets,
|
||||
temp_dir=str(tmp_path / "workdir"),
|
||||
download_func=mock_download,
|
||||
)
|
||||
|
||||
# Verify styles file was created
|
||||
assert os.path.exists(styles_path)
|
||||
|
||||
# The paths should now point to the local downloaded files, not the original paths
|
||||
assert "karaoke" in result_styles
|
||||
assert result_styles["karaoke"]["background_image"] != "/original/path/background.png"
|
||||
assert "karaoke_background.png" in result_styles["karaoke"]["background_image"]
|
||||
assert result_styles["karaoke"]["font_path"] != "/original/path/font.ttf"
|
||||
assert "font.ttf" in result_styles["karaoke"]["font_path"]
|
||||
|
||||
def test_load_styles_from_gcs_falls_back_to_minimal(self, tmp_path):
|
||||
"""Test that minimal styles are used when job has no custom styles."""
|
||||
import os
|
||||
from karaoke_gen.style_loader import load_styles_from_gcs
|
||||
|
||||
# Call with no custom styles
|
||||
styles_path, result_styles = load_styles_from_gcs(
|
||||
style_params_gcs_path=None, # No custom styles
|
||||
style_assets={},
|
||||
temp_dir=str(tmp_path),
|
||||
download_func=lambda gcs_path, local_path: None, # Won't be called
|
||||
)
|
||||
|
||||
# Verify styles file was created
|
||||
assert os.path.exists(styles_path)
|
||||
|
||||
# Should have karaoke section with minimal/default values
|
||||
assert "karaoke" in result_styles
|
||||
assert result_styles["karaoke"]["background_color"] == "#000000"
|
||||
assert result_styles["karaoke"]["font"] == "Arial"
|
||||
# Minimal styles have background_image as None (default)
|
||||
assert result_styles["karaoke"].get("background_image") is None
|
||||
|
||||
def test_asset_mapping_is_complete(self):
|
||||
"""Verify all required asset mappings are defined in the unified style loader."""
|
||||
from karaoke_gen.style_loader import ASSET_KEY_MAPPINGS
|
||||
|
||||
# These mappings must be present for styles to work correctly
|
||||
required_keys = [
|
||||
"karaoke_background",
|
||||
"intro_background",
|
||||
"end_background",
|
||||
"font",
|
||||
]
|
||||
|
||||
for key in required_keys:
|
||||
assert key in ASSET_KEY_MAPPINGS, f"Asset mapping '{key}' not found in ASSET_KEY_MAPPINGS"
|
||||
|
||||
# Verify karaoke_background maps to the correct path
|
||||
karaoke_mapping = ASSET_KEY_MAPPINGS["karaoke_background"]
|
||||
assert karaoke_mapping == ("karaoke", "background_image")
|
||||
|
||||
# Verify font maps to multiple sections
|
||||
font_mappings = ASSET_KEY_MAPPINGS["font"]
|
||||
assert isinstance(font_mappings, list)
|
||||
assert ("karaoke", "font_path") in font_mappings
|
||||
@@ -1,556 +0,0 @@
|
||||
"""
|
||||
Unit tests for service layer.
|
||||
|
||||
Tests AuthService, StorageService, WorkerService, and FirestoreService
|
||||
without requiring actual cloud resources.
|
||||
"""
|
||||
import pytest
|
||||
from unittest.mock import Mock, MagicMock, AsyncMock, patch, call
|
||||
from datetime import datetime, UTC
|
||||
|
||||
# Mock Google Cloud before imports
|
||||
import sys
|
||||
sys.modules['google.cloud.firestore'] = MagicMock()
|
||||
sys.modules['google.cloud.storage'] = MagicMock()
|
||||
sys.modules['google.cloud.tasks_v2'] = MagicMock()
|
||||
|
||||
from backend.services.auth_service import AuthService, UserType
|
||||
from backend.services.storage_service import StorageService
|
||||
from backend.services.worker_service import WorkerService, get_worker_service
|
||||
from backend.services.firestore_service import FirestoreService
|
||||
from backend.models.job import Job, JobStatus
|
||||
|
||||
|
||||
class TestAuthService:
|
||||
"""Test AuthService token validation and management."""
|
||||
|
||||
def test_validate_admin_token(self):
|
||||
"""Test validating hardcoded admin tokens."""
|
||||
with patch('backend.services.auth_service.get_settings') as mock_settings:
|
||||
mock_settings.return_value.admin_tokens = "admin123,secret456"
|
||||
|
||||
auth_service = AuthService()
|
||||
|
||||
# Valid admin token - returns (is_valid, user_type, usage_count, token)
|
||||
is_valid, user_type, usage_count, token = auth_service.validate_token('admin123')
|
||||
assert is_valid is True
|
||||
assert user_type == UserType.ADMIN
|
||||
|
||||
# Another valid admin token
|
||||
is_valid, user_type, _, _ = auth_service.validate_token('secret456')
|
||||
assert is_valid is True
|
||||
assert user_type == UserType.ADMIN
|
||||
|
||||
def test_validate_invalid_token(self):
|
||||
"""Test rejecting invalid tokens."""
|
||||
with patch('backend.services.auth_service.FirestoreService') as mock_fs:
|
||||
mock_fs_instance = Mock()
|
||||
mock_fs.return_value = mock_fs_instance
|
||||
mock_fs_instance.get_token.return_value = None # Token not in DB
|
||||
|
||||
with patch('backend.services.auth_service.get_settings') as mock_settings:
|
||||
mock_settings.return_value.admin_tokens = "admin123"
|
||||
|
||||
auth_service = AuthService()
|
||||
is_valid, _, _, _ = auth_service.validate_token('invalid_token')
|
||||
|
||||
assert is_valid is False
|
||||
|
||||
def test_validate_token_from_firestore(self):
|
||||
"""Test validating tokens stored in Firestore."""
|
||||
with patch('backend.services.auth_service.FirestoreService') as mock_fs:
|
||||
mock_fs_instance = Mock()
|
||||
mock_fs.return_value = mock_fs_instance
|
||||
|
||||
# Mock Firestore returning a token
|
||||
mock_fs_instance.get_token.return_value = {
|
||||
'token': 'db_token123',
|
||||
'type': 'unlimited',
|
||||
'valid': True,
|
||||
'usage_count': 5,
|
||||
'created_at': datetime.now(UTC).isoformat()
|
||||
}
|
||||
|
||||
with patch('backend.services.auth_service.get_settings') as mock_settings:
|
||||
mock_settings.return_value.admin_tokens = ""
|
||||
|
||||
auth_service = AuthService()
|
||||
is_valid, user_type, usage_count, token = auth_service.validate_token('db_token123')
|
||||
|
||||
assert is_valid is True
|
||||
assert user_type == UserType.UNLIMITED
|
||||
# usage_count can be -1 for unlimited tokens
|
||||
assert isinstance(usage_count, int)
|
||||
|
||||
|
||||
class TestStorageService:
|
||||
"""Test StorageService GCS operations."""
|
||||
|
||||
def test_upload_file(self):
|
||||
"""Test uploading a file to GCS."""
|
||||
with patch('backend.services.storage_service.storage') as mock_storage:
|
||||
mock_client = MagicMock()
|
||||
mock_storage.Client.return_value = mock_client
|
||||
|
||||
mock_bucket = MagicMock()
|
||||
mock_client.bucket.return_value = mock_bucket
|
||||
|
||||
mock_blob = MagicMock()
|
||||
mock_bucket.blob.return_value = mock_blob
|
||||
|
||||
storage_service = StorageService()
|
||||
|
||||
# Upload file - correct parameter names
|
||||
result = storage_service.upload_file(
|
||||
local_path="/tmp/test.flac",
|
||||
destination_path="uploads/test123/test.flac"
|
||||
)
|
||||
|
||||
# Verify blob was created and uploaded
|
||||
mock_bucket.blob.assert_called_once_with("uploads/test123/test.flac")
|
||||
mock_blob.upload_from_filename.assert_called_once_with("/tmp/test.flac")
|
||||
# Result should be the destination path
|
||||
assert result == "uploads/test123/test.flac"
|
||||
|
||||
def test_download_file(self):
|
||||
"""Test downloading a file from GCS."""
|
||||
with patch('backend.services.storage_service.storage') as mock_storage:
|
||||
mock_client = MagicMock()
|
||||
mock_storage.Client.return_value = mock_client
|
||||
|
||||
mock_bucket = MagicMock()
|
||||
mock_client.bucket.return_value = mock_bucket
|
||||
|
||||
mock_blob = MagicMock()
|
||||
mock_bucket.blob.return_value = mock_blob
|
||||
|
||||
storage_service = StorageService()
|
||||
|
||||
# Download file - correct parameter names
|
||||
storage_service.download_file(
|
||||
source_path="uploads/test123/test.flac",
|
||||
destination_path="/tmp/downloaded.flac"
|
||||
)
|
||||
|
||||
# Verify download was called
|
||||
mock_blob.download_to_filename.assert_called_once_with("/tmp/downloaded.flac")
|
||||
|
||||
def test_delete_file(self):
|
||||
"""Test deleting a file from GCS."""
|
||||
with patch('backend.services.storage_service.storage') as mock_storage:
|
||||
mock_client = MagicMock()
|
||||
mock_storage.Client.return_value = mock_client
|
||||
|
||||
mock_bucket = MagicMock()
|
||||
mock_client.bucket.return_value = mock_bucket
|
||||
|
||||
mock_blob = MagicMock()
|
||||
mock_bucket.blob.return_value = mock_blob
|
||||
|
||||
storage_service = StorageService()
|
||||
|
||||
# Delete file
|
||||
storage_service.delete_file("uploads/test123/test.flac")
|
||||
|
||||
# Verify delete was called
|
||||
mock_blob.delete.assert_called_once()
|
||||
|
||||
|
||||
class TestWorkerService:
|
||||
"""Test WorkerService internal HTTP calls."""
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_trigger_audio_worker(self):
|
||||
"""Test triggering audio worker via internal HTTP."""
|
||||
with patch('backend.services.worker_service.httpx.AsyncClient') as mock_client_cls:
|
||||
mock_client = AsyncMock()
|
||||
mock_client_cls.return_value.__aenter__.return_value = mock_client
|
||||
|
||||
mock_response = Mock()
|
||||
mock_response.status_code = 200
|
||||
mock_client.post.return_value = mock_response
|
||||
|
||||
worker_service = get_worker_service()
|
||||
|
||||
await worker_service.trigger_audio_worker("test123")
|
||||
|
||||
# Verify HTTP POST was made
|
||||
mock_client.post.assert_called_once()
|
||||
call_args = mock_client.post.call_args
|
||||
assert "/api/internal/workers/audio" in str(call_args)
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_trigger_lyrics_worker(self):
|
||||
"""Test triggering lyrics worker via internal HTTP."""
|
||||
with patch('backend.services.worker_service.httpx.AsyncClient') as mock_client_cls:
|
||||
mock_client = AsyncMock()
|
||||
mock_client_cls.return_value.__aenter__.return_value = mock_client
|
||||
|
||||
mock_response = Mock()
|
||||
mock_response.status_code = 200
|
||||
mock_client.post.return_value = mock_response
|
||||
|
||||
worker_service = get_worker_service()
|
||||
|
||||
await worker_service.trigger_lyrics_worker("test123")
|
||||
|
||||
# Verify HTTP POST was made
|
||||
mock_client.post.assert_called_once()
|
||||
call_args = mock_client.post.call_args
|
||||
assert "/api/internal/workers/lyrics" in str(call_args)
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_trigger_screens_worker(self):
|
||||
"""Test triggering screens worker via internal HTTP."""
|
||||
with patch('backend.services.worker_service.httpx.AsyncClient') as mock_client_cls:
|
||||
mock_client = AsyncMock()
|
||||
mock_client_cls.return_value.__aenter__.return_value = mock_client
|
||||
|
||||
mock_response = Mock()
|
||||
mock_response.status_code = 200
|
||||
mock_client.post.return_value = mock_response
|
||||
|
||||
worker_service = get_worker_service()
|
||||
|
||||
await worker_service.trigger_screens_worker("test123")
|
||||
|
||||
# Verify HTTP POST was made
|
||||
mock_client.post.assert_called_once()
|
||||
call_args = mock_client.post.call_args
|
||||
assert "/api/internal/workers/screens" in str(call_args)
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_trigger_video_worker(self):
|
||||
"""Test triggering video worker via internal HTTP."""
|
||||
with patch('backend.services.worker_service.httpx.AsyncClient') as mock_client_cls:
|
||||
mock_client = AsyncMock()
|
||||
mock_client_cls.return_value.__aenter__.return_value = mock_client
|
||||
|
||||
mock_response = Mock()
|
||||
mock_response.status_code = 200
|
||||
mock_client.post.return_value = mock_response
|
||||
|
||||
worker_service = get_worker_service()
|
||||
|
||||
await worker_service.trigger_video_worker("test123")
|
||||
|
||||
# Verify HTTP POST was made
|
||||
mock_client.post.assert_called_once()
|
||||
call_args = mock_client.post.call_args
|
||||
assert "/api/internal/workers/video" in str(call_args)
|
||||
|
||||
|
||||
class TestWorkerServiceCloudTasks:
|
||||
"""Test WorkerService Cloud Tasks integration."""
|
||||
|
||||
def test_should_use_cloud_tasks_default_false(self):
|
||||
"""Test that Cloud Tasks is disabled by default."""
|
||||
from backend.services.worker_service import WorkerService, reset_worker_service
|
||||
reset_worker_service()
|
||||
|
||||
with patch('backend.services.worker_service.os.getenv') as mock_getenv:
|
||||
mock_getenv.side_effect = lambda k, d=None: {
|
||||
'PORT': '8000',
|
||||
}.get(k, d)
|
||||
|
||||
with patch('backend.services.worker_service.get_settings') as mock_settings:
|
||||
mock_settings.return_value.admin_tokens = None
|
||||
mock_settings.return_value.google_cloud_project = "test-project"
|
||||
mock_settings.return_value.enable_cloud_tasks = False
|
||||
mock_settings.return_value.gcp_region = "us-central1"
|
||||
mock_settings.return_value.use_cloud_run_jobs_for_video = False
|
||||
|
||||
service = WorkerService()
|
||||
assert service._use_cloud_tasks is False
|
||||
|
||||
def test_should_use_cloud_tasks_enabled(self):
|
||||
"""Test that Cloud Tasks can be enabled via settings."""
|
||||
from backend.services.worker_service import WorkerService, reset_worker_service
|
||||
reset_worker_service()
|
||||
|
||||
with patch('backend.services.worker_service.os.getenv') as mock_getenv:
|
||||
mock_getenv.side_effect = lambda k, d=None: {
|
||||
'PORT': '8000',
|
||||
}.get(k, d)
|
||||
|
||||
with patch('backend.services.worker_service.get_settings') as mock_settings:
|
||||
mock_settings.return_value.admin_tokens = None
|
||||
mock_settings.return_value.google_cloud_project = "test-project"
|
||||
mock_settings.return_value.enable_cloud_tasks = True
|
||||
mock_settings.return_value.gcp_region = "us-central1"
|
||||
mock_settings.return_value.use_cloud_run_jobs_for_video = False
|
||||
|
||||
service = WorkerService()
|
||||
assert service._use_cloud_tasks is True
|
||||
|
||||
def test_worker_queues_mapping(self):
|
||||
"""Test that all worker types have queue mappings."""
|
||||
from backend.services.worker_service import WORKER_QUEUES
|
||||
|
||||
# Verify all expected workers have queues
|
||||
assert "audio" in WORKER_QUEUES
|
||||
assert "lyrics" in WORKER_QUEUES
|
||||
assert "screens" in WORKER_QUEUES
|
||||
assert "render-video" in WORKER_QUEUES
|
||||
assert "video" in WORKER_QUEUES
|
||||
|
||||
# Verify queue names are correct
|
||||
assert WORKER_QUEUES["audio"] == "audio-worker-queue"
|
||||
assert WORKER_QUEUES["lyrics"] == "lyrics-worker-queue"
|
||||
assert WORKER_QUEUES["screens"] == "screens-worker-queue"
|
||||
assert WORKER_QUEUES["render-video"] == "render-worker-queue"
|
||||
assert WORKER_QUEUES["video"] == "video-worker-queue"
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_trigger_worker_uses_http_when_cloud_tasks_disabled(self):
|
||||
"""Test that trigger_worker uses HTTP when Cloud Tasks is disabled."""
|
||||
from backend.services.worker_service import WorkerService, reset_worker_service
|
||||
reset_worker_service()
|
||||
|
||||
with patch('backend.services.worker_service.os.getenv') as mock_getenv:
|
||||
mock_getenv.side_effect = lambda k, d=None: {
|
||||
'PORT': '8000',
|
||||
}.get(k, d)
|
||||
|
||||
with patch('backend.services.worker_service.get_settings') as mock_settings:
|
||||
mock_settings.return_value.admin_tokens = "test-token"
|
||||
mock_settings.return_value.google_cloud_project = "test-project"
|
||||
mock_settings.return_value.enable_cloud_tasks = False
|
||||
mock_settings.return_value.gcp_region = "us-central1"
|
||||
mock_settings.return_value.use_cloud_run_jobs_for_video = False
|
||||
|
||||
with patch('backend.services.worker_service.httpx.AsyncClient') as mock_client_cls:
|
||||
mock_client = AsyncMock()
|
||||
mock_client_cls.return_value.__aenter__.return_value = mock_client
|
||||
|
||||
mock_response = Mock()
|
||||
mock_response.status_code = 200
|
||||
mock_client.post.return_value = mock_response
|
||||
|
||||
service = WorkerService()
|
||||
assert service._use_cloud_tasks is False
|
||||
|
||||
result = await service.trigger_worker("audio", "test-job-123")
|
||||
|
||||
# Verify HTTP was used
|
||||
assert result is True
|
||||
mock_client.post.assert_called_once()
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_trigger_worker_uses_cloud_tasks_when_enabled(self):
|
||||
"""Test that trigger_worker uses Cloud Tasks when enabled."""
|
||||
from backend.services.worker_service import WorkerService, reset_worker_service
|
||||
reset_worker_service()
|
||||
|
||||
with patch('backend.services.worker_service.os.getenv') as mock_getenv:
|
||||
mock_getenv.side_effect = lambda k, d=None: {
|
||||
'CLOUD_RUN_SERVICE_URL': 'https://api.example.com',
|
||||
}.get(k, d)
|
||||
|
||||
with patch('backend.services.worker_service.get_settings') as mock_settings:
|
||||
mock_settings.return_value.admin_tokens = "test-token"
|
||||
mock_settings.return_value.google_cloud_project = "test-project"
|
||||
mock_settings.return_value.enable_cloud_tasks = True
|
||||
mock_settings.return_value.gcp_region = "us-central1"
|
||||
mock_settings.return_value.use_cloud_run_jobs_for_video = False
|
||||
|
||||
# Mock Cloud Tasks client
|
||||
mock_tasks_client = MagicMock()
|
||||
mock_tasks_client.queue_path.return_value = "projects/test-project/locations/us-central1/queues/audio-worker-queue"
|
||||
mock_task_response = MagicMock()
|
||||
mock_task_response.name = "projects/test-project/locations/us-central1/queues/audio-worker-queue/tasks/abc123"
|
||||
mock_tasks_client.create_task.return_value = mock_task_response
|
||||
|
||||
# Mock the google.cloud.tasks_v2 module which is imported inside the method
|
||||
mock_tasks_module = MagicMock()
|
||||
mock_tasks_module.CloudTasksClient.return_value = mock_tasks_client
|
||||
mock_tasks_module.HttpMethod.POST = "POST"
|
||||
|
||||
with patch.dict('sys.modules', {'google.cloud.tasks_v2': mock_tasks_module}):
|
||||
with patch.dict(sys.modules, {'google.cloud': MagicMock()}):
|
||||
service = WorkerService()
|
||||
service._tasks_client = mock_tasks_client # Inject mocked client
|
||||
|
||||
result = await service.trigger_worker("audio", "test-job-123")
|
||||
|
||||
# Verify Cloud Tasks was used
|
||||
assert result is True
|
||||
mock_tasks_client.create_task.assert_called_once()
|
||||
|
||||
# Verify task payload
|
||||
call_args = mock_tasks_client.create_task.call_args
|
||||
assert call_args is not None
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_trigger_worker_returns_false_for_unknown_worker_type(self):
|
||||
"""Test that trigger_worker returns False for unknown worker types."""
|
||||
from backend.services.worker_service import WorkerService, reset_worker_service
|
||||
reset_worker_service()
|
||||
|
||||
with patch('backend.services.worker_service.os.getenv') as mock_getenv:
|
||||
mock_getenv.side_effect = lambda k, d=None: {
|
||||
'CLOUD_RUN_SERVICE_URL': 'https://api.example.com',
|
||||
}.get(k, d)
|
||||
|
||||
with patch('backend.services.worker_service.get_settings') as mock_settings:
|
||||
mock_settings.return_value.admin_tokens = "test-token"
|
||||
mock_settings.return_value.google_cloud_project = "test-project"
|
||||
mock_settings.return_value.enable_cloud_tasks = True
|
||||
mock_settings.return_value.gcp_region = "us-central1"
|
||||
mock_settings.return_value.use_cloud_run_jobs_for_video = False
|
||||
|
||||
service = WorkerService()
|
||||
|
||||
result = await service.trigger_worker("unknown-worker", "test-job-123")
|
||||
|
||||
assert result is False
|
||||
|
||||
def test_reset_worker_service(self):
|
||||
"""Test that reset_worker_service resets the singleton."""
|
||||
from backend.services.worker_service import get_worker_service, reset_worker_service
|
||||
|
||||
with patch('backend.services.worker_service.get_settings') as mock_settings:
|
||||
mock_settings.return_value.admin_tokens = None
|
||||
mock_settings.return_value.google_cloud_project = "test-project"
|
||||
mock_settings.return_value.enable_cloud_tasks = False
|
||||
mock_settings.return_value.gcp_region = "us-central1"
|
||||
mock_settings.return_value.use_cloud_run_jobs_for_video = False
|
||||
|
||||
service1 = get_worker_service()
|
||||
reset_worker_service()
|
||||
service2 = get_worker_service()
|
||||
|
||||
# After reset, should be different instances
|
||||
assert service1 is not service2
|
||||
|
||||
|
||||
class TestFirestoreService:
|
||||
"""Test FirestoreService database operations."""
|
||||
|
||||
def test_create_job(self):
|
||||
"""Test creating a job document in Firestore."""
|
||||
with patch('backend.services.firestore_service.firestore') as mock_firestore:
|
||||
mock_client = MagicMock()
|
||||
mock_firestore.Client.return_value = mock_client
|
||||
|
||||
mock_collection = MagicMock()
|
||||
mock_client.collection.return_value = mock_collection
|
||||
|
||||
mock_doc_ref = MagicMock()
|
||||
mock_collection.document.return_value = mock_doc_ref
|
||||
|
||||
firestore_service = FirestoreService()
|
||||
|
||||
job = Job(
|
||||
job_id="test123",
|
||||
status=JobStatus.PENDING,
|
||||
created_at=datetime.now(UTC),
|
||||
updated_at=datetime.now(UTC)
|
||||
)
|
||||
|
||||
firestore_service.create_job(job) # Returns None
|
||||
|
||||
# Verify document was set
|
||||
mock_doc_ref.set.assert_called_once()
|
||||
|
||||
def test_get_job(self):
|
||||
"""Test fetching a job from Firestore."""
|
||||
with patch('backend.services.firestore_service.firestore') as mock_firestore:
|
||||
mock_client = MagicMock()
|
||||
mock_firestore.Client.return_value = mock_client
|
||||
|
||||
mock_collection = MagicMock()
|
||||
mock_client.collection.return_value = mock_collection
|
||||
|
||||
mock_doc_ref = MagicMock()
|
||||
mock_collection.document.return_value = mock_doc_ref
|
||||
|
||||
mock_doc = MagicMock()
|
||||
mock_doc.exists = True
|
||||
mock_doc.to_dict.return_value = {
|
||||
'job_id': 'test123',
|
||||
'status': 'pending',
|
||||
'progress': 0,
|
||||
'created_at': datetime.now(UTC).isoformat(),
|
||||
'updated_at': datetime.now(UTC).isoformat()
|
||||
}
|
||||
mock_doc_ref.get.return_value = mock_doc
|
||||
|
||||
firestore_service = FirestoreService()
|
||||
|
||||
job = firestore_service.get_job("test123")
|
||||
|
||||
# Verify document was fetched
|
||||
mock_doc_ref.get.assert_called_once()
|
||||
assert job is not None
|
||||
assert job.job_id == "test123"
|
||||
|
||||
def test_get_nonexistent_job(self):
|
||||
"""Test fetching a job that doesn't exist."""
|
||||
with patch('backend.services.firestore_service.firestore') as mock_firestore:
|
||||
mock_client = MagicMock()
|
||||
mock_firestore.Client.return_value = mock_client
|
||||
|
||||
mock_collection = MagicMock()
|
||||
mock_client.collection.return_value = mock_collection
|
||||
|
||||
mock_doc_ref = MagicMock()
|
||||
mock_collection.document.return_value = mock_doc_ref
|
||||
|
||||
mock_doc = MagicMock()
|
||||
mock_doc.exists = False
|
||||
mock_doc_ref.get.return_value = mock_doc
|
||||
|
||||
firestore_service = FirestoreService()
|
||||
|
||||
job = firestore_service.get_job("nonexistent")
|
||||
|
||||
assert job is None
|
||||
|
||||
def test_update_job(self):
|
||||
"""Test updating a job in Firestore."""
|
||||
with patch('backend.services.firestore_service.firestore') as mock_firestore:
|
||||
mock_client = MagicMock()
|
||||
mock_firestore.Client.return_value = mock_client
|
||||
|
||||
mock_collection = MagicMock()
|
||||
mock_client.collection.return_value = mock_collection
|
||||
|
||||
mock_doc_ref = MagicMock()
|
||||
mock_collection.document.return_value = mock_doc_ref
|
||||
|
||||
firestore_service = FirestoreService()
|
||||
|
||||
updates = {
|
||||
'status': JobStatus.SEPARATING_STAGE1,
|
||||
'progress': 25
|
||||
}
|
||||
|
||||
firestore_service.update_job("test123", updates)
|
||||
|
||||
# Verify update was called
|
||||
mock_doc_ref.update.assert_called_once()
|
||||
|
||||
def test_delete_job(self):
|
||||
"""Test deleting a job from Firestore."""
|
||||
with patch('backend.services.firestore_service.firestore') as mock_firestore:
|
||||
mock_client = MagicMock()
|
||||
mock_firestore.Client.return_value = mock_client
|
||||
|
||||
mock_collection = MagicMock()
|
||||
mock_client.collection.return_value = mock_collection
|
||||
|
||||
mock_doc_ref = MagicMock()
|
||||
mock_collection.document.return_value = mock_doc_ref
|
||||
|
||||
firestore_service = FirestoreService()
|
||||
|
||||
firestore_service.delete_job("test123")
|
||||
|
||||
# Verify delete was called
|
||||
mock_doc_ref.delete.assert_called_once()
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
pytest.main([__file__, "-v"])
|
||||
|
||||
@@ -1,112 +0,0 @@
|
||||
"""
|
||||
Extended unit tests for services.
|
||||
|
||||
These tests provide additional coverage for service layer code.
|
||||
"""
|
||||
import pytest
|
||||
from datetime import datetime, UTC
|
||||
from unittest.mock import MagicMock, AsyncMock, patch
|
||||
|
||||
from backend.models.job import Job, JobStatus
|
||||
|
||||
|
||||
class TestJobManagerExtended:
|
||||
"""Extended tests for JobManager.
|
||||
|
||||
Note: These tests verify the module structure and basic behavior.
|
||||
Full JobManager testing is in test_job_manager.py.
|
||||
"""
|
||||
|
||||
def test_job_manager_module_imports(self):
|
||||
"""Test JobManager module can be imported."""
|
||||
from backend.services import job_manager
|
||||
assert hasattr(job_manager, 'JobManager')
|
||||
|
||||
def test_state_transitions_defined(self):
|
||||
"""Test STATE_TRANSITIONS dict is defined."""
|
||||
from backend.models.job import STATE_TRANSITIONS
|
||||
assert STATE_TRANSITIONS is not None
|
||||
assert len(STATE_TRANSITIONS) > 0
|
||||
|
||||
|
||||
class TestStorageServiceExtended:
|
||||
"""Extended tests for StorageService."""
|
||||
|
||||
def test_storage_service_initialization(self):
|
||||
"""Test StorageService can be initialized."""
|
||||
with patch('backend.services.storage_service.storage'):
|
||||
from backend.services.storage_service import StorageService
|
||||
service = StorageService()
|
||||
assert service is not None
|
||||
|
||||
def test_storage_service_bucket_name(self):
|
||||
"""Test StorageService uses configured bucket."""
|
||||
mock_client = MagicMock()
|
||||
mock_bucket = MagicMock()
|
||||
mock_client.bucket.return_value = mock_bucket
|
||||
|
||||
with patch('backend.services.storage_service.storage.Client', return_value=mock_client), \
|
||||
patch.dict('os.environ', {'GCS_BUCKET_NAME': 'test-bucket'}):
|
||||
from backend.services.storage_service import StorageService
|
||||
service = StorageService()
|
||||
# Service should use the configured bucket
|
||||
|
||||
|
||||
class TestAuthServiceExtended:
|
||||
"""Extended tests for AuthService.
|
||||
|
||||
Note: Full auth service testing is in test_services.py.
|
||||
"""
|
||||
|
||||
def test_auth_service_module_imports(self):
|
||||
"""Test AuthService module can be imported."""
|
||||
from backend.services import auth_service
|
||||
assert hasattr(auth_service, 'AuthService')
|
||||
|
||||
|
||||
class TestWorkerServiceExtended:
|
||||
"""Extended tests for WorkerService."""
|
||||
|
||||
def test_worker_service_initialization(self):
|
||||
"""Test WorkerService can be initialized."""
|
||||
from backend.services.worker_service import WorkerService
|
||||
service = WorkerService()
|
||||
assert service is not None
|
||||
|
||||
def test_worker_service_get_base_url(self):
|
||||
"""Test WorkerService constructs correct base URL."""
|
||||
from backend.services.worker_service import WorkerService
|
||||
service = WorkerService()
|
||||
|
||||
url = service._get_base_url()
|
||||
assert 'http' in url
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_worker_service_trigger_with_mock(self):
|
||||
"""Test triggering a worker with mocked HTTP."""
|
||||
from backend.services.worker_service import WorkerService
|
||||
|
||||
service = WorkerService()
|
||||
|
||||
# Mock the HTTP client
|
||||
with patch('backend.services.worker_service.httpx.AsyncClient') as mock_client_cls:
|
||||
mock_client = MagicMock()
|
||||
mock_response = MagicMock()
|
||||
mock_response.status_code = 200
|
||||
mock_client.post = AsyncMock(return_value=mock_response)
|
||||
mock_client_cls.return_value.__aenter__ = AsyncMock(return_value=mock_client)
|
||||
mock_client_cls.return_value.__aexit__ = AsyncMock(return_value=None)
|
||||
|
||||
# The trigger methods should work with mocked HTTP
|
||||
|
||||
|
||||
class TestFirestoreServiceExtended:
|
||||
"""Extended tests for FirestoreService."""
|
||||
|
||||
def test_firestore_service_initialization(self):
|
||||
"""Test FirestoreService can be initialized."""
|
||||
with patch('backend.services.firestore_service.firestore'):
|
||||
from backend.services.firestore_service import FirestoreService
|
||||
service = FirestoreService()
|
||||
assert service is not None
|
||||
|
||||
@@ -1,448 +0,0 @@
|
||||
"""
|
||||
Tests for storage_service.py - Google Cloud Storage operations.
|
||||
|
||||
These tests mock the GCS client to verify:
|
||||
- Upload operations (file, fileobj, JSON)
|
||||
- Download operations (file, JSON)
|
||||
- Signed URL generation
|
||||
- File operations (delete, list, exists)
|
||||
"""
|
||||
import json
|
||||
import pytest
|
||||
from unittest.mock import Mock, MagicMock, patch
|
||||
from io import BytesIO
|
||||
|
||||
from backend.services.storage_service import StorageService
|
||||
|
||||
|
||||
class TestStorageServiceInit:
|
||||
"""Test StorageService initialization."""
|
||||
|
||||
@patch("backend.services.storage_service.storage.Client")
|
||||
@patch("backend.services.storage_service.settings")
|
||||
def test_init_creates_client_and_bucket(self, mock_settings, mock_client_class):
|
||||
"""Test initialization creates GCS client and gets bucket."""
|
||||
mock_settings.google_cloud_project = "test-project"
|
||||
mock_settings.gcs_bucket_name = "test-bucket"
|
||||
|
||||
mock_client = Mock()
|
||||
mock_bucket = Mock()
|
||||
mock_client.bucket.return_value = mock_bucket
|
||||
mock_client_class.return_value = mock_client
|
||||
|
||||
service = StorageService()
|
||||
|
||||
mock_client_class.assert_called_once_with(project="test-project")
|
||||
mock_client.bucket.assert_called_once_with("test-bucket")
|
||||
assert service.client == mock_client
|
||||
assert service.bucket == mock_bucket
|
||||
|
||||
|
||||
class TestStorageServiceUpload:
|
||||
"""Test upload operations."""
|
||||
|
||||
@patch("backend.services.storage_service.storage.Client")
|
||||
@patch("backend.services.storage_service.settings")
|
||||
def test_upload_file(self, mock_settings, mock_client_class):
|
||||
"""Test uploading a file to GCS."""
|
||||
mock_settings.google_cloud_project = "test-project"
|
||||
mock_settings.gcs_bucket_name = "test-bucket"
|
||||
|
||||
mock_blob = Mock()
|
||||
mock_bucket = Mock()
|
||||
mock_bucket.blob.return_value = mock_blob
|
||||
mock_client = Mock()
|
||||
mock_client.bucket.return_value = mock_bucket
|
||||
mock_client_class.return_value = mock_client
|
||||
|
||||
service = StorageService()
|
||||
result = service.upload_file("/local/path/file.flac", "uploads/job123/file.flac")
|
||||
|
||||
mock_bucket.blob.assert_called_once_with("uploads/job123/file.flac")
|
||||
mock_blob.upload_from_filename.assert_called_once_with("/local/path/file.flac")
|
||||
assert result == "uploads/job123/file.flac"
|
||||
|
||||
@patch("backend.services.storage_service.storage.Client")
|
||||
@patch("backend.services.storage_service.settings")
|
||||
def test_upload_file_raises_on_error(self, mock_settings, mock_client_class):
|
||||
"""Test upload_file raises exception on GCS error."""
|
||||
mock_settings.google_cloud_project = "test-project"
|
||||
mock_settings.gcs_bucket_name = "test-bucket"
|
||||
|
||||
mock_blob = Mock()
|
||||
mock_blob.upload_from_filename.side_effect = Exception("GCS error")
|
||||
mock_bucket = Mock()
|
||||
mock_bucket.blob.return_value = mock_blob
|
||||
mock_client = Mock()
|
||||
mock_client.bucket.return_value = mock_bucket
|
||||
mock_client_class.return_value = mock_client
|
||||
|
||||
service = StorageService()
|
||||
|
||||
with pytest.raises(Exception) as exc_info:
|
||||
service.upload_file("/local/path/file.flac", "dest/file.flac")
|
||||
|
||||
assert "GCS error" in str(exc_info.value)
|
||||
|
||||
@patch("backend.services.storage_service.storage.Client")
|
||||
@patch("backend.services.storage_service.settings")
|
||||
def test_upload_fileobj(self, mock_settings, mock_client_class):
|
||||
"""Test uploading a file object to GCS."""
|
||||
mock_settings.google_cloud_project = "test-project"
|
||||
mock_settings.gcs_bucket_name = "test-bucket"
|
||||
|
||||
mock_blob = Mock()
|
||||
mock_bucket = Mock()
|
||||
mock_bucket.blob.return_value = mock_blob
|
||||
mock_client = Mock()
|
||||
mock_client.bucket.return_value = mock_bucket
|
||||
mock_client_class.return_value = mock_client
|
||||
|
||||
service = StorageService()
|
||||
file_obj = BytesIO(b"test content")
|
||||
|
||||
result = service.upload_fileobj(file_obj, "uploads/test.txt", content_type="text/plain")
|
||||
|
||||
mock_bucket.blob.assert_called_once_with("uploads/test.txt")
|
||||
assert mock_blob.content_type == "text/plain"
|
||||
mock_blob.upload_from_file.assert_called_once_with(file_obj, rewind=True)
|
||||
assert result == "uploads/test.txt"
|
||||
|
||||
@patch("backend.services.storage_service.storage.Client")
|
||||
@patch("backend.services.storage_service.settings")
|
||||
def test_upload_json(self, mock_settings, mock_client_class):
|
||||
"""Test uploading JSON data to GCS."""
|
||||
mock_settings.google_cloud_project = "test-project"
|
||||
mock_settings.gcs_bucket_name = "test-bucket"
|
||||
|
||||
mock_blob = Mock()
|
||||
mock_bucket = Mock()
|
||||
mock_bucket.blob.return_value = mock_blob
|
||||
mock_client = Mock()
|
||||
mock_client.bucket.return_value = mock_bucket
|
||||
mock_client_class.return_value = mock_client
|
||||
|
||||
service = StorageService()
|
||||
data = {"key": "value", "nested": {"data": True}}
|
||||
|
||||
result = service.upload_json("data/config.json", data)
|
||||
|
||||
mock_bucket.blob.assert_called_once_with("data/config.json")
|
||||
assert mock_blob.content_type == "application/json"
|
||||
# Check the uploaded content
|
||||
call_args = mock_blob.upload_from_string.call_args
|
||||
uploaded_content = call_args[0][0]
|
||||
assert json.loads(uploaded_content) == data
|
||||
assert result == "data/config.json"
|
||||
|
||||
|
||||
class TestStorageServiceDownload:
|
||||
"""Test download operations."""
|
||||
|
||||
@patch("backend.services.storage_service.storage.Client")
|
||||
@patch("backend.services.storage_service.settings")
|
||||
def test_download_file(self, mock_settings, mock_client_class):
|
||||
"""Test downloading a file from GCS."""
|
||||
mock_settings.google_cloud_project = "test-project"
|
||||
mock_settings.gcs_bucket_name = "test-bucket"
|
||||
|
||||
mock_blob = Mock()
|
||||
mock_bucket = Mock()
|
||||
mock_bucket.blob.return_value = mock_blob
|
||||
mock_client = Mock()
|
||||
mock_client.bucket.return_value = mock_bucket
|
||||
mock_client_class.return_value = mock_client
|
||||
|
||||
service = StorageService()
|
||||
result = service.download_file("uploads/job123/file.flac", "/local/path/file.flac")
|
||||
|
||||
mock_bucket.blob.assert_called_once_with("uploads/job123/file.flac")
|
||||
mock_blob.download_to_filename.assert_called_once_with("/local/path/file.flac")
|
||||
assert result == "/local/path/file.flac"
|
||||
|
||||
@patch("backend.services.storage_service.storage.Client")
|
||||
@patch("backend.services.storage_service.settings")
|
||||
def test_download_file_raises_on_error(self, mock_settings, mock_client_class):
|
||||
"""Test download_file raises exception on GCS error."""
|
||||
mock_settings.google_cloud_project = "test-project"
|
||||
mock_settings.gcs_bucket_name = "test-bucket"
|
||||
|
||||
mock_blob = Mock()
|
||||
mock_blob.download_to_filename.side_effect = Exception("File not found")
|
||||
mock_bucket = Mock()
|
||||
mock_bucket.blob.return_value = mock_blob
|
||||
mock_client = Mock()
|
||||
mock_client.bucket.return_value = mock_bucket
|
||||
mock_client_class.return_value = mock_client
|
||||
|
||||
service = StorageService()
|
||||
|
||||
with pytest.raises(Exception) as exc_info:
|
||||
service.download_file("missing/file.flac", "/local/path")
|
||||
|
||||
assert "File not found" in str(exc_info.value)
|
||||
|
||||
@patch("backend.services.storage_service.storage.Client")
|
||||
@patch("backend.services.storage_service.settings")
|
||||
def test_download_json(self, mock_settings, mock_client_class):
|
||||
"""Test downloading and parsing JSON from GCS."""
|
||||
mock_settings.google_cloud_project = "test-project"
|
||||
mock_settings.gcs_bucket_name = "test-bucket"
|
||||
|
||||
mock_blob = Mock()
|
||||
mock_blob.download_as_text.return_value = '{"key": "value", "count": 42}'
|
||||
mock_bucket = Mock()
|
||||
mock_bucket.blob.return_value = mock_blob
|
||||
mock_client = Mock()
|
||||
mock_client.bucket.return_value = mock_bucket
|
||||
mock_client_class.return_value = mock_client
|
||||
|
||||
service = StorageService()
|
||||
result = service.download_json("data/config.json")
|
||||
|
||||
assert result == {"key": "value", "count": 42}
|
||||
|
||||
|
||||
class TestStorageServiceDelete:
|
||||
"""Test delete operations."""
|
||||
|
||||
@patch("backend.services.storage_service.storage.Client")
|
||||
@patch("backend.services.storage_service.settings")
|
||||
def test_delete_file(self, mock_settings, mock_client_class):
|
||||
"""Test deleting a single file from GCS."""
|
||||
mock_settings.google_cloud_project = "test-project"
|
||||
mock_settings.gcs_bucket_name = "test-bucket"
|
||||
|
||||
mock_blob = Mock()
|
||||
mock_bucket = Mock()
|
||||
mock_bucket.blob.return_value = mock_blob
|
||||
mock_client = Mock()
|
||||
mock_client.bucket.return_value = mock_bucket
|
||||
mock_client_class.return_value = mock_client
|
||||
|
||||
service = StorageService()
|
||||
service.delete_file("uploads/job123/file.flac")
|
||||
|
||||
mock_bucket.blob.assert_called_once_with("uploads/job123/file.flac")
|
||||
mock_blob.delete.assert_called_once()
|
||||
|
||||
@patch("backend.services.storage_service.storage.Client")
|
||||
@patch("backend.services.storage_service.settings")
|
||||
def test_delete_folder(self, mock_settings, mock_client_class):
|
||||
"""Test deleting all files with a prefix (folder)."""
|
||||
mock_settings.google_cloud_project = "test-project"
|
||||
mock_settings.gcs_bucket_name = "test-bucket"
|
||||
|
||||
# Create mock blobs
|
||||
mock_blob1 = Mock()
|
||||
mock_blob1.name = "uploads/job123/file1.flac"
|
||||
mock_blob2 = Mock()
|
||||
mock_blob2.name = "uploads/job123/file2.flac"
|
||||
mock_blob3 = Mock()
|
||||
mock_blob3.name = "uploads/job123/subdir/file3.flac"
|
||||
|
||||
mock_bucket = Mock()
|
||||
mock_bucket.list_blobs.return_value = [mock_blob1, mock_blob2, mock_blob3]
|
||||
mock_client = Mock()
|
||||
mock_client.bucket.return_value = mock_bucket
|
||||
mock_client_class.return_value = mock_client
|
||||
|
||||
service = StorageService()
|
||||
count = service.delete_folder("uploads/job123/")
|
||||
|
||||
mock_bucket.list_blobs.assert_called_once_with(prefix="uploads/job123/")
|
||||
mock_blob1.delete.assert_called_once()
|
||||
mock_blob2.delete.assert_called_once()
|
||||
mock_blob3.delete.assert_called_once()
|
||||
assert count == 3
|
||||
|
||||
@patch("backend.services.storage_service.storage.Client")
|
||||
@patch("backend.services.storage_service.settings")
|
||||
def test_delete_folder_handles_errors_gracefully(self, mock_settings, mock_client_class):
|
||||
"""Test delete_folder continues even if some deletes fail."""
|
||||
mock_settings.google_cloud_project = "test-project"
|
||||
mock_settings.gcs_bucket_name = "test-bucket"
|
||||
|
||||
mock_blob1 = Mock()
|
||||
mock_blob1.name = "file1.flac"
|
||||
mock_blob2 = Mock()
|
||||
mock_blob2.name = "file2.flac"
|
||||
mock_blob2.delete.side_effect = Exception("Permission denied")
|
||||
mock_blob3 = Mock()
|
||||
mock_blob3.name = "file3.flac"
|
||||
|
||||
mock_bucket = Mock()
|
||||
mock_bucket.list_blobs.return_value = [mock_blob1, mock_blob2, mock_blob3]
|
||||
mock_client = Mock()
|
||||
mock_client.bucket.return_value = mock_bucket
|
||||
mock_client_class.return_value = mock_client
|
||||
|
||||
service = StorageService()
|
||||
count = service.delete_folder("uploads/")
|
||||
|
||||
# Should have deleted 2 files (blob2 failed)
|
||||
assert count == 2
|
||||
|
||||
|
||||
class TestStorageServiceFileOperations:
|
||||
"""Test file listing and existence checks."""
|
||||
|
||||
@patch("backend.services.storage_service.storage.Client")
|
||||
@patch("backend.services.storage_service.settings")
|
||||
def test_list_files(self, mock_settings, mock_client_class):
|
||||
"""Test listing files with a prefix."""
|
||||
mock_settings.google_cloud_project = "test-project"
|
||||
mock_settings.gcs_bucket_name = "test-bucket"
|
||||
|
||||
mock_blob1 = Mock()
|
||||
mock_blob1.name = "uploads/job123/audio.flac"
|
||||
mock_blob2 = Mock()
|
||||
mock_blob2.name = "uploads/job123/lyrics.txt"
|
||||
|
||||
mock_bucket = Mock()
|
||||
mock_bucket.list_blobs.return_value = [mock_blob1, mock_blob2]
|
||||
mock_client = Mock()
|
||||
mock_client.bucket.return_value = mock_bucket
|
||||
mock_client_class.return_value = mock_client
|
||||
|
||||
service = StorageService()
|
||||
files = service.list_files("uploads/job123/")
|
||||
|
||||
assert files == ["uploads/job123/audio.flac", "uploads/job123/lyrics.txt"]
|
||||
|
||||
@patch("backend.services.storage_service.storage.Client")
|
||||
@patch("backend.services.storage_service.settings")
|
||||
def test_file_exists_returns_true(self, mock_settings, mock_client_class):
|
||||
"""Test file_exists returns True for existing file."""
|
||||
mock_settings.google_cloud_project = "test-project"
|
||||
mock_settings.gcs_bucket_name = "test-bucket"
|
||||
|
||||
mock_blob = Mock()
|
||||
mock_blob.exists.return_value = True
|
||||
mock_bucket = Mock()
|
||||
mock_bucket.blob.return_value = mock_blob
|
||||
mock_client = Mock()
|
||||
mock_client.bucket.return_value = mock_bucket
|
||||
mock_client_class.return_value = mock_client
|
||||
|
||||
service = StorageService()
|
||||
result = service.file_exists("uploads/job123/file.flac")
|
||||
|
||||
assert result is True
|
||||
|
||||
@patch("backend.services.storage_service.storage.Client")
|
||||
@patch("backend.services.storage_service.settings")
|
||||
def test_file_exists_returns_false(self, mock_settings, mock_client_class):
|
||||
"""Test file_exists returns False for missing file."""
|
||||
mock_settings.google_cloud_project = "test-project"
|
||||
mock_settings.gcs_bucket_name = "test-bucket"
|
||||
|
||||
mock_blob = Mock()
|
||||
mock_blob.exists.return_value = False
|
||||
mock_bucket = Mock()
|
||||
mock_bucket.blob.return_value = mock_blob
|
||||
mock_client = Mock()
|
||||
mock_client.bucket.return_value = mock_bucket
|
||||
mock_client_class.return_value = mock_client
|
||||
|
||||
service = StorageService()
|
||||
result = service.file_exists("missing/file.flac")
|
||||
|
||||
assert result is False
|
||||
|
||||
|
||||
class TestStorageServiceSignedUrls:
|
||||
"""Test signed URL generation."""
|
||||
|
||||
@patch("backend.services.storage_service.storage.Client")
|
||||
@patch("backend.services.storage_service.settings")
|
||||
def test_generate_signed_url(self, mock_settings, mock_client_class):
|
||||
"""Test generating a signed download URL."""
|
||||
mock_settings.google_cloud_project = "test-project"
|
||||
mock_settings.gcs_bucket_name = "test-bucket"
|
||||
|
||||
mock_blob = Mock()
|
||||
mock_blob.generate_signed_url.return_value = "https://storage.googleapis.com/signed-url"
|
||||
mock_bucket = Mock()
|
||||
mock_bucket.blob.return_value = mock_blob
|
||||
mock_client = Mock()
|
||||
mock_client.bucket.return_value = mock_bucket
|
||||
mock_client_class.return_value = mock_client
|
||||
|
||||
# Mock google.auth.default to return credentials without service_account_email
|
||||
with patch("google.auth.default") as mock_auth_default:
|
||||
mock_credentials = Mock(spec=[]) # No service_account_email attr
|
||||
mock_auth_default.return_value = (mock_credentials, "test-project")
|
||||
|
||||
service = StorageService()
|
||||
url = service.generate_signed_url("uploads/file.flac", expiration_minutes=30)
|
||||
|
||||
assert url == "https://storage.googleapis.com/signed-url"
|
||||
mock_blob.generate_signed_url.assert_called_once()
|
||||
call_kwargs = mock_blob.generate_signed_url.call_args.kwargs
|
||||
assert call_kwargs["method"] == "GET"
|
||||
assert call_kwargs["version"] == "v4"
|
||||
|
||||
@patch("backend.services.storage_service.storage.Client")
|
||||
@patch("backend.services.storage_service.settings")
|
||||
def test_generate_signed_upload_url(self, mock_settings, mock_client_class):
|
||||
"""Test generating a signed upload URL."""
|
||||
mock_settings.google_cloud_project = "test-project"
|
||||
mock_settings.gcs_bucket_name = "test-bucket"
|
||||
|
||||
mock_blob = Mock()
|
||||
mock_blob.generate_signed_url.return_value = "https://storage.googleapis.com/signed-upload-url"
|
||||
mock_bucket = Mock()
|
||||
mock_bucket.blob.return_value = mock_blob
|
||||
mock_client = Mock()
|
||||
mock_client.bucket.return_value = mock_bucket
|
||||
mock_client_class.return_value = mock_client
|
||||
|
||||
with patch("google.auth.default") as mock_auth_default:
|
||||
mock_credentials = Mock(spec=[])
|
||||
mock_auth_default.return_value = (mock_credentials, "test-project")
|
||||
|
||||
service = StorageService()
|
||||
url = service.generate_signed_upload_url(
|
||||
"uploads/file.flac",
|
||||
content_type="audio/flac",
|
||||
expiration_minutes=15
|
||||
)
|
||||
|
||||
assert url == "https://storage.googleapis.com/signed-upload-url"
|
||||
call_kwargs = mock_blob.generate_signed_url.call_args.kwargs
|
||||
assert call_kwargs["method"] == "PUT"
|
||||
assert call_kwargs["headers"] == {"Content-Type": "audio/flac"}
|
||||
|
||||
@patch("backend.services.storage_service.storage.Client")
|
||||
@patch("backend.services.storage_service.settings")
|
||||
def test_signed_url_with_service_account(self, mock_settings, mock_client_class):
|
||||
"""Test signed URL generation with service account credentials."""
|
||||
mock_settings.google_cloud_project = "test-project"
|
||||
mock_settings.gcs_bucket_name = "test-bucket"
|
||||
|
||||
mock_blob = Mock()
|
||||
mock_blob.generate_signed_url.return_value = "https://storage.googleapis.com/signed-url"
|
||||
mock_bucket = Mock()
|
||||
mock_bucket.blob.return_value = mock_blob
|
||||
mock_client = Mock()
|
||||
mock_client.bucket.return_value = mock_bucket
|
||||
mock_client_class.return_value = mock_client
|
||||
|
||||
with patch("google.auth.default") as mock_auth_default:
|
||||
with patch("google.auth.transport.requests.Request") as mock_request:
|
||||
# Mock credentials with service_account_email
|
||||
mock_credentials = Mock()
|
||||
mock_credentials.service_account_email = "sa@project.iam.gserviceaccount.com"
|
||||
mock_credentials.token = "access-token-123"
|
||||
mock_auth_default.return_value = (mock_credentials, "test-project")
|
||||
|
||||
service = StorageService()
|
||||
url = service.generate_signed_url("uploads/file.flac")
|
||||
|
||||
# Should have refreshed credentials and used IAM signing
|
||||
mock_credentials.refresh.assert_called_once()
|
||||
call_kwargs = mock_blob.generate_signed_url.call_args.kwargs
|
||||
assert call_kwargs["service_account_email"] == "sa@project.iam.gserviceaccount.com"
|
||||
assert call_kwargs["access_token"] == "access-token-123"
|
||||
|
||||
Loaded 100 of 871 files, more files were not shown because too many files have changed in this diff.
Show more
Reference in new issue
Block a user