mirror of
https://github.com/nomadkaraoke/karaoke-gen.git
synced 2026-11-20 21:17:56 +09:00
Compare commits
| Author | SHA1 | Date | |
|---|---|---|---|
|
|
7eb082d382 | ||
|
|
1af7ab0bde | ||
|
|
7ba40af80d | ||
|
|
f434e3c93d | ||
|
|
4b6523dbd8 | ||
|
|
d67828b78f | ||
|
|
db55205127 | ||
|
|
01ff0eb042 | ||
|
|
438ea6a836 | ||
|
|
07f48b6213 | ||
|
|
5630cfd253 | ||
|
|
f543ea6efa | ||
|
|
5a2c77978b | ||
|
|
0f7ec8f71e | ||
|
|
dbb6f3b751 | ||
|
|
dd287f8c92 | ||
|
|
07c459258a | ||
|
|
3522fdd223 | ||
|
|
441f99edeb | ||
|
|
2e145dd927 | ||
|
|
985f5c7718 | ||
|
|
2b5ef28789 | ||
|
|
76940691b1 | ||
|
|
4b346dd369 | ||
|
|
7480950a1b | ||
|
|
64ada25e9c | ||
|
|
de6322dd87 | ||
|
|
49b472429d | ||
|
|
d57282c770 | ||
|
|
a41948fe37 | ||
|
|
ffd4e9e377 | ||
|
|
ef9c4fc243 | ||
|
|
1f030333ae | ||
|
|
707df6af72 |
No files matched your search
@@ -0,0 +1,368 @@
|
||||
---
|
||||
name: Anchor Sequence Optimization
|
||||
overview: Replace the current multiprocessing-based anchor sequence algorithm with an efficient single-threaded implementation using hash-based n-gram indexing, greedy longest-first matching, and simplified scoring to achieve sub-30-second performance in both local and cloud environments.
|
||||
todos:
|
||||
- id: create-golden-fixtures
|
||||
content: Create script to generate 10 golden test fixtures from representative song data (varying lengths/complexity), run current algorithm to capture expected anchor outputs
|
||||
status: completed
|
||||
- id: create-golden-test
|
||||
content: Create test_anchor_sequence_golden.py with regression tests that verify anchor finder output matches fixtures, ensure tests pass with current implementation before any changes
|
||||
status: completed
|
||||
- id: remove-multiprocessing
|
||||
content: Remove multiprocessing.Pool from find_anchors() and _remove_overlapping_sequences(), replace with single-threaded loops
|
||||
status: completed
|
||||
- id: build-ngram-index
|
||||
content: Add _build_ngram_index() method to create hash-based lookup table for O(1) n-gram matching
|
||||
status: completed
|
||||
- id: greedy-skip
|
||||
content: Add covered_positions set tracking and skip logic to avoid processing overlapping n-grams
|
||||
status: completed
|
||||
- id: simple-scoring
|
||||
content: Add _simple_score_anchor() with rule-based heuristics, make spaCy scoring optional via use_nlp_scoring flag
|
||||
status: completed
|
||||
- id: optimize-overlap
|
||||
content: Replace O(n^2) overlap removal with O(n log n) sorted single-pass algorithm
|
||||
status: completed
|
||||
- id: add-perf-tests
|
||||
content: Add performance benchmark test asserting <30 second completion for all fixture sizes
|
||||
status: pending
|
||||
---
|
||||
|
||||
# Anchor Sequence Algorithm Optimization
|
||||
|
||||
## Problem Analysis
|
||||
|
||||
The current `AnchorSequenceFinder` has critical performance issues:
|
||||
|
||||
### Root Causes of Hanging
|
||||
|
||||
1. **Multiprocessing Pool with fork()**: Python's `Pool()` uses `fork()` on Linux, which causes deadlocks in Cloud Run containers
|
||||
2. **spaCy model loading in forked processes**: The `PhraseAnalyzer` loads spaCy's `en_core_web_sm` model, which can hang when forked
|
||||
3. **Pool creation blocking**: The hang occurs before job submission, so timeout checks never trigger
|
||||
|
||||
### Algorithmic Inefficiencies
|
||||
|
||||
1. **O(n^2) n-gram matching**: Linear search through references for each n-gram
|
||||
2. **Redundant work**: Processes all n-gram lengths (3 to ~30) even though longer matches are preferred
|
||||
3. **O(n^2) overlap removal**: Each anchor checked against all previous anchors
|
||||
4. **Expensive phrase scoring**: spaCy NLP called twice per anchor (phrase + context)
|
||||
|
||||
## Proposed Solution: Hash-Based Greedy Algorithm
|
||||
|
||||
### Core Algorithm Changes
|
||||
|
||||
**1. Hash-Based N-gram Index (O(1) lookup instead of O(n))**
|
||||
|
||||
- Pre-build a dictionary mapping n-grams to their positions in each reference
|
||||
- Use tuple of cleaned words as key for O(1) lookup
|
||||
- Build index once during preprocessing
|
||||
|
||||
**2. Greedy Longest-First Matching**
|
||||
|
||||
- Process n-gram lengths from longest to shortest (current behavior)
|
||||
- BUT: Skip positions already covered by longer anchors immediately
|
||||
- Use a "covered positions" set to avoid redundant work
|
||||
|
||||
**3. Single-Threaded Processing**
|
||||
|
||||
- Remove all `multiprocessing.Pool` usage
|
||||
- Use efficient algorithms instead of parallelism
|
||||
- Works identically in local and cloud environments
|
||||
|
||||
**4. Simplified Phrase Scoring (Optional)**
|
||||
|
||||
- Make spaCy-based scoring optional via configuration flag
|
||||
- Default to rule-based heuristic scoring:
|
||||
- Prefer sequences starting/ending at word boundaries
|
||||
- Prefer sequences matching more reference sources
|
||||
- Prefer longer sequences
|
||||
- Option to enable full NLP scoring for higher-quality results offline
|
||||
|
||||
**5. O(n log n) Overlap Removal**
|
||||
|
||||
- Sort anchors by position
|
||||
- Single pass with interval tracking using sorted list
|
||||
- Skip anchors that overlap with any already-selected anchor
|
||||
|
||||
### New Data Structures
|
||||
|
||||
```python
|
||||
# N-gram index: maps (word1, word2, ...) -> {source: [positions]}
|
||||
ngram_index: Dict[Tuple[str, ...], Dict[str, List[int]]]
|
||||
|
||||
# Covered positions in transcription (for greedy skip)
|
||||
covered_trans_positions: Set[int]
|
||||
```
|
||||
|
||||
### Performance Targets
|
||||
|
||||
- **Target**: Complete in under 30 seconds for any song
|
||||
- **Typical song**: 200-400 words, 3 reference sources
|
||||
- **Expected performance**: 1-5 seconds for typical inputs
|
||||
|
||||
## Implementation Plan
|
||||
|
||||
### Execution Order (Critical)
|
||||
|
||||
1. **Phase 0**: Create golden test fixtures and verify tests pass with current code
|
||||
2. **Phase 1-5**: Implement optimizations incrementally, running golden tests after each phase
|
||||
3. **Final**: Add performance benchmarks
|
||||
|
||||
### Files to Create/Modify
|
||||
|
||||
- `lyrics_transcriber_temp/scripts/generate_anchor_fixtures.py` (new)
|
||||
- `lyrics_transcriber_temp/tests/fixtures/anchor_golden/*.json` (new)
|
||||
- `lyrics_transcriber_temp/tests/unit/correction/test_anchor_sequence_golden.py` (new)
|
||||
- `lyrics_transcriber_temp/lyrics_transcriber/correction/anchor_sequence.py` (modify)
|
||||
|
||||
### Key Changes
|
||||
|
||||
**Phase 0: Golden Test Setup** (MUST complete and pass before any algorithm changes)
|
||||
|
||||
- Create fixture generation script with 10 representative test cases
|
||||
- Run script to generate fixtures using current implementation
|
||||
- Create golden test file that loads fixtures and verifies outputs
|
||||
- Verify all golden tests pass with current code
|
||||
|
||||
**Phase 1: Remove Multiprocessing**
|
||||
|
||||
- Replace `Pool().apply_async()` with direct function calls
|
||||
- Remove all pool creation, cleanup, and timeout handling
|
||||
- Single-threaded loop through n-gram lengths
|
||||
|
||||
**Phase 2: Add Hash-Based Index**
|
||||
|
||||
- New method `_build_ngram_index()` that builds lookup table during preprocessing
|
||||
- Modify `_find_matching_sources()` to use O(1) index lookup
|
||||
- Index built once, shared across all n-gram length iterations
|
||||
|
||||
**Phase 3: Greedy Position Skipping**
|
||||
|
||||
- Track covered transcription positions in a set
|
||||
- Skip n-grams that start in already-covered positions
|
||||
- Reduces redundant work significantly for songs with good matches
|
||||
|
||||
**Phase 4: Simplify Scoring**
|
||||
|
||||
- Add `use_nlp_scoring: bool = False` parameter to constructor
|
||||
- New method `_simple_score_anchor()` using rule-based heuristics
|
||||
- Keep spaCy scoring available for offline/high-quality mode
|
||||
|
||||
**Phase 5: Optimize Overlap Removal**
|
||||
|
||||
- Sort anchors by (start_position, -length)
|
||||
- Single pass: keep anchor if no overlap with last kept anchor
|
||||
- O(n log n) total instead of O(n^2)
|
||||
|
||||
## API Compatibility
|
||||
|
||||
The public interface remains unchanged:
|
||||
|
||||
- `find_anchors(transcribed, references, transcription_result) -> List[ScoredAnchor]`
|
||||
- `find_gaps(transcribed, anchors, references, transcription_result) -> List[GapSequence]`
|
||||
|
||||
New optional constructor parameters:
|
||||
|
||||
- `use_nlp_scoring: bool = False` - Enable spaCy-based phrase scoring (slower but higher quality)
|
||||
- Timeout parameter retained but less critical with efficient algorithm
|
||||
|
||||
## Fallback Strategy
|
||||
|
||||
If performance is still insufficient for edge cases:
|
||||
|
||||
1. **Early termination**: Stop after finding sufficient coverage (e.g., 80% of words anchored)
|
||||
2. **Adaptive n-gram range**: Start with medium lengths (8-15), expand only if needed
|
||||
3. **Sampling**: For very long songs, sample n-gram positions instead of exhaustive search
|
||||
|
||||
## Phase 0: Golden Test Setup (BEFORE any algorithm changes)
|
||||
|
||||
### Goal
|
||||
|
||||
Create regression tests using REAL song data from local cache to verify the optimized algorithm produces equivalent results.
|
||||
|
||||
### Cache File Structure (Understanding)
|
||||
|
||||
The local cache at `~/lyrics-transcriber-cache/` contains:
|
||||
|
||||
- **Reference lyrics**: `{provider}_{hash}_converted.json` where provider is spotify/genius/lrclib/musixmatch
|
||||
- Hash is based on (artist, title) lookup - same song shares same hash across providers
|
||||
- `_raw.json` contains metadata (track name, artist)
|
||||
- `_converted.json` contains LyricsData structure with segments/words
|
||||
- **Transcriptions**: `audioshake_{hash}_converted.json`
|
||||
- Hash is based on audio file content (different from reference hash)
|
||||
- Contains TranscriptionData structure
|
||||
- **Anchor results**: `anchors_{hash}.json`
|
||||
- Hash is computed from combined inputs (transcription + all references)
|
||||
|
||||
### Manual Song Selection
|
||||
|
||||
Identify 10 songs from cache with good coverage. Search raw files for track names:
|
||||
|
||||
### Finding Song Mappings
|
||||
|
||||
To link audioshake transcriptions to reference lyrics, search cache files:
|
||||
|
||||
```bash
|
||||
# Find songs by searching raw files for track names
|
||||
grep -h "trackName" ~/lyrics-transcriber-cache/spotify_*_raw.json | head -20
|
||||
|
||||
# Find audioshake files for a known song (e.g., search for filename patterns)
|
||||
ls ~/lyrics-transcriber-cache/audioshake_*_raw.json | head -20
|
||||
|
||||
# Search converted files for specific lyrics text
|
||||
grep -l "Napoleon did surrender" ~/lyrics-transcriber-cache/*_converted.json
|
||||
```
|
||||
|
||||
**Target song list** (reference_hash to be paired with audioshake_hash):
|
||||
|
||||
1. ABBA - Waterloo (refs: 3365f6a1e42388d84bcbfc0ac6f8aa6c) - short, repetitive chorus
|
||||
2. The Format - Time Bomb (refs: 3a3802ec28c4ab6689909b7ef784bb6f) - medium length
|
||||
3. Rancid - Time Bomb (refs: d02d20b854cbc6b890f274bd6c278752) - fast punk lyrics
|
||||
4. Minutemen - Storm in My House (refs: bc625704b26b307d8680ae5235b74994) - short
|
||||
5. + 6 more songs to be identified during fixture generation by exploring cache
|
||||
|
||||
### Fixture Generation Script
|
||||
|
||||
Create `lyrics_transcriber_temp/scripts/generate_anchor_fixtures.py`:
|
||||
|
||||
```python
|
||||
# Input: List of (name, audioshake_hash, reference_hash) tuples
|
||||
# For each:
|
||||
# 1. Load audioshake_{hash}_converted.json as transcription
|
||||
# 2. Load {spotify,genius,lrclib}_{hash}_converted.json as references
|
||||
# 3. Run current AnchorSequenceFinder
|
||||
# 4. Save complete fixture with inputs + outputs
|
||||
```
|
||||
|
||||
The script will:
|
||||
|
||||
1. Accept a config file listing the 10 song mappings
|
||||
2. Load real LyricsData from cache files
|
||||
3. Run the CURRENT AnchorSequenceFinder implementation
|
||||
4. Save fixtures that capture the real behavior
|
||||
|
||||
### Fixture Format
|
||||
|
||||
```
|
||||
lyrics_transcriber_temp/tests/fixtures/anchor_golden/
|
||||
- waterloo_abba.json
|
||||
- time_bomb_format.json
|
||||
- time_bomb_rancid.json
|
||||
- ...
|
||||
```
|
||||
|
||||
Each fixture contains REAL data:
|
||||
|
||||
```json
|
||||
{
|
||||
"name": "waterloo_abba",
|
||||
"description": "ABBA - Waterloo, short song with repetitive chorus",
|
||||
"source_hashes": {
|
||||
"audioshake": "1654b7c0755fad7f23bc888308f274de",
|
||||
"references": "3365f6a1e42388d84bcbfc0ac6f8aa6c"
|
||||
},
|
||||
"transcription": {
|
||||
"segments": [...], // Full LyricsData from audioshake cache
|
||||
"word_count": 150
|
||||
},
|
||||
"references": {
|
||||
"spotify": { "segments": [...] },
|
||||
"genius": { "segments": [...] },
|
||||
"lrclib": { "segments": [...] }
|
||||
},
|
||||
"expected_anchors": [
|
||||
{
|
||||
"text": "My my at Waterloo Napoleon did surrender",
|
||||
"transcription_position": 0,
|
||||
"length": 7,
|
||||
"reference_positions": {"spotify": 0, "genius": 0, "lrclib": 0},
|
||||
"confidence": 1.0
|
||||
},
|
||||
// ... all anchors from current implementation
|
||||
],
|
||||
"expected_anchor_count": 25,
|
||||
"expected_word_coverage": 142,
|
||||
"expected_coverage_percent": 94.6
|
||||
}
|
||||
```
|
||||
|
||||
### Golden Test
|
||||
|
||||
Create `lyrics_transcriber_temp/tests/unit/correction/test_anchor_sequence_golden.py`:
|
||||
|
||||
```python
|
||||
import pytest
|
||||
import json
|
||||
from pathlib import Path
|
||||
|
||||
FIXTURES_DIR = Path(__file__).parent.parent.parent / "fixtures" / "anchor_golden"
|
||||
|
||||
@pytest.fixture(params=list(FIXTURES_DIR.glob("*.json")))
|
||||
def golden_fixture(request):
|
||||
with open(request.param) as f:
|
||||
return json.load(f)
|
||||
|
||||
def test_anchor_count_matches(golden_fixture, anchor_finder):
|
||||
"""Anchor count should be within 10% of expected."""
|
||||
anchors = run_anchor_finder(golden_fixture)
|
||||
expected = golden_fixture["expected_anchor_count"]
|
||||
assert abs(len(anchors) - expected) <= expected * 0.1
|
||||
|
||||
def test_key_anchors_match(golden_fixture, anchor_finder):
|
||||
"""First, last, and longest anchors should match exactly."""
|
||||
anchors = run_anchor_finder(golden_fixture)
|
||||
expected = golden_fixture["expected_anchors"]
|
||||
# Compare first anchor
|
||||
assert anchors[0].text == expected[0]["text"]
|
||||
# Compare positions match
|
||||
...
|
||||
|
||||
def test_word_coverage(golden_fixture, anchor_finder):
|
||||
"""Total word coverage should be within 5% of expected."""
|
||||
anchors = run_anchor_finder(golden_fixture)
|
||||
coverage = calculate_coverage(anchors, golden_fixture)
|
||||
expected = golden_fixture["expected_coverage_percent"]
|
||||
assert abs(coverage - expected) <= 5.0
|
||||
|
||||
def test_no_invalid_positions(golden_fixture, anchor_finder):
|
||||
"""All anchor positions should be valid."""
|
||||
anchors = run_anchor_finder(golden_fixture)
|
||||
word_count = golden_fixture["transcription"]["word_count"]
|
||||
for anchor in anchors:
|
||||
assert 0 <= anchor.transcription_position < word_count
|
||||
```
|
||||
|
||||
### Validation Steps
|
||||
|
||||
1. **Generate fixtures from current implementation**:
|
||||
```bash
|
||||
cd lyrics_transcriber_temp
|
||||
python scripts/generate_anchor_fixtures.py
|
||||
```
|
||||
|
||||
2. **Verify fixtures were created**:
|
||||
```bash
|
||||
ls tests/fixtures/anchor_golden/
|
||||
# Should show 10 .json files
|
||||
```
|
||||
|
||||
3. **Run golden tests with CURRENT implementation**:
|
||||
```bash
|
||||
pytest tests/unit/correction/test_anchor_sequence_golden.py -v
|
||||
```
|
||||
|
||||
4. **All tests MUST pass before any algorithm changes**
|
||||
|
||||
5. **After each optimization phase, re-run golden tests**:
|
||||
```bash
|
||||
pytest tests/unit/correction/test_anchor_sequence_golden.py -v
|
||||
# Must still pass!
|
||||
```
|
||||
|
||||
|
||||
## Testing (Post-Optimization)
|
||||
|
||||
- Golden tests verify correctness (same outputs as before)
|
||||
- Existing unit tests verify API compatibility
|
||||
- Performance benchmark test with 30-second timeout assertion
|
||||
- Test with various song lengths via golden fixtures
|
||||
File renamed without changes.
@@ -1,35 +1,167 @@
|
||||
name: Backend Tests
|
||||
name: CI
|
||||
|
||||
on:
|
||||
push: # Run on all branch pushes
|
||||
push:
|
||||
pull_request:
|
||||
branches: [ main ]
|
||||
branches: [main]
|
||||
workflow_dispatch:
|
||||
|
||||
permissions:
|
||||
contents: write
|
||||
pages: write
|
||||
id-token: write
|
||||
|
||||
concurrency:
|
||||
group: "ci-${{ github.ref }}"
|
||||
cancel-in-progress: true
|
||||
|
||||
jobs:
|
||||
unit-tests:
|
||||
name: Unit Tests
|
||||
# ============================================
|
||||
# PACKAGE TESTS (karaoke-gen Python CLI)
|
||||
# ============================================
|
||||
|
||||
package-unit-tests:
|
||||
name: "Package - Unit Tests (70% coverage)"
|
||||
runs-on: ubuntu-latest
|
||||
|
||||
steps:
|
||||
- name: Checkout code
|
||||
uses: actions/checkout@v4
|
||||
with:
|
||||
submodules: false
|
||||
|
||||
- uses: actions/checkout@v4
|
||||
|
||||
- name: Set up Python 3.13
|
||||
uses: actions/setup-python@v5
|
||||
with:
|
||||
python-version: '3.13'
|
||||
|
||||
python-version: "3.13"
|
||||
|
||||
- name: Install Poetry
|
||||
uses: snok/install-poetry@v1
|
||||
with:
|
||||
virtualenvs-create: true
|
||||
virtualenvs-in-project: true
|
||||
|
||||
|
||||
- name: Install dependencies
|
||||
run: poetry install --with dev
|
||||
|
||||
- name: Install system dependencies
|
||||
run: |
|
||||
sudo apt-get update
|
||||
sudo apt-get install -y ffmpeg
|
||||
|
||||
- name: Install spaCy language models
|
||||
run: poetry run python -m spacy download en_core_web_sm
|
||||
|
||||
- name: Run unit tests
|
||||
run: |
|
||||
poetry run pytest tests/unit/ -v --cov=karaoke_gen --cov-report=xml --cov-report=term-missing --cov-fail-under=70
|
||||
|
||||
- name: Upload coverage reports
|
||||
uses: codecov/codecov-action@v5
|
||||
with:
|
||||
token: ${{ secrets.CODECOV_TOKEN }}
|
||||
files: ./coverage.xml
|
||||
flags: unittests
|
||||
name: codecov-umbrella
|
||||
|
||||
package-integration-tests:
|
||||
name: "Package - Integration Tests"
|
||||
runs-on: ubuntu-latest
|
||||
steps:
|
||||
- uses: actions/checkout@v4
|
||||
|
||||
- name: Set up Python 3.13
|
||||
uses: actions/setup-python@v5
|
||||
with:
|
||||
python-version: "3.13"
|
||||
|
||||
- name: Install Poetry
|
||||
uses: snok/install-poetry@v1
|
||||
with:
|
||||
virtualenvs-create: true
|
||||
virtualenvs-in-project: true
|
||||
|
||||
- name: Install dependencies
|
||||
run: poetry install --with dev
|
||||
|
||||
- name: Install system dependencies
|
||||
run: |
|
||||
sudo apt-get update
|
||||
sudo apt-get install -y ffmpeg
|
||||
|
||||
- name: Install spaCy language models
|
||||
run: poetry run python -m spacy download en_core_web_sm
|
||||
|
||||
- name: Run integration tests
|
||||
run: poetry run pytest tests/integration/ -v
|
||||
|
||||
package-build:
|
||||
name: "Package - Build & Install"
|
||||
runs-on: ubuntu-latest
|
||||
needs: [package-unit-tests, package-integration-tests]
|
||||
steps:
|
||||
- uses: actions/checkout@v4
|
||||
|
||||
- name: Set up Python 3.13
|
||||
uses: actions/setup-python@v5
|
||||
with:
|
||||
python-version: "3.13"
|
||||
|
||||
- name: Install Poetry
|
||||
uses: snok/install-poetry@v1
|
||||
with:
|
||||
virtualenvs-create: true
|
||||
virtualenvs-in-project: true
|
||||
|
||||
- name: Build package
|
||||
run: poetry build
|
||||
|
||||
- name: Free disk space
|
||||
uses: jlumbroso/free-disk-space@main
|
||||
with:
|
||||
tool-cache: false
|
||||
android: true
|
||||
haskell: true
|
||||
dotnet: false
|
||||
large-packages: false
|
||||
docker-images: false
|
||||
swap-storage: false
|
||||
|
||||
- name: Test package installation
|
||||
run: |
|
||||
pip install dist/*.whl
|
||||
python -m karaoke_gen.utils.gen_cli --help
|
||||
|
||||
- name: Upload build artifacts
|
||||
uses: actions/upload-artifact@v4
|
||||
with:
|
||||
name: package-dist
|
||||
path: dist/
|
||||
retention-days: 1
|
||||
|
||||
# ============================================
|
||||
# BACKEND TESTS (Cloud Run API service)
|
||||
# ============================================
|
||||
|
||||
backend-unit-tests:
|
||||
name: "Backend - Unit Tests"
|
||||
runs-on: ubuntu-latest
|
||||
steps:
|
||||
- name: Checkout code
|
||||
uses: actions/checkout@v4
|
||||
with:
|
||||
submodules: false
|
||||
|
||||
- name: Set up Python 3.13
|
||||
uses: actions/setup-python@v5
|
||||
with:
|
||||
python-version: "3.13"
|
||||
|
||||
- name: Install Poetry
|
||||
uses: snok/install-poetry@v1
|
||||
with:
|
||||
virtualenvs-create: true
|
||||
virtualenvs-in-project: true
|
||||
|
||||
- name: Install dependencies
|
||||
run: poetry install
|
||||
|
||||
|
||||
- name: Run unit tests
|
||||
env:
|
||||
GOOGLE_CLOUD_PROJECT: test-project
|
||||
@@ -43,11 +175,10 @@ jobs:
|
||||
backend/tests/test_file_upload.py \
|
||||
backend/tests/test_services.py \
|
||||
-v --tb=short
|
||||
|
||||
emulator-tests:
|
||||
name: Emulator Integration Tests
|
||||
|
||||
backend-emulator-tests:
|
||||
name: "Backend - Emulator Integration Tests"
|
||||
runs-on: ubuntu-latest
|
||||
|
||||
steps:
|
||||
- name: Free disk space
|
||||
run: |
|
||||
@@ -55,144 +186,250 @@ jobs:
|
||||
sudo rm -rf /opt/ghc
|
||||
sudo rm -rf /usr/local/share/boost
|
||||
df -h
|
||||
|
||||
|
||||
- name: Checkout code
|
||||
uses: actions/checkout@v4
|
||||
with:
|
||||
submodules: false
|
||||
|
||||
|
||||
- name: Set up Python 3.13
|
||||
uses: actions/setup-python@v5
|
||||
with:
|
||||
python-version: '3.13'
|
||||
|
||||
python-version: "3.13"
|
||||
|
||||
- name: Install Poetry
|
||||
uses: snok/install-poetry@v1
|
||||
with:
|
||||
virtualenvs-create: true
|
||||
virtualenvs-in-project: true
|
||||
|
||||
|
||||
- name: Set up Java 21 (required by Firestore emulator)
|
||||
uses: actions/setup-java@v4
|
||||
with:
|
||||
distribution: 'temurin'
|
||||
java-version: '21'
|
||||
|
||||
distribution: "temurin"
|
||||
java-version: "21"
|
||||
|
||||
- name: Set up Cloud SDK
|
||||
uses: google-github-actions/setup-gcloud@v2
|
||||
with:
|
||||
version: 'latest'
|
||||
|
||||
version: "latest"
|
||||
|
||||
- name: Install Firestore emulator
|
||||
run: |
|
||||
gcloud components install beta cloud-firestore-emulator --quiet
|
||||
|
||||
run: gcloud components install beta cloud-firestore-emulator --quiet
|
||||
|
||||
- name: Install dependencies
|
||||
run: poetry install
|
||||
|
||||
|
||||
- name: Start Docker for GCS emulator
|
||||
run: |
|
||||
docker pull fsouza/fake-gcs-server:latest
|
||||
|
||||
run: docker pull fsouza/fake-gcs-server:latest
|
||||
|
||||
- name: Run emulator integration tests
|
||||
run: |
|
||||
chmod +x scripts/*.sh
|
||||
./scripts/run-emulator-tests.sh
|
||||
|
||||
|
||||
- name: Upload test results
|
||||
if: always()
|
||||
uses: actions/upload-artifact@v4
|
||||
with:
|
||||
name: test-results
|
||||
name: backend-test-results
|
||||
path: backend/htmlcov/
|
||||
retention-days: 7
|
||||
|
||||
lint:
|
||||
name: Code Quality
|
||||
backend-lint:
|
||||
name: "Backend - Code Quality"
|
||||
runs-on: ubuntu-latest
|
||||
|
||||
steps:
|
||||
- name: Checkout code
|
||||
uses: actions/checkout@v4
|
||||
with:
|
||||
submodules: false
|
||||
|
||||
|
||||
- name: Set up Python 3.13
|
||||
uses: actions/setup-python@v5
|
||||
with:
|
||||
python-version: '3.13'
|
||||
|
||||
python-version: "3.13"
|
||||
|
||||
- name: Install Poetry
|
||||
uses: snok/install-poetry@v1
|
||||
with:
|
||||
virtualenvs-create: true
|
||||
virtualenvs-in-project: true
|
||||
|
||||
|
||||
- name: Install dependencies
|
||||
run: poetry install
|
||||
|
||||
|
||||
- name: Run validation (syntax check only)
|
||||
run: |
|
||||
# Skip full validation as it requires GCP credentials
|
||||
# Just check Python syntax and imports
|
||||
poetry run python -m py_compile backend/main.py backend/config.py
|
||||
echo "✅ Basic syntax validation passed"
|
||||
|
||||
deploy:
|
||||
name: Deploy via Cloud Build
|
||||
|
||||
# ============================================
|
||||
# DEPLOYMENTS (main branch only)
|
||||
# ============================================
|
||||
|
||||
package-publish:
|
||||
name: "Deploy - Publish to PyPI"
|
||||
runs-on: ubuntu-latest
|
||||
needs: [unit-tests, emulator-tests, lint]
|
||||
# Deploy on any push where tests pass (pre-alpha: rapid iteration mode)
|
||||
# TODO: Change back to main-only when ready for production: github.ref == 'refs/heads/main'
|
||||
if: github.event_name == 'push'
|
||||
|
||||
needs: [package-build]
|
||||
if: github.ref == 'refs/heads/main' && github.event_name == 'push'
|
||||
steps:
|
||||
- uses: actions/checkout@v4
|
||||
|
||||
- name: Set up Python 3.13
|
||||
uses: actions/setup-python@v5
|
||||
with:
|
||||
python-version: "3.13"
|
||||
|
||||
- name: Install Poetry
|
||||
uses: snok/install-poetry@v1
|
||||
|
||||
- name: Download build artifacts
|
||||
uses: actions/download-artifact@v4
|
||||
with:
|
||||
name: package-dist
|
||||
path: dist/
|
||||
|
||||
- name: Extract version
|
||||
id: extract_version
|
||||
run: |
|
||||
VERSION=$(poetry version --short)
|
||||
echo "version=$VERSION" >> $GITHUB_OUTPUT
|
||||
echo "📋 Extracted version: $VERSION"
|
||||
|
||||
- name: Check if tag exists
|
||||
id: check_tag
|
||||
run: |
|
||||
VERSION="${{ steps.extract_version.outputs.version }}"
|
||||
if git ls-remote --tags origin | grep -q "refs/tags/v$VERSION$"; then
|
||||
echo "tag_exists=true" >> $GITHUB_OUTPUT
|
||||
echo "🏷️ Tag v$VERSION already exists, skipping release"
|
||||
else
|
||||
echo "tag_exists=false" >> $GITHUB_OUTPUT
|
||||
echo "✨ Tag v$VERSION does not exist, proceeding with release"
|
||||
fi
|
||||
|
||||
- name: Publish to PyPI
|
||||
if: steps.check_tag.outputs.tag_exists == 'false'
|
||||
run: |
|
||||
echo "📦 Publishing the following files to PyPI:"
|
||||
ls -la dist/
|
||||
echo "🚀 Publishing to PyPI..."
|
||||
poetry config pypi-token.pypi ${{ secrets.PYPI_API_TOKEN }}
|
||||
poetry publish
|
||||
|
||||
- name: Create Git Tag
|
||||
if: steps.check_tag.outputs.tag_exists == 'false'
|
||||
run: |
|
||||
git config user.name github-actions
|
||||
git config user.email github-actions@github.com
|
||||
git tag -a "v${{ steps.extract_version.outputs.version }}" -m "Release v${{ steps.extract_version.outputs.version }}"
|
||||
git push https://${{ secrets.GITHUB_TOKEN }}@github.com/${{ github.repository }} "v${{ steps.extract_version.outputs.version }}"
|
||||
|
||||
- name: Create GitHub Release
|
||||
if: steps.check_tag.outputs.tag_exists == 'false'
|
||||
uses: softprops/action-gh-release@v1
|
||||
with:
|
||||
tag_name: "v${{ steps.extract_version.outputs.version }}"
|
||||
name: "v${{ steps.extract_version.outputs.version }}"
|
||||
body: |
|
||||
## Changes in v${{ steps.extract_version.outputs.version }}
|
||||
|
||||
This release was automatically created from the main branch.
|
||||
|
||||
### Installation
|
||||
```bash
|
||||
pip install karaoke-gen==${{ steps.extract_version.outputs.version }}
|
||||
```
|
||||
|
||||
### Assets
|
||||
- Python package published to PyPI
|
||||
- Frontend deployed to GitHub Pages at gen.nomadkaraoke.com
|
||||
|
||||
For detailed changes, see the commit history.
|
||||
draft: false
|
||||
prerelease: false
|
||||
files: |
|
||||
dist/*.whl
|
||||
dist/*.tar.gz
|
||||
|
||||
deploy-frontend:
|
||||
name: "Deploy - Frontend (GitHub Pages)"
|
||||
runs-on: ubuntu-latest
|
||||
if: github.ref == 'refs/heads/main' && github.event_name == 'push'
|
||||
environment:
|
||||
name: github-pages
|
||||
url: ${{ steps.deployment.outputs.page_url }}
|
||||
steps:
|
||||
- name: Checkout
|
||||
uses: actions/checkout@v4
|
||||
|
||||
- name: Setup Pages
|
||||
uses: actions/configure-pages@v4
|
||||
|
||||
- name: Prepare frontend files
|
||||
run: |
|
||||
echo "📁 Preparing frontend files for deployment..."
|
||||
mkdir -p _site
|
||||
cp -r frontend/* _site/
|
||||
echo "✅ Frontend files copied to _site/"
|
||||
ls -la _site/
|
||||
|
||||
- name: Upload artifact
|
||||
uses: actions/upload-pages-artifact@v3
|
||||
with:
|
||||
path: "_site"
|
||||
|
||||
- name: Deploy to GitHub Pages
|
||||
id: deployment
|
||||
uses: actions/deploy-pages@v4
|
||||
|
||||
deploy-backend:
|
||||
name: "Deploy - Backend (Cloud Run)"
|
||||
runs-on: ubuntu-latest
|
||||
needs: [backend-unit-tests, backend-emulator-tests, backend-lint]
|
||||
if: github.ref == 'refs/heads/main' && github.event_name == 'push'
|
||||
permissions:
|
||||
contents: read
|
||||
id-token: write # Required for Workload Identity Federation
|
||||
|
||||
id-token: write
|
||||
steps:
|
||||
- name: Checkout code
|
||||
uses: actions/checkout@v4
|
||||
with:
|
||||
submodules: false
|
||||
|
||||
|
||||
- name: Authenticate to Google Cloud (Workload Identity)
|
||||
id: auth
|
||||
uses: google-github-actions/auth@v2
|
||||
with:
|
||||
workload_identity_provider: ${{ secrets.GCP_WORKLOAD_IDENTITY_PROVIDER }}
|
||||
service_account: ${{ secrets.GCP_SERVICE_ACCOUNT }}
|
||||
|
||||
|
||||
- name: Set up Cloud SDK
|
||||
uses: google-github-actions/setup-gcloud@v2
|
||||
with:
|
||||
version: 'latest'
|
||||
|
||||
version: "latest"
|
||||
|
||||
- name: Check if base image needs rebuild
|
||||
id: check-base
|
||||
run: |
|
||||
echo "🔍 Checking if base image needs rebuild..."
|
||||
|
||||
# Calculate current poetry.lock hash
|
||||
|
||||
CURRENT_HASH=$(sha256sum poetry.lock | cut -d' ' -f1)
|
||||
echo "Current poetry.lock hash: $CURRENT_HASH"
|
||||
|
||||
# Try to get the hash from the existing base image
|
||||
# First, authenticate to Artifact Registry
|
||||
|
||||
gcloud auth configure-docker us-central1-docker.pkg.dev --quiet
|
||||
|
||||
# Try to pull the base image and get its label
|
||||
|
||||
BASE_IMAGE="us-central1-docker.pkg.dev/nomadkaraoke/karaoke-repo/karaoke-backend-base:latest"
|
||||
EXISTING_HASH=""
|
||||
|
||||
|
||||
if docker pull $BASE_IMAGE 2>/dev/null; then
|
||||
EXISTING_HASH=$(docker inspect $BASE_IMAGE --format='{{index .Config.Labels "poetry.lock.hash"}}' 2>/dev/null || echo "")
|
||||
echo "Existing base image hash: $EXISTING_HASH"
|
||||
else
|
||||
echo "⚠️ No existing base image found"
|
||||
fi
|
||||
|
||||
# Compare hashes
|
||||
|
||||
if [ "$CURRENT_HASH" != "$EXISTING_HASH" ]; then
|
||||
echo "🔄 Base image needs rebuild (poetry.lock changed)"
|
||||
echo "needs_rebuild=true" >> $GITHUB_OUTPUT
|
||||
@@ -200,29 +437,27 @@ jobs:
|
||||
echo "✅ Base image is up to date"
|
||||
echo "needs_rebuild=false" >> $GITHUB_OUTPUT
|
||||
fi
|
||||
|
||||
|
||||
- name: Rebuild base image (if needed)
|
||||
if: steps.check-base.outputs.needs_rebuild == 'true'
|
||||
run: |
|
||||
echo "🏗️ Rebuilding base image with updated dependencies..."
|
||||
|
||||
# Submit base image build and wait for completion
|
||||
|
||||
BASE_BUILD_ID=$(gcloud builds submit \
|
||||
--config=cloudbuild-base.yaml \
|
||||
--project=nomadkaraoke \
|
||||
--format='value(id)' \
|
||||
--async \
|
||||
.)
|
||||
|
||||
|
||||
echo "Base Build ID: $BASE_BUILD_ID"
|
||||
echo "Build URL: https://console.cloud.google.com/cloud-build/builds/$BASE_BUILD_ID?project=nomadkaraoke"
|
||||
|
||||
# Poll for build completion
|
||||
|
||||
echo "Waiting for base image build to complete..."
|
||||
while true; do
|
||||
BUILD_STATUS=$(gcloud builds describe $BASE_BUILD_ID --project=nomadkaraoke --format='value(status)')
|
||||
echo "Base build status: $BUILD_STATUS"
|
||||
|
||||
|
||||
if [ "$BUILD_STATUS" = "SUCCESS" ]; then
|
||||
echo "✅ Base image build completed successfully"
|
||||
break
|
||||
@@ -239,32 +474,30 @@ jobs:
|
||||
gcloud builds log $BASE_BUILD_ID --project=nomadkaraoke 2>&1 || true
|
||||
exit 1
|
||||
fi
|
||||
|
||||
|
||||
sleep 30
|
||||
done
|
||||
|
||||
|
||||
- name: Trigger Cloud Build
|
||||
run: |
|
||||
echo "🚀 Triggering Cloud Build for deployment..."
|
||||
echo "Commit: ${{ github.sha }}"
|
||||
|
||||
# Submit build and wait for completion
|
||||
|
||||
BUILD_ID=$(gcloud builds submit \
|
||||
--config=cloudbuild.yaml \
|
||||
--project=nomadkaraoke \
|
||||
--format='value(id)' \
|
||||
--async \
|
||||
.)
|
||||
|
||||
|
||||
echo "Build ID: $BUILD_ID"
|
||||
echo "Build URL: https://console.cloud.google.com/cloud-build/builds/$BUILD_ID?project=nomadkaraoke"
|
||||
|
||||
# Poll for build completion
|
||||
|
||||
echo "Waiting for build to complete..."
|
||||
while true; do
|
||||
BUILD_STATUS=$(gcloud builds describe $BUILD_ID --project=nomadkaraoke --format='value(status)')
|
||||
echo "Build status: $BUILD_STATUS"
|
||||
|
||||
|
||||
if [ "$BUILD_STATUS" = "SUCCESS" ]; then
|
||||
echo "✅ Build completed successfully"
|
||||
break
|
||||
@@ -272,7 +505,6 @@ jobs:
|
||||
echo "❌ Build failed with status: $BUILD_STATUS"
|
||||
echo ""
|
||||
echo "==================== BUILD LOGS ===================="
|
||||
# Use Cloud Logging to get build logs (works with roles/logging.viewer)
|
||||
gcloud logging read "resource.type=build AND resource.labels.build_id=$BUILD_ID" \
|
||||
--project=nomadkaraoke \
|
||||
--format="value(textPayload)" \
|
||||
@@ -280,30 +512,24 @@ jobs:
|
||||
--order=asc || echo "Failed to retrieve logs from Cloud Logging"
|
||||
echo "===================================================="
|
||||
echo ""
|
||||
# Also try the direct logs command as fallback
|
||||
gcloud builds log $BUILD_ID --project=nomadkaraoke 2>&1 || true
|
||||
exit 1
|
||||
fi
|
||||
|
||||
|
||||
sleep 30
|
||||
done
|
||||
|
||||
|
||||
- name: Verify deployment
|
||||
run: |
|
||||
SERVICE_URL=$(gcloud run services describe karaoke-backend \
|
||||
--region us-central1 \
|
||||
--format 'value(status.url)')
|
||||
|
||||
|
||||
echo "Service URL: $SERVICE_URL"
|
||||
|
||||
# Wait a few seconds for the service to be ready
|
||||
sleep 10
|
||||
|
||||
# Test health endpoint
|
||||
curl -f "$SERVICE_URL/api/health" || exit 1
|
||||
|
||||
echo "✅ Deployment verified - service is healthy"
|
||||
|
||||
|
||||
- name: Summary
|
||||
if: success()
|
||||
run: |
|
||||
@@ -1,187 +0,0 @@
|
||||
name: Test and Publish
|
||||
|
||||
on: [push, pull_request, workflow_dispatch]
|
||||
|
||||
# Add permissions for GitHub Pages deployment and git operations
|
||||
permissions:
|
||||
contents: write # Changed from 'read' to 'write' to allow creating git tags
|
||||
pages: write
|
||||
id-token: write
|
||||
|
||||
# Allow only one concurrent deployment, skipping runs queued between the run in-progress and latest queued.
|
||||
concurrency:
|
||||
group: "pages"
|
||||
cancel-in-progress: false
|
||||
|
||||
jobs:
|
||||
test:
|
||||
runs-on: ubuntu-latest
|
||||
strategy:
|
||||
matrix:
|
||||
python-version: ['3.13']
|
||||
|
||||
steps:
|
||||
- uses: actions/checkout@v4
|
||||
|
||||
- name: Set up Python ${{ matrix.python-version }}
|
||||
uses: actions/setup-python@v5
|
||||
with:
|
||||
python-version: ${{ matrix.python-version }}
|
||||
|
||||
- name: Install Poetry
|
||||
uses: snok/install-poetry@v1
|
||||
with:
|
||||
virtualenvs-create: true
|
||||
virtualenvs-in-project: true
|
||||
|
||||
- name: Install dependencies
|
||||
run: |
|
||||
poetry install --with dev
|
||||
|
||||
- name: Install system dependencies
|
||||
run: |
|
||||
sudo apt-get update
|
||||
sudo apt-get install -y ffmpeg
|
||||
|
||||
- name: Install spaCy language models
|
||||
run: |
|
||||
poetry run python -m spacy download en_core_web_sm
|
||||
|
||||
- name: Run unit tests
|
||||
run: |
|
||||
poetry run pytest tests/unit/ -v --cov=karaoke_gen --cov-report=xml --cov-report=term-missing --cov-fail-under=70
|
||||
|
||||
- name: Upload coverage reports
|
||||
uses: codecov/codecov-action@v5
|
||||
with:
|
||||
token: ${{ secrets.CODECOV_TOKEN }}
|
||||
files: ./coverage.xml
|
||||
flags: unittests
|
||||
name: codecov-umbrella
|
||||
|
||||
- name: Run integration tests
|
||||
run: |
|
||||
poetry run pytest tests/integration/ -v
|
||||
|
||||
- name: Build package
|
||||
run: |
|
||||
poetry build
|
||||
|
||||
- name: Free Disk Space (GHA Ubuntu Runner)
|
||||
uses: jlumbroso/free-disk-space@main
|
||||
with:
|
||||
# this might remove tools that are actually needed,
|
||||
# if set to "true" but frees about 6 GB
|
||||
tool-cache: false
|
||||
|
||||
# all of these default to true, but feel free to set to
|
||||
# "false" if necessary for your workflow
|
||||
android: true
|
||||
haskell: true
|
||||
dotnet: false
|
||||
large-packages: false
|
||||
docker-images: false
|
||||
swap-storage: false
|
||||
|
||||
- name: Test package installation
|
||||
run: |
|
||||
pip install dist/*.whl
|
||||
python -m karaoke_gen.utils.gen_cli --help
|
||||
|
||||
- if: github.ref == 'refs/heads/main' && github.event_name == 'push'
|
||||
name: Extract version
|
||||
id: extract_version
|
||||
run: |
|
||||
VERSION=$(poetry version --short)
|
||||
echo "version=$VERSION" >> $GITHUB_OUTPUT
|
||||
echo "📋 Extracted version: $VERSION"
|
||||
|
||||
- if: github.ref == 'refs/heads/main' && github.event_name == 'push'
|
||||
name: Check if tag exists
|
||||
id: check_tag
|
||||
run: |
|
||||
VERSION="${{ steps.extract_version.outputs.version }}"
|
||||
if git ls-remote --tags origin | grep -q "refs/tags/v$VERSION$"; then
|
||||
echo "tag_exists=true" >> $GITHUB_OUTPUT
|
||||
echo "🏷️ Tag v$VERSION already exists, skipping release"
|
||||
else
|
||||
echo "tag_exists=false" >> $GITHUB_OUTPUT
|
||||
echo "✨ Tag v$VERSION does not exist, proceeding with release"
|
||||
fi
|
||||
|
||||
- if: github.ref == 'refs/heads/main' && github.event_name == 'push' && steps.check_tag.outputs.tag_exists == 'false'
|
||||
name: Publish to PyPI
|
||||
run: |
|
||||
echo "📦 Publishing the following files to PyPI:"
|
||||
ls -la dist/
|
||||
echo "🚀 Publishing to PyPI..."
|
||||
poetry config pypi-token.pypi ${{ secrets.PYPI_API_TOKEN }}
|
||||
poetry publish
|
||||
|
||||
- if: github.ref == 'refs/heads/main' && github.event_name == 'push' && steps.check_tag.outputs.tag_exists == 'false'
|
||||
name: Create Git Tag
|
||||
run: |
|
||||
git config user.name github-actions
|
||||
git config user.email github-actions@github.com
|
||||
git tag -a "v${{ steps.extract_version.outputs.version }}" -m "Release v${{ steps.extract_version.outputs.version }}"
|
||||
git push https://${{ secrets.GITHUB_TOKEN }}@github.com/${{ github.repository }} "v${{ steps.extract_version.outputs.version }}"
|
||||
|
||||
- if: github.ref == 'refs/heads/main' && github.event_name == 'push' && steps.check_tag.outputs.tag_exists == 'false'
|
||||
name: Create GitHub Release
|
||||
uses: softprops/action-gh-release@v1
|
||||
with:
|
||||
tag_name: "v${{ steps.extract_version.outputs.version }}"
|
||||
name: "v${{ steps.extract_version.outputs.version }}"
|
||||
body: |
|
||||
## Changes in v${{ steps.extract_version.outputs.version }}
|
||||
|
||||
This release was automatically created from the main branch.
|
||||
|
||||
### Installation
|
||||
```bash
|
||||
pip install karaoke-gen==${{ steps.extract_version.outputs.version }}
|
||||
```
|
||||
|
||||
### Assets
|
||||
- Python package published to PyPI
|
||||
- Frontend deployed to GitHub Pages at gen.nomadkaraoke.com
|
||||
|
||||
For detailed changes, see the commit history.
|
||||
draft: false
|
||||
prerelease: false
|
||||
files: |
|
||||
dist/*.whl
|
||||
dist/*.tar.gz
|
||||
|
||||
# Deploy frontend to GitHub Pages
|
||||
deploy-frontend:
|
||||
runs-on: ubuntu-latest
|
||||
if: github.ref == 'refs/heads/main' && github.event_name == 'push'
|
||||
environment:
|
||||
name: github-pages
|
||||
url: ${{ steps.deployment.outputs.page_url }}
|
||||
|
||||
steps:
|
||||
- name: Checkout
|
||||
uses: actions/checkout@v4
|
||||
|
||||
- name: Setup Pages
|
||||
uses: actions/configure-pages@v4
|
||||
|
||||
- name: Prepare frontend files
|
||||
run: |
|
||||
echo "📁 Preparing frontend files for deployment..."
|
||||
mkdir -p _site
|
||||
cp -r frontend/* _site/
|
||||
echo "✅ Frontend files copied to _site/"
|
||||
ls -la _site/
|
||||
|
||||
- name: Upload artifact
|
||||
uses: actions/upload-pages-artifact@v3
|
||||
with:
|
||||
path: '_site'
|
||||
|
||||
- name: Deploy to GitHub Pages
|
||||
id: deployment
|
||||
uses: actions/deploy-pages@v4
|
||||
|
||||
@@ -27,6 +27,10 @@ COPY lyrics_transcriber_temp /app/lyrics_transcriber_temp
|
||||
RUN pip install --no-cache-dir --upgrade pip && \
|
||||
pip install --no-cache-dir -e .
|
||||
|
||||
# Download spaCy language model for lyrics transcription
|
||||
# This must be done at build time, not runtime (Cloud Run network restrictions)
|
||||
RUN python -m spacy download en_core_web_sm
|
||||
|
||||
# Label with build info
|
||||
ARG BUILD_DATE
|
||||
ARG BUILD_ID
|
||||
|
||||
@@ -1,12 +1,25 @@
|
||||
"""
|
||||
File upload route for local file submission with style configuration support.
|
||||
|
||||
Supports two upload flows:
|
||||
1. Direct upload (original): All files sent as multipart form data to /api/jobs/upload
|
||||
- Simple but limited by Cloud Run's 32MB request body size
|
||||
|
||||
2. Signed URL upload (recommended for large files):
|
||||
- POST /api/jobs/create-with-upload-urls - Creates job, returns signed GCS upload URLs
|
||||
- Client uploads files directly to GCS using signed URLs (no size limit)
|
||||
- POST /api/jobs/{job_id}/uploads-complete - Validates uploads, triggers workers
|
||||
"""
|
||||
import asyncio
|
||||
import json
|
||||
import logging
|
||||
from fastapi import APIRouter, UploadFile, File, Form, HTTPException, BackgroundTasks, Request
|
||||
import tempfile
|
||||
import os
|
||||
from fastapi import APIRouter, UploadFile, File, Form, HTTPException, BackgroundTasks, Request, Body
|
||||
from pathlib import Path
|
||||
from typing import Optional, List, Dict, Any
|
||||
from typing import Optional, List, Dict, Any, Tuple
|
||||
|
||||
from pydantic import BaseModel, Field
|
||||
|
||||
from backend.models.job import JobCreate, JobStatus
|
||||
from backend.services.job_manager import JobManager
|
||||
@@ -19,6 +32,128 @@ from backend.version import VERSION
|
||||
logger = logging.getLogger(__name__)
|
||||
router = APIRouter(tags=["jobs"])
|
||||
|
||||
|
||||
# ============================================================================
|
||||
# Pydantic models for signed URL upload flow
|
||||
# ============================================================================
|
||||
|
||||
class FileUploadRequest(BaseModel):
|
||||
"""Information about a file to be uploaded."""
|
||||
filename: str = Field(..., description="Original filename with extension")
|
||||
content_type: str = Field(..., description="MIME type of the file")
|
||||
file_type: str = Field(..., description="Type of file: 'audio', 'style_params', 'style_intro_background', 'style_karaoke_background', 'style_end_background', 'style_font', 'style_cdg_instrumental_background', 'style_cdg_title_background', 'style_cdg_outro_background', 'lyrics_file'")
|
||||
|
||||
|
||||
class CreateJobFromUrlRequest(BaseModel):
|
||||
"""Request to create a job from a YouTube/online URL."""
|
||||
# Required fields
|
||||
url: str = Field(..., description="YouTube or other video URL to download audio from")
|
||||
|
||||
# Optional fields - will be auto-detected from URL if not provided
|
||||
artist: Optional[str] = Field(None, description="Artist name (auto-detected from URL if not provided)")
|
||||
title: Optional[str] = Field(None, description="Song title (auto-detected from URL if not provided)")
|
||||
|
||||
# Processing options
|
||||
enable_cdg: bool = Field(True, description="Generate CDG+MP3 package")
|
||||
enable_txt: bool = Field(True, description="Generate TXT+MP3 package")
|
||||
|
||||
# Finalisation options
|
||||
brand_prefix: Optional[str] = Field(None, description="Brand code prefix (e.g., NOMAD)")
|
||||
enable_youtube_upload: bool = Field(False, description="Upload to YouTube")
|
||||
youtube_description: Optional[str] = Field(None, description="YouTube video description text")
|
||||
discord_webhook_url: Optional[str] = Field(None, description="Discord webhook URL for notifications")
|
||||
webhook_url: Optional[str] = Field(None, description="Generic webhook URL")
|
||||
user_email: Optional[str] = Field(None, description="User email for notifications")
|
||||
|
||||
# Distribution options (native API - preferred for remote CLI)
|
||||
dropbox_path: Optional[str] = Field(None, description="Dropbox folder path for organized output")
|
||||
gdrive_folder_id: Optional[str] = Field(None, description="Google Drive folder ID for public share uploads")
|
||||
|
||||
# Legacy distribution options (rclone - deprecated)
|
||||
organised_dir_rclone_root: Optional[str] = Field(None, description="[Deprecated] rclone remote path")
|
||||
|
||||
# Lyrics configuration
|
||||
lyrics_artist: Optional[str] = Field(None, description="Override artist name for lyrics search")
|
||||
lyrics_title: Optional[str] = Field(None, description="Override title for lyrics search")
|
||||
subtitle_offset_ms: int = Field(0, description="Subtitle timing offset in milliseconds")
|
||||
|
||||
# Audio separation model configuration
|
||||
clean_instrumental_model: Optional[str] = Field(None, description="Model for clean instrumental separation")
|
||||
backing_vocals_models: Optional[List[str]] = Field(None, description="Models for backing vocals separation")
|
||||
other_stems_models: Optional[List[str]] = Field(None, description="Models for other stems")
|
||||
|
||||
|
||||
class CreateJobFromUrlResponse(BaseModel):
|
||||
"""Response from creating a job from URL."""
|
||||
status: str
|
||||
job_id: str
|
||||
message: str
|
||||
detected_artist: Optional[str]
|
||||
detected_title: Optional[str]
|
||||
server_version: str
|
||||
|
||||
|
||||
class CreateJobWithUploadUrlsRequest(BaseModel):
|
||||
"""Request to create a job and get signed upload URLs."""
|
||||
# Required fields
|
||||
artist: str = Field(..., description="Artist name")
|
||||
title: str = Field(..., description="Song title")
|
||||
files: List[FileUploadRequest] = Field(..., description="List of files to upload")
|
||||
|
||||
# Processing options
|
||||
enable_cdg: bool = Field(True, description="Generate CDG+MP3 package")
|
||||
enable_txt: bool = Field(True, description="Generate TXT+MP3 package")
|
||||
|
||||
# Finalisation options
|
||||
brand_prefix: Optional[str] = Field(None, description="Brand code prefix (e.g., NOMAD)")
|
||||
enable_youtube_upload: bool = Field(False, description="Upload to YouTube")
|
||||
youtube_description: Optional[str] = Field(None, description="YouTube video description text")
|
||||
discord_webhook_url: Optional[str] = Field(None, description="Discord webhook URL for notifications")
|
||||
webhook_url: Optional[str] = Field(None, description="Generic webhook URL")
|
||||
user_email: Optional[str] = Field(None, description="User email for notifications")
|
||||
|
||||
# Distribution options (native API - preferred for remote CLI)
|
||||
dropbox_path: Optional[str] = Field(None, description="Dropbox folder path for organized output")
|
||||
gdrive_folder_id: Optional[str] = Field(None, description="Google Drive folder ID for public share uploads")
|
||||
|
||||
# Legacy distribution options (rclone - deprecated)
|
||||
organised_dir_rclone_root: Optional[str] = Field(None, description="[Deprecated] rclone remote path")
|
||||
|
||||
# Lyrics configuration
|
||||
lyrics_artist: Optional[str] = Field(None, description="Override artist name for lyrics search")
|
||||
lyrics_title: Optional[str] = Field(None, description="Override title for lyrics search")
|
||||
subtitle_offset_ms: int = Field(0, description="Subtitle timing offset in milliseconds")
|
||||
|
||||
# Audio separation model configuration
|
||||
clean_instrumental_model: Optional[str] = Field(None, description="Model for clean instrumental separation")
|
||||
backing_vocals_models: Optional[List[str]] = Field(None, description="Models for backing vocals separation")
|
||||
other_stems_models: Optional[List[str]] = Field(None, description="Models for other stems")
|
||||
|
||||
# Existing instrumental configuration (Batch 3)
|
||||
existing_instrumental: bool = Field(False, description="Whether an existing instrumental file is being uploaded")
|
||||
|
||||
|
||||
class SignedUploadUrl(BaseModel):
|
||||
"""Signed URL for uploading a file."""
|
||||
file_type: str = Field(..., description="Type of file this URL is for")
|
||||
gcs_path: str = Field(..., description="The GCS path where file will be stored")
|
||||
upload_url: str = Field(..., description="Signed URL to PUT the file to")
|
||||
content_type: str = Field(..., description="Content-Type header to use when uploading")
|
||||
|
||||
|
||||
class CreateJobWithUploadUrlsResponse(BaseModel):
|
||||
"""Response from creating a job with upload URLs."""
|
||||
status: str
|
||||
job_id: str
|
||||
message: str
|
||||
upload_urls: List[SignedUploadUrl]
|
||||
server_version: str
|
||||
|
||||
|
||||
class UploadsCompleteRequest(BaseModel):
|
||||
"""Request to mark uploads as complete and start processing."""
|
||||
uploaded_files: List[str] = Field(..., description="List of file_types that were successfully uploaded")
|
||||
|
||||
# Initialize services
|
||||
job_manager = JobManager()
|
||||
storage_service = StorageService()
|
||||
@@ -126,6 +261,15 @@ async def upload_and_create_job(
|
||||
gdrive_folder_id: Optional[str] = Form(None, description="Google Drive folder ID for public share uploads"),
|
||||
# Legacy distribution options (rclone - deprecated)
|
||||
organised_dir_rclone_root: Optional[str] = Form(None, description="[Deprecated] rclone remote path for Dropbox upload"),
|
||||
# Lyrics configuration (overrides for search/transcription)
|
||||
lyrics_artist: Optional[str] = Form(None, description="Override artist name for lyrics search"),
|
||||
lyrics_title: Optional[str] = Form(None, description="Override title for lyrics search"),
|
||||
lyrics_file: Optional[UploadFile] = File(None, description="User-provided lyrics file (TXT, DOCX, RTF)"),
|
||||
subtitle_offset_ms: int = Form(0, description="Subtitle timing offset in milliseconds (positive = delay)"),
|
||||
# Audio separation model configuration
|
||||
clean_instrumental_model: Optional[str] = Form(None, description="Model for clean instrumental separation (e.g., model_bs_roformer_ep_317_sdr_12.9755.ckpt)"),
|
||||
backing_vocals_models: Optional[str] = Form(None, description="Comma-separated list of models for backing vocals separation"),
|
||||
other_stems_models: Optional[str] = Form(None, description="Comma-separated list of models for other stems (bass, drums, guitar, etc.)"),
|
||||
):
|
||||
"""
|
||||
Upload an audio file and create a karaoke generation job with full style configuration.
|
||||
@@ -182,6 +326,16 @@ async def upload_and_create_job(
|
||||
detail=f"Invalid font file type '{ext}'. Allowed: {', '.join(ALLOWED_FONT_EXTENSIONS)}"
|
||||
)
|
||||
|
||||
# Validate lyrics file if provided
|
||||
ALLOWED_LYRICS_EXTENSIONS = {'.txt', '.docx', '.rtf'}
|
||||
if lyrics_file:
|
||||
ext = Path(lyrics_file.filename).suffix.lower()
|
||||
if ext not in ALLOWED_LYRICS_EXTENSIONS:
|
||||
raise HTTPException(
|
||||
status_code=400,
|
||||
detail=f"Invalid lyrics file type '{ext}'. Allowed: {', '.join(ALLOWED_LYRICS_EXTENSIONS)}"
|
||||
)
|
||||
|
||||
# Apply default distribution settings from environment if not provided
|
||||
settings = get_settings()
|
||||
effective_dropbox_path = dropbox_path or settings.default_dropbox_path
|
||||
@@ -229,6 +383,15 @@ async def upload_and_create_job(
|
||||
# Extract request metadata for tracking and filtering
|
||||
request_metadata = extract_request_metadata(request, created_from="upload")
|
||||
|
||||
# Parse comma-separated model lists into arrays
|
||||
parsed_backing_vocals_models = None
|
||||
if backing_vocals_models:
|
||||
parsed_backing_vocals_models = [m.strip() for m in backing_vocals_models.split(',') if m.strip()]
|
||||
|
||||
parsed_other_stems_models = None
|
||||
if other_stems_models:
|
||||
parsed_other_stems_models = [m.strip() for m in other_stems_models.split(',') if m.strip()]
|
||||
|
||||
# Create job first to get job_id
|
||||
job_create = JobCreate(
|
||||
artist=artist,
|
||||
@@ -247,6 +410,14 @@ async def upload_and_create_job(
|
||||
gdrive_folder_id=effective_gdrive_folder_id,
|
||||
# Legacy rclone distribution (deprecated)
|
||||
organised_dir_rclone_root=organised_dir_rclone_root,
|
||||
# Lyrics configuration (overrides for search/transcription)
|
||||
lyrics_artist=lyrics_artist,
|
||||
lyrics_title=lyrics_title,
|
||||
subtitle_offset_ms=subtitle_offset_ms,
|
||||
# Audio separation model configuration
|
||||
clean_instrumental_model=clean_instrumental_model,
|
||||
backing_vocals_models=parsed_backing_vocals_models,
|
||||
other_stems_models=parsed_other_stems_models,
|
||||
# Request metadata for tracking and filtering
|
||||
request_metadata=request_metadata,
|
||||
)
|
||||
@@ -308,6 +479,17 @@ async def upload_and_create_job(
|
||||
)
|
||||
style_assets['font'] = font_gcs_path
|
||||
|
||||
# Upload lyrics file if provided
|
||||
lyrics_file_gcs_path = None
|
||||
if lyrics_file:
|
||||
lyrics_file_gcs_path = f"uploads/{job_id}/lyrics/user_lyrics{Path(lyrics_file.filename).suffix.lower()}"
|
||||
logger.info(f"Uploading user lyrics file to GCS: {lyrics_file_gcs_path}")
|
||||
storage_service.upload_fileobj(
|
||||
lyrics_file.file,
|
||||
lyrics_file_gcs_path,
|
||||
content_type='text/plain'
|
||||
)
|
||||
|
||||
# Update job with all GCS paths
|
||||
update_data = {
|
||||
'input_media_gcs_path': audio_gcs_path,
|
||||
@@ -338,6 +520,24 @@ async def upload_and_create_job(
|
||||
if organised_dir_rclone_root:
|
||||
update_data['organised_dir_rclone_root'] = organised_dir_rclone_root
|
||||
|
||||
# Lyrics configuration
|
||||
if lyrics_artist:
|
||||
update_data['lyrics_artist'] = lyrics_artist
|
||||
if lyrics_title:
|
||||
update_data['lyrics_title'] = lyrics_title
|
||||
if lyrics_file_gcs_path:
|
||||
update_data['lyrics_file_gcs_path'] = lyrics_file_gcs_path
|
||||
if subtitle_offset_ms != 0:
|
||||
update_data['subtitle_offset_ms'] = subtitle_offset_ms
|
||||
|
||||
# Audio separation model configuration
|
||||
if clean_instrumental_model:
|
||||
update_data['clean_instrumental_model'] = clean_instrumental_model
|
||||
if parsed_backing_vocals_models:
|
||||
update_data['backing_vocals_models'] = parsed_backing_vocals_models
|
||||
if parsed_other_stems_models:
|
||||
update_data['other_stems_models'] = parsed_other_stems_models
|
||||
|
||||
job_manager.update_job(job_id, update_data)
|
||||
|
||||
# Verify the update
|
||||
@@ -417,3 +617,629 @@ async def upload_and_create_job(
|
||||
except Exception as e:
|
||||
logger.error(f"Error uploading files: {e}", exc_info=True)
|
||||
raise HTTPException(status_code=500, detail=str(e)) from e
|
||||
|
||||
|
||||
# ============================================================================
|
||||
# Signed URL Upload Flow - For large files that exceed Cloud Run's 32MB limit
|
||||
# ============================================================================
|
||||
|
||||
# Valid file types and their expected extensions
|
||||
VALID_FILE_TYPES = {
|
||||
'audio': ALLOWED_AUDIO_EXTENSIONS,
|
||||
'style_params': {'.json'},
|
||||
'style_intro_background': ALLOWED_IMAGE_EXTENSIONS,
|
||||
'style_karaoke_background': ALLOWED_IMAGE_EXTENSIONS,
|
||||
'style_end_background': ALLOWED_IMAGE_EXTENSIONS,
|
||||
'style_font': ALLOWED_FONT_EXTENSIONS,
|
||||
'style_cdg_instrumental_background': ALLOWED_IMAGE_EXTENSIONS,
|
||||
'style_cdg_title_background': ALLOWED_IMAGE_EXTENSIONS,
|
||||
'style_cdg_outro_background': ALLOWED_IMAGE_EXTENSIONS,
|
||||
'lyrics_file': {'.txt', '.docx', '.rtf'},
|
||||
'existing_instrumental': ALLOWED_AUDIO_EXTENSIONS, # Batch 3: user-provided instrumental
|
||||
}
|
||||
|
||||
|
||||
async def _validate_audio_durations(
|
||||
storage: StorageService,
|
||||
audio_gcs_path: str,
|
||||
instrumental_gcs_path: str,
|
||||
tolerance_seconds: float = 0.5
|
||||
) -> Tuple[bool, float, float]:
|
||||
"""
|
||||
Validate that audio and instrumental files have matching durations.
|
||||
|
||||
Downloads both files to temp directory and checks their durations using pydub.
|
||||
|
||||
Args:
|
||||
storage: StorageService instance
|
||||
audio_gcs_path: GCS path to main audio file
|
||||
instrumental_gcs_path: GCS path to instrumental file
|
||||
tolerance_seconds: Maximum allowed difference in seconds (default 0.5s)
|
||||
|
||||
Returns:
|
||||
Tuple of (is_valid, audio_duration, instrumental_duration)
|
||||
"""
|
||||
from pydub import AudioSegment
|
||||
|
||||
temp_dir = tempfile.mkdtemp(prefix="duration_check_")
|
||||
try:
|
||||
# Download audio file
|
||||
audio_local = os.path.join(temp_dir, "audio" + Path(audio_gcs_path).suffix)
|
||||
storage.download_file(audio_gcs_path, audio_local)
|
||||
|
||||
# Download instrumental file
|
||||
instrumental_local = os.path.join(temp_dir, "instrumental" + Path(instrumental_gcs_path).suffix)
|
||||
storage.download_file(instrumental_gcs_path, instrumental_local)
|
||||
|
||||
# Get durations using pydub (returns milliseconds)
|
||||
audio_segment = AudioSegment.from_file(audio_local)
|
||||
audio_duration = len(audio_segment) / 1000.0 # Convert to seconds
|
||||
|
||||
instrumental_segment = AudioSegment.from_file(instrumental_local)
|
||||
instrumental_duration = len(instrumental_segment) / 1000.0 # Convert to seconds
|
||||
|
||||
# Check if durations are within tolerance
|
||||
difference = abs(audio_duration - instrumental_duration)
|
||||
is_valid = difference <= tolerance_seconds
|
||||
|
||||
return is_valid, audio_duration, instrumental_duration
|
||||
|
||||
finally:
|
||||
# Clean up temp files
|
||||
import shutil
|
||||
if os.path.exists(temp_dir):
|
||||
shutil.rmtree(temp_dir)
|
||||
|
||||
|
||||
def _get_gcs_path_for_file(job_id: str, file_type: str, filename: str) -> str:
|
||||
"""Generate the GCS path for a file based on its type."""
|
||||
ext = Path(filename).suffix.lower()
|
||||
|
||||
if file_type == 'audio':
|
||||
return f"uploads/{job_id}/audio/{filename}"
|
||||
elif file_type == 'style_params':
|
||||
return f"uploads/{job_id}/style/style_params.json"
|
||||
elif file_type.startswith('style_'):
|
||||
# Map style_intro_background -> intro_background, etc.
|
||||
asset_key = file_type.replace('style_', '')
|
||||
return f"uploads/{job_id}/style/{asset_key}{ext}"
|
||||
elif file_type == 'lyrics_file':
|
||||
return f"uploads/{job_id}/lyrics/user_lyrics{ext}"
|
||||
elif file_type == 'existing_instrumental':
|
||||
# Batch 3: user-provided instrumental file
|
||||
return f"uploads/{job_id}/audio/existing_instrumental{ext}"
|
||||
else:
|
||||
return f"uploads/{job_id}/other/{filename}"
|
||||
|
||||
|
||||
@router.post("/jobs/create-with-upload-urls", response_model=CreateJobWithUploadUrlsResponse)
|
||||
async def create_job_with_upload_urls(
|
||||
request: Request,
|
||||
body: CreateJobWithUploadUrlsRequest,
|
||||
):
|
||||
"""
|
||||
Create a karaoke generation job and return signed URLs for direct file upload to GCS.
|
||||
|
||||
This is the first step of the two-step upload flow for large files:
|
||||
1. Call this endpoint with job metadata and list of files to upload
|
||||
2. Upload each file directly to GCS using the returned signed URLs
|
||||
3. Call POST /api/jobs/{job_id}/uploads-complete to start processing
|
||||
|
||||
Benefits of this flow:
|
||||
- No file size limits (GCS supports up to 5TB)
|
||||
- Faster uploads (direct to storage, no proxy)
|
||||
- Works with any HTTP client (no HTTP/2 required)
|
||||
- Resumable uploads possible with GCS
|
||||
"""
|
||||
try:
|
||||
# Validate files list
|
||||
if not body.files:
|
||||
raise HTTPException(status_code=400, detail="At least one file is required")
|
||||
|
||||
# Check that audio file is included
|
||||
audio_files = [f for f in body.files if f.file_type == 'audio']
|
||||
if not audio_files:
|
||||
raise HTTPException(status_code=400, detail="An audio file is required")
|
||||
if len(audio_files) > 1:
|
||||
raise HTTPException(status_code=400, detail="Only one audio file is allowed")
|
||||
|
||||
# Validate file types and extensions
|
||||
for file_info in body.files:
|
||||
if file_info.file_type not in VALID_FILE_TYPES:
|
||||
raise HTTPException(
|
||||
status_code=400,
|
||||
detail=f"Invalid file_type: '{file_info.file_type}'. Valid types: {list(VALID_FILE_TYPES.keys())}"
|
||||
)
|
||||
|
||||
ext = Path(file_info.filename).suffix.lower()
|
||||
allowed_extensions = VALID_FILE_TYPES[file_info.file_type]
|
||||
if ext not in allowed_extensions:
|
||||
raise HTTPException(
|
||||
status_code=400,
|
||||
detail=f"Invalid file extension '{ext}' for file_type '{file_info.file_type}'. Allowed: {allowed_extensions}"
|
||||
)
|
||||
|
||||
# Apply default distribution settings from environment if not provided
|
||||
settings = get_settings()
|
||||
effective_dropbox_path = body.dropbox_path or settings.default_dropbox_path
|
||||
effective_gdrive_folder_id = body.gdrive_folder_id or settings.default_gdrive_folder_id
|
||||
effective_discord_webhook_url = body.discord_webhook_url or settings.default_discord_webhook_url
|
||||
|
||||
# Validate credentials for requested distribution services
|
||||
invalid_services = []
|
||||
credential_manager = get_credential_manager()
|
||||
|
||||
if body.enable_youtube_upload:
|
||||
result = credential_manager.check_youtube_credentials()
|
||||
if result.status != CredentialStatus.VALID:
|
||||
invalid_services.append(f"youtube ({result.message})")
|
||||
|
||||
if effective_dropbox_path:
|
||||
result = credential_manager.check_dropbox_credentials()
|
||||
if result.status != CredentialStatus.VALID:
|
||||
invalid_services.append(f"dropbox ({result.message})")
|
||||
|
||||
if effective_gdrive_folder_id:
|
||||
result = credential_manager.check_gdrive_credentials()
|
||||
if result.status != CredentialStatus.VALID:
|
||||
invalid_services.append(f"gdrive ({result.message})")
|
||||
|
||||
if invalid_services:
|
||||
raise HTTPException(
|
||||
status_code=400,
|
||||
detail={
|
||||
"error": "credentials_invalid",
|
||||
"message": f"The following distribution services need re-authorization: {', '.join(invalid_services)}",
|
||||
"invalid_services": invalid_services,
|
||||
"auth_url": "/api/auth/status"
|
||||
}
|
||||
)
|
||||
|
||||
# Extract request metadata for tracking
|
||||
request_metadata = extract_request_metadata(request, created_from="signed_url_upload")
|
||||
|
||||
# Get original audio filename
|
||||
audio_file = audio_files[0]
|
||||
|
||||
# Create job
|
||||
job_create = JobCreate(
|
||||
artist=body.artist,
|
||||
title=body.title,
|
||||
filename=audio_file.filename,
|
||||
enable_cdg=body.enable_cdg,
|
||||
enable_txt=body.enable_txt,
|
||||
brand_prefix=body.brand_prefix,
|
||||
enable_youtube_upload=body.enable_youtube_upload,
|
||||
youtube_description=body.youtube_description,
|
||||
discord_webhook_url=effective_discord_webhook_url,
|
||||
webhook_url=body.webhook_url,
|
||||
user_email=body.user_email,
|
||||
dropbox_path=effective_dropbox_path,
|
||||
gdrive_folder_id=effective_gdrive_folder_id,
|
||||
organised_dir_rclone_root=body.organised_dir_rclone_root,
|
||||
lyrics_artist=body.lyrics_artist,
|
||||
lyrics_title=body.lyrics_title,
|
||||
subtitle_offset_ms=body.subtitle_offset_ms,
|
||||
clean_instrumental_model=body.clean_instrumental_model,
|
||||
backing_vocals_models=body.backing_vocals_models,
|
||||
other_stems_models=body.other_stems_models,
|
||||
request_metadata=request_metadata,
|
||||
)
|
||||
job = job_manager.create_job(job_create)
|
||||
job_id = job.job_id
|
||||
|
||||
logger.info(f"Created job {job_id} for {body.artist} - {body.title} (signed URL upload flow)")
|
||||
|
||||
# Generate signed upload URLs for each file
|
||||
upload_urls = []
|
||||
for file_info in body.files:
|
||||
gcs_path = _get_gcs_path_for_file(job_id, file_info.file_type, file_info.filename)
|
||||
|
||||
# Generate signed upload URL (valid for 60 minutes)
|
||||
signed_url = storage_service.generate_signed_upload_url(
|
||||
gcs_path,
|
||||
content_type=file_info.content_type,
|
||||
expiration_minutes=60
|
||||
)
|
||||
|
||||
upload_urls.append(SignedUploadUrl(
|
||||
file_type=file_info.file_type,
|
||||
gcs_path=gcs_path,
|
||||
upload_url=signed_url,
|
||||
content_type=file_info.content_type
|
||||
))
|
||||
|
||||
logger.info(f"Generated signed upload URL for {file_info.file_type}: {gcs_path}")
|
||||
|
||||
return CreateJobWithUploadUrlsResponse(
|
||||
status="success",
|
||||
job_id=job_id,
|
||||
message="Job created. Upload files to the provided URLs, then call /api/jobs/{job_id}/uploads-complete",
|
||||
upload_urls=upload_urls,
|
||||
server_version=VERSION
|
||||
)
|
||||
|
||||
except HTTPException:
|
||||
raise
|
||||
except Exception as e:
|
||||
logger.error(f"Error creating job with upload URLs: {e}", exc_info=True)
|
||||
raise HTTPException(status_code=500, detail=str(e)) from e
|
||||
|
||||
|
||||
@router.post("/jobs/{job_id}/uploads-complete")
|
||||
async def mark_uploads_complete(
|
||||
job_id: str,
|
||||
background_tasks: BackgroundTasks,
|
||||
body: UploadsCompleteRequest,
|
||||
):
|
||||
"""
|
||||
Mark file uploads as complete and start job processing.
|
||||
|
||||
This is the second step of the signed URL upload flow:
|
||||
1. Create job with POST /api/jobs/create-with-upload-urls
|
||||
2. Upload files directly to GCS using signed URLs
|
||||
3. Call this endpoint to validate uploads and start processing
|
||||
|
||||
The endpoint will:
|
||||
- Verify the job exists and is in PENDING state
|
||||
- Validate that required files (audio) were uploaded
|
||||
- Update job with GCS paths
|
||||
- Trigger audio and lyrics workers
|
||||
"""
|
||||
try:
|
||||
# Get job and verify it exists
|
||||
job = job_manager.get_job(job_id)
|
||||
if not job:
|
||||
raise HTTPException(status_code=404, detail=f"Job {job_id} not found")
|
||||
|
||||
# Verify job is in pending state
|
||||
if job.status != JobStatus.PENDING:
|
||||
raise HTTPException(
|
||||
status_code=400,
|
||||
detail=f"Job {job_id} is not in pending state (current: {job.status}). Cannot complete uploads."
|
||||
)
|
||||
|
||||
# Validate required files
|
||||
if 'audio' not in body.uploaded_files:
|
||||
raise HTTPException(status_code=400, detail="Audio file upload is required")
|
||||
|
||||
# Build GCS paths for uploaded files and validate they exist
|
||||
update_data = {}
|
||||
style_assets = {}
|
||||
|
||||
for file_type in body.uploaded_files:
|
||||
if file_type not in VALID_FILE_TYPES:
|
||||
logger.warning(f"Unknown file_type in uploaded_files: {file_type}")
|
||||
continue
|
||||
|
||||
# Determine the GCS path - we need to find the actual file
|
||||
prefix = f"uploads/{job_id}/"
|
||||
if file_type == 'audio':
|
||||
prefix = f"uploads/{job_id}/audio/"
|
||||
elif file_type == 'style_params':
|
||||
prefix = f"uploads/{job_id}/style/style_params"
|
||||
elif file_type.startswith('style_'):
|
||||
asset_key = file_type.replace('style_', '')
|
||||
prefix = f"uploads/{job_id}/style/{asset_key}"
|
||||
elif file_type == 'lyrics_file':
|
||||
prefix = f"uploads/{job_id}/lyrics/user_lyrics"
|
||||
elif file_type == 'existing_instrumental':
|
||||
prefix = f"uploads/{job_id}/audio/existing_instrumental"
|
||||
|
||||
# List files with this prefix to find the actual uploaded file
|
||||
files = storage_service.list_files(prefix)
|
||||
if not files:
|
||||
raise HTTPException(
|
||||
status_code=400,
|
||||
detail=f"File for '{file_type}' was not uploaded to GCS. Expected prefix: {prefix}"
|
||||
)
|
||||
|
||||
# Use the first (and should be only) file found
|
||||
gcs_path = files[0]
|
||||
|
||||
# Update appropriate field based on file type
|
||||
if file_type == 'audio':
|
||||
update_data['input_media_gcs_path'] = gcs_path
|
||||
update_data['filename'] = Path(gcs_path).name
|
||||
elif file_type == 'style_params':
|
||||
update_data['style_params_gcs_path'] = gcs_path
|
||||
style_assets['style_params'] = gcs_path
|
||||
elif file_type.startswith('style_'):
|
||||
asset_key = file_type.replace('style_', '')
|
||||
style_assets[asset_key] = gcs_path
|
||||
elif file_type == 'lyrics_file':
|
||||
update_data['lyrics_file_gcs_path'] = gcs_path
|
||||
elif file_type == 'existing_instrumental':
|
||||
update_data['existing_instrumental_gcs_path'] = gcs_path
|
||||
|
||||
# Validate existing instrumental duration if provided (Batch 3)
|
||||
audio_gcs_path = update_data.get('input_media_gcs_path')
|
||||
instrumental_gcs_path = update_data.get('existing_instrumental_gcs_path')
|
||||
|
||||
if audio_gcs_path and instrumental_gcs_path:
|
||||
logger.info(f"Validating instrumental duration for job {job_id}")
|
||||
try:
|
||||
duration_valid, audio_duration, instrumental_duration = await _validate_audio_durations(
|
||||
storage_service, audio_gcs_path, instrumental_gcs_path
|
||||
)
|
||||
if not duration_valid:
|
||||
raise HTTPException(
|
||||
status_code=400,
|
||||
detail={
|
||||
"error": "duration_mismatch",
|
||||
"message": f"Instrumental duration ({instrumental_duration:.2f}s) does not match audio duration ({audio_duration:.2f}s). "
|
||||
f"Difference must be within 0.5 seconds.",
|
||||
"audio_duration": audio_duration,
|
||||
"instrumental_duration": instrumental_duration,
|
||||
"difference": abs(audio_duration - instrumental_duration),
|
||||
}
|
||||
)
|
||||
logger.info(f"Duration validation passed: audio={audio_duration:.2f}s, instrumental={instrumental_duration:.2f}s")
|
||||
except HTTPException:
|
||||
raise
|
||||
except Exception as e:
|
||||
logger.warning(f"Duration validation failed with error: {e}. Proceeding without validation.")
|
||||
# Don't block the job if we can't validate - the video worker will fail more gracefully
|
||||
|
||||
# Add style assets to update if any
|
||||
if style_assets:
|
||||
update_data['style_assets'] = style_assets
|
||||
|
||||
# Update job with GCS paths
|
||||
job_manager.update_job(job_id, update_data)
|
||||
|
||||
logger.info(f"Validated uploads for job {job_id}: {body.uploaded_files}")
|
||||
|
||||
# Transition job to DOWNLOADING state
|
||||
job_manager.transition_to_state(
|
||||
job_id=job_id,
|
||||
new_status=JobStatus.DOWNLOADING,
|
||||
progress=5,
|
||||
message="Files uploaded, preparing to process"
|
||||
)
|
||||
|
||||
# Trigger workers in parallel
|
||||
background_tasks.add_task(_trigger_workers_parallel, job_id)
|
||||
|
||||
# Get distribution services info for response
|
||||
credential_manager = get_credential_manager()
|
||||
distribution_services: Dict[str, Any] = {}
|
||||
|
||||
# Get fresh job data
|
||||
updated_job = job_manager.get_job(job_id)
|
||||
|
||||
if updated_job.dropbox_path:
|
||||
dropbox_result = credential_manager.check_dropbox_credentials()
|
||||
distribution_services["dropbox"] = {
|
||||
"enabled": True,
|
||||
"path": updated_job.dropbox_path,
|
||||
"credentials_valid": dropbox_result.status == CredentialStatus.VALID,
|
||||
}
|
||||
|
||||
if updated_job.gdrive_folder_id:
|
||||
gdrive_result = credential_manager.check_gdrive_credentials()
|
||||
distribution_services["gdrive"] = {
|
||||
"enabled": True,
|
||||
"folder_id": updated_job.gdrive_folder_id,
|
||||
"credentials_valid": gdrive_result.status == CredentialStatus.VALID,
|
||||
}
|
||||
|
||||
if updated_job.enable_youtube_upload:
|
||||
youtube_result = credential_manager.check_youtube_credentials()
|
||||
distribution_services["youtube"] = {
|
||||
"enabled": True,
|
||||
"credentials_valid": youtube_result.status == CredentialStatus.VALID,
|
||||
}
|
||||
|
||||
if updated_job.discord_webhook_url:
|
||||
distribution_services["discord"] = {
|
||||
"enabled": True,
|
||||
}
|
||||
|
||||
return {
|
||||
"status": "success",
|
||||
"job_id": job_id,
|
||||
"message": "Uploads validated. Processing started.",
|
||||
"files_validated": body.uploaded_files,
|
||||
"style_assets": list(style_assets.keys()) if style_assets else [],
|
||||
"server_version": VERSION,
|
||||
"distribution_services": distribution_services,
|
||||
}
|
||||
|
||||
except HTTPException:
|
||||
raise
|
||||
except Exception as e:
|
||||
logger.error(f"Error completing uploads for job {job_id}: {e}", exc_info=True)
|
||||
raise HTTPException(status_code=500, detail=str(e)) from e
|
||||
|
||||
|
||||
# ============================================================================
|
||||
# URL-based Job Creation - For YouTube and other online video URLs
|
||||
# ============================================================================
|
||||
|
||||
def _validate_url(url: Optional[str]) -> bool:
|
||||
"""
|
||||
Validate that a URL is a supported video/audio URL.
|
||||
|
||||
Supports YouTube, Vimeo, SoundCloud, and other platforms supported by yt-dlp.
|
||||
|
||||
Args:
|
||||
url: The URL to validate. Can be None or non-string.
|
||||
|
||||
Returns:
|
||||
True if valid URL, False otherwise.
|
||||
"""
|
||||
# Handle None and non-string inputs safely
|
||||
if url is None or not isinstance(url, str):
|
||||
return False
|
||||
|
||||
# Handle empty string
|
||||
if not url:
|
||||
return False
|
||||
|
||||
# Basic URL validation
|
||||
if not url.startswith(('http://', 'https://')):
|
||||
return False
|
||||
|
||||
# List of known supported domains (subset - yt-dlp supports many more)
|
||||
supported_domains = [
|
||||
'youtube.com', 'youtu.be', 'www.youtube.com', 'm.youtube.com',
|
||||
'vimeo.com', 'www.vimeo.com',
|
||||
'soundcloud.com', 'www.soundcloud.com',
|
||||
'dailymotion.com', 'www.dailymotion.com',
|
||||
'twitch.tv', 'www.twitch.tv',
|
||||
'twitter.com', 'www.twitter.com', 'x.com', 'www.x.com',
|
||||
'facebook.com', 'www.facebook.com', 'fb.watch',
|
||||
'instagram.com', 'www.instagram.com',
|
||||
'tiktok.com', 'www.tiktok.com',
|
||||
]
|
||||
|
||||
from urllib.parse import urlparse
|
||||
parsed = urlparse(url)
|
||||
domain = parsed.netloc.lower()
|
||||
|
||||
# Remove port if present
|
||||
if ':' in domain:
|
||||
domain = domain.split(':')[0]
|
||||
|
||||
# Check if domain matches any supported domain
|
||||
for supported in supported_domains:
|
||||
if domain == supported or domain.endswith('.' + supported):
|
||||
return True
|
||||
|
||||
# For other URLs, let yt-dlp try (it supports many more sites)
|
||||
return True
|
||||
|
||||
|
||||
@router.post("/jobs/create-from-url", response_model=CreateJobFromUrlResponse)
|
||||
async def create_job_from_url(
|
||||
request: Request,
|
||||
background_tasks: BackgroundTasks,
|
||||
body: CreateJobFromUrlRequest,
|
||||
):
|
||||
"""
|
||||
Create a karaoke generation job from a YouTube or other online video URL.
|
||||
|
||||
The backend will:
|
||||
1. Validate the URL
|
||||
2. Extract metadata (artist/title) from the URL if not provided
|
||||
3. Create the job
|
||||
4. Trigger the audio worker to download and process
|
||||
|
||||
This is an alternative to file upload for cases where the audio
|
||||
source is a YouTube video or other online content.
|
||||
|
||||
Note: YouTube rate limiting may cause occasional download failures.
|
||||
The backend will retry automatically.
|
||||
"""
|
||||
try:
|
||||
# Validate URL
|
||||
if not _validate_url(body.url):
|
||||
raise HTTPException(
|
||||
status_code=400,
|
||||
detail="Invalid URL. Please provide a valid YouTube, Vimeo, SoundCloud, or other supported video URL."
|
||||
)
|
||||
|
||||
# Apply default distribution settings from environment if not provided
|
||||
settings = get_settings()
|
||||
effective_dropbox_path = body.dropbox_path or settings.default_dropbox_path
|
||||
effective_gdrive_folder_id = body.gdrive_folder_id or settings.default_gdrive_folder_id
|
||||
effective_discord_webhook_url = body.discord_webhook_url or settings.default_discord_webhook_url
|
||||
|
||||
# Validate credentials for requested distribution services
|
||||
invalid_services = []
|
||||
credential_manager = get_credential_manager()
|
||||
|
||||
if body.enable_youtube_upload:
|
||||
result = credential_manager.check_youtube_credentials()
|
||||
if result.status != CredentialStatus.VALID:
|
||||
invalid_services.append(f"youtube ({result.message})")
|
||||
|
||||
if effective_dropbox_path:
|
||||
result = credential_manager.check_dropbox_credentials()
|
||||
if result.status != CredentialStatus.VALID:
|
||||
invalid_services.append(f"dropbox ({result.message})")
|
||||
|
||||
if effective_gdrive_folder_id:
|
||||
result = credential_manager.check_gdrive_credentials()
|
||||
if result.status != CredentialStatus.VALID:
|
||||
invalid_services.append(f"gdrive ({result.message})")
|
||||
|
||||
if invalid_services:
|
||||
raise HTTPException(
|
||||
status_code=400,
|
||||
detail={
|
||||
"error": "credentials_invalid",
|
||||
"message": f"The following distribution services need re-authorization: {', '.join(invalid_services)}",
|
||||
"invalid_services": invalid_services,
|
||||
"auth_url": "/api/auth/status"
|
||||
}
|
||||
)
|
||||
|
||||
# Extract request metadata for tracking
|
||||
request_metadata = extract_request_metadata(request, created_from="url")
|
||||
|
||||
# Use provided artist/title or leave as None (will be auto-detected by audio worker)
|
||||
artist = body.artist
|
||||
title = body.title
|
||||
|
||||
# Create job with URL
|
||||
job_create = JobCreate(
|
||||
url=body.url,
|
||||
artist=artist,
|
||||
title=title,
|
||||
filename=None, # No file uploaded
|
||||
enable_cdg=body.enable_cdg,
|
||||
enable_txt=body.enable_txt,
|
||||
brand_prefix=body.brand_prefix,
|
||||
enable_youtube_upload=body.enable_youtube_upload,
|
||||
youtube_description=body.youtube_description,
|
||||
discord_webhook_url=effective_discord_webhook_url,
|
||||
webhook_url=body.webhook_url,
|
||||
user_email=body.user_email,
|
||||
dropbox_path=effective_dropbox_path,
|
||||
gdrive_folder_id=effective_gdrive_folder_id,
|
||||
organised_dir_rclone_root=body.organised_dir_rclone_root,
|
||||
lyrics_artist=body.lyrics_artist,
|
||||
lyrics_title=body.lyrics_title,
|
||||
subtitle_offset_ms=body.subtitle_offset_ms,
|
||||
clean_instrumental_model=body.clean_instrumental_model,
|
||||
backing_vocals_models=body.backing_vocals_models,
|
||||
other_stems_models=body.other_stems_models,
|
||||
request_metadata=request_metadata,
|
||||
)
|
||||
job = job_manager.create_job(job_create)
|
||||
job_id = job.job_id
|
||||
|
||||
logger.info(f"Created URL-based job {job_id} for URL: {body.url}")
|
||||
if artist:
|
||||
logger.info(f" Artist: {artist}")
|
||||
if title:
|
||||
logger.info(f" Title: {title}")
|
||||
|
||||
# Transition job to DOWNLOADING state
|
||||
job_manager.transition_to_state(
|
||||
job_id=job_id,
|
||||
new_status=JobStatus.DOWNLOADING,
|
||||
progress=5,
|
||||
message="Starting audio download from URL"
|
||||
)
|
||||
|
||||
# Trigger workers in parallel
|
||||
# The audio worker will handle downloading from URL
|
||||
background_tasks.add_task(_trigger_workers_parallel, job_id)
|
||||
|
||||
return CreateJobFromUrlResponse(
|
||||
status="success",
|
||||
job_id=job_id,
|
||||
message="Job created. Audio will be downloaded from URL.",
|
||||
detected_artist=artist,
|
||||
detected_title=title,
|
||||
server_version=VERSION
|
||||
)
|
||||
|
||||
except HTTPException:
|
||||
raise
|
||||
except Exception as e:
|
||||
logger.error(f"Error creating job from URL: {e}", exc_info=True)
|
||||
raise HTTPException(status_code=500, detail=str(e)) from e
|
||||
@@ -3,10 +3,13 @@ Internal API routes for worker coordination.
|
||||
|
||||
These endpoints are for internal use only (backend → workers).
|
||||
They are protected by admin authentication.
|
||||
|
||||
With Cloud Tasks integration, these endpoints may be called multiple times
|
||||
(retry on failure). Idempotency checks prevent duplicate processing.
|
||||
"""
|
||||
import logging
|
||||
import asyncio
|
||||
from typing import Tuple
|
||||
from typing import Tuple, Optional
|
||||
from fastapi import APIRouter, HTTPException, BackgroundTasks, Depends
|
||||
from pydantic import BaseModel
|
||||
|
||||
@@ -17,6 +20,7 @@ from backend.workers.video_worker import generate_video
|
||||
from backend.workers.render_video_worker import process_render_video
|
||||
from backend.api.dependencies import require_admin
|
||||
from backend.services.auth_service import UserType
|
||||
from backend.services.job_manager import JobManager
|
||||
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
@@ -35,6 +39,57 @@ class WorkerResponse(BaseModel):
|
||||
message: str
|
||||
|
||||
|
||||
def _check_worker_idempotency(job_id: str, worker_name: str) -> Optional[WorkerResponse]:
|
||||
"""
|
||||
Check if a worker is already running or completed for this job.
|
||||
|
||||
This provides idempotency for Cloud Tasks retries - if a task is retried
|
||||
but the worker is already running or has completed, we skip processing.
|
||||
|
||||
Args:
|
||||
job_id: Job ID to check
|
||||
worker_name: Worker name (audio, lyrics, screens, render, video)
|
||||
|
||||
Returns:
|
||||
WorkerResponse if should skip (already running/complete), None to proceed
|
||||
"""
|
||||
job_manager = JobManager()
|
||||
job = job_manager.get_job(job_id)
|
||||
|
||||
if not job:
|
||||
logger.warning(f"Job {job_id} not found for {worker_name} worker")
|
||||
return WorkerResponse(
|
||||
status="not_found",
|
||||
job_id=job_id,
|
||||
message=f"Job {job_id} not found"
|
||||
)
|
||||
|
||||
# Check worker-specific progress in state_data
|
||||
progress_key = f"{worker_name}_progress"
|
||||
worker_progress = job.state_data.get(progress_key, {})
|
||||
stage = worker_progress.get('stage')
|
||||
|
||||
if stage == 'running':
|
||||
logger.info(f"{worker_name.capitalize()} worker already running for job {job_id}, skipping")
|
||||
return WorkerResponse(
|
||||
status="already_running",
|
||||
job_id=job_id,
|
||||
message=f"{worker_name.capitalize()} worker already in progress"
|
||||
)
|
||||
|
||||
if stage == 'complete':
|
||||
logger.info(f"{worker_name.capitalize()} worker already complete for job {job_id}, skipping")
|
||||
return WorkerResponse(
|
||||
status="already_complete",
|
||||
job_id=job_id,
|
||||
message=f"{worker_name.capitalize()} worker already completed"
|
||||
)
|
||||
|
||||
# Mark as running before starting (for idempotency on next retry)
|
||||
job_manager.update_state_data(job_id, progress_key, {'stage': 'running'})
|
||||
return None
|
||||
|
||||
|
||||
@router.post("/workers/audio", response_model=WorkerResponse)
|
||||
async def trigger_audio_worker(
|
||||
request: WorkerRequest,
|
||||
@@ -47,11 +102,18 @@ async def trigger_audio_worker(
|
||||
This endpoint is called internally after job creation to start
|
||||
the audio processing track (parallel with lyrics processing).
|
||||
|
||||
Idempotency: If worker is already running or complete, returns early.
|
||||
|
||||
The worker runs in the background and updates job state as it progresses.
|
||||
"""
|
||||
job_id = request.job_id
|
||||
logger.info(f"Triggering audio worker for job {job_id}")
|
||||
|
||||
# Idempotency check
|
||||
skip_response = _check_worker_idempotency(job_id, "audio")
|
||||
if skip_response:
|
||||
return skip_response
|
||||
|
||||
# Add task to background tasks
|
||||
# This allows the HTTP response to return immediately
|
||||
# while the worker continues processing
|
||||
@@ -76,11 +138,18 @@ async def trigger_lyrics_worker(
|
||||
This endpoint is called internally after job creation to start
|
||||
the lyrics processing track (parallel with audio processing).
|
||||
|
||||
Idempotency: If worker is already running or complete, returns early.
|
||||
|
||||
The worker runs in the background and updates job state as it progresses.
|
||||
"""
|
||||
job_id = request.job_id
|
||||
logger.info(f"Triggering lyrics worker for job {job_id}")
|
||||
|
||||
# Idempotency check
|
||||
skip_response = _check_worker_idempotency(job_id, "lyrics")
|
||||
if skip_response:
|
||||
return skip_response
|
||||
|
||||
# Add task to background tasks
|
||||
background_tasks.add_task(process_lyrics_transcription, job_id)
|
||||
|
||||
@@ -101,10 +170,17 @@ async def trigger_screens_worker(
|
||||
Trigger title/end screen generation worker.
|
||||
|
||||
This is called automatically when both audio and lyrics are complete.
|
||||
|
||||
Idempotency: If worker is already running or complete, returns early.
|
||||
"""
|
||||
job_id = request.job_id
|
||||
logger.info(f"Triggering screens worker for job {job_id}")
|
||||
|
||||
# Idempotency check
|
||||
skip_response = _check_worker_idempotency(job_id, "screens")
|
||||
if skip_response:
|
||||
return skip_response
|
||||
|
||||
# Add task to background tasks
|
||||
background_tasks.add_task(generate_screens, job_id)
|
||||
|
||||
@@ -126,10 +202,17 @@ async def trigger_video_worker(
|
||||
|
||||
This is called after user selects their preferred instrumental.
|
||||
This is the longest-running stage (15-20 minutes).
|
||||
|
||||
Idempotency: If worker is already running or complete, returns early.
|
||||
"""
|
||||
job_id = request.job_id
|
||||
logger.info(f"Triggering video worker for job {job_id}")
|
||||
|
||||
# Idempotency check
|
||||
skip_response = _check_worker_idempotency(job_id, "video")
|
||||
if skip_response:
|
||||
return skip_response
|
||||
|
||||
# Add task to background tasks
|
||||
background_tasks.add_task(generate_video, job_id)
|
||||
|
||||
@@ -153,12 +236,19 @@ async def trigger_render_video_worker(
|
||||
Uses OutputGenerator from LyricsTranscriber to generate the karaoke video
|
||||
with the corrected lyrics.
|
||||
|
||||
Idempotency: If worker is already running or complete, returns early.
|
||||
|
||||
Output: with_vocals.mkv in GCS
|
||||
Next state: AWAITING_INSTRUMENTAL_SELECTION
|
||||
"""
|
||||
job_id = request.job_id
|
||||
logger.info(f"Triggering render-video worker for job {job_id}")
|
||||
|
||||
# Idempotency check
|
||||
skip_response = _check_worker_idempotency(job_id, "render")
|
||||
if skip_response:
|
||||
return skip_response
|
||||
|
||||
# Add task to background tasks
|
||||
background_tasks.add_task(process_render_video, job_id)
|
||||
|
||||
|
||||
@@ -41,6 +41,18 @@ class Settings(BaseSettings):
|
||||
max_concurrent_jobs: int = int(os.getenv("MAX_CONCURRENT_JOBS", "5"))
|
||||
job_timeout_seconds: int = int(os.getenv("JOB_TIMEOUT_SECONDS", "3600"))
|
||||
|
||||
# Cloud Tasks (for scalable worker coordination)
|
||||
# When enabled, workers are triggered via Cloud Tasks for guaranteed delivery
|
||||
# When disabled (default), workers are triggered via direct HTTP (for development)
|
||||
enable_cloud_tasks: bool = os.getenv("ENABLE_CLOUD_TASKS", "false").lower() in ("true", "1", "yes")
|
||||
gcp_region: str = os.getenv("GCP_REGION", "us-central1")
|
||||
|
||||
# Cloud Run Jobs (for long-running video encoding)
|
||||
# When enabled AND enable_cloud_tasks is true, video worker uses Cloud Run Jobs
|
||||
# instead of Cloud Tasks. This supports encoding times >30 minutes (up to 24 hours).
|
||||
# Default is false - Cloud Tasks is sufficient for most videos (15-20 min).
|
||||
use_cloud_run_jobs_for_video: bool = os.getenv("USE_CLOUD_RUN_JOBS_FOR_VIDEO", "false").lower() in ("true", "1", "yes")
|
||||
|
||||
# Storage paths
|
||||
temp_dir: str = os.getenv("TEMP_DIR", "/tmp/karaoke-gen")
|
||||
|
||||
|
||||
@@ -211,6 +211,20 @@ class Job(BaseModel):
|
||||
# Legacy distribution configuration (rclone - for local CLI backward compat)
|
||||
organised_dir_rclone_root: Optional[str] = None # Deprecated: use dropbox_path instead
|
||||
|
||||
# Lyrics configuration (overrides for search/transcription)
|
||||
lyrics_artist: Optional[str] = None # Override artist name for lyrics search
|
||||
lyrics_title: Optional[str] = None # Override title for lyrics search
|
||||
lyrics_file_gcs_path: Optional[str] = None # GCS path to user-provided lyrics file
|
||||
subtitle_offset_ms: int = 0 # Offset for subtitle timing (positive = delay)
|
||||
|
||||
# Audio separation model configuration
|
||||
clean_instrumental_model: Optional[str] = None # Model for clean instrumental separation
|
||||
backing_vocals_models: Optional[List[str]] = None # Models for backing vocals separation
|
||||
other_stems_models: Optional[List[str]] = None # Models for other stems (bass, drums, etc.)
|
||||
|
||||
# Existing instrumental configuration (Batch 3)
|
||||
existing_instrumental_gcs_path: Optional[str] = None # GCS path to user-provided instrumental file
|
||||
|
||||
# Processing state
|
||||
track_output_dir: Optional[str] = None # Local output directory (temp)
|
||||
audio_hash: Optional[str] = None # Hash for deduplication
|
||||
@@ -373,6 +387,20 @@ class JobCreate(BaseModel):
|
||||
# Legacy (rclone - deprecated, use dropbox_path instead)
|
||||
organised_dir_rclone_root: Optional[str] = None
|
||||
|
||||
# Lyrics configuration (overrides for search/transcription)
|
||||
lyrics_artist: Optional[str] = None # Override artist name for lyrics search
|
||||
lyrics_title: Optional[str] = None # Override title for lyrics search
|
||||
lyrics_file_gcs_path: Optional[str] = None # GCS path to user-provided lyrics file
|
||||
subtitle_offset_ms: int = 0 # Offset for subtitle timing (positive = delay)
|
||||
|
||||
# Audio separation model configuration
|
||||
clean_instrumental_model: Optional[str] = None # Model for clean instrumental separation
|
||||
backing_vocals_models: Optional[List[str]] = None # Models for backing vocals separation
|
||||
other_stems_models: Optional[List[str]] = None # Models for other stems (bass, drums, etc.)
|
||||
|
||||
# Existing instrumental configuration (Batch 3)
|
||||
existing_instrumental_gcs_path: Optional[str] = None # GCS path to user-provided instrumental file
|
||||
|
||||
# Request metadata (set by API endpoint from request headers)
|
||||
request_metadata: Dict[str, Any] = Field(default_factory=dict)
|
||||
"""
|
||||
|
||||
@@ -11,6 +11,8 @@ httpx>=0.25.0,<1.0.0
|
||||
google-cloud-firestore>=2.14.0,<3.0.0
|
||||
google-cloud-storage>=2.14.0,<3.0.0
|
||||
google-cloud-secret-manager>=2.18.0,<3.0.0
|
||||
google-cloud-tasks>=2.14.0,<3.0.0
|
||||
google-cloud-run>=0.10.0,<1.0.0
|
||||
python-dotenv>=1.0.0,<2.0.0
|
||||
|
||||
# Audio processing (reuses karaoke_gen library)
|
||||
|
||||
@@ -138,7 +138,7 @@ def job_log_context(
|
||||
Args:
|
||||
job_id: Job ID to associate logs with
|
||||
worker: Worker name for log categorization
|
||||
logger_names: List of logger names to capture (None = capture root + common loggers)
|
||||
logger_names: List of logger names to capture (None = capture common loggers)
|
||||
min_level: Minimum log level to capture
|
||||
|
||||
Usage:
|
||||
@@ -147,14 +147,13 @@ def job_log_context(
|
||||
correction_operations.add_lyrics_source(...)
|
||||
"""
|
||||
# Default loggers to capture - these cover most of our processing
|
||||
# Note: We intentionally DON'T include the root logger ('') to avoid
|
||||
# duplicate log entries since logs propagate up the logger hierarchy.
|
||||
# Only capture at the top-level package loggers.
|
||||
if logger_names is None:
|
||||
logger_names = [
|
||||
'', # Root logger
|
||||
'backend.api.routes.review',
|
||||
'lyrics_transcriber',
|
||||
'lyrics_transcriber.correction',
|
||||
'lyrics_transcriber.correction.anchor_sequence',
|
||||
'lyrics_transcriber.correction.corrector',
|
||||
'karaoke_gen',
|
||||
]
|
||||
|
||||
|
||||
@@ -65,6 +65,26 @@ class StorageService:
|
||||
a private key available. Requires the service account to have
|
||||
roles/iam.serviceAccountTokenCreator on itself.
|
||||
"""
|
||||
return self._generate_signed_url_internal(blob_path, "GET", expiration_minutes)
|
||||
|
||||
def generate_signed_upload_url(self, blob_path: str, content_type: str = "application/octet-stream", expiration_minutes: int = 60) -> str:
|
||||
"""Generate a signed URL for uploading a file directly to GCS.
|
||||
|
||||
This allows clients to upload files directly to GCS without going through
|
||||
the backend, bypassing any request body size limits.
|
||||
|
||||
Args:
|
||||
blob_path: The destination path in GCS
|
||||
content_type: The expected content type of the upload
|
||||
expiration_minutes: How long the URL is valid for
|
||||
|
||||
Returns:
|
||||
A signed URL that accepts PUT requests with the file content
|
||||
"""
|
||||
return self._generate_signed_url_internal(blob_path, "PUT", expiration_minutes, content_type)
|
||||
|
||||
def _generate_signed_url_internal(self, blob_path: str, method: str, expiration_minutes: int = 60, content_type: Optional[str] = None) -> str:
|
||||
"""Internal method to generate signed URLs for GET or PUT operations."""
|
||||
import google.auth
|
||||
from google.auth.transport import requests
|
||||
|
||||
@@ -74,6 +94,17 @@ class StorageService:
|
||||
# Get default credentials and refresh to ensure we have a valid token
|
||||
credentials, project = google.auth.default()
|
||||
|
||||
# Common kwargs for signed URL generation
|
||||
kwargs = {
|
||||
"version": "v4",
|
||||
"expiration": timedelta(minutes=expiration_minutes),
|
||||
"method": method,
|
||||
}
|
||||
|
||||
# For PUT requests, we need to specify the content type in headers
|
||||
if method == "PUT" and content_type:
|
||||
kwargs["headers"] = {"Content-Type": content_type}
|
||||
|
||||
# Check if we're using compute credentials (Cloud Run/GCE)
|
||||
# These need to use IAM signBlob via service_account_email + access_token
|
||||
if hasattr(credentials, 'service_account_email'):
|
||||
@@ -81,25 +112,15 @@ class StorageService:
|
||||
auth_request = requests.Request()
|
||||
credentials.refresh(auth_request)
|
||||
|
||||
url = blob.generate_signed_url(
|
||||
version="v4",
|
||||
expiration=timedelta(minutes=expiration_minutes),
|
||||
method="GET",
|
||||
service_account_email=credentials.service_account_email,
|
||||
access_token=credentials.token
|
||||
)
|
||||
else:
|
||||
# Local development with service account key or user credentials
|
||||
url = blob.generate_signed_url(
|
||||
version="v4",
|
||||
expiration=timedelta(minutes=expiration_minutes),
|
||||
method="GET"
|
||||
)
|
||||
kwargs["service_account_email"] = credentials.service_account_email
|
||||
kwargs["access_token"] = credentials.token
|
||||
|
||||
logger.info(f"Generated signed URL for {blob_path}")
|
||||
url = blob.generate_signed_url(**kwargs)
|
||||
|
||||
logger.info(f"Generated signed {method} URL for {blob_path}")
|
||||
return url
|
||||
except Exception as e:
|
||||
logger.error(f"Error generating signed URL for {blob_path}: {e}")
|
||||
logger.error(f"Error generating signed {method} URL for {blob_path}: {e}")
|
||||
raise
|
||||
|
||||
def delete_file(self, blob_path: str) -> None:
|
||||
|
||||
@@ -2,7 +2,9 @@
|
||||
Worker coordination service.
|
||||
|
||||
Handles triggering and coordinating background workers.
|
||||
Implements proper HTTP-based triggers for production use.
|
||||
Supports two modes:
|
||||
- Cloud Tasks (production): Guaranteed delivery, automatic retries, horizontal scaling
|
||||
- Direct HTTP (development): Faster iteration, simpler debugging
|
||||
|
||||
SOLID Principles:
|
||||
- Single Responsibility: Only handles worker coordination
|
||||
@@ -11,20 +13,44 @@ SOLID Principles:
|
||||
"""
|
||||
import logging
|
||||
import os
|
||||
import httpx
|
||||
import json
|
||||
from typing import Optional
|
||||
import httpx
|
||||
|
||||
from backend.config import get_settings
|
||||
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
|
||||
# Mapping of worker types to their Cloud Tasks queue names
|
||||
# These queues are created by Pulumi in infrastructure/__main__.py
|
||||
WORKER_QUEUES = {
|
||||
"audio": "audio-worker-queue",
|
||||
"lyrics": "lyrics-worker-queue",
|
||||
"screens": "screens-worker-queue",
|
||||
"render-video": "render-worker-queue",
|
||||
"video": "video-worker-queue",
|
||||
}
|
||||
|
||||
|
||||
class WorkerService:
|
||||
"""
|
||||
Service for coordinating background workers.
|
||||
|
||||
Handles triggering workers via internal HTTP API.
|
||||
In production, workers run as separate Cloud Run requests or Cloud Tasks.
|
||||
Supports two execution modes controlled by ENABLE_CLOUD_TASKS env var:
|
||||
|
||||
1. Cloud Tasks mode (production, ENABLE_CLOUD_TASKS=true):
|
||||
- Tasks enqueued to Cloud Tasks for guaranteed delivery
|
||||
- Automatic retries on failure
|
||||
- Each task runs in dedicated Cloud Run instance
|
||||
- Rate limiting to protect external APIs (Modal, AudioShake)
|
||||
|
||||
2. Direct HTTP mode (development, ENABLE_CLOUD_TASKS=false):
|
||||
- Direct HTTP POST to internal worker endpoints
|
||||
- Worker runs as FastAPI BackgroundTask
|
||||
- Faster iteration for local development
|
||||
- No retry guarantees
|
||||
"""
|
||||
|
||||
def __init__(self):
|
||||
@@ -32,6 +58,44 @@ class WorkerService:
|
||||
self.settings = get_settings()
|
||||
self._base_url = self._get_base_url()
|
||||
self._admin_token = self._get_admin_token()
|
||||
self._use_cloud_tasks = self._should_use_cloud_tasks()
|
||||
self._tasks_client = None
|
||||
|
||||
if self._use_cloud_tasks:
|
||||
logger.info("WorkerService initialized with Cloud Tasks mode")
|
||||
else:
|
||||
logger.info("WorkerService initialized with direct HTTP mode")
|
||||
|
||||
def _should_use_cloud_tasks(self) -> bool:
|
||||
"""
|
||||
Check if Cloud Tasks should be used for worker coordination.
|
||||
|
||||
Controlled by ENABLE_CLOUD_TASKS setting (from environment variable).
|
||||
Default is false (direct HTTP mode) for backward compatibility.
|
||||
|
||||
Returns:
|
||||
True if Cloud Tasks should be used, False for direct HTTP
|
||||
"""
|
||||
return self.settings.enable_cloud_tasks
|
||||
|
||||
@property
|
||||
def tasks_client(self):
|
||||
"""
|
||||
Lazy-initialize Cloud Tasks client.
|
||||
|
||||
Only created when needed to avoid import overhead in development mode.
|
||||
"""
|
||||
if self._tasks_client is None and self._use_cloud_tasks:
|
||||
try:
|
||||
from google.cloud import tasks_v2
|
||||
self._tasks_client = tasks_v2.CloudTasksClient()
|
||||
except ImportError:
|
||||
logger.error(
|
||||
"google-cloud-tasks not installed. "
|
||||
"Install with: pip install google-cloud-tasks"
|
||||
)
|
||||
raise
|
||||
return self._tasks_client
|
||||
|
||||
def _get_admin_token(self) -> Optional[str]:
|
||||
"""
|
||||
@@ -49,9 +113,10 @@ class WorkerService:
|
||||
"""
|
||||
Get base URL for internal API calls.
|
||||
|
||||
In production: Cloud Run service URL
|
||||
In test: TEST_SERVER_URL
|
||||
In development: localhost
|
||||
Priority:
|
||||
1. TEST_SERVER_URL env var (for tests)
|
||||
2. CLOUD_RUN_SERVICE_URL env var (for Cloud Tasks to call back)
|
||||
3. localhost with PORT env var (for development)
|
||||
|
||||
Returns:
|
||||
Base URL for API calls
|
||||
@@ -61,14 +126,11 @@ class WorkerService:
|
||||
if test_url:
|
||||
return test_url
|
||||
|
||||
# Check if running in Cloud Run
|
||||
cloud_run_url = self.settings.google_cloud_project
|
||||
if cloud_run_url and self.settings.environment == "production":
|
||||
# TODO: Get actual Cloud Run service URL
|
||||
# For now, use environment variable
|
||||
service_url = os.getenv("CLOUD_RUN_SERVICE_URL")
|
||||
if service_url:
|
||||
return service_url
|
||||
# Production: Cloud Run service URL
|
||||
# This must be the publicly accessible URL for Cloud Tasks to call
|
||||
service_url = os.getenv("CLOUD_RUN_SERVICE_URL")
|
||||
if service_url:
|
||||
return service_url
|
||||
|
||||
# Development mode: use localhost with PORT env var
|
||||
port = os.getenv("PORT", "8000")
|
||||
@@ -81,10 +143,120 @@ class WorkerService:
|
||||
timeout_seconds: int = 30
|
||||
) -> bool:
|
||||
"""
|
||||
Trigger a background worker via internal API.
|
||||
Trigger a background worker for a job.
|
||||
|
||||
In production (ENABLE_CLOUD_TASKS=true):
|
||||
- Enqueues task to Cloud Tasks queue
|
||||
- Cloud Tasks delivers HTTP request to internal endpoint
|
||||
- Automatic retries on failure with exponential backoff
|
||||
- Rate limiting protects external APIs
|
||||
|
||||
In development (ENABLE_CLOUD_TASKS=false):
|
||||
- Direct HTTP POST to internal endpoint
|
||||
- Faster iteration, but no retry guarantees
|
||||
- Worker runs in same container as API
|
||||
|
||||
Args:
|
||||
worker_type: Worker type ("audio", "lyrics", "screens", "video")
|
||||
worker_type: Worker type ("audio", "lyrics", "screens", "render-video", "video")
|
||||
job_id: Job ID to process
|
||||
timeout_seconds: Request timeout (for direct HTTP mode)
|
||||
|
||||
Returns:
|
||||
True if trigger successful (or task enqueued), False otherwise
|
||||
"""
|
||||
if self._use_cloud_tasks:
|
||||
return await self._enqueue_cloud_task(worker_type, job_id)
|
||||
else:
|
||||
return await self._trigger_http(worker_type, job_id, timeout_seconds)
|
||||
|
||||
async def _enqueue_cloud_task(self, worker_type: str, job_id: str) -> bool:
|
||||
"""
|
||||
Enqueue task to Cloud Tasks for guaranteed delivery.
|
||||
|
||||
The task will be delivered as an HTTP POST to the internal worker endpoint.
|
||||
Cloud Tasks handles:
|
||||
- Retry on failure (with exponential backoff)
|
||||
- Rate limiting (max dispatches per second)
|
||||
- Deduplication (via task name if needed)
|
||||
- OIDC authentication for Cloud Run
|
||||
|
||||
Args:
|
||||
worker_type: Worker type
|
||||
job_id: Job ID to process
|
||||
|
||||
Returns:
|
||||
True if task enqueued successfully, False otherwise
|
||||
"""
|
||||
try:
|
||||
from google.cloud import tasks_v2
|
||||
|
||||
queue_name = WORKER_QUEUES.get(worker_type)
|
||||
if not queue_name:
|
||||
logger.error(f"Unknown worker type: {worker_type}")
|
||||
return False
|
||||
|
||||
project = self.settings.google_cloud_project
|
||||
if not project:
|
||||
logger.error("GOOGLE_CLOUD_PROJECT not set, cannot enqueue Cloud Task")
|
||||
return False
|
||||
|
||||
location = self.settings.gcp_region # Must match queue location
|
||||
|
||||
# Build queue path
|
||||
parent = self.tasks_client.queue_path(project, location, queue_name)
|
||||
|
||||
# Build task payload
|
||||
task = {
|
||||
"http_request": {
|
||||
"http_method": tasks_v2.HttpMethod.POST,
|
||||
"url": f"{self._base_url}/api/internal/workers/{worker_type}",
|
||||
"headers": {
|
||||
"Content-Type": "application/json",
|
||||
},
|
||||
"body": json.dumps({"job_id": job_id}).encode(),
|
||||
# Use OIDC token for Cloud Run authentication
|
||||
# This allows Cloud Tasks to invoke the Cloud Run service
|
||||
"oidc_token": {
|
||||
"service_account_email": f"karaoke-backend@{project}.iam.gserviceaccount.com",
|
||||
},
|
||||
},
|
||||
}
|
||||
|
||||
# Add admin auth header if available
|
||||
# This is needed for the internal endpoint's require_admin dependency
|
||||
if self._admin_token:
|
||||
task["http_request"]["headers"]["Authorization"] = f"Bearer {self._admin_token}"
|
||||
|
||||
# Create task
|
||||
# Note: We don't set a task name, allowing Cloud Tasks to generate one
|
||||
# This prevents duplicate task errors on retries
|
||||
response = self.tasks_client.create_task(parent=parent, task=task)
|
||||
logger.info(
|
||||
f"Created Cloud Task for {worker_type} worker, job {job_id}: {response.name}"
|
||||
)
|
||||
return True
|
||||
|
||||
except Exception as e:
|
||||
logger.error(
|
||||
f"Failed to enqueue Cloud Task for {worker_type}/{job_id}: {e}",
|
||||
exc_info=True
|
||||
)
|
||||
return False
|
||||
|
||||
async def _trigger_http(
|
||||
self,
|
||||
worker_type: str,
|
||||
job_id: str,
|
||||
timeout_seconds: int = 30
|
||||
) -> bool:
|
||||
"""
|
||||
Trigger worker via direct HTTP call (development mode).
|
||||
|
||||
This is the original implementation - direct HTTP POST to internal endpoint.
|
||||
The endpoint adds the worker function to FastAPI BackgroundTasks.
|
||||
|
||||
Args:
|
||||
worker_type: Worker type
|
||||
job_id: Job ID to process
|
||||
timeout_seconds: Request timeout
|
||||
|
||||
@@ -127,6 +299,9 @@ class WorkerService:
|
||||
)
|
||||
return False
|
||||
|
||||
# Convenience methods for specific workers
|
||||
# These provide a cleaner API and better IDE autocomplete
|
||||
|
||||
async def trigger_audio_worker(self, job_id: str) -> bool:
|
||||
"""Trigger audio separation worker."""
|
||||
return await self.trigger_worker("audio", job_id)
|
||||
@@ -140,9 +315,68 @@ class WorkerService:
|
||||
return await self.trigger_worker("screens", job_id)
|
||||
|
||||
async def trigger_video_worker(self, job_id: str) -> bool:
|
||||
"""Trigger video generation worker."""
|
||||
"""
|
||||
Trigger video generation worker.
|
||||
|
||||
When USE_CLOUD_RUN_JOBS_FOR_VIDEO=true and ENABLE_CLOUD_TASKS=true,
|
||||
uses Cloud Run Jobs for execution (supports >30 min encoding).
|
||||
Otherwise, uses Cloud Tasks or direct HTTP.
|
||||
"""
|
||||
if self._use_cloud_tasks and self.settings.use_cloud_run_jobs_for_video:
|
||||
return await self._trigger_cloud_run_job(job_id)
|
||||
return await self.trigger_worker("video", job_id)
|
||||
|
||||
async def _trigger_cloud_run_job(self, job_id: str) -> bool:
|
||||
"""
|
||||
Trigger video encoding as a Cloud Run Job.
|
||||
|
||||
Cloud Run Jobs support up to 24 hours of execution time,
|
||||
making them suitable for very long video encoding tasks.
|
||||
|
||||
Args:
|
||||
job_id: Job ID to process
|
||||
|
||||
Returns:
|
||||
True if job was triggered successfully, False otherwise
|
||||
"""
|
||||
try:
|
||||
from google.cloud import run_v2
|
||||
|
||||
project = self.settings.google_cloud_project
|
||||
if not project:
|
||||
logger.error("GOOGLE_CLOUD_PROJECT not set, cannot trigger Cloud Run Job")
|
||||
return False
|
||||
|
||||
location = self.settings.gcp_region
|
||||
job_name = f"projects/{project}/locations/{location}/jobs/video-encoding-job"
|
||||
|
||||
# Create Cloud Run Jobs client
|
||||
client = run_v2.JobsClient()
|
||||
|
||||
# Run the job with overrides for the specific job_id
|
||||
request = run_v2.RunJobRequest(
|
||||
name=job_name,
|
||||
overrides=run_v2.RunJobRequest.Overrides(
|
||||
container_overrides=[
|
||||
run_v2.RunJobRequest.Overrides.ContainerOverride(
|
||||
args=["python", "-m", "backend.workers.video_worker", "--job-id", job_id],
|
||||
)
|
||||
]
|
||||
)
|
||||
)
|
||||
|
||||
# Run the job (async operation)
|
||||
operation = client.run_job(request=request)
|
||||
logger.info(f"Started Cloud Run Job for video encoding, job {job_id}: {operation.metadata}")
|
||||
return True
|
||||
|
||||
except Exception as e:
|
||||
logger.error(
|
||||
f"Failed to trigger Cloud Run Job for video/{job_id}: {e}",
|
||||
exc_info=True
|
||||
)
|
||||
return False
|
||||
|
||||
async def trigger_render_video_worker(self, job_id: str) -> bool:
|
||||
"""Trigger render video worker (post-review)."""
|
||||
return await self.trigger_worker("render-video", job_id)
|
||||
@@ -166,3 +400,12 @@ def get_worker_service() -> WorkerService:
|
||||
_worker_service = WorkerService()
|
||||
return _worker_service
|
||||
|
||||
|
||||
def reset_worker_service() -> None:
|
||||
"""
|
||||
Reset the global worker service instance.
|
||||
|
||||
Used in tests to ensure clean state between test cases.
|
||||
"""
|
||||
global _worker_service
|
||||
_worker_service = None
|
||||
+1125
-10
File diff suppressed because it is too large.
Load diff
@@ -3,6 +3,9 @@ Unit tests for internal.py API routes using FastAPI TestClient.
|
||||
|
||||
Internal routes are used for worker communication within Cloud Run.
|
||||
These endpoints require admin authentication.
|
||||
|
||||
With Cloud Tasks integration, these endpoints include idempotency checks
|
||||
to prevent duplicate processing on retries.
|
||||
"""
|
||||
import os
|
||||
import pytest
|
||||
@@ -17,9 +20,20 @@ from backend.models.job import Job, JobStatus
|
||||
os.environ.setdefault('ADMIN_TOKENS', 'test-admin-token')
|
||||
|
||||
|
||||
def _create_mock_job(job_id: str = "test123") -> Job:
|
||||
"""Create a mock Job object for testing."""
|
||||
return Job(
|
||||
job_id=job_id,
|
||||
status=JobStatus.PROCESSING,
|
||||
created_at=datetime.now(UTC),
|
||||
updated_at=datetime.now(UTC),
|
||||
state_data={}, # Empty state_data - no worker progress yet
|
||||
)
|
||||
|
||||
|
||||
@pytest.fixture
|
||||
def client():
|
||||
"""Create TestClient with mocked workers."""
|
||||
"""Create TestClient with mocked workers and job manager."""
|
||||
mock_creds = MagicMock()
|
||||
mock_creds.universe_domain = 'googleapis.com'
|
||||
|
||||
@@ -28,11 +42,19 @@ def client():
|
||||
mock_lyrics = AsyncMock()
|
||||
mock_screens = AsyncMock()
|
||||
mock_video = AsyncMock()
|
||||
mock_render = AsyncMock()
|
||||
|
||||
# Mock JobManager for idempotency checks
|
||||
mock_job_manager = MagicMock()
|
||||
mock_job_manager.get_job.return_value = _create_mock_job()
|
||||
mock_job_manager.update_state_data.return_value = None
|
||||
|
||||
with patch('backend.api.routes.internal.process_audio_separation', mock_audio), \
|
||||
patch('backend.api.routes.internal.process_lyrics_transcription', mock_lyrics), \
|
||||
patch('backend.api.routes.internal.generate_screens', mock_screens), \
|
||||
patch('backend.api.routes.internal.generate_video', mock_video), \
|
||||
patch('backend.api.routes.internal.process_render_video', mock_render), \
|
||||
patch('backend.api.routes.internal.JobManager', return_value=mock_job_manager), \
|
||||
patch('backend.services.firestore_service.firestore'), \
|
||||
patch('backend.services.storage_service.storage'), \
|
||||
patch('google.auth.default', return_value=(mock_creds, 'test-project')):
|
||||
@@ -175,6 +197,36 @@ class TestVideoWorkerEndpoint:
|
||||
assert response.status_code == 401
|
||||
|
||||
|
||||
class TestRenderVideoWorkerEndpoint:
|
||||
"""Tests for POST /api/internal/workers/render-video."""
|
||||
|
||||
def test_render_video_worker_returns_200(self, client, auth_headers):
|
||||
"""Test render-video worker endpoint returns 200 with auth."""
|
||||
response = client.post(
|
||||
"/api/internal/workers/render-video",
|
||||
headers=auth_headers,
|
||||
json={"job_id": "test123"}
|
||||
)
|
||||
assert response.status_code == 200
|
||||
|
||||
def test_render_video_worker_requires_job_id(self, client, auth_headers):
|
||||
"""Test render-video worker requires job_id."""
|
||||
response = client.post(
|
||||
"/api/internal/workers/render-video",
|
||||
headers=auth_headers,
|
||||
json={}
|
||||
)
|
||||
assert response.status_code == 422
|
||||
|
||||
def test_render_video_worker_requires_auth(self, client):
|
||||
"""Test render-video worker requires authentication."""
|
||||
response = client.post(
|
||||
"/api/internal/workers/render-video",
|
||||
json={"job_id": "test123"}
|
||||
)
|
||||
assert response.status_code == 401
|
||||
|
||||
|
||||
class TestWorkerResponseFormat:
|
||||
"""Tests for worker response format."""
|
||||
|
||||
@@ -198,3 +250,94 @@ class TestWorkerResponseFormat:
|
||||
data = response.json()
|
||||
assert "status" in data or "message" in data
|
||||
|
||||
|
||||
class TestWorkerIdempotency:
|
||||
"""Tests for worker idempotency checks (Cloud Tasks retry support)."""
|
||||
|
||||
def test_audio_worker_returns_already_running_when_in_progress(self, auth_headers):
|
||||
"""Test audio worker returns already_running if worker is already running."""
|
||||
mock_creds = MagicMock()
|
||||
mock_creds.universe_domain = 'googleapis.com'
|
||||
|
||||
# Create job with audio already running
|
||||
mock_job = _create_mock_job()
|
||||
mock_job.state_data = {'audio_progress': {'stage': 'running'}}
|
||||
|
||||
mock_job_manager = MagicMock()
|
||||
mock_job_manager.get_job.return_value = mock_job
|
||||
|
||||
with patch('backend.api.routes.internal.process_audio_separation', AsyncMock()), \
|
||||
patch('backend.api.routes.internal.JobManager', return_value=mock_job_manager), \
|
||||
patch('backend.services.firestore_service.firestore'), \
|
||||
patch('backend.services.storage_service.storage'), \
|
||||
patch('google.auth.default', return_value=(mock_creds, 'test-project')):
|
||||
from backend.main import app
|
||||
client = TestClient(app)
|
||||
|
||||
response = client.post(
|
||||
"/api/internal/workers/audio",
|
||||
headers=auth_headers,
|
||||
json={"job_id": "test123"}
|
||||
)
|
||||
|
||||
assert response.status_code == 200
|
||||
data = response.json()
|
||||
assert data["status"] == "already_running"
|
||||
|
||||
def test_audio_worker_returns_already_complete_when_finished(self, auth_headers):
|
||||
"""Test audio worker returns already_complete if worker already finished."""
|
||||
mock_creds = MagicMock()
|
||||
mock_creds.universe_domain = 'googleapis.com'
|
||||
|
||||
# Create job with audio already complete
|
||||
mock_job = _create_mock_job()
|
||||
mock_job.state_data = {'audio_progress': {'stage': 'complete'}}
|
||||
|
||||
mock_job_manager = MagicMock()
|
||||
mock_job_manager.get_job.return_value = mock_job
|
||||
|
||||
with patch('backend.api.routes.internal.process_audio_separation', AsyncMock()), \
|
||||
patch('backend.api.routes.internal.JobManager', return_value=mock_job_manager), \
|
||||
patch('backend.services.firestore_service.firestore'), \
|
||||
patch('backend.services.storage_service.storage'), \
|
||||
patch('google.auth.default', return_value=(mock_creds, 'test-project')):
|
||||
from backend.main import app
|
||||
client = TestClient(app)
|
||||
|
||||
response = client.post(
|
||||
"/api/internal/workers/audio",
|
||||
headers=auth_headers,
|
||||
json={"job_id": "test123"}
|
||||
)
|
||||
|
||||
assert response.status_code == 200
|
||||
data = response.json()
|
||||
assert data["status"] == "already_complete"
|
||||
|
||||
def test_audio_worker_returns_not_found_when_job_missing(self, auth_headers):
|
||||
"""Test audio worker returns not_found if job doesn't exist."""
|
||||
mock_creds = MagicMock()
|
||||
mock_creds.universe_domain = 'googleapis.com'
|
||||
|
||||
# Return None for job not found
|
||||
mock_job_manager = MagicMock()
|
||||
mock_job_manager.get_job.return_value = None
|
||||
|
||||
with patch('backend.api.routes.internal.process_audio_separation', AsyncMock()), \
|
||||
patch('backend.api.routes.internal.JobManager', return_value=mock_job_manager), \
|
||||
patch('backend.services.firestore_service.firestore'), \
|
||||
patch('backend.services.storage_service.storage'), \
|
||||
patch('google.auth.default', return_value=(mock_creds, 'test-project')):
|
||||
from backend.main import app
|
||||
client = TestClient(app)
|
||||
|
||||
response = client.post(
|
||||
"/api/internal/workers/audio",
|
||||
headers=auth_headers,
|
||||
json={"job_id": "nonexistent"}
|
||||
)
|
||||
|
||||
assert response.status_code == 200
|
||||
data = response.json()
|
||||
assert data["status"] == "not_found"
|
||||
|
||||
@@ -577,6 +577,99 @@ class TestJobModelIntegration:
|
||||
assert restored_job.state_data["lyrics_metadata"]["line_count"] == 50
|
||||
|
||||
|
||||
class TestExistingInstrumentalModelFields:
|
||||
"""Test existing instrumental field in Job and JobCreate models (Batch 3)."""
|
||||
|
||||
def test_job_has_existing_instrumental_gcs_path(self):
|
||||
"""Test that Job model has existing_instrumental_gcs_path field."""
|
||||
job = Job(
|
||||
job_id="test123",
|
||||
status=JobStatus.PENDING,
|
||||
created_at=datetime.now(UTC),
|
||||
updated_at=datetime.now(UTC),
|
||||
existing_instrumental_gcs_path="uploads/test123/audio/existing_instrumental.flac"
|
||||
)
|
||||
|
||||
assert hasattr(job, 'existing_instrumental_gcs_path')
|
||||
assert job.existing_instrumental_gcs_path == "uploads/test123/audio/existing_instrumental.flac"
|
||||
|
||||
def test_job_existing_instrumental_defaults_to_none(self):
|
||||
"""Test that existing_instrumental_gcs_path defaults to None."""
|
||||
job = Job(
|
||||
job_id="test123",
|
||||
status=JobStatus.PENDING,
|
||||
created_at=datetime.now(UTC),
|
||||
updated_at=datetime.now(UTC)
|
||||
)
|
||||
|
||||
assert job.existing_instrumental_gcs_path is None
|
||||
|
||||
def test_job_create_has_existing_instrumental_gcs_path(self):
|
||||
"""Test that JobCreate model has existing_instrumental_gcs_path field."""
|
||||
from backend.models.job import JobCreate
|
||||
|
||||
job_create = JobCreate(
|
||||
artist="Artist",
|
||||
title="Title",
|
||||
existing_instrumental_gcs_path="uploads/test/audio/existing_instrumental.mp3"
|
||||
)
|
||||
|
||||
assert job_create.existing_instrumental_gcs_path == "uploads/test/audio/existing_instrumental.mp3"
|
||||
|
||||
def test_job_create_existing_instrumental_optional(self):
|
||||
"""Test that existing_instrumental_gcs_path is optional in JobCreate."""
|
||||
from backend.models.job import JobCreate
|
||||
|
||||
job_create = JobCreate(
|
||||
artist="Artist",
|
||||
title="Title"
|
||||
)
|
||||
|
||||
assert job_create.existing_instrumental_gcs_path is None
|
||||
|
||||
def test_existing_instrumental_roundtrip_serialization(self):
|
||||
"""Test that existing_instrumental_gcs_path survives serialization/deserialization."""
|
||||
job = Job(
|
||||
job_id="test123",
|
||||
status=JobStatus.PENDING,
|
||||
created_at=datetime.now(UTC),
|
||||
updated_at=datetime.now(UTC),
|
||||
existing_instrumental_gcs_path="uploads/test123/audio/existing_instrumental.wav"
|
||||
)
|
||||
|
||||
# Serialize to dict (simulates writing to Firestore)
|
||||
job_dict = job.model_dump()
|
||||
|
||||
# Convert to JSON and back (simulates Firestore storage)
|
||||
json_str = json.dumps(job_dict, default=str)
|
||||
restored_dict = json.loads(json_str)
|
||||
|
||||
# Deserialize back to Job (simulates reading from Firestore)
|
||||
restored_job = Job(**restored_dict)
|
||||
|
||||
assert restored_job.existing_instrumental_gcs_path == "uploads/test123/audio/existing_instrumental.wav"
|
||||
|
||||
def test_job_with_full_audio_config(self):
|
||||
"""Test Job with complete audio configuration including existing instrumental."""
|
||||
job = Job(
|
||||
job_id="test123",
|
||||
status=JobStatus.AUDIO_COMPLETE,
|
||||
created_at=datetime.now(UTC),
|
||||
updated_at=datetime.now(UTC),
|
||||
artist="Test Artist",
|
||||
title="Test Song",
|
||||
# Audio model configuration
|
||||
clean_instrumental_model="model_bs_roformer_ep_317_sdr_12.9755.ckpt",
|
||||
backing_vocals_models=["mel_band_roformer_karaoke.ckpt"],
|
||||
other_stems_models=["htdemucs_6s.yaml"],
|
||||
# Existing instrumental
|
||||
existing_instrumental_gcs_path="uploads/test123/audio/my_instrumental.flac",
|
||||
)
|
||||
|
||||
assert job.clean_instrumental_model == "model_bs_roformer_ep_317_sdr_12.9755.ckpt"
|
||||
assert job.existing_instrumental_gcs_path == "uploads/test123/audio/my_instrumental.flac"
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
pytest.main([__file__, "-v"])
|
||||
|
||||
@@ -12,6 +12,7 @@ from datetime import datetime, UTC
|
||||
import sys
|
||||
sys.modules['google.cloud.firestore'] = MagicMock()
|
||||
sys.modules['google.cloud.storage'] = MagicMock()
|
||||
sys.modules['google.cloud.tasks_v2'] = MagicMock()
|
||||
|
||||
from backend.services.auth_service import AuthService, UserType
|
||||
from backend.services.storage_service import StorageService
|
||||
@@ -239,6 +240,190 @@ class TestWorkerService:
|
||||
assert "/api/internal/workers/video" in str(call_args)
|
||||
|
||||
|
||||
class TestWorkerServiceCloudTasks:
|
||||
"""Test WorkerService Cloud Tasks integration."""
|
||||
|
||||
def test_should_use_cloud_tasks_default_false(self):
|
||||
"""Test that Cloud Tasks is disabled by default."""
|
||||
from backend.services.worker_service import WorkerService, reset_worker_service
|
||||
reset_worker_service()
|
||||
|
||||
with patch('backend.services.worker_service.os.getenv') as mock_getenv:
|
||||
mock_getenv.side_effect = lambda k, d=None: {
|
||||
'PORT': '8000',
|
||||
}.get(k, d)
|
||||
|
||||
with patch('backend.services.worker_service.get_settings') as mock_settings:
|
||||
mock_settings.return_value.admin_tokens = None
|
||||
mock_settings.return_value.google_cloud_project = "test-project"
|
||||
mock_settings.return_value.enable_cloud_tasks = False
|
||||
mock_settings.return_value.gcp_region = "us-central1"
|
||||
mock_settings.return_value.use_cloud_run_jobs_for_video = False
|
||||
|
||||
service = WorkerService()
|
||||
assert service._use_cloud_tasks is False
|
||||
|
||||
def test_should_use_cloud_tasks_enabled(self):
|
||||
"""Test that Cloud Tasks can be enabled via settings."""
|
||||
from backend.services.worker_service import WorkerService, reset_worker_service
|
||||
reset_worker_service()
|
||||
|
||||
with patch('backend.services.worker_service.os.getenv') as mock_getenv:
|
||||
mock_getenv.side_effect = lambda k, d=None: {
|
||||
'PORT': '8000',
|
||||
}.get(k, d)
|
||||
|
||||
with patch('backend.services.worker_service.get_settings') as mock_settings:
|
||||
mock_settings.return_value.admin_tokens = None
|
||||
mock_settings.return_value.google_cloud_project = "test-project"
|
||||
mock_settings.return_value.enable_cloud_tasks = True
|
||||
mock_settings.return_value.gcp_region = "us-central1"
|
||||
mock_settings.return_value.use_cloud_run_jobs_for_video = False
|
||||
|
||||
service = WorkerService()
|
||||
assert service._use_cloud_tasks is True
|
||||
|
||||
def test_worker_queues_mapping(self):
|
||||
"""Test that all worker types have queue mappings."""
|
||||
from backend.services.worker_service import WORKER_QUEUES
|
||||
|
||||
# Verify all expected workers have queues
|
||||
assert "audio" in WORKER_QUEUES
|
||||
assert "lyrics" in WORKER_QUEUES
|
||||
assert "screens" in WORKER_QUEUES
|
||||
assert "render-video" in WORKER_QUEUES
|
||||
assert "video" in WORKER_QUEUES
|
||||
|
||||
# Verify queue names are correct
|
||||
assert WORKER_QUEUES["audio"] == "audio-worker-queue"
|
||||
assert WORKER_QUEUES["lyrics"] == "lyrics-worker-queue"
|
||||
assert WORKER_QUEUES["screens"] == "screens-worker-queue"
|
||||
assert WORKER_QUEUES["render-video"] == "render-worker-queue"
|
||||
assert WORKER_QUEUES["video"] == "video-worker-queue"
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_trigger_worker_uses_http_when_cloud_tasks_disabled(self):
|
||||
"""Test that trigger_worker uses HTTP when Cloud Tasks is disabled."""
|
||||
from backend.services.worker_service import WorkerService, reset_worker_service
|
||||
reset_worker_service()
|
||||
|
||||
with patch('backend.services.worker_service.os.getenv') as mock_getenv:
|
||||
mock_getenv.side_effect = lambda k, d=None: {
|
||||
'PORT': '8000',
|
||||
}.get(k, d)
|
||||
|
||||
with patch('backend.services.worker_service.get_settings') as mock_settings:
|
||||
mock_settings.return_value.admin_tokens = "test-token"
|
||||
mock_settings.return_value.google_cloud_project = "test-project"
|
||||
mock_settings.return_value.enable_cloud_tasks = False
|
||||
mock_settings.return_value.gcp_region = "us-central1"
|
||||
mock_settings.return_value.use_cloud_run_jobs_for_video = False
|
||||
|
||||
with patch('backend.services.worker_service.httpx.AsyncClient') as mock_client_cls:
|
||||
mock_client = AsyncMock()
|
||||
mock_client_cls.return_value.__aenter__.return_value = mock_client
|
||||
|
||||
mock_response = Mock()
|
||||
mock_response.status_code = 200
|
||||
mock_client.post.return_value = mock_response
|
||||
|
||||
service = WorkerService()
|
||||
assert service._use_cloud_tasks is False
|
||||
|
||||
result = await service.trigger_worker("audio", "test-job-123")
|
||||
|
||||
# Verify HTTP was used
|
||||
assert result is True
|
||||
mock_client.post.assert_called_once()
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_trigger_worker_uses_cloud_tasks_when_enabled(self):
|
||||
"""Test that trigger_worker uses Cloud Tasks when enabled."""
|
||||
from backend.services.worker_service import WorkerService, reset_worker_service
|
||||
reset_worker_service()
|
||||
|
||||
with patch('backend.services.worker_service.os.getenv') as mock_getenv:
|
||||
mock_getenv.side_effect = lambda k, d=None: {
|
||||
'CLOUD_RUN_SERVICE_URL': 'https://api.example.com',
|
||||
}.get(k, d)
|
||||
|
||||
with patch('backend.services.worker_service.get_settings') as mock_settings:
|
||||
mock_settings.return_value.admin_tokens = "test-token"
|
||||
mock_settings.return_value.google_cloud_project = "test-project"
|
||||
mock_settings.return_value.enable_cloud_tasks = True
|
||||
mock_settings.return_value.gcp_region = "us-central1"
|
||||
mock_settings.return_value.use_cloud_run_jobs_for_video = False
|
||||
|
||||
# Mock Cloud Tasks client
|
||||
mock_tasks_client = MagicMock()
|
||||
mock_tasks_client.queue_path.return_value = "projects/test-project/locations/us-central1/queues/audio-worker-queue"
|
||||
mock_task_response = MagicMock()
|
||||
mock_task_response.name = "projects/test-project/locations/us-central1/queues/audio-worker-queue/tasks/abc123"
|
||||
mock_tasks_client.create_task.return_value = mock_task_response
|
||||
|
||||
# Mock the google.cloud.tasks_v2 module which is imported inside the method
|
||||
mock_tasks_module = MagicMock()
|
||||
mock_tasks_module.CloudTasksClient.return_value = mock_tasks_client
|
||||
mock_tasks_module.HttpMethod.POST = "POST"
|
||||
|
||||
with patch.dict('sys.modules', {'google.cloud.tasks_v2': mock_tasks_module}):
|
||||
with patch.dict(sys.modules, {'google.cloud': MagicMock()}):
|
||||
service = WorkerService()
|
||||
service._tasks_client = mock_tasks_client # Inject mocked client
|
||||
|
||||
result = await service.trigger_worker("audio", "test-job-123")
|
||||
|
||||
# Verify Cloud Tasks was used
|
||||
assert result is True
|
||||
mock_tasks_client.create_task.assert_called_once()
|
||||
|
||||
# Verify task payload
|
||||
call_args = mock_tasks_client.create_task.call_args
|
||||
assert call_args is not None
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_trigger_worker_returns_false_for_unknown_worker_type(self):
|
||||
"""Test that trigger_worker returns False for unknown worker types."""
|
||||
from backend.services.worker_service import WorkerService, reset_worker_service
|
||||
reset_worker_service()
|
||||
|
||||
with patch('backend.services.worker_service.os.getenv') as mock_getenv:
|
||||
mock_getenv.side_effect = lambda k, d=None: {
|
||||
'CLOUD_RUN_SERVICE_URL': 'https://api.example.com',
|
||||
}.get(k, d)
|
||||
|
||||
with patch('backend.services.worker_service.get_settings') as mock_settings:
|
||||
mock_settings.return_value.admin_tokens = "test-token"
|
||||
mock_settings.return_value.google_cloud_project = "test-project"
|
||||
mock_settings.return_value.enable_cloud_tasks = True
|
||||
mock_settings.return_value.gcp_region = "us-central1"
|
||||
mock_settings.return_value.use_cloud_run_jobs_for_video = False
|
||||
|
||||
service = WorkerService()
|
||||
|
||||
result = await service.trigger_worker("unknown-worker", "test-job-123")
|
||||
|
||||
assert result is False
|
||||
|
||||
def test_reset_worker_service(self):
|
||||
"""Test that reset_worker_service resets the singleton."""
|
||||
from backend.services.worker_service import get_worker_service, reset_worker_service
|
||||
|
||||
with patch('backend.services.worker_service.get_settings') as mock_settings:
|
||||
mock_settings.return_value.admin_tokens = None
|
||||
mock_settings.return_value.google_cloud_project = "test-project"
|
||||
mock_settings.return_value.enable_cloud_tasks = False
|
||||
mock_settings.return_value.gcp_region = "us-central1"
|
||||
mock_settings.return_value.use_cloud_run_jobs_for_video = False
|
||||
|
||||
service1 = get_worker_service()
|
||||
reset_worker_service()
|
||||
service2 = get_worker_service()
|
||||
|
||||
# After reset, should be different instances
|
||||
assert service1 is not service2
|
||||
|
||||
|
||||
class TestFirestoreService:
|
||||
"""Test FirestoreService database operations."""
|
||||
|
||||
|
||||
@@ -0,0 +1,156 @@
|
||||
"""
|
||||
Unit tests for backend validation script (validate.py).
|
||||
|
||||
Tests the validation functions that check for import errors, syntax errors,
|
||||
configuration issues, and FastAPI app creation.
|
||||
"""
|
||||
import pytest
|
||||
from unittest.mock import patch, MagicMock
|
||||
|
||||
|
||||
class TestValidateImports:
|
||||
"""Tests for the validate_imports function."""
|
||||
|
||||
def test_validate_imports_success(self):
|
||||
"""Test that validate_imports succeeds when all modules import."""
|
||||
from backend.validate import validate_imports
|
||||
|
||||
# This should succeed since we're in a valid test environment
|
||||
result = validate_imports()
|
||||
assert result is True
|
||||
|
||||
def test_validate_imports_with_failure(self):
|
||||
"""Test that validate_imports handles import failures."""
|
||||
from backend.validate import validate_imports
|
||||
|
||||
# Mock importlib to simulate a failure
|
||||
with patch('backend.validate.importlib.import_module') as mock_import:
|
||||
mock_import.side_effect = ImportError("Module not found")
|
||||
|
||||
result = validate_imports()
|
||||
assert result is False
|
||||
|
||||
|
||||
class TestValidateSyntax:
|
||||
"""Tests for the validate_syntax function."""
|
||||
|
||||
def test_validate_syntax_success(self):
|
||||
"""Test that validate_syntax succeeds with valid Python files."""
|
||||
from backend.validate import validate_syntax
|
||||
|
||||
# This should succeed since the backend code is valid
|
||||
result = validate_syntax()
|
||||
assert result is True
|
||||
|
||||
def test_validate_syntax_handles_invalid_file(self):
|
||||
"""Test that validate_syntax detects syntax errors."""
|
||||
import tempfile
|
||||
import os
|
||||
from pathlib import Path
|
||||
from backend.validate import validate_syntax
|
||||
|
||||
# We can't easily inject invalid files into the backend dir,
|
||||
# but we can verify the function runs and returns True for valid files
|
||||
result = validate_syntax()
|
||||
assert result is True
|
||||
|
||||
|
||||
class TestValidateConfig:
|
||||
"""Tests for the validate_config function."""
|
||||
|
||||
def test_validate_config_success(self):
|
||||
"""Test that validate_config succeeds with valid configuration."""
|
||||
from backend.validate import validate_config
|
||||
|
||||
# This should succeed in test environment
|
||||
result = validate_config()
|
||||
assert result is True
|
||||
|
||||
def test_validate_config_failure(self):
|
||||
"""Test that validate_config handles configuration errors."""
|
||||
from backend.validate import validate_config
|
||||
|
||||
# Patch at the source module
|
||||
with patch('backend.config.get_settings') as mock_settings:
|
||||
mock_settings.side_effect = Exception("Config error")
|
||||
|
||||
result = validate_config()
|
||||
assert result is False
|
||||
|
||||
|
||||
class TestValidateFastapiApp:
|
||||
"""Tests for the validate_fastapi_app function."""
|
||||
|
||||
def test_validate_fastapi_app_success(self):
|
||||
"""Test that validate_fastapi_app succeeds."""
|
||||
from backend.validate import validate_fastapi_app
|
||||
|
||||
# This should succeed since the FastAPI app is valid
|
||||
result = validate_fastapi_app()
|
||||
assert result is True
|
||||
|
||||
def test_validate_fastapi_app_failure(self):
|
||||
"""Test that validate_fastapi_app handles app creation errors."""
|
||||
from backend.validate import validate_fastapi_app
|
||||
|
||||
with patch.dict('sys.modules', {'backend.main': MagicMock(app=MagicMock(title="Test", version="1.0", routes=[]))}):
|
||||
# Even with a mock, the function should work
|
||||
result = validate_fastapi_app()
|
||||
# The original module is still accessible, so this should still pass
|
||||
assert result is True
|
||||
|
||||
|
||||
class TestMain:
|
||||
"""Tests for the main function."""
|
||||
|
||||
def test_main_runs(self):
|
||||
"""Test that main runs without errors."""
|
||||
from backend.validate import main
|
||||
|
||||
# In a valid test environment, main should return 0 (success)
|
||||
result = main()
|
||||
assert result == 0
|
||||
|
||||
def test_main_returns_1_on_failure(self):
|
||||
"""Test that main returns 1 when validations fail."""
|
||||
from backend.validate import main
|
||||
|
||||
# Mock one validation to fail
|
||||
with patch('backend.validate.validate_imports', return_value=False):
|
||||
result = main()
|
||||
assert result == 1
|
||||
|
||||
|
||||
class TestValidateModule:
|
||||
"""Test the validate module can be imported and has expected functions."""
|
||||
|
||||
def test_module_imports(self):
|
||||
"""Test that the validate module can be imported."""
|
||||
import backend.validate
|
||||
assert backend.validate is not None
|
||||
|
||||
def test_has_validate_imports(self):
|
||||
"""Test that validate_imports function exists."""
|
||||
from backend.validate import validate_imports
|
||||
assert callable(validate_imports)
|
||||
|
||||
def test_has_validate_syntax(self):
|
||||
"""Test that validate_syntax function exists."""
|
||||
from backend.validate import validate_syntax
|
||||
assert callable(validate_syntax)
|
||||
|
||||
def test_has_validate_config(self):
|
||||
"""Test that validate_config function exists."""
|
||||
from backend.validate import validate_config
|
||||
assert callable(validate_config)
|
||||
|
||||
def test_has_validate_fastapi_app(self):
|
||||
"""Test that validate_fastapi_app function exists."""
|
||||
from backend.validate import validate_fastapi_app
|
||||
assert callable(validate_fastapi_app)
|
||||
|
||||
def test_has_main(self):
|
||||
"""Test that main function exists."""
|
||||
from backend.validate import main
|
||||
assert callable(main)
|
||||
|
||||
@@ -0,0 +1,365 @@
|
||||
"""
|
||||
Unit tests for backend/workers/worker_logging.py.
|
||||
|
||||
Tests the job logging utilities including JobLogger, JobLogHandler,
|
||||
and job_logging_context for concurrent job isolation.
|
||||
"""
|
||||
import pytest
|
||||
import logging
|
||||
from unittest.mock import MagicMock, patch
|
||||
|
||||
|
||||
class TestJobLoggingContext:
|
||||
"""Tests for job_logging_context context manager."""
|
||||
|
||||
def test_job_logging_context_sets_and_resets_job_id(self):
|
||||
"""Test that job_logging_context sets and resets the current job ID."""
|
||||
from backend.workers.worker_logging import job_logging_context, _current_job_id
|
||||
|
||||
# Initially no job
|
||||
assert _current_job_id.get() is None
|
||||
|
||||
# Inside context, job is set
|
||||
with job_logging_context("job123"):
|
||||
assert _current_job_id.get() == "job123"
|
||||
|
||||
# After context, job is reset
|
||||
assert _current_job_id.get() is None
|
||||
|
||||
def test_job_logging_context_nested(self):
|
||||
"""Test nested job_logging_context calls."""
|
||||
from backend.workers.worker_logging import job_logging_context, _current_job_id
|
||||
|
||||
with job_logging_context("outer_job"):
|
||||
assert _current_job_id.get() == "outer_job"
|
||||
|
||||
with job_logging_context("inner_job"):
|
||||
assert _current_job_id.get() == "inner_job"
|
||||
|
||||
# After inner context, outer job is restored
|
||||
assert _current_job_id.get() == "outer_job"
|
||||
|
||||
assert _current_job_id.get() is None
|
||||
|
||||
def test_job_logging_context_handles_exception(self):
|
||||
"""Test that job_logging_context resets even on exception."""
|
||||
from backend.workers.worker_logging import job_logging_context, _current_job_id
|
||||
|
||||
try:
|
||||
with job_logging_context("job123"):
|
||||
assert _current_job_id.get() == "job123"
|
||||
raise ValueError("Test exception")
|
||||
except ValueError:
|
||||
pass
|
||||
|
||||
# Job should still be reset after exception
|
||||
assert _current_job_id.get() is None
|
||||
|
||||
|
||||
class TestJobLogHandler:
|
||||
"""Tests for JobLogHandler class."""
|
||||
|
||||
@pytest.fixture
|
||||
def mock_job_manager(self):
|
||||
"""Create a mock JobManager."""
|
||||
manager = MagicMock()
|
||||
manager.append_worker_log.return_value = None
|
||||
return manager
|
||||
|
||||
def test_job_log_handler_init(self, mock_job_manager):
|
||||
"""Test JobLogHandler initialization."""
|
||||
from backend.workers.worker_logging import JobLogHandler
|
||||
|
||||
handler = JobLogHandler(
|
||||
job_id="job123",
|
||||
worker_name="audio",
|
||||
job_manager=mock_job_manager
|
||||
)
|
||||
|
||||
assert handler.job_id == "job123"
|
||||
assert handler.worker_name == "audio"
|
||||
assert handler.job_manager == mock_job_manager
|
||||
assert handler.level == logging.INFO
|
||||
|
||||
def test_job_log_handler_custom_level(self, mock_job_manager):
|
||||
"""Test JobLogHandler with custom level."""
|
||||
from backend.workers.worker_logging import JobLogHandler
|
||||
|
||||
handler = JobLogHandler(
|
||||
job_id="job123",
|
||||
worker_name="audio",
|
||||
job_manager=mock_job_manager,
|
||||
level=logging.DEBUG
|
||||
)
|
||||
|
||||
assert handler.level == logging.DEBUG
|
||||
|
||||
def test_job_log_handler_emit_logs_to_firestore(self, mock_job_manager):
|
||||
"""Test that emit() calls job_manager.append_worker_log."""
|
||||
from backend.workers.worker_logging import JobLogHandler, job_logging_context
|
||||
|
||||
handler = JobLogHandler(
|
||||
job_id="job123",
|
||||
worker_name="audio",
|
||||
job_manager=mock_job_manager
|
||||
)
|
||||
handler.setFormatter(logging.Formatter('%(message)s'))
|
||||
|
||||
# Create a log record
|
||||
record = logging.LogRecord(
|
||||
name="test",
|
||||
level=logging.INFO,
|
||||
pathname="",
|
||||
lineno=0,
|
||||
msg="Test message",
|
||||
args=(),
|
||||
exc_info=None
|
||||
)
|
||||
|
||||
# Emit within the job context
|
||||
with job_logging_context("job123"):
|
||||
handler.emit(record)
|
||||
|
||||
# Should have called append_worker_log
|
||||
mock_job_manager.append_worker_log.assert_called()
|
||||
|
||||
def test_job_log_handler_filters_other_job_context(self, mock_job_manager):
|
||||
"""Test that handler filters logs from other job contexts."""
|
||||
from backend.workers.worker_logging import JobLogHandler, job_logging_context
|
||||
|
||||
handler = JobLogHandler(
|
||||
job_id="job123",
|
||||
worker_name="audio",
|
||||
job_manager=mock_job_manager
|
||||
)
|
||||
|
||||
record = logging.LogRecord(
|
||||
name="test",
|
||||
level=logging.INFO,
|
||||
pathname="",
|
||||
lineno=0,
|
||||
msg="Test message",
|
||||
args=(),
|
||||
exc_info=None
|
||||
)
|
||||
|
||||
# Emit from a different job's context
|
||||
with job_logging_context("different_job"):
|
||||
handler.emit(record)
|
||||
|
||||
# Should NOT have called append_worker_log
|
||||
mock_job_manager.append_worker_log.assert_not_called()
|
||||
|
||||
def test_job_log_handler_deduplication(self, mock_job_manager):
|
||||
"""Test that handler deduplicates repeated messages."""
|
||||
from backend.workers.worker_logging import JobLogHandler, job_logging_context
|
||||
|
||||
handler = JobLogHandler(
|
||||
job_id="job123",
|
||||
worker_name="audio",
|
||||
job_manager=mock_job_manager
|
||||
)
|
||||
handler.setFormatter(logging.Formatter('%(message)s'))
|
||||
|
||||
record = logging.LogRecord(
|
||||
name="test",
|
||||
level=logging.INFO,
|
||||
pathname="",
|
||||
lineno=0,
|
||||
msg="Duplicate message",
|
||||
args=(),
|
||||
exc_info=None
|
||||
)
|
||||
|
||||
with job_logging_context("job123"):
|
||||
# Emit the same record twice
|
||||
handler.emit(record)
|
||||
handler.emit(record)
|
||||
|
||||
# Should only be called once due to deduplication
|
||||
assert mock_job_manager.append_worker_log.call_count == 1
|
||||
|
||||
|
||||
class TestJobLogger:
|
||||
"""Tests for JobLogger class."""
|
||||
|
||||
@pytest.fixture
|
||||
def mock_job_manager(self):
|
||||
"""Create a mock JobManager."""
|
||||
manager = MagicMock()
|
||||
manager.append_worker_log.return_value = None
|
||||
return manager
|
||||
|
||||
def test_job_logger_init(self, mock_job_manager):
|
||||
"""Test JobLogger initialization."""
|
||||
from backend.workers.worker_logging import JobLogger
|
||||
|
||||
logger = JobLogger(
|
||||
job_id="job123",
|
||||
worker_name="lyrics",
|
||||
job_manager=mock_job_manager
|
||||
)
|
||||
|
||||
assert logger.job_id == "job123"
|
||||
assert logger.worker_name == "lyrics"
|
||||
assert logger.job_manager == mock_job_manager
|
||||
|
||||
def test_job_logger_info(self, mock_job_manager):
|
||||
"""Test JobLogger.info() method."""
|
||||
from backend.workers.worker_logging import JobLogger
|
||||
|
||||
logger = JobLogger(
|
||||
job_id="job123",
|
||||
worker_name="lyrics",
|
||||
job_manager=mock_job_manager
|
||||
)
|
||||
|
||||
logger.info("Processing started")
|
||||
|
||||
mock_job_manager.append_worker_log.assert_called_with(
|
||||
job_id="job123",
|
||||
worker="lyrics",
|
||||
level="INFO",
|
||||
message="Processing started"
|
||||
)
|
||||
|
||||
def test_job_logger_warning(self, mock_job_manager):
|
||||
"""Test JobLogger.warning() method."""
|
||||
from backend.workers.worker_logging import JobLogger
|
||||
|
||||
logger = JobLogger(
|
||||
job_id="job123",
|
||||
worker_name="audio",
|
||||
job_manager=mock_job_manager
|
||||
)
|
||||
|
||||
logger.warning("Low memory")
|
||||
|
||||
mock_job_manager.append_worker_log.assert_called_with(
|
||||
job_id="job123",
|
||||
worker="audio",
|
||||
level="WARNING",
|
||||
message="Low memory"
|
||||
)
|
||||
|
||||
def test_job_logger_error(self, mock_job_manager):
|
||||
"""Test JobLogger.error() method."""
|
||||
from backend.workers.worker_logging import JobLogger
|
||||
|
||||
logger = JobLogger(
|
||||
job_id="job123",
|
||||
worker_name="video",
|
||||
job_manager=mock_job_manager
|
||||
)
|
||||
|
||||
logger.error("Processing failed")
|
||||
|
||||
mock_job_manager.append_worker_log.assert_called_with(
|
||||
job_id="job123",
|
||||
worker="video",
|
||||
level="ERROR",
|
||||
message="Processing failed"
|
||||
)
|
||||
|
||||
def test_job_logger_debug(self, mock_job_manager):
|
||||
"""Test JobLogger.debug() method."""
|
||||
from backend.workers.worker_logging import JobLogger
|
||||
|
||||
logger = JobLogger(
|
||||
job_id="job123",
|
||||
worker_name="screens",
|
||||
job_manager=mock_job_manager
|
||||
)
|
||||
|
||||
logger.debug("Debug info")
|
||||
|
||||
mock_job_manager.append_worker_log.assert_called_with(
|
||||
job_id="job123",
|
||||
worker="screens",
|
||||
level="DEBUG",
|
||||
message="Debug info"
|
||||
)
|
||||
|
||||
def test_job_logger_with_format_args(self, mock_job_manager):
|
||||
"""Test JobLogger with format arguments."""
|
||||
from backend.workers.worker_logging import JobLogger
|
||||
|
||||
logger = JobLogger(
|
||||
job_id="job123",
|
||||
worker_name="audio",
|
||||
job_manager=mock_job_manager
|
||||
)
|
||||
|
||||
logger.info("Processing %s of %d", "audio", 10)
|
||||
|
||||
mock_job_manager.append_worker_log.assert_called_with(
|
||||
job_id="job123",
|
||||
worker="audio",
|
||||
level="INFO",
|
||||
message="Processing audio of 10"
|
||||
)
|
||||
|
||||
def test_job_logger_handles_firestore_error(self, mock_job_manager):
|
||||
"""Test that JobLogger handles Firestore errors gracefully."""
|
||||
from backend.workers.worker_logging import JobLogger
|
||||
|
||||
mock_job_manager.append_worker_log.side_effect = Exception("Firestore error")
|
||||
|
||||
logger = JobLogger(
|
||||
job_id="job123",
|
||||
worker_name="audio",
|
||||
job_manager=mock_job_manager
|
||||
)
|
||||
|
||||
# Should not raise exception
|
||||
logger.info("Test message")
|
||||
|
||||
|
||||
class TestCreateJobLogger:
|
||||
"""Tests for create_job_logger function."""
|
||||
|
||||
def test_create_job_logger(self):
|
||||
"""Test create_job_logger creates a JobLogger."""
|
||||
from backend.workers.worker_logging import create_job_logger, JobLogger
|
||||
|
||||
# Patch at the source module where JobManager is imported
|
||||
with patch('backend.services.job_manager.JobManager'):
|
||||
logger = create_job_logger("job123", "audio")
|
||||
|
||||
assert isinstance(logger, JobLogger)
|
||||
assert logger.job_id == "job123"
|
||||
assert logger.worker_name == "audio"
|
||||
|
||||
|
||||
class TestSetupJobLogging:
|
||||
"""Tests for setup_job_logging function."""
|
||||
|
||||
def test_setup_job_logging_returns_handler(self):
|
||||
"""Test setup_job_logging returns a JobLogHandler."""
|
||||
from backend.workers.worker_logging import setup_job_logging, JobLogHandler
|
||||
|
||||
# Patch at the source module
|
||||
with patch('backend.services.job_manager.JobManager'):
|
||||
handler = setup_job_logging("job123", "lyrics", "test_logger_wl1")
|
||||
|
||||
assert isinstance(handler, JobLogHandler)
|
||||
assert handler.job_id == "job123"
|
||||
assert handler.worker_name == "lyrics"
|
||||
|
||||
# Clean up - remove the handler
|
||||
logging.getLogger("test_logger_wl1").removeHandler(handler)
|
||||
|
||||
def test_setup_job_logging_adds_handler_to_loggers(self):
|
||||
"""Test setup_job_logging adds handler to specified loggers."""
|
||||
from backend.workers.worker_logging import setup_job_logging
|
||||
|
||||
with patch('backend.services.job_manager.JobManager'):
|
||||
handler = setup_job_logging("job123", "audio", "test_logger_wl2", "test_logger_wl3")
|
||||
|
||||
# Check handlers are added
|
||||
assert handler in logging.getLogger("test_logger_wl2").handlers
|
||||
assert handler in logging.getLogger("test_logger_wl3").handlers
|
||||
|
||||
# Clean up
|
||||
logging.getLogger("test_logger_wl2").removeHandler(handler)
|
||||
logging.getLogger("test_logger_wl3").removeHandler(handler)
|
||||
@@ -353,6 +353,108 @@ class TestLyricsWorker:
|
||||
mock_job_manager.mark_job_failed.assert_called()
|
||||
|
||||
|
||||
class TestLyricsWorkerConfiguration:
|
||||
"""Tests for lyrics worker configuration parameters."""
|
||||
|
||||
def test_create_lyrics_processor_with_defaults(self):
|
||||
"""Test creating LyricsProcessor with default parameters."""
|
||||
with patch('backend.workers.lyrics_worker.LyricsProcessor') as mock_processor:
|
||||
from backend.workers.lyrics_worker import create_lyrics_processor
|
||||
|
||||
result = create_lyrics_processor()
|
||||
|
||||
mock_processor.assert_called_once()
|
||||
call_kwargs = mock_processor.call_args[1]
|
||||
assert call_kwargs['lyrics_file'] is None
|
||||
assert call_kwargs['subtitle_offset_ms'] == 0
|
||||
|
||||
def test_create_lyrics_processor_with_lyrics_file(self):
|
||||
"""Test creating LyricsProcessor with custom lyrics file."""
|
||||
with patch('backend.workers.lyrics_worker.LyricsProcessor') as mock_processor:
|
||||
from backend.workers.lyrics_worker import create_lyrics_processor
|
||||
|
||||
result = create_lyrics_processor(
|
||||
lyrics_file="/path/to/lyrics.txt",
|
||||
subtitle_offset_ms=0
|
||||
)
|
||||
|
||||
mock_processor.assert_called_once()
|
||||
call_kwargs = mock_processor.call_args[1]
|
||||
assert call_kwargs['lyrics_file'] == "/path/to/lyrics.txt"
|
||||
|
||||
def test_create_lyrics_processor_with_subtitle_offset(self):
|
||||
"""Test creating LyricsProcessor with subtitle offset."""
|
||||
with patch('backend.workers.lyrics_worker.LyricsProcessor') as mock_processor:
|
||||
from backend.workers.lyrics_worker import create_lyrics_processor
|
||||
|
||||
result = create_lyrics_processor(
|
||||
subtitle_offset_ms=500
|
||||
)
|
||||
|
||||
mock_processor.assert_called_once()
|
||||
call_kwargs = mock_processor.call_args[1]
|
||||
assert call_kwargs['subtitle_offset_ms'] == 500
|
||||
|
||||
|
||||
class TestLyricsOverrideParameters:
|
||||
"""Tests for lyrics artist/title override functionality."""
|
||||
|
||||
@pytest.fixture
|
||||
def mock_job_with_overrides(self):
|
||||
"""Create a mock job with lyrics override fields."""
|
||||
return Job(
|
||||
job_id="test123",
|
||||
status=JobStatus.PENDING,
|
||||
created_at=datetime.now(UTC),
|
||||
updated_at=datetime.now(UTC),
|
||||
artist="Beatles, The",
|
||||
title="Hey Jude - 2009 Remaster",
|
||||
lyrics_artist="The Beatles",
|
||||
lyrics_title="Hey Jude",
|
||||
subtitle_offset_ms=250,
|
||||
input_media_gcs_path="uploads/test123/song.flac"
|
||||
)
|
||||
|
||||
def test_job_uses_lyrics_artist_override(self, mock_job_with_overrides):
|
||||
"""Test that job uses lyrics_artist when searching for lyrics."""
|
||||
job = mock_job_with_overrides
|
||||
|
||||
# Use override if present, else fall back to main artist
|
||||
lyrics_search_artist = job.lyrics_artist or job.artist
|
||||
|
||||
assert lyrics_search_artist == "The Beatles"
|
||||
assert lyrics_search_artist != job.artist # Override is different
|
||||
|
||||
def test_job_uses_lyrics_title_override(self, mock_job_with_overrides):
|
||||
"""Test that job uses lyrics_title when searching for lyrics."""
|
||||
job = mock_job_with_overrides
|
||||
|
||||
# Use override if present, else fall back to main title
|
||||
lyrics_search_title = job.lyrics_title or job.title
|
||||
|
||||
assert lyrics_search_title == "Hey Jude"
|
||||
assert lyrics_search_title != job.title # Override is different
|
||||
|
||||
def test_job_falls_back_when_no_override(self):
|
||||
"""Test that job falls back to main artist/title when no override."""
|
||||
job = Job(
|
||||
job_id="test123",
|
||||
status=JobStatus.PENDING,
|
||||
created_at=datetime.now(UTC),
|
||||
updated_at=datetime.now(UTC),
|
||||
artist="Test Artist",
|
||||
title="Test Song",
|
||||
input_media_gcs_path="uploads/test123/song.flac"
|
||||
)
|
||||
|
||||
# When override is None, use main values
|
||||
lyrics_search_artist = job.lyrics_artist or job.artist
|
||||
lyrics_search_title = job.lyrics_title or job.title
|
||||
|
||||
assert lyrics_search_artist == "Test Artist"
|
||||
assert lyrics_search_title == "Test Song"
|
||||
|
||||
|
||||
class TestScreensWorker:
|
||||
"""Tests for screens_worker.py functions.
|
||||
|
||||
@@ -379,6 +481,71 @@ class TestVideoWorker:
|
||||
assert hasattr(video_worker, 'logger')
|
||||
|
||||
|
||||
class TestRenderVideoWorkerConfiguration:
|
||||
"""Tests for render_video_worker subtitle_offset_ms support."""
|
||||
|
||||
@pytest.fixture
|
||||
def mock_job_with_offset(self):
|
||||
"""Create a mock job with subtitle offset."""
|
||||
return Job(
|
||||
job_id="test123",
|
||||
status=JobStatus.RENDERING_VIDEO,
|
||||
created_at=datetime.now(UTC),
|
||||
updated_at=datetime.now(UTC),
|
||||
artist="Test Artist",
|
||||
title="Test Song",
|
||||
subtitle_offset_ms=500,
|
||||
input_media_gcs_path="uploads/test123/song.flac"
|
||||
)
|
||||
|
||||
def test_job_has_subtitle_offset_ms(self, mock_job_with_offset):
|
||||
"""Test that job has subtitle_offset_ms field."""
|
||||
assert mock_job_with_offset.subtitle_offset_ms == 500
|
||||
|
||||
def test_subtitle_offset_from_job(self, mock_job_with_offset):
|
||||
"""Test extracting subtitle offset from job with getattr."""
|
||||
job = mock_job_with_offset
|
||||
|
||||
# This mirrors the logic in render_video_worker.py
|
||||
subtitle_offset = getattr(job, 'subtitle_offset_ms', 0) or 0
|
||||
|
||||
assert subtitle_offset == 500
|
||||
|
||||
def test_subtitle_offset_default_zero(self):
|
||||
"""Test that subtitle offset defaults to 0 when not set."""
|
||||
job = Job(
|
||||
job_id="test123",
|
||||
status=JobStatus.RENDERING_VIDEO,
|
||||
created_at=datetime.now(UTC),
|
||||
updated_at=datetime.now(UTC),
|
||||
artist="Test Artist",
|
||||
title="Test Song",
|
||||
input_media_gcs_path="uploads/test123/song.flac"
|
||||
)
|
||||
|
||||
subtitle_offset = getattr(job, 'subtitle_offset_ms', 0) or 0
|
||||
|
||||
assert subtitle_offset == 0
|
||||
|
||||
def test_subtitle_offset_negative_value(self):
|
||||
"""Test that subtitle offset can be negative (advance subtitles)."""
|
||||
job = Job(
|
||||
job_id="test123",
|
||||
status=JobStatus.RENDERING_VIDEO,
|
||||
created_at=datetime.now(UTC),
|
||||
updated_at=datetime.now(UTC),
|
||||
artist="Test Artist",
|
||||
title="Test Song",
|
||||
subtitle_offset_ms=-250, # Negative = advance subtitles
|
||||
input_media_gcs_path="uploads/test123/song.flac"
|
||||
)
|
||||
|
||||
subtitle_offset = getattr(job, 'subtitle_offset_ms', 0) or 0
|
||||
|
||||
# Negative values should be preserved (not converted to 0)
|
||||
assert subtitle_offset == -250
|
||||
|
||||
|
||||
class TestDownloadHelpers:
|
||||
"""Tests for download helper functions in workers."""
|
||||
|
||||
@@ -406,3 +573,299 @@ class TestDownloadHelpers:
|
||||
# Should have downloaded from GCS
|
||||
mock_storage.download_file.assert_called()
|
||||
|
||||
|
||||
class TestAudioWorkerModelConfiguration:
|
||||
"""Tests for audio worker model configuration parameters."""
|
||||
|
||||
def test_create_audio_processor_with_defaults(self):
|
||||
"""Test creating AudioProcessor with default models."""
|
||||
with patch('backend.workers.audio_worker.AudioProcessor') as mock_processor:
|
||||
from backend.workers.audio_worker import create_audio_processor
|
||||
|
||||
with tempfile.TemporaryDirectory() as temp_dir:
|
||||
result = create_audio_processor(temp_dir)
|
||||
|
||||
mock_processor.assert_called_once()
|
||||
call_kwargs = mock_processor.call_args[1]
|
||||
|
||||
# Should use default models
|
||||
assert call_kwargs['clean_instrumental_model'] == "model_bs_roformer_ep_317_sdr_12.9755.ckpt"
|
||||
assert call_kwargs['backing_vocals_models'] == ["mel_band_roformer_karaoke_aufr33_viperx_sdr_10.1956.ckpt"]
|
||||
assert call_kwargs['other_stems_models'] == ["htdemucs_6s.yaml"]
|
||||
|
||||
def test_create_audio_processor_with_custom_clean_model(self):
|
||||
"""Test creating AudioProcessor with custom clean instrumental model."""
|
||||
with patch('backend.workers.audio_worker.AudioProcessor') as mock_processor:
|
||||
from backend.workers.audio_worker import create_audio_processor
|
||||
|
||||
with tempfile.TemporaryDirectory() as temp_dir:
|
||||
result = create_audio_processor(
|
||||
temp_dir,
|
||||
clean_instrumental_model="custom_clean_model.ckpt"
|
||||
)
|
||||
|
||||
mock_processor.assert_called_once()
|
||||
call_kwargs = mock_processor.call_args[1]
|
||||
|
||||
# Should use custom clean model
|
||||
assert call_kwargs['clean_instrumental_model'] == "custom_clean_model.ckpt"
|
||||
# Other models should still be defaults
|
||||
assert call_kwargs['backing_vocals_models'] == ["mel_band_roformer_karaoke_aufr33_viperx_sdr_10.1956.ckpt"]
|
||||
|
||||
def test_create_audio_processor_with_custom_backing_models(self):
|
||||
"""Test creating AudioProcessor with custom backing vocals models."""
|
||||
with patch('backend.workers.audio_worker.AudioProcessor') as mock_processor:
|
||||
from backend.workers.audio_worker import create_audio_processor
|
||||
|
||||
with tempfile.TemporaryDirectory() as temp_dir:
|
||||
result = create_audio_processor(
|
||||
temp_dir,
|
||||
backing_vocals_models=["custom_bv1.ckpt", "custom_bv2.ckpt"]
|
||||
)
|
||||
|
||||
mock_processor.assert_called_once()
|
||||
call_kwargs = mock_processor.call_args[1]
|
||||
|
||||
# Should use custom backing vocals models
|
||||
assert call_kwargs['backing_vocals_models'] == ["custom_bv1.ckpt", "custom_bv2.ckpt"]
|
||||
|
||||
def test_create_audio_processor_with_custom_other_stems_models(self):
|
||||
"""Test creating AudioProcessor with custom other stems models."""
|
||||
with patch('backend.workers.audio_worker.AudioProcessor') as mock_processor:
|
||||
from backend.workers.audio_worker import create_audio_processor
|
||||
|
||||
with tempfile.TemporaryDirectory() as temp_dir:
|
||||
result = create_audio_processor(
|
||||
temp_dir,
|
||||
other_stems_models=["custom_demucs.yaml"]
|
||||
)
|
||||
|
||||
mock_processor.assert_called_once()
|
||||
call_kwargs = mock_processor.call_args[1]
|
||||
|
||||
# Should use custom other stems models
|
||||
assert call_kwargs['other_stems_models'] == ["custom_demucs.yaml"]
|
||||
|
||||
def test_create_audio_processor_with_all_custom_models(self):
|
||||
"""Test creating AudioProcessor with all custom models."""
|
||||
with patch('backend.workers.audio_worker.AudioProcessor') as mock_processor:
|
||||
from backend.workers.audio_worker import create_audio_processor
|
||||
|
||||
with tempfile.TemporaryDirectory() as temp_dir:
|
||||
result = create_audio_processor(
|
||||
temp_dir,
|
||||
clean_instrumental_model="custom_clean.ckpt",
|
||||
backing_vocals_models=["custom_bv.ckpt"],
|
||||
other_stems_models=["custom_stems.yaml"]
|
||||
)
|
||||
|
||||
mock_processor.assert_called_once()
|
||||
call_kwargs = mock_processor.call_args[1]
|
||||
|
||||
# All models should be custom
|
||||
assert call_kwargs['clean_instrumental_model'] == "custom_clean.ckpt"
|
||||
assert call_kwargs['backing_vocals_models'] == ["custom_bv.ckpt"]
|
||||
assert call_kwargs['other_stems_models'] == ["custom_stems.yaml"]
|
||||
|
||||
def test_job_model_fields_are_passed_to_processor(self):
|
||||
"""Test that job model fields can be passed to create_audio_processor."""
|
||||
mock_job = Job(
|
||||
job_id="test123",
|
||||
status=JobStatus.PENDING,
|
||||
created_at=datetime.now(UTC),
|
||||
updated_at=datetime.now(UTC),
|
||||
artist="Test",
|
||||
title="Test",
|
||||
clean_instrumental_model="job_clean.ckpt",
|
||||
backing_vocals_models=["job_bv.ckpt"],
|
||||
other_stems_models=["job_stems.yaml"]
|
||||
)
|
||||
|
||||
with patch('backend.workers.audio_worker.AudioProcessor') as mock_processor:
|
||||
from backend.workers.audio_worker import create_audio_processor
|
||||
|
||||
with tempfile.TemporaryDirectory() as temp_dir:
|
||||
# Simulate passing job model fields to create_audio_processor
|
||||
result = create_audio_processor(
|
||||
temp_dir,
|
||||
clean_instrumental_model=mock_job.clean_instrumental_model,
|
||||
backing_vocals_models=mock_job.backing_vocals_models,
|
||||
other_stems_models=mock_job.other_stems_models
|
||||
)
|
||||
|
||||
mock_processor.assert_called_once()
|
||||
call_kwargs = mock_processor.call_args[1]
|
||||
|
||||
# Models from job should be used
|
||||
assert call_kwargs['clean_instrumental_model'] == "job_clean.ckpt"
|
||||
assert call_kwargs['backing_vocals_models'] == ["job_bv.ckpt"]
|
||||
assert call_kwargs['other_stems_models'] == ["job_stems.yaml"]
|
||||
|
||||
def test_none_model_values_use_defaults(self):
|
||||
"""Test that None model values fall back to defaults."""
|
||||
mock_job = Job(
|
||||
job_id="test123",
|
||||
status=JobStatus.PENDING,
|
||||
created_at=datetime.now(UTC),
|
||||
updated_at=datetime.now(UTC),
|
||||
artist="Test",
|
||||
title="Test",
|
||||
clean_instrumental_model=None, # Not specified
|
||||
backing_vocals_models=None, # Not specified
|
||||
other_stems_models=None # Not specified
|
||||
)
|
||||
|
||||
with patch('backend.workers.audio_worker.AudioProcessor') as mock_processor:
|
||||
from backend.workers.audio_worker import create_audio_processor
|
||||
|
||||
with tempfile.TemporaryDirectory() as temp_dir:
|
||||
result = create_audio_processor(
|
||||
temp_dir,
|
||||
clean_instrumental_model=mock_job.clean_instrumental_model,
|
||||
backing_vocals_models=mock_job.backing_vocals_models,
|
||||
other_stems_models=mock_job.other_stems_models
|
||||
)
|
||||
|
||||
mock_processor.assert_called_once()
|
||||
call_kwargs = mock_processor.call_args[1]
|
||||
|
||||
# Should fall back to defaults
|
||||
assert call_kwargs['clean_instrumental_model'] == "model_bs_roformer_ep_317_sdr_12.9755.ckpt"
|
||||
assert call_kwargs['backing_vocals_models'] == ["mel_band_roformer_karaoke_aufr33_viperx_sdr_10.1956.ckpt"]
|
||||
assert call_kwargs['other_stems_models'] == ["htdemucs_6s.yaml"]
|
||||
|
||||
|
||||
class TestDownloadFromUrl:
|
||||
"""Tests for download_from_url function - URL-based audio download."""
|
||||
|
||||
def test_download_from_url_function_exists(self):
|
||||
"""Test that download_from_url function exists in audio_worker."""
|
||||
from backend.workers.audio_worker import download_from_url
|
||||
assert callable(download_from_url)
|
||||
|
||||
def test_download_from_url_signature(self):
|
||||
"""Test that download_from_url has the expected signature."""
|
||||
import inspect
|
||||
from backend.workers.audio_worker import download_from_url
|
||||
|
||||
sig = inspect.signature(download_from_url)
|
||||
params = list(sig.parameters.keys())
|
||||
|
||||
# Should have these parameters
|
||||
assert 'url' in params
|
||||
assert 'temp_dir' in params
|
||||
assert 'artist' in params
|
||||
assert 'title' in params
|
||||
assert 'job_manager' in params
|
||||
assert 'job_id' in params
|
||||
|
||||
def test_download_from_url_is_async(self):
|
||||
"""Test that download_from_url is an async function."""
|
||||
import inspect
|
||||
from backend.workers.audio_worker import download_from_url
|
||||
|
||||
assert inspect.iscoroutinefunction(download_from_url)
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_download_from_url_returns_none_for_invalid_url(self):
|
||||
"""Test that download_from_url handles errors gracefully."""
|
||||
from backend.workers.audio_worker import download_from_url
|
||||
|
||||
with tempfile.TemporaryDirectory() as temp_dir:
|
||||
# This should fail gracefully (no yt-dlp in test env or invalid URL)
|
||||
result = await download_from_url(
|
||||
url='not-a-valid-url',
|
||||
temp_dir=temp_dir,
|
||||
artist='Test',
|
||||
title='Test'
|
||||
)
|
||||
|
||||
# Should return None on error (graceful failure)
|
||||
assert result is None
|
||||
|
||||
|
||||
class TestDownloadAudioWithUrl:
|
||||
"""Tests for download_audio function with URL-based jobs."""
|
||||
|
||||
@pytest.fixture
|
||||
def mock_job_with_url(self):
|
||||
"""Create a mock job with a URL."""
|
||||
return Job(
|
||||
job_id="test123",
|
||||
status=JobStatus.PROCESSING,
|
||||
created_at=datetime.now(UTC),
|
||||
updated_at=datetime.now(UTC),
|
||||
artist="Test Artist",
|
||||
title="Test Song",
|
||||
url="https://www.youtube.com/watch?v=test123"
|
||||
)
|
||||
|
||||
@pytest.fixture
|
||||
def mock_job_manager(self, mock_job_with_url):
|
||||
"""Create a mock JobManager."""
|
||||
manager = MagicMock()
|
||||
manager.get_job.return_value = mock_job_with_url
|
||||
manager.update_job.return_value = None
|
||||
return manager
|
||||
|
||||
@pytest.fixture
|
||||
def mock_storage(self):
|
||||
"""Create a mock StorageService."""
|
||||
storage = MagicMock()
|
||||
return storage
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_download_audio_from_url(self, mock_job_with_url, mock_job_manager, mock_storage):
|
||||
"""Test download_audio routes to URL download when job has URL."""
|
||||
with patch('backend.workers.audio_worker.download_from_url', return_value='/tmp/test.wav') as mock_download:
|
||||
from backend.workers.audio_worker import download_audio
|
||||
|
||||
with tempfile.TemporaryDirectory() as temp_dir:
|
||||
result = await download_audio(
|
||||
"test123",
|
||||
temp_dir,
|
||||
mock_storage,
|
||||
mock_job_with_url,
|
||||
job_manager_instance=mock_job_manager
|
||||
)
|
||||
|
||||
# Should have called download_from_url
|
||||
mock_download.assert_called_once()
|
||||
# Check positional args
|
||||
args = mock_download.call_args[0]
|
||||
assert args[0] == "https://www.youtube.com/watch?v=test123" # url
|
||||
assert args[1] == temp_dir # temp_dir
|
||||
assert args[2] == "Test Artist" # artist
|
||||
assert args[3] == "Test Song" # title
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_download_audio_from_gcs_when_no_url(self, mock_job_manager, mock_storage):
|
||||
"""Test download_audio downloads from GCS when job has no URL."""
|
||||
mock_job_no_url = Job(
|
||||
job_id="test123",
|
||||
status=JobStatus.PROCESSING,
|
||||
created_at=datetime.now(UTC),
|
||||
updated_at=datetime.now(UTC),
|
||||
artist="Test Artist",
|
||||
title="Test Song",
|
||||
input_media_gcs_path="uploads/test123/song.flac"
|
||||
)
|
||||
|
||||
with patch('backend.workers.audio_worker.download_from_url') as mock_url_download:
|
||||
from backend.workers.audio_worker import download_audio
|
||||
|
||||
with tempfile.TemporaryDirectory() as temp_dir:
|
||||
result = await download_audio(
|
||||
"test123",
|
||||
temp_dir,
|
||||
mock_storage,
|
||||
mock_job_no_url,
|
||||
job_manager_instance=mock_job_manager
|
||||
)
|
||||
|
||||
# Should NOT have called download_from_url
|
||||
mock_url_download.assert_not_called()
|
||||
|
||||
# Should have called storage.download_file
|
||||
mock_storage.download_file.assert_called()
|
||||
|
||||
+159
-83
@@ -19,7 +19,7 @@ from backend.models.job import JobStatus
|
||||
from backend.services.job_manager import JobManager
|
||||
from backend.services.storage_service import StorageService
|
||||
from backend.config import get_settings
|
||||
from backend.workers.worker_logging import create_job_logger, setup_job_logging
|
||||
from backend.workers.worker_logging import create_job_logger, setup_job_logging, job_logging_context
|
||||
|
||||
# Import from karaoke_gen package
|
||||
from karaoke_gen.audio_processor import AudioProcessor
|
||||
@@ -34,7 +34,7 @@ AUDIO_WORKER_LOGGERS = [
|
||||
]
|
||||
|
||||
|
||||
async def download_from_url(url: str, temp_dir: str, artist: str, title: str) -> Optional[str]:
|
||||
async def download_from_url(url: str, temp_dir: str, artist: str, title: str, job_manager: JobManager = None, job_id: str = None) -> Optional[str]:
|
||||
"""
|
||||
Download audio from a URL (YouTube, etc.) using karaoke_gen.FileHandler.
|
||||
|
||||
@@ -43,11 +43,16 @@ async def download_from_url(url: str, temp_dir: str, artist: str, title: str) ->
|
||||
- Cookie support for authenticated downloads
|
||||
- Retry logic
|
||||
|
||||
If artist and/or title are not provided, attempts to extract them from
|
||||
the URL metadata.
|
||||
|
||||
Args:
|
||||
url: URL to download from
|
||||
temp_dir: Temporary directory to save to
|
||||
artist: Artist name for filename
|
||||
title: Song title for filename
|
||||
artist: Artist name for filename (can be None for auto-detection)
|
||||
title: Song title for filename (can be None for auto-detection)
|
||||
job_manager: Optional JobManager to update job with detected metadata
|
||||
job_id: Optional job ID to update
|
||||
|
||||
Returns:
|
||||
Path to downloaded audio file, or None if failed
|
||||
@@ -64,6 +69,34 @@ async def download_from_url(url: str, temp_dir: str, artist: str, title: str) ->
|
||||
dry_run=False
|
||||
)
|
||||
|
||||
# Try to extract metadata if artist or title not provided
|
||||
if not artist or not title:
|
||||
logger.info(f"Extracting metadata from URL: {url}")
|
||||
metadata = file_handler.extract_metadata_from_url(url)
|
||||
|
||||
if metadata:
|
||||
if not artist:
|
||||
artist = metadata.get('artist', 'Unknown')
|
||||
logger.info(f"Auto-detected artist: {artist}")
|
||||
if not title:
|
||||
title = metadata.get('title', 'Unknown')
|
||||
logger.info(f"Auto-detected title: {title}")
|
||||
|
||||
# Update job with detected metadata if job_manager provided
|
||||
if job_manager and job_id:
|
||||
update_data = {}
|
||||
if artist:
|
||||
update_data['artist'] = artist
|
||||
if title:
|
||||
update_data['title'] = title
|
||||
if update_data:
|
||||
job_manager.update_job(job_id, update_data)
|
||||
logger.info(f"Updated job {job_id} with detected metadata")
|
||||
else:
|
||||
logger.warning("Could not extract metadata from URL, using defaults")
|
||||
artist = artist or "Unknown"
|
||||
title = title or "Unknown"
|
||||
|
||||
# Create output filename (without extension)
|
||||
safe_artist = sanitize_filename(artist) if artist else "Unknown"
|
||||
safe_title = sanitize_filename(title) if title else "Unknown"
|
||||
@@ -112,7 +145,12 @@ async def download_from_url(url: str, temp_dir: str, artist: str, title: str) ->
|
||||
return None
|
||||
|
||||
|
||||
def create_audio_processor(temp_dir: str) -> AudioProcessor:
|
||||
def create_audio_processor(
|
||||
temp_dir: str,
|
||||
clean_instrumental_model: Optional[str] = None,
|
||||
backing_vocals_models: Optional[list] = None,
|
||||
other_stems_models: Optional[list] = None
|
||||
) -> AudioProcessor:
|
||||
"""
|
||||
Create an AudioProcessor instance configured for remote API processing.
|
||||
|
||||
@@ -120,10 +158,13 @@ def create_audio_processor(temp_dir: str) -> AudioProcessor:
|
||||
- Uses remote Modal API (via AUDIO_SEPARATOR_API_URL env var)
|
||||
- No local models needed (model_file_dir=None)
|
||||
- FLAC output format for quality
|
||||
- Standard model configurations from karaoke-gen CLI
|
||||
- Model configurations from job or CLI defaults
|
||||
|
||||
Args:
|
||||
temp_dir: Temporary directory for processing
|
||||
clean_instrumental_model: Model for clean instrumental separation (optional, uses default if not provided)
|
||||
backing_vocals_models: List of models for backing vocals separation (optional, uses default if not provided)
|
||||
other_stems_models: List of models for other stems separation (optional, uses default if not provided)
|
||||
|
||||
Returns:
|
||||
Configured AudioProcessor instance
|
||||
@@ -132,10 +173,10 @@ def create_audio_processor(temp_dir: str) -> AudioProcessor:
|
||||
audio_logger = logging.getLogger("karaoke_gen.audio_processor")
|
||||
audio_logger.setLevel(logging.INFO)
|
||||
|
||||
# Model configurations (same as CLI defaults)
|
||||
clean_instrumental_model = "model_bs_roformer_ep_317_sdr_12.9755.ckpt"
|
||||
backing_vocals_models = ["mel_band_roformer_karaoke_aufr33_viperx_sdr_10.1956.ckpt"]
|
||||
other_stems_models = ["htdemucs_6s.yaml"] # For 6-stem separation (bass, drums, etc.)
|
||||
# Model configurations - use provided values or CLI defaults
|
||||
effective_clean_model = clean_instrumental_model or "model_bs_roformer_ep_317_sdr_12.9755.ckpt"
|
||||
effective_backing_models = backing_vocals_models or ["mel_band_roformer_karaoke_aufr33_viperx_sdr_10.1956.ckpt"]
|
||||
effective_other_models = other_stems_models or ["htdemucs_6s.yaml"] # For 6-stem separation (bass, drums, etc.)
|
||||
|
||||
# FFmpeg command for combining audio files (must be a string, not a list)
|
||||
ffmpeg_base_command = "ffmpeg -hide_banner -loglevel error -nostats -y"
|
||||
@@ -146,9 +187,9 @@ def create_audio_processor(temp_dir: str) -> AudioProcessor:
|
||||
log_formatter=None, # Not needed for our use case
|
||||
model_file_dir=None, # No local models, using remote API
|
||||
lossless_output_format="FLAC",
|
||||
clean_instrumental_model=clean_instrumental_model,
|
||||
backing_vocals_models=backing_vocals_models,
|
||||
other_stems_models=other_stems_models,
|
||||
clean_instrumental_model=effective_clean_model,
|
||||
backing_vocals_models=effective_backing_models,
|
||||
other_stems_models=effective_other_models,
|
||||
ffmpeg_base_command=ffmpeg_base_command
|
||||
)
|
||||
|
||||
@@ -200,71 +241,88 @@ async def process_audio_separation(job_id: str) -> bool:
|
||||
job_log.info(f"Created temp directory: {temp_dir}")
|
||||
|
||||
try:
|
||||
job_log.info(f"Starting audio separation for {job.artist} - {job.title}")
|
||||
logger.info(f"Starting audio separation for job {job_id}")
|
||||
|
||||
# Ensure AUDIO_SEPARATOR_API_URL is set
|
||||
api_url = os.environ.get("AUDIO_SEPARATOR_API_URL")
|
||||
if not api_url:
|
||||
raise Exception("AUDIO_SEPARATOR_API_URL environment variable not set. "
|
||||
"Cannot perform audio separation without remote API access.")
|
||||
job_log.info(f"Audio separator API: {api_url}")
|
||||
|
||||
# Download audio file from GCS
|
||||
job_log.info("Downloading audio file from GCS...")
|
||||
audio_path = await download_audio(job_id, temp_dir, storage, job)
|
||||
if not audio_path:
|
||||
raise Exception("Failed to download audio file")
|
||||
job_log.info(f"Audio downloaded: {os.path.basename(audio_path)}")
|
||||
|
||||
# Update progress using state_data (don't change status during parallel processing)
|
||||
# The status is managed at a higher level - workers just track their progress
|
||||
job_manager.update_state_data(job_id, 'audio_progress', {
|
||||
'stage': 'separating_stage1',
|
||||
'progress': 10,
|
||||
'message': 'Starting audio separation (Stage 1: Clean instrumental)'
|
||||
})
|
||||
|
||||
# Create AudioProcessor instance (reuses karaoke_gen code)
|
||||
job_log.info("Creating AudioProcessor instance...")
|
||||
audio_processor = create_audio_processor(temp_dir)
|
||||
|
||||
# Format artist-title for file naming (matches CLI behavior)
|
||||
artist_title = f"{job.artist} - {job.title}"
|
||||
|
||||
# Run audio separation (calls Modal API internally)
|
||||
# This returns a dict with paths to all separated stems
|
||||
job_log.info("Starting audio separation (this may take 5-10 minutes)...")
|
||||
job_log.info(" Stage 1: Clean instrumental separation (MDX models)")
|
||||
job_log.info(" Stage 2: Backing vocals separation (Demucs model)")
|
||||
logger.info(f"Job {job_id}: Calling audio_processor.process_audio_separation()")
|
||||
separation_result = audio_processor.process_audio_separation(
|
||||
audio_file=audio_path,
|
||||
artist_title=artist_title,
|
||||
track_output_dir=temp_dir
|
||||
)
|
||||
|
||||
job_log.info("Audio separation complete!")
|
||||
job_log.info(f" Generated {len(separation_result)} stem files")
|
||||
logger.info(f"Job {job_id}: Audio separation complete, organizing results")
|
||||
|
||||
# Update progress using state_data (don't change status during parallel processing)
|
||||
job_manager.update_state_data(job_id, 'audio_progress', {
|
||||
'stage': 'audio_complete',
|
||||
'progress': 45,
|
||||
'message': 'Audio separation complete, uploading stems'
|
||||
})
|
||||
|
||||
# Upload all stems to GCS
|
||||
await upload_separation_results(job_id, separation_result, storage, job_manager)
|
||||
|
||||
logger.info(f"Job {job_id}: All stems uploaded successfully")
|
||||
|
||||
# Mark audio processing complete
|
||||
# This will check if lyrics are also complete and transition to next stage if so
|
||||
job_manager.mark_audio_complete(job_id)
|
||||
|
||||
return True
|
||||
# Use job_logging_context for proper log isolation when multiple jobs run concurrently
|
||||
# This ensures logs from third-party libraries (karaoke_gen.audio_processor) are
|
||||
# only captured by this job's handler, not handlers from other concurrent jobs
|
||||
with job_logging_context(job_id):
|
||||
job_log.info(f"Starting audio separation for {job.artist} - {job.title}")
|
||||
logger.info(f"Starting audio separation for job {job_id}")
|
||||
|
||||
# Ensure AUDIO_SEPARATOR_API_URL is set
|
||||
api_url = os.environ.get("AUDIO_SEPARATOR_API_URL")
|
||||
if not api_url:
|
||||
raise Exception("AUDIO_SEPARATOR_API_URL environment variable not set. "
|
||||
"Cannot perform audio separation without remote API access.")
|
||||
job_log.info(f"Audio separator API: {api_url}")
|
||||
|
||||
# Download audio file from GCS or URL
|
||||
job_log.info("Downloading audio file...")
|
||||
audio_path = await download_audio(job_id, temp_dir, storage, job, job_manager_instance=job_manager)
|
||||
if not audio_path:
|
||||
raise Exception("Failed to download audio file")
|
||||
job_log.info(f"Audio downloaded: {os.path.basename(audio_path)}")
|
||||
|
||||
# Update progress using state_data (don't change status during parallel processing)
|
||||
# The status is managed at a higher level - workers just track their progress
|
||||
job_manager.update_state_data(job_id, 'audio_progress', {
|
||||
'stage': 'separating_stage1',
|
||||
'progress': 10,
|
||||
'message': 'Starting audio separation (Stage 1: Clean instrumental)'
|
||||
})
|
||||
|
||||
# Create AudioProcessor instance (reuses karaoke_gen code)
|
||||
# Use model configuration from job if provided, otherwise use defaults
|
||||
job_log.info("Creating AudioProcessor instance...")
|
||||
if job.clean_instrumental_model:
|
||||
job_log.info(f" Using clean instrumental model: {job.clean_instrumental_model}")
|
||||
if job.backing_vocals_models:
|
||||
job_log.info(f" Using backing vocals models: {job.backing_vocals_models}")
|
||||
if job.other_stems_models:
|
||||
job_log.info(f" Using other stems models: {job.other_stems_models}")
|
||||
|
||||
audio_processor = create_audio_processor(
|
||||
temp_dir,
|
||||
clean_instrumental_model=job.clean_instrumental_model,
|
||||
backing_vocals_models=job.backing_vocals_models,
|
||||
other_stems_models=job.other_stems_models
|
||||
)
|
||||
|
||||
# Format artist-title for file naming (matches CLI behavior)
|
||||
artist_title = f"{job.artist} - {job.title}"
|
||||
|
||||
# Run audio separation (calls Modal API internally)
|
||||
# This returns a dict with paths to all separated stems
|
||||
job_log.info("Starting audio separation (this may take 5-10 minutes)...")
|
||||
job_log.info(" Stage 1: Clean instrumental separation (MDX models)")
|
||||
job_log.info(" Stage 2: Backing vocals separation (Demucs model)")
|
||||
logger.info(f"Job {job_id}: Calling audio_processor.process_audio_separation()")
|
||||
separation_result = audio_processor.process_audio_separation(
|
||||
audio_file=audio_path,
|
||||
artist_title=artist_title,
|
||||
track_output_dir=temp_dir
|
||||
)
|
||||
|
||||
job_log.info("Audio separation complete!")
|
||||
job_log.info(f" Generated {len(separation_result)} stem files")
|
||||
logger.info(f"Job {job_id}: Audio separation complete, organizing results")
|
||||
|
||||
# Update progress using state_data (don't change status during parallel processing)
|
||||
job_manager.update_state_data(job_id, 'audio_progress', {
|
||||
'stage': 'audio_complete',
|
||||
'progress': 45,
|
||||
'message': 'Audio separation complete, uploading stems'
|
||||
})
|
||||
|
||||
# Upload all stems to GCS
|
||||
await upload_separation_results(job_id, separation_result, storage, job_manager)
|
||||
|
||||
logger.info(f"Job {job_id}: All stems uploaded successfully")
|
||||
|
||||
# Mark audio processing complete
|
||||
# This will check if lyrics are also complete and transition to next stage if so
|
||||
job_manager.mark_audio_complete(job_id)
|
||||
|
||||
return True
|
||||
|
||||
except Exception as e:
|
||||
logger.error(f"Job {job_id}: Audio separation failed: {e}", exc_info=True)
|
||||
@@ -293,7 +351,8 @@ async def download_audio(
|
||||
job_id: str,
|
||||
temp_dir: str,
|
||||
storage: StorageService,
|
||||
job
|
||||
job,
|
||||
job_manager_instance: JobManager = None
|
||||
) -> Optional[str]:
|
||||
"""
|
||||
Download or fetch audio file to local temp directory.
|
||||
@@ -302,6 +361,13 @@ async def download_audio(
|
||||
1. Uploaded file: Download from GCS using input_media_gcs_path
|
||||
2. URL (YouTube, etc.): Download using yt-dlp or other tools
|
||||
|
||||
Args:
|
||||
job_id: Job ID
|
||||
temp_dir: Temporary directory to save to
|
||||
storage: StorageService instance
|
||||
job: Job object with URL or GCS path
|
||||
job_manager_instance: Optional JobManager instance for updating job metadata
|
||||
|
||||
Returns:
|
||||
Path to downloaded audio file, or None if failed
|
||||
"""
|
||||
@@ -326,7 +392,18 @@ async def download_audio(
|
||||
# Case 3: Fresh URL that needs downloading
|
||||
if job.url:
|
||||
logger.info(f"Job {job_id}: Downloading from URL: {job.url}")
|
||||
local_path = await download_from_url(job.url, temp_dir, job.artist, job.title)
|
||||
|
||||
# Use provided job_manager or create new one
|
||||
jm = job_manager_instance or JobManager()
|
||||
|
||||
local_path = await download_from_url(
|
||||
job.url,
|
||||
temp_dir,
|
||||
job.artist,
|
||||
job.title,
|
||||
job_manager=jm,
|
||||
job_id=job_id
|
||||
)
|
||||
|
||||
if local_path and os.path.exists(local_path):
|
||||
# Upload to GCS and update job
|
||||
@@ -334,9 +411,8 @@ async def download_audio(
|
||||
url = storage.upload_file(local_path, gcs_path)
|
||||
|
||||
# Update job with GCS path for lyrics worker
|
||||
job_manager = JobManager()
|
||||
job_manager.firestore.update_job(job_id, {'input_media_gcs_path': gcs_path})
|
||||
job_manager.update_file_url(job_id, 'input', 'audio', url)
|
||||
jm.update_job(job_id, {'input_media_gcs_path': gcs_path})
|
||||
jm.update_file_url(job_id, 'input', 'audio', url)
|
||||
|
||||
logger.info(f"Job {job_id}: Downloaded and uploaded audio to GCS: {gcs_path}")
|
||||
return local_path
|
||||
|
||||
+170
-138
@@ -26,7 +26,7 @@ from backend.models.job import JobStatus
|
||||
from karaoke_gen.utils import sanitize_filename
|
||||
from backend.services.job_manager import JobManager
|
||||
from backend.services.storage_service import StorageService
|
||||
from backend.workers.worker_logging import create_job_logger, setup_job_logging
|
||||
from backend.workers.worker_logging import create_job_logger, setup_job_logging, job_logging_context
|
||||
from backend.workers.style_helper import load_style_config
|
||||
|
||||
# Import from karaoke_gen package
|
||||
@@ -36,27 +36,21 @@ from karaoke_gen.lyrics_processor import LyricsProcessor
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
|
||||
# Loggers to capture for lyrics worker (includes LyricsTranscriber internals)
|
||||
# Loggers to capture for lyrics worker (top-level package loggers only)
|
||||
# Note: We only capture at top-level package loggers to avoid duplicate log entries.
|
||||
# Logs from sub-modules (e.g., lyrics_transcriber.correction.anchor_sequence) will
|
||||
# propagate up to their parent logger (lyrics_transcriber) where they get captured.
|
||||
LYRICS_WORKER_LOGGERS = [
|
||||
"karaoke_gen.lyrics_processor",
|
||||
"lyrics_transcriber",
|
||||
"lyrics_transcriber.core",
|
||||
"lyrics_transcriber.core.controller",
|
||||
"lyrics_transcriber.lyrics",
|
||||
"lyrics_transcriber.lyrics.genius",
|
||||
"lyrics_transcriber.lyrics.spotify",
|
||||
"lyrics_transcriber.lyrics.musixmatch",
|
||||
"lyrics_transcriber.lyrics.lrclib",
|
||||
"lyrics_transcriber.correction",
|
||||
"lyrics_transcriber.correction.corrector",
|
||||
"lyrics_transcriber.correction.anchor_sequence",
|
||||
"lyrics_transcriber.transcribers",
|
||||
"lyrics_transcriber.transcribers.audioshake",
|
||||
"lyrics_transcriber.output",
|
||||
]
|
||||
|
||||
|
||||
def create_lyrics_processor(style_params_json: Optional[str] = None) -> LyricsProcessor:
|
||||
def create_lyrics_processor(
|
||||
style_params_json: Optional[str] = None,
|
||||
lyrics_file: Optional[str] = None,
|
||||
subtitle_offset_ms: int = 0
|
||||
) -> LyricsProcessor:
|
||||
"""
|
||||
Create a LyricsProcessor instance configured for Cloud Run processing.
|
||||
|
||||
@@ -68,6 +62,8 @@ def create_lyrics_processor(style_params_json: Optional[str] = None) -> LyricsPr
|
||||
|
||||
Args:
|
||||
style_params_json: Optional path to style parameters JSON file
|
||||
lyrics_file: Optional path to user-provided lyrics file (overrides API fetch)
|
||||
subtitle_offset_ms: Offset for subtitle timing in milliseconds
|
||||
|
||||
Returns:
|
||||
Configured LyricsProcessor instance
|
||||
@@ -79,11 +75,11 @@ def create_lyrics_processor(style_params_json: Optional[str] = None) -> LyricsPr
|
||||
return LyricsProcessor(
|
||||
logger=lyrics_logger,
|
||||
style_params_json=style_params_json,
|
||||
lyrics_file=None, # Will fetch from APIs
|
||||
lyrics_file=lyrics_file, # Use user-provided lyrics if available
|
||||
skip_transcription=False, # We want transcription
|
||||
skip_transcription_review=True, # Skip interactive review (use React UI instead)
|
||||
render_video=False, # Skip video generation for now (will be done after review)
|
||||
subtitle_offset_ms=0 # No offset by default
|
||||
subtitle_offset_ms=subtitle_offset_ms
|
||||
)
|
||||
|
||||
|
||||
@@ -131,126 +127,162 @@ async def process_lyrics_transcription(job_id: str) -> bool:
|
||||
temp_dir = None
|
||||
|
||||
try:
|
||||
job = job_manager.get_job(job_id)
|
||||
if not job:
|
||||
logger.error(f"Job {job_id} not found")
|
||||
job_log.error(f"Job {job_id} not found in Firestore!")
|
||||
return False
|
||||
|
||||
# Create temporary working directory
|
||||
temp_dir = tempfile.mkdtemp(prefix=f"karaoke_lyrics_{job_id}_")
|
||||
job_log.info(f"Created temp directory: {temp_dir}")
|
||||
job_log.info(f"Starting lyrics transcription for {job.artist} - {job.title}")
|
||||
job_log.info(f"Log capture enabled for: {', '.join(LYRICS_WORKER_LOGGERS)}")
|
||||
logger.info(f"Starting lyrics transcription for job {job_id}")
|
||||
|
||||
# Log environment configuration
|
||||
job_log.info("Checking API configuration...")
|
||||
if os.environ.get("GENIUS_API_TOKEN"):
|
||||
job_log.info(" Genius API: configured")
|
||||
else:
|
||||
job_log.warning(" Genius API: NOT configured")
|
||||
if os.environ.get("SPOTIFY_COOKIE_SP_DC"):
|
||||
job_log.info(" Spotify: configured")
|
||||
else:
|
||||
job_log.warning(" Spotify: NOT configured")
|
||||
if os.environ.get("RAPIDAPI_KEY"):
|
||||
job_log.info(" Musixmatch (RapidAPI): configured")
|
||||
else:
|
||||
job_log.warning(" Musixmatch (RapidAPI): NOT configured")
|
||||
|
||||
# Ensure required environment variables are set
|
||||
if not os.environ.get("AUDIOSHAKE_API_TOKEN"):
|
||||
job_log.warning("AUDIOSHAKE_API_TOKEN not set - transcription may fail")
|
||||
logger.warning("AUDIOSHAKE_API_TOKEN not set - transcription may fail")
|
||||
|
||||
# Download audio file from GCS (waits for audio worker if URL job)
|
||||
job_log.info("Downloading audio file from GCS...")
|
||||
audio_path = await download_audio(job_id, temp_dir, storage, job, job_manager)
|
||||
if not audio_path:
|
||||
raise Exception("Failed to download audio file")
|
||||
job_log.info(f"Audio downloaded: {os.path.basename(audio_path)}")
|
||||
|
||||
# Update progress using state_data (don't change status during parallel processing)
|
||||
# The status is managed at a higher level - workers just track their progress
|
||||
job_manager.update_state_data(job_id, 'lyrics_progress', {
|
||||
'stage': 'transcribing',
|
||||
'progress': 10,
|
||||
'message': 'Starting lyrics transcription via AudioShake'
|
||||
})
|
||||
|
||||
# Load style configuration (downloads assets from GCS if available)
|
||||
# This is needed for max_line_length and other style settings in LyricsTranscriber
|
||||
job_log.info("Loading style configuration for lyrics processing...")
|
||||
style_config = await load_style_config(job, storage, temp_dir)
|
||||
style_params_json_path = style_config.get_style_params_path() if style_config.has_custom_styles() else None
|
||||
|
||||
if style_params_json_path:
|
||||
job_log.info(f"Using custom style params: {style_params_json_path}")
|
||||
# Log the contents of the style params for debugging
|
||||
try:
|
||||
with open(style_params_json_path, 'r') as f:
|
||||
style_content = json.load(f)
|
||||
job_log.info(f"Style params sections: {list(style_content.keys())}")
|
||||
if 'karaoke' in style_content:
|
||||
karaoke_style = style_content['karaoke']
|
||||
job_log.info(f" karaoke.background_image: {karaoke_style.get('background_image', 'NOT SET')}")
|
||||
job_log.info(f" karaoke.font_path: {karaoke_style.get('font_path', 'NOT SET')}")
|
||||
except Exception as e:
|
||||
job_log.warning(f"Could not read style params for logging: {e}")
|
||||
else:
|
||||
job_log.info("No custom style params found, using defaults")
|
||||
|
||||
# Create LyricsProcessor instance (reuses karaoke_gen code)
|
||||
job_log.info("Creating LyricsProcessor instance...")
|
||||
job_log.info(f" style_params_json: {style_params_json_path}")
|
||||
lyrics_processor = create_lyrics_processor(style_params_json=style_params_json_path)
|
||||
|
||||
# Run transcription + correction
|
||||
# This will:
|
||||
# 1. Fetch reference lyrics from Genius/Spotify/Musixmatch/LRCLib
|
||||
# 2. Transcribe audio via AudioShake API
|
||||
# 3. Run automatic correction with lyrics_transcriber
|
||||
# 4. Generate corrections JSON (no interactive review)
|
||||
# 5. Skip video generation (render_video=False)
|
||||
job_log.info("Starting LyricsTranscriber processing...")
|
||||
job_log.info(f" Artist: {job.artist}")
|
||||
job_log.info(f" Title: {job.title}")
|
||||
logger.info(f"Job {job_id}: Calling lyrics_processor.transcribe_lyrics()")
|
||||
|
||||
# Run transcription in thread pool to avoid blocking the event loop
|
||||
# (transcribe_lyrics is synchronous and takes 1-2 minutes)
|
||||
result = await asyncio.to_thread(
|
||||
lyrics_processor.transcribe_lyrics,
|
||||
input_audio_wav=audio_path,
|
||||
artist=job.artist,
|
||||
title=job.title,
|
||||
track_output_dir=temp_dir
|
||||
)
|
||||
|
||||
job_log.info("Transcription processing complete")
|
||||
logger.info(f"Job {job_id}: Transcription complete, uploading results")
|
||||
|
||||
# Upload lyrics results to GCS
|
||||
job_log.info("Uploading lyrics results to GCS...")
|
||||
await upload_lyrics_results(job_id, temp_dir, result, storage, job_manager, job_log)
|
||||
|
||||
job_log.info("All lyrics data uploaded successfully")
|
||||
logger.info(f"Job {job_id}: All lyrics data uploaded successfully")
|
||||
|
||||
# Update progress using state_data (don't change status during parallel processing)
|
||||
job_manager.update_state_data(job_id, 'lyrics_progress', {
|
||||
'stage': 'lyrics_complete',
|
||||
'progress': 45,
|
||||
'message': 'Lyrics transcription complete'
|
||||
})
|
||||
|
||||
# Mark lyrics processing complete
|
||||
# This will check if audio is also complete and transition to next stage if so
|
||||
job_log.info("Lyrics worker complete, checking if audio is also done...")
|
||||
job_manager.mark_lyrics_complete(job_id)
|
||||
|
||||
return True
|
||||
# Use job_logging_context for proper log isolation when multiple jobs run concurrently
|
||||
# This ensures logs from third-party libraries (lyrics_transcriber) are only captured
|
||||
# by this job's handler, not handlers from other concurrent jobs
|
||||
with job_logging_context(job_id):
|
||||
job = job_manager.get_job(job_id)
|
||||
if not job:
|
||||
logger.error(f"Job {job_id} not found")
|
||||
job_log.error(f"Job {job_id} not found in Firestore!")
|
||||
return False
|
||||
|
||||
# Create temporary working directory
|
||||
temp_dir = tempfile.mkdtemp(prefix=f"karaoke_lyrics_{job_id}_")
|
||||
job_log.info(f"Created temp directory: {temp_dir}")
|
||||
job_log.info(f"Starting lyrics transcription for {job.artist} - {job.title}")
|
||||
job_log.info(f"Log capture enabled for: {', '.join(LYRICS_WORKER_LOGGERS)}")
|
||||
logger.info(f"Starting lyrics transcription for job {job_id}")
|
||||
|
||||
# Log environment configuration
|
||||
job_log.info("Checking API configuration...")
|
||||
if os.environ.get("GENIUS_API_TOKEN"):
|
||||
job_log.info(" Genius API: configured")
|
||||
else:
|
||||
job_log.warning(" Genius API: NOT configured")
|
||||
if os.environ.get("SPOTIFY_COOKIE_SP_DC"):
|
||||
job_log.info(" Spotify: configured")
|
||||
else:
|
||||
job_log.warning(" Spotify: NOT configured")
|
||||
if os.environ.get("RAPIDAPI_KEY"):
|
||||
job_log.info(" Musixmatch (RapidAPI): configured")
|
||||
else:
|
||||
job_log.warning(" Musixmatch (RapidAPI): NOT configured")
|
||||
|
||||
# Ensure required environment variables are set
|
||||
if not os.environ.get("AUDIOSHAKE_API_TOKEN"):
|
||||
job_log.warning("AUDIOSHAKE_API_TOKEN not set - transcription may fail")
|
||||
logger.warning("AUDIOSHAKE_API_TOKEN not set - transcription may fail")
|
||||
|
||||
# Download audio file from GCS (waits for audio worker if URL job)
|
||||
job_log.info("Downloading audio file from GCS...")
|
||||
audio_path = await download_audio(job_id, temp_dir, storage, job, job_manager)
|
||||
if not audio_path:
|
||||
raise Exception("Failed to download audio file")
|
||||
job_log.info(f"Audio downloaded: {os.path.basename(audio_path)}")
|
||||
|
||||
# Update progress using state_data (don't change status during parallel processing)
|
||||
# The status is managed at a higher level - workers just track their progress
|
||||
job_manager.update_state_data(job_id, 'lyrics_progress', {
|
||||
'stage': 'transcribing',
|
||||
'progress': 10,
|
||||
'message': 'Starting lyrics transcription via AudioShake'
|
||||
})
|
||||
|
||||
# Load style configuration (downloads assets from GCS if available)
|
||||
# This is needed for max_line_length and other style settings in LyricsTranscriber
|
||||
job_log.info("Loading style configuration for lyrics processing...")
|
||||
style_config = await load_style_config(job, storage, temp_dir)
|
||||
style_params_json_path = style_config.get_style_params_path() if style_config.has_custom_styles() else None
|
||||
|
||||
if style_params_json_path:
|
||||
job_log.info(f"Using custom style params: {style_params_json_path}")
|
||||
# Log the contents of the style params for debugging
|
||||
try:
|
||||
with open(style_params_json_path, 'r') as f:
|
||||
style_content = json.load(f)
|
||||
job_log.info(f"Style params sections: {list(style_content.keys())}")
|
||||
if 'karaoke' in style_content:
|
||||
karaoke_style = style_content['karaoke']
|
||||
job_log.info(f" karaoke.background_image: {karaoke_style.get('background_image', 'NOT SET')}")
|
||||
job_log.info(f" karaoke.font_path: {karaoke_style.get('font_path', 'NOT SET')}")
|
||||
except Exception as e:
|
||||
job_log.warning(f"Could not read style params for logging: {e}")
|
||||
else:
|
||||
job_log.info("No custom style params found, using defaults")
|
||||
|
||||
# Download user-provided lyrics file if available
|
||||
lyrics_file_path = None
|
||||
if hasattr(job, 'lyrics_file_gcs_path') and job.lyrics_file_gcs_path:
|
||||
lyrics_file_path = os.path.join(temp_dir, "user_lyrics.txt")
|
||||
job_log.info(f"Downloading user-provided lyrics file: {job.lyrics_file_gcs_path}")
|
||||
storage.download_file(job.lyrics_file_gcs_path, lyrics_file_path)
|
||||
job_log.info(f"User lyrics file downloaded to: {lyrics_file_path}")
|
||||
|
||||
# Get lyrics configuration from job
|
||||
subtitle_offset = getattr(job, 'subtitle_offset_ms', 0) or 0
|
||||
if subtitle_offset != 0:
|
||||
job_log.info(f"Subtitle offset: {subtitle_offset}ms")
|
||||
|
||||
# Create LyricsProcessor instance (reuses karaoke_gen code)
|
||||
job_log.info("Creating LyricsProcessor instance...")
|
||||
job_log.info(f" style_params_json: {style_params_json_path}")
|
||||
job_log.info(f" lyrics_file: {lyrics_file_path}")
|
||||
job_log.info(f" subtitle_offset_ms: {subtitle_offset}")
|
||||
lyrics_processor = create_lyrics_processor(
|
||||
style_params_json=style_params_json_path,
|
||||
lyrics_file=lyrics_file_path,
|
||||
subtitle_offset_ms=subtitle_offset
|
||||
)
|
||||
|
||||
# Run transcription + correction
|
||||
# This will:
|
||||
# 1. Fetch reference lyrics from Genius/Spotify/Musixmatch/LRCLib
|
||||
# 2. Transcribe audio via AudioShake API
|
||||
# 3. Run automatic correction with lyrics_transcriber
|
||||
# 4. Generate corrections JSON (no interactive review)
|
||||
# 5. Skip video generation (render_video=False)
|
||||
|
||||
# Use lyrics_artist/lyrics_title overrides if provided, else fall back to job artist/title
|
||||
lyrics_search_artist = getattr(job, 'lyrics_artist', None) or job.artist
|
||||
lyrics_search_title = getattr(job, 'lyrics_title', None) or job.title
|
||||
|
||||
job_log.info("Starting LyricsTranscriber processing...")
|
||||
job_log.info(f" Artist: {job.artist}")
|
||||
job_log.info(f" Title: {job.title}")
|
||||
if lyrics_search_artist != job.artist:
|
||||
job_log.info(f" Lyrics search artist override: {lyrics_search_artist}")
|
||||
if lyrics_search_title != job.title:
|
||||
job_log.info(f" Lyrics search title override: {lyrics_search_title}")
|
||||
logger.info(f"Job {job_id}: Calling lyrics_processor.transcribe_lyrics()")
|
||||
|
||||
# Run transcription in thread pool to avoid blocking the event loop
|
||||
# (transcribe_lyrics is synchronous and takes 1-2 minutes)
|
||||
# Note: We pass the original artist/title for file naming, but the LyricsProcessor
|
||||
# will use lyrics_artist/lyrics_title for the search (set via constructor args)
|
||||
result = await asyncio.to_thread(
|
||||
lyrics_processor.transcribe_lyrics,
|
||||
input_audio_wav=audio_path,
|
||||
artist=job.artist, # Original artist for file naming
|
||||
title=job.title, # Original title for file naming
|
||||
track_output_dir=temp_dir,
|
||||
lyrics_artist=lyrics_search_artist, # Override for lyrics search
|
||||
lyrics_title=lyrics_search_title # Override for lyrics search
|
||||
)
|
||||
|
||||
job_log.info("Transcription processing complete")
|
||||
logger.info(f"Job {job_id}: Transcription complete, uploading results")
|
||||
|
||||
# Upload lyrics results to GCS
|
||||
job_log.info("Uploading lyrics results to GCS...")
|
||||
await upload_lyrics_results(job_id, temp_dir, result, storage, job_manager, job_log)
|
||||
|
||||
job_log.info("All lyrics data uploaded successfully")
|
||||
logger.info(f"Job {job_id}: All lyrics data uploaded successfully")
|
||||
|
||||
# Update progress using state_data (don't change status during parallel processing)
|
||||
job_manager.update_state_data(job_id, 'lyrics_progress', {
|
||||
'stage': 'lyrics_complete',
|
||||
'progress': 45,
|
||||
'message': 'Lyrics transcription complete'
|
||||
})
|
||||
|
||||
# Mark lyrics processing complete
|
||||
# This will check if audio is also complete and transition to next stage if so
|
||||
job_log.info("Lyrics worker complete, checking if audio is also done...")
|
||||
job_manager.mark_lyrics_complete(job_id)
|
||||
|
||||
return True
|
||||
|
||||
except Exception as e:
|
||||
job_log.error(f"Lyrics transcription failed: {str(e)}", exc_info=True)
|
||||
|
||||
@@ -26,7 +26,7 @@ from backend.models.job import JobStatus
|
||||
from backend.services.job_manager import JobManager
|
||||
from backend.services.storage_service import StorageService
|
||||
from backend.config import get_settings
|
||||
from backend.workers.worker_logging import create_job_logger, setup_job_logging
|
||||
from backend.workers.worker_logging import create_job_logger, setup_job_logging, job_logging_context
|
||||
|
||||
# Import from lyrics_transcriber (submodule)
|
||||
from lyrics_transcriber.output.generator import OutputGenerator
|
||||
@@ -85,189 +85,198 @@ async def process_render_video(job_id: str) -> bool:
|
||||
logger.info(f"Job {job_id}: Starting video render (post-review)")
|
||||
|
||||
try:
|
||||
# Transition to RENDERING_VIDEO
|
||||
job_manager.transition_to_state(
|
||||
job_id=job_id,
|
||||
new_status=JobStatus.RENDERING_VIDEO,
|
||||
progress=75,
|
||||
message="Rendering karaoke video with corrected lyrics"
|
||||
)
|
||||
|
||||
with tempfile.TemporaryDirectory() as temp_dir:
|
||||
job_log.info(f"Created temp directory: {temp_dir}")
|
||||
# 1. Download corrected corrections data
|
||||
corrections_path = os.path.join(temp_dir, "corrections.json")
|
||||
|
||||
# Try updated corrections first (from human review), fall back to original
|
||||
corrections_gcs_updated = f"jobs/{job_id}/lyrics/corrections_updated.json"
|
||||
corrections_gcs_original = f"jobs/{job_id}/lyrics/corrections.json"
|
||||
|
||||
# Get GCS path from file_urls
|
||||
corrections_url = job.file_urls.get('lyrics', {}).get('corrections_updated')
|
||||
if not corrections_url:
|
||||
corrections_url = job.file_urls.get('lyrics', {}).get('corrections')
|
||||
|
||||
if not corrections_url:
|
||||
# Try direct GCS paths
|
||||
if storage.file_exists(corrections_gcs_updated):
|
||||
corrections_gcs = corrections_gcs_updated
|
||||
elif storage.file_exists(corrections_gcs_original):
|
||||
corrections_gcs = corrections_gcs_original
|
||||
else:
|
||||
raise FileNotFoundError(f"No corrections file found for job {job_id}")
|
||||
else:
|
||||
# Extract GCS path from URL
|
||||
corrections_gcs = _extract_gcs_path(corrections_url)
|
||||
|
||||
# 2. Load the ORIGINAL corrections (has full structure)
|
||||
original_corrections_gcs = f"jobs/{job_id}/lyrics/corrections.json"
|
||||
original_corrections_path = os.path.join(temp_dir, "corrections_original.json")
|
||||
|
||||
job_log.info(f"Downloading original corrections from {original_corrections_gcs}")
|
||||
logger.info(f"Job {job_id}: Downloading original corrections from {original_corrections_gcs}")
|
||||
storage.download_file(original_corrections_gcs, original_corrections_path)
|
||||
|
||||
with open(original_corrections_path, 'r', encoding='utf-8') as f:
|
||||
original_data = json.load(f)
|
||||
|
||||
# 3. Check if there are updated corrections (from review UI)
|
||||
# The frontend sends only partial data: {corrections, corrected_segments}
|
||||
updated_corrections_gcs = f"jobs/{job_id}/lyrics/corrections_updated.json"
|
||||
|
||||
if storage.file_exists(updated_corrections_gcs):
|
||||
job_log.info("Found updated corrections from review, merging")
|
||||
logger.info(f"Job {job_id}: Found updated corrections, merging")
|
||||
updated_path = os.path.join(temp_dir, "corrections_updated.json")
|
||||
storage.download_file(updated_corrections_gcs, updated_path)
|
||||
|
||||
with open(updated_path, 'r', encoding='utf-8') as f:
|
||||
updated_data = json.load(f)
|
||||
|
||||
# Merge: update the original with the user's corrections
|
||||
if 'corrections' in updated_data:
|
||||
original_data['corrections'] = updated_data['corrections']
|
||||
if 'corrected_segments' in updated_data:
|
||||
original_data['corrected_segments'] = updated_data['corrected_segments']
|
||||
|
||||
job_log.info("Merged user corrections into original data")
|
||||
logger.info(f"Job {job_id}: Merged user corrections into original data")
|
||||
|
||||
# 4. Convert to CorrectionResult
|
||||
correction_result = CorrectionResult.from_dict(original_data)
|
||||
job_log.info(f"Loaded CorrectionResult with {len(correction_result.corrected_segments)} segments")
|
||||
logger.info(f"Job {job_id}: Loaded CorrectionResult with {len(correction_result.corrected_segments)} segments")
|
||||
|
||||
# 3. Download audio file
|
||||
audio_path = os.path.join(temp_dir, "audio.flac")
|
||||
audio_gcs_path = job.input_media_gcs_path
|
||||
|
||||
if not audio_gcs_path:
|
||||
raise FileNotFoundError(f"No input audio path for job {job_id}")
|
||||
|
||||
job_log.info(f"Downloading audio from {audio_gcs_path}")
|
||||
logger.info(f"Job {job_id}: Downloading audio from {audio_gcs_path}")
|
||||
storage.download_file(audio_gcs_path, audio_path)
|
||||
job_log.info(f"Audio downloaded: {os.path.getsize(audio_path)} bytes")
|
||||
|
||||
# 4. Get or create styles using the unified style loader
|
||||
job_log.info("Loading style configuration...")
|
||||
job_log.info(f" job.style_params_gcs_path: {job.style_params_gcs_path}")
|
||||
job_log.info(f" job.style_assets: {list(job.style_assets.keys()) if job.style_assets else 'None'}")
|
||||
|
||||
styles_path, style_data = load_styles_from_gcs(
|
||||
style_params_gcs_path=job.style_params_gcs_path,
|
||||
style_assets=job.style_assets,
|
||||
temp_dir=temp_dir,
|
||||
download_func=storage.download_file,
|
||||
logger=job_log,
|
||||
)
|
||||
|
||||
# 5. Configure OutputGenerator
|
||||
output_dir = os.path.join(temp_dir, "output")
|
||||
cache_dir = os.path.join(temp_dir, "cache")
|
||||
os.makedirs(output_dir, exist_ok=True)
|
||||
os.makedirs(cache_dir, exist_ok=True)
|
||||
|
||||
job_log.info(f"Using styles from: {styles_path}")
|
||||
|
||||
config = OutputConfig(
|
||||
output_dir=output_dir,
|
||||
cache_dir=cache_dir,
|
||||
output_styles_json=styles_path,
|
||||
render_video=True,
|
||||
generate_cdg=False, # CDG optional, generated separately
|
||||
generate_plain_text=True,
|
||||
generate_lrc=True,
|
||||
video_resolution="4k",
|
||||
subtitle_offset_ms=0
|
||||
)
|
||||
|
||||
job_log.info(f"OutputConfig: output_styles_json={config.output_styles_json}, render_video={config.render_video}")
|
||||
|
||||
output_generator = OutputGenerator(config, logger)
|
||||
|
||||
# 6. Generate outputs (video, LRC, ASS, etc.)
|
||||
output_prefix = f"{job.artist or 'Unknown'} - {job.title or 'Unknown'}"
|
||||
job_log.info(f"Generating outputs with prefix '{output_prefix}'")
|
||||
logger.info(f"Job {job_id}: Generating outputs with prefix '{output_prefix}'")
|
||||
|
||||
outputs = output_generator.generate_outputs(
|
||||
transcription_corrected=correction_result,
|
||||
lyrics_results={}, # Reference lyrics already in correction_result
|
||||
output_prefix=output_prefix,
|
||||
audio_filepath=audio_path,
|
||||
artist=job.artist,
|
||||
title=job.title
|
||||
)
|
||||
|
||||
# 7. Upload video to GCS
|
||||
if outputs.video and os.path.exists(outputs.video):
|
||||
video_size = os.path.getsize(outputs.video)
|
||||
job_log.info(f"Video generated: {video_size} bytes")
|
||||
video_gcs_path = f"jobs/{job_id}/videos/with_vocals.mkv"
|
||||
video_url = storage.upload_file(outputs.video, video_gcs_path)
|
||||
job_manager.update_file_url(job_id, 'videos', 'with_vocals', video_url)
|
||||
job_log.info(f"Uploaded with_vocals.mkv to GCS")
|
||||
logger.info(f"Job {job_id}: Uploaded with_vocals.mkv ({video_size} bytes)")
|
||||
else:
|
||||
job_log.error("Video generation failed - no output file produced!")
|
||||
raise Exception("Video generation failed - no output file produced")
|
||||
|
||||
# 8. Upload LRC file
|
||||
if outputs.lrc and os.path.exists(outputs.lrc):
|
||||
lrc_gcs_path = f"jobs/{job_id}/lyrics/karaoke.lrc"
|
||||
lrc_url = storage.upload_file(outputs.lrc, lrc_gcs_path)
|
||||
job_manager.update_file_url(job_id, 'lyrics', 'lrc', lrc_url)
|
||||
job_log.info("Uploaded karaoke.lrc")
|
||||
logger.info(f"Job {job_id}: Uploaded karaoke.lrc")
|
||||
|
||||
# 9. Upload ASS subtitle file
|
||||
if outputs.ass and os.path.exists(outputs.ass):
|
||||
ass_gcs_path = f"jobs/{job_id}/lyrics/karaoke.ass"
|
||||
ass_url = storage.upload_file(outputs.ass, ass_gcs_path)
|
||||
job_manager.update_file_url(job_id, 'lyrics', 'ass', ass_url)
|
||||
job_log.info("Uploaded karaoke.ass")
|
||||
logger.info(f"Job {job_id}: Uploaded karaoke.ass")
|
||||
|
||||
# 10. Upload corrected text files
|
||||
if outputs.corrected_txt and os.path.exists(outputs.corrected_txt):
|
||||
txt_gcs_path = f"jobs/{job_id}/lyrics/corrected.txt"
|
||||
txt_url = storage.upload_file(outputs.corrected_txt, txt_gcs_path)
|
||||
job_manager.update_file_url(job_id, 'lyrics', 'corrected_txt', txt_url)
|
||||
job_log.info("Uploaded corrected.txt")
|
||||
logger.info(f"Job {job_id}: Uploaded corrected.txt")
|
||||
|
||||
# 11. Transition to awaiting instrumental selection
|
||||
# Use job_logging_context for proper log isolation when multiple jobs run concurrently
|
||||
# This ensures logs from third-party libraries (lyrics_transcriber.output) are only
|
||||
# captured by this job's handler, not handlers from other concurrent jobs
|
||||
with job_logging_context(job_id):
|
||||
# Transition to RENDERING_VIDEO
|
||||
job_manager.transition_to_state(
|
||||
job_id=job_id,
|
||||
new_status=JobStatus.AWAITING_INSTRUMENTAL_SELECTION,
|
||||
progress=80,
|
||||
message="Video rendered - select your instrumental"
|
||||
new_status=JobStatus.RENDERING_VIDEO,
|
||||
progress=75,
|
||||
message="Rendering karaoke video with corrected lyrics"
|
||||
)
|
||||
|
||||
job_log.info("=== RENDER VIDEO WORKER COMPLETE ===")
|
||||
logger.info(f"Job {job_id}: Video render complete, awaiting instrumental selection")
|
||||
return True
|
||||
with tempfile.TemporaryDirectory() as temp_dir:
|
||||
job_log.info(f"Created temp directory: {temp_dir}")
|
||||
# 1. Download corrected corrections data
|
||||
corrections_path = os.path.join(temp_dir, "corrections.json")
|
||||
|
||||
# Try updated corrections first (from human review), fall back to original
|
||||
corrections_gcs_updated = f"jobs/{job_id}/lyrics/corrections_updated.json"
|
||||
corrections_gcs_original = f"jobs/{job_id}/lyrics/corrections.json"
|
||||
|
||||
# Get GCS path from file_urls
|
||||
corrections_url = job.file_urls.get('lyrics', {}).get('corrections_updated')
|
||||
if not corrections_url:
|
||||
corrections_url = job.file_urls.get('lyrics', {}).get('corrections')
|
||||
|
||||
if not corrections_url:
|
||||
# Try direct GCS paths
|
||||
if storage.file_exists(corrections_gcs_updated):
|
||||
corrections_gcs = corrections_gcs_updated
|
||||
elif storage.file_exists(corrections_gcs_original):
|
||||
corrections_gcs = corrections_gcs_original
|
||||
else:
|
||||
raise FileNotFoundError(f"No corrections file found for job {job_id}")
|
||||
else:
|
||||
# Extract GCS path from URL
|
||||
corrections_gcs = _extract_gcs_path(corrections_url)
|
||||
|
||||
# 2. Load the ORIGINAL corrections (has full structure)
|
||||
original_corrections_gcs = f"jobs/{job_id}/lyrics/corrections.json"
|
||||
original_corrections_path = os.path.join(temp_dir, "corrections_original.json")
|
||||
|
||||
job_log.info(f"Downloading original corrections from {original_corrections_gcs}")
|
||||
logger.info(f"Job {job_id}: Downloading original corrections from {original_corrections_gcs}")
|
||||
storage.download_file(original_corrections_gcs, original_corrections_path)
|
||||
|
||||
with open(original_corrections_path, 'r', encoding='utf-8') as f:
|
||||
original_data = json.load(f)
|
||||
|
||||
# 3. Check if there are updated corrections (from review UI)
|
||||
# The frontend sends only partial data: {corrections, corrected_segments}
|
||||
updated_corrections_gcs = f"jobs/{job_id}/lyrics/corrections_updated.json"
|
||||
|
||||
if storage.file_exists(updated_corrections_gcs):
|
||||
job_log.info("Found updated corrections from review, merging")
|
||||
logger.info(f"Job {job_id}: Found updated corrections, merging")
|
||||
updated_path = os.path.join(temp_dir, "corrections_updated.json")
|
||||
storage.download_file(updated_corrections_gcs, updated_path)
|
||||
|
||||
with open(updated_path, 'r', encoding='utf-8') as f:
|
||||
updated_data = json.load(f)
|
||||
|
||||
# Merge: update the original with the user's corrections
|
||||
if 'corrections' in updated_data:
|
||||
original_data['corrections'] = updated_data['corrections']
|
||||
if 'corrected_segments' in updated_data:
|
||||
original_data['corrected_segments'] = updated_data['corrected_segments']
|
||||
|
||||
job_log.info("Merged user corrections into original data")
|
||||
logger.info(f"Job {job_id}: Merged user corrections into original data")
|
||||
|
||||
# 4. Convert to CorrectionResult
|
||||
correction_result = CorrectionResult.from_dict(original_data)
|
||||
job_log.info(f"Loaded CorrectionResult with {len(correction_result.corrected_segments)} segments")
|
||||
logger.info(f"Job {job_id}: Loaded CorrectionResult with {len(correction_result.corrected_segments)} segments")
|
||||
|
||||
# 3. Download audio file
|
||||
audio_path = os.path.join(temp_dir, "audio.flac")
|
||||
audio_gcs_path = job.input_media_gcs_path
|
||||
|
||||
if not audio_gcs_path:
|
||||
raise FileNotFoundError(f"No input audio path for job {job_id}")
|
||||
|
||||
job_log.info(f"Downloading audio from {audio_gcs_path}")
|
||||
logger.info(f"Job {job_id}: Downloading audio from {audio_gcs_path}")
|
||||
storage.download_file(audio_gcs_path, audio_path)
|
||||
job_log.info(f"Audio downloaded: {os.path.getsize(audio_path)} bytes")
|
||||
|
||||
# 4. Get or create styles using the unified style loader
|
||||
job_log.info("Loading style configuration...")
|
||||
job_log.info(f" job.style_params_gcs_path: {job.style_params_gcs_path}")
|
||||
job_log.info(f" job.style_assets: {list(job.style_assets.keys()) if job.style_assets else 'None'}")
|
||||
|
||||
styles_path, style_data = load_styles_from_gcs(
|
||||
style_params_gcs_path=job.style_params_gcs_path,
|
||||
style_assets=job.style_assets,
|
||||
temp_dir=temp_dir,
|
||||
download_func=storage.download_file,
|
||||
logger=job_log,
|
||||
)
|
||||
|
||||
# 5. Configure OutputGenerator
|
||||
output_dir = os.path.join(temp_dir, "output")
|
||||
cache_dir = os.path.join(temp_dir, "cache")
|
||||
os.makedirs(output_dir, exist_ok=True)
|
||||
os.makedirs(cache_dir, exist_ok=True)
|
||||
|
||||
job_log.info(f"Using styles from: {styles_path}")
|
||||
|
||||
# Get subtitle offset from job (user-specified timing adjustment)
|
||||
subtitle_offset = getattr(job, 'subtitle_offset_ms', 0) or 0
|
||||
if subtitle_offset != 0:
|
||||
job_log.info(f"Applying subtitle offset: {subtitle_offset}ms")
|
||||
|
||||
config = OutputConfig(
|
||||
output_dir=output_dir,
|
||||
cache_dir=cache_dir,
|
||||
output_styles_json=styles_path,
|
||||
render_video=True,
|
||||
generate_cdg=False, # CDG optional, generated separately
|
||||
generate_plain_text=True,
|
||||
generate_lrc=True,
|
||||
video_resolution="4k",
|
||||
subtitle_offset_ms=subtitle_offset
|
||||
)
|
||||
|
||||
job_log.info(f"OutputConfig: output_styles_json={config.output_styles_json}, render_video={config.render_video}")
|
||||
|
||||
output_generator = OutputGenerator(config, logger)
|
||||
|
||||
# 6. Generate outputs (video, LRC, ASS, etc.)
|
||||
output_prefix = f"{job.artist or 'Unknown'} - {job.title or 'Unknown'}"
|
||||
job_log.info(f"Generating outputs with prefix '{output_prefix}'")
|
||||
logger.info(f"Job {job_id}: Generating outputs with prefix '{output_prefix}'")
|
||||
|
||||
outputs = output_generator.generate_outputs(
|
||||
transcription_corrected=correction_result,
|
||||
lyrics_results={}, # Reference lyrics already in correction_result
|
||||
output_prefix=output_prefix,
|
||||
audio_filepath=audio_path,
|
||||
artist=job.artist,
|
||||
title=job.title
|
||||
)
|
||||
|
||||
# 7. Upload video to GCS
|
||||
if outputs.video and os.path.exists(outputs.video):
|
||||
video_size = os.path.getsize(outputs.video)
|
||||
job_log.info(f"Video generated: {video_size} bytes")
|
||||
video_gcs_path = f"jobs/{job_id}/videos/with_vocals.mkv"
|
||||
video_url = storage.upload_file(outputs.video, video_gcs_path)
|
||||
job_manager.update_file_url(job_id, 'videos', 'with_vocals', video_url)
|
||||
job_log.info(f"Uploaded with_vocals.mkv to GCS")
|
||||
logger.info(f"Job {job_id}: Uploaded with_vocals.mkv ({video_size} bytes)")
|
||||
else:
|
||||
job_log.error("Video generation failed - no output file produced!")
|
||||
raise Exception("Video generation failed - no output file produced")
|
||||
|
||||
# 8. Upload LRC file
|
||||
if outputs.lrc and os.path.exists(outputs.lrc):
|
||||
lrc_gcs_path = f"jobs/{job_id}/lyrics/karaoke.lrc"
|
||||
lrc_url = storage.upload_file(outputs.lrc, lrc_gcs_path)
|
||||
job_manager.update_file_url(job_id, 'lyrics', 'lrc', lrc_url)
|
||||
job_log.info("Uploaded karaoke.lrc")
|
||||
logger.info(f"Job {job_id}: Uploaded karaoke.lrc")
|
||||
|
||||
# 9. Upload ASS subtitle file
|
||||
if outputs.ass and os.path.exists(outputs.ass):
|
||||
ass_gcs_path = f"jobs/{job_id}/lyrics/karaoke.ass"
|
||||
ass_url = storage.upload_file(outputs.ass, ass_gcs_path)
|
||||
job_manager.update_file_url(job_id, 'lyrics', 'ass', ass_url)
|
||||
job_log.info("Uploaded karaoke.ass")
|
||||
logger.info(f"Job {job_id}: Uploaded karaoke.ass")
|
||||
|
||||
# 10. Upload corrected text files
|
||||
if outputs.corrected_txt and os.path.exists(outputs.corrected_txt):
|
||||
txt_gcs_path = f"jobs/{job_id}/lyrics/corrected.txt"
|
||||
txt_url = storage.upload_file(outputs.corrected_txt, txt_gcs_path)
|
||||
job_manager.update_file_url(job_id, 'lyrics', 'corrected_txt', txt_url)
|
||||
job_log.info("Uploaded corrected.txt")
|
||||
logger.info(f"Job {job_id}: Uploaded corrected.txt")
|
||||
|
||||
# 11. Transition to awaiting instrumental selection
|
||||
job_manager.transition_to_state(
|
||||
job_id=job_id,
|
||||
new_status=JobStatus.AWAITING_INSTRUMENTAL_SELECTION,
|
||||
progress=80,
|
||||
message="Video rendered - select your instrumental"
|
||||
)
|
||||
|
||||
job_log.info("=== RENDER VIDEO WORKER COMPLETE ===")
|
||||
logger.info(f"Job {job_id}: Video render complete, awaiting instrumental selection")
|
||||
return True
|
||||
|
||||
except Exception as e:
|
||||
job_log.error(f"Video render failed: {e}")
|
||||
|
||||
@@ -35,7 +35,7 @@ from backend.services.job_manager import JobManager
|
||||
from backend.services.storage_service import StorageService
|
||||
from backend.config import get_settings
|
||||
from backend.workers.style_helper import load_style_config, StyleConfig
|
||||
from backend.workers.worker_logging import create_job_logger, setup_job_logging
|
||||
from backend.workers.worker_logging import create_job_logger, setup_job_logging, job_logging_context
|
||||
|
||||
# Import from karaoke_gen package
|
||||
from karaoke_gen.video_generator import VideoGenerator
|
||||
@@ -88,94 +88,98 @@ async def generate_screens(job_id: str) -> bool:
|
||||
temp_dir = tempfile.mkdtemp(prefix=f"karaoke_screens_{job_id}_")
|
||||
|
||||
try:
|
||||
job_log.info(f"Starting screen generation for {job.artist} - {job.title}")
|
||||
logger.info(f"Starting screen generation for job {job_id}")
|
||||
|
||||
# Transition to GENERATING_SCREENS state
|
||||
job_manager.transition_to_state(
|
||||
job_id=job_id,
|
||||
new_status=JobStatus.GENERATING_SCREENS,
|
||||
progress=50,
|
||||
message="Generating title and end screens"
|
||||
)
|
||||
|
||||
# Log style assets info
|
||||
style_assets = getattr(job, 'style_assets', {}) or {}
|
||||
job_log.info(f"Style assets from job: {list(style_assets.keys()) if style_assets else 'None'}")
|
||||
if style_assets:
|
||||
for key, path in style_assets.items():
|
||||
job_log.info(f" {key}: {path}")
|
||||
|
||||
# Load style configuration (downloads assets from GCS if available)
|
||||
job_log.info("Loading style configuration from GCS...")
|
||||
style_config = await load_style_config(job, storage, temp_dir)
|
||||
if style_config.has_custom_styles():
|
||||
job_log.info("Using CUSTOM style configuration")
|
||||
logger.info(f"Job {job_id}: Using custom style configuration")
|
||||
else:
|
||||
job_log.warning("Using DEFAULT style configuration (no custom styles found)")
|
||||
logger.info(f"Job {job_id}: Using default style configuration")
|
||||
|
||||
# Initialize video generator
|
||||
video_generator = _create_video_generator(temp_dir)
|
||||
|
||||
# Generate title screen with style config
|
||||
job_log.info("Generating title screen...")
|
||||
title_screen_path = await _generate_title_screen(
|
||||
job_id=job_id,
|
||||
job=job,
|
||||
video_generator=video_generator,
|
||||
style_config=style_config,
|
||||
temp_dir=temp_dir,
|
||||
job_log=job_log
|
||||
)
|
||||
|
||||
if not title_screen_path:
|
||||
raise Exception("Title screen generation failed")
|
||||
job_log.info(f"Title screen generated: {title_screen_path}")
|
||||
|
||||
# Generate end screen with style config
|
||||
job_log.info("Generating end screen...")
|
||||
end_screen_path = await _generate_end_screen(
|
||||
job_id=job_id,
|
||||
job=job,
|
||||
video_generator=video_generator,
|
||||
style_config=style_config,
|
||||
temp_dir=temp_dir,
|
||||
job_log=job_log
|
||||
)
|
||||
|
||||
if not end_screen_path:
|
||||
raise Exception("End screen generation failed")
|
||||
job_log.info(f"End screen generated: {end_screen_path}")
|
||||
|
||||
# Upload screens to GCS
|
||||
await _upload_screens(
|
||||
job_id=job_id,
|
||||
job_manager=job_manager,
|
||||
storage=storage,
|
||||
title_screen_path=title_screen_path,
|
||||
end_screen_path=end_screen_path
|
||||
)
|
||||
|
||||
# Apply countdown padding if needed
|
||||
# (This is handled automatically by checking lyrics metadata)
|
||||
await _apply_countdown_padding_if_needed(job_id, job_manager, job)
|
||||
|
||||
# Transition to AWAITING_REVIEW
|
||||
# Human must review lyrics before video can be rendered
|
||||
logger.info(f"Job {job_id}: Screens generated, awaiting lyrics review")
|
||||
job_manager.transition_to_state(
|
||||
job_id=job_id,
|
||||
new_status=JobStatus.AWAITING_REVIEW,
|
||||
progress=55,
|
||||
message="Ready for lyrics review. Please review and correct lyrics."
|
||||
)
|
||||
|
||||
# TODO: Send notification to user that selection is needed
|
||||
# await notify_user(job_id, "instrumental_selection_ready")
|
||||
|
||||
return True
|
||||
# Use job_logging_context for proper log isolation when multiple jobs run concurrently
|
||||
# This ensures logs from third-party libraries (karaoke_gen.video_generator) are only
|
||||
# captured by this job's handler, not handlers from other concurrent jobs
|
||||
with job_logging_context(job_id):
|
||||
job_log.info(f"Starting screen generation for {job.artist} - {job.title}")
|
||||
logger.info(f"Starting screen generation for job {job_id}")
|
||||
|
||||
# Transition to GENERATING_SCREENS state
|
||||
job_manager.transition_to_state(
|
||||
job_id=job_id,
|
||||
new_status=JobStatus.GENERATING_SCREENS,
|
||||
progress=50,
|
||||
message="Generating title and end screens"
|
||||
)
|
||||
|
||||
# Log style assets info
|
||||
style_assets = getattr(job, 'style_assets', {}) or {}
|
||||
job_log.info(f"Style assets from job: {list(style_assets.keys()) if style_assets else 'None'}")
|
||||
if style_assets:
|
||||
for key, path in style_assets.items():
|
||||
job_log.info(f" {key}: {path}")
|
||||
|
||||
# Load style configuration (downloads assets from GCS if available)
|
||||
job_log.info("Loading style configuration from GCS...")
|
||||
style_config = await load_style_config(job, storage, temp_dir)
|
||||
if style_config.has_custom_styles():
|
||||
job_log.info("Using CUSTOM style configuration")
|
||||
logger.info(f"Job {job_id}: Using custom style configuration")
|
||||
else:
|
||||
job_log.warning("Using DEFAULT style configuration (no custom styles found)")
|
||||
logger.info(f"Job {job_id}: Using default style configuration")
|
||||
|
||||
# Initialize video generator
|
||||
video_generator = _create_video_generator(temp_dir)
|
||||
|
||||
# Generate title screen with style config
|
||||
job_log.info("Generating title screen...")
|
||||
title_screen_path = await _generate_title_screen(
|
||||
job_id=job_id,
|
||||
job=job,
|
||||
video_generator=video_generator,
|
||||
style_config=style_config,
|
||||
temp_dir=temp_dir,
|
||||
job_log=job_log
|
||||
)
|
||||
|
||||
if not title_screen_path:
|
||||
raise Exception("Title screen generation failed")
|
||||
job_log.info(f"Title screen generated: {title_screen_path}")
|
||||
|
||||
# Generate end screen with style config
|
||||
job_log.info("Generating end screen...")
|
||||
end_screen_path = await _generate_end_screen(
|
||||
job_id=job_id,
|
||||
job=job,
|
||||
video_generator=video_generator,
|
||||
style_config=style_config,
|
||||
temp_dir=temp_dir,
|
||||
job_log=job_log
|
||||
)
|
||||
|
||||
if not end_screen_path:
|
||||
raise Exception("End screen generation failed")
|
||||
job_log.info(f"End screen generated: {end_screen_path}")
|
||||
|
||||
# Upload screens to GCS
|
||||
await _upload_screens(
|
||||
job_id=job_id,
|
||||
job_manager=job_manager,
|
||||
storage=storage,
|
||||
title_screen_path=title_screen_path,
|
||||
end_screen_path=end_screen_path
|
||||
)
|
||||
|
||||
# Apply countdown padding if needed
|
||||
# (This is handled automatically by checking lyrics metadata)
|
||||
await _apply_countdown_padding_if_needed(job_id, job_manager, job)
|
||||
|
||||
# Transition to AWAITING_REVIEW
|
||||
# Human must review lyrics before video can be rendered
|
||||
logger.info(f"Job {job_id}: Screens generated, awaiting lyrics review")
|
||||
job_manager.transition_to_state(
|
||||
job_id=job_id,
|
||||
new_status=JobStatus.AWAITING_REVIEW,
|
||||
progress=55,
|
||||
message="Ready for lyrics review. Please review and correct lyrics."
|
||||
)
|
||||
|
||||
# TODO: Send notification to user that selection is needed
|
||||
# await notify_user(job_id, "instrumental_selection_ready")
|
||||
|
||||
return True
|
||||
|
||||
except Exception as e:
|
||||
logger.error(f"Job {job_id}: Screen generation failed: {e}", exc_info=True)
|
||||
|
||||
+241
-152
@@ -31,7 +31,7 @@ from backend.services.rclone_service import get_rclone_service
|
||||
from backend.services.youtube_service import get_youtube_service
|
||||
from backend.config import get_settings
|
||||
from backend.workers.style_helper import load_style_config
|
||||
from backend.workers.worker_logging import create_job_logger, setup_job_logging
|
||||
from backend.workers.worker_logging import create_job_logger, setup_job_logging, job_logging_context
|
||||
|
||||
# Import from karaoke_gen package - reuse existing implementation
|
||||
from karaoke_gen.karaoke_finalise.karaoke_finalise import KaraokeFinalise
|
||||
@@ -105,149 +105,166 @@ async def generate_video(job_id: str) -> bool:
|
||||
job_log.warning("YouTube credentials not available - upload will be skipped")
|
||||
|
||||
try:
|
||||
job_log.info(f"Starting video finalization for {job.artist} - {job.title}")
|
||||
logger.info(f"Starting video generation for job {job_id}")
|
||||
|
||||
# Transition to GENERATING_VIDEO state
|
||||
job_manager.transition_to_state(
|
||||
job_id=job_id,
|
||||
new_status=JobStatus.GENERATING_VIDEO,
|
||||
progress=70,
|
||||
message="Preparing files for video generation"
|
||||
)
|
||||
|
||||
# Download and set up files in the format KaraokeFinalise expects
|
||||
base_name = f"{job.artist} - {job.title}"
|
||||
job_log.info("Downloading files from GCS (this may take a few minutes for large files)...")
|
||||
await _setup_working_directory(job_id, job, storage, temp_dir, base_name, job_log)
|
||||
job_log.info("All files downloaded successfully")
|
||||
|
||||
# Load style config for CDG styles
|
||||
job_log.info("Loading CDG style configuration...")
|
||||
style_config = await load_style_config(job, storage, temp_dir)
|
||||
cdg_styles = style_config.get_cdg_styles()
|
||||
if cdg_styles:
|
||||
job_log.info("CDG styles loaded from custom configuration")
|
||||
else:
|
||||
job_log.info("Using default CDG styles")
|
||||
|
||||
# Change to working directory (KaraokeFinalise works in cwd)
|
||||
os.chdir(temp_dir)
|
||||
|
||||
# Get the selected instrumental file path
|
||||
instrumental_selection = job.state_data['instrumental_selection']
|
||||
instrumental_suffix = "Clean" if instrumental_selection == 'clean' else "Backing"
|
||||
instrumental_file = os.path.join(temp_dir, f"{base_name} (Instrumental {instrumental_suffix}).flac")
|
||||
|
||||
# Transition to encoding state
|
||||
job_manager.transition_to_state(
|
||||
job_id=job_id,
|
||||
new_status=JobStatus.ENCODING,
|
||||
progress=75,
|
||||
message="Encoding videos (15-20 min)"
|
||||
)
|
||||
|
||||
# Log finalization parameters
|
||||
job_log.info("Creating KaraokeFinalise with parameters:")
|
||||
job_log.info(f" enable_cdg: {getattr(job, 'enable_cdg', False)}")
|
||||
job_log.info(f" enable_txt: {getattr(job, 'enable_txt', False)}")
|
||||
job_log.info(f" brand_prefix: {getattr(job, 'brand_prefix', None)}")
|
||||
job_log.info(f" discord_webhook: {'configured' if getattr(job, 'discord_webhook_url', None) else 'not configured'}")
|
||||
job_log.info(f" instrumental: {instrumental_selection}")
|
||||
|
||||
# Create KaraokeFinalise with ALL the parameters from the job
|
||||
# This reuses all existing functionality!
|
||||
|
||||
# Set up YouTube description file if template is provided
|
||||
youtube_desc_path = None
|
||||
if youtube_credentials and getattr(job, 'youtube_description_template', None):
|
||||
youtube_desc_path = os.path.join(temp_dir, "youtube_description.txt")
|
||||
with open(youtube_desc_path, 'w') as f:
|
||||
f.write(job.youtube_description_template)
|
||||
job_log.info("YouTube description template written to temp file")
|
||||
|
||||
finalise = KaraokeFinalise(
|
||||
logger=logger,
|
||||
log_level=logging.INFO,
|
||||
dry_run=False,
|
||||
instrumental_format="flac",
|
||||
# CDG/TXT generation
|
||||
enable_cdg=getattr(job, 'enable_cdg', False),
|
||||
enable_txt=getattr(job, 'enable_txt', False),
|
||||
cdg_styles=cdg_styles,
|
||||
# Brand code and organization (server-side mode uses rclone)
|
||||
brand_prefix=getattr(job, 'brand_prefix', None),
|
||||
organised_dir=None, # Not used in cloud - files stay in GCS
|
||||
organised_dir_rclone_root=getattr(job, 'organised_dir_rclone_root', None),
|
||||
public_share_dir=None, # Not used in cloud
|
||||
# Notifications
|
||||
discord_webhook_url=getattr(job, 'discord_webhook_url', None),
|
||||
# YouTube upload (server-side with pre-loaded credentials)
|
||||
youtube_client_secrets_file=None, # Not used with pre-stored credentials
|
||||
youtube_description_file=youtube_desc_path,
|
||||
user_youtube_credentials=youtube_credentials, # Pre-loaded from Secret Manager
|
||||
rclone_destination=None,
|
||||
email_template_file=None,
|
||||
# Server-side optimizations
|
||||
non_interactive=True,
|
||||
server_side_mode=True,
|
||||
selected_instrumental_file=instrumental_file,
|
||||
)
|
||||
|
||||
# Call process() - this does ALL the work:
|
||||
# - Encodes to 4 video formats
|
||||
# - Generates CDG/TXT packages if enabled
|
||||
# - Posts Discord notification if configured
|
||||
# - Handles brand code generation
|
||||
job_log.info("Starting KaraokeFinalise.process() - this may take 15-20 minutes...")
|
||||
logger.info(f"Job {job_id}: Starting KaraokeFinalise.process()")
|
||||
result = finalise.process()
|
||||
|
||||
job_log.info("KaraokeFinalise.process() complete!")
|
||||
if result.get('brand_code'):
|
||||
job_log.info(f"Brand code: {result.get('brand_code')}")
|
||||
logger.info(f"Job {job_id}: KaraokeFinalise.process() complete")
|
||||
logger.info(f"Job {job_id}: Brand code: {result.get('brand_code')}")
|
||||
|
||||
# Native API distribution uploads (used by remote CLI instead of rclone)
|
||||
await _handle_native_distribution(
|
||||
job_id=job_id,
|
||||
job=job,
|
||||
job_log=job_log,
|
||||
job_manager=job_manager,
|
||||
temp_dir=temp_dir,
|
||||
result=result,
|
||||
)
|
||||
|
||||
# Upload generated files to GCS
|
||||
job_manager.transition_to_state(
|
||||
job_id=job_id,
|
||||
new_status=JobStatus.PACKAGING,
|
||||
progress=95,
|
||||
message="Uploading final files"
|
||||
)
|
||||
|
||||
await _upload_results(job_id, job_manager, storage, temp_dir, result)
|
||||
|
||||
# Mark job as complete
|
||||
logger.info(f"Job {job_id}: Video generation complete")
|
||||
job_manager.transition_to_state(
|
||||
job_id=job_id,
|
||||
new_status=JobStatus.COMPLETE,
|
||||
progress=100,
|
||||
message="Karaoke generation complete!"
|
||||
)
|
||||
|
||||
# Store result metadata in job
|
||||
job_manager.update_job(job_id, {
|
||||
'state_data': {
|
||||
**job.state_data,
|
||||
'brand_code': result.get('brand_code'),
|
||||
'youtube_url': result.get('youtube_url'),
|
||||
}
|
||||
})
|
||||
|
||||
return True
|
||||
# Use job_logging_context for proper log isolation when multiple jobs run concurrently
|
||||
# This ensures logs from third-party libraries (karaoke_gen.karaoke_finalise) are only
|
||||
# captured by this job's handler, not handlers from other concurrent jobs
|
||||
with job_logging_context(job_id):
|
||||
job_log.info(f"Starting video finalization for {job.artist} - {job.title}")
|
||||
logger.info(f"Starting video generation for job {job_id}")
|
||||
|
||||
# Transition to GENERATING_VIDEO state
|
||||
job_manager.transition_to_state(
|
||||
job_id=job_id,
|
||||
new_status=JobStatus.GENERATING_VIDEO,
|
||||
progress=70,
|
||||
message="Preparing files for video generation"
|
||||
)
|
||||
|
||||
# Download and set up files in the format KaraokeFinalise expects
|
||||
base_name = f"{job.artist} - {job.title}"
|
||||
job_log.info("Downloading files from GCS (this may take a few minutes for large files)...")
|
||||
await _setup_working_directory(job_id, job, storage, temp_dir, base_name, job_log)
|
||||
job_log.info("All files downloaded successfully")
|
||||
|
||||
# Load style config for CDG styles
|
||||
job_log.info("Loading CDG style configuration...")
|
||||
style_config = await load_style_config(job, storage, temp_dir)
|
||||
cdg_styles = style_config.get_cdg_styles()
|
||||
if cdg_styles:
|
||||
job_log.info("CDG styles loaded from custom configuration")
|
||||
else:
|
||||
job_log.info("Using default CDG styles")
|
||||
|
||||
# Change to working directory (KaraokeFinalise works in cwd)
|
||||
os.chdir(temp_dir)
|
||||
|
||||
# Get the selected instrumental file path
|
||||
# Batch 3: If user provided existing instrumental, use that; otherwise use AI-separated
|
||||
instrumental_selection = job.state_data['instrumental_selection']
|
||||
existing_instrumental_path = getattr(job, 'existing_instrumental_gcs_path', None)
|
||||
|
||||
if existing_instrumental_path:
|
||||
# User provided existing instrumental
|
||||
ext = Path(existing_instrumental_path).suffix.lower()
|
||||
instrumental_file = os.path.join(temp_dir, f"{base_name} (Instrumental User){ext}")
|
||||
instrumental_source = "user-provided"
|
||||
else:
|
||||
# Use AI-separated instrumental
|
||||
instrumental_suffix = "Clean" if instrumental_selection == 'clean' else "Backing"
|
||||
instrumental_file = os.path.join(temp_dir, f"{base_name} (Instrumental {instrumental_suffix}).flac")
|
||||
instrumental_source = instrumental_selection
|
||||
|
||||
# Transition to encoding state
|
||||
job_manager.transition_to_state(
|
||||
job_id=job_id,
|
||||
new_status=JobStatus.ENCODING,
|
||||
progress=75,
|
||||
message="Encoding videos (15-20 min)"
|
||||
)
|
||||
|
||||
# Log finalization parameters
|
||||
job_log.info("Creating KaraokeFinalise with parameters:")
|
||||
job_log.info(f" enable_cdg: {getattr(job, 'enable_cdg', False)}")
|
||||
job_log.info(f" enable_txt: {getattr(job, 'enable_txt', False)}")
|
||||
job_log.info(f" brand_prefix: {getattr(job, 'brand_prefix', None)}")
|
||||
job_log.info(f" discord_webhook: {'configured' if getattr(job, 'discord_webhook_url', None) else 'not configured'}")
|
||||
job_log.info(f" instrumental source: {instrumental_source}")
|
||||
if existing_instrumental_path:
|
||||
job_log.info(f" using user-provided instrumental (selection was: {instrumental_selection})")
|
||||
|
||||
# Create KaraokeFinalise with ALL the parameters from the job
|
||||
# This reuses all existing functionality!
|
||||
|
||||
# Set up YouTube description file if template is provided
|
||||
youtube_desc_path = None
|
||||
if youtube_credentials and getattr(job, 'youtube_description_template', None):
|
||||
youtube_desc_path = os.path.join(temp_dir, "youtube_description.txt")
|
||||
with open(youtube_desc_path, 'w') as f:
|
||||
f.write(job.youtube_description_template)
|
||||
job_log.info("YouTube description template written to temp file")
|
||||
|
||||
finalise = KaraokeFinalise(
|
||||
logger=logger,
|
||||
log_level=logging.INFO,
|
||||
dry_run=False,
|
||||
instrumental_format="flac",
|
||||
# CDG/TXT generation
|
||||
enable_cdg=getattr(job, 'enable_cdg', False),
|
||||
enable_txt=getattr(job, 'enable_txt', False),
|
||||
cdg_styles=cdg_styles,
|
||||
# Brand code and organization (server-side mode uses rclone)
|
||||
brand_prefix=getattr(job, 'brand_prefix', None),
|
||||
organised_dir=None, # Not used in cloud - files stay in GCS
|
||||
organised_dir_rclone_root=getattr(job, 'organised_dir_rclone_root', None),
|
||||
public_share_dir=None, # Not used in cloud
|
||||
# Notifications
|
||||
discord_webhook_url=getattr(job, 'discord_webhook_url', None),
|
||||
# YouTube upload (server-side with pre-loaded credentials)
|
||||
youtube_client_secrets_file=None, # Not used with pre-stored credentials
|
||||
youtube_description_file=youtube_desc_path,
|
||||
user_youtube_credentials=youtube_credentials, # Pre-loaded from Secret Manager
|
||||
rclone_destination=None,
|
||||
email_template_file=None,
|
||||
# Server-side optimizations
|
||||
non_interactive=True,
|
||||
server_side_mode=True,
|
||||
selected_instrumental_file=instrumental_file,
|
||||
)
|
||||
|
||||
# Call process() - this does ALL the work:
|
||||
# - Encodes to 4 video formats
|
||||
# - Generates CDG/TXT packages if enabled
|
||||
# - Posts Discord notification if configured
|
||||
# - Handles brand code generation
|
||||
job_log.info("Starting KaraokeFinalise.process() - this may take 15-20 minutes...")
|
||||
logger.info(f"Job {job_id}: Starting KaraokeFinalise.process()")
|
||||
result = finalise.process()
|
||||
|
||||
job_log.info("KaraokeFinalise.process() complete!")
|
||||
if result.get('brand_code'):
|
||||
job_log.info(f"Brand code: {result.get('brand_code')}")
|
||||
logger.info(f"Job {job_id}: KaraokeFinalise.process() complete")
|
||||
logger.info(f"Job {job_id}: Brand code: {result.get('brand_code')}")
|
||||
|
||||
# Native API distribution uploads (used by remote CLI instead of rclone)
|
||||
await _handle_native_distribution(
|
||||
job_id=job_id,
|
||||
job=job,
|
||||
job_log=job_log,
|
||||
job_manager=job_manager,
|
||||
temp_dir=temp_dir,
|
||||
result=result,
|
||||
)
|
||||
|
||||
# Upload generated files to GCS
|
||||
job_manager.transition_to_state(
|
||||
job_id=job_id,
|
||||
new_status=JobStatus.PACKAGING,
|
||||
progress=95,
|
||||
message="Uploading final files"
|
||||
)
|
||||
|
||||
await _upload_results(job_id, job_manager, storage, temp_dir, result)
|
||||
|
||||
# Mark job as complete
|
||||
logger.info(f"Job {job_id}: Video generation complete")
|
||||
job_manager.transition_to_state(
|
||||
job_id=job_id,
|
||||
new_status=JobStatus.COMPLETE,
|
||||
progress=100,
|
||||
message="Karaoke generation complete!"
|
||||
)
|
||||
|
||||
# Store result metadata in job
|
||||
job_manager.update_job(job_id, {
|
||||
'state_data': {
|
||||
**job.state_data,
|
||||
'brand_code': result.get('brand_code'),
|
||||
'youtube_url': result.get('youtube_url'),
|
||||
}
|
||||
})
|
||||
|
||||
return True
|
||||
|
||||
except Exception as e:
|
||||
logger.error(f"Job {job_id}: Video generation failed: {e}", exc_info=True)
|
||||
@@ -487,15 +504,26 @@ async def _setup_working_directory(
|
||||
storage.download_file(lyrics_video_url, lyrics_video_path)
|
||||
log_progress("Downloaded karaoke video")
|
||||
|
||||
# Download selected instrumental
|
||||
# Download instrumental - either user-provided existing instrumental or AI-separated
|
||||
existing_instrumental_path = getattr(job, 'existing_instrumental_gcs_path', None)
|
||||
instrumental_selection = job.state_data['instrumental_selection']
|
||||
instrumental_key = 'instrumental_clean' if instrumental_selection == 'clean' else 'instrumental_with_backing'
|
||||
instrumental_url = job.file_urls['stems'][instrumental_key]
|
||||
instrumental_suffix = "Clean" if instrumental_selection == 'clean' else "Backing"
|
||||
instrumental_path = os.path.join(temp_dir, f"{base_name} (Instrumental {instrumental_suffix}).flac")
|
||||
log_progress(f"Downloading {instrumental_selection} instrumental audio...")
|
||||
storage.download_file(instrumental_url, instrumental_path)
|
||||
log_progress(f"Downloaded instrumental ({instrumental_selection})")
|
||||
|
||||
if existing_instrumental_path:
|
||||
# Batch 3: Use user-provided existing instrumental
|
||||
ext = Path(existing_instrumental_path).suffix.lower()
|
||||
instrumental_path = os.path.join(temp_dir, f"{base_name} (Instrumental User){ext}")
|
||||
log_progress(f"Downloading user-provided existing instrumental...")
|
||||
storage.download_file(existing_instrumental_path, instrumental_path)
|
||||
log_progress("Downloaded user-provided instrumental")
|
||||
else:
|
||||
# Use AI-separated instrumental based on selection
|
||||
instrumental_key = 'instrumental_clean' if instrumental_selection == 'clean' else 'instrumental_with_backing'
|
||||
instrumental_url = job.file_urls['stems'][instrumental_key]
|
||||
instrumental_suffix = "Clean" if instrumental_selection == 'clean' else "Backing"
|
||||
instrumental_path = os.path.join(temp_dir, f"{base_name} (Instrumental {instrumental_suffix}).flac")
|
||||
log_progress(f"Downloading {instrumental_selection} instrumental audio...")
|
||||
storage.download_file(instrumental_url, instrumental_path)
|
||||
log_progress(f"Downloaded instrumental ({instrumental_selection})")
|
||||
|
||||
# Download LRC file if available (needed for CDG/TXT)
|
||||
lyrics_urls = job.file_urls.get('lyrics', {})
|
||||
@@ -549,3 +577,64 @@ async def _upload_results(
|
||||
logger.info(f"Job {job_id}: Uploaded {file_key}")
|
||||
except Exception as e:
|
||||
logger.error(f"Job {job_id}: Failed to upload {file_key}: {e}")
|
||||
|
||||
|
||||
# ==================== CLI Entry Point for Cloud Run Jobs ====================
|
||||
# This allows the video worker to be run as a standalone Cloud Run Job.
|
||||
# Usage: python -m backend.workers.video_worker --job-id <job_id>
|
||||
|
||||
def main():
|
||||
"""
|
||||
CLI entry point for running video worker as a Cloud Run Job.
|
||||
|
||||
This is used when video encoding needs more than 30 minutes
|
||||
(the Cloud Tasks timeout limit). Cloud Run Jobs support up to 24 hours.
|
||||
|
||||
Usage:
|
||||
python -m backend.workers.video_worker --job-id abc123
|
||||
|
||||
Environment Variables:
|
||||
GOOGLE_CLOUD_PROJECT: GCP project ID (required)
|
||||
GCS_BUCKET_NAME: Storage bucket name (required)
|
||||
"""
|
||||
import argparse
|
||||
import asyncio
|
||||
import sys
|
||||
|
||||
# Set up argument parser
|
||||
parser = argparse.ArgumentParser(
|
||||
description="Video encoding worker for karaoke generation"
|
||||
)
|
||||
parser.add_argument(
|
||||
"--job-id",
|
||||
required=True,
|
||||
help="Job ID to process"
|
||||
)
|
||||
|
||||
args = parser.parse_args()
|
||||
job_id = args.job_id
|
||||
|
||||
# Configure logging
|
||||
logging.basicConfig(
|
||||
level=logging.INFO,
|
||||
format="%(asctime)s - %(name)s - %(levelname)s - %(message)s"
|
||||
)
|
||||
|
||||
logger.info(f"Starting video worker CLI for job {job_id}")
|
||||
|
||||
# Run the async worker function
|
||||
try:
|
||||
success = asyncio.run(generate_video(job_id))
|
||||
if success:
|
||||
logger.info(f"Video generation completed successfully for job {job_id}")
|
||||
sys.exit(0)
|
||||
else:
|
||||
logger.error(f"Video generation failed for job {job_id}")
|
||||
sys.exit(1)
|
||||
except Exception as e:
|
||||
logger.error(f"Video worker crashed: {e}", exc_info=True)
|
||||
sys.exit(1)
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
main()
|
||||
@@ -7,11 +7,48 @@ for remote debugging via the CLI.
|
||||
Two approaches are provided:
|
||||
1. JobLogger - A custom logger class for explicit logging in workers
|
||||
2. JobLogHandler - A logging.Handler that captures logs from any logger (including dependencies)
|
||||
|
||||
IMPORTANT: Uses contextvars to ensure log isolation between concurrent jobs.
|
||||
When multiple jobs run in parallel on the same Cloud Run instance, each job's
|
||||
worker logs are correctly filtered to only include logs from that job.
|
||||
"""
|
||||
import contextvars
|
||||
import logging
|
||||
from contextlib import contextmanager
|
||||
from typing import Optional, Set
|
||||
|
||||
|
||||
# Context variable to track the current job being processed
|
||||
# This is used to filter logs when multiple jobs run concurrently
|
||||
_current_job_id: contextvars.ContextVar[Optional[str]] = contextvars.ContextVar(
|
||||
'current_job_id', default=None
|
||||
)
|
||||
|
||||
|
||||
@contextmanager
|
||||
def job_logging_context(job_id: str):
|
||||
"""
|
||||
Context manager to set the current job ID for log filtering.
|
||||
|
||||
When multiple jobs run concurrently on the same Cloud Run instance,
|
||||
this ensures that each job's handler only captures logs from its own
|
||||
processing thread/context.
|
||||
|
||||
Usage:
|
||||
with job_logging_context(job_id):
|
||||
# All logs emitted here will be associated with this job_id
|
||||
process_job(job_id)
|
||||
|
||||
Args:
|
||||
job_id: The job ID to set as the current context
|
||||
"""
|
||||
token = _current_job_id.set(job_id)
|
||||
try:
|
||||
yield
|
||||
finally:
|
||||
_current_job_id.reset(token)
|
||||
|
||||
|
||||
class JobLogHandler(logging.Handler):
|
||||
"""
|
||||
A logging handler that captures log records and stores them in Firestore.
|
||||
@@ -53,6 +90,15 @@ class JobLogHandler(logging.Handler):
|
||||
def emit(self, record: logging.LogRecord) -> None:
|
||||
"""Process a log record."""
|
||||
try:
|
||||
# CRITICAL: Filter by current job context to prevent log mixing
|
||||
# When multiple jobs run concurrently, each attaches handlers to
|
||||
# the same global loggers. Without this check, logs from Job A
|
||||
# would be captured by Job B's handler and vice versa.
|
||||
current_job = _current_job_id.get()
|
||||
if current_job is not None and current_job != self.job_id:
|
||||
# This log is from a different job's context, skip it
|
||||
return
|
||||
|
||||
# Format the log message
|
||||
message = self.format(record)
|
||||
|
||||
@@ -182,6 +228,10 @@ def setup_job_logging(job_id: str, worker_name: str, *logger_names: str) -> JobL
|
||||
This adds a JobLogHandler to capture logs from specified loggers
|
||||
(including dependency loggers like lyrics_transcriber).
|
||||
|
||||
IMPORTANT: When using this function, wrap your job processing code in
|
||||
`job_logging_context(job_id)` to ensure proper log isolation when multiple
|
||||
jobs run concurrently on the same Cloud Run instance.
|
||||
|
||||
Args:
|
||||
job_id: Job ID
|
||||
worker_name: Worker name
|
||||
@@ -199,7 +249,10 @@ def setup_job_logging(job_id: str, worker_name: str, *logger_names: str) -> JobL
|
||||
"lyrics_transcriber", # Capture LyricsTranscriber logs
|
||||
)
|
||||
|
||||
# ... do work ...
|
||||
# IMPORTANT: Use job_logging_context for proper isolation
|
||||
with job_logging_context(job_id):
|
||||
# ... do work ...
|
||||
pass
|
||||
|
||||
# Optional: remove handler when done
|
||||
for name in logger_names:
|
||||
|
||||
@@ -1,10 +1,32 @@
|
||||
# Backend Feature Parity Plan
|
||||
|
||||
**Last Updated:** 2024-12-10
|
||||
**Status:** ✅ Core Feature Parity Achieved (v0.71.0)
|
||||
**Last Updated:** 2024-12-11
|
||||
**Status:** ✅ Core Feature Parity Achieved (v0.71.0) | Batch 1 ✅ | Batch 2 ✅ | Batch 3 ✅ | Batch 4 ✅
|
||||
|
||||
This document tracks the progress toward complete feature parity between the local `karaoke-gen` CLI and the cloud backend, enabling the `karaoke-gen-remote` CLI to have equivalent functionality.
|
||||
|
||||
### Quick Parity Summary
|
||||
|
||||
| Category | Supported | Needed | Parity | Notes |
|
||||
|----------|-----------|--------|--------|-------|
|
||||
| Core Processing | 12 | 12 | **100%** | ✅ Complete |
|
||||
| Distribution | 5 | 5 | **100%** | ✅ Complete |
|
||||
| Lyrics Configuration | 5 | 5 | **100%** | ✅ **Batch 1 Complete** (L1-L4) |
|
||||
| Style Configuration | 1 | 4 | **25%** | S2-S4 are MEDIUM priority |
|
||||
| Workflow Control | 0 | 3 | **0%** | W1/W2 HIGH, W7 MEDIUM (skip flags deferred) |
|
||||
| Audio Processing | 4 | 4 | **100%** | ✅ **Batch 2 & 3 Complete** (AP1-AP3, AP5) |
|
||||
| Input Modes | 2 | 3 | **67%** | ✅ **Batch 4 Complete** (P2), P3 remaining |
|
||||
| Audio Fetching | 0 | 1 | **0%** | A1 is HIGH priority |
|
||||
| Flags to Remove | - | - | - | I3-I6, AP6 (simplify codebase) |
|
||||
|
||||
**Overall:** Core workflow complete. See [CLI Parameter Parity Analysis](#-cli-parameter-parity-analysis) for detailed parameter list with implementation plans.
|
||||
|
||||
**Priority Summary:**
|
||||
- **HIGH**: L1-L4 (lyrics), AP1-AP5 (audio models/instrumental), P2-P3 (YouTube/flacfetch), A1 (interactive search), W1/W2 (prep/finalise phases), F15 (keep-brand-code)
|
||||
- **MEDIUM**: S2-S4 (style overrides/video bg), W7 (edit-lyrics), F14 (email template)
|
||||
- **LOW/DEFERRED**: W3-W6 (skip flags), D2-D3 (debugging)
|
||||
- **REMOVE**: I3, I4, I5, I6, AP6 (unnecessary options)
|
||||
|
||||
---
|
||||
|
||||
## 🎉 Milestone: First Successful End-to-End Run
|
||||
@@ -39,6 +61,507 @@ karaoke-gen-remote \
|
||||
|
||||
---
|
||||
|
||||
## 📊 CLI Parameter Parity Analysis
|
||||
|
||||
This section provides a comprehensive comparison of all CLI parameters between `karaoke-gen` (local) and `karaoke-gen-remote`, including detailed explanations of what each parameter does and how it could work remotely.
|
||||
|
||||
**Legend:**
|
||||
- ✅ Fully supported
|
||||
- ⚠️ Partially supported (limited functionality)
|
||||
- ❌ Not supported (ignored or errors)
|
||||
- 🔹 Remote-only (not applicable to local CLI)
|
||||
- N/A Not applicable to this mode
|
||||
|
||||
---
|
||||
|
||||
### Positional Arguments
|
||||
|
||||
| # | Parameter | Local | Remote | Description |
|
||||
|---|-----------|-------|--------|-------------|
|
||||
| P1 | `<file>` | ✅ | ✅ | **Local audio file path.** Copies file to output directory, converts to WAV for processing. Supports MP3, FLAC, WAV, M4A, OGG, AAC. |
|
||||
| P2 | `<url>` (YouTube) | ✅ | ✅ | **YouTube/online URL.** Uses yt-dlp to download audio and extract metadata (artist/title from video title). |
|
||||
| P3 | `<artist> <title>` | ✅ | ❌ | **Audio search mode.** When no file is provided, uses flacfetch to search Deezer/Tidal/etc for high-quality audio and downloads it. |
|
||||
| P4 | `<folder>` | ✅ | ❌ | **Batch folder processing.** Process all audio files in a folder, using `--filename_pattern` to extract track titles from filenames. |
|
||||
|
||||
**Remote Implementation Status:** ✅ **BATCH 4 COMPLETE** (v0.71.26)
|
||||
- **P2 YouTube URLs**: ✅ IMPLEMENTED. Backend accepts URL via `/api/jobs/create-from-url` endpoint. Uses yt-dlp to download audio and extract metadata (artist/title). Remote CLI detects URLs and routes to URL endpoint.
|
||||
- **P3 Flacfetch search**: HIGH PRIORITY. Backend integrates flacfetch to search Deezer/Tidal/etc. IMPORTANT: Must support interactive mode via API - return search results to client, let user select, then proceed. Default should be interactive (not auto-select). Use case: Eventually expose audio source selection in web UI.
|
||||
- **P4 Folder batch**: LOW PRIORITY. Could accept ZIP upload, process each as separate job. Defer for now.
|
||||
|
||||
---
|
||||
|
||||
### Workflow Control
|
||||
|
||||
| # | Parameter | Local | Remote | Description |
|
||||
|---|-----------|-------|--------|-------------|
|
||||
| W1 | `--prep-only` | ✅ | ❌ | **Stop after preparation phase.** Downloads audio, separates stems, transcribes lyrics, creates title/end screens, but does NOT run finalisation (encoding, YouTube upload, etc.). Useful for reviewing intermediate outputs before committing to final render. |
|
||||
| W2 | `--finalise-only` | ✅ | ❌ | **Run only finalisation phase.** Must be run from a directory that was previously prepared with `--prep-only`. Picks up where prep left off: encodes videos, creates CDG/TXT packages, uploads to YouTube/Dropbox, etc. |
|
||||
| W3 | `--skip-transcription` | ✅ | ❌ | **Skip automatic lyrics transcription.** Skips the AudioShake transcription and auto-correction steps. Use this if you want to manually provide lyrics or if transcription keeps failing. Lyrics review UI will still open if you have existing lyrics files. |
|
||||
| W4 | `--skip-separation` | ✅ | ❌ | **Skip audio separation.** Skips the AI-powered stem separation (Modal API). Useful when re-processing a track where you already have the stems from a previous run, or when combined with `--existing_instrumental`. |
|
||||
| W5 | `--skip-lyrics` | ✅ | ❌ | **Skip all lyrics processing.** Skips fetching lyrics from Genius/Spotify, transcription, and review. Output will have no lyrics overlay - just instrumental video with title/end screens. |
|
||||
| W6 | `--lyrics-only` | ✅ | ❌ | **Process only lyrics.** Sets `--skip-separation` and skips title/end screen generation. Useful for re-doing just the lyrics on an existing track without re-running expensive audio separation. |
|
||||
| W7 | `--edit-lyrics` | ✅ | ❌ | **Edit lyrics of existing track.** Run from inside an existing track directory (e.g., `NOMAD-1234 - Artist - Title/`). Backs up existing outputs, re-runs lyrics transcription with the existing audio, then re-renders and re-uploads. Uses `--keep-brand-code` implicitly. |
|
||||
| W8 | `--resume` | N/A | 🔹 | **Resume monitoring a job.** Reconnects to an existing remote job and continues monitoring its progress, handling review/instrumental selection if needed. |
|
||||
| W9 | `--cancel` | N/A | 🔹 | **Cancel a running job.** Stops processing but keeps the job record in Firestore. Useful for aborting stuck jobs. |
|
||||
| W10 | `--retry` | N/A | 🔹 | **Retry a failed job.** Restarts processing from the last successful checkpoint. Only works on jobs with `failed` status. |
|
||||
| W11 | `--delete` | N/A | 🔹 | **Delete a job.** Permanently removes job record from Firestore and all associated files from GCS. |
|
||||
| W12 | `--list` | N/A | 🔹 | **List all jobs.** Shows all jobs with their status, artist, title. Supports filtering with `--filter-environment` and `--filter-client-id`. |
|
||||
|
||||
**Remote Implementation Plan:**
|
||||
- **W1 `--prep-only`**: HIGH PRIORITY. Remote CLI submits job with `prep_only=true`. Backend runs all prep tasks (audio separation, lyrics transcription with review UI, initial render). After completion, CLI downloads the entire output folder and exits. No finalisation (encoding, YouTube, Dropbox, etc.) runs. Use case: Client needs custom edits to stems or lyrics before final render.
|
||||
- **W2 `--finalise-only`**: HIGH PRIORITY. Remote CLI checks current directory for expected prep output files. Uploads ALL files from that folder to backend (including any manual customizations user made since prep). Backend runs finalisation and distribution steps. Use case: User edited instrumental manually, now wants cloud to handle encoding/distribution.
|
||||
- **W3-W6 Skip flags**: LOW PRIORITY. Rarely used; typically all together to regenerate title/end screens with different artist/title. Defer until pipeline refactor provides cleaner stage-targeting.
|
||||
- **W7 `--edit-lyrics`**: MEDIUM PRIORITY. Re-runs lyrics worker with existing audio/stems. Creates NEW YouTube upload (doesn't delete/replace existing). Backend should handle gracefully if video with identical title already exists (user will have manually deleted old one). Use case: Customer found typo in published video.
|
||||
|
||||
---
|
||||
|
||||
### Logging & Debugging
|
||||
|
||||
| # | Parameter | Local | Remote | Description |
|
||||
|---|-----------|-------|--------|-------------|
|
||||
| D1 | `--log_level` | ✅ | ✅ | **Set logging verbosity.** Options: `debug`, `info`, `warning`, `error`. Default: `info`. Affects console output detail level. |
|
||||
| D2 | `--dry_run` | ✅ | ❌ | **Simulate without changes.** Runs through the workflow logic but doesn't actually download, process, or create files. Useful for testing argument combinations. |
|
||||
| D3 | `--render_bounding_boxes` | ✅ | ❌ | **Debug text positioning.** Renders red bounding boxes around text regions in title/end screen images. Helps debug custom style configurations. |
|
||||
|
||||
**Remote Implementation Plan:**
|
||||
- **D2 `--dry_run`**: LOW PRIORITY. Backend could accept `dry_run=true`, validate inputs and return what WOULD happen without processing. Nice to have for testing.
|
||||
- **D3 `--render_bounding_boxes`**: LOW PRIORITY. Backend could enable this, useful for testing styles before full render. Nice to have for debugging style configs.
|
||||
|
||||
---
|
||||
|
||||
### Input/Output Configuration
|
||||
|
||||
| # | Parameter | Local | Remote | Description |
|
||||
|---|-----------|-------|--------|-------------|
|
||||
| I1 | `--filename_pattern` | ✅ | N/A | **Regex for batch processing.** Python regex with named group `(?P<title>...)` to extract track title from filenames. Used with folder input. Example: `'(?P<index>\d+) - (?P<title>.+).mp3'` |
|
||||
| I2 | `--output_dir` | ✅ | ✅ | **Output directory.** Where to write output files. Local: creates subdirectory per track. Remote: where to download completed files. Default: current directory. |
|
||||
| I3 | `--no_track_subfolders` | 🗑️ | N/A | **REMOVE FROM LOCAL CLI.** Flat output structure. Not needed - always create subfolders. |
|
||||
| I4 | `--lossless_output_format` | 🗑️ | N/A | **REMOVE FROM LOCAL CLI.** Stem audio format. FLAC is always fine, no need for WAV option. |
|
||||
| I5 | `--output_png` | 🗑️ | N/A | **REMOVE FROM LOCAL CLI.** Always output both PNG and JPG, no need for flag. |
|
||||
| I6 | `--output_jpg` | 🗑️ | N/A | **REMOVE FROM LOCAL CLI.** Always output both PNG and JPG, no need for flag. |
|
||||
|
||||
**Remote Implementation Plan:**
|
||||
- **I3, I4, I5, I6**: REMOVE from local CLI entirely. Not worth maintaining. Backend always uses FLAC, outputs both PNG+JPG, creates subfolders.
|
||||
|
||||
---
|
||||
|
||||
### Audio Fetching Configuration
|
||||
|
||||
| # | Parameter | Local | Remote | Description |
|
||||
|---|-----------|-------|--------|-------------|
|
||||
| A1 | `--auto-download` | ✅ | ❌ | **Auto-select audio source.** When using artist+title search mode (flacfetch), automatically select the best quality source instead of prompting for manual selection. Useful for scripted/batch processing. |
|
||||
|
||||
**Remote Implementation Plan:**
|
||||
- **A1 `--auto-download`**: HIGH PRIORITY. Backend must support BOTH modes:
|
||||
- **Interactive (default)**: API returns search results to client. Client displays options, user selects, client sends selection back to backend. Use case: Web UI will show audio source picker.
|
||||
- **Auto-download (`--auto-download` or `-y`)**: Backend auto-selects best quality source. Use case: Automated/batch processing.
|
||||
|
||||
---
|
||||
|
||||
### Audio Processing Configuration
|
||||
|
||||
| # | Parameter | Local | Remote | Description |
|
||||
|---|-----------|-------|--------|-------------|
|
||||
| AP1 | `--clean_instrumental_model` | ✅ | ✅ | **Stage 1 separation model.** AI model for initial vocal/instrumental separation. Default: `model_bs_roformer_ep_317_sdr_12.9755.ckpt`. Different models have different quality/speed tradeoffs. |
|
||||
| AP2 | `--backing_vocals_models` | ✅ | ✅ | **Stage 2 separation model(s).** AI model(s) for separating lead vocals from backing vocals. Default: `mel_band_roformer_karaoke_aufr33_viperx_sdr_10.1956.ckpt`. |
|
||||
| AP3 | `--other_stems_models` | ✅ | ✅ | **Multi-stem separation model.** AI model for separating drums, bass, guitar, piano, other. Default: `htdemucs_6s.yaml`. |
|
||||
| AP4 | `--model_file_dir` | ✅ | N/A | **Model cache directory.** Where to store/load AI model files. Default: `/tmp/audio-separator-models/`. |
|
||||
| AP5 | `--existing_instrumental` | ✅ | ✅ | **Use pre-made instrumental.** Path to an existing instrumental file to use instead of running AI separation. Useful when you have a better instrumental from another source (official instrumental release, manual editing, etc.). Backend still runs separation for stems, but uses provided instrumental for final video. |
|
||||
| AP6 | `--instrumental_format` | 🗑️ | N/A | **REMOVE FROM LOCAL CLI.** FLAC is always fine, no need for format option. |
|
||||
|
||||
**Remote Implementation Status:** ✅ **BATCH 2 & 3 COMPLETE** (PR #9 v0.71.24, PR #10 v0.71.26)
|
||||
- **AP1 `--clean_instrumental_model`**: ✅ IMPLEMENTED. Sent as form field to backend. Audio worker passes to Modal API.
|
||||
- **AP2 `--backing_vocals_models`**: ✅ IMPLEMENTED. Sent as comma-separated list, parsed and passed to Modal API.
|
||||
- **AP3 `--other_stems_models`**: ✅ IMPLEMENTED. Sent as comma-separated list, parsed and passed to Modal API.
|
||||
- **AP4 `--model_file_dir`**: N/A for remote - Modal API handles model caching internally.
|
||||
- **AP5 `--existing_instrumental`**: ✅ IMPLEMENTED (Batch 3). User uploads instrumental file with job. Backend validates duration match (±0.5s), still runs separation for stems, but uses provided instrumental for final video encoding.
|
||||
- **AP6**: REMOVE from local CLI - FLAC is always fine.
|
||||
|
||||
---
|
||||
|
||||
### Lyrics Configuration
|
||||
|
||||
| # | Parameter | Local | Remote | Description |
|
||||
|---|-----------|-------|--------|-------------|
|
||||
| L1 | `--lyrics_artist` | ✅ | ✅ | **Override artist for lyrics search.** Use different artist name when searching Genius/Spotify/Musixmatch for lyrics. Useful for covers, remixes, or when artist name differs in lyrics databases. Example: `--lyrics_artist="The Beatles"` when your file says "Beatles, The". |
|
||||
| L2 | `--lyrics_title` | ✅ | ✅ | **Override title for lyrics search.** Use different song title when searching for lyrics. Useful for songs with alternate titles, subtitle variations, or "(Remastered)" suffixes. Example: `--lyrics_title="Hey Jude"` when file says "Hey Jude - 2009 Remaster". |
|
||||
| L3 | `--lyrics_file` | ✅ | ✅ | **Use existing lyrics file.** Path to a text file containing lyrics to use instead of fetching from online sources. Supports TXT, DOCX, RTF formats. Useful when online lyrics are wrong or unavailable. Transcription still runs to generate timestamps. |
|
||||
| L4 | `--subtitle_offset_ms` | ✅ | ✅ | **Adjust subtitle timing.** Shift all subtitle timestamps by N milliseconds. Positive values delay subtitles, negative values advance them. Useful when audio has intro padding or sync drift. Example: `--subtitle_offset_ms=500` delays by 0.5 seconds. |
|
||||
| L5 | `--skip_transcription_review` | ✅ | ⚠️ | **Skip review UI.** Don't open the browser-based lyrics review interface after transcription. Use existing auto-corrected lyrics as-is. Remote: use `-y` flag instead for non-interactive mode. |
|
||||
|
||||
**Remote Implementation Status:** ✅ **BATCH 1 COMPLETE** (PR #8, v0.71.22)
|
||||
- **L1 `--lyrics_artist`**: ✅ IMPLEMENTED. Sent as form field to backend. Lyrics worker uses for Genius/Spotify/AudioShake queries.
|
||||
- **L2 `--lyrics_title`**: ✅ IMPLEMENTED. Sent as form field to backend. Lyrics worker uses for search instead of main title.
|
||||
- **L3 `--lyrics_file`**: ✅ IMPLEMENTED. Uploaded with job, stored in GCS. Lyrics worker reads it instead of fetching online.
|
||||
- **L4 `--subtitle_offset_ms`**: ✅ IMPLEMENTED. Sent to backend, render worker applies offset when burning subtitles.
|
||||
- **L5 `--skip_transcription_review`**: Already works via `-y` flag which auto-accepts corrections.
|
||||
|
||||
---
|
||||
|
||||
### Style Configuration
|
||||
|
||||
| # | Parameter | Local | Remote | Description |
|
||||
|---|-----------|-------|--------|-------------|
|
||||
| S1 | `--style_params_json` | ✅ | ✅ | **Style configuration file.** Path to JSON file defining visual style: background images, fonts, colors, text positioning for intro/karaoke/end screens and CDG. All referenced image/font files are auto-uploaded in remote mode. |
|
||||
| S2 | `--style_override` | ✅ | ❌ | **Quick style tweaks.** Override specific style parameters without editing JSON. Can be used multiple times. Format: `section.key=value`. Example: `--style_override 'intro.background_image=/path/to/new_bg.png'`. Creates a temp JSON file with overrides merged. |
|
||||
| S3 | `--background_video` | ✅ | ❌ | **Video background for karaoke.** Path to video file to use as animated background instead of static image for the karaoke section. Video is looped/trimmed to match audio duration, with lyrics overlaid. |
|
||||
| S4 | `--background_video_darkness` | ✅ | ❌ | **Dim video background.** Darkness overlay percentage (0-100) applied to video background so lyrics are more readable. 0 = no darkening, 100 = completely black. Default: 0. |
|
||||
|
||||
**Remote Implementation Plan:**
|
||||
- **S2 `--style_override`**: MEDIUM PRIORITY. Send override strings to backend, merge server-side before passing to workers. Works same as local (creates merged config). Use case: Web UI will let users tweak font colors etc., needs to pass adjustments to backend.
|
||||
- **S3 `--background_video`**: MEDIUM PRIORITY. Upload video file with job. Backend processes with VideoBackgroundProcessor (loop/trim to audio duration). Requires larger file upload limits. Future: Web UI could allow YouTube URL or direct upload for video background.
|
||||
- **S4 `--background_video_darkness`**: Depends on S3. Send percentage to backend, apply overlay during video processing.
|
||||
|
||||
---
|
||||
|
||||
### Finalisation Configuration
|
||||
|
||||
| # | Parameter | Local | Remote | Description |
|
||||
|---|-----------|-------|--------|-------------|
|
||||
| F1 | `--enable_cdg` | ✅ | ✅ | **Generate CDG+MP3 package.** Create CD+Graphics karaoke format files (used by karaoke machines). Produces ZIP containing `.cdg` and `.mp3` files. Requires `cdg` section in style_params_json. |
|
||||
| F2 | `--enable_txt` | ✅ | ✅ | **Generate TXT+MP3 package.** Create text-based karaoke format with timed lyrics. Produces ZIP containing `.txt` (with timestamps) and `.mp3` files. |
|
||||
| F3 | `--brand_prefix` | ✅ | ✅ | **Brand code prefix.** Your brand identifier for sequential numbering. When set, output folder is renamed to `PREFIX-XXXX - Artist - Title` where XXXX is the next sequential number found in `organised_dir` or Dropbox. Example: `NOMAD` produces `NOMAD-1234`. |
|
||||
| F4 | `--organised_dir` | ✅ | N/A | **Local organized output.** Local filesystem path where final track folders are moved after processing. Used with `--brand_prefix` to maintain organized library. Example: `/Volumes/Media/Karaoke/Tracks-Organized/`. |
|
||||
| F5 | `--organised_dir_rclone_root` | ✅ | ✅ | **Rclone path for Dropbox.** Rclone remote path that maps to your organised_dir. Used to calculate next brand code number and for rclone-based uploads. Example: `dropbox:Media/Karaoke/Tracks-Organized`. |
|
||||
| F6 | `--public_share_dir` | ✅ | N/A | **Local public share folder.** Local filesystem path where shareable versions (720p video, CDG, TXT) are copied for public distribution. |
|
||||
| F7 | `--enable_youtube_upload` | ✅ | ✅ | **Upload to YouTube.** After encoding, upload the lossy 4K video to YouTube. Local: uses OAuth flow with client secrets. Remote: uses server-side stored credentials. |
|
||||
| F8 | `--youtube_client_secrets_file` | ✅ | N/A | **YouTube OAuth credentials.** Local path to Google OAuth client secrets JSON file for YouTube API. Only needed for local mode - remote uses server-stored credentials. |
|
||||
| F9 | `--youtube_description_file` | ✅ | ✅ | **YouTube video description.** Path to text file containing YouTube video description template. Supports placeholders like `{artist}`, `{title}`, `{brand_code}`. |
|
||||
| F10 | `--rclone_destination` | ✅ | N/A | **Rclone sync destination.** Rclone remote path to sync public_share_dir to. Used for Google Drive public sharing. Example: `googledrive:KaraokePublic`. |
|
||||
| F11 | `--dropbox_path` | N/A | 🔹 | **Dropbox folder (native API).** Remote-only: Dropbox folder path for organized output upload using native Dropbox API. Example: `/Karaoke/Tracks-Organized`. |
|
||||
| F12 | `--gdrive_folder_id` | N/A | 🔹 | **Google Drive folder (native API).** Remote-only: Google Drive folder ID for public share uploads using native API. Example: `1abc123xyz`. |
|
||||
| F13 | `--discord_webhook_url` | ✅ | ✅ | **Discord notification.** Webhook URL for sending completion notifications to Discord. Posts message with track info, brand code, and links. |
|
||||
| F14 | `--email_template_file` | ✅ | ❌ | **Email draft template.** Path to email template file for creating Gmail drafts with track info. Supports placeholders. Uses pyperclip for manual email composition. |
|
||||
| F15 | `--keep-brand-code` | ✅ | ❌ | **Preserve existing brand code.** When run from an existing track directory (with brand code in folder name), use that brand code instead of calculating a new one. Implicitly enabled with `--edit-lyrics`. |
|
||||
| F16 | `-y` / `--yes` | ✅ | ✅ | **Non-interactive mode.** Auto-accept all prompts: lyrics corrections, instrumental selection, confirmations. Useful for automated/CI pipelines. Remote: auto-completes review, selects clean instrumental. |
|
||||
| F17 | `--test_email_template` | ✅ | N/A | **Test email template.** Debug mode: test the email template rendering with fake data without processing any track. |
|
||||
|
||||
**Remote Implementation Plan:**
|
||||
- **F14 `--email_template_file`**: MEDIUM PRIORITY. Two behaviors needed:
|
||||
1. **Finalisation**: Same as local - use template to create Gmail draft (requires Gmail API OAuth). User just changes recipient and sends.
|
||||
2. **API endpoint**: Expose endpoint to fetch templated message text for a job. Use case: Web frontend button to copy message for pasting into Fiverr, etc.
|
||||
- **F15 `--keep-brand-code`**: HIGH PRIORITY (for `--finalise-only` workflow). When using `--finalise-only` remotely, user may be re-processing an already-published track. Must preserve existing brand code so updated files overwrite in Dropbox/GDrive rather than creating new sequence number. Implementation: Remote CLI reads brand code from local folder name, sends to backend.
|
||||
|
||||
---
|
||||
|
||||
### Remote CLI Specific Options
|
||||
|
||||
| # | Parameter | Local | Remote | Description |
|
||||
|---|-----------|-------|--------|-------------|
|
||||
| R1 | `--service-url` | N/A | 🔹 | **Backend service URL.** URL of the karaoke-gen cloud backend. Can also be set via `KARAOKE_GEN_URL` environment variable. Required for remote mode. |
|
||||
| R2 | `--review-ui-url` | N/A | 🔹 | **Lyrics review UI URL.** URL of the hosted lyrics review web application. Default: `https://lyrics.nomadkaraoke.com`. |
|
||||
| R3 | `--poll-interval` | N/A | 🔹 | **Status poll frequency.** Seconds between job status checks while monitoring. Default: 5 seconds. |
|
||||
| R4 | `--environment` | N/A | 🔹 | **Job environment tag.** Tag jobs with environment label (test/production/development) for filtering and cleanup. Sent as `X-Environment` header. |
|
||||
| R5 | `--client-id` | N/A | 🔹 | **Client identifier tag.** Tag jobs with client/user identifier for filtering. Sent as `X-Client-ID` header. |
|
||||
| R6 | `--filter-environment` | N/A | 🔹 | **Filter by environment.** When using `--list` or `--bulk-delete`, only show/delete jobs matching this environment. |
|
||||
| R7 | `--filter-client-id` | N/A | 🔹 | **Filter by client ID.** When using `--list` or `--bulk-delete`, only show/delete jobs matching this client ID. |
|
||||
| R8 | `--bulk-delete` | N/A | 🔹 | **Bulk delete jobs.** Delete all jobs matching filter criteria. Requires at least one filter (`--filter-environment` or `--filter-client-id`). |
|
||||
|
||||
---
|
||||
|
||||
## 🚀 Implementation Batches
|
||||
|
||||
The following batches are ordered by priority and grouped by similar functionality. Each batch is designed to be a self-contained unit of work that can be tackled by a single agent session.
|
||||
|
||||
### 📋 Instructions for Implementing a Batch
|
||||
|
||||
**Before starting a batch, ensure you follow these steps:**
|
||||
|
||||
#### 1. Test Coverage Requirements
|
||||
All code changes must have **>70% test coverage** on modified files. Run:
|
||||
```bash
|
||||
python -m pytest backend/tests/ --cov=backend.models.job --cov=backend.api.routes.file_upload --cov=backend.workers.<worker_name> --cov-report=term-missing
|
||||
```
|
||||
|
||||
**Required tests for each batch:**
|
||||
- **Model changes:** Test new fields exist, have correct defaults, serialize properly
|
||||
- **API changes:** Test form validation (valid/invalid inputs), file upload handling
|
||||
- **Worker changes:** Test parameter passing, configuration, error handling
|
||||
- **CLI changes:** Test submit_job() passes parameters correctly
|
||||
|
||||
#### 2. Documentation Updates (REQUIRED)
|
||||
As part of your PR, you MUST update this document:
|
||||
- [ ] Update the **Quick Parity Summary** table to reflect new parity percentages
|
||||
- [ ] Mark parameters as ✅ in the relevant **CLI Parameter Parity Analysis** section
|
||||
- [ ] Update **Remote Implementation Plan/Status** to show ✅ IMPLEMENTED
|
||||
- [ ] Mark the batch as **✅ COMPLETE** with PR number in the **Summary Table**
|
||||
- [ ] Add entry to **Recently Completed** table with version and PR number
|
||||
- [ ] Remove items from **Not Yet Implemented** list
|
||||
|
||||
#### 3. Version Bump
|
||||
Bump the patch version in `pyproject.toml` before committing.
|
||||
|
||||
#### 4. PR Requirements
|
||||
- Title: `feat: Batch N - <description>`
|
||||
- Include: Summary of changes, files modified, tests added
|
||||
- Link to this document in PR description
|
||||
|
||||
---
|
||||
|
||||
### Batch 1: Lyrics Configuration (HIGH) ✅ COMPLETE
|
||||
**Parameters:** L1 `--lyrics_artist`, L2 `--lyrics_title`, L3 `--lyrics_file`, L4 `--subtitle_offset_ms`
|
||||
|
||||
**Status:** ✅ **IMPLEMENTED** - PR #8 merged (v0.71.22)
|
||||
|
||||
**Implementation Summary:**
|
||||
- Remote CLI: Added 4 form fields to `submit_job()`, uploads lyrics file if provided
|
||||
- Backend: Added 4 Form parameters to `/api/jobs/upload` endpoint
|
||||
- Backend: Lyrics worker downloads user lyrics file, passes overrides to LyricsProcessor
|
||||
- Backend: Render worker applies subtitle offset via OutputConfig
|
||||
|
||||
**Files modified:**
|
||||
- `karaoke_gen/utils/remote_cli.py` - submit_job() with lyrics params
|
||||
- `backend/api/routes/file_upload.py` - Form fields + lyrics file upload
|
||||
- `backend/models/job.py` - lyrics_artist, lyrics_title, lyrics_file_gcs_path, subtitle_offset_ms fields
|
||||
- `backend/workers/lyrics_worker.py` - override values passed to LyricsProcessor
|
||||
- `backend/workers/render_video_worker.py` - subtitle_offset_ms applied
|
||||
|
||||
**Tests added:** 25+ unit tests covering lyrics file validation, Job model fields, worker configuration
|
||||
|
||||
**Complexity:** Low-Medium (straightforward form fields + file upload)
|
||||
|
||||
---
|
||||
|
||||
### Batch 2: Audio Separation Model Selection (HIGH) ✅ COMPLETE
|
||||
**Parameters:** AP1 `--clean_instrumental_model`, AP2 `--backing_vocals_models`, AP3 `--other_stems_models`
|
||||
|
||||
**Status:** ✅ **IMPLEMENTED** - PR #9 merged (v0.71.24)
|
||||
|
||||
**Implementation Summary:**
|
||||
- Remote CLI: Added 3 form fields to `submit_job()`, sends models as comma-separated strings
|
||||
- Backend: Added 3 Form parameters to `/api/jobs/upload` endpoint, parses comma-separated lists
|
||||
- Backend: Job model stores model configuration as optional fields
|
||||
- Backend: Audio worker passes model names to AudioProcessor which uses Modal API
|
||||
|
||||
**Files modified:**
|
||||
- `karaoke_gen/utils/remote_cli.py` - submit_job() with model params
|
||||
- `backend/api/routes/file_upload.py` - Form fields + comma parsing
|
||||
- `backend/models/job.py` - clean_instrumental_model, backing_vocals_models, other_stems_models fields
|
||||
- `backend/workers/audio_worker.py` - create_audio_processor() accepts model params from job
|
||||
|
||||
**Tests added:** 15+ unit tests covering Job model fields, form parsing, AudioProcessor configuration
|
||||
|
||||
**Complexity:** Low (Modal API already supports dynamic model selection)
|
||||
|
||||
---
|
||||
|
||||
### Batch 3: Existing Instrumental Support (HIGH) ✅ COMPLETE
|
||||
**Parameters:** AP5 `--existing_instrumental`
|
||||
|
||||
**Status:** ✅ **IMPLEMENTED** - PR #10 merged (v0.71.26)
|
||||
|
||||
**Implementation Summary:**
|
||||
- Remote CLI: Added `--existing_instrumental` parameter, uploads file as `existing_instrumental` type
|
||||
- Backend: Added `existing_instrumental` to VALID_FILE_TYPES with audio extension validation
|
||||
- Backend: Duration validation (±0.5s tolerance) at uploads-complete endpoint using pydub
|
||||
- Backend: Job model has `existing_instrumental_gcs_path` field
|
||||
- Backend: Video worker downloads user-provided instrumental if present, uses it for final encoding
|
||||
|
||||
**Files modified:**
|
||||
- `karaoke_gen/utils/remote_cli.py` - submit_job() with existing_instrumental param
|
||||
- `backend/api/routes/file_upload.py` - VALID_FILE_TYPES, GCS path generation, duration validation
|
||||
- `backend/models/job.py` - existing_instrumental_gcs_path field
|
||||
- `backend/workers/video_worker.py` - downloads and uses user-provided instrumental
|
||||
|
||||
**Tests added:** 20+ unit tests covering model fields, file type validation, GCS path generation, duration validation
|
||||
|
||||
**Complexity:** Medium (file upload + validation + conditional logic)
|
||||
|
||||
---
|
||||
|
||||
### Batch 4: YouTube URL Input (HIGH) ✅ COMPLETE
|
||||
**Parameters:** P2 `<url>` (YouTube/online URL)
|
||||
|
||||
**Status:** ✅ **IMPLEMENTED** (v0.71.26)
|
||||
|
||||
**Implementation Summary:**
|
||||
- Remote CLI: Detects URLs via `is_url()`, routes to `submit_job_from_url()` method
|
||||
- Backend: New `/api/jobs/create-from-url` endpoint accepts URL and optional artist/title
|
||||
- Backend: Audio worker downloads from URL using yt-dlp, auto-detects artist/title from metadata
|
||||
- FileHandler: Added `download_video()` and `extract_metadata_from_url()` methods
|
||||
- Dependencies: yt-dlp added to pyproject.toml
|
||||
|
||||
**Files modified:**
|
||||
- `karaoke_gen/utils/remote_cli.py` - URL detection, new `submit_job_from_url()` method
|
||||
- `karaoke_gen/file_handler.py` - `download_video()` and `extract_metadata_from_url()` methods
|
||||
- `backend/api/routes/file_upload.py` - `CreateJobFromUrlRequest`, `CreateJobFromUrlResponse`, `/api/jobs/create-from-url` endpoint
|
||||
- `backend/workers/audio_worker.py` - Enhanced `download_from_url()` with metadata extraction
|
||||
- `pyproject.toml` - Added yt-dlp dependency
|
||||
|
||||
**Tests added:** 20+ unit tests covering URL validation, request/response models, Job URL field
|
||||
|
||||
**Complexity:** Medium-High (new input mode, yt-dlp integration)
|
||||
|
||||
---
|
||||
|
||||
### Batch 5: Flacfetch Audio Search (HIGH)
|
||||
**Parameters:** P3 `<artist> <title>` (search mode), A1 `--auto-download`
|
||||
|
||||
**Scope:**
|
||||
- Remote CLI: Detect artist+title without file, trigger search flow
|
||||
- Backend: New endpoint to search for audio sources via flacfetch
|
||||
- Backend: Return search results to client for interactive selection
|
||||
- Remote CLI: Display results, let user select, send selection back
|
||||
- Backend: Download selected audio, proceed with job
|
||||
- Support `--auto-download` / `-y` to skip interactive selection
|
||||
|
||||
**Files to modify:**
|
||||
- `karaoke_gen/utils/remote_cli.py` - search flow, display results, selection
|
||||
- `backend/api/routes/` - new search endpoint
|
||||
- `backend/services/` - flacfetch integration service
|
||||
- `backend/workers/` - download from selected source
|
||||
|
||||
**Complexity:** High (new interactive API flow, new service integration)
|
||||
|
||||
---
|
||||
|
||||
### Batch 6: Two-Phase Workflow (HIGH)
|
||||
**Parameters:** W1 `--prep-only`, W2 `--finalise-only`, F15 `--keep-brand-code`
|
||||
|
||||
**Scope:**
|
||||
- **prep-only:** Remote CLI submits with `prep_only=true`. Backend runs through review, then stops. CLI downloads all outputs and exits.
|
||||
- **finalise-only:** Remote CLI checks local folder for prep outputs, uploads ALL files, sends `finalise_only=true`. Backend runs finalisation only.
|
||||
- **keep-brand-code:** Remote CLI extracts brand code from local folder name, sends to backend to preserve.
|
||||
|
||||
**Files to modify:**
|
||||
- `karaoke_gen/utils/remote_cli.py` - prep-only flow, finalise-only upload flow, brand code extraction
|
||||
- `backend/api/routes/file_upload.py` - accept prep_only, finalise_only flags
|
||||
- `backend/api/routes/` - new endpoint for finalise-only upload (multiple files)
|
||||
- `backend/workers/orchestrator.py` or job state machine - conditional worker execution
|
||||
|
||||
**Complexity:** High (significant workflow changes, new upload flow)
|
||||
|
||||
---
|
||||
|
||||
### Batch 7: Style Overrides (MEDIUM)
|
||||
**Parameters:** S2 `--style_override`
|
||||
|
||||
**Scope:**
|
||||
- Remote CLI: Parse override strings, send as array to backend
|
||||
- Backend: Accept style_override array
|
||||
- Backend: Merge overrides with uploaded style JSON before processing
|
||||
- Handle asset references in overrides (upload referenced files)
|
||||
|
||||
**Files to modify:**
|
||||
- `karaoke_gen/utils/remote_cli.py` - parse and send overrides
|
||||
- `backend/api/routes/file_upload.py` - accept overrides
|
||||
- `backend/services/style_service.py` or similar - merge logic
|
||||
|
||||
**Complexity:** Medium (string parsing, merge logic, asset handling)
|
||||
|
||||
---
|
||||
|
||||
### Batch 8: Video Background (MEDIUM)
|
||||
**Parameters:** S3 `--background_video`, S4 `--background_video_darkness`
|
||||
|
||||
**Scope:**
|
||||
- Remote CLI: Upload video file (larger file support needed)
|
||||
- Backend: Accept video upload, store in GCS
|
||||
- Backend: Render worker uses VideoBackgroundProcessor
|
||||
- Backend: Apply darkness overlay
|
||||
|
||||
**Files to modify:**
|
||||
- `karaoke_gen/utils/remote_cli.py` - upload video file
|
||||
- `backend/api/routes/file_upload.py` - accept video upload (larger limits)
|
||||
- `backend/workers/render_worker.py` - integrate VideoBackgroundProcessor
|
||||
|
||||
**Complexity:** Medium (large file upload, video processing)
|
||||
|
||||
---
|
||||
|
||||
### Batch 9: Edit Lyrics Mode (MEDIUM)
|
||||
**Parameters:** W7 `--edit-lyrics`
|
||||
|
||||
**Scope:**
|
||||
- Remote CLI: Accept job_id for existing job, trigger edit mode
|
||||
- Backend: Load existing job's audio/stems from GCS
|
||||
- Backend: Re-run lyrics worker only
|
||||
- Backend: Re-render with new lyrics
|
||||
- Backend: Create NEW YouTube upload (handle duplicate title gracefully)
|
||||
|
||||
**Files to modify:**
|
||||
- `karaoke_gen/utils/remote_cli.py` - edit mode flow
|
||||
- `backend/api/routes/` - edit endpoint or flag
|
||||
- `backend/workers/orchestrator.py` - partial re-processing logic
|
||||
|
||||
**Complexity:** Medium-High (partial re-processing, state management)
|
||||
|
||||
---
|
||||
|
||||
### Batch 10: Email Template + Cleanup (LOW)
|
||||
**Parameters:** F14 `--email_template_file`, D2 `--dry_run`, D3 `--render_bounding_boxes`
|
||||
**Removal:** I3 `--no_track_subfolders`, I4 `--lossless_output_format`, I5 `--output_png`, I6 `--output_jpg`, AP6 `--instrumental_format`
|
||||
|
||||
**Scope:**
|
||||
- **Email template:** Upload template, backend creates Gmail draft + new API endpoint to fetch formatted text
|
||||
- **Dry run:** Backend validates inputs, returns plan without executing
|
||||
- **Bounding boxes:** Enable debug rendering in screens worker
|
||||
- **Removal:** Remove 5 unnecessary CLI flags from local CLI
|
||||
|
||||
**Files to modify:**
|
||||
- `karaoke_gen/utils/cli_args.py` - remove flags
|
||||
- `karaoke_gen/utils/remote_cli.py` - email template upload
|
||||
- `backend/api/routes/` - email template endpoint
|
||||
- `backend/services/email_service.py` - Gmail API integration
|
||||
|
||||
**Complexity:** Mixed (Gmail OAuth is complex, removals are simple)
|
||||
|
||||
---
|
||||
|
||||
### Summary Table
|
||||
|
||||
| Batch | Parameters | Priority | Complexity | Status |
|
||||
|-------|------------|----------|------------|--------|
|
||||
| 1 | L1-L4 (lyrics config) | HIGH | Low-Medium | ✅ **COMPLETE** (PR #8) |
|
||||
| 2 | AP1-AP3 (model selection) | HIGH | Low | ✅ **COMPLETE** (PR #9) |
|
||||
| 3 | AP5 (existing instrumental) | HIGH | Medium | ✅ **COMPLETE** (PR #10) |
|
||||
| 4 | P2 (YouTube URLs) | HIGH | Medium-High | ✅ **COMPLETE** |
|
||||
| 5 | P3, A1 (flacfetch search) | HIGH | High | ⏳ Pending |
|
||||
| 6 | W1, W2, F15 (two-phase) | HIGH | High | ⏳ Pending |
|
||||
| 7 | S2 (style overrides) | MEDIUM | Medium | ⏳ Pending |
|
||||
| 8 | S3, S4 (video background) | MEDIUM | Medium | ⏳ Pending |
|
||||
| 9 | W7 (edit-lyrics) | MEDIUM | Medium-High | ⏳ Pending |
|
||||
| 10 | F14, D2, D3, removals | LOW | Mixed | ⏳ Pending |
|
||||
|
||||
**Recommended Order:** ~~1~~ → ~~2~~ → ~~3~~ → ~~4~~ → 5 → 6 → 7 → 8 → 9 → 10
|
||||
|
||||
Batches 1-4 are quick wins that add significant value. Batches 5-6 are larger but high priority. Batches 7-10 can be done as time permits.
|
||||
|
||||
---
|
||||
|
||||
## ✅ Implementation Decisions (Answered 2024-12-10)
|
||||
|
||||
All questions have been answered and incorporated into the "Remote Implementation Plan" sections above. Summary of key decisions:
|
||||
|
||||
### Workflow Control
|
||||
- **W1/W2**: YES, two-phase workflow needed. `--prep-only` downloads all prep outputs after review, `--finalise-only` uploads local folder (with any manual edits) for cloud finalisation.
|
||||
- **W3-W6 Skip flags**: LOW PRIORITY - rarely used, defer until pipeline refactor provides cleaner stage-targeting.
|
||||
- **W7 Edit-lyrics**: Creates NEW YouTube upload (user deletes old one manually). Backend handles duplicate title gracefully.
|
||||
|
||||
### Input Modes
|
||||
- **P2 YouTube**: HIGH PRIORITY - needed for niche live versions only on YouTube.
|
||||
- **P3 Flacfetch**: HIGH PRIORITY - MUST be interactive via API (not auto-select) for future web UI audio source picker.
|
||||
|
||||
### Audio Processing
|
||||
- **AP1-AP3 Models**: HIGH PRIORITY - all model options must work remotely (Modal API can load any model on demand).
|
||||
- **AP5 Existing Instrumental**: Still run separation (for consistent stem downloads), but use provided instrumental for final video. Validate duration match (±0.5s) at job start before proceeding.
|
||||
|
||||
### I/O & Audio Format Options
|
||||
- **I3, I4, I5, I6, AP6**: REMOVE from local CLI entirely - unnecessary options, simplifies codebase.
|
||||
|
||||
### Audio Fetching
|
||||
- **A1 `--auto-download`**: Backend must support BOTH interactive (default, for web UI) AND auto-select (with `-y` flag) modes.
|
||||
|
||||
### Style
|
||||
- **S2 Style Override**: MEDIUM PRIORITY - needed for web UI style tweaks (font colors, etc.).
|
||||
- **S3/S4 Background Video**: MEDIUM PRIORITY - implement, will eventually allow YouTube URL or direct upload in web UI.
|
||||
|
||||
### Finalisation
|
||||
- **F14 Email Template**: BOTH - create Gmail draft (like local) AND expose API endpoint to fetch templated text (for Fiverr copy-paste).
|
||||
- **F15 Keep-brand-code**: HIGH PRIORITY for `--finalise-only` workflow - preserves brand code so updates overwrite existing Dropbox/GDrive files.
|
||||
|
||||
---
|
||||
|
||||
## Vision
|
||||
|
||||
The karaoke-gen system supports multiple interfaces to the same core functionality:
|
||||
@@ -105,14 +628,27 @@ The karaoke-gen system supports multiple interfaces to the same core functionali
|
||||
| Google Drive upload | ✅ | Service account credentials |
|
||||
| Discord notification | ✅ | Webhook URL from job or default |
|
||||
|
||||
### ⏳ Not Yet Implemented (Lower Priority)
|
||||
### ✅ Recently Completed
|
||||
|
||||
| Feature | Version | PR | Notes |
|
||||
|---------|---------|-----|-------|
|
||||
| YouTube URL input | v0.71.26 | - | `<url>` positional argument for YouTube/online URLs |
|
||||
| Existing instrumental | v0.71.26 | #10 | `--existing_instrumental` - upload and use user-provided instrumental |
|
||||
| Audio model selection | v0.71.24 | #9 | `--clean_instrumental_model`, `--backing_vocals_models`, `--other_stems_models` |
|
||||
| Lyrics override params | v0.71.22 | #8 | `--lyrics_artist`, `--lyrics_title`, `--lyrics_file`, `--subtitle_offset_ms` |
|
||||
|
||||
### ⏳ Not Yet Implemented
|
||||
|
||||
| Feature | Priority | Notes |
|
||||
|---------|----------|-------|
|
||||
| Style override | MEDIUM | `--style_override` for quick tweaks |
|
||||
| Skip separation/transcription | MEDIUM | Workflow control for re-processing |
|
||||
| Edit lyrics mode | MEDIUM | `--edit-lyrics` for fixing existing tracks |
|
||||
| Background video | MEDIUM | `--background_video` requires video upload |
|
||||
| Audio search (flacfetch) | HIGH | Artist+title search, auto-download |
|
||||
| Two-phase workflow | HIGH | `--prep-only`, `--finalise-only` |
|
||||
| Batch folder processing | LOW | Process multiple files at once |
|
||||
| Gmail draft creation | LOW | Nice-to-have, not blocking |
|
||||
| YouTube URL input | LOW | Requires yt-dlp in container |
|
||||
| Batch processing | LOW | Queue management needed |
|
||||
| Existing instrumental support | LOW | For re-processing existing tracks |
|
||||
|
||||
---
|
||||
|
||||
@@ -168,96 +704,17 @@ karaoke-gen-remote \
|
||||
|
||||
---
|
||||
|
||||
## Future Architecture: Shared Pipeline
|
||||
## Future Architecture
|
||||
|
||||
Once feature parity is achieved, the codebase should be refactored toward a **shared pipeline architecture** where both local and remote execution use the same abstractions.
|
||||
For the comprehensive architecture plan covering both **horizontal scalability** (Cloud Tasks, Cloud Run Jobs) and **software architecture** (shared pipeline, SOLID, DRY), see:
|
||||
|
||||
### Current State (Divergent Paths)
|
||||
**[SCALABLE-ARCHITECTURE-PLAN.md](./SCALABLE-ARCHITECTURE-PLAN.md)**
|
||||
|
||||
```
|
||||
LOCAL CLI CLOUD BACKEND
|
||||
───────── ─────────────
|
||||
KaraokeGen.process() API Routes
|
||||
│ │
|
||||
├─► AudioProcessor ├─► audio_worker.py
|
||||
│ └── Modal API or local │ └── Modal API
|
||||
│ │
|
||||
├─► LyricsProcessor ├─► lyrics_worker.py
|
||||
│ └── Orchestrates everything │ └── Transcription only
|
||||
│ including video generation │
|
||||
│ ├─► screens_worker.py
|
||||
│ │
|
||||
│ ├─► render_video_worker.py
|
||||
│ │ └── OutputGenerator directly
|
||||
│ │
|
||||
└─► KaraokeFinalise └─► video_worker.py
|
||||
└── Encoding, distribution └── KaraokeFinalise
|
||||
```
|
||||
|
||||
**Problems:**
|
||||
- Video generation called differently (via LyricsProcessor vs OutputGenerator directly)
|
||||
- LyricsProcessor does too many things (fetching, transcription, video, file management)
|
||||
- Testing requires mocking different things for local vs remote
|
||||
- Bug fixes may need to be applied in multiple places
|
||||
|
||||
### Target State (Shared Pipeline)
|
||||
|
||||
```
|
||||
┌─────────────────────────────────────────────────────────────────────────────┐
|
||||
│ SHARED PIPELINE ARCHITECTURE │
|
||||
├─────────────────────────────────────────────────────────────────────────────┤
|
||||
│ │
|
||||
│ AudioInput → Separation → Transcription → Review → Render → Finalize │
|
||||
│ │ │ │ │ │ │ │
|
||||
│ ▼ ▼ ▼ ▼ ▼ ▼ │
|
||||
│ ┌────────┐ ┌────────┐ ┌────────┐ ┌────────┐ ┌────────┐ ┌────────┐ │
|
||||
│ │ Stage │ │ Stage │ │ Stage │ │ Stage │ │ Stage │ │ Stage │ │
|
||||
│ │ API │ │ API │ │ API │ │ API │ │ API │ │ API │ │
|
||||
│ └────┬───┘ └────┬───┘ └────┬───┘ └────┬───┘ └────┬───┘ └────┬───┘ │
|
||||
│ │ │ │ │ │ │ │
|
||||
│ ┌────┴───────────┴────────────┴────────────┴──────────┴──────────┴────┐ │
|
||||
│ │ EXECUTION LAYER │ │
|
||||
│ │ │ │
|
||||
│ │ Local Mode: │ Remote Mode: │ │
|
||||
│ │ - Direct calls │ - HTTP to backend │ │
|
||||
│ │ - Local GPU/CPU │ - Workers + Modal │ │
|
||||
│ │ - Blocking │ - Async + polling │ │
|
||||
│ └──────────────────────┴───────────────────────────────────────────────┘ │
|
||||
│ │
|
||||
└─────────────────────────────────────────────────────────────────────────────┘
|
||||
```
|
||||
|
||||
**Benefits:**
|
||||
- Single source of truth for each pipeline stage
|
||||
- Each stage independently testable
|
||||
- Same business logic regardless of execution mode
|
||||
- Easier to add new stages or modify existing ones
|
||||
- DRY - fix bugs once, works everywhere
|
||||
|
||||
### Implementation Plan for Shared Pipeline
|
||||
|
||||
This refactor should be done in a separate branch after the v0.71.0 release:
|
||||
|
||||
1. **Extract stage interfaces**
|
||||
```python
|
||||
class PipelineStage(Protocol):
|
||||
async def execute(self, context: PipelineContext) -> StageResult:
|
||||
...
|
||||
```
|
||||
|
||||
2. **Create execution adapters**
|
||||
```python
|
||||
class LocalExecutor:
|
||||
"""Runs stages directly in-process"""
|
||||
|
||||
class RemoteExecutor:
|
||||
"""Runs stages via backend API/workers"""
|
||||
```
|
||||
|
||||
3. **Refactor incrementally**
|
||||
- Start with one stage (e.g., Separation)
|
||||
- Prove the pattern works
|
||||
- Migrate other stages
|
||||
Key goals:
|
||||
- **100+ concurrent jobs** with no performance degradation
|
||||
- **Dedicated resources per job** (no CPU/memory contention)
|
||||
- **Shared pipeline architecture** - single source of truth for business logic
|
||||
- **DRY codebase** - bug fixes apply to both local and remote execution
|
||||
|
||||
---
|
||||
|
||||
@@ -355,6 +812,37 @@ DEFAULT_DISCORD_WEBHOOK_URL=https://discord.com/...
|
||||
|
||||
## Recent Changes Log
|
||||
|
||||
### 2024-12-11: v0.71.26 - Batch 3 Complete (Existing Instrumental Support)
|
||||
|
||||
**PR #10: Existing Instrumental Support for Remote CLI**
|
||||
|
||||
- ✅ `--existing_instrumental` - Upload and use user-provided instrumental file
|
||||
- Duration validation (±0.5s tolerance) before processing starts
|
||||
- Backend still runs AI separation for stem downloads, but uses provided instrumental for final video
|
||||
- Added 20+ unit tests for new functionality
|
||||
- Files: `remote_cli.py`, `file_upload.py`, `job.py`, `video_worker.py`
|
||||
|
||||
### 2024-12-11: v0.71.24 - Batch 2 Complete (Audio Model Selection)
|
||||
|
||||
**PR #9: Audio Model Selection for Remote CLI**
|
||||
|
||||
- ✅ `--clean_instrumental_model` - Custom Stage 1 separation model
|
||||
- ✅ `--backing_vocals_models` - Custom backing vocals separation models
|
||||
- ✅ `--other_stems_models` - Custom multi-stem separation models (bass, drums, etc.)
|
||||
- Added 15+ unit tests for new functionality
|
||||
- Files: `remote_cli.py`, `file_upload.py`, `job.py`, `audio_worker.py`
|
||||
|
||||
### 2024-12-10: v0.71.22 - Batch 1 Complete (Lyrics Configuration)
|
||||
|
||||
**PR #8: Lyrics Configuration for Remote CLI**
|
||||
|
||||
- ✅ `--lyrics_artist` - Override artist name for lyrics search
|
||||
- ✅ `--lyrics_title` - Override title for lyrics search
|
||||
- ✅ `--lyrics_file` - Upload custom lyrics file (TXT, DOCX, RTF)
|
||||
- ✅ `--subtitle_offset_ms` - Adjust subtitle timing offset
|
||||
- Added 25+ unit tests for new functionality
|
||||
- Files: `remote_cli.py`, `file_upload.py`, `job.py`, `lyrics_worker.py`, `render_video_worker.py`
|
||||
|
||||
### 2024-12-10: v0.71.0 - Core Feature Parity Complete
|
||||
|
||||
**First successful end-to-end remote run!**
|
||||
@@ -400,14 +888,46 @@ DEFAULT_DISCORD_WEBHOOK_URL=https://discord.com/...
|
||||
2. ✅ Tag v0.71.0 release
|
||||
3. Update PyPI package
|
||||
|
||||
### Short-Term
|
||||
1. Monitor production usage
|
||||
2. Address any edge cases found
|
||||
3. Improve error messages and logging
|
||||
### Short-Term: Feature Parity (Priority Order)
|
||||
|
||||
### Long-Term (Shared Pipeline Refactor)
|
||||
1. Create new branch for shared pipeline architecture
|
||||
2. Extract stage interfaces
|
||||
3. Implement execution adapters
|
||||
4. Migrate stages incrementally
|
||||
5. Achieve true code sharing between local and remote
|
||||
1. ✅ **Lyrics Configuration** (Batch 1 - COMPLETE)
|
||||
- `--lyrics_artist`, `--lyrics_title`, `--lyrics_file`, `--subtitle_offset_ms`
|
||||
- PR #8, v0.71.22
|
||||
|
||||
2. ✅ **Audio Model Selection** (Batch 2 - COMPLETE)
|
||||
- `--clean_instrumental_model`, `--backing_vocals_models`, `--other_stems_models`
|
||||
- PR #9, v0.71.24
|
||||
|
||||
3. ✅ **Existing Instrumental** (Batch 3 - COMPLETE)
|
||||
- Added `--existing_instrumental` upload support
|
||||
- PR #10, v0.71.26
|
||||
|
||||
4. **YouTube URL Input** (Batch 4 - NEXT)
|
||||
- Accept YouTube URLs as input
|
||||
- Requires: yt-dlp in container
|
||||
- Enables: Processing niche live versions only on YouTube
|
||||
|
||||
### Medium-Term: Workflow Features
|
||||
|
||||
1. **Flacfetch Audio Search** (Batch 5)
|
||||
- Interactive audio source selection via API
|
||||
|
||||
2. **Two-Phase Workflow** (Batch 6)
|
||||
- `--prep-only` and `--finalise-only` modes
|
||||
|
||||
3. **Style Override** (Batch 7)
|
||||
- Quick style tweaks without editing JSON
|
||||
|
||||
4. **Background Video Support** (Batch 8)
|
||||
- Upload and use video backgrounds
|
||||
|
||||
5. **Edit Lyrics Mode** (Batch 9)
|
||||
- Allow fixing lyrics on completed jobs
|
||||
|
||||
### Long-Term (Scalable Architecture)
|
||||
|
||||
See **[SCALABLE-ARCHITECTURE-PLAN.md](./SCALABLE-ARCHITECTURE-PLAN.md)** for the comprehensive plan covering:
|
||||
1. Cloud Tasks integration for horizontal scalability
|
||||
2. Cloud Run Jobs for long-running video encoding
|
||||
3. Shared pipeline architecture for code reuse
|
||||
4. Implementation roadmap and migration strategy
|
||||
@@ -0,0 +1,823 @@
|
||||
# Scalable Architecture Plan
|
||||
|
||||
**Last Updated:** 2025-12-11
|
||||
**Status:** Analysis Complete - Ready for Implementation
|
||||
|
||||
This document outlines the plan for evolving karaoke-gen's cloud backend to support **100+ concurrent jobs** with no performance degradation, while also achieving a clean, maintainable codebase with proper separation of concerns.
|
||||
|
||||
---
|
||||
|
||||
## Executive Summary
|
||||
|
||||
After thorough codebase analysis, I recommend a **phased approach** that prioritizes infrastructure scalability first (Cloud Tasks), followed by incremental adoption of the existing pipeline architecture for code sharing.
|
||||
|
||||
### Key Findings
|
||||
|
||||
1. **The infrastructure problem is well-isolated** - WorkerService already abstracts worker triggering behind HTTP calls
|
||||
2. **Pipeline architecture exists but is underutilized** - `karaoke_gen/pipeline/` has well-designed stages and executors, but backend workers don't use them
|
||||
3. **RemoteExecutor is a placeholder** - It submits jobs to the backend but doesn't execute stages remotely
|
||||
4. **Workers are self-contained** - Each worker only needs `job_id` and fetches everything from Firestore/GCS
|
||||
|
||||
### Recommended Priority
|
||||
|
||||
1. **Phase 1 (HIGH): Cloud Tasks Integration** - Solves the scalability problem immediately
|
||||
2. **Phase 2 (MEDIUM): Cloud Run Jobs for Video** - Removes 30-min timeout constraint
|
||||
3. **Phase 3 (LOW): Pipeline Unification** - Nice-to-have for code maintainability
|
||||
|
||||
---
|
||||
|
||||
## Goals
|
||||
|
||||
### Infrastructure Goals
|
||||
1. **Horizontal scalability** - Each job gets dedicated resources (no resource contention)
|
||||
2. **Maximum processing speed** - Same speed for 1 job or 100 concurrent jobs
|
||||
3. **Reliability** - Automatic retries, no lost jobs from container restarts
|
||||
4. **Cost efficiency** - Scale to zero when idle, pay only for active processing
|
||||
|
||||
### Software Architecture Goals
|
||||
1. **SOLID principles** - Single responsibility, clean interfaces
|
||||
2. **DRY** - Share code between local CLI and remote backend
|
||||
3. **Testability** - Each component independently testable
|
||||
4. **Maintainability** - Bug fixes apply everywhere, easy to add features
|
||||
|
||||
---
|
||||
|
||||
## Current Architecture Analysis
|
||||
|
||||
### What Works Well
|
||||
|
||||
```
|
||||
┌─────────────────────────────────────────────────────────────────────────────┐
|
||||
│ CURRENT ARCHITECTURE (v0.71.x) │
|
||||
├─────────────────────────────────────────────────────────────────────────────┤
|
||||
│ │
|
||||
│ ✅ Heavy work already offloaded to external services: │
|
||||
│ │
|
||||
│ Audio Separation ──────────────────▶ Modal API (GPU, ~5-8 min) │
|
||||
│ Lyrics Transcription ──────────────▶ AudioShake API (~2-3 min) │
|
||||
│ │
|
||||
│ ✅ State persistence in Firestore (survives restarts) │
|
||||
│ ✅ Files in GCS (no local storage dependencies) │
|
||||
│ ✅ Internal HTTP endpoints exist for worker triggers │
|
||||
│ ✅ Workers are self-contained (only need job_id) │
|
||||
│ ✅ WorkerService abstracts HTTP calls │
|
||||
│ ✅ Pipeline stages exist in karaoke_gen/pipeline/ │
|
||||
│ ✅ LocalExecutor for CLI, RemoteExecutor placeholder ready │
|
||||
│ │
|
||||
└─────────────────────────────────────────────────────────────────────────────┘
|
||||
```
|
||||
|
||||
### Current Code Structure
|
||||
|
||||
```
|
||||
WorkerService (backend/services/worker_service.py)
|
||||
│
|
||||
└── trigger_worker(worker_type, job_id)
|
||||
│
|
||||
└── HTTP POST /api/internal/workers/{worker_type}
|
||||
│
|
||||
└── BackgroundTasks.add_task(process_xxx, job_id) ⚠️ PROBLEM
|
||||
```
|
||||
|
||||
The **single point of change** is `WorkerService.trigger_worker()`. Currently it:
|
||||
1. Makes HTTP POST to internal endpoint
|
||||
2. Internal endpoint adds to FastAPI BackgroundTasks
|
||||
3. BackgroundTask runs in same container (problem!)
|
||||
|
||||
### Current Limitations
|
||||
|
||||
```
|
||||
┌─────────────────────────────────────────────────────────────────────────────┐
|
||||
│ CURRENT FLOW (PROBLEMATIC) │
|
||||
├─────────────────────────────────────────────────────────────────────────────┤
|
||||
│ │
|
||||
│ POST /api/jobs/upload │
|
||||
│ │ │
|
||||
│ ├──▶ Save job to Firestore ✅ │
|
||||
│ │ │
|
||||
│ └──▶ BackgroundTasks.add_task(trigger_workers) │
|
||||
│ │ │
|
||||
│ └──▶ HTTP POST /api/internal/workers/audio │
|
||||
│ │ │
|
||||
│ └──▶ BackgroundTasks.add_task(process_audio) ⚠️ │
|
||||
│ │ │
|
||||
│ └── Runs IN THE SAME CONTAINER │
|
||||
│ as the API request handler │
|
||||
│ │
|
||||
│ PROBLEMS: │
|
||||
│ ⚠️ Container may be killed before background tasks complete │
|
||||
│ ⚠️ No guaranteed delivery - if container crashes, job is lost │
|
||||
│ ⚠️ Multiple concurrent jobs share CPU/memory (resource contention) │
|
||||
│ ⚠️ Long-running video encoding (15-20 min) vulnerable to timeouts │
|
||||
│ ⚠️ FFmpeg renders compete for resources when jobs run in parallel │
|
||||
│ │
|
||||
└─────────────────────────────────────────────────────────────────────────────┘
|
||||
```
|
||||
|
||||
### Existing Pipeline Architecture (Underutilized)
|
||||
|
||||
The codebase already has a well-designed pipeline system:
|
||||
|
||||
```
|
||||
karaoke_gen/pipeline/
|
||||
├── base.py # PipelineStage, StageResult, PipelineExecutor abstracts
|
||||
├── context.py # PipelineContext for shared state
|
||||
├── executors/
|
||||
│ ├── local.py # LocalExecutor - runs stages in-process (for CLI)
|
||||
│ └── remote.py # RemoteExecutor - placeholder for backend integration
|
||||
└── stages/
|
||||
├── separation.py # SeparationStage - wraps AudioProcessor
|
||||
├── transcription.py # TranscriptionStage - wraps LyricsProcessor
|
||||
├── screens.py # ScreensStage - title/end screens
|
||||
├── render.py # RenderStage - lyrics video
|
||||
└── finalize.py # FinalizeStage - wraps KaraokeFinalise
|
||||
```
|
||||
|
||||
**However**: Backend workers (`backend/workers/*.py`) don't use these stages - they directly call the underlying processors. This creates two parallel implementations.
|
||||
|
||||
---
|
||||
|
||||
## Recommended Architecture
|
||||
|
||||
### Decision: Cloud Tasks First
|
||||
|
||||
**Why Cloud Tasks over Pub/Sub or Workflows:**
|
||||
|
||||
| Feature | Cloud Tasks | Pub/Sub | Workflows |
|
||||
|---------|-------------|---------|-----------|
|
||||
| HTTP delivery | ✅ Native | Requires push subscription | N/A |
|
||||
| Retry control | ✅ Built-in | Manual | Built-in |
|
||||
| Rate limiting | ✅ Built-in | Manual | N/A |
|
||||
| Timeout (max) | 30 min | 10 min ack deadline | 1 year |
|
||||
| Complexity | Low | Medium | High |
|
||||
| Cost | Very low | Low | Medium |
|
||||
| Existing code changes | Minimal | Medium | High |
|
||||
|
||||
**Cloud Tasks is the right choice because:**
|
||||
1. Workers already have HTTP endpoints - Cloud Tasks delivers HTTP requests
|
||||
2. Need rate limiting to protect Modal/AudioShake APIs
|
||||
3. Built-in retries with configurable backoff
|
||||
4. Minimal code changes (update WorkerService only)
|
||||
|
||||
### Target Architecture
|
||||
|
||||
```
|
||||
┌─────────────────────────────────────────────────────────────────────────────┐
|
||||
│ CLOUD TASKS + CLOUD RUN JOBS ARCHITECTURE │
|
||||
├─────────────────────────────────────────────────────────────────────────────┤
|
||||
│ │
|
||||
│ POST /api/jobs/upload │
|
||||
│ │ │
|
||||
│ ├──▶ Save job to Firestore │
|
||||
│ │ │
|
||||
│ └──▶ WorkerService.trigger_worker() │
|
||||
│ │ │
|
||||
│ │ if ENABLE_CLOUD_TASKS: │
|
||||
│ │ └──▶ Enqueue Cloud Task │
|
||||
│ │ │ │
|
||||
│ │ ├──▶ audio-worker-queue │
|
||||
│ │ │ {job_id, target: /internal/workers/audio} │
|
||||
│ │ │ │
|
||||
│ │ └──▶ lyrics-worker-queue │
|
||||
│ │ {job_id, target: /internal/workers/lyrics}│
|
||||
│ │ │
|
||||
│ │ else: (development/testing) │
|
||||
│ │ └──▶ HTTP POST directly (existing behavior) │
|
||||
│ │ │
|
||||
│ ┌──────────▼────────────────────────────────────────────────────────┐ │
|
||||
│ │ Cloud Tasks Queue │ │
|
||||
│ │ │ │
|
||||
│ │ • Guaranteed delivery (retries on failure) │ │
|
||||
│ │ • Rate limiting / concurrency control │ │
|
||||
│ │ • Task deduplication │ │
|
||||
│ │ • Configurable timeouts (up to 30 min per task) │ │
|
||||
│ └─────────────────────────────────┬───────────────────────────────────┘ │
|
||||
│ │ │
|
||||
│ ▼ │
|
||||
│ ┌─────────────────────────────────────────────────────────────────────┐ │
|
||||
│ │ Cloud Run (Auto-Scaled Worker Instances) │ │
|
||||
│ │ │ │
|
||||
│ │ Instance 1 Instance 2 Instance 3 ... │ │
|
||||
│ │ ┌──────────┐ ┌──────────┐ ┌──────────┐ │ │
|
||||
│ │ │ Job A │ │ Job B │ │ Job C │ │ │
|
||||
│ │ │ Audio │ │ Audio │ │ Lyrics │ │ │
|
||||
│ │ └──────────┘ └──────────┘ └──────────┘ │ │
|
||||
│ │ │ │
|
||||
│ │ Each task gets a DEDICATED container instance │ │
|
||||
│ │ • Full CPU/memory for that task alone │ │
|
||||
│ │ • No resource contention with other jobs │ │
|
||||
│ │ • Container lifecycle tied to task completion │ │
|
||||
│ └─────────────────────────────────────────────────────────────────────┘ │
|
||||
│ │
|
||||
│ For video encoding (>30 min timeout): │
|
||||
│ ┌─────────────────────────────────────────────────────────────────────┐ │
|
||||
│ │ Cloud Run Jobs (Long-Running) │ │
|
||||
│ │ │ │
|
||||
│ │ • Up to 24 hour timeout │ │
|
||||
│ │ • Separate from API service │ │
|
||||
│ │ • Higher CPU/memory (4 vCPU, 8GB) │ │
|
||||
│ │ • Triggered via gcloud or API │ │
|
||||
│ └─────────────────────────────────────────────────────────────────────┘ │
|
||||
│ │
|
||||
└─────────────────────────────────────────────────────────────────────────────┘
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Implementation Plan
|
||||
|
||||
### Phase 1: Cloud Tasks Integration (HIGH Priority)
|
||||
|
||||
**Goal:** Enable 100+ concurrent jobs with no performance degradation
|
||||
|
||||
**Effort:** 2-3 days | **Risk:** Low (feature-flagged, easy rollback)
|
||||
|
||||
#### Step 1.1: Add Cloud Tasks Infrastructure (Pulumi)
|
||||
|
||||
```python
|
||||
# infrastructure/__main__.py additions
|
||||
|
||||
from pulumi_gcp import cloudtasks
|
||||
|
||||
# Create Cloud Tasks queues
|
||||
audio_worker_queue = cloudtasks.Queue(
|
||||
"audio-worker-queue",
|
||||
name="audio-worker-queue",
|
||||
location="us-central1",
|
||||
rate_limits=cloudtasks.QueueRateLimitsArgs(
|
||||
max_dispatches_per_second=10, # Protect Modal API
|
||||
max_concurrent_dispatches=50, # Max parallel audio workers
|
||||
),
|
||||
retry_config=cloudtasks.QueueRetryConfigArgs(
|
||||
max_attempts=3,
|
||||
min_backoff="10s",
|
||||
max_backoff="300s",
|
||||
max_retry_duration="1800s", # 30 min total retry window
|
||||
),
|
||||
)
|
||||
|
||||
lyrics_worker_queue = cloudtasks.Queue(
|
||||
"lyrics-worker-queue",
|
||||
name="lyrics-worker-queue",
|
||||
location="us-central1",
|
||||
rate_limits=cloudtasks.QueueRateLimitsArgs(
|
||||
max_dispatches_per_second=10,
|
||||
max_concurrent_dispatches=50,
|
||||
),
|
||||
retry_config=cloudtasks.QueueRetryConfigArgs(
|
||||
max_attempts=3,
|
||||
min_backoff="10s",
|
||||
max_backoff="300s",
|
||||
),
|
||||
)
|
||||
|
||||
screens_worker_queue = cloudtasks.Queue(
|
||||
"screens-worker-queue",
|
||||
name="screens-worker-queue",
|
||||
location="us-central1",
|
||||
rate_limits=cloudtasks.QueueRateLimitsArgs(
|
||||
max_dispatches_per_second=50, # Fast, CPU-light
|
||||
max_concurrent_dispatches=100,
|
||||
),
|
||||
retry_config=cloudtasks.QueueRetryConfigArgs(
|
||||
max_attempts=3,
|
||||
min_backoff="5s",
|
||||
max_backoff="60s",
|
||||
),
|
||||
)
|
||||
|
||||
render_worker_queue = cloudtasks.Queue(
|
||||
"render-worker-queue",
|
||||
name="render-worker-queue",
|
||||
location="us-central1",
|
||||
rate_limits=cloudtasks.QueueRateLimitsArgs(
|
||||
max_dispatches_per_second=5, # CPU-intensive
|
||||
max_concurrent_dispatches=20,
|
||||
),
|
||||
retry_config=cloudtasks.QueueRetryConfigArgs(
|
||||
max_attempts=2,
|
||||
min_backoff="30s",
|
||||
max_backoff="300s",
|
||||
max_retry_duration="3600s", # 60 min for render
|
||||
),
|
||||
)
|
||||
|
||||
video_worker_queue = cloudtasks.Queue(
|
||||
"video-worker-queue",
|
||||
name="video-worker-queue",
|
||||
location="us-central1",
|
||||
rate_limits=cloudtasks.QueueRateLimitsArgs(
|
||||
max_dispatches_per_second=3, # Very CPU-intensive
|
||||
max_concurrent_dispatches=10,
|
||||
),
|
||||
retry_config=cloudtasks.QueueRetryConfigArgs(
|
||||
max_attempts=2,
|
||||
min_backoff="60s",
|
||||
max_backoff="600s",
|
||||
max_retry_duration="7200s", # 2 hour total (video is long)
|
||||
),
|
||||
)
|
||||
|
||||
# Grant Cloud Tasks permission to invoke Cloud Run
|
||||
cloud_tasks_invoker = gcp.projects.IAMMember(
|
||||
"cloud-tasks-invoker",
|
||||
project=project_id,
|
||||
role="roles/run.invoker",
|
||||
member=f"serviceAccount:service-{project.number}@gcp-sa-cloudtasks.iam.gserviceaccount.com",
|
||||
)
|
||||
|
||||
# Export queue names
|
||||
pulumi.export("audio_worker_queue", audio_worker_queue.name)
|
||||
pulumi.export("lyrics_worker_queue", lyrics_worker_queue.name)
|
||||
pulumi.export("screens_worker_queue", screens_worker_queue.name)
|
||||
pulumi.export("render_worker_queue", render_worker_queue.name)
|
||||
pulumi.export("video_worker_queue", video_worker_queue.name)
|
||||
```
|
||||
|
||||
#### Step 1.2: Update WorkerService for Cloud Tasks
|
||||
|
||||
```python
|
||||
# backend/services/worker_service.py - Updated
|
||||
|
||||
import logging
|
||||
import os
|
||||
import json
|
||||
from typing import Optional
|
||||
from google.cloud import tasks_v2
|
||||
from google.protobuf import timestamp_pb2
|
||||
import httpx
|
||||
|
||||
from backend.config import get_settings
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
# Queue mapping
|
||||
WORKER_QUEUES = {
|
||||
"audio": "audio-worker-queue",
|
||||
"lyrics": "lyrics-worker-queue",
|
||||
"screens": "screens-worker-queue",
|
||||
"render-video": "render-worker-queue",
|
||||
"video": "video-worker-queue",
|
||||
}
|
||||
|
||||
class WorkerService:
|
||||
"""
|
||||
Service for coordinating background workers.
|
||||
|
||||
Supports two modes:
|
||||
- Cloud Tasks (production): Guaranteed delivery, auto-scaling
|
||||
- Direct HTTP (development): Faster iteration, simpler debugging
|
||||
"""
|
||||
|
||||
def __init__(self):
|
||||
self.settings = get_settings()
|
||||
self._base_url = self._get_base_url()
|
||||
self._admin_token = self._get_admin_token()
|
||||
self._use_cloud_tasks = self._should_use_cloud_tasks()
|
||||
self._tasks_client = None
|
||||
|
||||
def _should_use_cloud_tasks(self) -> bool:
|
||||
"""Check if Cloud Tasks should be used."""
|
||||
# Feature flag with environment variable
|
||||
enable_flag = os.getenv("ENABLE_CLOUD_TASKS", "false").lower()
|
||||
return enable_flag in ("true", "1", "yes")
|
||||
|
||||
@property
|
||||
def tasks_client(self):
|
||||
"""Lazy-init Cloud Tasks client."""
|
||||
if self._tasks_client is None and self._use_cloud_tasks:
|
||||
self._tasks_client = tasks_v2.CloudTasksClient()
|
||||
return self._tasks_client
|
||||
|
||||
def _get_admin_token(self) -> Optional[str]:
|
||||
admin_tokens_str = self.settings.admin_tokens or ""
|
||||
tokens = [t.strip() for t in admin_tokens_str.split(",") if t.strip()]
|
||||
return tokens[0] if tokens else None
|
||||
|
||||
def _get_base_url(self) -> str:
|
||||
# Check for test environment override
|
||||
test_url = os.getenv("TEST_SERVER_URL")
|
||||
if test_url:
|
||||
return test_url
|
||||
|
||||
# Production: Cloud Run service URL
|
||||
service_url = os.getenv("CLOUD_RUN_SERVICE_URL")
|
||||
if service_url:
|
||||
return service_url
|
||||
|
||||
# Development: localhost
|
||||
port = os.getenv("PORT", "8000")
|
||||
return f"http://localhost:{port}"
|
||||
|
||||
async def trigger_worker(
|
||||
self,
|
||||
worker_type: str,
|
||||
job_id: str,
|
||||
timeout_seconds: int = 30
|
||||
) -> bool:
|
||||
"""
|
||||
Trigger a background worker.
|
||||
|
||||
In production (ENABLE_CLOUD_TASKS=true):
|
||||
- Enqueues task to Cloud Tasks queue
|
||||
- Cloud Tasks calls internal endpoint with retries
|
||||
|
||||
In development:
|
||||
- Direct HTTP call to internal endpoint
|
||||
- Faster iteration, but no retry guarantees
|
||||
"""
|
||||
if self._use_cloud_tasks:
|
||||
return await self._enqueue_cloud_task(worker_type, job_id)
|
||||
else:
|
||||
return await self._trigger_http(worker_type, job_id, timeout_seconds)
|
||||
|
||||
async def _enqueue_cloud_task(self, worker_type: str, job_id: str) -> bool:
|
||||
"""Enqueue task to Cloud Tasks for guaranteed delivery."""
|
||||
try:
|
||||
queue_name = WORKER_QUEUES.get(worker_type)
|
||||
if not queue_name:
|
||||
logger.error(f"Unknown worker type: {worker_type}")
|
||||
return False
|
||||
|
||||
project = self.settings.google_cloud_project
|
||||
location = "us-central1"
|
||||
|
||||
# Build queue path
|
||||
parent = self.tasks_client.queue_path(project, location, queue_name)
|
||||
|
||||
# Build task
|
||||
task = {
|
||||
"http_request": {
|
||||
"http_method": tasks_v2.HttpMethod.POST,
|
||||
"url": f"{self._base_url}/api/internal/workers/{worker_type}",
|
||||
"headers": {
|
||||
"Content-Type": "application/json",
|
||||
},
|
||||
"body": json.dumps({"job_id": job_id}).encode(),
|
||||
# Use OIDC token for Cloud Run authentication
|
||||
"oidc_token": {
|
||||
"service_account_email": f"karaoke-backend@{project}.iam.gserviceaccount.com",
|
||||
},
|
||||
},
|
||||
}
|
||||
|
||||
# Add admin auth header if available
|
||||
if self._admin_token:
|
||||
task["http_request"]["headers"]["Authorization"] = f"Bearer {self._admin_token}"
|
||||
|
||||
# Create task
|
||||
response = self.tasks_client.create_task(parent=parent, task=task)
|
||||
logger.info(f"Created Cloud Task for {worker_type} worker, job {job_id}: {response.name}")
|
||||
return True
|
||||
|
||||
except Exception as e:
|
||||
logger.error(f"Failed to enqueue Cloud Task for {worker_type}/{job_id}: {e}", exc_info=True)
|
||||
return False
|
||||
|
||||
async def _trigger_http(
|
||||
self,
|
||||
worker_type: str,
|
||||
job_id: str,
|
||||
timeout_seconds: int = 30
|
||||
) -> bool:
|
||||
"""Direct HTTP trigger (for development)."""
|
||||
try:
|
||||
headers = {}
|
||||
if self._admin_token:
|
||||
headers["Authorization"] = f"Bearer {self._admin_token}"
|
||||
|
||||
async with httpx.AsyncClient(timeout=timeout_seconds) as client:
|
||||
url = f"{self._base_url}/api/internal/workers/{worker_type}"
|
||||
response = await client.post(url, json={"job_id": job_id}, headers=headers)
|
||||
|
||||
if response.status_code == 200:
|
||||
logger.info(f"Triggered {worker_type} worker for job {job_id}")
|
||||
return True
|
||||
else:
|
||||
logger.error(f"Failed to trigger {worker_type}: HTTP {response.status_code}")
|
||||
return False
|
||||
|
||||
except Exception as e:
|
||||
logger.error(f"Error triggering {worker_type} worker: {e}", exc_info=True)
|
||||
return False
|
||||
|
||||
# Convenience methods (unchanged)
|
||||
async def trigger_audio_worker(self, job_id: str) -> bool:
|
||||
return await self.trigger_worker("audio", job_id)
|
||||
|
||||
async def trigger_lyrics_worker(self, job_id: str) -> bool:
|
||||
return await self.trigger_worker("lyrics", job_id)
|
||||
|
||||
async def trigger_screens_worker(self, job_id: str) -> bool:
|
||||
return await self.trigger_worker("screens", job_id)
|
||||
|
||||
async def trigger_video_worker(self, job_id: str) -> bool:
|
||||
return await self.trigger_worker("video", job_id)
|
||||
|
||||
async def trigger_render_video_worker(self, job_id: str) -> bool:
|
||||
return await self.trigger_worker("render-video", job_id)
|
||||
```
|
||||
|
||||
#### Step 1.3: Update Internal Endpoints for Cloud Tasks
|
||||
|
||||
The internal endpoints need minor updates to handle Cloud Tasks requests:
|
||||
|
||||
```python
|
||||
# backend/api/routes/internal.py - Updates
|
||||
|
||||
# Add idempotency check to prevent duplicate processing
|
||||
@router.post("/workers/audio", response_model=WorkerResponse)
|
||||
async def trigger_audio_worker(
|
||||
request: WorkerRequest,
|
||||
background_tasks: BackgroundTasks,
|
||||
auth_data: Tuple[str, UserType, int] = Depends(require_admin)
|
||||
):
|
||||
"""Trigger audio separation worker for a job."""
|
||||
job_id = request.job_id
|
||||
|
||||
# Idempotency: Check if already processing
|
||||
job_manager = JobManager()
|
||||
job = job_manager.get_job(job_id)
|
||||
|
||||
if job and job.state_data.get('audio_progress', {}).get('stage') == 'running':
|
||||
logger.info(f"Audio worker already running for job {job_id}, skipping")
|
||||
return WorkerResponse(
|
||||
status="already_running",
|
||||
job_id=job_id,
|
||||
message="Audio worker already in progress"
|
||||
)
|
||||
|
||||
# Mark as running before starting (for idempotency)
|
||||
job_manager.update_state_data(job_id, 'audio_progress', {'stage': 'running'})
|
||||
|
||||
logger.info(f"Triggering audio worker for job {job_id}")
|
||||
background_tasks.add_task(process_audio_separation, job_id)
|
||||
|
||||
return WorkerResponse(
|
||||
status="started",
|
||||
job_id=job_id,
|
||||
message="Audio separation worker started"
|
||||
)
|
||||
```
|
||||
|
||||
#### Step 1.4: Cloud Run Service Account Permissions
|
||||
|
||||
```python
|
||||
# infrastructure/__main__.py additions
|
||||
|
||||
# Grant Cloud Tasks admin permission to the service account
|
||||
# (needed to create tasks)
|
||||
cloud_tasks_admin = gcp.projects.IAMMember(
|
||||
"karaoke-backend-cloudtasks-admin",
|
||||
project=project_id,
|
||||
role="roles/cloudtasks.enqueuer",
|
||||
member=service_account.email.apply(lambda email: f"serviceAccount:{email}"),
|
||||
)
|
||||
```
|
||||
|
||||
### Phase 2: Cloud Run Jobs for Video Worker (MEDIUM Priority)
|
||||
|
||||
**Goal:** Remove 30-minute timeout constraint for video encoding
|
||||
|
||||
**Effort:** 1-2 days | **Risk:** Low
|
||||
|
||||
The video worker (final encoding) can take 15-20 minutes, sometimes longer. Cloud Tasks has a 30-minute max timeout. For safety, we use Cloud Run Jobs.
|
||||
|
||||
#### Step 2.1: Create Cloud Run Job
|
||||
|
||||
```yaml
|
||||
# cloudbuild-video-job.yaml
|
||||
steps:
|
||||
- name: 'gcr.io/cloud-builders/gcloud'
|
||||
args:
|
||||
- 'run'
|
||||
- 'jobs'
|
||||
- 'create'
|
||||
- 'video-encoding-job'
|
||||
- '--image=${_IMAGE}'
|
||||
- '--region=us-central1'
|
||||
- '--cpu=4'
|
||||
- '--memory=8Gi'
|
||||
- '--max-retries=2'
|
||||
- '--task-timeout=3600' # 1 hour max
|
||||
- '--service-account=karaoke-backend@${PROJECT_ID}.iam.gserviceaccount.com'
|
||||
- '--set-env-vars=GOOGLE_CLOUD_PROJECT=${PROJECT_ID}'
|
||||
```
|
||||
|
||||
#### Step 2.2: Update Video Worker Trigger
|
||||
|
||||
```python
|
||||
# backend/services/worker_service.py - Addition
|
||||
|
||||
async def trigger_video_worker_job(self, job_id: str) -> bool:
|
||||
"""Trigger video worker as Cloud Run Job (for long-running encoding)."""
|
||||
if not self._use_cloud_tasks:
|
||||
# Development: use HTTP trigger
|
||||
return await self._trigger_http("video", job_id)
|
||||
|
||||
try:
|
||||
from google.cloud import run_v2
|
||||
|
||||
client = run_v2.JobsClient()
|
||||
project = self.settings.google_cloud_project
|
||||
location = "us-central1"
|
||||
|
||||
job_name = f"projects/{project}/locations/{location}/jobs/video-encoding-job"
|
||||
|
||||
# Override container args with job_id
|
||||
request = run_v2.RunJobRequest(
|
||||
name=job_name,
|
||||
overrides=run_v2.RunJobRequest.Overrides(
|
||||
container_overrides=[
|
||||
run_v2.RunJobRequest.Overrides.ContainerOverride(
|
||||
args=["--job-id", job_id],
|
||||
)
|
||||
]
|
||||
)
|
||||
)
|
||||
|
||||
operation = client.run_job(request=request)
|
||||
logger.info(f"Started Cloud Run Job for video encoding: {job_id}")
|
||||
return True
|
||||
|
||||
except Exception as e:
|
||||
logger.error(f"Failed to start video encoding job: {e}", exc_info=True)
|
||||
return False
|
||||
```
|
||||
|
||||
### Phase 3: Pipeline Unification (LOW Priority - Future)
|
||||
|
||||
**Goal:** Share code between CLI and backend workers
|
||||
|
||||
**Effort:** 1-2 weeks | **Risk:** Medium (requires careful refactoring)
|
||||
|
||||
This phase can be deferred - the infrastructure scaling (Phase 1-2) solves the immediate problem.
|
||||
|
||||
**When to consider this:**
|
||||
- When adding new features that need to work in both CLI and backend
|
||||
- When a bug fix needs to be applied in multiple places
|
||||
- When the divergence between workers and pipeline stages causes maintenance burden
|
||||
|
||||
**Approach:**
|
||||
1. Make backend workers call pipeline stages instead of processors directly
|
||||
2. Have RemoteExecutor use Cloud Tasks to trigger stage execution
|
||||
3. Both CLI and backend use the same pipeline stages
|
||||
|
||||
---
|
||||
|
||||
## Resource Configuration
|
||||
|
||||
### Cloud Tasks Queues
|
||||
|
||||
| Queue | Max Concurrent | Rate Limit | Timeout | Retries | Notes |
|
||||
|-------|---------------|------------|---------|---------|-------|
|
||||
| audio-worker | 50 | 10/sec | 15 min | 3 | Mostly I/O (Modal API) |
|
||||
| lyrics-worker | 50 | 10/sec | 10 min | 3 | Mostly I/O (AudioShake) |
|
||||
| screens-worker | 100 | 50/sec | 2 min | 3 | FFmpeg (quick) |
|
||||
| render-worker | 20 | 5/sec | 30 min | 2 | LyricsTranscriber + FFmpeg |
|
||||
| video-worker | 10 | 3/sec | 30 min | 2 | Heavy FFmpeg (or use Jobs) |
|
||||
|
||||
### Cloud Run Instance Scaling
|
||||
|
||||
| Worker | CPU | Memory | Max Instances | Notes |
|
||||
|--------|-----|--------|---------------|-------|
|
||||
| audio | 2 | 2Gi | 50 | Mostly waiting on Modal API |
|
||||
| lyrics | 2 | 2Gi | 50 | Mostly waiting on AudioShake |
|
||||
| screens | 2 | 2Gi | 100 | Quick FFmpeg |
|
||||
| render | 4 | 4Gi | 20 | CPU-intensive |
|
||||
| video (Job) | 4 | 8Gi | 10 | Very CPU-intensive |
|
||||
|
||||
### Cost Estimate (100 concurrent jobs/day)
|
||||
|
||||
```
|
||||
Cloud Tasks: ~$0.40/million tasks ≈ $0.01/day
|
||||
Cloud Run: ~$0.00002/vCPU-sec ≈ $5-10/day (100 jobs)
|
||||
Cloud Run Jobs: ~$0.00002/vCPU-sec ≈ $2-5/day (100 jobs)
|
||||
Modal (GPU): ~$2.50/hour ≈ $20-40/day (100 jobs)
|
||||
AudioShake: Usage-based ≈ $10-20/day (100 jobs)
|
||||
|
||||
Total: ~$40-80/day for 100 concurrent jobs
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Migration Strategy
|
||||
|
||||
### Zero-Downtime Migration
|
||||
|
||||
```
|
||||
Week 1: Deploy Cloud Tasks infrastructure (Pulumi)
|
||||
└── Queues exist but aren't used yet (ENABLE_CLOUD_TASKS=false)
|
||||
|
||||
Week 2: Test in staging
|
||||
└── Deploy to staging with ENABLE_CLOUD_TASKS=true
|
||||
└── Run 10-20 test jobs concurrently
|
||||
└── Monitor for issues
|
||||
|
||||
Week 3: Enable in production
|
||||
└── Set ENABLE_CLOUD_TASKS=true
|
||||
└── Monitor metrics and logs
|
||||
└── Have rollback ready
|
||||
|
||||
Week 4: Cleanup
|
||||
└── Remove feature flag after stable week
|
||||
└── Update documentation
|
||||
```
|
||||
|
||||
### Rollback Plan
|
||||
|
||||
If issues occur after enabling Cloud Tasks:
|
||||
1. Set `ENABLE_CLOUD_TASKS=false` environment variable
|
||||
2. Redeploy (takes ~2 minutes)
|
||||
3. New jobs use direct HTTP (old behavior)
|
||||
4. In-flight Cloud Tasks jobs continue to completion
|
||||
5. No data loss - Firestore has job state
|
||||
|
||||
### Testing Strategy
|
||||
|
||||
```bash
|
||||
# Run unit tests (no Cloud Tasks)
|
||||
pytest backend/tests/ -v
|
||||
|
||||
# Run integration tests with Cloud Tasks (staging)
|
||||
ENABLE_CLOUD_TASKS=true pytest backend/tests/integration/ -v
|
||||
|
||||
# Load test (staging)
|
||||
# Submit 50 concurrent jobs, verify all complete
|
||||
for i in {1..50}; do
|
||||
curl -X POST "https://staging.api.nomadkaraoke.com/api/jobs/upload" \
|
||||
-F "file=@test.flac" -F "artist=Test" -F "title=Job$i" &
|
||||
done
|
||||
wait
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Success Criteria
|
||||
|
||||
### Phase 1 (Cloud Tasks)
|
||||
- [ ] 50 concurrent jobs complete without failures
|
||||
- [ ] Processing time per job unchanged (same as 1 job)
|
||||
- [ ] No resource contention (CPU/memory per job consistent)
|
||||
- [ ] Automatic retry on transient failures works
|
||||
- [ ] Feature flag rollback works
|
||||
|
||||
### Phase 2 (Cloud Run Jobs)
|
||||
- [ ] Video encoding >30 min completes successfully
|
||||
- [ ] Jobs triggered correctly from WorkerService
|
||||
- [ ] Progress updates visible in CLI
|
||||
|
||||
### Phase 3 (Pipeline Unification)
|
||||
- [ ] Single stage implementation used by both CLIs
|
||||
- [ ] Bug fixes apply to both local and remote execution
|
||||
- [ ] Each stage independently testable
|
||||
- [ ] 80%+ code coverage on pipeline module
|
||||
|
||||
---
|
||||
|
||||
## Files to Modify
|
||||
|
||||
### Phase 1
|
||||
|
||||
| File | Changes |
|
||||
|------|---------|
|
||||
| `infrastructure/__main__.py` | Add Cloud Tasks queues, IAM permissions |
|
||||
| `backend/services/worker_service.py` | Add Cloud Tasks client, feature flag |
|
||||
| `backend/api/routes/internal.py` | Add idempotency checks |
|
||||
| `backend/config.py` | Add ENABLE_CLOUD_TASKS setting |
|
||||
| `backend/requirements.txt` | Add google-cloud-tasks |
|
||||
|
||||
### Phase 2
|
||||
|
||||
| File | Changes |
|
||||
|------|---------|
|
||||
| `cloudbuild-video-job.yaml` | New file for Cloud Run Job |
|
||||
| `backend/services/worker_service.py` | Add Cloud Run Jobs trigger |
|
||||
| `backend/workers/video_worker.py` | CLI entry point for Jobs |
|
||||
|
||||
---
|
||||
|
||||
## Related Documents
|
||||
|
||||
- [ARCHITECTURE.md](../01-reference/ARCHITECTURE.md) - Current system architecture
|
||||
- [BACKEND-FEATURE-PARITY-PLAN.md](./BACKEND-FEATURE-PARITY-PLAN.md) - Feature parity roadmap
|
||||
- [WORKER-IMPLEMENTATION-PLAN.md](./WORKER-IMPLEMENTATION-PLAN.md) - Worker details
|
||||
|
||||
---
|
||||
|
||||
## Changelog
|
||||
|
||||
### 2025-12-11: Analysis Complete - Updated Plan
|
||||
|
||||
- **Analyzed codebase thoroughly** - found existing pipeline architecture underutilized
|
||||
- **Confirmed Cloud Tasks as best approach** - minimal code changes, single point of change
|
||||
- **Identified WorkerService as sole modification point** - clean abstraction exists
|
||||
- **Decided against immediate pipeline unification** - infrastructure scaling is the priority
|
||||
- **Added detailed implementation code** - ready to start Phase 1
|
||||
- **Created phased approach** - Phase 1 (Cloud Tasks) → Phase 2 (Jobs) → Phase 3 (Pipeline)
|
||||
|
||||
### 2025-12-11: Initial Plan
|
||||
- Documented current architecture limitations
|
||||
- Proposed Cloud Tasks + Cloud Run Jobs hybrid
|
||||
- Outlined shared pipeline architecture
|
||||
- Created implementation roadmap
|
||||
+182
-1
@@ -2,10 +2,20 @@
|
||||
Pulumi infrastructure for Karaoke Generator backend on Google Cloud Platform.
|
||||
|
||||
This replaces the manual setup scripts with proper Infrastructure as Code.
|
||||
|
||||
Infrastructure includes:
|
||||
- Firestore database for job state
|
||||
- Cloud Storage bucket for files
|
||||
- Artifact Registry for Docker images
|
||||
- Service accounts and IAM
|
||||
- GitHub Actions Workload Identity Federation
|
||||
- Cloud Tasks queues for worker coordination (Phase 1 scalability)
|
||||
- Secret Manager secrets
|
||||
- Cloud Run domain mapping
|
||||
"""
|
||||
import pulumi
|
||||
import pulumi_gcp as gcp
|
||||
from pulumi_gcp import storage, firestore, secretmanager, artifactregistry, cloudrun, serviceaccount
|
||||
from pulumi_gcp import storage, firestore, secretmanager, artifactregistry, cloudrun, serviceaccount, cloudtasks
|
||||
|
||||
# Get the current GCP project
|
||||
project = gcp.organizations.get_project()
|
||||
@@ -227,6 +237,167 @@ github_actions_wif_binding = gcp.serviceaccount.IAMBinding(
|
||||
)],
|
||||
)
|
||||
|
||||
# ==================== Cloud Tasks Queues (Phase 1 Scalability) ====================
|
||||
# These queues provide guaranteed delivery and horizontal scaling for worker tasks.
|
||||
# Workers are triggered via Cloud Tasks instead of BackgroundTasks, ensuring:
|
||||
# - Tasks survive container restarts
|
||||
# - Each task gets dedicated resources
|
||||
# - Automatic retries on failure
|
||||
# - Rate limiting to protect external APIs
|
||||
|
||||
# Audio worker queue - calls Modal API for audio separation
|
||||
audio_worker_queue = cloudtasks.Queue(
|
||||
"audio-worker-queue",
|
||||
name="audio-worker-queue",
|
||||
location="us-central1",
|
||||
rate_limits=cloudtasks.QueueRateLimitsArgs(
|
||||
max_dispatches_per_second=10, # Protect Modal API
|
||||
max_concurrent_dispatches=50, # Max parallel audio workers
|
||||
),
|
||||
retry_config=cloudtasks.QueueRetryConfigArgs(
|
||||
max_attempts=3,
|
||||
min_backoff="10s",
|
||||
max_backoff="300s",
|
||||
max_retry_duration="1800s", # 30 min total retry window
|
||||
),
|
||||
)
|
||||
|
||||
# Lyrics worker queue - calls AudioShake API for transcription
|
||||
lyrics_worker_queue = cloudtasks.Queue(
|
||||
"lyrics-worker-queue",
|
||||
name="lyrics-worker-queue",
|
||||
location="us-central1",
|
||||
rate_limits=cloudtasks.QueueRateLimitsArgs(
|
||||
max_dispatches_per_second=10,
|
||||
max_concurrent_dispatches=50,
|
||||
),
|
||||
retry_config=cloudtasks.QueueRetryConfigArgs(
|
||||
max_attempts=3,
|
||||
min_backoff="10s",
|
||||
max_backoff="300s",
|
||||
max_retry_duration="1200s", # 20 min total retry window
|
||||
),
|
||||
)
|
||||
|
||||
# Screens worker queue - generates title/end screens (fast, CPU-light)
|
||||
screens_worker_queue = cloudtasks.Queue(
|
||||
"screens-worker-queue",
|
||||
name="screens-worker-queue",
|
||||
location="us-central1",
|
||||
rate_limits=cloudtasks.QueueRateLimitsArgs(
|
||||
max_dispatches_per_second=50, # Fast operations
|
||||
max_concurrent_dispatches=100,
|
||||
),
|
||||
retry_config=cloudtasks.QueueRetryConfigArgs(
|
||||
max_attempts=3,
|
||||
min_backoff="5s",
|
||||
max_backoff="60s",
|
||||
max_retry_duration="300s", # 5 min total retry window
|
||||
),
|
||||
)
|
||||
|
||||
# Render worker queue - LyricsTranscriber + FFmpeg (CPU-intensive)
|
||||
render_worker_queue = cloudtasks.Queue(
|
||||
"render-worker-queue",
|
||||
name="render-worker-queue",
|
||||
location="us-central1",
|
||||
rate_limits=cloudtasks.QueueRateLimitsArgs(
|
||||
max_dispatches_per_second=5, # CPU-intensive
|
||||
max_concurrent_dispatches=20,
|
||||
),
|
||||
retry_config=cloudtasks.QueueRetryConfigArgs(
|
||||
max_attempts=2,
|
||||
min_backoff="30s",
|
||||
max_backoff="300s",
|
||||
max_retry_duration="3600s", # 60 min for render
|
||||
),
|
||||
)
|
||||
|
||||
# Video worker queue - final encoding (very CPU-intensive, longest running)
|
||||
# Note: For tasks >30 min, consider Cloud Run Jobs instead
|
||||
video_worker_queue = cloudtasks.Queue(
|
||||
"video-worker-queue",
|
||||
name="video-worker-queue",
|
||||
location="us-central1",
|
||||
rate_limits=cloudtasks.QueueRateLimitsArgs(
|
||||
max_dispatches_per_second=3, # Very CPU-intensive
|
||||
max_concurrent_dispatches=10,
|
||||
),
|
||||
retry_config=cloudtasks.QueueRetryConfigArgs(
|
||||
max_attempts=2,
|
||||
min_backoff="60s",
|
||||
max_backoff="600s",
|
||||
max_retry_duration="7200s", # 2 hour total (video is long)
|
||||
),
|
||||
)
|
||||
|
||||
# Grant Cloud Tasks permission to invoke Cloud Run service
|
||||
# Cloud Tasks service agent needs to authenticate to Cloud Run
|
||||
cloud_tasks_invoker = gcp.projects.IAMMember(
|
||||
"cloud-tasks-invoker",
|
||||
project=project_id,
|
||||
role="roles/run.invoker",
|
||||
member=f"serviceAccount:service-{project.number}@gcp-sa-cloudtasks.iam.gserviceaccount.com",
|
||||
)
|
||||
|
||||
# Grant backend service account permission to enqueue Cloud Tasks
|
||||
cloud_tasks_enqueuer = gcp.projects.IAMMember(
|
||||
"karaoke-backend-cloudtasks-enqueuer",
|
||||
project=project_id,
|
||||
role="roles/cloudtasks.enqueuer",
|
||||
member=service_account.email.apply(lambda email: f"serviceAccount:{email}"),
|
||||
)
|
||||
|
||||
# ==================== Cloud Run Job for Video Encoding (Phase 2 Scalability) ====================
|
||||
# Cloud Run Jobs provide up to 24-hour execution time for long-running video encoding.
|
||||
# This is used when video encoding might exceed the 30-minute Cloud Tasks timeout.
|
||||
# Triggered programmatically via WorkerService when ENABLE_CLOUD_TASKS=true and
|
||||
# USE_CLOUD_RUN_JOBS_FOR_VIDEO=true
|
||||
|
||||
video_encoding_job = gcp.cloudrunv2.Job(
|
||||
"video-encoding-job",
|
||||
name="video-encoding-job",
|
||||
location="us-central1",
|
||||
template=gcp.cloudrunv2.JobTemplateArgs(
|
||||
template=gcp.cloudrunv2.JobTemplateTemplateArgs(
|
||||
# Use the same container as the backend service
|
||||
# The job runs with --job-id argument to process a specific job
|
||||
containers=[gcp.cloudrunv2.JobTemplateTemplateContainerArgs(
|
||||
image=f"us-central1-docker.pkg.dev/{project_id}/karaoke-repo/karaoke-backend:latest",
|
||||
args=["python", "-m", "backend.workers.video_worker"],
|
||||
resources=gcp.cloudrunv2.JobTemplateTemplateContainerResourcesArgs(
|
||||
limits={
|
||||
"cpu": "4", # 4 vCPUs for encoding
|
||||
"memory": "8Gi", # 8GB RAM
|
||||
},
|
||||
),
|
||||
envs=[
|
||||
gcp.cloudrunv2.JobTemplateTemplateContainerEnvArgs(
|
||||
name="GOOGLE_CLOUD_PROJECT",
|
||||
value=project_id,
|
||||
),
|
||||
gcp.cloudrunv2.JobTemplateTemplateContainerEnvArgs(
|
||||
name="GCS_BUCKET_NAME",
|
||||
value=bucket.name,
|
||||
),
|
||||
],
|
||||
)],
|
||||
service_account=service_account.email,
|
||||
timeout="3600s", # 1 hour max per task
|
||||
max_retries=2,
|
||||
),
|
||||
),
|
||||
)
|
||||
|
||||
# Grant backend service account permission to run Cloud Run Jobs
|
||||
cloud_run_jobs_developer = gcp.projects.IAMMember(
|
||||
"karaoke-backend-run-jobs-developer",
|
||||
project=project_id,
|
||||
role="roles/run.developer",
|
||||
member=service_account.email.apply(lambda email: f"serviceAccount:{email}"),
|
||||
)
|
||||
|
||||
# ==================== Secrets ====================
|
||||
# Create secrets (you'll need to add the actual secret values manually or via pulumi config)
|
||||
config = pulumi.Config()
|
||||
|
||||
@@ -286,6 +457,16 @@ pulumi.export("backend_default_url", "https://karaoke-backend-ipzqd2k4yq-uc.a.ru
|
||||
pulumi.export("github_actions_service_account", github_actions_sa.email)
|
||||
pulumi.export("workload_identity_provider", workload_identity_provider.name)
|
||||
|
||||
# Cloud Tasks queue exports (Phase 1 scalability)
|
||||
pulumi.export("audio_worker_queue", audio_worker_queue.name)
|
||||
pulumi.export("lyrics_worker_queue", lyrics_worker_queue.name)
|
||||
pulumi.export("screens_worker_queue", screens_worker_queue.name)
|
||||
pulumi.export("render_worker_queue", render_worker_queue.name)
|
||||
pulumi.export("video_worker_queue", video_worker_queue.name)
|
||||
|
||||
# Cloud Run Job export (Phase 2 scalability)
|
||||
pulumi.export("video_encoding_job", video_encoding_job.name)
|
||||
|
||||
# Export DNS configuration needed for Cloudflare
|
||||
# These are the records to add to your Cloudflare DNS
|
||||
domain_mapping.statuses.apply(
|
||||
|
||||
@@ -5,6 +5,12 @@ import shutil
|
||||
import tempfile
|
||||
from .utils import sanitize_filename
|
||||
|
||||
try:
|
||||
import yt_dlp
|
||||
YT_DLP_AVAILABLE = True
|
||||
except ImportError:
|
||||
YT_DLP_AVAILABLE = False
|
||||
|
||||
|
||||
# Placeholder class or functions for file handling
|
||||
class FileHandler:
|
||||
@@ -71,6 +77,192 @@ class FileHandler:
|
||||
|
||||
return target_path
|
||||
|
||||
def download_video(self, url, output_filename_no_extension, cookies_str=None):
|
||||
"""
|
||||
Download audio from a URL (YouTube, etc.) using yt-dlp.
|
||||
|
||||
This method downloads the best quality audio from a URL and saves it
|
||||
to the specified output path. It handles YouTube and other video platforms
|
||||
supported by yt-dlp.
|
||||
|
||||
Args:
|
||||
url: URL to download from (YouTube, Vimeo, etc.)
|
||||
output_filename_no_extension: Output filename without extension
|
||||
cookies_str: Optional cookies string for authenticated downloads
|
||||
|
||||
Returns:
|
||||
Path to downloaded audio file, or None if failed
|
||||
"""
|
||||
if not YT_DLP_AVAILABLE:
|
||||
self.logger.error("yt-dlp is not installed. Install with: pip install yt-dlp")
|
||||
return None
|
||||
|
||||
self.logger.info(f"Downloading audio from URL: {url}")
|
||||
|
||||
# Configure yt-dlp options
|
||||
ydl_opts = {
|
||||
'format': 'bestaudio/best',
|
||||
'outtmpl': output_filename_no_extension + '.%(ext)s',
|
||||
'postprocessors': [{
|
||||
'key': 'FFmpegExtractAudio',
|
||||
'preferredcodec': 'best',
|
||||
'preferredquality': '0', # Best quality
|
||||
}],
|
||||
'quiet': True,
|
||||
'no_warnings': True,
|
||||
'extract_flat': False,
|
||||
# Anti-detection options
|
||||
'user_agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/120.0.0.0 Safari/537.36',
|
||||
'retries': 3,
|
||||
'fragment_retries': 3,
|
||||
'http_headers': {
|
||||
'Accept-Language': 'en-US,en;q=0.9',
|
||||
'Accept': 'text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8',
|
||||
},
|
||||
}
|
||||
|
||||
# Handle cookies if provided - use safe tempfile pattern to avoid leaks
|
||||
cookie_file_path = None
|
||||
if cookies_str:
|
||||
try:
|
||||
# Use context manager to safely write cookies file
|
||||
with tempfile.NamedTemporaryFile(mode='w', suffix='.txt', delete=False) as cookie_file:
|
||||
cookie_file.write(cookies_str)
|
||||
cookie_file_path = cookie_file.name
|
||||
ydl_opts['cookiefile'] = cookie_file_path
|
||||
except Exception as e:
|
||||
self.logger.warning(f"Failed to write cookies file: {e}")
|
||||
cookie_file_path = None
|
||||
|
||||
try:
|
||||
with yt_dlp.YoutubeDL(ydl_opts) as ydl:
|
||||
# Extract info first to get actual filename
|
||||
info = ydl.extract_info(url, download=True)
|
||||
|
||||
if info is None:
|
||||
self.logger.error("Failed to extract info from URL")
|
||||
return None
|
||||
|
||||
# Find the downloaded file
|
||||
# The actual filename might differ from template due to post-processing
|
||||
downloaded_file = None
|
||||
|
||||
# Check common extensions
|
||||
for ext in ['m4a', 'opus', 'webm', 'mp3', 'flac', 'wav', 'ogg', 'aac']:
|
||||
candidate = f"{output_filename_no_extension}.{ext}"
|
||||
if os.path.exists(candidate):
|
||||
downloaded_file = candidate
|
||||
break
|
||||
|
||||
if downloaded_file is None:
|
||||
# Try to find any audio file with matching prefix
|
||||
import glob
|
||||
matches = glob.glob(f"{output_filename_no_extension}.*")
|
||||
audio_extensions = ['.m4a', '.opus', '.webm', '.mp3', '.flac', '.wav', '.ogg', '.aac']
|
||||
for match in matches:
|
||||
if any(match.endswith(ext) for ext in audio_extensions):
|
||||
downloaded_file = match
|
||||
break
|
||||
|
||||
if downloaded_file and os.path.exists(downloaded_file):
|
||||
self.logger.info(f"Successfully downloaded: {downloaded_file}")
|
||||
return downloaded_file
|
||||
else:
|
||||
self.logger.error("Downloaded file not found after yt-dlp completed")
|
||||
return None
|
||||
|
||||
except yt_dlp.DownloadError as e:
|
||||
self.logger.error(f"yt-dlp download error: {e}")
|
||||
return None
|
||||
except Exception as e:
|
||||
self.logger.error(f"Failed to download from URL: {e}")
|
||||
return None
|
||||
finally:
|
||||
# Clean up cookie file if we created one
|
||||
if cookie_file_path is not None:
|
||||
try:
|
||||
os.unlink(cookie_file_path)
|
||||
except Exception:
|
||||
pass
|
||||
|
||||
def extract_metadata_from_url(self, url):
|
||||
"""
|
||||
Extract metadata (artist, title) from a URL without downloading.
|
||||
|
||||
Uses yt-dlp to fetch video metadata including title, uploader/artist,
|
||||
and other information that can be used for the karaoke generation.
|
||||
|
||||
Args:
|
||||
url: URL to extract metadata from
|
||||
|
||||
Returns:
|
||||
Dict with 'artist', 'title', 'duration', and 'raw_info', or None if failed
|
||||
"""
|
||||
if not YT_DLP_AVAILABLE:
|
||||
self.logger.error("yt-dlp is not installed. Install with: pip install yt-dlp")
|
||||
return None
|
||||
|
||||
self.logger.info(f"Extracting metadata from URL: {url}")
|
||||
|
||||
ydl_opts = {
|
||||
'quiet': True,
|
||||
'no_warnings': True,
|
||||
'extract_flat': False,
|
||||
'skip_download': True,
|
||||
}
|
||||
|
||||
try:
|
||||
with yt_dlp.YoutubeDL(ydl_opts) as ydl:
|
||||
info = ydl.extract_info(url, download=False)
|
||||
|
||||
if info is None:
|
||||
self.logger.error("Failed to extract metadata from URL")
|
||||
return None
|
||||
|
||||
# Try to extract artist and title from various fields
|
||||
raw_title = info.get('title', '')
|
||||
uploader = info.get('uploader', '') or info.get('channel', '') or info.get('artist', '')
|
||||
duration = info.get('duration', 0)
|
||||
|
||||
# Attempt to parse "Artist - Title" format from title
|
||||
artist = None
|
||||
title = raw_title
|
||||
|
||||
if ' - ' in raw_title:
|
||||
parts = raw_title.split(' - ', 1)
|
||||
if len(parts) == 2:
|
||||
artist = parts[0].strip()
|
||||
title = parts[1].strip()
|
||||
|
||||
# Fall back to uploader as artist if not found in title
|
||||
if not artist:
|
||||
artist = uploader
|
||||
|
||||
# Clean up title (remove common suffixes like "(Official Video)")
|
||||
title_cleanup_patterns = [
|
||||
'(official video)', '(official music video)', '(official audio)',
|
||||
'(lyric video)', '(lyrics)', '(visualizer)', '(music video)',
|
||||
'[official video]', '[official music video]', '[official audio]',
|
||||
'(hd)', '(4k)', '(remastered)', '| official video', '| official audio',
|
||||
]
|
||||
title_lower = title.lower()
|
||||
for pattern in title_cleanup_patterns:
|
||||
if pattern in title_lower:
|
||||
idx = title_lower.find(pattern)
|
||||
title = title[:idx].strip()
|
||||
title_lower = title.lower()
|
||||
|
||||
return {
|
||||
'artist': artist,
|
||||
'title': title,
|
||||
'duration': duration,
|
||||
'raw_info': info,
|
||||
}
|
||||
|
||||
except Exception as e:
|
||||
self.logger.error(f"Failed to extract metadata from URL: {e}")
|
||||
return None
|
||||
|
||||
def extract_still_image_from_video(self, input_filename, output_filename_no_extension):
|
||||
output_filename = output_filename_no_extension + ".png"
|
||||
self.logger.info(f"Extracting still image from position 30s input media")
|
||||
|
||||
+469
-112
@@ -143,6 +143,70 @@ class RemoteKaraokeClient:
|
||||
response = self.session.request(method, url, **kwargs)
|
||||
return response
|
||||
|
||||
def _upload_file_to_signed_url(self, signed_url: str, file_path: str, content_type: str) -> bool:
|
||||
"""
|
||||
Upload a file directly to GCS using a signed URL.
|
||||
|
||||
Args:
|
||||
signed_url: The signed URL from the backend
|
||||
file_path: Local path to the file to upload
|
||||
content_type: MIME type for the Content-Type header
|
||||
|
||||
Returns:
|
||||
True if upload succeeded, False otherwise
|
||||
"""
|
||||
try:
|
||||
with open(file_path, 'rb') as f:
|
||||
# Use a fresh requests session (not self.session) because
|
||||
# signed URLs should not have our auth headers
|
||||
response = requests.put(
|
||||
signed_url,
|
||||
data=f,
|
||||
headers={'Content-Type': content_type},
|
||||
timeout=600 # 10 minutes for large files
|
||||
)
|
||||
|
||||
if response.status_code in (200, 201):
|
||||
return True
|
||||
else:
|
||||
self.logger.error(f"Failed to upload to signed URL: HTTP {response.status_code} - {response.text}")
|
||||
return False
|
||||
except Exception as e:
|
||||
self.logger.error(f"Error uploading to signed URL: {e}")
|
||||
return False
|
||||
|
||||
def _get_content_type(self, file_path: str) -> str:
|
||||
"""Get the MIME content type for a file based on its extension."""
|
||||
ext = Path(file_path).suffix.lower()
|
||||
|
||||
content_types = {
|
||||
# Audio
|
||||
'.mp3': 'audio/mpeg',
|
||||
'.wav': 'audio/wav',
|
||||
'.flac': 'audio/flac',
|
||||
'.m4a': 'audio/mp4',
|
||||
'.ogg': 'audio/ogg',
|
||||
'.aac': 'audio/aac',
|
||||
# Images
|
||||
'.png': 'image/png',
|
||||
'.jpg': 'image/jpeg',
|
||||
'.jpeg': 'image/jpeg',
|
||||
'.gif': 'image/gif',
|
||||
'.webp': 'image/webp',
|
||||
# Fonts
|
||||
'.ttf': 'font/ttf',
|
||||
'.otf': 'font/otf',
|
||||
'.woff': 'font/woff',
|
||||
'.woff2': 'font/woff2',
|
||||
# Other
|
||||
'.json': 'application/json',
|
||||
'.txt': 'text/plain',
|
||||
'.docx': 'application/vnd.openxmlformats-officedocument.wordprocessingml.document',
|
||||
'.rtf': 'application/rtf',
|
||||
}
|
||||
|
||||
return content_types.get(ext, 'application/octet-stream')
|
||||
|
||||
def _parse_style_params(self, style_params_path: str) -> Dict[str, str]:
|
||||
"""
|
||||
Parse style_params.json and extract file paths that need to be uploaded.
|
||||
@@ -183,6 +247,124 @@ class RemoteKaraokeClient:
|
||||
|
||||
return asset_files
|
||||
|
||||
def submit_job_from_url(
|
||||
self,
|
||||
url: str,
|
||||
artist: Optional[str] = None,
|
||||
title: Optional[str] = None,
|
||||
enable_cdg: bool = True,
|
||||
enable_txt: bool = True,
|
||||
brand_prefix: Optional[str] = None,
|
||||
discord_webhook_url: Optional[str] = None,
|
||||
youtube_description: Optional[str] = None,
|
||||
organised_dir_rclone_root: Optional[str] = None,
|
||||
enable_youtube_upload: bool = False,
|
||||
dropbox_path: Optional[str] = None,
|
||||
gdrive_folder_id: Optional[str] = None,
|
||||
lyrics_artist: Optional[str] = None,
|
||||
lyrics_title: Optional[str] = None,
|
||||
subtitle_offset_ms: int = 0,
|
||||
clean_instrumental_model: Optional[str] = None,
|
||||
backing_vocals_models: Optional[list] = None,
|
||||
other_stems_models: Optional[list] = None,
|
||||
) -> Dict[str, Any]:
|
||||
"""
|
||||
Submit a new karaoke generation job from a YouTube/online URL.
|
||||
|
||||
The backend will download the audio from the URL and process it.
|
||||
Artist and title will be auto-detected from the URL if not provided.
|
||||
|
||||
Note: Custom style configuration is not supported for URL-based jobs.
|
||||
If you need custom styles, download the audio locally first and use
|
||||
the regular file upload flow with submit_job().
|
||||
|
||||
Args:
|
||||
url: YouTube or other video URL to download audio from
|
||||
artist: Artist name (optional - auto-detected if not provided)
|
||||
title: Song title (optional - auto-detected if not provided)
|
||||
enable_cdg: Generate CDG+MP3 package
|
||||
enable_txt: Generate TXT+MP3 package
|
||||
brand_prefix: Brand code prefix (e.g., "NOMAD")
|
||||
discord_webhook_url: Discord webhook for notifications
|
||||
youtube_description: YouTube video description
|
||||
organised_dir_rclone_root: Legacy rclone path (deprecated)
|
||||
enable_youtube_upload: Enable YouTube upload
|
||||
dropbox_path: Dropbox folder path for organized output (native API)
|
||||
gdrive_folder_id: Google Drive folder ID for public share (native API)
|
||||
lyrics_artist: Override artist name for lyrics search
|
||||
lyrics_title: Override title for lyrics search
|
||||
subtitle_offset_ms: Subtitle timing offset in milliseconds
|
||||
clean_instrumental_model: Model for clean instrumental separation
|
||||
backing_vocals_models: List of models for backing vocals separation
|
||||
other_stems_models: List of models for other stems (bass, drums, etc.)
|
||||
"""
|
||||
self.logger.info(f"Submitting URL-based job: {url}")
|
||||
|
||||
# Build request payload
|
||||
create_request = {
|
||||
'url': url,
|
||||
'enable_cdg': enable_cdg,
|
||||
'enable_txt': enable_txt,
|
||||
}
|
||||
|
||||
if artist:
|
||||
create_request['artist'] = artist
|
||||
if title:
|
||||
create_request['title'] = title
|
||||
if brand_prefix:
|
||||
create_request['brand_prefix'] = brand_prefix
|
||||
if discord_webhook_url:
|
||||
create_request['discord_webhook_url'] = discord_webhook_url
|
||||
if youtube_description:
|
||||
create_request['youtube_description'] = youtube_description
|
||||
if enable_youtube_upload:
|
||||
create_request['enable_youtube_upload'] = enable_youtube_upload
|
||||
if dropbox_path:
|
||||
create_request['dropbox_path'] = dropbox_path
|
||||
if gdrive_folder_id:
|
||||
create_request['gdrive_folder_id'] = gdrive_folder_id
|
||||
if organised_dir_rclone_root:
|
||||
create_request['organised_dir_rclone_root'] = organised_dir_rclone_root
|
||||
if lyrics_artist:
|
||||
create_request['lyrics_artist'] = lyrics_artist
|
||||
if lyrics_title:
|
||||
create_request['lyrics_title'] = lyrics_title
|
||||
if subtitle_offset_ms != 0:
|
||||
create_request['subtitle_offset_ms'] = subtitle_offset_ms
|
||||
if clean_instrumental_model:
|
||||
create_request['clean_instrumental_model'] = clean_instrumental_model
|
||||
if backing_vocals_models:
|
||||
create_request['backing_vocals_models'] = backing_vocals_models
|
||||
if other_stems_models:
|
||||
create_request['other_stems_models'] = other_stems_models
|
||||
|
||||
self.logger.info(f"Creating URL-based job at {self.config.service_url}/api/jobs/create-from-url")
|
||||
|
||||
response = self._request('POST', '/api/jobs/create-from-url', json=create_request)
|
||||
|
||||
if response.status_code != 200:
|
||||
try:
|
||||
error_detail = response.json()
|
||||
except Exception:
|
||||
error_detail = response.text
|
||||
raise RuntimeError(f"Error creating job from URL: {error_detail}")
|
||||
|
||||
result = response.json()
|
||||
if result.get('status') != 'success':
|
||||
raise RuntimeError(f"Error creating job from URL: {result}")
|
||||
|
||||
job_id = result['job_id']
|
||||
detected_artist = result.get('detected_artist')
|
||||
detected_title = result.get('detected_title')
|
||||
|
||||
self.logger.info(f"Job {job_id} created from URL")
|
||||
if detected_artist:
|
||||
self.logger.info(f" Artist: {detected_artist}")
|
||||
if detected_title:
|
||||
self.logger.info(f" Title: {detected_title}")
|
||||
|
||||
return result
|
||||
|
||||
def submit_job(
|
||||
self,
|
||||
filepath: str,
|
||||
@@ -199,10 +381,26 @@ class RemoteKaraokeClient:
|
||||
# Native API distribution (uses server-side credentials)
|
||||
dropbox_path: Optional[str] = None,
|
||||
gdrive_folder_id: Optional[str] = None,
|
||||
# Lyrics configuration
|
||||
lyrics_artist: Optional[str] = None,
|
||||
lyrics_title: Optional[str] = None,
|
||||
lyrics_file: Optional[str] = None,
|
||||
subtitle_offset_ms: int = 0,
|
||||
# Audio separation model configuration
|
||||
clean_instrumental_model: Optional[str] = None,
|
||||
backing_vocals_models: Optional[list] = None,
|
||||
other_stems_models: Optional[list] = None,
|
||||
# Existing instrumental (Batch 3)
|
||||
existing_instrumental: Optional[str] = None,
|
||||
) -> Dict[str, Any]:
|
||||
"""
|
||||
Submit a new karaoke generation job with optional style configuration.
|
||||
|
||||
Uses signed URL upload flow to bypass Cloud Run's 32MB request body limit:
|
||||
1. Create job and get signed upload URLs from backend
|
||||
2. Upload files directly to GCS using signed URLs
|
||||
3. Notify backend that uploads are complete to start processing
|
||||
|
||||
Args:
|
||||
filepath: Path to audio file
|
||||
artist: Artist name
|
||||
@@ -217,6 +415,14 @@ class RemoteKaraokeClient:
|
||||
enable_youtube_upload: Enable YouTube upload
|
||||
dropbox_path: Dropbox folder path for organized output (native API)
|
||||
gdrive_folder_id: Google Drive folder ID for public share (native API)
|
||||
lyrics_artist: Override artist name for lyrics search
|
||||
lyrics_title: Override title for lyrics search
|
||||
lyrics_file: Path to user-provided lyrics file
|
||||
subtitle_offset_ms: Subtitle timing offset in milliseconds
|
||||
clean_instrumental_model: Model for clean instrumental separation
|
||||
backing_vocals_models: List of models for backing vocals separation
|
||||
other_stems_models: List of models for other stems (bass, drums, etc.)
|
||||
existing_instrumental: Path to existing instrumental file to use instead of AI separation
|
||||
"""
|
||||
file_path = Path(filepath)
|
||||
|
||||
@@ -230,95 +436,168 @@ class RemoteKaraokeClient:
|
||||
f"Allowed: {', '.join(self.ALLOWED_AUDIO_EXTENSIONS)}"
|
||||
)
|
||||
|
||||
self.logger.info(f"Uploading audio file: {filepath}")
|
||||
# Step 1: Build list of files to upload
|
||||
files_info = []
|
||||
local_files = {} # file_type -> local_path
|
||||
|
||||
# Prepare files dict for multipart upload
|
||||
files_to_upload = {}
|
||||
files_to_close = []
|
||||
# Main audio file
|
||||
audio_content_type = self._get_content_type(filepath)
|
||||
files_info.append({
|
||||
'filename': file_path.name,
|
||||
'content_type': audio_content_type,
|
||||
'file_type': 'audio'
|
||||
})
|
||||
local_files['audio'] = filepath
|
||||
self.logger.info(f"Will upload audio: {filepath}")
|
||||
|
||||
try:
|
||||
# Main audio file
|
||||
audio_file = open(filepath, 'rb')
|
||||
files_to_close.append(audio_file)
|
||||
files_to_upload['file'] = (file_path.name, audio_file)
|
||||
# Parse style params and find referenced files
|
||||
style_assets = {}
|
||||
if style_params_path and os.path.isfile(style_params_path):
|
||||
self.logger.info(f"Parsing style configuration: {style_params_path}")
|
||||
style_assets = self._parse_style_params(style_params_path)
|
||||
|
||||
# Parse style params and find referenced files
|
||||
style_assets = {}
|
||||
if style_params_path and os.path.isfile(style_params_path):
|
||||
self.logger.info(f"Parsing style configuration: {style_params_path}")
|
||||
style_assets = self._parse_style_params(style_params_path)
|
||||
|
||||
# Upload style_params.json
|
||||
style_file = open(style_params_path, 'rb')
|
||||
files_to_close.append(style_file)
|
||||
files_to_upload['style_params'] = (Path(style_params_path).name, style_file, 'application/json')
|
||||
self.logger.info(f" Will upload style_params.json")
|
||||
|
||||
# Upload each style asset file
|
||||
for asset_key, asset_path in style_assets.items():
|
||||
if os.path.isfile(asset_path):
|
||||
asset_file = open(asset_path, 'rb')
|
||||
files_to_close.append(asset_file)
|
||||
# Determine content type
|
||||
ext = Path(asset_path).suffix.lower()
|
||||
if ext in self.ALLOWED_IMAGE_EXTENSIONS:
|
||||
content_type = f'image/{ext[1:]}'
|
||||
elif ext in self.ALLOWED_FONT_EXTENSIONS:
|
||||
content_type = 'font/ttf'
|
||||
else:
|
||||
content_type = 'application/octet-stream'
|
||||
files_to_upload[asset_key] = (Path(asset_path).name, asset_file, content_type)
|
||||
self.logger.info(f" Will upload {asset_key}: {asset_path}")
|
||||
|
||||
# Prepare form data
|
||||
data = {
|
||||
'artist': artist,
|
||||
'title': title,
|
||||
'enable_cdg': str(enable_cdg).lower(),
|
||||
'enable_txt': str(enable_txt).lower(),
|
||||
}
|
||||
|
||||
if brand_prefix:
|
||||
data['brand_prefix'] = brand_prefix
|
||||
if discord_webhook_url:
|
||||
data['discord_webhook_url'] = discord_webhook_url
|
||||
if youtube_description:
|
||||
data['youtube_description'] = youtube_description
|
||||
if enable_youtube_upload:
|
||||
data['enable_youtube_upload'] = str(enable_youtube_upload).lower()
|
||||
|
||||
# Native API distribution (preferred for remote CLI)
|
||||
if dropbox_path:
|
||||
data['dropbox_path'] = dropbox_path
|
||||
if gdrive_folder_id:
|
||||
data['gdrive_folder_id'] = gdrive_folder_id
|
||||
|
||||
# Legacy rclone distribution (deprecated)
|
||||
if organised_dir_rclone_root:
|
||||
data['organised_dir_rclone_root'] = organised_dir_rclone_root
|
||||
|
||||
self.logger.info(f"Submitting job to {self.config.service_url}/api/jobs/upload")
|
||||
|
||||
response = self._request('POST', '/api/jobs/upload', files=files_to_upload, data=data)
|
||||
|
||||
finally:
|
||||
# Close all opened files
|
||||
for f in files_to_close:
|
||||
try:
|
||||
f.close()
|
||||
except:
|
||||
pass
|
||||
# Add style_params.json
|
||||
files_info.append({
|
||||
'filename': Path(style_params_path).name,
|
||||
'content_type': 'application/json',
|
||||
'file_type': 'style_params'
|
||||
})
|
||||
local_files['style_params'] = style_params_path
|
||||
self.logger.info(f" Will upload style_params.json")
|
||||
|
||||
# Add each style asset file
|
||||
for asset_key, asset_path in style_assets.items():
|
||||
if os.path.isfile(asset_path):
|
||||
content_type = self._get_content_type(asset_path)
|
||||
files_info.append({
|
||||
'filename': Path(asset_path).name,
|
||||
'content_type': content_type,
|
||||
'file_type': asset_key # e.g., 'style_intro_background'
|
||||
})
|
||||
local_files[asset_key] = asset_path
|
||||
self.logger.info(f" Will upload {asset_key}: {asset_path}")
|
||||
|
||||
# Add lyrics file if provided
|
||||
if lyrics_file and os.path.isfile(lyrics_file):
|
||||
content_type = self._get_content_type(lyrics_file)
|
||||
files_info.append({
|
||||
'filename': Path(lyrics_file).name,
|
||||
'content_type': content_type,
|
||||
'file_type': 'lyrics_file'
|
||||
})
|
||||
local_files['lyrics_file'] = lyrics_file
|
||||
self.logger.info(f"Will upload lyrics file: {lyrics_file}")
|
||||
|
||||
# Add existing instrumental file if provided (Batch 3)
|
||||
if existing_instrumental and os.path.isfile(existing_instrumental):
|
||||
content_type = self._get_content_type(existing_instrumental)
|
||||
files_info.append({
|
||||
'filename': Path(existing_instrumental).name,
|
||||
'content_type': content_type,
|
||||
'file_type': 'existing_instrumental'
|
||||
})
|
||||
local_files['existing_instrumental'] = existing_instrumental
|
||||
self.logger.info(f"Will upload existing instrumental: {existing_instrumental}")
|
||||
|
||||
# Step 2: Create job and get signed upload URLs
|
||||
self.logger.info(f"Creating job at {self.config.service_url}/api/jobs/create-with-upload-urls")
|
||||
|
||||
create_request = {
|
||||
'artist': artist,
|
||||
'title': title,
|
||||
'files': files_info,
|
||||
'enable_cdg': enable_cdg,
|
||||
'enable_txt': enable_txt,
|
||||
}
|
||||
|
||||
if brand_prefix:
|
||||
create_request['brand_prefix'] = brand_prefix
|
||||
if discord_webhook_url:
|
||||
create_request['discord_webhook_url'] = discord_webhook_url
|
||||
if youtube_description:
|
||||
create_request['youtube_description'] = youtube_description
|
||||
if enable_youtube_upload:
|
||||
create_request['enable_youtube_upload'] = enable_youtube_upload
|
||||
if dropbox_path:
|
||||
create_request['dropbox_path'] = dropbox_path
|
||||
if gdrive_folder_id:
|
||||
create_request['gdrive_folder_id'] = gdrive_folder_id
|
||||
if organised_dir_rclone_root:
|
||||
create_request['organised_dir_rclone_root'] = organised_dir_rclone_root
|
||||
if lyrics_artist:
|
||||
create_request['lyrics_artist'] = lyrics_artist
|
||||
if lyrics_title:
|
||||
create_request['lyrics_title'] = lyrics_title
|
||||
if subtitle_offset_ms != 0:
|
||||
create_request['subtitle_offset_ms'] = subtitle_offset_ms
|
||||
if clean_instrumental_model:
|
||||
create_request['clean_instrumental_model'] = clean_instrumental_model
|
||||
if backing_vocals_models:
|
||||
create_request['backing_vocals_models'] = backing_vocals_models
|
||||
if other_stems_models:
|
||||
create_request['other_stems_models'] = other_stems_models
|
||||
|
||||
response = self._request('POST', '/api/jobs/create-with-upload-urls', json=create_request)
|
||||
|
||||
if response.status_code != 200:
|
||||
try:
|
||||
error_detail = response.json()
|
||||
except Exception:
|
||||
error_detail = response.text
|
||||
raise RuntimeError(f"Error submitting job: {error_detail}")
|
||||
raise RuntimeError(f"Error creating job: {error_detail}")
|
||||
|
||||
create_result = response.json()
|
||||
if create_result.get('status') != 'success':
|
||||
raise RuntimeError(f"Error creating job: {create_result}")
|
||||
|
||||
job_id = create_result['job_id']
|
||||
upload_urls = create_result['upload_urls']
|
||||
|
||||
self.logger.info(f"Job {job_id} created. Uploading {len(upload_urls)} files directly to storage...")
|
||||
|
||||
# Step 3: Upload each file directly to GCS using signed URLs
|
||||
uploaded_files = []
|
||||
for url_info in upload_urls:
|
||||
file_type = url_info['file_type']
|
||||
signed_url = url_info['upload_url']
|
||||
content_type = url_info['content_type']
|
||||
local_path = local_files.get(file_type)
|
||||
|
||||
if not local_path:
|
||||
self.logger.warning(f"No local file found for file_type: {file_type}")
|
||||
continue
|
||||
|
||||
# Calculate file size for logging
|
||||
file_size = os.path.getsize(local_path)
|
||||
file_size_mb = file_size / (1024 * 1024)
|
||||
self.logger.info(f" Uploading {file_type} ({file_size_mb:.1f} MB)...")
|
||||
|
||||
success = self._upload_file_to_signed_url(signed_url, local_path, content_type)
|
||||
if not success:
|
||||
raise RuntimeError(f"Failed to upload {file_type} to storage")
|
||||
|
||||
uploaded_files.append(file_type)
|
||||
self.logger.info(f" ✓ Uploaded {file_type}")
|
||||
|
||||
# Step 4: Notify backend that uploads are complete
|
||||
self.logger.info(f"Notifying backend that uploads are complete...")
|
||||
|
||||
complete_request = {
|
||||
'uploaded_files': uploaded_files
|
||||
}
|
||||
|
||||
response = self._request('POST', f'/api/jobs/{job_id}/uploads-complete', json=complete_request)
|
||||
|
||||
if response.status_code != 200:
|
||||
try:
|
||||
error_detail = response.json()
|
||||
except Exception:
|
||||
error_detail = response.text
|
||||
raise RuntimeError(f"Error completing uploads: {error_detail}")
|
||||
|
||||
result = response.json()
|
||||
if result.get('status') != 'success':
|
||||
raise RuntimeError(f"Error submitting job: {result}")
|
||||
raise RuntimeError(f"Error completing uploads: {result}")
|
||||
|
||||
# Log distribution services info if available
|
||||
if 'distribution_services' in result:
|
||||
@@ -1618,8 +1897,6 @@ def main():
|
||||
ignored_features.append("--skip-transcription")
|
||||
if args.lyrics_only:
|
||||
ignored_features.append("--lyrics-only")
|
||||
if args.existing_instrumental:
|
||||
ignored_features.append("--existing_instrumental")
|
||||
if args.background_video:
|
||||
ignored_features.append("--background_video")
|
||||
if getattr(args, 'auto_download', False):
|
||||
@@ -1644,6 +1921,7 @@ def main():
|
||||
|
||||
# Handle new job submission - parse input arguments same as gen_cli
|
||||
input_media, artist, title, filename_pattern = None, None, None, None
|
||||
is_url_input = False
|
||||
|
||||
if not args.args:
|
||||
parser.print_help()
|
||||
@@ -1652,17 +1930,22 @@ def main():
|
||||
# Allow 3 forms of positional arguments:
|
||||
# 1. URL or Media File only
|
||||
# 2. Artist and Title only
|
||||
# 3. URL, Artist, and Title
|
||||
# 3. URL/File, Artist, and Title
|
||||
if args.args and (is_url(args.args[0]) or is_file(args.args[0])):
|
||||
input_media = args.args[0]
|
||||
is_url_input = is_url(args.args[0])
|
||||
if len(args.args) > 2:
|
||||
artist = args.args[1]
|
||||
title = args.args[2]
|
||||
elif len(args.args) > 1:
|
||||
artist = args.args[1]
|
||||
else:
|
||||
logger.error("Input media provided without Artist and Title")
|
||||
return 1
|
||||
# For URLs, artist/title can be auto-detected
|
||||
if is_url_input:
|
||||
logger.info("URL provided without Artist and Title - will be auto-detected from video metadata")
|
||||
else:
|
||||
logger.error("Input media provided without Artist and Title")
|
||||
return 1
|
||||
elif os.path.isdir(args.args[0]):
|
||||
logger.error("Folder processing is not yet supported in remote mode")
|
||||
return 1
|
||||
@@ -1670,7 +1953,7 @@ def main():
|
||||
artist = args.args[0]
|
||||
title = args.args[1]
|
||||
logger.error("Audio search (artist+title) is not yet supported in remote mode.")
|
||||
logger.error("Please provide a local audio file path instead.")
|
||||
logger.error("Please provide a local audio file path or YouTube URL instead.")
|
||||
logger.error("")
|
||||
logger.error("For local flacfetch search, use karaoke-gen instead:")
|
||||
logger.error(f" karaoke-gen \"{artist}\" \"{title}\"")
|
||||
@@ -1679,24 +1962,36 @@ def main():
|
||||
parser.print_help()
|
||||
return 1
|
||||
|
||||
# For now, remote mode only supports file uploads
|
||||
if not input_media or not os.path.isfile(input_media):
|
||||
logger.error("Remote mode currently only supports local file uploads")
|
||||
logger.error("Please provide a path to an audio file (mp3, wav, flac, m4a, ogg, aac)")
|
||||
# Validate input
|
||||
if not input_media:
|
||||
logger.error("No input media or URL provided")
|
||||
return 1
|
||||
|
||||
# Validate artist and title are provided
|
||||
if not artist or not title:
|
||||
logger.error("Artist and Title are required")
|
||||
parser.print_help()
|
||||
return 1
|
||||
# For file input, validate file exists and artist/title are provided
|
||||
if not is_url_input:
|
||||
if not os.path.isfile(input_media):
|
||||
logger.error(f"File not found: {input_media}")
|
||||
logger.error("Please provide a valid path to an audio file (mp3, wav, flac, m4a, ogg, aac)")
|
||||
return 1
|
||||
|
||||
if not artist or not title:
|
||||
logger.error("Artist and Title are required for file uploads")
|
||||
parser.print_help()
|
||||
return 1
|
||||
|
||||
logger.info("=" * 60)
|
||||
logger.info("Karaoke Generator (Remote) - Job Submission")
|
||||
logger.info("=" * 60)
|
||||
logger.info(f"File: {input_media}")
|
||||
logger.info(f"Artist: {artist}")
|
||||
logger.info(f"Title: {title}")
|
||||
if is_url_input:
|
||||
logger.info(f"URL: {input_media}")
|
||||
else:
|
||||
logger.info(f"File: {input_media}")
|
||||
if artist:
|
||||
logger.info(f"Artist: {artist}")
|
||||
if title:
|
||||
logger.info(f"Title: {title}")
|
||||
if not artist and not title and is_url_input:
|
||||
logger.info(f"Artist/Title: (will be auto-detected from URL)")
|
||||
if args.style_params_json:
|
||||
logger.info(f"Style: {args.style_params_json}")
|
||||
logger.info(f"CDG: {args.enable_cdg}, TXT: {args.enable_txt}")
|
||||
@@ -1714,6 +2009,24 @@ def main():
|
||||
logger.info(f"Dropbox (rclone): {args.organised_dir_rclone_root}")
|
||||
if args.discord_webhook_url:
|
||||
logger.info(f"Discord: enabled")
|
||||
# Lyrics configuration
|
||||
if getattr(args, 'lyrics_artist', None):
|
||||
logger.info(f"Lyrics Artist Override: {args.lyrics_artist}")
|
||||
if getattr(args, 'lyrics_title', None):
|
||||
logger.info(f"Lyrics Title Override: {args.lyrics_title}")
|
||||
if getattr(args, 'lyrics_file', None):
|
||||
logger.info(f"Lyrics File: {args.lyrics_file}")
|
||||
if getattr(args, 'subtitle_offset_ms', 0):
|
||||
logger.info(f"Subtitle Offset: {args.subtitle_offset_ms}ms")
|
||||
# Audio model configuration
|
||||
if getattr(args, 'clean_instrumental_model', None):
|
||||
logger.info(f"Clean Instrumental Model: {args.clean_instrumental_model}")
|
||||
if getattr(args, 'backing_vocals_models', None):
|
||||
logger.info(f"Backing Vocals Models: {args.backing_vocals_models}")
|
||||
if getattr(args, 'other_stems_models', None):
|
||||
logger.info(f"Other Stems Models: {args.other_stems_models}")
|
||||
if getattr(args, 'existing_instrumental', None):
|
||||
logger.info(f"Existing Instrumental: {args.existing_instrumental}")
|
||||
logger.info(f"Service URL: {config.service_url}")
|
||||
logger.info(f"Review UI: {config.review_ui_url}")
|
||||
if config.non_interactive:
|
||||
@@ -1731,23 +2044,67 @@ def main():
|
||||
logger.warning(f"Failed to read YouTube description file: {e}")
|
||||
|
||||
try:
|
||||
# Submit job with all options
|
||||
result = client.submit_job(
|
||||
filepath=input_media,
|
||||
artist=artist,
|
||||
title=title,
|
||||
style_params_path=args.style_params_json,
|
||||
enable_cdg=args.enable_cdg,
|
||||
enable_txt=args.enable_txt,
|
||||
brand_prefix=args.brand_prefix,
|
||||
discord_webhook_url=args.discord_webhook_url,
|
||||
youtube_description=youtube_description,
|
||||
organised_dir_rclone_root=args.organised_dir_rclone_root,
|
||||
enable_youtube_upload=getattr(args, 'enable_youtube_upload', False),
|
||||
# Native API distribution (preferred for remote CLI)
|
||||
dropbox_path=getattr(args, 'dropbox_path', None),
|
||||
gdrive_folder_id=getattr(args, 'gdrive_folder_id', None),
|
||||
)
|
||||
# Submit job - different endpoint for URL vs file
|
||||
if is_url_input:
|
||||
# URL-based job submission
|
||||
# Note: style_params_path is not supported for URL-based jobs
|
||||
# If custom styles are needed, download the audio locally first
|
||||
if args.style_params_json:
|
||||
logger.warning("Custom styles (--style_params_json) are not supported for URL-based jobs. "
|
||||
"Download the audio locally first and use file upload for custom styles.")
|
||||
|
||||
result = client.submit_job_from_url(
|
||||
url=input_media,
|
||||
artist=artist,
|
||||
title=title,
|
||||
enable_cdg=args.enable_cdg,
|
||||
enable_txt=args.enable_txt,
|
||||
brand_prefix=args.brand_prefix,
|
||||
discord_webhook_url=args.discord_webhook_url,
|
||||
youtube_description=youtube_description,
|
||||
organised_dir_rclone_root=args.organised_dir_rclone_root,
|
||||
enable_youtube_upload=getattr(args, 'enable_youtube_upload', False),
|
||||
# Native API distribution (preferred for remote CLI)
|
||||
dropbox_path=getattr(args, 'dropbox_path', None),
|
||||
gdrive_folder_id=getattr(args, 'gdrive_folder_id', None),
|
||||
# Lyrics configuration
|
||||
lyrics_artist=getattr(args, 'lyrics_artist', None),
|
||||
lyrics_title=getattr(args, 'lyrics_title', None),
|
||||
subtitle_offset_ms=getattr(args, 'subtitle_offset_ms', 0) or 0,
|
||||
# Audio separation model configuration
|
||||
clean_instrumental_model=getattr(args, 'clean_instrumental_model', None),
|
||||
backing_vocals_models=getattr(args, 'backing_vocals_models', None),
|
||||
other_stems_models=getattr(args, 'other_stems_models', None),
|
||||
)
|
||||
else:
|
||||
# File-based job submission
|
||||
result = client.submit_job(
|
||||
filepath=input_media,
|
||||
artist=artist,
|
||||
title=title,
|
||||
style_params_path=args.style_params_json,
|
||||
enable_cdg=args.enable_cdg,
|
||||
enable_txt=args.enable_txt,
|
||||
brand_prefix=args.brand_prefix,
|
||||
discord_webhook_url=args.discord_webhook_url,
|
||||
youtube_description=youtube_description,
|
||||
organised_dir_rclone_root=args.organised_dir_rclone_root,
|
||||
enable_youtube_upload=getattr(args, 'enable_youtube_upload', False),
|
||||
# Native API distribution (preferred for remote CLI)
|
||||
dropbox_path=getattr(args, 'dropbox_path', None),
|
||||
gdrive_folder_id=getattr(args, 'gdrive_folder_id', None),
|
||||
# Lyrics configuration
|
||||
lyrics_artist=getattr(args, 'lyrics_artist', None),
|
||||
lyrics_title=getattr(args, 'lyrics_title', None),
|
||||
lyrics_file=getattr(args, 'lyrics_file', None),
|
||||
subtitle_offset_ms=getattr(args, 'subtitle_offset_ms', 0) or 0,
|
||||
# Audio separation model configuration
|
||||
clean_instrumental_model=getattr(args, 'clean_instrumental_model', None),
|
||||
backing_vocals_models=getattr(args, 'backing_vocals_models', None),
|
||||
other_stems_models=getattr(args, 'other_stems_models', None),
|
||||
# Existing instrumental (Batch 3)
|
||||
existing_instrumental=getattr(args, 'existing_instrumental', None),
|
||||
)
|
||||
job_id = result.get('job_id')
|
||||
style_assets = result.get('style_assets_uploaded', [])
|
||||
server_version = result.get('server_version', 'unknown')
|
||||
|
||||
@@ -1,9 +1,8 @@
|
||||
import threading
|
||||
import time
|
||||
from typing import Any, Dict, List, Optional, Tuple, Union
|
||||
from typing import Any, Dict, List, Optional, Set, Tuple, Union
|
||||
import logging
|
||||
from tqdm import tqdm
|
||||
from multiprocessing import Pool, cpu_count
|
||||
from functools import partial
|
||||
from pathlib import Path
|
||||
import json
|
||||
@@ -64,8 +63,70 @@ class AnchorSequenceFinder:
|
||||
# self.logger.debug(f"_find_ngrams called with {len(words)} words, n={n}")
|
||||
return [(words[i : i + n], i) for i in range(len(words) - n + 1)]
|
||||
|
||||
def _build_ngram_index(
|
||||
self,
|
||||
references: Dict[str, List[str]],
|
||||
n: int
|
||||
) -> Dict[Tuple[str, ...], Dict[str, List[int]]]:
|
||||
"""
|
||||
Build a hash-based index mapping n-grams to their positions in each reference.
|
||||
|
||||
Args:
|
||||
references: Dict mapping source names to lists of cleaned words
|
||||
n: The n-gram length to index
|
||||
|
||||
Returns:
|
||||
Dict mapping n-gram tuples to {source: [positions]} dicts
|
||||
"""
|
||||
index: Dict[Tuple[str, ...], Dict[str, List[int]]] = {}
|
||||
|
||||
for source, words in references.items():
|
||||
for i in range(len(words) - n + 1):
|
||||
ngram_tuple = tuple(words[i:i + n])
|
||||
if ngram_tuple not in index:
|
||||
index[ngram_tuple] = {}
|
||||
if source not in index[ngram_tuple]:
|
||||
index[ngram_tuple][source] = []
|
||||
index[ngram_tuple][source].append(i)
|
||||
|
||||
return index
|
||||
|
||||
def _find_matching_sources_indexed(
|
||||
self,
|
||||
ngram: List[str],
|
||||
ngram_index: Dict[Tuple[str, ...], Dict[str, List[int]]]
|
||||
) -> Dict[str, int]:
|
||||
"""
|
||||
Find which sources contain the given n-gram using pre-built index (O(1) lookup).
|
||||
|
||||
Args:
|
||||
ngram: List of words to find
|
||||
ngram_index: Pre-built index from _build_ngram_index()
|
||||
|
||||
Returns:
|
||||
Dict mapping source names to first unused position
|
||||
"""
|
||||
matches = {}
|
||||
ngram_tuple = tuple(ngram)
|
||||
|
||||
# O(1) lookup in the index
|
||||
if ngram_tuple not in ngram_index:
|
||||
return matches
|
||||
|
||||
source_positions = ngram_index[ngram_tuple]
|
||||
|
||||
# For each source that contains this n-gram, find first unused position
|
||||
for source, positions in source_positions.items():
|
||||
used = self.used_positions.get(source, set())
|
||||
for pos in positions:
|
||||
if pos not in used:
|
||||
matches[source] = pos
|
||||
break
|
||||
|
||||
return matches
|
||||
|
||||
def _find_matching_sources(self, ngram: List[str], references: Dict[str, List[str]], n: int) -> Dict[str, int]:
|
||||
"""Find which sources contain the given n-gram and at what positions."""
|
||||
"""Find which sources contain the given n-gram and at what positions (legacy O(n) method)."""
|
||||
# self.logger.debug(f"_find_matching_sources called for ngram: '{' '.join(ngram)}'")
|
||||
matches = {}
|
||||
all_positions = {source: [] for source in references}
|
||||
@@ -193,121 +254,59 @@ class AnchorSequenceFinder:
|
||||
ref_words: Dict[str, List[Word]],
|
||||
min_sources: int,
|
||||
) -> List[AnchorSequence]:
|
||||
"""Process a single n-gram length to find matching sequences with timeout and early termination."""
|
||||
self.logger.info(f"🔍 N-GRAM {n}: Starting processing with {len(trans_words)} transcription words")
|
||||
self.logger.info(f"🔍 N-GRAM {n}: Reference sources: {list(ref_texts_clean.keys())}")
|
||||
self.logger.info(f"🔍 N-GRAM {n}: Max iterations limit: {self.max_iterations_per_ngram}")
|
||||
"""Process a single n-gram length to find matching sequences using hash-based index."""
|
||||
self.logger.debug(f"🔍 N-GRAM {n}: Starting processing with {len(trans_words)} transcription words")
|
||||
|
||||
candidate_anchors = []
|
||||
used_positions = {source: set() for source in ref_texts_clean.keys()}
|
||||
used_trans_positions = set()
|
||||
used_trans_positions: Set[int] = set()
|
||||
|
||||
iteration_count = 0
|
||||
last_progress_check = 0
|
||||
last_anchor_count = 0
|
||||
stagnation_count = 0
|
||||
|
||||
self.logger.debug(f"🔍 N-GRAM {n}: Processing n-gram length {n} with max {self.max_iterations_per_ngram} iterations")
|
||||
# Build hash-based index for O(1) lookups
|
||||
ngram_index = self._build_ngram_index(ref_texts_clean, n)
|
||||
self.logger.debug(f"🔍 N-GRAM {n}: Built index with {len(ngram_index)} unique n-grams")
|
||||
|
||||
# Generate n-grams from transcribed text once
|
||||
# Generate n-grams from transcribed text
|
||||
trans_ngrams = self._find_ngrams(trans_words, n)
|
||||
self.logger.info(f"🔍 N-GRAM {n}: Generated {len(trans_ngrams)} n-grams for processing")
|
||||
self.logger.debug(f"🔍 N-GRAM {n}: Processing {len(trans_ngrams)} transcription n-grams")
|
||||
|
||||
# Process all n-grams efficiently in multiple passes
|
||||
found_new_match = True
|
||||
while found_new_match and iteration_count < self.max_iterations_per_ngram:
|
||||
found_new_match = False
|
||||
iteration_count += 1
|
||||
anchors_found_this_iteration = 0
|
||||
# Single pass through all transcription n-grams
|
||||
for ngram, trans_pos in trans_ngrams:
|
||||
# Skip if we've already used this transcription position
|
||||
if trans_pos in used_trans_positions:
|
||||
continue
|
||||
|
||||
# Log every 10th iteration to track progress
|
||||
if iteration_count % 10 == 0:
|
||||
self.logger.debug(f"🔍 N-GRAM {n}: Iteration {iteration_count}, anchors found: {len(candidate_anchors)}")
|
||||
|
||||
# Check for progress stagnation every N iterations
|
||||
if iteration_count - last_progress_check >= self.progress_check_interval:
|
||||
current_anchor_count = len(candidate_anchors)
|
||||
if current_anchor_count == last_anchor_count:
|
||||
stagnation_count += 1
|
||||
self.logger.debug(f"🔍 N-GRAM {n}: Stagnation check {stagnation_count}/3 at iteration {iteration_count}")
|
||||
if stagnation_count >= 3: # No progress for 3 consecutive checks
|
||||
self.logger.info(f"🔍 N-GRAM {n}: ⏹️ Early termination due to stagnation after {iteration_count} iterations")
|
||||
break
|
||||
else:
|
||||
stagnation_count = 0 # Reset stagnation counter
|
||||
|
||||
last_anchor_count = current_anchor_count
|
||||
last_progress_check = iteration_count
|
||||
|
||||
self.logger.debug(f"🔍 N-GRAM {n}: iteration {iteration_count}, anchors: {current_anchor_count}, stagnation: {stagnation_count}")
|
||||
|
||||
# Process all n-grams in this iteration
|
||||
for ngram, trans_pos in trans_ngrams:
|
||||
# Skip if we've already used this transcription position
|
||||
if trans_pos in used_trans_positions:
|
||||
continue
|
||||
|
||||
# Get the actual words from the transcription at this position
|
||||
actual_words = [w.text.lower().strip('.,?!"\n') for w in all_words[trans_pos : trans_pos + n]]
|
||||
ngram_words = [w.lower() for w in ngram]
|
||||
|
||||
if actual_words != ngram_words:
|
||||
self.logger.error(f"🔍 N-GRAM {n}: ❌ Mismatch between ngram and actual words at position {trans_pos}:")
|
||||
self.logger.error(f"🔍 N-GRAM {n}: Ngram words: {ngram_words}")
|
||||
self.logger.error(f"🔍 N-GRAM {n}: Actual words: {actual_words}")
|
||||
self.logger.error(f"🔍 N-GRAM {n}: Full trans_words: {trans_words}")
|
||||
self.logger.error(f"🔍 N-GRAM {n}: Full all_words: {[w.text for w in all_words]}")
|
||||
raise AssertionError(
|
||||
f"Ngram words don't match actual words at position {trans_pos}. "
|
||||
f"This should never happen as trans_words should be derived from all_words."
|
||||
)
|
||||
|
||||
matches = self._find_matching_sources(ngram, ref_texts_clean, n)
|
||||
if len(matches) >= min_sources:
|
||||
# Log successful match
|
||||
if len(candidate_anchors) < 5: # Only log first few matches to avoid spam
|
||||
self.logger.debug(f"🔍 N-GRAM {n}: ✅ Found match: '{' '.join(ngram)}' at pos {trans_pos} with {len(matches)} sources")
|
||||
|
||||
# Get Word IDs for transcribed words
|
||||
transcribed_word_ids = [w.id for w in all_words[trans_pos : trans_pos + n]]
|
||||
|
||||
# Get Word IDs for reference words
|
||||
reference_word_ids = {source: [w.id for w in ref_words[source][pos : pos + n]] for source, pos in matches.items()}
|
||||
|
||||
# Mark positions as used
|
||||
for source, pos in matches.items():
|
||||
used_positions[source].add(pos)
|
||||
used_trans_positions.add(trans_pos)
|
||||
|
||||
anchor = AnchorSequence(
|
||||
id=WordUtils.generate_id(),
|
||||
transcribed_word_ids=transcribed_word_ids,
|
||||
transcription_position=trans_pos,
|
||||
reference_positions=matches,
|
||||
reference_word_ids=reference_word_ids,
|
||||
confidence=len(matches) / len(ref_texts_clean),
|
||||
)
|
||||
candidate_anchors.append(anchor)
|
||||
anchors_found_this_iteration += 1
|
||||
found_new_match = True
|
||||
|
||||
# For efficiency, if we have very low iteration limits, find one match per iteration
|
||||
if self.max_iterations_per_ngram <= 10:
|
||||
break
|
||||
# Use indexed lookup (O(1) instead of O(n))
|
||||
matches = self._find_matching_sources_indexed(ngram, ngram_index)
|
||||
|
||||
# Log progress for this iteration
|
||||
if anchors_found_this_iteration > 0:
|
||||
self.logger.debug(f"🔍 N-GRAM {n}: Found {anchors_found_this_iteration} anchors in iteration {iteration_count}")
|
||||
|
||||
# Early termination if we've found enough anchors or processed all positions
|
||||
if len(used_trans_positions) >= len(trans_ngrams) or len(candidate_anchors) >= len(trans_ngrams):
|
||||
self.logger.info(f"🔍 N-GRAM {n}: ⏹️ Early termination - processed all positions after {iteration_count} iterations")
|
||||
break
|
||||
if len(matches) >= min_sources:
|
||||
# Get Word IDs for transcribed words
|
||||
transcribed_word_ids = [w.id for w in all_words[trans_pos : trans_pos + n]]
|
||||
|
||||
if iteration_count >= self.max_iterations_per_ngram:
|
||||
self.logger.warning(f"🔍 N-GRAM {n}: ⏰ Processing terminated after reaching max iterations ({self.max_iterations_per_ngram})")
|
||||
# Get Word IDs for reference words
|
||||
reference_word_ids = {
|
||||
source: [w.id for w in ref_words[source][pos : pos + n]]
|
||||
for source, pos in matches.items()
|
||||
}
|
||||
|
||||
# Mark transcription position as used
|
||||
used_trans_positions.add(trans_pos)
|
||||
|
||||
# Mark reference positions as used
|
||||
for source, pos in matches.items():
|
||||
if source not in self.used_positions:
|
||||
self.used_positions[source] = set()
|
||||
self.used_positions[source].add(pos)
|
||||
|
||||
anchor = AnchorSequence(
|
||||
id=WordUtils.generate_id(),
|
||||
transcribed_word_ids=transcribed_word_ids,
|
||||
transcription_position=trans_pos,
|
||||
reference_positions=matches,
|
||||
reference_word_ids=reference_word_ids,
|
||||
confidence=len(matches) / len(ref_texts_clean),
|
||||
)
|
||||
candidate_anchors.append(anchor)
|
||||
|
||||
self.logger.info(f"🔍 N-GRAM {n}: ✅ Completed processing after {iteration_count} iterations, found {len(candidate_anchors)} anchors")
|
||||
self.logger.debug(f"🔍 N-GRAM {n}: Found {len(candidate_anchors)} anchors")
|
||||
return candidate_anchors
|
||||
|
||||
def find_anchors(
|
||||
@@ -347,8 +346,10 @@ class AnchorSequenceFinder:
|
||||
self.logger.error("Could not serialize first cached anchor for logging")
|
||||
|
||||
# If not in cache or cache format invalid, perform the computation
|
||||
self.logger.info(f"🔍 ANCHOR SEARCH: ❌ Cache miss - computing anchors with timeout {self.timeout_seconds}s")
|
||||
self.logger.info(f"🔍 ANCHOR SEARCH: Finding anchor sequences for transcription with length {len(transcribed)}")
|
||||
self.logger.info(f"🔍 ANCHOR SEARCH: Cache miss - computing anchors")
|
||||
|
||||
# Reset used positions for fresh computation
|
||||
self.used_positions = {}
|
||||
|
||||
# Check timeout before starting computation
|
||||
self._check_timeout(start_time, "anchor computation initialization")
|
||||
@@ -407,122 +408,45 @@ class AnchorSequenceFinder:
|
||||
min_sources=self.min_sources,
|
||||
)
|
||||
|
||||
# Process n-gram lengths in parallel with timeout
|
||||
# Process n-gram lengths sequentially (single-threaded for cloud compatibility)
|
||||
candidate_anchors = []
|
||||
pool_timeout = max(60, self.timeout_seconds // 2) if self.timeout_seconds > 0 else 300 # Use half the total timeout for pool operations
|
||||
|
||||
# Check timeout before parallel processing
|
||||
self._check_timeout(start_time, "parallel processing start")
|
||||
self.logger.info(f"🔍 ANCHOR SEARCH: ✅ Timeout check passed - about to start parallel processing")
|
||||
# Check timeout before processing
|
||||
self._check_timeout(start_time, "n-gram processing start")
|
||||
self.logger.info(f"🔍 ANCHOR SEARCH: Starting sequential n-gram processing ({len(n_gram_lengths)} lengths)")
|
||||
|
||||
pool = None
|
||||
try:
|
||||
self.logger.info(f"🔍 ANCHOR SEARCH: 🚀 Starting parallel processing with {max(cpu_count() - 1, 1)} processes, pool timeout: {pool_timeout}s")
|
||||
pool = Pool(processes=max(cpu_count() - 1, 1))
|
||||
self.logger.debug(f"🔍 ANCHOR SEARCH: Pool created successfully")
|
||||
results = []
|
||||
|
||||
# Submit all jobs first
|
||||
self.logger.info(f"🔍 ANCHOR SEARCH: Submitting {len(n_gram_lengths)} n-gram processing jobs...")
|
||||
async_results = []
|
||||
for i, n in enumerate(n_gram_lengths):
|
||||
self.logger.debug(f"🔍 ANCHOR SEARCH: Submitting job {i+1}/{len(n_gram_lengths)} for n-gram length {n}")
|
||||
async_result = pool.apply_async(process_length_partial, (n,))
|
||||
async_results.append(async_result)
|
||||
|
||||
self.logger.info(f"🔍 ANCHOR SEARCH: ✅ All {len(async_results)} jobs submitted")
|
||||
|
||||
# Collect results with individual timeouts
|
||||
batch_results = []
|
||||
batch_size = 10
|
||||
|
||||
for i, async_result in enumerate(async_results):
|
||||
n_gram_length = n_gram_lengths[i]
|
||||
try:
|
||||
# Check remaining time for pool timeout (more lenient than overall timeout)
|
||||
batch_size = 10
|
||||
batch_results = []
|
||||
|
||||
for i, n in enumerate(n_gram_lengths):
|
||||
try:
|
||||
# Check timeout periodically
|
||||
if self.timeout_seconds > 0:
|
||||
elapsed_time = time.time() - start_time
|
||||
remaining_time = max(10, self.timeout_seconds - elapsed_time) if self.timeout_seconds > 0 else pool_timeout
|
||||
|
||||
self.logger.debug(f"🔍 ANCHOR SEARCH: Remaining time for n-gram {n_gram_length}: {remaining_time}s")
|
||||
|
||||
# Use a more lenient timeout for individual results to allow fallback
|
||||
individual_timeout = min(pool_timeout, remaining_time) if self.timeout_seconds > 0 else pool_timeout
|
||||
|
||||
result = async_result.get(timeout=individual_timeout)
|
||||
results.append(result)
|
||||
|
||||
# Batch logging - collect info for batched logging
|
||||
batch_results.append((n_gram_length, len(result)))
|
||||
|
||||
# Log progress every batch_size results or on the last result
|
||||
if (i + 1) % batch_size == 0 or (i + 1) == len(async_results):
|
||||
total_anchors_in_batch = sum(anchor_count for _, anchor_count in batch_results)
|
||||
n_gram_ranges = [str(ng) for ng, _ in batch_results]
|
||||
range_str = f"{n_gram_ranges[0]}-{n_gram_ranges[-1]}" if len(n_gram_ranges) > 1 else n_gram_ranges[0]
|
||||
self.logger.info(f"🔍 ANCHOR SEARCH: ✅ Completed n-gram lengths {range_str} ({i+1-len(batch_results)+1}-{i+1}/{len(async_results)}) - found {total_anchors_in_batch} anchors total")
|
||||
batch_results = [] # Reset batch
|
||||
|
||||
except Exception as e:
|
||||
self.logger.warning(f"🔍 ANCHOR SEARCH: ⚠️ n-gram length {n_gram_length} failed or timed out: {str(e)}")
|
||||
results.append([]) # Add empty result to maintain order
|
||||
|
||||
# Add failed result to batch for logging
|
||||
batch_results.append((n_gram_length, 0))
|
||||
|
||||
# If we're running short on time, trigger fallback early
|
||||
if self.timeout_seconds > 0 and (time.time() - start_time) > (self.timeout_seconds * 0.8):
|
||||
self.logger.warning(f"🔍 ANCHOR SEARCH: ⚠️ Approaching timeout limit, triggering early fallback")
|
||||
# Raise exception to trigger fallback to sequential processing
|
||||
raise Exception("Parallel processing timeout, triggering fallback")
|
||||
|
||||
self.logger.info(f"🔍 ANCHOR SEARCH: ✅ Parallel processing completed, combining results...")
|
||||
for anchors in results:
|
||||
if elapsed_time > self.timeout_seconds:
|
||||
self.logger.warning(f"🔍 ANCHOR SEARCH: ⏰ Timeout reached at n-gram {n}, stopping")
|
||||
break
|
||||
|
||||
anchors = self._process_ngram_length(
|
||||
n, trans_words, all_words, ref_texts_clean, ref_words, self.min_sources
|
||||
)
|
||||
candidate_anchors.extend(anchors)
|
||||
|
||||
# Explicitly cleanup pool to avoid hangs in containerized environments
|
||||
self.logger.info(f"🔍 ANCHOR SEARCH: 🧹 Cleaning up pool...")
|
||||
pool.close()
|
||||
pool.terminate()
|
||||
self.logger.info(f"🔍 ANCHOR SEARCH: ✅ Pool cleanup completed")
|
||||
|
||||
# Batch logging
|
||||
batch_results.append((n, len(anchors)))
|
||||
|
||||
# Log progress every batch_size results or on the last result
|
||||
if (i + 1) % batch_size == 0 or (i + 1) == len(n_gram_lengths):
|
||||
total_anchors_in_batch = sum(anchor_count for _, anchor_count in batch_results)
|
||||
n_gram_ranges = [str(ng) for ng, _ in batch_results]
|
||||
range_str = f"{n_gram_ranges[0]}-{n_gram_ranges[-1]}" if len(n_gram_ranges) > 1 else n_gram_ranges[0]
|
||||
self.logger.debug(f"🔍 ANCHOR SEARCH: Completed n-gram lengths {range_str} - found {total_anchors_in_batch} anchors")
|
||||
batch_results = []
|
||||
|
||||
except AnchorSequenceTimeoutError:
|
||||
self.logger.error(f"🔍 ANCHOR SEARCH: ❌ Parallel processing timed out")
|
||||
# Re-raise timeout errors
|
||||
raise
|
||||
except Exception as e:
|
||||
self.logger.error(f"🔍 ANCHOR SEARCH: ❌ Parallel processing failed: {str(e)}")
|
||||
# Fall back to sequential processing with timeout checks
|
||||
self.logger.info("🔍 ANCHOR SEARCH: 🔄 Falling back to sequential processing")
|
||||
for n in n_gram_lengths:
|
||||
try:
|
||||
# Check timeout more leniently during sequential processing
|
||||
if self.timeout_seconds > 0:
|
||||
elapsed_time = time.time() - start_time
|
||||
# Allow more time for sequential processing (up to 2x the original timeout)
|
||||
if elapsed_time > (self.timeout_seconds * 2.0):
|
||||
self.logger.warning(f"🔍 ANCHOR SEARCH: ⏰ Sequential processing timeout for n-gram {n}")
|
||||
break
|
||||
|
||||
self.logger.info(f"🔍 ANCHOR SEARCH: 🔄 Sequential processing n-gram length {n}")
|
||||
|
||||
anchors = self._process_ngram_length(
|
||||
n, trans_words, all_words, ref_texts_clean, ref_words, self.min_sources
|
||||
)
|
||||
candidate_anchors.extend(anchors)
|
||||
self.logger.info(f"🔍 ANCHOR SEARCH: ✅ Sequential n-gram {n} completed - found {len(anchors)} anchors")
|
||||
except Exception as e:
|
||||
self.logger.warning(f"🔍 ANCHOR SEARCH: ⚠️ Sequential processing failed for n-gram length {n}: {str(e)}")
|
||||
continue
|
||||
finally:
|
||||
# Always ensure pool is cleaned up to avoid hangs in containerized environments
|
||||
if pool is not None:
|
||||
try:
|
||||
self.logger.info(f"🔍 ANCHOR SEARCH: 🧹 Final pool cleanup...")
|
||||
pool.terminate()
|
||||
pool.join(timeout=5) # Wait max 5 seconds for workers to terminate
|
||||
self.logger.info(f"🔍 ANCHOR SEARCH: ✅ Final pool cleanup completed")
|
||||
except Exception as cleanup_error:
|
||||
self.logger.warning(f"🔍 ANCHOR SEARCH: ⚠️ Pool cleanup error (ignored): {cleanup_error}")
|
||||
except Exception as e:
|
||||
self.logger.warning(f"🔍 ANCHOR SEARCH: ⚠️ n-gram length {n} failed: {str(e)}")
|
||||
batch_results.append((n, 0))
|
||||
continue
|
||||
|
||||
self.logger.info(f"🔍 ANCHOR SEARCH: ✅ Found {len(candidate_anchors)} candidate anchors in {time.time() - start_time:.1f}s")
|
||||
|
||||
@@ -669,123 +593,75 @@ class AnchorSequenceFinder:
|
||||
|
||||
start_time = time.time()
|
||||
|
||||
# Try different pool sizes with timeout
|
||||
num_processes = max(cpu_count() - 1, 1) # Leave one CPU free
|
||||
self.logger.info(f"🔍 FILTERING: Using {num_processes} processes for scoring")
|
||||
|
||||
# Create a partial function with the context parameter fixed
|
||||
score_anchor_partial = partial(self._score_anchor_static, context=context)
|
||||
|
||||
# Use multiprocessing to score anchors in parallel with timeout
|
||||
# Score anchors sequentially using simple rule-based scoring
|
||||
# (Avoids expensive spaCy NLP and works in cloud environments)
|
||||
scored_anchors = []
|
||||
pool_timeout = 300 # 5 minutes for scoring phase
|
||||
|
||||
scoring_pool = None
|
||||
try:
|
||||
self.logger.info(f"🔍 FILTERING: 🚀 Starting parallel scoring with timeout {pool_timeout}s")
|
||||
scoring_pool = Pool(processes=num_processes)
|
||||
# Submit scoring jobs with timeout
|
||||
async_results = []
|
||||
batch_size = 50
|
||||
|
||||
self.logger.info(f"🔍 FILTERING: Splitting {len(anchors)} anchors into batches of {batch_size}")
|
||||
for i in range(0, len(anchors), batch_size):
|
||||
batch = anchors[i:i + batch_size]
|
||||
async_result = scoring_pool.apply_async(self._score_batch_static, (batch, context))
|
||||
async_results.append(async_result)
|
||||
|
||||
self.logger.info(f"🔍 FILTERING: Submitted {len(async_results)} scoring batches")
|
||||
|
||||
# Collect results with timeout
|
||||
for i, async_result in enumerate(async_results):
|
||||
try:
|
||||
self.logger.debug(f"🔍 FILTERING: ⏳ Collecting batch {i+1}/{len(async_results)}")
|
||||
batch_results = async_result.get(timeout=pool_timeout)
|
||||
scored_anchors.extend(batch_results)
|
||||
self.logger.debug(f"🔍 FILTERING: ✅ Completed scoring batch {i+1}/{len(async_results)}")
|
||||
except Exception as e:
|
||||
self.logger.warning(f"🔍 FILTERING: ⚠️ Scoring batch {i+1} failed or timed out: {str(e)}")
|
||||
# Add basic scores for failed batch
|
||||
start_idx = i * batch_size
|
||||
end_idx = min((i + 1) * batch_size, len(anchors))
|
||||
for j in range(start_idx, end_idx):
|
||||
if j < len(anchors):
|
||||
try:
|
||||
phrase_score = PhraseScore(
|
||||
total_score=1.0,
|
||||
natural_break_score=1.0,
|
||||
phrase_type=PhraseType.COMPLETE
|
||||
)
|
||||
scored_anchors.append(ScoredAnchor(anchor=anchors[j], phrase_score=phrase_score))
|
||||
except:
|
||||
continue
|
||||
|
||||
# Explicitly cleanup pool to avoid hangs in containerized environments
|
||||
self.logger.info(f"🔍 FILTERING: 🧹 Cleaning up scoring pool...")
|
||||
scoring_pool.close()
|
||||
scoring_pool.terminate()
|
||||
self.logger.info(f"🔍 FILTERING: ✅ Scoring pool cleanup completed")
|
||||
|
||||
except Exception as e:
|
||||
self.logger.warning(f"🔍 FILTERING: ❌ Parallel scoring failed: {str(e)}, falling back to basic scoring")
|
||||
# Fall back to basic scoring
|
||||
for anchor in anchors:
|
||||
try:
|
||||
phrase_score = PhraseScore(
|
||||
total_score=1.0,
|
||||
natural_break_score=1.0,
|
||||
phrase_type=PhraseType.COMPLETE
|
||||
)
|
||||
scored_anchors.append(ScoredAnchor(anchor=anchor, phrase_score=phrase_score))
|
||||
except:
|
||||
continue
|
||||
finally:
|
||||
# Always ensure scoring pool is cleaned up to avoid hangs
|
||||
if scoring_pool is not None:
|
||||
try:
|
||||
self.logger.info(f"🔍 FILTERING: 🧹 Final scoring pool cleanup...")
|
||||
scoring_pool.terminate()
|
||||
scoring_pool.join(timeout=5) # Wait max 5 seconds for workers to terminate
|
||||
self.logger.info(f"🔍 FILTERING: ✅ Final scoring pool cleanup completed")
|
||||
except Exception as cleanup_error:
|
||||
self.logger.warning(f"🔍 FILTERING: ⚠️ Scoring pool cleanup error (ignored): {cleanup_error}")
|
||||
self.logger.debug(f"🔍 FILTERING: Scoring {len(anchors)} anchors sequentially")
|
||||
|
||||
parallel_time = time.time() - start_time
|
||||
self.logger.info(f"🔍 FILTERING: ✅ Parallel scoring completed in {parallel_time:.2f}s, scored {len(scored_anchors)} anchors")
|
||||
for i, anchor in enumerate(anchors):
|
||||
try:
|
||||
# Simple rule-based scoring based on anchor properties
|
||||
phrase_score = self._simple_score_anchor(anchor)
|
||||
scored_anchors.append(ScoredAnchor(anchor=anchor, phrase_score=phrase_score))
|
||||
except Exception as e:
|
||||
# Fallback to default score on error
|
||||
self.logger.debug(f"🔍 FILTERING: Scoring failed for anchor {i}: {e}")
|
||||
phrase_score = PhraseScore(
|
||||
phrase_type=PhraseType.COMPLETE,
|
||||
natural_break_score=1.0,
|
||||
length_score=1.0
|
||||
)
|
||||
scored_anchors.append(ScoredAnchor(anchor=anchor, phrase_score=phrase_score))
|
||||
|
||||
# Sort and filter as before
|
||||
self.logger.info(f"🔍 FILTERING: 🔄 Sorting anchors by priority...")
|
||||
scoring_time = time.time() - start_time
|
||||
self.logger.debug(f"🔍 FILTERING: Scoring completed in {scoring_time:.2f}s, scored {len(scored_anchors)} anchors")
|
||||
|
||||
# Sort anchors by priority (highest first)
|
||||
self.logger.debug(f"🔍 FILTERING: Sorting anchors by priority...")
|
||||
scored_anchors.sort(key=self._get_sequence_priority, reverse=True)
|
||||
self.logger.info(f"🔍 FILTERING: ✅ Sorting completed")
|
||||
|
||||
self.logger.info(f"🔍 FILTERING: 🔄 Filtering {len(scored_anchors)} overlapping sequences")
|
||||
# O(N) overlap filtering using covered positions set
|
||||
self.logger.debug(f"🔍 FILTERING: Filtering {len(scored_anchors)} overlapping sequences")
|
||||
filtered_scored = []
|
||||
|
||||
for i, scored_anchor in enumerate(scored_anchors):
|
||||
# Check timeout every 100 anchors using our timeout mechanism (more lenient)
|
||||
if i % 100 == 0 and i > 0:
|
||||
# Only check timeout if we're significantly over the limit
|
||||
if self.timeout_seconds > 0:
|
||||
elapsed_time = time.time() - start_time
|
||||
# Use a more lenient timeout for filtering (allow 50% more time)
|
||||
if elapsed_time > (self.timeout_seconds * 1.5):
|
||||
self.logger.warning(f"🔍 FILTERING: ⏰ Filtering timed out, returning {len(filtered_scored)} anchors out of {len(scored_anchors)}")
|
||||
break
|
||||
|
||||
self.logger.debug(f"🔍 FILTERING: Progress: {i}/{len(scored_anchors)} processed, {len(filtered_scored)} kept")
|
||||
covered_positions: Set[int] = set()
|
||||
|
||||
for scored_anchor in scored_anchors:
|
||||
anchor = scored_anchor.anchor
|
||||
start_pos = anchor.transcription_position
|
||||
end_pos = start_pos + anchor.length
|
||||
|
||||
overlaps = False
|
||||
for existing in filtered_scored:
|
||||
if self._sequences_overlap(scored_anchor.anchor, existing.anchor):
|
||||
overlaps = True
|
||||
break
|
||||
|
||||
if not overlaps:
|
||||
# Check if any position in this anchor's range is already covered
|
||||
anchor_positions = set(range(start_pos, end_pos))
|
||||
if not anchor_positions & covered_positions: # No overlap with covered
|
||||
filtered_scored.append(scored_anchor)
|
||||
covered_positions.update(anchor_positions)
|
||||
|
||||
self.logger.info(f"🔍 FILTERING: ✅ Filtering completed - kept {len(filtered_scored)} non-overlapping anchors out of {len(scored_anchors)}")
|
||||
self.logger.debug(f"🔍 FILTERING: Kept {len(filtered_scored)} non-overlapping anchors out of {len(scored_anchors)}")
|
||||
return filtered_scored
|
||||
|
||||
def _simple_score_anchor(self, anchor: AnchorSequence) -> PhraseScore:
|
||||
"""
|
||||
Simple rule-based scoring for anchors without expensive NLP.
|
||||
|
||||
Scoring criteria:
|
||||
- Longer sequences are preferred (length_score)
|
||||
- Sequences matching more reference sources are preferred (natural_break_score)
|
||||
- All sequences treated as COMPLETE type for simplicity
|
||||
"""
|
||||
# Length score: normalize to 0-1 range (3-15 words typical)
|
||||
length = anchor.length
|
||||
length_score = min(1.0, (length - 2) / 10.0) # 3 words = 0.1, 12 words = 1.0
|
||||
|
||||
# Source match score: more sources = higher score
|
||||
num_sources = len(anchor.reference_positions)
|
||||
natural_break_score = min(1.0, num_sources / 3.0) # 1 source = 0.33, 3+ sources = 1.0
|
||||
|
||||
return PhraseScore(
|
||||
phrase_type=PhraseType.COMPLETE,
|
||||
natural_break_score=natural_break_score,
|
||||
length_score=length_score
|
||||
)
|
||||
|
||||
@staticmethod
|
||||
def _score_anchor_static(anchor: AnchorSequence, context: str) -> ScoredAnchor:
|
||||
"""Static version of _score_anchor for multiprocessing compatibility."""
|
||||
@@ -816,9 +692,9 @@ class AnchorSequenceFinder:
|
||||
except Exception:
|
||||
# Add basic score for failed anchor
|
||||
phrase_score = PhraseScore(
|
||||
total_score=1.0,
|
||||
phrase_type=PhraseType.COMPLETE,
|
||||
natural_break_score=1.0,
|
||||
phrase_type=PhraseType.COMPLETE
|
||||
length_score=1.0
|
||||
)
|
||||
scored_anchors.append(ScoredAnchor(anchor=anchor, phrase_score=phrase_score))
|
||||
|
||||
|
||||
@@ -0,0 +1,363 @@
|
||||
#!/usr/bin/env python3
|
||||
"""
|
||||
Generate golden test fixtures for AnchorSequenceFinder using real cache data.
|
||||
|
||||
This script loads real transcription and reference data from the local cache,
|
||||
runs the current AnchorSequenceFinder implementation, and saves the results
|
||||
as fixtures for regression testing.
|
||||
|
||||
Usage:
|
||||
python scripts/generate_anchor_fixtures.py
|
||||
"""
|
||||
|
||||
import json
|
||||
import os
|
||||
import sys
|
||||
from pathlib import Path
|
||||
from dataclasses import dataclass
|
||||
from typing import Dict, List, Optional, Any
|
||||
|
||||
# Add the parent directory to the path so we can import lyrics_transcriber
|
||||
sys.path.insert(0, str(Path(__file__).parent.parent))
|
||||
|
||||
from lyrics_transcriber.types import (
|
||||
LyricsData,
|
||||
LyricsSegment,
|
||||
LyricsMetadata,
|
||||
TranscriptionData,
|
||||
TranscriptionResult,
|
||||
Word,
|
||||
)
|
||||
from lyrics_transcriber.correction.anchor_sequence import AnchorSequenceFinder
|
||||
|
||||
|
||||
@dataclass
|
||||
class SongMapping:
|
||||
"""Maps an audioshake transcription to its reference lyrics."""
|
||||
name: str
|
||||
description: str
|
||||
audioshake_hash: str
|
||||
reference_hash: str
|
||||
|
||||
|
||||
# 10 diverse songs with varying lengths and characteristics
|
||||
SONG_MAPPINGS = [
|
||||
SongMapping(
|
||||
name="waterloo_abba",
|
||||
description="ABBA - Waterloo, short song snippet (30 words)",
|
||||
audioshake_hash="1654b7c0755fad7f23bc888308f274de",
|
||||
reference_hash="3365f6a1e42388d84bcbfc0ac6f8aa6c",
|
||||
),
|
||||
SongMapping(
|
||||
name="let_your_hair_down",
|
||||
description="Let Your Hair Down, short song (86 words)",
|
||||
audioshake_hash="b19b4fca60015bbec16e4e0d68c2e777",
|
||||
reference_hash="a2e1d435193bf261db0246b535a87de9",
|
||||
),
|
||||
SongMapping(
|
||||
name="daddy_fell_into_sun",
|
||||
description="Daddy Fell Into The Sun, short (97 words)",
|
||||
audioshake_hash="2f7e53617228901f67c2256e5648fbd6",
|
||||
reference_hash="b1aac2e85a3800fddae4a77c74b37280",
|
||||
),
|
||||
SongMapping(
|
||||
name="pull_the_curtain",
|
||||
description="Pull the Curtain, medium-short (140 words)",
|
||||
audioshake_hash="87a40e919d0ba2066fa22c2b74cacc19",
|
||||
reference_hash="3448335d15fc3b842eef858e59681300",
|
||||
),
|
||||
SongMapping(
|
||||
name="bright_eyes",
|
||||
description="Bright Eyes, medium (164 words)",
|
||||
audioshake_hash="6cfe3e59cb5c0a10d935396ff8b96752",
|
||||
reference_hash="748e5a24b339ab66e8bbb55e42e67770",
|
||||
),
|
||||
SongMapping(
|
||||
name="texas_sun",
|
||||
description="Texas Sun, medium (183 words)",
|
||||
audioshake_hash="d0f851c8b3e7d56784c185f02416485b",
|
||||
reference_hash="2f0173340cb1169caf003cbfe535315b",
|
||||
),
|
||||
SongMapping(
|
||||
name="shut_up_and_let_me_go",
|
||||
description="Ting Tings - Shut Up and Let Me Go, medium-long (231 words)",
|
||||
audioshake_hash="489c65695fca5bbe9cd11325af8e34ca",
|
||||
reference_hash="e96729ca29274f64ddc9ad461058949d",
|
||||
),
|
||||
SongMapping(
|
||||
name="for_your_pleasure",
|
||||
description="For Your Pleasure, medium-long (251 words)",
|
||||
audioshake_hash="89fec5f0c8520e98bbeb17fe349e3182",
|
||||
reference_hash="7b2afa24bfb3f9611b45fb70698a8263",
|
||||
),
|
||||
SongMapping(
|
||||
name="mayday_situation",
|
||||
description="Mayday Situation Overload, long (292 words)",
|
||||
audioshake_hash="0655f3ddeac47008cee8f484f4a1ed55",
|
||||
reference_hash="35d368e93554c99d18d666eafe37f16c",
|
||||
),
|
||||
SongMapping(
|
||||
name="devil_went_down_to_georgia",
|
||||
description="The Devil Went Down to Georgia, long (391 words)",
|
||||
audioshake_hash="6f76f504e6a3c6dff7ee99191973579a",
|
||||
reference_hash="a98b3484d65c132218575f9cc118994d",
|
||||
),
|
||||
]
|
||||
|
||||
|
||||
def get_cache_dir() -> Path:
|
||||
"""Get the lyrics transcriber cache directory."""
|
||||
return Path(os.path.expanduser("~/lyrics-transcriber-cache"))
|
||||
|
||||
|
||||
def load_audioshake_as_transcription(cache_dir: Path, audioshake_hash: str) -> Optional[TranscriptionResult]:
|
||||
"""Load an audioshake converted file and convert it to TranscriptionResult."""
|
||||
file_path = cache_dir / f"audioshake_{audioshake_hash}_converted.json"
|
||||
if not file_path.exists():
|
||||
print(f" WARNING: Audioshake file not found: {file_path}")
|
||||
return None
|
||||
|
||||
with open(file_path) as f:
|
||||
data = json.load(f)
|
||||
|
||||
# Convert the raw cache format to LyricsSegment objects
|
||||
segments = []
|
||||
all_words = []
|
||||
|
||||
for seg_data in data.get("segments", []):
|
||||
words = []
|
||||
for w_data in seg_data.get("words", []):
|
||||
word = Word(
|
||||
id=w_data.get("id", f"w_{len(all_words)}"),
|
||||
text=w_data.get("text", ""),
|
||||
start_time=w_data.get("start_time") or 0.0,
|
||||
end_time=w_data.get("end_time") or 0.0,
|
||||
confidence=w_data.get("confidence"),
|
||||
created_during_correction=w_data.get("created_during_correction", False),
|
||||
)
|
||||
words.append(word)
|
||||
all_words.append(word)
|
||||
|
||||
segment = LyricsSegment(
|
||||
id=seg_data.get("id", f"s_{len(segments)}"),
|
||||
text=seg_data.get("text", ""),
|
||||
words=words,
|
||||
start_time=seg_data.get("start_time") or 0.0,
|
||||
end_time=seg_data.get("end_time") or 0.0,
|
||||
)
|
||||
segments.append(segment)
|
||||
|
||||
# Create TranscriptionData
|
||||
full_text = " ".join(seg.text for seg in segments)
|
||||
transcription_data = TranscriptionData(
|
||||
segments=segments,
|
||||
words=all_words,
|
||||
text=full_text,
|
||||
source="audioshake",
|
||||
)
|
||||
|
||||
return TranscriptionResult(
|
||||
name="audioshake",
|
||||
priority=1,
|
||||
result=transcription_data,
|
||||
)
|
||||
|
||||
|
||||
def load_reference_lyrics(cache_dir: Path, reference_hash: str) -> Dict[str, LyricsData]:
|
||||
"""Load all available reference lyrics for a given hash."""
|
||||
references = {}
|
||||
providers = ["spotify", "genius", "lrclib", "musixmatch"]
|
||||
|
||||
for provider in providers:
|
||||
file_path = cache_dir / f"{provider}_{reference_hash}_converted.json"
|
||||
if not file_path.exists():
|
||||
continue
|
||||
|
||||
with open(file_path) as f:
|
||||
data = json.load(f)
|
||||
|
||||
# Convert to LyricsSegment objects
|
||||
segments = []
|
||||
for seg_data in data.get("segments", []):
|
||||
words = []
|
||||
for w_data in seg_data.get("words", []):
|
||||
word = Word(
|
||||
id=w_data.get("id", f"w_{len(words)}"),
|
||||
text=w_data.get("text", ""),
|
||||
start_time=w_data.get("start_time") or 0.0,
|
||||
end_time=w_data.get("end_time") or 0.0,
|
||||
confidence=w_data.get("confidence"),
|
||||
created_during_correction=w_data.get("created_during_correction", False),
|
||||
)
|
||||
words.append(word)
|
||||
|
||||
segment = LyricsSegment(
|
||||
id=seg_data.get("id", f"s_{len(segments)}"),
|
||||
text=seg_data.get("text", ""),
|
||||
words=words,
|
||||
start_time=seg_data.get("start_time") or 0.0,
|
||||
end_time=seg_data.get("end_time") or 0.0,
|
||||
)
|
||||
segments.append(segment)
|
||||
|
||||
# Try to get metadata from raw file
|
||||
raw_path = cache_dir / f"{provider}_{reference_hash}_raw.json"
|
||||
track_name = "Unknown"
|
||||
artist_names = "Unknown"
|
||||
if raw_path.exists():
|
||||
try:
|
||||
with open(raw_path) as f:
|
||||
raw_data = json.load(f)
|
||||
track_name = raw_data.get("trackName") or raw_data.get("name") or raw_data.get("track_data", {}).get("name", "Unknown")
|
||||
artist_names = raw_data.get("artistName") or raw_data.get("track_data", {}).get("artists", [{}])[0].get("name", "Unknown")
|
||||
except:
|
||||
pass
|
||||
|
||||
metadata = LyricsMetadata(
|
||||
source=provider,
|
||||
track_name=track_name,
|
||||
artist_names=artist_names,
|
||||
is_synced=True,
|
||||
)
|
||||
|
||||
lyrics_data = LyricsData(
|
||||
segments=segments,
|
||||
metadata=metadata,
|
||||
source=provider,
|
||||
)
|
||||
references[provider] = lyrics_data
|
||||
|
||||
return references
|
||||
|
||||
|
||||
def extract_words_from_segments(segments: List[LyricsSegment]) -> List[str]:
|
||||
"""Extract all word texts from segments."""
|
||||
words = []
|
||||
for seg in segments:
|
||||
for word in seg.words:
|
||||
words.append(word.text)
|
||||
return words
|
||||
|
||||
|
||||
def generate_fixture(mapping: SongMapping, cache_dir: Path, output_dir: Path) -> bool:
|
||||
"""Generate a fixture for a single song mapping."""
|
||||
print(f"Generating fixture for: {mapping.name}")
|
||||
print(f" Description: {mapping.description}")
|
||||
|
||||
# Load transcription
|
||||
transcription_result = load_audioshake_as_transcription(cache_dir, mapping.audioshake_hash)
|
||||
if transcription_result is None:
|
||||
print(f" SKIPPED: Could not load transcription")
|
||||
return False
|
||||
|
||||
# Load references
|
||||
references = load_reference_lyrics(cache_dir, mapping.reference_hash)
|
||||
if not references:
|
||||
print(f" SKIPPED: No reference lyrics found")
|
||||
return False
|
||||
|
||||
print(f" Loaded {len(references)} reference sources: {list(references.keys())}")
|
||||
|
||||
# Get transcribed words and text
|
||||
transcribed_words = extract_words_from_segments(transcription_result.result.segments)
|
||||
transcribed_text = " ".join(transcribed_words)
|
||||
print(f" Transcription word count: {len(transcribed_words)}")
|
||||
|
||||
# Run AnchorSequenceFinder
|
||||
print(f" Running AnchorSequenceFinder...")
|
||||
try:
|
||||
finder = AnchorSequenceFinder(
|
||||
cache_dir=str(cache_dir),
|
||||
timeout_seconds=300, # 5 minute timeout for fixture generation
|
||||
)
|
||||
|
||||
anchors = finder.find_anchors(
|
||||
transcribed=transcribed_text,
|
||||
references=references,
|
||||
transcription_result=transcription_result,
|
||||
)
|
||||
|
||||
print(f" Found {len(anchors)} anchors")
|
||||
except Exception as e:
|
||||
print(f" ERROR running AnchorSequenceFinder: {e}")
|
||||
import traceback
|
||||
traceback.print_exc()
|
||||
return False
|
||||
|
||||
# Calculate coverage
|
||||
covered_positions = set()
|
||||
for anchor in anchors:
|
||||
pos = anchor.anchor.transcription_position
|
||||
length = anchor.anchor.length
|
||||
for i in range(pos, pos + length):
|
||||
covered_positions.add(i)
|
||||
|
||||
coverage_percent = (len(covered_positions) / len(transcribed_words) * 100) if transcribed_words else 0
|
||||
|
||||
# Build fixture
|
||||
fixture = {
|
||||
"name": mapping.name,
|
||||
"description": mapping.description,
|
||||
"source_hashes": {
|
||||
"audioshake": mapping.audioshake_hash,
|
||||
"references": mapping.reference_hash,
|
||||
},
|
||||
"transcription": {
|
||||
"segments": [seg.to_dict() for seg in transcription_result.result.segments],
|
||||
"word_count": len(transcribed_words),
|
||||
"text": transcription_result.result.text,
|
||||
},
|
||||
"references": {
|
||||
source: {
|
||||
"segments": [seg.to_dict() for seg in lyrics.segments],
|
||||
"word_count": sum(len(seg.words) for seg in lyrics.segments),
|
||||
}
|
||||
for source, lyrics in references.items()
|
||||
},
|
||||
"expected_anchors": [anchor.to_dict() for anchor in anchors],
|
||||
"expected_anchor_count": len(anchors),
|
||||
"expected_word_coverage": len(covered_positions),
|
||||
"expected_coverage_percent": round(coverage_percent, 2),
|
||||
}
|
||||
|
||||
# Save fixture
|
||||
output_path = output_dir / f"{mapping.name}.json"
|
||||
with open(output_path, 'w') as f:
|
||||
json.dump(fixture, f, indent=2)
|
||||
|
||||
print(f" Saved fixture to: {output_path}")
|
||||
print(f" Coverage: {coverage_percent:.1f}% ({len(covered_positions)}/{len(transcribed_words)} words)")
|
||||
|
||||
return True
|
||||
|
||||
|
||||
def main():
|
||||
"""Generate all fixtures."""
|
||||
cache_dir = get_cache_dir()
|
||||
output_dir = Path(__file__).parent.parent / "tests" / "fixtures" / "anchor_golden"
|
||||
|
||||
print(f"Cache directory: {cache_dir}")
|
||||
print(f"Output directory: {output_dir}")
|
||||
print()
|
||||
|
||||
# Ensure output directory exists
|
||||
output_dir.mkdir(parents=True, exist_ok=True)
|
||||
|
||||
# Generate fixtures
|
||||
success_count = 0
|
||||
for mapping in SONG_MAPPINGS:
|
||||
if generate_fixture(mapping, cache_dir, output_dir):
|
||||
success_count += 1
|
||||
print()
|
||||
|
||||
print(f"Generated {success_count}/{len(SONG_MAPPINGS)} fixtures successfully")
|
||||
|
||||
if success_count < len(SONG_MAPPINGS):
|
||||
print("\nWARNING: Some fixtures could not be generated. Check the errors above.")
|
||||
return 1
|
||||
|
||||
return 0
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
sys.exit(main())
|
||||
@@ -11,9 +11,35 @@ sys.path.insert(0, os.path.abspath(os.path.join(os.path.dirname(__file__), '..')
|
||||
# All tests are now working! No more skip patterns needed.
|
||||
SKIP_PATTERNS = []
|
||||
|
||||
|
||||
def pytest_configure(config):
|
||||
"""Register custom markers."""
|
||||
config.addinivalue_line(
|
||||
"markers", "slow: marks tests as slow (run with --run-slow)"
|
||||
)
|
||||
|
||||
|
||||
def pytest_addoption(parser):
|
||||
"""Add custom command line options."""
|
||||
parser.addoption(
|
||||
"--run-slow",
|
||||
action="store_true",
|
||||
default=False,
|
||||
help="run slow tests that execute expensive algorithms"
|
||||
)
|
||||
|
||||
|
||||
def pytest_collection_modifyitems(config, items):
|
||||
"""Modify test collection to skip slow tests unless --run-slow is specified."""
|
||||
# Skip slow tests unless --run-slow is specified
|
||||
if not config.getoption("--run-slow", default=False):
|
||||
skip_slow = pytest.mark.skip(reason="need --run-slow option to run")
|
||||
for item in items:
|
||||
if "slow" in item.keywords:
|
||||
item.add_marker(skip_slow)
|
||||
|
||||
# Legacy skip patterns
|
||||
for item in items:
|
||||
# Check if any test path pattern matches the current test
|
||||
for pattern in SKIP_PATTERNS:
|
||||
if pattern in item.nodeid:
|
||||
item.add_marker(pytest.mark.skip(reason="Skipped due to codebase changes"))
|
||||
|
||||
File diff suppressed because it is too large.
Load diff
+2990
File diff suppressed because it is too large.
Load diff
+12399
File diff suppressed because it is too large.
Load diff
+6501
File diff suppressed because it is too large.
Load diff
+3051
File diff suppressed because it is too large.
Load diff
File diff suppressed because it is too large.
Load diff
File diff suppressed because it is too large.
Load diff
+7490
File diff suppressed because it is too large.
Load diff
File diff suppressed because it is too large.
Load diff
File diff suppressed because it is too large.
Load diff
@@ -304,17 +304,24 @@ def test_real_world_lyrics_scenario(setup_teardown):
|
||||
"source2": "son of a mother\nson of a father\nyou can look inside",
|
||||
}
|
||||
finder = AnchorSequenceFinder(min_sequence_length=3, min_sources=1, cache_dir=setup_teardown)
|
||||
|
||||
|
||||
# Create proper test data objects
|
||||
transcription_result = create_test_transcription_result_from_text(transcribed)
|
||||
lyrics_data_references = convert_references_to_lyrics_data(references)
|
||||
|
||||
|
||||
anchors = finder.find_anchors(transcribed, lyrics_data_references, transcription_result)
|
||||
|
||||
# Should find at least one of these sequences
|
||||
expected_sequences = ["son of a father", "son of a", "of a father"]
|
||||
# Should find at least one anchor that contains these key phrases
|
||||
# Note: The algorithm may find longer matches like "son of a father you" which is valid
|
||||
expected_phrases = ["son of a father", "son of a", "of a father"]
|
||||
found_sequences = [anchor.anchor.text for anchor in anchors]
|
||||
assert any(seq in found_sequences for seq in expected_sequences), f"Expected one of {expected_sequences}, but found {found_sequences}"
|
||||
|
||||
# Check if any found sequence contains one of the expected phrases
|
||||
found_expected = any(
|
||||
any(phrase in seq for phrase in expected_phrases)
|
||||
for seq in found_sequences
|
||||
)
|
||||
assert found_expected, f"Expected anchors containing one of {expected_phrases}, but found {found_sequences}"
|
||||
|
||||
|
||||
def test_find_anchors_with_punctuation(setup_teardown):
|
||||
|
||||
@@ -0,0 +1,369 @@
|
||||
"""
|
||||
Golden tests for AnchorSequenceFinder using real song data fixtures.
|
||||
|
||||
These tests verify that the AnchorSequenceFinder produces equivalent results
|
||||
to the baseline fixtures generated from real cache data. They serve as
|
||||
regression tests to ensure algorithm changes don't break correctness.
|
||||
|
||||
Usage:
|
||||
pytest tests/unit/correction/test_anchor_sequence_golden.py -v
|
||||
"""
|
||||
|
||||
import json
|
||||
import pytest
|
||||
from pathlib import Path
|
||||
from typing import Dict, List, Any, Tuple
|
||||
|
||||
from lyrics_transcriber.types import (
|
||||
LyricsData,
|
||||
LyricsSegment,
|
||||
LyricsMetadata,
|
||||
TranscriptionData,
|
||||
TranscriptionResult,
|
||||
Word,
|
||||
ScoredAnchor,
|
||||
)
|
||||
from lyrics_transcriber.correction.anchor_sequence import AnchorSequenceFinder
|
||||
|
||||
|
||||
# Path to golden test fixtures
|
||||
FIXTURES_DIR = Path(__file__).parent.parent.parent / "fixtures" / "anchor_golden"
|
||||
|
||||
|
||||
def load_fixture(fixture_path: Path) -> Dict[str, Any]:
|
||||
"""Load a fixture JSON file."""
|
||||
with open(fixture_path) as f:
|
||||
return json.load(f)
|
||||
|
||||
|
||||
def fixture_to_transcription_result(fixture: Dict[str, Any]) -> TranscriptionResult:
|
||||
"""Convert fixture data to TranscriptionResult."""
|
||||
segments = []
|
||||
all_words = []
|
||||
|
||||
for seg_data in fixture["transcription"]["segments"]:
|
||||
words = []
|
||||
for w_data in seg_data["words"]:
|
||||
word = Word(
|
||||
id=w_data["id"],
|
||||
text=w_data["text"],
|
||||
start_time=w_data.get("start_time") or 0.0,
|
||||
end_time=w_data.get("end_time") or 0.0,
|
||||
confidence=w_data.get("confidence"),
|
||||
created_during_correction=w_data.get("created_during_correction", False),
|
||||
)
|
||||
words.append(word)
|
||||
all_words.append(word)
|
||||
|
||||
segment = LyricsSegment(
|
||||
id=seg_data["id"],
|
||||
text=seg_data["text"],
|
||||
words=words,
|
||||
start_time=seg_data.get("start_time") or 0.0,
|
||||
end_time=seg_data.get("end_time") or 0.0,
|
||||
)
|
||||
segments.append(segment)
|
||||
|
||||
transcription_data = TranscriptionData(
|
||||
segments=segments,
|
||||
words=all_words,
|
||||
text=fixture["transcription"]["text"],
|
||||
source="audioshake",
|
||||
)
|
||||
|
||||
return TranscriptionResult(
|
||||
name="audioshake",
|
||||
priority=1,
|
||||
result=transcription_data,
|
||||
)
|
||||
|
||||
|
||||
def fixture_to_references(fixture: Dict[str, Any]) -> Dict[str, LyricsData]:
|
||||
"""Convert fixture data to references dict."""
|
||||
references = {}
|
||||
|
||||
for source, ref_data in fixture["references"].items():
|
||||
segments = []
|
||||
for seg_data in ref_data["segments"]:
|
||||
words = []
|
||||
for w_data in seg_data["words"]:
|
||||
word = Word(
|
||||
id=w_data["id"],
|
||||
text=w_data["text"],
|
||||
start_time=w_data.get("start_time") or 0.0,
|
||||
end_time=w_data.get("end_time") or 0.0,
|
||||
confidence=w_data.get("confidence"),
|
||||
created_during_correction=w_data.get("created_during_correction", False),
|
||||
)
|
||||
words.append(word)
|
||||
|
||||
segment = LyricsSegment(
|
||||
id=seg_data["id"],
|
||||
text=seg_data["text"],
|
||||
words=words,
|
||||
start_time=seg_data.get("start_time") or 0.0,
|
||||
end_time=seg_data.get("end_time") or 0.0,
|
||||
)
|
||||
segments.append(segment)
|
||||
|
||||
metadata = LyricsMetadata(
|
||||
source=source,
|
||||
track_name="Unknown",
|
||||
artist_names="Unknown",
|
||||
is_synced=True,
|
||||
)
|
||||
|
||||
references[source] = LyricsData(
|
||||
segments=segments,
|
||||
metadata=metadata,
|
||||
source=source,
|
||||
)
|
||||
|
||||
return references
|
||||
|
||||
|
||||
def get_transcribed_text(transcription_result: TranscriptionResult) -> str:
|
||||
"""Extract full text from transcription result."""
|
||||
words = []
|
||||
for segment in transcription_result.result.segments:
|
||||
for word in segment.words:
|
||||
words.append(word.text)
|
||||
return " ".join(words)
|
||||
|
||||
|
||||
def calculate_coverage(anchors: List[ScoredAnchor], word_count: int) -> float:
|
||||
"""Calculate the percentage of words covered by anchors."""
|
||||
if word_count == 0:
|
||||
return 0.0
|
||||
|
||||
covered_positions = set()
|
||||
for anchor in anchors:
|
||||
pos = anchor.anchor.transcription_position
|
||||
length = anchor.anchor.length
|
||||
for i in range(pos, pos + length):
|
||||
covered_positions.add(i)
|
||||
|
||||
return len(covered_positions) / word_count * 100
|
||||
|
||||
|
||||
# Collect all fixture files
|
||||
FIXTURE_FILES = list(FIXTURES_DIR.glob("*.json")) if FIXTURES_DIR.exists() else []
|
||||
|
||||
|
||||
# Cache for computed anchor results - avoid running expensive algorithm multiple times
|
||||
_anchor_cache: Dict[str, Tuple[Dict[str, Any], List[ScoredAnchor]]] = {}
|
||||
|
||||
|
||||
def get_anchors_for_fixture(fixture_path: Path, tmp_path: Path) -> Tuple[Dict[str, Any], List[ScoredAnchor]]:
|
||||
"""
|
||||
Get anchors for a fixture, using cache to avoid re-running expensive algorithm.
|
||||
Returns (fixture_data, anchors).
|
||||
"""
|
||||
cache_key = str(fixture_path)
|
||||
|
||||
if cache_key in _anchor_cache:
|
||||
return _anchor_cache[cache_key]
|
||||
|
||||
fixture = load_fixture(fixture_path)
|
||||
transcription_result = fixture_to_transcription_result(fixture)
|
||||
references = fixture_to_references(fixture)
|
||||
transcribed = get_transcribed_text(transcription_result)
|
||||
|
||||
# Use tmp_path to avoid using cached results from user's actual cache
|
||||
finder = AnchorSequenceFinder(
|
||||
cache_dir=str(tmp_path),
|
||||
timeout_seconds=120, # 2 minute timeout per fixture
|
||||
)
|
||||
|
||||
anchors = finder.find_anchors(
|
||||
transcribed=transcribed,
|
||||
references=references,
|
||||
transcription_result=transcription_result,
|
||||
)
|
||||
|
||||
_anchor_cache[cache_key] = (fixture, anchors)
|
||||
return fixture, anchors
|
||||
|
||||
|
||||
class TestAnchorSequenceGolden:
|
||||
"""
|
||||
Golden tests to verify AnchorSequenceFinder produces expected results.
|
||||
|
||||
These tests use the EXPECTED values from fixtures (not re-running the algorithm)
|
||||
to verify fixture validity. The algorithm correctness tests are separate.
|
||||
"""
|
||||
|
||||
@pytest.mark.parametrize("fixture_path", FIXTURE_FILES, ids=lambda f: f.stem)
|
||||
def test_fixture_has_reasonable_coverage(self, fixture_path):
|
||||
"""Fixture should have reasonable coverage (at least 30%)."""
|
||||
fixture = load_fixture(fixture_path)
|
||||
expected_coverage = fixture["expected_coverage_percent"]
|
||||
|
||||
# All fixtures should have at least some coverage
|
||||
assert expected_coverage >= 30.0, (
|
||||
f"Fixture {fixture_path.name} has low coverage: {expected_coverage:.1f}%"
|
||||
)
|
||||
|
||||
@pytest.mark.parametrize("fixture_path", FIXTURE_FILES, ids=lambda f: f.stem)
|
||||
def test_expected_anchors_are_valid(self, fixture_path):
|
||||
"""Expected anchors in fixture should have valid positions."""
|
||||
fixture = load_fixture(fixture_path)
|
||||
word_count = fixture["transcription"]["word_count"]
|
||||
expected_anchors = fixture["expected_anchors"]
|
||||
|
||||
for i, anchor_data in enumerate(expected_anchors):
|
||||
# Handle both nested and flat anchor structures
|
||||
if "anchor" in anchor_data:
|
||||
pos = anchor_data["anchor"]["transcription_position"]
|
||||
length = anchor_data["anchor"].get("length", len(anchor_data["anchor"].get("transcribed_word_ids", [])))
|
||||
else:
|
||||
pos = anchor_data["transcription_position"]
|
||||
length = anchor_data.get("length", len(anchor_data.get("transcribed_word_ids", [])))
|
||||
|
||||
end_pos = pos + length
|
||||
|
||||
assert pos >= 0, f"Anchor {i} has negative position: {pos}"
|
||||
assert end_pos <= word_count, (
|
||||
f"Anchor {i} end position {end_pos} exceeds word count {word_count}"
|
||||
)
|
||||
|
||||
@pytest.mark.parametrize("fixture_path", FIXTURE_FILES, ids=lambda f: f.stem)
|
||||
def test_expected_anchors_dont_overlap(self, fixture_path):
|
||||
"""Expected anchors in fixture should not overlap."""
|
||||
fixture = load_fixture(fixture_path)
|
||||
expected_anchors = fixture["expected_anchors"]
|
||||
|
||||
covered_positions = set()
|
||||
for i, anchor_data in enumerate(expected_anchors):
|
||||
# Handle both nested and flat anchor structures
|
||||
if "anchor" in anchor_data:
|
||||
pos = anchor_data["anchor"]["transcription_position"]
|
||||
length = anchor_data["anchor"].get("length", len(anchor_data["anchor"].get("transcribed_word_ids", [])))
|
||||
else:
|
||||
pos = anchor_data["transcription_position"]
|
||||
length = anchor_data.get("length", len(anchor_data.get("transcribed_word_ids", [])))
|
||||
|
||||
for j in range(pos, pos + length):
|
||||
assert j not in covered_positions, (
|
||||
f"Position {j} is covered by multiple anchors"
|
||||
)
|
||||
covered_positions.add(j)
|
||||
|
||||
|
||||
class TestGoldenFixturesExist:
|
||||
"""Verify that golden fixtures exist and are valid."""
|
||||
|
||||
def test_fixtures_directory_exists(self):
|
||||
"""The fixtures directory should exist."""
|
||||
assert FIXTURES_DIR.exists(), f"Fixtures directory not found: {FIXTURES_DIR}"
|
||||
|
||||
def test_at_least_10_fixtures(self):
|
||||
"""There should be at least 10 fixture files."""
|
||||
fixture_count = len(FIXTURE_FILES)
|
||||
assert fixture_count >= 10, f"Expected at least 10 fixtures, found {fixture_count}"
|
||||
|
||||
@pytest.mark.parametrize("fixture_path", FIXTURE_FILES, ids=lambda f: f.stem)
|
||||
def test_fixture_has_required_fields(self, fixture_path):
|
||||
"""Each fixture should have all required fields."""
|
||||
required_fields = [
|
||||
"name",
|
||||
"description",
|
||||
"transcription",
|
||||
"references",
|
||||
"expected_anchors",
|
||||
"expected_anchor_count",
|
||||
"expected_coverage_percent",
|
||||
]
|
||||
|
||||
fixture = load_fixture(fixture_path)
|
||||
for field in required_fields:
|
||||
assert field in fixture, (
|
||||
f"Fixture {fixture_path.name} missing required field: {field}"
|
||||
)
|
||||
|
||||
|
||||
# Skip running the expensive algorithm tests by default - run with --run-slow
|
||||
@pytest.mark.slow
|
||||
class TestAnchorSequenceAlgorithm:
|
||||
"""
|
||||
Tests that actually run the AnchorSequenceFinder algorithm.
|
||||
These are slow and should only be run when verifying algorithm changes.
|
||||
|
||||
Run with: pytest -v -m slow --run-slow
|
||||
"""
|
||||
|
||||
@pytest.mark.parametrize("fixture_path", FIXTURE_FILES[:3], ids=lambda f: f.stem) # Only first 3 for speed
|
||||
def test_algorithm_produces_similar_count(self, fixture_path, tmp_path):
|
||||
"""Re-running algorithm should produce similar anchor count."""
|
||||
fixture, anchors = get_anchors_for_fixture(fixture_path, tmp_path)
|
||||
|
||||
expected_count = fixture["expected_anchor_count"]
|
||||
actual_count = len(anchors)
|
||||
|
||||
# Allow 30% tolerance for anchor count (algorithm may vary slightly)
|
||||
tolerance = max(expected_count * 0.3, 3)
|
||||
assert abs(actual_count - expected_count) <= tolerance, (
|
||||
f"Anchor count {actual_count} differs from expected {expected_count} "
|
||||
f"by more than {tolerance:.0f} (30% tolerance)"
|
||||
)
|
||||
|
||||
|
||||
@pytest.mark.slow
|
||||
class TestAnchorSequencePerformance:
|
||||
"""
|
||||
Performance benchmark tests for the AnchorSequenceFinder.
|
||||
|
||||
These tests verify that the algorithm completes within acceptable time limits
|
||||
for real-world song data of varying complexity.
|
||||
|
||||
Run with: pytest -v -m slow --run-slow
|
||||
"""
|
||||
|
||||
@pytest.mark.parametrize("fixture_path", FIXTURE_FILES, ids=lambda f: f.stem)
|
||||
def test_algorithm_completes_within_30_seconds(self, fixture_path, tmp_path):
|
||||
"""Each fixture should complete anchor finding in under 30 seconds."""
|
||||
import time
|
||||
|
||||
fixture = load_fixture(fixture_path)
|
||||
transcription_result = fixture_to_transcription_result(fixture)
|
||||
references = fixture_to_references(fixture)
|
||||
transcribed = get_transcribed_text(transcription_result)
|
||||
|
||||
# Create finder with 30 second timeout
|
||||
finder = AnchorSequenceFinder(
|
||||
cache_dir=str(tmp_path),
|
||||
timeout_seconds=30,
|
||||
)
|
||||
|
||||
start_time = time.time()
|
||||
anchors = finder.find_anchors(
|
||||
transcribed=transcribed,
|
||||
references=references,
|
||||
transcription_result=transcription_result,
|
||||
)
|
||||
elapsed = time.time() - start_time
|
||||
|
||||
# Assert it completed in under 30 seconds
|
||||
assert elapsed < 30, (
|
||||
f"Fixture {fixture_path.stem} took {elapsed:.1f}s, exceeding 30s limit. "
|
||||
f"Found {len(anchors)} anchors."
|
||||
)
|
||||
|
||||
# Also verify we got some anchors (sanity check)
|
||||
assert len(anchors) > 0, f"No anchors found for {fixture_path.stem}"
|
||||
|
||||
@pytest.mark.parametrize("fixture_path", FIXTURE_FILES[:3], ids=lambda f: f.stem)
|
||||
def test_algorithm_produces_similar_coverage(self, fixture_path, tmp_path):
|
||||
"""Re-running algorithm should produce similar coverage."""
|
||||
fixture, anchors = get_anchors_for_fixture(fixture_path, tmp_path)
|
||||
|
||||
word_count = fixture["transcription"]["word_count"]
|
||||
expected_coverage = fixture["expected_coverage_percent"]
|
||||
actual_coverage = calculate_coverage(anchors, word_count)
|
||||
|
||||
# Allow 15% tolerance for coverage
|
||||
tolerance = 15.0
|
||||
assert abs(actual_coverage - expected_coverage) <= tolerance, (
|
||||
f"Coverage {actual_coverage:.1f}% differs from expected {expected_coverage:.1f}% "
|
||||
f"by more than {tolerance:.1f}%"
|
||||
)
|
||||
Generated
+47
-1
@@ -2187,6 +2187,29 @@ proto-plus = [
|
||||
]
|
||||
protobuf = ">=3.20.2,<4.21.0 || >4.21.0,<4.21.1 || >4.21.1,<4.21.2 || >4.21.2,<4.21.3 || >4.21.3,<4.21.4 || >4.21.4,<4.21.5 || >4.21.5,<7.0.0dev"
|
||||
|
||||
[[package]]
|
||||
name = "google-cloud-run"
|
||||
version = "0.13.0"
|
||||
description = "Google Cloud Run API client library"
|
||||
optional = false
|
||||
python-versions = ">=3.7"
|
||||
groups = ["main"]
|
||||
files = [
|
||||
{file = "google_cloud_run-0.13.0-py3-none-any.whl", hash = "sha256:ec61a8ec2e38e04d482643ef8a6f371630c8a7d6173ab6c69c1d80ad9754220f"},
|
||||
{file = "google_cloud_run-0.13.0.tar.gz", hash = "sha256:97534ad76d3a2c2047d124c0a0a294a4822fcc24331eba0f294cadfb193c49ad"},
|
||||
]
|
||||
|
||||
[package.dependencies]
|
||||
google-api-core = {version = ">=1.34.1,<2.0.dev0 || >=2.11.dev0,<3.0.0", extras = ["grpc"]}
|
||||
google-auth = ">=2.14.1,<2.24.0 || >2.24.0,<2.25.0 || >2.25.0,<3.0.0"
|
||||
grpc-google-iam-v1 = ">=0.14.0,<1.0.0"
|
||||
grpcio = ">=1.33.2,<2.0.0"
|
||||
proto-plus = [
|
||||
{version = ">=1.25.0,<2.0.0", markers = "python_version >= \"3.13\""},
|
||||
{version = ">=1.22.3,<2.0.0"},
|
||||
]
|
||||
protobuf = ">=3.20.2,<4.21.0 || >4.21.0,<4.21.1 || >4.21.1,<4.21.2 || >4.21.2,<4.21.3 || >4.21.3,<4.21.4 || >4.21.4,<4.21.5 || >4.21.5,<7.0.0"
|
||||
|
||||
[[package]]
|
||||
name = "google-cloud-secret-manager"
|
||||
version = "2.25.0"
|
||||
@@ -2235,6 +2258,29 @@ grpc = ["google-api-core[grpc] (>=2.27.0,<3.0.0)", "grpc-google-iam-v1 (>=0.14.0
|
||||
protobuf = ["protobuf (>=3.20.2,<7.0.0)"]
|
||||
tracing = ["opentelemetry-api (>=1.1.0,<2.0.0)"]
|
||||
|
||||
[[package]]
|
||||
name = "google-cloud-tasks"
|
||||
version = "2.20.0"
|
||||
description = "Google Cloud Tasks API client library"
|
||||
optional = false
|
||||
python-versions = ">=3.7"
|
||||
groups = ["main"]
|
||||
files = [
|
||||
{file = "google_cloud_tasks-2.20.0-py3-none-any.whl", hash = "sha256:3d261a8d1f3ab90454b3e2780f484373031b0b6061506571190dc5c300768422"},
|
||||
{file = "google_cloud_tasks-2.20.0.tar.gz", hash = "sha256:733cf440a25d34fe1cdb4e6c946d173bd83a2c22c3d7b2c76a24591f3c160412"},
|
||||
]
|
||||
|
||||
[package.dependencies]
|
||||
google-api-core = {version = ">=1.34.1,<2.0.dev0 || >=2.11.dev0,<3.0.0", extras = ["grpc"]}
|
||||
google-auth = ">=2.14.1,<2.24.0 || >2.24.0,<2.25.0 || >2.25.0,<3.0.0"
|
||||
grpc-google-iam-v1 = ">=0.14.0,<1.0.0"
|
||||
grpcio = ">=1.33.2,<2.0.0"
|
||||
proto-plus = [
|
||||
{version = ">=1.25.0,<2.0.0", markers = "python_version >= \"3.13\""},
|
||||
{version = ">=1.22.3,<2.0.0"},
|
||||
]
|
||||
protobuf = ">=3.20.2,<4.21.0 || >4.21.0,<4.21.1 || >4.21.1,<4.21.2 || >4.21.2,<4.21.3 || >4.21.3,<4.21.4 || >4.21.4,<4.21.5 || >4.21.5,<7.0.0"
|
||||
|
||||
[[package]]
|
||||
name = "google-cloud-trace"
|
||||
version = "1.17.0"
|
||||
@@ -9512,4 +9558,4 @@ cffi = ["cffi (>=1.17,<2.0) ; platform_python_implementation != \"PyPy\" and pyt
|
||||
[metadata]
|
||||
lock-version = "2.1"
|
||||
python-versions = ">=3.10,<3.14"
|
||||
content-hash = "833da1a111ed1fc56eff6e363e36d7196d9dd144050d95c0aff4a301497853b5"
|
||||
content-hash = "5a3ba6b78930536884c633eaf9ba584675f519b411da87f6b7581c8deac2c2ff"
|
||||
+4
-1
@@ -1,6 +1,6 @@
|
||||
[tool.poetry]
|
||||
name = "karaoke-gen"
|
||||
version = "0.71.19"
|
||||
version = "0.71.35"
|
||||
description = "Generate karaoke videos with synchronized lyrics. Handles the entire process from downloading audio and lyrics to creating the final video with title screens."
|
||||
authors = ["Andrew Beveridge <andrew@beveridge.uk>"]
|
||||
license = "MIT"
|
||||
@@ -75,6 +75,7 @@ pyperclip = "*"
|
||||
pytest-asyncio = "*"
|
||||
ffmpeg-python = "^0.2.0"
|
||||
python-multipart = "^0.0.20"
|
||||
yt-dlp = ">=2024.0.0"
|
||||
matplotlib = ">=3"
|
||||
# Backend API dependencies
|
||||
fastapi = ">=0.104.0"
|
||||
@@ -85,6 +86,8 @@ httpx = ">=0.25.0"
|
||||
google-cloud-firestore = ">=2.14.0"
|
||||
google-cloud-storage = ">=2.14.0"
|
||||
google-cloud-secret-manager = ">=2.18.0"
|
||||
google-cloud-tasks = ">=2.16.0" # Cloud Tasks for worker coordination (Phase 1 scalability)
|
||||
google-cloud-run = ">=0.10.0" # Cloud Run Jobs for long-running video encoding (Phase 2 scalability)
|
||||
python-dotenv = ">=1.0.0"
|
||||
pydub = ">=0.25.1"
|
||||
# OpenTelemetry for distributed tracing
|
||||
|
||||
@@ -202,3 +202,437 @@ class TestFileOperations:
|
||||
"""Test setting up output paths with no inputs."""
|
||||
with pytest.raises(ValueError, match="Error: At least title or artist must be provided"):
|
||||
basic_karaoke_gen.file_handler.setup_output_paths(basic_karaoke_gen.output_dir, None, None)
|
||||
|
||||
def test_download_audio_from_fetcher_result_file_not_found(self, basic_karaoke_gen, temp_dir):
|
||||
"""Test download_audio_from_fetcher_result when file doesn't exist."""
|
||||
fake_path = os.path.join(temp_dir, "nonexistent.flac")
|
||||
output_filename = os.path.join(temp_dir, "output")
|
||||
|
||||
result = basic_karaoke_gen.file_handler.download_audio_from_fetcher_result(
|
||||
fake_path, output_filename
|
||||
)
|
||||
|
||||
assert result is None
|
||||
|
||||
|
||||
class TestDownloadVideo:
|
||||
"""Tests for download_video method."""
|
||||
|
||||
def test_download_video_yt_dlp_not_available(self, basic_karaoke_gen):
|
||||
"""Test download_video when yt-dlp is not available."""
|
||||
with patch.dict('karaoke_gen.file_handler.__dict__', {'YT_DLP_AVAILABLE': False}):
|
||||
# Re-import to apply patch
|
||||
from karaoke_gen.file_handler import FileHandler
|
||||
handler = FileHandler(
|
||||
logger=basic_karaoke_gen.logger,
|
||||
ffmpeg_base_command="ffmpeg",
|
||||
create_track_subfolders=False,
|
||||
dry_run=False,
|
||||
)
|
||||
|
||||
# Mock the YT_DLP_AVAILABLE at module level
|
||||
with patch('karaoke_gen.file_handler.YT_DLP_AVAILABLE', False):
|
||||
result = handler.download_video("https://youtube.com/watch?v=test", "/output/file")
|
||||
assert result is None
|
||||
|
||||
def test_download_video_success(self, basic_karaoke_gen, temp_dir):
|
||||
"""Test successful video download."""
|
||||
output_filename = os.path.join(temp_dir, "output")
|
||||
|
||||
# Create a mock downloaded file
|
||||
downloaded_file = f"{output_filename}.m4a"
|
||||
|
||||
mock_ydl_instance = MagicMock()
|
||||
mock_ydl_instance.__enter__ = MagicMock(return_value=mock_ydl_instance)
|
||||
mock_ydl_instance.__exit__ = MagicMock(return_value=False)
|
||||
mock_ydl_instance.extract_info.return_value = {"title": "Test Video"}
|
||||
|
||||
with patch('karaoke_gen.file_handler.YT_DLP_AVAILABLE', True):
|
||||
with patch('karaoke_gen.file_handler.yt_dlp') as mock_yt_dlp:
|
||||
mock_yt_dlp.YoutubeDL.return_value = mock_ydl_instance
|
||||
# Create the file that would be downloaded
|
||||
with open(downloaded_file, 'w') as f:
|
||||
f.write("audio content")
|
||||
|
||||
result = basic_karaoke_gen.file_handler.download_video(
|
||||
"https://youtube.com/watch?v=test",
|
||||
output_filename
|
||||
)
|
||||
|
||||
assert result == downloaded_file
|
||||
|
||||
def test_download_video_extract_info_returns_none(self, basic_karaoke_gen, temp_dir):
|
||||
"""Test download_video when extract_info returns None."""
|
||||
output_filename = os.path.join(temp_dir, "output")
|
||||
|
||||
mock_ydl_instance = MagicMock()
|
||||
mock_ydl_instance.__enter__ = MagicMock(return_value=mock_ydl_instance)
|
||||
mock_ydl_instance.__exit__ = MagicMock(return_value=False)
|
||||
mock_ydl_instance.extract_info.return_value = None
|
||||
|
||||
with patch('karaoke_gen.file_handler.YT_DLP_AVAILABLE', True):
|
||||
with patch('karaoke_gen.file_handler.yt_dlp') as mock_yt_dlp:
|
||||
mock_yt_dlp.YoutubeDL.return_value = mock_ydl_instance
|
||||
|
||||
result = basic_karaoke_gen.file_handler.download_video(
|
||||
"https://youtube.com/watch?v=test",
|
||||
output_filename
|
||||
)
|
||||
|
||||
assert result is None
|
||||
|
||||
def test_download_video_with_cookies(self, basic_karaoke_gen, temp_dir):
|
||||
"""Test download_video with cookies string."""
|
||||
output_filename = os.path.join(temp_dir, "output")
|
||||
downloaded_file = f"{output_filename}.m4a"
|
||||
|
||||
mock_ydl_instance = MagicMock()
|
||||
mock_ydl_instance.__enter__ = MagicMock(return_value=mock_ydl_instance)
|
||||
mock_ydl_instance.__exit__ = MagicMock(return_value=False)
|
||||
mock_ydl_instance.extract_info.return_value = {"title": "Test Video"}
|
||||
|
||||
with patch('karaoke_gen.file_handler.YT_DLP_AVAILABLE', True):
|
||||
with patch('karaoke_gen.file_handler.yt_dlp') as mock_yt_dlp:
|
||||
mock_yt_dlp.YoutubeDL.return_value = mock_ydl_instance
|
||||
# Create the file
|
||||
with open(downloaded_file, 'w') as f:
|
||||
f.write("audio content")
|
||||
|
||||
result = basic_karaoke_gen.file_handler.download_video(
|
||||
"https://youtube.com/watch?v=test",
|
||||
output_filename,
|
||||
cookies_str="cookie_data_here"
|
||||
)
|
||||
|
||||
assert result == downloaded_file
|
||||
|
||||
def test_download_video_download_error(self, basic_karaoke_gen, temp_dir):
|
||||
"""Test download_video when yt-dlp raises DownloadError."""
|
||||
output_filename = os.path.join(temp_dir, "output")
|
||||
|
||||
mock_ydl_instance = MagicMock()
|
||||
mock_ydl_instance.__enter__ = MagicMock(return_value=mock_ydl_instance)
|
||||
mock_ydl_instance.__exit__ = MagicMock(return_value=False)
|
||||
|
||||
with patch('karaoke_gen.file_handler.YT_DLP_AVAILABLE', True):
|
||||
with patch('karaoke_gen.file_handler.yt_dlp') as mock_yt_dlp:
|
||||
mock_yt_dlp.DownloadError = Exception
|
||||
mock_ydl_instance.extract_info.side_effect = mock_yt_dlp.DownloadError("Download failed")
|
||||
mock_yt_dlp.YoutubeDL.return_value = mock_ydl_instance
|
||||
|
||||
result = basic_karaoke_gen.file_handler.download_video(
|
||||
"https://youtube.com/watch?v=test",
|
||||
output_filename
|
||||
)
|
||||
|
||||
assert result is None
|
||||
|
||||
def test_download_video_generic_exception(self, basic_karaoke_gen, temp_dir):
|
||||
"""Test download_video when generic exception occurs."""
|
||||
output_filename = os.path.join(temp_dir, "output")
|
||||
|
||||
mock_ydl_instance = MagicMock()
|
||||
mock_ydl_instance.__enter__ = MagicMock(return_value=mock_ydl_instance)
|
||||
mock_ydl_instance.__exit__ = MagicMock(return_value=False)
|
||||
mock_ydl_instance.extract_info.side_effect = Exception("Generic error")
|
||||
|
||||
with patch('karaoke_gen.file_handler.YT_DLP_AVAILABLE', True):
|
||||
with patch('karaoke_gen.file_handler.yt_dlp') as mock_yt_dlp:
|
||||
mock_yt_dlp.DownloadError = type('DownloadError', (Exception,), {})
|
||||
mock_yt_dlp.YoutubeDL.return_value = mock_ydl_instance
|
||||
|
||||
result = basic_karaoke_gen.file_handler.download_video(
|
||||
"https://youtube.com/watch?v=test",
|
||||
output_filename
|
||||
)
|
||||
|
||||
assert result is None
|
||||
|
||||
def test_download_video_file_not_found_after_download(self, basic_karaoke_gen, temp_dir):
|
||||
"""Test download_video when downloaded file is not found."""
|
||||
output_filename = os.path.join(temp_dir, "output")
|
||||
|
||||
mock_ydl_instance = MagicMock()
|
||||
mock_ydl_instance.__enter__ = MagicMock(return_value=mock_ydl_instance)
|
||||
mock_ydl_instance.__exit__ = MagicMock(return_value=False)
|
||||
mock_ydl_instance.extract_info.return_value = {"title": "Test Video"}
|
||||
|
||||
with patch('karaoke_gen.file_handler.YT_DLP_AVAILABLE', True):
|
||||
with patch('karaoke_gen.file_handler.yt_dlp') as mock_yt_dlp:
|
||||
mock_yt_dlp.YoutubeDL.return_value = mock_ydl_instance
|
||||
# Don't create the file - simulate download failure
|
||||
|
||||
result = basic_karaoke_gen.file_handler.download_video(
|
||||
"https://youtube.com/watch?v=test",
|
||||
output_filename
|
||||
)
|
||||
|
||||
assert result is None
|
||||
|
||||
|
||||
class TestExtractMetadataFromUrl:
|
||||
"""Tests for extract_metadata_from_url method."""
|
||||
|
||||
def test_extract_metadata_yt_dlp_not_available(self, basic_karaoke_gen):
|
||||
"""Test extract_metadata_from_url when yt-dlp is not available."""
|
||||
with patch('karaoke_gen.file_handler.YT_DLP_AVAILABLE', False):
|
||||
result = basic_karaoke_gen.file_handler.extract_metadata_from_url(
|
||||
"https://youtube.com/watch?v=test"
|
||||
)
|
||||
assert result is None
|
||||
|
||||
def test_extract_metadata_success_with_dash_format(self, basic_karaoke_gen):
|
||||
"""Test extracting metadata with 'Artist - Title' format."""
|
||||
mock_ydl_instance = MagicMock()
|
||||
mock_ydl_instance.__enter__ = MagicMock(return_value=mock_ydl_instance)
|
||||
mock_ydl_instance.__exit__ = MagicMock(return_value=False)
|
||||
mock_ydl_instance.extract_info.return_value = {
|
||||
"title": "Test Artist - Test Song",
|
||||
"uploader": "Test Channel",
|
||||
"duration": 240,
|
||||
}
|
||||
|
||||
with patch('karaoke_gen.file_handler.YT_DLP_AVAILABLE', True):
|
||||
with patch('karaoke_gen.file_handler.yt_dlp') as mock_yt_dlp:
|
||||
mock_yt_dlp.YoutubeDL.return_value = mock_ydl_instance
|
||||
|
||||
result = basic_karaoke_gen.file_handler.extract_metadata_from_url(
|
||||
"https://youtube.com/watch?v=test"
|
||||
)
|
||||
|
||||
assert result is not None
|
||||
assert result['artist'] == "Test Artist"
|
||||
assert result['title'] == "Test Song"
|
||||
assert result['duration'] == 240
|
||||
|
||||
def test_extract_metadata_success_without_dash(self, basic_karaoke_gen):
|
||||
"""Test extracting metadata without 'Artist - Title' format."""
|
||||
mock_ydl_instance = MagicMock()
|
||||
mock_ydl_instance.__enter__ = MagicMock(return_value=mock_ydl_instance)
|
||||
mock_ydl_instance.__exit__ = MagicMock(return_value=False)
|
||||
mock_ydl_instance.extract_info.return_value = {
|
||||
"title": "Just A Song Title",
|
||||
"uploader": "The Artist",
|
||||
"duration": 180,
|
||||
}
|
||||
|
||||
with patch('karaoke_gen.file_handler.YT_DLP_AVAILABLE', True):
|
||||
with patch('karaoke_gen.file_handler.yt_dlp') as mock_yt_dlp:
|
||||
mock_yt_dlp.YoutubeDL.return_value = mock_ydl_instance
|
||||
|
||||
result = basic_karaoke_gen.file_handler.extract_metadata_from_url(
|
||||
"https://youtube.com/watch?v=test"
|
||||
)
|
||||
|
||||
assert result is not None
|
||||
assert result['artist'] == "The Artist"
|
||||
assert result['title'] == "Just A Song Title"
|
||||
|
||||
def test_extract_metadata_cleans_title_suffixes(self, basic_karaoke_gen):
|
||||
"""Test that common suffixes are cleaned from title."""
|
||||
mock_ydl_instance = MagicMock()
|
||||
mock_ydl_instance.__enter__ = MagicMock(return_value=mock_ydl_instance)
|
||||
mock_ydl_instance.__exit__ = MagicMock(return_value=False)
|
||||
mock_ydl_instance.extract_info.return_value = {
|
||||
"title": "Artist - Song Title (Official Video)",
|
||||
"uploader": "VEVO",
|
||||
"duration": 200,
|
||||
}
|
||||
|
||||
with patch('karaoke_gen.file_handler.YT_DLP_AVAILABLE', True):
|
||||
with patch('karaoke_gen.file_handler.yt_dlp') as mock_yt_dlp:
|
||||
mock_yt_dlp.YoutubeDL.return_value = mock_ydl_instance
|
||||
|
||||
result = basic_karaoke_gen.file_handler.extract_metadata_from_url(
|
||||
"https://youtube.com/watch?v=test"
|
||||
)
|
||||
|
||||
assert result is not None
|
||||
assert result['title'] == "Song Title"
|
||||
|
||||
def test_extract_metadata_cleans_multiple_suffixes(self, basic_karaoke_gen):
|
||||
"""Test cleaning multiple suffixes from title."""
|
||||
test_titles = [
|
||||
("Song (Official Music Video)", "Song"),
|
||||
("Song (Official Audio)", "Song"),
|
||||
("Song (Lyric Video)", "Song"),
|
||||
("Song (HD)", "Song"),
|
||||
("Song (4K)", "Song"),
|
||||
("Song (Remastered)", "Song"),
|
||||
("Song | Official Video", "Song"),
|
||||
("Song [Official Video]", "Song"),
|
||||
]
|
||||
|
||||
for input_title, expected_title in test_titles:
|
||||
mock_ydl_instance = MagicMock()
|
||||
mock_ydl_instance.__enter__ = MagicMock(return_value=mock_ydl_instance)
|
||||
mock_ydl_instance.__exit__ = MagicMock(return_value=False)
|
||||
mock_ydl_instance.extract_info.return_value = {
|
||||
"title": f"Artist - {input_title}",
|
||||
"uploader": "Channel",
|
||||
"duration": 200,
|
||||
}
|
||||
|
||||
with patch('karaoke_gen.file_handler.YT_DLP_AVAILABLE', True):
|
||||
with patch('karaoke_gen.file_handler.yt_dlp') as mock_yt_dlp:
|
||||
mock_yt_dlp.YoutubeDL.return_value = mock_ydl_instance
|
||||
|
||||
result = basic_karaoke_gen.file_handler.extract_metadata_from_url(
|
||||
"https://youtube.com/watch?v=test"
|
||||
)
|
||||
|
||||
assert result['title'] == expected_title, f"Failed for input: {input_title}"
|
||||
|
||||
def test_extract_metadata_returns_none_on_error(self, basic_karaoke_gen):
|
||||
"""Test that None is returned on extraction error."""
|
||||
mock_ydl_instance = MagicMock()
|
||||
mock_ydl_instance.__enter__ = MagicMock(return_value=mock_ydl_instance)
|
||||
mock_ydl_instance.__exit__ = MagicMock(return_value=False)
|
||||
mock_ydl_instance.extract_info.side_effect = Exception("Network error")
|
||||
|
||||
with patch('karaoke_gen.file_handler.YT_DLP_AVAILABLE', True):
|
||||
with patch('karaoke_gen.file_handler.yt_dlp') as mock_yt_dlp:
|
||||
mock_yt_dlp.YoutubeDL.return_value = mock_ydl_instance
|
||||
|
||||
result = basic_karaoke_gen.file_handler.extract_metadata_from_url(
|
||||
"https://youtube.com/watch?v=test"
|
||||
)
|
||||
|
||||
assert result is None
|
||||
|
||||
def test_extract_metadata_returns_none_when_info_is_none(self, basic_karaoke_gen):
|
||||
"""Test that None is returned when extract_info returns None."""
|
||||
mock_ydl_instance = MagicMock()
|
||||
mock_ydl_instance.__enter__ = MagicMock(return_value=mock_ydl_instance)
|
||||
mock_ydl_instance.__exit__ = MagicMock(return_value=False)
|
||||
mock_ydl_instance.extract_info.return_value = None
|
||||
|
||||
with patch('karaoke_gen.file_handler.YT_DLP_AVAILABLE', True):
|
||||
with patch('karaoke_gen.file_handler.yt_dlp') as mock_yt_dlp:
|
||||
mock_yt_dlp.YoutubeDL.return_value = mock_ydl_instance
|
||||
|
||||
result = basic_karaoke_gen.file_handler.extract_metadata_from_url(
|
||||
"https://youtube.com/watch?v=test"
|
||||
)
|
||||
|
||||
assert result is None
|
||||
|
||||
def test_extract_metadata_uses_channel_as_fallback(self, basic_karaoke_gen):
|
||||
"""Test that channel is used as artist when uploader is not available."""
|
||||
mock_ydl_instance = MagicMock()
|
||||
mock_ydl_instance.__enter__ = MagicMock(return_value=mock_ydl_instance)
|
||||
mock_ydl_instance.__exit__ = MagicMock(return_value=False)
|
||||
mock_ydl_instance.extract_info.return_value = {
|
||||
"title": "Song Title",
|
||||
"channel": "The Channel",
|
||||
"duration": 200,
|
||||
}
|
||||
|
||||
with patch('karaoke_gen.file_handler.YT_DLP_AVAILABLE', True):
|
||||
with patch('karaoke_gen.file_handler.yt_dlp') as mock_yt_dlp:
|
||||
mock_yt_dlp.YoutubeDL.return_value = mock_ydl_instance
|
||||
|
||||
result = basic_karaoke_gen.file_handler.extract_metadata_from_url(
|
||||
"https://youtube.com/watch?v=test"
|
||||
)
|
||||
|
||||
assert result['artist'] == "The Channel"
|
||||
|
||||
|
||||
class TestBackupExistingOutputs:
|
||||
"""Tests for backup_existing_outputs method."""
|
||||
|
||||
def test_backup_existing_outputs_creates_version_dir(self, basic_karaoke_gen, temp_dir):
|
||||
"""Test that backup creates a versioned directory."""
|
||||
# Setup artist and title
|
||||
artist = "Test Artist"
|
||||
title = "Test Song"
|
||||
|
||||
# Create the necessary files
|
||||
base_name = f"{artist} - {title}"
|
||||
wav_file = os.path.join(temp_dir, f"{base_name}.wav")
|
||||
with open(wav_file, 'w') as f:
|
||||
f.write("audio content")
|
||||
|
||||
# Create a file to be moved
|
||||
karaoke_file = os.path.join(temp_dir, f"{base_name} (Karaoke).mp4")
|
||||
with open(karaoke_file, 'w') as f:
|
||||
f.write("video content")
|
||||
|
||||
result = basic_karaoke_gen.file_handler.backup_existing_outputs(
|
||||
temp_dir, artist, title
|
||||
)
|
||||
|
||||
# Check that version directory was created
|
||||
version_dir = os.path.join(temp_dir, "version-1")
|
||||
assert os.path.exists(version_dir)
|
||||
|
||||
# Check that the file was moved
|
||||
moved_file = os.path.join(version_dir, f"{base_name} (Karaoke).mp4")
|
||||
assert os.path.exists(moved_file)
|
||||
assert not os.path.exists(karaoke_file)
|
||||
|
||||
# Check that the WAV file path was returned
|
||||
assert result == wav_file
|
||||
|
||||
def test_backup_existing_outputs_increments_version(self, basic_karaoke_gen, temp_dir):
|
||||
"""Test that backup increments version number."""
|
||||
artist = "Test Artist"
|
||||
title = "Test Song"
|
||||
|
||||
# Create existing version directories
|
||||
os.makedirs(os.path.join(temp_dir, "version-1"))
|
||||
os.makedirs(os.path.join(temp_dir, "version-2"))
|
||||
|
||||
# Create the WAV file
|
||||
base_name = f"{artist} - {title}"
|
||||
wav_file = os.path.join(temp_dir, f"{base_name}.wav")
|
||||
with open(wav_file, 'w') as f:
|
||||
f.write("audio content")
|
||||
|
||||
basic_karaoke_gen.file_handler.backup_existing_outputs(
|
||||
temp_dir, artist, title
|
||||
)
|
||||
|
||||
# Check that version-3 was created
|
||||
assert os.path.exists(os.path.join(temp_dir, "version-3"))
|
||||
|
||||
def test_backup_existing_outputs_raises_on_missing_wav(self, basic_karaoke_gen, temp_dir):
|
||||
"""Test that exception is raised when no WAV file is found."""
|
||||
artist = "Test Artist"
|
||||
title = "Test Song"
|
||||
|
||||
with pytest.raises(Exception, match="No input audio file found"):
|
||||
basic_karaoke_gen.file_handler.backup_existing_outputs(
|
||||
temp_dir, artist, title
|
||||
)
|
||||
|
||||
def test_backup_existing_outputs_backs_up_lyrics_dir(self, basic_karaoke_gen, temp_dir):
|
||||
"""Test that lyrics directory is backed up."""
|
||||
artist = "Test Artist"
|
||||
title = "Test Song"
|
||||
|
||||
# Create WAV file
|
||||
base_name = f"{artist} - {title}"
|
||||
wav_file = os.path.join(temp_dir, f"{base_name}.wav")
|
||||
with open(wav_file, 'w') as f:
|
||||
f.write("audio content")
|
||||
|
||||
# Create lyrics directory with a file
|
||||
lyrics_dir = os.path.join(temp_dir, "lyrics")
|
||||
os.makedirs(lyrics_dir)
|
||||
lyrics_file = os.path.join(lyrics_dir, "lyrics.txt")
|
||||
with open(lyrics_file, 'w') as f:
|
||||
f.write("lyrics content")
|
||||
|
||||
basic_karaoke_gen.file_handler.backup_existing_outputs(
|
||||
temp_dir, artist, title
|
||||
)
|
||||
|
||||
# Check that lyrics were backed up
|
||||
version_dir = os.path.join(temp_dir, "version-1")
|
||||
backed_up_lyrics = os.path.join(version_dir, "lyrics", "lyrics.txt")
|
||||
assert os.path.exists(backed_up_lyrics)
|
||||
|
||||
# Original lyrics directory should be removed
|
||||
assert not os.path.exists(lyrics_dir)
|
||||
@@ -337,6 +337,256 @@ class TestRemoteExecutor:
|
||||
assert result["status"] == "complete"
|
||||
assert result["progress"] == 100
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_run_pipeline_success(self):
|
||||
"""Test run_pipeline with successful job completion."""
|
||||
from karaoke_gen.pipeline.executors.remote import RemoteExecutor
|
||||
from karaoke_gen.pipeline.base import PipelineStage, StageResult, StageStatus
|
||||
from karaoke_gen.pipeline.context import PipelineContext
|
||||
|
||||
class TestStage(PipelineStage):
|
||||
@property
|
||||
def name(self):
|
||||
return "test"
|
||||
|
||||
async def execute(self, context):
|
||||
return StageResult(status=StageStatus.COMPLETED)
|
||||
|
||||
executor = RemoteExecutor(service_url="https://api.example.com")
|
||||
context = PipelineContext(
|
||||
job_id="test",
|
||||
artist="Artist",
|
||||
title="Title",
|
||||
input_audio_path="/audio.flac",
|
||||
output_dir="/output",
|
||||
)
|
||||
|
||||
# Mock _submit_job and _monitor_job
|
||||
with patch.object(executor, '_submit_job', new_callable=AsyncMock, return_value="job-123"):
|
||||
with patch.object(executor, '_monitor_job', new_callable=AsyncMock, return_value="complete"):
|
||||
results = await executor.run_pipeline([TestStage()], context)
|
||||
|
||||
assert "test" in results
|
||||
assert results["test"].status == StageStatus.COMPLETED
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_run_pipeline_failure(self):
|
||||
"""Test run_pipeline with job failure."""
|
||||
from karaoke_gen.pipeline.executors.remote import RemoteExecutor
|
||||
from karaoke_gen.pipeline.base import PipelineStage, StageResult, StageStatus
|
||||
from karaoke_gen.pipeline.context import PipelineContext
|
||||
|
||||
class TestStage(PipelineStage):
|
||||
@property
|
||||
def name(self):
|
||||
return "test"
|
||||
|
||||
async def execute(self, context):
|
||||
return StageResult(status=StageStatus.COMPLETED)
|
||||
|
||||
executor = RemoteExecutor(service_url="https://api.example.com")
|
||||
context = PipelineContext(
|
||||
job_id="test",
|
||||
artist="Artist",
|
||||
title="Title",
|
||||
input_audio_path="/audio.flac",
|
||||
output_dir="/output",
|
||||
)
|
||||
|
||||
# Mock _submit_job and _monitor_job to simulate failure
|
||||
with patch.object(executor, '_submit_job', new_callable=AsyncMock, return_value="job-123"):
|
||||
with patch.object(executor, '_monitor_job', new_callable=AsyncMock, return_value="failed"):
|
||||
results = await executor.run_pipeline([TestStage()], context)
|
||||
|
||||
assert "test" in results
|
||||
assert results["test"].status == StageStatus.FAILED
|
||||
assert "failed" in results["test"].error_message
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_run_pipeline_exception(self):
|
||||
"""Test run_pipeline handles exceptions."""
|
||||
from karaoke_gen.pipeline.executors.remote import RemoteExecutor
|
||||
from karaoke_gen.pipeline.base import PipelineStage, StageResult, StageStatus
|
||||
from karaoke_gen.pipeline.context import PipelineContext
|
||||
|
||||
class TestStage(PipelineStage):
|
||||
@property
|
||||
def name(self):
|
||||
return "test"
|
||||
|
||||
async def execute(self, context):
|
||||
return StageResult(status=StageStatus.COMPLETED)
|
||||
|
||||
executor = RemoteExecutor(service_url="https://api.example.com")
|
||||
context = PipelineContext(
|
||||
job_id="test",
|
||||
artist="Artist",
|
||||
title="Title",
|
||||
input_audio_path="/audio.flac",
|
||||
output_dir="/output",
|
||||
)
|
||||
|
||||
# Mock _submit_job to raise exception
|
||||
with patch.object(executor, '_submit_job', new_callable=AsyncMock, side_effect=Exception("Connection error")):
|
||||
results = await executor.run_pipeline([TestStage()], context)
|
||||
|
||||
assert "test" in results
|
||||
assert results["test"].status == StageStatus.FAILED
|
||||
assert "Connection error" in results["test"].error_message
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_submit_job(self):
|
||||
"""Test _submit_job method."""
|
||||
from karaoke_gen.pipeline.executors.remote import RemoteExecutor
|
||||
from karaoke_gen.pipeline.context import PipelineContext
|
||||
import tempfile
|
||||
import os
|
||||
|
||||
executor = RemoteExecutor(service_url="https://api.example.com")
|
||||
|
||||
# Create a temporary audio file
|
||||
with tempfile.NamedTemporaryFile(suffix=".flac", delete=False) as f:
|
||||
f.write(b"fake audio data")
|
||||
audio_path = f.name
|
||||
|
||||
try:
|
||||
context = PipelineContext(
|
||||
job_id="test",
|
||||
artist="Artist",
|
||||
title="Title",
|
||||
input_audio_path=audio_path,
|
||||
output_dir="/output",
|
||||
enable_cdg=True,
|
||||
enable_txt=True,
|
||||
)
|
||||
|
||||
mock_response = MagicMock()
|
||||
mock_response.json.return_value = {"job_id": "job-123"}
|
||||
|
||||
mock_session = MagicMock()
|
||||
mock_session.post.return_value = mock_response
|
||||
executor._session = mock_session
|
||||
|
||||
result = await executor._submit_job(context)
|
||||
|
||||
assert result == "job-123"
|
||||
mock_session.post.assert_called_once()
|
||||
finally:
|
||||
os.unlink(audio_path)
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_submit_job_with_optional_params(self):
|
||||
"""Test _submit_job with optional parameters."""
|
||||
from karaoke_gen.pipeline.executors.remote import RemoteExecutor
|
||||
from karaoke_gen.pipeline.context import PipelineContext
|
||||
import tempfile
|
||||
import os
|
||||
|
||||
executor = RemoteExecutor(service_url="https://api.example.com")
|
||||
|
||||
# Create a temporary audio file
|
||||
with tempfile.NamedTemporaryFile(suffix=".flac", delete=False) as f:
|
||||
f.write(b"fake audio data")
|
||||
audio_path = f.name
|
||||
|
||||
try:
|
||||
context = PipelineContext(
|
||||
job_id="test",
|
||||
artist="Artist",
|
||||
title="Title",
|
||||
input_audio_path=audio_path,
|
||||
output_dir="/output",
|
||||
brand_prefix="TestBrand",
|
||||
discord_webhook_url="https://discord.com/webhook",
|
||||
enable_youtube_upload=True,
|
||||
dropbox_path="/dropbox/path",
|
||||
gdrive_folder_id="folder123",
|
||||
)
|
||||
|
||||
mock_response = MagicMock()
|
||||
mock_response.json.return_value = {"job_id": "job-456"}
|
||||
|
||||
mock_session = MagicMock()
|
||||
mock_session.post.return_value = mock_response
|
||||
executor._session = mock_session
|
||||
|
||||
result = await executor._submit_job(context)
|
||||
|
||||
assert result == "job-456"
|
||||
# Check that optional params were included
|
||||
call_args = mock_session.post.call_args
|
||||
data = call_args.kwargs.get('data') or call_args[1].get('data')
|
||||
assert data['brand_prefix'] == "TestBrand"
|
||||
finally:
|
||||
os.unlink(audio_path)
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_monitor_job_complete(self):
|
||||
"""Test _monitor_job until job completes."""
|
||||
from karaoke_gen.pipeline.executors.remote import RemoteExecutor
|
||||
from karaoke_gen.pipeline.context import PipelineContext
|
||||
|
||||
executor = RemoteExecutor(service_url="https://api.example.com", poll_interval=0.01)
|
||||
context = PipelineContext(
|
||||
job_id="test",
|
||||
artist="Artist",
|
||||
title="Title",
|
||||
input_audio_path="/audio.flac",
|
||||
output_dir="/output",
|
||||
)
|
||||
|
||||
# Simulate job progression
|
||||
call_count = [0]
|
||||
def mock_get(*args, **kwargs):
|
||||
call_count[0] += 1
|
||||
mock_response = MagicMock()
|
||||
if call_count[0] < 3:
|
||||
mock_response.json.return_value = {"status": "processing", "progress": 50}
|
||||
else:
|
||||
mock_response.json.return_value = {"status": "complete", "progress": 100}
|
||||
return mock_response
|
||||
|
||||
mock_session = MagicMock()
|
||||
mock_session.get.side_effect = mock_get
|
||||
executor._session = mock_session
|
||||
|
||||
result = await executor._monitor_job("job-123", context)
|
||||
|
||||
assert result == "complete"
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_monitor_job_handles_error(self):
|
||||
"""Test _monitor_job handles polling errors gracefully."""
|
||||
from karaoke_gen.pipeline.executors.remote import RemoteExecutor
|
||||
from karaoke_gen.pipeline.context import PipelineContext
|
||||
|
||||
executor = RemoteExecutor(service_url="https://api.example.com", poll_interval=0.01)
|
||||
context = PipelineContext(
|
||||
job_id="test",
|
||||
artist="Artist",
|
||||
title="Title",
|
||||
input_audio_path="/audio.flac",
|
||||
output_dir="/output",
|
||||
)
|
||||
|
||||
# Simulate error then success
|
||||
call_count = [0]
|
||||
def mock_get(*args, **kwargs):
|
||||
call_count[0] += 1
|
||||
if call_count[0] == 1:
|
||||
raise Exception("Network error")
|
||||
mock_response = MagicMock()
|
||||
mock_response.json.return_value = {"status": "complete", "progress": 100}
|
||||
return mock_response
|
||||
|
||||
mock_session = MagicMock()
|
||||
mock_session.get.side_effect = mock_get
|
||||
executor._session = mock_session
|
||||
|
||||
result = await executor._monitor_job("job-123", context)
|
||||
|
||||
assert result == "complete"
|
||||
|
||||
|
||||
class TestExecutorModuleExports:
|
||||
"""Tests for module exports."""
|
||||
|
||||
+342
-24
@@ -549,44 +549,94 @@ class TestRemoteKaraokeClient:
|
||||
client.submit_job(str(test_file), "Artist", "Title")
|
||||
|
||||
@patch.object(RemoteKaraokeClient, '_request')
|
||||
def test_submit_job_success(self, mock_request, client, tmp_path):
|
||||
"""Test successful job submission."""
|
||||
@patch.object(RemoteKaraokeClient, '_upload_file_to_signed_url')
|
||||
def test_submit_job_success(self, mock_upload, mock_request, client, tmp_path):
|
||||
"""Test successful job submission via signed URL flow."""
|
||||
# Create test audio file
|
||||
audio_file = tmp_path / "test.mp3"
|
||||
audio_file.write_bytes(b"audio data")
|
||||
|
||||
mock_response = MagicMock()
|
||||
mock_response.status_code = 200
|
||||
mock_response.json.return_value = {
|
||||
|
||||
# Mock create job response (first API call)
|
||||
create_response = MagicMock()
|
||||
create_response.status_code = 200
|
||||
create_response.json.return_value = {
|
||||
"status": "success",
|
||||
"job_id": "new-job-123"
|
||||
"job_id": "new-job-123",
|
||||
"upload_urls": [
|
||||
{
|
||||
"file_type": "audio",
|
||||
"gcs_path": "uploads/new-job-123/audio/test.mp3",
|
||||
"upload_url": "https://storage.googleapis.com/signed-url",
|
||||
"content_type": "audio/mpeg"
|
||||
}
|
||||
]
|
||||
}
|
||||
mock_request.return_value = mock_response
|
||||
|
||||
# Mock uploads complete response (second API call)
|
||||
complete_response = MagicMock()
|
||||
complete_response.status_code = 200
|
||||
complete_response.json.return_value = {
|
||||
"status": "success",
|
||||
"job_id": "new-job-123",
|
||||
"message": "Processing started"
|
||||
}
|
||||
|
||||
mock_request.side_effect = [create_response, complete_response]
|
||||
mock_upload.return_value = True
|
||||
|
||||
result = client.submit_job(str(audio_file), "Test Artist", "Test Title")
|
||||
|
||||
|
||||
assert result["status"] == "success"
|
||||
assert result["job_id"] == "new-job-123"
|
||||
|
||||
# Verify the signed URL upload was called
|
||||
mock_upload.assert_called_once()
|
||||
|
||||
@patch.object(RemoteKaraokeClient, '_upload_file_to_signed_url')
|
||||
@patch.object(RemoteKaraokeClient, '_request')
|
||||
def test_submit_job_with_style_params(self, mock_request, client, tmp_path):
|
||||
"""Test job submission with style params."""
|
||||
def test_submit_job_with_style_params(self, mock_request, mock_upload, client, tmp_path):
|
||||
"""Test job submission with style params via signed URL flow."""
|
||||
# Create test files
|
||||
audio_file = tmp_path / "test.mp3"
|
||||
audio_file.write_bytes(b"audio data")
|
||||
|
||||
|
||||
style_file = tmp_path / "style_params.json"
|
||||
style_file.write_text(json.dumps({"intro": {}}))
|
||||
|
||||
mock_response = MagicMock()
|
||||
mock_response.status_code = 200
|
||||
mock_response.json.return_value = {
|
||||
|
||||
# Mock create job response (first API call)
|
||||
create_response = MagicMock()
|
||||
create_response.status_code = 200
|
||||
create_response.json.return_value = {
|
||||
"status": "success",
|
||||
"job_id": "new-job-456",
|
||||
"style_assets_uploaded": []
|
||||
"upload_urls": [
|
||||
{
|
||||
"file_type": "audio",
|
||||
"gcs_path": "uploads/new-job-456/audio/test.mp3",
|
||||
"upload_url": "https://storage.googleapis.com/signed-url-1",
|
||||
"content_type": "audio/mpeg"
|
||||
},
|
||||
{
|
||||
"file_type": "style_params",
|
||||
"gcs_path": "uploads/new-job-456/style/style_params.json",
|
||||
"upload_url": "https://storage.googleapis.com/signed-url-2",
|
||||
"content_type": "application/json"
|
||||
}
|
||||
]
|
||||
}
|
||||
mock_request.return_value = mock_response
|
||||
|
||||
# Mock uploads complete response (second API call)
|
||||
complete_response = MagicMock()
|
||||
complete_response.status_code = 200
|
||||
complete_response.json.return_value = {
|
||||
"status": "success",
|
||||
"job_id": "new-job-456",
|
||||
"message": "Processing started",
|
||||
"style_assets": ["style_params"]
|
||||
}
|
||||
|
||||
mock_request.side_effect = [create_response, complete_response]
|
||||
mock_upload.return_value = True
|
||||
|
||||
result = client.submit_job(
|
||||
str(audio_file),
|
||||
"Test Artist",
|
||||
@@ -597,21 +647,128 @@ class TestRemoteKaraokeClient:
|
||||
brand_prefix="TEST",
|
||||
discord_webhook_url="https://discord.webhook/url"
|
||||
)
|
||||
|
||||
|
||||
assert result["status"] == "success"
|
||||
# Verify uploads were called for both files
|
||||
assert mock_upload.call_count == 2
|
||||
|
||||
@patch.object(RemoteKaraokeClient, '_upload_file_to_signed_url')
|
||||
@patch.object(RemoteKaraokeClient, '_request')
|
||||
def test_submit_job_api_error(self, mock_request, client, tmp_path):
|
||||
"""Test handling API error during job submission."""
|
||||
def test_submit_job_with_existing_instrumental(self, mock_request, mock_upload, client, tmp_path):
|
||||
"""Test job submission with existing instrumental via signed URL flow (Batch 3)."""
|
||||
# Create test files
|
||||
audio_file = tmp_path / "test.mp3"
|
||||
audio_file.write_bytes(b"audio data")
|
||||
|
||||
instrumental_file = tmp_path / "instrumental.flac"
|
||||
instrumental_file.write_bytes(b"instrumental data")
|
||||
|
||||
# Mock create job response (first API call)
|
||||
create_response = MagicMock()
|
||||
create_response.status_code = 200
|
||||
create_response.json.return_value = {
|
||||
"status": "success",
|
||||
"job_id": "job-with-instrumental-123",
|
||||
"upload_urls": [
|
||||
{
|
||||
"file_type": "audio",
|
||||
"gcs_path": "uploads/job-with-instrumental-123/audio/test.mp3",
|
||||
"upload_url": "https://storage.googleapis.com/signed-url-audio",
|
||||
"content_type": "audio/mpeg"
|
||||
},
|
||||
{
|
||||
"file_type": "existing_instrumental",
|
||||
"gcs_path": "uploads/job-with-instrumental-123/audio/existing_instrumental.flac",
|
||||
"upload_url": "https://storage.googleapis.com/signed-url-instrumental",
|
||||
"content_type": "audio/flac"
|
||||
}
|
||||
]
|
||||
}
|
||||
|
||||
# Mock uploads complete response (second API call)
|
||||
complete_response = MagicMock()
|
||||
complete_response.status_code = 200
|
||||
complete_response.json.return_value = {
|
||||
"status": "success",
|
||||
"job_id": "job-with-instrumental-123",
|
||||
"message": "Processing started"
|
||||
}
|
||||
|
||||
mock_request.side_effect = [create_response, complete_response]
|
||||
mock_upload.return_value = True
|
||||
|
||||
result = client.submit_job(
|
||||
str(audio_file),
|
||||
"Test Artist",
|
||||
"Test Title",
|
||||
existing_instrumental=str(instrumental_file)
|
||||
)
|
||||
|
||||
assert result["status"] == "success"
|
||||
assert result["job_id"] == "job-with-instrumental-123"
|
||||
# Verify uploads were called for both audio and instrumental
|
||||
assert mock_upload.call_count == 2
|
||||
|
||||
@patch.object(RemoteKaraokeClient, '_upload_file_to_signed_url')
|
||||
@patch.object(RemoteKaraokeClient, '_request')
|
||||
def test_submit_job_existing_instrumental_nonexistent_file(self, mock_request, mock_upload, client, tmp_path):
|
||||
"""Test that nonexistent instrumental file is silently ignored."""
|
||||
# Create test audio file
|
||||
audio_file = tmp_path / "test.mp3"
|
||||
audio_file.write_bytes(b"audio data")
|
||||
|
||||
# Mock create job response (instrumental not included since file doesn't exist)
|
||||
create_response = MagicMock()
|
||||
create_response.status_code = 200
|
||||
create_response.json.return_value = {
|
||||
"status": "success",
|
||||
"job_id": "job-no-instrumental-456",
|
||||
"upload_urls": [
|
||||
{
|
||||
"file_type": "audio",
|
||||
"gcs_path": "uploads/job-no-instrumental-456/audio/test.mp3",
|
||||
"upload_url": "https://storage.googleapis.com/signed-url-audio",
|
||||
"content_type": "audio/mpeg"
|
||||
}
|
||||
]
|
||||
}
|
||||
|
||||
# Mock uploads complete response
|
||||
complete_response = MagicMock()
|
||||
complete_response.status_code = 200
|
||||
complete_response.json.return_value = {
|
||||
"status": "success",
|
||||
"job_id": "job-no-instrumental-456",
|
||||
"message": "Processing started"
|
||||
}
|
||||
|
||||
mock_request.side_effect = [create_response, complete_response]
|
||||
mock_upload.return_value = True
|
||||
|
||||
# Pass nonexistent instrumental path - should be silently ignored
|
||||
result = client.submit_job(
|
||||
str(audio_file),
|
||||
"Test Artist",
|
||||
"Test Title",
|
||||
existing_instrumental="/nonexistent/instrumental.flac"
|
||||
)
|
||||
|
||||
assert result["status"] == "success"
|
||||
# Only audio should be uploaded (instrumental file doesn't exist)
|
||||
assert mock_upload.call_count == 1
|
||||
|
||||
@patch.object(RemoteKaraokeClient, '_request')
|
||||
def test_submit_job_api_error(self, mock_request, client, tmp_path):
|
||||
"""Test handling API error during job creation."""
|
||||
audio_file = tmp_path / "test.mp3"
|
||||
audio_file.write_bytes(b"audio data")
|
||||
|
||||
mock_response = MagicMock()
|
||||
mock_response.status_code = 500
|
||||
mock_response.json.return_value = {"error": "Server error"}
|
||||
mock_request.return_value = mock_response
|
||||
|
||||
with pytest.raises(RuntimeError, match="Error submitting job"):
|
||||
|
||||
with pytest.raises(RuntimeError, match="Error creating job"):
|
||||
client.submit_job(str(audio_file), "Artist", "Title")
|
||||
|
||||
|
||||
@@ -1717,3 +1874,164 @@ class TestJobMonitorHandleReview:
|
||||
|
||||
mock_open_ui.assert_called_once_with("job-123")
|
||||
assert mock_get_job.call_count == 2
|
||||
|
||||
|
||||
class TestSignedUrlUploadFlow:
|
||||
"""Tests for the signed URL upload flow."""
|
||||
|
||||
@pytest.fixture
|
||||
def config(self):
|
||||
return Config(
|
||||
service_url="https://api.example.com",
|
||||
review_ui_url="https://review.example.com",
|
||||
poll_interval=5,
|
||||
output_dir="/tmp/output"
|
||||
)
|
||||
|
||||
@pytest.fixture
|
||||
def logger(self):
|
||||
return logging.getLogger("test")
|
||||
|
||||
@pytest.fixture
|
||||
def client(self, config, logger):
|
||||
return RemoteKaraokeClient(config, logger)
|
||||
|
||||
def test_get_content_type_audio(self, client):
|
||||
"""Test content type detection for audio files."""
|
||||
assert client._get_content_type("song.flac") == "audio/flac"
|
||||
assert client._get_content_type("song.mp3") == "audio/mpeg"
|
||||
assert client._get_content_type("song.wav") == "audio/wav"
|
||||
assert client._get_content_type("song.m4a") == "audio/mp4"
|
||||
assert client._get_content_type("song.ogg") == "audio/ogg"
|
||||
assert client._get_content_type("song.aac") == "audio/aac"
|
||||
|
||||
def test_get_content_type_images(self, client):
|
||||
"""Test content type detection for image files."""
|
||||
assert client._get_content_type("bg.png") == "image/png"
|
||||
assert client._get_content_type("bg.jpg") == "image/jpeg"
|
||||
assert client._get_content_type("bg.jpeg") == "image/jpeg"
|
||||
assert client._get_content_type("bg.gif") == "image/gif"
|
||||
assert client._get_content_type("bg.webp") == "image/webp"
|
||||
|
||||
def test_get_content_type_fonts(self, client):
|
||||
"""Test content type detection for font files."""
|
||||
assert client._get_content_type("font.ttf") == "font/ttf"
|
||||
assert client._get_content_type("font.otf") == "font/otf"
|
||||
assert client._get_content_type("font.woff") == "font/woff"
|
||||
assert client._get_content_type("font.woff2") == "font/woff2"
|
||||
|
||||
def test_get_content_type_other(self, client):
|
||||
"""Test content type detection for other files."""
|
||||
assert client._get_content_type("style.json") == "application/json"
|
||||
assert client._get_content_type("lyrics.txt") == "text/plain"
|
||||
assert client._get_content_type("unknown.xyz") == "application/octet-stream"
|
||||
|
||||
@patch('requests.put')
|
||||
def test_upload_file_to_signed_url_success(self, mock_put, client):
|
||||
"""Test successful upload to signed URL."""
|
||||
mock_put.return_value = MagicMock(status_code=200)
|
||||
|
||||
with tempfile.NamedTemporaryFile(delete=False, suffix=".flac") as f:
|
||||
f.write(b"fake audio data")
|
||||
temp_path = f.name
|
||||
|
||||
try:
|
||||
result = client._upload_file_to_signed_url(
|
||||
"https://storage.googleapis.com/signed-url",
|
||||
temp_path,
|
||||
"audio/flac"
|
||||
)
|
||||
assert result is True
|
||||
mock_put.assert_called_once()
|
||||
finally:
|
||||
os.unlink(temp_path)
|
||||
|
||||
@patch('requests.put')
|
||||
def test_upload_file_to_signed_url_failure(self, mock_put, client):
|
||||
"""Test failed upload to signed URL."""
|
||||
mock_put.return_value = MagicMock(status_code=403, text="Forbidden")
|
||||
|
||||
with tempfile.NamedTemporaryFile(delete=False, suffix=".flac") as f:
|
||||
f.write(b"fake audio data")
|
||||
temp_path = f.name
|
||||
|
||||
try:
|
||||
result = client._upload_file_to_signed_url(
|
||||
"https://storage.googleapis.com/signed-url",
|
||||
temp_path,
|
||||
"audio/flac"
|
||||
)
|
||||
assert result is False
|
||||
finally:
|
||||
os.unlink(temp_path)
|
||||
|
||||
@patch.object(RemoteKaraokeClient, '_upload_file_to_signed_url')
|
||||
@patch.object(RemoteKaraokeClient, '_request')
|
||||
def test_submit_job_uses_signed_url_flow(self, mock_request, mock_upload, client):
|
||||
"""Test that submit_job uses the signed URL upload flow."""
|
||||
# Mock create job response
|
||||
create_response = MagicMock()
|
||||
create_response.status_code = 200
|
||||
create_response.json.return_value = {
|
||||
"status": "success",
|
||||
"job_id": "test-job-123",
|
||||
"upload_urls": [
|
||||
{
|
||||
"file_type": "audio",
|
||||
"gcs_path": "uploads/test-job-123/audio/test.flac",
|
||||
"upload_url": "https://storage.googleapis.com/signed-url-1",
|
||||
"content_type": "audio/flac"
|
||||
}
|
||||
]
|
||||
}
|
||||
|
||||
# Mock uploads complete response
|
||||
complete_response = MagicMock()
|
||||
complete_response.status_code = 200
|
||||
complete_response.json.return_value = {
|
||||
"status": "success",
|
||||
"job_id": "test-job-123",
|
||||
"message": "Processing started"
|
||||
}
|
||||
|
||||
mock_request.side_effect = [create_response, complete_response]
|
||||
mock_upload.return_value = True
|
||||
|
||||
with tempfile.NamedTemporaryFile(delete=False, suffix=".flac") as f:
|
||||
f.write(b"fake audio data")
|
||||
temp_path = f.name
|
||||
|
||||
try:
|
||||
result = client.submit_job(
|
||||
filepath=temp_path,
|
||||
artist="Test Artist",
|
||||
title="Test Song"
|
||||
)
|
||||
|
||||
# Verify the flow: create -> upload -> complete
|
||||
assert mock_request.call_count == 2
|
||||
|
||||
# First call should be to create-with-upload-urls
|
||||
first_call = mock_request.call_args_list[0]
|
||||
assert first_call[0][0] == 'POST'
|
||||
assert 'create-with-upload-urls' in first_call[0][1]
|
||||
|
||||
# Second call should be to uploads-complete
|
||||
second_call = mock_request.call_args_list[1]
|
||||
assert second_call[0][0] == 'POST'
|
||||
assert 'uploads-complete' in second_call[0][1]
|
||||
|
||||
# Upload should have been called for the audio file
|
||||
mock_upload.assert_called_once()
|
||||
|
||||
assert result["status"] == "success"
|
||||
assert result["job_id"] == "test-job-123"
|
||||
finally:
|
||||
os.unlink(temp_path)
|
||||
|
||||
def test_client_has_signed_url_methods(self, client):
|
||||
"""Test that RemoteKaraokeClient has the required signed URL methods."""
|
||||
assert hasattr(client, '_upload_file_to_signed_url')
|
||||
assert hasattr(client, '_get_content_type')
|
||||
assert callable(client._upload_file_to_signed_url)
|
||||
assert callable(client._get_content_type)
|
||||
@@ -0,0 +1,587 @@
|
||||
"""Tests for the style_loader module."""
|
||||
import json
|
||||
import os
|
||||
import pytest
|
||||
import tempfile
|
||||
from unittest.mock import MagicMock, patch, mock_open
|
||||
|
||||
from karaoke_gen.style_loader import (
|
||||
DEFAULT_INTRO_STYLE,
|
||||
DEFAULT_END_STYLE,
|
||||
DEFAULT_KARAOKE_STYLE,
|
||||
DEFAULT_CDG_STYLE,
|
||||
DEFAULT_STYLE_PARAMS,
|
||||
ASSET_KEY_MAPPINGS,
|
||||
load_style_params_from_file,
|
||||
apply_style_overrides,
|
||||
update_asset_paths,
|
||||
save_style_params,
|
||||
load_styles_from_gcs,
|
||||
get_default_style_params,
|
||||
get_minimal_karaoke_styles,
|
||||
get_intro_format,
|
||||
get_end_format,
|
||||
get_karaoke_format,
|
||||
get_cdg_format,
|
||||
get_video_durations,
|
||||
get_existing_images,
|
||||
)
|
||||
|
||||
|
||||
class TestDefaultStyles:
|
||||
"""Tests for default style configurations."""
|
||||
|
||||
def test_default_intro_style_has_required_keys(self):
|
||||
"""Test that DEFAULT_INTRO_STYLE has all required keys."""
|
||||
required_keys = [
|
||||
"video_duration", "existing_image", "background_color",
|
||||
"background_image", "font", "artist_color", "title_color",
|
||||
]
|
||||
for key in required_keys:
|
||||
assert key in DEFAULT_INTRO_STYLE
|
||||
|
||||
def test_default_end_style_has_required_keys(self):
|
||||
"""Test that DEFAULT_END_STYLE has all required keys."""
|
||||
required_keys = [
|
||||
"video_duration", "background_color", "font",
|
||||
"extra_text", "extra_text_color",
|
||||
]
|
||||
for key in required_keys:
|
||||
assert key in DEFAULT_END_STYLE
|
||||
|
||||
def test_default_karaoke_style_has_required_keys(self):
|
||||
"""Test that DEFAULT_KARAOKE_STYLE has all required keys."""
|
||||
required_keys = [
|
||||
"background_color", "font", "font_path", "primary_color",
|
||||
"secondary_color", "outline_color", "back_color",
|
||||
"font_size", "max_line_length",
|
||||
]
|
||||
for key in required_keys:
|
||||
assert key in DEFAULT_KARAOKE_STYLE
|
||||
|
||||
def test_default_cdg_style_has_required_keys(self):
|
||||
"""Test that DEFAULT_CDG_STYLE has all required keys."""
|
||||
required_keys = [
|
||||
"font_path", "instrumental_background",
|
||||
"title_screen_background", "outro_background",
|
||||
]
|
||||
for key in required_keys:
|
||||
assert key in DEFAULT_CDG_STYLE
|
||||
|
||||
def test_default_style_params_has_all_sections(self):
|
||||
"""Test that DEFAULT_STYLE_PARAMS has all sections."""
|
||||
assert "intro" in DEFAULT_STYLE_PARAMS
|
||||
assert "end" in DEFAULT_STYLE_PARAMS
|
||||
assert "karaoke" in DEFAULT_STYLE_PARAMS
|
||||
assert "cdg" in DEFAULT_STYLE_PARAMS
|
||||
|
||||
|
||||
class TestAssetKeyMappings:
|
||||
"""Tests for asset key mappings."""
|
||||
|
||||
def test_background_mappings_exist(self):
|
||||
"""Test that background image mappings exist."""
|
||||
assert "intro_background" in ASSET_KEY_MAPPINGS
|
||||
assert "karaoke_background" in ASSET_KEY_MAPPINGS
|
||||
assert "end_background" in ASSET_KEY_MAPPINGS
|
||||
|
||||
def test_font_mapping_is_list(self):
|
||||
"""Test that font mapping is a list (maps to multiple sections)."""
|
||||
assert isinstance(ASSET_KEY_MAPPINGS["font"], list)
|
||||
assert len(ASSET_KEY_MAPPINGS["font"]) == 4 # intro, karaoke, end, cdg
|
||||
|
||||
def test_cdg_mappings_exist(self):
|
||||
"""Test that CDG-specific mappings exist."""
|
||||
assert "cdg_instrumental_background" in ASSET_KEY_MAPPINGS
|
||||
assert "cdg_title_background" in ASSET_KEY_MAPPINGS
|
||||
assert "cdg_outro_background" in ASSET_KEY_MAPPINGS
|
||||
|
||||
|
||||
class TestLoadStyleParamsFromFile:
|
||||
"""Tests for load_style_params_from_file function."""
|
||||
|
||||
def test_returns_defaults_when_no_path(self):
|
||||
"""Test that defaults are returned when no path is provided."""
|
||||
result = load_style_params_from_file(None)
|
||||
assert result == get_default_style_params()
|
||||
|
||||
def test_loads_valid_json_file(self, temp_dir):
|
||||
"""Test loading a valid JSON file."""
|
||||
style_data = {"intro": {"video_duration": 10}}
|
||||
style_path = os.path.join(temp_dir, "styles.json")
|
||||
|
||||
with open(style_path, "w") as f:
|
||||
json.dump(style_data, f)
|
||||
|
||||
result = load_style_params_from_file(style_path)
|
||||
assert result == style_data
|
||||
|
||||
def test_raises_on_file_not_found(self, temp_dir):
|
||||
"""Test that FileNotFoundError is raised when file doesn't exist."""
|
||||
fake_path = os.path.join(temp_dir, "nonexistent.json")
|
||||
|
||||
with pytest.raises(FileNotFoundError):
|
||||
load_style_params_from_file(fake_path, exit_on_error=False)
|
||||
|
||||
def test_raises_on_invalid_json(self, temp_dir):
|
||||
"""Test that JSONDecodeError is raised for invalid JSON."""
|
||||
invalid_path = os.path.join(temp_dir, "invalid.json")
|
||||
|
||||
with open(invalid_path, "w") as f:
|
||||
f.write("not valid json {{{")
|
||||
|
||||
with pytest.raises(json.JSONDecodeError):
|
||||
load_style_params_from_file(invalid_path, exit_on_error=False)
|
||||
|
||||
def test_raises_on_generic_error(self, temp_dir):
|
||||
"""Test that generic exceptions are raised."""
|
||||
style_path = os.path.join(temp_dir, "styles.json")
|
||||
|
||||
# Create a file that will cause a read error by mocking open
|
||||
with patch("builtins.open", side_effect=PermissionError("Permission denied")):
|
||||
with pytest.raises(PermissionError):
|
||||
load_style_params_from_file(style_path, exit_on_error=False)
|
||||
|
||||
def test_exit_on_error_file_not_found(self, temp_dir):
|
||||
"""Test that sys.exit is called when exit_on_error=True and file not found."""
|
||||
fake_path = os.path.join(temp_dir, "nonexistent.json")
|
||||
|
||||
with patch("sys.exit") as mock_exit:
|
||||
# sys.exit is mocked so exception still propagates after sys.exit call
|
||||
with pytest.raises(FileNotFoundError):
|
||||
load_style_params_from_file(fake_path, exit_on_error=True)
|
||||
mock_exit.assert_called_once_with(1)
|
||||
|
||||
def test_exit_on_error_invalid_json(self, temp_dir):
|
||||
"""Test that sys.exit is called when exit_on_error=True and invalid JSON."""
|
||||
invalid_path = os.path.join(temp_dir, "invalid.json")
|
||||
|
||||
with open(invalid_path, "w") as f:
|
||||
f.write("not valid json")
|
||||
|
||||
with patch("sys.exit") as mock_exit:
|
||||
# sys.exit is mocked so exception still propagates after sys.exit call
|
||||
with pytest.raises(json.JSONDecodeError):
|
||||
load_style_params_from_file(invalid_path, exit_on_error=True)
|
||||
mock_exit.assert_called_once_with(1)
|
||||
|
||||
|
||||
class TestApplyStyleOverrides:
|
||||
"""Tests for apply_style_overrides function."""
|
||||
|
||||
def test_applies_simple_override(self):
|
||||
"""Test applying a simple override."""
|
||||
style_params = {"intro": {"video_duration": 5}}
|
||||
overrides = {"intro.video_duration": "10"}
|
||||
|
||||
apply_style_overrides(style_params, overrides)
|
||||
|
||||
assert style_params["intro"]["video_duration"] == 10
|
||||
|
||||
def test_applies_string_override(self):
|
||||
"""Test applying a string override."""
|
||||
style_params = {"intro": {"font": "Arial"}}
|
||||
overrides = {"intro.font": "Helvetica"}
|
||||
|
||||
apply_style_overrides(style_params, overrides)
|
||||
|
||||
assert style_params["intro"]["font"] == "Helvetica"
|
||||
|
||||
def test_applies_bool_override_true(self):
|
||||
"""Test applying a boolean true override."""
|
||||
style_params = {"karaoke": {"bold": False}}
|
||||
overrides = {"karaoke.bold": "true"}
|
||||
|
||||
apply_style_overrides(style_params, overrides)
|
||||
|
||||
assert style_params["karaoke"]["bold"] is True
|
||||
|
||||
def test_applies_bool_override_false(self):
|
||||
"""Test applying a boolean false override."""
|
||||
style_params = {"karaoke": {"bold": True}}
|
||||
overrides = {"karaoke.bold": "false"}
|
||||
|
||||
apply_style_overrides(style_params, overrides)
|
||||
|
||||
assert style_params["karaoke"]["bold"] is False
|
||||
|
||||
def test_applies_bool_override_yes(self):
|
||||
"""Test applying a boolean 'yes' override."""
|
||||
style_params = {"karaoke": {"bold": False}}
|
||||
overrides = {"karaoke.bold": "yes"}
|
||||
|
||||
apply_style_overrides(style_params, overrides)
|
||||
|
||||
assert style_params["karaoke"]["bold"] is True
|
||||
|
||||
def test_warns_on_missing_key(self, mock_logger):
|
||||
"""Test that a warning is logged for missing keys."""
|
||||
style_params = {"intro": {"video_duration": 5}}
|
||||
overrides = {"intro.nonexistent": "value"}
|
||||
|
||||
apply_style_overrides(style_params, overrides, logger=mock_logger)
|
||||
|
||||
mock_logger.warning.assert_called()
|
||||
|
||||
def test_warns_on_missing_section(self, mock_logger):
|
||||
"""Test that a warning is logged for missing sections."""
|
||||
style_params = {"intro": {"video_duration": 5}}
|
||||
overrides = {"nonexistent.key": "value"}
|
||||
|
||||
apply_style_overrides(style_params, overrides, logger=mock_logger)
|
||||
|
||||
mock_logger.warning.assert_called()
|
||||
|
||||
def test_handles_type_conversion_error(self, mock_logger):
|
||||
"""Test handling of type conversion errors."""
|
||||
style_params = {"intro": {"video_duration": 5}}
|
||||
# This will try to convert "not_a_number" to int and fail gracefully
|
||||
overrides = {"intro.video_duration": "not_a_number"}
|
||||
|
||||
apply_style_overrides(style_params, overrides, logger=mock_logger)
|
||||
|
||||
# Should use string value as fallback
|
||||
mock_logger.warning.assert_called()
|
||||
|
||||
def test_handles_none_original_value(self):
|
||||
"""Test handling when original value is None."""
|
||||
style_params = {"intro": {"background_image": None}}
|
||||
overrides = {"intro.background_image": "/path/to/image.png"}
|
||||
|
||||
apply_style_overrides(style_params, overrides)
|
||||
|
||||
assert style_params["intro"]["background_image"] == "/path/to/image.png"
|
||||
|
||||
|
||||
class TestUpdateAssetPaths:
|
||||
"""Tests for update_asset_paths function."""
|
||||
|
||||
def test_updates_single_mapping(self):
|
||||
"""Test updating a single asset path."""
|
||||
style_data = {"intro": {"background_image": None}}
|
||||
local_assets = {"intro_background": "/local/path/background.png"}
|
||||
|
||||
result = update_asset_paths(style_data, local_assets)
|
||||
|
||||
assert result is True
|
||||
assert style_data["intro"]["background_image"] == "/local/path/background.png"
|
||||
|
||||
def test_updates_multiple_mappings(self):
|
||||
"""Test updating multiple asset paths from font mapping."""
|
||||
style_data = {
|
||||
"intro": {"font": None},
|
||||
"karaoke": {"font_path": None},
|
||||
"end": {"font": None},
|
||||
"cdg": {"font_path": None},
|
||||
}
|
||||
local_assets = {"font": "/local/path/font.ttf"}
|
||||
|
||||
result = update_asset_paths(style_data, local_assets)
|
||||
|
||||
assert result is True
|
||||
assert style_data["intro"]["font"] == "/local/path/font.ttf"
|
||||
assert style_data["karaoke"]["font_path"] == "/local/path/font.ttf"
|
||||
assert style_data["end"]["font"] == "/local/path/font.ttf"
|
||||
assert style_data["cdg"]["font_path"] == "/local/path/font.ttf"
|
||||
|
||||
def test_skips_unknown_asset_keys(self):
|
||||
"""Test that unknown asset keys are skipped."""
|
||||
style_data = {"intro": {"background_image": None}}
|
||||
local_assets = {"unknown_key": "/local/path/file.png"}
|
||||
|
||||
result = update_asset_paths(style_data, local_assets)
|
||||
|
||||
assert result is False
|
||||
assert style_data["intro"]["background_image"] is None
|
||||
|
||||
def test_skips_missing_sections(self):
|
||||
"""Test that missing sections don't cause errors."""
|
||||
style_data = {} # No sections
|
||||
local_assets = {"intro_background": "/local/path/background.png"}
|
||||
|
||||
result = update_asset_paths(style_data, local_assets)
|
||||
|
||||
assert result is False
|
||||
|
||||
def test_returns_false_when_no_updates(self):
|
||||
"""Test that False is returned when no updates are made."""
|
||||
style_data = {"intro": {"background_image": None}}
|
||||
local_assets = {}
|
||||
|
||||
result = update_asset_paths(style_data, local_assets)
|
||||
|
||||
assert result is False
|
||||
|
||||
|
||||
class TestSaveStyleParams:
|
||||
"""Tests for save_style_params function."""
|
||||
|
||||
def test_saves_to_file(self, temp_dir):
|
||||
"""Test saving style params to a file."""
|
||||
style_data = {"intro": {"video_duration": 10}}
|
||||
output_path = os.path.join(temp_dir, "output.json")
|
||||
|
||||
result = save_style_params(style_data, output_path)
|
||||
|
||||
assert result == output_path
|
||||
assert os.path.exists(output_path)
|
||||
|
||||
with open(output_path) as f:
|
||||
loaded = json.load(f)
|
||||
assert loaded == style_data
|
||||
|
||||
def test_formats_json_with_indent(self, temp_dir):
|
||||
"""Test that JSON is formatted with indentation."""
|
||||
style_data = {"intro": {"video_duration": 10}}
|
||||
output_path = os.path.join(temp_dir, "output.json")
|
||||
|
||||
save_style_params(style_data, output_path)
|
||||
|
||||
with open(output_path) as f:
|
||||
content = f.read()
|
||||
# Check for indentation (2 spaces)
|
||||
assert " " in content
|
||||
|
||||
|
||||
class TestLoadStylesFromGcs:
|
||||
"""Tests for load_styles_from_gcs function."""
|
||||
|
||||
def test_returns_minimal_styles_when_no_gcs_path(self, temp_dir):
|
||||
"""Test that minimal styles are returned when no GCS path provided."""
|
||||
mock_download = MagicMock()
|
||||
|
||||
styles_path, style_data = load_styles_from_gcs(
|
||||
style_params_gcs_path=None,
|
||||
style_assets=None,
|
||||
temp_dir=temp_dir,
|
||||
download_func=mock_download,
|
||||
)
|
||||
|
||||
assert os.path.exists(styles_path)
|
||||
assert "karaoke" in style_data
|
||||
assert "cdg" in style_data
|
||||
mock_download.assert_not_called()
|
||||
|
||||
def test_downloads_and_loads_custom_styles(self, temp_dir):
|
||||
"""Test downloading and loading custom styles from GCS."""
|
||||
custom_styles = {"karaoke": {"font_size": 120}}
|
||||
|
||||
def mock_download(gcs_path, local_path):
|
||||
if "styles.json" in local_path:
|
||||
with open(local_path, "w") as f:
|
||||
json.dump(custom_styles, f)
|
||||
|
||||
styles_path, style_data = load_styles_from_gcs(
|
||||
style_params_gcs_path="gs://bucket/styles.json",
|
||||
style_assets=None,
|
||||
temp_dir=temp_dir,
|
||||
download_func=mock_download,
|
||||
)
|
||||
|
||||
assert style_data["karaoke"]["font_size"] == 120
|
||||
|
||||
def test_downloads_style_assets(self, temp_dir):
|
||||
"""Test downloading style assets."""
|
||||
custom_styles = {"karaoke": {"background_image": None}}
|
||||
download_calls = []
|
||||
|
||||
def mock_download(gcs_path, local_path):
|
||||
download_calls.append((gcs_path, local_path))
|
||||
if "styles.json" in local_path:
|
||||
with open(local_path, "w") as f:
|
||||
json.dump(custom_styles, f)
|
||||
else:
|
||||
# Create a dummy asset file
|
||||
with open(local_path, "w") as f:
|
||||
f.write("asset content")
|
||||
|
||||
style_assets = {
|
||||
"style_params": "gs://bucket/styles.json",
|
||||
"karaoke_background": "gs://bucket/background.png",
|
||||
}
|
||||
|
||||
styles_path, style_data = load_styles_from_gcs(
|
||||
style_params_gcs_path="gs://bucket/styles.json",
|
||||
style_assets=style_assets,
|
||||
temp_dir=temp_dir,
|
||||
download_func=mock_download,
|
||||
)
|
||||
|
||||
# Should have downloaded both styles.json and the background
|
||||
assert len(download_calls) >= 2
|
||||
|
||||
def test_skips_style_params_asset(self, temp_dir):
|
||||
"""Test that style_params asset key is skipped (already downloaded)."""
|
||||
custom_styles = {"karaoke": {"background_image": None}}
|
||||
download_calls = []
|
||||
|
||||
def mock_download(gcs_path, local_path):
|
||||
download_calls.append((gcs_path, local_path))
|
||||
if "styles.json" in local_path:
|
||||
with open(local_path, "w") as f:
|
||||
json.dump(custom_styles, f)
|
||||
|
||||
style_assets = {"style_params": "gs://bucket/styles.json"}
|
||||
|
||||
load_styles_from_gcs(
|
||||
style_params_gcs_path="gs://bucket/styles.json",
|
||||
style_assets=style_assets,
|
||||
temp_dir=temp_dir,
|
||||
download_func=mock_download,
|
||||
)
|
||||
|
||||
# Should only download styles.json once (not from style_assets)
|
||||
assert len(download_calls) == 1
|
||||
|
||||
def test_handles_asset_download_failure(self, temp_dir, mock_logger):
|
||||
"""Test handling of asset download failures."""
|
||||
custom_styles = {"karaoke": {"background_image": None}}
|
||||
|
||||
def mock_download(gcs_path, local_path):
|
||||
if "background" in gcs_path:
|
||||
raise Exception("Download failed")
|
||||
with open(local_path, "w") as f:
|
||||
json.dump(custom_styles, f)
|
||||
|
||||
style_assets = {"karaoke_background": "gs://bucket/background.png"}
|
||||
|
||||
# Should not raise, just log warning
|
||||
styles_path, style_data = load_styles_from_gcs(
|
||||
style_params_gcs_path="gs://bucket/styles.json",
|
||||
style_assets=style_assets,
|
||||
temp_dir=temp_dir,
|
||||
download_func=mock_download,
|
||||
logger=mock_logger,
|
||||
)
|
||||
|
||||
mock_logger.warning.assert_called()
|
||||
|
||||
def test_returns_defaults_on_download_failure(self, temp_dir, mock_logger):
|
||||
"""Test that defaults are returned when download fails."""
|
||||
def mock_download(gcs_path, local_path):
|
||||
raise Exception("Download failed")
|
||||
|
||||
styles_path, style_data = load_styles_from_gcs(
|
||||
style_params_gcs_path="gs://bucket/styles.json",
|
||||
style_assets=None,
|
||||
temp_dir=temp_dir,
|
||||
download_func=mock_download,
|
||||
logger=mock_logger,
|
||||
)
|
||||
|
||||
assert "karaoke" in style_data
|
||||
mock_logger.warning.assert_called()
|
||||
|
||||
|
||||
class TestHelperFunctions:
|
||||
"""Tests for helper functions."""
|
||||
|
||||
def test_get_default_style_params(self):
|
||||
"""Test get_default_style_params returns fresh copy."""
|
||||
params1 = get_default_style_params()
|
||||
params2 = get_default_style_params()
|
||||
|
||||
# Modify params1
|
||||
params1["intro"]["video_duration"] = 999
|
||||
|
||||
# params2 should be unaffected
|
||||
assert params2["intro"]["video_duration"] != 999
|
||||
|
||||
def test_get_minimal_karaoke_styles(self):
|
||||
"""Test get_minimal_karaoke_styles returns minimal set."""
|
||||
styles = get_minimal_karaoke_styles()
|
||||
|
||||
assert "karaoke" in styles
|
||||
assert "cdg" in styles
|
||||
assert "intro" not in styles
|
||||
assert "end" not in styles
|
||||
|
||||
def test_get_intro_format_with_defaults(self):
|
||||
"""Test get_intro_format with empty params uses defaults."""
|
||||
result = get_intro_format({})
|
||||
|
||||
assert result["video_duration"] == DEFAULT_INTRO_STYLE["video_duration"]
|
||||
|
||||
def test_get_intro_format_merges_custom(self):
|
||||
"""Test get_intro_format merges custom params."""
|
||||
style_params = {"intro": {"video_duration": 10}}
|
||||
result = get_intro_format(style_params)
|
||||
|
||||
assert result["video_duration"] == 10
|
||||
# Should still have default keys
|
||||
assert "font" in result
|
||||
|
||||
def test_get_end_format_with_defaults(self):
|
||||
"""Test get_end_format with empty params uses defaults."""
|
||||
result = get_end_format({})
|
||||
|
||||
assert result["video_duration"] == DEFAULT_END_STYLE["video_duration"]
|
||||
|
||||
def test_get_end_format_merges_custom(self):
|
||||
"""Test get_end_format merges custom params."""
|
||||
style_params = {"end": {"video_duration": 8}}
|
||||
result = get_end_format(style_params)
|
||||
|
||||
assert result["video_duration"] == 8
|
||||
|
||||
def test_get_karaoke_format_with_defaults(self):
|
||||
"""Test get_karaoke_format with empty params uses defaults."""
|
||||
result = get_karaoke_format({})
|
||||
|
||||
assert result["font_size"] == DEFAULT_KARAOKE_STYLE["font_size"]
|
||||
|
||||
def test_get_karaoke_format_merges_custom(self):
|
||||
"""Test get_karaoke_format merges custom params."""
|
||||
style_params = {"karaoke": {"font_size": 120}}
|
||||
result = get_karaoke_format(style_params)
|
||||
|
||||
assert result["font_size"] == 120
|
||||
|
||||
def test_get_cdg_format_returns_none_when_missing(self):
|
||||
"""Test get_cdg_format returns None when section is missing."""
|
||||
result = get_cdg_format({})
|
||||
|
||||
assert result is None
|
||||
|
||||
def test_get_cdg_format_merges_custom(self):
|
||||
"""Test get_cdg_format merges custom params."""
|
||||
style_params = {"cdg": {"font_path": "/custom/font.ttf"}}
|
||||
result = get_cdg_format(style_params)
|
||||
|
||||
assert result["font_path"] == "/custom/font.ttf"
|
||||
|
||||
def test_get_video_durations_defaults(self):
|
||||
"""Test get_video_durations with defaults."""
|
||||
intro_dur, end_dur = get_video_durations({})
|
||||
|
||||
assert intro_dur == DEFAULT_INTRO_STYLE["video_duration"]
|
||||
assert end_dur == DEFAULT_END_STYLE["video_duration"]
|
||||
|
||||
def test_get_video_durations_custom(self):
|
||||
"""Test get_video_durations with custom values."""
|
||||
style_params = {
|
||||
"intro": {"video_duration": 10},
|
||||
"end": {"video_duration": 8},
|
||||
}
|
||||
intro_dur, end_dur = get_video_durations(style_params)
|
||||
|
||||
assert intro_dur == 10
|
||||
assert end_dur == 8
|
||||
|
||||
def test_get_existing_images_defaults(self):
|
||||
"""Test get_existing_images with no existing images."""
|
||||
title_img, end_img = get_existing_images({})
|
||||
|
||||
assert title_img is None
|
||||
assert end_img is None
|
||||
|
||||
def test_get_existing_images_custom(self):
|
||||
"""Test get_existing_images with custom images."""
|
||||
style_params = {
|
||||
"intro": {"existing_image": "/path/to/title.png"},
|
||||
"end": {"existing_image": "/path/to/end.png"},
|
||||
}
|
||||
title_img, end_img = get_existing_images(style_params)
|
||||
|
||||
assert title_img == "/path/to/title.png"
|
||||
assert end_img == "/path/to/end.png"
|
||||
Reference in new issue
Block a user