mirror of
https://github.com/nomadkaraoke/karaoke-gen.git
synced 2026-11-21 05:18:08 +09:00
Compare commits
5
Commits
| Author | SHA1 | Date | |
|---|---|---|---|
|
|
8e5e81d6b8 | ||
|
|
73faacaebe | ||
|
|
499c386ad7 | ||
|
|
3f5a423f98 | ||
|
|
b87e29d622 |
No files matched your search
@@ -1,401 +0,0 @@
|
||||
# Plan: Video Worker Stage Refactor
|
||||
|
||||
**Created:** 2026-01-03
|
||||
**Branch:** feat/sess-20260103-2250-investigate-youtube-upload
|
||||
**Status:** COMPLETE - Ready for PR
|
||||
**Last Updated:** 2026-01-04
|
||||
|
||||
## Overview
|
||||
|
||||
Refactor `video_worker.py` and `KaraokeFinalise` into separate, composable stages following SOLID principles. This eliminates the GCE vs KaraokeFinalise code path divergence that causes features like YouTube upload to be missing from the GCE encoding path.
|
||||
|
||||
## Three Execution Modes
|
||||
|
||||
Understanding the three ways this system runs is critical to the refactor:
|
||||
|
||||
### 1. Local CLI (`karaoke-gen`)
|
||||
- Runs entirely on user's machine
|
||||
- KaraokeFinalise.process() orchestrates everything
|
||||
- Writes directly to local Dropbox sync folder (e.g., `~/Dropbox/Karaoke/Tracks-Organized/`)
|
||||
- **Produces full output structure** with stems/, lyrics/, visualizations/, etc.
|
||||
- **This is the source of truth for output format**
|
||||
|
||||
### 2. Remote CLI (`karaoke-gen-remote`)
|
||||
- User's CLI talks to cloud backend (api.nomadkaraoke.com)
|
||||
- Cloud does ALL processing (audio, lyrics, video encoding)
|
||||
- CLI just monitors job status and downloads final outputs
|
||||
- With Dropbox configured: cloud uploads full structure to user's Dropbox
|
||||
- Without Dropbox: user downloads individual files
|
||||
|
||||
### 3. Web Frontend (`gen.nomadkaraoke.com`)
|
||||
- Same cloud backend as remote CLI
|
||||
- User sees:
|
||||
- YouTube video link
|
||||
- Dropbox folder link
|
||||
- Direct download: final karaoke video (720p)
|
||||
- Direct download: 4K video
|
||||
- Direct download: "with vocals" video
|
||||
- Direct download: CDG zip (if enabled)
|
||||
|
||||
## Problem Statement
|
||||
|
||||
Currently, `video_worker.py` has two completely divergent code paths:
|
||||
|
||||
```python
|
||||
if use_gce_encoding:
|
||||
# GCE PATH: Only encoding, missing YouTube/Discord/CDG/TXT
|
||||
result = await _encode_via_gce(...)
|
||||
# result DOES NOT include youtube_url!
|
||||
else:
|
||||
# STANDARD PATH: KaraokeFinalise does everything
|
||||
finalise = KaraokeFinalise(...)
|
||||
result = finalise.process()
|
||||
# result includes youtube_url from upload
|
||||
```
|
||||
|
||||
### What GCE Encoding Broke
|
||||
|
||||
| Feature | KaraokeFinalise Path | GCE Path |
|
||||
|---------|---------------------|----------|
|
||||
| Video encoding | ✅ 4 formats | ✅ 4 formats |
|
||||
| YouTube upload | ✅ | ❌ **BROKEN** |
|
||||
| Discord notification | ✅ | ❌ **BROKEN** |
|
||||
| CDG/TXT packaging | ✅ | ❌ **BROKEN** |
|
||||
| Brand code generation | ✅ | ✅ (via Dropbox service) |
|
||||
| Dropbox upload | ✅ (rclone) | ✅ (native SDK) |
|
||||
| Google Drive upload | ❌ | ✅ (native SDK) |
|
||||
|
||||
### Root Cause
|
||||
|
||||
`KaraokeFinalise.process()` is a monolith that handles:
|
||||
1. CDG/TXT packaging
|
||||
2. Video encoding (the CPU-intensive part)
|
||||
3. YouTube upload
|
||||
4. Discord notifications
|
||||
5. Folder organization
|
||||
|
||||
The GCE encoding was added as a bypass around the entire class, rather than replacing just the encoding step.
|
||||
|
||||
## Requirements
|
||||
|
||||
### Functional Requirements
|
||||
- [ ] YouTube upload works regardless of encoding backend (GCE or local)
|
||||
- [ ] Discord notifications work regardless of encoding backend
|
||||
- [ ] CDG/TXT packaging works regardless of encoding backend
|
||||
- [ ] Brand code generation works consistently
|
||||
- [ ] Dropbox/Google Drive upload works (already works in both paths)
|
||||
- [ ] All existing CLI functionality continues to work
|
||||
- [ ] Output structure matches local CLI exactly
|
||||
|
||||
### Output Structure Parity
|
||||
|
||||
Cloud backend must produce identical output to local CLI:
|
||||
|
||||
```
|
||||
NOMAD-XXXX - Artist - Title/
|
||||
├── Artist - Title (Title).mov
|
||||
├── Artist - Title (Title).jpg
|
||||
├── Artist - Title (End).mov
|
||||
├── Artist - Title (End).jpg
|
||||
├── Artist - Title (Final Karaoke Lossless 4k).mp4
|
||||
├── Artist - Title (Final Karaoke Lossless 4k).mkv
|
||||
├── Artist - Title (Final Karaoke Lossy 4k).mp4
|
||||
├── Artist - Title (Final Karaoke Lossy 720p).mp4
|
||||
├── Artist - Title (Final Karaoke CDG).zip
|
||||
├── Artist - Title (Final Karaoke TXT).zip
|
||||
├── Artist - Title (Instrumental {model}).flac
|
||||
├── Artist - Title (Instrumental +BV {model}).flac
|
||||
├── Artist - Title (Karaoke).mp4
|
||||
├── Artist - Title (Karaoke).cdg
|
||||
├── Artist - Title (Karaoke).mp3
|
||||
├── Artist - Title (Karaoke).lrc
|
||||
├── Artist - Title (With Vocals).mp4
|
||||
├── stems/
|
||||
│ ├── Artist - Title (Instrumental {model}).flac
|
||||
│ ├── Artist - Title (Vocals {model}).flac
|
||||
│ ├── Artist - Title (Lead Vocals {model}).flac
|
||||
│ ├── Artist - Title (Backing Vocals {model}).flac
|
||||
│ └── ... (other stems)
|
||||
├── lyrics/
|
||||
│ ├── Artist - Title (Karaoke).lrc
|
||||
│ ├── Artist - Title (Karaoke).ass
|
||||
│ ├── Artist - Title (Karaoke).srt
|
||||
│ └── ... (corrections JSONs)
|
||||
├── previews/
|
||||
├── visualizations/
|
||||
└── styles_assets/
|
||||
```
|
||||
|
||||
### Non-Functional Requirements
|
||||
- [ ] Single Responsibility: Each stage has one job
|
||||
- [ ] Open/Closed: Easy to add new encoding backends without modifying stages
|
||||
- [ ] Testable: Each stage can be unit tested in isolation
|
||||
- [ ] No duplicate code between paths
|
||||
- [ ] Backwards compatible with existing KaraokeFinalise CLI usage
|
||||
|
||||
## Technical Approach
|
||||
|
||||
### Architecture: Stage-Based Pipeline
|
||||
|
||||
```
|
||||
video_worker.py (VideoWorkerOrchestrator)
|
||||
│
|
||||
├── 1. EncodingStage (interface)
|
||||
│ ├── LocalEncodingService (wraps KaraokeFinalise encoding methods)
|
||||
│ └── GCEEncodingService (existing encoding_service.py)
|
||||
│
|
||||
├── 2. PackagingStage
|
||||
│ ├── CDGPackager (extracts from KaraokeFinalise)
|
||||
│ └── TXTPackager (extracts from KaraokeFinalise)
|
||||
│
|
||||
├── 3. DistributionStage
|
||||
│ ├── YouTubeUploadService (extracts from KaraokeFinalise)
|
||||
│ ├── DropboxService (existing)
|
||||
│ ├── GoogleDriveService (existing)
|
||||
│ └── DiscordService (extracts from KaraokeFinalise)
|
||||
│
|
||||
└── 4. OrganizationStage
|
||||
└── BrandCodeService (extracts from KaraokeFinalise + DropboxService)
|
||||
```
|
||||
|
||||
### Key Design Decisions
|
||||
|
||||
1. **Extract, Don't Rewrite**: Pull methods out of KaraokeFinalise into services, don't rewrite from scratch
|
||||
2. **Keep KaraokeFinalise for CLI**: Local CLI still uses KaraokeFinalise.process() - we extract services that it can also use
|
||||
3. **Interface-Based Encoding**: EncodingStage interface allows swapping GCE/local without changing orchestration
|
||||
4. **Existing Service Pattern**: Follow singleton + factory pattern already used in codebase
|
||||
5. **Local CLI stays sync**: Local karaoke-gen CLI remains synchronous as it is today
|
||||
|
||||
### Data Flow
|
||||
|
||||
```
|
||||
JobConfig (from Firestore)
|
||||
│
|
||||
▼
|
||||
┌─────────────────────────────────────────────────────────────────┐
|
||||
│ VideoWorkerOrchestrator │
|
||||
│ ┌─────────────────────────────────────────────────────────────┐│
|
||||
│ │ 1. Setup: Download files, prepare directories ││
|
||||
│ │ - Download title.mov, end.mov, with_vocals.mkv ││
|
||||
│ │ - Download instrumental audio ││
|
||||
│ │ - Prepare temp directory ││
|
||||
│ └─────────────────────────────────────────────────────────────┘│
|
||||
│ │ │
|
||||
│ ▼ │
|
||||
│ ┌─────────────────────────────────────────────────────────────┐│
|
||||
│ │ 2. Packaging: CDG/TXT (if enabled) - runs BEFORE encoding ││
|
||||
│ │ Reason: CDG depends on LRC file which is available ││
|
||||
│ │ Input: LRC file, instrumental audio ││
|
||||
│ │ Output: CDG.zip, TXT.zip, .cdg, .mp3 ││
|
||||
│ └─────────────────────────────────────────────────────────────┘│
|
||||
│ │ │
|
||||
│ ▼ │
|
||||
│ ┌─────────────────────────────────────────────────────────────┐│
|
||||
│ │ 3. Encoding: GCE or Local (via interface) ││
|
||||
│ │ Input: title.mov, karaoke.mov, end.mov ││
|
||||
│ │ Output: 4K lossless, 4K lossy, 720p, MKV ││
|
||||
│ └─────────────────────────────────────────────────────────────┘│
|
||||
│ │ │
|
||||
│ ▼ │
|
||||
│ ┌─────────────────────────────────────────────────────────────┐│
|
||||
│ │ 4. Organization: Brand code, folder structure ││
|
||||
│ │ - Generate brand code (NOMAD-XXXX) ││
|
||||
│ │ - Prepare full output directory structure ││
|
||||
│ └─────────────────────────────────────────────────────────────┘│
|
||||
│ │ │
|
||||
│ ▼ │
|
||||
│ ┌─────────────────────────────────────────────────────────────┐│
|
||||
│ │ 5. Distribution: Upload to destinations ││
|
||||
│ │ - YouTube upload (if enabled) ││
|
||||
│ │ - Dropbox upload (if configured) ││
|
||||
│ │ - Google Drive upload (if configured) ││
|
||||
│ └─────────────────────────────────────────────────────────────┘│
|
||||
│ │ │
|
||||
│ ▼ │
|
||||
│ ┌─────────────────────────────────────────────────────────────┐│
|
||||
│ │ 6. Notification: Discord, Email ││
|
||||
│ │ Input: YouTube URL, brand code, Dropbox link ││
|
||||
│ └─────────────────────────────────────────────────────────────┘│
|
||||
└─────────────────────────────────────────────────────────────────┘
|
||||
│
|
||||
▼
|
||||
Result (youtube_url, brand_code, dropbox_link, file_paths, etc.)
|
||||
```
|
||||
|
||||
## Implementation Steps
|
||||
|
||||
### Phase 1: Extract Services from KaraokeFinalise (Non-Breaking) ✅ COMPLETE
|
||||
|
||||
These services can be used by both the cloud orchestrator AND KaraokeFinalise.
|
||||
|
||||
1. [x] **Create `backend/services/youtube_upload_service.py`** ✅
|
||||
- Extract YouTube upload logic from KaraokeFinalise
|
||||
- Methods: `upload_video()`, `check_duplicate()`, `delete_video()`, `truncate_title()`
|
||||
- Supports pre-loaded credentials (server-side) or client secrets file (CLI)
|
||||
- 23 unit tests with mocked YouTube API
|
||||
|
||||
2. [x] **Create `backend/services/discord_service.py`** ✅
|
||||
- Extract Discord notification from KaraokeFinalise
|
||||
- Methods: `post_message()`, `post_video_notification()`, `is_enabled()`
|
||||
- Simple HTTP POST to webhook with validation
|
||||
- 20 unit tests with mocked requests
|
||||
|
||||
3. [x] **Create `backend/services/packaging_service.py`** ✅
|
||||
- Extract CDG/TXT generation from KaraokeFinalise
|
||||
- Methods: `create_cdg_package()`, `create_txt_package()`
|
||||
- Uses lyrics_converter and CDGGenerator (existing dependencies)
|
||||
- 18 unit tests with temp files
|
||||
|
||||
4. [x] **Create `backend/services/local_encoding_service.py`** ✅
|
||||
- Extract encoding methods from KaraokeFinalise
|
||||
- Methods: `encode_all_formats()`, individual encoding methods
|
||||
- Hardware acceleration detection with software fallback
|
||||
- 28 unit tests with mocked subprocess
|
||||
|
||||
### Phase 2: Create Encoding Interface ✅ COMPLETE
|
||||
|
||||
5. [x] **Create `backend/services/encoding_interface.py`** ✅
|
||||
- Define ABC `EncodingBackend` with methods: `encode()`, `is_available()`, `get_status()`
|
||||
- `EncodingInput` and `EncodingOutput` dataclasses for config/results
|
||||
- Factory function `get_encoding_backend()` with "local", "gce", "auto" options
|
||||
|
||||
6. [x] **`LocalEncodingBackend`** ✅
|
||||
- Implements `EncodingBackend` interface
|
||||
- Wraps `LocalEncodingService` with `asyncio.to_thread()`
|
||||
- 17 unit tests
|
||||
|
||||
7. [x] **`GCEEncodingBackend`** ✅
|
||||
- Implements `EncodingBackend` interface
|
||||
- Wraps existing `EncodingService` (encoding_service.py)
|
||||
- Requires GCS paths in options for cloud execution
|
||||
- 6 additional unit tests
|
||||
|
||||
### Phase 3: Create Orchestrator ✅ COMPLETE
|
||||
|
||||
8. [x] **Create `backend/workers/video_worker_orchestrator.py`** ✅
|
||||
- New class that coordinates all stages
|
||||
- Replaces the branching logic in `generate_video()`
|
||||
- Methods:
|
||||
- `__init__(config, job_manager, storage, job_logger)`
|
||||
- `async run() -> OrchestratorResult`
|
||||
- `_run_packaging()` - CDG/TXT
|
||||
- `_run_encoding()` - GCE or local via EncodingBackend interface
|
||||
- `_run_organization()` - Brand code, folder structure
|
||||
- `_run_distribution()` - YouTube, Dropbox, GDrive
|
||||
- `_run_notifications()` - Discord
|
||||
- 27 unit tests (test_video_worker_orchestrator.py)
|
||||
|
||||
9. [x] **Update `backend/workers/video_worker.py`** ✅
|
||||
- Added `USE_NEW_ORCHESTRATOR` feature flag (default: true)
|
||||
- `generate_video()` routes to orchestrator or legacy path
|
||||
- `generate_video_orchestrated()` - new unified pipeline
|
||||
- `generate_video_legacy()` - original code for rollback
|
||||
- Legacy GCE and KaraokeFinalise paths preserved for safety
|
||||
|
||||
### Phase 4: Update KaraokeFinalise to Use Services (OPTIONAL)
|
||||
|
||||
10. [ ] **Update `karaoke_gen/karaoke_finalise/karaoke_finalise.py`**
|
||||
- Import and use the new services
|
||||
- Replace inline YouTube/Discord/CDG code with service calls
|
||||
- Keep process() working for CLI backwards compatibility
|
||||
- Services become the single source of truth
|
||||
- **NOTE**: This is optional for MVP - the orchestrator already uses the extracted services directly
|
||||
|
||||
### Phase 5: Testing & Validation ✅ COMPLETE
|
||||
|
||||
11. [x] **Add unit tests for each service** ✅
|
||||
- test_youtube_upload_service.py (23 tests)
|
||||
- test_discord_service.py (20 tests)
|
||||
- test_packaging_service.py (18 tests)
|
||||
- test_local_encoding_service.py (28 tests)
|
||||
- test_encoding_interface.py (23 tests)
|
||||
- test_video_worker_orchestrator.py (27 tests)
|
||||
- **Total: 139 new tests**
|
||||
|
||||
12. [x] **Full test suite passes** ✅
|
||||
- 976 tests passed (86 skipped - emulator/integration tests)
|
||||
- No regressions from changes
|
||||
|
||||
13. [ ] **Manual E2E testing** (After deployment)
|
||||
- [ ] GCE encoding job with YouTube upload enabled
|
||||
- [ ] Local encoding job with YouTube upload enabled
|
||||
- [ ] CLI karaoke-gen with all features
|
||||
- [ ] Job with CDG/TXT enabled
|
||||
- [ ] Verify output structure matches local CLI
|
||||
|
||||
## Files Created/Modified
|
||||
|
||||
| File | Action | Description | Status |
|
||||
|------|--------|-------------|--------|
|
||||
| `backend/services/youtube_upload_service.py` | Create | YouTube upload logic (~446 lines) | ✅ |
|
||||
| `backend/services/discord_service.py` | Create | Discord notification service (~167 lines) | ✅ |
|
||||
| `backend/services/packaging_service.py` | Create | CDG/TXT package generation (~266 lines) | ✅ |
|
||||
| `backend/services/local_encoding_service.py` | Create | Local FFmpeg encoding service (~512 lines) | ✅ |
|
||||
| `backend/services/encoding_interface.py` | Create | Abstract interface + backends (~418 lines) | ✅ |
|
||||
| `backend/workers/video_worker_orchestrator.py` | Create | Stage orchestration logic (~550 lines) | ✅ |
|
||||
| `backend/workers/video_worker.py` | Modify | Added orchestrator path + feature flag | ✅ |
|
||||
| `backend/tests/test_youtube_upload_service.py` | Create | 23 unit tests | ✅ |
|
||||
| `backend/tests/test_discord_service.py` | Create | 20 unit tests | ✅ |
|
||||
| `backend/tests/test_packaging_service.py` | Create | 18 unit tests | ✅ |
|
||||
| `backend/tests/test_local_encoding_service.py` | Create | 28 unit tests | ✅ |
|
||||
| `backend/tests/test_encoding_interface.py` | Create | 23 unit tests | ✅ |
|
||||
| `backend/tests/test_video_worker_orchestrator.py` | Create | 27 unit tests | ✅ |
|
||||
| `karaoke_gen/karaoke_finalise/karaoke_finalise.py` | Modify | Use extracted services (optional) | Deferred |
|
||||
|
||||
## Testing Strategy
|
||||
|
||||
### Unit Tests
|
||||
- **youtube_upload_service**: Mock YouTube API, test upload/delete/duplicate check
|
||||
- **discord_service**: Mock requests, test webhook POST
|
||||
- **packaging_service**: Use temp files, test CDG/TXT generation
|
||||
- **local_encoding_service**: Mock subprocess, test FFmpeg command construction
|
||||
|
||||
### Integration Tests
|
||||
- **video_worker_orchestrator**: Mock all services, test stage sequencing
|
||||
- **video_worker**: Test with mocked orchestrator
|
||||
|
||||
### E2E Verification
|
||||
- [ ] GCE encoding job with YouTube upload enabled → YouTube link appears
|
||||
- [ ] Local encoding job with YouTube upload enabled → Same behavior
|
||||
- [ ] CLI `karaoke-gen` with all features → No regression
|
||||
- [ ] Job with CDG/TXT enabled → Packages generated
|
||||
- [ ] Dropbox output structure → Matches local CLI exactly
|
||||
|
||||
## Open Questions
|
||||
|
||||
- [x] Should we keep KaraokeFinalise.process() or deprecate it?
|
||||
- **Decision**: Keep it for CLI backwards compatibility. It will use the new services internally.
|
||||
|
||||
- [x] Should local_encoding_service run FFmpeg sync or async?
|
||||
- **Decision**: Sync implementation, async wrapper via `asyncio.to_thread()` for interface compatibility.
|
||||
|
||||
- [x] Order of packaging vs encoding?
|
||||
- **Decision**: CDG/TXT runs BEFORE encoding (depends on LRC file, not videos). KaraokeFinalise does it this way too.
|
||||
|
||||
## Rollback Plan
|
||||
|
||||
1. The refactor is additive - new services don't replace old code until orchestrator is wired up
|
||||
2. Feature flag: `USE_NEW_ORCHESTRATOR=true` to enable new code path
|
||||
3. If issues found in production:
|
||||
- Set `USE_NEW_ORCHESTRATOR=false` to revert to old behavior
|
||||
- Old `generate_video()` code preserved behind feature flag
|
||||
4. After 1 week stable, remove feature flag and old code
|
||||
|
||||
## Success Criteria
|
||||
|
||||
1. Job `5b78ad46` scenario (GCE + YouTube) would upload to YouTube
|
||||
2. All 803+ existing tests pass
|
||||
3. E2E happy path test passes
|
||||
4. No regression in local CLI functionality
|
||||
5. Output structure matches local CLI exactly
|
||||
6. Code coverage maintained or improved
|
||||
|
||||
## Estimated Scope
|
||||
|
||||
- **Phase 1** (Extract Services): 4 new service files + tests (~400-600 lines each)
|
||||
- **Phase 2** (Interface): 1 new file, 2 modifications (~100 lines)
|
||||
- **Phase 3** (Orchestrator): 2 files (~300-400 lines)
|
||||
- **Phase 4** (KaraokeFinalise): 1 modification (~100 lines of changes)
|
||||
- **Phase 5** (Testing): Integration tests + manual verification
|
||||
|
||||
This is a medium-sized refactor that can be done incrementally. Each phase produces working, tested code.
|
||||
@@ -225,8 +225,7 @@ jobs:
|
||||
- name: Run Happy Path Production Test
|
||||
working-directory: frontend
|
||||
env:
|
||||
TESTMAIL_API_KEY: ${{ secrets.TESTMAIL_API_KEY }}
|
||||
TESTMAIL_NAMESPACE: ${{ secrets.TESTMAIL_NAMESPACE }}
|
||||
MAILSLURP_API_KEY: ${{ secrets.MAILSLURP_API_KEY }}
|
||||
E2E_TEST_TOKEN: ${{ secrets.E2E_TEST_TOKEN }}
|
||||
E2E_ADMIN_TOKEN: ${{ secrets.E2E_ADMIN_TOKEN }}
|
||||
run: |
|
||||
|
||||
@@ -70,9 +70,8 @@ jobs:
|
||||
- name: Run E2E Happy Path Test
|
||||
working-directory: frontend
|
||||
env:
|
||||
TESTMAIL_API_KEY: ${{ secrets.TESTMAIL_API_KEY }}
|
||||
TESTMAIL_NAMESPACE: ${{ secrets.TESTMAIL_NAMESPACE }}
|
||||
E2E_TEST_TOKEN: ${{ inputs.use_test_token && secrets.E2E_TEST_TOKEN || '' }}
|
||||
MAILSLURP_API_KEY: ${{ secrets.MAILSLURP_API_KEY }}
|
||||
E2E_TEST_TOKEN: ${{ inputs.use_test_token == 'true' && secrets.E2E_TEST_TOKEN || '' }}
|
||||
E2E_ADMIN_TOKEN: ${{ secrets.E2E_ADMIN_TOKEN }}
|
||||
DEBUG_MODE: ${{ inputs.debug_mode || 'false' }}
|
||||
run: |
|
||||
|
||||
@@ -18,23 +18,6 @@ RUN apt-get update && apt-get install -y \
|
||||
xz-utils \
|
||||
&& rm -rf /var/lib/apt/lists/*
|
||||
|
||||
# Install comprehensive font support for international lyrics and symbols
|
||||
# Noto fonts provide full Unicode coverage including:
|
||||
# - Latin, Greek, Cyrillic (core)
|
||||
# - CJK: Chinese, Japanese, Korean (~150MB)
|
||||
# - Arabic, Hebrew, Thai, Hindi, and other scripts (extra)
|
||||
# - Color emoji (limited support in libass, but available)
|
||||
# - Musical symbols (♪ ♫ etc.)
|
||||
# Note: This adds ~200-300MB to the image but ensures all languages render correctly
|
||||
RUN apt-get update && apt-get install -y \
|
||||
fonts-noto-core \
|
||||
fonts-noto-cjk \
|
||||
fonts-noto-extra \
|
||||
fonts-noto-color-emoji \
|
||||
fontconfig \
|
||||
&& fc-cache -f \
|
||||
&& rm -rf /var/lib/apt/lists/*
|
||||
|
||||
# Install optimized static FFmpeg build (7.0.2) instead of Debian package (5.1.8)
|
||||
# John Van Sickle's builds are compiled with better optimizations for x86_64
|
||||
# This provides ~20-40% performance improvement for encoding tasks
|
||||
|
||||
+2
-385
@@ -5,20 +5,18 @@ Handles:
|
||||
- Dashboard overview statistics
|
||||
- System-wide metrics
|
||||
- Admin-only operations
|
||||
- Audio search cache management
|
||||
"""
|
||||
import logging
|
||||
from datetime import datetime, timedelta
|
||||
from typing import Tuple, List, Optional, Any, Dict
|
||||
from typing import Tuple
|
||||
|
||||
from fastapi import APIRouter, Depends, HTTPException
|
||||
from fastapi import APIRouter, Depends
|
||||
from pydantic import BaseModel
|
||||
|
||||
from backend.api.dependencies import require_admin
|
||||
from backend.services.auth_service import UserType
|
||||
from backend.services.user_service import get_user_service, UserService, USERS_COLLECTION
|
||||
from backend.services.job_manager import JobManager
|
||||
from backend.services.flacfetch_client import get_flacfetch_client, FlacfetchServiceError
|
||||
from backend.models.job import JobStatus
|
||||
|
||||
|
||||
@@ -199,384 +197,3 @@ async def get_admin_stats_overview(
|
||||
total_credits_issued_30d=total_credits_issued_30d,
|
||||
total_beta_testers=total_beta_testers,
|
||||
)
|
||||
|
||||
|
||||
# =============================================================================
|
||||
# Audio Search Management Models
|
||||
# =============================================================================
|
||||
|
||||
class AudioSearchResultSummary(BaseModel):
|
||||
"""Summary of a single audio search result."""
|
||||
index: int
|
||||
provider: str
|
||||
artist: str
|
||||
title: str
|
||||
is_lossless: bool
|
||||
quality: Optional[str] = None
|
||||
seeders: Optional[int] = None
|
||||
|
||||
|
||||
class AudioSearchJobSummary(BaseModel):
|
||||
"""Summary of a job with audio search results."""
|
||||
job_id: str
|
||||
status: str
|
||||
user_email: Optional[str] = None
|
||||
audio_search_artist: Optional[str] = None
|
||||
audio_search_title: Optional[str] = None
|
||||
created_at: Optional[datetime] = None
|
||||
results_count: int
|
||||
results_summary: List[AudioSearchResultSummary]
|
||||
has_lossless: bool
|
||||
providers: List[str]
|
||||
|
||||
|
||||
class AudioSearchListResponse(BaseModel):
|
||||
"""Response for listing audio search jobs."""
|
||||
jobs: List[AudioSearchJobSummary]
|
||||
total: int
|
||||
|
||||
|
||||
class ClearSearchCacheResponse(BaseModel):
|
||||
"""Response for clearing search cache."""
|
||||
status: str
|
||||
job_id: str
|
||||
message: str
|
||||
previous_status: str
|
||||
new_status: str
|
||||
results_cleared: int
|
||||
flacfetch_cache_cleared: bool = False
|
||||
flacfetch_error: Optional[str] = None
|
||||
|
||||
|
||||
class ClearAllCacheResponse(BaseModel):
|
||||
"""Response for clearing all flacfetch cache."""
|
||||
status: str
|
||||
message: str
|
||||
deleted_count: int
|
||||
|
||||
|
||||
class CacheStatsResponse(BaseModel):
|
||||
"""Response for cache statistics."""
|
||||
count: int
|
||||
total_size_bytes: int
|
||||
oldest_entry: Optional[str] = None
|
||||
newest_entry: Optional[str] = None
|
||||
configured: bool
|
||||
|
||||
|
||||
# =============================================================================
|
||||
# Audio Search Management Endpoints
|
||||
# =============================================================================
|
||||
|
||||
@router.get("/audio-searches", response_model=AudioSearchListResponse)
|
||||
async def list_audio_searches(
|
||||
limit: int = 50,
|
||||
status_filter: Optional[str] = None,
|
||||
auth_data: Tuple[str, UserType, int] = Depends(require_admin),
|
||||
user_service: UserService = Depends(get_user_service),
|
||||
):
|
||||
"""
|
||||
List jobs with audio search results.
|
||||
|
||||
Returns jobs that have cached audio search results, useful for:
|
||||
- Monitoring search activity
|
||||
- Identifying stale cached results
|
||||
- Clearing cache for specific jobs
|
||||
|
||||
Args:
|
||||
limit: Maximum number of jobs to return (default 50)
|
||||
status_filter: Optional filter by job status (e.g., 'awaiting_audio_selection')
|
||||
"""
|
||||
from google.cloud.firestore_v1 import FieldFilter
|
||||
|
||||
db = user_service.db
|
||||
jobs_collection = db.collection("jobs")
|
||||
|
||||
# Query jobs - we'll filter for those with audio_search_results in Python
|
||||
# since Firestore can't query for existence of nested fields efficiently
|
||||
query = jobs_collection.order_by("created_at", direction="DESCENDING").limit(500)
|
||||
|
||||
if status_filter:
|
||||
query = jobs_collection.where(
|
||||
filter=FieldFilter("status", "==", status_filter)
|
||||
).order_by("created_at", direction="DESCENDING").limit(500)
|
||||
|
||||
jobs_with_searches = []
|
||||
|
||||
for doc in query.stream():
|
||||
data = doc.to_dict()
|
||||
state_data = data.get("state_data", {})
|
||||
audio_results = state_data.get("audio_search_results", [])
|
||||
|
||||
if not audio_results:
|
||||
continue
|
||||
|
||||
# Compute has_lossless and providers from ALL results (not just first 10)
|
||||
has_lossless = any(r.get("is_lossless", False) for r in audio_results)
|
||||
providers = {r.get("provider", "Unknown") for r in audio_results}
|
||||
|
||||
# Build summary from first 10 results only
|
||||
results_summary = []
|
||||
for r in audio_results[:10]:
|
||||
results_summary.append(AudioSearchResultSummary(
|
||||
index=r.get("index", 0),
|
||||
provider=r.get("provider", "Unknown"),
|
||||
artist=r.get("artist", ""),
|
||||
title=r.get("title", ""),
|
||||
is_lossless=r.get("is_lossless", False),
|
||||
quality=r.get("quality"),
|
||||
seeders=r.get("seeders"),
|
||||
))
|
||||
|
||||
jobs_with_searches.append(AudioSearchJobSummary(
|
||||
job_id=doc.id,
|
||||
status=data.get("status", "unknown"),
|
||||
user_email=data.get("user_email"),
|
||||
audio_search_artist=data.get("audio_search_artist"),
|
||||
audio_search_title=data.get("audio_search_title"),
|
||||
created_at=data.get("created_at"),
|
||||
results_count=len(audio_results),
|
||||
results_summary=results_summary,
|
||||
has_lossless=has_lossless,
|
||||
providers=sorted(providers),
|
||||
))
|
||||
|
||||
if len(jobs_with_searches) >= limit:
|
||||
break
|
||||
|
||||
return AudioSearchListResponse(
|
||||
jobs=jobs_with_searches,
|
||||
total=len(jobs_with_searches),
|
||||
)
|
||||
|
||||
|
||||
@router.post("/audio-searches/{job_id}/clear-cache", response_model=ClearSearchCacheResponse)
|
||||
async def clear_audio_search_cache(
|
||||
job_id: str,
|
||||
auth_data: Tuple[str, UserType, int] = Depends(require_admin),
|
||||
user_service: UserService = Depends(get_user_service),
|
||||
):
|
||||
"""
|
||||
Clear the cached audio search results for a job.
|
||||
|
||||
This will:
|
||||
1. Remove the cached search results from job.state_data
|
||||
2. Reset the job status to 'pending' so a new search can be performed
|
||||
3. Clear the flacfetch GCS cache for this artist/title (if available)
|
||||
|
||||
Use this when:
|
||||
- Cached results are stale (e.g., flacfetch was updated)
|
||||
- User wants to search again with different terms
|
||||
- Results appear incomplete or incorrect
|
||||
"""
|
||||
job_manager = JobManager()
|
||||
job = job_manager.get_job(job_id)
|
||||
if not job:
|
||||
raise HTTPException(status_code=404, detail=f"Job {job_id} not found")
|
||||
|
||||
# Get current state
|
||||
state_data = job.state_data or {}
|
||||
audio_results = state_data.get("audio_search_results", [])
|
||||
results_count = len(audio_results)
|
||||
previous_status = job.status
|
||||
|
||||
if not audio_results:
|
||||
raise HTTPException(
|
||||
status_code=400,
|
||||
detail=f"Job {job_id} has no cached audio search results"
|
||||
)
|
||||
|
||||
# Validate job status - only allow cache clear for appropriate states
|
||||
# Don't allow clearing cache for jobs that are actively processing or complete
|
||||
forbidden_statuses = {
|
||||
"downloading", "downloading_audio", "searching_audio",
|
||||
"separating_stage1", "separating_stage2", "transcribing", "correcting",
|
||||
"generating_screens", "applying_padding", "rendering_video",
|
||||
"generating_video", "encoding", "packaging", "uploading",
|
||||
"complete", "prep_complete",
|
||||
}
|
||||
if previous_status in forbidden_statuses:
|
||||
raise HTTPException(
|
||||
status_code=400,
|
||||
detail=f"Cannot clear cache for job in '{previous_status}' state. "
|
||||
f"Only jobs in pending, awaiting_audio_selection, failed, or cancelled states can have cache cleared."
|
||||
)
|
||||
|
||||
# Clear the cache by removing the keys
|
||||
db = user_service.db
|
||||
job_ref = db.collection("jobs").document(job_id)
|
||||
|
||||
from google.cloud.firestore_v1 import DELETE_FIELD
|
||||
|
||||
# Update job: clear cache and reset status
|
||||
job_ref.update({
|
||||
"state_data.audio_search_results": DELETE_FIELD,
|
||||
"state_data.audio_search_count": DELETE_FIELD,
|
||||
"state_data.remote_search_id": DELETE_FIELD,
|
||||
"status": "pending",
|
||||
"progress": 0,
|
||||
"message": "Audio search cache cleared by admin. Ready for new search.",
|
||||
"updated_at": datetime.utcnow(),
|
||||
})
|
||||
|
||||
# Also clear flacfetch's GCS cache if we have artist/title
|
||||
flacfetch_cache_cleared = False
|
||||
flacfetch_error = None
|
||||
artist = job.audio_search_artist
|
||||
title = job.audio_search_title
|
||||
|
||||
if artist and title:
|
||||
flacfetch_client = get_flacfetch_client()
|
||||
if flacfetch_client:
|
||||
try:
|
||||
flacfetch_cache_cleared = await flacfetch_client.clear_search_cache(artist, title)
|
||||
logger.info(
|
||||
f"Cleared flacfetch cache for '{artist}' - '{title}': "
|
||||
f"{'deleted' if flacfetch_cache_cleared else 'no entry found'}"
|
||||
)
|
||||
except FlacfetchServiceError as e:
|
||||
flacfetch_error = str(e)
|
||||
logger.warning(f"Failed to clear flacfetch cache: {e}")
|
||||
else:
|
||||
flacfetch_error = "flacfetch client not configured"
|
||||
logger.debug("Skipping flacfetch cache clear - client not configured")
|
||||
else:
|
||||
flacfetch_error = "missing artist or title"
|
||||
logger.debug(f"Skipping flacfetch cache clear - missing artist ({artist}) or title ({title})")
|
||||
|
||||
logger.info(
|
||||
f"Admin {auth_data[0]} cleared audio search cache for job {job_id}. "
|
||||
f"Cleared {results_count} results. Status changed from {previous_status} to pending. "
|
||||
f"Flacfetch cache cleared: {flacfetch_cache_cleared}"
|
||||
)
|
||||
|
||||
message = f"Cleared {results_count} cached search results. Job reset to pending."
|
||||
if flacfetch_cache_cleared:
|
||||
message += " Flacfetch cache also cleared."
|
||||
elif flacfetch_error:
|
||||
message += f" Note: flacfetch cache not cleared ({flacfetch_error})."
|
||||
|
||||
return ClearSearchCacheResponse(
|
||||
status="success",
|
||||
job_id=job_id,
|
||||
message=message,
|
||||
previous_status=previous_status,
|
||||
new_status="pending",
|
||||
results_cleared=results_count,
|
||||
flacfetch_cache_cleared=flacfetch_cache_cleared,
|
||||
flacfetch_error=flacfetch_error,
|
||||
)
|
||||
|
||||
|
||||
@router.post("/jobs/{job_id}/reset-worker-state")
|
||||
async def reset_worker_state(
|
||||
job_id: str,
|
||||
auth_data: Tuple[str, UserType, int] = Depends(require_admin),
|
||||
):
|
||||
"""
|
||||
Reset stale worker progress state for a job.
|
||||
|
||||
This clears the video_progress, render_progress, and screens_progress
|
||||
from state_data, allowing workers to be re-triggered.
|
||||
|
||||
Use this when a job is stuck because worker progress shows 'running'
|
||||
from a previous failed attempt.
|
||||
"""
|
||||
from backend.services.job_manager import JobManager
|
||||
|
||||
job_manager = JobManager()
|
||||
job = job_manager.get_job(job_id)
|
||||
|
||||
if not job:
|
||||
raise HTTPException(status_code=404, detail=f"Job {job_id} not found")
|
||||
|
||||
# Reset worker progress states
|
||||
job_manager.update_state_data(job_id, 'video_progress', {'stage': 'pending'})
|
||||
job_manager.update_state_data(job_id, 'render_progress', {'stage': 'pending'})
|
||||
job_manager.update_state_data(job_id, 'screens_progress', {'stage': 'pending'})
|
||||
|
||||
logger.info(f"Admin {auth_data[0]} reset worker state for job {job_id}")
|
||||
|
||||
return {
|
||||
"status": "success",
|
||||
"job_id": job_id,
|
||||
"message": "Worker progress states reset to pending"
|
||||
}
|
||||
|
||||
|
||||
@router.delete("/cache", response_model=ClearAllCacheResponse)
|
||||
async def clear_all_flacfetch_cache(
|
||||
auth_data: Tuple[str, UserType, int] = Depends(require_admin),
|
||||
):
|
||||
"""
|
||||
Clear the entire flacfetch search cache.
|
||||
|
||||
This will delete all cached search results from flacfetch's GCS cache.
|
||||
Use with caution - this will cause all subsequent searches to hit
|
||||
the trackers fresh.
|
||||
|
||||
Note: This does NOT clear Firestore job.state_data caches, only the
|
||||
flacfetch-side GCS cache.
|
||||
"""
|
||||
flacfetch_client = get_flacfetch_client()
|
||||
if not flacfetch_client:
|
||||
raise HTTPException(
|
||||
status_code=503,
|
||||
detail="flacfetch client not configured"
|
||||
)
|
||||
|
||||
try:
|
||||
deleted_count = await flacfetch_client.clear_all_cache()
|
||||
logger.info(
|
||||
f"Admin {auth_data[0]} cleared all flacfetch cache. "
|
||||
f"Deleted {deleted_count} entries."
|
||||
)
|
||||
return ClearAllCacheResponse(
|
||||
status="success",
|
||||
message=f"Cleared {deleted_count} cache entries from flacfetch.",
|
||||
deleted_count=deleted_count,
|
||||
)
|
||||
except FlacfetchServiceError as e:
|
||||
logger.error(f"Failed to clear all flacfetch cache: {e}")
|
||||
raise HTTPException(
|
||||
status_code=502,
|
||||
detail=f"Failed to clear flacfetch cache: {e}"
|
||||
)
|
||||
|
||||
|
||||
@router.get("/cache/stats", response_model=CacheStatsResponse)
|
||||
async def get_flacfetch_cache_stats(
|
||||
auth_data: Tuple[str, UserType, int] = Depends(require_admin),
|
||||
):
|
||||
"""
|
||||
Get statistics about the flacfetch search cache.
|
||||
|
||||
Returns information about:
|
||||
- Number of cached entries
|
||||
- Total size in bytes
|
||||
- Oldest and newest cache entries
|
||||
- Whether cache is configured
|
||||
"""
|
||||
flacfetch_client = get_flacfetch_client()
|
||||
if not flacfetch_client:
|
||||
raise HTTPException(
|
||||
status_code=503,
|
||||
detail="flacfetch client not configured"
|
||||
)
|
||||
|
||||
try:
|
||||
stats = await flacfetch_client.get_cache_stats()
|
||||
return CacheStatsResponse(
|
||||
count=stats.get("count", 0),
|
||||
total_size_bytes=stats.get("total_size_bytes", 0),
|
||||
oldest_entry=stats.get("oldest_entry"),
|
||||
newest_entry=stats.get("newest_entry"),
|
||||
configured=stats.get("configured", True),
|
||||
)
|
||||
except FlacfetchServiceError as e:
|
||||
logger.error(f"Failed to get flacfetch cache stats: {e}")
|
||||
raise HTTPException(
|
||||
status_code=502,
|
||||
detail=f"Failed to get cache stats: {e}"
|
||||
)
|
||||
# Trigger deployment
|
||||
@@ -1291,13 +1291,12 @@ async def retry_job(
|
||||
|
||||
logger.info(f"Job {job_id}: Has rendered video and instrumental selection, retrying video generation")
|
||||
|
||||
# Clear error state and reset worker progress for idempotency
|
||||
# Clear error state
|
||||
job_manager.update_job(job_id, {
|
||||
'error_message': None,
|
||||
'error_details': None,
|
||||
})
|
||||
job_manager.update_state_data(job_id, 'video_progress', {'stage': 'pending'})
|
||||
|
||||
|
||||
# Reset to INSTRUMENTAL_SELECTED and trigger video worker
|
||||
if not job_manager.transition_to_state(
|
||||
job_id=job_id,
|
||||
@@ -1325,14 +1324,13 @@ async def retry_job(
|
||||
file_urls.get('screens', {}).get('title')):
|
||||
|
||||
logger.info(f"Job {job_id}: Has corrections and screens, retrying from render stage")
|
||||
|
||||
# Clear error state and reset worker progress for idempotency
|
||||
|
||||
# Clear error state
|
||||
job_manager.update_job(job_id, {
|
||||
'error_message': None,
|
||||
'error_details': None,
|
||||
})
|
||||
job_manager.update_state_data(job_id, 'render_progress', {'stage': 'pending'})
|
||||
|
||||
|
||||
# Reset to REVIEW_COMPLETE and trigger render worker
|
||||
if not job_manager.transition_to_state(
|
||||
job_id=job_id,
|
||||
@@ -1360,14 +1358,13 @@ async def retry_job(
|
||||
file_urls.get('lyrics', {}).get('corrections')):
|
||||
|
||||
logger.info(f"Job {job_id}: Has stems and corrections, retrying from screens stage")
|
||||
|
||||
# Clear error state and reset worker progress for idempotency
|
||||
|
||||
# Clear error state
|
||||
job_manager.update_job(job_id, {
|
||||
'error_message': None,
|
||||
'error_details': None,
|
||||
})
|
||||
job_manager.update_state_data(job_id, 'screens_progress', {'stage': 'pending'})
|
||||
|
||||
|
||||
# Reset to a state before screens and trigger screens worker
|
||||
if not job_manager.transition_to_state(
|
||||
job_id=job_id,
|
||||
|
||||
@@ -1,172 +0,0 @@
|
||||
"""
|
||||
Discord Notification Service.
|
||||
|
||||
Provides Discord webhook notification functionality, extracted from KaraokeFinalise
|
||||
for use by both the cloud backend (video_worker) and local CLI.
|
||||
|
||||
This service handles:
|
||||
- Posting messages to Discord webhooks
|
||||
- Video upload notifications
|
||||
- Validation of webhook URLs
|
||||
"""
|
||||
|
||||
import logging
|
||||
import re
|
||||
from typing import Optional
|
||||
|
||||
import requests
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
|
||||
class DiscordNotificationService:
|
||||
"""
|
||||
Service for sending Discord webhook notifications.
|
||||
|
||||
Supports posting messages to Discord channels via webhooks,
|
||||
commonly used to notify when new karaoke videos are uploaded.
|
||||
"""
|
||||
|
||||
# Discord webhook URL pattern
|
||||
WEBHOOK_URL_PATTERN = re.compile(
|
||||
r"^https://discord\.com/api/webhooks/\d+/[a-zA-Z0-9_-]+$"
|
||||
)
|
||||
|
||||
def __init__(
|
||||
self,
|
||||
webhook_url: Optional[str] = None,
|
||||
dry_run: bool = False,
|
||||
logger: Optional[logging.Logger] = None,
|
||||
):
|
||||
"""
|
||||
Initialize the Discord notification service.
|
||||
|
||||
Args:
|
||||
webhook_url: Discord webhook URL for posting notifications
|
||||
dry_run: If True, log actions without performing them
|
||||
logger: Optional logger instance
|
||||
"""
|
||||
self.webhook_url = webhook_url
|
||||
self.dry_run = dry_run
|
||||
self.logger = logger or logging.getLogger(__name__)
|
||||
|
||||
# Validate webhook URL if provided
|
||||
if webhook_url:
|
||||
self._validate_webhook_url(webhook_url)
|
||||
|
||||
def _validate_webhook_url(self, url: str) -> None:
|
||||
"""
|
||||
Validate that the webhook URL is a valid Discord webhook URL.
|
||||
|
||||
Args:
|
||||
url: URL to validate
|
||||
|
||||
Raises:
|
||||
ValueError: If URL is not a valid Discord webhook URL
|
||||
"""
|
||||
url = url.strip()
|
||||
if not url.startswith("https://discord.com/api/webhooks/"):
|
||||
raise ValueError(
|
||||
f"Invalid Discord webhook URL: {url}. "
|
||||
"URL must start with 'https://discord.com/api/webhooks/'"
|
||||
)
|
||||
|
||||
def post_message(
|
||||
self,
|
||||
message: str,
|
||||
webhook_url: Optional[str] = None,
|
||||
) -> bool:
|
||||
"""
|
||||
Post a message to a Discord channel via webhook.
|
||||
|
||||
Args:
|
||||
message: The message content to post
|
||||
webhook_url: Optional webhook URL (uses instance URL if not provided)
|
||||
|
||||
Returns:
|
||||
True if message was posted successfully, False otherwise
|
||||
|
||||
Raises:
|
||||
ValueError: If no webhook URL is available
|
||||
requests.HTTPError: If the webhook request fails
|
||||
"""
|
||||
url = webhook_url or self.webhook_url
|
||||
if not url:
|
||||
raise ValueError("No Discord webhook URL provided")
|
||||
|
||||
url = url.strip()
|
||||
self._validate_webhook_url(url)
|
||||
|
||||
if self.dry_run:
|
||||
self.logger.info(
|
||||
f"DRY RUN: Would post Discord message: '{message}' to {url}"
|
||||
)
|
||||
return True
|
||||
|
||||
self.logger.info(f"Posting message to Discord webhook...")
|
||||
data = {"content": message}
|
||||
response = requests.post(url, json=data, timeout=30)
|
||||
response.raise_for_status()
|
||||
self.logger.info("Message posted to Discord successfully")
|
||||
return True
|
||||
|
||||
def post_video_notification(
|
||||
self,
|
||||
youtube_url: str,
|
||||
webhook_url: Optional[str] = None,
|
||||
) -> bool:
|
||||
"""
|
||||
Post a notification about a new video upload.
|
||||
|
||||
Args:
|
||||
youtube_url: The YouTube URL of the uploaded video
|
||||
webhook_url: Optional webhook URL (uses instance URL if not provided)
|
||||
|
||||
Returns:
|
||||
True if notification was posted successfully, False otherwise
|
||||
"""
|
||||
if not youtube_url:
|
||||
self.logger.info("Skipping Discord notification - no YouTube URL available")
|
||||
return False
|
||||
|
||||
message = f"New upload: {youtube_url}"
|
||||
return self.post_message(message, webhook_url)
|
||||
|
||||
def is_enabled(self) -> bool:
|
||||
"""
|
||||
Check if Discord notifications are enabled.
|
||||
|
||||
Returns:
|
||||
True if a valid webhook URL is configured
|
||||
"""
|
||||
return bool(self.webhook_url)
|
||||
|
||||
|
||||
# Singleton instance and factory function (following existing service pattern)
|
||||
_discord_notification_service: Optional[DiscordNotificationService] = None
|
||||
|
||||
|
||||
def get_discord_notification_service(
|
||||
webhook_url: Optional[str] = None,
|
||||
**kwargs
|
||||
) -> DiscordNotificationService:
|
||||
"""
|
||||
Get a Discord notification service instance.
|
||||
|
||||
Args:
|
||||
webhook_url: Discord webhook URL for notifications
|
||||
**kwargs: Additional arguments passed to DiscordNotificationService
|
||||
|
||||
Returns:
|
||||
DiscordNotificationService instance
|
||||
"""
|
||||
global _discord_notification_service
|
||||
|
||||
# Create new instance if webhook URL changed or not yet created
|
||||
if _discord_notification_service is None or webhook_url:
|
||||
_discord_notification_service = DiscordNotificationService(
|
||||
webhook_url=webhook_url,
|
||||
**kwargs
|
||||
)
|
||||
|
||||
return _discord_notification_service
|
||||
@@ -1,429 +0,0 @@
|
||||
"""
|
||||
Encoding Interface.
|
||||
|
||||
Defines the abstract interface for video encoding backends, allowing
|
||||
the video worker orchestrator to use either GCE or local encoding
|
||||
interchangeably.
|
||||
|
||||
This follows the Strategy pattern - different encoding implementations
|
||||
can be swapped without changing the orchestration logic.
|
||||
"""
|
||||
|
||||
import logging
|
||||
from abc import ABC, abstractmethod
|
||||
from dataclasses import dataclass, field
|
||||
from typing import Dict, List, Optional, Any
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
|
||||
@dataclass
|
||||
class EncodingInput:
|
||||
"""
|
||||
Input configuration for video encoding.
|
||||
|
||||
Contains all the paths and metadata needed to encode a karaoke video.
|
||||
"""
|
||||
# Required input files
|
||||
title_video_path: str # Title card video (MOV)
|
||||
karaoke_video_path: str # Main karaoke video with vocals (MOV/MKV)
|
||||
instrumental_audio_path: str # Instrumental audio track (FLAC)
|
||||
|
||||
# Optional input files
|
||||
end_video_path: Optional[str] = None # End credits video (MOV)
|
||||
|
||||
# Metadata for output naming
|
||||
artist: str = ""
|
||||
title: str = ""
|
||||
brand_code: Optional[str] = None
|
||||
|
||||
# Output directory
|
||||
output_dir: str = ""
|
||||
|
||||
# Additional options
|
||||
options: Dict[str, Any] = field(default_factory=dict)
|
||||
|
||||
|
||||
@dataclass
|
||||
class EncodingOutput:
|
||||
"""
|
||||
Output from video encoding.
|
||||
|
||||
Contains paths to all generated video files and status information.
|
||||
"""
|
||||
success: bool
|
||||
error_message: Optional[str] = None
|
||||
|
||||
# Output file paths (relative to output_dir or absolute)
|
||||
karaoke_mp4_path: Optional[str] = None # Karaoke video with instrumental audio
|
||||
with_vocals_mp4_path: Optional[str] = None # With vocals MP4 version
|
||||
lossless_4k_mp4_path: Optional[str] = None # Final lossless 4K MP4
|
||||
lossy_4k_mp4_path: Optional[str] = None # Final lossy 4K MP4
|
||||
lossless_mkv_path: Optional[str] = None # MKV with FLAC (for YouTube)
|
||||
lossy_720p_mp4_path: Optional[str] = None # 720p web version
|
||||
|
||||
# All output files as a dict for convenience
|
||||
output_files: Dict[str, str] = field(default_factory=dict)
|
||||
|
||||
# Encoding metadata
|
||||
encoding_time_seconds: Optional[float] = None
|
||||
encoding_backend: Optional[str] = None # "gce" or "local"
|
||||
|
||||
|
||||
class EncodingBackend(ABC):
|
||||
"""
|
||||
Abstract base class for video encoding backends.
|
||||
|
||||
Implementations:
|
||||
- GCEEncodingBackend: Cloud-based encoding using Google Compute Engine
|
||||
- LocalEncodingBackend: Local FFmpeg-based encoding
|
||||
"""
|
||||
|
||||
@property
|
||||
@abstractmethod
|
||||
def name(self) -> str:
|
||||
"""Return the name of this encoding backend."""
|
||||
pass
|
||||
|
||||
@abstractmethod
|
||||
async def encode(self, input_config: EncodingInput) -> EncodingOutput:
|
||||
"""
|
||||
Encode video files according to the input configuration.
|
||||
|
||||
This method should:
|
||||
1. Create karaoke video with instrumental audio
|
||||
2. Convert with-vocals video to MP4 if needed
|
||||
3. Encode lossless 4K MP4 (concatenating title + karaoke + end)
|
||||
4. Encode lossy 4K MP4 with AAC audio
|
||||
5. Create MKV with FLAC audio for YouTube
|
||||
6. Encode 720p version for web
|
||||
|
||||
Args:
|
||||
input_config: EncodingInput with all required paths and options
|
||||
|
||||
Returns:
|
||||
EncodingOutput with paths to generated files and status
|
||||
"""
|
||||
pass
|
||||
|
||||
@abstractmethod
|
||||
async def is_available(self) -> bool:
|
||||
"""
|
||||
Check if this encoding backend is available and configured.
|
||||
|
||||
Returns:
|
||||
True if the backend can be used, False otherwise
|
||||
"""
|
||||
pass
|
||||
|
||||
@abstractmethod
|
||||
async def get_status(self) -> Dict[str, Any]:
|
||||
"""
|
||||
Get the current status of the encoding backend.
|
||||
|
||||
Returns:
|
||||
Dict with status information (availability, queue length, etc.)
|
||||
"""
|
||||
pass
|
||||
|
||||
|
||||
class LocalEncodingBackend(EncodingBackend):
|
||||
"""
|
||||
Local FFmpeg-based encoding backend.
|
||||
|
||||
Wraps the LocalEncodingService to implement the EncodingBackend interface.
|
||||
Uses asyncio.to_thread() to run synchronous FFmpeg operations asynchronously.
|
||||
"""
|
||||
|
||||
def __init__(
|
||||
self,
|
||||
dry_run: bool = False,
|
||||
logger: Optional[logging.Logger] = None,
|
||||
):
|
||||
"""
|
||||
Initialize the local encoding backend.
|
||||
|
||||
Args:
|
||||
dry_run: If True, log operations without executing
|
||||
logger: Optional logger instance
|
||||
"""
|
||||
self.dry_run = dry_run
|
||||
self.logger = logger or logging.getLogger(__name__)
|
||||
self._service = None
|
||||
|
||||
@property
|
||||
def name(self) -> str:
|
||||
return "local"
|
||||
|
||||
def _get_service(self):
|
||||
"""Lazy-load the local encoding service."""
|
||||
if self._service is None:
|
||||
from backend.services.local_encoding_service import LocalEncodingService
|
||||
self._service = LocalEncodingService(
|
||||
dry_run=self.dry_run,
|
||||
logger=self.logger
|
||||
)
|
||||
return self._service
|
||||
|
||||
async def encode(self, input_config: EncodingInput) -> EncodingOutput:
|
||||
"""
|
||||
Encode video using local FFmpeg.
|
||||
|
||||
Args:
|
||||
input_config: Encoding input configuration
|
||||
|
||||
Returns:
|
||||
EncodingOutput with results
|
||||
"""
|
||||
import asyncio
|
||||
import time
|
||||
from backend.services.local_encoding_service import EncodingConfig
|
||||
|
||||
start_time = time.time()
|
||||
|
||||
# Build output file paths
|
||||
base_name = f"{input_config.artist} - {input_config.title}"
|
||||
output_dir = input_config.output_dir or "."
|
||||
|
||||
import os
|
||||
config = EncodingConfig(
|
||||
title_video=input_config.title_video_path,
|
||||
karaoke_video=input_config.karaoke_video_path,
|
||||
instrumental_audio=input_config.instrumental_audio_path,
|
||||
end_video=input_config.end_video_path,
|
||||
output_karaoke_mp4=os.path.join(output_dir, f"{base_name} (Karaoke).mp4"),
|
||||
output_with_vocals_mp4=os.path.join(output_dir, f"{base_name} (With Vocals).mp4"),
|
||||
output_lossless_4k_mp4=os.path.join(output_dir, f"{base_name} (Final Karaoke Lossless 4k).mp4"),
|
||||
output_lossy_4k_mp4=os.path.join(output_dir, f"{base_name} (Final Karaoke Lossy 4k).mp4"),
|
||||
output_lossless_mkv=os.path.join(output_dir, f"{base_name} (Final Karaoke Lossless 4k).mkv"),
|
||||
output_720p_mp4=os.path.join(output_dir, f"{base_name} (Final Karaoke Lossy 720p).mp4"),
|
||||
)
|
||||
|
||||
# Run encoding in thread pool to avoid blocking
|
||||
service = self._get_service()
|
||||
result = await asyncio.to_thread(service.encode_all_formats, config)
|
||||
|
||||
encoding_time = time.time() - start_time
|
||||
|
||||
if result.success:
|
||||
return EncodingOutput(
|
||||
success=True,
|
||||
karaoke_mp4_path=config.output_karaoke_mp4,
|
||||
with_vocals_mp4_path=config.output_with_vocals_mp4,
|
||||
lossless_4k_mp4_path=config.output_lossless_4k_mp4,
|
||||
lossy_4k_mp4_path=config.output_lossy_4k_mp4,
|
||||
lossless_mkv_path=config.output_lossless_mkv,
|
||||
lossy_720p_mp4_path=config.output_720p_mp4,
|
||||
output_files=result.output_files,
|
||||
encoding_time_seconds=encoding_time,
|
||||
encoding_backend=self.name
|
||||
)
|
||||
else:
|
||||
return EncodingOutput(
|
||||
success=False,
|
||||
error_message=result.error,
|
||||
output_files=result.output_files,
|
||||
encoding_time_seconds=encoding_time,
|
||||
encoding_backend=self.name
|
||||
)
|
||||
|
||||
async def is_available(self) -> bool:
|
||||
"""Check if local encoding is available (FFmpeg installed)."""
|
||||
import subprocess
|
||||
import asyncio
|
||||
|
||||
try:
|
||||
result = await asyncio.to_thread(
|
||||
subprocess.run,
|
||||
["ffmpeg", "-version"],
|
||||
capture_output=True,
|
||||
timeout=10
|
||||
)
|
||||
return result.returncode == 0
|
||||
except (FileNotFoundError, subprocess.TimeoutExpired):
|
||||
return False
|
||||
|
||||
async def get_status(self) -> Dict[str, Any]:
|
||||
"""Get local encoding status."""
|
||||
available = await self.is_available()
|
||||
service = self._get_service()
|
||||
|
||||
return {
|
||||
"backend": self.name,
|
||||
"available": available,
|
||||
"hwaccel_available": service.hwaccel_available if available else False,
|
||||
"video_encoder": service.video_encoder if available else None,
|
||||
}
|
||||
|
||||
|
||||
class GCEEncodingBackend(EncodingBackend):
|
||||
"""
|
||||
GCE-based encoding backend.
|
||||
|
||||
Wraps the existing EncodingService to implement the EncodingBackend interface.
|
||||
Submits jobs to a remote GCE worker for high-performance encoding.
|
||||
"""
|
||||
|
||||
def __init__(
|
||||
self,
|
||||
dry_run: bool = False,
|
||||
logger: Optional[logging.Logger] = None,
|
||||
):
|
||||
"""
|
||||
Initialize the GCE encoding backend.
|
||||
|
||||
Args:
|
||||
dry_run: Ignored for GCE backend (remote service doesn't support dry run)
|
||||
logger: Optional logger instance
|
||||
"""
|
||||
self.dry_run = dry_run # Stored but not used (GCE is remote)
|
||||
self.logger = logger or logging.getLogger(__name__)
|
||||
self._service = None
|
||||
|
||||
@property
|
||||
def name(self) -> str:
|
||||
return "gce"
|
||||
|
||||
def _get_service(self):
|
||||
"""Lazy-load the GCE encoding service."""
|
||||
if self._service is None:
|
||||
from backend.services.encoding_service import get_encoding_service
|
||||
self._service = get_encoding_service()
|
||||
return self._service
|
||||
|
||||
async def encode(self, input_config: EncodingInput) -> EncodingOutput:
|
||||
"""
|
||||
Encode video using GCE worker.
|
||||
|
||||
Note: GCE encoding requires files to be in GCS, so this method
|
||||
expects input_config.options to contain 'input_gcs_path' and
|
||||
'output_gcs_path'.
|
||||
|
||||
Args:
|
||||
input_config: Encoding input configuration
|
||||
|
||||
Returns:
|
||||
EncodingOutput with results
|
||||
"""
|
||||
import time
|
||||
|
||||
start_time = time.time()
|
||||
service = self._get_service()
|
||||
|
||||
# GCE requires GCS paths
|
||||
input_gcs_path = input_config.options.get("input_gcs_path")
|
||||
output_gcs_path = input_config.options.get("output_gcs_path")
|
||||
job_id = input_config.options.get("job_id", input_config.brand_code or "unknown")
|
||||
|
||||
if not input_gcs_path or not output_gcs_path:
|
||||
return EncodingOutput(
|
||||
success=False,
|
||||
error_message="GCE encoding requires input_gcs_path and output_gcs_path in options",
|
||||
encoding_backend=self.name
|
||||
)
|
||||
|
||||
try:
|
||||
# Build encoding config
|
||||
encoding_config = {
|
||||
"formats": ["mp4_4k_lossless", "mp4_4k_lossy", "mkv_4k", "mp4_720p"],
|
||||
"artist": input_config.artist,
|
||||
"title": input_config.title,
|
||||
}
|
||||
|
||||
# Submit and wait for completion
|
||||
result = await service.encode_videos(
|
||||
job_id=job_id,
|
||||
input_gcs_path=input_gcs_path,
|
||||
output_gcs_path=output_gcs_path,
|
||||
encoding_config=encoding_config,
|
||||
)
|
||||
|
||||
encoding_time = time.time() - start_time
|
||||
|
||||
# Extract output file paths from result
|
||||
# Handle case where GCE worker returns a list or unexpected format
|
||||
if isinstance(result, list):
|
||||
# If result is a list, try to find the output_files in the first dict
|
||||
self.logger.warning(f"GCE returned list instead of dict: {result}")
|
||||
result = result[0] if result and isinstance(result[0], dict) else {}
|
||||
if not isinstance(result, dict):
|
||||
self.logger.error(f"Unexpected GCE result type: {type(result)}")
|
||||
result = {}
|
||||
output_files = result.get("output_files", {})
|
||||
|
||||
return EncodingOutput(
|
||||
success=True,
|
||||
lossless_4k_mp4_path=output_files.get("mp4_4k_lossless"),
|
||||
lossy_4k_mp4_path=output_files.get("mp4_4k_lossy"),
|
||||
lossless_mkv_path=output_files.get("mkv_4k"),
|
||||
lossy_720p_mp4_path=output_files.get("mp4_720p"),
|
||||
output_files=output_files,
|
||||
encoding_time_seconds=encoding_time,
|
||||
encoding_backend=self.name
|
||||
)
|
||||
|
||||
except Exception as e:
|
||||
encoding_time = time.time() - start_time
|
||||
self.logger.error(f"GCE encoding failed: {e}")
|
||||
return EncodingOutput(
|
||||
success=False,
|
||||
error_message=str(e),
|
||||
encoding_time_seconds=encoding_time,
|
||||
encoding_backend=self.name
|
||||
)
|
||||
|
||||
async def is_available(self) -> bool:
|
||||
"""Check if GCE encoding is available and configured."""
|
||||
service = self._get_service()
|
||||
return service.is_enabled
|
||||
|
||||
async def get_status(self) -> Dict[str, Any]:
|
||||
"""Get GCE encoding status."""
|
||||
service = self._get_service()
|
||||
|
||||
status = {
|
||||
"backend": self.name,
|
||||
"available": service.is_enabled,
|
||||
"configured": service.is_configured,
|
||||
}
|
||||
|
||||
if service.is_configured:
|
||||
try:
|
||||
health = await service.health_check()
|
||||
status["health"] = health
|
||||
except Exception as e:
|
||||
status["health_error"] = str(e)
|
||||
|
||||
return status
|
||||
|
||||
|
||||
# Factory function to get an encoding backend
|
||||
def get_encoding_backend(
|
||||
backend_type: str = "auto",
|
||||
**kwargs
|
||||
) -> EncodingBackend:
|
||||
"""
|
||||
Get an encoding backend instance.
|
||||
|
||||
Args:
|
||||
backend_type: Type of backend - "local", "gce", or "auto"
|
||||
**kwargs: Additional arguments for the backend
|
||||
|
||||
Returns:
|
||||
EncodingBackend instance
|
||||
|
||||
Raises:
|
||||
ValueError: If backend_type is unknown
|
||||
"""
|
||||
if backend_type == "local":
|
||||
return LocalEncodingBackend(**kwargs)
|
||||
elif backend_type == "gce":
|
||||
return GCEEncodingBackend(**kwargs)
|
||||
elif backend_type == "auto":
|
||||
# Check if GCE is available, otherwise use local
|
||||
gce_backend = GCEEncodingBackend(**kwargs)
|
||||
if gce_backend._get_service().is_enabled:
|
||||
return gce_backend
|
||||
return LocalEncodingBackend(**kwargs)
|
||||
else:
|
||||
raise ValueError(f"Unknown encoding backend type: {backend_type}")
|
||||
@@ -408,11 +408,11 @@ class FlacfetchClient:
|
||||
) -> Dict[str, Any]:
|
||||
"""
|
||||
Trigger disk cleanup.
|
||||
|
||||
|
||||
Args:
|
||||
strategy: Cleanup strategy (oldest, largest, lowest_ratio)
|
||||
target_free_gb: Target free space to achieve
|
||||
|
||||
|
||||
Returns:
|
||||
Cleanup response with removed count and freed space
|
||||
"""
|
||||
@@ -429,112 +429,12 @@ class FlacfetchClient:
|
||||
)
|
||||
resp.raise_for_status()
|
||||
return resp.json()
|
||||
|
||||
|
||||
except httpx.RequestError as e:
|
||||
raise FlacfetchServiceError(f"Cleanup failed: {e}")
|
||||
except httpx.HTTPStatusError as e:
|
||||
raise FlacfetchServiceError(f"Cleanup failed: {e.response.status_code}")
|
||||
|
||||
# =========================================================================
|
||||
# Cache Management
|
||||
# =========================================================================
|
||||
|
||||
async def clear_search_cache(self, artist: str, title: str) -> bool:
|
||||
"""
|
||||
Clear a specific cached search result by artist and title.
|
||||
|
||||
Args:
|
||||
artist: Artist name
|
||||
title: Track title
|
||||
|
||||
Returns:
|
||||
True if a cache entry was deleted, False if no entry existed
|
||||
|
||||
Raises:
|
||||
FlacfetchServiceError: On request failure
|
||||
"""
|
||||
try:
|
||||
async with httpx.AsyncClient() as client:
|
||||
resp = await client.request(
|
||||
"DELETE",
|
||||
f"{self.base_url}/cache/search",
|
||||
headers=self._headers(),
|
||||
json={"artist": artist, "title": title},
|
||||
timeout=30,
|
||||
)
|
||||
resp.raise_for_status()
|
||||
data = resp.json()
|
||||
deleted = data.get("deleted", False)
|
||||
logger.info(
|
||||
f"Cleared flacfetch cache for '{artist}' - '{title}': "
|
||||
f"{'deleted' if deleted else 'no entry found'}"
|
||||
)
|
||||
return deleted
|
||||
|
||||
except httpx.RequestError as e:
|
||||
raise FlacfetchServiceError(f"Clear search cache request failed: {e}")
|
||||
except httpx.HTTPStatusError as e:
|
||||
raise FlacfetchServiceError(
|
||||
f"Clear search cache failed: {e.response.status_code} - {e.response.text}"
|
||||
)
|
||||
|
||||
async def clear_all_cache(self) -> int:
|
||||
"""
|
||||
Clear all cached search results.
|
||||
|
||||
Returns:
|
||||
Number of cache entries deleted
|
||||
|
||||
Raises:
|
||||
FlacfetchServiceError: On request failure
|
||||
"""
|
||||
try:
|
||||
async with httpx.AsyncClient() as client:
|
||||
resp = await client.delete(
|
||||
f"{self.base_url}/cache",
|
||||
headers=self._headers(),
|
||||
timeout=60,
|
||||
)
|
||||
resp.raise_for_status()
|
||||
data = resp.json()
|
||||
deleted_count = data.get("deleted_count", 0)
|
||||
logger.info(f"Cleared all flacfetch cache: {deleted_count} entries deleted")
|
||||
return deleted_count
|
||||
|
||||
except httpx.RequestError as e:
|
||||
raise FlacfetchServiceError(f"Clear all cache request failed: {e}")
|
||||
except httpx.HTTPStatusError as e:
|
||||
raise FlacfetchServiceError(
|
||||
f"Clear all cache failed: {e.response.status_code} - {e.response.text}"
|
||||
)
|
||||
|
||||
async def get_cache_stats(self) -> Dict[str, Any]:
|
||||
"""
|
||||
Get statistics about the cache.
|
||||
|
||||
Returns:
|
||||
Dict with count, total_size_bytes, oldest_entry, newest_entry, configured
|
||||
|
||||
Raises:
|
||||
FlacfetchServiceError: On request failure
|
||||
"""
|
||||
try:
|
||||
async with httpx.AsyncClient() as client:
|
||||
resp = await client.get(
|
||||
f"{self.base_url}/cache/stats",
|
||||
headers=self._headers(),
|
||||
timeout=30,
|
||||
)
|
||||
resp.raise_for_status()
|
||||
return resp.json()
|
||||
|
||||
except httpx.RequestError as e:
|
||||
raise FlacfetchServiceError(f"Get cache stats request failed: {e}")
|
||||
except httpx.HTTPStatusError as e:
|
||||
raise FlacfetchServiceError(
|
||||
f"Get cache stats failed: {e.response.status_code} - {e.response.text}"
|
||||
)
|
||||
|
||||
|
||||
# Singleton client instance
|
||||
_client: Optional[FlacfetchClient] = None
|
||||
|
||||
@@ -1,590 +0,0 @@
|
||||
"""
|
||||
Local Encoding Service.
|
||||
|
||||
Provides local FFmpeg-based video encoding functionality, extracted from KaraokeFinalise
|
||||
for use by both the cloud backend (video_worker as a fallback) and local CLI.
|
||||
|
||||
This service handles:
|
||||
- Concatenating title, karaoke, and end videos
|
||||
- Encoding to multiple output formats (4K lossless, 4K lossy, 720p, MKV)
|
||||
- Hardware acceleration detection and fallback to software encoding
|
||||
- Remuxing video with instrumental audio
|
||||
"""
|
||||
|
||||
import logging
|
||||
import os
|
||||
import shlex
|
||||
import subprocess
|
||||
import sys
|
||||
from dataclasses import dataclass
|
||||
from typing import Optional, List, Dict, Any, Tuple
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
|
||||
@dataclass
|
||||
class EncodingConfig:
|
||||
"""Configuration for video encoding."""
|
||||
title_video: str # Path to title video
|
||||
karaoke_video: str # Path to karaoke video (with vocals)
|
||||
instrumental_audio: str # Path to instrumental audio
|
||||
end_video: Optional[str] = None # Optional path to end credits video
|
||||
|
||||
# Output paths
|
||||
output_karaoke_mp4: Optional[str] = None
|
||||
output_with_vocals_mp4: Optional[str] = None
|
||||
output_lossless_4k_mp4: Optional[str] = None
|
||||
output_lossy_4k_mp4: Optional[str] = None
|
||||
output_lossless_mkv: Optional[str] = None
|
||||
output_720p_mp4: Optional[str] = None
|
||||
|
||||
|
||||
@dataclass
|
||||
class EncodingResult:
|
||||
"""Result of video encoding operation."""
|
||||
success: bool
|
||||
output_files: Dict[str, str]
|
||||
error: Optional[str] = None
|
||||
|
||||
|
||||
class LocalEncodingService:
|
||||
"""
|
||||
Service for local FFmpeg-based video encoding.
|
||||
|
||||
Supports hardware acceleration (NVENC) with automatic fallback
|
||||
to software encoding (libx264) when unavailable.
|
||||
"""
|
||||
|
||||
# MP4 flags for better compatibility
|
||||
MP4_FLAGS = "-movflags +faststart"
|
||||
|
||||
def __init__(
|
||||
self,
|
||||
dry_run: bool = False,
|
||||
log_level: int = logging.INFO,
|
||||
logger: Optional[logging.Logger] = None,
|
||||
):
|
||||
"""
|
||||
Initialize the local encoding service.
|
||||
|
||||
Args:
|
||||
dry_run: If True, log commands without executing them
|
||||
log_level: Logging level (affects FFmpeg verbosity)
|
||||
logger: Optional logger instance
|
||||
"""
|
||||
self.dry_run = dry_run
|
||||
self.log_level = log_level
|
||||
self.logger = logger or logging.getLogger(__name__)
|
||||
|
||||
# Hardware acceleration settings (detected on first use)
|
||||
self._hwaccel_available: Optional[bool] = None
|
||||
self._video_encoder: Optional[str] = None
|
||||
self._scale_filter: Optional[str] = None
|
||||
self._hwaccel_decode_flags: Optional[str] = None
|
||||
self._aac_codec: Optional[str] = None
|
||||
|
||||
# Build FFmpeg base command
|
||||
self._ffmpeg_base_command = self._build_ffmpeg_base_command()
|
||||
|
||||
def _build_ffmpeg_base_command(self) -> str:
|
||||
"""Build the FFmpeg base command with appropriate flags."""
|
||||
# Use bundled FFmpeg for frozen builds
|
||||
ffmpeg_path = os.path.join(sys._MEIPASS, "ffmpeg.exe") if getattr(sys, "frozen", False) else "ffmpeg"
|
||||
|
||||
base_cmd = f"{ffmpeg_path} -hide_banner -nostats -y"
|
||||
|
||||
if self.log_level == logging.DEBUG:
|
||||
base_cmd += " -loglevel verbose"
|
||||
else:
|
||||
base_cmd += " -loglevel fatal"
|
||||
|
||||
return base_cmd
|
||||
|
||||
def _detect_hardware_acceleration(self) -> Tuple[bool, str, str, str, str]:
|
||||
"""
|
||||
Detect available hardware acceleration.
|
||||
|
||||
Returns:
|
||||
Tuple of (hwaccel_available, video_encoder, scale_filter,
|
||||
hwaccel_decode_flags, aac_codec)
|
||||
"""
|
||||
self.logger.info("Detecting hardware acceleration capabilities...")
|
||||
|
||||
# Try NVENC (NVIDIA)
|
||||
try:
|
||||
test_cmd = f"{self._ffmpeg_base_command} -hide_banner -loglevel error " \
|
||||
f"-f lavfi -i testsrc=duration=1:size=320x240:rate=1 " \
|
||||
f"-c:v h264_nvenc -f null -"
|
||||
subprocess.run(
|
||||
test_cmd, shell=True, check=True,
|
||||
capture_output=True, timeout=30
|
||||
)
|
||||
self.logger.info("NVIDIA NVENC hardware acceleration available")
|
||||
return (
|
||||
True,
|
||||
"h264_nvenc",
|
||||
"scale_cuda",
|
||||
"-hwaccel cuda -hwaccel_output_format cuda",
|
||||
"aac"
|
||||
)
|
||||
except (subprocess.CalledProcessError, subprocess.TimeoutExpired, FileNotFoundError):
|
||||
pass
|
||||
|
||||
# No hardware acceleration available
|
||||
self.logger.info("No hardware acceleration available, using software encoding")
|
||||
return (False, "libx264", "scale", "", "aac")
|
||||
|
||||
@property
|
||||
def hwaccel_available(self) -> bool:
|
||||
"""Check if hardware acceleration is available."""
|
||||
if self._hwaccel_available is None:
|
||||
self._detect_and_set_hwaccel()
|
||||
return self._hwaccel_available
|
||||
|
||||
@property
|
||||
def video_encoder(self) -> str:
|
||||
"""Get the video encoder to use."""
|
||||
if self._video_encoder is None:
|
||||
self._detect_and_set_hwaccel()
|
||||
return self._video_encoder
|
||||
|
||||
@property
|
||||
def scale_filter(self) -> str:
|
||||
"""Get the scale filter to use."""
|
||||
if self._scale_filter is None:
|
||||
self._detect_and_set_hwaccel()
|
||||
return self._scale_filter
|
||||
|
||||
@property
|
||||
def hwaccel_decode_flags(self) -> str:
|
||||
"""Get hardware acceleration decode flags."""
|
||||
if self._hwaccel_decode_flags is None:
|
||||
self._detect_and_set_hwaccel()
|
||||
return self._hwaccel_decode_flags
|
||||
|
||||
@property
|
||||
def aac_codec(self) -> str:
|
||||
"""Get the AAC codec to use."""
|
||||
if self._aac_codec is None:
|
||||
self._detect_and_set_hwaccel()
|
||||
return self._aac_codec
|
||||
|
||||
def _detect_and_set_hwaccel(self) -> None:
|
||||
"""Detect and set hardware acceleration settings."""
|
||||
(
|
||||
self._hwaccel_available,
|
||||
self._video_encoder,
|
||||
self._scale_filter,
|
||||
self._hwaccel_decode_flags,
|
||||
self._aac_codec
|
||||
) = self._detect_hardware_acceleration()
|
||||
|
||||
def _get_nvenc_quality_settings(self, preset: str = "medium") -> str:
|
||||
"""Get NVENC quality settings for different presets."""
|
||||
if not self.hwaccel_available:
|
||||
return ""
|
||||
|
||||
settings = {
|
||||
"lossless": "-preset p7 -tune hq -rc vbr -cq 0 -qmin 0 -qmax 0",
|
||||
"high": "-preset p7 -tune hq -rc vbr -cq 19 -b:v 0",
|
||||
"medium": "-preset p4 -tune hq -rc vbr -cq 23 -b:v 0",
|
||||
"fast": "-preset p2 -tune ll -rc vbr -cq 28 -b:v 0",
|
||||
}
|
||||
return settings.get(preset, settings["medium"])
|
||||
|
||||
def _execute_command(
|
||||
self,
|
||||
command: str,
|
||||
description: str,
|
||||
timeout: int = 3600, # 1 hour default
|
||||
) -> bool:
|
||||
"""
|
||||
Execute an FFmpeg command.
|
||||
|
||||
Args:
|
||||
command: The FFmpeg command to execute
|
||||
description: Human-readable description of the operation
|
||||
timeout: Command timeout in seconds
|
||||
|
||||
Returns:
|
||||
True if successful, False otherwise
|
||||
"""
|
||||
self.logger.info(f"Executing: {description}")
|
||||
self.logger.debug(f"Command: {command}")
|
||||
|
||||
if self.dry_run:
|
||||
self.logger.info(f"DRY RUN: Would execute: {command}")
|
||||
return True
|
||||
|
||||
try:
|
||||
result = subprocess.run(
|
||||
command,
|
||||
shell=True,
|
||||
check=True,
|
||||
capture_output=True,
|
||||
text=True,
|
||||
timeout=timeout
|
||||
)
|
||||
self.logger.info(f"Completed: {description}")
|
||||
return True
|
||||
except subprocess.CalledProcessError as e:
|
||||
self.logger.error(f"Failed: {description}")
|
||||
self.logger.error(f"Error: {e.stderr}")
|
||||
return False
|
||||
except subprocess.TimeoutExpired:
|
||||
self.logger.error(f"Timeout: {description}")
|
||||
return False
|
||||
|
||||
def _execute_command_with_fallback(
|
||||
self,
|
||||
gpu_command: str,
|
||||
cpu_command: str,
|
||||
description: str,
|
||||
) -> bool:
|
||||
"""
|
||||
Execute a command with GPU, falling back to CPU if it fails.
|
||||
|
||||
Args:
|
||||
gpu_command: Hardware-accelerated command
|
||||
cpu_command: Software fallback command
|
||||
description: Human-readable description
|
||||
|
||||
Returns:
|
||||
True if successful, False otherwise
|
||||
"""
|
||||
if self.hwaccel_available:
|
||||
self.logger.info(f"Trying hardware-accelerated encoding for: {description}")
|
||||
if self._execute_command(gpu_command, description):
|
||||
return True
|
||||
self.logger.warning(f"Hardware encoding failed, falling back to software")
|
||||
|
||||
return self._execute_command(cpu_command, description)
|
||||
|
||||
def remux_with_instrumental(
|
||||
self,
|
||||
input_video: str,
|
||||
instrumental_audio: str,
|
||||
output_file: str,
|
||||
) -> bool:
|
||||
"""
|
||||
Remux video with instrumental audio track.
|
||||
|
||||
Args:
|
||||
input_video: Path to input video file
|
||||
instrumental_audio: Path to instrumental audio file
|
||||
output_file: Path for output file
|
||||
|
||||
Returns:
|
||||
True if successful, False otherwise
|
||||
"""
|
||||
command = (
|
||||
f'{self._ffmpeg_base_command} -i "{input_video}" '
|
||||
f'-i "{instrumental_audio}" -map 0:v -map 1:a -c copy '
|
||||
f'{self.MP4_FLAGS} "{output_file}"'
|
||||
)
|
||||
return self._execute_command(command, "Remuxing with instrumental audio")
|
||||
|
||||
def convert_mov_to_mp4(
|
||||
self,
|
||||
input_file: str,
|
||||
output_file: str,
|
||||
) -> bool:
|
||||
"""
|
||||
Convert MOV to MP4 format.
|
||||
|
||||
Args:
|
||||
input_file: Path to input MOV file
|
||||
output_file: Path for output MP4 file
|
||||
|
||||
Returns:
|
||||
True if successful, False otherwise
|
||||
"""
|
||||
gpu_command = (
|
||||
f'{self._ffmpeg_base_command} {self.hwaccel_decode_flags} -i "{input_file}" '
|
||||
f'-c:v {self.video_encoder} -c:a copy {self.MP4_FLAGS} "{output_file}"'
|
||||
)
|
||||
cpu_command = (
|
||||
f'{self._ffmpeg_base_command} -i "{input_file}" '
|
||||
f'-c:v libx264 -c:a copy {self.MP4_FLAGS} "{output_file}"'
|
||||
)
|
||||
return self._execute_command_with_fallback(
|
||||
gpu_command, cpu_command, "Converting MOV to MP4"
|
||||
)
|
||||
|
||||
def encode_lossless_mp4(
|
||||
self,
|
||||
title_video: str,
|
||||
karaoke_video: str,
|
||||
output_file: str,
|
||||
end_video: Optional[str] = None,
|
||||
) -> bool:
|
||||
"""
|
||||
Create lossless 4K MP4 by concatenating title, karaoke, and optionally end videos.
|
||||
|
||||
Args:
|
||||
title_video: Path to title video
|
||||
karaoke_video: Path to karaoke video
|
||||
output_file: Path for output file
|
||||
end_video: Optional path to end credits video
|
||||
|
||||
Returns:
|
||||
True if successful, False otherwise
|
||||
"""
|
||||
# Quote file paths
|
||||
title_quoted = shlex.quote(os.path.abspath(title_video))
|
||||
karaoke_quoted = shlex.quote(os.path.abspath(karaoke_video))
|
||||
|
||||
# Build filter and inputs for concatenation
|
||||
if end_video and os.path.isfile(end_video):
|
||||
end_quoted = shlex.quote(os.path.abspath(end_video))
|
||||
extra_input = f"-i {end_quoted}"
|
||||
concat_filter = (
|
||||
'-filter_complex "[0:v:0][0:a:0][1:v:0][1:a:0][2:v:0][2:a:0]'
|
||||
'concat=n=3:v=1:a=1[outv][outa]"'
|
||||
)
|
||||
else:
|
||||
extra_input = ""
|
||||
concat_filter = (
|
||||
'-filter_complex "[0:v:0][0:a:0][1:v:0][1:a:0]'
|
||||
'concat=n=2:v=1:a=1[outv][outa]"'
|
||||
)
|
||||
|
||||
gpu_command = (
|
||||
f"{self._ffmpeg_base_command} {self.hwaccel_decode_flags} -i {title_quoted} "
|
||||
f"{self.hwaccel_decode_flags} -i {karaoke_quoted} {extra_input} "
|
||||
f'{concat_filter} -map "[outv]" -map "[outa]" -c:v {self.video_encoder} '
|
||||
f'{self._get_nvenc_quality_settings("lossless")} -c:a pcm_s16le '
|
||||
f'{self.MP4_FLAGS} "{output_file}"'
|
||||
)
|
||||
cpu_command = (
|
||||
f"{self._ffmpeg_base_command} -i {title_quoted} -i {karaoke_quoted} {extra_input} "
|
||||
f'{concat_filter} -map "[outv]" -map "[outa]" -c:v libx264 -c:a pcm_s16le '
|
||||
f'{self.MP4_FLAGS} "{output_file}"'
|
||||
)
|
||||
|
||||
return self._execute_command_with_fallback(
|
||||
gpu_command, cpu_command, "Encoding lossless 4K MP4"
|
||||
)
|
||||
|
||||
def encode_lossy_mp4(
|
||||
self,
|
||||
input_file: str,
|
||||
output_file: str,
|
||||
) -> bool:
|
||||
"""
|
||||
Create lossy 4K MP4 with AAC audio.
|
||||
|
||||
Args:
|
||||
input_file: Path to input file (typically lossless MP4)
|
||||
output_file: Path for output file
|
||||
|
||||
Returns:
|
||||
True if successful, False otherwise
|
||||
"""
|
||||
command = (
|
||||
f'{self._ffmpeg_base_command} -i "{input_file}" '
|
||||
f'-c:v copy -c:a {self.aac_codec} -ar 48000 -b:a 320k '
|
||||
f'{self.MP4_FLAGS} "{output_file}"'
|
||||
)
|
||||
return self._execute_command(command, "Encoding lossy 4K MP4 with AAC")
|
||||
|
||||
def encode_lossless_mkv(
|
||||
self,
|
||||
input_file: str,
|
||||
output_file: str,
|
||||
) -> bool:
|
||||
"""
|
||||
Create MKV with FLAC audio (for YouTube upload).
|
||||
|
||||
Args:
|
||||
input_file: Path to input file
|
||||
output_file: Path for output file
|
||||
|
||||
Returns:
|
||||
True if successful, False otherwise
|
||||
"""
|
||||
command = (
|
||||
f'{self._ffmpeg_base_command} -i "{input_file}" '
|
||||
f'-c:v copy -c:a flac "{output_file}"'
|
||||
)
|
||||
return self._execute_command(command, "Creating MKV with FLAC for YouTube")
|
||||
|
||||
def encode_720p(
|
||||
self,
|
||||
input_file: str,
|
||||
output_file: str,
|
||||
) -> bool:
|
||||
"""
|
||||
Create 720p MP4 with AAC audio.
|
||||
|
||||
Args:
|
||||
input_file: Path to input file
|
||||
output_file: Path for output file
|
||||
|
||||
Returns:
|
||||
True if successful, False otherwise
|
||||
"""
|
||||
gpu_command = (
|
||||
f'{self._ffmpeg_base_command} {self.hwaccel_decode_flags} -i "{input_file}" '
|
||||
f'-c:v {self.video_encoder} -vf "{self.scale_filter}=1280:720" '
|
||||
f'{self._get_nvenc_quality_settings("medium")} -b:v 2000k '
|
||||
f'-c:a {self.aac_codec} -ar 48000 -b:a 128k '
|
||||
f'{self.MP4_FLAGS} "{output_file}"'
|
||||
)
|
||||
cpu_command = (
|
||||
f'{self._ffmpeg_base_command} -i "{input_file}" '
|
||||
f'-c:v libx264 -vf "scale=1280:720" -b:v 2000k -preset medium -tune animation '
|
||||
f'-c:a {self.aac_codec} -ar 48000 -b:a 128k '
|
||||
f'{self.MP4_FLAGS} "{output_file}"'
|
||||
)
|
||||
return self._execute_command_with_fallback(
|
||||
gpu_command, cpu_command, "Encoding 720p MP4"
|
||||
)
|
||||
|
||||
def encode_all_formats(
|
||||
self,
|
||||
config: EncodingConfig,
|
||||
) -> EncodingResult:
|
||||
"""
|
||||
Encode video to all output formats.
|
||||
|
||||
This performs the full encoding pipeline:
|
||||
1. Remux with instrumental audio
|
||||
2. Convert to MP4 if needed
|
||||
3. Encode lossless 4K MP4 (concatenated)
|
||||
4. Encode lossy 4K MP4
|
||||
5. Encode lossless MKV (for YouTube)
|
||||
6. Encode 720p MP4
|
||||
|
||||
Args:
|
||||
config: Encoding configuration with input/output paths
|
||||
|
||||
Returns:
|
||||
EncodingResult with success status and output file paths
|
||||
"""
|
||||
output_files = {}
|
||||
|
||||
try:
|
||||
# Step 1: Remux with instrumental audio
|
||||
if config.output_karaoke_mp4:
|
||||
self.logger.info("[Step 1/6] Remuxing video with instrumental audio...")
|
||||
if not self.remux_with_instrumental(
|
||||
config.karaoke_video,
|
||||
config.instrumental_audio,
|
||||
config.output_karaoke_mp4
|
||||
):
|
||||
return EncodingResult(
|
||||
success=False,
|
||||
output_files=output_files,
|
||||
error="Failed to remux with instrumental audio"
|
||||
)
|
||||
output_files["karaoke_mp4"] = config.output_karaoke_mp4
|
||||
|
||||
# Step 2: Convert to MP4 if needed
|
||||
if config.output_with_vocals_mp4:
|
||||
if not config.karaoke_video.endswith(".mp4"):
|
||||
self.logger.info("[Step 2/6] Converting karaoke video to MP4...")
|
||||
if not self.convert_mov_to_mp4(
|
||||
config.karaoke_video,
|
||||
config.output_with_vocals_mp4
|
||||
):
|
||||
return EncodingResult(
|
||||
success=False,
|
||||
output_files=output_files,
|
||||
error="Failed to convert to MP4"
|
||||
)
|
||||
output_files["with_vocals_mp4"] = config.output_with_vocals_mp4
|
||||
else:
|
||||
self.logger.info("[Step 2/6] Skipped - video already MP4")
|
||||
|
||||
# Step 3: Encode lossless 4K MP4
|
||||
if config.output_lossless_4k_mp4:
|
||||
self.logger.info("[Step 3/6] Encoding lossless 4K MP4...")
|
||||
karaoke_for_concat = config.output_karaoke_mp4 or config.karaoke_video
|
||||
if not self.encode_lossless_mp4(
|
||||
config.title_video,
|
||||
karaoke_for_concat,
|
||||
config.output_lossless_4k_mp4,
|
||||
config.end_video
|
||||
):
|
||||
return EncodingResult(
|
||||
success=False,
|
||||
output_files=output_files,
|
||||
error="Failed to encode lossless 4K MP4"
|
||||
)
|
||||
output_files["lossless_4k_mp4"] = config.output_lossless_4k_mp4
|
||||
|
||||
# Step 4: Encode lossy 4K MP4
|
||||
if config.output_lossy_4k_mp4 and config.output_lossless_4k_mp4:
|
||||
self.logger.info("[Step 4/6] Encoding lossy 4K MP4...")
|
||||
if not self.encode_lossy_mp4(
|
||||
config.output_lossless_4k_mp4,
|
||||
config.output_lossy_4k_mp4
|
||||
):
|
||||
return EncodingResult(
|
||||
success=False,
|
||||
output_files=output_files,
|
||||
error="Failed to encode lossy 4K MP4"
|
||||
)
|
||||
output_files["lossy_4k_mp4"] = config.output_lossy_4k_mp4
|
||||
|
||||
# Step 5: Create MKV with FLAC audio
|
||||
if config.output_lossless_mkv and config.output_lossless_4k_mp4:
|
||||
self.logger.info("[Step 5/6] Creating MKV with FLAC audio...")
|
||||
if not self.encode_lossless_mkv(
|
||||
config.output_lossless_4k_mp4,
|
||||
config.output_lossless_mkv
|
||||
):
|
||||
return EncodingResult(
|
||||
success=False,
|
||||
output_files=output_files,
|
||||
error="Failed to create MKV"
|
||||
)
|
||||
output_files["lossless_mkv"] = config.output_lossless_mkv
|
||||
|
||||
# Step 6: Encode 720p version
|
||||
if config.output_720p_mp4 and config.output_lossless_4k_mp4:
|
||||
self.logger.info("[Step 6/6] Encoding 720p MP4...")
|
||||
if not self.encode_720p(
|
||||
config.output_lossless_4k_mp4,
|
||||
config.output_720p_mp4
|
||||
):
|
||||
return EncodingResult(
|
||||
success=False,
|
||||
output_files=output_files,
|
||||
error="Failed to encode 720p"
|
||||
)
|
||||
output_files["720p_mp4"] = config.output_720p_mp4
|
||||
|
||||
self.logger.info("All encoding steps completed successfully")
|
||||
return EncodingResult(success=True, output_files=output_files)
|
||||
|
||||
except Exception as e:
|
||||
self.logger.error(f"Encoding failed with exception: {e}")
|
||||
return EncodingResult(
|
||||
success=False,
|
||||
output_files=output_files,
|
||||
error=str(e)
|
||||
)
|
||||
|
||||
|
||||
# Singleton instance and factory function (following existing service pattern)
|
||||
_local_encoding_service: Optional[LocalEncodingService] = None
|
||||
|
||||
|
||||
def get_local_encoding_service(**kwargs) -> LocalEncodingService:
|
||||
"""
|
||||
Get a local encoding service instance.
|
||||
|
||||
Args:
|
||||
**kwargs: Arguments passed to LocalEncodingService
|
||||
|
||||
Returns:
|
||||
LocalEncodingService instance
|
||||
"""
|
||||
global _local_encoding_service
|
||||
|
||||
if _local_encoding_service is None:
|
||||
_local_encoding_service = LocalEncodingService(**kwargs)
|
||||
|
||||
return _local_encoding_service
|
||||
@@ -1,287 +0,0 @@
|
||||
"""
|
||||
Packaging Service.
|
||||
|
||||
Provides CDG and TXT package generation functionality, extracted from KaraokeFinalise
|
||||
for use by both the cloud backend (video_worker) and local CLI.
|
||||
|
||||
This service handles:
|
||||
- CDG (CD+G) file generation from LRC files
|
||||
- TXT lyric file generation from LRC files
|
||||
- ZIP packaging of CDG/MP3 and TXT/MP3 pairs
|
||||
"""
|
||||
|
||||
import logging
|
||||
import os
|
||||
import zipfile
|
||||
from typing import Optional, Dict, Any, Tuple
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
|
||||
class PackagingService:
|
||||
"""
|
||||
Service for creating CDG and TXT karaoke packages.
|
||||
|
||||
CDG (CD+Graphics) is a format used by karaoke machines.
|
||||
TXT packages are used by software karaoke players.
|
||||
Both formats are packaged as ZIP files with an MP3 audio track.
|
||||
"""
|
||||
|
||||
def __init__(
|
||||
self,
|
||||
cdg_styles: Optional[Dict[str, Any]] = None,
|
||||
dry_run: bool = False,
|
||||
non_interactive: bool = False,
|
||||
logger: Optional[logging.Logger] = None,
|
||||
):
|
||||
"""
|
||||
Initialize the packaging service.
|
||||
|
||||
Args:
|
||||
cdg_styles: CDG style configuration for CDG generation
|
||||
dry_run: If True, log actions without performing them
|
||||
non_interactive: If True, skip interactive prompts
|
||||
logger: Optional logger instance
|
||||
"""
|
||||
self.cdg_styles = cdg_styles
|
||||
self.dry_run = dry_run
|
||||
self.non_interactive = non_interactive
|
||||
self.logger = logger or logging.getLogger(__name__)
|
||||
|
||||
def create_cdg_package(
|
||||
self,
|
||||
lrc_file: str,
|
||||
audio_file: str,
|
||||
output_zip_path: str,
|
||||
artist: str,
|
||||
title: str,
|
||||
output_mp3_path: Optional[str] = None,
|
||||
output_cdg_path: Optional[str] = None,
|
||||
) -> Tuple[str, Optional[str], Optional[str]]:
|
||||
"""
|
||||
Create a CDG package (ZIP containing CDG and MP3 files).
|
||||
|
||||
Args:
|
||||
lrc_file: Path to the LRC lyrics file
|
||||
audio_file: Path to the instrumental audio file
|
||||
output_zip_path: Path for the output ZIP file
|
||||
artist: Artist name for metadata
|
||||
title: Song title for metadata
|
||||
output_mp3_path: Optional path for the extracted MP3 file
|
||||
output_cdg_path: Optional path for the extracted CDG file
|
||||
|
||||
Returns:
|
||||
Tuple of (zip_path, mp3_path, cdg_path)
|
||||
|
||||
Raises:
|
||||
ValueError: If CDG styles are not configured
|
||||
FileNotFoundError: If input files are missing
|
||||
Exception: If CDG generation fails
|
||||
"""
|
||||
self.logger.info(f"Creating CDG package for {artist} - {title}")
|
||||
|
||||
# Validate inputs
|
||||
if not os.path.isfile(lrc_file):
|
||||
raise FileNotFoundError(f"LRC file not found: {lrc_file}")
|
||||
if not os.path.isfile(audio_file):
|
||||
raise FileNotFoundError(f"Audio file not found: {audio_file}")
|
||||
|
||||
# Check if ZIP already exists
|
||||
if os.path.isfile(output_zip_path):
|
||||
if self.non_interactive:
|
||||
self.logger.info(
|
||||
f"CDG ZIP exists, will be overwritten: {output_zip_path}"
|
||||
)
|
||||
else:
|
||||
self.logger.info(f"CDG ZIP already exists: {output_zip_path}")
|
||||
|
||||
# Check if individual files exist (allows skipping generation)
|
||||
if output_mp3_path and output_cdg_path:
|
||||
if os.path.isfile(output_mp3_path) and os.path.isfile(output_cdg_path):
|
||||
self.logger.info("Found existing MP3 and CDG files, creating ZIP directly")
|
||||
if not self.dry_run:
|
||||
self._create_zip_from_files(
|
||||
output_zip_path,
|
||||
[(output_mp3_path, os.path.basename(output_mp3_path)),
|
||||
(output_cdg_path, os.path.basename(output_cdg_path))]
|
||||
)
|
||||
return output_zip_path, output_mp3_path, output_cdg_path
|
||||
|
||||
if self.dry_run:
|
||||
self.logger.info(
|
||||
f"DRY RUN: Would generate CDG package: {output_zip_path}"
|
||||
)
|
||||
return output_zip_path, output_mp3_path, output_cdg_path
|
||||
|
||||
# Generate CDG files
|
||||
if self.cdg_styles is None:
|
||||
raise ValueError(
|
||||
"CDG styles configuration is required for CDG generation"
|
||||
)
|
||||
|
||||
self.logger.info("Generating CDG and MP3 files")
|
||||
from lyrics_transcriber.output.cdg import CDGGenerator
|
||||
|
||||
output_dir = os.path.dirname(output_zip_path) or os.getcwd()
|
||||
generator = CDGGenerator(output_dir=output_dir, logger=self.logger)
|
||||
|
||||
cdg_file, mp3_file, zip_file = generator.generate_cdg_from_lrc(
|
||||
lrc_file=lrc_file,
|
||||
audio_file=audio_file,
|
||||
title=title,
|
||||
artist=artist,
|
||||
cdg_styles=self.cdg_styles,
|
||||
)
|
||||
|
||||
# Rename ZIP to expected output path if different
|
||||
if os.path.isfile(zip_file) and zip_file != output_zip_path:
|
||||
os.rename(zip_file, output_zip_path)
|
||||
self.logger.info(f"Renamed CDG ZIP: {zip_file} -> {output_zip_path}")
|
||||
|
||||
if not os.path.isfile(output_zip_path):
|
||||
raise Exception(f"Failed to create CDG ZIP file: {output_zip_path}")
|
||||
|
||||
# Extract the ZIP to get individual files if paths provided
|
||||
extracted_mp3 = None
|
||||
extracted_cdg = None
|
||||
if output_mp3_path or output_cdg_path:
|
||||
self.logger.info(f"Extracting CDG ZIP file: {output_zip_path}")
|
||||
with zipfile.ZipFile(output_zip_path, "r") as zip_ref:
|
||||
zip_ref.extractall(output_dir)
|
||||
|
||||
# Find extracted files
|
||||
if output_mp3_path and os.path.isfile(output_mp3_path):
|
||||
extracted_mp3 = output_mp3_path
|
||||
self.logger.info(f"Extracted MP3: {extracted_mp3}")
|
||||
if output_cdg_path and os.path.isfile(output_cdg_path):
|
||||
extracted_cdg = output_cdg_path
|
||||
self.logger.info(f"Extracted CDG: {extracted_cdg}")
|
||||
|
||||
self.logger.info(f"CDG package created: {output_zip_path}")
|
||||
return output_zip_path, extracted_mp3, extracted_cdg
|
||||
|
||||
def create_txt_package(
|
||||
self,
|
||||
lrc_file: str,
|
||||
mp3_file: str,
|
||||
output_zip_path: str,
|
||||
output_txt_path: Optional[str] = None,
|
||||
) -> Tuple[str, Optional[str]]:
|
||||
"""
|
||||
Create a TXT package (ZIP containing TXT lyrics and MP3 files).
|
||||
|
||||
Args:
|
||||
lrc_file: Path to the LRC lyrics file
|
||||
mp3_file: Path to the MP3 audio file
|
||||
output_zip_path: Path for the output ZIP file
|
||||
output_txt_path: Optional path for the generated TXT file
|
||||
|
||||
Returns:
|
||||
Tuple of (zip_path, txt_path)
|
||||
|
||||
Raises:
|
||||
FileNotFoundError: If input files are missing
|
||||
Exception: If TXT generation fails
|
||||
"""
|
||||
self.logger.info(f"Creating TXT package from {lrc_file}")
|
||||
|
||||
# Validate inputs
|
||||
if not os.path.isfile(lrc_file):
|
||||
raise FileNotFoundError(f"LRC file not found: {lrc_file}")
|
||||
if not os.path.isfile(mp3_file):
|
||||
raise FileNotFoundError(f"MP3 file not found: {mp3_file}")
|
||||
|
||||
# Check if ZIP already exists
|
||||
if os.path.isfile(output_zip_path):
|
||||
if self.non_interactive:
|
||||
self.logger.info(
|
||||
f"TXT ZIP exists, will be overwritten: {output_zip_path}"
|
||||
)
|
||||
else:
|
||||
self.logger.info(f"TXT ZIP already exists: {output_zip_path}")
|
||||
|
||||
if self.dry_run:
|
||||
self.logger.info(
|
||||
f"DRY RUN: Would create TXT package: {output_zip_path}"
|
||||
)
|
||||
return output_zip_path, output_txt_path
|
||||
|
||||
# Generate TXT from LRC
|
||||
self.logger.info(f"Converting LRC to TXT format: {lrc_file}")
|
||||
from lyrics_converter import LyricsConverter
|
||||
|
||||
txt_converter = LyricsConverter(output_format="txt", filepath=lrc_file)
|
||||
converted_txt = txt_converter.convert_file()
|
||||
|
||||
# Write TXT file
|
||||
if output_txt_path is None:
|
||||
# Default to same name as ZIP but with .txt extension
|
||||
output_txt_path = output_zip_path.replace(".zip", ".txt")
|
||||
|
||||
with open(output_txt_path, "w") as txt_file:
|
||||
txt_file.write(converted_txt)
|
||||
self.logger.info(f"TXT file written: {output_txt_path}")
|
||||
|
||||
# Create ZIP containing MP3 and TXT
|
||||
self.logger.info(f"Creating TXT ZIP: {output_zip_path}")
|
||||
self._create_zip_from_files(
|
||||
output_zip_path,
|
||||
[(mp3_file, os.path.basename(mp3_file)),
|
||||
(output_txt_path, os.path.basename(output_txt_path))]
|
||||
)
|
||||
|
||||
if not os.path.isfile(output_zip_path):
|
||||
raise Exception(f"Failed to create TXT ZIP file: {output_zip_path}")
|
||||
|
||||
self.logger.info(f"TXT package created: {output_zip_path}")
|
||||
return output_zip_path, output_txt_path
|
||||
|
||||
def _create_zip_from_files(
|
||||
self,
|
||||
zip_path: str,
|
||||
files: list,
|
||||
) -> None:
|
||||
"""
|
||||
Create a ZIP file from a list of files.
|
||||
|
||||
Args:
|
||||
zip_path: Path for the output ZIP file
|
||||
files: List of (file_path, archive_name) tuples
|
||||
"""
|
||||
with zipfile.ZipFile(zip_path, "w") as zipf:
|
||||
for file_path, archive_name in files:
|
||||
if os.path.isfile(file_path):
|
||||
zipf.write(file_path, archive_name)
|
||||
self.logger.debug(f"Added to ZIP: {archive_name}")
|
||||
else:
|
||||
self.logger.warning(f"File not found for ZIP: {file_path}")
|
||||
|
||||
|
||||
# Singleton instance and factory function (following existing service pattern)
|
||||
_packaging_service: Optional[PackagingService] = None
|
||||
|
||||
|
||||
def get_packaging_service(
|
||||
cdg_styles: Optional[Dict[str, Any]] = None,
|
||||
**kwargs
|
||||
) -> PackagingService:
|
||||
"""
|
||||
Get a packaging service instance.
|
||||
|
||||
Args:
|
||||
cdg_styles: CDG style configuration
|
||||
**kwargs: Additional arguments passed to PackagingService
|
||||
|
||||
Returns:
|
||||
PackagingService instance
|
||||
"""
|
||||
global _packaging_service
|
||||
|
||||
# Create new instance if settings changed
|
||||
if _packaging_service is None or cdg_styles:
|
||||
_packaging_service = PackagingService(
|
||||
cdg_styles=cdg_styles,
|
||||
**kwargs
|
||||
)
|
||||
|
||||
return _packaging_service
|
||||
@@ -1,445 +0,0 @@
|
||||
"""
|
||||
YouTube Upload Service.
|
||||
|
||||
Provides video upload functionality for YouTube, extracted from KaraokeFinalise
|
||||
for use by both the cloud backend (video_worker) and local CLI.
|
||||
|
||||
This service handles:
|
||||
- Video uploads with metadata (title, description, tags)
|
||||
- Thumbnail uploads
|
||||
- Duplicate video detection and replacement
|
||||
- Authentication via pre-stored credentials or client secrets file
|
||||
"""
|
||||
|
||||
import logging
|
||||
import os
|
||||
from typing import Optional, Dict, Any, Tuple
|
||||
|
||||
from thefuzz import fuzz
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
|
||||
class YouTubeUploadService:
|
||||
"""
|
||||
Service for uploading videos to YouTube.
|
||||
|
||||
Supports two authentication modes:
|
||||
1. Pre-stored credentials (for server-side/non-interactive use)
|
||||
2. Client secrets file (for interactive CLI use with browser OAuth)
|
||||
"""
|
||||
|
||||
YOUTUBE_URL_PREFIX = "https://www.youtube.com/watch?v="
|
||||
SCOPES = ["https://www.googleapis.com/auth/youtube"]
|
||||
|
||||
def __init__(
|
||||
self,
|
||||
credentials: Optional[Dict[str, Any]] = None,
|
||||
client_secrets_file: Optional[str] = None,
|
||||
non_interactive: bool = False,
|
||||
server_side_mode: bool = False,
|
||||
dry_run: bool = False,
|
||||
logger: Optional[logging.Logger] = None,
|
||||
):
|
||||
"""
|
||||
Initialize the YouTube upload service.
|
||||
|
||||
Args:
|
||||
credentials: Pre-stored OAuth credentials dict with keys:
|
||||
- token: Access token
|
||||
- refresh_token: Refresh token
|
||||
- token_uri: Token endpoint URL
|
||||
- client_id: OAuth client ID
|
||||
- client_secret: OAuth client secret
|
||||
- scopes: List of OAuth scopes
|
||||
client_secrets_file: Path to OAuth client secrets JSON file
|
||||
(for interactive authentication)
|
||||
non_interactive: If True, skip interactive prompts
|
||||
server_side_mode: If True, use exact matching for duplicate detection
|
||||
dry_run: If True, log actions without performing them
|
||||
logger: Optional logger instance
|
||||
"""
|
||||
self.credentials = credentials
|
||||
self.client_secrets_file = client_secrets_file
|
||||
self.non_interactive = non_interactive
|
||||
self.server_side_mode = server_side_mode
|
||||
self.dry_run = dry_run
|
||||
self.logger = logger or logging.getLogger(__name__)
|
||||
|
||||
self._youtube_service = None
|
||||
self._channel_id = None
|
||||
|
||||
def _authenticate(self):
|
||||
"""
|
||||
Authenticate with YouTube API and return service object.
|
||||
|
||||
Returns:
|
||||
YouTube API service object
|
||||
|
||||
Raises:
|
||||
Exception: If authentication fails or is required in non-interactive mode
|
||||
"""
|
||||
from google.auth.transport.requests import Request
|
||||
from google.oauth2.credentials import Credentials
|
||||
from googleapiclient.discovery import build
|
||||
from google_auth_oauthlib.flow import InstalledAppFlow
|
||||
import pickle
|
||||
|
||||
# Check if we have pre-stored credentials (for non-interactive mode)
|
||||
if self.credentials and self.non_interactive:
|
||||
try:
|
||||
creds = Credentials(
|
||||
token=self.credentials.get('token'),
|
||||
refresh_token=self.credentials.get('refresh_token'),
|
||||
token_uri=self.credentials.get('token_uri'),
|
||||
client_id=self.credentials.get('client_id'),
|
||||
client_secret=self.credentials.get('client_secret'),
|
||||
scopes=self.credentials.get('scopes', self.SCOPES)
|
||||
)
|
||||
|
||||
# Refresh token if needed
|
||||
if creds.expired and creds.refresh_token:
|
||||
creds.refresh(Request())
|
||||
|
||||
youtube = build('youtube', 'v3', credentials=creds)
|
||||
self.logger.info("Successfully authenticated with YouTube using pre-stored credentials")
|
||||
return youtube
|
||||
|
||||
except Exception as e:
|
||||
self.logger.error(f"Failed to authenticate with pre-stored credentials: {str(e)}")
|
||||
# Fall through to original authentication if pre-stored credentials fail
|
||||
|
||||
# For non-interactive mode without pre-stored credentials, raise error
|
||||
if self.non_interactive:
|
||||
raise Exception(
|
||||
"YouTube authentication required but running in non-interactive mode. "
|
||||
"Please pre-authenticate or disable YouTube upload."
|
||||
)
|
||||
|
||||
# Interactive authentication using client secrets file
|
||||
if not self.client_secrets_file:
|
||||
raise Exception("No YouTube credentials or client secrets file provided")
|
||||
|
||||
# Token file stores the user's access and refresh tokens
|
||||
youtube_token_file = "/tmp/karaoke-finalise-youtube-token.pickle"
|
||||
|
||||
creds = None
|
||||
|
||||
# Check if we have saved credentials
|
||||
if os.path.exists(youtube_token_file):
|
||||
with open(youtube_token_file, "rb") as token:
|
||||
creds = pickle.load(token)
|
||||
|
||||
# If there are no valid credentials, let the user log in
|
||||
if not creds or not creds.valid:
|
||||
if creds and creds.expired and creds.refresh_token:
|
||||
creds.refresh(Request())
|
||||
else:
|
||||
flow = InstalledAppFlow.from_client_secrets_file(
|
||||
self.client_secrets_file, scopes=self.SCOPES
|
||||
)
|
||||
creds = flow.run_local_server(port=0)
|
||||
|
||||
# Save the credentials for the next run
|
||||
with open(youtube_token_file, "wb") as token:
|
||||
pickle.dump(creds, token)
|
||||
|
||||
return build("youtube", "v3", credentials=creds)
|
||||
|
||||
@property
|
||||
def youtube_service(self):
|
||||
"""Lazy-load YouTube service on first access."""
|
||||
if self._youtube_service is None:
|
||||
self._youtube_service = self._authenticate()
|
||||
return self._youtube_service
|
||||
|
||||
def get_channel_id(self) -> Optional[str]:
|
||||
"""
|
||||
Get the authenticated user's YouTube channel ID.
|
||||
|
||||
Returns:
|
||||
Channel ID string, or None if not found
|
||||
"""
|
||||
if self._channel_id:
|
||||
return self._channel_id
|
||||
|
||||
request = self.youtube_service.channels().list(part="snippet", mine=True)
|
||||
response = request.execute()
|
||||
|
||||
if "items" in response and len(response["items"]) > 0:
|
||||
self._channel_id = response["items"][0]["id"]
|
||||
return self._channel_id
|
||||
|
||||
return None
|
||||
|
||||
def check_duplicate(
|
||||
self,
|
||||
title: str,
|
||||
exact_match: Optional[bool] = None
|
||||
) -> Tuple[bool, Optional[str], Optional[str]]:
|
||||
"""
|
||||
Check if a video with the given title already exists on the channel.
|
||||
|
||||
Args:
|
||||
title: Video title to search for
|
||||
exact_match: If True, require exact title match. If None, uses
|
||||
server_side_mode setting (exact in server mode, fuzzy in CLI)
|
||||
|
||||
Returns:
|
||||
Tuple of (exists: bool, video_id: Optional[str], video_url: Optional[str])
|
||||
"""
|
||||
channel_id = self.get_channel_id()
|
||||
if not channel_id:
|
||||
self.logger.warning("Could not get channel ID, skipping duplicate check")
|
||||
return False, None, None
|
||||
|
||||
use_exact_match = exact_match if exact_match is not None else self.server_side_mode
|
||||
|
||||
self.logger.info(f"Searching YouTube channel {channel_id} for title: {title}")
|
||||
request = self.youtube_service.search().list(
|
||||
part="snippet",
|
||||
channelId=channel_id,
|
||||
q=title,
|
||||
type="video",
|
||||
maxResults=10
|
||||
)
|
||||
response = request.execute()
|
||||
|
||||
if "items" not in response or len(response["items"]) == 0:
|
||||
self.logger.info(f"No matching video found with title: {title}")
|
||||
return False, None, None
|
||||
|
||||
for item in response["items"]:
|
||||
# Verify the video actually belongs to our channel
|
||||
result_channel_id = item["snippet"]["channelId"]
|
||||
if result_channel_id != channel_id:
|
||||
self.logger.debug(
|
||||
f"Skipping video from different channel: {item['snippet']['title']} "
|
||||
f"(channel: {result_channel_id})"
|
||||
)
|
||||
continue
|
||||
|
||||
found_title = item["snippet"]["title"]
|
||||
|
||||
# Determine if this is a match
|
||||
if use_exact_match:
|
||||
is_match = title.lower() == found_title.lower()
|
||||
similarity_score = 100 if is_match else 0
|
||||
else:
|
||||
similarity_score = fuzz.ratio(title.lower(), found_title.lower())
|
||||
is_match = similarity_score >= 70
|
||||
|
||||
if is_match:
|
||||
video_id = item["id"]["videoId"]
|
||||
video_url = f"{self.YOUTUBE_URL_PREFIX}{video_id}"
|
||||
self.logger.info(
|
||||
f"Potential match found on YouTube channel with ID: {video_id} "
|
||||
f"and title: {found_title} (similarity: {similarity_score}%)"
|
||||
)
|
||||
|
||||
# In non-interactive mode, return the match directly
|
||||
if self.non_interactive:
|
||||
self.logger.info("Non-interactive mode, found a match.")
|
||||
return True, video_id, video_url
|
||||
|
||||
# Interactive confirmation
|
||||
confirmation = input(
|
||||
f"Is '{found_title}' the video you are finalising? (y/n): "
|
||||
).strip().lower()
|
||||
if confirmation == "y":
|
||||
return True, video_id, video_url
|
||||
|
||||
self.logger.info(f"No matching video found with title: {title}")
|
||||
return False, None, None
|
||||
|
||||
def delete_video(self, video_id: str) -> bool:
|
||||
"""
|
||||
Delete a YouTube video by its ID.
|
||||
|
||||
Args:
|
||||
video_id: The YouTube video ID to delete
|
||||
|
||||
Returns:
|
||||
True if successful, False otherwise
|
||||
"""
|
||||
self.logger.info(f"Deleting YouTube video with ID: {video_id}")
|
||||
|
||||
if self.dry_run:
|
||||
self.logger.info(f"DRY RUN: Would delete YouTube video with ID: {video_id}")
|
||||
return True
|
||||
|
||||
try:
|
||||
self.youtube_service.videos().delete(id=video_id).execute()
|
||||
self.logger.info(f"Successfully deleted YouTube video with ID: {video_id}")
|
||||
return True
|
||||
except Exception as e:
|
||||
self.logger.error(f"Failed to delete YouTube video with ID {video_id}: {e}")
|
||||
return False
|
||||
|
||||
@staticmethod
|
||||
def truncate_title(title: str, max_length: int = 95) -> str:
|
||||
"""
|
||||
Truncate title to the nearest whole word within max_length.
|
||||
|
||||
Args:
|
||||
title: Title to truncate
|
||||
max_length: Maximum length (default 95 for YouTube)
|
||||
|
||||
Returns:
|
||||
Truncated title with "..." if needed
|
||||
"""
|
||||
if len(title) <= max_length:
|
||||
return title
|
||||
truncated_title = title[:max_length].rsplit(" ", 1)[0]
|
||||
if len(truncated_title) < max_length:
|
||||
truncated_title += " ..."
|
||||
return truncated_title
|
||||
|
||||
def upload_video(
|
||||
self,
|
||||
video_path: str,
|
||||
title: str,
|
||||
description: str,
|
||||
thumbnail_path: Optional[str] = None,
|
||||
tags: Optional[list] = None,
|
||||
category_id: str = "10", # Music category
|
||||
privacy_status: str = "public",
|
||||
replace_existing: bool = False,
|
||||
) -> Tuple[Optional[str], Optional[str]]:
|
||||
"""
|
||||
Upload a video to YouTube with metadata and optional thumbnail.
|
||||
|
||||
Args:
|
||||
video_path: Path to the video file to upload
|
||||
title: Video title (will be truncated to 95 chars if needed)
|
||||
description: Video description
|
||||
thumbnail_path: Optional path to thumbnail image
|
||||
tags: Optional list of tags/keywords
|
||||
category_id: YouTube category ID (default "10" for Music)
|
||||
privacy_status: "public", "private", or "unlisted"
|
||||
replace_existing: If True, delete existing video with same title
|
||||
|
||||
Returns:
|
||||
Tuple of (video_id, video_url) or (None, None) if upload failed/skipped
|
||||
"""
|
||||
from googleapiclient.http import MediaFileUpload
|
||||
|
||||
# Truncate title if needed
|
||||
youtube_title = self.truncate_title(title)
|
||||
|
||||
self.logger.info(f"Uploading video to YouTube: {youtube_title}")
|
||||
|
||||
if self.dry_run:
|
||||
self.logger.info(
|
||||
f"DRY RUN: Would upload {video_path} to YouTube with title: {youtube_title}"
|
||||
)
|
||||
return "dry_run_video_id", f"{self.YOUTUBE_URL_PREFIX}dry_run_video_id"
|
||||
|
||||
# Check for existing video
|
||||
should_replace = True if self.server_side_mode else replace_existing
|
||||
exists, existing_id, existing_url = self.check_duplicate(youtube_title)
|
||||
|
||||
if exists:
|
||||
if should_replace:
|
||||
self.logger.info(f"Video already exists on YouTube, deleting before re-upload: {existing_url}")
|
||||
if self.delete_video(existing_id):
|
||||
self.logger.info("Successfully deleted existing video, proceeding with upload")
|
||||
else:
|
||||
self.logger.error("Failed to delete existing video, aborting upload")
|
||||
return None, None
|
||||
else:
|
||||
self.logger.warning(f"Video already exists on YouTube, skipping upload: {existing_url}")
|
||||
return existing_id, existing_url
|
||||
|
||||
# Prepare video metadata
|
||||
body = {
|
||||
"snippet": {
|
||||
"title": youtube_title,
|
||||
"description": description,
|
||||
"tags": tags or [],
|
||||
"categoryId": category_id,
|
||||
},
|
||||
"status": {"privacyStatus": privacy_status},
|
||||
}
|
||||
|
||||
# Determine MIME type based on file extension
|
||||
ext = os.path.splitext(video_path)[1].lower()
|
||||
mime_type = {
|
||||
".mkv": "video/x-matroska",
|
||||
".mp4": "video/mp4",
|
||||
".mov": "video/quicktime",
|
||||
".avi": "video/x-msvideo",
|
||||
}.get(ext, "video/*")
|
||||
|
||||
# Upload video
|
||||
self.logger.info(f"Authenticating with YouTube...")
|
||||
media_file = MediaFileUpload(video_path, mimetype=mime_type, resumable=True)
|
||||
|
||||
self.logger.info(f"Uploading video to YouTube...")
|
||||
request = self.youtube_service.videos().insert(
|
||||
part="snippet,status",
|
||||
body=body,
|
||||
media_body=media_file
|
||||
)
|
||||
response = request.execute()
|
||||
|
||||
video_id = response.get("id")
|
||||
video_url = f"{self.YOUTUBE_URL_PREFIX}{video_id}"
|
||||
self.logger.info(f"Uploaded video to YouTube: {video_url}")
|
||||
|
||||
# Upload thumbnail if provided
|
||||
if thumbnail_path and os.path.isfile(thumbnail_path):
|
||||
try:
|
||||
self.logger.info(f"Uploading thumbnail from: {thumbnail_path}")
|
||||
media_thumbnail = MediaFileUpload(thumbnail_path, mimetype="image/jpeg")
|
||||
self.youtube_service.thumbnails().set(
|
||||
videoId=video_id,
|
||||
media_body=media_thumbnail
|
||||
).execute()
|
||||
self.logger.info(f"Uploaded thumbnail for video ID {video_id}")
|
||||
except Exception as e:
|
||||
self.logger.error(f"Failed to upload thumbnail: {e}")
|
||||
self.logger.warning(
|
||||
"Video uploaded but thumbnail not set. "
|
||||
"You may need to set it manually on YouTube."
|
||||
)
|
||||
elif thumbnail_path:
|
||||
self.logger.warning(f"Thumbnail file not found, skipping: {thumbnail_path}")
|
||||
|
||||
return video_id, video_url
|
||||
|
||||
|
||||
# Singleton instance and factory function (following existing service pattern)
|
||||
_youtube_upload_service: Optional[YouTubeUploadService] = None
|
||||
|
||||
|
||||
def get_youtube_upload_service(
|
||||
credentials: Optional[Dict[str, Any]] = None,
|
||||
client_secrets_file: Optional[str] = None,
|
||||
**kwargs
|
||||
) -> YouTubeUploadService:
|
||||
"""
|
||||
Get a YouTube upload service instance.
|
||||
|
||||
For server-side use, pass credentials from YouTubeService.
|
||||
For CLI use, pass client_secrets_file.
|
||||
|
||||
Args:
|
||||
credentials: Pre-stored OAuth credentials dict
|
||||
client_secrets_file: Path to OAuth client secrets JSON file
|
||||
**kwargs: Additional arguments passed to YouTubeUploadService
|
||||
|
||||
Returns:
|
||||
YouTubeUploadService instance
|
||||
"""
|
||||
global _youtube_upload_service
|
||||
|
||||
# Create new instance if credentials/settings changed
|
||||
if _youtube_upload_service is None or credentials or client_secrets_file:
|
||||
_youtube_upload_service = YouTubeUploadService(
|
||||
credentials=credentials,
|
||||
client_secrets_file=client_secrets_file,
|
||||
**kwargs
|
||||
)
|
||||
|
||||
return _youtube_upload_service
|
||||
@@ -6,30 +6,24 @@ Provides common mocks and test utilities across all test modules.
|
||||
NOTE: Module-level mocks are NOT applied here to allow emulator tests to work.
|
||||
Individual tests must mock dependencies as needed.
|
||||
"""
|
||||
# IMPORTANT: Set environment variables FIRST, before ANY imports that might
|
||||
# trigger backend module loading and Settings singleton creation.
|
||||
import os
|
||||
|
||||
# Set up test environment variables BEFORE importing any backend modules
|
||||
# This must happen before Settings is instantiated, which occurs on first import
|
||||
if 'FIRESTORE_EMULATOR_HOST' not in os.environ:
|
||||
os.environ.setdefault('ADMIN_TOKENS', 'test-admin-token')
|
||||
os.environ.setdefault('GOOGLE_CLOUD_PROJECT', 'test-project')
|
||||
os.environ.setdefault('GCS_BUCKET_NAME', 'test-bucket')
|
||||
os.environ.setdefault('FIRESTORE_COLLECTION', 'jobs')
|
||||
os.environ.setdefault('ENVIRONMENT', 'test')
|
||||
|
||||
import pytest
|
||||
from unittest.mock import Mock, MagicMock, AsyncMock, patch
|
||||
from datetime import datetime, UTC
|
||||
from fastapi.testclient import TestClient
|
||||
|
||||
|
||||
# Mock google.auth.default AND firestore for unit tests if not using emulator
|
||||
# This prevents DefaultCredentialsError and FirestoreClient initialization during imports
|
||||
if 'FIRESTORE_EMULATOR_HOST' not in os.environ:
|
||||
from unittest.mock import MagicMock
|
||||
|
||||
# Set up test environment variables BEFORE importing any backend modules
|
||||
os.environ.setdefault('ADMIN_TOKENS', 'test-admin-token')
|
||||
os.environ.setdefault('GOOGLE_CLOUD_PROJECT', 'test-project')
|
||||
os.environ.setdefault('GCS_BUCKET_NAME', 'test-bucket')
|
||||
os.environ.setdefault('FIRESTORE_COLLECTION', 'jobs')
|
||||
os.environ.setdefault('ENVIRONMENT', 'test')
|
||||
|
||||
# Mock google.auth.default - prevents credential errors
|
||||
try:
|
||||
import google.auth
|
||||
|
||||
@@ -1,244 +0,0 @@
|
||||
"""
|
||||
Tests for DiscordNotificationService.
|
||||
|
||||
Tests cover:
|
||||
- Service initialization
|
||||
- Webhook URL validation
|
||||
- Message posting
|
||||
- Video notification posting
|
||||
- Dry run mode
|
||||
"""
|
||||
|
||||
import pytest
|
||||
from unittest.mock import MagicMock, patch
|
||||
|
||||
from backend.services.discord_service import (
|
||||
DiscordNotificationService,
|
||||
get_discord_notification_service,
|
||||
)
|
||||
|
||||
|
||||
class TestDiscordNotificationServiceInit:
|
||||
"""Test service initialization."""
|
||||
|
||||
def test_init_with_webhook_url(self):
|
||||
"""Test initialization with valid webhook URL."""
|
||||
url = "https://discord.com/api/webhooks/123456789/abcdef123456"
|
||||
service = DiscordNotificationService(webhook_url=url)
|
||||
assert service.webhook_url == url
|
||||
assert service.is_enabled() is True
|
||||
|
||||
def test_init_without_webhook_url(self):
|
||||
"""Test initialization without webhook URL."""
|
||||
service = DiscordNotificationService()
|
||||
assert service.webhook_url is None
|
||||
assert service.is_enabled() is False
|
||||
|
||||
def test_init_with_dry_run(self):
|
||||
"""Test initialization with dry run mode."""
|
||||
service = DiscordNotificationService(dry_run=True)
|
||||
assert service.dry_run is True
|
||||
|
||||
def test_init_with_invalid_webhook_url_raises(self):
|
||||
"""Test that invalid webhook URL raises ValueError."""
|
||||
with pytest.raises(ValueError) as exc_info:
|
||||
DiscordNotificationService(webhook_url="https://example.com/webhook")
|
||||
assert "Invalid Discord webhook URL" in str(exc_info.value)
|
||||
|
||||
def test_init_with_non_discord_url_raises(self):
|
||||
"""Test that non-Discord URL raises ValueError."""
|
||||
with pytest.raises(ValueError):
|
||||
DiscordNotificationService(
|
||||
webhook_url="https://slack.com/api/webhooks/123"
|
||||
)
|
||||
|
||||
|
||||
class TestDiscordNotificationServiceValidation:
|
||||
"""Test webhook URL validation."""
|
||||
|
||||
def test_validate_valid_webhook_url(self):
|
||||
"""Test that valid webhook URL passes validation."""
|
||||
service = DiscordNotificationService()
|
||||
# Should not raise
|
||||
service._validate_webhook_url(
|
||||
"https://discord.com/api/webhooks/123456789/abcdef"
|
||||
)
|
||||
|
||||
def test_validate_invalid_webhook_url(self):
|
||||
"""Test that invalid webhook URL raises ValueError."""
|
||||
service = DiscordNotificationService()
|
||||
with pytest.raises(ValueError) as exc_info:
|
||||
service._validate_webhook_url("https://example.com/webhook")
|
||||
assert "Invalid Discord webhook URL" in str(exc_info.value)
|
||||
|
||||
def test_validate_strips_whitespace(self):
|
||||
"""Test that whitespace is stripped from URL."""
|
||||
service = DiscordNotificationService()
|
||||
# Should not raise - whitespace is stripped
|
||||
service._validate_webhook_url(
|
||||
" https://discord.com/api/webhooks/123456789/abcdef "
|
||||
)
|
||||
|
||||
|
||||
class TestDiscordNotificationServicePostMessage:
|
||||
"""Test message posting."""
|
||||
|
||||
@patch("backend.services.discord_service.requests.post")
|
||||
def test_post_message_success(self, mock_post):
|
||||
"""Test successful message posting."""
|
||||
mock_response = MagicMock()
|
||||
mock_response.raise_for_status = MagicMock()
|
||||
mock_post.return_value = mock_response
|
||||
|
||||
url = "https://discord.com/api/webhooks/123456789/abcdef"
|
||||
service = DiscordNotificationService(webhook_url=url)
|
||||
|
||||
result = service.post_message("Test message")
|
||||
|
||||
assert result is True
|
||||
mock_post.assert_called_once_with(
|
||||
url,
|
||||
json={"content": "Test message"},
|
||||
timeout=30
|
||||
)
|
||||
|
||||
@patch("backend.services.discord_service.requests.post")
|
||||
def test_post_message_with_custom_webhook(self, mock_post):
|
||||
"""Test message posting with custom webhook URL."""
|
||||
mock_response = MagicMock()
|
||||
mock_response.raise_for_status = MagicMock()
|
||||
mock_post.return_value = mock_response
|
||||
|
||||
service = DiscordNotificationService()
|
||||
custom_url = "https://discord.com/api/webhooks/987654321/fedcba"
|
||||
|
||||
result = service.post_message("Test message", webhook_url=custom_url)
|
||||
|
||||
assert result is True
|
||||
mock_post.assert_called_once_with(
|
||||
custom_url,
|
||||
json={"content": "Test message"},
|
||||
timeout=30
|
||||
)
|
||||
|
||||
def test_post_message_no_webhook_raises(self):
|
||||
"""Test that posting without webhook URL raises ValueError."""
|
||||
service = DiscordNotificationService()
|
||||
|
||||
with pytest.raises(ValueError) as exc_info:
|
||||
service.post_message("Test message")
|
||||
assert "No Discord webhook URL provided" in str(exc_info.value)
|
||||
|
||||
def test_post_message_dry_run(self):
|
||||
"""Test message posting in dry run mode."""
|
||||
url = "https://discord.com/api/webhooks/123456789/abcdef"
|
||||
service = DiscordNotificationService(webhook_url=url, dry_run=True)
|
||||
|
||||
result = service.post_message("Test message")
|
||||
|
||||
assert result is True
|
||||
# In dry run mode, no actual request should be made
|
||||
|
||||
@patch("backend.services.discord_service.requests.post")
|
||||
def test_post_message_http_error(self, mock_post):
|
||||
"""Test handling of HTTP errors."""
|
||||
import requests as req
|
||||
mock_response = MagicMock()
|
||||
mock_response.raise_for_status.side_effect = req.HTTPError("404 Not Found")
|
||||
mock_post.return_value = mock_response
|
||||
|
||||
url = "https://discord.com/api/webhooks/123456789/abcdef"
|
||||
service = DiscordNotificationService(webhook_url=url)
|
||||
|
||||
with pytest.raises(req.HTTPError):
|
||||
service.post_message("Test message")
|
||||
|
||||
|
||||
class TestDiscordNotificationServiceVideoNotification:
|
||||
"""Test video notification posting."""
|
||||
|
||||
@patch("backend.services.discord_service.requests.post")
|
||||
def test_post_video_notification_success(self, mock_post):
|
||||
"""Test successful video notification."""
|
||||
mock_response = MagicMock()
|
||||
mock_response.raise_for_status = MagicMock()
|
||||
mock_post.return_value = mock_response
|
||||
|
||||
url = "https://discord.com/api/webhooks/123456789/abcdef"
|
||||
service = DiscordNotificationService(webhook_url=url)
|
||||
|
||||
result = service.post_video_notification(
|
||||
"https://www.youtube.com/watch?v=abc123"
|
||||
)
|
||||
|
||||
assert result is True
|
||||
mock_post.assert_called_once()
|
||||
call_args = mock_post.call_args
|
||||
assert "New upload:" in call_args[1]["json"]["content"]
|
||||
assert "abc123" in call_args[1]["json"]["content"]
|
||||
|
||||
def test_post_video_notification_no_youtube_url(self):
|
||||
"""Test video notification with empty YouTube URL."""
|
||||
url = "https://discord.com/api/webhooks/123456789/abcdef"
|
||||
service = DiscordNotificationService(webhook_url=url)
|
||||
|
||||
result = service.post_video_notification("")
|
||||
|
||||
assert result is False
|
||||
|
||||
def test_post_video_notification_none_youtube_url(self):
|
||||
"""Test video notification with None YouTube URL."""
|
||||
url = "https://discord.com/api/webhooks/123456789/abcdef"
|
||||
service = DiscordNotificationService(webhook_url=url)
|
||||
|
||||
result = service.post_video_notification(None)
|
||||
|
||||
assert result is False
|
||||
|
||||
def test_post_video_notification_dry_run(self):
|
||||
"""Test video notification in dry run mode."""
|
||||
url = "https://discord.com/api/webhooks/123456789/abcdef"
|
||||
service = DiscordNotificationService(webhook_url=url, dry_run=True)
|
||||
|
||||
result = service.post_video_notification(
|
||||
"https://www.youtube.com/watch?v=abc123"
|
||||
)
|
||||
|
||||
assert result is True
|
||||
|
||||
|
||||
class TestGetDiscordNotificationService:
|
||||
"""Test factory function."""
|
||||
|
||||
def test_get_service_creates_instance(self):
|
||||
"""Test that factory function creates a new instance."""
|
||||
import backend.services.discord_service as module
|
||||
module._discord_notification_service = None
|
||||
|
||||
url = "https://discord.com/api/webhooks/123456789/abcdef"
|
||||
service = get_discord_notification_service(webhook_url=url)
|
||||
|
||||
assert service is not None
|
||||
assert isinstance(service, DiscordNotificationService)
|
||||
assert service.webhook_url == url
|
||||
|
||||
def test_get_service_without_webhook(self):
|
||||
"""Test factory function without webhook URL."""
|
||||
import backend.services.discord_service as module
|
||||
module._discord_notification_service = None
|
||||
|
||||
service = get_discord_notification_service()
|
||||
|
||||
assert service is not None
|
||||
assert service.webhook_url is None
|
||||
assert service.is_enabled() is False
|
||||
|
||||
def test_get_service_with_dry_run(self):
|
||||
"""Test factory function with dry run mode."""
|
||||
import backend.services.discord_service as module
|
||||
module._discord_notification_service = None
|
||||
|
||||
url = "https://discord.com/api/webhooks/123456789/abcdef"
|
||||
service = get_discord_notification_service(webhook_url=url, dry_run=True)
|
||||
|
||||
assert service.dry_run is True
|
||||
@@ -1,412 +0,0 @@
|
||||
"""
|
||||
Tests for Encoding Interface.
|
||||
|
||||
Tests cover:
|
||||
- EncodingInput and EncodingOutput dataclasses
|
||||
- LocalEncodingBackend implementation
|
||||
- Backend factory function
|
||||
"""
|
||||
|
||||
import pytest
|
||||
from unittest.mock import MagicMock, patch, AsyncMock
|
||||
|
||||
from backend.services.encoding_interface import (
|
||||
EncodingInput,
|
||||
EncodingOutput,
|
||||
EncodingBackend,
|
||||
LocalEncodingBackend,
|
||||
GCEEncodingBackend,
|
||||
get_encoding_backend,
|
||||
)
|
||||
|
||||
|
||||
class TestEncodingInput:
|
||||
"""Test EncodingInput dataclass."""
|
||||
|
||||
def test_required_fields(self):
|
||||
"""Test that required fields must be provided."""
|
||||
input_config = EncodingInput(
|
||||
title_video_path="/path/title.mov",
|
||||
karaoke_video_path="/path/karaoke.mov",
|
||||
instrumental_audio_path="/path/audio.flac",
|
||||
)
|
||||
assert input_config.title_video_path == "/path/title.mov"
|
||||
assert input_config.end_video_path is None
|
||||
|
||||
def test_all_fields(self):
|
||||
"""Test all fields including optional ones."""
|
||||
input_config = EncodingInput(
|
||||
title_video_path="/path/title.mov",
|
||||
karaoke_video_path="/path/karaoke.mov",
|
||||
instrumental_audio_path="/path/audio.flac",
|
||||
end_video_path="/path/end.mov",
|
||||
artist="Test Artist",
|
||||
title="Test Title",
|
||||
brand_code="NOMAD-1234",
|
||||
output_dir="/output",
|
||||
options={"quality": "high"}
|
||||
)
|
||||
assert input_config.artist == "Test Artist"
|
||||
assert input_config.brand_code == "NOMAD-1234"
|
||||
assert input_config.options["quality"] == "high"
|
||||
|
||||
|
||||
class TestEncodingOutput:
|
||||
"""Test EncodingOutput dataclass."""
|
||||
|
||||
def test_success_output(self):
|
||||
"""Test successful output."""
|
||||
output = EncodingOutput(
|
||||
success=True,
|
||||
lossless_4k_mp4_path="/output/video.mp4",
|
||||
encoding_backend="local"
|
||||
)
|
||||
assert output.success is True
|
||||
assert output.error_message is None
|
||||
|
||||
def test_failure_output(self):
|
||||
"""Test failure output."""
|
||||
output = EncodingOutput(
|
||||
success=False,
|
||||
error_message="Encoding failed",
|
||||
encoding_backend="gce"
|
||||
)
|
||||
assert output.success is False
|
||||
assert output.error_message == "Encoding failed"
|
||||
|
||||
def test_output_files_dict(self):
|
||||
"""Test output_files dictionary."""
|
||||
output = EncodingOutput(
|
||||
success=True,
|
||||
output_files={
|
||||
"lossless_4k_mp4": "/output/lossless.mp4",
|
||||
"720p_mp4": "/output/720p.mp4"
|
||||
}
|
||||
)
|
||||
assert "lossless_4k_mp4" in output.output_files
|
||||
assert output.output_files["720p_mp4"] == "/output/720p.mp4"
|
||||
|
||||
|
||||
class TestLocalEncodingBackend:
|
||||
"""Test LocalEncodingBackend implementation."""
|
||||
|
||||
def test_name(self):
|
||||
"""Test backend name."""
|
||||
backend = LocalEncodingBackend()
|
||||
assert backend.name == "local"
|
||||
|
||||
def test_init_with_dry_run(self):
|
||||
"""Test initialization with dry run."""
|
||||
backend = LocalEncodingBackend(dry_run=True)
|
||||
assert backend.dry_run is True
|
||||
|
||||
@pytest.mark.asyncio
|
||||
@patch("subprocess.run")
|
||||
async def test_is_available_success(self, mock_run):
|
||||
"""Test availability check when FFmpeg is installed."""
|
||||
mock_run.return_value = MagicMock(returncode=0)
|
||||
|
||||
backend = LocalEncodingBackend()
|
||||
available = await backend.is_available()
|
||||
|
||||
assert available is True
|
||||
|
||||
@pytest.mark.asyncio
|
||||
@patch("subprocess.run")
|
||||
async def test_is_available_not_found(self, mock_run):
|
||||
"""Test availability check when FFmpeg is not installed."""
|
||||
mock_run.side_effect = FileNotFoundError()
|
||||
|
||||
backend = LocalEncodingBackend()
|
||||
available = await backend.is_available()
|
||||
|
||||
assert available is False
|
||||
|
||||
@pytest.mark.asyncio
|
||||
@patch.object(LocalEncodingBackend, "is_available")
|
||||
@patch.object(LocalEncodingBackend, "_get_service")
|
||||
async def test_get_status(self, mock_get_service, mock_is_available):
|
||||
"""Test status retrieval."""
|
||||
mock_is_available.return_value = True
|
||||
mock_service = MagicMock()
|
||||
mock_service.hwaccel_available = True
|
||||
mock_service.video_encoder = "h264_nvenc"
|
||||
mock_get_service.return_value = mock_service
|
||||
|
||||
backend = LocalEncodingBackend()
|
||||
status = await backend.get_status()
|
||||
|
||||
assert status["backend"] == "local"
|
||||
assert status["available"] is True
|
||||
|
||||
@pytest.mark.asyncio
|
||||
@patch("asyncio.to_thread")
|
||||
@patch.object(LocalEncodingBackend, "_get_service")
|
||||
async def test_encode_success(self, mock_get_service, mock_to_thread):
|
||||
"""Test successful encoding."""
|
||||
from backend.services.local_encoding_service import EncodingResult
|
||||
|
||||
mock_result = EncodingResult(
|
||||
success=True,
|
||||
output_files={"key": "/path/file.mp4"}
|
||||
)
|
||||
mock_to_thread.return_value = mock_result
|
||||
|
||||
mock_service = MagicMock()
|
||||
mock_get_service.return_value = mock_service
|
||||
|
||||
backend = LocalEncodingBackend()
|
||||
input_config = EncodingInput(
|
||||
title_video_path="/input/title.mov",
|
||||
karaoke_video_path="/input/karaoke.mov",
|
||||
instrumental_audio_path="/input/audio.flac",
|
||||
artist="Test Artist",
|
||||
title="Test Title",
|
||||
output_dir="/output"
|
||||
)
|
||||
|
||||
output = await backend.encode(input_config)
|
||||
|
||||
assert output.success is True
|
||||
assert output.encoding_backend == "local"
|
||||
assert output.encoding_time_seconds is not None
|
||||
|
||||
@pytest.mark.asyncio
|
||||
@patch("asyncio.to_thread")
|
||||
@patch.object(LocalEncodingBackend, "_get_service")
|
||||
async def test_encode_failure(self, mock_get_service, mock_to_thread):
|
||||
"""Test encoding failure."""
|
||||
from backend.services.local_encoding_service import EncodingResult
|
||||
|
||||
mock_result = EncodingResult(
|
||||
success=False,
|
||||
output_files={},
|
||||
error="FFmpeg failed"
|
||||
)
|
||||
mock_to_thread.return_value = mock_result
|
||||
|
||||
mock_service = MagicMock()
|
||||
mock_get_service.return_value = mock_service
|
||||
|
||||
backend = LocalEncodingBackend()
|
||||
input_config = EncodingInput(
|
||||
title_video_path="/input/title.mov",
|
||||
karaoke_video_path="/input/karaoke.mov",
|
||||
instrumental_audio_path="/input/audio.flac",
|
||||
artist="Test",
|
||||
title="Test",
|
||||
)
|
||||
|
||||
output = await backend.encode(input_config)
|
||||
|
||||
assert output.success is False
|
||||
assert output.error_message == "FFmpeg failed"
|
||||
|
||||
|
||||
class TestGCEEncodingBackend:
|
||||
"""Test GCEEncodingBackend implementation."""
|
||||
|
||||
def test_name(self):
|
||||
"""Test backend name."""
|
||||
backend = GCEEncodingBackend()
|
||||
assert backend.name == "gce"
|
||||
|
||||
@patch.object(GCEEncodingBackend, "_get_service")
|
||||
@pytest.mark.asyncio
|
||||
async def test_is_available_enabled(self, mock_get_service):
|
||||
"""Test availability when GCE is enabled."""
|
||||
mock_service = MagicMock()
|
||||
mock_service.is_enabled = True
|
||||
mock_get_service.return_value = mock_service
|
||||
|
||||
backend = GCEEncodingBackend()
|
||||
available = await backend.is_available()
|
||||
|
||||
assert available is True
|
||||
|
||||
@patch.object(GCEEncodingBackend, "_get_service")
|
||||
@pytest.mark.asyncio
|
||||
async def test_is_available_disabled(self, mock_get_service):
|
||||
"""Test availability when GCE is disabled."""
|
||||
mock_service = MagicMock()
|
||||
mock_service.is_enabled = False
|
||||
mock_get_service.return_value = mock_service
|
||||
|
||||
backend = GCEEncodingBackend()
|
||||
available = await backend.is_available()
|
||||
|
||||
assert available is False
|
||||
|
||||
@patch.object(GCEEncodingBackend, "_get_service")
|
||||
@pytest.mark.asyncio
|
||||
async def test_encode_missing_gcs_paths(self, mock_get_service):
|
||||
"""Test encoding fails without GCS paths."""
|
||||
backend = GCEEncodingBackend()
|
||||
input_config = EncodingInput(
|
||||
title_video_path="/input/title.mov",
|
||||
karaoke_video_path="/input/karaoke.mov",
|
||||
instrumental_audio_path="/input/audio.flac",
|
||||
# Missing GCS paths in options
|
||||
)
|
||||
|
||||
output = await backend.encode(input_config)
|
||||
|
||||
assert output.success is False
|
||||
assert "gcs_path" in output.error_message.lower()
|
||||
|
||||
@patch.object(GCEEncodingBackend, "_get_service")
|
||||
@pytest.mark.asyncio
|
||||
async def test_encode_success(self, mock_get_service):
|
||||
"""Test successful GCE encoding."""
|
||||
mock_service = MagicMock()
|
||||
mock_service.encode_videos = AsyncMock(return_value={
|
||||
"status": "complete",
|
||||
"output_files": {
|
||||
"mp4_4k_lossless": "gs://bucket/output/lossless.mp4",
|
||||
"mp4_720p": "gs://bucket/output/720p.mp4",
|
||||
}
|
||||
})
|
||||
mock_get_service.return_value = mock_service
|
||||
|
||||
backend = GCEEncodingBackend()
|
||||
input_config = EncodingInput(
|
||||
title_video_path="/input/title.mov",
|
||||
karaoke_video_path="/input/karaoke.mov",
|
||||
instrumental_audio_path="/input/audio.flac",
|
||||
artist="Test Artist",
|
||||
title="Test Title",
|
||||
options={
|
||||
"job_id": "test-job",
|
||||
"input_gcs_path": "gs://bucket/input/",
|
||||
"output_gcs_path": "gs://bucket/output/",
|
||||
}
|
||||
)
|
||||
|
||||
output = await backend.encode(input_config)
|
||||
|
||||
assert output.success is True
|
||||
assert output.encoding_backend == "gce"
|
||||
mock_service.encode_videos.assert_called_once()
|
||||
|
||||
@patch.object(GCEEncodingBackend, "_get_service")
|
||||
@pytest.mark.asyncio
|
||||
async def test_encode_failure(self, mock_get_service):
|
||||
"""Test GCE encoding failure handling."""
|
||||
mock_service = MagicMock()
|
||||
mock_service.encode_videos = AsyncMock(side_effect=Exception("GCE worker error"))
|
||||
mock_get_service.return_value = mock_service
|
||||
|
||||
backend = GCEEncodingBackend()
|
||||
input_config = EncodingInput(
|
||||
title_video_path="/input/title.mov",
|
||||
karaoke_video_path="/input/karaoke.mov",
|
||||
instrumental_audio_path="/input/audio.flac",
|
||||
options={
|
||||
"job_id": "test-job",
|
||||
"input_gcs_path": "gs://bucket/input/",
|
||||
"output_gcs_path": "gs://bucket/output/",
|
||||
}
|
||||
)
|
||||
|
||||
output = await backend.encode(input_config)
|
||||
|
||||
assert output.success is False
|
||||
assert "GCE worker error" in output.error_message
|
||||
|
||||
@patch.object(GCEEncodingBackend, "_get_service")
|
||||
@pytest.mark.asyncio
|
||||
async def test_encode_handles_list_result(self, mock_get_service):
|
||||
"""Test GCE encoding handles list response gracefully.
|
||||
|
||||
This would have caught: 'list' object has no attribute 'get' error
|
||||
when GCE worker returns a list instead of a dict.
|
||||
"""
|
||||
mock_service = MagicMock()
|
||||
# Simulate GCE worker returning a list instead of dict
|
||||
mock_service.encode_videos = AsyncMock(return_value=[
|
||||
{"output_files": {"mp4_4k_lossless": "gs://bucket/output/lossless.mp4"}}
|
||||
])
|
||||
mock_get_service.return_value = mock_service
|
||||
|
||||
backend = GCEEncodingBackend()
|
||||
input_config = EncodingInput(
|
||||
title_video_path="/input/title.mov",
|
||||
karaoke_video_path="/input/karaoke.mov",
|
||||
instrumental_audio_path="/input/audio.flac",
|
||||
options={
|
||||
"job_id": "test-job",
|
||||
"input_gcs_path": "gs://bucket/input/",
|
||||
"output_gcs_path": "gs://bucket/output/",
|
||||
}
|
||||
)
|
||||
|
||||
# This should not raise an error
|
||||
output = await backend.encode(input_config)
|
||||
|
||||
# Should still succeed by extracting from the list
|
||||
assert output.success is True
|
||||
assert output.lossless_4k_mp4_path == "gs://bucket/output/lossless.mp4"
|
||||
|
||||
|
||||
class TestGetEncodingBackend:
|
||||
"""Test encoding backend factory function."""
|
||||
|
||||
def test_get_local_backend(self):
|
||||
"""Test getting local backend."""
|
||||
backend = get_encoding_backend("local")
|
||||
assert isinstance(backend, LocalEncodingBackend)
|
||||
assert backend.name == "local"
|
||||
|
||||
@patch.object(GCEEncodingBackend, "_get_service")
|
||||
def test_get_auto_backend_gce_disabled(self, mock_get_service):
|
||||
"""Test getting auto backend falls back to local when GCE disabled."""
|
||||
mock_service = MagicMock()
|
||||
mock_service.is_enabled = False
|
||||
mock_get_service.return_value = mock_service
|
||||
|
||||
backend = get_encoding_backend("auto")
|
||||
assert isinstance(backend, LocalEncodingBackend)
|
||||
|
||||
def test_get_local_backend_with_options(self):
|
||||
"""Test getting local backend with options."""
|
||||
backend = get_encoding_backend("local", dry_run=True)
|
||||
assert backend.dry_run is True
|
||||
|
||||
def test_get_gce_backend(self):
|
||||
"""Test getting GCE backend."""
|
||||
backend = get_encoding_backend("gce")
|
||||
assert isinstance(backend, GCEEncodingBackend)
|
||||
assert backend.name == "gce"
|
||||
|
||||
def test_get_unknown_backend_raises(self):
|
||||
"""Test that unknown backend raises ValueError."""
|
||||
with pytest.raises(ValueError) as exc_info:
|
||||
get_encoding_backend("unknown")
|
||||
assert "Unknown encoding backend type" in str(exc_info.value)
|
||||
|
||||
def test_get_gce_backend_with_options(self):
|
||||
"""Test getting GCE backend with common options like dry_run.
|
||||
|
||||
This ensures all backends accept the same kwargs, preventing
|
||||
errors when get_encoding_backend() passes **kwargs to different backends.
|
||||
"""
|
||||
# This would have caught: GCEEncodingBackend.__init__() got an unexpected keyword argument 'dry_run'
|
||||
backend = get_encoding_backend("gce", dry_run=True)
|
||||
assert isinstance(backend, GCEEncodingBackend)
|
||||
assert backend.dry_run is True
|
||||
|
||||
@patch.object(GCEEncodingBackend, "_get_service")
|
||||
def test_all_backends_accept_dry_run(self, mock_get_service):
|
||||
"""Test that all backend types accept dry_run parameter.
|
||||
|
||||
This is an integration test to ensure the factory function
|
||||
can pass dry_run to any backend without TypeError.
|
||||
"""
|
||||
mock_service = MagicMock()
|
||||
mock_service.is_enabled = False
|
||||
mock_get_service.return_value = mock_service
|
||||
|
||||
for backend_type in ["local", "gce", "auto"]:
|
||||
# This should not raise TypeError
|
||||
backend = get_encoding_backend(backend_type, dry_run=True)
|
||||
assert backend is not None
|
||||
@@ -1,423 +0,0 @@
|
||||
"""
|
||||
Tests for LocalEncodingService.
|
||||
|
||||
Tests cover:
|
||||
- Service initialization
|
||||
- Hardware acceleration detection
|
||||
- FFmpeg command execution
|
||||
- Individual encoding methods
|
||||
- Full encoding pipeline
|
||||
- Dry run mode
|
||||
"""
|
||||
|
||||
import os
|
||||
import tempfile
|
||||
import pytest
|
||||
from unittest.mock import MagicMock, patch, call
|
||||
|
||||
from backend.services.local_encoding_service import (
|
||||
LocalEncodingService,
|
||||
EncodingConfig,
|
||||
EncodingResult,
|
||||
get_local_encoding_service,
|
||||
)
|
||||
|
||||
|
||||
class TestLocalEncodingServiceInit:
|
||||
"""Test service initialization."""
|
||||
|
||||
def test_init_default_values(self):
|
||||
"""Test default initialization."""
|
||||
service = LocalEncodingService()
|
||||
assert service.dry_run is False
|
||||
assert "ffmpeg" in service._ffmpeg_base_command
|
||||
|
||||
def test_init_with_dry_run(self):
|
||||
"""Test initialization with dry run mode."""
|
||||
service = LocalEncodingService(dry_run=True)
|
||||
assert service.dry_run is True
|
||||
|
||||
def test_init_with_debug_logging(self):
|
||||
"""Test initialization with debug logging level."""
|
||||
import logging
|
||||
service = LocalEncodingService(log_level=logging.DEBUG)
|
||||
assert "-loglevel verbose" in service._ffmpeg_base_command
|
||||
|
||||
def test_init_with_info_logging(self):
|
||||
"""Test initialization with info logging level."""
|
||||
import logging
|
||||
service = LocalEncodingService(log_level=logging.INFO)
|
||||
assert "-loglevel fatal" in service._ffmpeg_base_command
|
||||
|
||||
|
||||
class TestLocalEncodingServiceHWAccel:
|
||||
"""Test hardware acceleration detection."""
|
||||
|
||||
@patch("subprocess.run")
|
||||
def test_detect_nvenc_available(self, mock_run):
|
||||
"""Test NVENC detection when available."""
|
||||
mock_run.return_value = MagicMock(returncode=0)
|
||||
|
||||
service = LocalEncodingService()
|
||||
# Force detection
|
||||
service._detect_and_set_hwaccel()
|
||||
|
||||
assert service._hwaccel_available is True
|
||||
assert service._video_encoder == "h264_nvenc"
|
||||
assert service._scale_filter == "scale_cuda"
|
||||
|
||||
@patch("subprocess.run")
|
||||
def test_detect_no_hwaccel(self, mock_run):
|
||||
"""Test fallback when no hardware acceleration available."""
|
||||
import subprocess
|
||||
mock_run.side_effect = subprocess.CalledProcessError(1, "test")
|
||||
|
||||
service = LocalEncodingService()
|
||||
# Force detection
|
||||
service._detect_and_set_hwaccel()
|
||||
|
||||
assert service._hwaccel_available is False
|
||||
assert service._video_encoder == "libx264"
|
||||
assert service._scale_filter == "scale"
|
||||
|
||||
def test_nvenc_quality_settings(self):
|
||||
"""Test NVENC quality settings for different presets."""
|
||||
service = LocalEncodingService()
|
||||
service._hwaccel_available = True
|
||||
|
||||
lossless = service._get_nvenc_quality_settings("lossless")
|
||||
assert "cq 0" in lossless
|
||||
|
||||
medium = service._get_nvenc_quality_settings("medium")
|
||||
assert "p4" in medium
|
||||
|
||||
def test_nvenc_quality_settings_disabled(self):
|
||||
"""Test NVENC quality settings when hwaccel is disabled."""
|
||||
service = LocalEncodingService()
|
||||
service._hwaccel_available = False
|
||||
|
||||
settings = service._get_nvenc_quality_settings("lossless")
|
||||
assert settings == ""
|
||||
|
||||
|
||||
class TestLocalEncodingServiceExecuteCommand:
|
||||
"""Test command execution."""
|
||||
|
||||
@patch("subprocess.run")
|
||||
def test_execute_command_success(self, mock_run):
|
||||
"""Test successful command execution."""
|
||||
mock_run.return_value = MagicMock(returncode=0)
|
||||
|
||||
service = LocalEncodingService()
|
||||
result = service._execute_command("echo test", "Test command")
|
||||
|
||||
assert result is True
|
||||
mock_run.assert_called_once()
|
||||
|
||||
@patch("subprocess.run")
|
||||
def test_execute_command_failure(self, mock_run):
|
||||
"""Test command execution failure."""
|
||||
import subprocess
|
||||
mock_run.side_effect = subprocess.CalledProcessError(
|
||||
1, "test", stderr="Error message"
|
||||
)
|
||||
|
||||
service = LocalEncodingService()
|
||||
result = service._execute_command("echo test", "Test command")
|
||||
|
||||
assert result is False
|
||||
|
||||
@patch("subprocess.run")
|
||||
def test_execute_command_timeout(self, mock_run):
|
||||
"""Test command execution timeout."""
|
||||
import subprocess
|
||||
mock_run.side_effect = subprocess.TimeoutExpired("test", 30)
|
||||
|
||||
service = LocalEncodingService()
|
||||
result = service._execute_command("echo test", "Test command", timeout=30)
|
||||
|
||||
assert result is False
|
||||
|
||||
def test_execute_command_dry_run(self):
|
||||
"""Test command execution in dry run mode."""
|
||||
service = LocalEncodingService(dry_run=True)
|
||||
result = service._execute_command("echo test", "Test command")
|
||||
|
||||
assert result is True
|
||||
# No actual subprocess should be called in dry run mode
|
||||
|
||||
|
||||
class TestLocalEncodingServiceEncodingMethods:
|
||||
"""Test individual encoding methods."""
|
||||
|
||||
@patch.object(LocalEncodingService, "_execute_command")
|
||||
def test_remux_with_instrumental(self, mock_execute):
|
||||
"""Test remuxing with instrumental audio."""
|
||||
mock_execute.return_value = True
|
||||
|
||||
service = LocalEncodingService()
|
||||
result = service.remux_with_instrumental(
|
||||
"/input/video.mov",
|
||||
"/input/audio.flac",
|
||||
"/output/video.mp4"
|
||||
)
|
||||
|
||||
assert result is True
|
||||
mock_execute.assert_called_once()
|
||||
call_args = mock_execute.call_args[0][0]
|
||||
assert "/input/video.mov" in call_args
|
||||
assert "/input/audio.flac" in call_args
|
||||
assert "-map 0:v -map 1:a" in call_args
|
||||
|
||||
@patch.object(LocalEncodingService, "_execute_command_with_fallback")
|
||||
def test_convert_mov_to_mp4(self, mock_execute):
|
||||
"""Test MOV to MP4 conversion."""
|
||||
mock_execute.return_value = True
|
||||
|
||||
service = LocalEncodingService()
|
||||
result = service.convert_mov_to_mp4(
|
||||
"/input/video.mov",
|
||||
"/output/video.mp4"
|
||||
)
|
||||
|
||||
assert result is True
|
||||
mock_execute.assert_called_once()
|
||||
|
||||
@patch.object(LocalEncodingService, "_execute_command_with_fallback")
|
||||
def test_encode_lossless_mp4_without_end(self, mock_execute):
|
||||
"""Test lossless 4K MP4 encoding without end credits."""
|
||||
mock_execute.return_value = True
|
||||
|
||||
service = LocalEncodingService()
|
||||
result = service.encode_lossless_mp4(
|
||||
"/input/title.mov",
|
||||
"/input/karaoke.mp4",
|
||||
"/output/lossless.mp4"
|
||||
)
|
||||
|
||||
assert result is True
|
||||
mock_execute.assert_called_once()
|
||||
call_args = mock_execute.call_args
|
||||
assert "concat=n=2" in call_args[0][0] # GPU command
|
||||
|
||||
@patch.object(LocalEncodingService, "_execute_command_with_fallback")
|
||||
def test_encode_lossless_mp4_with_end(self, mock_execute):
|
||||
"""Test lossless 4K MP4 encoding with end credits."""
|
||||
mock_execute.return_value = True
|
||||
|
||||
with tempfile.TemporaryDirectory() as tmpdir:
|
||||
end_file = os.path.join(tmpdir, "end.mov")
|
||||
with open(end_file, "w") as f:
|
||||
f.write("fake video")
|
||||
|
||||
service = LocalEncodingService()
|
||||
result = service.encode_lossless_mp4(
|
||||
"/input/title.mov",
|
||||
"/input/karaoke.mp4",
|
||||
"/output/lossless.mp4",
|
||||
end_video=end_file
|
||||
)
|
||||
|
||||
assert result is True
|
||||
call_args = mock_execute.call_args
|
||||
assert "concat=n=3" in call_args[0][0] # 3 videos
|
||||
|
||||
@patch.object(LocalEncodingService, "_execute_command")
|
||||
def test_encode_lossy_mp4(self, mock_execute):
|
||||
"""Test lossy 4K MP4 encoding."""
|
||||
mock_execute.return_value = True
|
||||
|
||||
service = LocalEncodingService()
|
||||
result = service.encode_lossy_mp4(
|
||||
"/input/lossless.mp4",
|
||||
"/output/lossy.mp4"
|
||||
)
|
||||
|
||||
assert result is True
|
||||
call_args = mock_execute.call_args[0][0]
|
||||
assert "-c:v copy" in call_args
|
||||
assert "aac" in call_args.lower()
|
||||
|
||||
@patch.object(LocalEncodingService, "_execute_command")
|
||||
def test_encode_lossless_mkv(self, mock_execute):
|
||||
"""Test MKV encoding with FLAC audio."""
|
||||
mock_execute.return_value = True
|
||||
|
||||
service = LocalEncodingService()
|
||||
result = service.encode_lossless_mkv(
|
||||
"/input/lossless.mp4",
|
||||
"/output/video.mkv"
|
||||
)
|
||||
|
||||
assert result is True
|
||||
call_args = mock_execute.call_args[0][0]
|
||||
assert "-c:v copy" in call_args
|
||||
assert "-c:a flac" in call_args
|
||||
|
||||
@patch.object(LocalEncodingService, "_execute_command_with_fallback")
|
||||
def test_encode_720p(self, mock_execute):
|
||||
"""Test 720p encoding."""
|
||||
mock_execute.return_value = True
|
||||
|
||||
service = LocalEncodingService()
|
||||
result = service.encode_720p(
|
||||
"/input/lossless.mp4",
|
||||
"/output/720p.mp4"
|
||||
)
|
||||
|
||||
assert result is True
|
||||
call_args = mock_execute.call_args
|
||||
# Should contain scale filter
|
||||
assert "1280:720" in call_args[0][0] or "1280:720" in call_args[0][1]
|
||||
|
||||
|
||||
class TestLocalEncodingServiceFullPipeline:
|
||||
"""Test full encoding pipeline."""
|
||||
|
||||
@patch.object(LocalEncodingService, "remux_with_instrumental")
|
||||
@patch.object(LocalEncodingService, "convert_mov_to_mp4")
|
||||
@patch.object(LocalEncodingService, "encode_lossless_mp4")
|
||||
@patch.object(LocalEncodingService, "encode_lossy_mp4")
|
||||
@patch.object(LocalEncodingService, "encode_lossless_mkv")
|
||||
@patch.object(LocalEncodingService, "encode_720p")
|
||||
def test_encode_all_formats_success(
|
||||
self, mock_720p, mock_mkv, mock_lossy, mock_lossless, mock_convert, mock_remux
|
||||
):
|
||||
"""Test successful full encoding pipeline."""
|
||||
mock_remux.return_value = True
|
||||
mock_convert.return_value = True
|
||||
mock_lossless.return_value = True
|
||||
mock_lossy.return_value = True
|
||||
mock_mkv.return_value = True
|
||||
mock_720p.return_value = True
|
||||
|
||||
service = LocalEncodingService()
|
||||
config = EncodingConfig(
|
||||
title_video="/input/title.mov",
|
||||
karaoke_video="/input/karaoke.mov",
|
||||
instrumental_audio="/input/instrumental.flac",
|
||||
output_karaoke_mp4="/output/karaoke.mp4",
|
||||
output_with_vocals_mp4="/output/with_vocals.mp4",
|
||||
output_lossless_4k_mp4="/output/lossless_4k.mp4",
|
||||
output_lossy_4k_mp4="/output/lossy_4k.mp4",
|
||||
output_lossless_mkv="/output/lossless.mkv",
|
||||
output_720p_mp4="/output/720p.mp4",
|
||||
)
|
||||
|
||||
result = service.encode_all_formats(config)
|
||||
|
||||
assert result.success is True
|
||||
assert "karaoke_mp4" in result.output_files
|
||||
assert "lossless_4k_mp4" in result.output_files
|
||||
assert "720p_mp4" in result.output_files
|
||||
|
||||
@patch.object(LocalEncodingService, "remux_with_instrumental")
|
||||
def test_encode_all_formats_failure_early(self, mock_remux):
|
||||
"""Test encoding pipeline failure at early step."""
|
||||
mock_remux.return_value = False
|
||||
|
||||
service = LocalEncodingService()
|
||||
config = EncodingConfig(
|
||||
title_video="/input/title.mov",
|
||||
karaoke_video="/input/karaoke.mov",
|
||||
instrumental_audio="/input/instrumental.flac",
|
||||
output_karaoke_mp4="/output/karaoke.mp4",
|
||||
output_lossless_4k_mp4="/output/lossless_4k.mp4",
|
||||
)
|
||||
|
||||
result = service.encode_all_formats(config)
|
||||
|
||||
assert result.success is False
|
||||
assert "Failed to remux" in result.error
|
||||
|
||||
def test_encode_all_formats_dry_run(self):
|
||||
"""Test encoding pipeline in dry run mode."""
|
||||
service = LocalEncodingService(dry_run=True)
|
||||
config = EncodingConfig(
|
||||
title_video="/input/title.mov",
|
||||
karaoke_video="/input/karaoke.mp4", # Already MP4
|
||||
instrumental_audio="/input/instrumental.flac",
|
||||
output_karaoke_mp4="/output/karaoke.mp4",
|
||||
output_lossless_4k_mp4="/output/lossless_4k.mp4",
|
||||
output_lossy_4k_mp4="/output/lossy_4k.mp4",
|
||||
output_lossless_mkv="/output/lossless.mkv",
|
||||
output_720p_mp4="/output/720p.mp4",
|
||||
)
|
||||
|
||||
result = service.encode_all_formats(config)
|
||||
|
||||
# In dry run mode, all operations should "succeed"
|
||||
assert result.success is True
|
||||
|
||||
|
||||
class TestEncodingConfig:
|
||||
"""Test EncodingConfig dataclass."""
|
||||
|
||||
def test_config_required_fields(self):
|
||||
"""Test that required fields must be provided."""
|
||||
config = EncodingConfig(
|
||||
title_video="/path/title.mov",
|
||||
karaoke_video="/path/karaoke.mov",
|
||||
instrumental_audio="/path/audio.flac",
|
||||
)
|
||||
assert config.title_video == "/path/title.mov"
|
||||
assert config.end_video is None # Optional field
|
||||
|
||||
def test_config_all_fields(self):
|
||||
"""Test config with all fields."""
|
||||
config = EncodingConfig(
|
||||
title_video="/path/title.mov",
|
||||
karaoke_video="/path/karaoke.mov",
|
||||
instrumental_audio="/path/audio.flac",
|
||||
end_video="/path/end.mov",
|
||||
output_karaoke_mp4="/output/karaoke.mp4",
|
||||
output_720p_mp4="/output/720p.mp4",
|
||||
)
|
||||
assert config.end_video == "/path/end.mov"
|
||||
assert config.output_karaoke_mp4 == "/output/karaoke.mp4"
|
||||
|
||||
|
||||
class TestEncodingResult:
|
||||
"""Test EncodingResult dataclass."""
|
||||
|
||||
def test_result_success(self):
|
||||
"""Test successful result."""
|
||||
result = EncodingResult(
|
||||
success=True,
|
||||
output_files={"key": "/path/file.mp4"}
|
||||
)
|
||||
assert result.success is True
|
||||
assert result.error is None
|
||||
|
||||
def test_result_failure(self):
|
||||
"""Test failure result."""
|
||||
result = EncodingResult(
|
||||
success=False,
|
||||
output_files={},
|
||||
error="Something went wrong"
|
||||
)
|
||||
assert result.success is False
|
||||
assert result.error == "Something went wrong"
|
||||
|
||||
|
||||
class TestGetLocalEncodingService:
|
||||
"""Test factory function."""
|
||||
|
||||
def test_get_service_creates_instance(self):
|
||||
"""Test that factory function creates a new instance."""
|
||||
import backend.services.local_encoding_service as module
|
||||
module._local_encoding_service = None
|
||||
|
||||
service = get_local_encoding_service()
|
||||
|
||||
assert service is not None
|
||||
assert isinstance(service, LocalEncodingService)
|
||||
|
||||
def test_get_service_with_dry_run(self):
|
||||
"""Test factory function with dry run option."""
|
||||
import backend.services.local_encoding_service as module
|
||||
module._local_encoding_service = None
|
||||
|
||||
service = get_local_encoding_service(dry_run=True)
|
||||
|
||||
assert service.dry_run is True
|
||||
@@ -1,382 +0,0 @@
|
||||
"""
|
||||
Tests for PackagingService.
|
||||
|
||||
Tests cover:
|
||||
- Service initialization
|
||||
- CDG package creation
|
||||
- TXT package creation
|
||||
- Dry run mode
|
||||
- Error handling
|
||||
"""
|
||||
|
||||
import os
|
||||
import zipfile
|
||||
import tempfile
|
||||
import pytest
|
||||
from unittest.mock import MagicMock, patch
|
||||
|
||||
from backend.services.packaging_service import (
|
||||
PackagingService,
|
||||
get_packaging_service,
|
||||
)
|
||||
|
||||
|
||||
class TestPackagingServiceInit:
|
||||
"""Test service initialization."""
|
||||
|
||||
def test_init_default_values(self):
|
||||
"""Test default initialization."""
|
||||
service = PackagingService()
|
||||
assert service.cdg_styles is None
|
||||
assert service.dry_run is False
|
||||
assert service.non_interactive is False
|
||||
|
||||
def test_init_with_cdg_styles(self):
|
||||
"""Test initialization with CDG styles."""
|
||||
styles = {"font_size": 24, "bg_color": "black"}
|
||||
service = PackagingService(cdg_styles=styles)
|
||||
assert service.cdg_styles == styles
|
||||
|
||||
def test_init_with_dry_run(self):
|
||||
"""Test initialization with dry run mode."""
|
||||
service = PackagingService(dry_run=True)
|
||||
assert service.dry_run is True
|
||||
|
||||
def test_init_with_non_interactive(self):
|
||||
"""Test initialization with non-interactive mode."""
|
||||
service = PackagingService(non_interactive=True)
|
||||
assert service.non_interactive is True
|
||||
|
||||
|
||||
class TestPackagingServiceCreateZipFromFiles:
|
||||
"""Test internal ZIP creation."""
|
||||
|
||||
def test_create_zip_from_files(self):
|
||||
"""Test creating a ZIP file from multiple files."""
|
||||
with tempfile.TemporaryDirectory() as tmpdir:
|
||||
# Create test files
|
||||
file1 = os.path.join(tmpdir, "test1.txt")
|
||||
file2 = os.path.join(tmpdir, "test2.txt")
|
||||
zip_path = os.path.join(tmpdir, "output.zip")
|
||||
|
||||
with open(file1, "w") as f:
|
||||
f.write("content1")
|
||||
with open(file2, "w") as f:
|
||||
f.write("content2")
|
||||
|
||||
service = PackagingService()
|
||||
service._create_zip_from_files(
|
||||
zip_path,
|
||||
[(file1, "test1.txt"), (file2, "test2.txt")]
|
||||
)
|
||||
|
||||
assert os.path.isfile(zip_path)
|
||||
with zipfile.ZipFile(zip_path, "r") as zf:
|
||||
names = zf.namelist()
|
||||
assert "test1.txt" in names
|
||||
assert "test2.txt" in names
|
||||
|
||||
|
||||
class TestPackagingServiceCDGPackage:
|
||||
"""Test CDG package creation."""
|
||||
|
||||
def test_create_cdg_package_missing_lrc_file(self):
|
||||
"""Test that missing LRC file raises FileNotFoundError."""
|
||||
service = PackagingService()
|
||||
|
||||
with pytest.raises(FileNotFoundError) as exc_info:
|
||||
service.create_cdg_package(
|
||||
lrc_file="/nonexistent/file.lrc",
|
||||
audio_file="/some/audio.flac",
|
||||
output_zip_path="/output/test.zip",
|
||||
artist="Test Artist",
|
||||
title="Test Title",
|
||||
)
|
||||
assert "LRC file not found" in str(exc_info.value)
|
||||
|
||||
def test_create_cdg_package_missing_audio_file(self):
|
||||
"""Test that missing audio file raises FileNotFoundError."""
|
||||
with tempfile.TemporaryDirectory() as tmpdir:
|
||||
lrc_file = os.path.join(tmpdir, "test.lrc")
|
||||
with open(lrc_file, "w") as f:
|
||||
f.write("[00:00.00]Test lyrics")
|
||||
|
||||
service = PackagingService()
|
||||
|
||||
with pytest.raises(FileNotFoundError) as exc_info:
|
||||
service.create_cdg_package(
|
||||
lrc_file=lrc_file,
|
||||
audio_file="/nonexistent/audio.flac",
|
||||
output_zip_path="/output/test.zip",
|
||||
artist="Test Artist",
|
||||
title="Test Title",
|
||||
)
|
||||
assert "Audio file not found" in str(exc_info.value)
|
||||
|
||||
def test_create_cdg_package_no_styles_raises(self):
|
||||
"""Test that CDG generation without styles raises ValueError."""
|
||||
with tempfile.TemporaryDirectory() as tmpdir:
|
||||
lrc_file = os.path.join(tmpdir, "test.lrc")
|
||||
audio_file = os.path.join(tmpdir, "test.flac")
|
||||
zip_path = os.path.join(tmpdir, "output.zip")
|
||||
|
||||
with open(lrc_file, "w") as f:
|
||||
f.write("[00:00.00]Test lyrics")
|
||||
with open(audio_file, "w") as f:
|
||||
f.write("fake audio")
|
||||
|
||||
# Service without CDG styles
|
||||
service = PackagingService(cdg_styles=None)
|
||||
|
||||
with pytest.raises(ValueError) as exc_info:
|
||||
service.create_cdg_package(
|
||||
lrc_file=lrc_file,
|
||||
audio_file=audio_file,
|
||||
output_zip_path=zip_path,
|
||||
artist="Test Artist",
|
||||
title="Test Title",
|
||||
)
|
||||
assert "CDG styles configuration is required" in str(exc_info.value)
|
||||
|
||||
def test_create_cdg_package_dry_run(self):
|
||||
"""Test CDG package creation in dry run mode."""
|
||||
with tempfile.TemporaryDirectory() as tmpdir:
|
||||
lrc_file = os.path.join(tmpdir, "test.lrc")
|
||||
audio_file = os.path.join(tmpdir, "test.flac")
|
||||
zip_path = os.path.join(tmpdir, "output.zip")
|
||||
|
||||
with open(lrc_file, "w") as f:
|
||||
f.write("[00:00.00]Test lyrics")
|
||||
with open(audio_file, "w") as f:
|
||||
f.write("fake audio")
|
||||
|
||||
service = PackagingService(dry_run=True)
|
||||
|
||||
result = service.create_cdg_package(
|
||||
lrc_file=lrc_file,
|
||||
audio_file=audio_file,
|
||||
output_zip_path=zip_path,
|
||||
artist="Test Artist",
|
||||
title="Test Title",
|
||||
)
|
||||
|
||||
# In dry run, ZIP should not be created
|
||||
assert not os.path.isfile(zip_path)
|
||||
assert result[0] == zip_path
|
||||
|
||||
def test_create_cdg_package_existing_files(self):
|
||||
"""Test CDG package creation with existing MP3 and CDG files."""
|
||||
with tempfile.TemporaryDirectory() as tmpdir:
|
||||
lrc_file = os.path.join(tmpdir, "test.lrc")
|
||||
audio_file = os.path.join(tmpdir, "test.flac")
|
||||
mp3_file = os.path.join(tmpdir, "test.mp3")
|
||||
cdg_file = os.path.join(tmpdir, "test.cdg")
|
||||
zip_path = os.path.join(tmpdir, "output.zip")
|
||||
|
||||
# Create all files
|
||||
with open(lrc_file, "w") as f:
|
||||
f.write("[00:00.00]Test lyrics")
|
||||
with open(audio_file, "w") as f:
|
||||
f.write("fake audio")
|
||||
with open(mp3_file, "w") as f:
|
||||
f.write("fake mp3")
|
||||
with open(cdg_file, "w") as f:
|
||||
f.write("fake cdg")
|
||||
|
||||
service = PackagingService()
|
||||
|
||||
result = service.create_cdg_package(
|
||||
lrc_file=lrc_file,
|
||||
audio_file=audio_file,
|
||||
output_zip_path=zip_path,
|
||||
artist="Test Artist",
|
||||
title="Test Title",
|
||||
output_mp3_path=mp3_file,
|
||||
output_cdg_path=cdg_file,
|
||||
)
|
||||
|
||||
# ZIP should be created from existing files
|
||||
assert os.path.isfile(zip_path)
|
||||
assert result[0] == zip_path
|
||||
assert result[1] == mp3_file
|
||||
assert result[2] == cdg_file
|
||||
|
||||
@patch("lyrics_transcriber.output.cdg.CDGGenerator")
|
||||
def test_create_cdg_package_with_generator(self, mock_cdg_generator_class):
|
||||
"""Test CDG package creation using CDGGenerator."""
|
||||
with tempfile.TemporaryDirectory() as tmpdir:
|
||||
lrc_file = os.path.join(tmpdir, "test.lrc")
|
||||
audio_file = os.path.join(tmpdir, "test.flac")
|
||||
zip_path = os.path.join(tmpdir, "output.zip")
|
||||
generated_zip = os.path.join(tmpdir, "generated.zip")
|
||||
|
||||
with open(lrc_file, "w") as f:
|
||||
f.write("[00:00.00]Test lyrics")
|
||||
with open(audio_file, "w") as f:
|
||||
f.write("fake audio")
|
||||
|
||||
# Create the zip file that would be generated
|
||||
with zipfile.ZipFile(generated_zip, "w") as zf:
|
||||
zf.writestr("test.mp3", "fake mp3")
|
||||
zf.writestr("test.cdg", "fake cdg")
|
||||
|
||||
mock_generator = MagicMock()
|
||||
mock_generator.generate_cdg_from_lrc.return_value = (
|
||||
"test.cdg", "test.mp3", generated_zip
|
||||
)
|
||||
mock_cdg_generator_class.return_value = mock_generator
|
||||
|
||||
styles = {"font_size": 24}
|
||||
service = PackagingService(cdg_styles=styles)
|
||||
|
||||
result = service.create_cdg_package(
|
||||
lrc_file=lrc_file,
|
||||
audio_file=audio_file,
|
||||
output_zip_path=zip_path,
|
||||
artist="Test Artist",
|
||||
title="Test Title",
|
||||
)
|
||||
|
||||
mock_generator.generate_cdg_from_lrc.assert_called_once_with(
|
||||
lrc_file=lrc_file,
|
||||
audio_file=audio_file,
|
||||
title="Test Title",
|
||||
artist="Test Artist",
|
||||
cdg_styles=styles,
|
||||
)
|
||||
assert os.path.isfile(zip_path)
|
||||
|
||||
|
||||
class TestPackagingServiceTXTPackage:
|
||||
"""Test TXT package creation."""
|
||||
|
||||
def test_create_txt_package_missing_lrc_file(self):
|
||||
"""Test that missing LRC file raises FileNotFoundError."""
|
||||
service = PackagingService()
|
||||
|
||||
with pytest.raises(FileNotFoundError) as exc_info:
|
||||
service.create_txt_package(
|
||||
lrc_file="/nonexistent/file.lrc",
|
||||
mp3_file="/some/audio.mp3",
|
||||
output_zip_path="/output/test.zip",
|
||||
)
|
||||
assert "LRC file not found" in str(exc_info.value)
|
||||
|
||||
def test_create_txt_package_missing_mp3_file(self):
|
||||
"""Test that missing MP3 file raises FileNotFoundError."""
|
||||
with tempfile.TemporaryDirectory() as tmpdir:
|
||||
lrc_file = os.path.join(tmpdir, "test.lrc")
|
||||
with open(lrc_file, "w") as f:
|
||||
f.write("[00:00.00]Test lyrics")
|
||||
|
||||
service = PackagingService()
|
||||
|
||||
with pytest.raises(FileNotFoundError) as exc_info:
|
||||
service.create_txt_package(
|
||||
lrc_file=lrc_file,
|
||||
mp3_file="/nonexistent/audio.mp3",
|
||||
output_zip_path="/output/test.zip",
|
||||
)
|
||||
assert "MP3 file not found" in str(exc_info.value)
|
||||
|
||||
def test_create_txt_package_dry_run(self):
|
||||
"""Test TXT package creation in dry run mode."""
|
||||
with tempfile.TemporaryDirectory() as tmpdir:
|
||||
lrc_file = os.path.join(tmpdir, "test.lrc")
|
||||
mp3_file = os.path.join(tmpdir, "test.mp3")
|
||||
zip_path = os.path.join(tmpdir, "output.zip")
|
||||
|
||||
with open(lrc_file, "w") as f:
|
||||
f.write("[00:00.00]Test lyrics")
|
||||
with open(mp3_file, "w") as f:
|
||||
f.write("fake mp3")
|
||||
|
||||
service = PackagingService(dry_run=True)
|
||||
|
||||
result = service.create_txt_package(
|
||||
lrc_file=lrc_file,
|
||||
mp3_file=mp3_file,
|
||||
output_zip_path=zip_path,
|
||||
)
|
||||
|
||||
# In dry run, ZIP should not be created
|
||||
assert not os.path.isfile(zip_path)
|
||||
assert result[0] == zip_path
|
||||
|
||||
@patch("lyrics_converter.LyricsConverter")
|
||||
def test_create_txt_package_success(self, mock_converter_class):
|
||||
"""Test successful TXT package creation."""
|
||||
with tempfile.TemporaryDirectory() as tmpdir:
|
||||
lrc_file = os.path.join(tmpdir, "test.lrc")
|
||||
mp3_file = os.path.join(tmpdir, "test.mp3")
|
||||
zip_path = os.path.join(tmpdir, "output.zip")
|
||||
txt_path = os.path.join(tmpdir, "output.txt")
|
||||
|
||||
with open(lrc_file, "w") as f:
|
||||
f.write("[00:00.00]Test lyrics")
|
||||
with open(mp3_file, "w") as f:
|
||||
f.write("fake mp3")
|
||||
|
||||
mock_converter = MagicMock()
|
||||
mock_converter.convert_file.return_value = "Converted lyrics text"
|
||||
mock_converter_class.return_value = mock_converter
|
||||
|
||||
service = PackagingService()
|
||||
|
||||
result = service.create_txt_package(
|
||||
lrc_file=lrc_file,
|
||||
mp3_file=mp3_file,
|
||||
output_zip_path=zip_path,
|
||||
output_txt_path=txt_path,
|
||||
)
|
||||
|
||||
mock_converter_class.assert_called_once_with(
|
||||
output_format="txt",
|
||||
filepath=lrc_file
|
||||
)
|
||||
mock_converter.convert_file.assert_called_once()
|
||||
|
||||
assert os.path.isfile(zip_path)
|
||||
assert os.path.isfile(txt_path)
|
||||
assert result[0] == zip_path
|
||||
assert result[1] == txt_path
|
||||
|
||||
# Verify ZIP contents
|
||||
with zipfile.ZipFile(zip_path, "r") as zf:
|
||||
names = zf.namelist()
|
||||
assert "test.mp3" in names
|
||||
assert "output.txt" in names
|
||||
|
||||
|
||||
class TestGetPackagingService:
|
||||
"""Test factory function."""
|
||||
|
||||
def test_get_service_creates_instance(self):
|
||||
"""Test that factory function creates a new instance."""
|
||||
import backend.services.packaging_service as module
|
||||
module._packaging_service = None
|
||||
|
||||
service = get_packaging_service()
|
||||
|
||||
assert service is not None
|
||||
assert isinstance(service, PackagingService)
|
||||
|
||||
def test_get_service_with_cdg_styles(self):
|
||||
"""Test factory function with CDG styles."""
|
||||
import backend.services.packaging_service as module
|
||||
module._packaging_service = None
|
||||
|
||||
styles = {"font_size": 24}
|
||||
service = get_packaging_service(cdg_styles=styles)
|
||||
|
||||
assert service.cdg_styles == styles
|
||||
|
||||
def test_get_service_with_options(self):
|
||||
"""Test factory function with additional options."""
|
||||
import backend.services.packaging_service as module
|
||||
module._packaging_service = None
|
||||
|
||||
service = get_packaging_service(dry_run=True, non_interactive=True)
|
||||
|
||||
assert service.dry_run is True
|
||||
assert service.non_interactive is True
|
||||
@@ -99,7 +99,7 @@ class TestStylesConfigRequirements:
|
||||
"karaoke": {
|
||||
"background_color": "#000000",
|
||||
"font_path": "", # MUST be string, NOT None
|
||||
"font": "Noto Sans",
|
||||
"font": "Arial",
|
||||
"ass_name": "Default", # REQUIRED
|
||||
"primary_color": "112, 112, 247, 255",
|
||||
"secondary_color": "255, 255, 255, 255",
|
||||
@@ -241,7 +241,7 @@ class TestPreviewStyleLoading:
|
||||
"background_image": "/original/path/background.png",
|
||||
"font_path": "/original/path/font.ttf",
|
||||
"background_color": "#000000",
|
||||
"font": "Noto Sans",
|
||||
"font": "Arial",
|
||||
"ass_name": "Default",
|
||||
}
|
||||
}
|
||||
@@ -308,7 +308,7 @@ class TestPreviewStyleLoading:
|
||||
# Should have karaoke section with minimal/default values
|
||||
assert "karaoke" in result_styles
|
||||
assert result_styles["karaoke"]["background_color"] == "#000000"
|
||||
assert result_styles["karaoke"]["font"] == "Noto Sans"
|
||||
assert result_styles["karaoke"]["font"] == "Arial"
|
||||
# Minimal styles have background_image as None (default)
|
||||
assert result_styles["karaoke"].get("background_image") is None
|
||||
|
||||
|
||||
@@ -1,742 +0,0 @@
|
||||
"""
|
||||
Tests for Video Worker Orchestrator.
|
||||
|
||||
Tests cover:
|
||||
- OrchestratorConfig and OrchestratorResult dataclasses
|
||||
- VideoWorkerOrchestrator initialization
|
||||
- Individual stage methods
|
||||
- Full pipeline execution
|
||||
- Error handling and recovery
|
||||
"""
|
||||
|
||||
import os
|
||||
import tempfile
|
||||
import pytest
|
||||
from unittest.mock import MagicMock, patch, AsyncMock
|
||||
from dataclasses import asdict
|
||||
|
||||
from backend.workers.video_worker_orchestrator import (
|
||||
OrchestratorConfig,
|
||||
OrchestratorResult,
|
||||
VideoWorkerOrchestrator,
|
||||
create_orchestrator_config_from_job,
|
||||
)
|
||||
|
||||
|
||||
class TestOrchestratorConfig:
|
||||
"""Test OrchestratorConfig dataclass."""
|
||||
|
||||
def test_required_fields(self):
|
||||
"""Test that required fields must be provided."""
|
||||
config = OrchestratorConfig(
|
||||
job_id="test-job",
|
||||
artist="Test Artist",
|
||||
title="Test Title",
|
||||
title_video_path="/path/title.mov",
|
||||
karaoke_video_path="/path/karaoke.mov",
|
||||
instrumental_audio_path="/path/audio.flac",
|
||||
)
|
||||
assert config.job_id == "test-job"
|
||||
assert config.artist == "Test Artist"
|
||||
assert config.encoding_backend == "auto"
|
||||
|
||||
def test_default_values(self):
|
||||
"""Test default values for optional fields."""
|
||||
config = OrchestratorConfig(
|
||||
job_id="test-job",
|
||||
artist="Test Artist",
|
||||
title="Test Title",
|
||||
title_video_path="/path/title.mov",
|
||||
karaoke_video_path="/path/karaoke.mov",
|
||||
instrumental_audio_path="/path/audio.flac",
|
||||
)
|
||||
assert config.enable_cdg is False
|
||||
assert config.enable_txt is False
|
||||
assert config.enable_youtube_upload is False
|
||||
assert config.dry_run is False
|
||||
assert config.non_interactive is True
|
||||
assert config.end_video_path is None
|
||||
|
||||
def test_all_fields(self):
|
||||
"""Test all fields including optional ones."""
|
||||
config = OrchestratorConfig(
|
||||
job_id="test-job",
|
||||
artist="Test Artist",
|
||||
title="Test Title",
|
||||
title_video_path="/path/title.mov",
|
||||
karaoke_video_path="/path/karaoke.mov",
|
||||
instrumental_audio_path="/path/audio.flac",
|
||||
end_video_path="/path/end.mov",
|
||||
enable_cdg=True,
|
||||
enable_txt=True,
|
||||
enable_youtube_upload=True,
|
||||
brand_prefix="TEST",
|
||||
discord_webhook_url="https://discord.com/api/webhooks/123/abc",
|
||||
encoding_backend="gce",
|
||||
)
|
||||
assert config.enable_cdg is True
|
||||
assert config.encoding_backend == "gce"
|
||||
assert config.discord_webhook_url is not None
|
||||
|
||||
|
||||
class TestOrchestratorResult:
|
||||
"""Test OrchestratorResult dataclass."""
|
||||
|
||||
def test_success_result(self):
|
||||
"""Test successful result."""
|
||||
result = OrchestratorResult(
|
||||
success=True,
|
||||
final_video="/output/video.mp4",
|
||||
brand_code="TEST-1234",
|
||||
youtube_url="https://youtube.com/watch?v=test",
|
||||
)
|
||||
assert result.success is True
|
||||
assert result.error_message is None
|
||||
assert result.brand_code == "TEST-1234"
|
||||
|
||||
def test_failure_result(self):
|
||||
"""Test failure result."""
|
||||
result = OrchestratorResult(
|
||||
success=False,
|
||||
error_message="Encoding failed",
|
||||
)
|
||||
assert result.success is False
|
||||
assert result.error_message == "Encoding failed"
|
||||
assert result.final_video is None
|
||||
|
||||
def test_all_output_files(self):
|
||||
"""Test all output file fields."""
|
||||
result = OrchestratorResult(
|
||||
success=True,
|
||||
final_video="/output/lossless.mp4",
|
||||
final_video_mkv="/output/lossless.mkv",
|
||||
final_video_lossy="/output/lossy.mp4",
|
||||
final_video_720p="/output/720p.mp4",
|
||||
final_karaoke_cdg_zip="/output/cdg.zip",
|
||||
final_karaoke_txt_zip="/output/txt.zip",
|
||||
)
|
||||
assert result.final_video == "/output/lossless.mp4"
|
||||
assert result.final_video_mkv == "/output/lossless.mkv"
|
||||
|
||||
|
||||
class TestVideoWorkerOrchestratorInit:
|
||||
"""Test VideoWorkerOrchestrator initialization."""
|
||||
|
||||
def test_init_with_config(self):
|
||||
"""Test initialization with config."""
|
||||
config = OrchestratorConfig(
|
||||
job_id="test-job",
|
||||
artist="Test Artist",
|
||||
title="Test Title",
|
||||
title_video_path="/path/title.mov",
|
||||
karaoke_video_path="/path/karaoke.mov",
|
||||
instrumental_audio_path="/path/audio.flac",
|
||||
)
|
||||
orchestrator = VideoWorkerOrchestrator(config)
|
||||
|
||||
assert orchestrator.config == config
|
||||
assert orchestrator.result.success is False
|
||||
|
||||
def test_init_with_job_manager(self):
|
||||
"""Test initialization with job manager."""
|
||||
config = OrchestratorConfig(
|
||||
job_id="test-job",
|
||||
artist="Test Artist",
|
||||
title="Test Title",
|
||||
title_video_path="/path/title.mov",
|
||||
karaoke_video_path="/path/karaoke.mov",
|
||||
instrumental_audio_path="/path/audio.flac",
|
||||
)
|
||||
job_manager = MagicMock()
|
||||
orchestrator = VideoWorkerOrchestrator(config, job_manager=job_manager)
|
||||
|
||||
assert orchestrator.job_manager == job_manager
|
||||
|
||||
|
||||
class TestVideoWorkerOrchestratorServices:
|
||||
"""Test service lazy-loading."""
|
||||
|
||||
def test_get_encoding_backend(self):
|
||||
"""Test encoding backend lazy loading."""
|
||||
config = OrchestratorConfig(
|
||||
job_id="test-job",
|
||||
artist="Test Artist",
|
||||
title="Test Title",
|
||||
title_video_path="/path/title.mov",
|
||||
karaoke_video_path="/path/karaoke.mov",
|
||||
instrumental_audio_path="/path/audio.flac",
|
||||
encoding_backend="local",
|
||||
)
|
||||
orchestrator = VideoWorkerOrchestrator(config)
|
||||
|
||||
with patch("backend.services.encoding_interface.get_encoding_backend") as mock_get:
|
||||
mock_backend = MagicMock()
|
||||
mock_backend.name = "local"
|
||||
mock_get.return_value = mock_backend
|
||||
|
||||
backend = orchestrator._get_encoding_backend()
|
||||
|
||||
assert backend == mock_backend
|
||||
mock_get.assert_called_once()
|
||||
|
||||
def test_get_packaging_service(self):
|
||||
"""Test packaging service lazy loading."""
|
||||
config = OrchestratorConfig(
|
||||
job_id="test-job",
|
||||
artist="Test Artist",
|
||||
title="Test Title",
|
||||
title_video_path="/path/title.mov",
|
||||
karaoke_video_path="/path/karaoke.mov",
|
||||
instrumental_audio_path="/path/audio.flac",
|
||||
cdg_styles={"background_color": "black"},
|
||||
)
|
||||
orchestrator = VideoWorkerOrchestrator(config)
|
||||
|
||||
with patch("backend.services.packaging_service.PackagingService") as MockService:
|
||||
mock_service = MagicMock()
|
||||
MockService.return_value = mock_service
|
||||
|
||||
service = orchestrator._get_packaging_service()
|
||||
|
||||
assert service == mock_service
|
||||
MockService.assert_called_once()
|
||||
|
||||
def test_get_youtube_service(self):
|
||||
"""Test YouTube service lazy loading."""
|
||||
config = OrchestratorConfig(
|
||||
job_id="test-job",
|
||||
artist="Test Artist",
|
||||
title="Test Title",
|
||||
title_video_path="/path/title.mov",
|
||||
karaoke_video_path="/path/karaoke.mov",
|
||||
instrumental_audio_path="/path/audio.flac",
|
||||
youtube_credentials={"token": "test"},
|
||||
)
|
||||
orchestrator = VideoWorkerOrchestrator(config)
|
||||
|
||||
with patch("backend.services.youtube_upload_service.YouTubeUploadService") as MockService:
|
||||
mock_service = MagicMock()
|
||||
MockService.return_value = mock_service
|
||||
|
||||
service = orchestrator._get_youtube_service()
|
||||
|
||||
assert service == mock_service
|
||||
MockService.assert_called_once()
|
||||
|
||||
def test_get_discord_service(self):
|
||||
"""Test Discord service lazy loading."""
|
||||
config = OrchestratorConfig(
|
||||
job_id="test-job",
|
||||
artist="Test Artist",
|
||||
title="Test Title",
|
||||
title_video_path="/path/title.mov",
|
||||
karaoke_video_path="/path/karaoke.mov",
|
||||
instrumental_audio_path="/path/audio.flac",
|
||||
discord_webhook_url="https://discord.com/api/webhooks/123/abc",
|
||||
)
|
||||
orchestrator = VideoWorkerOrchestrator(config)
|
||||
|
||||
with patch("backend.services.discord_service.DiscordNotificationService") as MockService:
|
||||
mock_service = MagicMock()
|
||||
MockService.return_value = mock_service
|
||||
|
||||
service = orchestrator._get_discord_service()
|
||||
|
||||
assert service == mock_service
|
||||
MockService.assert_called_once()
|
||||
|
||||
|
||||
class TestVideoWorkerOrchestratorPackaging:
|
||||
"""Test packaging stage."""
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_run_packaging_no_lrc(self):
|
||||
"""Test packaging stage with no LRC file."""
|
||||
config = OrchestratorConfig(
|
||||
job_id="test-job",
|
||||
artist="Test Artist",
|
||||
title="Test Title",
|
||||
title_video_path="/path/title.mov",
|
||||
karaoke_video_path="/path/karaoke.mov",
|
||||
instrumental_audio_path="/path/audio.flac",
|
||||
enable_cdg=True,
|
||||
lrc_file_path=None,
|
||||
)
|
||||
orchestrator = VideoWorkerOrchestrator(config)
|
||||
|
||||
# Should not raise, just skip
|
||||
await orchestrator._run_packaging()
|
||||
|
||||
assert orchestrator.result.final_karaoke_cdg_zip is None
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_run_packaging_cdg(self):
|
||||
"""Test CDG packaging."""
|
||||
with tempfile.TemporaryDirectory() as temp_dir:
|
||||
lrc_file = os.path.join(temp_dir, "test.lrc")
|
||||
audio_file = os.path.join(temp_dir, "test.flac")
|
||||
|
||||
# Create dummy files
|
||||
with open(lrc_file, "w") as f:
|
||||
f.write("[00:00.00]Test lyrics")
|
||||
with open(audio_file, "w") as f:
|
||||
f.write("dummy audio")
|
||||
|
||||
config = OrchestratorConfig(
|
||||
job_id="test-job",
|
||||
artist="Test Artist",
|
||||
title="Test Title",
|
||||
title_video_path="/path/title.mov",
|
||||
karaoke_video_path="/path/karaoke.mov",
|
||||
instrumental_audio_path=audio_file,
|
||||
lrc_file_path=lrc_file,
|
||||
output_dir=temp_dir,
|
||||
enable_cdg=True,
|
||||
cdg_styles={"background_color": "black"},
|
||||
)
|
||||
orchestrator = VideoWorkerOrchestrator(config)
|
||||
|
||||
with patch.object(orchestrator, "_get_packaging_service") as mock_get:
|
||||
mock_service = MagicMock()
|
||||
mock_service.create_cdg_package.return_value = (
|
||||
f"{temp_dir}/cdg.zip",
|
||||
f"{temp_dir}/test.mp3",
|
||||
f"{temp_dir}/test.cdg",
|
||||
)
|
||||
mock_get.return_value = mock_service
|
||||
|
||||
await orchestrator._run_packaging()
|
||||
|
||||
mock_service.create_cdg_package.assert_called_once()
|
||||
assert orchestrator.result.final_karaoke_cdg_zip == f"{temp_dir}/cdg.zip"
|
||||
|
||||
|
||||
class TestVideoWorkerOrchestratorEncoding:
|
||||
"""Test encoding stage."""
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_run_encoding_success(self):
|
||||
"""Test successful encoding."""
|
||||
config = OrchestratorConfig(
|
||||
job_id="test-job",
|
||||
artist="Test Artist",
|
||||
title="Test Title",
|
||||
title_video_path="/path/title.mov",
|
||||
karaoke_video_path="/path/karaoke.mov",
|
||||
instrumental_audio_path="/path/audio.flac",
|
||||
output_dir="/output",
|
||||
)
|
||||
orchestrator = VideoWorkerOrchestrator(config)
|
||||
|
||||
with patch.object(orchestrator, "_get_encoding_backend") as mock_get:
|
||||
from backend.services.encoding_interface import EncodingOutput
|
||||
|
||||
mock_backend = MagicMock()
|
||||
mock_backend.name = "local"
|
||||
mock_backend.encode = AsyncMock(return_value=EncodingOutput(
|
||||
success=True,
|
||||
lossless_4k_mp4_path="/output/lossless.mp4",
|
||||
lossy_4k_mp4_path="/output/lossy.mp4",
|
||||
lossy_720p_mp4_path="/output/720p.mp4",
|
||||
lossless_mkv_path="/output/lossless.mkv",
|
||||
encoding_time_seconds=120.5,
|
||||
encoding_backend="local",
|
||||
))
|
||||
mock_get.return_value = mock_backend
|
||||
|
||||
await orchestrator._run_encoding()
|
||||
|
||||
mock_backend.encode.assert_called_once()
|
||||
assert orchestrator.result.final_video == "/output/lossless.mp4"
|
||||
assert orchestrator.result.final_video_720p == "/output/720p.mp4"
|
||||
assert orchestrator.result.encoding_time_seconds == 120.5
|
||||
|
||||
def test_encoding_input_gcs_paths_pattern(self):
|
||||
"""Test that EncodingInput.options contains proper GCS paths structure.
|
||||
|
||||
This test verifies the GCS path construction pattern that GCE encoding
|
||||
requires. It tests the structure without running _run_encoding().
|
||||
|
||||
This would have caught: 'GCE encoding requires input_gcs_path and
|
||||
output_gcs_path in options' error when the orchestrator didn't pass
|
||||
the required paths for GCE encoding.
|
||||
"""
|
||||
from backend.services.encoding_interface import EncodingInput
|
||||
|
||||
# Test that EncodingInput can hold GCS paths in options
|
||||
# This mirrors what the orchestrator should build
|
||||
job_id = "test-job-123"
|
||||
bucket = "test-bucket"
|
||||
input_gcs_path = f"gs://{bucket}/jobs/{job_id}/"
|
||||
output_gcs_path = f"gs://{bucket}/jobs/{job_id}/finals/"
|
||||
|
||||
encoding_input = EncodingInput(
|
||||
title_video_path="/path/title.mov",
|
||||
karaoke_video_path="/path/karaoke.mov",
|
||||
instrumental_audio_path="/path/audio.flac",
|
||||
artist="Test Artist",
|
||||
title="Test Title",
|
||||
brand_code="TEST-001",
|
||||
output_dir="/output",
|
||||
options={
|
||||
"input_gcs_path": input_gcs_path,
|
||||
"output_gcs_path": output_gcs_path,
|
||||
},
|
||||
)
|
||||
|
||||
# Verify the structure that GCEEncodingBackend expects
|
||||
assert "input_gcs_path" in encoding_input.options, \
|
||||
"EncodingInput.options must include input_gcs_path for GCE encoding"
|
||||
assert "output_gcs_path" in encoding_input.options, \
|
||||
"EncodingInput.options must include output_gcs_path for GCE encoding"
|
||||
# Verify path format
|
||||
assert encoding_input.options["input_gcs_path"].startswith("gs://")
|
||||
assert encoding_input.options["output_gcs_path"].startswith("gs://")
|
||||
assert job_id in encoding_input.options["input_gcs_path"]
|
||||
assert job_id in encoding_input.options["output_gcs_path"]
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_run_encoding_failure(self):
|
||||
"""Test encoding failure."""
|
||||
config = OrchestratorConfig(
|
||||
job_id="test-job",
|
||||
artist="Test Artist",
|
||||
title="Test Title",
|
||||
title_video_path="/path/title.mov",
|
||||
karaoke_video_path="/path/karaoke.mov",
|
||||
instrumental_audio_path="/path/audio.flac",
|
||||
)
|
||||
orchestrator = VideoWorkerOrchestrator(config)
|
||||
|
||||
with patch.object(orchestrator, "_get_encoding_backend") as mock_get:
|
||||
from backend.services.encoding_interface import EncodingOutput
|
||||
|
||||
mock_backend = MagicMock()
|
||||
mock_backend.name = "local"
|
||||
mock_backend.encode = AsyncMock(return_value=EncodingOutput(
|
||||
success=False,
|
||||
error_message="FFmpeg failed",
|
||||
encoding_backend="local",
|
||||
))
|
||||
mock_get.return_value = mock_backend
|
||||
|
||||
with pytest.raises(Exception) as exc_info:
|
||||
await orchestrator._run_encoding()
|
||||
|
||||
assert "Encoding failed" in str(exc_info.value)
|
||||
|
||||
|
||||
class TestVideoWorkerOrchestratorOrganization:
|
||||
"""Test organization stage."""
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_run_organization_keep_brand_code(self):
|
||||
"""Test organization with existing brand code."""
|
||||
config = OrchestratorConfig(
|
||||
job_id="test-job",
|
||||
artist="Test Artist",
|
||||
title="Test Title",
|
||||
title_video_path="/path/title.mov",
|
||||
karaoke_video_path="/path/karaoke.mov",
|
||||
instrumental_audio_path="/path/audio.flac",
|
||||
keep_brand_code="NOMAD-9999",
|
||||
)
|
||||
orchestrator = VideoWorkerOrchestrator(config)
|
||||
|
||||
await orchestrator._run_organization()
|
||||
|
||||
assert orchestrator.result.brand_code == "NOMAD-9999"
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_run_organization_generate_brand_code(self):
|
||||
"""Test brand code generation from Dropbox."""
|
||||
config = OrchestratorConfig(
|
||||
job_id="test-job",
|
||||
artist="Test Artist",
|
||||
title="Test Title",
|
||||
title_video_path="/path/title.mov",
|
||||
karaoke_video_path="/path/karaoke.mov",
|
||||
instrumental_audio_path="/path/audio.flac",
|
||||
dropbox_path="/Karaoke/Tracks",
|
||||
brand_prefix="TEST",
|
||||
)
|
||||
orchestrator = VideoWorkerOrchestrator(config)
|
||||
|
||||
with patch("backend.services.dropbox_service.get_dropbox_service") as mock_get:
|
||||
mock_dropbox = MagicMock()
|
||||
mock_dropbox.is_configured = True
|
||||
mock_dropbox.get_next_brand_code.return_value = "TEST-0001"
|
||||
mock_get.return_value = mock_dropbox
|
||||
|
||||
await orchestrator._run_organization()
|
||||
|
||||
assert orchestrator.result.brand_code == "TEST-0001"
|
||||
|
||||
|
||||
class TestVideoWorkerOrchestratorDistribution:
|
||||
"""Test distribution stage."""
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_upload_to_youtube(self):
|
||||
"""Test YouTube upload."""
|
||||
with tempfile.TemporaryDirectory() as temp_dir:
|
||||
video_file = os.path.join(temp_dir, "test.mp4")
|
||||
with open(video_file, "w") as f:
|
||||
f.write("dummy video")
|
||||
|
||||
config = OrchestratorConfig(
|
||||
job_id="test-job",
|
||||
artist="Test Artist",
|
||||
title="Test Title",
|
||||
title_video_path="/path/title.mov",
|
||||
karaoke_video_path="/path/karaoke.mov",
|
||||
instrumental_audio_path="/path/audio.flac",
|
||||
enable_youtube_upload=True,
|
||||
youtube_credentials={"token": "test"},
|
||||
)
|
||||
orchestrator = VideoWorkerOrchestrator(config)
|
||||
orchestrator.result.final_video_lossy = video_file
|
||||
|
||||
with patch.object(orchestrator, "_get_youtube_service") as mock_get:
|
||||
mock_service = MagicMock()
|
||||
mock_service.upload_video.return_value = (
|
||||
"video123",
|
||||
"https://youtube.com/watch?v=video123"
|
||||
)
|
||||
mock_get.return_value = mock_service
|
||||
|
||||
await orchestrator._upload_to_youtube()
|
||||
|
||||
mock_service.upload_video.assert_called_once()
|
||||
assert orchestrator.result.youtube_url == "https://youtube.com/watch?v=video123"
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_upload_to_dropbox(self):
|
||||
"""Test Dropbox upload."""
|
||||
with tempfile.TemporaryDirectory() as temp_dir:
|
||||
config = OrchestratorConfig(
|
||||
job_id="test-job",
|
||||
artist="Test Artist",
|
||||
title="Test Title",
|
||||
title_video_path="/path/title.mov",
|
||||
karaoke_video_path="/path/karaoke.mov",
|
||||
instrumental_audio_path="/path/audio.flac",
|
||||
output_dir=temp_dir,
|
||||
dropbox_path="/Karaoke/Tracks",
|
||||
brand_prefix="TEST",
|
||||
)
|
||||
orchestrator = VideoWorkerOrchestrator(config)
|
||||
orchestrator.result.brand_code = "TEST-0001"
|
||||
|
||||
with patch("backend.services.dropbox_service.get_dropbox_service") as mock_get:
|
||||
mock_dropbox = MagicMock()
|
||||
mock_dropbox.is_configured = True
|
||||
mock_dropbox.create_shared_link.return_value = "https://dropbox.com/link"
|
||||
mock_get.return_value = mock_dropbox
|
||||
|
||||
await orchestrator._upload_to_dropbox()
|
||||
|
||||
mock_dropbox.upload_folder.assert_called_once()
|
||||
assert orchestrator.result.dropbox_link == "https://dropbox.com/link"
|
||||
|
||||
|
||||
class TestVideoWorkerOrchestratorNotifications:
|
||||
"""Test notifications stage."""
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_run_notifications_with_youtube_url(self):
|
||||
"""Test Discord notification with YouTube URL."""
|
||||
config = OrchestratorConfig(
|
||||
job_id="test-job",
|
||||
artist="Test Artist",
|
||||
title="Test Title",
|
||||
title_video_path="/path/title.mov",
|
||||
karaoke_video_path="/path/karaoke.mov",
|
||||
instrumental_audio_path="/path/audio.flac",
|
||||
discord_webhook_url="https://discord.com/api/webhooks/123/abc",
|
||||
)
|
||||
orchestrator = VideoWorkerOrchestrator(config)
|
||||
orchestrator.result.youtube_url = "https://youtube.com/watch?v=test"
|
||||
|
||||
with patch.object(orchestrator, "_get_discord_service") as mock_get:
|
||||
mock_service = MagicMock()
|
||||
mock_service.post_video_notification.return_value = True
|
||||
mock_get.return_value = mock_service
|
||||
|
||||
await orchestrator._run_notifications()
|
||||
|
||||
mock_service.post_video_notification.assert_called_once_with(
|
||||
"https://youtube.com/watch?v=test"
|
||||
)
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_run_notifications_no_youtube_url(self):
|
||||
"""Test notification skipped without YouTube URL."""
|
||||
config = OrchestratorConfig(
|
||||
job_id="test-job",
|
||||
artist="Test Artist",
|
||||
title="Test Title",
|
||||
title_video_path="/path/title.mov",
|
||||
karaoke_video_path="/path/karaoke.mov",
|
||||
instrumental_audio_path="/path/audio.flac",
|
||||
discord_webhook_url="https://discord.com/api/webhooks/123/abc",
|
||||
)
|
||||
orchestrator = VideoWorkerOrchestrator(config)
|
||||
# No youtube_url set
|
||||
|
||||
with patch.object(orchestrator, "_get_discord_service") as mock_get:
|
||||
mock_service = MagicMock()
|
||||
mock_get.return_value = mock_service
|
||||
|
||||
await orchestrator._run_notifications()
|
||||
|
||||
mock_service.post_video_notification.assert_not_called()
|
||||
|
||||
|
||||
class TestVideoWorkerOrchestratorFullPipeline:
|
||||
"""Test full pipeline execution."""
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_run_full_pipeline_success(self):
|
||||
"""Test successful full pipeline."""
|
||||
config = OrchestratorConfig(
|
||||
job_id="test-job",
|
||||
artist="Test Artist",
|
||||
title="Test Title",
|
||||
title_video_path="/path/title.mov",
|
||||
karaoke_video_path="/path/karaoke.mov",
|
||||
instrumental_audio_path="/path/audio.flac",
|
||||
enable_cdg=True, # Enable to trigger packaging stage
|
||||
)
|
||||
orchestrator = VideoWorkerOrchestrator(config)
|
||||
|
||||
# Mock all stages
|
||||
with patch.object(orchestrator, "_run_packaging", new_callable=AsyncMock) as mock_packaging, \
|
||||
patch.object(orchestrator, "_run_encoding", new_callable=AsyncMock) as mock_encoding, \
|
||||
patch.object(orchestrator, "_run_organization", new_callable=AsyncMock) as mock_org, \
|
||||
patch.object(orchestrator, "_run_distribution", new_callable=AsyncMock) as mock_dist, \
|
||||
patch.object(orchestrator, "_run_notifications", new_callable=AsyncMock) as mock_notify:
|
||||
|
||||
result = await orchestrator.run()
|
||||
|
||||
assert result.success is True
|
||||
mock_packaging.assert_called_once()
|
||||
mock_encoding.assert_called_once()
|
||||
mock_org.assert_called_once()
|
||||
mock_dist.assert_called_once()
|
||||
mock_notify.assert_called_once()
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_run_pipeline_skips_packaging_when_disabled(self):
|
||||
"""Test pipeline skips packaging when CDG/TXT disabled."""
|
||||
config = OrchestratorConfig(
|
||||
job_id="test-job",
|
||||
artist="Test Artist",
|
||||
title="Test Title",
|
||||
title_video_path="/path/title.mov",
|
||||
karaoke_video_path="/path/karaoke.mov",
|
||||
instrumental_audio_path="/path/audio.flac",
|
||||
enable_cdg=False,
|
||||
enable_txt=False,
|
||||
)
|
||||
orchestrator = VideoWorkerOrchestrator(config)
|
||||
|
||||
# Mock all stages
|
||||
with patch.object(orchestrator, "_run_packaging", new_callable=AsyncMock) as mock_packaging, \
|
||||
patch.object(orchestrator, "_run_encoding", new_callable=AsyncMock) as mock_encoding, \
|
||||
patch.object(orchestrator, "_run_organization", new_callable=AsyncMock), \
|
||||
patch.object(orchestrator, "_run_distribution", new_callable=AsyncMock), \
|
||||
patch.object(orchestrator, "_run_notifications", new_callable=AsyncMock):
|
||||
|
||||
result = await orchestrator.run()
|
||||
|
||||
assert result.success is True
|
||||
mock_packaging.assert_not_called()
|
||||
mock_encoding.assert_called_once()
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_run_full_pipeline_encoding_failure(self):
|
||||
"""Test pipeline failure during encoding."""
|
||||
config = OrchestratorConfig(
|
||||
job_id="test-job",
|
||||
artist="Test Artist",
|
||||
title="Test Title",
|
||||
title_video_path="/path/title.mov",
|
||||
karaoke_video_path="/path/karaoke.mov",
|
||||
instrumental_audio_path="/path/audio.flac",
|
||||
)
|
||||
orchestrator = VideoWorkerOrchestrator(config)
|
||||
|
||||
with patch.object(orchestrator, "_run_packaging", new_callable=AsyncMock), \
|
||||
patch.object(orchestrator, "_run_encoding", new_callable=AsyncMock) as mock_encoding, \
|
||||
patch.object(orchestrator, "_run_organization", new_callable=AsyncMock) as mock_org:
|
||||
|
||||
mock_encoding.side_effect = Exception("Encoding failed")
|
||||
|
||||
result = await orchestrator.run()
|
||||
|
||||
assert result.success is False
|
||||
assert "Encoding failed" in result.error_message
|
||||
mock_org.assert_not_called()
|
||||
|
||||
|
||||
class TestCreateOrchestratorConfigFromJob:
|
||||
"""Test helper function for creating config from job."""
|
||||
|
||||
def test_create_config_from_job(self):
|
||||
"""Test creating config from a job object."""
|
||||
job = MagicMock()
|
||||
job.job_id = "test-123"
|
||||
job.artist = "Test Artist"
|
||||
job.title = "Test Title"
|
||||
job.state_data = {"instrumental_selection": "clean"}
|
||||
job.enable_cdg = True
|
||||
job.enable_txt = False
|
||||
job.enable_youtube_upload = True
|
||||
job.brand_prefix = "NOMAD"
|
||||
job.discord_webhook_url = "https://discord.com/api/webhooks/123/abc"
|
||||
job.youtube_description_template = "Test description"
|
||||
job.dropbox_path = "/Karaoke"
|
||||
job.gdrive_folder_id = None
|
||||
job.keep_brand_code = None
|
||||
job.existing_instrumental_gcs_path = None
|
||||
|
||||
config = create_orchestrator_config_from_job(
|
||||
job=job,
|
||||
temp_dir="/tmp/test",
|
||||
youtube_credentials={"token": "test"},
|
||||
cdg_styles={"background": "black"},
|
||||
)
|
||||
|
||||
assert config.job_id == "test-123"
|
||||
assert config.artist == "Test Artist"
|
||||
assert config.title == "Test Title"
|
||||
assert config.enable_cdg is True
|
||||
assert config.enable_youtube_upload is True
|
||||
assert config.title_video_path == "/tmp/test/Test Artist - Test Title (Title).mov"
|
||||
assert config.instrumental_audio_path == "/tmp/test/Test Artist - Test Title (Instrumental Clean).flac"
|
||||
|
||||
def test_create_config_from_job_with_existing_instrumental(self):
|
||||
"""Test config with user-provided instrumental."""
|
||||
job = MagicMock()
|
||||
job.job_id = "test-123"
|
||||
job.artist = "Test Artist"
|
||||
job.title = "Test Title"
|
||||
job.state_data = {"instrumental_selection": "custom"}
|
||||
job.enable_cdg = False
|
||||
job.enable_txt = False
|
||||
job.enable_youtube_upload = False
|
||||
job.brand_prefix = None
|
||||
job.discord_webhook_url = None
|
||||
job.youtube_description_template = None
|
||||
job.dropbox_path = None
|
||||
job.gdrive_folder_id = None
|
||||
job.keep_brand_code = None
|
||||
job.existing_instrumental_gcs_path = "gs://bucket/instrumental.mp3"
|
||||
|
||||
config = create_orchestrator_config_from_job(
|
||||
job=job,
|
||||
temp_dir="/tmp/test",
|
||||
)
|
||||
|
||||
assert config.instrumental_audio_path == "/tmp/test/Test Artist - Test Title (Instrumental User).mp3"
|
||||
@@ -1,568 +0,0 @@
|
||||
"""
|
||||
Tests for YouTubeUploadService.
|
||||
|
||||
Tests cover:
|
||||
- Authentication with pre-stored credentials
|
||||
- Authentication with client secrets file
|
||||
- Duplicate video detection (exact and fuzzy matching)
|
||||
- Video deletion
|
||||
- Video upload with metadata and thumbnail
|
||||
- Title truncation
|
||||
- Dry run mode
|
||||
"""
|
||||
|
||||
import os
|
||||
import pytest
|
||||
from unittest.mock import MagicMock, patch, mock_open
|
||||
|
||||
from backend.services.youtube_upload_service import YouTubeUploadService, get_youtube_upload_service
|
||||
|
||||
|
||||
class TestYouTubeUploadServiceInit:
|
||||
"""Test service initialization."""
|
||||
|
||||
def test_init_with_credentials(self):
|
||||
"""Test initialization with pre-stored credentials."""
|
||||
creds = {
|
||||
"token": "test_token",
|
||||
"refresh_token": "test_refresh",
|
||||
"token_uri": "https://oauth2.googleapis.com/token",
|
||||
"client_id": "test_client_id",
|
||||
"client_secret": "test_secret",
|
||||
}
|
||||
service = YouTubeUploadService(
|
||||
credentials=creds,
|
||||
non_interactive=True
|
||||
)
|
||||
assert service.credentials == creds
|
||||
assert service.non_interactive is True
|
||||
|
||||
def test_init_with_client_secrets(self):
|
||||
"""Test initialization with client secrets file."""
|
||||
service = YouTubeUploadService(
|
||||
client_secrets_file="/path/to/secrets.json"
|
||||
)
|
||||
assert service.client_secrets_file == "/path/to/secrets.json"
|
||||
assert service.non_interactive is False
|
||||
|
||||
def test_init_default_values(self):
|
||||
"""Test default values on initialization."""
|
||||
service = YouTubeUploadService()
|
||||
assert service.credentials is None
|
||||
assert service.client_secrets_file is None
|
||||
assert service.non_interactive is False
|
||||
assert service.server_side_mode is False
|
||||
assert service.dry_run is False
|
||||
|
||||
|
||||
class TestYouTubeUploadServiceAuthentication:
|
||||
"""Test authentication methods."""
|
||||
|
||||
@patch("googleapiclient.discovery.build")
|
||||
@patch("google.oauth2.credentials.Credentials")
|
||||
@patch("google.auth.transport.requests.Request")
|
||||
def test_authenticate_with_prestored_credentials(
|
||||
self, mock_request, mock_credentials_class, mock_build
|
||||
):
|
||||
"""Test authentication using pre-stored credentials."""
|
||||
# Setup mocks
|
||||
mock_creds = MagicMock()
|
||||
mock_creds.expired = False
|
||||
mock_credentials_class.return_value = mock_creds
|
||||
mock_youtube = MagicMock()
|
||||
mock_build.return_value = mock_youtube
|
||||
|
||||
creds = {
|
||||
"token": "test_token",
|
||||
"refresh_token": "test_refresh",
|
||||
"token_uri": "https://oauth2.googleapis.com/token",
|
||||
"client_id": "test_client_id",
|
||||
"client_secret": "test_secret",
|
||||
"scopes": ["https://www.googleapis.com/auth/youtube"],
|
||||
}
|
||||
service = YouTubeUploadService(credentials=creds, non_interactive=True)
|
||||
|
||||
# Access youtube_service to trigger authentication
|
||||
result = service.youtube_service
|
||||
|
||||
assert result == mock_youtube
|
||||
mock_credentials_class.assert_called_once()
|
||||
mock_build.assert_called_once_with('youtube', 'v3', credentials=mock_creds)
|
||||
|
||||
@patch("googleapiclient.discovery.build")
|
||||
@patch("google.oauth2.credentials.Credentials")
|
||||
@patch("google.auth.transport.requests.Request")
|
||||
def test_authenticate_refreshes_expired_token(
|
||||
self, mock_request, mock_credentials_class, mock_build
|
||||
):
|
||||
"""Test that expired tokens are refreshed."""
|
||||
mock_creds = MagicMock()
|
||||
mock_creds.expired = True
|
||||
mock_creds.refresh_token = "test_refresh"
|
||||
mock_credentials_class.return_value = mock_creds
|
||||
|
||||
creds = {
|
||||
"token": "expired_token",
|
||||
"refresh_token": "test_refresh",
|
||||
"token_uri": "https://oauth2.googleapis.com/token",
|
||||
"client_id": "test_client_id",
|
||||
"client_secret": "test_secret",
|
||||
}
|
||||
service = YouTubeUploadService(credentials=creds, non_interactive=True)
|
||||
service.youtube_service
|
||||
|
||||
mock_creds.refresh.assert_called_once()
|
||||
|
||||
def test_authenticate_non_interactive_no_credentials_raises(self):
|
||||
"""Test that non-interactive mode without credentials raises error."""
|
||||
service = YouTubeUploadService(non_interactive=True)
|
||||
|
||||
with pytest.raises(Exception) as exc_info:
|
||||
service._authenticate()
|
||||
|
||||
assert "non-interactive mode" in str(exc_info.value).lower()
|
||||
|
||||
|
||||
class TestYouTubeUploadServiceChannelId:
|
||||
"""Test channel ID retrieval."""
|
||||
|
||||
@patch("googleapiclient.discovery.build")
|
||||
@patch("google.oauth2.credentials.Credentials")
|
||||
@patch("google.auth.transport.requests.Request")
|
||||
def test_get_channel_id_success(
|
||||
self, mock_request, mock_credentials_class, mock_build
|
||||
):
|
||||
"""Test successful channel ID retrieval."""
|
||||
mock_creds = MagicMock()
|
||||
mock_creds.expired = False
|
||||
mock_credentials_class.return_value = mock_creds
|
||||
|
||||
mock_youtube = MagicMock()
|
||||
mock_channels = MagicMock()
|
||||
mock_list = MagicMock()
|
||||
mock_list.execute.return_value = {
|
||||
"items": [{"id": "UC123456"}]
|
||||
}
|
||||
mock_channels.list.return_value = mock_list
|
||||
mock_youtube.channels.return_value = mock_channels
|
||||
mock_build.return_value = mock_youtube
|
||||
|
||||
creds = {"token": "test", "refresh_token": "test", "token_uri": "uri",
|
||||
"client_id": "id", "client_secret": "secret"}
|
||||
service = YouTubeUploadService(credentials=creds, non_interactive=True)
|
||||
|
||||
channel_id = service.get_channel_id()
|
||||
|
||||
assert channel_id == "UC123456"
|
||||
|
||||
@patch("googleapiclient.discovery.build")
|
||||
@patch("google.oauth2.credentials.Credentials")
|
||||
@patch("google.auth.transport.requests.Request")
|
||||
def test_get_channel_id_no_items(
|
||||
self, mock_request, mock_credentials_class, mock_build
|
||||
):
|
||||
"""Test channel ID returns None when no items found."""
|
||||
mock_creds = MagicMock()
|
||||
mock_creds.expired = False
|
||||
mock_credentials_class.return_value = mock_creds
|
||||
|
||||
mock_youtube = MagicMock()
|
||||
mock_channels = MagicMock()
|
||||
mock_list = MagicMock()
|
||||
mock_list.execute.return_value = {"items": []}
|
||||
mock_channels.list.return_value = mock_list
|
||||
mock_youtube.channels.return_value = mock_channels
|
||||
mock_build.return_value = mock_youtube
|
||||
|
||||
creds = {"token": "test", "refresh_token": "test", "token_uri": "uri",
|
||||
"client_id": "id", "client_secret": "secret"}
|
||||
service = YouTubeUploadService(credentials=creds, non_interactive=True)
|
||||
|
||||
channel_id = service.get_channel_id()
|
||||
|
||||
assert channel_id is None
|
||||
|
||||
|
||||
class TestYouTubeUploadServiceDuplicateCheck:
|
||||
"""Test duplicate video detection."""
|
||||
|
||||
@patch("googleapiclient.discovery.build")
|
||||
@patch("google.oauth2.credentials.Credentials")
|
||||
@patch("google.auth.transport.requests.Request")
|
||||
def test_check_duplicate_exact_match_found(
|
||||
self, mock_request, mock_credentials_class, mock_build
|
||||
):
|
||||
"""Test duplicate detection with exact match in server mode."""
|
||||
mock_creds = MagicMock()
|
||||
mock_creds.expired = False
|
||||
mock_credentials_class.return_value = mock_creds
|
||||
|
||||
mock_youtube = MagicMock()
|
||||
# Mock channels().list() for get_channel_id
|
||||
mock_channels = MagicMock()
|
||||
mock_channels_list = MagicMock()
|
||||
mock_channels_list.execute.return_value = {"items": [{"id": "UC123456"}]}
|
||||
mock_channels.list.return_value = mock_channels_list
|
||||
mock_youtube.channels.return_value = mock_channels
|
||||
|
||||
# Mock search().list() for duplicate check
|
||||
mock_search = MagicMock()
|
||||
mock_search_list = MagicMock()
|
||||
mock_search_list.execute.return_value = {
|
||||
"items": [{
|
||||
"id": {"videoId": "VIDEO123"},
|
||||
"snippet": {
|
||||
"channelId": "UC123456",
|
||||
"title": "Test Artist - Test Song (Karaoke)"
|
||||
}
|
||||
}]
|
||||
}
|
||||
mock_search.list.return_value = mock_search_list
|
||||
mock_youtube.search.return_value = mock_search
|
||||
mock_build.return_value = mock_youtube
|
||||
|
||||
creds = {"token": "test", "refresh_token": "test", "token_uri": "uri",
|
||||
"client_id": "id", "client_secret": "secret"}
|
||||
service = YouTubeUploadService(
|
||||
credentials=creds,
|
||||
non_interactive=True,
|
||||
server_side_mode=True
|
||||
)
|
||||
|
||||
exists, video_id, video_url = service.check_duplicate(
|
||||
"Test Artist - Test Song (Karaoke)"
|
||||
)
|
||||
|
||||
assert exists is True
|
||||
assert video_id == "VIDEO123"
|
||||
assert "VIDEO123" in video_url
|
||||
|
||||
@patch("googleapiclient.discovery.build")
|
||||
@patch("google.oauth2.credentials.Credentials")
|
||||
@patch("google.auth.transport.requests.Request")
|
||||
def test_check_duplicate_no_match(
|
||||
self, mock_request, mock_credentials_class, mock_build
|
||||
):
|
||||
"""Test duplicate detection when no match found."""
|
||||
mock_creds = MagicMock()
|
||||
mock_creds.expired = False
|
||||
mock_credentials_class.return_value = mock_creds
|
||||
|
||||
mock_youtube = MagicMock()
|
||||
mock_channels = MagicMock()
|
||||
mock_channels_list = MagicMock()
|
||||
mock_channels_list.execute.return_value = {"items": [{"id": "UC123456"}]}
|
||||
mock_channels.list.return_value = mock_channels_list
|
||||
mock_youtube.channels.return_value = mock_channels
|
||||
|
||||
mock_search = MagicMock()
|
||||
mock_search_list = MagicMock()
|
||||
mock_search_list.execute.return_value = {"items": []}
|
||||
mock_search.list.return_value = mock_search_list
|
||||
mock_youtube.search.return_value = mock_search
|
||||
mock_build.return_value = mock_youtube
|
||||
|
||||
creds = {"token": "test", "refresh_token": "test", "token_uri": "uri",
|
||||
"client_id": "id", "client_secret": "secret"}
|
||||
service = YouTubeUploadService(credentials=creds, non_interactive=True)
|
||||
|
||||
exists, video_id, video_url = service.check_duplicate("Some Title")
|
||||
|
||||
assert exists is False
|
||||
assert video_id is None
|
||||
assert video_url is None
|
||||
|
||||
@patch("googleapiclient.discovery.build")
|
||||
@patch("google.oauth2.credentials.Credentials")
|
||||
@patch("google.auth.transport.requests.Request")
|
||||
def test_check_duplicate_skips_other_channels(
|
||||
self, mock_request, mock_credentials_class, mock_build
|
||||
):
|
||||
"""Test that videos from other channels are skipped."""
|
||||
mock_creds = MagicMock()
|
||||
mock_creds.expired = False
|
||||
mock_credentials_class.return_value = mock_creds
|
||||
|
||||
mock_youtube = MagicMock()
|
||||
mock_channels = MagicMock()
|
||||
mock_channels_list = MagicMock()
|
||||
mock_channels_list.execute.return_value = {"items": [{"id": "UC123456"}]}
|
||||
mock_channels.list.return_value = mock_channels_list
|
||||
mock_youtube.channels.return_value = mock_channels
|
||||
|
||||
# Return video from different channel
|
||||
mock_search = MagicMock()
|
||||
mock_search_list = MagicMock()
|
||||
mock_search_list.execute.return_value = {
|
||||
"items": [{
|
||||
"id": {"videoId": "VIDEO123"},
|
||||
"snippet": {
|
||||
"channelId": "UC_DIFFERENT", # Different channel
|
||||
"title": "Test Title"
|
||||
}
|
||||
}]
|
||||
}
|
||||
mock_search.list.return_value = mock_search_list
|
||||
mock_youtube.search.return_value = mock_search
|
||||
mock_build.return_value = mock_youtube
|
||||
|
||||
creds = {"token": "test", "refresh_token": "test", "token_uri": "uri",
|
||||
"client_id": "id", "client_secret": "secret"}
|
||||
service = YouTubeUploadService(credentials=creds, non_interactive=True)
|
||||
|
||||
exists, video_id, video_url = service.check_duplicate("Test Title")
|
||||
|
||||
assert exists is False
|
||||
|
||||
|
||||
class TestYouTubeUploadServiceDelete:
|
||||
"""Test video deletion."""
|
||||
|
||||
@patch("googleapiclient.discovery.build")
|
||||
@patch("google.oauth2.credentials.Credentials")
|
||||
@patch("google.auth.transport.requests.Request")
|
||||
def test_delete_video_success(
|
||||
self, mock_request, mock_credentials_class, mock_build
|
||||
):
|
||||
"""Test successful video deletion."""
|
||||
mock_creds = MagicMock()
|
||||
mock_creds.expired = False
|
||||
mock_credentials_class.return_value = mock_creds
|
||||
|
||||
mock_youtube = MagicMock()
|
||||
mock_videos = MagicMock()
|
||||
mock_delete = MagicMock()
|
||||
mock_delete.execute.return_value = None
|
||||
mock_videos.delete.return_value = mock_delete
|
||||
mock_youtube.videos.return_value = mock_videos
|
||||
mock_build.return_value = mock_youtube
|
||||
|
||||
creds = {"token": "test", "refresh_token": "test", "token_uri": "uri",
|
||||
"client_id": "id", "client_secret": "secret"}
|
||||
service = YouTubeUploadService(credentials=creds, non_interactive=True)
|
||||
|
||||
result = service.delete_video("VIDEO123")
|
||||
|
||||
assert result is True
|
||||
mock_videos.delete.assert_called_once_with(id="VIDEO123")
|
||||
|
||||
@patch("googleapiclient.discovery.build")
|
||||
@patch("google.oauth2.credentials.Credentials")
|
||||
@patch("google.auth.transport.requests.Request")
|
||||
def test_delete_video_failure(
|
||||
self, mock_request, mock_credentials_class, mock_build
|
||||
):
|
||||
"""Test video deletion failure handling."""
|
||||
mock_creds = MagicMock()
|
||||
mock_creds.expired = False
|
||||
mock_credentials_class.return_value = mock_creds
|
||||
|
||||
mock_youtube = MagicMock()
|
||||
mock_videos = MagicMock()
|
||||
mock_delete = MagicMock()
|
||||
mock_delete.execute.side_effect = Exception("API Error")
|
||||
mock_videos.delete.return_value = mock_delete
|
||||
mock_youtube.videos.return_value = mock_videos
|
||||
mock_build.return_value = mock_youtube
|
||||
|
||||
creds = {"token": "test", "refresh_token": "test", "token_uri": "uri",
|
||||
"client_id": "id", "client_secret": "secret"}
|
||||
service = YouTubeUploadService(credentials=creds, non_interactive=True)
|
||||
|
||||
result = service.delete_video("VIDEO123")
|
||||
|
||||
assert result is False
|
||||
|
||||
def test_delete_video_dry_run(self):
|
||||
"""Test video deletion in dry run mode."""
|
||||
service = YouTubeUploadService(dry_run=True)
|
||||
|
||||
result = service.delete_video("VIDEO123")
|
||||
|
||||
assert result is True
|
||||
|
||||
|
||||
class TestYouTubeUploadServiceTitleTruncation:
|
||||
"""Test title truncation."""
|
||||
|
||||
def test_truncate_title_short_title(self):
|
||||
"""Test that short titles are not truncated."""
|
||||
title = "Short Title"
|
||||
result = YouTubeUploadService.truncate_title(title)
|
||||
assert result == title
|
||||
|
||||
def test_truncate_title_exact_length(self):
|
||||
"""Test title at exact max length."""
|
||||
title = "A" * 95
|
||||
result = YouTubeUploadService.truncate_title(title, max_length=95)
|
||||
assert result == title
|
||||
assert len(result) == 95
|
||||
|
||||
def test_truncate_title_long_title(self):
|
||||
"""Test that long titles are truncated at word boundary."""
|
||||
title = "This is a very long title that exceeds the maximum length and needs to be truncated properly at a word boundary"
|
||||
result = YouTubeUploadService.truncate_title(title, max_length=50)
|
||||
assert len(result) <= 50
|
||||
assert result.endswith("...")
|
||||
|
||||
def test_truncate_title_no_space(self):
|
||||
"""Test truncation of title without spaces."""
|
||||
title = "A" * 100
|
||||
result = YouTubeUploadService.truncate_title(title, max_length=50)
|
||||
assert len(result) <= 50
|
||||
|
||||
|
||||
class TestYouTubeUploadServiceUpload:
|
||||
"""Test video upload."""
|
||||
|
||||
@patch("googleapiclient.http.MediaFileUpload")
|
||||
@patch("googleapiclient.discovery.build")
|
||||
@patch("google.oauth2.credentials.Credentials")
|
||||
@patch("google.auth.transport.requests.Request")
|
||||
def test_upload_video_success(
|
||||
self, mock_request, mock_credentials_class, mock_build, mock_media_upload
|
||||
):
|
||||
"""Test successful video upload."""
|
||||
mock_creds = MagicMock()
|
||||
mock_creds.expired = False
|
||||
mock_credentials_class.return_value = mock_creds
|
||||
|
||||
mock_youtube = MagicMock()
|
||||
# Mock channels for get_channel_id
|
||||
mock_channels = MagicMock()
|
||||
mock_channels_list = MagicMock()
|
||||
mock_channels_list.execute.return_value = {"items": [{"id": "UC123456"}]}
|
||||
mock_channels.list.return_value = mock_channels_list
|
||||
mock_youtube.channels.return_value = mock_channels
|
||||
|
||||
# Mock search for duplicate check (no duplicates)
|
||||
mock_search = MagicMock()
|
||||
mock_search_list = MagicMock()
|
||||
mock_search_list.execute.return_value = {"items": []}
|
||||
mock_search.list.return_value = mock_search_list
|
||||
mock_youtube.search.return_value = mock_search
|
||||
|
||||
# Mock video insert
|
||||
mock_videos = MagicMock()
|
||||
mock_insert = MagicMock()
|
||||
mock_insert.execute.return_value = {"id": "NEW_VIDEO_ID"}
|
||||
mock_videos.insert.return_value = mock_insert
|
||||
mock_youtube.videos.return_value = mock_videos
|
||||
mock_build.return_value = mock_youtube
|
||||
|
||||
mock_media_upload.return_value = MagicMock()
|
||||
|
||||
creds = {"token": "test", "refresh_token": "test", "token_uri": "uri",
|
||||
"client_id": "id", "client_secret": "secret"}
|
||||
service = YouTubeUploadService(credentials=creds, non_interactive=True)
|
||||
|
||||
video_id, video_url = service.upload_video(
|
||||
video_path="/path/to/video.mkv",
|
||||
title="Test Artist - Test Song (Karaoke)",
|
||||
description="Test description",
|
||||
tags=["karaoke", "test"]
|
||||
)
|
||||
|
||||
assert video_id == "NEW_VIDEO_ID"
|
||||
assert "NEW_VIDEO_ID" in video_url
|
||||
|
||||
def test_upload_video_dry_run(self):
|
||||
"""Test video upload in dry run mode."""
|
||||
service = YouTubeUploadService(dry_run=True)
|
||||
|
||||
video_id, video_url = service.upload_video(
|
||||
video_path="/path/to/video.mkv",
|
||||
title="Test Title",
|
||||
description="Test description"
|
||||
)
|
||||
|
||||
assert video_id == "dry_run_video_id"
|
||||
assert "dry_run_video_id" in video_url
|
||||
|
||||
@patch("googleapiclient.http.MediaFileUpload")
|
||||
@patch("googleapiclient.discovery.build")
|
||||
@patch("google.oauth2.credentials.Credentials")
|
||||
@patch("google.auth.transport.requests.Request")
|
||||
@patch("os.path.isfile")
|
||||
def test_upload_video_with_thumbnail(
|
||||
self, mock_isfile, mock_request, mock_credentials_class,
|
||||
mock_build, mock_media_upload
|
||||
):
|
||||
"""Test video upload with thumbnail."""
|
||||
mock_isfile.return_value = True
|
||||
mock_creds = MagicMock()
|
||||
mock_creds.expired = False
|
||||
mock_credentials_class.return_value = mock_creds
|
||||
|
||||
mock_youtube = MagicMock()
|
||||
mock_channels = MagicMock()
|
||||
mock_channels_list = MagicMock()
|
||||
mock_channels_list.execute.return_value = {"items": [{"id": "UC123456"}]}
|
||||
mock_channels.list.return_value = mock_channels_list
|
||||
mock_youtube.channels.return_value = mock_channels
|
||||
|
||||
mock_search = MagicMock()
|
||||
mock_search_list = MagicMock()
|
||||
mock_search_list.execute.return_value = {"items": []}
|
||||
mock_search.list.return_value = mock_search_list
|
||||
mock_youtube.search.return_value = mock_search
|
||||
|
||||
mock_videos = MagicMock()
|
||||
mock_insert = MagicMock()
|
||||
mock_insert.execute.return_value = {"id": "NEW_VIDEO_ID"}
|
||||
mock_videos.insert.return_value = mock_insert
|
||||
mock_youtube.videos.return_value = mock_videos
|
||||
|
||||
# Mock thumbnail upload
|
||||
mock_thumbnails = MagicMock()
|
||||
mock_set = MagicMock()
|
||||
mock_set.execute.return_value = None
|
||||
mock_thumbnails.set.return_value = mock_set
|
||||
mock_youtube.thumbnails.return_value = mock_thumbnails
|
||||
|
||||
mock_build.return_value = mock_youtube
|
||||
mock_media_upload.return_value = MagicMock()
|
||||
|
||||
creds = {"token": "test", "refresh_token": "test", "token_uri": "uri",
|
||||
"client_id": "id", "client_secret": "secret"}
|
||||
service = YouTubeUploadService(credentials=creds, non_interactive=True)
|
||||
|
||||
video_id, video_url = service.upload_video(
|
||||
video_path="/path/to/video.mkv",
|
||||
title="Test Title",
|
||||
description="Test description",
|
||||
thumbnail_path="/path/to/thumbnail.jpg"
|
||||
)
|
||||
|
||||
assert video_id == "NEW_VIDEO_ID"
|
||||
mock_thumbnails.set.assert_called_once()
|
||||
|
||||
|
||||
class TestGetYouTubeUploadService:
|
||||
"""Test factory function."""
|
||||
|
||||
def test_get_service_creates_instance(self):
|
||||
"""Test that factory function creates a new instance."""
|
||||
# Reset global
|
||||
import backend.services.youtube_upload_service as module
|
||||
module._youtube_upload_service = None
|
||||
|
||||
service = get_youtube_upload_service(
|
||||
credentials={"token": "test"},
|
||||
non_interactive=True
|
||||
)
|
||||
|
||||
assert service is not None
|
||||
assert isinstance(service, YouTubeUploadService)
|
||||
|
||||
def test_get_service_with_client_secrets(self):
|
||||
"""Test factory function with client secrets file."""
|
||||
import backend.services.youtube_upload_service as module
|
||||
module._youtube_upload_service = None
|
||||
|
||||
service = get_youtube_upload_service(
|
||||
client_secrets_file="/path/to/secrets.json"
|
||||
)
|
||||
|
||||
assert service is not None
|
||||
assert service.client_secrets_file == "/path/to/secrets.json"
|
||||
+18
-256
@@ -1,21 +1,20 @@
|
||||
"""
|
||||
Video generation and finalization worker.
|
||||
|
||||
This worker has two code paths controlled by USE_NEW_ORCHESTRATOR:
|
||||
This worker uses KaraokeFinalise from the karaoke_gen package to handle:
|
||||
1. Video encoding (4 formats)
|
||||
2. CDG/TXT package generation
|
||||
3. Discord notifications
|
||||
4. Brand code generation
|
||||
5. All other finalisation features
|
||||
|
||||
1. NEW PATH (USE_NEW_ORCHESTRATOR=true):
|
||||
Uses VideoWorkerOrchestrator which provides a unified pipeline for all
|
||||
encoding backends (GCE or local). This ensures features like YouTube upload,
|
||||
Discord notifications, and CDG/TXT packaging work regardless of encoding backend.
|
||||
The worker's job is simply to:
|
||||
1. Download files from GCS and set up the directory structure KaraokeFinalise expects
|
||||
2. Instantiate KaraokeFinalise with parameters from the job
|
||||
3. Call process()
|
||||
4. Upload results back to GCS
|
||||
|
||||
2. LEGACY PATH (USE_NEW_ORCHESTRATOR=false):
|
||||
Uses KaraokeFinalise from the karaoke_gen package. This path has divergent
|
||||
behavior between GCE and local encoding - GCE encoding bypasses many features.
|
||||
|
||||
The orchestrator-based approach is preferred as it:
|
||||
- Eliminates GCE vs local code path divergence
|
||||
- Ensures all features work consistently
|
||||
- Provides better testability and maintainability
|
||||
This reuses 100% of the existing karaoke_gen functionality.
|
||||
|
||||
Observability:
|
||||
- All operations wrapped in tracing spans for Cloud Trace visibility
|
||||
@@ -49,12 +48,6 @@ from karaoke_gen.karaoke_finalise.karaoke_finalise import KaraokeFinalise
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
|
||||
# Feature flag for new orchestrator-based pipeline
|
||||
# Set to True to use the new unified pipeline that works with any encoding backend
|
||||
# Set to False to use the legacy KaraokeFinalise-based pipeline
|
||||
USE_NEW_ORCHESTRATOR = os.environ.get("USE_NEW_ORCHESTRATOR", "true").lower() == "true"
|
||||
|
||||
|
||||
# Loggers to capture for video worker
|
||||
# Include the full module path to properly capture KaraokeFinalise logs
|
||||
VIDEO_WORKER_LOGGERS = [
|
||||
@@ -179,247 +172,16 @@ async def _encode_via_gce(
|
||||
raise
|
||||
|
||||
|
||||
async def generate_video_orchestrated(job_id: str) -> bool:
|
||||
"""
|
||||
Generate final karaoke videos using the new VideoWorkerOrchestrator.
|
||||
|
||||
This provides a unified pipeline that works with any encoding backend
|
||||
(GCE or local), ensuring all features like YouTube upload, Discord
|
||||
notifications, and CDG/TXT packaging work consistently.
|
||||
|
||||
Args:
|
||||
job_id: Job ID to process
|
||||
|
||||
Returns:
|
||||
True if successful, False otherwise
|
||||
"""
|
||||
from backend.workers.video_worker_orchestrator import (
|
||||
VideoWorkerOrchestrator,
|
||||
create_orchestrator_config_from_job,
|
||||
)
|
||||
|
||||
start_time = time.time()
|
||||
job_manager = JobManager()
|
||||
storage = StorageService()
|
||||
|
||||
# Create job logger for remote debugging
|
||||
job_log = create_job_logger(job_id, "video")
|
||||
|
||||
# Log with structured markers for easy Cloud Logging queries
|
||||
logger.info(f"[job:{job_id}] WORKER_START worker=video orchestrator=true")
|
||||
|
||||
# Set up log capture for KaraokeFinalise (still used by some services)
|
||||
log_handler = setup_job_logging(job_id, "video", *VIDEO_WORKER_LOGGERS)
|
||||
|
||||
job = job_manager.get_job(job_id)
|
||||
if not job:
|
||||
logger.error(f"[job:{job_id}] Job not found")
|
||||
return False
|
||||
|
||||
# Validate prerequisites
|
||||
if not _validate_prerequisites(job):
|
||||
logger.error(f"[job:{job_id}] Prerequisites not met for video generation")
|
||||
return False
|
||||
|
||||
# Create temporary working directory
|
||||
temp_dir = tempfile.mkdtemp(prefix=f"karaoke_video_{job_id}_")
|
||||
original_cwd = os.getcwd()
|
||||
|
||||
# Set up rclone config if needed (for legacy Dropbox upload path)
|
||||
rclone_service = None
|
||||
if getattr(job, 'organised_dir_rclone_root', None):
|
||||
rclone_service = get_rclone_service()
|
||||
if rclone_service.setup_rclone_config():
|
||||
job_log.info("Rclone config loaded for Dropbox upload")
|
||||
|
||||
# Load YouTube credentials if needed
|
||||
youtube_credentials = None
|
||||
if getattr(job, 'enable_youtube_upload', False):
|
||||
youtube_service = get_youtube_service()
|
||||
if youtube_service.is_configured:
|
||||
youtube_credentials = youtube_service.get_credentials_dict()
|
||||
job_log.info("YouTube credentials loaded for video upload")
|
||||
else:
|
||||
job_log.warning("YouTube credentials not available - upload will be skipped")
|
||||
|
||||
try:
|
||||
# Wrap entire worker in a tracing span
|
||||
with job_span("video-worker-orchestrated", job_id, {"artist": job.artist, "title": job.title}) as root_span:
|
||||
with job_logging_context(job_id):
|
||||
job_log.info(f"Starting orchestrated video generation for {job.artist} - {job.title}")
|
||||
logger.info(f"[job:{job_id}] Starting orchestrated video generation")
|
||||
|
||||
# Transition to GENERATING_VIDEO state
|
||||
job_manager.transition_to_state(
|
||||
job_id=job_id,
|
||||
new_status=JobStatus.GENERATING_VIDEO,
|
||||
progress=70,
|
||||
message="Preparing files for video generation"
|
||||
)
|
||||
|
||||
# Download and set up files
|
||||
base_name = f"{job.artist} - {job.title}"
|
||||
with job_span("download-files", job_id):
|
||||
job_log.info("Downloading files from GCS...")
|
||||
await _setup_working_directory(job_id, job, storage, temp_dir, base_name, job_log)
|
||||
job_log.info("All files downloaded successfully")
|
||||
|
||||
# Load style config for CDG styles
|
||||
style_config = await load_style_config(job, storage, temp_dir)
|
||||
cdg_styles = style_config.get_cdg_styles()
|
||||
|
||||
# Change to working directory
|
||||
os.chdir(temp_dir)
|
||||
|
||||
# Create orchestrator config from job
|
||||
config = create_orchestrator_config_from_job(
|
||||
job=job,
|
||||
temp_dir=temp_dir,
|
||||
youtube_credentials=youtube_credentials,
|
||||
cdg_styles=cdg_styles,
|
||||
)
|
||||
|
||||
# Run the orchestrated pipeline
|
||||
orchestrator = VideoWorkerOrchestrator(
|
||||
config=config,
|
||||
job_manager=job_manager,
|
||||
storage=storage,
|
||||
job_logger=job_log,
|
||||
)
|
||||
|
||||
with job_span("orchestrator-run", job_id):
|
||||
result = await orchestrator.run()
|
||||
|
||||
if not result.success:
|
||||
raise Exception(result.error_message or "Orchestrator failed")
|
||||
|
||||
# Prepare distribution directory for native uploads
|
||||
with job_span("distribution", job_id):
|
||||
await _handle_native_distribution(
|
||||
job_id=job_id,
|
||||
job=job,
|
||||
job_log=job_log,
|
||||
job_manager=job_manager,
|
||||
temp_dir=temp_dir,
|
||||
result={
|
||||
'brand_code': result.brand_code,
|
||||
'youtube_url': result.youtube_url,
|
||||
'final_video': result.final_video,
|
||||
'final_video_lossy': result.final_video_lossy,
|
||||
'final_video_720p': result.final_video_720p,
|
||||
'final_karaoke_cdg_zip': result.final_karaoke_cdg_zip,
|
||||
'dropbox_link': result.dropbox_link,
|
||||
'gdrive_files': result.gdrive_files,
|
||||
},
|
||||
storage=storage,
|
||||
)
|
||||
|
||||
# Upload generated files to GCS
|
||||
job_manager.transition_to_state(
|
||||
job_id=job_id,
|
||||
new_status=JobStatus.PACKAGING,
|
||||
progress=95,
|
||||
message="Uploading final files"
|
||||
)
|
||||
|
||||
with job_span("upload-results", job_id):
|
||||
await _upload_results(job_id, job_manager, storage, temp_dir, {
|
||||
'final_video': result.final_video,
|
||||
'final_video_mkv': result.final_video_mkv,
|
||||
'final_video_lossy': result.final_video_lossy,
|
||||
'final_video_720p': result.final_video_720p,
|
||||
'final_karaoke_cdg_zip': result.final_karaoke_cdg_zip,
|
||||
'final_karaoke_txt_zip': result.final_karaoke_txt_zip,
|
||||
})
|
||||
|
||||
# Mark job as complete
|
||||
logger.info(f"[job:{job_id}] Video generation complete")
|
||||
job_manager.transition_to_state(
|
||||
job_id=job_id,
|
||||
new_status=JobStatus.COMPLETE,
|
||||
progress=100,
|
||||
message="Karaoke generation complete!"
|
||||
)
|
||||
|
||||
# Store result metadata in job
|
||||
job_manager.update_job(job_id, {
|
||||
'state_data': {
|
||||
**job.state_data,
|
||||
'brand_code': result.brand_code,
|
||||
'youtube_url': result.youtube_url,
|
||||
'dropbox_link': result.dropbox_link,
|
||||
'gdrive_files': result.gdrive_files,
|
||||
}
|
||||
})
|
||||
|
||||
duration = time.time() - start_time
|
||||
root_span.set_attribute("duration_seconds", duration)
|
||||
root_span.set_attribute("brand_code", result.brand_code or '')
|
||||
logger.info(f"[job:{job_id}] WORKER_END worker=video orchestrator=true status=success duration={duration:.1f}s")
|
||||
return True
|
||||
|
||||
except Exception as e:
|
||||
duration = time.time() - start_time
|
||||
logger.error(f"[job:{job_id}] WORKER_END worker=video orchestrator=true status=error duration={duration:.1f}s error={e}")
|
||||
job_manager.mark_job_failed(
|
||||
job_id=job_id,
|
||||
error_message=f"Video generation failed: {str(e)}",
|
||||
error_details={"stage": "video_generation", "error": str(e)}
|
||||
)
|
||||
return False
|
||||
|
||||
finally:
|
||||
# Restore original working directory
|
||||
os.chdir(original_cwd)
|
||||
|
||||
# Remove log handler
|
||||
for logger_name in VIDEO_WORKER_LOGGERS:
|
||||
try:
|
||||
logging.getLogger(logger_name).removeHandler(log_handler)
|
||||
except Exception:
|
||||
pass
|
||||
|
||||
# Cleanup rclone config file
|
||||
if rclone_service:
|
||||
rclone_service.cleanup()
|
||||
|
||||
# Cleanup temporary directory
|
||||
if os.path.exists(temp_dir):
|
||||
shutil.rmtree(temp_dir)
|
||||
logger.debug(f"Cleaned up temp directory: {temp_dir}")
|
||||
|
||||
|
||||
async def generate_video(job_id: str) -> bool:
|
||||
"""
|
||||
Generate final karaoke videos.
|
||||
|
||||
Routes to either the new orchestrator-based pipeline or the legacy
|
||||
KaraokeFinalise-based pipeline based on USE_NEW_ORCHESTRATOR flag.
|
||||
|
||||
Generate final karaoke videos using KaraokeFinalise.
|
||||
|
||||
This sets up the directory structure and calls KaraokeFinalise.process()
|
||||
which handles all encoding, packaging, and notifications.
|
||||
|
||||
Args:
|
||||
job_id: Job ID to process
|
||||
|
||||
Returns:
|
||||
True if successful, False otherwise
|
||||
"""
|
||||
if USE_NEW_ORCHESTRATOR:
|
||||
logger.info(f"[job:{job_id}] Using new orchestrator-based pipeline")
|
||||
return await generate_video_orchestrated(job_id)
|
||||
else:
|
||||
logger.info(f"[job:{job_id}] Using legacy KaraokeFinalise pipeline")
|
||||
return await generate_video_legacy(job_id)
|
||||
|
||||
|
||||
async def generate_video_legacy(job_id: str) -> bool:
|
||||
"""
|
||||
Generate final karaoke videos using KaraokeFinalise (legacy path).
|
||||
|
||||
This is the original implementation that has divergent behavior between
|
||||
GCE and local encoding paths. Kept for rollback purposes.
|
||||
|
||||
Args:
|
||||
job_id: Job ID to process
|
||||
|
||||
|
||||
Returns:
|
||||
True if successful, False otherwise
|
||||
"""
|
||||
|
||||
@@ -1,638 +0,0 @@
|
||||
"""
|
||||
Video Worker Orchestrator.
|
||||
|
||||
Coordinates the video generation pipeline stages in a unified way,
|
||||
regardless of whether encoding happens locally or on GCE.
|
||||
|
||||
This resolves the code path divergence where GCE encoding bypassed
|
||||
features like YouTube upload, Discord notifications, and CDG/TXT packaging.
|
||||
|
||||
Pipeline stages:
|
||||
1. Setup - Download files, prepare directories
|
||||
2. Packaging - CDG/TXT generation (if enabled)
|
||||
3. Encoding - GCE or Local via EncodingBackend interface
|
||||
4. Organization - Brand code, folder structure
|
||||
5. Distribution - YouTube, Dropbox, Google Drive uploads
|
||||
6. Notification - Discord notifications
|
||||
"""
|
||||
|
||||
import logging
|
||||
import os
|
||||
import time
|
||||
from dataclasses import dataclass, field
|
||||
from typing import Optional, Dict, Any, List
|
||||
from pathlib import Path
|
||||
|
||||
from backend.models.job import JobStatus
|
||||
from backend.services.job_manager import JobManager
|
||||
from backend.services.storage_service import StorageService
|
||||
from backend.services.tracing import job_span, add_span_event
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
|
||||
@dataclass
|
||||
class OrchestratorConfig:
|
||||
"""Configuration for the video worker orchestrator."""
|
||||
job_id: str
|
||||
artist: str
|
||||
title: str
|
||||
|
||||
# Input file paths (in temp_dir)
|
||||
title_video_path: str
|
||||
karaoke_video_path: str
|
||||
instrumental_audio_path: str
|
||||
end_video_path: Optional[str] = None
|
||||
lrc_file_path: Optional[str] = None
|
||||
title_jpg_path: Optional[str] = None
|
||||
|
||||
# Output directory
|
||||
output_dir: str = ""
|
||||
|
||||
# Feature flags
|
||||
enable_cdg: bool = False
|
||||
enable_txt: bool = False
|
||||
enable_youtube_upload: bool = False
|
||||
|
||||
# Service configurations
|
||||
brand_prefix: Optional[str] = None
|
||||
discord_webhook_url: Optional[str] = None
|
||||
youtube_credentials: Optional[Dict[str, Any]] = None
|
||||
youtube_description_template: Optional[str] = None
|
||||
cdg_styles: Optional[Dict[str, Any]] = None
|
||||
|
||||
# Dropbox/GDrive configuration
|
||||
dropbox_path: Optional[str] = None
|
||||
gdrive_folder_id: Optional[str] = None
|
||||
|
||||
# Keep existing brand code (for re-processing)
|
||||
keep_brand_code: Optional[str] = None
|
||||
|
||||
# Encoding backend preference
|
||||
encoding_backend: str = "auto" # "auto", "local", "gce"
|
||||
|
||||
# Additional options
|
||||
dry_run: bool = False
|
||||
non_interactive: bool = True
|
||||
|
||||
|
||||
@dataclass
|
||||
class OrchestratorResult:
|
||||
"""Result from the video worker orchestrator."""
|
||||
success: bool
|
||||
error_message: Optional[str] = None
|
||||
|
||||
# Generated files
|
||||
final_video: Optional[str] = None # Lossless 4K MP4
|
||||
final_video_mkv: Optional[str] = None # Lossless 4K MKV
|
||||
final_video_lossy: Optional[str] = None # Lossy 4K MP4
|
||||
final_video_720p: Optional[str] = None # Lossy 720p MP4
|
||||
final_karaoke_cdg_zip: Optional[str] = None
|
||||
final_karaoke_txt_zip: Optional[str] = None
|
||||
|
||||
# Organization
|
||||
brand_code: Optional[str] = None
|
||||
|
||||
# Distribution results
|
||||
youtube_url: Optional[str] = None
|
||||
dropbox_link: Optional[str] = None
|
||||
gdrive_files: Optional[Dict[str, str]] = field(default_factory=dict)
|
||||
|
||||
# Timing
|
||||
encoding_time_seconds: Optional[float] = None
|
||||
total_time_seconds: Optional[float] = None
|
||||
|
||||
|
||||
class VideoWorkerOrchestrator:
|
||||
"""
|
||||
Orchestrates the video generation pipeline.
|
||||
|
||||
This class coordinates all stages of video generation in a unified way,
|
||||
ensuring that features like YouTube upload and Discord notifications
|
||||
work regardless of whether GCE or local encoding is used.
|
||||
"""
|
||||
|
||||
def __init__(
|
||||
self,
|
||||
config: OrchestratorConfig,
|
||||
job_manager: Optional[JobManager] = None,
|
||||
storage: Optional[StorageService] = None,
|
||||
job_logger: Optional[logging.Logger] = None,
|
||||
):
|
||||
"""
|
||||
Initialize the orchestrator.
|
||||
|
||||
Args:
|
||||
config: Orchestrator configuration
|
||||
job_manager: Job manager for state updates (optional)
|
||||
storage: Storage service for file downloads (optional)
|
||||
job_logger: Job-specific logger (optional)
|
||||
"""
|
||||
self.config = config
|
||||
self.job_manager = job_manager
|
||||
self.storage = storage
|
||||
self.job_log = job_logger or logging.getLogger(__name__)
|
||||
|
||||
# Services (lazy-loaded)
|
||||
self._encoding_backend = None
|
||||
self._packaging_service = None
|
||||
self._youtube_service = None
|
||||
self._discord_service = None
|
||||
|
||||
# State
|
||||
self.result = OrchestratorResult(success=False)
|
||||
|
||||
def _get_encoding_backend(self):
|
||||
"""Lazy-load the encoding backend."""
|
||||
if self._encoding_backend is None:
|
||||
from backend.services.encoding_interface import get_encoding_backend
|
||||
self._encoding_backend = get_encoding_backend(
|
||||
backend_type=self.config.encoding_backend,
|
||||
dry_run=self.config.dry_run,
|
||||
logger=self.job_log,
|
||||
)
|
||||
return self._encoding_backend
|
||||
|
||||
def _get_packaging_service(self):
|
||||
"""Lazy-load the packaging service."""
|
||||
if self._packaging_service is None:
|
||||
from backend.services.packaging_service import PackagingService
|
||||
self._packaging_service = PackagingService(
|
||||
cdg_styles=self.config.cdg_styles,
|
||||
dry_run=self.config.dry_run,
|
||||
non_interactive=self.config.non_interactive,
|
||||
logger=self.job_log,
|
||||
)
|
||||
return self._packaging_service
|
||||
|
||||
def _get_youtube_service(self):
|
||||
"""Lazy-load the YouTube upload service."""
|
||||
if self._youtube_service is None:
|
||||
from backend.services.youtube_upload_service import YouTubeUploadService
|
||||
self._youtube_service = YouTubeUploadService(
|
||||
credentials=self.config.youtube_credentials,
|
||||
non_interactive=self.config.non_interactive,
|
||||
server_side_mode=True,
|
||||
dry_run=self.config.dry_run,
|
||||
logger=self.job_log,
|
||||
)
|
||||
return self._youtube_service
|
||||
|
||||
def _get_discord_service(self):
|
||||
"""Lazy-load the Discord notification service."""
|
||||
if self._discord_service is None:
|
||||
from backend.services.discord_service import DiscordNotificationService
|
||||
self._discord_service = DiscordNotificationService(
|
||||
webhook_url=self.config.discord_webhook_url,
|
||||
dry_run=self.config.dry_run,
|
||||
logger=self.job_log,
|
||||
)
|
||||
return self._discord_service
|
||||
|
||||
def _update_progress(self, status: JobStatus, progress: int, message: str):
|
||||
"""Update job progress if job_manager is available."""
|
||||
if self.job_manager:
|
||||
self.job_manager.transition_to_state(
|
||||
job_id=self.config.job_id,
|
||||
new_status=status,
|
||||
progress=progress,
|
||||
message=message
|
||||
)
|
||||
|
||||
async def run(self) -> OrchestratorResult:
|
||||
"""
|
||||
Run the full video generation pipeline.
|
||||
|
||||
Returns:
|
||||
OrchestratorResult with generated files and metadata
|
||||
"""
|
||||
start_time = time.time()
|
||||
|
||||
try:
|
||||
with job_span("orchestrator", self.config.job_id) as span:
|
||||
self.job_log.info(f"Starting orchestrated video generation for {self.config.artist} - {self.config.title}")
|
||||
|
||||
# Stage 1: Packaging (CDG/TXT) - runs BEFORE encoding
|
||||
# This depends on LRC file, not on encoded videos
|
||||
if self.config.enable_cdg or self.config.enable_txt:
|
||||
await self._run_packaging()
|
||||
|
||||
# Stage 2: Encoding
|
||||
await self._run_encoding()
|
||||
|
||||
# Stage 3: Organization (brand code)
|
||||
await self._run_organization()
|
||||
|
||||
# Stage 4: Distribution (YouTube, Dropbox, GDrive)
|
||||
await self._run_distribution()
|
||||
|
||||
# Stage 5: Notifications (Discord)
|
||||
await self._run_notifications()
|
||||
|
||||
self.result.success = True
|
||||
self.result.total_time_seconds = time.time() - start_time
|
||||
|
||||
self.job_log.info(f"Orchestrated video generation complete in {self.result.total_time_seconds:.1f}s")
|
||||
|
||||
except Exception as e:
|
||||
self.result.success = False
|
||||
self.result.error_message = str(e)
|
||||
self.result.total_time_seconds = time.time() - start_time
|
||||
self.job_log.error(f"Orchestrated video generation failed: {e}")
|
||||
logger.error(f"[job:{self.config.job_id}] Orchestration failed: {e}")
|
||||
|
||||
return self.result
|
||||
|
||||
async def _run_packaging(self):
|
||||
"""Run the packaging stage (CDG/TXT generation)."""
|
||||
self.job_log.info("Starting packaging stage (CDG/TXT)")
|
||||
|
||||
if not self.config.lrc_file_path or not os.path.isfile(self.config.lrc_file_path):
|
||||
self.job_log.warning("No LRC file available, skipping CDG/TXT packaging")
|
||||
return
|
||||
|
||||
base_name = f"{self.config.artist} - {self.config.title}"
|
||||
packaging_service = self._get_packaging_service()
|
||||
|
||||
# Generate CDG package
|
||||
if self.config.enable_cdg:
|
||||
self.job_log.info("Generating CDG package")
|
||||
try:
|
||||
cdg_zip_path = os.path.join(
|
||||
self.config.output_dir,
|
||||
f"{base_name} (Final Karaoke CDG).zip"
|
||||
)
|
||||
mp3_path = os.path.join(
|
||||
self.config.output_dir,
|
||||
f"{base_name} (Karaoke).mp3"
|
||||
)
|
||||
cdg_path = os.path.join(
|
||||
self.config.output_dir,
|
||||
f"{base_name} (Karaoke).cdg"
|
||||
)
|
||||
|
||||
zip_file, mp3_file, cdg_file = packaging_service.create_cdg_package(
|
||||
lrc_file=self.config.lrc_file_path,
|
||||
audio_file=self.config.instrumental_audio_path,
|
||||
output_zip_path=cdg_zip_path,
|
||||
artist=self.config.artist,
|
||||
title=self.config.title,
|
||||
output_mp3_path=mp3_path,
|
||||
output_cdg_path=cdg_path,
|
||||
)
|
||||
|
||||
self.result.final_karaoke_cdg_zip = zip_file
|
||||
self.job_log.info(f"CDG package created: {zip_file}")
|
||||
|
||||
except Exception as e:
|
||||
self.job_log.error(f"CDG generation failed: {e}")
|
||||
# Don't fail the pipeline, CDG is optional
|
||||
|
||||
# Generate TXT package
|
||||
if self.config.enable_txt:
|
||||
self.job_log.info("Generating TXT package")
|
||||
try:
|
||||
# TXT package needs MP3 file (from CDG generation or create it)
|
||||
mp3_path = os.path.join(
|
||||
self.config.output_dir,
|
||||
f"{base_name} (Karaoke).mp3"
|
||||
)
|
||||
|
||||
if not os.path.isfile(mp3_path):
|
||||
self.job_log.info("MP3 not found, CDG must be enabled first for TXT")
|
||||
return
|
||||
|
||||
txt_zip_path = os.path.join(
|
||||
self.config.output_dir,
|
||||
f"{base_name} (Final Karaoke TXT).zip"
|
||||
)
|
||||
|
||||
zip_file, txt_file = packaging_service.create_txt_package(
|
||||
lrc_file=self.config.lrc_file_path,
|
||||
mp3_file=mp3_path,
|
||||
output_zip_path=txt_zip_path,
|
||||
)
|
||||
|
||||
self.result.final_karaoke_txt_zip = zip_file
|
||||
self.job_log.info(f"TXT package created: {zip_file}")
|
||||
|
||||
except Exception as e:
|
||||
self.job_log.error(f"TXT generation failed: {e}")
|
||||
# Don't fail the pipeline, TXT is optional
|
||||
|
||||
async def _run_encoding(self):
|
||||
"""Run the encoding stage."""
|
||||
self.job_log.info("Starting encoding stage")
|
||||
self._update_progress(JobStatus.ENCODING, 75, "Encoding videos")
|
||||
|
||||
encoding_backend = self._get_encoding_backend()
|
||||
self.job_log.info(f"Using encoding backend: {encoding_backend.name}")
|
||||
|
||||
# Build encoding input
|
||||
from backend.services.encoding_interface import EncodingInput
|
||||
from backend.config import settings
|
||||
|
||||
# For GCE encoding, we need to provide GCS paths
|
||||
gcs_bucket = settings.gcs_bucket_name
|
||||
input_gcs_path = f"gs://{gcs_bucket}/jobs/{self.config.job_id}/"
|
||||
output_gcs_path = f"gs://{gcs_bucket}/jobs/{self.config.job_id}/finals/"
|
||||
|
||||
encoding_input = EncodingInput(
|
||||
title_video_path=self.config.title_video_path,
|
||||
karaoke_video_path=self.config.karaoke_video_path,
|
||||
instrumental_audio_path=self.config.instrumental_audio_path,
|
||||
end_video_path=self.config.end_video_path,
|
||||
artist=self.config.artist,
|
||||
title=self.config.title,
|
||||
brand_code=self.config.keep_brand_code,
|
||||
output_dir=self.config.output_dir,
|
||||
options={
|
||||
"input_gcs_path": input_gcs_path,
|
||||
"output_gcs_path": output_gcs_path,
|
||||
},
|
||||
)
|
||||
|
||||
# Run encoding
|
||||
with job_span("encoding", self.config.job_id) as span:
|
||||
add_span_event("encoding_started", {"backend": encoding_backend.name})
|
||||
|
||||
output = await encoding_backend.encode(encoding_input)
|
||||
|
||||
add_span_event("encoding_completed", {
|
||||
"success": output.success,
|
||||
"duration": output.encoding_time_seconds or 0
|
||||
})
|
||||
|
||||
if not output.success:
|
||||
raise Exception(f"Encoding failed: {output.error_message}")
|
||||
|
||||
# Store results
|
||||
self.result.final_video = output.lossless_4k_mp4_path
|
||||
self.result.final_video_mkv = output.lossless_mkv_path
|
||||
self.result.final_video_lossy = output.lossy_4k_mp4_path
|
||||
self.result.final_video_720p = output.lossy_720p_mp4_path
|
||||
self.result.encoding_time_seconds = output.encoding_time_seconds
|
||||
|
||||
self.job_log.info(f"Encoding complete ({encoding_backend.name}) in {output.encoding_time_seconds:.1f}s")
|
||||
|
||||
async def _run_organization(self):
|
||||
"""Run the organization stage (brand code generation)."""
|
||||
self.job_log.info("Starting organization stage")
|
||||
|
||||
# Use existing brand code if provided
|
||||
if self.config.keep_brand_code:
|
||||
self.result.brand_code = self.config.keep_brand_code
|
||||
self.job_log.info(f"Using preserved brand code: {self.result.brand_code}")
|
||||
return
|
||||
|
||||
# Generate brand code from Dropbox if configured
|
||||
if self.config.dropbox_path and self.config.brand_prefix:
|
||||
try:
|
||||
from backend.services.dropbox_service import get_dropbox_service
|
||||
|
||||
dropbox = get_dropbox_service()
|
||||
if dropbox.is_configured:
|
||||
brand_code = dropbox.get_next_brand_code(
|
||||
self.config.dropbox_path,
|
||||
self.config.brand_prefix
|
||||
)
|
||||
self.result.brand_code = brand_code
|
||||
self.job_log.info(f"Generated brand code: {brand_code}")
|
||||
else:
|
||||
self.job_log.warning("Dropbox not configured, skipping brand code generation")
|
||||
|
||||
except Exception as e:
|
||||
self.job_log.error(f"Brand code generation failed: {e}")
|
||||
# Don't fail - brand code is optional
|
||||
|
||||
async def _run_distribution(self):
|
||||
"""Run the distribution stage (YouTube, Dropbox, GDrive uploads)."""
|
||||
self.job_log.info("Starting distribution stage")
|
||||
self._update_progress(JobStatus.PACKAGING, 90, "Uploading files")
|
||||
|
||||
# YouTube upload
|
||||
if self.config.enable_youtube_upload and self.config.youtube_credentials:
|
||||
await self._upload_to_youtube()
|
||||
|
||||
# Dropbox upload
|
||||
if self.config.dropbox_path and self.config.brand_prefix:
|
||||
await self._upload_to_dropbox()
|
||||
|
||||
# Google Drive upload
|
||||
if self.config.gdrive_folder_id:
|
||||
await self._upload_to_gdrive()
|
||||
|
||||
async def _upload_to_youtube(self):
|
||||
"""Upload video to YouTube."""
|
||||
self.job_log.info("Uploading to YouTube")
|
||||
|
||||
# Find the best video file to upload (prefer MKV for FLAC audio, then lossless MP4)
|
||||
video_to_upload = None
|
||||
if self.result.final_video_mkv and os.path.isfile(self.result.final_video_mkv):
|
||||
video_to_upload = self.result.final_video_mkv
|
||||
elif self.result.final_video and os.path.isfile(self.result.final_video):
|
||||
video_to_upload = self.result.final_video
|
||||
elif self.result.final_video_lossy and os.path.isfile(self.result.final_video_lossy):
|
||||
video_to_upload = self.result.final_video_lossy
|
||||
|
||||
if not video_to_upload:
|
||||
self.job_log.warning("No video file available for YouTube upload")
|
||||
return
|
||||
|
||||
try:
|
||||
youtube_service = self._get_youtube_service()
|
||||
|
||||
# Build video title
|
||||
title = f"{self.config.artist} - {self.config.title} (Karaoke)"
|
||||
|
||||
# Build description
|
||||
description = self.config.youtube_description_template or ""
|
||||
if self.result.brand_code:
|
||||
description = f"{description}\n\nBrand Code: {self.result.brand_code}".strip()
|
||||
|
||||
# Upload
|
||||
video_id, video_url = youtube_service.upload_video(
|
||||
video_path=video_to_upload,
|
||||
title=title,
|
||||
description=description,
|
||||
thumbnail_path=self.config.title_jpg_path,
|
||||
tags=["karaoke", self.config.artist, self.config.title],
|
||||
replace_existing=True, # Server-side always replaces
|
||||
)
|
||||
|
||||
if video_url:
|
||||
self.result.youtube_url = video_url
|
||||
self.job_log.info(f"Uploaded to YouTube: {video_url}")
|
||||
else:
|
||||
self.job_log.warning("YouTube upload did not return a URL")
|
||||
|
||||
except Exception as e:
|
||||
self.job_log.error(f"YouTube upload failed: {e}")
|
||||
# Don't fail the pipeline - YouTube is optional
|
||||
|
||||
async def _upload_to_dropbox(self):
|
||||
"""Upload files to Dropbox."""
|
||||
self.job_log.info("Uploading to Dropbox")
|
||||
|
||||
try:
|
||||
from backend.services.dropbox_service import get_dropbox_service
|
||||
|
||||
dropbox = get_dropbox_service()
|
||||
if not dropbox.is_configured:
|
||||
self.job_log.warning("Dropbox not configured, skipping upload")
|
||||
return
|
||||
|
||||
base_name = f"{self.config.artist} - {self.config.title}"
|
||||
folder_name = f"{self.result.brand_code or 'TRACK-0000'} - {base_name}"
|
||||
remote_folder = f"{self.config.dropbox_path}/{folder_name}"
|
||||
|
||||
# Upload entire output directory
|
||||
dropbox.upload_folder(self.config.output_dir, remote_folder)
|
||||
|
||||
# Create sharing link
|
||||
try:
|
||||
sharing_link = dropbox.create_shared_link(remote_folder)
|
||||
self.result.dropbox_link = sharing_link
|
||||
self.job_log.info(f"Dropbox sharing link: {sharing_link}")
|
||||
except Exception as e:
|
||||
self.job_log.warning(f"Failed to create Dropbox sharing link: {e}")
|
||||
|
||||
self.job_log.info("Dropbox upload complete")
|
||||
|
||||
except Exception as e:
|
||||
self.job_log.error(f"Dropbox upload failed: {e}")
|
||||
# Don't fail the pipeline - Dropbox is optional
|
||||
|
||||
async def _upload_to_gdrive(self):
|
||||
"""Upload files to Google Drive."""
|
||||
self.job_log.info("Uploading to Google Drive")
|
||||
|
||||
try:
|
||||
from backend.services.gdrive_service import get_gdrive_service
|
||||
|
||||
gdrive = get_gdrive_service()
|
||||
if not gdrive.is_configured:
|
||||
self.job_log.warning("Google Drive not configured, skipping upload")
|
||||
return
|
||||
|
||||
base_name = f"{self.config.artist} - {self.config.title}"
|
||||
brand_code = self.result.brand_code or f"{self.config.brand_prefix or 'TRACK'}-0000"
|
||||
|
||||
# Map result files to expected keys
|
||||
output_files = {
|
||||
'final_karaoke_lossy_mp4': self.result.final_video_lossy,
|
||||
'final_karaoke_lossy_720p_mp4': self.result.final_video_720p,
|
||||
'final_karaoke_cdg_zip': self.result.final_karaoke_cdg_zip,
|
||||
}
|
||||
|
||||
uploaded = gdrive.upload_to_public_share(
|
||||
root_folder_id=self.config.gdrive_folder_id,
|
||||
brand_code=brand_code,
|
||||
base_name=base_name,
|
||||
output_files=output_files,
|
||||
)
|
||||
|
||||
self.result.gdrive_files = uploaded
|
||||
self.job_log.info(f"Google Drive upload complete: {len(uploaded)} files")
|
||||
|
||||
except Exception as e:
|
||||
self.job_log.error(f"Google Drive upload failed: {e}")
|
||||
# Don't fail the pipeline - GDrive is optional
|
||||
|
||||
async def _run_notifications(self):
|
||||
"""Run the notifications stage (Discord)."""
|
||||
self.job_log.info("Starting notifications stage")
|
||||
|
||||
if not self.config.discord_webhook_url:
|
||||
self.job_log.debug("No Discord webhook configured, skipping notification")
|
||||
return
|
||||
|
||||
if not self.result.youtube_url:
|
||||
self.job_log.info("No YouTube URL available, skipping Discord notification")
|
||||
return
|
||||
|
||||
try:
|
||||
discord_service = self._get_discord_service()
|
||||
discord_service.post_video_notification(self.result.youtube_url)
|
||||
self.job_log.info("Discord notification sent")
|
||||
except Exception as e:
|
||||
self.job_log.error(f"Discord notification failed: {e}")
|
||||
# Don't fail the pipeline - notifications are optional
|
||||
|
||||
|
||||
def create_orchestrator_config_from_job(
|
||||
job,
|
||||
temp_dir: str,
|
||||
youtube_credentials: Optional[Dict[str, Any]] = None,
|
||||
cdg_styles: Optional[Dict[str, Any]] = None,
|
||||
) -> OrchestratorConfig:
|
||||
"""
|
||||
Create an OrchestratorConfig from a job object.
|
||||
|
||||
This is a helper function to bridge the existing job structure
|
||||
with the new orchestrator configuration.
|
||||
|
||||
Args:
|
||||
job: Job object from Firestore
|
||||
temp_dir: Temporary directory with downloaded files
|
||||
youtube_credentials: Pre-loaded YouTube credentials
|
||||
cdg_styles: CDG style configuration
|
||||
|
||||
Returns:
|
||||
OrchestratorConfig for the orchestrator
|
||||
"""
|
||||
base_name = f"{job.artist} - {job.title}"
|
||||
|
||||
# Determine instrumental file path
|
||||
instrumental_selection = job.state_data.get('instrumental_selection', 'clean')
|
||||
existing_instrumental = getattr(job, 'existing_instrumental_gcs_path', None)
|
||||
|
||||
if existing_instrumental:
|
||||
ext = Path(existing_instrumental).suffix.lower()
|
||||
instrumental_path = os.path.join(temp_dir, f"{base_name} (Instrumental User){ext}")
|
||||
else:
|
||||
instrumental_suffix = "Clean" if instrumental_selection == 'clean' else "Backing"
|
||||
instrumental_path = os.path.join(temp_dir, f"{base_name} (Instrumental {instrumental_suffix}).flac")
|
||||
|
||||
return OrchestratorConfig(
|
||||
job_id=job.job_id,
|
||||
artist=job.artist,
|
||||
title=job.title,
|
||||
|
||||
# Input files
|
||||
title_video_path=os.path.join(temp_dir, f"{base_name} (Title).mov"),
|
||||
karaoke_video_path=os.path.join(temp_dir, f"{base_name} (With Vocals).mov"),
|
||||
instrumental_audio_path=instrumental_path,
|
||||
end_video_path=os.path.join(temp_dir, f"{base_name} (End).mov"),
|
||||
lrc_file_path=os.path.join(temp_dir, f"{base_name} (Karaoke).lrc"),
|
||||
title_jpg_path=os.path.join(temp_dir, f"{base_name} (Title).jpg"),
|
||||
|
||||
# Output directory
|
||||
output_dir=temp_dir,
|
||||
|
||||
# Feature flags
|
||||
enable_cdg=getattr(job, 'enable_cdg', False),
|
||||
enable_txt=getattr(job, 'enable_txt', False),
|
||||
enable_youtube_upload=getattr(job, 'enable_youtube_upload', False),
|
||||
|
||||
# Service configurations
|
||||
brand_prefix=getattr(job, 'brand_prefix', None),
|
||||
discord_webhook_url=getattr(job, 'discord_webhook_url', None),
|
||||
youtube_credentials=youtube_credentials,
|
||||
youtube_description_template=getattr(job, 'youtube_description_template', None),
|
||||
cdg_styles=cdg_styles,
|
||||
|
||||
# Dropbox/GDrive
|
||||
dropbox_path=getattr(job, 'dropbox_path', None),
|
||||
gdrive_folder_id=getattr(job, 'gdrive_folder_id', None),
|
||||
|
||||
# Keep existing brand code
|
||||
keep_brand_code=getattr(job, 'keep_brand_code', None),
|
||||
|
||||
# Encoding backend - auto selects GCE if available
|
||||
encoding_backend="auto",
|
||||
|
||||
# Server-side defaults
|
||||
dry_run=False,
|
||||
non_interactive=True,
|
||||
)
|
||||
-104
@@ -383,110 +383,6 @@ Authorization: Bearer ADMIN_TOKEN
|
||||
|
||||
Admins can delete any job. Regular users can only delete their own jobs.
|
||||
|
||||
### Audio Search Management (Admin)
|
||||
|
||||
#### List Audio Searches
|
||||
|
||||
```http
|
||||
GET /api/admin/audio-searches
|
||||
GET /api/admin/audio-searches?limit=50&status_filter=awaiting_audio_selection
|
||||
Authorization: Bearer ADMIN_TOKEN
|
||||
```
|
||||
|
||||
Returns jobs with cached audio search results. Useful for:
|
||||
- Monitoring search activity
|
||||
- Identifying stale cached results (YouTube-only when lossless should be available)
|
||||
- Clearing cache for specific jobs
|
||||
|
||||
Response:
|
||||
```json
|
||||
{
|
||||
"jobs": [
|
||||
{
|
||||
"job_id": "abc123",
|
||||
"status": "awaiting_audio_selection",
|
||||
"user_email": "user@example.com",
|
||||
"audio_search_artist": "Artist Name",
|
||||
"audio_search_title": "Song Title",
|
||||
"created_at": "2026-01-03T12:00:00Z",
|
||||
"results_count": 5,
|
||||
"has_lossless": false,
|
||||
"providers": ["YouTube"],
|
||||
"results_summary": [...]
|
||||
}
|
||||
],
|
||||
"total": 1
|
||||
}
|
||||
```
|
||||
|
||||
#### Clear Audio Search Cache
|
||||
|
||||
```http
|
||||
POST /api/admin/audio-searches/{job_id}/clear-cache
|
||||
Authorization: Bearer ADMIN_TOKEN
|
||||
```
|
||||
|
||||
Clears cached search results and resets job to `pending` status, allowing a new search.
|
||||
|
||||
Use when:
|
||||
- Cached results are stale (e.g., flacfetch was updated with new providers)
|
||||
- User wants to search again
|
||||
- Results appear incomplete (YouTube-only when lossless should exist)
|
||||
|
||||
Response:
|
||||
```json
|
||||
{
|
||||
"status": "success",
|
||||
"job_id": "abc123",
|
||||
"message": "Cleared 5 cached search results. Job reset to pending. Flacfetch cache also cleared.",
|
||||
"previous_status": "awaiting_audio_selection",
|
||||
"new_status": "pending",
|
||||
"results_cleared": 5,
|
||||
"flacfetch_cache_cleared": true,
|
||||
"flacfetch_error": null
|
||||
}
|
||||
```
|
||||
|
||||
Note: This endpoint now also clears the flacfetch GCS cache for the artist/title combination, ensuring the next search hits trackers fresh.
|
||||
|
||||
#### Clear All Flacfetch Cache
|
||||
|
||||
```http
|
||||
DELETE /api/admin/cache
|
||||
Authorization: Bearer ADMIN_TOKEN
|
||||
```
|
||||
|
||||
Clears the entire flacfetch search cache (GCS-backed). Use when flacfetch has been updated and you want all subsequent searches to use fresh tracker results.
|
||||
|
||||
Response:
|
||||
```json
|
||||
{
|
||||
"status": "success",
|
||||
"message": "Cleared 15 cache entries from flacfetch.",
|
||||
"deleted_count": 15
|
||||
}
|
||||
```
|
||||
|
||||
#### Get Flacfetch Cache Stats
|
||||
|
||||
```http
|
||||
GET /api/admin/cache/stats
|
||||
Authorization: Bearer ADMIN_TOKEN
|
||||
```
|
||||
|
||||
Returns statistics about the flacfetch search cache.
|
||||
|
||||
Response:
|
||||
```json
|
||||
{
|
||||
"count": 42,
|
||||
"total_size_bytes": 128000,
|
||||
"oldest_entry": "2025-12-15T10:30:00Z",
|
||||
"newest_entry": "2026-01-03T15:45:00Z",
|
||||
"configured": true
|
||||
}
|
||||
```
|
||||
|
||||
## Rate Limits
|
||||
|
||||
No rate limits currently implemented.
|
||||
|
||||
@@ -169,43 +169,6 @@ karaoke-gen shares a GCP project (`nomadkaraoke`) with karaoke-decide, but uses
|
||||
|
||||
**Note**: The `users` collection in the same Firestore instance belongs to karaoke-decide (different schema: user_id, is_guest, quiz_* fields). Don't use it for karaoke-gen.
|
||||
|
||||
## Video Worker Orchestrator
|
||||
|
||||
The Video Worker uses an orchestrator pattern to ensure all features work regardless of encoding backend (local Cloud Run or GCE VM).
|
||||
|
||||
```text
|
||||
┌─────────────────────────────────────────────────────────────┐
|
||||
│ VideoWorkerOrchestrator │
|
||||
│ │
|
||||
│ Stage 1: PACKAGING Stage 2: ENCODING │
|
||||
│ ├─ CDG file generation ├─ Local (FFmpeg) │
|
||||
│ └─ TXT lyrics export └─ GCE (remote VM) │
|
||||
│ │
|
||||
│ Stage 3: ORGANIZATION Stage 4: DISTRIBUTION │
|
||||
│ ├─ Final path setup ├─ Dropbox upload │
|
||||
│ └─ File organization ├─ Google Drive upload │
|
||||
│ └─ YouTube upload │
|
||||
│ │
|
||||
│ Stage 5: NOTIFICATIONS │
|
||||
│ └─ Discord webhooks │
|
||||
└─────────────────────────────────────────────────────────────┘
|
||||
```
|
||||
|
||||
**Key Design Decision**: The orchestrator coordinates stages rather than embedding logic in monolithic functions. This:
|
||||
- Ensures all features work with both local and GCE encoding
|
||||
- Makes each stage independently testable (139 new tests)
|
||||
- Allows feature flag rollback via `USE_NEW_ORCHESTRATOR` env var
|
||||
|
||||
**Service Modules**:
|
||||
| Module | Purpose |
|
||||
|--------|---------|
|
||||
| `youtube_upload_service.py` | YouTube OAuth flow and upload |
|
||||
| `discord_service.py` | Discord webhook notifications |
|
||||
| `packaging_service.py` | CDG/TXT package generation |
|
||||
| `local_encoding_service.py` | FFmpeg encoding (Cloud Run) |
|
||||
| `encoding_interface.py` | Abstract interface + GCE backend |
|
||||
| `video_worker_orchestrator.py` | Stage coordination |
|
||||
|
||||
## Tech Stack
|
||||
|
||||
- **Backend**: FastAPI, Python 3.12, Cloud Run
|
||||
|
||||
@@ -14,20 +14,18 @@
|
||||
|
||||
## Test Requirements
|
||||
|
||||
### 1. New User Registration (testmail.app)
|
||||
### 1. New User Registration (MailSlurp)
|
||||
|
||||
- [ ] Navigate to landing page (`gen.nomadkaraoke.com`)
|
||||
- [ ] Click "Sign In" or "Get Started" button
|
||||
- [ ] Generate a unique test email address (testmail.app namespace)
|
||||
- [ ] Create a NEW MailSlurp inbox (fresh email address for each test run)
|
||||
- [ ] Enter the test email in the beta enrollment form
|
||||
- [ ] Submit the form
|
||||
- [ ] Wait for welcome/magic-link email via testmail.app API
|
||||
- [ ] Wait for welcome/magic-link email in MailSlurp inbox
|
||||
- [ ] Extract the magic link from the email
|
||||
- [ ] Click/navigate to the magic link to authenticate
|
||||
- [ ] Verify user is logged in and has credits
|
||||
|
||||
**Note**: When `use_test_token=true` workflow input is set, steps 1-9 above are skipped and the test uses a pre-authenticated token instead.
|
||||
|
||||
### 2. Create Karaoke Job (UI Only)
|
||||
|
||||
- [ ] Navigate to the karaoke creation page
|
||||
@@ -120,10 +118,8 @@
|
||||
### Environment Variables
|
||||
|
||||
```bash
|
||||
TESTMAIL_NAMESPACE=xxx # Required - for email testing (testmail.app)
|
||||
TESTMAIL_API_KEY=xxx # Required - for email testing (testmail.app)
|
||||
E2E_TEST_TOKEN=xxx # Optional - skip signup, use existing account
|
||||
# When use_test_token=true, skips email signup and uses pre-authenticated token
|
||||
MAILSLURP_API_KEY=xxx # Required - for email testing
|
||||
# No pre-auth tokens - test handles full signup
|
||||
```
|
||||
|
||||
### Test Song
|
||||
@@ -164,8 +160,7 @@ Log detailed progress at each step with timestamps for debugging.
|
||||
|
||||
### Secrets Required
|
||||
|
||||
- `TESTMAIL_API_KEY` - For email testing (testmail.app)
|
||||
- `TESTMAIL_NAMESPACE` - testmail.app namespace for unique email addresses
|
||||
- `MAILSLURP_API_KEY` - For email testing
|
||||
|
||||
### Workflow Steps
|
||||
|
||||
|
||||
@@ -633,40 +633,6 @@ See `docs/archive/2025-12-28-self-hosted-github-runner.md` for full details.
|
||||
- Pre-job checks provide fail-fast behavior with clear error messages
|
||||
- `docker builder prune -af` catches buildkit cache that `system prune` misses
|
||||
|
||||
### Fonts in Docker for Video Rendering
|
||||
|
||||
**Problem**: Video rendering showed replacement characters (squares with question marks) for special Unicode symbols like `♪` in intro/instrumental screens.
|
||||
|
||||
**Root cause**: The `python:3.11-slim` Docker base image has NO fonts installed. FFmpeg/libass falls back to whatever it can find, which often lacks support for musical symbols, emoji, or non-Latin scripts.
|
||||
|
||||
**Symptoms**:
|
||||
- `♪ INTRO (19 seconds) ♪` displayed as `□ INTRO (19 seconds) □`
|
||||
- Unicode characters render as tofu (replacement boxes)
|
||||
- Only appears in cloud-generated videos, not local dev (which uses system fonts)
|
||||
|
||||
**Solution**: Install comprehensive Noto fonts in the Docker base image:
|
||||
```dockerfile
|
||||
RUN apt-get update && apt-get install -y \
|
||||
fonts-noto-core \
|
||||
fonts-noto-cjk \
|
||||
fonts-noto-extra \
|
||||
fonts-noto-color-emoji \
|
||||
fontconfig \
|
||||
&& fc-cache -f \
|
||||
&& rm -rf /var/lib/apt/lists/*
|
||||
```
|
||||
|
||||
**Package breakdown**:
|
||||
- `fonts-noto-core` (~10MB): Latin, Greek, Cyrillic, musical symbols
|
||||
- `fonts-noto-cjk` (~150MB): Chinese, Japanese, Korean
|
||||
- `fonts-noto-extra` (~50MB): Arabic, Hebrew, Thai, Hindi, other scripts
|
||||
- `fonts-noto-color-emoji` (~10MB): Color emoji (limited libass support)
|
||||
- `fontconfig`: Font configuration system (required for `fc-cache`)
|
||||
|
||||
**Note on color emoji**: FFmpeg's libass subtitle renderer doesn't fully support color emoji fonts - it renders text as vectors and can't display color bitmap glyphs. Color emoji appears as monochrome outlines or may not render at all in ASS subtitles.
|
||||
|
||||
**Lesson**: When building Docker images for video/subtitle rendering, install fonts explicitly. The default slim images have nothing, and font fallback behavior differs from local development machines.
|
||||
|
||||
## Performance Observations
|
||||
|
||||
| Operation | Duration |
|
||||
@@ -823,37 +789,6 @@ result = job.state_data['search_results'][selection_index]
|
||||
|
||||
**Lesson**: Design for stateless request handling. Any state that must survive between requests must be persisted externally.
|
||||
|
||||
### Firestore Cache Has No Automatic Invalidation
|
||||
|
||||
**Problem**: Audio search results are cached in Firestore (`job.state_data.audio_search_results`) with no expiration or invalidation. When a library like flacfetch is updated with new providers or bug fixes, existing cached results remain stale forever.
|
||||
|
||||
**Symptoms**:
|
||||
- Job shows only 5 YouTube results when CLI shows 59 results including lossless sources
|
||||
- User cannot get better search results without admin intervention
|
||||
- No "re-search" option in the UI
|
||||
|
||||
**Example**: Job `6ddbbef1` was searched when flacfetch had a bug returning only YouTube results. After flacfetch was fixed, the job still showed the old 5-result cache.
|
||||
|
||||
**Solution**: Added admin endpoints and UI to manage cached searches:
|
||||
- `GET /api/admin/audio-searches` - List jobs with cached results
|
||||
- `POST /api/admin/audio-searches/{job_id}/clear-cache` - Clear both Firestore and flacfetch GCS cache
|
||||
- `DELETE /api/admin/cache` - Clear entire flacfetch cache
|
||||
- `GET /api/admin/cache/stats` - View cache statistics
|
||||
- Admin UI at `/admin/searches` shows cache stats and "Clear All Cache" button
|
||||
|
||||
**Key insight**: There are TWO caches involved:
|
||||
1. **Firestore** (`job.state_data.audio_search_results`) - stores results for user selection UI
|
||||
2. **Flacfetch GCS** - caches tracker search responses (30-day TTL)
|
||||
|
||||
Both must be cleared for a truly fresh search. The clear-cache endpoint now clears both automatically.
|
||||
|
||||
**Future consideration**: Add automatic cache invalidation based on:
|
||||
- Time (TTL)
|
||||
- flacfetch version change
|
||||
- User-initiated "re-search" button
|
||||
|
||||
**Lesson**: When caching external API results, consider how/when the cache should be invalidated. Permanent caches need admin tooling to manage stale data. When multiple layers cache the same data, clearing one layer isn't enough - you need to clear all of them.
|
||||
|
||||
### GCS Downloads Preserve Directory Structure
|
||||
|
||||
**Problem**: When downloading files from GCS, the directory structure is preserved. Code that uses non-recursive glob patterns won't find files in subdirectories.
|
||||
@@ -879,49 +814,6 @@ input_files = [f for f in input_files if "outputs" not in str(f)]
|
||||
|
||||
**Lesson**: When writing code that processes files downloaded from GCS, always use recursive glob patterns and verify the expected directory structure matches what's actually stored.
|
||||
|
||||
### GCS Blob Paths vs Full URIs
|
||||
|
||||
**Problem**: When uploading files to GCS and returning paths to other services, there's confusion between "blob paths" (just the path within a bucket) and "full GCS URIs" (gs://bucket/path).
|
||||
|
||||
**Symptoms**:
|
||||
- 404 errors when downloading files that were just uploaded
|
||||
- Paths look like: `bucket/gs://bucket/path/file.mp4` (doubled prefix)
|
||||
- Storage service prepends bucket name to what's already a full URI
|
||||
|
||||
**Example of the bug**:
|
||||
```python
|
||||
# GCE worker uploads to gs://bucket/jobs/id/encoded/
|
||||
# Then returns full URI in response: "gs://bucket/jobs/id/encoded/file.mp4"
|
||||
# Backend tries: storage.download_file("gs://bucket/jobs/id/encoded/file.mp4")
|
||||
# Storage service prepends bucket: "bucket/gs://bucket/jobs/id/encoded/file.mp4"
|
||||
# Result: 404 - No such object
|
||||
```
|
||||
|
||||
**Solution**: Return blob paths (path within bucket), not full GCS URIs:
|
||||
```python
|
||||
# output_gcs_path = "gs://bucket/jobs/id/encoded/"
|
||||
gcs_path = output_gcs_path.replace("gs://", "")
|
||||
parts = gcs_path.split("/", 1)
|
||||
bucket = parts[0]
|
||||
prefix = parts[1].rstrip("/") if len(parts) > 1 else ""
|
||||
|
||||
# Return blob paths like: "jobs/id/encoded/file.mp4"
|
||||
blob_path = f"{prefix}/{filename}" if prefix else filename
|
||||
```
|
||||
|
||||
**Related gotcha**: Double slashes in paths. When the prefix already ends with "/" and you concatenate with another "/":
|
||||
```python
|
||||
# BAD - creates "encoded//file.mp4"
|
||||
prefix = "jobs/id/encoded/" # Has trailing slash
|
||||
blob_name = f"{prefix}/{filename}" # Adds another slash
|
||||
|
||||
# GOOD - strip trailing slash first
|
||||
prefix = parts[1].rstrip("/")
|
||||
blob_name = f"{prefix}/{filename}" if prefix else filename
|
||||
```
|
||||
|
||||
**Lesson**: When building systems that pass GCS paths between services, establish a clear contract: either always use full URIs (`gs://bucket/path`) or always use blob paths (`path/within/bucket`). Document which format each API expects.
|
||||
|
||||
### Library Caches Need GCS Sync in Cloud Run
|
||||
|
||||
**Problem**: Libraries like LyricsTranscriber cache API responses to local directories (e.g., `~/lyrics-transcriber-cache`). In Cloud Run, each container is ephemeral - the cache is lost on every restart or scale-up, causing redundant API calls.
|
||||
@@ -1002,42 +894,6 @@ async def upload(auth_result: AuthResult = Depends(require_auth)):
|
||||
|
||||
See `docs/archive/2024-12-31-job-failure-investigation.md` for full details.
|
||||
|
||||
### Alternative Code Paths Must Implement All Features
|
||||
|
||||
**Problem**: Adding a GCE encoding option created a code path that completely bypassed the existing `KaraokeFinalise` class, which contained YouTube upload, Discord notifications, and CDG/TXT packaging. Jobs using GCE encoding silently missed these features.
|
||||
|
||||
```python
|
||||
# BAD - two divergent code paths
|
||||
if use_gce_encoding:
|
||||
await gce_encoding_worker.encode(job_id) # Returns, does nothing else
|
||||
return # YouTube upload, Discord, CDG never happen!
|
||||
else:
|
||||
finalise = KaraokeFinalise(...) # Has all the features
|
||||
await finalise.run()
|
||||
```
|
||||
|
||||
**Symptoms**:
|
||||
- Jobs complete successfully but YouTube URL is never set
|
||||
- No Discord notifications for GCE-encoded jobs
|
||||
- CDG/TXT files never generated for GCE jobs
|
||||
- No errors in logs (the code path simply doesn't include those features)
|
||||
|
||||
**Solution**: Create an orchestrator that runs all stages regardless of encoding backend:
|
||||
|
||||
```python
|
||||
# GOOD - unified pipeline with swappable encoding
|
||||
orchestrator = VideoWorkerOrchestrator(config)
|
||||
result = await orchestrator.run()
|
||||
# Orchestrator always runs: packaging → encoding → organization → distribution → notifications
|
||||
# The encoding step uses either local or GCE backend based on config
|
||||
```
|
||||
|
||||
**Key insight**: When adding an alternative implementation (like GCE encoding), audit ALL side effects of the original code path. The new path must replicate them or delegate to shared code.
|
||||
|
||||
**Architecture pattern**: The orchestrator pattern separates "what stages run" from "how each stage runs". Stages are always executed; only the implementation varies.
|
||||
|
||||
See `docs/archive/2026-01-04-video-worker-orchestrator-refactor.md` for full details.
|
||||
|
||||
## What We'd Do Differently
|
||||
|
||||
1. **Add Pydantic model fields test first** - Would have caught the silent field issue immediately
|
||||
|
||||
@@ -36,14 +36,6 @@
|
||||
|
||||
## Recent Changes
|
||||
|
||||
- **Video Worker Orchestrator** (2026-01-04): Major refactor to unify video generation pipeline. Created VideoWorkerOrchestrator that coordinates all stages (packaging, encoding, distribution, notifications) regardless of encoding backend (GCE or local). Fixes issue where GCE encoding path bypassed YouTube upload, Discord notifications, and CDG/TXT packaging. Feature flag `USE_NEW_ORCHESTRATOR` (default: true) enables rollback. 139 new tests across 6 new service modules. See [ARCHITECTURE.md](ARCHITECTURE.md#video-worker-orchestrator) and [archive/2026-01-04-video-worker-orchestrator-refactor.md](archive/2026-01-04-video-worker-orchestrator-refactor.md).
|
||||
|
||||
- **Full Unicode Font Support** (2026-01-03): Fixed rendering of musical symbols (♪) and added comprehensive international font support to Docker base image. Installed Noto fonts covering Latin, CJK (Chinese/Japanese/Korean), Arabic, Hebrew, Thai, and other scripts. Changed default karaoke font from Arial to Noto Sans. See [LESSONS-LEARNED.md](LESSONS-LEARNED.md#fonts-in-docker-for-video-rendering).
|
||||
|
||||
- **Flacfetch Cache API Integration** (2026-01-03): Enhanced audio search cache management to also clear flacfetch's GCS cache. When admins clear a job's cache, both Firestore (job state) and flacfetch (GCS search results) are cleared simultaneously. Added "Clear All Cache" button and cache stats display to admin UI. New endpoints: `DELETE /api/admin/cache`, `GET /api/admin/cache/stats`. See [API.md](API.md#audio-search-management-admin).
|
||||
|
||||
- **Audio Search Cache Management** (2026-01-03): Added admin UI at `/admin/searches` to view and manage cached audio search results. Admins can clear stale caches (e.g., when flacfetch is updated with new providers) and allow users to re-search. Fixes issue where jobs showed only YouTube results when lossless sources should be available. See [API.md](API.md#audio-search-management-admin) and [LESSONS-LEARNED.md](LESSONS-LEARNED.md#firestore-cache-has-no-automatic-invalidation).
|
||||
|
||||
- **User Database Separation** (2026-01-03): Migrated karaoke-gen users from shared `users` collection to dedicated `gen_users` collection. This separates karaoke-gen user data from karaoke-decide user data (both apps share the same GCP project/Firestore instance). Migration script in `scripts/migrate_users_to_gen_users.py`. See [ARCHITECTURE.md](ARCHITECTURE.md#firestore-collections) for collection details.
|
||||
|
||||
- **Admin Dashboard** (2026-01-03): Added comprehensive admin dashboard at `/admin` with: stats overview (users, jobs, credits), user management (search, sort, add credits, toggle role, enable/disable), job browser (filter by status/user, view details, delete), and beta program monitoring. Query-param based routing for static export compatibility. See [API.md](API.md#admin-endpoints) for backend endpoints.
|
||||
|
||||
@@ -1,101 +0,0 @@
|
||||
# Video Worker Orchestrator Refactor - 2026-01-04
|
||||
|
||||
## Summary
|
||||
|
||||
Major refactor to unify video generation pipeline with a new VideoWorkerOrchestrator. This resolves the GCE vs KaraokeFinalise code path divergence that caused features like YouTube upload, Discord notifications, and CDG/TXT packaging to be missing from the GCE encoding path.
|
||||
|
||||
## Problem
|
||||
|
||||
Job 5b78ad46 completed successfully via GCE encoding but had no YouTube URL. Investigation revealed:
|
||||
|
||||
1. The GCE encoding path was a completely separate code branch
|
||||
2. When `USE_GCE_ENCODING=true`, the worker triggered GCE encoding and returned immediately
|
||||
3. This bypassed the entire `KaraokeFinalise` class which contained:
|
||||
- YouTube upload logic
|
||||
- Discord webhook notifications
|
||||
- CDG/TXT package generation
|
||||
- Dropbox/Google Drive uploads
|
||||
|
||||
The root cause wasn't a bug in any individual feature - it was an architectural issue where alternative implementations didn't share common post-processing stages.
|
||||
|
||||
## Solution
|
||||
|
||||
Created a stage-based orchestrator pattern:
|
||||
|
||||
```
|
||||
VideoWorkerOrchestrator
|
||||
├── Stage 1: Packaging (CDG, TXT)
|
||||
├── Stage 2: Encoding (Local or GCE - swappable)
|
||||
├── Stage 3: Organization (file paths, structure)
|
||||
├── Stage 4: Distribution (Dropbox, GDrive, YouTube)
|
||||
└── Stage 5: Notifications (Discord)
|
||||
```
|
||||
|
||||
The orchestrator always runs all stages. Only the encoding implementation varies based on backend selection.
|
||||
|
||||
## Changes
|
||||
|
||||
### New Service Modules
|
||||
|
||||
| File | Lines | Purpose |
|
||||
|------|-------|---------|
|
||||
| `youtube_upload_service.py` | ~445 | YouTube OAuth and upload |
|
||||
| `discord_service.py` | ~172 | Discord webhook notifications |
|
||||
| `packaging_service.py` | ~287 | CDG/TXT package generation |
|
||||
| `local_encoding_service.py` | ~590 | FFmpeg encoding on Cloud Run |
|
||||
| `encoding_interface.py` | ~418 | Abstract interface + GCE backend |
|
||||
| `video_worker_orchestrator.py` | ~628 | Stage coordination |
|
||||
|
||||
### Test Coverage
|
||||
|
||||
| Test File | Tests | Coverage |
|
||||
|-----------|-------|----------|
|
||||
| `test_youtube_upload_service.py` | 25 | OAuth, upload, error handling |
|
||||
| `test_discord_service.py` | 18 | Webhook formatting, delivery |
|
||||
| `test_packaging_service.py` | 22 | CDG/TXT generation |
|
||||
| `test_local_encoding_service.py` | 24 | FFmpeg encoding variants |
|
||||
| `test_encoding_interface.py` | 23 | Interface + GCE backend |
|
||||
| `test_video_worker_orchestrator.py` | 27 | Full pipeline, stages |
|
||||
| **Total** | **139** | |
|
||||
|
||||
### video_worker.py Updates
|
||||
|
||||
- Added `USE_NEW_ORCHESTRATOR` feature flag (default: `true`)
|
||||
- `generate_video()` routes to orchestrator or legacy path
|
||||
- Legacy code preserved as `generate_video_legacy()` for rollback
|
||||
- New `generate_video_orchestrated()` builds config and runs orchestrator
|
||||
|
||||
## Feature Flag
|
||||
|
||||
```bash
|
||||
# Enable new orchestrator (default)
|
||||
USE_NEW_ORCHESTRATOR=true
|
||||
|
||||
# Rollback to legacy behavior
|
||||
USE_NEW_ORCHESTRATOR=false
|
||||
```
|
||||
|
||||
## Key Design Decisions
|
||||
|
||||
1. **Stage-based orchestration**: Each stage is independent and testable
|
||||
2. **Strategy pattern for encoding**: `EncodingBackend` interface with local and GCE implementations
|
||||
3. **Lazy service loading**: Services instantiated only when needed via `_get_*()` methods
|
||||
4. **Dataclass configs**: `OrchestratorConfig` and `OrchestratorResult` for clear contracts
|
||||
5. **Feature flag default true**: New code active immediately since tests are comprehensive
|
||||
|
||||
## Testing Notes
|
||||
|
||||
- All 976 existing tests continue to pass
|
||||
- 139 new tests added across 6 test files
|
||||
- E2E verification: Jobs 501258e1 and d21e6ef0 successfully processed through new pipeline
|
||||
- Both jobs reached `awaiting_instrumental_selection` as expected (requires user input to complete)
|
||||
|
||||
## PR
|
||||
|
||||
- PR #182: https://github.com/nomadkaraoke/karaoke-gen/pull/182
|
||||
- Merged: 2026-01-04
|
||||
- Version: 0.89.0
|
||||
|
||||
## Lessons Learned
|
||||
|
||||
See [LESSONS-LEARNED.md](../LESSONS-LEARNED.md#alternative-code-paths-must-implement-all-features) for the key architectural lesson about alternative code paths.
|
||||
@@ -46,8 +46,6 @@ function AdminBreadcrumb() {
|
||||
breadcrumbs.push({ label: "Jobs", href: "/admin/jobs" })
|
||||
breadcrumbs.push({ label: decodeURIComponent(segments[2]) })
|
||||
}
|
||||
} else if (segments[1] === "searches") {
|
||||
breadcrumbs.push({ label: "Audio Searches" })
|
||||
} else if (segments[1] === "beta") {
|
||||
breadcrumbs.push({ label: "Beta Program" })
|
||||
}
|
||||
|
||||
@@ -1,473 +0,0 @@
|
||||
"use client"
|
||||
|
||||
import { useEffect, useState, useCallback } from "react"
|
||||
import { adminApi, AudioSearchJobSummary, CacheStatsResponse } from "@/lib/api"
|
||||
import { Button } from "@/components/ui/button"
|
||||
import { Badge } from "@/components/ui/badge"
|
||||
import {
|
||||
Table,
|
||||
TableBody,
|
||||
TableCell,
|
||||
TableHead,
|
||||
TableHeader,
|
||||
TableRow,
|
||||
} from "@/components/ui/table"
|
||||
import {
|
||||
Select,
|
||||
SelectContent,
|
||||
SelectItem,
|
||||
SelectTrigger,
|
||||
SelectValue,
|
||||
} from "@/components/ui/select"
|
||||
import {
|
||||
AlertDialog,
|
||||
AlertDialogAction,
|
||||
AlertDialogCancel,
|
||||
AlertDialogContent,
|
||||
AlertDialogDescription,
|
||||
AlertDialogFooter,
|
||||
AlertDialogHeader,
|
||||
AlertDialogTitle,
|
||||
} from "@/components/ui/alert-dialog"
|
||||
import {
|
||||
Tooltip,
|
||||
TooltipContent,
|
||||
TooltipProvider,
|
||||
TooltipTrigger,
|
||||
} from "@/components/ui/tooltip"
|
||||
import {
|
||||
RefreshCw,
|
||||
Loader2,
|
||||
Trash2,
|
||||
Music,
|
||||
CheckCircle,
|
||||
XCircle,
|
||||
ExternalLink,
|
||||
Database,
|
||||
HardDrive,
|
||||
} from "lucide-react"
|
||||
import { useToast } from "@/hooks/use-toast"
|
||||
|
||||
const statusOptions = [
|
||||
{ value: "all", label: "All Statuses" },
|
||||
{ value: "awaiting_audio_selection", label: "Awaiting Selection" },
|
||||
{ value: "pending", label: "Pending" },
|
||||
{ value: "complete", label: "Complete" },
|
||||
{ value: "failed", label: "Failed" },
|
||||
]
|
||||
|
||||
export default function AdminSearchesPage() {
|
||||
const { toast } = useToast()
|
||||
const [searches, setSearches] = useState<AudioSearchJobSummary[]>([])
|
||||
const [loading, setLoading] = useState(true)
|
||||
const [statusFilter, setStatusFilter] = useState("all")
|
||||
|
||||
// Clear cache dialog (single job)
|
||||
const [clearDialogOpen, setClearDialogOpen] = useState(false)
|
||||
const [jobToClear, setJobToClear] = useState<AudioSearchJobSummary | null>(null)
|
||||
const [clearing, setClearing] = useState(false)
|
||||
|
||||
// Clear all cache dialog
|
||||
const [clearAllDialogOpen, setClearAllDialogOpen] = useState(false)
|
||||
const [clearingAll, setClearingAll] = useState(false)
|
||||
|
||||
// Cache stats
|
||||
const [cacheStats, setCacheStats] = useState<CacheStatsResponse | null>(null)
|
||||
const [statsLoading, setStatsLoading] = useState(false)
|
||||
|
||||
const loadSearches = useCallback(async () => {
|
||||
try {
|
||||
setLoading(true)
|
||||
const data = await adminApi.listAudioSearches({
|
||||
limit: 100,
|
||||
status_filter: statusFilter !== "all" ? statusFilter : undefined,
|
||||
})
|
||||
setSearches(data.jobs)
|
||||
} catch (err: any) {
|
||||
console.error("Failed to load searches:", err)
|
||||
toast({
|
||||
title: "Error",
|
||||
description: err.message || "Failed to load audio searches",
|
||||
variant: "destructive",
|
||||
})
|
||||
} finally {
|
||||
setLoading(false)
|
||||
}
|
||||
}, [statusFilter, toast])
|
||||
|
||||
const loadCacheStats = useCallback(async () => {
|
||||
try {
|
||||
setStatsLoading(true)
|
||||
const stats = await adminApi.getCacheStats()
|
||||
setCacheStats(stats)
|
||||
} catch (err: any) {
|
||||
// Silently fail - stats are optional
|
||||
console.error("Failed to load cache stats:", err)
|
||||
setCacheStats(null)
|
||||
} finally {
|
||||
setStatsLoading(false)
|
||||
}
|
||||
}, [])
|
||||
|
||||
useEffect(() => {
|
||||
loadSearches()
|
||||
loadCacheStats()
|
||||
}, [loadSearches, loadCacheStats])
|
||||
|
||||
const handleClearCache = async () => {
|
||||
if (!jobToClear) return
|
||||
|
||||
try {
|
||||
setClearing(true)
|
||||
const result = await adminApi.clearAudioSearchCache(jobToClear.job_id)
|
||||
toast({
|
||||
title: "Cache Cleared",
|
||||
description: result.message,
|
||||
})
|
||||
setClearDialogOpen(false)
|
||||
setJobToClear(null)
|
||||
loadSearches()
|
||||
loadCacheStats()
|
||||
} catch (err: any) {
|
||||
toast({
|
||||
title: "Error",
|
||||
description: err.message || "Failed to clear cache",
|
||||
variant: "destructive",
|
||||
})
|
||||
} finally {
|
||||
setClearing(false)
|
||||
}
|
||||
}
|
||||
|
||||
const handleClearAllCache = async () => {
|
||||
try {
|
||||
setClearingAll(true)
|
||||
const result = await adminApi.clearAllCache()
|
||||
toast({
|
||||
title: "All Cache Cleared",
|
||||
description: result.message,
|
||||
})
|
||||
setClearAllDialogOpen(false)
|
||||
loadCacheStats()
|
||||
} catch (err: any) {
|
||||
toast({
|
||||
title: "Error",
|
||||
description: err.message || "Failed to clear all cache",
|
||||
variant: "destructive",
|
||||
})
|
||||
} finally {
|
||||
setClearingAll(false)
|
||||
}
|
||||
}
|
||||
|
||||
const formatBytes = (bytes: number): string => {
|
||||
if (bytes === 0) return "0 B"
|
||||
const k = 1024
|
||||
const sizes = ["B", "KB", "MB", "GB"]
|
||||
const i = Math.floor(Math.log(bytes) / Math.log(k))
|
||||
return parseFloat((bytes / Math.pow(k, i)).toFixed(1)) + " " + sizes[i]
|
||||
}
|
||||
|
||||
const getStatusVariant = (status: string) => {
|
||||
if (status === "complete" || status === "prep_complete") return "default"
|
||||
if (status === "failed") return "destructive"
|
||||
if (status === "cancelled") return "outline"
|
||||
if (status.includes("awaiting")) return "secondary"
|
||||
return "secondary"
|
||||
}
|
||||
|
||||
const formatDate = (dateStr?: string) => {
|
||||
if (!dateStr) return "—"
|
||||
return new Date(dateStr).toLocaleString()
|
||||
}
|
||||
|
||||
const getProviderBadgeClass = (provider: string) => {
|
||||
switch (provider.toLowerCase()) {
|
||||
case "youtube":
|
||||
return "bg-red-500/20 text-red-400 border-red-500/30"
|
||||
case "red":
|
||||
return "bg-red-600/30 text-red-300 border-red-500/30"
|
||||
case "ops":
|
||||
return "bg-blue-500/20 text-blue-400 border-blue-500/30"
|
||||
default:
|
||||
return "bg-slate-500/20 text-slate-400 border-slate-500/30"
|
||||
}
|
||||
}
|
||||
|
||||
return (
|
||||
<div className="space-y-6">
|
||||
{/* Header */}
|
||||
<div className="flex items-center justify-between">
|
||||
<div>
|
||||
<h1 className="text-2xl font-bold tracking-tight">Audio Searches</h1>
|
||||
<p className="text-muted-foreground">
|
||||
View and manage cached audio search results
|
||||
</p>
|
||||
</div>
|
||||
<div className="flex items-center gap-2">
|
||||
<Button variant="outline" size="sm" onClick={() => { loadSearches(); loadCacheStats() }} disabled={loading}>
|
||||
<RefreshCw className={`w-4 h-4 mr-2 ${loading ? "animate-spin" : ""}`} />
|
||||
Refresh
|
||||
</Button>
|
||||
<TooltipProvider>
|
||||
<Tooltip>
|
||||
<TooltipTrigger asChild>
|
||||
<Button
|
||||
variant="outline"
|
||||
size="sm"
|
||||
className="text-red-500 hover:text-red-400 hover:bg-red-500/10 border-red-500/30"
|
||||
onClick={() => setClearAllDialogOpen(true)}
|
||||
>
|
||||
<Database className="w-4 h-4 mr-2" />
|
||||
Clear All Cache
|
||||
</Button>
|
||||
</TooltipTrigger>
|
||||
<TooltipContent>
|
||||
<p>Clear entire flacfetch search cache</p>
|
||||
</TooltipContent>
|
||||
</Tooltip>
|
||||
</TooltipProvider>
|
||||
</div>
|
||||
</div>
|
||||
|
||||
{/* Cache Stats */}
|
||||
{cacheStats && (
|
||||
<div className="flex items-center gap-4 text-sm text-muted-foreground bg-muted/30 rounded-lg px-4 py-2">
|
||||
<div className="flex items-center gap-2">
|
||||
<HardDrive className="w-4 h-4" />
|
||||
<span>Flacfetch Cache:</span>
|
||||
</div>
|
||||
<Badge variant="outline" className="font-mono">
|
||||
{cacheStats.count} entries
|
||||
</Badge>
|
||||
<Badge variant="outline" className="font-mono">
|
||||
{formatBytes(cacheStats.total_size_bytes)}
|
||||
</Badge>
|
||||
{cacheStats.newest_entry && (
|
||||
<span className="text-xs">
|
||||
Latest: {new Date(cacheStats.newest_entry).toLocaleDateString()}
|
||||
</span>
|
||||
)}
|
||||
</div>
|
||||
)}
|
||||
|
||||
{/* Filters */}
|
||||
<div className="flex gap-4">
|
||||
<Select value={statusFilter} onValueChange={setStatusFilter}>
|
||||
<SelectTrigger className="w-[200px]">
|
||||
<SelectValue placeholder="Filter by status" />
|
||||
</SelectTrigger>
|
||||
<SelectContent>
|
||||
{statusOptions.map((opt) => (
|
||||
<SelectItem key={opt.value} value={opt.value}>
|
||||
{opt.label}
|
||||
</SelectItem>
|
||||
))}
|
||||
</SelectContent>
|
||||
</Select>
|
||||
</div>
|
||||
|
||||
{/* Table */}
|
||||
<div className="rounded-md border">
|
||||
<Table>
|
||||
<TableHeader>
|
||||
<TableRow>
|
||||
<TableHead>Job ID</TableHead>
|
||||
<TableHead>Search Query</TableHead>
|
||||
<TableHead>Results</TableHead>
|
||||
<TableHead>Providers</TableHead>
|
||||
<TableHead>Status</TableHead>
|
||||
<TableHead>User</TableHead>
|
||||
<TableHead>Created</TableHead>
|
||||
<TableHead className="text-right">Actions</TableHead>
|
||||
</TableRow>
|
||||
</TableHeader>
|
||||
<TableBody>
|
||||
{loading ? (
|
||||
<TableRow>
|
||||
<TableCell colSpan={8} className="text-center py-8">
|
||||
<Loader2 className="w-6 h-6 animate-spin mx-auto text-muted-foreground" />
|
||||
</TableCell>
|
||||
</TableRow>
|
||||
) : searches.length === 0 ? (
|
||||
<TableRow>
|
||||
<TableCell colSpan={8} className="text-center py-8 text-muted-foreground">
|
||||
<Music className="w-8 h-8 mx-auto mb-2 opacity-50" />
|
||||
No audio searches found
|
||||
</TableCell>
|
||||
</TableRow>
|
||||
) : (
|
||||
searches.map((search) => (
|
||||
<TableRow key={search.job_id}>
|
||||
<TableCell className="font-mono text-xs">
|
||||
<a
|
||||
href={`/admin/jobs?id=${search.job_id}`}
|
||||
className="text-blue-500 hover:underline"
|
||||
>
|
||||
{search.job_id.slice(0, 8)}...
|
||||
</a>
|
||||
</TableCell>
|
||||
<TableCell>
|
||||
<div className="max-w-[200px]">
|
||||
<div className="font-medium truncate">
|
||||
{search.audio_search_artist || "—"}
|
||||
</div>
|
||||
<div className="text-sm text-muted-foreground truncate">
|
||||
{search.audio_search_title || "—"}
|
||||
</div>
|
||||
</div>
|
||||
</TableCell>
|
||||
<TableCell>
|
||||
<TooltipProvider>
|
||||
<Tooltip>
|
||||
<TooltipTrigger asChild>
|
||||
<div className="flex items-center gap-2">
|
||||
<span className="font-medium">{search.results_count}</span>
|
||||
{search.has_lossless ? (
|
||||
<CheckCircle className="w-4 h-4 text-green-500" />
|
||||
) : (
|
||||
<XCircle className="w-4 h-4 text-red-400" />
|
||||
)}
|
||||
</div>
|
||||
</TooltipTrigger>
|
||||
<TooltipContent>
|
||||
<p>{search.has_lossless ? "Has lossless sources" : "No lossless sources (YouTube only)"}</p>
|
||||
</TooltipContent>
|
||||
</Tooltip>
|
||||
</TooltipProvider>
|
||||
</TableCell>
|
||||
<TableCell>
|
||||
<div className="flex flex-wrap gap-1">
|
||||
{search.providers.map((provider) => (
|
||||
<Badge
|
||||
key={provider}
|
||||
variant="outline"
|
||||
className={`text-[10px] ${getProviderBadgeClass(provider)}`}
|
||||
>
|
||||
{provider}
|
||||
</Badge>
|
||||
))}
|
||||
</div>
|
||||
</TableCell>
|
||||
<TableCell>
|
||||
<Badge variant={getStatusVariant(search.status)}>
|
||||
{search.status.replace(/_/g, " ")}
|
||||
</Badge>
|
||||
</TableCell>
|
||||
<TableCell className="text-xs text-muted-foreground">
|
||||
{search.user_email?.split("@")[0] || "—"}
|
||||
</TableCell>
|
||||
<TableCell className="text-xs text-muted-foreground">
|
||||
{formatDate(search.created_at)}
|
||||
</TableCell>
|
||||
<TableCell className="text-right">
|
||||
<TooltipProvider>
|
||||
<Tooltip>
|
||||
<TooltipTrigger asChild>
|
||||
<Button
|
||||
variant="ghost"
|
||||
size="icon"
|
||||
className="h-8 w-8 text-orange-500 hover:text-orange-400 hover:bg-orange-500/10"
|
||||
onClick={() => {
|
||||
setJobToClear(search)
|
||||
setClearDialogOpen(true)
|
||||
}}
|
||||
>
|
||||
<Trash2 className="w-4 h-4" />
|
||||
</Button>
|
||||
</TooltipTrigger>
|
||||
<TooltipContent>
|
||||
<p>Clear cache & allow re-search</p>
|
||||
</TooltipContent>
|
||||
</Tooltip>
|
||||
</TooltipProvider>
|
||||
</TableCell>
|
||||
</TableRow>
|
||||
))
|
||||
)}
|
||||
</TableBody>
|
||||
</Table>
|
||||
</div>
|
||||
|
||||
{/* Summary */}
|
||||
{!loading && searches.length > 0 && (
|
||||
<div className="text-sm text-muted-foreground">
|
||||
Showing {searches.length} jobs with cached search results.
|
||||
{searches.filter(s => !s.has_lossless).length > 0 && (
|
||||
<span className="ml-2 text-orange-400">
|
||||
{searches.filter(s => !s.has_lossless).length} have YouTube-only results (may need cache refresh).
|
||||
</span>
|
||||
)}
|
||||
</div>
|
||||
)}
|
||||
|
||||
{/* Clear Cache Dialog (single job) */}
|
||||
<AlertDialog open={clearDialogOpen} onOpenChange={setClearDialogOpen}>
|
||||
<AlertDialogContent>
|
||||
<AlertDialogHeader>
|
||||
<AlertDialogTitle>Clear Search Cache?</AlertDialogTitle>
|
||||
<AlertDialogDescription>
|
||||
This will clear the cached search results for job{" "}
|
||||
<span className="font-mono">{jobToClear?.job_id}</span> and reset it to pending
|
||||
status, allowing a new search to be performed.
|
||||
{jobToClear && !jobToClear.has_lossless && (
|
||||
<span className="block mt-2 text-orange-400">
|
||||
This job only has YouTube results - clearing the cache may fix this if flacfetch has been updated.
|
||||
</span>
|
||||
)}
|
||||
</AlertDialogDescription>
|
||||
</AlertDialogHeader>
|
||||
<AlertDialogFooter>
|
||||
<AlertDialogCancel>Cancel</AlertDialogCancel>
|
||||
<AlertDialogAction
|
||||
onClick={handleClearCache}
|
||||
disabled={clearing}
|
||||
className="bg-orange-600 hover:bg-orange-500"
|
||||
>
|
||||
{clearing ? (
|
||||
<Loader2 className="w-4 h-4 animate-spin mr-2" />
|
||||
) : (
|
||||
<Trash2 className="w-4 h-4 mr-2" />
|
||||
)}
|
||||
Clear Cache
|
||||
</AlertDialogAction>
|
||||
</AlertDialogFooter>
|
||||
</AlertDialogContent>
|
||||
</AlertDialog>
|
||||
|
||||
{/* Clear All Cache Dialog */}
|
||||
<AlertDialog open={clearAllDialogOpen} onOpenChange={setClearAllDialogOpen}>
|
||||
<AlertDialogContent>
|
||||
<AlertDialogHeader>
|
||||
<AlertDialogTitle>Clear Entire Flacfetch Cache?</AlertDialogTitle>
|
||||
<AlertDialogDescription>
|
||||
This will delete all cached search results from flacfetch's GCS cache
|
||||
{cacheStats && (
|
||||
<span className="font-medium"> ({cacheStats.count} entries, {formatBytes(cacheStats.total_size_bytes)})</span>
|
||||
)}.
|
||||
<span className="block mt-2 text-orange-400">
|
||||
All subsequent searches will hit the trackers fresh. This does NOT affect
|
||||
the Firestore job caches - use individual "Clear Cache" buttons for those.
|
||||
</span>
|
||||
</AlertDialogDescription>
|
||||
</AlertDialogHeader>
|
||||
<AlertDialogFooter>
|
||||
<AlertDialogCancel>Cancel</AlertDialogCancel>
|
||||
<AlertDialogAction
|
||||
onClick={handleClearAllCache}
|
||||
disabled={clearingAll}
|
||||
className="bg-red-600 hover:bg-red-500"
|
||||
>
|
||||
{clearingAll ? (
|
||||
<Loader2 className="w-4 h-4 animate-spin mr-2" />
|
||||
) : (
|
||||
<Database className="w-4 h-4 mr-2" />
|
||||
)}
|
||||
Clear All Cache
|
||||
</AlertDialogAction>
|
||||
</AlertDialogFooter>
|
||||
</AlertDialogContent>
|
||||
</AlertDialog>
|
||||
</div>
|
||||
)
|
||||
}
|
||||
@@ -45,7 +45,7 @@ describe('AuthStatus', () => {
|
||||
|
||||
// Should show user email and credits
|
||||
expect(screen.getByText('test@example.com')).toBeInTheDocument()
|
||||
expect(screen.getByText('5 credits')).toBeInTheDocument()
|
||||
expect(screen.getByText('5')).toBeInTheDocument()
|
||||
})
|
||||
|
||||
it('opens auth dialog when Login is clicked', () => {
|
||||
|
||||
@@ -9,7 +9,6 @@ import {
|
||||
TestTube2,
|
||||
ArrowLeft,
|
||||
Settings,
|
||||
Search,
|
||||
} from "lucide-react"
|
||||
|
||||
import {
|
||||
@@ -42,11 +41,6 @@ const navItems = [
|
||||
href: "/admin/jobs",
|
||||
icon: Briefcase,
|
||||
},
|
||||
{
|
||||
title: "Audio Searches",
|
||||
href: "/admin/searches",
|
||||
icon: Search,
|
||||
},
|
||||
{
|
||||
title: "Beta Program",
|
||||
href: "/admin/beta",
|
||||
|
||||
@@ -12,7 +12,7 @@ import {
|
||||
DropdownMenuSeparator,
|
||||
DropdownMenuTrigger,
|
||||
} from "./ui/dropdown-menu"
|
||||
import { User, LogOut, Settings, Shield, Mail } from "lucide-react"
|
||||
import { User, LogOut, Settings } from "lucide-react"
|
||||
import Link from "next/link"
|
||||
|
||||
export function AppHeader() {
|
||||
@@ -40,7 +40,7 @@ export function AppHeader() {
|
||||
<Link href="/jobs" className="text-sm font-medium hover:text-primary transition-colors">
|
||||
My Jobs
|
||||
</Link>
|
||||
{(user.role === "admin" || user.email?.endsWith("@nomadkaraoke.com")) && (
|
||||
{user.role === "admin" && (
|
||||
<Link href="/admin" className="text-sm font-medium hover:text-primary transition-colors">
|
||||
Admin
|
||||
</Link>
|
||||
@@ -48,14 +48,6 @@ export function AppHeader() {
|
||||
</nav>
|
||||
|
||||
<div className="flex items-center gap-4">
|
||||
<a
|
||||
href="mailto:andrew@nomadkaraoke.com"
|
||||
className="hidden lg:flex items-center gap-1.5 text-xs text-muted-foreground hover:text-foreground transition-colors"
|
||||
>
|
||||
<Mail className="w-3.5 h-3.5" />
|
||||
<span>Need help?</span>
|
||||
</a>
|
||||
|
||||
<div className="hidden sm:flex items-center gap-2 px-4 py-2 rounded-lg bg-primary/10 border border-primary/20">
|
||||
<div className="w-2 h-2 rounded-full bg-primary animate-pulse" />
|
||||
<span className="text-sm font-medium text-primary">{user.credits} credits</span>
|
||||
@@ -81,24 +73,10 @@ export function AppHeader() {
|
||||
<span>{user.credits} credits</span>
|
||||
</div>
|
||||
</DropdownMenuItem>
|
||||
{(user.role === "admin" || user.email?.endsWith("@nomadkaraoke.com")) && (
|
||||
<DropdownMenuItem asChild>
|
||||
<Link href="/admin">
|
||||
<Shield className="w-4 h-4 mr-2" />
|
||||
Admin Dashboard
|
||||
</Link>
|
||||
</DropdownMenuItem>
|
||||
)}
|
||||
<DropdownMenuItem>
|
||||
<Settings className="w-4 h-4 mr-2" />
|
||||
Settings
|
||||
</DropdownMenuItem>
|
||||
<DropdownMenuItem asChild>
|
||||
<a href="mailto:andrew@nomadkaraoke.com">
|
||||
<Mail className="w-4 h-4 mr-2" />
|
||||
Need help? Contact us
|
||||
</a>
|
||||
</DropdownMenuItem>
|
||||
<DropdownMenuSeparator />
|
||||
<DropdownMenuItem onClick={handleLogout} className="text-destructive focus:text-destructive">
|
||||
<LogOut className="w-4 h-4 mr-2" />
|
||||
|
||||
@@ -1,8 +1,7 @@
|
||||
"use client"
|
||||
|
||||
import { useState, useEffect } from "react"
|
||||
import { User, LogOut, CreditCard, Coins, KeyRound, Shield } from "lucide-react"
|
||||
import Link from "next/link"
|
||||
import { User, LogOut, CreditCard, Coins, KeyRound } from "lucide-react"
|
||||
import { Button } from "@/components/ui/button"
|
||||
import {
|
||||
DropdownMenu,
|
||||
@@ -70,7 +69,7 @@ export function AuthStatus({ onAuthChange }: AuthStatusProps) {
|
||||
</span>
|
||||
<span className="flex items-center gap-1 text-amber-400 font-medium">
|
||||
<Coins className="w-3 h-3" />
|
||||
{user.credits} {user.credits === 1 ? 'credit' : 'credits'}
|
||||
{user.credits}
|
||||
</span>
|
||||
</Button>
|
||||
</DropdownMenuTrigger>
|
||||
@@ -98,17 +97,6 @@ export function AuthStatus({ onAuthChange }: AuthStatusProps) {
|
||||
<CreditCard className="w-4 h-4 mr-2" />
|
||||
<span>Buy More Credits</span>
|
||||
</DropdownMenuItem>
|
||||
{(user.role === "admin" || user.email?.endsWith("@nomadkaraoke.com")) && (
|
||||
<>
|
||||
<DropdownMenuSeparator className="bg-slate-700" />
|
||||
<DropdownMenuItem asChild className="text-slate-300 focus:text-white focus:bg-slate-800">
|
||||
<Link href="/admin">
|
||||
<Shield className="w-4 h-4 mr-2" />
|
||||
<span>Admin Dashboard</span>
|
||||
</Link>
|
||||
</DropdownMenuItem>
|
||||
</>
|
||||
)}
|
||||
<DropdownMenuSeparator className="bg-slate-700" />
|
||||
<DropdownMenuItem
|
||||
onClick={handleLogout}
|
||||
|
||||
@@ -272,8 +272,8 @@ export function JobSubmission({ onJobCreated }: JobSubmissionProps) {
|
||||
Skip lyrics review (non-interactive)
|
||||
</Label>
|
||||
<p className="text-xs mt-1" style={{ color: 'var(--text-muted)' }}>
|
||||
<span className="text-amber-500 font-medium">Warning:</span> auto lyrics correction isn't perfect.
|
||||
Skipping review will likely result in incorrect words in your karaoke video.
|
||||
<span className="text-amber-500 font-medium">Warning:</span> AI lyrics correction isn't perfect.
|
||||
Skipping review will likely result in timing errors and incorrect words in your karaoke video.
|
||||
</p>
|
||||
</div>
|
||||
</div>
|
||||
@@ -377,8 +377,8 @@ export function JobSubmission({ onJobCreated }: JobSubmissionProps) {
|
||||
Skip lyrics review (non-interactive)
|
||||
</Label>
|
||||
<p className="text-xs mt-1" style={{ color: 'var(--text-muted)' }}>
|
||||
<span className="text-amber-500 font-medium">Warning:</span> auto lyrics correction isn't perfect.
|
||||
Skipping review will likely result in incorrect words in your karaoke video.
|
||||
<span className="text-amber-500 font-medium">Warning:</span> AI lyrics correction isn't perfect.
|
||||
Skipping review will likely result in timing errors and incorrect words in your karaoke video.
|
||||
</p>
|
||||
</div>
|
||||
</div>
|
||||
@@ -465,8 +465,8 @@ export function JobSubmission({ onJobCreated }: JobSubmissionProps) {
|
||||
Skip lyrics review (non-interactive)
|
||||
</Label>
|
||||
<p className="text-xs mt-1" style={{ color: 'var(--text-muted)' }}>
|
||||
<span className="text-amber-500 font-medium">Warning:</span> auto lyrics correction isn't perfect.
|
||||
Skipping review will likely result in incorrect words in your karaoke video.
|
||||
<span className="text-amber-500 font-medium">Warning:</span> AI lyrics correction isn't perfect.
|
||||
Skipping review will likely result in timing errors and incorrect words in your karaoke video.
|
||||
</p>
|
||||
</div>
|
||||
</div>
|
||||
|
||||
@@ -453,8 +453,8 @@ test.describe('Production E2E - Full User Journey', () => {
|
||||
jobId,
|
||||
['awaiting_instrumental_selection', 'complete'],
|
||||
accessToken!,
|
||||
TIMEOUTS.download // Video rendering can take 10-15 minutes with orchestrator
|
||||
);
|
||||
TIMEOUTS.jobProcessing // Video rendering can take several minutes
|
||||
).catch(() => ({ status: 'unknown', job: null }));
|
||||
|
||||
if (statusBeforeInstrumental === 'awaiting_instrumental_selection') {
|
||||
console.log(' Selecting clean instrumental via API...');
|
||||
|
||||
@@ -11,7 +11,7 @@ import { createEmailHelper, isEmailTestingAvailable } from '../helpers/email-tes
|
||||
*
|
||||
* Test Flow:
|
||||
* 1. Landing Page - Navigate and verify
|
||||
* 2. New User Signup - Create testmail.app inbox, submit beta form, verify email
|
||||
* 2. New User Signup - Create MailSlurp inbox, submit beta form, verify email
|
||||
* 3. Magic Link Auth - Click magic link to authenticate
|
||||
* 4. Create Job - Search for test song via UI
|
||||
* 5. Audio Selection - Select audio source via UI dialog
|
||||
@@ -24,7 +24,7 @@ import { createEmailHelper, isEmailTestingAvailable } from '../helpers/email-tes
|
||||
* 12. Cleanup - Delete distributed content and job
|
||||
*
|
||||
* Environment Variables:
|
||||
* - TESTMAIL_API_KEY + TESTMAIL_NAMESPACE: Required for email testing (can be skipped with E2E_TEST_TOKEN)
|
||||
* - MAILSLURP_API_KEY: Required for email testing (can be skipped with E2E_TEST_TOKEN)
|
||||
* - E2E_TEST_TOKEN: Pre-configured user token to skip signup flow (for iterations)
|
||||
* - E2E_ADMIN_TOKEN: Required for cleanup endpoint (optional - cleanup skipped if not set)
|
||||
*/
|
||||
@@ -138,12 +138,12 @@ test.describe('E2E Happy Path - Real User with Full UI Interactions', () => {
|
||||
}) => {
|
||||
test.setTimeout(TIMEOUTS.fullTest);
|
||||
|
||||
// Check if we have a pre-configured token (for iterations without using testmail.app)
|
||||
// Check if we have a pre-configured token (for iterations without using MailSlurp)
|
||||
const usePreConfiguredToken = hasPreConfiguredToken();
|
||||
|
||||
// Skip if neither pre-configured token nor testmail.app is available
|
||||
// Skip if neither pre-configured token nor MailSlurp is available
|
||||
if (!usePreConfiguredToken && !isEmailTestingAvailable()) {
|
||||
test.skip(true, 'Neither E2E_TEST_TOKEN nor TESTMAIL_API_KEY/TESTMAIL_NAMESPACE set - cannot run test');
|
||||
test.skip(true, 'Neither E2E_TEST_TOKEN nor MAILSLURP_API_KEY set - cannot run test');
|
||||
return;
|
||||
}
|
||||
|
||||
@@ -217,7 +217,7 @@ test.describe('E2E Happy Path - Real User with Full UI Interactions', () => {
|
||||
console.log('STEP 3 COMPLETE: User authenticated with pre-configured token');
|
||||
|
||||
} else {
|
||||
// ----- FULL PATH: Beta signup with testmail.app -----
|
||||
// ----- FULL PATH: Beta signup with MailSlurp -----
|
||||
console.log('\n========================================');
|
||||
console.log('STEP 2: New User Beta Enrollment');
|
||||
console.log('========================================');
|
||||
|
||||
@@ -782,60 +782,6 @@ export interface AdminJobListParams {
|
||||
limit?: number;
|
||||
}
|
||||
|
||||
// Audio Search Admin Types
|
||||
export interface AudioSearchResultSummary {
|
||||
index: number;
|
||||
provider: string;
|
||||
artist: string;
|
||||
title: string;
|
||||
is_lossless: boolean;
|
||||
quality?: string;
|
||||
seeders?: number;
|
||||
}
|
||||
|
||||
export interface AudioSearchJobSummary {
|
||||
job_id: string;
|
||||
status: string;
|
||||
user_email?: string;
|
||||
audio_search_artist?: string;
|
||||
audio_search_title?: string;
|
||||
created_at?: string;
|
||||
results_count: number;
|
||||
results_summary: AudioSearchResultSummary[];
|
||||
has_lossless: boolean;
|
||||
providers: string[];
|
||||
}
|
||||
|
||||
export interface AudioSearchListResponse {
|
||||
jobs: AudioSearchJobSummary[];
|
||||
total: number;
|
||||
}
|
||||
|
||||
export interface ClearSearchCacheResponse {
|
||||
status: string;
|
||||
job_id: string;
|
||||
message: string;
|
||||
previous_status: string;
|
||||
new_status: string;
|
||||
results_cleared: number;
|
||||
flacfetch_cache_cleared: boolean;
|
||||
flacfetch_error?: string;
|
||||
}
|
||||
|
||||
export interface ClearAllCacheResponse {
|
||||
status: string;
|
||||
message: string;
|
||||
deleted_count: number;
|
||||
}
|
||||
|
||||
export interface CacheStatsResponse {
|
||||
count: number;
|
||||
total_size_bytes: number;
|
||||
oldest_entry?: string;
|
||||
newest_entry?: string;
|
||||
configured: boolean;
|
||||
}
|
||||
|
||||
// Admin API namespace
|
||||
export const adminApi = {
|
||||
/**
|
||||
@@ -987,57 +933,6 @@ export const adminApi = {
|
||||
);
|
||||
return handleResponse(response);
|
||||
},
|
||||
|
||||
/**
|
||||
* List jobs with audio search results
|
||||
*/
|
||||
async listAudioSearches(params?: {
|
||||
limit?: number;
|
||||
status_filter?: string;
|
||||
}): Promise<AudioSearchListResponse> {
|
||||
const searchParams = new URLSearchParams();
|
||||
if (params?.limit) searchParams.set('limit', String(params.limit));
|
||||
if (params?.status_filter) searchParams.set('status_filter', params.status_filter);
|
||||
|
||||
const url = `${API_BASE_URL}/api/admin/audio-searches${searchParams.toString() ? '?' + searchParams.toString() : ''}`;
|
||||
const response = await fetch(url, {
|
||||
headers: getAuthHeaders()
|
||||
});
|
||||
return handleResponse(response);
|
||||
},
|
||||
|
||||
/**
|
||||
* Clear audio search cache for a job
|
||||
*/
|
||||
async clearAudioSearchCache(jobId: string): Promise<ClearSearchCacheResponse> {
|
||||
const response = await fetch(
|
||||
`${API_BASE_URL}/api/admin/audio-searches/${jobId}/clear-cache`,
|
||||
{ method: 'POST', headers: getAuthHeaders() }
|
||||
);
|
||||
return handleResponse(response);
|
||||
},
|
||||
|
||||
/**
|
||||
* Clear all flacfetch search cache
|
||||
*/
|
||||
async clearAllCache(): Promise<ClearAllCacheResponse> {
|
||||
const response = await fetch(
|
||||
`${API_BASE_URL}/api/admin/cache`,
|
||||
{ method: 'DELETE', headers: getAuthHeaders() }
|
||||
);
|
||||
return handleResponse(response);
|
||||
},
|
||||
|
||||
/**
|
||||
* Get flacfetch cache statistics
|
||||
*/
|
||||
async getCacheStats(): Promise<CacheStatsResponse> {
|
||||
const response = await fetch(
|
||||
`${API_BASE_URL}/api/admin/cache/stats`,
|
||||
{ headers: getAuthHeaders() }
|
||||
);
|
||||
return handleResponse(response);
|
||||
},
|
||||
};
|
||||
|
||||
export { ApiError };
|
||||
|
||||
@@ -1,6 +1,6 @@
|
||||
{
|
||||
"name": "karaoke-gen-frontend",
|
||||
"version": "0.80.0",
|
||||
"version": "0.79.0",
|
||||
"private": true,
|
||||
"scripts": {
|
||||
"build": "next build",
|
||||
|
||||
@@ -1439,7 +1439,7 @@ async def process_job(job_id: str, request: EncodeRequest):
|
||||
|
||||
# Convert local paths to blob paths (backend expects blob paths, not full gs:// URIs)
|
||||
# output_gcs_path is like "gs://bucket/jobs/id/encoded/"
|
||||
# We need paths like "jobs/id/encoded/output_4k_lossless.mp4"
|
||||
# We need paths like "jobs/id/encoded/output_4k.mp4"
|
||||
gcs_path = request.output_gcs_path.replace("gs://", "")
|
||||
parts = gcs_path.split("/", 1)
|
||||
prefix = parts[1].rstrip("/") if len(parts) > 1 else ""
|
||||
|
||||
@@ -70,9 +70,7 @@ DEFAULT_KARAOKE_STYLE = {
|
||||
"background_color": "#000000",
|
||||
"background_image": None,
|
||||
# Font settings
|
||||
# Using "Noto Sans" for full Unicode support (CJK, Arabic, symbols, etc.)
|
||||
# Requires fonts-noto-* packages installed in Docker image
|
||||
"font": "Noto Sans",
|
||||
"font": "Arial",
|
||||
"font_path": "", # Must be string, not None (for ASS generator)
|
||||
"ass_name": "Default",
|
||||
# Colors in "R, G, B, A" format (required by ASS)
|
||||
|
||||
+1
-1
@@ -1,6 +1,6 @@
|
||||
[tool.poetry]
|
||||
name = "karaoke-gen"
|
||||
version = "0.89.3"
|
||||
version = "0.87.1"
|
||||
description = "Generate karaoke videos with synchronized lyrics. Handles the entire process from downloading audio and lyrics to creating the final video with title screens."
|
||||
authors = ["Andrew Beveridge <andrew@beveridge.uk>"]
|
||||
license = "MIT"
|
||||
|
||||
Reference in new issue
Block a user