refactor: Unify video worker pipeline with new orchestrator (#182)

Resolves GCE vs KaraokeFinalise code path divergence that caused features like YouTube upload, Discord notifications, and CDG/TXT packaging to be missing from the GCE encoding path.

- New services: youtube_upload_service, discord_service, packaging_service, local_encoding_service, encoding_interface
- New VideoWorkerOrchestrator coordinates all stages (packaging, encoding, distribution, notifications)
- Works with any encoding backend (GCE or local)
- 139 new tests, 976 total tests passing
- Feature flag USE_NEW_ORCHESTRATOR (default: true) with rollback capability
This commit is contained in:
Andrew Beveridge ☄️ authored and GitHub committed 2026-01-04 00:25:23 -05:00
1 parent b3045c5487
commit c1588992ce
15 files changed
+5865 -20

No files matched your search

@@ -0,0 +1,401 @@
# Plan: Video Worker Stage Refactor
**Created:** 2026-01-03
**Branch:** feat/sess-20260103-2250-investigate-youtube-upload
**Status:** COMPLETE - Ready for PR
**Last Updated:** 2026-01-04
## Overview
Refactor `video_worker.py` and `KaraokeFinalise` into separate, composable stages following SOLID principles. This eliminates the GCE vs KaraokeFinalise code path divergence that causes features like YouTube upload to be missing from the GCE encoding path.
## Three Execution Modes
Understanding the three ways this system runs is critical to the refactor:
### 1. Local CLI (`karaoke-gen`)
- Runs entirely on user's machine
- KaraokeFinalise.process() orchestrates everything
- Writes directly to local Dropbox sync folder (e.g., `~/Dropbox/Karaoke/Tracks-Organized/`)
- **Produces full output structure** with stems/, lyrics/, visualizations/, etc.
- **This is the source of truth for output format**
### 2. Remote CLI (`karaoke-gen-remote`)
- User's CLI talks to cloud backend (api.nomadkaraoke.com)
- Cloud does ALL processing (audio, lyrics, video encoding)
- CLI just monitors job status and downloads final outputs
- With Dropbox configured: cloud uploads full structure to user's Dropbox
- Without Dropbox: user downloads individual files
### 3. Web Frontend (`gen.nomadkaraoke.com`)
- Same cloud backend as remote CLI
- User sees:
- YouTube video link
- Dropbox folder link
- Direct download: final karaoke video (720p)
- Direct download: 4K video
- Direct download: "with vocals" video
- Direct download: CDG zip (if enabled)
## Problem Statement
Currently, `video_worker.py` has two completely divergent code paths:
```python
if use_gce_encoding:
# GCE PATH: Only encoding, missing YouTube/Discord/CDG/TXT
result = await _encode_via_gce(...)
# result DOES NOT include youtube_url!
else:
# STANDARD PATH: KaraokeFinalise does everything
finalise = KaraokeFinalise(...)
result = finalise.process()
# result includes youtube_url from upload
```
### What GCE Encoding Broke
| Feature | KaraokeFinalise Path | GCE Path |
|---------|---------------------|----------|
| Video encoding | ✅ 4 formats | ✅ 4 formats |
| YouTube upload | ✅ | ❌ **BROKEN** |
| Discord notification | ✅ | ❌ **BROKEN** |
| CDG/TXT packaging | ✅ | ❌ **BROKEN** |
| Brand code generation | ✅ | ✅ (via Dropbox service) |
| Dropbox upload | ✅ (rclone) | ✅ (native SDK) |
| Google Drive upload | ❌ | ✅ (native SDK) |
### Root Cause
`KaraokeFinalise.process()` is a monolith that handles:
1. CDG/TXT packaging
2. Video encoding (the CPU-intensive part)
3. YouTube upload
4. Discord notifications
5. Folder organization
The GCE encoding was added as a bypass around the entire class, rather than replacing just the encoding step.
## Requirements
### Functional Requirements
- [ ] YouTube upload works regardless of encoding backend (GCE or local)
- [ ] Discord notifications work regardless of encoding backend
- [ ] CDG/TXT packaging works regardless of encoding backend
- [ ] Brand code generation works consistently
- [ ] Dropbox/Google Drive upload works (already works in both paths)
- [ ] All existing CLI functionality continues to work
- [ ] Output structure matches local CLI exactly
### Output Structure Parity
Cloud backend must produce identical output to local CLI:
```
NOMAD-XXXX - Artist - Title/
├── Artist - Title (Title).mov
├── Artist - Title (Title).jpg
├── Artist - Title (End).mov
├── Artist - Title (End).jpg
├── Artist - Title (Final Karaoke Lossless 4k).mp4
├── Artist - Title (Final Karaoke Lossless 4k).mkv
├── Artist - Title (Final Karaoke Lossy 4k).mp4
├── Artist - Title (Final Karaoke Lossy 720p).mp4
├── Artist - Title (Final Karaoke CDG).zip
├── Artist - Title (Final Karaoke TXT).zip
├── Artist - Title (Instrumental {model}).flac
├── Artist - Title (Instrumental +BV {model}).flac
├── Artist - Title (Karaoke).mp4
├── Artist - Title (Karaoke).cdg
├── Artist - Title (Karaoke).mp3
├── Artist - Title (Karaoke).lrc
├── Artist - Title (With Vocals).mp4
├── stems/
│ ├── Artist - Title (Instrumental {model}).flac
│ ├── Artist - Title (Vocals {model}).flac
│ ├── Artist - Title (Lead Vocals {model}).flac
│ ├── Artist - Title (Backing Vocals {model}).flac
│ └── ... (other stems)
├── lyrics/
│ ├── Artist - Title (Karaoke).lrc
│ ├── Artist - Title (Karaoke).ass
│ ├── Artist - Title (Karaoke).srt
│ └── ... (corrections JSONs)
├── previews/
├── visualizations/
└── styles_assets/
```
### Non-Functional Requirements
- [ ] Single Responsibility: Each stage has one job
- [ ] Open/Closed: Easy to add new encoding backends without modifying stages
- [ ] Testable: Each stage can be unit tested in isolation
- [ ] No duplicate code between paths
- [ ] Backwards compatible with existing KaraokeFinalise CLI usage
## Technical Approach
### Architecture: Stage-Based Pipeline
```
video_worker.py (VideoWorkerOrchestrator)
│
├── 1. EncodingStage (interface)
│ ├── LocalEncodingService (wraps KaraokeFinalise encoding methods)
│ └── GCEEncodingService (existing encoding_service.py)
│
├── 2. PackagingStage
│ ├── CDGPackager (extracts from KaraokeFinalise)
│ └── TXTPackager (extracts from KaraokeFinalise)
│
├── 3. DistributionStage
│ ├── YouTubeUploadService (extracts from KaraokeFinalise)
│ ├── DropboxService (existing)
│ ├── GoogleDriveService (existing)
│ └── DiscordService (extracts from KaraokeFinalise)
│
└── 4. OrganizationStage
└── BrandCodeService (extracts from KaraokeFinalise + DropboxService)
```
### Key Design Decisions
1. **Extract, Don't Rewrite**: Pull methods out of KaraokeFinalise into services, don't rewrite from scratch
2. **Keep KaraokeFinalise for CLI**: Local CLI still uses KaraokeFinalise.process() - we extract services that it can also use
3. **Interface-Based Encoding**: EncodingStage interface allows swapping GCE/local without changing orchestration
4. **Existing Service Pattern**: Follow singleton + factory pattern already used in codebase
5. **Local CLI stays sync**: Local karaoke-gen CLI remains synchronous as it is today
### Data Flow
```
JobConfig (from Firestore)
│
▼
┌─────────────────────────────────────────────────────────────────┐
│ VideoWorkerOrchestrator │
│ ┌─────────────────────────────────────────────────────────────┐│
│ │ 1. Setup: Download files, prepare directories ││
│ │ - Download title.mov, end.mov, with_vocals.mkv ││
│ │ - Download instrumental audio ││
│ │ - Prepare temp directory ││
│ └─────────────────────────────────────────────────────────────┘│
│ │ │
│ ▼ │
│ ┌─────────────────────────────────────────────────────────────┐│
│ │ 2. Packaging: CDG/TXT (if enabled) - runs BEFORE encoding ││
│ │ Reason: CDG depends on LRC file which is available ││
│ │ Input: LRC file, instrumental audio ││
│ │ Output: CDG.zip, TXT.zip, .cdg, .mp3 ││
│ └─────────────────────────────────────────────────────────────┘│
│ │ │
│ ▼ │
│ ┌─────────────────────────────────────────────────────────────┐│
│ │ 3. Encoding: GCE or Local (via interface) ││
│ │ Input: title.mov, karaoke.mov, end.mov ││
│ │ Output: 4K lossless, 4K lossy, 720p, MKV ││
│ └─────────────────────────────────────────────────────────────┘│
│ │ │
│ ▼ │
│ ┌─────────────────────────────────────────────────────────────┐│
│ │ 4. Organization: Brand code, folder structure ││
│ │ - Generate brand code (NOMAD-XXXX) ││
│ │ - Prepare full output directory structure ││
│ └─────────────────────────────────────────────────────────────┘│
│ │ │
│ ▼ │
│ ┌─────────────────────────────────────────────────────────────┐│
│ │ 5. Distribution: Upload to destinations ││
│ │ - YouTube upload (if enabled) ││
│ │ - Dropbox upload (if configured) ││
│ │ - Google Drive upload (if configured) ││
│ └─────────────────────────────────────────────────────────────┘│
│ │ │
│ ▼ │
│ ┌─────────────────────────────────────────────────────────────┐│
│ │ 6. Notification: Discord, Email ││
│ │ Input: YouTube URL, brand code, Dropbox link ││
│ └─────────────────────────────────────────────────────────────┘│
└─────────────────────────────────────────────────────────────────┘
│
▼
Result (youtube_url, brand_code, dropbox_link, file_paths, etc.)
```
## Implementation Steps
### Phase 1: Extract Services from KaraokeFinalise (Non-Breaking) ✅ COMPLETE
These services can be used by both the cloud orchestrator AND KaraokeFinalise.
1. [x] **Create `backend/services/youtube_upload_service.py`** ✅
- Extract YouTube upload logic from KaraokeFinalise
- Methods: `upload_video()`, `check_duplicate()`, `delete_video()`, `truncate_title()`
- Supports pre-loaded credentials (server-side) or client secrets file (CLI)
- 23 unit tests with mocked YouTube API
2. [x] **Create `backend/services/discord_service.py`** ✅
- Extract Discord notification from KaraokeFinalise
- Methods: `post_message()`, `post_video_notification()`, `is_enabled()`
- Simple HTTP POST to webhook with validation
- 20 unit tests with mocked requests
3. [x] **Create `backend/services/packaging_service.py`** ✅
- Extract CDG/TXT generation from KaraokeFinalise
- Methods: `create_cdg_package()`, `create_txt_package()`
- Uses lyrics_converter and CDGGenerator (existing dependencies)
- 18 unit tests with temp files
4. [x] **Create `backend/services/local_encoding_service.py`** ✅
- Extract encoding methods from KaraokeFinalise
- Methods: `encode_all_formats()`, individual encoding methods
- Hardware acceleration detection with software fallback
- 28 unit tests with mocked subprocess
### Phase 2: Create Encoding Interface ✅ COMPLETE
5. [x] **Create `backend/services/encoding_interface.py`** ✅
- Define ABC `EncodingBackend` with methods: `encode()`, `is_available()`, `get_status()`
- `EncodingInput` and `EncodingOutput` dataclasses for config/results
- Factory function `get_encoding_backend()` with "local", "gce", "auto" options
6. [x] **`LocalEncodingBackend`** ✅
- Implements `EncodingBackend` interface
- Wraps `LocalEncodingService` with `asyncio.to_thread()`
- 17 unit tests
7. [x] **`GCEEncodingBackend`** ✅
- Implements `EncodingBackend` interface
- Wraps existing `EncodingService` (encoding_service.py)
- Requires GCS paths in options for cloud execution
- 6 additional unit tests
### Phase 3: Create Orchestrator ✅ COMPLETE
8. [x] **Create `backend/workers/video_worker_orchestrator.py`** ✅
- New class that coordinates all stages
- Replaces the branching logic in `generate_video()`
- Methods:
- `__init__(config, job_manager, storage, job_logger)`
- `async run() -> OrchestratorResult`
- `_run_packaging()` - CDG/TXT
- `_run_encoding()` - GCE or local via EncodingBackend interface
- `_run_organization()` - Brand code, folder structure
- `_run_distribution()` - YouTube, Dropbox, GDrive
- `_run_notifications()` - Discord
- 27 unit tests (test_video_worker_orchestrator.py)
9. [x] **Update `backend/workers/video_worker.py`** ✅
- Added `USE_NEW_ORCHESTRATOR` feature flag (default: true)
- `generate_video()` routes to orchestrator or legacy path
- `generate_video_orchestrated()` - new unified pipeline
- `generate_video_legacy()` - original code for rollback
- Legacy GCE and KaraokeFinalise paths preserved for safety
### Phase 4: Update KaraokeFinalise to Use Services (OPTIONAL)
10. [ ] **Update `karaoke_gen/karaoke_finalise/karaoke_finalise.py`**
- Import and use the new services
- Replace inline YouTube/Discord/CDG code with service calls
- Keep process() working for CLI backwards compatibility
- Services become the single source of truth
- **NOTE**: This is optional for MVP - the orchestrator already uses the extracted services directly
### Phase 5: Testing & Validation ✅ COMPLETE
11. [x] **Add unit tests for each service** ✅
- test_youtube_upload_service.py (23 tests)
- test_discord_service.py (20 tests)
- test_packaging_service.py (18 tests)
- test_local_encoding_service.py (28 tests)
- test_encoding_interface.py (23 tests)
- test_video_worker_orchestrator.py (27 tests)
- **Total: 139 new tests**
12. [x] **Full test suite passes** ✅
- 976 tests passed (86 skipped - emulator/integration tests)
- No regressions from changes
13. [ ] **Manual E2E testing** (After deployment)
- [ ] GCE encoding job with YouTube upload enabled
- [ ] Local encoding job with YouTube upload enabled
- [ ] CLI karaoke-gen with all features
- [ ] Job with CDG/TXT enabled
- [ ] Verify output structure matches local CLI
## Files Created/Modified
| File | Action | Description | Status |
|------|--------|-------------|--------|
| `backend/services/youtube_upload_service.py` | Create | YouTube upload logic (~446 lines) | ✅ |
| `backend/services/discord_service.py` | Create | Discord notification service (~167 lines) | ✅ |
| `backend/services/packaging_service.py` | Create | CDG/TXT package generation (~266 lines) | ✅ |
| `backend/services/local_encoding_service.py` | Create | Local FFmpeg encoding service (~512 lines) | ✅ |
| `backend/services/encoding_interface.py` | Create | Abstract interface + backends (~418 lines) | ✅ |
| `backend/workers/video_worker_orchestrator.py` | Create | Stage orchestration logic (~550 lines) | ✅ |
| `backend/workers/video_worker.py` | Modify | Added orchestrator path + feature flag | ✅ |
| `backend/tests/test_youtube_upload_service.py` | Create | 23 unit tests | ✅ |
| `backend/tests/test_discord_service.py` | Create | 20 unit tests | ✅ |
| `backend/tests/test_packaging_service.py` | Create | 18 unit tests | ✅ |
| `backend/tests/test_local_encoding_service.py` | Create | 28 unit tests | ✅ |
| `backend/tests/test_encoding_interface.py` | Create | 23 unit tests | ✅ |
| `backend/tests/test_video_worker_orchestrator.py` | Create | 27 unit tests | ✅ |
| `karaoke_gen/karaoke_finalise/karaoke_finalise.py` | Modify | Use extracted services (optional) | Deferred |
## Testing Strategy
### Unit Tests
- **youtube_upload_service**: Mock YouTube API, test upload/delete/duplicate check
- **discord_service**: Mock requests, test webhook POST
- **packaging_service**: Use temp files, test CDG/TXT generation
- **local_encoding_service**: Mock subprocess, test FFmpeg command construction
### Integration Tests
- **video_worker_orchestrator**: Mock all services, test stage sequencing
- **video_worker**: Test with mocked orchestrator
### E2E Verification
- [ ] GCE encoding job with YouTube upload enabled → YouTube link appears
- [ ] Local encoding job with YouTube upload enabled → Same behavior
- [ ] CLI `karaoke-gen` with all features → No regression
- [ ] Job with CDG/TXT enabled → Packages generated
- [ ] Dropbox output structure → Matches local CLI exactly
## Open Questions
- [x] Should we keep KaraokeFinalise.process() or deprecate it?
- **Decision**: Keep it for CLI backwards compatibility. It will use the new services internally.
- [x] Should local_encoding_service run FFmpeg sync or async?
- **Decision**: Sync implementation, async wrapper via `asyncio.to_thread()` for interface compatibility.
- [x] Order of packaging vs encoding?
- **Decision**: CDG/TXT runs BEFORE encoding (depends on LRC file, not videos). KaraokeFinalise does it this way too.
## Rollback Plan
1. The refactor is additive - new services don't replace old code until orchestrator is wired up
2. Feature flag: `USE_NEW_ORCHESTRATOR=true` to enable new code path
3. If issues found in production:
- Set `USE_NEW_ORCHESTRATOR=false` to revert to old behavior
- Old `generate_video()` code preserved behind feature flag
4. After 1 week stable, remove feature flag and old code
## Success Criteria
1. Job `5b78ad46` scenario (GCE + YouTube) would upload to YouTube
2. All 803+ existing tests pass
3. E2E happy path test passes
4. No regression in local CLI functionality
5. Output structure matches local CLI exactly
6. Code coverage maintained or improved
## Estimated Scope
- **Phase 1** (Extract Services): 4 new service files + tests (~400-600 lines each)
- **Phase 2** (Interface): 1 new file, 2 modifications (~100 lines)
- **Phase 3** (Orchestrator): 2 files (~300-400 lines)
- **Phase 4** (KaraokeFinalise): 1 modification (~100 lines of changes)
- **Phase 5** (Testing): Integration tests + manual verification
This is a medium-sized refactor that can be done incrementally. Each phase produces working, tested code.
+172
View File
@@ -0,0 +1,172 @@
"""
Discord Notification Service.
Provides Discord webhook notification functionality, extracted from KaraokeFinalise
for use by both the cloud backend (video_worker) and local CLI.
This service handles:
- Posting messages to Discord webhooks
- Video upload notifications
- Validation of webhook URLs
"""
import logging
import re
from typing import Optional
import requests
logger = logging.getLogger(__name__)
class DiscordNotificationService:
"""
Service for sending Discord webhook notifications.
Supports posting messages to Discord channels via webhooks,
commonly used to notify when new karaoke videos are uploaded.
"""
# Discord webhook URL pattern
WEBHOOK_URL_PATTERN = re.compile(
r"^https://discord\.com/api/webhooks/\d+/[a-zA-Z0-9_-]+$"
)
def __init__(
self,
webhook_url: Optional[str] = None,
dry_run: bool = False,
logger: Optional[logging.Logger] = None,
):
"""
Initialize the Discord notification service.
Args:
webhook_url: Discord webhook URL for posting notifications
dry_run: If True, log actions without performing them
logger: Optional logger instance
"""
self.webhook_url = webhook_url
self.dry_run = dry_run
self.logger = logger or logging.getLogger(__name__)
# Validate webhook URL if provided
if webhook_url:
self._validate_webhook_url(webhook_url)
def _validate_webhook_url(self, url: str) -> None:
"""
Validate that the webhook URL is a valid Discord webhook URL.
Args:
url: URL to validate
Raises:
ValueError: If URL is not a valid Discord webhook URL
"""
url = url.strip()
if not url.startswith("https://discord.com/api/webhooks/"):
raise ValueError(
f"Invalid Discord webhook URL: {url}. "
"URL must start with 'https://discord.com/api/webhooks/'"
)
def post_message(
self,
message: str,
webhook_url: Optional[str] = None,
) -> bool:
"""
Post a message to a Discord channel via webhook.
Args:
message: The message content to post
webhook_url: Optional webhook URL (uses instance URL if not provided)
Returns:
True if message was posted successfully, False otherwise
Raises:
ValueError: If no webhook URL is available
requests.HTTPError: If the webhook request fails
"""
url = webhook_url or self.webhook_url
if not url:
raise ValueError("No Discord webhook URL provided")
url = url.strip()
self._validate_webhook_url(url)
if self.dry_run:
self.logger.info(
f"DRY RUN: Would post Discord message: '{message}' to {url}"
)
return True
self.logger.info(f"Posting message to Discord webhook...")
data = {"content": message}
response = requests.post(url, json=data, timeout=30)
response.raise_for_status()
self.logger.info("Message posted to Discord successfully")
return True
def post_video_notification(
self,
youtube_url: str,
webhook_url: Optional[str] = None,
) -> bool:
"""
Post a notification about a new video upload.
Args:
youtube_url: The YouTube URL of the uploaded video
webhook_url: Optional webhook URL (uses instance URL if not provided)
Returns:
True if notification was posted successfully, False otherwise
"""
if not youtube_url:
self.logger.info("Skipping Discord notification - no YouTube URL available")
return False
message = f"New upload: {youtube_url}"
return self.post_message(message, webhook_url)
def is_enabled(self) -> bool:
"""
Check if Discord notifications are enabled.
Returns:
True if a valid webhook URL is configured
"""
return bool(self.webhook_url)
# Singleton instance and factory function (following existing service pattern)
_discord_notification_service: Optional[DiscordNotificationService] = None
def get_discord_notification_service(
webhook_url: Optional[str] = None,
**kwargs
) -> DiscordNotificationService:
"""
Get a Discord notification service instance.
Args:
webhook_url: Discord webhook URL for notifications
**kwargs: Additional arguments passed to DiscordNotificationService
Returns:
DiscordNotificationService instance
"""
global _discord_notification_service
# Create new instance if webhook URL changed or not yet created
if _discord_notification_service is None or webhook_url:
_discord_notification_service = DiscordNotificationService(
webhook_url=webhook_url,
**kwargs
)
return _discord_notification_service
+418
View File
@@ -0,0 +1,418 @@
"""
Encoding Interface.
Defines the abstract interface for video encoding backends, allowing
the video worker orchestrator to use either GCE or local encoding
interchangeably.
This follows the Strategy pattern - different encoding implementations
can be swapped without changing the orchestration logic.
"""
import logging
from abc import ABC, abstractmethod
from dataclasses import dataclass, field
from typing import Dict, List, Optional, Any
logger = logging.getLogger(__name__)
@dataclass
class EncodingInput:
"""
Input configuration for video encoding.
Contains all the paths and metadata needed to encode a karaoke video.
"""
# Required input files
title_video_path: str # Title card video (MOV)
karaoke_video_path: str # Main karaoke video with vocals (MOV/MKV)
instrumental_audio_path: str # Instrumental audio track (FLAC)
# Optional input files
end_video_path: Optional[str] = None # End credits video (MOV)
# Metadata for output naming
artist: str = ""
title: str = ""
brand_code: Optional[str] = None
# Output directory
output_dir: str = ""
# Additional options
options: Dict[str, Any] = field(default_factory=dict)
@dataclass
class EncodingOutput:
"""
Output from video encoding.
Contains paths to all generated video files and status information.
"""
success: bool
error_message: Optional[str] = None
# Output file paths (relative to output_dir or absolute)
karaoke_mp4_path: Optional[str] = None # Karaoke video with instrumental audio
with_vocals_mp4_path: Optional[str] = None # With vocals MP4 version
lossless_4k_mp4_path: Optional[str] = None # Final lossless 4K MP4
lossy_4k_mp4_path: Optional[str] = None # Final lossy 4K MP4
lossless_mkv_path: Optional[str] = None # MKV with FLAC (for YouTube)
lossy_720p_mp4_path: Optional[str] = None # 720p web version
# All output files as a dict for convenience
output_files: Dict[str, str] = field(default_factory=dict)
# Encoding metadata
encoding_time_seconds: Optional[float] = None
encoding_backend: Optional[str] = None # "gce" or "local"
class EncodingBackend(ABC):
"""
Abstract base class for video encoding backends.
Implementations:
- GCEEncodingBackend: Cloud-based encoding using Google Compute Engine
- LocalEncodingBackend: Local FFmpeg-based encoding
"""
@property
@abstractmethod
def name(self) -> str:
"""Return the name of this encoding backend."""
pass
@abstractmethod
async def encode(self, input_config: EncodingInput) -> EncodingOutput:
"""
Encode video files according to the input configuration.
This method should:
1. Create karaoke video with instrumental audio
2. Convert with-vocals video to MP4 if needed
3. Encode lossless 4K MP4 (concatenating title + karaoke + end)
4. Encode lossy 4K MP4 with AAC audio
5. Create MKV with FLAC audio for YouTube
6. Encode 720p version for web
Args:
input_config: EncodingInput with all required paths and options
Returns:
EncodingOutput with paths to generated files and status
"""
pass
@abstractmethod
async def is_available(self) -> bool:
"""
Check if this encoding backend is available and configured.
Returns:
True if the backend can be used, False otherwise
"""
pass
@abstractmethod
async def get_status(self) -> Dict[str, Any]:
"""
Get the current status of the encoding backend.
Returns:
Dict with status information (availability, queue length, etc.)
"""
pass
class LocalEncodingBackend(EncodingBackend):
"""
Local FFmpeg-based encoding backend.
Wraps the LocalEncodingService to implement the EncodingBackend interface.
Uses asyncio.to_thread() to run synchronous FFmpeg operations asynchronously.
"""
def __init__(
self,
dry_run: bool = False,
logger: Optional[logging.Logger] = None,
):
"""
Initialize the local encoding backend.
Args:
dry_run: If True, log operations without executing
logger: Optional logger instance
"""
self.dry_run = dry_run
self.logger = logger or logging.getLogger(__name__)
self._service = None
@property
def name(self) -> str:
return "local"
def _get_service(self):
"""Lazy-load the local encoding service."""
if self._service is None:
from backend.services.local_encoding_service import LocalEncodingService
self._service = LocalEncodingService(
dry_run=self.dry_run,
logger=self.logger
)
return self._service
async def encode(self, input_config: EncodingInput) -> EncodingOutput:
"""
Encode video using local FFmpeg.
Args:
input_config: Encoding input configuration
Returns:
EncodingOutput with results
"""
import asyncio
import time
from backend.services.local_encoding_service import EncodingConfig
start_time = time.time()
# Build output file paths
base_name = f"{input_config.artist} - {input_config.title}"
output_dir = input_config.output_dir or "."
import os
config = EncodingConfig(
title_video=input_config.title_video_path,
karaoke_video=input_config.karaoke_video_path,
instrumental_audio=input_config.instrumental_audio_path,
end_video=input_config.end_video_path,
output_karaoke_mp4=os.path.join(output_dir, f"{base_name} (Karaoke).mp4"),
output_with_vocals_mp4=os.path.join(output_dir, f"{base_name} (With Vocals).mp4"),
output_lossless_4k_mp4=os.path.join(output_dir, f"{base_name} (Final Karaoke Lossless 4k).mp4"),
output_lossy_4k_mp4=os.path.join(output_dir, f"{base_name} (Final Karaoke Lossy 4k).mp4"),
output_lossless_mkv=os.path.join(output_dir, f"{base_name} (Final Karaoke Lossless 4k).mkv"),
output_720p_mp4=os.path.join(output_dir, f"{base_name} (Final Karaoke Lossy 720p).mp4"),
)
# Run encoding in thread pool to avoid blocking
service = self._get_service()
result = await asyncio.to_thread(service.encode_all_formats, config)
encoding_time = time.time() - start_time
if result.success:
return EncodingOutput(
success=True,
karaoke_mp4_path=config.output_karaoke_mp4,
with_vocals_mp4_path=config.output_with_vocals_mp4,
lossless_4k_mp4_path=config.output_lossless_4k_mp4,
lossy_4k_mp4_path=config.output_lossy_4k_mp4,
lossless_mkv_path=config.output_lossless_mkv,
lossy_720p_mp4_path=config.output_720p_mp4,
output_files=result.output_files,
encoding_time_seconds=encoding_time,
encoding_backend=self.name
)
else:
return EncodingOutput(
success=False,
error_message=result.error,
output_files=result.output_files,
encoding_time_seconds=encoding_time,
encoding_backend=self.name
)
async def is_available(self) -> bool:
"""Check if local encoding is available (FFmpeg installed)."""
import subprocess
import asyncio
try:
result = await asyncio.to_thread(
subprocess.run,
["ffmpeg", "-version"],
capture_output=True,
timeout=10
)
return result.returncode == 0
except (FileNotFoundError, subprocess.TimeoutExpired):
return False
async def get_status(self) -> Dict[str, Any]:
"""Get local encoding status."""
available = await self.is_available()
service = self._get_service()
return {
"backend": self.name,
"available": available,
"hwaccel_available": service.hwaccel_available if available else False,
"video_encoder": service.video_encoder if available else None,
}
class GCEEncodingBackend(EncodingBackend):
"""
GCE-based encoding backend.
Wraps the existing EncodingService to implement the EncodingBackend interface.
Submits jobs to a remote GCE worker for high-performance encoding.
"""
def __init__(
self,
logger: Optional[logging.Logger] = None,
):
"""
Initialize the GCE encoding backend.
Args:
logger: Optional logger instance
"""
self.logger = logger or logging.getLogger(__name__)
self._service = None
@property
def name(self) -> str:
return "gce"
def _get_service(self):
"""Lazy-load the GCE encoding service."""
if self._service is None:
from backend.services.encoding_service import get_encoding_service
self._service = get_encoding_service()
return self._service
async def encode(self, input_config: EncodingInput) -> EncodingOutput:
"""
Encode video using GCE worker.
Note: GCE encoding requires files to be in GCS, so this method
expects input_config.options to contain 'input_gcs_path' and
'output_gcs_path'.
Args:
input_config: Encoding input configuration
Returns:
EncodingOutput with results
"""
import time
start_time = time.time()
service = self._get_service()
# GCE requires GCS paths
input_gcs_path = input_config.options.get("input_gcs_path")
output_gcs_path = input_config.options.get("output_gcs_path")
job_id = input_config.options.get("job_id", input_config.brand_code or "unknown")
if not input_gcs_path or not output_gcs_path:
return EncodingOutput(
success=False,
error_message="GCE encoding requires input_gcs_path and output_gcs_path in options",
encoding_backend=self.name
)
try:
# Build encoding config
encoding_config = {
"formats": ["mp4_4k_lossless", "mp4_4k_lossy", "mkv_4k", "mp4_720p"],
"artist": input_config.artist,
"title": input_config.title,
}
# Submit and wait for completion
result = await service.encode_videos(
job_id=job_id,
input_gcs_path=input_gcs_path,
output_gcs_path=output_gcs_path,
encoding_config=encoding_config,
)
encoding_time = time.time() - start_time
# Extract output file paths from result
output_files = result.get("output_files", {})
return EncodingOutput(
success=True,
lossless_4k_mp4_path=output_files.get("mp4_4k_lossless"),
lossy_4k_mp4_path=output_files.get("mp4_4k_lossy"),
lossless_mkv_path=output_files.get("mkv_4k"),
lossy_720p_mp4_path=output_files.get("mp4_720p"),
output_files=output_files,
encoding_time_seconds=encoding_time,
encoding_backend=self.name
)
except Exception as e:
encoding_time = time.time() - start_time
self.logger.error(f"GCE encoding failed: {e}")
return EncodingOutput(
success=False,
error_message=str(e),
encoding_time_seconds=encoding_time,
encoding_backend=self.name
)
async def is_available(self) -> bool:
"""Check if GCE encoding is available and configured."""
service = self._get_service()
return service.is_enabled
async def get_status(self) -> Dict[str, Any]:
"""Get GCE encoding status."""
service = self._get_service()
status = {
"backend": self.name,
"available": service.is_enabled,
"configured": service.is_configured,
}
if service.is_configured:
try:
health = await service.health_check()
status["health"] = health
except Exception as e:
status["health_error"] = str(e)
return status
# Factory function to get an encoding backend
def get_encoding_backend(
backend_type: str = "auto",
**kwargs
) -> EncodingBackend:
"""
Get an encoding backend instance.
Args:
backend_type: Type of backend - "local", "gce", or "auto"
**kwargs: Additional arguments for the backend
Returns:
EncodingBackend instance
Raises:
ValueError: If backend_type is unknown
"""
if backend_type == "local":
return LocalEncodingBackend(**kwargs)
elif backend_type == "gce":
return GCEEncodingBackend(**kwargs)
elif backend_type == "auto":
# Check if GCE is available, otherwise use local
gce_backend = GCEEncodingBackend(**kwargs)
if gce_backend._get_service().is_enabled:
return gce_backend
return LocalEncodingBackend(**kwargs)
else:
raise ValueError(f"Unknown encoding backend type: {backend_type}")
+590
View File
@@ -0,0 +1,590 @@
"""
Local Encoding Service.
Provides local FFmpeg-based video encoding functionality, extracted from KaraokeFinalise
for use by both the cloud backend (video_worker as a fallback) and local CLI.
This service handles:
- Concatenating title, karaoke, and end videos
- Encoding to multiple output formats (4K lossless, 4K lossy, 720p, MKV)
- Hardware acceleration detection and fallback to software encoding
- Remuxing video with instrumental audio
"""
import logging
import os
import shlex
import subprocess
import sys
from dataclasses import dataclass
from typing import Optional, List, Dict, Any, Tuple
logger = logging.getLogger(__name__)
@dataclass
class EncodingConfig:
"""Configuration for video encoding."""
title_video: str # Path to title video
karaoke_video: str # Path to karaoke video (with vocals)
instrumental_audio: str # Path to instrumental audio
end_video: Optional[str] = None # Optional path to end credits video
# Output paths
output_karaoke_mp4: Optional[str] = None
output_with_vocals_mp4: Optional[str] = None
output_lossless_4k_mp4: Optional[str] = None
output_lossy_4k_mp4: Optional[str] = None
output_lossless_mkv: Optional[str] = None
output_720p_mp4: Optional[str] = None
@dataclass
class EncodingResult:
"""Result of video encoding operation."""
success: bool
output_files: Dict[str, str]
error: Optional[str] = None
class LocalEncodingService:
"""
Service for local FFmpeg-based video encoding.
Supports hardware acceleration (NVENC) with automatic fallback
to software encoding (libx264) when unavailable.
"""
# MP4 flags for better compatibility
MP4_FLAGS = "-movflags +faststart"
def __init__(
self,
dry_run: bool = False,
log_level: int = logging.INFO,
logger: Optional[logging.Logger] = None,
):
"""
Initialize the local encoding service.
Args:
dry_run: If True, log commands without executing them
log_level: Logging level (affects FFmpeg verbosity)
logger: Optional logger instance
"""
self.dry_run = dry_run
self.log_level = log_level
self.logger = logger or logging.getLogger(__name__)
# Hardware acceleration settings (detected on first use)
self._hwaccel_available: Optional[bool] = None
self._video_encoder: Optional[str] = None
self._scale_filter: Optional[str] = None
self._hwaccel_decode_flags: Optional[str] = None
self._aac_codec: Optional[str] = None
# Build FFmpeg base command
self._ffmpeg_base_command = self._build_ffmpeg_base_command()
def _build_ffmpeg_base_command(self) -> str:
"""Build the FFmpeg base command with appropriate flags."""
# Use bundled FFmpeg for frozen builds
ffmpeg_path = os.path.join(sys._MEIPASS, "ffmpeg.exe") if getattr(sys, "frozen", False) else "ffmpeg"
base_cmd = f"{ffmpeg_path} -hide_banner -nostats -y"
if self.log_level == logging.DEBUG:
base_cmd += " -loglevel verbose"
else:
base_cmd += " -loglevel fatal"
return base_cmd
def _detect_hardware_acceleration(self) -> Tuple[bool, str, str, str, str]:
"""
Detect available hardware acceleration.
Returns:
Tuple of (hwaccel_available, video_encoder, scale_filter,
hwaccel_decode_flags, aac_codec)
"""
self.logger.info("Detecting hardware acceleration capabilities...")
# Try NVENC (NVIDIA)
try:
test_cmd = f"{self._ffmpeg_base_command} -hide_banner -loglevel error " \
f"-f lavfi -i testsrc=duration=1:size=320x240:rate=1 " \
f"-c:v h264_nvenc -f null -"
subprocess.run(
test_cmd, shell=True, check=True,
capture_output=True, timeout=30
)
self.logger.info("NVIDIA NVENC hardware acceleration available")
return (
True,
"h264_nvenc",
"scale_cuda",
"-hwaccel cuda -hwaccel_output_format cuda",
"aac"
)
except (subprocess.CalledProcessError, subprocess.TimeoutExpired, FileNotFoundError):
pass
# No hardware acceleration available
self.logger.info("No hardware acceleration available, using software encoding")
return (False, "libx264", "scale", "", "aac")
@property
def hwaccel_available(self) -> bool:
"""Check if hardware acceleration is available."""
if self._hwaccel_available is None:
self._detect_and_set_hwaccel()
return self._hwaccel_available
@property
def video_encoder(self) -> str:
"""Get the video encoder to use."""
if self._video_encoder is None:
self._detect_and_set_hwaccel()
return self._video_encoder
@property
def scale_filter(self) -> str:
"""Get the scale filter to use."""
if self._scale_filter is None:
self._detect_and_set_hwaccel()
return self._scale_filter
@property
def hwaccel_decode_flags(self) -> str:
"""Get hardware acceleration decode flags."""
if self._hwaccel_decode_flags is None:
self._detect_and_set_hwaccel()
return self._hwaccel_decode_flags
@property
def aac_codec(self) -> str:
"""Get the AAC codec to use."""
if self._aac_codec is None:
self._detect_and_set_hwaccel()
return self._aac_codec
def _detect_and_set_hwaccel(self) -> None:
"""Detect and set hardware acceleration settings."""
(
self._hwaccel_available,
self._video_encoder,
self._scale_filter,
self._hwaccel_decode_flags,
self._aac_codec
) = self._detect_hardware_acceleration()
def _get_nvenc_quality_settings(self, preset: str = "medium") -> str:
"""Get NVENC quality settings for different presets."""
if not self.hwaccel_available:
return ""
settings = {
"lossless": "-preset p7 -tune hq -rc vbr -cq 0 -qmin 0 -qmax 0",
"high": "-preset p7 -tune hq -rc vbr -cq 19 -b:v 0",
"medium": "-preset p4 -tune hq -rc vbr -cq 23 -b:v 0",
"fast": "-preset p2 -tune ll -rc vbr -cq 28 -b:v 0",
}
return settings.get(preset, settings["medium"])
def _execute_command(
self,
command: str,
description: str,
timeout: int = 3600, # 1 hour default
) -> bool:
"""
Execute an FFmpeg command.
Args:
command: The FFmpeg command to execute
description: Human-readable description of the operation
timeout: Command timeout in seconds
Returns:
True if successful, False otherwise
"""
self.logger.info(f"Executing: {description}")
self.logger.debug(f"Command: {command}")
if self.dry_run:
self.logger.info(f"DRY RUN: Would execute: {command}")
return True
try:
result = subprocess.run(
command,
shell=True,
check=True,
capture_output=True,
text=True,
timeout=timeout
)
self.logger.info(f"Completed: {description}")
return True
except subprocess.CalledProcessError as e:
self.logger.error(f"Failed: {description}")
self.logger.error(f"Error: {e.stderr}")
return False
except subprocess.TimeoutExpired:
self.logger.error(f"Timeout: {description}")
return False
def _execute_command_with_fallback(
self,
gpu_command: str,
cpu_command: str,
description: str,
) -> bool:
"""
Execute a command with GPU, falling back to CPU if it fails.
Args:
gpu_command: Hardware-accelerated command
cpu_command: Software fallback command
description: Human-readable description
Returns:
True if successful, False otherwise
"""
if self.hwaccel_available:
self.logger.info(f"Trying hardware-accelerated encoding for: {description}")
if self._execute_command(gpu_command, description):
return True
self.logger.warning(f"Hardware encoding failed, falling back to software")
return self._execute_command(cpu_command, description)
def remux_with_instrumental(
self,
input_video: str,
instrumental_audio: str,
output_file: str,
) -> bool:
"""
Remux video with instrumental audio track.
Args:
input_video: Path to input video file
instrumental_audio: Path to instrumental audio file
output_file: Path for output file
Returns:
True if successful, False otherwise
"""
command = (
f'{self._ffmpeg_base_command} -i "{input_video}" '
f'-i "{instrumental_audio}" -map 0:v -map 1:a -c copy '
f'{self.MP4_FLAGS} "{output_file}"'
)
return self._execute_command(command, "Remuxing with instrumental audio")
def convert_mov_to_mp4(
self,
input_file: str,
output_file: str,
) -> bool:
"""
Convert MOV to MP4 format.
Args:
input_file: Path to input MOV file
output_file: Path for output MP4 file
Returns:
True if successful, False otherwise
"""
gpu_command = (
f'{self._ffmpeg_base_command} {self.hwaccel_decode_flags} -i "{input_file}" '
f'-c:v {self.video_encoder} -c:a copy {self.MP4_FLAGS} "{output_file}"'
)
cpu_command = (
f'{self._ffmpeg_base_command} -i "{input_file}" '
f'-c:v libx264 -c:a copy {self.MP4_FLAGS} "{output_file}"'
)
return self._execute_command_with_fallback(
gpu_command, cpu_command, "Converting MOV to MP4"
)
def encode_lossless_mp4(
self,
title_video: str,
karaoke_video: str,
output_file: str,
end_video: Optional[str] = None,
) -> bool:
"""
Create lossless 4K MP4 by concatenating title, karaoke, and optionally end videos.
Args:
title_video: Path to title video
karaoke_video: Path to karaoke video
output_file: Path for output file
end_video: Optional path to end credits video
Returns:
True if successful, False otherwise
"""
# Quote file paths
title_quoted = shlex.quote(os.path.abspath(title_video))
karaoke_quoted = shlex.quote(os.path.abspath(karaoke_video))
# Build filter and inputs for concatenation
if end_video and os.path.isfile(end_video):
end_quoted = shlex.quote(os.path.abspath(end_video))
extra_input = f"-i {end_quoted}"
concat_filter = (
'-filter_complex "[0:v:0][0:a:0][1:v:0][1:a:0][2:v:0][2:a:0]'
'concat=n=3:v=1:a=1[outv][outa]"'
)
else:
extra_input = ""
concat_filter = (
'-filter_complex "[0:v:0][0:a:0][1:v:0][1:a:0]'
'concat=n=2:v=1:a=1[outv][outa]"'
)
gpu_command = (
f"{self._ffmpeg_base_command} {self.hwaccel_decode_flags} -i {title_quoted} "
f"{self.hwaccel_decode_flags} -i {karaoke_quoted} {extra_input} "
f'{concat_filter} -map "[outv]" -map "[outa]" -c:v {self.video_encoder} '
f'{self._get_nvenc_quality_settings("lossless")} -c:a pcm_s16le '
f'{self.MP4_FLAGS} "{output_file}"'
)
cpu_command = (
f"{self._ffmpeg_base_command} -i {title_quoted} -i {karaoke_quoted} {extra_input} "
f'{concat_filter} -map "[outv]" -map "[outa]" -c:v libx264 -c:a pcm_s16le '
f'{self.MP4_FLAGS} "{output_file}"'
)
return self._execute_command_with_fallback(
gpu_command, cpu_command, "Encoding lossless 4K MP4"
)
def encode_lossy_mp4(
self,
input_file: str,
output_file: str,
) -> bool:
"""
Create lossy 4K MP4 with AAC audio.
Args:
input_file: Path to input file (typically lossless MP4)
output_file: Path for output file
Returns:
True if successful, False otherwise
"""
command = (
f'{self._ffmpeg_base_command} -i "{input_file}" '
f'-c:v copy -c:a {self.aac_codec} -ar 48000 -b:a 320k '
f'{self.MP4_FLAGS} "{output_file}"'
)
return self._execute_command(command, "Encoding lossy 4K MP4 with AAC")
def encode_lossless_mkv(
self,
input_file: str,
output_file: str,
) -> bool:
"""
Create MKV with FLAC audio (for YouTube upload).
Args:
input_file: Path to input file
output_file: Path for output file
Returns:
True if successful, False otherwise
"""
command = (
f'{self._ffmpeg_base_command} -i "{input_file}" '
f'-c:v copy -c:a flac "{output_file}"'
)
return self._execute_command(command, "Creating MKV with FLAC for YouTube")
def encode_720p(
self,
input_file: str,
output_file: str,
) -> bool:
"""
Create 720p MP4 with AAC audio.
Args:
input_file: Path to input file
output_file: Path for output file
Returns:
True if successful, False otherwise
"""
gpu_command = (
f'{self._ffmpeg_base_command} {self.hwaccel_decode_flags} -i "{input_file}" '
f'-c:v {self.video_encoder} -vf "{self.scale_filter}=1280:720" '
f'{self._get_nvenc_quality_settings("medium")} -b:v 2000k '
f'-c:a {self.aac_codec} -ar 48000 -b:a 128k '
f'{self.MP4_FLAGS} "{output_file}"'
)
cpu_command = (
f'{self._ffmpeg_base_command} -i "{input_file}" '
f'-c:v libx264 -vf "scale=1280:720" -b:v 2000k -preset medium -tune animation '
f'-c:a {self.aac_codec} -ar 48000 -b:a 128k '
f'{self.MP4_FLAGS} "{output_file}"'
)
return self._execute_command_with_fallback(
gpu_command, cpu_command, "Encoding 720p MP4"
)
def encode_all_formats(
self,
config: EncodingConfig,
) -> EncodingResult:
"""
Encode video to all output formats.
This performs the full encoding pipeline:
1. Remux with instrumental audio
2. Convert to MP4 if needed
3. Encode lossless 4K MP4 (concatenated)
4. Encode lossy 4K MP4
5. Encode lossless MKV (for YouTube)
6. Encode 720p MP4
Args:
config: Encoding configuration with input/output paths
Returns:
EncodingResult with success status and output file paths
"""
output_files = {}
try:
# Step 1: Remux with instrumental audio
if config.output_karaoke_mp4:
self.logger.info("[Step 1/6] Remuxing video with instrumental audio...")
if not self.remux_with_instrumental(
config.karaoke_video,
config.instrumental_audio,
config.output_karaoke_mp4
):
return EncodingResult(
success=False,
output_files=output_files,
error="Failed to remux with instrumental audio"
)
output_files["karaoke_mp4"] = config.output_karaoke_mp4
# Step 2: Convert to MP4 if needed
if config.output_with_vocals_mp4:
if not config.karaoke_video.endswith(".mp4"):
self.logger.info("[Step 2/6] Converting karaoke video to MP4...")
if not self.convert_mov_to_mp4(
config.karaoke_video,
config.output_with_vocals_mp4
):
return EncodingResult(
success=False,
output_files=output_files,
error="Failed to convert to MP4"
)
output_files["with_vocals_mp4"] = config.output_with_vocals_mp4
else:
self.logger.info("[Step 2/6] Skipped - video already MP4")
# Step 3: Encode lossless 4K MP4
if config.output_lossless_4k_mp4:
self.logger.info("[Step 3/6] Encoding lossless 4K MP4...")
karaoke_for_concat = config.output_karaoke_mp4 or config.karaoke_video
if not self.encode_lossless_mp4(
config.title_video,
karaoke_for_concat,
config.output_lossless_4k_mp4,
config.end_video
):
return EncodingResult(
success=False,
output_files=output_files,
error="Failed to encode lossless 4K MP4"
)
output_files["lossless_4k_mp4"] = config.output_lossless_4k_mp4
# Step 4: Encode lossy 4K MP4
if config.output_lossy_4k_mp4 and config.output_lossless_4k_mp4:
self.logger.info("[Step 4/6] Encoding lossy 4K MP4...")
if not self.encode_lossy_mp4(
config.output_lossless_4k_mp4,
config.output_lossy_4k_mp4
):
return EncodingResult(
success=False,
output_files=output_files,
error="Failed to encode lossy 4K MP4"
)
output_files["lossy_4k_mp4"] = config.output_lossy_4k_mp4
# Step 5: Create MKV with FLAC audio
if config.output_lossless_mkv and config.output_lossless_4k_mp4:
self.logger.info("[Step 5/6] Creating MKV with FLAC audio...")
if not self.encode_lossless_mkv(
config.output_lossless_4k_mp4,
config.output_lossless_mkv
):
return EncodingResult(
success=False,
output_files=output_files,
error="Failed to create MKV"
)
output_files["lossless_mkv"] = config.output_lossless_mkv
# Step 6: Encode 720p version
if config.output_720p_mp4 and config.output_lossless_4k_mp4:
self.logger.info("[Step 6/6] Encoding 720p MP4...")
if not self.encode_720p(
config.output_lossless_4k_mp4,
config.output_720p_mp4
):
return EncodingResult(
success=False,
output_files=output_files,
error="Failed to encode 720p"
)
output_files["720p_mp4"] = config.output_720p_mp4
self.logger.info("All encoding steps completed successfully")
return EncodingResult(success=True, output_files=output_files)
except Exception as e:
self.logger.error(f"Encoding failed with exception: {e}")
return EncodingResult(
success=False,
output_files=output_files,
error=str(e)
)
# Singleton instance and factory function (following existing service pattern)
_local_encoding_service: Optional[LocalEncodingService] = None
def get_local_encoding_service(**kwargs) -> LocalEncodingService:
"""
Get a local encoding service instance.
Args:
**kwargs: Arguments passed to LocalEncodingService
Returns:
LocalEncodingService instance
"""
global _local_encoding_service
if _local_encoding_service is None:
_local_encoding_service = LocalEncodingService(**kwargs)
return _local_encoding_service
+287
View File
@@ -0,0 +1,287 @@
"""
Packaging Service.
Provides CDG and TXT package generation functionality, extracted from KaraokeFinalise
for use by both the cloud backend (video_worker) and local CLI.
This service handles:
- CDG (CD+G) file generation from LRC files
- TXT lyric file generation from LRC files
- ZIP packaging of CDG/MP3 and TXT/MP3 pairs
"""
import logging
import os
import zipfile
from typing import Optional, Dict, Any, Tuple
logger = logging.getLogger(__name__)
class PackagingService:
"""
Service for creating CDG and TXT karaoke packages.
CDG (CD+Graphics) is a format used by karaoke machines.
TXT packages are used by software karaoke players.
Both formats are packaged as ZIP files with an MP3 audio track.
"""
def __init__(
self,
cdg_styles: Optional[Dict[str, Any]] = None,
dry_run: bool = False,
non_interactive: bool = False,
logger: Optional[logging.Logger] = None,
):
"""
Initialize the packaging service.
Args:
cdg_styles: CDG style configuration for CDG generation
dry_run: If True, log actions without performing them
non_interactive: If True, skip interactive prompts
logger: Optional logger instance
"""
self.cdg_styles = cdg_styles
self.dry_run = dry_run
self.non_interactive = non_interactive
self.logger = logger or logging.getLogger(__name__)
def create_cdg_package(
self,
lrc_file: str,
audio_file: str,
output_zip_path: str,
artist: str,
title: str,
output_mp3_path: Optional[str] = None,
output_cdg_path: Optional[str] = None,
) -> Tuple[str, Optional[str], Optional[str]]:
"""
Create a CDG package (ZIP containing CDG and MP3 files).
Args:
lrc_file: Path to the LRC lyrics file
audio_file: Path to the instrumental audio file
output_zip_path: Path for the output ZIP file
artist: Artist name for metadata
title: Song title for metadata
output_mp3_path: Optional path for the extracted MP3 file
output_cdg_path: Optional path for the extracted CDG file
Returns:
Tuple of (zip_path, mp3_path, cdg_path)
Raises:
ValueError: If CDG styles are not configured
FileNotFoundError: If input files are missing
Exception: If CDG generation fails
"""
self.logger.info(f"Creating CDG package for {artist} - {title}")
# Validate inputs
if not os.path.isfile(lrc_file):
raise FileNotFoundError(f"LRC file not found: {lrc_file}")
if not os.path.isfile(audio_file):
raise FileNotFoundError(f"Audio file not found: {audio_file}")
# Check if ZIP already exists
if os.path.isfile(output_zip_path):
if self.non_interactive:
self.logger.info(
f"CDG ZIP exists, will be overwritten: {output_zip_path}"
)
else:
self.logger.info(f"CDG ZIP already exists: {output_zip_path}")
# Check if individual files exist (allows skipping generation)
if output_mp3_path and output_cdg_path:
if os.path.isfile(output_mp3_path) and os.path.isfile(output_cdg_path):
self.logger.info("Found existing MP3 and CDG files, creating ZIP directly")
if not self.dry_run:
self._create_zip_from_files(
output_zip_path,
[(output_mp3_path, os.path.basename(output_mp3_path)),
(output_cdg_path, os.path.basename(output_cdg_path))]
)
return output_zip_path, output_mp3_path, output_cdg_path
if self.dry_run:
self.logger.info(
f"DRY RUN: Would generate CDG package: {output_zip_path}"
)
return output_zip_path, output_mp3_path, output_cdg_path
# Generate CDG files
if self.cdg_styles is None:
raise ValueError(
"CDG styles configuration is required for CDG generation"
)
self.logger.info("Generating CDG and MP3 files")
from lyrics_transcriber.output.cdg import CDGGenerator
output_dir = os.path.dirname(output_zip_path) or os.getcwd()
generator = CDGGenerator(output_dir=output_dir, logger=self.logger)
cdg_file, mp3_file, zip_file = generator.generate_cdg_from_lrc(
lrc_file=lrc_file,
audio_file=audio_file,
title=title,
artist=artist,
cdg_styles=self.cdg_styles,
)
# Rename ZIP to expected output path if different
if os.path.isfile(zip_file) and zip_file != output_zip_path:
os.rename(zip_file, output_zip_path)
self.logger.info(f"Renamed CDG ZIP: {zip_file} -> {output_zip_path}")
if not os.path.isfile(output_zip_path):
raise Exception(f"Failed to create CDG ZIP file: {output_zip_path}")
# Extract the ZIP to get individual files if paths provided
extracted_mp3 = None
extracted_cdg = None
if output_mp3_path or output_cdg_path:
self.logger.info(f"Extracting CDG ZIP file: {output_zip_path}")
with zipfile.ZipFile(output_zip_path, "r") as zip_ref:
zip_ref.extractall(output_dir)
# Find extracted files
if output_mp3_path and os.path.isfile(output_mp3_path):
extracted_mp3 = output_mp3_path
self.logger.info(f"Extracted MP3: {extracted_mp3}")
if output_cdg_path and os.path.isfile(output_cdg_path):
extracted_cdg = output_cdg_path
self.logger.info(f"Extracted CDG: {extracted_cdg}")
self.logger.info(f"CDG package created: {output_zip_path}")
return output_zip_path, extracted_mp3, extracted_cdg
def create_txt_package(
self,
lrc_file: str,
mp3_file: str,
output_zip_path: str,
output_txt_path: Optional[str] = None,
) -> Tuple[str, Optional[str]]:
"""
Create a TXT package (ZIP containing TXT lyrics and MP3 files).
Args:
lrc_file: Path to the LRC lyrics file
mp3_file: Path to the MP3 audio file
output_zip_path: Path for the output ZIP file
output_txt_path: Optional path for the generated TXT file
Returns:
Tuple of (zip_path, txt_path)
Raises:
FileNotFoundError: If input files are missing
Exception: If TXT generation fails
"""
self.logger.info(f"Creating TXT package from {lrc_file}")
# Validate inputs
if not os.path.isfile(lrc_file):
raise FileNotFoundError(f"LRC file not found: {lrc_file}")
if not os.path.isfile(mp3_file):
raise FileNotFoundError(f"MP3 file not found: {mp3_file}")
# Check if ZIP already exists
if os.path.isfile(output_zip_path):
if self.non_interactive:
self.logger.info(
f"TXT ZIP exists, will be overwritten: {output_zip_path}"
)
else:
self.logger.info(f"TXT ZIP already exists: {output_zip_path}")
if self.dry_run:
self.logger.info(
f"DRY RUN: Would create TXT package: {output_zip_path}"
)
return output_zip_path, output_txt_path
# Generate TXT from LRC
self.logger.info(f"Converting LRC to TXT format: {lrc_file}")
from lyrics_converter import LyricsConverter
txt_converter = LyricsConverter(output_format="txt", filepath=lrc_file)
converted_txt = txt_converter.convert_file()
# Write TXT file
if output_txt_path is None:
# Default to same name as ZIP but with .txt extension
output_txt_path = output_zip_path.replace(".zip", ".txt")
with open(output_txt_path, "w") as txt_file:
txt_file.write(converted_txt)
self.logger.info(f"TXT file written: {output_txt_path}")
# Create ZIP containing MP3 and TXT
self.logger.info(f"Creating TXT ZIP: {output_zip_path}")
self._create_zip_from_files(
output_zip_path,
[(mp3_file, os.path.basename(mp3_file)),
(output_txt_path, os.path.basename(output_txt_path))]
)
if not os.path.isfile(output_zip_path):
raise Exception(f"Failed to create TXT ZIP file: {output_zip_path}")
self.logger.info(f"TXT package created: {output_zip_path}")
return output_zip_path, output_txt_path
def _create_zip_from_files(
self,
zip_path: str,
files: list,
) -> None:
"""
Create a ZIP file from a list of files.
Args:
zip_path: Path for the output ZIP file
files: List of (file_path, archive_name) tuples
"""
with zipfile.ZipFile(zip_path, "w") as zipf:
for file_path, archive_name in files:
if os.path.isfile(file_path):
zipf.write(file_path, archive_name)
self.logger.debug(f"Added to ZIP: {archive_name}")
else:
self.logger.warning(f"File not found for ZIP: {file_path}")
# Singleton instance and factory function (following existing service pattern)
_packaging_service: Optional[PackagingService] = None
def get_packaging_service(
cdg_styles: Optional[Dict[str, Any]] = None,
**kwargs
) -> PackagingService:
"""
Get a packaging service instance.
Args:
cdg_styles: CDG style configuration
**kwargs: Additional arguments passed to PackagingService
Returns:
PackagingService instance
"""
global _packaging_service
# Create new instance if settings changed
if _packaging_service is None or cdg_styles:
_packaging_service = PackagingService(
cdg_styles=cdg_styles,
**kwargs
)
return _packaging_service
+445
View File
@@ -0,0 +1,445 @@
"""
YouTube Upload Service.
Provides video upload functionality for YouTube, extracted from KaraokeFinalise
for use by both the cloud backend (video_worker) and local CLI.
This service handles:
- Video uploads with metadata (title, description, tags)
- Thumbnail uploads
- Duplicate video detection and replacement
- Authentication via pre-stored credentials or client secrets file
"""
import logging
import os
from typing import Optional, Dict, Any, Tuple
from thefuzz import fuzz
logger = logging.getLogger(__name__)
class YouTubeUploadService:
"""
Service for uploading videos to YouTube.
Supports two authentication modes:
1. Pre-stored credentials (for server-side/non-interactive use)
2. Client secrets file (for interactive CLI use with browser OAuth)
"""
YOUTUBE_URL_PREFIX = "https://www.youtube.com/watch?v="
SCOPES = ["https://www.googleapis.com/auth/youtube"]
def __init__(
self,
credentials: Optional[Dict[str, Any]] = None,
client_secrets_file: Optional[str] = None,
non_interactive: bool = False,
server_side_mode: bool = False,
dry_run: bool = False,
logger: Optional[logging.Logger] = None,
):
"""
Initialize the YouTube upload service.
Args:
credentials: Pre-stored OAuth credentials dict with keys:
- token: Access token
- refresh_token: Refresh token
- token_uri: Token endpoint URL
- client_id: OAuth client ID
- client_secret: OAuth client secret
- scopes: List of OAuth scopes
client_secrets_file: Path to OAuth client secrets JSON file
(for interactive authentication)
non_interactive: If True, skip interactive prompts
server_side_mode: If True, use exact matching for duplicate detection
dry_run: If True, log actions without performing them
logger: Optional logger instance
"""
self.credentials = credentials
self.client_secrets_file = client_secrets_file
self.non_interactive = non_interactive
self.server_side_mode = server_side_mode
self.dry_run = dry_run
self.logger = logger or logging.getLogger(__name__)
self._youtube_service = None
self._channel_id = None
def _authenticate(self):
"""
Authenticate with YouTube API and return service object.
Returns:
YouTube API service object
Raises:
Exception: If authentication fails or is required in non-interactive mode
"""
from google.auth.transport.requests import Request
from google.oauth2.credentials import Credentials
from googleapiclient.discovery import build
from google_auth_oauthlib.flow import InstalledAppFlow
import pickle
# Check if we have pre-stored credentials (for non-interactive mode)
if self.credentials and self.non_interactive:
try:
creds = Credentials(
token=self.credentials.get('token'),
refresh_token=self.credentials.get('refresh_token'),
token_uri=self.credentials.get('token_uri'),
client_id=self.credentials.get('client_id'),
client_secret=self.credentials.get('client_secret'),
scopes=self.credentials.get('scopes', self.SCOPES)
)
# Refresh token if needed
if creds.expired and creds.refresh_token:
creds.refresh(Request())
youtube = build('youtube', 'v3', credentials=creds)
self.logger.info("Successfully authenticated with YouTube using pre-stored credentials")
return youtube
except Exception as e:
self.logger.error(f"Failed to authenticate with pre-stored credentials: {str(e)}")
# Fall through to original authentication if pre-stored credentials fail
# For non-interactive mode without pre-stored credentials, raise error
if self.non_interactive:
raise Exception(
"YouTube authentication required but running in non-interactive mode. "
"Please pre-authenticate or disable YouTube upload."
)
# Interactive authentication using client secrets file
if not self.client_secrets_file:
raise Exception("No YouTube credentials or client secrets file provided")
# Token file stores the user's access and refresh tokens
youtube_token_file = "/tmp/karaoke-finalise-youtube-token.pickle"
creds = None
# Check if we have saved credentials
if os.path.exists(youtube_token_file):
with open(youtube_token_file, "rb") as token:
creds = pickle.load(token)
# If there are no valid credentials, let the user log in
if not creds or not creds.valid:
if creds and creds.expired and creds.refresh_token:
creds.refresh(Request())
else:
flow = InstalledAppFlow.from_client_secrets_file(
self.client_secrets_file, scopes=self.SCOPES
)
creds = flow.run_local_server(port=0)
# Save the credentials for the next run
with open(youtube_token_file, "wb") as token:
pickle.dump(creds, token)
return build("youtube", "v3", credentials=creds)
@property
def youtube_service(self):
"""Lazy-load YouTube service on first access."""
if self._youtube_service is None:
self._youtube_service = self._authenticate()
return self._youtube_service
def get_channel_id(self) -> Optional[str]:
"""
Get the authenticated user's YouTube channel ID.
Returns:
Channel ID string, or None if not found
"""
if self._channel_id:
return self._channel_id
request = self.youtube_service.channels().list(part="snippet", mine=True)
response = request.execute()
if "items" in response and len(response["items"]) > 0:
self._channel_id = response["items"][0]["id"]
return self._channel_id
return None
def check_duplicate(
self,
title: str,
exact_match: Optional[bool] = None
) -> Tuple[bool, Optional[str], Optional[str]]:
"""
Check if a video with the given title already exists on the channel.
Args:
title: Video title to search for
exact_match: If True, require exact title match. If None, uses
server_side_mode setting (exact in server mode, fuzzy in CLI)
Returns:
Tuple of (exists: bool, video_id: Optional[str], video_url: Optional[str])
"""
channel_id = self.get_channel_id()
if not channel_id:
self.logger.warning("Could not get channel ID, skipping duplicate check")
return False, None, None
use_exact_match = exact_match if exact_match is not None else self.server_side_mode
self.logger.info(f"Searching YouTube channel {channel_id} for title: {title}")
request = self.youtube_service.search().list(
part="snippet",
channelId=channel_id,
q=title,
type="video",
maxResults=10
)
response = request.execute()
if "items" not in response or len(response["items"]) == 0:
self.logger.info(f"No matching video found with title: {title}")
return False, None, None
for item in response["items"]:
# Verify the video actually belongs to our channel
result_channel_id = item["snippet"]["channelId"]
if result_channel_id != channel_id:
self.logger.debug(
f"Skipping video from different channel: {item['snippet']['title']} "
f"(channel: {result_channel_id})"
)
continue
found_title = item["snippet"]["title"]
# Determine if this is a match
if use_exact_match:
is_match = title.lower() == found_title.lower()
similarity_score = 100 if is_match else 0
else:
similarity_score = fuzz.ratio(title.lower(), found_title.lower())
is_match = similarity_score >= 70
if is_match:
video_id = item["id"]["videoId"]
video_url = f"{self.YOUTUBE_URL_PREFIX}{video_id}"
self.logger.info(
f"Potential match found on YouTube channel with ID: {video_id} "
f"and title: {found_title} (similarity: {similarity_score}%)"
)
# In non-interactive mode, return the match directly
if self.non_interactive:
self.logger.info("Non-interactive mode, found a match.")
return True, video_id, video_url
# Interactive confirmation
confirmation = input(
f"Is '{found_title}' the video you are finalising? (y/n): "
).strip().lower()
if confirmation == "y":
return True, video_id, video_url
self.logger.info(f"No matching video found with title: {title}")
return False, None, None
def delete_video(self, video_id: str) -> bool:
"""
Delete a YouTube video by its ID.
Args:
video_id: The YouTube video ID to delete
Returns:
True if successful, False otherwise
"""
self.logger.info(f"Deleting YouTube video with ID: {video_id}")
if self.dry_run:
self.logger.info(f"DRY RUN: Would delete YouTube video with ID: {video_id}")
return True
try:
self.youtube_service.videos().delete(id=video_id).execute()
self.logger.info(f"Successfully deleted YouTube video with ID: {video_id}")
return True
except Exception as e:
self.logger.error(f"Failed to delete YouTube video with ID {video_id}: {e}")
return False
@staticmethod
def truncate_title(title: str, max_length: int = 95) -> str:
"""
Truncate title to the nearest whole word within max_length.
Args:
title: Title to truncate
max_length: Maximum length (default 95 for YouTube)
Returns:
Truncated title with "..." if needed
"""
if len(title) <= max_length:
return title
truncated_title = title[:max_length].rsplit(" ", 1)[0]
if len(truncated_title) < max_length:
truncated_title += " ..."
return truncated_title
def upload_video(
self,
video_path: str,
title: str,
description: str,
thumbnail_path: Optional[str] = None,
tags: Optional[list] = None,
category_id: str = "10", # Music category
privacy_status: str = "public",
replace_existing: bool = False,
) -> Tuple[Optional[str], Optional[str]]:
"""
Upload a video to YouTube with metadata and optional thumbnail.
Args:
video_path: Path to the video file to upload
title: Video title (will be truncated to 95 chars if needed)
description: Video description
thumbnail_path: Optional path to thumbnail image
tags: Optional list of tags/keywords
category_id: YouTube category ID (default "10" for Music)
privacy_status: "public", "private", or "unlisted"
replace_existing: If True, delete existing video with same title
Returns:
Tuple of (video_id, video_url) or (None, None) if upload failed/skipped
"""
from googleapiclient.http import MediaFileUpload
# Truncate title if needed
youtube_title = self.truncate_title(title)
self.logger.info(f"Uploading video to YouTube: {youtube_title}")
if self.dry_run:
self.logger.info(
f"DRY RUN: Would upload {video_path} to YouTube with title: {youtube_title}"
)
return "dry_run_video_id", f"{self.YOUTUBE_URL_PREFIX}dry_run_video_id"
# Check for existing video
should_replace = True if self.server_side_mode else replace_existing
exists, existing_id, existing_url = self.check_duplicate(youtube_title)
if exists:
if should_replace:
self.logger.info(f"Video already exists on YouTube, deleting before re-upload: {existing_url}")
if self.delete_video(existing_id):
self.logger.info("Successfully deleted existing video, proceeding with upload")
else:
self.logger.error("Failed to delete existing video, aborting upload")
return None, None
else:
self.logger.warning(f"Video already exists on YouTube, skipping upload: {existing_url}")
return existing_id, existing_url
# Prepare video metadata
body = {
"snippet": {
"title": youtube_title,
"description": description,
"tags": tags or [],
"categoryId": category_id,
},
"status": {"privacyStatus": privacy_status},
}
# Determine MIME type based on file extension
ext = os.path.splitext(video_path)[1].lower()
mime_type = {
".mkv": "video/x-matroska",
".mp4": "video/mp4",
".mov": "video/quicktime",
".avi": "video/x-msvideo",
}.get(ext, "video/*")
# Upload video
self.logger.info(f"Authenticating with YouTube...")
media_file = MediaFileUpload(video_path, mimetype=mime_type, resumable=True)
self.logger.info(f"Uploading video to YouTube...")
request = self.youtube_service.videos().insert(
part="snippet,status",
body=body,
media_body=media_file
)
response = request.execute()
video_id = response.get("id")
video_url = f"{self.YOUTUBE_URL_PREFIX}{video_id}"
self.logger.info(f"Uploaded video to YouTube: {video_url}")
# Upload thumbnail if provided
if thumbnail_path and os.path.isfile(thumbnail_path):
try:
self.logger.info(f"Uploading thumbnail from: {thumbnail_path}")
media_thumbnail = MediaFileUpload(thumbnail_path, mimetype="image/jpeg")
self.youtube_service.thumbnails().set(
videoId=video_id,
media_body=media_thumbnail
).execute()
self.logger.info(f"Uploaded thumbnail for video ID {video_id}")
except Exception as e:
self.logger.error(f"Failed to upload thumbnail: {e}")
self.logger.warning(
"Video uploaded but thumbnail not set. "
"You may need to set it manually on YouTube."
)
elif thumbnail_path:
self.logger.warning(f"Thumbnail file not found, skipping: {thumbnail_path}")
return video_id, video_url
# Singleton instance and factory function (following existing service pattern)
_youtube_upload_service: Optional[YouTubeUploadService] = None
def get_youtube_upload_service(
credentials: Optional[Dict[str, Any]] = None,
client_secrets_file: Optional[str] = None,
**kwargs
) -> YouTubeUploadService:
"""
Get a YouTube upload service instance.
For server-side use, pass credentials from YouTubeService.
For CLI use, pass client_secrets_file.
Args:
credentials: Pre-stored OAuth credentials dict
client_secrets_file: Path to OAuth client secrets JSON file
**kwargs: Additional arguments passed to YouTubeUploadService
Returns:
YouTubeUploadService instance
"""
global _youtube_upload_service
# Create new instance if credentials/settings changed
if _youtube_upload_service is None or credentials or client_secrets_file:
_youtube_upload_service = YouTubeUploadService(
credentials=credentials,
client_secrets_file=client_secrets_file,
**kwargs
)
return _youtube_upload_service
+244
View File
@@ -0,0 +1,244 @@
"""
Tests for DiscordNotificationService.
Tests cover:
- Service initialization
- Webhook URL validation
- Message posting
- Video notification posting
- Dry run mode
"""
import pytest
from unittest.mock import MagicMock, patch
from backend.services.discord_service import (
DiscordNotificationService,
get_discord_notification_service,
)
class TestDiscordNotificationServiceInit:
"""Test service initialization."""
def test_init_with_webhook_url(self):
"""Test initialization with valid webhook URL."""
url = "https://discord.com/api/webhooks/123456789/abcdef123456"
service = DiscordNotificationService(webhook_url=url)
assert service.webhook_url == url
assert service.is_enabled() is True
def test_init_without_webhook_url(self):
"""Test initialization without webhook URL."""
service = DiscordNotificationService()
assert service.webhook_url is None
assert service.is_enabled() is False
def test_init_with_dry_run(self):
"""Test initialization with dry run mode."""
service = DiscordNotificationService(dry_run=True)
assert service.dry_run is True
def test_init_with_invalid_webhook_url_raises(self):
"""Test that invalid webhook URL raises ValueError."""
with pytest.raises(ValueError) as exc_info:
DiscordNotificationService(webhook_url="https://example.com/webhook")
assert "Invalid Discord webhook URL" in str(exc_info.value)
def test_init_with_non_discord_url_raises(self):
"""Test that non-Discord URL raises ValueError."""
with pytest.raises(ValueError):
DiscordNotificationService(
webhook_url="https://slack.com/api/webhooks/123"
)
class TestDiscordNotificationServiceValidation:
"""Test webhook URL validation."""
def test_validate_valid_webhook_url(self):
"""Test that valid webhook URL passes validation."""
service = DiscordNotificationService()
# Should not raise
service._validate_webhook_url(
"https://discord.com/api/webhooks/123456789/abcdef"
)
def test_validate_invalid_webhook_url(self):
"""Test that invalid webhook URL raises ValueError."""
service = DiscordNotificationService()
with pytest.raises(ValueError) as exc_info:
service._validate_webhook_url("https://example.com/webhook")
assert "Invalid Discord webhook URL" in str(exc_info.value)
def test_validate_strips_whitespace(self):
"""Test that whitespace is stripped from URL."""
service = DiscordNotificationService()
# Should not raise - whitespace is stripped
service._validate_webhook_url(
" https://discord.com/api/webhooks/123456789/abcdef "
)
class TestDiscordNotificationServicePostMessage:
"""Test message posting."""
@patch("backend.services.discord_service.requests.post")
def test_post_message_success(self, mock_post):
"""Test successful message posting."""
mock_response = MagicMock()
mock_response.raise_for_status = MagicMock()
mock_post.return_value = mock_response
url = "https://discord.com/api/webhooks/123456789/abcdef"
service = DiscordNotificationService(webhook_url=url)
result = service.post_message("Test message")
assert result is True
mock_post.assert_called_once_with(
url,
json={"content": "Test message"},
timeout=30
)
@patch("backend.services.discord_service.requests.post")
def test_post_message_with_custom_webhook(self, mock_post):
"""Test message posting with custom webhook URL."""
mock_response = MagicMock()
mock_response.raise_for_status = MagicMock()
mock_post.return_value = mock_response
service = DiscordNotificationService()
custom_url = "https://discord.com/api/webhooks/987654321/fedcba"
result = service.post_message("Test message", webhook_url=custom_url)
assert result is True
mock_post.assert_called_once_with(
custom_url,
json={"content": "Test message"},
timeout=30
)
def test_post_message_no_webhook_raises(self):
"""Test that posting without webhook URL raises ValueError."""
service = DiscordNotificationService()
with pytest.raises(ValueError) as exc_info:
service.post_message("Test message")
assert "No Discord webhook URL provided" in str(exc_info.value)
def test_post_message_dry_run(self):
"""Test message posting in dry run mode."""
url = "https://discord.com/api/webhooks/123456789/abcdef"
service = DiscordNotificationService(webhook_url=url, dry_run=True)
result = service.post_message("Test message")
assert result is True
# In dry run mode, no actual request should be made
@patch("backend.services.discord_service.requests.post")
def test_post_message_http_error(self, mock_post):
"""Test handling of HTTP errors."""
import requests as req
mock_response = MagicMock()
mock_response.raise_for_status.side_effect = req.HTTPError("404 Not Found")
mock_post.return_value = mock_response
url = "https://discord.com/api/webhooks/123456789/abcdef"
service = DiscordNotificationService(webhook_url=url)
with pytest.raises(req.HTTPError):
service.post_message("Test message")
class TestDiscordNotificationServiceVideoNotification:
"""Test video notification posting."""
@patch("backend.services.discord_service.requests.post")
def test_post_video_notification_success(self, mock_post):
"""Test successful video notification."""
mock_response = MagicMock()
mock_response.raise_for_status = MagicMock()
mock_post.return_value = mock_response
url = "https://discord.com/api/webhooks/123456789/abcdef"
service = DiscordNotificationService(webhook_url=url)
result = service.post_video_notification(
"https://www.youtube.com/watch?v=abc123"
)
assert result is True
mock_post.assert_called_once()
call_args = mock_post.call_args
assert "New upload:" in call_args[1]["json"]["content"]
assert "abc123" in call_args[1]["json"]["content"]
def test_post_video_notification_no_youtube_url(self):
"""Test video notification with empty YouTube URL."""
url = "https://discord.com/api/webhooks/123456789/abcdef"
service = DiscordNotificationService(webhook_url=url)
result = service.post_video_notification("")
assert result is False
def test_post_video_notification_none_youtube_url(self):
"""Test video notification with None YouTube URL."""
url = "https://discord.com/api/webhooks/123456789/abcdef"
service = DiscordNotificationService(webhook_url=url)
result = service.post_video_notification(None)
assert result is False
def test_post_video_notification_dry_run(self):
"""Test video notification in dry run mode."""
url = "https://discord.com/api/webhooks/123456789/abcdef"
service = DiscordNotificationService(webhook_url=url, dry_run=True)
result = service.post_video_notification(
"https://www.youtube.com/watch?v=abc123"
)
assert result is True
class TestGetDiscordNotificationService:
"""Test factory function."""
def test_get_service_creates_instance(self):
"""Test that factory function creates a new instance."""
import backend.services.discord_service as module
module._discord_notification_service = None
url = "https://discord.com/api/webhooks/123456789/abcdef"
service = get_discord_notification_service(webhook_url=url)
assert service is not None
assert isinstance(service, DiscordNotificationService)
assert service.webhook_url == url
def test_get_service_without_webhook(self):
"""Test factory function without webhook URL."""
import backend.services.discord_service as module
module._discord_notification_service = None
service = get_discord_notification_service()
assert service is not None
assert service.webhook_url is None
assert service.is_enabled() is False
def test_get_service_with_dry_run(self):
"""Test factory function with dry run mode."""
import backend.services.discord_service as module
module._discord_notification_service = None
url = "https://discord.com/api/webhooks/123456789/abcdef"
service = get_discord_notification_service(webhook_url=url, dry_run=True)
assert service.dry_run is True
+351
View File
@@ -0,0 +1,351 @@
"""
Tests for Encoding Interface.
Tests cover:
- EncodingInput and EncodingOutput dataclasses
- LocalEncodingBackend implementation
- Backend factory function
"""
import pytest
from unittest.mock import MagicMock, patch, AsyncMock
from backend.services.encoding_interface import (
EncodingInput,
EncodingOutput,
EncodingBackend,
LocalEncodingBackend,
GCEEncodingBackend,
get_encoding_backend,
)
class TestEncodingInput:
"""Test EncodingInput dataclass."""
def test_required_fields(self):
"""Test that required fields must be provided."""
input_config = EncodingInput(
title_video_path="/path/title.mov",
karaoke_video_path="/path/karaoke.mov",
instrumental_audio_path="/path/audio.flac",
)
assert input_config.title_video_path == "/path/title.mov"
assert input_config.end_video_path is None
def test_all_fields(self):
"""Test all fields including optional ones."""
input_config = EncodingInput(
title_video_path="/path/title.mov",
karaoke_video_path="/path/karaoke.mov",
instrumental_audio_path="/path/audio.flac",
end_video_path="/path/end.mov",
artist="Test Artist",
title="Test Title",
brand_code="NOMAD-1234",
output_dir="/output",
options={"quality": "high"}
)
assert input_config.artist == "Test Artist"
assert input_config.brand_code == "NOMAD-1234"
assert input_config.options["quality"] == "high"
class TestEncodingOutput:
"""Test EncodingOutput dataclass."""
def test_success_output(self):
"""Test successful output."""
output = EncodingOutput(
success=True,
lossless_4k_mp4_path="/output/video.mp4",
encoding_backend="local"
)
assert output.success is True
assert output.error_message is None
def test_failure_output(self):
"""Test failure output."""
output = EncodingOutput(
success=False,
error_message="Encoding failed",
encoding_backend="gce"
)
assert output.success is False
assert output.error_message == "Encoding failed"
def test_output_files_dict(self):
"""Test output_files dictionary."""
output = EncodingOutput(
success=True,
output_files={
"lossless_4k_mp4": "/output/lossless.mp4",
"720p_mp4": "/output/720p.mp4"
}
)
assert "lossless_4k_mp4" in output.output_files
assert output.output_files["720p_mp4"] == "/output/720p.mp4"
class TestLocalEncodingBackend:
"""Test LocalEncodingBackend implementation."""
def test_name(self):
"""Test backend name."""
backend = LocalEncodingBackend()
assert backend.name == "local"
def test_init_with_dry_run(self):
"""Test initialization with dry run."""
backend = LocalEncodingBackend(dry_run=True)
assert backend.dry_run is True
@pytest.mark.asyncio
@patch("subprocess.run")
async def test_is_available_success(self, mock_run):
"""Test availability check when FFmpeg is installed."""
mock_run.return_value = MagicMock(returncode=0)
backend = LocalEncodingBackend()
available = await backend.is_available()
assert available is True
@pytest.mark.asyncio
@patch("subprocess.run")
async def test_is_available_not_found(self, mock_run):
"""Test availability check when FFmpeg is not installed."""
mock_run.side_effect = FileNotFoundError()
backend = LocalEncodingBackend()
available = await backend.is_available()
assert available is False
@pytest.mark.asyncio
@patch.object(LocalEncodingBackend, "is_available")
@patch.object(LocalEncodingBackend, "_get_service")
async def test_get_status(self, mock_get_service, mock_is_available):
"""Test status retrieval."""
mock_is_available.return_value = True
mock_service = MagicMock()
mock_service.hwaccel_available = True
mock_service.video_encoder = "h264_nvenc"
mock_get_service.return_value = mock_service
backend = LocalEncodingBackend()
status = await backend.get_status()
assert status["backend"] == "local"
assert status["available"] is True
@pytest.mark.asyncio
@patch("asyncio.to_thread")
@patch.object(LocalEncodingBackend, "_get_service")
async def test_encode_success(self, mock_get_service, mock_to_thread):
"""Test successful encoding."""
from backend.services.local_encoding_service import EncodingResult
mock_result = EncodingResult(
success=True,
output_files={"key": "/path/file.mp4"}
)
mock_to_thread.return_value = mock_result
mock_service = MagicMock()
mock_get_service.return_value = mock_service
backend = LocalEncodingBackend()
input_config = EncodingInput(
title_video_path="/input/title.mov",
karaoke_video_path="/input/karaoke.mov",
instrumental_audio_path="/input/audio.flac",
artist="Test Artist",
title="Test Title",
output_dir="/output"
)
output = await backend.encode(input_config)
assert output.success is True
assert output.encoding_backend == "local"
assert output.encoding_time_seconds is not None
@pytest.mark.asyncio
@patch("asyncio.to_thread")
@patch.object(LocalEncodingBackend, "_get_service")
async def test_encode_failure(self, mock_get_service, mock_to_thread):
"""Test encoding failure."""
from backend.services.local_encoding_service import EncodingResult
mock_result = EncodingResult(
success=False,
output_files={},
error="FFmpeg failed"
)
mock_to_thread.return_value = mock_result
mock_service = MagicMock()
mock_get_service.return_value = mock_service
backend = LocalEncodingBackend()
input_config = EncodingInput(
title_video_path="/input/title.mov",
karaoke_video_path="/input/karaoke.mov",
instrumental_audio_path="/input/audio.flac",
artist="Test",
title="Test",
)
output = await backend.encode(input_config)
assert output.success is False
assert output.error_message == "FFmpeg failed"
class TestGCEEncodingBackend:
"""Test GCEEncodingBackend implementation."""
def test_name(self):
"""Test backend name."""
backend = GCEEncodingBackend()
assert backend.name == "gce"
@patch.object(GCEEncodingBackend, "_get_service")
@pytest.mark.asyncio
async def test_is_available_enabled(self, mock_get_service):
"""Test availability when GCE is enabled."""
mock_service = MagicMock()
mock_service.is_enabled = True
mock_get_service.return_value = mock_service
backend = GCEEncodingBackend()
available = await backend.is_available()
assert available is True
@patch.object(GCEEncodingBackend, "_get_service")
@pytest.mark.asyncio
async def test_is_available_disabled(self, mock_get_service):
"""Test availability when GCE is disabled."""
mock_service = MagicMock()
mock_service.is_enabled = False
mock_get_service.return_value = mock_service
backend = GCEEncodingBackend()
available = await backend.is_available()
assert available is False
@patch.object(GCEEncodingBackend, "_get_service")
@pytest.mark.asyncio
async def test_encode_missing_gcs_paths(self, mock_get_service):
"""Test encoding fails without GCS paths."""
backend = GCEEncodingBackend()
input_config = EncodingInput(
title_video_path="/input/title.mov",
karaoke_video_path="/input/karaoke.mov",
instrumental_audio_path="/input/audio.flac",
# Missing GCS paths in options
)
output = await backend.encode(input_config)
assert output.success is False
assert "gcs_path" in output.error_message.lower()
@patch.object(GCEEncodingBackend, "_get_service")
@pytest.mark.asyncio
async def test_encode_success(self, mock_get_service):
"""Test successful GCE encoding."""
mock_service = MagicMock()
mock_service.encode_videos = AsyncMock(return_value={
"status": "complete",
"output_files": {
"mp4_4k_lossless": "gs://bucket/output/lossless.mp4",
"mp4_720p": "gs://bucket/output/720p.mp4",
}
})
mock_get_service.return_value = mock_service
backend = GCEEncodingBackend()
input_config = EncodingInput(
title_video_path="/input/title.mov",
karaoke_video_path="/input/karaoke.mov",
instrumental_audio_path="/input/audio.flac",
artist="Test Artist",
title="Test Title",
options={
"job_id": "test-job",
"input_gcs_path": "gs://bucket/input/",
"output_gcs_path": "gs://bucket/output/",
}
)
output = await backend.encode(input_config)
assert output.success is True
assert output.encoding_backend == "gce"
mock_service.encode_videos.assert_called_once()
@patch.object(GCEEncodingBackend, "_get_service")
@pytest.mark.asyncio
async def test_encode_failure(self, mock_get_service):
"""Test GCE encoding failure handling."""
mock_service = MagicMock()
mock_service.encode_videos = AsyncMock(side_effect=Exception("GCE worker error"))
mock_get_service.return_value = mock_service
backend = GCEEncodingBackend()
input_config = EncodingInput(
title_video_path="/input/title.mov",
karaoke_video_path="/input/karaoke.mov",
instrumental_audio_path="/input/audio.flac",
options={
"job_id": "test-job",
"input_gcs_path": "gs://bucket/input/",
"output_gcs_path": "gs://bucket/output/",
}
)
output = await backend.encode(input_config)
assert output.success is False
assert "GCE worker error" in output.error_message
class TestGetEncodingBackend:
"""Test encoding backend factory function."""
def test_get_local_backend(self):
"""Test getting local backend."""
backend = get_encoding_backend("local")
assert isinstance(backend, LocalEncodingBackend)
assert backend.name == "local"
@patch.object(GCEEncodingBackend, "_get_service")
def test_get_auto_backend_gce_disabled(self, mock_get_service):
"""Test getting auto backend falls back to local when GCE disabled."""
mock_service = MagicMock()
mock_service.is_enabled = False
mock_get_service.return_value = mock_service
backend = get_encoding_backend("auto")
assert isinstance(backend, LocalEncodingBackend)
def test_get_local_backend_with_options(self):
"""Test getting local backend with options."""
backend = get_encoding_backend("local", dry_run=True)
assert backend.dry_run is True
def test_get_gce_backend(self):
"""Test getting GCE backend."""
backend = get_encoding_backend("gce")
assert isinstance(backend, GCEEncodingBackend)
assert backend.name == "gce"
def test_get_unknown_backend_raises(self):
"""Test that unknown backend raises ValueError."""
with pytest.raises(ValueError) as exc_info:
get_encoding_backend("unknown")
assert "Unknown encoding backend type" in str(exc_info.value)
@@ -0,0 +1,423 @@
"""
Tests for LocalEncodingService.
Tests cover:
- Service initialization
- Hardware acceleration detection
- FFmpeg command execution
- Individual encoding methods
- Full encoding pipeline
- Dry run mode
"""
import os
import tempfile
import pytest
from unittest.mock import MagicMock, patch, call
from backend.services.local_encoding_service import (
LocalEncodingService,
EncodingConfig,
EncodingResult,
get_local_encoding_service,
)
class TestLocalEncodingServiceInit:
"""Test service initialization."""
def test_init_default_values(self):
"""Test default initialization."""
service = LocalEncodingService()
assert service.dry_run is False
assert "ffmpeg" in service._ffmpeg_base_command
def test_init_with_dry_run(self):
"""Test initialization with dry run mode."""
service = LocalEncodingService(dry_run=True)
assert service.dry_run is True
def test_init_with_debug_logging(self):
"""Test initialization with debug logging level."""
import logging
service = LocalEncodingService(log_level=logging.DEBUG)
assert "-loglevel verbose" in service._ffmpeg_base_command
def test_init_with_info_logging(self):
"""Test initialization with info logging level."""
import logging
service = LocalEncodingService(log_level=logging.INFO)
assert "-loglevel fatal" in service._ffmpeg_base_command
class TestLocalEncodingServiceHWAccel:
"""Test hardware acceleration detection."""
@patch("subprocess.run")
def test_detect_nvenc_available(self, mock_run):
"""Test NVENC detection when available."""
mock_run.return_value = MagicMock(returncode=0)
service = LocalEncodingService()
# Force detection
service._detect_and_set_hwaccel()
assert service._hwaccel_available is True
assert service._video_encoder == "h264_nvenc"
assert service._scale_filter == "scale_cuda"
@patch("subprocess.run")
def test_detect_no_hwaccel(self, mock_run):
"""Test fallback when no hardware acceleration available."""
import subprocess
mock_run.side_effect = subprocess.CalledProcessError(1, "test")
service = LocalEncodingService()
# Force detection
service._detect_and_set_hwaccel()
assert service._hwaccel_available is False
assert service._video_encoder == "libx264"
assert service._scale_filter == "scale"
def test_nvenc_quality_settings(self):
"""Test NVENC quality settings for different presets."""
service = LocalEncodingService()
service._hwaccel_available = True
lossless = service._get_nvenc_quality_settings("lossless")
assert "cq 0" in lossless
medium = service._get_nvenc_quality_settings("medium")
assert "p4" in medium
def test_nvenc_quality_settings_disabled(self):
"""Test NVENC quality settings when hwaccel is disabled."""
service = LocalEncodingService()
service._hwaccel_available = False
settings = service._get_nvenc_quality_settings("lossless")
assert settings == ""
class TestLocalEncodingServiceExecuteCommand:
"""Test command execution."""
@patch("subprocess.run")
def test_execute_command_success(self, mock_run):
"""Test successful command execution."""
mock_run.return_value = MagicMock(returncode=0)
service = LocalEncodingService()
result = service._execute_command("echo test", "Test command")
assert result is True
mock_run.assert_called_once()
@patch("subprocess.run")
def test_execute_command_failure(self, mock_run):
"""Test command execution failure."""
import subprocess
mock_run.side_effect = subprocess.CalledProcessError(
1, "test", stderr="Error message"
)
service = LocalEncodingService()
result = service._execute_command("echo test", "Test command")
assert result is False
@patch("subprocess.run")
def test_execute_command_timeout(self, mock_run):
"""Test command execution timeout."""
import subprocess
mock_run.side_effect = subprocess.TimeoutExpired("test", 30)
service = LocalEncodingService()
result = service._execute_command("echo test", "Test command", timeout=30)
assert result is False
def test_execute_command_dry_run(self):
"""Test command execution in dry run mode."""
service = LocalEncodingService(dry_run=True)
result = service._execute_command("echo test", "Test command")
assert result is True
# No actual subprocess should be called in dry run mode
class TestLocalEncodingServiceEncodingMethods:
"""Test individual encoding methods."""
@patch.object(LocalEncodingService, "_execute_command")
def test_remux_with_instrumental(self, mock_execute):
"""Test remuxing with instrumental audio."""
mock_execute.return_value = True
service = LocalEncodingService()
result = service.remux_with_instrumental(
"/input/video.mov",
"/input/audio.flac",
"/output/video.mp4"
)
assert result is True
mock_execute.assert_called_once()
call_args = mock_execute.call_args[0][0]
assert "/input/video.mov" in call_args
assert "/input/audio.flac" in call_args
assert "-map 0:v -map 1:a" in call_args
@patch.object(LocalEncodingService, "_execute_command_with_fallback")
def test_convert_mov_to_mp4(self, mock_execute):
"""Test MOV to MP4 conversion."""
mock_execute.return_value = True
service = LocalEncodingService()
result = service.convert_mov_to_mp4(
"/input/video.mov",
"/output/video.mp4"
)
assert result is True
mock_execute.assert_called_once()
@patch.object(LocalEncodingService, "_execute_command_with_fallback")
def test_encode_lossless_mp4_without_end(self, mock_execute):
"""Test lossless 4K MP4 encoding without end credits."""
mock_execute.return_value = True
service = LocalEncodingService()
result = service.encode_lossless_mp4(
"/input/title.mov",
"/input/karaoke.mp4",
"/output/lossless.mp4"
)
assert result is True
mock_execute.assert_called_once()
call_args = mock_execute.call_args
assert "concat=n=2" in call_args[0][0] # GPU command
@patch.object(LocalEncodingService, "_execute_command_with_fallback")
def test_encode_lossless_mp4_with_end(self, mock_execute):
"""Test lossless 4K MP4 encoding with end credits."""
mock_execute.return_value = True
with tempfile.TemporaryDirectory() as tmpdir:
end_file = os.path.join(tmpdir, "end.mov")
with open(end_file, "w") as f:
f.write("fake video")
service = LocalEncodingService()
result = service.encode_lossless_mp4(
"/input/title.mov",
"/input/karaoke.mp4",
"/output/lossless.mp4",
end_video=end_file
)
assert result is True
call_args = mock_execute.call_args
assert "concat=n=3" in call_args[0][0] # 3 videos
@patch.object(LocalEncodingService, "_execute_command")
def test_encode_lossy_mp4(self, mock_execute):
"""Test lossy 4K MP4 encoding."""
mock_execute.return_value = True
service = LocalEncodingService()
result = service.encode_lossy_mp4(
"/input/lossless.mp4",
"/output/lossy.mp4"
)
assert result is True
call_args = mock_execute.call_args[0][0]
assert "-c:v copy" in call_args
assert "aac" in call_args.lower()
@patch.object(LocalEncodingService, "_execute_command")
def test_encode_lossless_mkv(self, mock_execute):
"""Test MKV encoding with FLAC audio."""
mock_execute.return_value = True
service = LocalEncodingService()
result = service.encode_lossless_mkv(
"/input/lossless.mp4",
"/output/video.mkv"
)
assert result is True
call_args = mock_execute.call_args[0][0]
assert "-c:v copy" in call_args
assert "-c:a flac" in call_args
@patch.object(LocalEncodingService, "_execute_command_with_fallback")
def test_encode_720p(self, mock_execute):
"""Test 720p encoding."""
mock_execute.return_value = True
service = LocalEncodingService()
result = service.encode_720p(
"/input/lossless.mp4",
"/output/720p.mp4"
)
assert result is True
call_args = mock_execute.call_args
# Should contain scale filter
assert "1280:720" in call_args[0][0] or "1280:720" in call_args[0][1]
class TestLocalEncodingServiceFullPipeline:
"""Test full encoding pipeline."""
@patch.object(LocalEncodingService, "remux_with_instrumental")
@patch.object(LocalEncodingService, "convert_mov_to_mp4")
@patch.object(LocalEncodingService, "encode_lossless_mp4")
@patch.object(LocalEncodingService, "encode_lossy_mp4")
@patch.object(LocalEncodingService, "encode_lossless_mkv")
@patch.object(LocalEncodingService, "encode_720p")
def test_encode_all_formats_success(
self, mock_720p, mock_mkv, mock_lossy, mock_lossless, mock_convert, mock_remux
):
"""Test successful full encoding pipeline."""
mock_remux.return_value = True
mock_convert.return_value = True
mock_lossless.return_value = True
mock_lossy.return_value = True
mock_mkv.return_value = True
mock_720p.return_value = True
service = LocalEncodingService()
config = EncodingConfig(
title_video="/input/title.mov",
karaoke_video="/input/karaoke.mov",
instrumental_audio="/input/instrumental.flac",
output_karaoke_mp4="/output/karaoke.mp4",
output_with_vocals_mp4="/output/with_vocals.mp4",
output_lossless_4k_mp4="/output/lossless_4k.mp4",
output_lossy_4k_mp4="/output/lossy_4k.mp4",
output_lossless_mkv="/output/lossless.mkv",
output_720p_mp4="/output/720p.mp4",
)
result = service.encode_all_formats(config)
assert result.success is True
assert "karaoke_mp4" in result.output_files
assert "lossless_4k_mp4" in result.output_files
assert "720p_mp4" in result.output_files
@patch.object(LocalEncodingService, "remux_with_instrumental")
def test_encode_all_formats_failure_early(self, mock_remux):
"""Test encoding pipeline failure at early step."""
mock_remux.return_value = False
service = LocalEncodingService()
config = EncodingConfig(
title_video="/input/title.mov",
karaoke_video="/input/karaoke.mov",
instrumental_audio="/input/instrumental.flac",
output_karaoke_mp4="/output/karaoke.mp4",
output_lossless_4k_mp4="/output/lossless_4k.mp4",
)
result = service.encode_all_formats(config)
assert result.success is False
assert "Failed to remux" in result.error
def test_encode_all_formats_dry_run(self):
"""Test encoding pipeline in dry run mode."""
service = LocalEncodingService(dry_run=True)
config = EncodingConfig(
title_video="/input/title.mov",
karaoke_video="/input/karaoke.mp4", # Already MP4
instrumental_audio="/input/instrumental.flac",
output_karaoke_mp4="/output/karaoke.mp4",
output_lossless_4k_mp4="/output/lossless_4k.mp4",
output_lossy_4k_mp4="/output/lossy_4k.mp4",
output_lossless_mkv="/output/lossless.mkv",
output_720p_mp4="/output/720p.mp4",
)
result = service.encode_all_formats(config)
# In dry run mode, all operations should "succeed"
assert result.success is True
class TestEncodingConfig:
"""Test EncodingConfig dataclass."""
def test_config_required_fields(self):
"""Test that required fields must be provided."""
config = EncodingConfig(
title_video="/path/title.mov",
karaoke_video="/path/karaoke.mov",
instrumental_audio="/path/audio.flac",
)
assert config.title_video == "/path/title.mov"
assert config.end_video is None # Optional field
def test_config_all_fields(self):
"""Test config with all fields."""
config = EncodingConfig(
title_video="/path/title.mov",
karaoke_video="/path/karaoke.mov",
instrumental_audio="/path/audio.flac",
end_video="/path/end.mov",
output_karaoke_mp4="/output/karaoke.mp4",
output_720p_mp4="/output/720p.mp4",
)
assert config.end_video == "/path/end.mov"
assert config.output_karaoke_mp4 == "/output/karaoke.mp4"
class TestEncodingResult:
"""Test EncodingResult dataclass."""
def test_result_success(self):
"""Test successful result."""
result = EncodingResult(
success=True,
output_files={"key": "/path/file.mp4"}
)
assert result.success is True
assert result.error is None
def test_result_failure(self):
"""Test failure result."""
result = EncodingResult(
success=False,
output_files={},
error="Something went wrong"
)
assert result.success is False
assert result.error == "Something went wrong"
class TestGetLocalEncodingService:
"""Test factory function."""
def test_get_service_creates_instance(self):
"""Test that factory function creates a new instance."""
import backend.services.local_encoding_service as module
module._local_encoding_service = None
service = get_local_encoding_service()
assert service is not None
assert isinstance(service, LocalEncodingService)
def test_get_service_with_dry_run(self):
"""Test factory function with dry run option."""
import backend.services.local_encoding_service as module
module._local_encoding_service = None
service = get_local_encoding_service(dry_run=True)
assert service.dry_run is True
+382
View File
@@ -0,0 +1,382 @@
"""
Tests for PackagingService.
Tests cover:
- Service initialization
- CDG package creation
- TXT package creation
- Dry run mode
- Error handling
"""
import os
import zipfile
import tempfile
import pytest
from unittest.mock import MagicMock, patch
from backend.services.packaging_service import (
PackagingService,
get_packaging_service,
)
class TestPackagingServiceInit:
"""Test service initialization."""
def test_init_default_values(self):
"""Test default initialization."""
service = PackagingService()
assert service.cdg_styles is None
assert service.dry_run is False
assert service.non_interactive is False
def test_init_with_cdg_styles(self):
"""Test initialization with CDG styles."""
styles = {"font_size": 24, "bg_color": "black"}
service = PackagingService(cdg_styles=styles)
assert service.cdg_styles == styles
def test_init_with_dry_run(self):
"""Test initialization with dry run mode."""
service = PackagingService(dry_run=True)
assert service.dry_run is True
def test_init_with_non_interactive(self):
"""Test initialization with non-interactive mode."""
service = PackagingService(non_interactive=True)
assert service.non_interactive is True
class TestPackagingServiceCreateZipFromFiles:
"""Test internal ZIP creation."""
def test_create_zip_from_files(self):
"""Test creating a ZIP file from multiple files."""
with tempfile.TemporaryDirectory() as tmpdir:
# Create test files
file1 = os.path.join(tmpdir, "test1.txt")
file2 = os.path.join(tmpdir, "test2.txt")
zip_path = os.path.join(tmpdir, "output.zip")
with open(file1, "w") as f:
f.write("content1")
with open(file2, "w") as f:
f.write("content2")
service = PackagingService()
service._create_zip_from_files(
zip_path,
[(file1, "test1.txt"), (file2, "test2.txt")]
)
assert os.path.isfile(zip_path)
with zipfile.ZipFile(zip_path, "r") as zf:
names = zf.namelist()
assert "test1.txt" in names
assert "test2.txt" in names
class TestPackagingServiceCDGPackage:
"""Test CDG package creation."""
def test_create_cdg_package_missing_lrc_file(self):
"""Test that missing LRC file raises FileNotFoundError."""
service = PackagingService()
with pytest.raises(FileNotFoundError) as exc_info:
service.create_cdg_package(
lrc_file="/nonexistent/file.lrc",
audio_file="/some/audio.flac",
output_zip_path="/output/test.zip",
artist="Test Artist",
title="Test Title",
)
assert "LRC file not found" in str(exc_info.value)
def test_create_cdg_package_missing_audio_file(self):
"""Test that missing audio file raises FileNotFoundError."""
with tempfile.TemporaryDirectory() as tmpdir:
lrc_file = os.path.join(tmpdir, "test.lrc")
with open(lrc_file, "w") as f:
f.write("[00:00.00]Test lyrics")
service = PackagingService()
with pytest.raises(FileNotFoundError) as exc_info:
service.create_cdg_package(
lrc_file=lrc_file,
audio_file="/nonexistent/audio.flac",
output_zip_path="/output/test.zip",
artist="Test Artist",
title="Test Title",
)
assert "Audio file not found" in str(exc_info.value)
def test_create_cdg_package_no_styles_raises(self):
"""Test that CDG generation without styles raises ValueError."""
with tempfile.TemporaryDirectory() as tmpdir:
lrc_file = os.path.join(tmpdir, "test.lrc")
audio_file = os.path.join(tmpdir, "test.flac")
zip_path = os.path.join(tmpdir, "output.zip")
with open(lrc_file, "w") as f:
f.write("[00:00.00]Test lyrics")
with open(audio_file, "w") as f:
f.write("fake audio")
# Service without CDG styles
service = PackagingService(cdg_styles=None)
with pytest.raises(ValueError) as exc_info:
service.create_cdg_package(
lrc_file=lrc_file,
audio_file=audio_file,
output_zip_path=zip_path,
artist="Test Artist",
title="Test Title",
)
assert "CDG styles configuration is required" in str(exc_info.value)
def test_create_cdg_package_dry_run(self):
"""Test CDG package creation in dry run mode."""
with tempfile.TemporaryDirectory() as tmpdir:
lrc_file = os.path.join(tmpdir, "test.lrc")
audio_file = os.path.join(tmpdir, "test.flac")
zip_path = os.path.join(tmpdir, "output.zip")
with open(lrc_file, "w") as f:
f.write("[00:00.00]Test lyrics")
with open(audio_file, "w") as f:
f.write("fake audio")
service = PackagingService(dry_run=True)
result = service.create_cdg_package(
lrc_file=lrc_file,
audio_file=audio_file,
output_zip_path=zip_path,
artist="Test Artist",
title="Test Title",
)
# In dry run, ZIP should not be created
assert not os.path.isfile(zip_path)
assert result[0] == zip_path
def test_create_cdg_package_existing_files(self):
"""Test CDG package creation with existing MP3 and CDG files."""
with tempfile.TemporaryDirectory() as tmpdir:
lrc_file = os.path.join(tmpdir, "test.lrc")
audio_file = os.path.join(tmpdir, "test.flac")
mp3_file = os.path.join(tmpdir, "test.mp3")
cdg_file = os.path.join(tmpdir, "test.cdg")
zip_path = os.path.join(tmpdir, "output.zip")
# Create all files
with open(lrc_file, "w") as f:
f.write("[00:00.00]Test lyrics")
with open(audio_file, "w") as f:
f.write("fake audio")
with open(mp3_file, "w") as f:
f.write("fake mp3")
with open(cdg_file, "w") as f:
f.write("fake cdg")
service = PackagingService()
result = service.create_cdg_package(
lrc_file=lrc_file,
audio_file=audio_file,
output_zip_path=zip_path,
artist="Test Artist",
title="Test Title",
output_mp3_path=mp3_file,
output_cdg_path=cdg_file,
)
# ZIP should be created from existing files
assert os.path.isfile(zip_path)
assert result[0] == zip_path
assert result[1] == mp3_file
assert result[2] == cdg_file
@patch("lyrics_transcriber.output.cdg.CDGGenerator")
def test_create_cdg_package_with_generator(self, mock_cdg_generator_class):
"""Test CDG package creation using CDGGenerator."""
with tempfile.TemporaryDirectory() as tmpdir:
lrc_file = os.path.join(tmpdir, "test.lrc")
audio_file = os.path.join(tmpdir, "test.flac")
zip_path = os.path.join(tmpdir, "output.zip")
generated_zip = os.path.join(tmpdir, "generated.zip")
with open(lrc_file, "w") as f:
f.write("[00:00.00]Test lyrics")
with open(audio_file, "w") as f:
f.write("fake audio")
# Create the zip file that would be generated
with zipfile.ZipFile(generated_zip, "w") as zf:
zf.writestr("test.mp3", "fake mp3")
zf.writestr("test.cdg", "fake cdg")
mock_generator = MagicMock()
mock_generator.generate_cdg_from_lrc.return_value = (
"test.cdg", "test.mp3", generated_zip
)
mock_cdg_generator_class.return_value = mock_generator
styles = {"font_size": 24}
service = PackagingService(cdg_styles=styles)
result = service.create_cdg_package(
lrc_file=lrc_file,
audio_file=audio_file,
output_zip_path=zip_path,
artist="Test Artist",
title="Test Title",
)
mock_generator.generate_cdg_from_lrc.assert_called_once_with(
lrc_file=lrc_file,
audio_file=audio_file,
title="Test Title",
artist="Test Artist",
cdg_styles=styles,
)
assert os.path.isfile(zip_path)
class TestPackagingServiceTXTPackage:
"""Test TXT package creation."""
def test_create_txt_package_missing_lrc_file(self):
"""Test that missing LRC file raises FileNotFoundError."""
service = PackagingService()
with pytest.raises(FileNotFoundError) as exc_info:
service.create_txt_package(
lrc_file="/nonexistent/file.lrc",
mp3_file="/some/audio.mp3",
output_zip_path="/output/test.zip",
)
assert "LRC file not found" in str(exc_info.value)
def test_create_txt_package_missing_mp3_file(self):
"""Test that missing MP3 file raises FileNotFoundError."""
with tempfile.TemporaryDirectory() as tmpdir:
lrc_file = os.path.join(tmpdir, "test.lrc")
with open(lrc_file, "w") as f:
f.write("[00:00.00]Test lyrics")
service = PackagingService()
with pytest.raises(FileNotFoundError) as exc_info:
service.create_txt_package(
lrc_file=lrc_file,
mp3_file="/nonexistent/audio.mp3",
output_zip_path="/output/test.zip",
)
assert "MP3 file not found" in str(exc_info.value)
def test_create_txt_package_dry_run(self):
"""Test TXT package creation in dry run mode."""
with tempfile.TemporaryDirectory() as tmpdir:
lrc_file = os.path.join(tmpdir, "test.lrc")
mp3_file = os.path.join(tmpdir, "test.mp3")
zip_path = os.path.join(tmpdir, "output.zip")
with open(lrc_file, "w") as f:
f.write("[00:00.00]Test lyrics")
with open(mp3_file, "w") as f:
f.write("fake mp3")
service = PackagingService(dry_run=True)
result = service.create_txt_package(
lrc_file=lrc_file,
mp3_file=mp3_file,
output_zip_path=zip_path,
)
# In dry run, ZIP should not be created
assert not os.path.isfile(zip_path)
assert result[0] == zip_path
@patch("lyrics_converter.LyricsConverter")
def test_create_txt_package_success(self, mock_converter_class):
"""Test successful TXT package creation."""
with tempfile.TemporaryDirectory() as tmpdir:
lrc_file = os.path.join(tmpdir, "test.lrc")
mp3_file = os.path.join(tmpdir, "test.mp3")
zip_path = os.path.join(tmpdir, "output.zip")
txt_path = os.path.join(tmpdir, "output.txt")
with open(lrc_file, "w") as f:
f.write("[00:00.00]Test lyrics")
with open(mp3_file, "w") as f:
f.write("fake mp3")
mock_converter = MagicMock()
mock_converter.convert_file.return_value = "Converted lyrics text"
mock_converter_class.return_value = mock_converter
service = PackagingService()
result = service.create_txt_package(
lrc_file=lrc_file,
mp3_file=mp3_file,
output_zip_path=zip_path,
output_txt_path=txt_path,
)
mock_converter_class.assert_called_once_with(
output_format="txt",
filepath=lrc_file
)
mock_converter.convert_file.assert_called_once()
assert os.path.isfile(zip_path)
assert os.path.isfile(txt_path)
assert result[0] == zip_path
assert result[1] == txt_path
# Verify ZIP contents
with zipfile.ZipFile(zip_path, "r") as zf:
names = zf.namelist()
assert "test.mp3" in names
assert "output.txt" in names
class TestGetPackagingService:
"""Test factory function."""
def test_get_service_creates_instance(self):
"""Test that factory function creates a new instance."""
import backend.services.packaging_service as module
module._packaging_service = None
service = get_packaging_service()
assert service is not None
assert isinstance(service, PackagingService)
def test_get_service_with_cdg_styles(self):
"""Test factory function with CDG styles."""
import backend.services.packaging_service as module
module._packaging_service = None
styles = {"font_size": 24}
service = get_packaging_service(cdg_styles=styles)
assert service.cdg_styles == styles
def test_get_service_with_options(self):
"""Test factory function with additional options."""
import backend.services.packaging_service as module
module._packaging_service = None
service = get_packaging_service(dry_run=True, non_interactive=True)
assert service.dry_run is True
assert service.non_interactive is True
@@ -0,0 +1,698 @@
"""
Tests for Video Worker Orchestrator.
Tests cover:
- OrchestratorConfig and OrchestratorResult dataclasses
- VideoWorkerOrchestrator initialization
- Individual stage methods
- Full pipeline execution
- Error handling and recovery
"""
import os
import tempfile
import pytest
from unittest.mock import MagicMock, patch, AsyncMock
from dataclasses import asdict
from backend.workers.video_worker_orchestrator import (
OrchestratorConfig,
OrchestratorResult,
VideoWorkerOrchestrator,
create_orchestrator_config_from_job,
)
class TestOrchestratorConfig:
"""Test OrchestratorConfig dataclass."""
def test_required_fields(self):
"""Test that required fields must be provided."""
config = OrchestratorConfig(
job_id="test-job",
artist="Test Artist",
title="Test Title",
title_video_path="/path/title.mov",
karaoke_video_path="/path/karaoke.mov",
instrumental_audio_path="/path/audio.flac",
)
assert config.job_id == "test-job"
assert config.artist == "Test Artist"
assert config.encoding_backend == "auto"
def test_default_values(self):
"""Test default values for optional fields."""
config = OrchestratorConfig(
job_id="test-job",
artist="Test Artist",
title="Test Title",
title_video_path="/path/title.mov",
karaoke_video_path="/path/karaoke.mov",
instrumental_audio_path="/path/audio.flac",
)
assert config.enable_cdg is False
assert config.enable_txt is False
assert config.enable_youtube_upload is False
assert config.dry_run is False
assert config.non_interactive is True
assert config.end_video_path is None
def test_all_fields(self):
"""Test all fields including optional ones."""
config = OrchestratorConfig(
job_id="test-job",
artist="Test Artist",
title="Test Title",
title_video_path="/path/title.mov",
karaoke_video_path="/path/karaoke.mov",
instrumental_audio_path="/path/audio.flac",
end_video_path="/path/end.mov",
enable_cdg=True,
enable_txt=True,
enable_youtube_upload=True,
brand_prefix="TEST",
discord_webhook_url="https://discord.com/api/webhooks/123/abc",
encoding_backend="gce",
)
assert config.enable_cdg is True
assert config.encoding_backend == "gce"
assert config.discord_webhook_url is not None
class TestOrchestratorResult:
"""Test OrchestratorResult dataclass."""
def test_success_result(self):
"""Test successful result."""
result = OrchestratorResult(
success=True,
final_video="/output/video.mp4",
brand_code="TEST-1234",
youtube_url="https://youtube.com/watch?v=test",
)
assert result.success is True
assert result.error_message is None
assert result.brand_code == "TEST-1234"
def test_failure_result(self):
"""Test failure result."""
result = OrchestratorResult(
success=False,
error_message="Encoding failed",
)
assert result.success is False
assert result.error_message == "Encoding failed"
assert result.final_video is None
def test_all_output_files(self):
"""Test all output file fields."""
result = OrchestratorResult(
success=True,
final_video="/output/lossless.mp4",
final_video_mkv="/output/lossless.mkv",
final_video_lossy="/output/lossy.mp4",
final_video_720p="/output/720p.mp4",
final_karaoke_cdg_zip="/output/cdg.zip",
final_karaoke_txt_zip="/output/txt.zip",
)
assert result.final_video == "/output/lossless.mp4"
assert result.final_video_mkv == "/output/lossless.mkv"
class TestVideoWorkerOrchestratorInit:
"""Test VideoWorkerOrchestrator initialization."""
def test_init_with_config(self):
"""Test initialization with config."""
config = OrchestratorConfig(
job_id="test-job",
artist="Test Artist",
title="Test Title",
title_video_path="/path/title.mov",
karaoke_video_path="/path/karaoke.mov",
instrumental_audio_path="/path/audio.flac",
)
orchestrator = VideoWorkerOrchestrator(config)
assert orchestrator.config == config
assert orchestrator.result.success is False
def test_init_with_job_manager(self):
"""Test initialization with job manager."""
config = OrchestratorConfig(
job_id="test-job",
artist="Test Artist",
title="Test Title",
title_video_path="/path/title.mov",
karaoke_video_path="/path/karaoke.mov",
instrumental_audio_path="/path/audio.flac",
)
job_manager = MagicMock()
orchestrator = VideoWorkerOrchestrator(config, job_manager=job_manager)
assert orchestrator.job_manager == job_manager
class TestVideoWorkerOrchestratorServices:
"""Test service lazy-loading."""
def test_get_encoding_backend(self):
"""Test encoding backend lazy loading."""
config = OrchestratorConfig(
job_id="test-job",
artist="Test Artist",
title="Test Title",
title_video_path="/path/title.mov",
karaoke_video_path="/path/karaoke.mov",
instrumental_audio_path="/path/audio.flac",
encoding_backend="local",
)
orchestrator = VideoWorkerOrchestrator(config)
with patch("backend.services.encoding_interface.get_encoding_backend") as mock_get:
mock_backend = MagicMock()
mock_backend.name = "local"
mock_get.return_value = mock_backend
backend = orchestrator._get_encoding_backend()
assert backend == mock_backend
mock_get.assert_called_once()
def test_get_packaging_service(self):
"""Test packaging service lazy loading."""
config = OrchestratorConfig(
job_id="test-job",
artist="Test Artist",
title="Test Title",
title_video_path="/path/title.mov",
karaoke_video_path="/path/karaoke.mov",
instrumental_audio_path="/path/audio.flac",
cdg_styles={"background_color": "black"},
)
orchestrator = VideoWorkerOrchestrator(config)
with patch("backend.services.packaging_service.PackagingService") as MockService:
mock_service = MagicMock()
MockService.return_value = mock_service
service = orchestrator._get_packaging_service()
assert service == mock_service
MockService.assert_called_once()
def test_get_youtube_service(self):
"""Test YouTube service lazy loading."""
config = OrchestratorConfig(
job_id="test-job",
artist="Test Artist",
title="Test Title",
title_video_path="/path/title.mov",
karaoke_video_path="/path/karaoke.mov",
instrumental_audio_path="/path/audio.flac",
youtube_credentials={"token": "test"},
)
orchestrator = VideoWorkerOrchestrator(config)
with patch("backend.services.youtube_upload_service.YouTubeUploadService") as MockService:
mock_service = MagicMock()
MockService.return_value = mock_service
service = orchestrator._get_youtube_service()
assert service == mock_service
MockService.assert_called_once()
def test_get_discord_service(self):
"""Test Discord service lazy loading."""
config = OrchestratorConfig(
job_id="test-job",
artist="Test Artist",
title="Test Title",
title_video_path="/path/title.mov",
karaoke_video_path="/path/karaoke.mov",
instrumental_audio_path="/path/audio.flac",
discord_webhook_url="https://discord.com/api/webhooks/123/abc",
)
orchestrator = VideoWorkerOrchestrator(config)
with patch("backend.services.discord_service.DiscordNotificationService") as MockService:
mock_service = MagicMock()
MockService.return_value = mock_service
service = orchestrator._get_discord_service()
assert service == mock_service
MockService.assert_called_once()
class TestVideoWorkerOrchestratorPackaging:
"""Test packaging stage."""
@pytest.mark.asyncio
async def test_run_packaging_no_lrc(self):
"""Test packaging stage with no LRC file."""
config = OrchestratorConfig(
job_id="test-job",
artist="Test Artist",
title="Test Title",
title_video_path="/path/title.mov",
karaoke_video_path="/path/karaoke.mov",
instrumental_audio_path="/path/audio.flac",
enable_cdg=True,
lrc_file_path=None,
)
orchestrator = VideoWorkerOrchestrator(config)
# Should not raise, just skip
await orchestrator._run_packaging()
assert orchestrator.result.final_karaoke_cdg_zip is None
@pytest.mark.asyncio
async def test_run_packaging_cdg(self):
"""Test CDG packaging."""
with tempfile.TemporaryDirectory() as temp_dir:
lrc_file = os.path.join(temp_dir, "test.lrc")
audio_file = os.path.join(temp_dir, "test.flac")
# Create dummy files
with open(lrc_file, "w") as f:
f.write("[00:00.00]Test lyrics")
with open(audio_file, "w") as f:
f.write("dummy audio")
config = OrchestratorConfig(
job_id="test-job",
artist="Test Artist",
title="Test Title",
title_video_path="/path/title.mov",
karaoke_video_path="/path/karaoke.mov",
instrumental_audio_path=audio_file,
lrc_file_path=lrc_file,
output_dir=temp_dir,
enable_cdg=True,
cdg_styles={"background_color": "black"},
)
orchestrator = VideoWorkerOrchestrator(config)
with patch.object(orchestrator, "_get_packaging_service") as mock_get:
mock_service = MagicMock()
mock_service.create_cdg_package.return_value = (
f"{temp_dir}/cdg.zip",
f"{temp_dir}/test.mp3",
f"{temp_dir}/test.cdg",
)
mock_get.return_value = mock_service
await orchestrator._run_packaging()
mock_service.create_cdg_package.assert_called_once()
assert orchestrator.result.final_karaoke_cdg_zip == f"{temp_dir}/cdg.zip"
class TestVideoWorkerOrchestratorEncoding:
"""Test encoding stage."""
@pytest.mark.asyncio
async def test_run_encoding_success(self):
"""Test successful encoding."""
config = OrchestratorConfig(
job_id="test-job",
artist="Test Artist",
title="Test Title",
title_video_path="/path/title.mov",
karaoke_video_path="/path/karaoke.mov",
instrumental_audio_path="/path/audio.flac",
output_dir="/output",
)
orchestrator = VideoWorkerOrchestrator(config)
with patch.object(orchestrator, "_get_encoding_backend") as mock_get:
from backend.services.encoding_interface import EncodingOutput
mock_backend = MagicMock()
mock_backend.name = "local"
mock_backend.encode = AsyncMock(return_value=EncodingOutput(
success=True,
lossless_4k_mp4_path="/output/lossless.mp4",
lossy_4k_mp4_path="/output/lossy.mp4",
lossy_720p_mp4_path="/output/720p.mp4",
lossless_mkv_path="/output/lossless.mkv",
encoding_time_seconds=120.5,
encoding_backend="local",
))
mock_get.return_value = mock_backend
await orchestrator._run_encoding()
mock_backend.encode.assert_called_once()
assert orchestrator.result.final_video == "/output/lossless.mp4"
assert orchestrator.result.final_video_720p == "/output/720p.mp4"
assert orchestrator.result.encoding_time_seconds == 120.5
@pytest.mark.asyncio
async def test_run_encoding_failure(self):
"""Test encoding failure."""
config = OrchestratorConfig(
job_id="test-job",
artist="Test Artist",
title="Test Title",
title_video_path="/path/title.mov",
karaoke_video_path="/path/karaoke.mov",
instrumental_audio_path="/path/audio.flac",
)
orchestrator = VideoWorkerOrchestrator(config)
with patch.object(orchestrator, "_get_encoding_backend") as mock_get:
from backend.services.encoding_interface import EncodingOutput
mock_backend = MagicMock()
mock_backend.name = "local"
mock_backend.encode = AsyncMock(return_value=EncodingOutput(
success=False,
error_message="FFmpeg failed",
encoding_backend="local",
))
mock_get.return_value = mock_backend
with pytest.raises(Exception) as exc_info:
await orchestrator._run_encoding()
assert "Encoding failed" in str(exc_info.value)
class TestVideoWorkerOrchestratorOrganization:
"""Test organization stage."""
@pytest.mark.asyncio
async def test_run_organization_keep_brand_code(self):
"""Test organization with existing brand code."""
config = OrchestratorConfig(
job_id="test-job",
artist="Test Artist",
title="Test Title",
title_video_path="/path/title.mov",
karaoke_video_path="/path/karaoke.mov",
instrumental_audio_path="/path/audio.flac",
keep_brand_code="NOMAD-9999",
)
orchestrator = VideoWorkerOrchestrator(config)
await orchestrator._run_organization()
assert orchestrator.result.brand_code == "NOMAD-9999"
@pytest.mark.asyncio
async def test_run_organization_generate_brand_code(self):
"""Test brand code generation from Dropbox."""
config = OrchestratorConfig(
job_id="test-job",
artist="Test Artist",
title="Test Title",
title_video_path="/path/title.mov",
karaoke_video_path="/path/karaoke.mov",
instrumental_audio_path="/path/audio.flac",
dropbox_path="/Karaoke/Tracks",
brand_prefix="TEST",
)
orchestrator = VideoWorkerOrchestrator(config)
with patch("backend.services.dropbox_service.get_dropbox_service") as mock_get:
mock_dropbox = MagicMock()
mock_dropbox.is_configured = True
mock_dropbox.get_next_brand_code.return_value = "TEST-0001"
mock_get.return_value = mock_dropbox
await orchestrator._run_organization()
assert orchestrator.result.brand_code == "TEST-0001"
class TestVideoWorkerOrchestratorDistribution:
"""Test distribution stage."""
@pytest.mark.asyncio
async def test_upload_to_youtube(self):
"""Test YouTube upload."""
with tempfile.TemporaryDirectory() as temp_dir:
video_file = os.path.join(temp_dir, "test.mp4")
with open(video_file, "w") as f:
f.write("dummy video")
config = OrchestratorConfig(
job_id="test-job",
artist="Test Artist",
title="Test Title",
title_video_path="/path/title.mov",
karaoke_video_path="/path/karaoke.mov",
instrumental_audio_path="/path/audio.flac",
enable_youtube_upload=True,
youtube_credentials={"token": "test"},
)
orchestrator = VideoWorkerOrchestrator(config)
orchestrator.result.final_video_lossy = video_file
with patch.object(orchestrator, "_get_youtube_service") as mock_get:
mock_service = MagicMock()
mock_service.upload_video.return_value = (
"video123",
"https://youtube.com/watch?v=video123"
)
mock_get.return_value = mock_service
await orchestrator._upload_to_youtube()
mock_service.upload_video.assert_called_once()
assert orchestrator.result.youtube_url == "https://youtube.com/watch?v=video123"
@pytest.mark.asyncio
async def test_upload_to_dropbox(self):
"""Test Dropbox upload."""
with tempfile.TemporaryDirectory() as temp_dir:
config = OrchestratorConfig(
job_id="test-job",
artist="Test Artist",
title="Test Title",
title_video_path="/path/title.mov",
karaoke_video_path="/path/karaoke.mov",
instrumental_audio_path="/path/audio.flac",
output_dir=temp_dir,
dropbox_path="/Karaoke/Tracks",
brand_prefix="TEST",
)
orchestrator = VideoWorkerOrchestrator(config)
orchestrator.result.brand_code = "TEST-0001"
with patch("backend.services.dropbox_service.get_dropbox_service") as mock_get:
mock_dropbox = MagicMock()
mock_dropbox.is_configured = True
mock_dropbox.create_shared_link.return_value = "https://dropbox.com/link"
mock_get.return_value = mock_dropbox
await orchestrator._upload_to_dropbox()
mock_dropbox.upload_folder.assert_called_once()
assert orchestrator.result.dropbox_link == "https://dropbox.com/link"
class TestVideoWorkerOrchestratorNotifications:
"""Test notifications stage."""
@pytest.mark.asyncio
async def test_run_notifications_with_youtube_url(self):
"""Test Discord notification with YouTube URL."""
config = OrchestratorConfig(
job_id="test-job",
artist="Test Artist",
title="Test Title",
title_video_path="/path/title.mov",
karaoke_video_path="/path/karaoke.mov",
instrumental_audio_path="/path/audio.flac",
discord_webhook_url="https://discord.com/api/webhooks/123/abc",
)
orchestrator = VideoWorkerOrchestrator(config)
orchestrator.result.youtube_url = "https://youtube.com/watch?v=test"
with patch.object(orchestrator, "_get_discord_service") as mock_get:
mock_service = MagicMock()
mock_service.post_video_notification.return_value = True
mock_get.return_value = mock_service
await orchestrator._run_notifications()
mock_service.post_video_notification.assert_called_once_with(
"https://youtube.com/watch?v=test"
)
@pytest.mark.asyncio
async def test_run_notifications_no_youtube_url(self):
"""Test notification skipped without YouTube URL."""
config = OrchestratorConfig(
job_id="test-job",
artist="Test Artist",
title="Test Title",
title_video_path="/path/title.mov",
karaoke_video_path="/path/karaoke.mov",
instrumental_audio_path="/path/audio.flac",
discord_webhook_url="https://discord.com/api/webhooks/123/abc",
)
orchestrator = VideoWorkerOrchestrator(config)
# No youtube_url set
with patch.object(orchestrator, "_get_discord_service") as mock_get:
mock_service = MagicMock()
mock_get.return_value = mock_service
await orchestrator._run_notifications()
mock_service.post_video_notification.assert_not_called()
class TestVideoWorkerOrchestratorFullPipeline:
"""Test full pipeline execution."""
@pytest.mark.asyncio
async def test_run_full_pipeline_success(self):
"""Test successful full pipeline."""
config = OrchestratorConfig(
job_id="test-job",
artist="Test Artist",
title="Test Title",
title_video_path="/path/title.mov",
karaoke_video_path="/path/karaoke.mov",
instrumental_audio_path="/path/audio.flac",
enable_cdg=True, # Enable to trigger packaging stage
)
orchestrator = VideoWorkerOrchestrator(config)
# Mock all stages
with patch.object(orchestrator, "_run_packaging", new_callable=AsyncMock) as mock_packaging, \
patch.object(orchestrator, "_run_encoding", new_callable=AsyncMock) as mock_encoding, \
patch.object(orchestrator, "_run_organization", new_callable=AsyncMock) as mock_org, \
patch.object(orchestrator, "_run_distribution", new_callable=AsyncMock) as mock_dist, \
patch.object(orchestrator, "_run_notifications", new_callable=AsyncMock) as mock_notify:
result = await orchestrator.run()
assert result.success is True
mock_packaging.assert_called_once()
mock_encoding.assert_called_once()
mock_org.assert_called_once()
mock_dist.assert_called_once()
mock_notify.assert_called_once()
@pytest.mark.asyncio
async def test_run_pipeline_skips_packaging_when_disabled(self):
"""Test pipeline skips packaging when CDG/TXT disabled."""
config = OrchestratorConfig(
job_id="test-job",
artist="Test Artist",
title="Test Title",
title_video_path="/path/title.mov",
karaoke_video_path="/path/karaoke.mov",
instrumental_audio_path="/path/audio.flac",
enable_cdg=False,
enable_txt=False,
)
orchestrator = VideoWorkerOrchestrator(config)
# Mock all stages
with patch.object(orchestrator, "_run_packaging", new_callable=AsyncMock) as mock_packaging, \
patch.object(orchestrator, "_run_encoding", new_callable=AsyncMock) as mock_encoding, \
patch.object(orchestrator, "_run_organization", new_callable=AsyncMock), \
patch.object(orchestrator, "_run_distribution", new_callable=AsyncMock), \
patch.object(orchestrator, "_run_notifications", new_callable=AsyncMock):
result = await orchestrator.run()
assert result.success is True
mock_packaging.assert_not_called()
mock_encoding.assert_called_once()
@pytest.mark.asyncio
async def test_run_full_pipeline_encoding_failure(self):
"""Test pipeline failure during encoding."""
config = OrchestratorConfig(
job_id="test-job",
artist="Test Artist",
title="Test Title",
title_video_path="/path/title.mov",
karaoke_video_path="/path/karaoke.mov",
instrumental_audio_path="/path/audio.flac",
)
orchestrator = VideoWorkerOrchestrator(config)
with patch.object(orchestrator, "_run_packaging", new_callable=AsyncMock), \
patch.object(orchestrator, "_run_encoding", new_callable=AsyncMock) as mock_encoding, \
patch.object(orchestrator, "_run_organization", new_callable=AsyncMock) as mock_org:
mock_encoding.side_effect = Exception("Encoding failed")
result = await orchestrator.run()
assert result.success is False
assert "Encoding failed" in result.error_message
mock_org.assert_not_called()
class TestCreateOrchestratorConfigFromJob:
"""Test helper function for creating config from job."""
def test_create_config_from_job(self):
"""Test creating config from a job object."""
job = MagicMock()
job.job_id = "test-123"
job.artist = "Test Artist"
job.title = "Test Title"
job.state_data = {"instrumental_selection": "clean"}
job.enable_cdg = True
job.enable_txt = False
job.enable_youtube_upload = True
job.brand_prefix = "NOMAD"
job.discord_webhook_url = "https://discord.com/api/webhooks/123/abc"
job.youtube_description_template = "Test description"
job.dropbox_path = "/Karaoke"
job.gdrive_folder_id = None
job.keep_brand_code = None
job.existing_instrumental_gcs_path = None
config = create_orchestrator_config_from_job(
job=job,
temp_dir="/tmp/test",
youtube_credentials={"token": "test"},
cdg_styles={"background": "black"},
)
assert config.job_id == "test-123"
assert config.artist == "Test Artist"
assert config.title == "Test Title"
assert config.enable_cdg is True
assert config.enable_youtube_upload is True
assert config.title_video_path == "/tmp/test/Test Artist - Test Title (Title).mov"
assert config.instrumental_audio_path == "/tmp/test/Test Artist - Test Title (Instrumental Clean).flac"
def test_create_config_from_job_with_existing_instrumental(self):
"""Test config with user-provided instrumental."""
job = MagicMock()
job.job_id = "test-123"
job.artist = "Test Artist"
job.title = "Test Title"
job.state_data = {"instrumental_selection": "custom"}
job.enable_cdg = False
job.enable_txt = False
job.enable_youtube_upload = False
job.brand_prefix = None
job.discord_webhook_url = None
job.youtube_description_template = None
job.dropbox_path = None
job.gdrive_folder_id = None
job.keep_brand_code = None
job.existing_instrumental_gcs_path = "gs://bucket/instrumental.mp3"
config = create_orchestrator_config_from_job(
job=job,
temp_dir="/tmp/test",
)
assert config.instrumental_audio_path == "/tmp/test/Test Artist - Test Title (Instrumental User).mp3"
@@ -0,0 +1,568 @@
"""
Tests for YouTubeUploadService.
Tests cover:
- Authentication with pre-stored credentials
- Authentication with client secrets file
- Duplicate video detection (exact and fuzzy matching)
- Video deletion
- Video upload with metadata and thumbnail
- Title truncation
- Dry run mode
"""
import os
import pytest
from unittest.mock import MagicMock, patch, mock_open
from backend.services.youtube_upload_service import YouTubeUploadService, get_youtube_upload_service
class TestYouTubeUploadServiceInit:
"""Test service initialization."""
def test_init_with_credentials(self):
"""Test initialization with pre-stored credentials."""
creds = {
"token": "test_token",
"refresh_token": "test_refresh",
"token_uri": "https://oauth2.googleapis.com/token",
"client_id": "test_client_id",
"client_secret": "test_secret",
}
service = YouTubeUploadService(
credentials=creds,
non_interactive=True
)
assert service.credentials == creds
assert service.non_interactive is True
def test_init_with_client_secrets(self):
"""Test initialization with client secrets file."""
service = YouTubeUploadService(
client_secrets_file="/path/to/secrets.json"
)
assert service.client_secrets_file == "/path/to/secrets.json"
assert service.non_interactive is False
def test_init_default_values(self):
"""Test default values on initialization."""
service = YouTubeUploadService()
assert service.credentials is None
assert service.client_secrets_file is None
assert service.non_interactive is False
assert service.server_side_mode is False
assert service.dry_run is False
class TestYouTubeUploadServiceAuthentication:
"""Test authentication methods."""
@patch("googleapiclient.discovery.build")
@patch("google.oauth2.credentials.Credentials")
@patch("google.auth.transport.requests.Request")
def test_authenticate_with_prestored_credentials(
self, mock_request, mock_credentials_class, mock_build
):
"""Test authentication using pre-stored credentials."""
# Setup mocks
mock_creds = MagicMock()
mock_creds.expired = False
mock_credentials_class.return_value = mock_creds
mock_youtube = MagicMock()
mock_build.return_value = mock_youtube
creds = {
"token": "test_token",
"refresh_token": "test_refresh",
"token_uri": "https://oauth2.googleapis.com/token",
"client_id": "test_client_id",
"client_secret": "test_secret",
"scopes": ["https://www.googleapis.com/auth/youtube"],
}
service = YouTubeUploadService(credentials=creds, non_interactive=True)
# Access youtube_service to trigger authentication
result = service.youtube_service
assert result == mock_youtube
mock_credentials_class.assert_called_once()
mock_build.assert_called_once_with('youtube', 'v3', credentials=mock_creds)
@patch("googleapiclient.discovery.build")
@patch("google.oauth2.credentials.Credentials")
@patch("google.auth.transport.requests.Request")
def test_authenticate_refreshes_expired_token(
self, mock_request, mock_credentials_class, mock_build
):
"""Test that expired tokens are refreshed."""
mock_creds = MagicMock()
mock_creds.expired = True
mock_creds.refresh_token = "test_refresh"
mock_credentials_class.return_value = mock_creds
creds = {
"token": "expired_token",
"refresh_token": "test_refresh",
"token_uri": "https://oauth2.googleapis.com/token",
"client_id": "test_client_id",
"client_secret": "test_secret",
}
service = YouTubeUploadService(credentials=creds, non_interactive=True)
service.youtube_service
mock_creds.refresh.assert_called_once()
def test_authenticate_non_interactive_no_credentials_raises(self):
"""Test that non-interactive mode without credentials raises error."""
service = YouTubeUploadService(non_interactive=True)
with pytest.raises(Exception) as exc_info:
service._authenticate()
assert "non-interactive mode" in str(exc_info.value).lower()
class TestYouTubeUploadServiceChannelId:
"""Test channel ID retrieval."""
@patch("googleapiclient.discovery.build")
@patch("google.oauth2.credentials.Credentials")
@patch("google.auth.transport.requests.Request")
def test_get_channel_id_success(
self, mock_request, mock_credentials_class, mock_build
):
"""Test successful channel ID retrieval."""
mock_creds = MagicMock()
mock_creds.expired = False
mock_credentials_class.return_value = mock_creds
mock_youtube = MagicMock()
mock_channels = MagicMock()
mock_list = MagicMock()
mock_list.execute.return_value = {
"items": [{"id": "UC123456"}]
}
mock_channels.list.return_value = mock_list
mock_youtube.channels.return_value = mock_channels
mock_build.return_value = mock_youtube
creds = {"token": "test", "refresh_token": "test", "token_uri": "uri",
"client_id": "id", "client_secret": "secret"}
service = YouTubeUploadService(credentials=creds, non_interactive=True)
channel_id = service.get_channel_id()
assert channel_id == "UC123456"
@patch("googleapiclient.discovery.build")
@patch("google.oauth2.credentials.Credentials")
@patch("google.auth.transport.requests.Request")
def test_get_channel_id_no_items(
self, mock_request, mock_credentials_class, mock_build
):
"""Test channel ID returns None when no items found."""
mock_creds = MagicMock()
mock_creds.expired = False
mock_credentials_class.return_value = mock_creds
mock_youtube = MagicMock()
mock_channels = MagicMock()
mock_list = MagicMock()
mock_list.execute.return_value = {"items": []}
mock_channels.list.return_value = mock_list
mock_youtube.channels.return_value = mock_channels
mock_build.return_value = mock_youtube
creds = {"token": "test", "refresh_token": "test", "token_uri": "uri",
"client_id": "id", "client_secret": "secret"}
service = YouTubeUploadService(credentials=creds, non_interactive=True)
channel_id = service.get_channel_id()
assert channel_id is None
class TestYouTubeUploadServiceDuplicateCheck:
"""Test duplicate video detection."""
@patch("googleapiclient.discovery.build")
@patch("google.oauth2.credentials.Credentials")
@patch("google.auth.transport.requests.Request")
def test_check_duplicate_exact_match_found(
self, mock_request, mock_credentials_class, mock_build
):
"""Test duplicate detection with exact match in server mode."""
mock_creds = MagicMock()
mock_creds.expired = False
mock_credentials_class.return_value = mock_creds
mock_youtube = MagicMock()
# Mock channels().list() for get_channel_id
mock_channels = MagicMock()
mock_channels_list = MagicMock()
mock_channels_list.execute.return_value = {"items": [{"id": "UC123456"}]}
mock_channels.list.return_value = mock_channels_list
mock_youtube.channels.return_value = mock_channels
# Mock search().list() for duplicate check
mock_search = MagicMock()
mock_search_list = MagicMock()
mock_search_list.execute.return_value = {
"items": [{
"id": {"videoId": "VIDEO123"},
"snippet": {
"channelId": "UC123456",
"title": "Test Artist - Test Song (Karaoke)"
}
}]
}
mock_search.list.return_value = mock_search_list
mock_youtube.search.return_value = mock_search
mock_build.return_value = mock_youtube
creds = {"token": "test", "refresh_token": "test", "token_uri": "uri",
"client_id": "id", "client_secret": "secret"}
service = YouTubeUploadService(
credentials=creds,
non_interactive=True,
server_side_mode=True
)
exists, video_id, video_url = service.check_duplicate(
"Test Artist - Test Song (Karaoke)"
)
assert exists is True
assert video_id == "VIDEO123"
assert "VIDEO123" in video_url
@patch("googleapiclient.discovery.build")
@patch("google.oauth2.credentials.Credentials")
@patch("google.auth.transport.requests.Request")
def test_check_duplicate_no_match(
self, mock_request, mock_credentials_class, mock_build
):
"""Test duplicate detection when no match found."""
mock_creds = MagicMock()
mock_creds.expired = False
mock_credentials_class.return_value = mock_creds
mock_youtube = MagicMock()
mock_channels = MagicMock()
mock_channels_list = MagicMock()
mock_channels_list.execute.return_value = {"items": [{"id": "UC123456"}]}
mock_channels.list.return_value = mock_channels_list
mock_youtube.channels.return_value = mock_channels
mock_search = MagicMock()
mock_search_list = MagicMock()
mock_search_list.execute.return_value = {"items": []}
mock_search.list.return_value = mock_search_list
mock_youtube.search.return_value = mock_search
mock_build.return_value = mock_youtube
creds = {"token": "test", "refresh_token": "test", "token_uri": "uri",
"client_id": "id", "client_secret": "secret"}
service = YouTubeUploadService(credentials=creds, non_interactive=True)
exists, video_id, video_url = service.check_duplicate("Some Title")
assert exists is False
assert video_id is None
assert video_url is None
@patch("googleapiclient.discovery.build")
@patch("google.oauth2.credentials.Credentials")
@patch("google.auth.transport.requests.Request")
def test_check_duplicate_skips_other_channels(
self, mock_request, mock_credentials_class, mock_build
):
"""Test that videos from other channels are skipped."""
mock_creds = MagicMock()
mock_creds.expired = False
mock_credentials_class.return_value = mock_creds
mock_youtube = MagicMock()
mock_channels = MagicMock()
mock_channels_list = MagicMock()
mock_channels_list.execute.return_value = {"items": [{"id": "UC123456"}]}
mock_channels.list.return_value = mock_channels_list
mock_youtube.channels.return_value = mock_channels
# Return video from different channel
mock_search = MagicMock()
mock_search_list = MagicMock()
mock_search_list.execute.return_value = {
"items": [{
"id": {"videoId": "VIDEO123"},
"snippet": {
"channelId": "UC_DIFFERENT", # Different channel
"title": "Test Title"
}
}]
}
mock_search.list.return_value = mock_search_list
mock_youtube.search.return_value = mock_search
mock_build.return_value = mock_youtube
creds = {"token": "test", "refresh_token": "test", "token_uri": "uri",
"client_id": "id", "client_secret": "secret"}
service = YouTubeUploadService(credentials=creds, non_interactive=True)
exists, video_id, video_url = service.check_duplicate("Test Title")
assert exists is False
class TestYouTubeUploadServiceDelete:
"""Test video deletion."""
@patch("googleapiclient.discovery.build")
@patch("google.oauth2.credentials.Credentials")
@patch("google.auth.transport.requests.Request")
def test_delete_video_success(
self, mock_request, mock_credentials_class, mock_build
):
"""Test successful video deletion."""
mock_creds = MagicMock()
mock_creds.expired = False
mock_credentials_class.return_value = mock_creds
mock_youtube = MagicMock()
mock_videos = MagicMock()
mock_delete = MagicMock()
mock_delete.execute.return_value = None
mock_videos.delete.return_value = mock_delete
mock_youtube.videos.return_value = mock_videos
mock_build.return_value = mock_youtube
creds = {"token": "test", "refresh_token": "test", "token_uri": "uri",
"client_id": "id", "client_secret": "secret"}
service = YouTubeUploadService(credentials=creds, non_interactive=True)
result = service.delete_video("VIDEO123")
assert result is True
mock_videos.delete.assert_called_once_with(id="VIDEO123")
@patch("googleapiclient.discovery.build")
@patch("google.oauth2.credentials.Credentials")
@patch("google.auth.transport.requests.Request")
def test_delete_video_failure(
self, mock_request, mock_credentials_class, mock_build
):
"""Test video deletion failure handling."""
mock_creds = MagicMock()
mock_creds.expired = False
mock_credentials_class.return_value = mock_creds
mock_youtube = MagicMock()
mock_videos = MagicMock()
mock_delete = MagicMock()
mock_delete.execute.side_effect = Exception("API Error")
mock_videos.delete.return_value = mock_delete
mock_youtube.videos.return_value = mock_videos
mock_build.return_value = mock_youtube
creds = {"token": "test", "refresh_token": "test", "token_uri": "uri",
"client_id": "id", "client_secret": "secret"}
service = YouTubeUploadService(credentials=creds, non_interactive=True)
result = service.delete_video("VIDEO123")
assert result is False
def test_delete_video_dry_run(self):
"""Test video deletion in dry run mode."""
service = YouTubeUploadService(dry_run=True)
result = service.delete_video("VIDEO123")
assert result is True
class TestYouTubeUploadServiceTitleTruncation:
"""Test title truncation."""
def test_truncate_title_short_title(self):
"""Test that short titles are not truncated."""
title = "Short Title"
result = YouTubeUploadService.truncate_title(title)
assert result == title
def test_truncate_title_exact_length(self):
"""Test title at exact max length."""
title = "A" * 95
result = YouTubeUploadService.truncate_title(title, max_length=95)
assert result == title
assert len(result) == 95
def test_truncate_title_long_title(self):
"""Test that long titles are truncated at word boundary."""
title = "This is a very long title that exceeds the maximum length and needs to be truncated properly at a word boundary"
result = YouTubeUploadService.truncate_title(title, max_length=50)
assert len(result) <= 50
assert result.endswith("...")
def test_truncate_title_no_space(self):
"""Test truncation of title without spaces."""
title = "A" * 100
result = YouTubeUploadService.truncate_title(title, max_length=50)
assert len(result) <= 50
class TestYouTubeUploadServiceUpload:
"""Test video upload."""
@patch("googleapiclient.http.MediaFileUpload")
@patch("googleapiclient.discovery.build")
@patch("google.oauth2.credentials.Credentials")
@patch("google.auth.transport.requests.Request")
def test_upload_video_success(
self, mock_request, mock_credentials_class, mock_build, mock_media_upload
):
"""Test successful video upload."""
mock_creds = MagicMock()
mock_creds.expired = False
mock_credentials_class.return_value = mock_creds
mock_youtube = MagicMock()
# Mock channels for get_channel_id
mock_channels = MagicMock()
mock_channels_list = MagicMock()
mock_channels_list.execute.return_value = {"items": [{"id": "UC123456"}]}
mock_channels.list.return_value = mock_channels_list
mock_youtube.channels.return_value = mock_channels
# Mock search for duplicate check (no duplicates)
mock_search = MagicMock()
mock_search_list = MagicMock()
mock_search_list.execute.return_value = {"items": []}
mock_search.list.return_value = mock_search_list
mock_youtube.search.return_value = mock_search
# Mock video insert
mock_videos = MagicMock()
mock_insert = MagicMock()
mock_insert.execute.return_value = {"id": "NEW_VIDEO_ID"}
mock_videos.insert.return_value = mock_insert
mock_youtube.videos.return_value = mock_videos
mock_build.return_value = mock_youtube
mock_media_upload.return_value = MagicMock()
creds = {"token": "test", "refresh_token": "test", "token_uri": "uri",
"client_id": "id", "client_secret": "secret"}
service = YouTubeUploadService(credentials=creds, non_interactive=True)
video_id, video_url = service.upload_video(
video_path="/path/to/video.mkv",
title="Test Artist - Test Song (Karaoke)",
description="Test description",
tags=["karaoke", "test"]
)
assert video_id == "NEW_VIDEO_ID"
assert "NEW_VIDEO_ID" in video_url
def test_upload_video_dry_run(self):
"""Test video upload in dry run mode."""
service = YouTubeUploadService(dry_run=True)
video_id, video_url = service.upload_video(
video_path="/path/to/video.mkv",
title="Test Title",
description="Test description"
)
assert video_id == "dry_run_video_id"
assert "dry_run_video_id" in video_url
@patch("googleapiclient.http.MediaFileUpload")
@patch("googleapiclient.discovery.build")
@patch("google.oauth2.credentials.Credentials")
@patch("google.auth.transport.requests.Request")
@patch("os.path.isfile")
def test_upload_video_with_thumbnail(
self, mock_isfile, mock_request, mock_credentials_class,
mock_build, mock_media_upload
):
"""Test video upload with thumbnail."""
mock_isfile.return_value = True
mock_creds = MagicMock()
mock_creds.expired = False
mock_credentials_class.return_value = mock_creds
mock_youtube = MagicMock()
mock_channels = MagicMock()
mock_channels_list = MagicMock()
mock_channels_list.execute.return_value = {"items": [{"id": "UC123456"}]}
mock_channels.list.return_value = mock_channels_list
mock_youtube.channels.return_value = mock_channels
mock_search = MagicMock()
mock_search_list = MagicMock()
mock_search_list.execute.return_value = {"items": []}
mock_search.list.return_value = mock_search_list
mock_youtube.search.return_value = mock_search
mock_videos = MagicMock()
mock_insert = MagicMock()
mock_insert.execute.return_value = {"id": "NEW_VIDEO_ID"}
mock_videos.insert.return_value = mock_insert
mock_youtube.videos.return_value = mock_videos
# Mock thumbnail upload
mock_thumbnails = MagicMock()
mock_set = MagicMock()
mock_set.execute.return_value = None
mock_thumbnails.set.return_value = mock_set
mock_youtube.thumbnails.return_value = mock_thumbnails
mock_build.return_value = mock_youtube
mock_media_upload.return_value = MagicMock()
creds = {"token": "test", "refresh_token": "test", "token_uri": "uri",
"client_id": "id", "client_secret": "secret"}
service = YouTubeUploadService(credentials=creds, non_interactive=True)
video_id, video_url = service.upload_video(
video_path="/path/to/video.mkv",
title="Test Title",
description="Test description",
thumbnail_path="/path/to/thumbnail.jpg"
)
assert video_id == "NEW_VIDEO_ID"
mock_thumbnails.set.assert_called_once()
class TestGetYouTubeUploadService:
"""Test factory function."""
def test_get_service_creates_instance(self):
"""Test that factory function creates a new instance."""
# Reset global
import backend.services.youtube_upload_service as module
module._youtube_upload_service = None
service = get_youtube_upload_service(
credentials={"token": "test"},
non_interactive=True
)
assert service is not None
assert isinstance(service, YouTubeUploadService)
def test_get_service_with_client_secrets(self):
"""Test factory function with client secrets file."""
import backend.services.youtube_upload_service as module
module._youtube_upload_service = None
service = get_youtube_upload_service(
client_secrets_file="/path/to/secrets.json"
)
assert service is not None
assert service.client_secrets_file == "/path/to/secrets.json"
+257 -19
View File
@@ -1,20 +1,21 @@
"""
Video generation and finalization worker.
This worker uses KaraokeFinalise from the karaoke_gen package to handle:
1. Video encoding (4 formats)
2. CDG/TXT package generation
3. Discord notifications
4. Brand code generation
5. All other finalisation features
This worker has two code paths controlled by USE_NEW_ORCHESTRATOR:
The worker's job is simply to:
1. Download files from GCS and set up the directory structure KaraokeFinalise expects
2. Instantiate KaraokeFinalise with parameters from the job
3. Call process()
4. Upload results back to GCS
1. NEW PATH (USE_NEW_ORCHESTRATOR=true):
Uses VideoWorkerOrchestrator which provides a unified pipeline for all
encoding backends (GCE or local). This ensures features like YouTube upload,
Discord notifications, and CDG/TXT packaging work regardless of encoding backend.
This reuses 100% of the existing karaoke_gen functionality.
2. LEGACY PATH (USE_NEW_ORCHESTRATOR=false):
Uses KaraokeFinalise from the karaoke_gen package. This path has divergent
behavior between GCE and local encoding - GCE encoding bypasses many features.
The orchestrator-based approach is preferred as it:
- Eliminates GCE vs local code path divergence
- Ensures all features work consistently
- Provides better testability and maintainability
Observability:
- All operations wrapped in tracing spans for Cloud Trace visibility
@@ -48,6 +49,12 @@ from karaoke_gen.karaoke_finalise.karaoke_finalise import KaraokeFinalise
logger = logging.getLogger(__name__)
# Feature flag for new orchestrator-based pipeline
# Set to True to use the new unified pipeline that works with any encoding backend
# Set to False to use the legacy KaraokeFinalise-based pipeline
USE_NEW_ORCHESTRATOR = os.environ.get("USE_NEW_ORCHESTRATOR", "true").lower() == "true"
# Loggers to capture for video worker
# Include the full module path to properly capture KaraokeFinalise logs
VIDEO_WORKER_LOGGERS = [
@@ -172,16 +179,247 @@ async def _encode_via_gce(
raise
async def generate_video(job_id: str) -> bool:
async def generate_video_orchestrated(job_id: str) -> bool:
"""
Generate final karaoke videos using KaraokeFinalise.
This sets up the directory structure and calls KaraokeFinalise.process()
which handles all encoding, packaging, and notifications.
Generate final karaoke videos using the new VideoWorkerOrchestrator.
This provides a unified pipeline that works with any encoding backend
(GCE or local), ensuring all features like YouTube upload, Discord
notifications, and CDG/TXT packaging work consistently.
Args:
job_id: Job ID to process
Returns:
True if successful, False otherwise
"""
from backend.workers.video_worker_orchestrator import (
VideoWorkerOrchestrator,
create_orchestrator_config_from_job,
)
start_time = time.time()
job_manager = JobManager()
storage = StorageService()
# Create job logger for remote debugging
job_log = create_job_logger(job_id, "video")
# Log with structured markers for easy Cloud Logging queries
logger.info(f"[job:{job_id}] WORKER_START worker=video orchestrator=true")
# Set up log capture for KaraokeFinalise (still used by some services)
log_handler = setup_job_logging(job_id, "video", *VIDEO_WORKER_LOGGERS)
job = job_manager.get_job(job_id)
if not job:
logger.error(f"[job:{job_id}] Job not found")
return False
# Validate prerequisites
if not _validate_prerequisites(job):
logger.error(f"[job:{job_id}] Prerequisites not met for video generation")
return False
# Create temporary working directory
temp_dir = tempfile.mkdtemp(prefix=f"karaoke_video_{job_id}_")
original_cwd = os.getcwd()
# Set up rclone config if needed (for legacy Dropbox upload path)
rclone_service = None
if getattr(job, 'organised_dir_rclone_root', None):
rclone_service = get_rclone_service()
if rclone_service.setup_rclone_config():
job_log.info("Rclone config loaded for Dropbox upload")
# Load YouTube credentials if needed
youtube_credentials = None
if getattr(job, 'enable_youtube_upload', False):
youtube_service = get_youtube_service()
if youtube_service.is_configured:
youtube_credentials = youtube_service.get_credentials_dict()
job_log.info("YouTube credentials loaded for video upload")
else:
job_log.warning("YouTube credentials not available - upload will be skipped")
try:
# Wrap entire worker in a tracing span
with job_span("video-worker-orchestrated", job_id, {"artist": job.artist, "title": job.title}) as root_span:
with job_logging_context(job_id):
job_log.info(f"Starting orchestrated video generation for {job.artist} - {job.title}")
logger.info(f"[job:{job_id}] Starting orchestrated video generation")
# Transition to GENERATING_VIDEO state
job_manager.transition_to_state(
job_id=job_id,
new_status=JobStatus.GENERATING_VIDEO,
progress=70,
message="Preparing files for video generation"
)
# Download and set up files
base_name = f"{job.artist} - {job.title}"
with job_span("download-files", job_id):
job_log.info("Downloading files from GCS...")
await _setup_working_directory(job_id, job, storage, temp_dir, base_name, job_log)
job_log.info("All files downloaded successfully")
# Load style config for CDG styles
style_config = await load_style_config(job, storage, temp_dir)
cdg_styles = style_config.get_cdg_styles()
# Change to working directory
os.chdir(temp_dir)
# Create orchestrator config from job
config = create_orchestrator_config_from_job(
job=job,
temp_dir=temp_dir,
youtube_credentials=youtube_credentials,
cdg_styles=cdg_styles,
)
# Run the orchestrated pipeline
orchestrator = VideoWorkerOrchestrator(
config=config,
job_manager=job_manager,
storage=storage,
job_logger=job_log,
)
with job_span("orchestrator-run", job_id):
result = await orchestrator.run()
if not result.success:
raise Exception(result.error_message or "Orchestrator failed")
# Prepare distribution directory for native uploads
with job_span("distribution", job_id):
await _handle_native_distribution(
job_id=job_id,
job=job,
job_log=job_log,
job_manager=job_manager,
temp_dir=temp_dir,
result={
'brand_code': result.brand_code,
'youtube_url': result.youtube_url,
'final_video': result.final_video,
'final_video_lossy': result.final_video_lossy,
'final_video_720p': result.final_video_720p,
'final_karaoke_cdg_zip': result.final_karaoke_cdg_zip,
'dropbox_link': result.dropbox_link,
'gdrive_files': result.gdrive_files,
},
storage=storage,
)
# Upload generated files to GCS
job_manager.transition_to_state(
job_id=job_id,
new_status=JobStatus.PACKAGING,
progress=95,
message="Uploading final files"
)
with job_span("upload-results", job_id):
await _upload_results(job_id, job_manager, storage, temp_dir, {
'final_video': result.final_video,
'final_video_mkv': result.final_video_mkv,
'final_video_lossy': result.final_video_lossy,
'final_video_720p': result.final_video_720p,
'final_karaoke_cdg_zip': result.final_karaoke_cdg_zip,
'final_karaoke_txt_zip': result.final_karaoke_txt_zip,
})
# Mark job as complete
logger.info(f"[job:{job_id}] Video generation complete")
job_manager.transition_to_state(
job_id=job_id,
new_status=JobStatus.COMPLETE,
progress=100,
message="Karaoke generation complete!"
)
# Store result metadata in job
job_manager.update_job(job_id, {
'state_data': {
**job.state_data,
'brand_code': result.brand_code,
'youtube_url': result.youtube_url,
'dropbox_link': result.dropbox_link,
'gdrive_files': result.gdrive_files,
}
})
duration = time.time() - start_time
root_span.set_attribute("duration_seconds", duration)
root_span.set_attribute("brand_code", result.brand_code or '')
logger.info(f"[job:{job_id}] WORKER_END worker=video orchestrator=true status=success duration={duration:.1f}s")
return True
except Exception as e:
duration = time.time() - start_time
logger.error(f"[job:{job_id}] WORKER_END worker=video orchestrator=true status=error duration={duration:.1f}s error={e}")
job_manager.mark_job_failed(
job_id=job_id,
error_message=f"Video generation failed: {str(e)}",
error_details={"stage": "video_generation", "error": str(e)}
)
return False
finally:
# Restore original working directory
os.chdir(original_cwd)
# Remove log handler
for logger_name in VIDEO_WORKER_LOGGERS:
try:
logging.getLogger(logger_name).removeHandler(log_handler)
except Exception:
pass
# Cleanup rclone config file
if rclone_service:
rclone_service.cleanup()
# Cleanup temporary directory
if os.path.exists(temp_dir):
shutil.rmtree(temp_dir)
logger.debug(f"Cleaned up temp directory: {temp_dir}")
async def generate_video(job_id: str) -> bool:
"""
Generate final karaoke videos.
Routes to either the new orchestrator-based pipeline or the legacy
KaraokeFinalise-based pipeline based on USE_NEW_ORCHESTRATOR flag.
Args:
job_id: Job ID to process
Returns:
True if successful, False otherwise
"""
if USE_NEW_ORCHESTRATOR:
logger.info(f"[job:{job_id}] Using new orchestrator-based pipeline")
return await generate_video_orchestrated(job_id)
else:
logger.info(f"[job:{job_id}] Using legacy KaraokeFinalise pipeline")
return await generate_video_legacy(job_id)
async def generate_video_legacy(job_id: str) -> bool:
"""
Generate final karaoke videos using KaraokeFinalise (legacy path).
This is the original implementation that has divergent behavior between
GCE and local encoding paths. Kept for rollback purposes.
Args:
job_id: Job ID to process
Returns:
True if successful, False otherwise
"""
@@ -0,0 +1,628 @@
"""
Video Worker Orchestrator.
Coordinates the video generation pipeline stages in a unified way,
regardless of whether encoding happens locally or on GCE.
This resolves the code path divergence where GCE encoding bypassed
features like YouTube upload, Discord notifications, and CDG/TXT packaging.
Pipeline stages:
1. Setup - Download files, prepare directories
2. Packaging - CDG/TXT generation (if enabled)
3. Encoding - GCE or Local via EncodingBackend interface
4. Organization - Brand code, folder structure
5. Distribution - YouTube, Dropbox, Google Drive uploads
6. Notification - Discord notifications
"""
import logging
import os
import time
from dataclasses import dataclass, field
from typing import Optional, Dict, Any, List
from pathlib import Path
from backend.models.job import JobStatus
from backend.services.job_manager import JobManager
from backend.services.storage_service import StorageService
from backend.services.tracing import job_span, add_span_event
logger = logging.getLogger(__name__)
@dataclass
class OrchestratorConfig:
"""Configuration for the video worker orchestrator."""
job_id: str
artist: str
title: str
# Input file paths (in temp_dir)
title_video_path: str
karaoke_video_path: str
instrumental_audio_path: str
end_video_path: Optional[str] = None
lrc_file_path: Optional[str] = None
title_jpg_path: Optional[str] = None
# Output directory
output_dir: str = ""
# Feature flags
enable_cdg: bool = False
enable_txt: bool = False
enable_youtube_upload: bool = False
# Service configurations
brand_prefix: Optional[str] = None
discord_webhook_url: Optional[str] = None
youtube_credentials: Optional[Dict[str, Any]] = None
youtube_description_template: Optional[str] = None
cdg_styles: Optional[Dict[str, Any]] = None
# Dropbox/GDrive configuration
dropbox_path: Optional[str] = None
gdrive_folder_id: Optional[str] = None
# Keep existing brand code (for re-processing)
keep_brand_code: Optional[str] = None
# Encoding backend preference
encoding_backend: str = "auto" # "auto", "local", "gce"
# Additional options
dry_run: bool = False
non_interactive: bool = True
@dataclass
class OrchestratorResult:
"""Result from the video worker orchestrator."""
success: bool
error_message: Optional[str] = None
# Generated files
final_video: Optional[str] = None # Lossless 4K MP4
final_video_mkv: Optional[str] = None # Lossless 4K MKV
final_video_lossy: Optional[str] = None # Lossy 4K MP4
final_video_720p: Optional[str] = None # Lossy 720p MP4
final_karaoke_cdg_zip: Optional[str] = None
final_karaoke_txt_zip: Optional[str] = None
# Organization
brand_code: Optional[str] = None
# Distribution results
youtube_url: Optional[str] = None
dropbox_link: Optional[str] = None
gdrive_files: Optional[Dict[str, str]] = field(default_factory=dict)
# Timing
encoding_time_seconds: Optional[float] = None
total_time_seconds: Optional[float] = None
class VideoWorkerOrchestrator:
"""
Orchestrates the video generation pipeline.
This class coordinates all stages of video generation in a unified way,
ensuring that features like YouTube upload and Discord notifications
work regardless of whether GCE or local encoding is used.
"""
def __init__(
self,
config: OrchestratorConfig,
job_manager: Optional[JobManager] = None,
storage: Optional[StorageService] = None,
job_logger: Optional[logging.Logger] = None,
):
"""
Initialize the orchestrator.
Args:
config: Orchestrator configuration
job_manager: Job manager for state updates (optional)
storage: Storage service for file downloads (optional)
job_logger: Job-specific logger (optional)
"""
self.config = config
self.job_manager = job_manager
self.storage = storage
self.job_log = job_logger or logging.getLogger(__name__)
# Services (lazy-loaded)
self._encoding_backend = None
self._packaging_service = None
self._youtube_service = None
self._discord_service = None
# State
self.result = OrchestratorResult(success=False)
def _get_encoding_backend(self):
"""Lazy-load the encoding backend."""
if self._encoding_backend is None:
from backend.services.encoding_interface import get_encoding_backend
self._encoding_backend = get_encoding_backend(
backend_type=self.config.encoding_backend,
dry_run=self.config.dry_run,
logger=self.job_log,
)
return self._encoding_backend
def _get_packaging_service(self):
"""Lazy-load the packaging service."""
if self._packaging_service is None:
from backend.services.packaging_service import PackagingService
self._packaging_service = PackagingService(
cdg_styles=self.config.cdg_styles,
dry_run=self.config.dry_run,
non_interactive=self.config.non_interactive,
logger=self.job_log,
)
return self._packaging_service
def _get_youtube_service(self):
"""Lazy-load the YouTube upload service."""
if self._youtube_service is None:
from backend.services.youtube_upload_service import YouTubeUploadService
self._youtube_service = YouTubeUploadService(
credentials=self.config.youtube_credentials,
non_interactive=self.config.non_interactive,
server_side_mode=True,
dry_run=self.config.dry_run,
logger=self.job_log,
)
return self._youtube_service
def _get_discord_service(self):
"""Lazy-load the Discord notification service."""
if self._discord_service is None:
from backend.services.discord_service import DiscordNotificationService
self._discord_service = DiscordNotificationService(
webhook_url=self.config.discord_webhook_url,
dry_run=self.config.dry_run,
logger=self.job_log,
)
return self._discord_service
def _update_progress(self, status: JobStatus, progress: int, message: str):
"""Update job progress if job_manager is available."""
if self.job_manager:
self.job_manager.transition_to_state(
job_id=self.config.job_id,
new_status=status,
progress=progress,
message=message
)
async def run(self) -> OrchestratorResult:
"""
Run the full video generation pipeline.
Returns:
OrchestratorResult with generated files and metadata
"""
start_time = time.time()
try:
with job_span("orchestrator", self.config.job_id) as span:
self.job_log.info(f"Starting orchestrated video generation for {self.config.artist} - {self.config.title}")
# Stage 1: Packaging (CDG/TXT) - runs BEFORE encoding
# This depends on LRC file, not on encoded videos
if self.config.enable_cdg or self.config.enable_txt:
await self._run_packaging()
# Stage 2: Encoding
await self._run_encoding()
# Stage 3: Organization (brand code)
await self._run_organization()
# Stage 4: Distribution (YouTube, Dropbox, GDrive)
await self._run_distribution()
# Stage 5: Notifications (Discord)
await self._run_notifications()
self.result.success = True
self.result.total_time_seconds = time.time() - start_time
self.job_log.info(f"Orchestrated video generation complete in {self.result.total_time_seconds:.1f}s")
except Exception as e:
self.result.success = False
self.result.error_message = str(e)
self.result.total_time_seconds = time.time() - start_time
self.job_log.error(f"Orchestrated video generation failed: {e}")
logger.error(f"[job:{self.config.job_id}] Orchestration failed: {e}")
return self.result
async def _run_packaging(self):
"""Run the packaging stage (CDG/TXT generation)."""
self.job_log.info("Starting packaging stage (CDG/TXT)")
if not self.config.lrc_file_path or not os.path.isfile(self.config.lrc_file_path):
self.job_log.warning("No LRC file available, skipping CDG/TXT packaging")
return
base_name = f"{self.config.artist} - {self.config.title}"
packaging_service = self._get_packaging_service()
# Generate CDG package
if self.config.enable_cdg:
self.job_log.info("Generating CDG package")
try:
cdg_zip_path = os.path.join(
self.config.output_dir,
f"{base_name} (Final Karaoke CDG).zip"
)
mp3_path = os.path.join(
self.config.output_dir,
f"{base_name} (Karaoke).mp3"
)
cdg_path = os.path.join(
self.config.output_dir,
f"{base_name} (Karaoke).cdg"
)
zip_file, mp3_file, cdg_file = packaging_service.create_cdg_package(
lrc_file=self.config.lrc_file_path,
audio_file=self.config.instrumental_audio_path,
output_zip_path=cdg_zip_path,
artist=self.config.artist,
title=self.config.title,
output_mp3_path=mp3_path,
output_cdg_path=cdg_path,
)
self.result.final_karaoke_cdg_zip = zip_file
self.job_log.info(f"CDG package created: {zip_file}")
except Exception as e:
self.job_log.error(f"CDG generation failed: {e}")
# Don't fail the pipeline, CDG is optional
# Generate TXT package
if self.config.enable_txt:
self.job_log.info("Generating TXT package")
try:
# TXT package needs MP3 file (from CDG generation or create it)
mp3_path = os.path.join(
self.config.output_dir,
f"{base_name} (Karaoke).mp3"
)
if not os.path.isfile(mp3_path):
self.job_log.info("MP3 not found, CDG must be enabled first for TXT")
return
txt_zip_path = os.path.join(
self.config.output_dir,
f"{base_name} (Final Karaoke TXT).zip"
)
zip_file, txt_file = packaging_service.create_txt_package(
lrc_file=self.config.lrc_file_path,
mp3_file=mp3_path,
output_zip_path=txt_zip_path,
)
self.result.final_karaoke_txt_zip = zip_file
self.job_log.info(f"TXT package created: {zip_file}")
except Exception as e:
self.job_log.error(f"TXT generation failed: {e}")
# Don't fail the pipeline, TXT is optional
async def _run_encoding(self):
"""Run the encoding stage."""
self.job_log.info("Starting encoding stage")
self._update_progress(JobStatus.ENCODING, 75, "Encoding videos")
encoding_backend = self._get_encoding_backend()
self.job_log.info(f"Using encoding backend: {encoding_backend.name}")
# Build encoding input
from backend.services.encoding_interface import EncodingInput
encoding_input = EncodingInput(
title_video_path=self.config.title_video_path,
karaoke_video_path=self.config.karaoke_video_path,
instrumental_audio_path=self.config.instrumental_audio_path,
end_video_path=self.config.end_video_path,
artist=self.config.artist,
title=self.config.title,
brand_code=self.config.keep_brand_code,
output_dir=self.config.output_dir,
)
# Run encoding
with job_span("encoding", self.config.job_id) as span:
add_span_event("encoding_started", {"backend": encoding_backend.name})
output = await encoding_backend.encode(encoding_input)
add_span_event("encoding_completed", {
"success": output.success,
"duration": output.encoding_time_seconds or 0
})
if not output.success:
raise Exception(f"Encoding failed: {output.error_message}")
# Store results
self.result.final_video = output.lossless_4k_mp4_path
self.result.final_video_mkv = output.lossless_mkv_path
self.result.final_video_lossy = output.lossy_4k_mp4_path
self.result.final_video_720p = output.lossy_720p_mp4_path
self.result.encoding_time_seconds = output.encoding_time_seconds
self.job_log.info(f"Encoding complete ({encoding_backend.name}) in {output.encoding_time_seconds:.1f}s")
async def _run_organization(self):
"""Run the organization stage (brand code generation)."""
self.job_log.info("Starting organization stage")
# Use existing brand code if provided
if self.config.keep_brand_code:
self.result.brand_code = self.config.keep_brand_code
self.job_log.info(f"Using preserved brand code: {self.result.brand_code}")
return
# Generate brand code from Dropbox if configured
if self.config.dropbox_path and self.config.brand_prefix:
try:
from backend.services.dropbox_service import get_dropbox_service
dropbox = get_dropbox_service()
if dropbox.is_configured:
brand_code = dropbox.get_next_brand_code(
self.config.dropbox_path,
self.config.brand_prefix
)
self.result.brand_code = brand_code
self.job_log.info(f"Generated brand code: {brand_code}")
else:
self.job_log.warning("Dropbox not configured, skipping brand code generation")
except Exception as e:
self.job_log.error(f"Brand code generation failed: {e}")
# Don't fail - brand code is optional
async def _run_distribution(self):
"""Run the distribution stage (YouTube, Dropbox, GDrive uploads)."""
self.job_log.info("Starting distribution stage")
self._update_progress(JobStatus.PACKAGING, 90, "Uploading files")
# YouTube upload
if self.config.enable_youtube_upload and self.config.youtube_credentials:
await self._upload_to_youtube()
# Dropbox upload
if self.config.dropbox_path and self.config.brand_prefix:
await self._upload_to_dropbox()
# Google Drive upload
if self.config.gdrive_folder_id:
await self._upload_to_gdrive()
async def _upload_to_youtube(self):
"""Upload video to YouTube."""
self.job_log.info("Uploading to YouTube")
# Find the best video file to upload (prefer MKV for FLAC audio, then lossless MP4)
video_to_upload = None
if self.result.final_video_mkv and os.path.isfile(self.result.final_video_mkv):
video_to_upload = self.result.final_video_mkv
elif self.result.final_video and os.path.isfile(self.result.final_video):
video_to_upload = self.result.final_video
elif self.result.final_video_lossy and os.path.isfile(self.result.final_video_lossy):
video_to_upload = self.result.final_video_lossy
if not video_to_upload:
self.job_log.warning("No video file available for YouTube upload")
return
try:
youtube_service = self._get_youtube_service()
# Build video title
title = f"{self.config.artist} - {self.config.title} (Karaoke)"
# Build description
description = self.config.youtube_description_template or ""
if self.result.brand_code:
description = f"{description}\n\nBrand Code: {self.result.brand_code}".strip()
# Upload
video_id, video_url = youtube_service.upload_video(
video_path=video_to_upload,
title=title,
description=description,
thumbnail_path=self.config.title_jpg_path,
tags=["karaoke", self.config.artist, self.config.title],
replace_existing=True, # Server-side always replaces
)
if video_url:
self.result.youtube_url = video_url
self.job_log.info(f"Uploaded to YouTube: {video_url}")
else:
self.job_log.warning("YouTube upload did not return a URL")
except Exception as e:
self.job_log.error(f"YouTube upload failed: {e}")
# Don't fail the pipeline - YouTube is optional
async def _upload_to_dropbox(self):
"""Upload files to Dropbox."""
self.job_log.info("Uploading to Dropbox")
try:
from backend.services.dropbox_service import get_dropbox_service
dropbox = get_dropbox_service()
if not dropbox.is_configured:
self.job_log.warning("Dropbox not configured, skipping upload")
return
base_name = f"{self.config.artist} - {self.config.title}"
folder_name = f"{self.result.brand_code or 'TRACK-0000'} - {base_name}"
remote_folder = f"{self.config.dropbox_path}/{folder_name}"
# Upload entire output directory
dropbox.upload_folder(self.config.output_dir, remote_folder)
# Create sharing link
try:
sharing_link = dropbox.create_shared_link(remote_folder)
self.result.dropbox_link = sharing_link
self.job_log.info(f"Dropbox sharing link: {sharing_link}")
except Exception as e:
self.job_log.warning(f"Failed to create Dropbox sharing link: {e}")
self.job_log.info("Dropbox upload complete")
except Exception as e:
self.job_log.error(f"Dropbox upload failed: {e}")
# Don't fail the pipeline - Dropbox is optional
async def _upload_to_gdrive(self):
"""Upload files to Google Drive."""
self.job_log.info("Uploading to Google Drive")
try:
from backend.services.gdrive_service import get_gdrive_service
gdrive = get_gdrive_service()
if not gdrive.is_configured:
self.job_log.warning("Google Drive not configured, skipping upload")
return
base_name = f"{self.config.artist} - {self.config.title}"
brand_code = self.result.brand_code or f"{self.config.brand_prefix or 'TRACK'}-0000"
# Map result files to expected keys
output_files = {
'final_karaoke_lossy_mp4': self.result.final_video_lossy,
'final_karaoke_lossy_720p_mp4': self.result.final_video_720p,
'final_karaoke_cdg_zip': self.result.final_karaoke_cdg_zip,
}
uploaded = gdrive.upload_to_public_share(
root_folder_id=self.config.gdrive_folder_id,
brand_code=brand_code,
base_name=base_name,
output_files=output_files,
)
self.result.gdrive_files = uploaded
self.job_log.info(f"Google Drive upload complete: {len(uploaded)} files")
except Exception as e:
self.job_log.error(f"Google Drive upload failed: {e}")
# Don't fail the pipeline - GDrive is optional
async def _run_notifications(self):
"""Run the notifications stage (Discord)."""
self.job_log.info("Starting notifications stage")
if not self.config.discord_webhook_url:
self.job_log.debug("No Discord webhook configured, skipping notification")
return
if not self.result.youtube_url:
self.job_log.info("No YouTube URL available, skipping Discord notification")
return
try:
discord_service = self._get_discord_service()
discord_service.post_video_notification(self.result.youtube_url)
self.job_log.info("Discord notification sent")
except Exception as e:
self.job_log.error(f"Discord notification failed: {e}")
# Don't fail the pipeline - notifications are optional
def create_orchestrator_config_from_job(
job,
temp_dir: str,
youtube_credentials: Optional[Dict[str, Any]] = None,
cdg_styles: Optional[Dict[str, Any]] = None,
) -> OrchestratorConfig:
"""
Create an OrchestratorConfig from a job object.
This is a helper function to bridge the existing job structure
with the new orchestrator configuration.
Args:
job: Job object from Firestore
temp_dir: Temporary directory with downloaded files
youtube_credentials: Pre-loaded YouTube credentials
cdg_styles: CDG style configuration
Returns:
OrchestratorConfig for the orchestrator
"""
base_name = f"{job.artist} - {job.title}"
# Determine instrumental file path
instrumental_selection = job.state_data.get('instrumental_selection', 'clean')
existing_instrumental = getattr(job, 'existing_instrumental_gcs_path', None)
if existing_instrumental:
ext = Path(existing_instrumental).suffix.lower()
instrumental_path = os.path.join(temp_dir, f"{base_name} (Instrumental User){ext}")
else:
instrumental_suffix = "Clean" if instrumental_selection == 'clean' else "Backing"
instrumental_path = os.path.join(temp_dir, f"{base_name} (Instrumental {instrumental_suffix}).flac")
return OrchestratorConfig(
job_id=job.job_id,
artist=job.artist,
title=job.title,
# Input files
title_video_path=os.path.join(temp_dir, f"{base_name} (Title).mov"),
karaoke_video_path=os.path.join(temp_dir, f"{base_name} (With Vocals).mov"),
instrumental_audio_path=instrumental_path,
end_video_path=os.path.join(temp_dir, f"{base_name} (End).mov"),
lrc_file_path=os.path.join(temp_dir, f"{base_name} (Karaoke).lrc"),
title_jpg_path=os.path.join(temp_dir, f"{base_name} (Title).jpg"),
# Output directory
output_dir=temp_dir,
# Feature flags
enable_cdg=getattr(job, 'enable_cdg', False),
enable_txt=getattr(job, 'enable_txt', False),
enable_youtube_upload=getattr(job, 'enable_youtube_upload', False),
# Service configurations
brand_prefix=getattr(job, 'brand_prefix', None),
discord_webhook_url=getattr(job, 'discord_webhook_url', None),
youtube_credentials=youtube_credentials,
youtube_description_template=getattr(job, 'youtube_description_template', None),
cdg_styles=cdg_styles,
# Dropbox/GDrive
dropbox_path=getattr(job, 'dropbox_path', None),
gdrive_folder_id=getattr(job, 'gdrive_folder_id', None),
# Keep existing brand code
keep_brand_code=getattr(job, 'keep_brand_code', None),
# Encoding backend - auto selects GCE if available
encoding_backend="auto",
# Server-side defaults
dry_run=False,
non_interactive=True,
)
+1 -1
View File
@@ -1,6 +1,6 @@
[tool.poetry]
name = "karaoke-gen"
version = "0.88.0"
version = "0.89.0"
description = "Generate karaoke videos with synchronized lyrics. Handles the entire process from downloading audio and lyrics to creating the final video with title screens."
authors = ["Andrew Beveridge <andrew@beveridge.uk>"]
license = "MIT"