Compare commits

..
Author SHA1 Message Date
Andrew Beveridge 8aa10fd090 chore: force backend deployment 2026-01-04 15:07:03 -05:00
Andrew Beveridge ☄️andClaude Opus 4.5 7acc2c9f17 feat: Add admin endpoint to reset stale worker state (#188)
## Summary
- Add `/api/admin/jobs/{id}/reset-worker-state` endpoint
- Allows admins to reset stale worker progress when jobs get stuck

## Context
When a job fails, the worker progress (e.g., `video_progress.stage =
'running'`)
can get stuck, preventing retries. This endpoint resets all worker
progress to 'pending'.

## Testing
- [ ] Test with job `501258e1` after deployment

@coderabbitai ignore

---
🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-authored-by: Claude Opus 4.5 <noreply@anthropic.com>
2026-01-04 20:00:21 +00:00
Andrew Beveridge ☄️andClaude Opus 4.5 a3f5c68660 fix: Reset worker progress state on job retry (#187)
## Summary
- Fix job retry getting stuck because worker progress state wasn't reset
- When retrying, idempotency check saw `video_progress.stage =
'running'` from previous failed run
- Now properly resets progress state to 'pending' for video, render, and
screens workers

## Changes
- `backend/api/routes/jobs.py` - Add `update_state_data` calls to reset
progress

## Testing
- [x] Encoding tests pass locally
- [ ] Test with job `501258e1` after deployment

@coderabbitai ignore

---
🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-authored-by: Claude Opus 4.5 <noreply@anthropic.com>
2026-01-04 19:24:02 +00:00
Andrew Beveridge ☄️andClaude Opus 4.5 27495fb4fe fix: Handle list response from GCE encoding worker (#186)
## Summary
- Fix "'list' object has no attribute 'get'" error when GCE worker
returns a list instead of dict
- Add defensive handling in `GCEEncodingBackend.encode()` to extract
first dict from list responses
- Add local tests to catch encoding integration issues earlier

## Changes
- `backend/services/encoding_interface.py` - Handle list/non-dict
responses from GCE worker
- `backend/tests/test_encoding_interface.py` - Add tests for list
response handling and backend options
- `backend/tests/test_video_worker_orchestrator.py` - Add test for GCS
paths structure
- `backend/tests/conftest.py` - Move env var setup to top before imports

## Testing
- [x] All 57 tests pass locally (`make test`)
- [x] `test_encode_handles_list_result` validates fix
- [ ] Deploy and test with existing job at
`awaiting_instrumental_selection`

@coderabbitai ignore

---
🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 4.5 <noreply@anthropic.com>
2026-01-04 18:48:22 +00:00
Andrew Beveridge ☄️andClaude Opus 4.5 b32014a9a0 fix: Pass GCS paths to encoding backend for GCE encoding (#185)
## Summary
- Fixed `GCE encoding requires input_gcs_path and output_gcs_path in
options` error
- Jobs were failing at video generation because orchestrator wasn't
passing required GCS paths

## Changes
- Added GCS path construction in
`VideoWorkerOrchestrator._run_encoding()`
- Paths are now passed in `EncodingInput.options` dict

## Testing
- [x] All 27 orchestrator tests pass
- [x] Ready for quick deploy and test with existing job at
`awaiting_instrumental_selection`

@coderabbitai ignore

---
🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 4.5 <noreply@anthropic.com>
2026-01-04 17:34:35 +00:00
Andrew Beveridge ☄️andClaude Opus 4.5 bb19a6e1fc fix: Add dry_run parameter to GCEEncodingBackend (#184)
## Summary
- Fixed `GCEEncodingBackend.__init__() got an unexpected keyword
argument 'dry_run'` error
- Jobs were failing during video generation when using GCE encoding

## Changes
- Added `dry_run` parameter to `GCEEncodingBackend.__init__()` for
interface consistency with `LocalEncodingBackend`
- Parameter is stored but not used since GCE is a remote service that
doesn't support dry run mode

## Testing
- [x] All 23 encoding interface tests pass
- [x] No breaking changes to existing functionality

@coderabbitai ignore

---
🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 4.5 <noreply@anthropic.com>
2026-01-04 11:54:27 -05:00
Andrew Beveridge ☄️andClaude Opus 4.5 1b0521cb83 docs: Add video worker orchestrator documentation (#183)
## Summary
- Add entry to README.md Recent Changes section for orchestrator
refactor
- Add Video Worker Orchestrator section to ARCHITECTURE.md with diagram
- Add "Alternative Code Paths Must Implement All Features" lesson to
LESSONS-LEARNED.md
- Create archive entry documenting the refactor

## Changes
- `docs/README.md` - Added recent change entry
- `docs/ARCHITECTURE.md` - Added orchestrator section with stage diagram
- `docs/LESSONS-LEARNED.md` - Added lesson about code path divergence
- `docs/archive/2026-01-04-video-worker-orchestrator-refactor.md` - Full
documentation

## Context
Documentation for PR #182 which unified the video worker pipeline.

@coderabbitai ignore

---
🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 4.5 <noreply@anthropic.com>
2026-01-04 11:07:16 -05:00
Andrew Beveridge ☄️ c1588992ce refactor: Unify video worker pipeline with new orchestrator (#182)
Resolves GCE vs KaraokeFinalise code path divergence that caused features like YouTube upload, Discord notifications, and CDG/TXT packaging to be missing from the GCE encoding path.

- New services: youtube_upload_service, discord_service, packaging_service, local_encoding_service, encoding_interface
- New VideoWorkerOrchestrator coordinates all stages (packaging, encoding, distribution, notifications)
- Works with any encoding backend (GCE or local)
- 139 new tests, 976 total tests passing
- Feature flag USE_NEW_ORCHESTRATOR (default: true) with rollback capability
2026-01-04 00:25:23 -05:00
Andrew Beveridge ☄️andClaude b3045c5487 fix: Add Noto fonts for full Unicode support in video rendering (#181)
## Summary

- Fixes rendering of musical symbols (♪) in INTRO/INSTRUMENTAL/OUTRO
screens
- Adds comprehensive international font support for CJK and other
scripts
- Changes default karaoke font from Arial to Noto Sans

## Changes

- **backend/Dockerfile.base**: Install Noto font packages (core, CJK,
extra, color-emoji) with fontconfig
- **karaoke_gen/style_loader.py**: Change default font from "Arial" to
"Noto Sans"
- **backend/tests/test_routes_review.py**: Update test assertions to
match new default font
- **docs/LESSONS-LEARNED.md**: Add "Fonts in Docker for Video Rendering"
section
- **docs/README.md**: Add "Full Unicode Font Support" to Recent Changes

## Testing

- [x] Tests pass locally (`make test`)
- [x] Manual review of changes completed

## Notes

- Adds ~200-300MB to Docker base image for comprehensive font coverage
- Color emoji has limited support in FFmpeg/libass (renders as
monochrome)
- Requires base image rebuild to take effect

@coderabbitai ignore

---
🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-authored-by: Claude <noreply@anthropic.com>
2026-01-03 23:11:30 -05:00
Andrew Beveridge ☄️andClaude 8a7cbb4a8b feat: Integrate flacfetch cache management API (#180)
## Summary

- Integrate karaoke-gen with flacfetch's new cache management API
(v0.15.4)
- When admins clear a job's cache, both Firestore AND flacfetch GCS
cache are now cleared
- Add bulk cache management (clear all, view stats) to admin UI

## Changes

### Backend
- **FlacfetchClient** (`backend/services/flacfetch_client.py`): Add
cache management methods
  - `clear_search_cache(artist, title)` - clears specific cache entry
  - `clear_all_cache()` - clears entire flacfetch cache  
  - `get_cache_stats()` - returns cache statistics

- **Admin API** (`backend/api/routes/admin.py`): 
- Update `POST /api/admin/audio-searches/{job_id}/clear-cache` to also
call flacfetch cache API
  - Add `DELETE /api/admin/cache` - clear entire flacfetch cache
  - Add `GET /api/admin/cache/stats` - get cache statistics

### Frontend
- **API Client** (`frontend/lib/api.ts`): Add `clearAllCache()` and
`getCacheStats()` methods
- **Admin Searches Page** (`frontend/app/admin/searches/page.tsx`):
  - Add "Clear All Cache" button with confirmation dialog
  - Display cache stats (entry count, size, newest entry)

### Documentation
- Update API.md with new endpoints
- Update README.md with recent changes
- Update LESSONS-LEARNED.md with two-layer caching insight

## Architecture

```
User clicks "Clear Cache" on job
    ↓
karaoke-gen clears Firestore state_data (existing)
    ↓
karaoke-gen calls flacfetch DELETE /cache/search (NEW)
    ↓
flacfetch deletes GCS cache entry
    ↓
Next search hits trackers fresh ✓
```

## Testing

- [x] Backend tests pass (`make test` - 57 tests)
- [x] Frontend tests pass (`npm run test:unit` - 128 tests)
- [x] Frontend builds successfully
- [x] CodeRabbit CLI review completed locally

@coderabbitai ignore

---
🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude <noreply@anthropic.com>
2026-01-03 22:48:29 -05:00
Andrew Beveridge ☄️andClaude Opus 4.5 3ffed23880 feat: Add customer support link to app header (#179)
## Summary
- Add "Need help?" mailto link visible in header on large screens
- Add support contact option in user dropdown menu for mobile/tablet
users
- Links to andrew@nomadkaraoke.com

## Test plan
- [ ] Verify link appears in header on desktop (lg breakpoint and above)
- [ ] Verify link appears in user dropdown menu
- [ ] Verify clicking opens email client with correct address

@coderabbitai ignore

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-authored-by: Claude Opus 4.5 <noreply@anthropic.com>
2026-01-03 21:01:47 -05:00
Andrew Beveridge ☄️andClaude Opus 4.5 ac39cc5ff7 fix: Add admin link to AuthStatus component dropdown (#178)
## Summary
- The `/app` page uses the `AuthStatus` component for its header
dropdown menu, not the `app-header.tsx` component
- This adds the admin link for users with admin role or
@nomadkaraoke.com email to the correct component

## Test plan
- [ ] Log in as admin user at https://gen.nomadkaraoke.com/app
- [ ] Click user dropdown in header
- [ ] Verify "Admin Dashboard" link appears between "Buy More Credits"
and "Sign Out"

@coderabbitai ignore

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-authored-by: Claude Opus 4.5 <noreply@anthropic.com>
2026-01-03 20:27:43 -05:00
Andrew Beveridge ☄️andClaude d47634016d feat: Add admin link to header for admin users (#177)
## Summary

- Add "Admin" link to header navigation for users with admin access
- Add "Admin Dashboard" option to user dropdown menu
- Uses same access logic as admin layout (role=admin OR
@nomadkaraoke.com email)

## Changes

- `frontend/components/app-header.tsx` - Added admin link to nav and
dropdown menu with Shield icon

## Testing

- [x] TypeScript compiles
- [ ] Manual testing needed

@coderabbitai ignore

---
🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-authored-by: Claude <noreply@anthropic.com>
2026-01-03 20:20:04 -05:00
Andrew Beveridge ☄️andClaude 68188e9a28 feat: Add admin UI for audio search cache management (#176)
## Summary

- Add admin UI at `/admin/searches` to view and manage cached audio
search results
- Allow admins to clear stale caches when flacfetch is updated with new
providers
- Fixes issue where jobs showed only YouTube results when lossless
sources should be available

## Changes

- **Backend**: New admin endpoints `GET /api/admin/audio-searches` and
`POST /api/admin/audio-searches/{job_id}/clear-cache`
- **Frontend**: New admin page with search results table, status
indicators, and cache clear action
- **Safety**: Status validation prevents clearing cache on
active/complete jobs
- **Docs**: Updated API.md, LESSONS-LEARNED.md, and README.md

## Testing

- [x] Backend module imports successfully
- [x] Frontend TypeScript compiles without errors
- [x] CodeRabbit CLI review completed and issues addressed

## Review

- [x] CodeRabbit CLI review completed locally (3 cycles)
- [x] Fixed: `has_lossless` now computed from ALL results, not just
first 10
- [x] Fixed: Added status validation before cache clear
- [x] Noted (not fixed): Scalability optimization for large-scale -
acceptable for admin tool

@coderabbitai ignore

---
🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-authored-by: Claude <noreply@anthropic.com>
2026-01-03 20:08:43 -05:00
Andrew Beveridge ☄️andClaude Opus 4.5 cbf3b815dd docs: Add gen_users migration documentation (#174)
## Summary
Adds documentation for the gen_users migration completed in PR #173.

- **README.md**: Add entry in Recent Changes about user database
separation
- **ARCHITECTURE.md**: Add Firestore Collections section documenting
`gen_users` and other collections
- **LESSONS-LEARNED.md**: Add lesson about separate collections for
multi-app projects

## Test plan
- [x] Documentation-only change, no testing required

@coderabbitai ignore

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 4.5 <noreply@anthropic.com>
2026-01-03 18:09:57 -05:00
Andrew Beveridge ☄️andClaude Opus 4.5 6aa8f150df docs: Add GCS path lessons and update E2E spec for testmail.app (#175)
## Summary

- Add "GCS Blob Paths vs Full URIs" section to LESSONS-LEARNED.md
documenting the path format confusion bug and double-slash gotcha that
was fixed in PR #168
- Update E2E-HAPPY-PATH-TEST-SPEC.md secrets section to correctly
reflect testmail.app migration (was referencing MAILSLURP)

## Test plan

- [ ] Documentation review - verify changes are accurate and helpful

@coderabbitai ignore

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-authored-by: Claude Opus 4.5 <noreply@anthropic.com>
2026-01-03 18:08:09 -05:00
Andrew Beveridge ☄️andClaude Opus 4.5 925dd3c262 fix: Complete mailslurp to testmail.app migration (#168)
## Summary
- Complete the migration from MailSlurp to testmail.app for E2E testing
- Update all workflow files to use `TESTMAIL_API_KEY` and
`TESTMAIL_NAMESPACE` secrets
- Update documentation and test comments

## Changes
- `.github/workflows/e2e-happy-path.yml` - Use testmail.app secrets
- `.github/workflows/ci.yml` - Use testmail.app secrets  
- `docs/E2E-HAPPY-PATH-TEST-SPEC.md` - Update documentation
- `frontend/e2e/production/happy-path-real-user.spec.ts` - Update
comments

## Test plan
- [x] Verify no remaining mailslurp references in active files (archives
excluded)
- [ ] E2E Happy Path test should run (not skip) with testmail.app
secrets

🤖 Generated with [Claude Code](https://claude.com/claude-code)

@coderabbitai ignore

---------

Co-authored-by: Claude Opus 4.5 <noreply@anthropic.com>
2026-01-03 17:32:53 -05:00
42 changed files with 7579 additions and 76 deletions

No files matched your search

@@ -0,0 +1,401 @@
# Plan: Video Worker Stage Refactor
**Created:** 2026-01-03
**Branch:** feat/sess-20260103-2250-investigate-youtube-upload
**Status:** COMPLETE - Ready for PR
**Last Updated:** 2026-01-04
## Overview
Refactor `video_worker.py` and `KaraokeFinalise` into separate, composable stages following SOLID principles. This eliminates the GCE vs KaraokeFinalise code path divergence that causes features like YouTube upload to be missing from the GCE encoding path.
## Three Execution Modes
Understanding the three ways this system runs is critical to the refactor:
### 1. Local CLI (`karaoke-gen`)
- Runs entirely on user's machine
- KaraokeFinalise.process() orchestrates everything
- Writes directly to local Dropbox sync folder (e.g., `~/Dropbox/Karaoke/Tracks-Organized/`)
- **Produces full output structure** with stems/, lyrics/, visualizations/, etc.
- **This is the source of truth for output format**
### 2. Remote CLI (`karaoke-gen-remote`)
- User's CLI talks to cloud backend (api.nomadkaraoke.com)
- Cloud does ALL processing (audio, lyrics, video encoding)
- CLI just monitors job status and downloads final outputs
- With Dropbox configured: cloud uploads full structure to user's Dropbox
- Without Dropbox: user downloads individual files
### 3. Web Frontend (`gen.nomadkaraoke.com`)
- Same cloud backend as remote CLI
- User sees:
- YouTube video link
- Dropbox folder link
- Direct download: final karaoke video (720p)
- Direct download: 4K video
- Direct download: "with vocals" video
- Direct download: CDG zip (if enabled)
## Problem Statement
Currently, `video_worker.py` has two completely divergent code paths:
```python
if use_gce_encoding:
# GCE PATH: Only encoding, missing YouTube/Discord/CDG/TXT
result = await _encode_via_gce(...)
# result DOES NOT include youtube_url!
else:
# STANDARD PATH: KaraokeFinalise does everything
finalise = KaraokeFinalise(...)
result = finalise.process()
# result includes youtube_url from upload
```
### What GCE Encoding Broke
| Feature | KaraokeFinalise Path | GCE Path |
|---------|---------------------|----------|
| Video encoding | ✅ 4 formats | ✅ 4 formats |
| YouTube upload | ✅ | ❌ **BROKEN** |
| Discord notification | ✅ | ❌ **BROKEN** |
| CDG/TXT packaging | ✅ | ❌ **BROKEN** |
| Brand code generation | ✅ | ✅ (via Dropbox service) |
| Dropbox upload | ✅ (rclone) | ✅ (native SDK) |
| Google Drive upload | ❌ | ✅ (native SDK) |
### Root Cause
`KaraokeFinalise.process()` is a monolith that handles:
1. CDG/TXT packaging
2. Video encoding (the CPU-intensive part)
3. YouTube upload
4. Discord notifications
5. Folder organization
The GCE encoding was added as a bypass around the entire class, rather than replacing just the encoding step.
## Requirements
### Functional Requirements
- [ ] YouTube upload works regardless of encoding backend (GCE or local)
- [ ] Discord notifications work regardless of encoding backend
- [ ] CDG/TXT packaging works regardless of encoding backend
- [ ] Brand code generation works consistently
- [ ] Dropbox/Google Drive upload works (already works in both paths)
- [ ] All existing CLI functionality continues to work
- [ ] Output structure matches local CLI exactly
### Output Structure Parity
Cloud backend must produce identical output to local CLI:
```
NOMAD-XXXX - Artist - Title/
├── Artist - Title (Title).mov
├── Artist - Title (Title).jpg
├── Artist - Title (End).mov
├── Artist - Title (End).jpg
├── Artist - Title (Final Karaoke Lossless 4k).mp4
├── Artist - Title (Final Karaoke Lossless 4k).mkv
├── Artist - Title (Final Karaoke Lossy 4k).mp4
├── Artist - Title (Final Karaoke Lossy 720p).mp4
├── Artist - Title (Final Karaoke CDG).zip
├── Artist - Title (Final Karaoke TXT).zip
├── Artist - Title (Instrumental {model}).flac
├── Artist - Title (Instrumental +BV {model}).flac
├── Artist - Title (Karaoke).mp4
├── Artist - Title (Karaoke).cdg
├── Artist - Title (Karaoke).mp3
├── Artist - Title (Karaoke).lrc
├── Artist - Title (With Vocals).mp4
├── stems/
│ ├── Artist - Title (Instrumental {model}).flac
│ ├── Artist - Title (Vocals {model}).flac
│ ├── Artist - Title (Lead Vocals {model}).flac
│ ├── Artist - Title (Backing Vocals {model}).flac
│ └── ... (other stems)
├── lyrics/
│ ├── Artist - Title (Karaoke).lrc
│ ├── Artist - Title (Karaoke).ass
│ ├── Artist - Title (Karaoke).srt
│ └── ... (corrections JSONs)
├── previews/
├── visualizations/
└── styles_assets/
```
### Non-Functional Requirements
- [ ] Single Responsibility: Each stage has one job
- [ ] Open/Closed: Easy to add new encoding backends without modifying stages
- [ ] Testable: Each stage can be unit tested in isolation
- [ ] No duplicate code between paths
- [ ] Backwards compatible with existing KaraokeFinalise CLI usage
## Technical Approach
### Architecture: Stage-Based Pipeline
```
video_worker.py (VideoWorkerOrchestrator)
│
├── 1. EncodingStage (interface)
│ ├── LocalEncodingService (wraps KaraokeFinalise encoding methods)
│ └── GCEEncodingService (existing encoding_service.py)
│
├── 2. PackagingStage
│ ├── CDGPackager (extracts from KaraokeFinalise)
│ └── TXTPackager (extracts from KaraokeFinalise)
│
├── 3. DistributionStage
│ ├── YouTubeUploadService (extracts from KaraokeFinalise)
│ ├── DropboxService (existing)
│ ├── GoogleDriveService (existing)
│ └── DiscordService (extracts from KaraokeFinalise)
│
└── 4. OrganizationStage
└── BrandCodeService (extracts from KaraokeFinalise + DropboxService)
```
### Key Design Decisions
1. **Extract, Don't Rewrite**: Pull methods out of KaraokeFinalise into services, don't rewrite from scratch
2. **Keep KaraokeFinalise for CLI**: Local CLI still uses KaraokeFinalise.process() - we extract services that it can also use
3. **Interface-Based Encoding**: EncodingStage interface allows swapping GCE/local without changing orchestration
4. **Existing Service Pattern**: Follow singleton + factory pattern already used in codebase
5. **Local CLI stays sync**: Local karaoke-gen CLI remains synchronous as it is today
### Data Flow
```
JobConfig (from Firestore)
│
▼
┌─────────────────────────────────────────────────────────────────┐
│ VideoWorkerOrchestrator │
│ ┌─────────────────────────────────────────────────────────────┐│
│ │ 1. Setup: Download files, prepare directories ││
│ │ - Download title.mov, end.mov, with_vocals.mkv ││
│ │ - Download instrumental audio ││
│ │ - Prepare temp directory ││
│ └─────────────────────────────────────────────────────────────┘│
│ │ │
│ ▼ │
│ ┌─────────────────────────────────────────────────────────────┐│
│ │ 2. Packaging: CDG/TXT (if enabled) - runs BEFORE encoding ││
│ │ Reason: CDG depends on LRC file which is available ││
│ │ Input: LRC file, instrumental audio ││
│ │ Output: CDG.zip, TXT.zip, .cdg, .mp3 ││
│ └─────────────────────────────────────────────────────────────┘│
│ │ │
│ ▼ │
│ ┌─────────────────────────────────────────────────────────────┐│
│ │ 3. Encoding: GCE or Local (via interface) ││
│ │ Input: title.mov, karaoke.mov, end.mov ││
│ │ Output: 4K lossless, 4K lossy, 720p, MKV ││
│ └─────────────────────────────────────────────────────────────┘│
│ │ │
│ ▼ │
│ ┌─────────────────────────────────────────────────────────────┐│
│ │ 4. Organization: Brand code, folder structure ││
│ │ - Generate brand code (NOMAD-XXXX) ││
│ │ - Prepare full output directory structure ││
│ └─────────────────────────────────────────────────────────────┘│
│ │ │
│ ▼ │
│ ┌─────────────────────────────────────────────────────────────┐│
│ │ 5. Distribution: Upload to destinations ││
│ │ - YouTube upload (if enabled) ││
│ │ - Dropbox upload (if configured) ││
│ │ - Google Drive upload (if configured) ││
│ └─────────────────────────────────────────────────────────────┘│
│ │ │
│ ▼ │
│ ┌─────────────────────────────────────────────────────────────┐│
│ │ 6. Notification: Discord, Email ││
│ │ Input: YouTube URL, brand code, Dropbox link ││
│ └─────────────────────────────────────────────────────────────┘│
└─────────────────────────────────────────────────────────────────┘
│
▼
Result (youtube_url, brand_code, dropbox_link, file_paths, etc.)
```
## Implementation Steps
### Phase 1: Extract Services from KaraokeFinalise (Non-Breaking) ✅ COMPLETE
These services can be used by both the cloud orchestrator AND KaraokeFinalise.
1. [x] **Create `backend/services/youtube_upload_service.py`** ✅
- Extract YouTube upload logic from KaraokeFinalise
- Methods: `upload_video()`, `check_duplicate()`, `delete_video()`, `truncate_title()`
- Supports pre-loaded credentials (server-side) or client secrets file (CLI)
- 23 unit tests with mocked YouTube API
2. [x] **Create `backend/services/discord_service.py`** ✅
- Extract Discord notification from KaraokeFinalise
- Methods: `post_message()`, `post_video_notification()`, `is_enabled()`
- Simple HTTP POST to webhook with validation
- 20 unit tests with mocked requests
3. [x] **Create `backend/services/packaging_service.py`** ✅
- Extract CDG/TXT generation from KaraokeFinalise
- Methods: `create_cdg_package()`, `create_txt_package()`
- Uses lyrics_converter and CDGGenerator (existing dependencies)
- 18 unit tests with temp files
4. [x] **Create `backend/services/local_encoding_service.py`** ✅
- Extract encoding methods from KaraokeFinalise
- Methods: `encode_all_formats()`, individual encoding methods
- Hardware acceleration detection with software fallback
- 28 unit tests with mocked subprocess
### Phase 2: Create Encoding Interface ✅ COMPLETE
5. [x] **Create `backend/services/encoding_interface.py`** ✅
- Define ABC `EncodingBackend` with methods: `encode()`, `is_available()`, `get_status()`
- `EncodingInput` and `EncodingOutput` dataclasses for config/results
- Factory function `get_encoding_backend()` with "local", "gce", "auto" options
6. [x] **`LocalEncodingBackend`** ✅
- Implements `EncodingBackend` interface
- Wraps `LocalEncodingService` with `asyncio.to_thread()`
- 17 unit tests
7. [x] **`GCEEncodingBackend`** ✅
- Implements `EncodingBackend` interface
- Wraps existing `EncodingService` (encoding_service.py)
- Requires GCS paths in options for cloud execution
- 6 additional unit tests
### Phase 3: Create Orchestrator ✅ COMPLETE
8. [x] **Create `backend/workers/video_worker_orchestrator.py`** ✅
- New class that coordinates all stages
- Replaces the branching logic in `generate_video()`
- Methods:
- `__init__(config, job_manager, storage, job_logger)`
- `async run() -> OrchestratorResult`
- `_run_packaging()` - CDG/TXT
- `_run_encoding()` - GCE or local via EncodingBackend interface
- `_run_organization()` - Brand code, folder structure
- `_run_distribution()` - YouTube, Dropbox, GDrive
- `_run_notifications()` - Discord
- 27 unit tests (test_video_worker_orchestrator.py)
9. [x] **Update `backend/workers/video_worker.py`** ✅
- Added `USE_NEW_ORCHESTRATOR` feature flag (default: true)
- `generate_video()` routes to orchestrator or legacy path
- `generate_video_orchestrated()` - new unified pipeline
- `generate_video_legacy()` - original code for rollback
- Legacy GCE and KaraokeFinalise paths preserved for safety
### Phase 4: Update KaraokeFinalise to Use Services (OPTIONAL)
10. [ ] **Update `karaoke_gen/karaoke_finalise/karaoke_finalise.py`**
- Import and use the new services
- Replace inline YouTube/Discord/CDG code with service calls
- Keep process() working for CLI backwards compatibility
- Services become the single source of truth
- **NOTE**: This is optional for MVP - the orchestrator already uses the extracted services directly
### Phase 5: Testing & Validation ✅ COMPLETE
11. [x] **Add unit tests for each service** ✅
- test_youtube_upload_service.py (23 tests)
- test_discord_service.py (20 tests)
- test_packaging_service.py (18 tests)
- test_local_encoding_service.py (28 tests)
- test_encoding_interface.py (23 tests)
- test_video_worker_orchestrator.py (27 tests)
- **Total: 139 new tests**
12. [x] **Full test suite passes** ✅
- 976 tests passed (86 skipped - emulator/integration tests)
- No regressions from changes
13. [ ] **Manual E2E testing** (After deployment)
- [ ] GCE encoding job with YouTube upload enabled
- [ ] Local encoding job with YouTube upload enabled
- [ ] CLI karaoke-gen with all features
- [ ] Job with CDG/TXT enabled
- [ ] Verify output structure matches local CLI
## Files Created/Modified
| File | Action | Description | Status |
|------|--------|-------------|--------|
| `backend/services/youtube_upload_service.py` | Create | YouTube upload logic (~446 lines) | ✅ |
| `backend/services/discord_service.py` | Create | Discord notification service (~167 lines) | ✅ |
| `backend/services/packaging_service.py` | Create | CDG/TXT package generation (~266 lines) | ✅ |
| `backend/services/local_encoding_service.py` | Create | Local FFmpeg encoding service (~512 lines) | ✅ |
| `backend/services/encoding_interface.py` | Create | Abstract interface + backends (~418 lines) | ✅ |
| `backend/workers/video_worker_orchestrator.py` | Create | Stage orchestration logic (~550 lines) | ✅ |
| `backend/workers/video_worker.py` | Modify | Added orchestrator path + feature flag | ✅ |
| `backend/tests/test_youtube_upload_service.py` | Create | 23 unit tests | ✅ |
| `backend/tests/test_discord_service.py` | Create | 20 unit tests | ✅ |
| `backend/tests/test_packaging_service.py` | Create | 18 unit tests | ✅ |
| `backend/tests/test_local_encoding_service.py` | Create | 28 unit tests | ✅ |
| `backend/tests/test_encoding_interface.py` | Create | 23 unit tests | ✅ |
| `backend/tests/test_video_worker_orchestrator.py` | Create | 27 unit tests | ✅ |
| `karaoke_gen/karaoke_finalise/karaoke_finalise.py` | Modify | Use extracted services (optional) | Deferred |
## Testing Strategy
### Unit Tests
- **youtube_upload_service**: Mock YouTube API, test upload/delete/duplicate check
- **discord_service**: Mock requests, test webhook POST
- **packaging_service**: Use temp files, test CDG/TXT generation
- **local_encoding_service**: Mock subprocess, test FFmpeg command construction
### Integration Tests
- **video_worker_orchestrator**: Mock all services, test stage sequencing
- **video_worker**: Test with mocked orchestrator
### E2E Verification
- [ ] GCE encoding job with YouTube upload enabled → YouTube link appears
- [ ] Local encoding job with YouTube upload enabled → Same behavior
- [ ] CLI `karaoke-gen` with all features → No regression
- [ ] Job with CDG/TXT enabled → Packages generated
- [ ] Dropbox output structure → Matches local CLI exactly
## Open Questions
- [x] Should we keep KaraokeFinalise.process() or deprecate it?
- **Decision**: Keep it for CLI backwards compatibility. It will use the new services internally.
- [x] Should local_encoding_service run FFmpeg sync or async?
- **Decision**: Sync implementation, async wrapper via `asyncio.to_thread()` for interface compatibility.
- [x] Order of packaging vs encoding?
- **Decision**: CDG/TXT runs BEFORE encoding (depends on LRC file, not videos). KaraokeFinalise does it this way too.
## Rollback Plan
1. The refactor is additive - new services don't replace old code until orchestrator is wired up
2. Feature flag: `USE_NEW_ORCHESTRATOR=true` to enable new code path
3. If issues found in production:
- Set `USE_NEW_ORCHESTRATOR=false` to revert to old behavior
- Old `generate_video()` code preserved behind feature flag
4. After 1 week stable, remove feature flag and old code
## Success Criteria
1. Job `5b78ad46` scenario (GCE + YouTube) would upload to YouTube
2. All 803+ existing tests pass
3. E2E happy path test passes
4. No regression in local CLI functionality
5. Output structure matches local CLI exactly
6. Code coverage maintained or improved
## Estimated Scope
- **Phase 1** (Extract Services): 4 new service files + tests (~400-600 lines each)
- **Phase 2** (Interface): 1 new file, 2 modifications (~100 lines)
- **Phase 3** (Orchestrator): 2 files (~300-400 lines)
- **Phase 4** (KaraokeFinalise): 1 modification (~100 lines of changes)
- **Phase 5** (Testing): Integration tests + manual verification
This is a medium-sized refactor that can be done incrementally. Each phase produces working, tested code.
+2 -1
View File
@@ -225,7 +225,8 @@ jobs:
- name: Run Happy Path Production Test
working-directory: frontend
env:
MAILSLURP_API_KEY: ${{ secrets.MAILSLURP_API_KEY }}
TESTMAIL_API_KEY: ${{ secrets.TESTMAIL_API_KEY }}
TESTMAIL_NAMESPACE: ${{ secrets.TESTMAIL_NAMESPACE }}
E2E_TEST_TOKEN: ${{ secrets.E2E_TEST_TOKEN }}
E2E_ADMIN_TOKEN: ${{ secrets.E2E_ADMIN_TOKEN }}
run: |
+3 -2
View File
@@ -70,8 +70,9 @@ jobs:
- name: Run E2E Happy Path Test
working-directory: frontend
env:
MAILSLURP_API_KEY: ${{ secrets.MAILSLURP_API_KEY }}
E2E_TEST_TOKEN: ${{ inputs.use_test_token == 'true' && secrets.E2E_TEST_TOKEN || '' }}
TESTMAIL_API_KEY: ${{ secrets.TESTMAIL_API_KEY }}
TESTMAIL_NAMESPACE: ${{ secrets.TESTMAIL_NAMESPACE }}
E2E_TEST_TOKEN: ${{ inputs.use_test_token && secrets.E2E_TEST_TOKEN || '' }}
E2E_ADMIN_TOKEN: ${{ secrets.E2E_ADMIN_TOKEN }}
DEBUG_MODE: ${{ inputs.debug_mode || 'false' }}
run: |
+17
View File
@@ -18,6 +18,23 @@ RUN apt-get update && apt-get install -y \
xz-utils \
&& rm -rf /var/lib/apt/lists/*
# Install comprehensive font support for international lyrics and symbols
# Noto fonts provide full Unicode coverage including:
# - Latin, Greek, Cyrillic (core)
# - CJK: Chinese, Japanese, Korean (~150MB)
# - Arabic, Hebrew, Thai, Hindi, and other scripts (extra)
# - Color emoji (limited support in libass, but available)
# - Musical symbols (♪ ♫ etc.)
# Note: This adds ~200-300MB to the image but ensures all languages render correctly
RUN apt-get update && apt-get install -y \
fonts-noto-core \
fonts-noto-cjk \
fonts-noto-extra \
fonts-noto-color-emoji \
fontconfig \
&& fc-cache -f \
&& rm -rf /var/lib/apt/lists/*
# Install optimized static FFmpeg build (7.0.2) instead of Debian package (5.1.8)
# John Van Sickle's builds are compiled with better optimizations for x86_64
# This provides ~20-40% performance improvement for encoding tasks
+385 -2
View File
@@ -5,18 +5,20 @@ Handles:
- Dashboard overview statistics
- System-wide metrics
- Admin-only operations
- Audio search cache management
"""
import logging
from datetime import datetime, timedelta
from typing import Tuple
from typing import Tuple, List, Optional, Any, Dict
from fastapi import APIRouter, Depends
from fastapi import APIRouter, Depends, HTTPException
from pydantic import BaseModel
from backend.api.dependencies import require_admin
from backend.services.auth_service import UserType
from backend.services.user_service import get_user_service, UserService, USERS_COLLECTION
from backend.services.job_manager import JobManager
from backend.services.flacfetch_client import get_flacfetch_client, FlacfetchServiceError
from backend.models.job import JobStatus
@@ -197,3 +199,384 @@ async def get_admin_stats_overview(
total_credits_issued_30d=total_credits_issued_30d,
total_beta_testers=total_beta_testers,
)
# =============================================================================
# Audio Search Management Models
# =============================================================================
class AudioSearchResultSummary(BaseModel):
"""Summary of a single audio search result."""
index: int
provider: str
artist: str
title: str
is_lossless: bool
quality: Optional[str] = None
seeders: Optional[int] = None
class AudioSearchJobSummary(BaseModel):
"""Summary of a job with audio search results."""
job_id: str
status: str
user_email: Optional[str] = None
audio_search_artist: Optional[str] = None
audio_search_title: Optional[str] = None
created_at: Optional[datetime] = None
results_count: int
results_summary: List[AudioSearchResultSummary]
has_lossless: bool
providers: List[str]
class AudioSearchListResponse(BaseModel):
"""Response for listing audio search jobs."""
jobs: List[AudioSearchJobSummary]
total: int
class ClearSearchCacheResponse(BaseModel):
"""Response for clearing search cache."""
status: str
job_id: str
message: str
previous_status: str
new_status: str
results_cleared: int
flacfetch_cache_cleared: bool = False
flacfetch_error: Optional[str] = None
class ClearAllCacheResponse(BaseModel):
"""Response for clearing all flacfetch cache."""
status: str
message: str
deleted_count: int
class CacheStatsResponse(BaseModel):
"""Response for cache statistics."""
count: int
total_size_bytes: int
oldest_entry: Optional[str] = None
newest_entry: Optional[str] = None
configured: bool
# =============================================================================
# Audio Search Management Endpoints
# =============================================================================
@router.get("/audio-searches", response_model=AudioSearchListResponse)
async def list_audio_searches(
limit: int = 50,
status_filter: Optional[str] = None,
auth_data: Tuple[str, UserType, int] = Depends(require_admin),
user_service: UserService = Depends(get_user_service),
):
"""
List jobs with audio search results.
Returns jobs that have cached audio search results, useful for:
- Monitoring search activity
- Identifying stale cached results
- Clearing cache for specific jobs
Args:
limit: Maximum number of jobs to return (default 50)
status_filter: Optional filter by job status (e.g., 'awaiting_audio_selection')
"""
from google.cloud.firestore_v1 import FieldFilter
db = user_service.db
jobs_collection = db.collection("jobs")
# Query jobs - we'll filter for those with audio_search_results in Python
# since Firestore can't query for existence of nested fields efficiently
query = jobs_collection.order_by("created_at", direction="DESCENDING").limit(500)
if status_filter:
query = jobs_collection.where(
filter=FieldFilter("status", "==", status_filter)
).order_by("created_at", direction="DESCENDING").limit(500)
jobs_with_searches = []
for doc in query.stream():
data = doc.to_dict()
state_data = data.get("state_data", {})
audio_results = state_data.get("audio_search_results", [])
if not audio_results:
continue
# Compute has_lossless and providers from ALL results (not just first 10)
has_lossless = any(r.get("is_lossless", False) for r in audio_results)
providers = {r.get("provider", "Unknown") for r in audio_results}
# Build summary from first 10 results only
results_summary = []
for r in audio_results[:10]:
results_summary.append(AudioSearchResultSummary(
index=r.get("index", 0),
provider=r.get("provider", "Unknown"),
artist=r.get("artist", ""),
title=r.get("title", ""),
is_lossless=r.get("is_lossless", False),
quality=r.get("quality"),
seeders=r.get("seeders"),
))
jobs_with_searches.append(AudioSearchJobSummary(
job_id=doc.id,
status=data.get("status", "unknown"),
user_email=data.get("user_email"),
audio_search_artist=data.get("audio_search_artist"),
audio_search_title=data.get("audio_search_title"),
created_at=data.get("created_at"),
results_count=len(audio_results),
results_summary=results_summary,
has_lossless=has_lossless,
providers=sorted(providers),
))
if len(jobs_with_searches) >= limit:
break
return AudioSearchListResponse(
jobs=jobs_with_searches,
total=len(jobs_with_searches),
)
@router.post("/audio-searches/{job_id}/clear-cache", response_model=ClearSearchCacheResponse)
async def clear_audio_search_cache(
job_id: str,
auth_data: Tuple[str, UserType, int] = Depends(require_admin),
user_service: UserService = Depends(get_user_service),
):
"""
Clear the cached audio search results for a job.
This will:
1. Remove the cached search results from job.state_data
2. Reset the job status to 'pending' so a new search can be performed
3. Clear the flacfetch GCS cache for this artist/title (if available)
Use this when:
- Cached results are stale (e.g., flacfetch was updated)
- User wants to search again with different terms
- Results appear incomplete or incorrect
"""
job_manager = JobManager()
job = job_manager.get_job(job_id)
if not job:
raise HTTPException(status_code=404, detail=f"Job {job_id} not found")
# Get current state
state_data = job.state_data or {}
audio_results = state_data.get("audio_search_results", [])
results_count = len(audio_results)
previous_status = job.status
if not audio_results:
raise HTTPException(
status_code=400,
detail=f"Job {job_id} has no cached audio search results"
)
# Validate job status - only allow cache clear for appropriate states
# Don't allow clearing cache for jobs that are actively processing or complete
forbidden_statuses = {
"downloading", "downloading_audio", "searching_audio",
"separating_stage1", "separating_stage2", "transcribing", "correcting",
"generating_screens", "applying_padding", "rendering_video",
"generating_video", "encoding", "packaging", "uploading",
"complete", "prep_complete",
}
if previous_status in forbidden_statuses:
raise HTTPException(
status_code=400,
detail=f"Cannot clear cache for job in '{previous_status}' state. "
f"Only jobs in pending, awaiting_audio_selection, failed, or cancelled states can have cache cleared."
)
# Clear the cache by removing the keys
db = user_service.db
job_ref = db.collection("jobs").document(job_id)
from google.cloud.firestore_v1 import DELETE_FIELD
# Update job: clear cache and reset status
job_ref.update({
"state_data.audio_search_results": DELETE_FIELD,
"state_data.audio_search_count": DELETE_FIELD,
"state_data.remote_search_id": DELETE_FIELD,
"status": "pending",
"progress": 0,
"message": "Audio search cache cleared by admin. Ready for new search.",
"updated_at": datetime.utcnow(),
})
# Also clear flacfetch's GCS cache if we have artist/title
flacfetch_cache_cleared = False
flacfetch_error = None
artist = job.audio_search_artist
title = job.audio_search_title
if artist and title:
flacfetch_client = get_flacfetch_client()
if flacfetch_client:
try:
flacfetch_cache_cleared = await flacfetch_client.clear_search_cache(artist, title)
logger.info(
f"Cleared flacfetch cache for '{artist}' - '{title}': "
f"{'deleted' if flacfetch_cache_cleared else 'no entry found'}"
)
except FlacfetchServiceError as e:
flacfetch_error = str(e)
logger.warning(f"Failed to clear flacfetch cache: {e}")
else:
flacfetch_error = "flacfetch client not configured"
logger.debug("Skipping flacfetch cache clear - client not configured")
else:
flacfetch_error = "missing artist or title"
logger.debug(f"Skipping flacfetch cache clear - missing artist ({artist}) or title ({title})")
logger.info(
f"Admin {auth_data[0]} cleared audio search cache for job {job_id}. "
f"Cleared {results_count} results. Status changed from {previous_status} to pending. "
f"Flacfetch cache cleared: {flacfetch_cache_cleared}"
)
message = f"Cleared {results_count} cached search results. Job reset to pending."
if flacfetch_cache_cleared:
message += " Flacfetch cache also cleared."
elif flacfetch_error:
message += f" Note: flacfetch cache not cleared ({flacfetch_error})."
return ClearSearchCacheResponse(
status="success",
job_id=job_id,
message=message,
previous_status=previous_status,
new_status="pending",
results_cleared=results_count,
flacfetch_cache_cleared=flacfetch_cache_cleared,
flacfetch_error=flacfetch_error,
)
@router.post("/jobs/{job_id}/reset-worker-state")
async def reset_worker_state(
job_id: str,
auth_data: Tuple[str, UserType, int] = Depends(require_admin),
):
"""
Reset stale worker progress state for a job.
This clears the video_progress, render_progress, and screens_progress
from state_data, allowing workers to be re-triggered.
Use this when a job is stuck because worker progress shows 'running'
from a previous failed attempt.
"""
from backend.services.job_manager import JobManager
job_manager = JobManager()
job = job_manager.get_job(job_id)
if not job:
raise HTTPException(status_code=404, detail=f"Job {job_id} not found")
# Reset worker progress states
job_manager.update_state_data(job_id, 'video_progress', {'stage': 'pending'})
job_manager.update_state_data(job_id, 'render_progress', {'stage': 'pending'})
job_manager.update_state_data(job_id, 'screens_progress', {'stage': 'pending'})
logger.info(f"Admin {auth_data[0]} reset worker state for job {job_id}")
return {
"status": "success",
"job_id": job_id,
"message": "Worker progress states reset to pending"
}
@router.delete("/cache", response_model=ClearAllCacheResponse)
async def clear_all_flacfetch_cache(
auth_data: Tuple[str, UserType, int] = Depends(require_admin),
):
"""
Clear the entire flacfetch search cache.
This will delete all cached search results from flacfetch's GCS cache.
Use with caution - this will cause all subsequent searches to hit
the trackers fresh.
Note: This does NOT clear Firestore job.state_data caches, only the
flacfetch-side GCS cache.
"""
flacfetch_client = get_flacfetch_client()
if not flacfetch_client:
raise HTTPException(
status_code=503,
detail="flacfetch client not configured"
)
try:
deleted_count = await flacfetch_client.clear_all_cache()
logger.info(
f"Admin {auth_data[0]} cleared all flacfetch cache. "
f"Deleted {deleted_count} entries."
)
return ClearAllCacheResponse(
status="success",
message=f"Cleared {deleted_count} cache entries from flacfetch.",
deleted_count=deleted_count,
)
except FlacfetchServiceError as e:
logger.error(f"Failed to clear all flacfetch cache: {e}")
raise HTTPException(
status_code=502,
detail=f"Failed to clear flacfetch cache: {e}"
)
@router.get("/cache/stats", response_model=CacheStatsResponse)
async def get_flacfetch_cache_stats(
auth_data: Tuple[str, UserType, int] = Depends(require_admin),
):
"""
Get statistics about the flacfetch search cache.
Returns information about:
- Number of cached entries
- Total size in bytes
- Oldest and newest cache entries
- Whether cache is configured
"""
flacfetch_client = get_flacfetch_client()
if not flacfetch_client:
raise HTTPException(
status_code=503,
detail="flacfetch client not configured"
)
try:
stats = await flacfetch_client.get_cache_stats()
return CacheStatsResponse(
count=stats.get("count", 0),
total_size_bytes=stats.get("total_size_bytes", 0),
oldest_entry=stats.get("oldest_entry"),
newest_entry=stats.get("newest_entry"),
configured=stats.get("configured", True),
)
except FlacfetchServiceError as e:
logger.error(f"Failed to get flacfetch cache stats: {e}")
raise HTTPException(
status_code=502,
detail=f"Failed to get cache stats: {e}"
)
# Trigger deployment
+11 -8
View File
@@ -1291,12 +1291,13 @@ async def retry_job(
logger.info(f"Job {job_id}: Has rendered video and instrumental selection, retrying video generation")
# Clear error state
# Clear error state and reset worker progress for idempotency
job_manager.update_job(job_id, {
'error_message': None,
'error_details': None,
})
job_manager.update_state_data(job_id, 'video_progress', {'stage': 'pending'})
# Reset to INSTRUMENTAL_SELECTED and trigger video worker
if not job_manager.transition_to_state(
job_id=job_id,
@@ -1324,13 +1325,14 @@ async def retry_job(
file_urls.get('screens', {}).get('title')):
logger.info(f"Job {job_id}: Has corrections and screens, retrying from render stage")
# Clear error state
# Clear error state and reset worker progress for idempotency
job_manager.update_job(job_id, {
'error_message': None,
'error_details': None,
})
job_manager.update_state_data(job_id, 'render_progress', {'stage': 'pending'})
# Reset to REVIEW_COMPLETE and trigger render worker
if not job_manager.transition_to_state(
job_id=job_id,
@@ -1358,13 +1360,14 @@ async def retry_job(
file_urls.get('lyrics', {}).get('corrections')):
logger.info(f"Job {job_id}: Has stems and corrections, retrying from screens stage")
# Clear error state
# Clear error state and reset worker progress for idempotency
job_manager.update_job(job_id, {
'error_message': None,
'error_details': None,
})
job_manager.update_state_data(job_id, 'screens_progress', {'stage': 'pending'})
# Reset to a state before screens and trigger screens worker
if not job_manager.transition_to_state(
job_id=job_id,
+172
View File
@@ -0,0 +1,172 @@
"""
Discord Notification Service.
Provides Discord webhook notification functionality, extracted from KaraokeFinalise
for use by both the cloud backend (video_worker) and local CLI.
This service handles:
- Posting messages to Discord webhooks
- Video upload notifications
- Validation of webhook URLs
"""
import logging
import re
from typing import Optional
import requests
logger = logging.getLogger(__name__)
class DiscordNotificationService:
"""
Service for sending Discord webhook notifications.
Supports posting messages to Discord channels via webhooks,
commonly used to notify when new karaoke videos are uploaded.
"""
# Discord webhook URL pattern
WEBHOOK_URL_PATTERN = re.compile(
r"^https://discord\.com/api/webhooks/\d+/[a-zA-Z0-9_-]+$"
)
def __init__(
self,
webhook_url: Optional[str] = None,
dry_run: bool = False,
logger: Optional[logging.Logger] = None,
):
"""
Initialize the Discord notification service.
Args:
webhook_url: Discord webhook URL for posting notifications
dry_run: If True, log actions without performing them
logger: Optional logger instance
"""
self.webhook_url = webhook_url
self.dry_run = dry_run
self.logger = logger or logging.getLogger(__name__)
# Validate webhook URL if provided
if webhook_url:
self._validate_webhook_url(webhook_url)
def _validate_webhook_url(self, url: str) -> None:
"""
Validate that the webhook URL is a valid Discord webhook URL.
Args:
url: URL to validate
Raises:
ValueError: If URL is not a valid Discord webhook URL
"""
url = url.strip()
if not url.startswith("https://discord.com/api/webhooks/"):
raise ValueError(
f"Invalid Discord webhook URL: {url}. "
"URL must start with 'https://discord.com/api/webhooks/'"
)
def post_message(
self,
message: str,
webhook_url: Optional[str] = None,
) -> bool:
"""
Post a message to a Discord channel via webhook.
Args:
message: The message content to post
webhook_url: Optional webhook URL (uses instance URL if not provided)
Returns:
True if message was posted successfully, False otherwise
Raises:
ValueError: If no webhook URL is available
requests.HTTPError: If the webhook request fails
"""
url = webhook_url or self.webhook_url
if not url:
raise ValueError("No Discord webhook URL provided")
url = url.strip()
self._validate_webhook_url(url)
if self.dry_run:
self.logger.info(
f"DRY RUN: Would post Discord message: '{message}' to {url}"
)
return True
self.logger.info(f"Posting message to Discord webhook...")
data = {"content": message}
response = requests.post(url, json=data, timeout=30)
response.raise_for_status()
self.logger.info("Message posted to Discord successfully")
return True
def post_video_notification(
self,
youtube_url: str,
webhook_url: Optional[str] = None,
) -> bool:
"""
Post a notification about a new video upload.
Args:
youtube_url: The YouTube URL of the uploaded video
webhook_url: Optional webhook URL (uses instance URL if not provided)
Returns:
True if notification was posted successfully, False otherwise
"""
if not youtube_url:
self.logger.info("Skipping Discord notification - no YouTube URL available")
return False
message = f"New upload: {youtube_url}"
return self.post_message(message, webhook_url)
def is_enabled(self) -> bool:
"""
Check if Discord notifications are enabled.
Returns:
True if a valid webhook URL is configured
"""
return bool(self.webhook_url)
# Singleton instance and factory function (following existing service pattern)
_discord_notification_service: Optional[DiscordNotificationService] = None
def get_discord_notification_service(
webhook_url: Optional[str] = None,
**kwargs
) -> DiscordNotificationService:
"""
Get a Discord notification service instance.
Args:
webhook_url: Discord webhook URL for notifications
**kwargs: Additional arguments passed to DiscordNotificationService
Returns:
DiscordNotificationService instance
"""
global _discord_notification_service
# Create new instance if webhook URL changed or not yet created
if _discord_notification_service is None or webhook_url:
_discord_notification_service = DiscordNotificationService(
webhook_url=webhook_url,
**kwargs
)
return _discord_notification_service
+429
View File
@@ -0,0 +1,429 @@
"""
Encoding Interface.
Defines the abstract interface for video encoding backends, allowing
the video worker orchestrator to use either GCE or local encoding
interchangeably.
This follows the Strategy pattern - different encoding implementations
can be swapped without changing the orchestration logic.
"""
import logging
from abc import ABC, abstractmethod
from dataclasses import dataclass, field
from typing import Dict, List, Optional, Any
logger = logging.getLogger(__name__)
@dataclass
class EncodingInput:
"""
Input configuration for video encoding.
Contains all the paths and metadata needed to encode a karaoke video.
"""
# Required input files
title_video_path: str # Title card video (MOV)
karaoke_video_path: str # Main karaoke video with vocals (MOV/MKV)
instrumental_audio_path: str # Instrumental audio track (FLAC)
# Optional input files
end_video_path: Optional[str] = None # End credits video (MOV)
# Metadata for output naming
artist: str = ""
title: str = ""
brand_code: Optional[str] = None
# Output directory
output_dir: str = ""
# Additional options
options: Dict[str, Any] = field(default_factory=dict)
@dataclass
class EncodingOutput:
"""
Output from video encoding.
Contains paths to all generated video files and status information.
"""
success: bool
error_message: Optional[str] = None
# Output file paths (relative to output_dir or absolute)
karaoke_mp4_path: Optional[str] = None # Karaoke video with instrumental audio
with_vocals_mp4_path: Optional[str] = None # With vocals MP4 version
lossless_4k_mp4_path: Optional[str] = None # Final lossless 4K MP4
lossy_4k_mp4_path: Optional[str] = None # Final lossy 4K MP4
lossless_mkv_path: Optional[str] = None # MKV with FLAC (for YouTube)
lossy_720p_mp4_path: Optional[str] = None # 720p web version
# All output files as a dict for convenience
output_files: Dict[str, str] = field(default_factory=dict)
# Encoding metadata
encoding_time_seconds: Optional[float] = None
encoding_backend: Optional[str] = None # "gce" or "local"
class EncodingBackend(ABC):
"""
Abstract base class for video encoding backends.
Implementations:
- GCEEncodingBackend: Cloud-based encoding using Google Compute Engine
- LocalEncodingBackend: Local FFmpeg-based encoding
"""
@property
@abstractmethod
def name(self) -> str:
"""Return the name of this encoding backend."""
pass
@abstractmethod
async def encode(self, input_config: EncodingInput) -> EncodingOutput:
"""
Encode video files according to the input configuration.
This method should:
1. Create karaoke video with instrumental audio
2. Convert with-vocals video to MP4 if needed
3. Encode lossless 4K MP4 (concatenating title + karaoke + end)
4. Encode lossy 4K MP4 with AAC audio
5. Create MKV with FLAC audio for YouTube
6. Encode 720p version for web
Args:
input_config: EncodingInput with all required paths and options
Returns:
EncodingOutput with paths to generated files and status
"""
pass
@abstractmethod
async def is_available(self) -> bool:
"""
Check if this encoding backend is available and configured.
Returns:
True if the backend can be used, False otherwise
"""
pass
@abstractmethod
async def get_status(self) -> Dict[str, Any]:
"""
Get the current status of the encoding backend.
Returns:
Dict with status information (availability, queue length, etc.)
"""
pass
class LocalEncodingBackend(EncodingBackend):
"""
Local FFmpeg-based encoding backend.
Wraps the LocalEncodingService to implement the EncodingBackend interface.
Uses asyncio.to_thread() to run synchronous FFmpeg operations asynchronously.
"""
def __init__(
self,
dry_run: bool = False,
logger: Optional[logging.Logger] = None,
):
"""
Initialize the local encoding backend.
Args:
dry_run: If True, log operations without executing
logger: Optional logger instance
"""
self.dry_run = dry_run
self.logger = logger or logging.getLogger(__name__)
self._service = None
@property
def name(self) -> str:
return "local"
def _get_service(self):
"""Lazy-load the local encoding service."""
if self._service is None:
from backend.services.local_encoding_service import LocalEncodingService
self._service = LocalEncodingService(
dry_run=self.dry_run,
logger=self.logger
)
return self._service
async def encode(self, input_config: EncodingInput) -> EncodingOutput:
"""
Encode video using local FFmpeg.
Args:
input_config: Encoding input configuration
Returns:
EncodingOutput with results
"""
import asyncio
import time
from backend.services.local_encoding_service import EncodingConfig
start_time = time.time()
# Build output file paths
base_name = f"{input_config.artist} - {input_config.title}"
output_dir = input_config.output_dir or "."
import os
config = EncodingConfig(
title_video=input_config.title_video_path,
karaoke_video=input_config.karaoke_video_path,
instrumental_audio=input_config.instrumental_audio_path,
end_video=input_config.end_video_path,
output_karaoke_mp4=os.path.join(output_dir, f"{base_name} (Karaoke).mp4"),
output_with_vocals_mp4=os.path.join(output_dir, f"{base_name} (With Vocals).mp4"),
output_lossless_4k_mp4=os.path.join(output_dir, f"{base_name} (Final Karaoke Lossless 4k).mp4"),
output_lossy_4k_mp4=os.path.join(output_dir, f"{base_name} (Final Karaoke Lossy 4k).mp4"),
output_lossless_mkv=os.path.join(output_dir, f"{base_name} (Final Karaoke Lossless 4k).mkv"),
output_720p_mp4=os.path.join(output_dir, f"{base_name} (Final Karaoke Lossy 720p).mp4"),
)
# Run encoding in thread pool to avoid blocking
service = self._get_service()
result = await asyncio.to_thread(service.encode_all_formats, config)
encoding_time = time.time() - start_time
if result.success:
return EncodingOutput(
success=True,
karaoke_mp4_path=config.output_karaoke_mp4,
with_vocals_mp4_path=config.output_with_vocals_mp4,
lossless_4k_mp4_path=config.output_lossless_4k_mp4,
lossy_4k_mp4_path=config.output_lossy_4k_mp4,
lossless_mkv_path=config.output_lossless_mkv,
lossy_720p_mp4_path=config.output_720p_mp4,
output_files=result.output_files,
encoding_time_seconds=encoding_time,
encoding_backend=self.name
)
else:
return EncodingOutput(
success=False,
error_message=result.error,
output_files=result.output_files,
encoding_time_seconds=encoding_time,
encoding_backend=self.name
)
async def is_available(self) -> bool:
"""Check if local encoding is available (FFmpeg installed)."""
import subprocess
import asyncio
try:
result = await asyncio.to_thread(
subprocess.run,
["ffmpeg", "-version"],
capture_output=True,
timeout=10
)
return result.returncode == 0
except (FileNotFoundError, subprocess.TimeoutExpired):
return False
async def get_status(self) -> Dict[str, Any]:
"""Get local encoding status."""
available = await self.is_available()
service = self._get_service()
return {
"backend": self.name,
"available": available,
"hwaccel_available": service.hwaccel_available if available else False,
"video_encoder": service.video_encoder if available else None,
}
class GCEEncodingBackend(EncodingBackend):
"""
GCE-based encoding backend.
Wraps the existing EncodingService to implement the EncodingBackend interface.
Submits jobs to a remote GCE worker for high-performance encoding.
"""
def __init__(
self,
dry_run: bool = False,
logger: Optional[logging.Logger] = None,
):
"""
Initialize the GCE encoding backend.
Args:
dry_run: Ignored for GCE backend (remote service doesn't support dry run)
logger: Optional logger instance
"""
self.dry_run = dry_run # Stored but not used (GCE is remote)
self.logger = logger or logging.getLogger(__name__)
self._service = None
@property
def name(self) -> str:
return "gce"
def _get_service(self):
"""Lazy-load the GCE encoding service."""
if self._service is None:
from backend.services.encoding_service import get_encoding_service
self._service = get_encoding_service()
return self._service
async def encode(self, input_config: EncodingInput) -> EncodingOutput:
"""
Encode video using GCE worker.
Note: GCE encoding requires files to be in GCS, so this method
expects input_config.options to contain 'input_gcs_path' and
'output_gcs_path'.
Args:
input_config: Encoding input configuration
Returns:
EncodingOutput with results
"""
import time
start_time = time.time()
service = self._get_service()
# GCE requires GCS paths
input_gcs_path = input_config.options.get("input_gcs_path")
output_gcs_path = input_config.options.get("output_gcs_path")
job_id = input_config.options.get("job_id", input_config.brand_code or "unknown")
if not input_gcs_path or not output_gcs_path:
return EncodingOutput(
success=False,
error_message="GCE encoding requires input_gcs_path and output_gcs_path in options",
encoding_backend=self.name
)
try:
# Build encoding config
encoding_config = {
"formats": ["mp4_4k_lossless", "mp4_4k_lossy", "mkv_4k", "mp4_720p"],
"artist": input_config.artist,
"title": input_config.title,
}
# Submit and wait for completion
result = await service.encode_videos(
job_id=job_id,
input_gcs_path=input_gcs_path,
output_gcs_path=output_gcs_path,
encoding_config=encoding_config,
)
encoding_time = time.time() - start_time
# Extract output file paths from result
# Handle case where GCE worker returns a list or unexpected format
if isinstance(result, list):
# If result is a list, try to find the output_files in the first dict
self.logger.warning(f"GCE returned list instead of dict: {result}")
result = result[0] if result and isinstance(result[0], dict) else {}
if not isinstance(result, dict):
self.logger.error(f"Unexpected GCE result type: {type(result)}")
result = {}
output_files = result.get("output_files", {})
return EncodingOutput(
success=True,
lossless_4k_mp4_path=output_files.get("mp4_4k_lossless"),
lossy_4k_mp4_path=output_files.get("mp4_4k_lossy"),
lossless_mkv_path=output_files.get("mkv_4k"),
lossy_720p_mp4_path=output_files.get("mp4_720p"),
output_files=output_files,
encoding_time_seconds=encoding_time,
encoding_backend=self.name
)
except Exception as e:
encoding_time = time.time() - start_time
self.logger.error(f"GCE encoding failed: {e}")
return EncodingOutput(
success=False,
error_message=str(e),
encoding_time_seconds=encoding_time,
encoding_backend=self.name
)
async def is_available(self) -> bool:
"""Check if GCE encoding is available and configured."""
service = self._get_service()
return service.is_enabled
async def get_status(self) -> Dict[str, Any]:
"""Get GCE encoding status."""
service = self._get_service()
status = {
"backend": self.name,
"available": service.is_enabled,
"configured": service.is_configured,
}
if service.is_configured:
try:
health = await service.health_check()
status["health"] = health
except Exception as e:
status["health_error"] = str(e)
return status
# Factory function to get an encoding backend
def get_encoding_backend(
backend_type: str = "auto",
**kwargs
) -> EncodingBackend:
"""
Get an encoding backend instance.
Args:
backend_type: Type of backend - "local", "gce", or "auto"
**kwargs: Additional arguments for the backend
Returns:
EncodingBackend instance
Raises:
ValueError: If backend_type is unknown
"""
if backend_type == "local":
return LocalEncodingBackend(**kwargs)
elif backend_type == "gce":
return GCEEncodingBackend(**kwargs)
elif backend_type == "auto":
# Check if GCE is available, otherwise use local
gce_backend = GCEEncodingBackend(**kwargs)
if gce_backend._get_service().is_enabled:
return gce_backend
return LocalEncodingBackend(**kwargs)
else:
raise ValueError(f"Unknown encoding backend type: {backend_type}")
+103 -3
View File
@@ -408,11 +408,11 @@ class FlacfetchClient:
) -> Dict[str, Any]:
"""
Trigger disk cleanup.
Args:
strategy: Cleanup strategy (oldest, largest, lowest_ratio)
target_free_gb: Target free space to achieve
Returns:
Cleanup response with removed count and freed space
"""
@@ -429,12 +429,112 @@ class FlacfetchClient:
)
resp.raise_for_status()
return resp.json()
except httpx.RequestError as e:
raise FlacfetchServiceError(f"Cleanup failed: {e}")
except httpx.HTTPStatusError as e:
raise FlacfetchServiceError(f"Cleanup failed: {e.response.status_code}")
# =========================================================================
# Cache Management
# =========================================================================
async def clear_search_cache(self, artist: str, title: str) -> bool:
"""
Clear a specific cached search result by artist and title.
Args:
artist: Artist name
title: Track title
Returns:
True if a cache entry was deleted, False if no entry existed
Raises:
FlacfetchServiceError: On request failure
"""
try:
async with httpx.AsyncClient() as client:
resp = await client.request(
"DELETE",
f"{self.base_url}/cache/search",
headers=self._headers(),
json={"artist": artist, "title": title},
timeout=30,
)
resp.raise_for_status()
data = resp.json()
deleted = data.get("deleted", False)
logger.info(
f"Cleared flacfetch cache for '{artist}' - '{title}': "
f"{'deleted' if deleted else 'no entry found'}"
)
return deleted
except httpx.RequestError as e:
raise FlacfetchServiceError(f"Clear search cache request failed: {e}")
except httpx.HTTPStatusError as e:
raise FlacfetchServiceError(
f"Clear search cache failed: {e.response.status_code} - {e.response.text}"
)
async def clear_all_cache(self) -> int:
"""
Clear all cached search results.
Returns:
Number of cache entries deleted
Raises:
FlacfetchServiceError: On request failure
"""
try:
async with httpx.AsyncClient() as client:
resp = await client.delete(
f"{self.base_url}/cache",
headers=self._headers(),
timeout=60,
)
resp.raise_for_status()
data = resp.json()
deleted_count = data.get("deleted_count", 0)
logger.info(f"Cleared all flacfetch cache: {deleted_count} entries deleted")
return deleted_count
except httpx.RequestError as e:
raise FlacfetchServiceError(f"Clear all cache request failed: {e}")
except httpx.HTTPStatusError as e:
raise FlacfetchServiceError(
f"Clear all cache failed: {e.response.status_code} - {e.response.text}"
)
async def get_cache_stats(self) -> Dict[str, Any]:
"""
Get statistics about the cache.
Returns:
Dict with count, total_size_bytes, oldest_entry, newest_entry, configured
Raises:
FlacfetchServiceError: On request failure
"""
try:
async with httpx.AsyncClient() as client:
resp = await client.get(
f"{self.base_url}/cache/stats",
headers=self._headers(),
timeout=30,
)
resp.raise_for_status()
return resp.json()
except httpx.RequestError as e:
raise FlacfetchServiceError(f"Get cache stats request failed: {e}")
except httpx.HTTPStatusError as e:
raise FlacfetchServiceError(
f"Get cache stats failed: {e.response.status_code} - {e.response.text}"
)
# Singleton client instance
_client: Optional[FlacfetchClient] = None
+590
View File
@@ -0,0 +1,590 @@
"""
Local Encoding Service.
Provides local FFmpeg-based video encoding functionality, extracted from KaraokeFinalise
for use by both the cloud backend (video_worker as a fallback) and local CLI.
This service handles:
- Concatenating title, karaoke, and end videos
- Encoding to multiple output formats (4K lossless, 4K lossy, 720p, MKV)
- Hardware acceleration detection and fallback to software encoding
- Remuxing video with instrumental audio
"""
import logging
import os
import shlex
import subprocess
import sys
from dataclasses import dataclass
from typing import Optional, List, Dict, Any, Tuple
logger = logging.getLogger(__name__)
@dataclass
class EncodingConfig:
"""Configuration for video encoding."""
title_video: str # Path to title video
karaoke_video: str # Path to karaoke video (with vocals)
instrumental_audio: str # Path to instrumental audio
end_video: Optional[str] = None # Optional path to end credits video
# Output paths
output_karaoke_mp4: Optional[str] = None
output_with_vocals_mp4: Optional[str] = None
output_lossless_4k_mp4: Optional[str] = None
output_lossy_4k_mp4: Optional[str] = None
output_lossless_mkv: Optional[str] = None
output_720p_mp4: Optional[str] = None
@dataclass
class EncodingResult:
"""Result of video encoding operation."""
success: bool
output_files: Dict[str, str]
error: Optional[str] = None
class LocalEncodingService:
"""
Service for local FFmpeg-based video encoding.
Supports hardware acceleration (NVENC) with automatic fallback
to software encoding (libx264) when unavailable.
"""
# MP4 flags for better compatibility
MP4_FLAGS = "-movflags +faststart"
def __init__(
self,
dry_run: bool = False,
log_level: int = logging.INFO,
logger: Optional[logging.Logger] = None,
):
"""
Initialize the local encoding service.
Args:
dry_run: If True, log commands without executing them
log_level: Logging level (affects FFmpeg verbosity)
logger: Optional logger instance
"""
self.dry_run = dry_run
self.log_level = log_level
self.logger = logger or logging.getLogger(__name__)
# Hardware acceleration settings (detected on first use)
self._hwaccel_available: Optional[bool] = None
self._video_encoder: Optional[str] = None
self._scale_filter: Optional[str] = None
self._hwaccel_decode_flags: Optional[str] = None
self._aac_codec: Optional[str] = None
# Build FFmpeg base command
self._ffmpeg_base_command = self._build_ffmpeg_base_command()
def _build_ffmpeg_base_command(self) -> str:
"""Build the FFmpeg base command with appropriate flags."""
# Use bundled FFmpeg for frozen builds
ffmpeg_path = os.path.join(sys._MEIPASS, "ffmpeg.exe") if getattr(sys, "frozen", False) else "ffmpeg"
base_cmd = f"{ffmpeg_path} -hide_banner -nostats -y"
if self.log_level == logging.DEBUG:
base_cmd += " -loglevel verbose"
else:
base_cmd += " -loglevel fatal"
return base_cmd
def _detect_hardware_acceleration(self) -> Tuple[bool, str, str, str, str]:
"""
Detect available hardware acceleration.
Returns:
Tuple of (hwaccel_available, video_encoder, scale_filter,
hwaccel_decode_flags, aac_codec)
"""
self.logger.info("Detecting hardware acceleration capabilities...")
# Try NVENC (NVIDIA)
try:
test_cmd = f"{self._ffmpeg_base_command} -hide_banner -loglevel error " \
f"-f lavfi -i testsrc=duration=1:size=320x240:rate=1 " \
f"-c:v h264_nvenc -f null -"
subprocess.run(
test_cmd, shell=True, check=True,
capture_output=True, timeout=30
)
self.logger.info("NVIDIA NVENC hardware acceleration available")
return (
True,
"h264_nvenc",
"scale_cuda",
"-hwaccel cuda -hwaccel_output_format cuda",
"aac"
)
except (subprocess.CalledProcessError, subprocess.TimeoutExpired, FileNotFoundError):
pass
# No hardware acceleration available
self.logger.info("No hardware acceleration available, using software encoding")
return (False, "libx264", "scale", "", "aac")
@property
def hwaccel_available(self) -> bool:
"""Check if hardware acceleration is available."""
if self._hwaccel_available is None:
self._detect_and_set_hwaccel()
return self._hwaccel_available
@property
def video_encoder(self) -> str:
"""Get the video encoder to use."""
if self._video_encoder is None:
self._detect_and_set_hwaccel()
return self._video_encoder
@property
def scale_filter(self) -> str:
"""Get the scale filter to use."""
if self._scale_filter is None:
self._detect_and_set_hwaccel()
return self._scale_filter
@property
def hwaccel_decode_flags(self) -> str:
"""Get hardware acceleration decode flags."""
if self._hwaccel_decode_flags is None:
self._detect_and_set_hwaccel()
return self._hwaccel_decode_flags
@property
def aac_codec(self) -> str:
"""Get the AAC codec to use."""
if self._aac_codec is None:
self._detect_and_set_hwaccel()
return self._aac_codec
def _detect_and_set_hwaccel(self) -> None:
"""Detect and set hardware acceleration settings."""
(
self._hwaccel_available,
self._video_encoder,
self._scale_filter,
self._hwaccel_decode_flags,
self._aac_codec
) = self._detect_hardware_acceleration()
def _get_nvenc_quality_settings(self, preset: str = "medium") -> str:
"""Get NVENC quality settings for different presets."""
if not self.hwaccel_available:
return ""
settings = {
"lossless": "-preset p7 -tune hq -rc vbr -cq 0 -qmin 0 -qmax 0",
"high": "-preset p7 -tune hq -rc vbr -cq 19 -b:v 0",
"medium": "-preset p4 -tune hq -rc vbr -cq 23 -b:v 0",
"fast": "-preset p2 -tune ll -rc vbr -cq 28 -b:v 0",
}
return settings.get(preset, settings["medium"])
def _execute_command(
self,
command: str,
description: str,
timeout: int = 3600, # 1 hour default
) -> bool:
"""
Execute an FFmpeg command.
Args:
command: The FFmpeg command to execute
description: Human-readable description of the operation
timeout: Command timeout in seconds
Returns:
True if successful, False otherwise
"""
self.logger.info(f"Executing: {description}")
self.logger.debug(f"Command: {command}")
if self.dry_run:
self.logger.info(f"DRY RUN: Would execute: {command}")
return True
try:
result = subprocess.run(
command,
shell=True,
check=True,
capture_output=True,
text=True,
timeout=timeout
)
self.logger.info(f"Completed: {description}")
return True
except subprocess.CalledProcessError as e:
self.logger.error(f"Failed: {description}")
self.logger.error(f"Error: {e.stderr}")
return False
except subprocess.TimeoutExpired:
self.logger.error(f"Timeout: {description}")
return False
def _execute_command_with_fallback(
self,
gpu_command: str,
cpu_command: str,
description: str,
) -> bool:
"""
Execute a command with GPU, falling back to CPU if it fails.
Args:
gpu_command: Hardware-accelerated command
cpu_command: Software fallback command
description: Human-readable description
Returns:
True if successful, False otherwise
"""
if self.hwaccel_available:
self.logger.info(f"Trying hardware-accelerated encoding for: {description}")
if self._execute_command(gpu_command, description):
return True
self.logger.warning(f"Hardware encoding failed, falling back to software")
return self._execute_command(cpu_command, description)
def remux_with_instrumental(
self,
input_video: str,
instrumental_audio: str,
output_file: str,
) -> bool:
"""
Remux video with instrumental audio track.
Args:
input_video: Path to input video file
instrumental_audio: Path to instrumental audio file
output_file: Path for output file
Returns:
True if successful, False otherwise
"""
command = (
f'{self._ffmpeg_base_command} -i "{input_video}" '
f'-i "{instrumental_audio}" -map 0:v -map 1:a -c copy '
f'{self.MP4_FLAGS} "{output_file}"'
)
return self._execute_command(command, "Remuxing with instrumental audio")
def convert_mov_to_mp4(
self,
input_file: str,
output_file: str,
) -> bool:
"""
Convert MOV to MP4 format.
Args:
input_file: Path to input MOV file
output_file: Path for output MP4 file
Returns:
True if successful, False otherwise
"""
gpu_command = (
f'{self._ffmpeg_base_command} {self.hwaccel_decode_flags} -i "{input_file}" '
f'-c:v {self.video_encoder} -c:a copy {self.MP4_FLAGS} "{output_file}"'
)
cpu_command = (
f'{self._ffmpeg_base_command} -i "{input_file}" '
f'-c:v libx264 -c:a copy {self.MP4_FLAGS} "{output_file}"'
)
return self._execute_command_with_fallback(
gpu_command, cpu_command, "Converting MOV to MP4"
)
def encode_lossless_mp4(
self,
title_video: str,
karaoke_video: str,
output_file: str,
end_video: Optional[str] = None,
) -> bool:
"""
Create lossless 4K MP4 by concatenating title, karaoke, and optionally end videos.
Args:
title_video: Path to title video
karaoke_video: Path to karaoke video
output_file: Path for output file
end_video: Optional path to end credits video
Returns:
True if successful, False otherwise
"""
# Quote file paths
title_quoted = shlex.quote(os.path.abspath(title_video))
karaoke_quoted = shlex.quote(os.path.abspath(karaoke_video))
# Build filter and inputs for concatenation
if end_video and os.path.isfile(end_video):
end_quoted = shlex.quote(os.path.abspath(end_video))
extra_input = f"-i {end_quoted}"
concat_filter = (
'-filter_complex "[0:v:0][0:a:0][1:v:0][1:a:0][2:v:0][2:a:0]'
'concat=n=3:v=1:a=1[outv][outa]"'
)
else:
extra_input = ""
concat_filter = (
'-filter_complex "[0:v:0][0:a:0][1:v:0][1:a:0]'
'concat=n=2:v=1:a=1[outv][outa]"'
)
gpu_command = (
f"{self._ffmpeg_base_command} {self.hwaccel_decode_flags} -i {title_quoted} "
f"{self.hwaccel_decode_flags} -i {karaoke_quoted} {extra_input} "
f'{concat_filter} -map "[outv]" -map "[outa]" -c:v {self.video_encoder} '
f'{self._get_nvenc_quality_settings("lossless")} -c:a pcm_s16le '
f'{self.MP4_FLAGS} "{output_file}"'
)
cpu_command = (
f"{self._ffmpeg_base_command} -i {title_quoted} -i {karaoke_quoted} {extra_input} "
f'{concat_filter} -map "[outv]" -map "[outa]" -c:v libx264 -c:a pcm_s16le '
f'{self.MP4_FLAGS} "{output_file}"'
)
return self._execute_command_with_fallback(
gpu_command, cpu_command, "Encoding lossless 4K MP4"
)
def encode_lossy_mp4(
self,
input_file: str,
output_file: str,
) -> bool:
"""
Create lossy 4K MP4 with AAC audio.
Args:
input_file: Path to input file (typically lossless MP4)
output_file: Path for output file
Returns:
True if successful, False otherwise
"""
command = (
f'{self._ffmpeg_base_command} -i "{input_file}" '
f'-c:v copy -c:a {self.aac_codec} -ar 48000 -b:a 320k '
f'{self.MP4_FLAGS} "{output_file}"'
)
return self._execute_command(command, "Encoding lossy 4K MP4 with AAC")
def encode_lossless_mkv(
self,
input_file: str,
output_file: str,
) -> bool:
"""
Create MKV with FLAC audio (for YouTube upload).
Args:
input_file: Path to input file
output_file: Path for output file
Returns:
True if successful, False otherwise
"""
command = (
f'{self._ffmpeg_base_command} -i "{input_file}" '
f'-c:v copy -c:a flac "{output_file}"'
)
return self._execute_command(command, "Creating MKV with FLAC for YouTube")
def encode_720p(
self,
input_file: str,
output_file: str,
) -> bool:
"""
Create 720p MP4 with AAC audio.
Args:
input_file: Path to input file
output_file: Path for output file
Returns:
True if successful, False otherwise
"""
gpu_command = (
f'{self._ffmpeg_base_command} {self.hwaccel_decode_flags} -i "{input_file}" '
f'-c:v {self.video_encoder} -vf "{self.scale_filter}=1280:720" '
f'{self._get_nvenc_quality_settings("medium")} -b:v 2000k '
f'-c:a {self.aac_codec} -ar 48000 -b:a 128k '
f'{self.MP4_FLAGS} "{output_file}"'
)
cpu_command = (
f'{self._ffmpeg_base_command} -i "{input_file}" '
f'-c:v libx264 -vf "scale=1280:720" -b:v 2000k -preset medium -tune animation '
f'-c:a {self.aac_codec} -ar 48000 -b:a 128k '
f'{self.MP4_FLAGS} "{output_file}"'
)
return self._execute_command_with_fallback(
gpu_command, cpu_command, "Encoding 720p MP4"
)
def encode_all_formats(
self,
config: EncodingConfig,
) -> EncodingResult:
"""
Encode video to all output formats.
This performs the full encoding pipeline:
1. Remux with instrumental audio
2. Convert to MP4 if needed
3. Encode lossless 4K MP4 (concatenated)
4. Encode lossy 4K MP4
5. Encode lossless MKV (for YouTube)
6. Encode 720p MP4
Args:
config: Encoding configuration with input/output paths
Returns:
EncodingResult with success status and output file paths
"""
output_files = {}
try:
# Step 1: Remux with instrumental audio
if config.output_karaoke_mp4:
self.logger.info("[Step 1/6] Remuxing video with instrumental audio...")
if not self.remux_with_instrumental(
config.karaoke_video,
config.instrumental_audio,
config.output_karaoke_mp4
):
return EncodingResult(
success=False,
output_files=output_files,
error="Failed to remux with instrumental audio"
)
output_files["karaoke_mp4"] = config.output_karaoke_mp4
# Step 2: Convert to MP4 if needed
if config.output_with_vocals_mp4:
if not config.karaoke_video.endswith(".mp4"):
self.logger.info("[Step 2/6] Converting karaoke video to MP4...")
if not self.convert_mov_to_mp4(
config.karaoke_video,
config.output_with_vocals_mp4
):
return EncodingResult(
success=False,
output_files=output_files,
error="Failed to convert to MP4"
)
output_files["with_vocals_mp4"] = config.output_with_vocals_mp4
else:
self.logger.info("[Step 2/6] Skipped - video already MP4")
# Step 3: Encode lossless 4K MP4
if config.output_lossless_4k_mp4:
self.logger.info("[Step 3/6] Encoding lossless 4K MP4...")
karaoke_for_concat = config.output_karaoke_mp4 or config.karaoke_video
if not self.encode_lossless_mp4(
config.title_video,
karaoke_for_concat,
config.output_lossless_4k_mp4,
config.end_video
):
return EncodingResult(
success=False,
output_files=output_files,
error="Failed to encode lossless 4K MP4"
)
output_files["lossless_4k_mp4"] = config.output_lossless_4k_mp4
# Step 4: Encode lossy 4K MP4
if config.output_lossy_4k_mp4 and config.output_lossless_4k_mp4:
self.logger.info("[Step 4/6] Encoding lossy 4K MP4...")
if not self.encode_lossy_mp4(
config.output_lossless_4k_mp4,
config.output_lossy_4k_mp4
):
return EncodingResult(
success=False,
output_files=output_files,
error="Failed to encode lossy 4K MP4"
)
output_files["lossy_4k_mp4"] = config.output_lossy_4k_mp4
# Step 5: Create MKV with FLAC audio
if config.output_lossless_mkv and config.output_lossless_4k_mp4:
self.logger.info("[Step 5/6] Creating MKV with FLAC audio...")
if not self.encode_lossless_mkv(
config.output_lossless_4k_mp4,
config.output_lossless_mkv
):
return EncodingResult(
success=False,
output_files=output_files,
error="Failed to create MKV"
)
output_files["lossless_mkv"] = config.output_lossless_mkv
# Step 6: Encode 720p version
if config.output_720p_mp4 and config.output_lossless_4k_mp4:
self.logger.info("[Step 6/6] Encoding 720p MP4...")
if not self.encode_720p(
config.output_lossless_4k_mp4,
config.output_720p_mp4
):
return EncodingResult(
success=False,
output_files=output_files,
error="Failed to encode 720p"
)
output_files["720p_mp4"] = config.output_720p_mp4
self.logger.info("All encoding steps completed successfully")
return EncodingResult(success=True, output_files=output_files)
except Exception as e:
self.logger.error(f"Encoding failed with exception: {e}")
return EncodingResult(
success=False,
output_files=output_files,
error=str(e)
)
# Singleton instance and factory function (following existing service pattern)
_local_encoding_service: Optional[LocalEncodingService] = None
def get_local_encoding_service(**kwargs) -> LocalEncodingService:
"""
Get a local encoding service instance.
Args:
**kwargs: Arguments passed to LocalEncodingService
Returns:
LocalEncodingService instance
"""
global _local_encoding_service
if _local_encoding_service is None:
_local_encoding_service = LocalEncodingService(**kwargs)
return _local_encoding_service
+287
View File
@@ -0,0 +1,287 @@
"""
Packaging Service.
Provides CDG and TXT package generation functionality, extracted from KaraokeFinalise
for use by both the cloud backend (video_worker) and local CLI.
This service handles:
- CDG (CD+G) file generation from LRC files
- TXT lyric file generation from LRC files
- ZIP packaging of CDG/MP3 and TXT/MP3 pairs
"""
import logging
import os
import zipfile
from typing import Optional, Dict, Any, Tuple
logger = logging.getLogger(__name__)
class PackagingService:
"""
Service for creating CDG and TXT karaoke packages.
CDG (CD+Graphics) is a format used by karaoke machines.
TXT packages are used by software karaoke players.
Both formats are packaged as ZIP files with an MP3 audio track.
"""
def __init__(
self,
cdg_styles: Optional[Dict[str, Any]] = None,
dry_run: bool = False,
non_interactive: bool = False,
logger: Optional[logging.Logger] = None,
):
"""
Initialize the packaging service.
Args:
cdg_styles: CDG style configuration for CDG generation
dry_run: If True, log actions without performing them
non_interactive: If True, skip interactive prompts
logger: Optional logger instance
"""
self.cdg_styles = cdg_styles
self.dry_run = dry_run
self.non_interactive = non_interactive
self.logger = logger or logging.getLogger(__name__)
def create_cdg_package(
self,
lrc_file: str,
audio_file: str,
output_zip_path: str,
artist: str,
title: str,
output_mp3_path: Optional[str] = None,
output_cdg_path: Optional[str] = None,
) -> Tuple[str, Optional[str], Optional[str]]:
"""
Create a CDG package (ZIP containing CDG and MP3 files).
Args:
lrc_file: Path to the LRC lyrics file
audio_file: Path to the instrumental audio file
output_zip_path: Path for the output ZIP file
artist: Artist name for metadata
title: Song title for metadata
output_mp3_path: Optional path for the extracted MP3 file
output_cdg_path: Optional path for the extracted CDG file
Returns:
Tuple of (zip_path, mp3_path, cdg_path)
Raises:
ValueError: If CDG styles are not configured
FileNotFoundError: If input files are missing
Exception: If CDG generation fails
"""
self.logger.info(f"Creating CDG package for {artist} - {title}")
# Validate inputs
if not os.path.isfile(lrc_file):
raise FileNotFoundError(f"LRC file not found: {lrc_file}")
if not os.path.isfile(audio_file):
raise FileNotFoundError(f"Audio file not found: {audio_file}")
# Check if ZIP already exists
if os.path.isfile(output_zip_path):
if self.non_interactive:
self.logger.info(
f"CDG ZIP exists, will be overwritten: {output_zip_path}"
)
else:
self.logger.info(f"CDG ZIP already exists: {output_zip_path}")
# Check if individual files exist (allows skipping generation)
if output_mp3_path and output_cdg_path:
if os.path.isfile(output_mp3_path) and os.path.isfile(output_cdg_path):
self.logger.info("Found existing MP3 and CDG files, creating ZIP directly")
if not self.dry_run:
self._create_zip_from_files(
output_zip_path,
[(output_mp3_path, os.path.basename(output_mp3_path)),
(output_cdg_path, os.path.basename(output_cdg_path))]
)
return output_zip_path, output_mp3_path, output_cdg_path
if self.dry_run:
self.logger.info(
f"DRY RUN: Would generate CDG package: {output_zip_path}"
)
return output_zip_path, output_mp3_path, output_cdg_path
# Generate CDG files
if self.cdg_styles is None:
raise ValueError(
"CDG styles configuration is required for CDG generation"
)
self.logger.info("Generating CDG and MP3 files")
from lyrics_transcriber.output.cdg import CDGGenerator
output_dir = os.path.dirname(output_zip_path) or os.getcwd()
generator = CDGGenerator(output_dir=output_dir, logger=self.logger)
cdg_file, mp3_file, zip_file = generator.generate_cdg_from_lrc(
lrc_file=lrc_file,
audio_file=audio_file,
title=title,
artist=artist,
cdg_styles=self.cdg_styles,
)
# Rename ZIP to expected output path if different
if os.path.isfile(zip_file) and zip_file != output_zip_path:
os.rename(zip_file, output_zip_path)
self.logger.info(f"Renamed CDG ZIP: {zip_file} -> {output_zip_path}")
if not os.path.isfile(output_zip_path):
raise Exception(f"Failed to create CDG ZIP file: {output_zip_path}")
# Extract the ZIP to get individual files if paths provided
extracted_mp3 = None
extracted_cdg = None
if output_mp3_path or output_cdg_path:
self.logger.info(f"Extracting CDG ZIP file: {output_zip_path}")
with zipfile.ZipFile(output_zip_path, "r") as zip_ref:
zip_ref.extractall(output_dir)
# Find extracted files
if output_mp3_path and os.path.isfile(output_mp3_path):
extracted_mp3 = output_mp3_path
self.logger.info(f"Extracted MP3: {extracted_mp3}")
if output_cdg_path and os.path.isfile(output_cdg_path):
extracted_cdg = output_cdg_path
self.logger.info(f"Extracted CDG: {extracted_cdg}")
self.logger.info(f"CDG package created: {output_zip_path}")
return output_zip_path, extracted_mp3, extracted_cdg
def create_txt_package(
self,
lrc_file: str,
mp3_file: str,
output_zip_path: str,
output_txt_path: Optional[str] = None,
) -> Tuple[str, Optional[str]]:
"""
Create a TXT package (ZIP containing TXT lyrics and MP3 files).
Args:
lrc_file: Path to the LRC lyrics file
mp3_file: Path to the MP3 audio file
output_zip_path: Path for the output ZIP file
output_txt_path: Optional path for the generated TXT file
Returns:
Tuple of (zip_path, txt_path)
Raises:
FileNotFoundError: If input files are missing
Exception: If TXT generation fails
"""
self.logger.info(f"Creating TXT package from {lrc_file}")
# Validate inputs
if not os.path.isfile(lrc_file):
raise FileNotFoundError(f"LRC file not found: {lrc_file}")
if not os.path.isfile(mp3_file):
raise FileNotFoundError(f"MP3 file not found: {mp3_file}")
# Check if ZIP already exists
if os.path.isfile(output_zip_path):
if self.non_interactive:
self.logger.info(
f"TXT ZIP exists, will be overwritten: {output_zip_path}"
)
else:
self.logger.info(f"TXT ZIP already exists: {output_zip_path}")
if self.dry_run:
self.logger.info(
f"DRY RUN: Would create TXT package: {output_zip_path}"
)
return output_zip_path, output_txt_path
# Generate TXT from LRC
self.logger.info(f"Converting LRC to TXT format: {lrc_file}")
from lyrics_converter import LyricsConverter
txt_converter = LyricsConverter(output_format="txt", filepath=lrc_file)
converted_txt = txt_converter.convert_file()
# Write TXT file
if output_txt_path is None:
# Default to same name as ZIP but with .txt extension
output_txt_path = output_zip_path.replace(".zip", ".txt")
with open(output_txt_path, "w") as txt_file:
txt_file.write(converted_txt)
self.logger.info(f"TXT file written: {output_txt_path}")
# Create ZIP containing MP3 and TXT
self.logger.info(f"Creating TXT ZIP: {output_zip_path}")
self._create_zip_from_files(
output_zip_path,
[(mp3_file, os.path.basename(mp3_file)),
(output_txt_path, os.path.basename(output_txt_path))]
)
if not os.path.isfile(output_zip_path):
raise Exception(f"Failed to create TXT ZIP file: {output_zip_path}")
self.logger.info(f"TXT package created: {output_zip_path}")
return output_zip_path, output_txt_path
def _create_zip_from_files(
self,
zip_path: str,
files: list,
) -> None:
"""
Create a ZIP file from a list of files.
Args:
zip_path: Path for the output ZIP file
files: List of (file_path, archive_name) tuples
"""
with zipfile.ZipFile(zip_path, "w") as zipf:
for file_path, archive_name in files:
if os.path.isfile(file_path):
zipf.write(file_path, archive_name)
self.logger.debug(f"Added to ZIP: {archive_name}")
else:
self.logger.warning(f"File not found for ZIP: {file_path}")
# Singleton instance and factory function (following existing service pattern)
_packaging_service: Optional[PackagingService] = None
def get_packaging_service(
cdg_styles: Optional[Dict[str, Any]] = None,
**kwargs
) -> PackagingService:
"""
Get a packaging service instance.
Args:
cdg_styles: CDG style configuration
**kwargs: Additional arguments passed to PackagingService
Returns:
PackagingService instance
"""
global _packaging_service
# Create new instance if settings changed
if _packaging_service is None or cdg_styles:
_packaging_service = PackagingService(
cdg_styles=cdg_styles,
**kwargs
)
return _packaging_service
+445
View File
@@ -0,0 +1,445 @@
"""
YouTube Upload Service.
Provides video upload functionality for YouTube, extracted from KaraokeFinalise
for use by both the cloud backend (video_worker) and local CLI.
This service handles:
- Video uploads with metadata (title, description, tags)
- Thumbnail uploads
- Duplicate video detection and replacement
- Authentication via pre-stored credentials or client secrets file
"""
import logging
import os
from typing import Optional, Dict, Any, Tuple
from thefuzz import fuzz
logger = logging.getLogger(__name__)
class YouTubeUploadService:
"""
Service for uploading videos to YouTube.
Supports two authentication modes:
1. Pre-stored credentials (for server-side/non-interactive use)
2. Client secrets file (for interactive CLI use with browser OAuth)
"""
YOUTUBE_URL_PREFIX = "https://www.youtube.com/watch?v="
SCOPES = ["https://www.googleapis.com/auth/youtube"]
def __init__(
self,
credentials: Optional[Dict[str, Any]] = None,
client_secrets_file: Optional[str] = None,
non_interactive: bool = False,
server_side_mode: bool = False,
dry_run: bool = False,
logger: Optional[logging.Logger] = None,
):
"""
Initialize the YouTube upload service.
Args:
credentials: Pre-stored OAuth credentials dict with keys:
- token: Access token
- refresh_token: Refresh token
- token_uri: Token endpoint URL
- client_id: OAuth client ID
- client_secret: OAuth client secret
- scopes: List of OAuth scopes
client_secrets_file: Path to OAuth client secrets JSON file
(for interactive authentication)
non_interactive: If True, skip interactive prompts
server_side_mode: If True, use exact matching for duplicate detection
dry_run: If True, log actions without performing them
logger: Optional logger instance
"""
self.credentials = credentials
self.client_secrets_file = client_secrets_file
self.non_interactive = non_interactive
self.server_side_mode = server_side_mode
self.dry_run = dry_run
self.logger = logger or logging.getLogger(__name__)
self._youtube_service = None
self._channel_id = None
def _authenticate(self):
"""
Authenticate with YouTube API and return service object.
Returns:
YouTube API service object
Raises:
Exception: If authentication fails or is required in non-interactive mode
"""
from google.auth.transport.requests import Request
from google.oauth2.credentials import Credentials
from googleapiclient.discovery import build
from google_auth_oauthlib.flow import InstalledAppFlow
import pickle
# Check if we have pre-stored credentials (for non-interactive mode)
if self.credentials and self.non_interactive:
try:
creds = Credentials(
token=self.credentials.get('token'),
refresh_token=self.credentials.get('refresh_token'),
token_uri=self.credentials.get('token_uri'),
client_id=self.credentials.get('client_id'),
client_secret=self.credentials.get('client_secret'),
scopes=self.credentials.get('scopes', self.SCOPES)
)
# Refresh token if needed
if creds.expired and creds.refresh_token:
creds.refresh(Request())
youtube = build('youtube', 'v3', credentials=creds)
self.logger.info("Successfully authenticated with YouTube using pre-stored credentials")
return youtube
except Exception as e:
self.logger.error(f"Failed to authenticate with pre-stored credentials: {str(e)}")
# Fall through to original authentication if pre-stored credentials fail
# For non-interactive mode without pre-stored credentials, raise error
if self.non_interactive:
raise Exception(
"YouTube authentication required but running in non-interactive mode. "
"Please pre-authenticate or disable YouTube upload."
)
# Interactive authentication using client secrets file
if not self.client_secrets_file:
raise Exception("No YouTube credentials or client secrets file provided")
# Token file stores the user's access and refresh tokens
youtube_token_file = "/tmp/karaoke-finalise-youtube-token.pickle"
creds = None
# Check if we have saved credentials
if os.path.exists(youtube_token_file):
with open(youtube_token_file, "rb") as token:
creds = pickle.load(token)
# If there are no valid credentials, let the user log in
if not creds or not creds.valid:
if creds and creds.expired and creds.refresh_token:
creds.refresh(Request())
else:
flow = InstalledAppFlow.from_client_secrets_file(
self.client_secrets_file, scopes=self.SCOPES
)
creds = flow.run_local_server(port=0)
# Save the credentials for the next run
with open(youtube_token_file, "wb") as token:
pickle.dump(creds, token)
return build("youtube", "v3", credentials=creds)
@property
def youtube_service(self):
"""Lazy-load YouTube service on first access."""
if self._youtube_service is None:
self._youtube_service = self._authenticate()
return self._youtube_service
def get_channel_id(self) -> Optional[str]:
"""
Get the authenticated user's YouTube channel ID.
Returns:
Channel ID string, or None if not found
"""
if self._channel_id:
return self._channel_id
request = self.youtube_service.channels().list(part="snippet", mine=True)
response = request.execute()
if "items" in response and len(response["items"]) > 0:
self._channel_id = response["items"][0]["id"]
return self._channel_id
return None
def check_duplicate(
self,
title: str,
exact_match: Optional[bool] = None
) -> Tuple[bool, Optional[str], Optional[str]]:
"""
Check if a video with the given title already exists on the channel.
Args:
title: Video title to search for
exact_match: If True, require exact title match. If None, uses
server_side_mode setting (exact in server mode, fuzzy in CLI)
Returns:
Tuple of (exists: bool, video_id: Optional[str], video_url: Optional[str])
"""
channel_id = self.get_channel_id()
if not channel_id:
self.logger.warning("Could not get channel ID, skipping duplicate check")
return False, None, None
use_exact_match = exact_match if exact_match is not None else self.server_side_mode
self.logger.info(f"Searching YouTube channel {channel_id} for title: {title}")
request = self.youtube_service.search().list(
part="snippet",
channelId=channel_id,
q=title,
type="video",
maxResults=10
)
response = request.execute()
if "items" not in response or len(response["items"]) == 0:
self.logger.info(f"No matching video found with title: {title}")
return False, None, None
for item in response["items"]:
# Verify the video actually belongs to our channel
result_channel_id = item["snippet"]["channelId"]
if result_channel_id != channel_id:
self.logger.debug(
f"Skipping video from different channel: {item['snippet']['title']} "
f"(channel: {result_channel_id})"
)
continue
found_title = item["snippet"]["title"]
# Determine if this is a match
if use_exact_match:
is_match = title.lower() == found_title.lower()
similarity_score = 100 if is_match else 0
else:
similarity_score = fuzz.ratio(title.lower(), found_title.lower())
is_match = similarity_score >= 70
if is_match:
video_id = item["id"]["videoId"]
video_url = f"{self.YOUTUBE_URL_PREFIX}{video_id}"
self.logger.info(
f"Potential match found on YouTube channel with ID: {video_id} "
f"and title: {found_title} (similarity: {similarity_score}%)"
)
# In non-interactive mode, return the match directly
if self.non_interactive:
self.logger.info("Non-interactive mode, found a match.")
return True, video_id, video_url
# Interactive confirmation
confirmation = input(
f"Is '{found_title}' the video you are finalising? (y/n): "
).strip().lower()
if confirmation == "y":
return True, video_id, video_url
self.logger.info(f"No matching video found with title: {title}")
return False, None, None
def delete_video(self, video_id: str) -> bool:
"""
Delete a YouTube video by its ID.
Args:
video_id: The YouTube video ID to delete
Returns:
True if successful, False otherwise
"""
self.logger.info(f"Deleting YouTube video with ID: {video_id}")
if self.dry_run:
self.logger.info(f"DRY RUN: Would delete YouTube video with ID: {video_id}")
return True
try:
self.youtube_service.videos().delete(id=video_id).execute()
self.logger.info(f"Successfully deleted YouTube video with ID: {video_id}")
return True
except Exception as e:
self.logger.error(f"Failed to delete YouTube video with ID {video_id}: {e}")
return False
@staticmethod
def truncate_title(title: str, max_length: int = 95) -> str:
"""
Truncate title to the nearest whole word within max_length.
Args:
title: Title to truncate
max_length: Maximum length (default 95 for YouTube)
Returns:
Truncated title with "..." if needed
"""
if len(title) <= max_length:
return title
truncated_title = title[:max_length].rsplit(" ", 1)[0]
if len(truncated_title) < max_length:
truncated_title += " ..."
return truncated_title
def upload_video(
self,
video_path: str,
title: str,
description: str,
thumbnail_path: Optional[str] = None,
tags: Optional[list] = None,
category_id: str = "10", # Music category
privacy_status: str = "public",
replace_existing: bool = False,
) -> Tuple[Optional[str], Optional[str]]:
"""
Upload a video to YouTube with metadata and optional thumbnail.
Args:
video_path: Path to the video file to upload
title: Video title (will be truncated to 95 chars if needed)
description: Video description
thumbnail_path: Optional path to thumbnail image
tags: Optional list of tags/keywords
category_id: YouTube category ID (default "10" for Music)
privacy_status: "public", "private", or "unlisted"
replace_existing: If True, delete existing video with same title
Returns:
Tuple of (video_id, video_url) or (None, None) if upload failed/skipped
"""
from googleapiclient.http import MediaFileUpload
# Truncate title if needed
youtube_title = self.truncate_title(title)
self.logger.info(f"Uploading video to YouTube: {youtube_title}")
if self.dry_run:
self.logger.info(
f"DRY RUN: Would upload {video_path} to YouTube with title: {youtube_title}"
)
return "dry_run_video_id", f"{self.YOUTUBE_URL_PREFIX}dry_run_video_id"
# Check for existing video
should_replace = True if self.server_side_mode else replace_existing
exists, existing_id, existing_url = self.check_duplicate(youtube_title)
if exists:
if should_replace:
self.logger.info(f"Video already exists on YouTube, deleting before re-upload: {existing_url}")
if self.delete_video(existing_id):
self.logger.info("Successfully deleted existing video, proceeding with upload")
else:
self.logger.error("Failed to delete existing video, aborting upload")
return None, None
else:
self.logger.warning(f"Video already exists on YouTube, skipping upload: {existing_url}")
return existing_id, existing_url
# Prepare video metadata
body = {
"snippet": {
"title": youtube_title,
"description": description,
"tags": tags or [],
"categoryId": category_id,
},
"status": {"privacyStatus": privacy_status},
}
# Determine MIME type based on file extension
ext = os.path.splitext(video_path)[1].lower()
mime_type = {
".mkv": "video/x-matroska",
".mp4": "video/mp4",
".mov": "video/quicktime",
".avi": "video/x-msvideo",
}.get(ext, "video/*")
# Upload video
self.logger.info(f"Authenticating with YouTube...")
media_file = MediaFileUpload(video_path, mimetype=mime_type, resumable=True)
self.logger.info(f"Uploading video to YouTube...")
request = self.youtube_service.videos().insert(
part="snippet,status",
body=body,
media_body=media_file
)
response = request.execute()
video_id = response.get("id")
video_url = f"{self.YOUTUBE_URL_PREFIX}{video_id}"
self.logger.info(f"Uploaded video to YouTube: {video_url}")
# Upload thumbnail if provided
if thumbnail_path and os.path.isfile(thumbnail_path):
try:
self.logger.info(f"Uploading thumbnail from: {thumbnail_path}")
media_thumbnail = MediaFileUpload(thumbnail_path, mimetype="image/jpeg")
self.youtube_service.thumbnails().set(
videoId=video_id,
media_body=media_thumbnail
).execute()
self.logger.info(f"Uploaded thumbnail for video ID {video_id}")
except Exception as e:
self.logger.error(f"Failed to upload thumbnail: {e}")
self.logger.warning(
"Video uploaded but thumbnail not set. "
"You may need to set it manually on YouTube."
)
elif thumbnail_path:
self.logger.warning(f"Thumbnail file not found, skipping: {thumbnail_path}")
return video_id, video_url
# Singleton instance and factory function (following existing service pattern)
_youtube_upload_service: Optional[YouTubeUploadService] = None
def get_youtube_upload_service(
credentials: Optional[Dict[str, Any]] = None,
client_secrets_file: Optional[str] = None,
**kwargs
) -> YouTubeUploadService:
"""
Get a YouTube upload service instance.
For server-side use, pass credentials from YouTubeService.
For CLI use, pass client_secrets_file.
Args:
credentials: Pre-stored OAuth credentials dict
client_secrets_file: Path to OAuth client secrets JSON file
**kwargs: Additional arguments passed to YouTubeUploadService
Returns:
YouTubeUploadService instance
"""
global _youtube_upload_service
# Create new instance if credentials/settings changed
if _youtube_upload_service is None or credentials or client_secrets_file:
_youtube_upload_service = YouTubeUploadService(
credentials=credentials,
client_secrets_file=client_secrets_file,
**kwargs
)
return _youtube_upload_service
+15 -9
View File
@@ -6,23 +6,29 @@ Provides common mocks and test utilities across all test modules.
NOTE: Module-level mocks are NOT applied here to allow emulator tests to work.
Individual tests must mock dependencies as needed.
"""
# IMPORTANT: Set environment variables FIRST, before ANY imports that might
# trigger backend module loading and Settings singleton creation.
import os
import pytest
from unittest.mock import Mock, MagicMock, AsyncMock, patch
from datetime import datetime, UTC
from fastapi.testclient import TestClient
# Mock google.auth.default AND firestore for unit tests if not using emulator
# This prevents DefaultCredentialsError and FirestoreClient initialization during imports
# Set up test environment variables BEFORE importing any backend modules
# This must happen before Settings is instantiated, which occurs on first import
if 'FIRESTORE_EMULATOR_HOST' not in os.environ:
from unittest.mock import MagicMock
# Set up test environment variables BEFORE importing any backend modules
os.environ.setdefault('ADMIN_TOKENS', 'test-admin-token')
os.environ.setdefault('GOOGLE_CLOUD_PROJECT', 'test-project')
os.environ.setdefault('GCS_BUCKET_NAME', 'test-bucket')
os.environ.setdefault('FIRESTORE_COLLECTION', 'jobs')
os.environ.setdefault('ENVIRONMENT', 'test')
import pytest
from unittest.mock import Mock, MagicMock, AsyncMock, patch
from datetime import datetime, UTC
from fastapi.testclient import TestClient
# Mock google.auth.default AND firestore for unit tests if not using emulator
# This prevents DefaultCredentialsError and FirestoreClient initialization during imports
if 'FIRESTORE_EMULATOR_HOST' not in os.environ:
from unittest.mock import MagicMock
# Mock google.auth.default - prevents credential errors
try:
+244
View File
@@ -0,0 +1,244 @@
"""
Tests for DiscordNotificationService.
Tests cover:
- Service initialization
- Webhook URL validation
- Message posting
- Video notification posting
- Dry run mode
"""
import pytest
from unittest.mock import MagicMock, patch
from backend.services.discord_service import (
DiscordNotificationService,
get_discord_notification_service,
)
class TestDiscordNotificationServiceInit:
"""Test service initialization."""
def test_init_with_webhook_url(self):
"""Test initialization with valid webhook URL."""
url = "https://discord.com/api/webhooks/123456789/abcdef123456"
service = DiscordNotificationService(webhook_url=url)
assert service.webhook_url == url
assert service.is_enabled() is True
def test_init_without_webhook_url(self):
"""Test initialization without webhook URL."""
service = DiscordNotificationService()
assert service.webhook_url is None
assert service.is_enabled() is False
def test_init_with_dry_run(self):
"""Test initialization with dry run mode."""
service = DiscordNotificationService(dry_run=True)
assert service.dry_run is True
def test_init_with_invalid_webhook_url_raises(self):
"""Test that invalid webhook URL raises ValueError."""
with pytest.raises(ValueError) as exc_info:
DiscordNotificationService(webhook_url="https://example.com/webhook")
assert "Invalid Discord webhook URL" in str(exc_info.value)
def test_init_with_non_discord_url_raises(self):
"""Test that non-Discord URL raises ValueError."""
with pytest.raises(ValueError):
DiscordNotificationService(
webhook_url="https://slack.com/api/webhooks/123"
)
class TestDiscordNotificationServiceValidation:
"""Test webhook URL validation."""
def test_validate_valid_webhook_url(self):
"""Test that valid webhook URL passes validation."""
service = DiscordNotificationService()
# Should not raise
service._validate_webhook_url(
"https://discord.com/api/webhooks/123456789/abcdef"
)
def test_validate_invalid_webhook_url(self):
"""Test that invalid webhook URL raises ValueError."""
service = DiscordNotificationService()
with pytest.raises(ValueError) as exc_info:
service._validate_webhook_url("https://example.com/webhook")
assert "Invalid Discord webhook URL" in str(exc_info.value)
def test_validate_strips_whitespace(self):
"""Test that whitespace is stripped from URL."""
service = DiscordNotificationService()
# Should not raise - whitespace is stripped
service._validate_webhook_url(
" https://discord.com/api/webhooks/123456789/abcdef "
)
class TestDiscordNotificationServicePostMessage:
"""Test message posting."""
@patch("backend.services.discord_service.requests.post")
def test_post_message_success(self, mock_post):
"""Test successful message posting."""
mock_response = MagicMock()
mock_response.raise_for_status = MagicMock()
mock_post.return_value = mock_response
url = "https://discord.com/api/webhooks/123456789/abcdef"
service = DiscordNotificationService(webhook_url=url)
result = service.post_message("Test message")
assert result is True
mock_post.assert_called_once_with(
url,
json={"content": "Test message"},
timeout=30
)
@patch("backend.services.discord_service.requests.post")
def test_post_message_with_custom_webhook(self, mock_post):
"""Test message posting with custom webhook URL."""
mock_response = MagicMock()
mock_response.raise_for_status = MagicMock()
mock_post.return_value = mock_response
service = DiscordNotificationService()
custom_url = "https://discord.com/api/webhooks/987654321/fedcba"
result = service.post_message("Test message", webhook_url=custom_url)
assert result is True
mock_post.assert_called_once_with(
custom_url,
json={"content": "Test message"},
timeout=30
)
def test_post_message_no_webhook_raises(self):
"""Test that posting without webhook URL raises ValueError."""
service = DiscordNotificationService()
with pytest.raises(ValueError) as exc_info:
service.post_message("Test message")
assert "No Discord webhook URL provided" in str(exc_info.value)
def test_post_message_dry_run(self):
"""Test message posting in dry run mode."""
url = "https://discord.com/api/webhooks/123456789/abcdef"
service = DiscordNotificationService(webhook_url=url, dry_run=True)
result = service.post_message("Test message")
assert result is True
# In dry run mode, no actual request should be made
@patch("backend.services.discord_service.requests.post")
def test_post_message_http_error(self, mock_post):
"""Test handling of HTTP errors."""
import requests as req
mock_response = MagicMock()
mock_response.raise_for_status.side_effect = req.HTTPError("404 Not Found")
mock_post.return_value = mock_response
url = "https://discord.com/api/webhooks/123456789/abcdef"
service = DiscordNotificationService(webhook_url=url)
with pytest.raises(req.HTTPError):
service.post_message("Test message")
class TestDiscordNotificationServiceVideoNotification:
"""Test video notification posting."""
@patch("backend.services.discord_service.requests.post")
def test_post_video_notification_success(self, mock_post):
"""Test successful video notification."""
mock_response = MagicMock()
mock_response.raise_for_status = MagicMock()
mock_post.return_value = mock_response
url = "https://discord.com/api/webhooks/123456789/abcdef"
service = DiscordNotificationService(webhook_url=url)
result = service.post_video_notification(
"https://www.youtube.com/watch?v=abc123"
)
assert result is True
mock_post.assert_called_once()
call_args = mock_post.call_args
assert "New upload:" in call_args[1]["json"]["content"]
assert "abc123" in call_args[1]["json"]["content"]
def test_post_video_notification_no_youtube_url(self):
"""Test video notification with empty YouTube URL."""
url = "https://discord.com/api/webhooks/123456789/abcdef"
service = DiscordNotificationService(webhook_url=url)
result = service.post_video_notification("")
assert result is False
def test_post_video_notification_none_youtube_url(self):
"""Test video notification with None YouTube URL."""
url = "https://discord.com/api/webhooks/123456789/abcdef"
service = DiscordNotificationService(webhook_url=url)
result = service.post_video_notification(None)
assert result is False
def test_post_video_notification_dry_run(self):
"""Test video notification in dry run mode."""
url = "https://discord.com/api/webhooks/123456789/abcdef"
service = DiscordNotificationService(webhook_url=url, dry_run=True)
result = service.post_video_notification(
"https://www.youtube.com/watch?v=abc123"
)
assert result is True
class TestGetDiscordNotificationService:
"""Test factory function."""
def test_get_service_creates_instance(self):
"""Test that factory function creates a new instance."""
import backend.services.discord_service as module
module._discord_notification_service = None
url = "https://discord.com/api/webhooks/123456789/abcdef"
service = get_discord_notification_service(webhook_url=url)
assert service is not None
assert isinstance(service, DiscordNotificationService)
assert service.webhook_url == url
def test_get_service_without_webhook(self):
"""Test factory function without webhook URL."""
import backend.services.discord_service as module
module._discord_notification_service = None
service = get_discord_notification_service()
assert service is not None
assert service.webhook_url is None
assert service.is_enabled() is False
def test_get_service_with_dry_run(self):
"""Test factory function with dry run mode."""
import backend.services.discord_service as module
module._discord_notification_service = None
url = "https://discord.com/api/webhooks/123456789/abcdef"
service = get_discord_notification_service(webhook_url=url, dry_run=True)
assert service.dry_run is True
+412
View File
@@ -0,0 +1,412 @@
"""
Tests for Encoding Interface.
Tests cover:
- EncodingInput and EncodingOutput dataclasses
- LocalEncodingBackend implementation
- Backend factory function
"""
import pytest
from unittest.mock import MagicMock, patch, AsyncMock
from backend.services.encoding_interface import (
EncodingInput,
EncodingOutput,
EncodingBackend,
LocalEncodingBackend,
GCEEncodingBackend,
get_encoding_backend,
)
class TestEncodingInput:
"""Test EncodingInput dataclass."""
def test_required_fields(self):
"""Test that required fields must be provided."""
input_config = EncodingInput(
title_video_path="/path/title.mov",
karaoke_video_path="/path/karaoke.mov",
instrumental_audio_path="/path/audio.flac",
)
assert input_config.title_video_path == "/path/title.mov"
assert input_config.end_video_path is None
def test_all_fields(self):
"""Test all fields including optional ones."""
input_config = EncodingInput(
title_video_path="/path/title.mov",
karaoke_video_path="/path/karaoke.mov",
instrumental_audio_path="/path/audio.flac",
end_video_path="/path/end.mov",
artist="Test Artist",
title="Test Title",
brand_code="NOMAD-1234",
output_dir="/output",
options={"quality": "high"}
)
assert input_config.artist == "Test Artist"
assert input_config.brand_code == "NOMAD-1234"
assert input_config.options["quality"] == "high"
class TestEncodingOutput:
"""Test EncodingOutput dataclass."""
def test_success_output(self):
"""Test successful output."""
output = EncodingOutput(
success=True,
lossless_4k_mp4_path="/output/video.mp4",
encoding_backend="local"
)
assert output.success is True
assert output.error_message is None
def test_failure_output(self):
"""Test failure output."""
output = EncodingOutput(
success=False,
error_message="Encoding failed",
encoding_backend="gce"
)
assert output.success is False
assert output.error_message == "Encoding failed"
def test_output_files_dict(self):
"""Test output_files dictionary."""
output = EncodingOutput(
success=True,
output_files={
"lossless_4k_mp4": "/output/lossless.mp4",
"720p_mp4": "/output/720p.mp4"
}
)
assert "lossless_4k_mp4" in output.output_files
assert output.output_files["720p_mp4"] == "/output/720p.mp4"
class TestLocalEncodingBackend:
"""Test LocalEncodingBackend implementation."""
def test_name(self):
"""Test backend name."""
backend = LocalEncodingBackend()
assert backend.name == "local"
def test_init_with_dry_run(self):
"""Test initialization with dry run."""
backend = LocalEncodingBackend(dry_run=True)
assert backend.dry_run is True
@pytest.mark.asyncio
@patch("subprocess.run")
async def test_is_available_success(self, mock_run):
"""Test availability check when FFmpeg is installed."""
mock_run.return_value = MagicMock(returncode=0)
backend = LocalEncodingBackend()
available = await backend.is_available()
assert available is True
@pytest.mark.asyncio
@patch("subprocess.run")
async def test_is_available_not_found(self, mock_run):
"""Test availability check when FFmpeg is not installed."""
mock_run.side_effect = FileNotFoundError()
backend = LocalEncodingBackend()
available = await backend.is_available()
assert available is False
@pytest.mark.asyncio
@patch.object(LocalEncodingBackend, "is_available")
@patch.object(LocalEncodingBackend, "_get_service")
async def test_get_status(self, mock_get_service, mock_is_available):
"""Test status retrieval."""
mock_is_available.return_value = True
mock_service = MagicMock()
mock_service.hwaccel_available = True
mock_service.video_encoder = "h264_nvenc"
mock_get_service.return_value = mock_service
backend = LocalEncodingBackend()
status = await backend.get_status()
assert status["backend"] == "local"
assert status["available"] is True
@pytest.mark.asyncio
@patch("asyncio.to_thread")
@patch.object(LocalEncodingBackend, "_get_service")
async def test_encode_success(self, mock_get_service, mock_to_thread):
"""Test successful encoding."""
from backend.services.local_encoding_service import EncodingResult
mock_result = EncodingResult(
success=True,
output_files={"key": "/path/file.mp4"}
)
mock_to_thread.return_value = mock_result
mock_service = MagicMock()
mock_get_service.return_value = mock_service
backend = LocalEncodingBackend()
input_config = EncodingInput(
title_video_path="/input/title.mov",
karaoke_video_path="/input/karaoke.mov",
instrumental_audio_path="/input/audio.flac",
artist="Test Artist",
title="Test Title",
output_dir="/output"
)
output = await backend.encode(input_config)
assert output.success is True
assert output.encoding_backend == "local"
assert output.encoding_time_seconds is not None
@pytest.mark.asyncio
@patch("asyncio.to_thread")
@patch.object(LocalEncodingBackend, "_get_service")
async def test_encode_failure(self, mock_get_service, mock_to_thread):
"""Test encoding failure."""
from backend.services.local_encoding_service import EncodingResult
mock_result = EncodingResult(
success=False,
output_files={},
error="FFmpeg failed"
)
mock_to_thread.return_value = mock_result
mock_service = MagicMock()
mock_get_service.return_value = mock_service
backend = LocalEncodingBackend()
input_config = EncodingInput(
title_video_path="/input/title.mov",
karaoke_video_path="/input/karaoke.mov",
instrumental_audio_path="/input/audio.flac",
artist="Test",
title="Test",
)
output = await backend.encode(input_config)
assert output.success is False
assert output.error_message == "FFmpeg failed"
class TestGCEEncodingBackend:
"""Test GCEEncodingBackend implementation."""
def test_name(self):
"""Test backend name."""
backend = GCEEncodingBackend()
assert backend.name == "gce"
@patch.object(GCEEncodingBackend, "_get_service")
@pytest.mark.asyncio
async def test_is_available_enabled(self, mock_get_service):
"""Test availability when GCE is enabled."""
mock_service = MagicMock()
mock_service.is_enabled = True
mock_get_service.return_value = mock_service
backend = GCEEncodingBackend()
available = await backend.is_available()
assert available is True
@patch.object(GCEEncodingBackend, "_get_service")
@pytest.mark.asyncio
async def test_is_available_disabled(self, mock_get_service):
"""Test availability when GCE is disabled."""
mock_service = MagicMock()
mock_service.is_enabled = False
mock_get_service.return_value = mock_service
backend = GCEEncodingBackend()
available = await backend.is_available()
assert available is False
@patch.object(GCEEncodingBackend, "_get_service")
@pytest.mark.asyncio
async def test_encode_missing_gcs_paths(self, mock_get_service):
"""Test encoding fails without GCS paths."""
backend = GCEEncodingBackend()
input_config = EncodingInput(
title_video_path="/input/title.mov",
karaoke_video_path="/input/karaoke.mov",
instrumental_audio_path="/input/audio.flac",
# Missing GCS paths in options
)
output = await backend.encode(input_config)
assert output.success is False
assert "gcs_path" in output.error_message.lower()
@patch.object(GCEEncodingBackend, "_get_service")
@pytest.mark.asyncio
async def test_encode_success(self, mock_get_service):
"""Test successful GCE encoding."""
mock_service = MagicMock()
mock_service.encode_videos = AsyncMock(return_value={
"status": "complete",
"output_files": {
"mp4_4k_lossless": "gs://bucket/output/lossless.mp4",
"mp4_720p": "gs://bucket/output/720p.mp4",
}
})
mock_get_service.return_value = mock_service
backend = GCEEncodingBackend()
input_config = EncodingInput(
title_video_path="/input/title.mov",
karaoke_video_path="/input/karaoke.mov",
instrumental_audio_path="/input/audio.flac",
artist="Test Artist",
title="Test Title",
options={
"job_id": "test-job",
"input_gcs_path": "gs://bucket/input/",
"output_gcs_path": "gs://bucket/output/",
}
)
output = await backend.encode(input_config)
assert output.success is True
assert output.encoding_backend == "gce"
mock_service.encode_videos.assert_called_once()
@patch.object(GCEEncodingBackend, "_get_service")
@pytest.mark.asyncio
async def test_encode_failure(self, mock_get_service):
"""Test GCE encoding failure handling."""
mock_service = MagicMock()
mock_service.encode_videos = AsyncMock(side_effect=Exception("GCE worker error"))
mock_get_service.return_value = mock_service
backend = GCEEncodingBackend()
input_config = EncodingInput(
title_video_path="/input/title.mov",
karaoke_video_path="/input/karaoke.mov",
instrumental_audio_path="/input/audio.flac",
options={
"job_id": "test-job",
"input_gcs_path": "gs://bucket/input/",
"output_gcs_path": "gs://bucket/output/",
}
)
output = await backend.encode(input_config)
assert output.success is False
assert "GCE worker error" in output.error_message
@patch.object(GCEEncodingBackend, "_get_service")
@pytest.mark.asyncio
async def test_encode_handles_list_result(self, mock_get_service):
"""Test GCE encoding handles list response gracefully.
This would have caught: 'list' object has no attribute 'get' error
when GCE worker returns a list instead of a dict.
"""
mock_service = MagicMock()
# Simulate GCE worker returning a list instead of dict
mock_service.encode_videos = AsyncMock(return_value=[
{"output_files": {"mp4_4k_lossless": "gs://bucket/output/lossless.mp4"}}
])
mock_get_service.return_value = mock_service
backend = GCEEncodingBackend()
input_config = EncodingInput(
title_video_path="/input/title.mov",
karaoke_video_path="/input/karaoke.mov",
instrumental_audio_path="/input/audio.flac",
options={
"job_id": "test-job",
"input_gcs_path": "gs://bucket/input/",
"output_gcs_path": "gs://bucket/output/",
}
)
# This should not raise an error
output = await backend.encode(input_config)
# Should still succeed by extracting from the list
assert output.success is True
assert output.lossless_4k_mp4_path == "gs://bucket/output/lossless.mp4"
class TestGetEncodingBackend:
"""Test encoding backend factory function."""
def test_get_local_backend(self):
"""Test getting local backend."""
backend = get_encoding_backend("local")
assert isinstance(backend, LocalEncodingBackend)
assert backend.name == "local"
@patch.object(GCEEncodingBackend, "_get_service")
def test_get_auto_backend_gce_disabled(self, mock_get_service):
"""Test getting auto backend falls back to local when GCE disabled."""
mock_service = MagicMock()
mock_service.is_enabled = False
mock_get_service.return_value = mock_service
backend = get_encoding_backend("auto")
assert isinstance(backend, LocalEncodingBackend)
def test_get_local_backend_with_options(self):
"""Test getting local backend with options."""
backend = get_encoding_backend("local", dry_run=True)
assert backend.dry_run is True
def test_get_gce_backend(self):
"""Test getting GCE backend."""
backend = get_encoding_backend("gce")
assert isinstance(backend, GCEEncodingBackend)
assert backend.name == "gce"
def test_get_unknown_backend_raises(self):
"""Test that unknown backend raises ValueError."""
with pytest.raises(ValueError) as exc_info:
get_encoding_backend("unknown")
assert "Unknown encoding backend type" in str(exc_info.value)
def test_get_gce_backend_with_options(self):
"""Test getting GCE backend with common options like dry_run.
This ensures all backends accept the same kwargs, preventing
errors when get_encoding_backend() passes **kwargs to different backends.
"""
# This would have caught: GCEEncodingBackend.__init__() got an unexpected keyword argument 'dry_run'
backend = get_encoding_backend("gce", dry_run=True)
assert isinstance(backend, GCEEncodingBackend)
assert backend.dry_run is True
@patch.object(GCEEncodingBackend, "_get_service")
def test_all_backends_accept_dry_run(self, mock_get_service):
"""Test that all backend types accept dry_run parameter.
This is an integration test to ensure the factory function
can pass dry_run to any backend without TypeError.
"""
mock_service = MagicMock()
mock_service.is_enabled = False
mock_get_service.return_value = mock_service
for backend_type in ["local", "gce", "auto"]:
# This should not raise TypeError
backend = get_encoding_backend(backend_type, dry_run=True)
assert backend is not None
@@ -0,0 +1,423 @@
"""
Tests for LocalEncodingService.
Tests cover:
- Service initialization
- Hardware acceleration detection
- FFmpeg command execution
- Individual encoding methods
- Full encoding pipeline
- Dry run mode
"""
import os
import tempfile
import pytest
from unittest.mock import MagicMock, patch, call
from backend.services.local_encoding_service import (
LocalEncodingService,
EncodingConfig,
EncodingResult,
get_local_encoding_service,
)
class TestLocalEncodingServiceInit:
"""Test service initialization."""
def test_init_default_values(self):
"""Test default initialization."""
service = LocalEncodingService()
assert service.dry_run is False
assert "ffmpeg" in service._ffmpeg_base_command
def test_init_with_dry_run(self):
"""Test initialization with dry run mode."""
service = LocalEncodingService(dry_run=True)
assert service.dry_run is True
def test_init_with_debug_logging(self):
"""Test initialization with debug logging level."""
import logging
service = LocalEncodingService(log_level=logging.DEBUG)
assert "-loglevel verbose" in service._ffmpeg_base_command
def test_init_with_info_logging(self):
"""Test initialization with info logging level."""
import logging
service = LocalEncodingService(log_level=logging.INFO)
assert "-loglevel fatal" in service._ffmpeg_base_command
class TestLocalEncodingServiceHWAccel:
"""Test hardware acceleration detection."""
@patch("subprocess.run")
def test_detect_nvenc_available(self, mock_run):
"""Test NVENC detection when available."""
mock_run.return_value = MagicMock(returncode=0)
service = LocalEncodingService()
# Force detection
service._detect_and_set_hwaccel()
assert service._hwaccel_available is True
assert service._video_encoder == "h264_nvenc"
assert service._scale_filter == "scale_cuda"
@patch("subprocess.run")
def test_detect_no_hwaccel(self, mock_run):
"""Test fallback when no hardware acceleration available."""
import subprocess
mock_run.side_effect = subprocess.CalledProcessError(1, "test")
service = LocalEncodingService()
# Force detection
service._detect_and_set_hwaccel()
assert service._hwaccel_available is False
assert service._video_encoder == "libx264"
assert service._scale_filter == "scale"
def test_nvenc_quality_settings(self):
"""Test NVENC quality settings for different presets."""
service = LocalEncodingService()
service._hwaccel_available = True
lossless = service._get_nvenc_quality_settings("lossless")
assert "cq 0" in lossless
medium = service._get_nvenc_quality_settings("medium")
assert "p4" in medium
def test_nvenc_quality_settings_disabled(self):
"""Test NVENC quality settings when hwaccel is disabled."""
service = LocalEncodingService()
service._hwaccel_available = False
settings = service._get_nvenc_quality_settings("lossless")
assert settings == ""
class TestLocalEncodingServiceExecuteCommand:
"""Test command execution."""
@patch("subprocess.run")
def test_execute_command_success(self, mock_run):
"""Test successful command execution."""
mock_run.return_value = MagicMock(returncode=0)
service = LocalEncodingService()
result = service._execute_command("echo test", "Test command")
assert result is True
mock_run.assert_called_once()
@patch("subprocess.run")
def test_execute_command_failure(self, mock_run):
"""Test command execution failure."""
import subprocess
mock_run.side_effect = subprocess.CalledProcessError(
1, "test", stderr="Error message"
)
service = LocalEncodingService()
result = service._execute_command("echo test", "Test command")
assert result is False
@patch("subprocess.run")
def test_execute_command_timeout(self, mock_run):
"""Test command execution timeout."""
import subprocess
mock_run.side_effect = subprocess.TimeoutExpired("test", 30)
service = LocalEncodingService()
result = service._execute_command("echo test", "Test command", timeout=30)
assert result is False
def test_execute_command_dry_run(self):
"""Test command execution in dry run mode."""
service = LocalEncodingService(dry_run=True)
result = service._execute_command("echo test", "Test command")
assert result is True
# No actual subprocess should be called in dry run mode
class TestLocalEncodingServiceEncodingMethods:
"""Test individual encoding methods."""
@patch.object(LocalEncodingService, "_execute_command")
def test_remux_with_instrumental(self, mock_execute):
"""Test remuxing with instrumental audio."""
mock_execute.return_value = True
service = LocalEncodingService()
result = service.remux_with_instrumental(
"/input/video.mov",
"/input/audio.flac",
"/output/video.mp4"
)
assert result is True
mock_execute.assert_called_once()
call_args = mock_execute.call_args[0][0]
assert "/input/video.mov" in call_args
assert "/input/audio.flac" in call_args
assert "-map 0:v -map 1:a" in call_args
@patch.object(LocalEncodingService, "_execute_command_with_fallback")
def test_convert_mov_to_mp4(self, mock_execute):
"""Test MOV to MP4 conversion."""
mock_execute.return_value = True
service = LocalEncodingService()
result = service.convert_mov_to_mp4(
"/input/video.mov",
"/output/video.mp4"
)
assert result is True
mock_execute.assert_called_once()
@patch.object(LocalEncodingService, "_execute_command_with_fallback")
def test_encode_lossless_mp4_without_end(self, mock_execute):
"""Test lossless 4K MP4 encoding without end credits."""
mock_execute.return_value = True
service = LocalEncodingService()
result = service.encode_lossless_mp4(
"/input/title.mov",
"/input/karaoke.mp4",
"/output/lossless.mp4"
)
assert result is True
mock_execute.assert_called_once()
call_args = mock_execute.call_args
assert "concat=n=2" in call_args[0][0] # GPU command
@patch.object(LocalEncodingService, "_execute_command_with_fallback")
def test_encode_lossless_mp4_with_end(self, mock_execute):
"""Test lossless 4K MP4 encoding with end credits."""
mock_execute.return_value = True
with tempfile.TemporaryDirectory() as tmpdir:
end_file = os.path.join(tmpdir, "end.mov")
with open(end_file, "w") as f:
f.write("fake video")
service = LocalEncodingService()
result = service.encode_lossless_mp4(
"/input/title.mov",
"/input/karaoke.mp4",
"/output/lossless.mp4",
end_video=end_file
)
assert result is True
call_args = mock_execute.call_args
assert "concat=n=3" in call_args[0][0] # 3 videos
@patch.object(LocalEncodingService, "_execute_command")
def test_encode_lossy_mp4(self, mock_execute):
"""Test lossy 4K MP4 encoding."""
mock_execute.return_value = True
service = LocalEncodingService()
result = service.encode_lossy_mp4(
"/input/lossless.mp4",
"/output/lossy.mp4"
)
assert result is True
call_args = mock_execute.call_args[0][0]
assert "-c:v copy" in call_args
assert "aac" in call_args.lower()
@patch.object(LocalEncodingService, "_execute_command")
def test_encode_lossless_mkv(self, mock_execute):
"""Test MKV encoding with FLAC audio."""
mock_execute.return_value = True
service = LocalEncodingService()
result = service.encode_lossless_mkv(
"/input/lossless.mp4",
"/output/video.mkv"
)
assert result is True
call_args = mock_execute.call_args[0][0]
assert "-c:v copy" in call_args
assert "-c:a flac" in call_args
@patch.object(LocalEncodingService, "_execute_command_with_fallback")
def test_encode_720p(self, mock_execute):
"""Test 720p encoding."""
mock_execute.return_value = True
service = LocalEncodingService()
result = service.encode_720p(
"/input/lossless.mp4",
"/output/720p.mp4"
)
assert result is True
call_args = mock_execute.call_args
# Should contain scale filter
assert "1280:720" in call_args[0][0] or "1280:720" in call_args[0][1]
class TestLocalEncodingServiceFullPipeline:
"""Test full encoding pipeline."""
@patch.object(LocalEncodingService, "remux_with_instrumental")
@patch.object(LocalEncodingService, "convert_mov_to_mp4")
@patch.object(LocalEncodingService, "encode_lossless_mp4")
@patch.object(LocalEncodingService, "encode_lossy_mp4")
@patch.object(LocalEncodingService, "encode_lossless_mkv")
@patch.object(LocalEncodingService, "encode_720p")
def test_encode_all_formats_success(
self, mock_720p, mock_mkv, mock_lossy, mock_lossless, mock_convert, mock_remux
):
"""Test successful full encoding pipeline."""
mock_remux.return_value = True
mock_convert.return_value = True
mock_lossless.return_value = True
mock_lossy.return_value = True
mock_mkv.return_value = True
mock_720p.return_value = True
service = LocalEncodingService()
config = EncodingConfig(
title_video="/input/title.mov",
karaoke_video="/input/karaoke.mov",
instrumental_audio="/input/instrumental.flac",
output_karaoke_mp4="/output/karaoke.mp4",
output_with_vocals_mp4="/output/with_vocals.mp4",
output_lossless_4k_mp4="/output/lossless_4k.mp4",
output_lossy_4k_mp4="/output/lossy_4k.mp4",
output_lossless_mkv="/output/lossless.mkv",
output_720p_mp4="/output/720p.mp4",
)
result = service.encode_all_formats(config)
assert result.success is True
assert "karaoke_mp4" in result.output_files
assert "lossless_4k_mp4" in result.output_files
assert "720p_mp4" in result.output_files
@patch.object(LocalEncodingService, "remux_with_instrumental")
def test_encode_all_formats_failure_early(self, mock_remux):
"""Test encoding pipeline failure at early step."""
mock_remux.return_value = False
service = LocalEncodingService()
config = EncodingConfig(
title_video="/input/title.mov",
karaoke_video="/input/karaoke.mov",
instrumental_audio="/input/instrumental.flac",
output_karaoke_mp4="/output/karaoke.mp4",
output_lossless_4k_mp4="/output/lossless_4k.mp4",
)
result = service.encode_all_formats(config)
assert result.success is False
assert "Failed to remux" in result.error
def test_encode_all_formats_dry_run(self):
"""Test encoding pipeline in dry run mode."""
service = LocalEncodingService(dry_run=True)
config = EncodingConfig(
title_video="/input/title.mov",
karaoke_video="/input/karaoke.mp4", # Already MP4
instrumental_audio="/input/instrumental.flac",
output_karaoke_mp4="/output/karaoke.mp4",
output_lossless_4k_mp4="/output/lossless_4k.mp4",
output_lossy_4k_mp4="/output/lossy_4k.mp4",
output_lossless_mkv="/output/lossless.mkv",
output_720p_mp4="/output/720p.mp4",
)
result = service.encode_all_formats(config)
# In dry run mode, all operations should "succeed"
assert result.success is True
class TestEncodingConfig:
"""Test EncodingConfig dataclass."""
def test_config_required_fields(self):
"""Test that required fields must be provided."""
config = EncodingConfig(
title_video="/path/title.mov",
karaoke_video="/path/karaoke.mov",
instrumental_audio="/path/audio.flac",
)
assert config.title_video == "/path/title.mov"
assert config.end_video is None # Optional field
def test_config_all_fields(self):
"""Test config with all fields."""
config = EncodingConfig(
title_video="/path/title.mov",
karaoke_video="/path/karaoke.mov",
instrumental_audio="/path/audio.flac",
end_video="/path/end.mov",
output_karaoke_mp4="/output/karaoke.mp4",
output_720p_mp4="/output/720p.mp4",
)
assert config.end_video == "/path/end.mov"
assert config.output_karaoke_mp4 == "/output/karaoke.mp4"
class TestEncodingResult:
"""Test EncodingResult dataclass."""
def test_result_success(self):
"""Test successful result."""
result = EncodingResult(
success=True,
output_files={"key": "/path/file.mp4"}
)
assert result.success is True
assert result.error is None
def test_result_failure(self):
"""Test failure result."""
result = EncodingResult(
success=False,
output_files={},
error="Something went wrong"
)
assert result.success is False
assert result.error == "Something went wrong"
class TestGetLocalEncodingService:
"""Test factory function."""
def test_get_service_creates_instance(self):
"""Test that factory function creates a new instance."""
import backend.services.local_encoding_service as module
module._local_encoding_service = None
service = get_local_encoding_service()
assert service is not None
assert isinstance(service, LocalEncodingService)
def test_get_service_with_dry_run(self):
"""Test factory function with dry run option."""
import backend.services.local_encoding_service as module
module._local_encoding_service = None
service = get_local_encoding_service(dry_run=True)
assert service.dry_run is True
+382
View File
@@ -0,0 +1,382 @@
"""
Tests for PackagingService.
Tests cover:
- Service initialization
- CDG package creation
- TXT package creation
- Dry run mode
- Error handling
"""
import os
import zipfile
import tempfile
import pytest
from unittest.mock import MagicMock, patch
from backend.services.packaging_service import (
PackagingService,
get_packaging_service,
)
class TestPackagingServiceInit:
"""Test service initialization."""
def test_init_default_values(self):
"""Test default initialization."""
service = PackagingService()
assert service.cdg_styles is None
assert service.dry_run is False
assert service.non_interactive is False
def test_init_with_cdg_styles(self):
"""Test initialization with CDG styles."""
styles = {"font_size": 24, "bg_color": "black"}
service = PackagingService(cdg_styles=styles)
assert service.cdg_styles == styles
def test_init_with_dry_run(self):
"""Test initialization with dry run mode."""
service = PackagingService(dry_run=True)
assert service.dry_run is True
def test_init_with_non_interactive(self):
"""Test initialization with non-interactive mode."""
service = PackagingService(non_interactive=True)
assert service.non_interactive is True
class TestPackagingServiceCreateZipFromFiles:
"""Test internal ZIP creation."""
def test_create_zip_from_files(self):
"""Test creating a ZIP file from multiple files."""
with tempfile.TemporaryDirectory() as tmpdir:
# Create test files
file1 = os.path.join(tmpdir, "test1.txt")
file2 = os.path.join(tmpdir, "test2.txt")
zip_path = os.path.join(tmpdir, "output.zip")
with open(file1, "w") as f:
f.write("content1")
with open(file2, "w") as f:
f.write("content2")
service = PackagingService()
service._create_zip_from_files(
zip_path,
[(file1, "test1.txt"), (file2, "test2.txt")]
)
assert os.path.isfile(zip_path)
with zipfile.ZipFile(zip_path, "r") as zf:
names = zf.namelist()
assert "test1.txt" in names
assert "test2.txt" in names
class TestPackagingServiceCDGPackage:
"""Test CDG package creation."""
def test_create_cdg_package_missing_lrc_file(self):
"""Test that missing LRC file raises FileNotFoundError."""
service = PackagingService()
with pytest.raises(FileNotFoundError) as exc_info:
service.create_cdg_package(
lrc_file="/nonexistent/file.lrc",
audio_file="/some/audio.flac",
output_zip_path="/output/test.zip",
artist="Test Artist",
title="Test Title",
)
assert "LRC file not found" in str(exc_info.value)
def test_create_cdg_package_missing_audio_file(self):
"""Test that missing audio file raises FileNotFoundError."""
with tempfile.TemporaryDirectory() as tmpdir:
lrc_file = os.path.join(tmpdir, "test.lrc")
with open(lrc_file, "w") as f:
f.write("[00:00.00]Test lyrics")
service = PackagingService()
with pytest.raises(FileNotFoundError) as exc_info:
service.create_cdg_package(
lrc_file=lrc_file,
audio_file="/nonexistent/audio.flac",
output_zip_path="/output/test.zip",
artist="Test Artist",
title="Test Title",
)
assert "Audio file not found" in str(exc_info.value)
def test_create_cdg_package_no_styles_raises(self):
"""Test that CDG generation without styles raises ValueError."""
with tempfile.TemporaryDirectory() as tmpdir:
lrc_file = os.path.join(tmpdir, "test.lrc")
audio_file = os.path.join(tmpdir, "test.flac")
zip_path = os.path.join(tmpdir, "output.zip")
with open(lrc_file, "w") as f:
f.write("[00:00.00]Test lyrics")
with open(audio_file, "w") as f:
f.write("fake audio")
# Service without CDG styles
service = PackagingService(cdg_styles=None)
with pytest.raises(ValueError) as exc_info:
service.create_cdg_package(
lrc_file=lrc_file,
audio_file=audio_file,
output_zip_path=zip_path,
artist="Test Artist",
title="Test Title",
)
assert "CDG styles configuration is required" in str(exc_info.value)
def test_create_cdg_package_dry_run(self):
"""Test CDG package creation in dry run mode."""
with tempfile.TemporaryDirectory() as tmpdir:
lrc_file = os.path.join(tmpdir, "test.lrc")
audio_file = os.path.join(tmpdir, "test.flac")
zip_path = os.path.join(tmpdir, "output.zip")
with open(lrc_file, "w") as f:
f.write("[00:00.00]Test lyrics")
with open(audio_file, "w") as f:
f.write("fake audio")
service = PackagingService(dry_run=True)
result = service.create_cdg_package(
lrc_file=lrc_file,
audio_file=audio_file,
output_zip_path=zip_path,
artist="Test Artist",
title="Test Title",
)
# In dry run, ZIP should not be created
assert not os.path.isfile(zip_path)
assert result[0] == zip_path
def test_create_cdg_package_existing_files(self):
"""Test CDG package creation with existing MP3 and CDG files."""
with tempfile.TemporaryDirectory() as tmpdir:
lrc_file = os.path.join(tmpdir, "test.lrc")
audio_file = os.path.join(tmpdir, "test.flac")
mp3_file = os.path.join(tmpdir, "test.mp3")
cdg_file = os.path.join(tmpdir, "test.cdg")
zip_path = os.path.join(tmpdir, "output.zip")
# Create all files
with open(lrc_file, "w") as f:
f.write("[00:00.00]Test lyrics")
with open(audio_file, "w") as f:
f.write("fake audio")
with open(mp3_file, "w") as f:
f.write("fake mp3")
with open(cdg_file, "w") as f:
f.write("fake cdg")
service = PackagingService()
result = service.create_cdg_package(
lrc_file=lrc_file,
audio_file=audio_file,
output_zip_path=zip_path,
artist="Test Artist",
title="Test Title",
output_mp3_path=mp3_file,
output_cdg_path=cdg_file,
)
# ZIP should be created from existing files
assert os.path.isfile(zip_path)
assert result[0] == zip_path
assert result[1] == mp3_file
assert result[2] == cdg_file
@patch("lyrics_transcriber.output.cdg.CDGGenerator")
def test_create_cdg_package_with_generator(self, mock_cdg_generator_class):
"""Test CDG package creation using CDGGenerator."""
with tempfile.TemporaryDirectory() as tmpdir:
lrc_file = os.path.join(tmpdir, "test.lrc")
audio_file = os.path.join(tmpdir, "test.flac")
zip_path = os.path.join(tmpdir, "output.zip")
generated_zip = os.path.join(tmpdir, "generated.zip")
with open(lrc_file, "w") as f:
f.write("[00:00.00]Test lyrics")
with open(audio_file, "w") as f:
f.write("fake audio")
# Create the zip file that would be generated
with zipfile.ZipFile(generated_zip, "w") as zf:
zf.writestr("test.mp3", "fake mp3")
zf.writestr("test.cdg", "fake cdg")
mock_generator = MagicMock()
mock_generator.generate_cdg_from_lrc.return_value = (
"test.cdg", "test.mp3", generated_zip
)
mock_cdg_generator_class.return_value = mock_generator
styles = {"font_size": 24}
service = PackagingService(cdg_styles=styles)
result = service.create_cdg_package(
lrc_file=lrc_file,
audio_file=audio_file,
output_zip_path=zip_path,
artist="Test Artist",
title="Test Title",
)
mock_generator.generate_cdg_from_lrc.assert_called_once_with(
lrc_file=lrc_file,
audio_file=audio_file,
title="Test Title",
artist="Test Artist",
cdg_styles=styles,
)
assert os.path.isfile(zip_path)
class TestPackagingServiceTXTPackage:
"""Test TXT package creation."""
def test_create_txt_package_missing_lrc_file(self):
"""Test that missing LRC file raises FileNotFoundError."""
service = PackagingService()
with pytest.raises(FileNotFoundError) as exc_info:
service.create_txt_package(
lrc_file="/nonexistent/file.lrc",
mp3_file="/some/audio.mp3",
output_zip_path="/output/test.zip",
)
assert "LRC file not found" in str(exc_info.value)
def test_create_txt_package_missing_mp3_file(self):
"""Test that missing MP3 file raises FileNotFoundError."""
with tempfile.TemporaryDirectory() as tmpdir:
lrc_file = os.path.join(tmpdir, "test.lrc")
with open(lrc_file, "w") as f:
f.write("[00:00.00]Test lyrics")
service = PackagingService()
with pytest.raises(FileNotFoundError) as exc_info:
service.create_txt_package(
lrc_file=lrc_file,
mp3_file="/nonexistent/audio.mp3",
output_zip_path="/output/test.zip",
)
assert "MP3 file not found" in str(exc_info.value)
def test_create_txt_package_dry_run(self):
"""Test TXT package creation in dry run mode."""
with tempfile.TemporaryDirectory() as tmpdir:
lrc_file = os.path.join(tmpdir, "test.lrc")
mp3_file = os.path.join(tmpdir, "test.mp3")
zip_path = os.path.join(tmpdir, "output.zip")
with open(lrc_file, "w") as f:
f.write("[00:00.00]Test lyrics")
with open(mp3_file, "w") as f:
f.write("fake mp3")
service = PackagingService(dry_run=True)
result = service.create_txt_package(
lrc_file=lrc_file,
mp3_file=mp3_file,
output_zip_path=zip_path,
)
# In dry run, ZIP should not be created
assert not os.path.isfile(zip_path)
assert result[0] == zip_path
@patch("lyrics_converter.LyricsConverter")
def test_create_txt_package_success(self, mock_converter_class):
"""Test successful TXT package creation."""
with tempfile.TemporaryDirectory() as tmpdir:
lrc_file = os.path.join(tmpdir, "test.lrc")
mp3_file = os.path.join(tmpdir, "test.mp3")
zip_path = os.path.join(tmpdir, "output.zip")
txt_path = os.path.join(tmpdir, "output.txt")
with open(lrc_file, "w") as f:
f.write("[00:00.00]Test lyrics")
with open(mp3_file, "w") as f:
f.write("fake mp3")
mock_converter = MagicMock()
mock_converter.convert_file.return_value = "Converted lyrics text"
mock_converter_class.return_value = mock_converter
service = PackagingService()
result = service.create_txt_package(
lrc_file=lrc_file,
mp3_file=mp3_file,
output_zip_path=zip_path,
output_txt_path=txt_path,
)
mock_converter_class.assert_called_once_with(
output_format="txt",
filepath=lrc_file
)
mock_converter.convert_file.assert_called_once()
assert os.path.isfile(zip_path)
assert os.path.isfile(txt_path)
assert result[0] == zip_path
assert result[1] == txt_path
# Verify ZIP contents
with zipfile.ZipFile(zip_path, "r") as zf:
names = zf.namelist()
assert "test.mp3" in names
assert "output.txt" in names
class TestGetPackagingService:
"""Test factory function."""
def test_get_service_creates_instance(self):
"""Test that factory function creates a new instance."""
import backend.services.packaging_service as module
module._packaging_service = None
service = get_packaging_service()
assert service is not None
assert isinstance(service, PackagingService)
def test_get_service_with_cdg_styles(self):
"""Test factory function with CDG styles."""
import backend.services.packaging_service as module
module._packaging_service = None
styles = {"font_size": 24}
service = get_packaging_service(cdg_styles=styles)
assert service.cdg_styles == styles
def test_get_service_with_options(self):
"""Test factory function with additional options."""
import backend.services.packaging_service as module
module._packaging_service = None
service = get_packaging_service(dry_run=True, non_interactive=True)
assert service.dry_run is True
assert service.non_interactive is True
+3 -3
View File
@@ -99,7 +99,7 @@ class TestStylesConfigRequirements:
"karaoke": {
"background_color": "#000000",
"font_path": "", # MUST be string, NOT None
"font": "Arial",
"font": "Noto Sans",
"ass_name": "Default", # REQUIRED
"primary_color": "112, 112, 247, 255",
"secondary_color": "255, 255, 255, 255",
@@ -241,7 +241,7 @@ class TestPreviewStyleLoading:
"background_image": "/original/path/background.png",
"font_path": "/original/path/font.ttf",
"background_color": "#000000",
"font": "Arial",
"font": "Noto Sans",
"ass_name": "Default",
}
}
@@ -308,7 +308,7 @@ class TestPreviewStyleLoading:
# Should have karaoke section with minimal/default values
assert "karaoke" in result_styles
assert result_styles["karaoke"]["background_color"] == "#000000"
assert result_styles["karaoke"]["font"] == "Arial"
assert result_styles["karaoke"]["font"] == "Noto Sans"
# Minimal styles have background_image as None (default)
assert result_styles["karaoke"].get("background_image") is None
@@ -0,0 +1,742 @@
"""
Tests for Video Worker Orchestrator.
Tests cover:
- OrchestratorConfig and OrchestratorResult dataclasses
- VideoWorkerOrchestrator initialization
- Individual stage methods
- Full pipeline execution
- Error handling and recovery
"""
import os
import tempfile
import pytest
from unittest.mock import MagicMock, patch, AsyncMock
from dataclasses import asdict
from backend.workers.video_worker_orchestrator import (
OrchestratorConfig,
OrchestratorResult,
VideoWorkerOrchestrator,
create_orchestrator_config_from_job,
)
class TestOrchestratorConfig:
"""Test OrchestratorConfig dataclass."""
def test_required_fields(self):
"""Test that required fields must be provided."""
config = OrchestratorConfig(
job_id="test-job",
artist="Test Artist",
title="Test Title",
title_video_path="/path/title.mov",
karaoke_video_path="/path/karaoke.mov",
instrumental_audio_path="/path/audio.flac",
)
assert config.job_id == "test-job"
assert config.artist == "Test Artist"
assert config.encoding_backend == "auto"
def test_default_values(self):
"""Test default values for optional fields."""
config = OrchestratorConfig(
job_id="test-job",
artist="Test Artist",
title="Test Title",
title_video_path="/path/title.mov",
karaoke_video_path="/path/karaoke.mov",
instrumental_audio_path="/path/audio.flac",
)
assert config.enable_cdg is False
assert config.enable_txt is False
assert config.enable_youtube_upload is False
assert config.dry_run is False
assert config.non_interactive is True
assert config.end_video_path is None
def test_all_fields(self):
"""Test all fields including optional ones."""
config = OrchestratorConfig(
job_id="test-job",
artist="Test Artist",
title="Test Title",
title_video_path="/path/title.mov",
karaoke_video_path="/path/karaoke.mov",
instrumental_audio_path="/path/audio.flac",
end_video_path="/path/end.mov",
enable_cdg=True,
enable_txt=True,
enable_youtube_upload=True,
brand_prefix="TEST",
discord_webhook_url="https://discord.com/api/webhooks/123/abc",
encoding_backend="gce",
)
assert config.enable_cdg is True
assert config.encoding_backend == "gce"
assert config.discord_webhook_url is not None
class TestOrchestratorResult:
"""Test OrchestratorResult dataclass."""
def test_success_result(self):
"""Test successful result."""
result = OrchestratorResult(
success=True,
final_video="/output/video.mp4",
brand_code="TEST-1234",
youtube_url="https://youtube.com/watch?v=test",
)
assert result.success is True
assert result.error_message is None
assert result.brand_code == "TEST-1234"
def test_failure_result(self):
"""Test failure result."""
result = OrchestratorResult(
success=False,
error_message="Encoding failed",
)
assert result.success is False
assert result.error_message == "Encoding failed"
assert result.final_video is None
def test_all_output_files(self):
"""Test all output file fields."""
result = OrchestratorResult(
success=True,
final_video="/output/lossless.mp4",
final_video_mkv="/output/lossless.mkv",
final_video_lossy="/output/lossy.mp4",
final_video_720p="/output/720p.mp4",
final_karaoke_cdg_zip="/output/cdg.zip",
final_karaoke_txt_zip="/output/txt.zip",
)
assert result.final_video == "/output/lossless.mp4"
assert result.final_video_mkv == "/output/lossless.mkv"
class TestVideoWorkerOrchestratorInit:
"""Test VideoWorkerOrchestrator initialization."""
def test_init_with_config(self):
"""Test initialization with config."""
config = OrchestratorConfig(
job_id="test-job",
artist="Test Artist",
title="Test Title",
title_video_path="/path/title.mov",
karaoke_video_path="/path/karaoke.mov",
instrumental_audio_path="/path/audio.flac",
)
orchestrator = VideoWorkerOrchestrator(config)
assert orchestrator.config == config
assert orchestrator.result.success is False
def test_init_with_job_manager(self):
"""Test initialization with job manager."""
config = OrchestratorConfig(
job_id="test-job",
artist="Test Artist",
title="Test Title",
title_video_path="/path/title.mov",
karaoke_video_path="/path/karaoke.mov",
instrumental_audio_path="/path/audio.flac",
)
job_manager = MagicMock()
orchestrator = VideoWorkerOrchestrator(config, job_manager=job_manager)
assert orchestrator.job_manager == job_manager
class TestVideoWorkerOrchestratorServices:
"""Test service lazy-loading."""
def test_get_encoding_backend(self):
"""Test encoding backend lazy loading."""
config = OrchestratorConfig(
job_id="test-job",
artist="Test Artist",
title="Test Title",
title_video_path="/path/title.mov",
karaoke_video_path="/path/karaoke.mov",
instrumental_audio_path="/path/audio.flac",
encoding_backend="local",
)
orchestrator = VideoWorkerOrchestrator(config)
with patch("backend.services.encoding_interface.get_encoding_backend") as mock_get:
mock_backend = MagicMock()
mock_backend.name = "local"
mock_get.return_value = mock_backend
backend = orchestrator._get_encoding_backend()
assert backend == mock_backend
mock_get.assert_called_once()
def test_get_packaging_service(self):
"""Test packaging service lazy loading."""
config = OrchestratorConfig(
job_id="test-job",
artist="Test Artist",
title="Test Title",
title_video_path="/path/title.mov",
karaoke_video_path="/path/karaoke.mov",
instrumental_audio_path="/path/audio.flac",
cdg_styles={"background_color": "black"},
)
orchestrator = VideoWorkerOrchestrator(config)
with patch("backend.services.packaging_service.PackagingService") as MockService:
mock_service = MagicMock()
MockService.return_value = mock_service
service = orchestrator._get_packaging_service()
assert service == mock_service
MockService.assert_called_once()
def test_get_youtube_service(self):
"""Test YouTube service lazy loading."""
config = OrchestratorConfig(
job_id="test-job",
artist="Test Artist",
title="Test Title",
title_video_path="/path/title.mov",
karaoke_video_path="/path/karaoke.mov",
instrumental_audio_path="/path/audio.flac",
youtube_credentials={"token": "test"},
)
orchestrator = VideoWorkerOrchestrator(config)
with patch("backend.services.youtube_upload_service.YouTubeUploadService") as MockService:
mock_service = MagicMock()
MockService.return_value = mock_service
service = orchestrator._get_youtube_service()
assert service == mock_service
MockService.assert_called_once()
def test_get_discord_service(self):
"""Test Discord service lazy loading."""
config = OrchestratorConfig(
job_id="test-job",
artist="Test Artist",
title="Test Title",
title_video_path="/path/title.mov",
karaoke_video_path="/path/karaoke.mov",
instrumental_audio_path="/path/audio.flac",
discord_webhook_url="https://discord.com/api/webhooks/123/abc",
)
orchestrator = VideoWorkerOrchestrator(config)
with patch("backend.services.discord_service.DiscordNotificationService") as MockService:
mock_service = MagicMock()
MockService.return_value = mock_service
service = orchestrator._get_discord_service()
assert service == mock_service
MockService.assert_called_once()
class TestVideoWorkerOrchestratorPackaging:
"""Test packaging stage."""
@pytest.mark.asyncio
async def test_run_packaging_no_lrc(self):
"""Test packaging stage with no LRC file."""
config = OrchestratorConfig(
job_id="test-job",
artist="Test Artist",
title="Test Title",
title_video_path="/path/title.mov",
karaoke_video_path="/path/karaoke.mov",
instrumental_audio_path="/path/audio.flac",
enable_cdg=True,
lrc_file_path=None,
)
orchestrator = VideoWorkerOrchestrator(config)
# Should not raise, just skip
await orchestrator._run_packaging()
assert orchestrator.result.final_karaoke_cdg_zip is None
@pytest.mark.asyncio
async def test_run_packaging_cdg(self):
"""Test CDG packaging."""
with tempfile.TemporaryDirectory() as temp_dir:
lrc_file = os.path.join(temp_dir, "test.lrc")
audio_file = os.path.join(temp_dir, "test.flac")
# Create dummy files
with open(lrc_file, "w") as f:
f.write("[00:00.00]Test lyrics")
with open(audio_file, "w") as f:
f.write("dummy audio")
config = OrchestratorConfig(
job_id="test-job",
artist="Test Artist",
title="Test Title",
title_video_path="/path/title.mov",
karaoke_video_path="/path/karaoke.mov",
instrumental_audio_path=audio_file,
lrc_file_path=lrc_file,
output_dir=temp_dir,
enable_cdg=True,
cdg_styles={"background_color": "black"},
)
orchestrator = VideoWorkerOrchestrator(config)
with patch.object(orchestrator, "_get_packaging_service") as mock_get:
mock_service = MagicMock()
mock_service.create_cdg_package.return_value = (
f"{temp_dir}/cdg.zip",
f"{temp_dir}/test.mp3",
f"{temp_dir}/test.cdg",
)
mock_get.return_value = mock_service
await orchestrator._run_packaging()
mock_service.create_cdg_package.assert_called_once()
assert orchestrator.result.final_karaoke_cdg_zip == f"{temp_dir}/cdg.zip"
class TestVideoWorkerOrchestratorEncoding:
"""Test encoding stage."""
@pytest.mark.asyncio
async def test_run_encoding_success(self):
"""Test successful encoding."""
config = OrchestratorConfig(
job_id="test-job",
artist="Test Artist",
title="Test Title",
title_video_path="/path/title.mov",
karaoke_video_path="/path/karaoke.mov",
instrumental_audio_path="/path/audio.flac",
output_dir="/output",
)
orchestrator = VideoWorkerOrchestrator(config)
with patch.object(orchestrator, "_get_encoding_backend") as mock_get:
from backend.services.encoding_interface import EncodingOutput
mock_backend = MagicMock()
mock_backend.name = "local"
mock_backend.encode = AsyncMock(return_value=EncodingOutput(
success=True,
lossless_4k_mp4_path="/output/lossless.mp4",
lossy_4k_mp4_path="/output/lossy.mp4",
lossy_720p_mp4_path="/output/720p.mp4",
lossless_mkv_path="/output/lossless.mkv",
encoding_time_seconds=120.5,
encoding_backend="local",
))
mock_get.return_value = mock_backend
await orchestrator._run_encoding()
mock_backend.encode.assert_called_once()
assert orchestrator.result.final_video == "/output/lossless.mp4"
assert orchestrator.result.final_video_720p == "/output/720p.mp4"
assert orchestrator.result.encoding_time_seconds == 120.5
def test_encoding_input_gcs_paths_pattern(self):
"""Test that EncodingInput.options contains proper GCS paths structure.
This test verifies the GCS path construction pattern that GCE encoding
requires. It tests the structure without running _run_encoding().
This would have caught: 'GCE encoding requires input_gcs_path and
output_gcs_path in options' error when the orchestrator didn't pass
the required paths for GCE encoding.
"""
from backend.services.encoding_interface import EncodingInput
# Test that EncodingInput can hold GCS paths in options
# This mirrors what the orchestrator should build
job_id = "test-job-123"
bucket = "test-bucket"
input_gcs_path = f"gs://{bucket}/jobs/{job_id}/"
output_gcs_path = f"gs://{bucket}/jobs/{job_id}/finals/"
encoding_input = EncodingInput(
title_video_path="/path/title.mov",
karaoke_video_path="/path/karaoke.mov",
instrumental_audio_path="/path/audio.flac",
artist="Test Artist",
title="Test Title",
brand_code="TEST-001",
output_dir="/output",
options={
"input_gcs_path": input_gcs_path,
"output_gcs_path": output_gcs_path,
},
)
# Verify the structure that GCEEncodingBackend expects
assert "input_gcs_path" in encoding_input.options, \
"EncodingInput.options must include input_gcs_path for GCE encoding"
assert "output_gcs_path" in encoding_input.options, \
"EncodingInput.options must include output_gcs_path for GCE encoding"
# Verify path format
assert encoding_input.options["input_gcs_path"].startswith("gs://")
assert encoding_input.options["output_gcs_path"].startswith("gs://")
assert job_id in encoding_input.options["input_gcs_path"]
assert job_id in encoding_input.options["output_gcs_path"]
@pytest.mark.asyncio
async def test_run_encoding_failure(self):
"""Test encoding failure."""
config = OrchestratorConfig(
job_id="test-job",
artist="Test Artist",
title="Test Title",
title_video_path="/path/title.mov",
karaoke_video_path="/path/karaoke.mov",
instrumental_audio_path="/path/audio.flac",
)
orchestrator = VideoWorkerOrchestrator(config)
with patch.object(orchestrator, "_get_encoding_backend") as mock_get:
from backend.services.encoding_interface import EncodingOutput
mock_backend = MagicMock()
mock_backend.name = "local"
mock_backend.encode = AsyncMock(return_value=EncodingOutput(
success=False,
error_message="FFmpeg failed",
encoding_backend="local",
))
mock_get.return_value = mock_backend
with pytest.raises(Exception) as exc_info:
await orchestrator._run_encoding()
assert "Encoding failed" in str(exc_info.value)
class TestVideoWorkerOrchestratorOrganization:
"""Test organization stage."""
@pytest.mark.asyncio
async def test_run_organization_keep_brand_code(self):
"""Test organization with existing brand code."""
config = OrchestratorConfig(
job_id="test-job",
artist="Test Artist",
title="Test Title",
title_video_path="/path/title.mov",
karaoke_video_path="/path/karaoke.mov",
instrumental_audio_path="/path/audio.flac",
keep_brand_code="NOMAD-9999",
)
orchestrator = VideoWorkerOrchestrator(config)
await orchestrator._run_organization()
assert orchestrator.result.brand_code == "NOMAD-9999"
@pytest.mark.asyncio
async def test_run_organization_generate_brand_code(self):
"""Test brand code generation from Dropbox."""
config = OrchestratorConfig(
job_id="test-job",
artist="Test Artist",
title="Test Title",
title_video_path="/path/title.mov",
karaoke_video_path="/path/karaoke.mov",
instrumental_audio_path="/path/audio.flac",
dropbox_path="/Karaoke/Tracks",
brand_prefix="TEST",
)
orchestrator = VideoWorkerOrchestrator(config)
with patch("backend.services.dropbox_service.get_dropbox_service") as mock_get:
mock_dropbox = MagicMock()
mock_dropbox.is_configured = True
mock_dropbox.get_next_brand_code.return_value = "TEST-0001"
mock_get.return_value = mock_dropbox
await orchestrator._run_organization()
assert orchestrator.result.brand_code == "TEST-0001"
class TestVideoWorkerOrchestratorDistribution:
"""Test distribution stage."""
@pytest.mark.asyncio
async def test_upload_to_youtube(self):
"""Test YouTube upload."""
with tempfile.TemporaryDirectory() as temp_dir:
video_file = os.path.join(temp_dir, "test.mp4")
with open(video_file, "w") as f:
f.write("dummy video")
config = OrchestratorConfig(
job_id="test-job",
artist="Test Artist",
title="Test Title",
title_video_path="/path/title.mov",
karaoke_video_path="/path/karaoke.mov",
instrumental_audio_path="/path/audio.flac",
enable_youtube_upload=True,
youtube_credentials={"token": "test"},
)
orchestrator = VideoWorkerOrchestrator(config)
orchestrator.result.final_video_lossy = video_file
with patch.object(orchestrator, "_get_youtube_service") as mock_get:
mock_service = MagicMock()
mock_service.upload_video.return_value = (
"video123",
"https://youtube.com/watch?v=video123"
)
mock_get.return_value = mock_service
await orchestrator._upload_to_youtube()
mock_service.upload_video.assert_called_once()
assert orchestrator.result.youtube_url == "https://youtube.com/watch?v=video123"
@pytest.mark.asyncio
async def test_upload_to_dropbox(self):
"""Test Dropbox upload."""
with tempfile.TemporaryDirectory() as temp_dir:
config = OrchestratorConfig(
job_id="test-job",
artist="Test Artist",
title="Test Title",
title_video_path="/path/title.mov",
karaoke_video_path="/path/karaoke.mov",
instrumental_audio_path="/path/audio.flac",
output_dir=temp_dir,
dropbox_path="/Karaoke/Tracks",
brand_prefix="TEST",
)
orchestrator = VideoWorkerOrchestrator(config)
orchestrator.result.brand_code = "TEST-0001"
with patch("backend.services.dropbox_service.get_dropbox_service") as mock_get:
mock_dropbox = MagicMock()
mock_dropbox.is_configured = True
mock_dropbox.create_shared_link.return_value = "https://dropbox.com/link"
mock_get.return_value = mock_dropbox
await orchestrator._upload_to_dropbox()
mock_dropbox.upload_folder.assert_called_once()
assert orchestrator.result.dropbox_link == "https://dropbox.com/link"
class TestVideoWorkerOrchestratorNotifications:
"""Test notifications stage."""
@pytest.mark.asyncio
async def test_run_notifications_with_youtube_url(self):
"""Test Discord notification with YouTube URL."""
config = OrchestratorConfig(
job_id="test-job",
artist="Test Artist",
title="Test Title",
title_video_path="/path/title.mov",
karaoke_video_path="/path/karaoke.mov",
instrumental_audio_path="/path/audio.flac",
discord_webhook_url="https://discord.com/api/webhooks/123/abc",
)
orchestrator = VideoWorkerOrchestrator(config)
orchestrator.result.youtube_url = "https://youtube.com/watch?v=test"
with patch.object(orchestrator, "_get_discord_service") as mock_get:
mock_service = MagicMock()
mock_service.post_video_notification.return_value = True
mock_get.return_value = mock_service
await orchestrator._run_notifications()
mock_service.post_video_notification.assert_called_once_with(
"https://youtube.com/watch?v=test"
)
@pytest.mark.asyncio
async def test_run_notifications_no_youtube_url(self):
"""Test notification skipped without YouTube URL."""
config = OrchestratorConfig(
job_id="test-job",
artist="Test Artist",
title="Test Title",
title_video_path="/path/title.mov",
karaoke_video_path="/path/karaoke.mov",
instrumental_audio_path="/path/audio.flac",
discord_webhook_url="https://discord.com/api/webhooks/123/abc",
)
orchestrator = VideoWorkerOrchestrator(config)
# No youtube_url set
with patch.object(orchestrator, "_get_discord_service") as mock_get:
mock_service = MagicMock()
mock_get.return_value = mock_service
await orchestrator._run_notifications()
mock_service.post_video_notification.assert_not_called()
class TestVideoWorkerOrchestratorFullPipeline:
"""Test full pipeline execution."""
@pytest.mark.asyncio
async def test_run_full_pipeline_success(self):
"""Test successful full pipeline."""
config = OrchestratorConfig(
job_id="test-job",
artist="Test Artist",
title="Test Title",
title_video_path="/path/title.mov",
karaoke_video_path="/path/karaoke.mov",
instrumental_audio_path="/path/audio.flac",
enable_cdg=True, # Enable to trigger packaging stage
)
orchestrator = VideoWorkerOrchestrator(config)
# Mock all stages
with patch.object(orchestrator, "_run_packaging", new_callable=AsyncMock) as mock_packaging, \
patch.object(orchestrator, "_run_encoding", new_callable=AsyncMock) as mock_encoding, \
patch.object(orchestrator, "_run_organization", new_callable=AsyncMock) as mock_org, \
patch.object(orchestrator, "_run_distribution", new_callable=AsyncMock) as mock_dist, \
patch.object(orchestrator, "_run_notifications", new_callable=AsyncMock) as mock_notify:
result = await orchestrator.run()
assert result.success is True
mock_packaging.assert_called_once()
mock_encoding.assert_called_once()
mock_org.assert_called_once()
mock_dist.assert_called_once()
mock_notify.assert_called_once()
@pytest.mark.asyncio
async def test_run_pipeline_skips_packaging_when_disabled(self):
"""Test pipeline skips packaging when CDG/TXT disabled."""
config = OrchestratorConfig(
job_id="test-job",
artist="Test Artist",
title="Test Title",
title_video_path="/path/title.mov",
karaoke_video_path="/path/karaoke.mov",
instrumental_audio_path="/path/audio.flac",
enable_cdg=False,
enable_txt=False,
)
orchestrator = VideoWorkerOrchestrator(config)
# Mock all stages
with patch.object(orchestrator, "_run_packaging", new_callable=AsyncMock) as mock_packaging, \
patch.object(orchestrator, "_run_encoding", new_callable=AsyncMock) as mock_encoding, \
patch.object(orchestrator, "_run_organization", new_callable=AsyncMock), \
patch.object(orchestrator, "_run_distribution", new_callable=AsyncMock), \
patch.object(orchestrator, "_run_notifications", new_callable=AsyncMock):
result = await orchestrator.run()
assert result.success is True
mock_packaging.assert_not_called()
mock_encoding.assert_called_once()
@pytest.mark.asyncio
async def test_run_full_pipeline_encoding_failure(self):
"""Test pipeline failure during encoding."""
config = OrchestratorConfig(
job_id="test-job",
artist="Test Artist",
title="Test Title",
title_video_path="/path/title.mov",
karaoke_video_path="/path/karaoke.mov",
instrumental_audio_path="/path/audio.flac",
)
orchestrator = VideoWorkerOrchestrator(config)
with patch.object(orchestrator, "_run_packaging", new_callable=AsyncMock), \
patch.object(orchestrator, "_run_encoding", new_callable=AsyncMock) as mock_encoding, \
patch.object(orchestrator, "_run_organization", new_callable=AsyncMock) as mock_org:
mock_encoding.side_effect = Exception("Encoding failed")
result = await orchestrator.run()
assert result.success is False
assert "Encoding failed" in result.error_message
mock_org.assert_not_called()
class TestCreateOrchestratorConfigFromJob:
"""Test helper function for creating config from job."""
def test_create_config_from_job(self):
"""Test creating config from a job object."""
job = MagicMock()
job.job_id = "test-123"
job.artist = "Test Artist"
job.title = "Test Title"
job.state_data = {"instrumental_selection": "clean"}
job.enable_cdg = True
job.enable_txt = False
job.enable_youtube_upload = True
job.brand_prefix = "NOMAD"
job.discord_webhook_url = "https://discord.com/api/webhooks/123/abc"
job.youtube_description_template = "Test description"
job.dropbox_path = "/Karaoke"
job.gdrive_folder_id = None
job.keep_brand_code = None
job.existing_instrumental_gcs_path = None
config = create_orchestrator_config_from_job(
job=job,
temp_dir="/tmp/test",
youtube_credentials={"token": "test"},
cdg_styles={"background": "black"},
)
assert config.job_id == "test-123"
assert config.artist == "Test Artist"
assert config.title == "Test Title"
assert config.enable_cdg is True
assert config.enable_youtube_upload is True
assert config.title_video_path == "/tmp/test/Test Artist - Test Title (Title).mov"
assert config.instrumental_audio_path == "/tmp/test/Test Artist - Test Title (Instrumental Clean).flac"
def test_create_config_from_job_with_existing_instrumental(self):
"""Test config with user-provided instrumental."""
job = MagicMock()
job.job_id = "test-123"
job.artist = "Test Artist"
job.title = "Test Title"
job.state_data = {"instrumental_selection": "custom"}
job.enable_cdg = False
job.enable_txt = False
job.enable_youtube_upload = False
job.brand_prefix = None
job.discord_webhook_url = None
job.youtube_description_template = None
job.dropbox_path = None
job.gdrive_folder_id = None
job.keep_brand_code = None
job.existing_instrumental_gcs_path = "gs://bucket/instrumental.mp3"
config = create_orchestrator_config_from_job(
job=job,
temp_dir="/tmp/test",
)
assert config.instrumental_audio_path == "/tmp/test/Test Artist - Test Title (Instrumental User).mp3"
@@ -0,0 +1,568 @@
"""
Tests for YouTubeUploadService.
Tests cover:
- Authentication with pre-stored credentials
- Authentication with client secrets file
- Duplicate video detection (exact and fuzzy matching)
- Video deletion
- Video upload with metadata and thumbnail
- Title truncation
- Dry run mode
"""
import os
import pytest
from unittest.mock import MagicMock, patch, mock_open
from backend.services.youtube_upload_service import YouTubeUploadService, get_youtube_upload_service
class TestYouTubeUploadServiceInit:
"""Test service initialization."""
def test_init_with_credentials(self):
"""Test initialization with pre-stored credentials."""
creds = {
"token": "test_token",
"refresh_token": "test_refresh",
"token_uri": "https://oauth2.googleapis.com/token",
"client_id": "test_client_id",
"client_secret": "test_secret",
}
service = YouTubeUploadService(
credentials=creds,
non_interactive=True
)
assert service.credentials == creds
assert service.non_interactive is True
def test_init_with_client_secrets(self):
"""Test initialization with client secrets file."""
service = YouTubeUploadService(
client_secrets_file="/path/to/secrets.json"
)
assert service.client_secrets_file == "/path/to/secrets.json"
assert service.non_interactive is False
def test_init_default_values(self):
"""Test default values on initialization."""
service = YouTubeUploadService()
assert service.credentials is None
assert service.client_secrets_file is None
assert service.non_interactive is False
assert service.server_side_mode is False
assert service.dry_run is False
class TestYouTubeUploadServiceAuthentication:
"""Test authentication methods."""
@patch("googleapiclient.discovery.build")
@patch("google.oauth2.credentials.Credentials")
@patch("google.auth.transport.requests.Request")
def test_authenticate_with_prestored_credentials(
self, mock_request, mock_credentials_class, mock_build
):
"""Test authentication using pre-stored credentials."""
# Setup mocks
mock_creds = MagicMock()
mock_creds.expired = False
mock_credentials_class.return_value = mock_creds
mock_youtube = MagicMock()
mock_build.return_value = mock_youtube
creds = {
"token": "test_token",
"refresh_token": "test_refresh",
"token_uri": "https://oauth2.googleapis.com/token",
"client_id": "test_client_id",
"client_secret": "test_secret",
"scopes": ["https://www.googleapis.com/auth/youtube"],
}
service = YouTubeUploadService(credentials=creds, non_interactive=True)
# Access youtube_service to trigger authentication
result = service.youtube_service
assert result == mock_youtube
mock_credentials_class.assert_called_once()
mock_build.assert_called_once_with('youtube', 'v3', credentials=mock_creds)
@patch("googleapiclient.discovery.build")
@patch("google.oauth2.credentials.Credentials")
@patch("google.auth.transport.requests.Request")
def test_authenticate_refreshes_expired_token(
self, mock_request, mock_credentials_class, mock_build
):
"""Test that expired tokens are refreshed."""
mock_creds = MagicMock()
mock_creds.expired = True
mock_creds.refresh_token = "test_refresh"
mock_credentials_class.return_value = mock_creds
creds = {
"token": "expired_token",
"refresh_token": "test_refresh",
"token_uri": "https://oauth2.googleapis.com/token",
"client_id": "test_client_id",
"client_secret": "test_secret",
}
service = YouTubeUploadService(credentials=creds, non_interactive=True)
service.youtube_service
mock_creds.refresh.assert_called_once()
def test_authenticate_non_interactive_no_credentials_raises(self):
"""Test that non-interactive mode without credentials raises error."""
service = YouTubeUploadService(non_interactive=True)
with pytest.raises(Exception) as exc_info:
service._authenticate()
assert "non-interactive mode" in str(exc_info.value).lower()
class TestYouTubeUploadServiceChannelId:
"""Test channel ID retrieval."""
@patch("googleapiclient.discovery.build")
@patch("google.oauth2.credentials.Credentials")
@patch("google.auth.transport.requests.Request")
def test_get_channel_id_success(
self, mock_request, mock_credentials_class, mock_build
):
"""Test successful channel ID retrieval."""
mock_creds = MagicMock()
mock_creds.expired = False
mock_credentials_class.return_value = mock_creds
mock_youtube = MagicMock()
mock_channels = MagicMock()
mock_list = MagicMock()
mock_list.execute.return_value = {
"items": [{"id": "UC123456"}]
}
mock_channels.list.return_value = mock_list
mock_youtube.channels.return_value = mock_channels
mock_build.return_value = mock_youtube
creds = {"token": "test", "refresh_token": "test", "token_uri": "uri",
"client_id": "id", "client_secret": "secret"}
service = YouTubeUploadService(credentials=creds, non_interactive=True)
channel_id = service.get_channel_id()
assert channel_id == "UC123456"
@patch("googleapiclient.discovery.build")
@patch("google.oauth2.credentials.Credentials")
@patch("google.auth.transport.requests.Request")
def test_get_channel_id_no_items(
self, mock_request, mock_credentials_class, mock_build
):
"""Test channel ID returns None when no items found."""
mock_creds = MagicMock()
mock_creds.expired = False
mock_credentials_class.return_value = mock_creds
mock_youtube = MagicMock()
mock_channels = MagicMock()
mock_list = MagicMock()
mock_list.execute.return_value = {"items": []}
mock_channels.list.return_value = mock_list
mock_youtube.channels.return_value = mock_channels
mock_build.return_value = mock_youtube
creds = {"token": "test", "refresh_token": "test", "token_uri": "uri",
"client_id": "id", "client_secret": "secret"}
service = YouTubeUploadService(credentials=creds, non_interactive=True)
channel_id = service.get_channel_id()
assert channel_id is None
class TestYouTubeUploadServiceDuplicateCheck:
"""Test duplicate video detection."""
@patch("googleapiclient.discovery.build")
@patch("google.oauth2.credentials.Credentials")
@patch("google.auth.transport.requests.Request")
def test_check_duplicate_exact_match_found(
self, mock_request, mock_credentials_class, mock_build
):
"""Test duplicate detection with exact match in server mode."""
mock_creds = MagicMock()
mock_creds.expired = False
mock_credentials_class.return_value = mock_creds
mock_youtube = MagicMock()
# Mock channels().list() for get_channel_id
mock_channels = MagicMock()
mock_channels_list = MagicMock()
mock_channels_list.execute.return_value = {"items": [{"id": "UC123456"}]}
mock_channels.list.return_value = mock_channels_list
mock_youtube.channels.return_value = mock_channels
# Mock search().list() for duplicate check
mock_search = MagicMock()
mock_search_list = MagicMock()
mock_search_list.execute.return_value = {
"items": [{
"id": {"videoId": "VIDEO123"},
"snippet": {
"channelId": "UC123456",
"title": "Test Artist - Test Song (Karaoke)"
}
}]
}
mock_search.list.return_value = mock_search_list
mock_youtube.search.return_value = mock_search
mock_build.return_value = mock_youtube
creds = {"token": "test", "refresh_token": "test", "token_uri": "uri",
"client_id": "id", "client_secret": "secret"}
service = YouTubeUploadService(
credentials=creds,
non_interactive=True,
server_side_mode=True
)
exists, video_id, video_url = service.check_duplicate(
"Test Artist - Test Song (Karaoke)"
)
assert exists is True
assert video_id == "VIDEO123"
assert "VIDEO123" in video_url
@patch("googleapiclient.discovery.build")
@patch("google.oauth2.credentials.Credentials")
@patch("google.auth.transport.requests.Request")
def test_check_duplicate_no_match(
self, mock_request, mock_credentials_class, mock_build
):
"""Test duplicate detection when no match found."""
mock_creds = MagicMock()
mock_creds.expired = False
mock_credentials_class.return_value = mock_creds
mock_youtube = MagicMock()
mock_channels = MagicMock()
mock_channels_list = MagicMock()
mock_channels_list.execute.return_value = {"items": [{"id": "UC123456"}]}
mock_channels.list.return_value = mock_channels_list
mock_youtube.channels.return_value = mock_channels
mock_search = MagicMock()
mock_search_list = MagicMock()
mock_search_list.execute.return_value = {"items": []}
mock_search.list.return_value = mock_search_list
mock_youtube.search.return_value = mock_search
mock_build.return_value = mock_youtube
creds = {"token": "test", "refresh_token": "test", "token_uri": "uri",
"client_id": "id", "client_secret": "secret"}
service = YouTubeUploadService(credentials=creds, non_interactive=True)
exists, video_id, video_url = service.check_duplicate("Some Title")
assert exists is False
assert video_id is None
assert video_url is None
@patch("googleapiclient.discovery.build")
@patch("google.oauth2.credentials.Credentials")
@patch("google.auth.transport.requests.Request")
def test_check_duplicate_skips_other_channels(
self, mock_request, mock_credentials_class, mock_build
):
"""Test that videos from other channels are skipped."""
mock_creds = MagicMock()
mock_creds.expired = False
mock_credentials_class.return_value = mock_creds
mock_youtube = MagicMock()
mock_channels = MagicMock()
mock_channels_list = MagicMock()
mock_channels_list.execute.return_value = {"items": [{"id": "UC123456"}]}
mock_channels.list.return_value = mock_channels_list
mock_youtube.channels.return_value = mock_channels
# Return video from different channel
mock_search = MagicMock()
mock_search_list = MagicMock()
mock_search_list.execute.return_value = {
"items": [{
"id": {"videoId": "VIDEO123"},
"snippet": {
"channelId": "UC_DIFFERENT", # Different channel
"title": "Test Title"
}
}]
}
mock_search.list.return_value = mock_search_list
mock_youtube.search.return_value = mock_search
mock_build.return_value = mock_youtube
creds = {"token": "test", "refresh_token": "test", "token_uri": "uri",
"client_id": "id", "client_secret": "secret"}
service = YouTubeUploadService(credentials=creds, non_interactive=True)
exists, video_id, video_url = service.check_duplicate("Test Title")
assert exists is False
class TestYouTubeUploadServiceDelete:
"""Test video deletion."""
@patch("googleapiclient.discovery.build")
@patch("google.oauth2.credentials.Credentials")
@patch("google.auth.transport.requests.Request")
def test_delete_video_success(
self, mock_request, mock_credentials_class, mock_build
):
"""Test successful video deletion."""
mock_creds = MagicMock()
mock_creds.expired = False
mock_credentials_class.return_value = mock_creds
mock_youtube = MagicMock()
mock_videos = MagicMock()
mock_delete = MagicMock()
mock_delete.execute.return_value = None
mock_videos.delete.return_value = mock_delete
mock_youtube.videos.return_value = mock_videos
mock_build.return_value = mock_youtube
creds = {"token": "test", "refresh_token": "test", "token_uri": "uri",
"client_id": "id", "client_secret": "secret"}
service = YouTubeUploadService(credentials=creds, non_interactive=True)
result = service.delete_video("VIDEO123")
assert result is True
mock_videos.delete.assert_called_once_with(id="VIDEO123")
@patch("googleapiclient.discovery.build")
@patch("google.oauth2.credentials.Credentials")
@patch("google.auth.transport.requests.Request")
def test_delete_video_failure(
self, mock_request, mock_credentials_class, mock_build
):
"""Test video deletion failure handling."""
mock_creds = MagicMock()
mock_creds.expired = False
mock_credentials_class.return_value = mock_creds
mock_youtube = MagicMock()
mock_videos = MagicMock()
mock_delete = MagicMock()
mock_delete.execute.side_effect = Exception("API Error")
mock_videos.delete.return_value = mock_delete
mock_youtube.videos.return_value = mock_videos
mock_build.return_value = mock_youtube
creds = {"token": "test", "refresh_token": "test", "token_uri": "uri",
"client_id": "id", "client_secret": "secret"}
service = YouTubeUploadService(credentials=creds, non_interactive=True)
result = service.delete_video("VIDEO123")
assert result is False
def test_delete_video_dry_run(self):
"""Test video deletion in dry run mode."""
service = YouTubeUploadService(dry_run=True)
result = service.delete_video("VIDEO123")
assert result is True
class TestYouTubeUploadServiceTitleTruncation:
"""Test title truncation."""
def test_truncate_title_short_title(self):
"""Test that short titles are not truncated."""
title = "Short Title"
result = YouTubeUploadService.truncate_title(title)
assert result == title
def test_truncate_title_exact_length(self):
"""Test title at exact max length."""
title = "A" * 95
result = YouTubeUploadService.truncate_title(title, max_length=95)
assert result == title
assert len(result) == 95
def test_truncate_title_long_title(self):
"""Test that long titles are truncated at word boundary."""
title = "This is a very long title that exceeds the maximum length and needs to be truncated properly at a word boundary"
result = YouTubeUploadService.truncate_title(title, max_length=50)
assert len(result) <= 50
assert result.endswith("...")
def test_truncate_title_no_space(self):
"""Test truncation of title without spaces."""
title = "A" * 100
result = YouTubeUploadService.truncate_title(title, max_length=50)
assert len(result) <= 50
class TestYouTubeUploadServiceUpload:
"""Test video upload."""
@patch("googleapiclient.http.MediaFileUpload")
@patch("googleapiclient.discovery.build")
@patch("google.oauth2.credentials.Credentials")
@patch("google.auth.transport.requests.Request")
def test_upload_video_success(
self, mock_request, mock_credentials_class, mock_build, mock_media_upload
):
"""Test successful video upload."""
mock_creds = MagicMock()
mock_creds.expired = False
mock_credentials_class.return_value = mock_creds
mock_youtube = MagicMock()
# Mock channels for get_channel_id
mock_channels = MagicMock()
mock_channels_list = MagicMock()
mock_channels_list.execute.return_value = {"items": [{"id": "UC123456"}]}
mock_channels.list.return_value = mock_channels_list
mock_youtube.channels.return_value = mock_channels
# Mock search for duplicate check (no duplicates)
mock_search = MagicMock()
mock_search_list = MagicMock()
mock_search_list.execute.return_value = {"items": []}
mock_search.list.return_value = mock_search_list
mock_youtube.search.return_value = mock_search
# Mock video insert
mock_videos = MagicMock()
mock_insert = MagicMock()
mock_insert.execute.return_value = {"id": "NEW_VIDEO_ID"}
mock_videos.insert.return_value = mock_insert
mock_youtube.videos.return_value = mock_videos
mock_build.return_value = mock_youtube
mock_media_upload.return_value = MagicMock()
creds = {"token": "test", "refresh_token": "test", "token_uri": "uri",
"client_id": "id", "client_secret": "secret"}
service = YouTubeUploadService(credentials=creds, non_interactive=True)
video_id, video_url = service.upload_video(
video_path="/path/to/video.mkv",
title="Test Artist - Test Song (Karaoke)",
description="Test description",
tags=["karaoke", "test"]
)
assert video_id == "NEW_VIDEO_ID"
assert "NEW_VIDEO_ID" in video_url
def test_upload_video_dry_run(self):
"""Test video upload in dry run mode."""
service = YouTubeUploadService(dry_run=True)
video_id, video_url = service.upload_video(
video_path="/path/to/video.mkv",
title="Test Title",
description="Test description"
)
assert video_id == "dry_run_video_id"
assert "dry_run_video_id" in video_url
@patch("googleapiclient.http.MediaFileUpload")
@patch("googleapiclient.discovery.build")
@patch("google.oauth2.credentials.Credentials")
@patch("google.auth.transport.requests.Request")
@patch("os.path.isfile")
def test_upload_video_with_thumbnail(
self, mock_isfile, mock_request, mock_credentials_class,
mock_build, mock_media_upload
):
"""Test video upload with thumbnail."""
mock_isfile.return_value = True
mock_creds = MagicMock()
mock_creds.expired = False
mock_credentials_class.return_value = mock_creds
mock_youtube = MagicMock()
mock_channels = MagicMock()
mock_channels_list = MagicMock()
mock_channels_list.execute.return_value = {"items": [{"id": "UC123456"}]}
mock_channels.list.return_value = mock_channels_list
mock_youtube.channels.return_value = mock_channels
mock_search = MagicMock()
mock_search_list = MagicMock()
mock_search_list.execute.return_value = {"items": []}
mock_search.list.return_value = mock_search_list
mock_youtube.search.return_value = mock_search
mock_videos = MagicMock()
mock_insert = MagicMock()
mock_insert.execute.return_value = {"id": "NEW_VIDEO_ID"}
mock_videos.insert.return_value = mock_insert
mock_youtube.videos.return_value = mock_videos
# Mock thumbnail upload
mock_thumbnails = MagicMock()
mock_set = MagicMock()
mock_set.execute.return_value = None
mock_thumbnails.set.return_value = mock_set
mock_youtube.thumbnails.return_value = mock_thumbnails
mock_build.return_value = mock_youtube
mock_media_upload.return_value = MagicMock()
creds = {"token": "test", "refresh_token": "test", "token_uri": "uri",
"client_id": "id", "client_secret": "secret"}
service = YouTubeUploadService(credentials=creds, non_interactive=True)
video_id, video_url = service.upload_video(
video_path="/path/to/video.mkv",
title="Test Title",
description="Test description",
thumbnail_path="/path/to/thumbnail.jpg"
)
assert video_id == "NEW_VIDEO_ID"
mock_thumbnails.set.assert_called_once()
class TestGetYouTubeUploadService:
"""Test factory function."""
def test_get_service_creates_instance(self):
"""Test that factory function creates a new instance."""
# Reset global
import backend.services.youtube_upload_service as module
module._youtube_upload_service = None
service = get_youtube_upload_service(
credentials={"token": "test"},
non_interactive=True
)
assert service is not None
assert isinstance(service, YouTubeUploadService)
def test_get_service_with_client_secrets(self):
"""Test factory function with client secrets file."""
import backend.services.youtube_upload_service as module
module._youtube_upload_service = None
service = get_youtube_upload_service(
client_secrets_file="/path/to/secrets.json"
)
assert service is not None
assert service.client_secrets_file == "/path/to/secrets.json"
+257 -19
View File
@@ -1,20 +1,21 @@
"""
Video generation and finalization worker.
This worker uses KaraokeFinalise from the karaoke_gen package to handle:
1. Video encoding (4 formats)
2. CDG/TXT package generation
3. Discord notifications
4. Brand code generation
5. All other finalisation features
This worker has two code paths controlled by USE_NEW_ORCHESTRATOR:
The worker's job is simply to:
1. Download files from GCS and set up the directory structure KaraokeFinalise expects
2. Instantiate KaraokeFinalise with parameters from the job
3. Call process()
4. Upload results back to GCS
1. NEW PATH (USE_NEW_ORCHESTRATOR=true):
Uses VideoWorkerOrchestrator which provides a unified pipeline for all
encoding backends (GCE or local). This ensures features like YouTube upload,
Discord notifications, and CDG/TXT packaging work regardless of encoding backend.
This reuses 100% of the existing karaoke_gen functionality.
2. LEGACY PATH (USE_NEW_ORCHESTRATOR=false):
Uses KaraokeFinalise from the karaoke_gen package. This path has divergent
behavior between GCE and local encoding - GCE encoding bypasses many features.
The orchestrator-based approach is preferred as it:
- Eliminates GCE vs local code path divergence
- Ensures all features work consistently
- Provides better testability and maintainability
Observability:
- All operations wrapped in tracing spans for Cloud Trace visibility
@@ -48,6 +49,12 @@ from karaoke_gen.karaoke_finalise.karaoke_finalise import KaraokeFinalise
logger = logging.getLogger(__name__)
# Feature flag for new orchestrator-based pipeline
# Set to True to use the new unified pipeline that works with any encoding backend
# Set to False to use the legacy KaraokeFinalise-based pipeline
USE_NEW_ORCHESTRATOR = os.environ.get("USE_NEW_ORCHESTRATOR", "true").lower() == "true"
# Loggers to capture for video worker
# Include the full module path to properly capture KaraokeFinalise logs
VIDEO_WORKER_LOGGERS = [
@@ -172,16 +179,247 @@ async def _encode_via_gce(
raise
async def generate_video(job_id: str) -> bool:
async def generate_video_orchestrated(job_id: str) -> bool:
"""
Generate final karaoke videos using KaraokeFinalise.
This sets up the directory structure and calls KaraokeFinalise.process()
which handles all encoding, packaging, and notifications.
Generate final karaoke videos using the new VideoWorkerOrchestrator.
This provides a unified pipeline that works with any encoding backend
(GCE or local), ensuring all features like YouTube upload, Discord
notifications, and CDG/TXT packaging work consistently.
Args:
job_id: Job ID to process
Returns:
True if successful, False otherwise
"""
from backend.workers.video_worker_orchestrator import (
VideoWorkerOrchestrator,
create_orchestrator_config_from_job,
)
start_time = time.time()
job_manager = JobManager()
storage = StorageService()
# Create job logger for remote debugging
job_log = create_job_logger(job_id, "video")
# Log with structured markers for easy Cloud Logging queries
logger.info(f"[job:{job_id}] WORKER_START worker=video orchestrator=true")
# Set up log capture for KaraokeFinalise (still used by some services)
log_handler = setup_job_logging(job_id, "video", *VIDEO_WORKER_LOGGERS)
job = job_manager.get_job(job_id)
if not job:
logger.error(f"[job:{job_id}] Job not found")
return False
# Validate prerequisites
if not _validate_prerequisites(job):
logger.error(f"[job:{job_id}] Prerequisites not met for video generation")
return False
# Create temporary working directory
temp_dir = tempfile.mkdtemp(prefix=f"karaoke_video_{job_id}_")
original_cwd = os.getcwd()
# Set up rclone config if needed (for legacy Dropbox upload path)
rclone_service = None
if getattr(job, 'organised_dir_rclone_root', None):
rclone_service = get_rclone_service()
if rclone_service.setup_rclone_config():
job_log.info("Rclone config loaded for Dropbox upload")
# Load YouTube credentials if needed
youtube_credentials = None
if getattr(job, 'enable_youtube_upload', False):
youtube_service = get_youtube_service()
if youtube_service.is_configured:
youtube_credentials = youtube_service.get_credentials_dict()
job_log.info("YouTube credentials loaded for video upload")
else:
job_log.warning("YouTube credentials not available - upload will be skipped")
try:
# Wrap entire worker in a tracing span
with job_span("video-worker-orchestrated", job_id, {"artist": job.artist, "title": job.title}) as root_span:
with job_logging_context(job_id):
job_log.info(f"Starting orchestrated video generation for {job.artist} - {job.title}")
logger.info(f"[job:{job_id}] Starting orchestrated video generation")
# Transition to GENERATING_VIDEO state
job_manager.transition_to_state(
job_id=job_id,
new_status=JobStatus.GENERATING_VIDEO,
progress=70,
message="Preparing files for video generation"
)
# Download and set up files
base_name = f"{job.artist} - {job.title}"
with job_span("download-files", job_id):
job_log.info("Downloading files from GCS...")
await _setup_working_directory(job_id, job, storage, temp_dir, base_name, job_log)
job_log.info("All files downloaded successfully")
# Load style config for CDG styles
style_config = await load_style_config(job, storage, temp_dir)
cdg_styles = style_config.get_cdg_styles()
# Change to working directory
os.chdir(temp_dir)
# Create orchestrator config from job
config = create_orchestrator_config_from_job(
job=job,
temp_dir=temp_dir,
youtube_credentials=youtube_credentials,
cdg_styles=cdg_styles,
)
# Run the orchestrated pipeline
orchestrator = VideoWorkerOrchestrator(
config=config,
job_manager=job_manager,
storage=storage,
job_logger=job_log,
)
with job_span("orchestrator-run", job_id):
result = await orchestrator.run()
if not result.success:
raise Exception(result.error_message or "Orchestrator failed")
# Prepare distribution directory for native uploads
with job_span("distribution", job_id):
await _handle_native_distribution(
job_id=job_id,
job=job,
job_log=job_log,
job_manager=job_manager,
temp_dir=temp_dir,
result={
'brand_code': result.brand_code,
'youtube_url': result.youtube_url,
'final_video': result.final_video,
'final_video_lossy': result.final_video_lossy,
'final_video_720p': result.final_video_720p,
'final_karaoke_cdg_zip': result.final_karaoke_cdg_zip,
'dropbox_link': result.dropbox_link,
'gdrive_files': result.gdrive_files,
},
storage=storage,
)
# Upload generated files to GCS
job_manager.transition_to_state(
job_id=job_id,
new_status=JobStatus.PACKAGING,
progress=95,
message="Uploading final files"
)
with job_span("upload-results", job_id):
await _upload_results(job_id, job_manager, storage, temp_dir, {
'final_video': result.final_video,
'final_video_mkv': result.final_video_mkv,
'final_video_lossy': result.final_video_lossy,
'final_video_720p': result.final_video_720p,
'final_karaoke_cdg_zip': result.final_karaoke_cdg_zip,
'final_karaoke_txt_zip': result.final_karaoke_txt_zip,
})
# Mark job as complete
logger.info(f"[job:{job_id}] Video generation complete")
job_manager.transition_to_state(
job_id=job_id,
new_status=JobStatus.COMPLETE,
progress=100,
message="Karaoke generation complete!"
)
# Store result metadata in job
job_manager.update_job(job_id, {
'state_data': {
**job.state_data,
'brand_code': result.brand_code,
'youtube_url': result.youtube_url,
'dropbox_link': result.dropbox_link,
'gdrive_files': result.gdrive_files,
}
})
duration = time.time() - start_time
root_span.set_attribute("duration_seconds", duration)
root_span.set_attribute("brand_code", result.brand_code or '')
logger.info(f"[job:{job_id}] WORKER_END worker=video orchestrator=true status=success duration={duration:.1f}s")
return True
except Exception as e:
duration = time.time() - start_time
logger.error(f"[job:{job_id}] WORKER_END worker=video orchestrator=true status=error duration={duration:.1f}s error={e}")
job_manager.mark_job_failed(
job_id=job_id,
error_message=f"Video generation failed: {str(e)}",
error_details={"stage": "video_generation", "error": str(e)}
)
return False
finally:
# Restore original working directory
os.chdir(original_cwd)
# Remove log handler
for logger_name in VIDEO_WORKER_LOGGERS:
try:
logging.getLogger(logger_name).removeHandler(log_handler)
except Exception:
pass
# Cleanup rclone config file
if rclone_service:
rclone_service.cleanup()
# Cleanup temporary directory
if os.path.exists(temp_dir):
shutil.rmtree(temp_dir)
logger.debug(f"Cleaned up temp directory: {temp_dir}")
async def generate_video(job_id: str) -> bool:
"""
Generate final karaoke videos.
Routes to either the new orchestrator-based pipeline or the legacy
KaraokeFinalise-based pipeline based on USE_NEW_ORCHESTRATOR flag.
Args:
job_id: Job ID to process
Returns:
True if successful, False otherwise
"""
if USE_NEW_ORCHESTRATOR:
logger.info(f"[job:{job_id}] Using new orchestrator-based pipeline")
return await generate_video_orchestrated(job_id)
else:
logger.info(f"[job:{job_id}] Using legacy KaraokeFinalise pipeline")
return await generate_video_legacy(job_id)
async def generate_video_legacy(job_id: str) -> bool:
"""
Generate final karaoke videos using KaraokeFinalise (legacy path).
This is the original implementation that has divergent behavior between
GCE and local encoding paths. Kept for rollback purposes.
Args:
job_id: Job ID to process
Returns:
True if successful, False otherwise
"""
@@ -0,0 +1,638 @@
"""
Video Worker Orchestrator.
Coordinates the video generation pipeline stages in a unified way,
regardless of whether encoding happens locally or on GCE.
This resolves the code path divergence where GCE encoding bypassed
features like YouTube upload, Discord notifications, and CDG/TXT packaging.
Pipeline stages:
1. Setup - Download files, prepare directories
2. Packaging - CDG/TXT generation (if enabled)
3. Encoding - GCE or Local via EncodingBackend interface
4. Organization - Brand code, folder structure
5. Distribution - YouTube, Dropbox, Google Drive uploads
6. Notification - Discord notifications
"""
import logging
import os
import time
from dataclasses import dataclass, field
from typing import Optional, Dict, Any, List
from pathlib import Path
from backend.models.job import JobStatus
from backend.services.job_manager import JobManager
from backend.services.storage_service import StorageService
from backend.services.tracing import job_span, add_span_event
logger = logging.getLogger(__name__)
@dataclass
class OrchestratorConfig:
"""Configuration for the video worker orchestrator."""
job_id: str
artist: str
title: str
# Input file paths (in temp_dir)
title_video_path: str
karaoke_video_path: str
instrumental_audio_path: str
end_video_path: Optional[str] = None
lrc_file_path: Optional[str] = None
title_jpg_path: Optional[str] = None
# Output directory
output_dir: str = ""
# Feature flags
enable_cdg: bool = False
enable_txt: bool = False
enable_youtube_upload: bool = False
# Service configurations
brand_prefix: Optional[str] = None
discord_webhook_url: Optional[str] = None
youtube_credentials: Optional[Dict[str, Any]] = None
youtube_description_template: Optional[str] = None
cdg_styles: Optional[Dict[str, Any]] = None
# Dropbox/GDrive configuration
dropbox_path: Optional[str] = None
gdrive_folder_id: Optional[str] = None
# Keep existing brand code (for re-processing)
keep_brand_code: Optional[str] = None
# Encoding backend preference
encoding_backend: str = "auto" # "auto", "local", "gce"
# Additional options
dry_run: bool = False
non_interactive: bool = True
@dataclass
class OrchestratorResult:
"""Result from the video worker orchestrator."""
success: bool
error_message: Optional[str] = None
# Generated files
final_video: Optional[str] = None # Lossless 4K MP4
final_video_mkv: Optional[str] = None # Lossless 4K MKV
final_video_lossy: Optional[str] = None # Lossy 4K MP4
final_video_720p: Optional[str] = None # Lossy 720p MP4
final_karaoke_cdg_zip: Optional[str] = None
final_karaoke_txt_zip: Optional[str] = None
# Organization
brand_code: Optional[str] = None
# Distribution results
youtube_url: Optional[str] = None
dropbox_link: Optional[str] = None
gdrive_files: Optional[Dict[str, str]] = field(default_factory=dict)
# Timing
encoding_time_seconds: Optional[float] = None
total_time_seconds: Optional[float] = None
class VideoWorkerOrchestrator:
"""
Orchestrates the video generation pipeline.
This class coordinates all stages of video generation in a unified way,
ensuring that features like YouTube upload and Discord notifications
work regardless of whether GCE or local encoding is used.
"""
def __init__(
self,
config: OrchestratorConfig,
job_manager: Optional[JobManager] = None,
storage: Optional[StorageService] = None,
job_logger: Optional[logging.Logger] = None,
):
"""
Initialize the orchestrator.
Args:
config: Orchestrator configuration
job_manager: Job manager for state updates (optional)
storage: Storage service for file downloads (optional)
job_logger: Job-specific logger (optional)
"""
self.config = config
self.job_manager = job_manager
self.storage = storage
self.job_log = job_logger or logging.getLogger(__name__)
# Services (lazy-loaded)
self._encoding_backend = None
self._packaging_service = None
self._youtube_service = None
self._discord_service = None
# State
self.result = OrchestratorResult(success=False)
def _get_encoding_backend(self):
"""Lazy-load the encoding backend."""
if self._encoding_backend is None:
from backend.services.encoding_interface import get_encoding_backend
self._encoding_backend = get_encoding_backend(
backend_type=self.config.encoding_backend,
dry_run=self.config.dry_run,
logger=self.job_log,
)
return self._encoding_backend
def _get_packaging_service(self):
"""Lazy-load the packaging service."""
if self._packaging_service is None:
from backend.services.packaging_service import PackagingService
self._packaging_service = PackagingService(
cdg_styles=self.config.cdg_styles,
dry_run=self.config.dry_run,
non_interactive=self.config.non_interactive,
logger=self.job_log,
)
return self._packaging_service
def _get_youtube_service(self):
"""Lazy-load the YouTube upload service."""
if self._youtube_service is None:
from backend.services.youtube_upload_service import YouTubeUploadService
self._youtube_service = YouTubeUploadService(
credentials=self.config.youtube_credentials,
non_interactive=self.config.non_interactive,
server_side_mode=True,
dry_run=self.config.dry_run,
logger=self.job_log,
)
return self._youtube_service
def _get_discord_service(self):
"""Lazy-load the Discord notification service."""
if self._discord_service is None:
from backend.services.discord_service import DiscordNotificationService
self._discord_service = DiscordNotificationService(
webhook_url=self.config.discord_webhook_url,
dry_run=self.config.dry_run,
logger=self.job_log,
)
return self._discord_service
def _update_progress(self, status: JobStatus, progress: int, message: str):
"""Update job progress if job_manager is available."""
if self.job_manager:
self.job_manager.transition_to_state(
job_id=self.config.job_id,
new_status=status,
progress=progress,
message=message
)
async def run(self) -> OrchestratorResult:
"""
Run the full video generation pipeline.
Returns:
OrchestratorResult with generated files and metadata
"""
start_time = time.time()
try:
with job_span("orchestrator", self.config.job_id) as span:
self.job_log.info(f"Starting orchestrated video generation for {self.config.artist} - {self.config.title}")
# Stage 1: Packaging (CDG/TXT) - runs BEFORE encoding
# This depends on LRC file, not on encoded videos
if self.config.enable_cdg or self.config.enable_txt:
await self._run_packaging()
# Stage 2: Encoding
await self._run_encoding()
# Stage 3: Organization (brand code)
await self._run_organization()
# Stage 4: Distribution (YouTube, Dropbox, GDrive)
await self._run_distribution()
# Stage 5: Notifications (Discord)
await self._run_notifications()
self.result.success = True
self.result.total_time_seconds = time.time() - start_time
self.job_log.info(f"Orchestrated video generation complete in {self.result.total_time_seconds:.1f}s")
except Exception as e:
self.result.success = False
self.result.error_message = str(e)
self.result.total_time_seconds = time.time() - start_time
self.job_log.error(f"Orchestrated video generation failed: {e}")
logger.error(f"[job:{self.config.job_id}] Orchestration failed: {e}")
return self.result
async def _run_packaging(self):
"""Run the packaging stage (CDG/TXT generation)."""
self.job_log.info("Starting packaging stage (CDG/TXT)")
if not self.config.lrc_file_path or not os.path.isfile(self.config.lrc_file_path):
self.job_log.warning("No LRC file available, skipping CDG/TXT packaging")
return
base_name = f"{self.config.artist} - {self.config.title}"
packaging_service = self._get_packaging_service()
# Generate CDG package
if self.config.enable_cdg:
self.job_log.info("Generating CDG package")
try:
cdg_zip_path = os.path.join(
self.config.output_dir,
f"{base_name} (Final Karaoke CDG).zip"
)
mp3_path = os.path.join(
self.config.output_dir,
f"{base_name} (Karaoke).mp3"
)
cdg_path = os.path.join(
self.config.output_dir,
f"{base_name} (Karaoke).cdg"
)
zip_file, mp3_file, cdg_file = packaging_service.create_cdg_package(
lrc_file=self.config.lrc_file_path,
audio_file=self.config.instrumental_audio_path,
output_zip_path=cdg_zip_path,
artist=self.config.artist,
title=self.config.title,
output_mp3_path=mp3_path,
output_cdg_path=cdg_path,
)
self.result.final_karaoke_cdg_zip = zip_file
self.job_log.info(f"CDG package created: {zip_file}")
except Exception as e:
self.job_log.error(f"CDG generation failed: {e}")
# Don't fail the pipeline, CDG is optional
# Generate TXT package
if self.config.enable_txt:
self.job_log.info("Generating TXT package")
try:
# TXT package needs MP3 file (from CDG generation or create it)
mp3_path = os.path.join(
self.config.output_dir,
f"{base_name} (Karaoke).mp3"
)
if not os.path.isfile(mp3_path):
self.job_log.info("MP3 not found, CDG must be enabled first for TXT")
return
txt_zip_path = os.path.join(
self.config.output_dir,
f"{base_name} (Final Karaoke TXT).zip"
)
zip_file, txt_file = packaging_service.create_txt_package(
lrc_file=self.config.lrc_file_path,
mp3_file=mp3_path,
output_zip_path=txt_zip_path,
)
self.result.final_karaoke_txt_zip = zip_file
self.job_log.info(f"TXT package created: {zip_file}")
except Exception as e:
self.job_log.error(f"TXT generation failed: {e}")
# Don't fail the pipeline, TXT is optional
async def _run_encoding(self):
"""Run the encoding stage."""
self.job_log.info("Starting encoding stage")
self._update_progress(JobStatus.ENCODING, 75, "Encoding videos")
encoding_backend = self._get_encoding_backend()
self.job_log.info(f"Using encoding backend: {encoding_backend.name}")
# Build encoding input
from backend.services.encoding_interface import EncodingInput
from backend.config import settings
# For GCE encoding, we need to provide GCS paths
gcs_bucket = settings.gcs_bucket_name
input_gcs_path = f"gs://{gcs_bucket}/jobs/{self.config.job_id}/"
output_gcs_path = f"gs://{gcs_bucket}/jobs/{self.config.job_id}/finals/"
encoding_input = EncodingInput(
title_video_path=self.config.title_video_path,
karaoke_video_path=self.config.karaoke_video_path,
instrumental_audio_path=self.config.instrumental_audio_path,
end_video_path=self.config.end_video_path,
artist=self.config.artist,
title=self.config.title,
brand_code=self.config.keep_brand_code,
output_dir=self.config.output_dir,
options={
"input_gcs_path": input_gcs_path,
"output_gcs_path": output_gcs_path,
},
)
# Run encoding
with job_span("encoding", self.config.job_id) as span:
add_span_event("encoding_started", {"backend": encoding_backend.name})
output = await encoding_backend.encode(encoding_input)
add_span_event("encoding_completed", {
"success": output.success,
"duration": output.encoding_time_seconds or 0
})
if not output.success:
raise Exception(f"Encoding failed: {output.error_message}")
# Store results
self.result.final_video = output.lossless_4k_mp4_path
self.result.final_video_mkv = output.lossless_mkv_path
self.result.final_video_lossy = output.lossy_4k_mp4_path
self.result.final_video_720p = output.lossy_720p_mp4_path
self.result.encoding_time_seconds = output.encoding_time_seconds
self.job_log.info(f"Encoding complete ({encoding_backend.name}) in {output.encoding_time_seconds:.1f}s")
async def _run_organization(self):
"""Run the organization stage (brand code generation)."""
self.job_log.info("Starting organization stage")
# Use existing brand code if provided
if self.config.keep_brand_code:
self.result.brand_code = self.config.keep_brand_code
self.job_log.info(f"Using preserved brand code: {self.result.brand_code}")
return
# Generate brand code from Dropbox if configured
if self.config.dropbox_path and self.config.brand_prefix:
try:
from backend.services.dropbox_service import get_dropbox_service
dropbox = get_dropbox_service()
if dropbox.is_configured:
brand_code = dropbox.get_next_brand_code(
self.config.dropbox_path,
self.config.brand_prefix
)
self.result.brand_code = brand_code
self.job_log.info(f"Generated brand code: {brand_code}")
else:
self.job_log.warning("Dropbox not configured, skipping brand code generation")
except Exception as e:
self.job_log.error(f"Brand code generation failed: {e}")
# Don't fail - brand code is optional
async def _run_distribution(self):
"""Run the distribution stage (YouTube, Dropbox, GDrive uploads)."""
self.job_log.info("Starting distribution stage")
self._update_progress(JobStatus.PACKAGING, 90, "Uploading files")
# YouTube upload
if self.config.enable_youtube_upload and self.config.youtube_credentials:
await self._upload_to_youtube()
# Dropbox upload
if self.config.dropbox_path and self.config.brand_prefix:
await self._upload_to_dropbox()
# Google Drive upload
if self.config.gdrive_folder_id:
await self._upload_to_gdrive()
async def _upload_to_youtube(self):
"""Upload video to YouTube."""
self.job_log.info("Uploading to YouTube")
# Find the best video file to upload (prefer MKV for FLAC audio, then lossless MP4)
video_to_upload = None
if self.result.final_video_mkv and os.path.isfile(self.result.final_video_mkv):
video_to_upload = self.result.final_video_mkv
elif self.result.final_video and os.path.isfile(self.result.final_video):
video_to_upload = self.result.final_video
elif self.result.final_video_lossy and os.path.isfile(self.result.final_video_lossy):
video_to_upload = self.result.final_video_lossy
if not video_to_upload:
self.job_log.warning("No video file available for YouTube upload")
return
try:
youtube_service = self._get_youtube_service()
# Build video title
title = f"{self.config.artist} - {self.config.title} (Karaoke)"
# Build description
description = self.config.youtube_description_template or ""
if self.result.brand_code:
description = f"{description}\n\nBrand Code: {self.result.brand_code}".strip()
# Upload
video_id, video_url = youtube_service.upload_video(
video_path=video_to_upload,
title=title,
description=description,
thumbnail_path=self.config.title_jpg_path,
tags=["karaoke", self.config.artist, self.config.title],
replace_existing=True, # Server-side always replaces
)
if video_url:
self.result.youtube_url = video_url
self.job_log.info(f"Uploaded to YouTube: {video_url}")
else:
self.job_log.warning("YouTube upload did not return a URL")
except Exception as e:
self.job_log.error(f"YouTube upload failed: {e}")
# Don't fail the pipeline - YouTube is optional
async def _upload_to_dropbox(self):
"""Upload files to Dropbox."""
self.job_log.info("Uploading to Dropbox")
try:
from backend.services.dropbox_service import get_dropbox_service
dropbox = get_dropbox_service()
if not dropbox.is_configured:
self.job_log.warning("Dropbox not configured, skipping upload")
return
base_name = f"{self.config.artist} - {self.config.title}"
folder_name = f"{self.result.brand_code or 'TRACK-0000'} - {base_name}"
remote_folder = f"{self.config.dropbox_path}/{folder_name}"
# Upload entire output directory
dropbox.upload_folder(self.config.output_dir, remote_folder)
# Create sharing link
try:
sharing_link = dropbox.create_shared_link(remote_folder)
self.result.dropbox_link = sharing_link
self.job_log.info(f"Dropbox sharing link: {sharing_link}")
except Exception as e:
self.job_log.warning(f"Failed to create Dropbox sharing link: {e}")
self.job_log.info("Dropbox upload complete")
except Exception as e:
self.job_log.error(f"Dropbox upload failed: {e}")
# Don't fail the pipeline - Dropbox is optional
async def _upload_to_gdrive(self):
"""Upload files to Google Drive."""
self.job_log.info("Uploading to Google Drive")
try:
from backend.services.gdrive_service import get_gdrive_service
gdrive = get_gdrive_service()
if not gdrive.is_configured:
self.job_log.warning("Google Drive not configured, skipping upload")
return
base_name = f"{self.config.artist} - {self.config.title}"
brand_code = self.result.brand_code or f"{self.config.brand_prefix or 'TRACK'}-0000"
# Map result files to expected keys
output_files = {
'final_karaoke_lossy_mp4': self.result.final_video_lossy,
'final_karaoke_lossy_720p_mp4': self.result.final_video_720p,
'final_karaoke_cdg_zip': self.result.final_karaoke_cdg_zip,
}
uploaded = gdrive.upload_to_public_share(
root_folder_id=self.config.gdrive_folder_id,
brand_code=brand_code,
base_name=base_name,
output_files=output_files,
)
self.result.gdrive_files = uploaded
self.job_log.info(f"Google Drive upload complete: {len(uploaded)} files")
except Exception as e:
self.job_log.error(f"Google Drive upload failed: {e}")
# Don't fail the pipeline - GDrive is optional
async def _run_notifications(self):
"""Run the notifications stage (Discord)."""
self.job_log.info("Starting notifications stage")
if not self.config.discord_webhook_url:
self.job_log.debug("No Discord webhook configured, skipping notification")
return
if not self.result.youtube_url:
self.job_log.info("No YouTube URL available, skipping Discord notification")
return
try:
discord_service = self._get_discord_service()
discord_service.post_video_notification(self.result.youtube_url)
self.job_log.info("Discord notification sent")
except Exception as e:
self.job_log.error(f"Discord notification failed: {e}")
# Don't fail the pipeline - notifications are optional
def create_orchestrator_config_from_job(
job,
temp_dir: str,
youtube_credentials: Optional[Dict[str, Any]] = None,
cdg_styles: Optional[Dict[str, Any]] = None,
) -> OrchestratorConfig:
"""
Create an OrchestratorConfig from a job object.
This is a helper function to bridge the existing job structure
with the new orchestrator configuration.
Args:
job: Job object from Firestore
temp_dir: Temporary directory with downloaded files
youtube_credentials: Pre-loaded YouTube credentials
cdg_styles: CDG style configuration
Returns:
OrchestratorConfig for the orchestrator
"""
base_name = f"{job.artist} - {job.title}"
# Determine instrumental file path
instrumental_selection = job.state_data.get('instrumental_selection', 'clean')
existing_instrumental = getattr(job, 'existing_instrumental_gcs_path', None)
if existing_instrumental:
ext = Path(existing_instrumental).suffix.lower()
instrumental_path = os.path.join(temp_dir, f"{base_name} (Instrumental User){ext}")
else:
instrumental_suffix = "Clean" if instrumental_selection == 'clean' else "Backing"
instrumental_path = os.path.join(temp_dir, f"{base_name} (Instrumental {instrumental_suffix}).flac")
return OrchestratorConfig(
job_id=job.job_id,
artist=job.artist,
title=job.title,
# Input files
title_video_path=os.path.join(temp_dir, f"{base_name} (Title).mov"),
karaoke_video_path=os.path.join(temp_dir, f"{base_name} (With Vocals).mov"),
instrumental_audio_path=instrumental_path,
end_video_path=os.path.join(temp_dir, f"{base_name} (End).mov"),
lrc_file_path=os.path.join(temp_dir, f"{base_name} (Karaoke).lrc"),
title_jpg_path=os.path.join(temp_dir, f"{base_name} (Title).jpg"),
# Output directory
output_dir=temp_dir,
# Feature flags
enable_cdg=getattr(job, 'enable_cdg', False),
enable_txt=getattr(job, 'enable_txt', False),
enable_youtube_upload=getattr(job, 'enable_youtube_upload', False),
# Service configurations
brand_prefix=getattr(job, 'brand_prefix', None),
discord_webhook_url=getattr(job, 'discord_webhook_url', None),
youtube_credentials=youtube_credentials,
youtube_description_template=getattr(job, 'youtube_description_template', None),
cdg_styles=cdg_styles,
# Dropbox/GDrive
dropbox_path=getattr(job, 'dropbox_path', None),
gdrive_folder_id=getattr(job, 'gdrive_folder_id', None),
# Keep existing brand code
keep_brand_code=getattr(job, 'keep_brand_code', None),
# Encoding backend - auto selects GCE if available
encoding_backend="auto",
# Server-side defaults
dry_run=False,
non_interactive=True,
)
+104
View File
@@ -383,6 +383,110 @@ Authorization: Bearer ADMIN_TOKEN
Admins can delete any job. Regular users can only delete their own jobs.
### Audio Search Management (Admin)
#### List Audio Searches
```http
GET /api/admin/audio-searches
GET /api/admin/audio-searches?limit=50&status_filter=awaiting_audio_selection
Authorization: Bearer ADMIN_TOKEN
```
Returns jobs with cached audio search results. Useful for:
- Monitoring search activity
- Identifying stale cached results (YouTube-only when lossless should be available)
- Clearing cache for specific jobs
Response:
```json
{
"jobs": [
{
"job_id": "abc123",
"status": "awaiting_audio_selection",
"user_email": "user@example.com",
"audio_search_artist": "Artist Name",
"audio_search_title": "Song Title",
"created_at": "2026-01-03T12:00:00Z",
"results_count": 5,
"has_lossless": false,
"providers": ["YouTube"],
"results_summary": [...]
}
],
"total": 1
}
```
#### Clear Audio Search Cache
```http
POST /api/admin/audio-searches/{job_id}/clear-cache
Authorization: Bearer ADMIN_TOKEN
```
Clears cached search results and resets job to `pending` status, allowing a new search.
Use when:
- Cached results are stale (e.g., flacfetch was updated with new providers)
- User wants to search again
- Results appear incomplete (YouTube-only when lossless should exist)
Response:
```json
{
"status": "success",
"job_id": "abc123",
"message": "Cleared 5 cached search results. Job reset to pending. Flacfetch cache also cleared.",
"previous_status": "awaiting_audio_selection",
"new_status": "pending",
"results_cleared": 5,
"flacfetch_cache_cleared": true,
"flacfetch_error": null
}
```
Note: This endpoint now also clears the flacfetch GCS cache for the artist/title combination, ensuring the next search hits trackers fresh.
#### Clear All Flacfetch Cache
```http
DELETE /api/admin/cache
Authorization: Bearer ADMIN_TOKEN
```
Clears the entire flacfetch search cache (GCS-backed). Use when flacfetch has been updated and you want all subsequent searches to use fresh tracker results.
Response:
```json
{
"status": "success",
"message": "Cleared 15 cache entries from flacfetch.",
"deleted_count": 15
}
```
#### Get Flacfetch Cache Stats
```http
GET /api/admin/cache/stats
Authorization: Bearer ADMIN_TOKEN
```
Returns statistics about the flacfetch search cache.
Response:
```json
{
"count": 42,
"total_size_bytes": 128000,
"oldest_entry": "2025-12-15T10:30:00Z",
"newest_entry": "2026-01-03T15:45:00Z",
"configured": true
}
```
## Rate Limits
No rate limits currently implemented.
+37
View File
@@ -169,6 +169,43 @@ karaoke-gen shares a GCP project (`nomadkaraoke`) with karaoke-decide, but uses
**Note**: The `users` collection in the same Firestore instance belongs to karaoke-decide (different schema: user_id, is_guest, quiz_* fields). Don't use it for karaoke-gen.
## Video Worker Orchestrator
The Video Worker uses an orchestrator pattern to ensure all features work regardless of encoding backend (local Cloud Run or GCE VM).
```text
┌─────────────────────────────────────────────────────────────┐
│ VideoWorkerOrchestrator │
│ │
│ Stage 1: PACKAGING Stage 2: ENCODING │
│ ├─ CDG file generation ├─ Local (FFmpeg) │
│ └─ TXT lyrics export └─ GCE (remote VM) │
│ │
│ Stage 3: ORGANIZATION Stage 4: DISTRIBUTION │
│ ├─ Final path setup ├─ Dropbox upload │
│ └─ File organization ├─ Google Drive upload │
│ └─ YouTube upload │
│ │
│ Stage 5: NOTIFICATIONS │
│ └─ Discord webhooks │
└─────────────────────────────────────────────────────────────┘
```
**Key Design Decision**: The orchestrator coordinates stages rather than embedding logic in monolithic functions. This:
- Ensures all features work with both local and GCE encoding
- Makes each stage independently testable (139 new tests)
- Allows feature flag rollback via `USE_NEW_ORCHESTRATOR` env var
**Service Modules**:
| Module | Purpose |
|--------|---------|
| `youtube_upload_service.py` | YouTube OAuth flow and upload |
| `discord_service.py` | Discord webhook notifications |
| `packaging_service.py` | CDG/TXT package generation |
| `local_encoding_service.py` | FFmpeg encoding (Cloud Run) |
| `encoding_interface.py` | Abstract interface + GCE backend |
| `video_worker_orchestrator.py` | Stage coordination |
## Tech Stack
- **Backend**: FastAPI, Python 3.12, Cloud Run
+11 -6
View File
@@ -14,18 +14,20 @@
## Test Requirements
### 1. New User Registration (MailSlurp)
### 1. New User Registration (testmail.app)
- [ ] Navigate to landing page (`gen.nomadkaraoke.com`)
- [ ] Click "Sign In" or "Get Started" button
- [ ] Create a NEW MailSlurp inbox (fresh email address for each test run)
- [ ] Generate a unique test email address (testmail.app namespace)
- [ ] Enter the test email in the beta enrollment form
- [ ] Submit the form
- [ ] Wait for welcome/magic-link email in MailSlurp inbox
- [ ] Wait for welcome/magic-link email via testmail.app API
- [ ] Extract the magic link from the email
- [ ] Click/navigate to the magic link to authenticate
- [ ] Verify user is logged in and has credits
**Note**: When `use_test_token=true` workflow input is set, steps 1-9 above are skipped and the test uses a pre-authenticated token instead.
### 2. Create Karaoke Job (UI Only)
- [ ] Navigate to the karaoke creation page
@@ -118,8 +120,10 @@
### Environment Variables
```bash
MAILSLURP_API_KEY=xxx # Required - for email testing
# No pre-auth tokens - test handles full signup
TESTMAIL_NAMESPACE=xxx # Required - for email testing (testmail.app)
TESTMAIL_API_KEY=xxx # Required - for email testing (testmail.app)
E2E_TEST_TOKEN=xxx # Optional - skip signup, use existing account
# When use_test_token=true, skips email signup and uses pre-authenticated token
```
### Test Song
@@ -160,7 +164,8 @@ Log detailed progress at each step with timestamps for debugging.
### Secrets Required
- `MAILSLURP_API_KEY` - For email testing
- `TESTMAIL_API_KEY` - For email testing (testmail.app)
- `TESTMAIL_NAMESPACE` - testmail.app namespace for unique email addresses
### Workflow Steps
+144
View File
@@ -633,6 +633,40 @@ See `docs/archive/2025-12-28-self-hosted-github-runner.md` for full details.
- Pre-job checks provide fail-fast behavior with clear error messages
- `docker builder prune -af` catches buildkit cache that `system prune` misses
### Fonts in Docker for Video Rendering
**Problem**: Video rendering showed replacement characters (squares with question marks) for special Unicode symbols like `♪` in intro/instrumental screens.
**Root cause**: The `python:3.11-slim` Docker base image has NO fonts installed. FFmpeg/libass falls back to whatever it can find, which often lacks support for musical symbols, emoji, or non-Latin scripts.
**Symptoms**:
- `♪ INTRO (19 seconds) ♪` displayed as `□ INTRO (19 seconds) □`
- Unicode characters render as tofu (replacement boxes)
- Only appears in cloud-generated videos, not local dev (which uses system fonts)
**Solution**: Install comprehensive Noto fonts in the Docker base image:
```dockerfile
RUN apt-get update && apt-get install -y \
fonts-noto-core \
fonts-noto-cjk \
fonts-noto-extra \
fonts-noto-color-emoji \
fontconfig \
&& fc-cache -f \
&& rm -rf /var/lib/apt/lists/*
```
**Package breakdown**:
- `fonts-noto-core` (~10MB): Latin, Greek, Cyrillic, musical symbols
- `fonts-noto-cjk` (~150MB): Chinese, Japanese, Korean
- `fonts-noto-extra` (~50MB): Arabic, Hebrew, Thai, Hindi, other scripts
- `fonts-noto-color-emoji` (~10MB): Color emoji (limited libass support)
- `fontconfig`: Font configuration system (required for `fc-cache`)
**Note on color emoji**: FFmpeg's libass subtitle renderer doesn't fully support color emoji fonts - it renders text as vectors and can't display color bitmap glyphs. Color emoji appears as monochrome outlines or may not render at all in ASS subtitles.
**Lesson**: When building Docker images for video/subtitle rendering, install fonts explicitly. The default slim images have nothing, and font fallback behavior differs from local development machines.
## Performance Observations
| Operation | Duration |
@@ -789,6 +823,37 @@ result = job.state_data['search_results'][selection_index]
**Lesson**: Design for stateless request handling. Any state that must survive between requests must be persisted externally.
### Firestore Cache Has No Automatic Invalidation
**Problem**: Audio search results are cached in Firestore (`job.state_data.audio_search_results`) with no expiration or invalidation. When a library like flacfetch is updated with new providers or bug fixes, existing cached results remain stale forever.
**Symptoms**:
- Job shows only 5 YouTube results when CLI shows 59 results including lossless sources
- User cannot get better search results without admin intervention
- No "re-search" option in the UI
**Example**: Job `6ddbbef1` was searched when flacfetch had a bug returning only YouTube results. After flacfetch was fixed, the job still showed the old 5-result cache.
**Solution**: Added admin endpoints and UI to manage cached searches:
- `GET /api/admin/audio-searches` - List jobs with cached results
- `POST /api/admin/audio-searches/{job_id}/clear-cache` - Clear both Firestore and flacfetch GCS cache
- `DELETE /api/admin/cache` - Clear entire flacfetch cache
- `GET /api/admin/cache/stats` - View cache statistics
- Admin UI at `/admin/searches` shows cache stats and "Clear All Cache" button
**Key insight**: There are TWO caches involved:
1. **Firestore** (`job.state_data.audio_search_results`) - stores results for user selection UI
2. **Flacfetch GCS** - caches tracker search responses (30-day TTL)
Both must be cleared for a truly fresh search. The clear-cache endpoint now clears both automatically.
**Future consideration**: Add automatic cache invalidation based on:
- Time (TTL)
- flacfetch version change
- User-initiated "re-search" button
**Lesson**: When caching external API results, consider how/when the cache should be invalidated. Permanent caches need admin tooling to manage stale data. When multiple layers cache the same data, clearing one layer isn't enough - you need to clear all of them.
### GCS Downloads Preserve Directory Structure
**Problem**: When downloading files from GCS, the directory structure is preserved. Code that uses non-recursive glob patterns won't find files in subdirectories.
@@ -814,6 +879,49 @@ input_files = [f for f in input_files if "outputs" not in str(f)]
**Lesson**: When writing code that processes files downloaded from GCS, always use recursive glob patterns and verify the expected directory structure matches what's actually stored.
### GCS Blob Paths vs Full URIs
**Problem**: When uploading files to GCS and returning paths to other services, there's confusion between "blob paths" (just the path within a bucket) and "full GCS URIs" (gs://bucket/path).
**Symptoms**:
- 404 errors when downloading files that were just uploaded
- Paths look like: `bucket/gs://bucket/path/file.mp4` (doubled prefix)
- Storage service prepends bucket name to what's already a full URI
**Example of the bug**:
```python
# GCE worker uploads to gs://bucket/jobs/id/encoded/
# Then returns full URI in response: "gs://bucket/jobs/id/encoded/file.mp4"
# Backend tries: storage.download_file("gs://bucket/jobs/id/encoded/file.mp4")
# Storage service prepends bucket: "bucket/gs://bucket/jobs/id/encoded/file.mp4"
# Result: 404 - No such object
```
**Solution**: Return blob paths (path within bucket), not full GCS URIs:
```python
# output_gcs_path = "gs://bucket/jobs/id/encoded/"
gcs_path = output_gcs_path.replace("gs://", "")
parts = gcs_path.split("/", 1)
bucket = parts[0]
prefix = parts[1].rstrip("/") if len(parts) > 1 else ""
# Return blob paths like: "jobs/id/encoded/file.mp4"
blob_path = f"{prefix}/{filename}" if prefix else filename
```
**Related gotcha**: Double slashes in paths. When the prefix already ends with "/" and you concatenate with another "/":
```python
# BAD - creates "encoded//file.mp4"
prefix = "jobs/id/encoded/" # Has trailing slash
blob_name = f"{prefix}/{filename}" # Adds another slash
# GOOD - strip trailing slash first
prefix = parts[1].rstrip("/")
blob_name = f"{prefix}/{filename}" if prefix else filename
```
**Lesson**: When building systems that pass GCS paths between services, establish a clear contract: either always use full URIs (`gs://bucket/path`) or always use blob paths (`path/within/bucket`). Document which format each API expects.
### Library Caches Need GCS Sync in Cloud Run
**Problem**: Libraries like LyricsTranscriber cache API responses to local directories (e.g., `~/lyrics-transcriber-cache`). In Cloud Run, each container is ephemeral - the cache is lost on every restart or scale-up, causing redundant API calls.
@@ -894,6 +1002,42 @@ async def upload(auth_result: AuthResult = Depends(require_auth)):
See `docs/archive/2024-12-31-job-failure-investigation.md` for full details.
### Alternative Code Paths Must Implement All Features
**Problem**: Adding a GCE encoding option created a code path that completely bypassed the existing `KaraokeFinalise` class, which contained YouTube upload, Discord notifications, and CDG/TXT packaging. Jobs using GCE encoding silently missed these features.
```python
# BAD - two divergent code paths
if use_gce_encoding:
await gce_encoding_worker.encode(job_id) # Returns, does nothing else
return # YouTube upload, Discord, CDG never happen!
else:
finalise = KaraokeFinalise(...) # Has all the features
await finalise.run()
```
**Symptoms**:
- Jobs complete successfully but YouTube URL is never set
- No Discord notifications for GCE-encoded jobs
- CDG/TXT files never generated for GCE jobs
- No errors in logs (the code path simply doesn't include those features)
**Solution**: Create an orchestrator that runs all stages regardless of encoding backend:
```python
# GOOD - unified pipeline with swappable encoding
orchestrator = VideoWorkerOrchestrator(config)
result = await orchestrator.run()
# Orchestrator always runs: packaging → encoding → organization → distribution → notifications
# The encoding step uses either local or GCE backend based on config
```
**Key insight**: When adding an alternative implementation (like GCE encoding), audit ALL side effects of the original code path. The new path must replicate them or delegate to shared code.
**Architecture pattern**: The orchestrator pattern separates "what stages run" from "how each stage runs". Stages are always executed; only the implementation varies.
See `docs/archive/2026-01-04-video-worker-orchestrator-refactor.md` for full details.
## What We'd Do Differently
1. **Add Pydantic model fields test first** - Would have caught the silent field issue immediately
+8
View File
@@ -36,6 +36,14 @@
## Recent Changes
- **Video Worker Orchestrator** (2026-01-04): Major refactor to unify video generation pipeline. Created VideoWorkerOrchestrator that coordinates all stages (packaging, encoding, distribution, notifications) regardless of encoding backend (GCE or local). Fixes issue where GCE encoding path bypassed YouTube upload, Discord notifications, and CDG/TXT packaging. Feature flag `USE_NEW_ORCHESTRATOR` (default: true) enables rollback. 139 new tests across 6 new service modules. See [ARCHITECTURE.md](ARCHITECTURE.md#video-worker-orchestrator) and [archive/2026-01-04-video-worker-orchestrator-refactor.md](archive/2026-01-04-video-worker-orchestrator-refactor.md).
- **Full Unicode Font Support** (2026-01-03): Fixed rendering of musical symbols (♪) and added comprehensive international font support to Docker base image. Installed Noto fonts covering Latin, CJK (Chinese/Japanese/Korean), Arabic, Hebrew, Thai, and other scripts. Changed default karaoke font from Arial to Noto Sans. See [LESSONS-LEARNED.md](LESSONS-LEARNED.md#fonts-in-docker-for-video-rendering).
- **Flacfetch Cache API Integration** (2026-01-03): Enhanced audio search cache management to also clear flacfetch's GCS cache. When admins clear a job's cache, both Firestore (job state) and flacfetch (GCS search results) are cleared simultaneously. Added "Clear All Cache" button and cache stats display to admin UI. New endpoints: `DELETE /api/admin/cache`, `GET /api/admin/cache/stats`. See [API.md](API.md#audio-search-management-admin).
- **Audio Search Cache Management** (2026-01-03): Added admin UI at `/admin/searches` to view and manage cached audio search results. Admins can clear stale caches (e.g., when flacfetch is updated with new providers) and allow users to re-search. Fixes issue where jobs showed only YouTube results when lossless sources should be available. See [API.md](API.md#audio-search-management-admin) and [LESSONS-LEARNED.md](LESSONS-LEARNED.md#firestore-cache-has-no-automatic-invalidation).
- **User Database Separation** (2026-01-03): Migrated karaoke-gen users from shared `users` collection to dedicated `gen_users` collection. This separates karaoke-gen user data from karaoke-decide user data (both apps share the same GCP project/Firestore instance). Migration script in `scripts/migrate_users_to_gen_users.py`. See [ARCHITECTURE.md](ARCHITECTURE.md#firestore-collections) for collection details.
- **Admin Dashboard** (2026-01-03): Added comprehensive admin dashboard at `/admin` with: stats overview (users, jobs, credits), user management (search, sort, add credits, toggle role, enable/disable), job browser (filter by status/user, view details, delete), and beta program monitoring. Query-param based routing for static export compatibility. See [API.md](API.md#admin-endpoints) for backend endpoints.
@@ -0,0 +1,101 @@
# Video Worker Orchestrator Refactor - 2026-01-04
## Summary
Major refactor to unify video generation pipeline with a new VideoWorkerOrchestrator. This resolves the GCE vs KaraokeFinalise code path divergence that caused features like YouTube upload, Discord notifications, and CDG/TXT packaging to be missing from the GCE encoding path.
## Problem
Job 5b78ad46 completed successfully via GCE encoding but had no YouTube URL. Investigation revealed:
1. The GCE encoding path was a completely separate code branch
2. When `USE_GCE_ENCODING=true`, the worker triggered GCE encoding and returned immediately
3. This bypassed the entire `KaraokeFinalise` class which contained:
- YouTube upload logic
- Discord webhook notifications
- CDG/TXT package generation
- Dropbox/Google Drive uploads
The root cause wasn't a bug in any individual feature - it was an architectural issue where alternative implementations didn't share common post-processing stages.
## Solution
Created a stage-based orchestrator pattern:
```
VideoWorkerOrchestrator
├── Stage 1: Packaging (CDG, TXT)
├── Stage 2: Encoding (Local or GCE - swappable)
├── Stage 3: Organization (file paths, structure)
├── Stage 4: Distribution (Dropbox, GDrive, YouTube)
└── Stage 5: Notifications (Discord)
```
The orchestrator always runs all stages. Only the encoding implementation varies based on backend selection.
## Changes
### New Service Modules
| File | Lines | Purpose |
|------|-------|---------|
| `youtube_upload_service.py` | ~445 | YouTube OAuth and upload |
| `discord_service.py` | ~172 | Discord webhook notifications |
| `packaging_service.py` | ~287 | CDG/TXT package generation |
| `local_encoding_service.py` | ~590 | FFmpeg encoding on Cloud Run |
| `encoding_interface.py` | ~418 | Abstract interface + GCE backend |
| `video_worker_orchestrator.py` | ~628 | Stage coordination |
### Test Coverage
| Test File | Tests | Coverage |
|-----------|-------|----------|
| `test_youtube_upload_service.py` | 25 | OAuth, upload, error handling |
| `test_discord_service.py` | 18 | Webhook formatting, delivery |
| `test_packaging_service.py` | 22 | CDG/TXT generation |
| `test_local_encoding_service.py` | 24 | FFmpeg encoding variants |
| `test_encoding_interface.py` | 23 | Interface + GCE backend |
| `test_video_worker_orchestrator.py` | 27 | Full pipeline, stages |
| **Total** | **139** | |
### video_worker.py Updates
- Added `USE_NEW_ORCHESTRATOR` feature flag (default: `true`)
- `generate_video()` routes to orchestrator or legacy path
- Legacy code preserved as `generate_video_legacy()` for rollback
- New `generate_video_orchestrated()` builds config and runs orchestrator
## Feature Flag
```bash
# Enable new orchestrator (default)
USE_NEW_ORCHESTRATOR=true
# Rollback to legacy behavior
USE_NEW_ORCHESTRATOR=false
```
## Key Design Decisions
1. **Stage-based orchestration**: Each stage is independent and testable
2. **Strategy pattern for encoding**: `EncodingBackend` interface with local and GCE implementations
3. **Lazy service loading**: Services instantiated only when needed via `_get_*()` methods
4. **Dataclass configs**: `OrchestratorConfig` and `OrchestratorResult` for clear contracts
5. **Feature flag default true**: New code active immediately since tests are comprehensive
## Testing Notes
- All 976 existing tests continue to pass
- 139 new tests added across 6 test files
- E2E verification: Jobs 501258e1 and d21e6ef0 successfully processed through new pipeline
- Both jobs reached `awaiting_instrumental_selection` as expected (requires user input to complete)
## PR
- PR #182: https://github.com/nomadkaraoke/karaoke-gen/pull/182
- Merged: 2026-01-04
- Version: 0.89.0
## Lessons Learned
See [LESSONS-LEARNED.md](../LESSONS-LEARNED.md#alternative-code-paths-must-implement-all-features) for the key architectural lesson about alternative code paths.
+2
View File
@@ -46,6 +46,8 @@ function AdminBreadcrumb() {
breadcrumbs.push({ label: "Jobs", href: "/admin/jobs" })
breadcrumbs.push({ label: decodeURIComponent(segments[2]) })
}
} else if (segments[1] === "searches") {
breadcrumbs.push({ label: "Audio Searches" })
} else if (segments[1] === "beta") {
breadcrumbs.push({ label: "Beta Program" })
}
+473
View File
@@ -0,0 +1,473 @@
"use client"
import { useEffect, useState, useCallback } from "react"
import { adminApi, AudioSearchJobSummary, CacheStatsResponse } from "@/lib/api"
import { Button } from "@/components/ui/button"
import { Badge } from "@/components/ui/badge"
import {
Table,
TableBody,
TableCell,
TableHead,
TableHeader,
TableRow,
} from "@/components/ui/table"
import {
Select,
SelectContent,
SelectItem,
SelectTrigger,
SelectValue,
} from "@/components/ui/select"
import {
AlertDialog,
AlertDialogAction,
AlertDialogCancel,
AlertDialogContent,
AlertDialogDescription,
AlertDialogFooter,
AlertDialogHeader,
AlertDialogTitle,
} from "@/components/ui/alert-dialog"
import {
Tooltip,
TooltipContent,
TooltipProvider,
TooltipTrigger,
} from "@/components/ui/tooltip"
import {
RefreshCw,
Loader2,
Trash2,
Music,
CheckCircle,
XCircle,
ExternalLink,
Database,
HardDrive,
} from "lucide-react"
import { useToast } from "@/hooks/use-toast"
const statusOptions = [
{ value: "all", label: "All Statuses" },
{ value: "awaiting_audio_selection", label: "Awaiting Selection" },
{ value: "pending", label: "Pending" },
{ value: "complete", label: "Complete" },
{ value: "failed", label: "Failed" },
]
export default function AdminSearchesPage() {
const { toast } = useToast()
const [searches, setSearches] = useState<AudioSearchJobSummary[]>([])
const [loading, setLoading] = useState(true)
const [statusFilter, setStatusFilter] = useState("all")
// Clear cache dialog (single job)
const [clearDialogOpen, setClearDialogOpen] = useState(false)
const [jobToClear, setJobToClear] = useState<AudioSearchJobSummary | null>(null)
const [clearing, setClearing] = useState(false)
// Clear all cache dialog
const [clearAllDialogOpen, setClearAllDialogOpen] = useState(false)
const [clearingAll, setClearingAll] = useState(false)
// Cache stats
const [cacheStats, setCacheStats] = useState<CacheStatsResponse | null>(null)
const [statsLoading, setStatsLoading] = useState(false)
const loadSearches = useCallback(async () => {
try {
setLoading(true)
const data = await adminApi.listAudioSearches({
limit: 100,
status_filter: statusFilter !== "all" ? statusFilter : undefined,
})
setSearches(data.jobs)
} catch (err: any) {
console.error("Failed to load searches:", err)
toast({
title: "Error",
description: err.message || "Failed to load audio searches",
variant: "destructive",
})
} finally {
setLoading(false)
}
}, [statusFilter, toast])
const loadCacheStats = useCallback(async () => {
try {
setStatsLoading(true)
const stats = await adminApi.getCacheStats()
setCacheStats(stats)
} catch (err: any) {
// Silently fail - stats are optional
console.error("Failed to load cache stats:", err)
setCacheStats(null)
} finally {
setStatsLoading(false)
}
}, [])
useEffect(() => {
loadSearches()
loadCacheStats()
}, [loadSearches, loadCacheStats])
const handleClearCache = async () => {
if (!jobToClear) return
try {
setClearing(true)
const result = await adminApi.clearAudioSearchCache(jobToClear.job_id)
toast({
title: "Cache Cleared",
description: result.message,
})
setClearDialogOpen(false)
setJobToClear(null)
loadSearches()
loadCacheStats()
} catch (err: any) {
toast({
title: "Error",
description: err.message || "Failed to clear cache",
variant: "destructive",
})
} finally {
setClearing(false)
}
}
const handleClearAllCache = async () => {
try {
setClearingAll(true)
const result = await adminApi.clearAllCache()
toast({
title: "All Cache Cleared",
description: result.message,
})
setClearAllDialogOpen(false)
loadCacheStats()
} catch (err: any) {
toast({
title: "Error",
description: err.message || "Failed to clear all cache",
variant: "destructive",
})
} finally {
setClearingAll(false)
}
}
const formatBytes = (bytes: number): string => {
if (bytes === 0) return "0 B"
const k = 1024
const sizes = ["B", "KB", "MB", "GB"]
const i = Math.floor(Math.log(bytes) / Math.log(k))
return parseFloat((bytes / Math.pow(k, i)).toFixed(1)) + " " + sizes[i]
}
const getStatusVariant = (status: string) => {
if (status === "complete" || status === "prep_complete") return "default"
if (status === "failed") return "destructive"
if (status === "cancelled") return "outline"
if (status.includes("awaiting")) return "secondary"
return "secondary"
}
const formatDate = (dateStr?: string) => {
if (!dateStr) return "—"
return new Date(dateStr).toLocaleString()
}
const getProviderBadgeClass = (provider: string) => {
switch (provider.toLowerCase()) {
case "youtube":
return "bg-red-500/20 text-red-400 border-red-500/30"
case "red":
return "bg-red-600/30 text-red-300 border-red-500/30"
case "ops":
return "bg-blue-500/20 text-blue-400 border-blue-500/30"
default:
return "bg-slate-500/20 text-slate-400 border-slate-500/30"
}
}
return (
<div className="space-y-6">
{/* Header */}
<div className="flex items-center justify-between">
<div>
<h1 className="text-2xl font-bold tracking-tight">Audio Searches</h1>
<p className="text-muted-foreground">
View and manage cached audio search results
</p>
</div>
<div className="flex items-center gap-2">
<Button variant="outline" size="sm" onClick={() => { loadSearches(); loadCacheStats() }} disabled={loading}>
<RefreshCw className={`w-4 h-4 mr-2 ${loading ? "animate-spin" : ""}`} />
Refresh
</Button>
<TooltipProvider>
<Tooltip>
<TooltipTrigger asChild>
<Button
variant="outline"
size="sm"
className="text-red-500 hover:text-red-400 hover:bg-red-500/10 border-red-500/30"
onClick={() => setClearAllDialogOpen(true)}
>
<Database className="w-4 h-4 mr-2" />
Clear All Cache
</Button>
</TooltipTrigger>
<TooltipContent>
<p>Clear entire flacfetch search cache</p>
</TooltipContent>
</Tooltip>
</TooltipProvider>
</div>
</div>
{/* Cache Stats */}
{cacheStats && (
<div className="flex items-center gap-4 text-sm text-muted-foreground bg-muted/30 rounded-lg px-4 py-2">
<div className="flex items-center gap-2">
<HardDrive className="w-4 h-4" />
<span>Flacfetch Cache:</span>
</div>
<Badge variant="outline" className="font-mono">
{cacheStats.count} entries
</Badge>
<Badge variant="outline" className="font-mono">
{formatBytes(cacheStats.total_size_bytes)}
</Badge>
{cacheStats.newest_entry && (
<span className="text-xs">
Latest: {new Date(cacheStats.newest_entry).toLocaleDateString()}
</span>
)}
</div>
)}
{/* Filters */}
<div className="flex gap-4">
<Select value={statusFilter} onValueChange={setStatusFilter}>
<SelectTrigger className="w-[200px]">
<SelectValue placeholder="Filter by status" />
</SelectTrigger>
<SelectContent>
{statusOptions.map((opt) => (
<SelectItem key={opt.value} value={opt.value}>
{opt.label}
</SelectItem>
))}
</SelectContent>
</Select>
</div>
{/* Table */}
<div className="rounded-md border">
<Table>
<TableHeader>
<TableRow>
<TableHead>Job ID</TableHead>
<TableHead>Search Query</TableHead>
<TableHead>Results</TableHead>
<TableHead>Providers</TableHead>
<TableHead>Status</TableHead>
<TableHead>User</TableHead>
<TableHead>Created</TableHead>
<TableHead className="text-right">Actions</TableHead>
</TableRow>
</TableHeader>
<TableBody>
{loading ? (
<TableRow>
<TableCell colSpan={8} className="text-center py-8">
<Loader2 className="w-6 h-6 animate-spin mx-auto text-muted-foreground" />
</TableCell>
</TableRow>
) : searches.length === 0 ? (
<TableRow>
<TableCell colSpan={8} className="text-center py-8 text-muted-foreground">
<Music className="w-8 h-8 mx-auto mb-2 opacity-50" />
No audio searches found
</TableCell>
</TableRow>
) : (
searches.map((search) => (
<TableRow key={search.job_id}>
<TableCell className="font-mono text-xs">
<a
href={`/admin/jobs?id=${search.job_id}`}
className="text-blue-500 hover:underline"
>
{search.job_id.slice(0, 8)}...
</a>
</TableCell>
<TableCell>
<div className="max-w-[200px]">
<div className="font-medium truncate">
{search.audio_search_artist || "—"}
</div>
<div className="text-sm text-muted-foreground truncate">
{search.audio_search_title || "—"}
</div>
</div>
</TableCell>
<TableCell>
<TooltipProvider>
<Tooltip>
<TooltipTrigger asChild>
<div className="flex items-center gap-2">
<span className="font-medium">{search.results_count}</span>
{search.has_lossless ? (
<CheckCircle className="w-4 h-4 text-green-500" />
) : (
<XCircle className="w-4 h-4 text-red-400" />
)}
</div>
</TooltipTrigger>
<TooltipContent>
<p>{search.has_lossless ? "Has lossless sources" : "No lossless sources (YouTube only)"}</p>
</TooltipContent>
</Tooltip>
</TooltipProvider>
</TableCell>
<TableCell>
<div className="flex flex-wrap gap-1">
{search.providers.map((provider) => (
<Badge
key={provider}
variant="outline"
className={`text-[10px] ${getProviderBadgeClass(provider)}`}
>
{provider}
</Badge>
))}
</div>
</TableCell>
<TableCell>
<Badge variant={getStatusVariant(search.status)}>
{search.status.replace(/_/g, " ")}
</Badge>
</TableCell>
<TableCell className="text-xs text-muted-foreground">
{search.user_email?.split("@")[0] || "—"}
</TableCell>
<TableCell className="text-xs text-muted-foreground">
{formatDate(search.created_at)}
</TableCell>
<TableCell className="text-right">
<TooltipProvider>
<Tooltip>
<TooltipTrigger asChild>
<Button
variant="ghost"
size="icon"
className="h-8 w-8 text-orange-500 hover:text-orange-400 hover:bg-orange-500/10"
onClick={() => {
setJobToClear(search)
setClearDialogOpen(true)
}}
>
<Trash2 className="w-4 h-4" />
</Button>
</TooltipTrigger>
<TooltipContent>
<p>Clear cache & allow re-search</p>
</TooltipContent>
</Tooltip>
</TooltipProvider>
</TableCell>
</TableRow>
))
)}
</TableBody>
</Table>
</div>
{/* Summary */}
{!loading && searches.length > 0 && (
<div className="text-sm text-muted-foreground">
Showing {searches.length} jobs with cached search results.
{searches.filter(s => !s.has_lossless).length > 0 && (
<span className="ml-2 text-orange-400">
{searches.filter(s => !s.has_lossless).length} have YouTube-only results (may need cache refresh).
</span>
)}
</div>
)}
{/* Clear Cache Dialog (single job) */}
<AlertDialog open={clearDialogOpen} onOpenChange={setClearDialogOpen}>
<AlertDialogContent>
<AlertDialogHeader>
<AlertDialogTitle>Clear Search Cache?</AlertDialogTitle>
<AlertDialogDescription>
This will clear the cached search results for job{" "}
<span className="font-mono">{jobToClear?.job_id}</span> and reset it to pending
status, allowing a new search to be performed.
{jobToClear && !jobToClear.has_lossless && (
<span className="block mt-2 text-orange-400">
This job only has YouTube results - clearing the cache may fix this if flacfetch has been updated.
</span>
)}
</AlertDialogDescription>
</AlertDialogHeader>
<AlertDialogFooter>
<AlertDialogCancel>Cancel</AlertDialogCancel>
<AlertDialogAction
onClick={handleClearCache}
disabled={clearing}
className="bg-orange-600 hover:bg-orange-500"
>
{clearing ? (
<Loader2 className="w-4 h-4 animate-spin mr-2" />
) : (
<Trash2 className="w-4 h-4 mr-2" />
)}
Clear Cache
</AlertDialogAction>
</AlertDialogFooter>
</AlertDialogContent>
</AlertDialog>
{/* Clear All Cache Dialog */}
<AlertDialog open={clearAllDialogOpen} onOpenChange={setClearAllDialogOpen}>
<AlertDialogContent>
<AlertDialogHeader>
<AlertDialogTitle>Clear Entire Flacfetch Cache?</AlertDialogTitle>
<AlertDialogDescription>
This will delete all cached search results from flacfetch&apos;s GCS cache
{cacheStats && (
<span className="font-medium"> ({cacheStats.count} entries, {formatBytes(cacheStats.total_size_bytes)})</span>
)}.
<span className="block mt-2 text-orange-400">
All subsequent searches will hit the trackers fresh. This does NOT affect
the Firestore job caches - use individual &quot;Clear Cache&quot; buttons for those.
</span>
</AlertDialogDescription>
</AlertDialogHeader>
<AlertDialogFooter>
<AlertDialogCancel>Cancel</AlertDialogCancel>
<AlertDialogAction
onClick={handleClearAllCache}
disabled={clearingAll}
className="bg-red-600 hover:bg-red-500"
>
{clearingAll ? (
<Loader2 className="w-4 h-4 animate-spin mr-2" />
) : (
<Database className="w-4 h-4 mr-2" />
)}
Clear All Cache
</AlertDialogAction>
</AlertDialogFooter>
</AlertDialogContent>
</AlertDialog>
</div>
)
}
@@ -45,7 +45,7 @@ describe('AuthStatus', () => {
// Should show user email and credits
expect(screen.getByText('test@example.com')).toBeInTheDocument()
expect(screen.getByText('5')).toBeInTheDocument()
expect(screen.getByText('5 credits')).toBeInTheDocument()
})
it('opens auth dialog when Login is clicked', () => {
@@ -9,6 +9,7 @@ import {
TestTube2,
ArrowLeft,
Settings,
Search,
} from "lucide-react"
import {
@@ -41,6 +42,11 @@ const navItems = [
href: "/admin/jobs",
icon: Briefcase,
},
{
title: "Audio Searches",
href: "/admin/searches",
icon: Search,
},
{
title: "Beta Program",
href: "/admin/beta",
+24 -2
View File
@@ -12,7 +12,7 @@ import {
DropdownMenuSeparator,
DropdownMenuTrigger,
} from "./ui/dropdown-menu"
import { User, LogOut, Settings } from "lucide-react"
import { User, LogOut, Settings, Shield, Mail } from "lucide-react"
import Link from "next/link"
export function AppHeader() {
@@ -40,7 +40,7 @@ export function AppHeader() {
<Link href="/jobs" className="text-sm font-medium hover:text-primary transition-colors">
My Jobs
</Link>
{user.role === "admin" && (
{(user.role === "admin" || user.email?.endsWith("@nomadkaraoke.com")) && (
<Link href="/admin" className="text-sm font-medium hover:text-primary transition-colors">
Admin
</Link>
@@ -48,6 +48,14 @@ export function AppHeader() {
</nav>
<div className="flex items-center gap-4">
<a
href="mailto:andrew@nomadkaraoke.com"
className="hidden lg:flex items-center gap-1.5 text-xs text-muted-foreground hover:text-foreground transition-colors"
>
<Mail className="w-3.5 h-3.5" />
<span>Need help?</span>
</a>
<div className="hidden sm:flex items-center gap-2 px-4 py-2 rounded-lg bg-primary/10 border border-primary/20">
<div className="w-2 h-2 rounded-full bg-primary animate-pulse" />
<span className="text-sm font-medium text-primary">{user.credits} credits</span>
@@ -73,10 +81,24 @@ export function AppHeader() {
<span>{user.credits} credits</span>
</div>
</DropdownMenuItem>
{(user.role === "admin" || user.email?.endsWith("@nomadkaraoke.com")) && (
<DropdownMenuItem asChild>
<Link href="/admin">
<Shield className="w-4 h-4 mr-2" />
Admin Dashboard
</Link>
</DropdownMenuItem>
)}
<DropdownMenuItem>
<Settings className="w-4 h-4 mr-2" />
Settings
</DropdownMenuItem>
<DropdownMenuItem asChild>
<a href="mailto:andrew@nomadkaraoke.com">
<Mail className="w-4 h-4 mr-2" />
Need help? Contact us
</a>
</DropdownMenuItem>
<DropdownMenuSeparator />
<DropdownMenuItem onClick={handleLogout} className="text-destructive focus:text-destructive">
<LogOut className="w-4 h-4 mr-2" />
+14 -2
View File
@@ -1,7 +1,8 @@
"use client"
import { useState, useEffect } from "react"
import { User, LogOut, CreditCard, Coins, KeyRound } from "lucide-react"
import { User, LogOut, CreditCard, Coins, KeyRound, Shield } from "lucide-react"
import Link from "next/link"
import { Button } from "@/components/ui/button"
import {
DropdownMenu,
@@ -69,7 +70,7 @@ export function AuthStatus({ onAuthChange }: AuthStatusProps) {
</span>
<span className="flex items-center gap-1 text-amber-400 font-medium">
<Coins className="w-3 h-3" />
{user.credits}
{user.credits} {user.credits === 1 ? 'credit' : 'credits'}
</span>
</Button>
</DropdownMenuTrigger>
@@ -97,6 +98,17 @@ export function AuthStatus({ onAuthChange }: AuthStatusProps) {
<CreditCard className="w-4 h-4 mr-2" />
<span>Buy More Credits</span>
</DropdownMenuItem>
{(user.role === "admin" || user.email?.endsWith("@nomadkaraoke.com")) && (
<>
<DropdownMenuSeparator className="bg-slate-700" />
<DropdownMenuItem asChild className="text-slate-300 focus:text-white focus:bg-slate-800">
<Link href="/admin">
<Shield className="w-4 h-4 mr-2" />
<span>Admin Dashboard</span>
</Link>
</DropdownMenuItem>
</>
)}
<DropdownMenuSeparator className="bg-slate-700" />
<DropdownMenuItem
onClick={handleLogout}
+6 -6
View File
@@ -272,8 +272,8 @@ export function JobSubmission({ onJobCreated }: JobSubmissionProps) {
Skip lyrics review (non-interactive)
</Label>
<p className="text-xs mt-1" style={{ color: 'var(--text-muted)' }}>
<span className="text-amber-500 font-medium">Warning:</span> AI lyrics correction isn&apos;t perfect.
Skipping review will likely result in timing errors and incorrect words in your karaoke video.
<span className="text-amber-500 font-medium">Warning:</span> auto lyrics correction isn&apos;t perfect.
Skipping review will likely result in incorrect words in your karaoke video.
</p>
</div>
</div>
@@ -377,8 +377,8 @@ export function JobSubmission({ onJobCreated }: JobSubmissionProps) {
Skip lyrics review (non-interactive)
</Label>
<p className="text-xs mt-1" style={{ color: 'var(--text-muted)' }}>
<span className="text-amber-500 font-medium">Warning:</span> AI lyrics correction isn&apos;t perfect.
Skipping review will likely result in timing errors and incorrect words in your karaoke video.
<span className="text-amber-500 font-medium">Warning:</span> auto lyrics correction isn&apos;t perfect.
Skipping review will likely result in incorrect words in your karaoke video.
</p>
</div>
</div>
@@ -465,8 +465,8 @@ export function JobSubmission({ onJobCreated }: JobSubmissionProps) {
Skip lyrics review (non-interactive)
</Label>
<p className="text-xs mt-1" style={{ color: 'var(--text-muted)' }}>
<span className="text-amber-500 font-medium">Warning:</span> AI lyrics correction isn&apos;t perfect.
Skipping review will likely result in timing errors and incorrect words in your karaoke video.
<span className="text-amber-500 font-medium">Warning:</span> auto lyrics correction isn&apos;t perfect.
Skipping review will likely result in incorrect words in your karaoke video.
</p>
</div>
</div>
@@ -453,8 +453,8 @@ test.describe('Production E2E - Full User Journey', () => {
jobId,
['awaiting_instrumental_selection', 'complete'],
accessToken!,
TIMEOUTS.jobProcessing // Video rendering can take several minutes
).catch(() => ({ status: 'unknown', job: null }));
TIMEOUTS.download // Video rendering can take 10-15 minutes with orchestrator
);
if (statusBeforeInstrumental === 'awaiting_instrumental_selection') {
console.log(' Selecting clean instrumental via API...');
@@ -11,7 +11,7 @@ import { createEmailHelper, isEmailTestingAvailable } from '../helpers/email-tes
*
* Test Flow:
* 1. Landing Page - Navigate and verify
* 2. New User Signup - Create MailSlurp inbox, submit beta form, verify email
* 2. New User Signup - Create testmail.app inbox, submit beta form, verify email
* 3. Magic Link Auth - Click magic link to authenticate
* 4. Create Job - Search for test song via UI
* 5. Audio Selection - Select audio source via UI dialog
@@ -24,7 +24,7 @@ import { createEmailHelper, isEmailTestingAvailable } from '../helpers/email-tes
* 12. Cleanup - Delete distributed content and job
*
* Environment Variables:
* - MAILSLURP_API_KEY: Required for email testing (can be skipped with E2E_TEST_TOKEN)
* - TESTMAIL_API_KEY + TESTMAIL_NAMESPACE: Required for email testing (can be skipped with E2E_TEST_TOKEN)
* - E2E_TEST_TOKEN: Pre-configured user token to skip signup flow (for iterations)
* - E2E_ADMIN_TOKEN: Required for cleanup endpoint (optional - cleanup skipped if not set)
*/
@@ -138,12 +138,12 @@ test.describe('E2E Happy Path - Real User with Full UI Interactions', () => {
}) => {
test.setTimeout(TIMEOUTS.fullTest);
// Check if we have a pre-configured token (for iterations without using MailSlurp)
// Check if we have a pre-configured token (for iterations without using testmail.app)
const usePreConfiguredToken = hasPreConfiguredToken();
// Skip if neither pre-configured token nor MailSlurp is available
// Skip if neither pre-configured token nor testmail.app is available
if (!usePreConfiguredToken && !isEmailTestingAvailable()) {
test.skip(true, 'Neither E2E_TEST_TOKEN nor MAILSLURP_API_KEY set - cannot run test');
test.skip(true, 'Neither E2E_TEST_TOKEN nor TESTMAIL_API_KEY/TESTMAIL_NAMESPACE set - cannot run test');
return;
}
@@ -217,7 +217,7 @@ test.describe('E2E Happy Path - Real User with Full UI Interactions', () => {
console.log('STEP 3 COMPLETE: User authenticated with pre-configured token');
} else {
// ----- FULL PATH: Beta signup with MailSlurp -----
// ----- FULL PATH: Beta signup with testmail.app -----
console.log('\n========================================');
console.log('STEP 2: New User Beta Enrollment');
console.log('========================================');
+105
View File
@@ -782,6 +782,60 @@ export interface AdminJobListParams {
limit?: number;
}
// Audio Search Admin Types
export interface AudioSearchResultSummary {
index: number;
provider: string;
artist: string;
title: string;
is_lossless: boolean;
quality?: string;
seeders?: number;
}
export interface AudioSearchJobSummary {
job_id: string;
status: string;
user_email?: string;
audio_search_artist?: string;
audio_search_title?: string;
created_at?: string;
results_count: number;
results_summary: AudioSearchResultSummary[];
has_lossless: boolean;
providers: string[];
}
export interface AudioSearchListResponse {
jobs: AudioSearchJobSummary[];
total: number;
}
export interface ClearSearchCacheResponse {
status: string;
job_id: string;
message: string;
previous_status: string;
new_status: string;
results_cleared: number;
flacfetch_cache_cleared: boolean;
flacfetch_error?: string;
}
export interface ClearAllCacheResponse {
status: string;
message: string;
deleted_count: number;
}
export interface CacheStatsResponse {
count: number;
total_size_bytes: number;
oldest_entry?: string;
newest_entry?: string;
configured: boolean;
}
// Admin API namespace
export const adminApi = {
/**
@@ -933,6 +987,57 @@ export const adminApi = {
);
return handleResponse(response);
},
/**
* List jobs with audio search results
*/
async listAudioSearches(params?: {
limit?: number;
status_filter?: string;
}): Promise<AudioSearchListResponse> {
const searchParams = new URLSearchParams();
if (params?.limit) searchParams.set('limit', String(params.limit));
if (params?.status_filter) searchParams.set('status_filter', params.status_filter);
const url = `${API_BASE_URL}/api/admin/audio-searches${searchParams.toString() ? '?' + searchParams.toString() : ''}`;
const response = await fetch(url, {
headers: getAuthHeaders()
});
return handleResponse(response);
},
/**
* Clear audio search cache for a job
*/
async clearAudioSearchCache(jobId: string): Promise<ClearSearchCacheResponse> {
const response = await fetch(
`${API_BASE_URL}/api/admin/audio-searches/${jobId}/clear-cache`,
{ method: 'POST', headers: getAuthHeaders() }
);
return handleResponse(response);
},
/**
* Clear all flacfetch search cache
*/
async clearAllCache(): Promise<ClearAllCacheResponse> {
const response = await fetch(
`${API_BASE_URL}/api/admin/cache`,
{ method: 'DELETE', headers: getAuthHeaders() }
);
return handleResponse(response);
},
/**
* Get flacfetch cache statistics
*/
async getCacheStats(): Promise<CacheStatsResponse> {
const response = await fetch(
`${API_BASE_URL}/api/admin/cache/stats`,
{ headers: getAuthHeaders() }
);
return handleResponse(response);
},
};
export { ApiError };
+1 -1
View File
@@ -1,6 +1,6 @@
{
"name": "karaoke-gen-frontend",
"version": "0.79.0",
"version": "0.80.0",
"private": true,
"scripts": {
"build": "next build",
+1 -1
View File
@@ -1439,7 +1439,7 @@ async def process_job(job_id: str, request: EncodeRequest):
# Convert local paths to blob paths (backend expects blob paths, not full gs:// URIs)
# output_gcs_path is like "gs://bucket/jobs/id/encoded/"
# We need paths like "jobs/id/encoded/output_4k.mp4"
# We need paths like "jobs/id/encoded/output_4k_lossless.mp4"
gcs_path = request.output_gcs_path.replace("gs://", "")
parts = gcs_path.split("/", 1)
prefix = parts[1].rstrip("/") if len(parts) > 1 else ""
+3 -1
View File
@@ -70,7 +70,9 @@ DEFAULT_KARAOKE_STYLE = {
"background_color": "#000000",
"background_image": None,
# Font settings
"font": "Arial",
# Using "Noto Sans" for full Unicode support (CJK, Arabic, symbols, etc.)
# Requires fonts-noto-* packages installed in Docker image
"font": "Noto Sans",
"font_path": "", # Must be string, not None (for ASS generator)
"ass_name": "Default",
# Colors in "R, G, B, A" format (required by ASS)
+1 -1
View File
@@ -1,6 +1,6 @@
[tool.poetry]
name = "karaoke-gen"
version = "0.87.1"
version = "0.89.3"
description = "Generate karaoke videos with synchronized lyrics. Handles the entire process from downloading audio and lyrics to creating the final video with title screens."
authors = ["Andrew Beveridge <andrew@beveridge.uk>"]
license = "MIT"