Compare commits

..
Author SHA1 Message Date
Andrew BeveridgeandClaude Opus 4.5 97f1a7640b feat: add production debug script template for agents
Add infrastructure for ad-hoc production testing:
- Debug script template at frontend/e2e/helpers/debug-prod-template.mjs
- Gitignore patterns for test-*.local.* and debug-*.local.* files
- Documentation in CLAUDE.md, TESTING.md, and DEVELOPMENT.md

This allows agents to quickly test production issues when asked to
"test it yourself in prod" without risking token exposure in commits.

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
2026-01-20 10:15:25 -08:00
519 changed files with 6410 additions and 615207 deletions

No files matched your search

-236
View File
@@ -1,236 +0,0 @@
# Plan: Combined Lyrics + Instrumental Review Phase
**Created:** 2026-01-24
**Branch:** feat/sess-20260124-1444-combined-review-flow
**Status:** Draft
## Overview
Combine the lyrics review and instrumental review phases into a single unified "Human Review" step. Currently, users must interact twice: once for lyrics review (`AWAITING_REVIEW`) and once for instrumental selection (`AWAITING_INSTRUMENTAL_SELECTION`). After this change, users will complete both reviews in a single session.
**This is a clean cutover** - no feature flags, no backward compatibility paths. The old two-step flow will be completely removed.
### Key Insight
By the time users enter lyrics review, **audio separation is already complete** - the instrumental stems (`instrumental_clean` and `instrumental_with_backing`) are ready. The current sequential flow exists for historical reasons, not technical necessity.
### Benefits
1. **Better UX** - One interaction point instead of two
2. **Faster turnaround** - Users complete both reviews in one session
3. **Reduced drop-off** - Users less likely to abandon between phases
4. **Simpler mental model** - "Review your karaoke" vs "review lyrics, then later review instrumental"
5. **Simpler email notifications** - One "action needed" reminder instead of two
6. **Cleaner codebase** - No conditional paths or legacy state handling
## Requirements
- [x] Functional: Users complete lyrics review AND instrumental selection in a single UI flow
- [x] Functional: Single email notification for "review needed" instead of two separate ones
- [x] Functional: Remove `AWAITING_INSTRUMENTAL_SELECTION` state from normal flow
- [x] Functional: Local CLI continues to work (uses `non_interactive=true`)
- [x] Functional: Admin "reset to review" action works with combined flow
- [x] Non-functional: No regression in existing test coverage
- [x] Non-functional: Clean removal of old code paths (no dead code)
## Technical Approach
### Strategy: "Combined Review - Clean Cutover"
**State Machine - Before:**
```
AWAITING_REVIEW → IN_REVIEW → REVIEW_COMPLETE → RENDERING_VIDEO
→ AWAITING_INSTRUMENTAL_SELECTION → INSTRUMENTAL_SELECTED → GENERATING_VIDEO
```
**State Machine - After:**
```
AWAITING_REVIEW → IN_REVIEW → REVIEW_COMPLETE → RENDERING_VIDEO
→ INSTRUMENTAL_SELECTED → GENERATING_VIDEO
```
The `AWAITING_INSTRUMENTAL_SELECTION` state is **removed from normal operation**. Instrumental selection is now required during lyrics review. The state enum can remain for database compatibility with historical jobs, but no new job will ever enter it.
### Key Changes
1. **Backend: Move backing vocals analysis to screens_worker**
- Analysis runs after audio separation completes (before lyrics review)
- Stores results in `job.state_data['backing_vocals_analysis']`
- Remove analysis from render_video_worker entirely
2. **Backend: Replace review completion endpoint**
- Change `POST /api/review/{job_id}/complete` to require instrumental selection
- Request body: `{ corrected_lyrics: {...}, instrumental_selection: "clean" | "with_backing" }`
- Remove the old endpoint that didn't require instrumental
3. **Backend: Simplify render_video_worker**
- Always expect `instrumental_selection` in state_data
- Always transition to `INSTRUMENTAL_SELECTED` after rendering
- Remove all conditional logic for `AWAITING_INSTRUMENTAL_SELECTION`
4. **Frontend: Single combined review page**
- Combine LyricsAnalyzer and InstrumentalSelector into one flow
- Remove separate `/instrumental` route entirely
- Single "Complete Review" button submits both
5. **Backend: Remove instrumental-only endpoints**
- Remove `GET /api/jobs/{job_id}/instrumental-options`
- Remove `GET /api/jobs/{job_id}/instrumental-analysis`
- Remove `POST /api/jobs/{job_id}/select-instrumental`
- Keep custom instrumental creation for future advanced use
6. **Email notifications: Simplify**
- Remove "Instrumental selection needed" email trigger entirely
- Update "Review needed" email to mention instrumental selection
- Remove `instrumental_token` generation (no longer needed)
## Implementation Steps
### Phase 1: Backend Changes
1. [x] **Move backing vocals analysis to screens_worker**
- Move `_analyze_backing_vocals()` from `render_video_worker.py` to `screens_worker.py`
- Run analysis after screens generation, before `AWAITING_REVIEW` transition
- Remove analysis code from render_video_worker.py entirely
2. [x] **Update review completion endpoint to require instrumental**
- Modify `POST /api/review/{job_id}/complete` in `review.py`
- Add required `instrumental_selection` field to request body
- Store selection in state_data before transitioning to `REVIEW_COMPLETE`
3. [x] **Add instrumental data to correction-data endpoint**
- Modify `GET /api/review/{job_id}/correction-data` to include:
- `instrumental_options`: List of {id, label, audio_url}
- `backing_vocals_analysis`: Analysis data from state_data
4. [x] **Simplify render_video_worker**
- Remove conditional check for instrumental pre-selection
- Always transition directly to `INSTRUMENTAL_SELECTED`
- Always trigger video worker immediately after render
5. [x] **Update state machine**
- Remove `AWAITING_INSTRUMENTAL_SELECTION` from `STATE_TRANSITIONS` destinations
- Add `RENDERING_VIDEO → INSTRUMENTAL_SELECTED` as the only path
- Remove `RENDERING_VIDEO → AWAITING_INSTRUMENTAL_SELECTION` transition
6. [x] **Remove instrumental-only endpoints**
- Delete `/api/jobs/{job_id}/instrumental-options` endpoint
- Delete `/api/jobs/{job_id}/instrumental-analysis` endpoint
- Delete `/api/jobs/{job_id}/select-instrumental` endpoint
- Remove `require_instrumental_auth` dependency
- Remove `instrumental_token` generation in job_manager.py
### Phase 2: Frontend Changes
7. [x] **Create combined review page**
- Modified `ReviewChangesModal` to include InstrumentalSelectorEmbedded component
- Instrumental selection UI appears after preview video section
- Instrumental data comes from correction-data endpoint (via CorrectionData type)
8. [x] **Create InstrumentalSelectorEmbedded component**
- Created embedded version of InstrumentalSelector
- Accepts options, analysis, value, onChange props
- Supports compact mode for modal embedding
9. [x] **Update review completion flow**
- "Complete Review" button requires instrumental selection
- Validation error shown if no instrumental selected
- Single API call submits both lyrics and instrumental selection
10. [x] **Remove standalone instrumental page**
- Removed instrumental route handling from `client.tsx`
- Removed `getExpectedStates` case for "instrumental"
- InstrumentalSelector component stubbed out (deprecated)
- Kept `selectInstrumental` API for finalise-only jobs
11. [x] **Update lib/api.ts**
- Modified `completeReview()` to require instrumental selection
- Kept `selectInstrumental()` for finalise-only jobs
- Removed other instrumental-only functions
### Phase 3: Email & Notification Cleanup
12. [x] **Remove instrumental notification trigger**
- Keep for finalise-only jobs which still use AWAITING_INSTRUMENTAL_SELECTION
- Updated docstrings to clarify finalise-only vs normal flow
- Instrumental reminder scheduling still works for finalise-only jobs
13. [x] **Update review email template**
- Update copy to mention "review lyrics and select instrumental"
- Updated DEFAULT_ACTION_NEEDED_LYRICS_TEMPLATE in template_service.py
### Phase 4: Testing & Cleanup
14. [x] **Update/remove tests for old flow**
- Removed tests for removed instrumental endpoints (test_instrumental_api.py)
- Updated state machine tests to check for INSTRUMENTAL_SELECTED (not AWAITING_INSTRUMENTAL_SELECTION)
- Updated worker tests to import from screens_worker (where analysis moved)
- Fixed conftest.py mocks to return AuthResult objects
15. [x] **Add tests for combined flow**
- Existing tests already cover combined completion endpoint
- Test infrastructure validates combined review flow works
- 1741 tests pass (only 1 pre-existing unrelated failure)
16. [x] **Update documentation**
- Updated `docs/API.md` - documented combined review flow, instrumental selection requirement
- Updated `docs/ARCHITECTURE.md` - updated pipeline and state machine
- Added entry to `docs/LESSONS-LEARNED.md` - Combined Review Flow decision
- Updated `docs/README.md` - added Recent Changes entry, updated Current State table
17. [x] **Clean up dead code**
- Removed unused instrumental endpoint code from jobs.py
- Stubbed InstrumentalSelector component (deprecated)
- Kept select-instrumental endpoint for finalise-only jobs
## Files to Modify/Delete
| File | Action | Description |
|------|--------|-------------|
| `backend/models/job.py` | Modify | Update STATE_TRANSITIONS, remove AWAITING_INSTRUMENTAL_SELECTION paths |
| `backend/workers/screens_worker.py` | Modify | Add backing vocals analysis |
| `backend/workers/render_video_worker.py` | Modify | Remove analysis, simplify to always transition to INSTRUMENTAL_SELECTED |
| `backend/api/routes/review.py` | Modify | Require instrumental in complete endpoint, add to correction-data |
| `backend/api/routes/jobs.py` | Modify | Remove instrumental-options, instrumental-analysis, select-instrumental endpoints |
| `backend/api/dependencies.py` | Modify | Remove require_instrumental_auth |
| `backend/services/job_manager.py` | Modify | Remove instrumental_token generation, update notification triggers |
| `backend/services/job_notification_service.py` | Modify | Remove instrumental notification |
| `frontend/app/app/jobs/[[...slug]]/client.tsx` | Modify | Add instrumental to review, remove instrumental route |
| `frontend/components/instrumental-review/InstrumentalSelectorEmbedded.tsx` | Create | Embeddable selector component |
| `frontend/lib/api.ts` | Modify | Update completeReview, remove instrumental functions |
## Testing Strategy
### Unit Tests
- `test_combined_completion_endpoint`: Test endpoint requires instrumental selection
- `test_screens_worker_with_analysis`: Test backing vocals analysis runs during screens
- `test_render_video_worker_direct_transition`: Test always goes to INSTRUMENTAL_SELECTED
- `test_state_transitions_no_awaiting_instrumental`: Verify old state unreachable
### Integration Tests
- `test_combined_review_flow`: Full flow from AWAITING_REVIEW to GENERATING_VIDEO
- `test_non_interactive_job`: CLI jobs skip review entirely as before
### E2E Tests
- `test_combined_review_ui`: User completes both reviews in single session
- `test_instrumental_route_rejected`: Verify /instrumental URL shows error
## Risks & Mitigations
| Risk | Mitigation |
|------|------------|
| Jobs currently in AWAITING_INSTRUMENTAL_SELECTION | Admin can manually reset to AWAITING_REVIEW |
| CLI regression | Test `non_interactive` flag behavior explicitly |
| Analysis adds latency to screens worker | Analysis is fast (~2-5s), acceptable tradeoff |
## Migration Notes
**For any jobs stuck in `AWAITING_INSTRUMENTAL_SELECTION` after deploy:**
1. Admin resets job to `AWAITING_REVIEW` via admin panel
2. User completes combined review
3. Job proceeds normally
This is acceptable because:
- Very few jobs should be in this state at any time
- Admin reset is already a supported operation
- Clean cutover is worth the minor migration friction
@@ -1,321 +0,0 @@
# Plan: Consolidate YouTube Download Paths
**Created:** 2026-01-20
**Status:** Planning
## Problem Statement
YouTube downloads currently have **3 different code paths** with different behaviors:
1. **Audio Search → Select YouTube** (`audio_search.py`)
- Can use remote flacfetch, local flacfetch, OR direct `download_from_url()`
- Multiple conditional branches with subtle bugs (e.g., `source_id` extraction)
2. **Direct URL Submission** (`file_upload.py` → `audio_worker.py`)
- Always uses `download_from_url()` → `FileHandler.download_video()` → yt_dlp
- Runs on Cloud Run, blocked by YouTube bot detection
- Never uses remote flacfetch even when configured
3. **Local CLI** (`karaoke_gen/audio_fetcher.py`)
- Uses `FlacFetcher.download_from_url()` → local yt_dlp
- Works fine locally, not relevant to cloud issues
### Current Issues
1. **Direct URL path ignores remote flacfetch** - Users submitting YouTube URLs directly hit bot detection because Cloud Run IPs are blocked
2. **Multiple implementations** - `FileHandler.download_video()`, `download_from_url()`, `FlacFetcher.download_from_url()` all do similar things
3. **Inconsistent error handling** - Different paths handle errors differently
4. **Testing gaps** - No integration tests verifying the full flow
5. **Dead/confusing code** - Fallback paths that "may fail on multi-instance deployments"
## Goals
1. **Single source of truth** - ONE function handles YouTube downloads for the cloud backend
2. **Always use remote flacfetch** when configured - All YouTube downloads should go through the flacfetch VM
3. **Graceful fallback** - When remote is disabled, use local download with clear error messaging
4. **Comprehensive testing** - Unit tests for each component, integration tests for full flow
5. **Remove dead code** - Clean up redundant implementations
## Proposed Architecture
```
┌─────────────────────────────────────────────────────────────────────┐
│ Entry Points │
├──────────────────────┬──────────────────────┬───────────────────────┤
│ Audio Search Select │ Direct URL Submit │ File Upload │
│ (audio_search.py) │ (file_upload.py) │ (file_upload.py) │
└──────────┬───────────┴──────────┬───────────┴───────────┬───────────┘
│ │ │
▼ ▼ │
┌──────────────────────────────────────────────┐ │
│ YouTubeDownloadService │ │
│ (NEW: backend/services/youtube_service.py) │ │
│ │ │
│ - download_by_url(url, job_id) → gcs_path │ │
│ - download_by_id(video_id, job_id) → gcs_path│ │
│ │ │
│ Internally: │ │
│ 1. Check if remote flacfetch enabled │ │
│ 2. If yes: Use FlacfetchClient.download_by_id│ │
│ 3. If no: Use local yt_dlp with clear error │ │
│ messaging about bot detection risk │ │
└──────────────────┬───────────────────────────┘ │
│ │
▼ ▼
┌─────────────────────────────────────────────────────────────────────┐
│ GCS │
│ uploads/{job_id}/audio/ │
└─────────────────────────────────────────────────────────────────────┘
```
## Implementation Steps
### Phase 1: Create YouTubeDownloadService
**File:** `backend/services/youtube_download_service.py`
```python
class YouTubeDownloadService:
"""
Single point of entry for all YouTube downloads in the cloud backend.
When remote flacfetch is configured (FLACFETCH_API_URL), downloads happen
on the flacfetch VM which has YouTube cookies and avoids bot detection.
When remote is not configured, falls back to local yt_dlp with a warning
that downloads may be blocked.
"""
def __init__(self):
self._flacfetch_client = get_flacfetch_client()
async def download(
self,
url: str,
job_id: str,
artist: Optional[str] = None,
title: Optional[str] = None,
) -> str:
"""
Download YouTube audio and upload to GCS.
Args:
url: YouTube URL (any format - watch, youtu.be, shorts, etc.)
job_id: Job ID for GCS path
artist: Optional artist name for filename
title: Optional title for filename
Returns:
GCS path (not gs:// prefix, just the path portion)
Raises:
YouTubeDownloadError: If download fails
"""
video_id = self._extract_video_id(url)
if self._flacfetch_client:
return await self._download_remote(video_id, job_id, artist, title)
else:
logger.warning(
"Remote flacfetch not configured. Attempting local download, "
"but this may fail due to YouTube bot detection on Cloud Run IPs."
)
return await self._download_local(url, job_id, artist, title)
async def download_by_id(
self,
video_id: str,
job_id: str,
artist: Optional[str] = None,
title: Optional[str] = None,
) -> str:
"""Download by video ID instead of URL."""
url = f"https://www.youtube.com/watch?v={video_id}"
return await self.download(url, job_id, artist, title)
```
### Phase 2: Refactor Audio Search Path
**File:** `backend/api/routes/audio_search.py`
Changes to `_download_and_start_processing()`:
```python
# BEFORE: Complex branching with multiple paths
if is_torrent_source and is_remote_enabled:
# Torrent path...
elif is_remote_enabled and source_id and source_name:
# YouTube remote path (buggy source_id extraction)...
else:
# Local path with another YouTube branch...
# AFTER: Clean separation by source type
if is_torrent_source:
result = await _download_torrent(source_name, source_id, target_file, job_id)
elif source_name == 'YouTube':
result = await youtube_service.download_by_id(source_id, job_id, artist, title)
elif source_name == 'Spotify':
result = await _download_spotify(source_id, job_id, artist, title)
else:
raise DownloadError(f"Unknown source: {source_name}")
```
### Phase 3: Refactor Direct URL Path
**File:** `backend/api/routes/file_upload.py`
Changes to `create_job_from_url()`:
```python
# BEFORE: Creates job and triggers audio_worker which uses download_from_url()
# This always runs on Cloud Run and gets blocked
# AFTER: If YouTube URL, download immediately using YouTubeDownloadService
if is_youtube_url(body.url):
youtube_service = get_youtube_download_service()
gcs_path = await youtube_service.download(body.url, job_id, body.artist, body.title)
# Update job with audio already downloaded
job.input_media_gcs_path = gcs_path
# Skip audio download in worker
```
**File:** `backend/workers/audio_worker.py`
Changes to `download_audio()`:
```python
# BEFORE: Has YouTube download logic with download_from_url()
# AFTER: Remove YouTube-specific logic - it's handled before worker runs
async def download_audio(job_id: str, job: Job) -> str:
"""Download audio from GCS (already uploaded by service layer)."""
if not job.input_media_gcs_path:
raise Exception("No audio path - this shouldn't happen for URL jobs")
# Download from GCS to local for processing
local_path = f"/tmp/{job_id}/audio/{os.path.basename(job.input_media_gcs_path)}"
storage_service.download_file(job.input_media_gcs_path, local_path)
return local_path
```
### Phase 4: Clean Up Dead Code
Remove or consolidate:
1. **`backend/workers/audio_worker.py:download_from_url()`** - Move to youtube_service or delete
2. **`karaoke_gen/file_handler.py:FileHandler.download_video()`** - Keep for CLI, but backend doesn't use it
3. **Multiple fallback paths in audio_search.py** - Simplify to single path per source type
4. **`_trigger_lyrics_worker_after_url_download()`** - Inline or remove if no longer needed
### Phase 5: Comprehensive Testing
#### Unit Tests (`backend/tests/test_youtube_download_service.py`)
```python
class TestYouTubeDownloadService:
def test_download_uses_remote_when_configured(self):
"""When FLACFETCH_API_URL is set, should use remote client."""
def test_download_falls_back_to_local_when_remote_disabled(self):
"""When remote not configured, should use local yt_dlp."""
def test_download_extracts_video_id_from_various_url_formats(self):
"""Should handle youtube.com, youtu.be, shorts, etc."""
def test_download_returns_correct_gcs_path_format(self):
"""Should return path without gs:// prefix."""
def test_download_handles_remote_failure_gracefully(self):
"""If remote fails, should raise clear error, not fall back silently."""
```
#### Integration Tests (`backend/tests/integration/test_youtube_flow.py`)
```python
class TestYouTubeDownloadFlow:
@pytest.mark.integration
async def test_audio_search_youtube_selection_downloads_correctly(self):
"""Full flow: search → select YouTube → verify GCS upload."""
@pytest.mark.integration
async def test_direct_url_submission_downloads_correctly(self):
"""Full flow: submit YouTube URL → verify GCS upload."""
@pytest.mark.integration
async def test_youtube_download_with_remote_flacfetch(self):
"""Verify remote flacfetch integration works end-to-end."""
```
#### Flacfetch Tests (in flacfetch repo)
```python
class TestYouTubeDownloadByIdEndpoint:
def test_download_by_id_with_youtube_source(self):
"""POST /download-by-id with source_name=YouTube works."""
def test_download_by_id_uploads_to_gcs_when_requested(self):
"""When gcs_path provided, should upload and return gs:// path."""
def test_download_by_id_handles_bot_detection_error(self):
"""Should return clear error if YouTube blocks the request."""
```
## Files to Create/Modify
| File | Action | Description |
|------|--------|-------------|
| `backend/services/youtube_download_service.py` | Create | New consolidated YouTube download service |
| `backend/api/routes/audio_search.py` | Modify | Use YouTubeDownloadService, remove complex branching |
| `backend/api/routes/file_upload.py` | Modify | Download YouTube before creating job |
| `backend/workers/audio_worker.py` | Modify | Remove YouTube download logic |
| `backend/services/audio_search_service.py` | Modify | Fix source_id extraction (quick fix already done) |
| `backend/tests/test_youtube_download_service.py` | Create | Unit tests for new service |
| `backend/tests/integration/test_youtube_flow.py` | Create | Integration tests |
## Migration Strategy
1. **Phase 1-2** can be deployed independently - new service + audio search refactor
2. **Phase 3** changes the URL submission flow - may need careful rollout
3. **Phase 4** cleanup can happen after Phase 1-3 are stable
4. **Phase 5** tests should be written alongside each phase
## Quick Fix vs Full Refactor
### Quick Fix (do now)
- Fix `source_id` extraction in `audio_search_service.py` ✅ (done)
- This unblocks the "search → select YouTube" flow
### Full Refactor (this plan)
- Consolidate all YouTube downloads into `YouTubeDownloadService`
- Fix the "direct URL submission" flow to use remote flacfetch
- Remove dead code and simplify branching
- Add comprehensive tests
## Risk Assessment
| Risk | Mitigation |
|------|------------|
| Breaking existing flows | Comprehensive tests before deployment |
| Remote flacfetch unavailable | Graceful fallback with clear error messaging |
| YouTube bot detection | Remote flacfetch with cookies is the solution |
| GCS path format inconsistencies | Standardize on path-only format (no gs:// prefix) |
## Success Criteria
1. **One code path** for YouTube downloads in cloud backend
2. **Both entry points** (search+select, direct URL) use remote flacfetch when configured
3. **All tests pass** with >80% coverage on new code
4. **Production verified** with real YouTube downloads
5. **No dead code** remaining in the codebase
## Open Questions
1. Should we support local YouTube downloads at all in Cloud Run? (Current: yes with warning)
2. Should direct URL submission be synchronous (wait for download) or async (background)?
3. How do we handle YouTube cookies management for remote flacfetch?
## Related Documentation
- `docs/LESSONS-LEARNED.md` - "Fix Both Sides of Dual Code Paths" pattern
- `docs/ARCHITECTURE.md` - Remote flacfetch integration
- Previous PR #320 - Initial YouTube remote download fix (incomplete)
@@ -1,110 +0,0 @@
# Plan: Disable Auto-Correction (Agentic and Heuristic)
**Created:** 2026-01-20
**Branch:** feat/sess-20260120-1914-disable-auto-correction
**Status:** Draft
## Overview
Disable all auto-correction during the lyrics transcription & correction phase. Currently, auto-correction (both agentic AI-based and heuristic rule-based) creates more work for human reviewers by introducing errors that need to be manually fixed. The solution is to skip all correction processing and pass the raw transcription directly to human review.
## Requirements
- [x] Disable agentic AI correction (LLM-based via Gemini)
- [x] Disable heuristic correction (rule-based handlers like ExtendAnchorHandler, WordCountMatchHandler, etc.)
- [x] Preserve the transcription → human review → video generation flow
- [x] Minimal code changes (configuration-based if possible)
- [x] Easy to re-enable in the future if correction quality improves
## Technical Approach
The simplest approach is to change the backend configuration default for `use_agentic_ai` from `true` to `false`, AND ensure that when agentic AI is disabled, the heuristic handlers also don't run.
**Current behavior:**
1. `backend/config.py` line 46: `use_agentic_ai` defaults to `true`
2. `lyrics_worker.py` sets `USE_AGENTIC_AI` env var based on config
3. `corrector.py` line 84/132: Checks `USE_AGENTIC_AI` env var
4. When agentic is disabled, rule-based handlers still run (lines 669-733)
**Proposed changes:**
### Option A: Add a new `skip_correction` config (Recommended)
Add a single boolean `skip_correction` setting that:
- When `true`: Skips the entire correction process, returning raw transcription
- When `false` (future): Falls back to existing agentic/heuristic behavior
This is cleaner than disabling both systems separately because:
1. Single config controls both
2. Clear intent ("skip correction" vs "disable two different systems")
3. Easy to understand for future maintainers
### Option B: Just disable both individually
- Set `use_agentic_ai` default to `false`
- Disable default heuristic handlers by setting `enabled_handlers=[]`
Option A is cleaner, but Option B requires fewer code changes.
**Decision: Option A** - Add `skip_correction` config for clarity and easier future re-enablement.
## Implementation Steps
1. [x] **Add `skip_correction` config to backend/config.py**
- Add `skip_correction: bool` setting, default `true`
- Comment explaining this disables both agentic and heuristic correction
2. [x] **Update lyrics_worker.py to pass skip_correction**
- Pass the config value through to LyricsProcessor/LyricsTranscriber
3. [x] **Update corrector.py to support skip_correction**
- When `skip_correction=True`, return immediately with uncorrected segments
- No handler processing, no agentic AI
- Still need to do anchor/gap finding? No - skip entirely
4. [x] **Update LyricsProcessor to support skip_correction**
- Pass through to LyricsTranscriber
5. [x] **Update LyricsTranscriber controller to support skip_correction**
- Pass through to corrector or skip corrector.run() entirely
6. [x] **Update docs/README.md**
- Note that auto-correction is disabled
- Update "Agentic AI correction" status to "Disabled"
7. [x] **Run tests to verify no regressions**
## Files to Create/Modify
| File | Action | Description |
|------|--------|-------------|
| `backend/config.py` | Modify | Add `skip_correction` config (default: true) |
| `backend/workers/lyrics_worker.py` | Modify | Pass `skip_correction` to transcription |
| `karaoke_gen/lyrics_transcriber/correction/corrector.py` | Modify | Return early when skip_correction=True |
| `karaoke_gen/lyrics_transcriber/controller.py` | Modify | Pass skip_correction to corrector |
| `karaoke_gen/lyrics_processor.py` | Modify | Pass skip_correction through |
| `docs/README.md` | Modify | Update status to reflect disabled correction |
## Testing Strategy
- **Unit tests**: No new tests needed - this is a config change
- **Existing tests**: Run `make test` to ensure no regressions
- **Manual verification**: The correction metadata in the corrections.json should show no corrections made
## Open Questions
None - the approach is straightforward.
## Rollback Plan
To re-enable auto-correction:
1. Set `SKIP_CORRECTION=false` environment variable, OR
2. Change the default back to `false` in `backend/config.py`
The code paths for agentic and heuristic correction remain intact and can be re-enabled at any time.
## Alternative Considered
Instead of adding a new config, we could:
1. Set `USE_AGENTIC_AI=false` (disables agentic)
2. Set `enabled_handlers=[]` somewhere (disables heuristic)
But this is more fragmented and harder to understand. A single `skip_correction` config is clearer.
@@ -1,99 +0,0 @@
# Plan: Encoding Worker Immutable Deployment Pattern
## Status: In Progress
## Problem Statement
The GCE encoding worker has recurring version mismatch issues because:
1. `startup.sh` is baked into the Packer image (requires image rebuild + VM recreation)
2. Wheel selection uses `sort -V` which can fail if image is outdated
3. CI only restarts service - doesn't verify version strictly
4. Multiple actors (CI, Pulumi, manual SSH) can cause drift
5. 51 old wheel files accumulate in GCS bucket
## Solution: Immutable Deployment Pattern
### Core Principles
1. **Single Source of Truth**: All code and config from GCS, nothing baked in image
2. **Fixed Wheel Path**: `karaoke_gen-current.whl` eliminates sorting
3. **Version Manifest**: CI writes expected version, service MUST match
4. **Strict Verification**: CI FAILS if deployed version doesn't match
5. **Self-Updating Startup**: Download startup.sh from GCS on every boot
## Implementation Steps
### Phase 1: GCS Structure & Scripts
- [x] **Step 1.1**: Create `infrastructure/encoding-worker/` directory
- [x] **Step 1.2**: Create `bootstrap.sh` (minimal, baked in image)
- [x] **Step 1.3**: Create `startup.sh` (CI-managed, deployed to GCS)
- [x] **Step 1.4**: Create README.md with documentation
### Phase 2: CI Workflow Updates
- [x] **Step 2.1**: Update CI to upload wheel to fixed path
- [x] **Step 2.2**: Update CI to upload startup.sh to GCS
- [x] **Step 2.3**: Update CI to write version.txt manifest
- [x] **Step 2.4**: Update CI to strictly verify deployed version (fail on mismatch)
### Phase 3: Packer Image Updates
- [x] **Step 3.1**: Update provision.sh to use bootstrap.sh
- [ ] **Step 3.2**: Rebuild Packer image (manual step after merge)
- [ ] **Step 3.3**: Recreate VM to use new image (manual step after merge)
### Phase 4: Documentation & Cleanup
- [x] **Step 4.1**: Update LESSONS-LEARNED.md with new pattern
- [x] **Step 4.2**: Update infrastructure/packer/README.md
- [ ] **Step 4.3**: Add lifecycle rule for old wheels (optional, later)
## New GCS Structure
```
gs://karaoke-gen-storage-nomadkaraoke/
├── wheels/
│ ├── karaoke_gen-current.whl # Latest (CI overwrites)
│ └── karaoke_gen-{version}.whl # Versioned for rollback
│
└── encoding-worker/
├── bootstrap.sh # Reference copy
├── startup.sh # CI-managed, downloaded on boot
└── version.txt # Expected version number
```
## Files to Create/Modify
| File | Action | Purpose |
|------|--------|---------|
| `infrastructure/encoding-worker/bootstrap.sh` | Create | Minimal loader (baked in image) |
| `infrastructure/encoding-worker/startup.sh` | Create | Main logic (deployed to GCS) |
| `infrastructure/packer/scripts/provision.sh` | Modify | Use bootstrap.sh |
| `.github/workflows/ci.yml` | Modify | New deployment flow |
| `docs/LESSONS-LEARNED.md` | Modify | Document pattern |
## Testing Strategy
1. **Unit test**: Verify startup.sh logic locally
2. **Integration test**: After merge, verify:
- CI uploads files to correct GCS paths
- Service starts with correct version
- Health endpoint returns expected version
3. **Manual verification**: Check VM logs after first deployment
## Rollback Plan
If issues occur:
1. CI can re-upload previous wheel to `karaoke_gen-current.whl`
2. Previous versioned wheels remain in GCS for manual rollback
3. Old Packer images remain for VM rollback
## Open Questions
None - design is complete.
## Notes
- Bootstrap.sh is ~10 lines and should never need changes
- All logic changes go through startup.sh in GCS
- Version verification is strict - CI fails on mismatch
@@ -1,166 +0,0 @@
# Plan: Fix YouTube Remote Download Missing GCS Path
**Created:** 2026-01-20
**Branch:** feat/sess-20260120-1756-fix-youtube-remote-download
**Status:** Complete
## Overview
When remote flacfetch is enabled and a user selects a YouTube audio source, the backend calls `download_by_id()` without the `gcs_path` parameter. This causes flacfetch to download the file locally on the VM instead of uploading to GCS. The backend then tries to open the VM's local path, which doesn't exist on Cloud Run, causing the job to fail.
This is the same class of bug that was fixed on 2025-12-22 for RED/OPS torrent sources (see `docs/archive/2025-12-phase1-testing-gaps.md`), but the fix only covered torrent sources, not YouTube.
## Root Cause
In `backend/api/routes/audio_search.py` lines 354-363:
```python
# Use download_by_id if we have source_id and remote is enabled
if source_id and source_name and is_remote_enabled:
logger.info(f"Using download_by_id for local download: {source_name} ID={source_id}")
result = audio_search_service.download_by_id(
source_name=source_name,
source_id=source_id,
output_dir=temp_dir, # <-- This is a local temp dir, but flacfetch runs on VM!
target_file=target_file,
download_url=download_url,
# <-- MISSING: gcs_path parameter
)
```
When `is_remote_enabled=True`, `download_by_id()` routes to the remote flacfetch VM. Without `gcs_path`, the VM downloads locally and returns a local path like `/var/lib/transmission-daemon/downloads/...` which doesn't exist on Cloud Run.
## Requirements
- [x] YouTube downloads with remote flacfetch should upload directly to GCS
- [x] Backend should receive a `gs://` path from flacfetch
- [x] No changes to flacfetch itself (it already supports `upload_to_gcs` and `gcs_destination`)
- [x] Maintain backwards compatibility (local-only deployments should still work)
- [x] Add test coverage for this code path
## Technical Approach
**Option A (chosen):** Pass `gcs_path` to `download_by_id()` for ALL remote downloads, not just torrent sources.
The current code structure has two branches:
1. `if is_torrent_source and is_remote_enabled:` - handles RED/OPS with GCS upload ✅
2. `else:` branch - handles YouTube, but incorrectly assumes "local download" even when remote is enabled
The fix refactors to check `is_remote_enabled` first, then handle torrent vs YouTube within that:
- If remote enabled: always pass `gcs_path` to let flacfetch upload directly
- If remote disabled: download locally and upload to GCS from Cloud Run
## Implementation Steps
1. [x] Refactor the `else` branch in `_download_and_start_processing()` to check for remote-enabled YouTube downloads
2. [x] Pass `gcs_path` when calling `download_by_id()` for remote YouTube downloads
3. [x] Add test case for YouTube with remote enabled to verify `gcs_path` is passed
4. [x] Update logging to clarify the download path being used
5. [x] Bump version in `pyproject.toml`
6. [x] Run tests to verify no regressions
## Files to Create/Modify
| File | Action | Description |
|------|--------|-------------|
| `backend/api/routes/audio_search.py` | Modify | Add `gcs_path` parameter for remote YouTube downloads |
| `backend/tests/test_audio_search.py` | Modify | Add test case for YouTube remote download with GCS path |
| `pyproject.toml` | Modify | Bump version |
## Detailed Code Changes
### audio_search.py (lines 349-393)
Current structure:
```python
else:
# Local download (YouTube or fallback) <-- WRONG: not always local!
temp_dir = tempfile.mkdtemp(...)
if source_id and source_name and is_remote_enabled:
result = audio_search_service.download_by_id(
...
output_dir=temp_dir, # <-- BUG: unused for remote
# Missing gcs_path!
)
```
Fixed structure:
```python
else:
# Non-torrent download (YouTube or fallback)
if is_remote_enabled and source_id and source_name:
# Remote download via flacfetch VM - upload directly to GCS
gcs_destination = f"uploads/{job_id}/audio/"
logger.info(f"Using remote download_by_id for {source_name} ID={source_id} with GCS upload to: {gcs_destination}")
result = audio_search_service.download_by_id(
source_name=source_name,
source_id=source_id,
output_dir="", # Not used for remote
target_file=target_file,
download_url=download_url,
gcs_path=gcs_destination, # <-- FIX: Add this!
)
# Handle GCS path response (same logic as torrent branch)
if result.filepath.startswith("gs://"):
parts = result.filepath.replace("gs://", "").split("/", 1)
if len(parts) == 2:
audio_gcs_path = parts[1]
else:
audio_gcs_path = result.filepath
filename = os.path.basename(result.filepath)
else:
# Fallback: upload manually
filename = os.path.basename(result.filepath)
audio_gcs_path = f"uploads/{job_id}/audio/{filename}"
logger.warning(f"Remote download returned local path: {result.filepath}, uploading manually")
with open(result.filepath, 'rb') as f:
storage_service.upload_fileobj(f, audio_gcs_path, content_type='audio/flac')
logger.info(f"Remote download complete, GCS path: {audio_gcs_path}")
else:
# Local download (remote disabled or no source_id)
temp_dir = tempfile.mkdtemp(prefix=f"audio_download_{job_id}_")
# ... existing local download logic ...
```
## Testing Strategy
### Unit Tests
Add to `test_audio_search.py` in `TestAudioSearchApiRouteDownload`:
```python
def test_youtube_remote_download_passes_gcs_path(self, mock_job_manager, mock_storage_service):
"""
Test that YouTube downloads via remote flacfetch include GCS path.
This is the same bug pattern as the torrent download bug fixed in 2025-12:
when remote is enabled, we must pass gcs_path so flacfetch uploads to GCS.
"""
# Setup: YouTube source with remote enabled
# Verify: download_by_id called with gcs_path parameter
```
### Manual Testing (post-deploy)
1. Create a job and select a YouTube audio source
2. Verify job completes successfully
3. Check GCS path is properly set in job data
## Open Questions
None - the fix is straightforward and follows the pattern established for torrent sources.
## Rollback Plan
This is a minor code change. If issues occur:
1. Revert the PR
2. The failure mode returns to the current state (YouTube downloads fail when remote is enabled)
## Related Documentation
- `docs/archive/2025-12-phase1-testing-gaps.md` - Original bug fix for torrent sources
- `docs/LESSONS-LEARNED.md` - "Fix Both Sides of Dual Code Paths" pattern
@@ -1,329 +0,0 @@
# Plan: Unified Combined Review for Local CLI
**Created:** 2026-01-24
**Branch:** feat/sess-20260124-1444-combined-review-flow
**Status:** Draft
## Problem Statement
The combined lyrics + instrumental review flow was implemented for the cloud web app, but the local CLI is broken in two ways:
1. **Bug: Lyrics review is skipped when `SKIP_CORRECTION=true`** (the default)
- Review code lives inside `correct_lyrics()` method in controller.py
- When auto-correction is disabled, `correct_lyrics()` is never called
- Users never get a chance to review lyrics
2. **Standalone instrumental UI still exists for CLI**
- `InstrumentalReviewServer` in `instrumental_review/server.py` is a separate server
- CLI launches this standalone server after lyrics review
- The combined review flow we built only works for the web app
## Goals
1. **Fix the skip bug** - Review should happen even when auto-correction is disabled
2. **Unify all review UIs** - One combined review flow for CLI, cloud CLI, and web app
3. **Remove dead code** - Delete standalone `InstrumentalReviewServer` and related code
4. **Keep it simple** - Minimal changes to achieve the goal
## Architecture Overview
### Current State (Broken)
```
CLI Flow:
controller.py::process()
→ correct_lyrics() [SKIPPED if SKIP_CORRECTION=true]
→ ReviewServer starts (lyrics only)
→ User reviews
→ ReviewServer stops
→ gen_cli.py::run_instrumental_review()
→ InstrumentalReviewServer starts (instrumental only)
→ User selects instrumental
→ InstrumentalReviewServer stops
Cloud Web App Flow:
Job enters AWAITING_REVIEW
→ User visits /app/jobs/{id}/review
→ Combined UI: lyrics + instrumental
→ POST /api/jobs/{id}/complete-review (includes instrumental_selection)
```
### Target State
```
CLI Flow (and Cloud CLI):
controller.py::process()
→ _run_human_review() [NEW - always called when review enabled]
→ ReviewServer starts (combined: lyrics + instrumental)
→ User reviews lyrics AND selects instrumental
→ ReviewServer stops, returns both results
→ Continue with selected instrumental
Cloud Web App Flow: (unchanged)
Same as before
```
## Key Insight
The ReviewServer already serves the bundled Next.js frontend. That frontend already has the combined review UI with `InstrumentalSelectorEmbedded`. We just need to:
1. Make `ReviewServer` serve instrumental options in its `/correction-data` endpoint
2. Make `ReviewServer` accept instrumental selection in its `/complete-review` endpoint
3. Call the review flow from the right place (not inside `correct_lyrics()`)
4. Delete `InstrumentalReviewServer` entirely
## Implementation Steps
### Phase 1: Fix ReviewServer to Support Instrumental Selection
**1.1. Add instrumental data to ReviewServer's correction-data endpoint**
File: `karaoke_gen/lyrics_transcriber/review/server.py`
The `ReviewServer.__init__()` accepts a `CorrectionResult`. We need to also accept instrumental options:
```python
def __init__(
self,
correction_result: CorrectionResult,
instrumental_options: list[dict] | None = None, # NEW
backing_vocals_analysis: dict | None = None, # NEW
...
):
```
Update `_correction_data()` endpoint to include these in response:
```python
@app.get("/api/review/{job_id}/correction-data")
def _correction_data(job_id: str):
return {
"lyrics_data": self.correction_result.to_dict(),
"instrumental_options": self.instrumental_options, # NEW
"backing_vocals_analysis": self.backing_vocals_analysis # NEW
}
```
**1.2. Update ReviewServer's complete-review endpoint to accept instrumental selection**
File: `karaoke_gen/lyrics_transcriber/review/server.py`
The existing endpoint accepts `corrected_lyrics`. Add `instrumental_selection`:
```python
@app.post("/api/review/{job_id}/complete-review")
def _complete_review(job_id: str, request: ReviewCompleteRequest):
# Existing: store corrected lyrics
self.corrected_lyrics = request.corrected_lyrics
# NEW: store instrumental selection
self.instrumental_selection = request.instrumental_selection # "clean" | "with_backing"
self.review_completed = True
```
**1.3. Expose instrumental selection after review completes**
The calling code needs to get the selection. Update the return value or add an attribute:
```python
# After review loop completes:
return {
"corrected_lyrics": self.corrected_lyrics,
"instrumental_selection": self.instrumental_selection
}
```
### Phase 2: Fix controller.py to Call Review at the Right Time
**2.1. Extract review logic from `correct_lyrics()` into standalone method**
File: `karaoke_gen/lyrics_transcriber/core/controller.py`
Create new method `_run_human_review()` that can be called independently:
```python
def _run_human_review(
self,
correction_result: CorrectionResult,
instrumental_options: list[dict] | None = None,
backing_vocals_analysis: dict | None = None
) -> tuple[CorrectionResult, str | None]:
"""Run human review server and return corrected result + instrumental selection."""
if not self.output_config.review_enabled:
return correction_result, None
server = ReviewServer(
correction_result=correction_result,
instrumental_options=instrumental_options,
backing_vocals_analysis=backing_vocals_analysis,
...
)
result = server.start() # Blocks until review complete
return result["corrected_lyrics"], result.get("instrumental_selection")
```
**2.2. Update `process()` to always call review when appropriate**
File: `karaoke_gen/lyrics_transcriber/core/controller.py`
Current flow (simplified):
```python
def process(self):
self.transcribe()
self.separate_audio()
if self.output_config.run_correction:
self.correct_lyrics() # Review is inside here!
# ...
```
New flow:
```python
def process(self):
self.transcribe()
self.separate_audio()
# Build correction result (with or without auto-correction)
if self.output_config.run_correction:
correction_result = self._run_auto_correction()
else:
correction_result = self._create_uncorrected_result()
# Human review is separate from auto-correction
if self.output_config.review_enabled:
instrumental_options = self._get_instrumental_options()
backing_vocals_analysis = self._analyze_backing_vocals()
corrected, instrumental_selection = self._run_human_review(
correction_result,
instrumental_options,
backing_vocals_analysis
)
self.results.correction_result = corrected
self.results.instrumental_selection = instrumental_selection
```
### Phase 3: Update gen_cli.py to Use Combined Review
**3.1. Remove standalone instrumental review call**
File: `karaoke_gen/utils/gen_cli.py`
Remove the `run_instrumental_review()` function call from main flow. The instrumental selection now comes from the combined review.
Before:
```python
# After lyrics processing
if not args.non_interactive:
instrumental_selection = run_instrumental_review(...)
```
After:
```python
# Instrumental selection comes from controller.results.instrumental_selection
instrumental_selection = controller.results.instrumental_selection
```
**3.2. Pass instrumental options to controller**
The controller needs instrumental options to display in review. These come from the separated audio stems:
```python
controller = LyricsTranscriber(
# ... existing params ...
)
# Before calling process(), set up instrumental options
# These are typically discovered during audio separation
```
### Phase 4: Delete Standalone InstrumentalReviewServer
**4.1. Delete files**
- Delete `karaoke_gen/instrumental_review/server.py`
- Delete `karaoke_gen/instrumental_review/__init__.py` (if empty after)
- Remove `run_instrumental_review()` from `karaoke_gen/utils/gen_cli.py`
**4.2. Clean up imports**
Remove any imports of `InstrumentalReviewServer` throughout the codebase.
### Phase 5: Update Frontend for Local Mode
**5.1. Verify local mode detection works**
File: `frontend/lib/api.ts`
The frontend already has local mode detection based on hostname and port. Verify it works:
```typescript
const isLocalMode = () => {
if (typeof window === 'undefined') return false;
const hostname = window.location.hostname;
const port = parseInt(window.location.port, 10);
return hostname === 'localhost' && port >= 8000 && port <= 8770;
};
```
**5.2. Verify correction-data endpoint parsing**
Ensure the frontend correctly parses instrumental_options and backing_vocals_analysis from the local server's response.
### Phase 6: Testing
**6.1. Manual testing checklist**
- [ ] Run CLI with `SKIP_CORRECTION=true` - review should still appear
- [ ] Run CLI with `SKIP_CORRECTION=false` - review should appear after correction
- [ ] Combined UI shows both lyrics editor and instrumental selector
- [ ] Selecting instrumental and completing review works
- [ ] No standalone instrumental review page appears
- [ ] Cloud web app still works (no regression)
**6.2. Update unit tests**
- Remove tests for `InstrumentalReviewServer`
- Add tests for `ReviewServer` with instrumental options
- Add tests for `controller._run_human_review()`
## Files to Modify
| File | Action | Description |
|------|--------|-------------|
| `lyrics_transcriber/review/server.py` | Modify | Add instrumental options/selection support |
| `lyrics_transcriber/core/controller.py` | Modify | Extract review into standalone method, call at right time |
| `utils/gen_cli.py` | Modify | Remove `run_instrumental_review()`, use combined result |
| `instrumental_review/server.py` | Delete | Standalone server no longer needed |
| `instrumental_review/__init__.py` | Modify/Delete | Clean up imports |
## Risks & Mitigations
| Risk | Mitigation |
|------|------------|
| Breaking cloud flow | Cloud doesn't use ReviewServer, uses backend API - no risk |
| Missing instrumental options in local mode | Fail clearly if options not provided |
| Review server port conflicts | Keep existing port allocation logic |
## Open Questions
1. **Where does backing vocals analysis run for local CLI?**
- Currently runs in `render_video_worker.py` for cloud
- For local, need to run it after audio separation, before review
- May need to call analysis function from controller
2. **How are instrumental stems exposed to local review?**
- Need to serve audio files from ReviewServer
- Or use file:// URLs (may have CORS issues)
- Existing approach: ReviewServer already serves static files
## Success Criteria
1. `SKIP_CORRECTION=true` no longer skips human review
2. Single combined review UI appears (no separate instrumental step)
3. All platforms (CLI, cloud CLI, web) use same review flow
4. No dead code remaining for standalone instrumental review
+26 -106
View File
@@ -967,16 +967,8 @@ jobs:
version: "latest"
# ============================================
# Deploy to GCE Encoding Worker (Immutable Deployment Pattern)
#
# This pattern ensures version consistency:
# 1. Upload wheel to FIXED path (no version sorting)
# 2. Upload startup.sh (logic can change without image rebuild)
# 3. Write version manifest for verification
# 4. Restart service (triggers bootstrap.sh -> startup.sh)
# 5. STRICTLY verify deployed version (fail CI on mismatch)
#
# See infrastructure/encoding-worker/README.md for details
# Upload wheel to GCS for GCE encoding worker
# The GCE worker installs this wheel at job start for hot code updates
# ============================================
- name: Download wheel artifact
@@ -985,114 +977,52 @@ jobs:
name: package-dist
path: dist/
- name: Deploy to GCE Encoding Worker
- name: Upload wheel to GCS for GCE worker
run: |
VERSION="${{ steps.version_check.outputs.version }}"
BUCKET="gs://karaoke-gen-storage-${{ env.PROJECT_ID }}"
echo "🚀 Deploying encoding worker v${VERSION}..."
# 1. Find the wheel file
echo "📦 Uploading wheel to GCS for GCE encoding worker..."
WHEEL_FILE=$(ls dist/*.whl | head -1)
if [ -z "$WHEEL_FILE" ]; then
echo "::error::No wheel file found in dist/"
exit 1
fi
echo "📦 Found wheel: $WHEEL_FILE"
echo "Found wheel: $WHEEL_FILE"
# 2. Upload wheel to FIXED path (atomic via temp file)
# This eliminates version sorting - always use karaoke_gen-current.whl
echo "📤 Uploading wheel to fixed path..."
gsutil cp "$WHEEL_FILE" "${BUCKET}/wheels/karaoke_gen-current.whl.tmp"
gsutil mv "${BUCKET}/wheels/karaoke_gen-current.whl.tmp" \
"${BUCKET}/wheels/karaoke_gen-current.whl"
# Upload to GCS (GCE worker downloads latest at job start)
gsutil cp "$WHEEL_FILE" gs://karaoke-gen-storage-${{ env.PROJECT_ID }}/wheels/
# Also keep versioned copy for rollback/debugging
gsutil cp "$WHEEL_FILE" "${BUCKET}/wheels/"
# 3. Upload startup.sh (allows logic changes without image rebuild)
echo "📤 Uploading startup.sh..."
gsutil cp infrastructure/encoding-worker/startup.sh \
"${BUCKET}/encoding-worker/startup.sh"
# 4. Write version manifest for verification
echo "📝 Writing version manifest..."
echo "${VERSION}" | gsutil cp - "${BUCKET}/encoding-worker/version.txt"
echo "✅ GCS deployment complete"
echo "✅ Wheel uploaded to GCS"
- name: Restart GCE Encoding Worker
run: |
VERSION="${{ steps.version_check.outputs.version }}"
echo "🔄 Restarting encoding worker..."
echo "🔄 Restarting encoding worker to pick up new wheel..."
# Use IAP tunneling - works directly with service account credentials
# (avoids SSH key propagation delays that cause silent failures)
gcloud compute ssh encoding-worker \
--zone=us-central1-c \
--project=nomadkaraoke \
--tunnel-through-iap \
--command="sudo systemctl restart encoding-worker"
echo "⏳ Waiting for service to start (up to 60s)..."
sleep 20
echo "⏳ Waiting for service to start..."
sleep 15
- name: Verify Encoding Worker Version
run: |
VERSION="${{ steps.version_check.outputs.version }}"
# Verify the restart worked by checking the wheel version
NEW_VERSION=$(gcloud compute ssh encoding-worker \
--zone=us-central1-c \
--project=nomadkaraoke \
--tunnel-through-iap \
--command="curl -s http://localhost:8080/health | grep -o '\"wheel_version\":\"[^\"]*\"' | cut -d'\"' -f4")
echo "🔍 Verifying encoding worker version..."
EXPECTED_VERSION=$(cat pyproject.toml | grep 'version = ' | head -1 | cut -d'"' -f2)
# Retry loop - service may take time to fully start
MAX_RETRIES=6
RETRY_DELAY=10
for i in $(seq 1 $MAX_RETRIES); do
echo "Attempt $i/$MAX_RETRIES..."
NEW_VERSION=$(gcloud compute ssh encoding-worker \
--zone=us-central1-c \
--project=nomadkaraoke \
--tunnel-through-iap \
--command="curl -s http://localhost:8080/health 2>/dev/null | grep -o '\"wheel_version\":\"[^\"]*\"' | cut -d'\"' -f4" 2>/dev/null || echo "")
if [ "$NEW_VERSION" = "$VERSION" ]; then
echo "✅ Encoding worker version verified: v$NEW_VERSION"
exit 0
fi
if [ -n "$NEW_VERSION" ]; then
echo "⚠️ Version: $NEW_VERSION (expected: $VERSION)"
else
echo "⚠️ Health endpoint not responding yet"
fi
if [ $i -lt $MAX_RETRIES ]; then
echo " Retrying in ${RETRY_DELAY}s..."
sleep $RETRY_DELAY
fi
done
# STRICT VERIFICATION: Fail CI on version mismatch
echo ""
echo "=============================================="
echo "❌ ENCODING WORKER DEPLOYMENT FAILED"
echo "=============================================="
echo "Expected version: $VERSION"
echo "Deployed version: $NEW_VERSION"
echo ""
echo "Possible causes:"
echo " - GCS upload failed (check 'Deploy to GCE' step)"
echo " - startup.sh download failed (check VM logs)"
echo " - Wheel installation failed (check VM logs)"
echo ""
echo "To debug, SSH to VM and check logs:"
echo " gcloud compute ssh encoding-worker --zone=us-central1-c --project=nomadkaraoke --tunnel-through-iap"
echo " sudo journalctl -u encoding-worker -n 100"
echo " cat /var/log/encoding-worker-startup.log"
echo "=============================================="
exit 1
if [ "$NEW_VERSION" = "$EXPECTED_VERSION" ]; then
echo "✅ Encoding worker restarted successfully (v$NEW_VERSION)"
else
echo "⚠️ Version mismatch: encoder has v$NEW_VERSION, expected v$EXPECTED_VERSION"
echo " This may indicate the restart failed or the wheel wasn't downloaded"
# Don't fail the build - the encoder will still work, just with old version
fi
- name: Configure Docker for Artifact Registry
run: gcloud auth configure-docker ${{ env.REGISTRY }} --quiet
@@ -1395,16 +1325,6 @@ jobs:
run: |
pulumi login
- name: Refresh Infrastructure State
working-directory: infrastructure
env:
PULUMI_ACCESS_TOKEN: ${{ secrets.PULUMI_ACCESS_TOKEN }}
run: |
echo "🔄 Refreshing Pulumi state to sync with actual cloud resources..."
# Refresh detects resources deleted outside of Pulumi (e.g., via gcloud CLI)
# This ensures Pulumi recreates missing resources
pulumi refresh --yes --skip-preview --stack nomadkaraoke/karaoke-gen-infrastructure/prod || true
- name: Deploy Infrastructure
working-directory: infrastructure
env:
-1
View File
@@ -3,7 +3,6 @@
.vscode
*.csv
*.mp3
ABBA*
# Andrew functional adds
/audio/
+1 -10
View File
@@ -1,7 +1,7 @@
# Makefile for karaoke-gen project
# Run `make help` to see available commands
.PHONY: help install install-backend install-frontend build-frontend test test-unit test-backend test-e2e test-frontend test-all lint clean emulators-start emulators-stop
.PHONY: help install install-backend install-frontend test test-unit test-backend test-e2e test-frontend test-all lint clean emulators-start emulators-stop
# Default target
help:
@@ -12,7 +12,6 @@ help:
@echo " make test-unit - Run unit tests only (karaoke_gen package)"
@echo " make test-e2e - Run emulator tests with auto-start/stop"
@echo " make install - Install all dependencies (backend + frontend)"
@echo " make build-frontend - Build frontend and copy to Python package (for local testing)"
@echo " make lint - Run linter checks"
@echo " make emulators-start - Start GCP emulators for local development"
@echo " make emulators-stop - Stop GCP emulators"
@@ -34,14 +33,6 @@ install-frontend:
install: install-backend install-frontend
# Build frontend and copy to Python package (for local CLI testing)
build-frontend: install-frontend
@echo "=== Building frontend ==="
cd frontend && npm run build
@echo "=== Copying build to Python package ==="
cp -r frontend/out/* karaoke_gen/nextjs_frontend/out/
@echo "✅ Frontend built and ready for local testing"
# Run unit tests for karaoke_gen package
test-unit: install-backend
@echo "=== Running karaoke_gen unit tests ==="
+20 -2
View File
@@ -406,8 +406,26 @@ gcloud secrets versions add gdrive-service-account --data-file=./gdrive-sa.json
### Deploy Cloud Run
Deployments happen automatically via GitHub Actions CI when pushing to `main`.
See `.github/workflows/ci.yml` for the full deployment workflow.
```bash
# Build and deploy
gcloud builds submit --config=cloudbuild.yaml
# Get outputs from Pulumi
SA_EMAIL=$(pulumi stack output service_account_email)
BUCKET_NAME=$(pulumi stack output bucket_name)
# Deploy Cloud Run service
gcloud run deploy karaoke-backend \
--image us-central1-docker.pkg.dev/YOUR-PROJECT/karaoke-repo/karaoke-backend:latest \
--platform managed \
--region us-central1 \
--allow-unauthenticated \
--service-account $SA_EMAIL \
--memory 2Gi \
--cpu 2 \
--timeout 600 \
--set-env-vars="GOOGLE_CLOUD_PROJECT=YOUR-PROJECT,GCS_BUCKET_NAME=$BUCKET_NAME"
```
### Point CLI to Your Backend
+114
View File
@@ -341,3 +341,117 @@ def require_review_auth_factory(job_id_param: str = "job_id"):
require_review_auth = require_review_auth_factory()
def require_instrumental_auth_factory(job_id_param: str = "job_id"):
"""
Factory to create an instrumental review authentication dependency.
Accepts either:
1. Full user authentication (admin/user token) - also validates job ownership
2. Job-specific instrumental token (only valid for the specific job)
Args:
job_id_param: Name of the path parameter containing the job ID
Returns:
Dependency function that validates instrumental review access
"""
async def require_instrumental_auth(
request: Request,
job_id: str = Path(...),
instrumental_token: Optional[str] = Query(None, description="Job-specific instrumental review token"),
auth_service: AuthService = Depends(get_auth_service),
credentials: Optional[HTTPAuthorizationCredentials] = Depends(security),
token: Optional[str] = Query(None, alias="token", description="Full access token")
) -> Tuple[str, str]:
"""
Validate instrumental review access for a job.
Returns:
(job_id, auth_type) where auth_type is "full" or "instrumental_token"
Raises:
HTTPException: 401 if authentication fails
HTTPException: 403 if user doesn't own the job (for full auth)
"""
# Import here to avoid circular dependency
from backend.services.job_manager import JobManager
job_manager = JobManager()
# Try full authentication first
full_token = None
if credentials:
full_token = credentials.credentials
elif token:
full_token = token
if full_token:
auth_result = auth_service.validate_token_full(full_token)
if auth_result.is_valid:
# For full auth, also verify job ownership
job = job_manager.get_job(job_id)
if not job:
raise HTTPException(status_code=404, detail=f"Job {job_id} not found")
# Check ownership: admin can access all, users only their own jobs
if not auth_result.is_admin:
if auth_result.user_email and job.user_email:
if auth_result.user_email.lower() != job.user_email.lower():
logger.warning(
f"Instrumental access denied: user {auth_result.user_email} "
f"tried to access job {job_id} owned by {job.user_email}"
)
raise HTTPException(
status_code=403,
detail="You don't have permission to access this job's instrumental review"
)
elif job.user_email:
# Token auth without email trying to access a job with owner
raise HTTPException(
status_code=403,
detail="You don't have permission to access this job's instrumental review"
)
logger.info(f"Instrumental access granted via full auth ({auth_result.user_type}) for job {job_id}")
return job_id, "full"
# Try instrumental token
if instrumental_token:
job = job_manager.get_job(job_id)
if not job:
raise HTTPException(status_code=404, detail=f"Job {job_id} not found")
# Validate instrumental token matches
if job.instrumental_token and secrets.compare_digest(job.instrumental_token, instrumental_token):
# Check expiry if set
if job.instrumental_token_expires_at:
now = datetime.now(UTC)
# Handle timezone-naive datetimes from Firestore
expiry = job.instrumental_token_expires_at
if expiry.tzinfo is None:
expiry = expiry.replace(tzinfo=UTC)
if now > expiry:
logger.warning(f"Instrumental token expired for job {job_id}")
raise HTTPException(
status_code=401,
detail="Instrumental review token has expired. Please request a new review link."
)
logger.info(f"Instrumental access granted via instrumental_token for job {job_id}")
return job_id, "instrumental_token"
else:
logger.warning(f"Invalid instrumental token for job {job_id}")
# No valid authentication
raise HTTPException(
status_code=401,
detail="Authentication required. Provide either a full access token or a valid instrumental_token for this job."
)
return require_instrumental_auth
# Default instance for most instrumental review endpoints
require_instrumental_auth = require_instrumental_auth_factory()
+14 -294
View File
@@ -989,23 +989,6 @@ class JobResetResponse(BaseModel):
new_status: str
message: str
cleared_data: List[str]
worker_triggered: Optional[bool] = None # Was worker auto-triggered? (only for instrumental_selected)
worker_trigger_error: Optional[str] = None # Error message if trigger failed
class TriggerWorkerRequest(BaseModel):
"""Request to trigger a worker for a job."""
worker_type: str = "video" # Currently only video supported
class TriggerWorkerResponse(BaseModel):
"""Response from trigger worker endpoint."""
status: str
job_id: str
worker_type: str
triggered: bool
message: str
error: Optional[str] = None
# States that are allowed as reset targets
@@ -1014,13 +997,10 @@ ALLOWED_RESET_STATES = {
"awaiting_audio_selection",
"awaiting_review",
"awaiting_instrumental_selection",
"instrumental_selected", # Re-run video generation with same settings
}
# State data keys to clear for each reset target
# Keys not in this mapping are preserved
# Note: encoding_progress is included in all states because workers check {worker}_progress.stage
# to determine if they should skip (idempotency). Stale 'complete' markers cause workers to skip.
STATE_DATA_CLEAR_KEYS = {
"pending": [
"audio_search_results",
@@ -1033,7 +1013,6 @@ STATE_DATA_CLEAR_KEYS = {
"video_progress",
"render_progress",
"screens_progress",
"encoding_progress",
],
"awaiting_audio_selection": [
"audio_selection",
@@ -1043,7 +1022,6 @@ STATE_DATA_CLEAR_KEYS = {
"video_progress",
"render_progress",
"screens_progress",
"encoding_progress",
],
"awaiting_review": [
"review_complete",
@@ -1052,28 +1030,12 @@ STATE_DATA_CLEAR_KEYS = {
"video_progress",
"render_progress",
"screens_progress",
"encoding_progress",
],
"awaiting_instrumental_selection": [
"instrumental_selection",
"video_progress",
"render_progress",
"screens_progress",
"encoding_progress",
],
"instrumental_selected": [
# Clear video/encoding/distribution state to allow re-processing
# Preserves: instrumental_selection, lyrics_metadata, review data
"video_progress",
"render_progress",
"screens_progress",
"encoding_progress",
"distribution",
"brand_code",
"youtube_url",
"youtube_video_id",
"dropbox_link",
"gdrive_files",
],
}
@@ -1157,263 +1119,39 @@ async def reset_job(
detail="Failed to reset job. Please try again."
)
# Clear the state data keys and error state using direct Firestore update
# Clear the state data keys separately using direct Firestore update
from google.cloud.firestore_v1 import DELETE_FIELD, ArrayUnion
job_ref = user_service.db.collection("jobs").document(job_id)
# Build the update payload
clear_updates = {}
# Clear state_data keys
if cleared_keys:
clear_updates = {}
for key in cleared_keys:
clear_updates[f"state_data.{key}"] = DELETE_FIELD
# Always clear error state on reset (confusing to have old errors after reset)
clear_updates["error_message"] = DELETE_FIELD
clear_updates["error_details"] = DELETE_FIELD
# Add timeline event
clear_updates["timeline"] = ArrayUnion([timeline_event])
# Add timeline event
clear_updates["timeline"] = ArrayUnion([timeline_event])
job_ref.update(clear_updates)
else:
# Just add timeline event
job_ref.update({
"timeline": ArrayUnion([timeline_event])
})
# Execute the update
job_ref.update(clear_updates)
# Debug logging to verify keys were actually cleared
# Log the admin action
logger.info(
f"Admin {admin_email} reset job {job_id} from {previous_status} to {target_state}. "
f"Attempted to clear state_data keys: {cleared_keys}"
f"Cleared state_data keys: {cleared_keys}"
)
# Verify the keys were actually deleted (debug check)
job_after = job_manager.get_job(job_id)
if job_after and cleared_keys:
remaining_keys = [k for k in cleared_keys if k in (job_after.state_data or {})]
if remaining_keys:
logger.error(
f"Admin reset job {job_id}: Some keys were NOT cleared: {remaining_keys}. "
f"Current state_data keys: {list((job_after.state_data or {}).keys())}"
)
else:
logger.debug(f"Admin reset job {job_id}: Verified all keys were cleared successfully")
# Trigger video worker for states that should auto-continue
# instrumental_selected: Re-run video generation with same instrumental selection
worker_triggered: Optional[bool] = None
worker_trigger_error: Optional[str] = None
if target_state == "instrumental_selected":
try:
from backend.services.worker_service import get_worker_service
worker_service = get_worker_service()
worker_triggered = await worker_service.trigger_video_worker(job_id)
if worker_triggered:
logger.info(f"Admin reset: Triggered video worker for job {job_id}")
else:
worker_trigger_error = "Worker trigger returned False - check worker service logs"
logger.warning(f"Admin reset: Failed to trigger video worker for job {job_id}")
except Exception as e:
worker_triggered = False
worker_trigger_error = str(e)
logger.error(f"Admin reset: Exception triggering video worker for job {job_id}: {e}", exc_info=True)
message = f"Job reset from {previous_status} to {target_state}"
if worker_triggered:
message += " (video worker triggered)"
elif target_state == "instrumental_selected" and not worker_triggered:
message += " (WARNING: video worker NOT triggered - use manual trigger)"
return JobResetResponse(
status="success",
job_id=job_id,
previous_status=previous_status,
new_status=target_state,
message=message,
message=f"Job reset from {previous_status} to {target_state}",
cleared_data=cleared_keys,
worker_triggered=worker_triggered,
worker_trigger_error=worker_trigger_error,
)
# =============================================================================
# Trigger Worker Endpoint
# =============================================================================
@router.post("/jobs/{job_id}/trigger-worker", response_model=TriggerWorkerResponse)
async def trigger_worker(
job_id: str,
request: TriggerWorkerRequest,
auth_data: AuthResult = Depends(require_admin),
):
"""
Manually trigger a worker for a job (admin only).
Use this when:
- Auto-trigger fails after a reset to instrumental_selected
- Need to re-run processing without resetting state
- Debugging worker trigger issues
Currently supports:
- video: Triggers the video generation worker
The job should be in an appropriate state for the worker type:
- video: Job should be in 'instrumental_selected' status
Args:
job_id: Job ID to trigger worker for
request: Contains worker_type (default: "video")
Returns:
Trigger result with success/failure and error details
"""
admin_email = auth_data.user_email or "unknown"
worker_type = request.worker_type.lower()
# Validate worker type
if worker_type not in ["video"]:
raise HTTPException(
status_code=400,
detail=f"Unsupported worker type '{worker_type}'. Supported: video"
)
# Get job to validate it exists
job_manager = JobManager()
job = job_manager.get_job(job_id)
if not job:
raise HTTPException(status_code=404, detail=f"Job {job_id} not found")
# Warn if job is not in the expected state (but don't block)
expected_status = "instrumental_selected"
if job.status != expected_status:
logger.warning(
f"Admin {admin_email} triggering {worker_type} worker for job {job_id} "
f"in unexpected status '{job.status}' (expected '{expected_status}')"
)
# Trigger the worker
triggered = False
error_msg = None
try:
from backend.services.worker_service import get_worker_service
worker_service = get_worker_service()
if worker_type == "video":
triggered = await worker_service.trigger_video_worker(job_id)
if triggered:
logger.info(
f"Admin {admin_email} manually triggered {worker_type} worker for job {job_id}"
)
message = f"Successfully triggered {worker_type} worker"
else:
error_msg = "Worker service returned False - check service logs"
logger.warning(
f"Admin {admin_email} failed to trigger {worker_type} worker for job {job_id}: "
f"service returned False"
)
message = f"Failed to trigger {worker_type} worker"
except Exception as e:
triggered = False
error_msg = str(e)
logger.error(
f"Admin {admin_email} exception triggering {worker_type} worker for job {job_id}: {e}",
exc_info=True
)
message = f"Error triggering {worker_type} worker"
return TriggerWorkerResponse(
status="success" if triggered else "error",
job_id=job_id,
worker_type=worker_type,
triggered=triggered,
message=message,
error=error_msg,
)
# =============================================================================
# Clear Worker State Endpoint
# =============================================================================
class ClearWorkersResponse(BaseModel):
"""Response from clear workers endpoint."""
status: str
job_id: str
message: str
cleared_keys: List[str]
# All worker progress keys that can be cleared
ALL_WORKER_PROGRESS_KEYS = [
"audio_progress",
"lyrics_progress",
"render_progress",
"screens_progress",
"video_progress",
"encoding_progress",
]
@router.post("/jobs/{job_id}/clear-workers", response_model=ClearWorkersResponse)
async def clear_worker_state(
job_id: str,
auth_data: AuthResult = Depends(require_admin),
):
"""
Clear all worker completion markers to allow re-execution (admin only).
This is an escape hatch for edge cases where you need workers to run again
without doing a full status reset. It clears all *_progress keys from state_data.
Use cases:
- Worker was interrupted and left stale "running" progress
- Worker completed but needs to run again (e.g., after code fix)
- Debugging worker idempotency issues
Note: This does NOT change the job status. If the job is in a terminal state
(complete, failed, cancelled), workers may not be triggered automatically.
Use the reset endpoint to also change status if needed.
"""
from google.cloud.firestore_v1 import DELETE_FIELD
admin_email = auth_data.user_email or "unknown"
job_manager = JobManager()
job = job_manager.get_job(job_id)
if not job:
raise HTTPException(status_code=404, detail=f"Job {job_id} not found")
# Check which keys actually exist in state_data
state_data = job.state_data or {}
keys_to_clear = [k for k in ALL_WORKER_PROGRESS_KEYS if k in state_data]
if not keys_to_clear:
return ClearWorkersResponse(
status="success",
job_id=job_id,
message="No worker progress keys found in state_data",
cleared_keys=[],
)
# Clear the worker progress keys
db = JobManager().firestore.db
job_ref = db.collection("jobs").document(job_id)
update_payload = {f"state_data.{key}": DELETE_FIELD for key in keys_to_clear}
job_ref.update(update_payload)
logger.info(
f"Admin {admin_email} cleared worker state for job {job_id}. "
f"Cleared keys: {keys_to_clear}"
)
return ClearWorkersResponse(
status="success",
job_id=job_id,
message=f"Cleared {len(keys_to_clear)} worker progress key(s)",
cleared_keys=keys_to_clear,
)
@@ -1575,23 +1313,6 @@ async def delete_job_outputs(
logger.error(f"Error deleting Google Drive files for job {job_id}: {e}", exc_info=True)
results["gdrive"] = {"status": "error", "error": str(e)}
# Clean up GCS finals folder - prevents stale files from being picked up during re-encoding
try:
from backend.services.storage_service import StorageService
storage = StorageService()
finals_prefix = f"jobs/{job_id}/finals/"
deleted_count = storage.delete_folder(finals_prefix)
if deleted_count > 0:
logger.info(f"Deleted {deleted_count} files from GCS finals folder for job {job_id}")
results["gcs_finals"] = {
"status": "success",
"deleted_count": deleted_count,
"path": finals_prefix,
}
except Exception as e:
logger.error(f"Error deleting GCS finals folder for job {job_id}: {e}", exc_info=True)
results["gcs_finals"] = {"status": "error", "error": str(e)}
# Update job record
deletion_timestamp = datetime.now(timezone.utc)
user_service = get_user_service()
@@ -1643,8 +1364,7 @@ async def delete_job_outputs(
f"Admin {admin_email} deleted outputs for job {job_id}. "
f"YouTube: {results['youtube']['status']}, "
f"Dropbox: {results['dropbox']['status']}, "
f"GDrive: {results['gdrive']['status']}, "
f"GCS Finals: {results['gcs_finals']['status']}. "
f"GDrive: {results['gdrive']['status']}. "
f"Cleared state_data keys: {cleared_keys}"
)
+98 -120
View File
@@ -13,9 +13,7 @@ The flow is:
"""
import asyncio
import logging
import mimetypes
import os
import shutil
import tempfile
from typing import Optional, List, Dict, Any
@@ -35,10 +33,6 @@ from backend.services.audio_search_service import (
NoResultsError,
DownloadError,
)
from backend.services.youtube_download_service import (
get_youtube_download_service,
YouTubeDownloadError,
)
from backend.services.theme_service import get_theme_service
from backend.services.job_defaults_service import resolve_cdg_txt_defaults
from backend.config import get_settings
@@ -244,24 +238,6 @@ async def _trigger_workers_parallel(job_id: str) -> None:
)
def _extract_gcs_path(filepath: str) -> str:
"""
Extract the path portion from a GCS path or gs:// URL.
Args:
filepath: Either a gs:// URL or a plain path
Returns:
The path portion (e.g., "uploads/job123/audio/file.flac")
"""
if filepath.startswith("gs://"):
# Format: gs://bucket/path/to/file
parts = filepath.replace("gs://", "").split("/", 1)
if len(parts) == 2:
return parts[1]
return filepath
async def _download_and_start_processing(
job_id: str,
selection_index: int,
@@ -312,127 +288,129 @@ async def _download_and_start_processing(
)
try:
# Get source info from selected result
source_name = selected.get('provider')
# Determine if this is a remote torrent download
is_torrent_source = selected.get('provider') in ['RED', 'OPS']
is_remote_enabled = audio_search_service.is_remote_enabled()
# Check if we can use download_by_id (preferred - avoids re-searching)
source_id = selected.get('source_id')
source_name = selected.get('provider')
target_file = selected.get('target_file')
download_url = selected.get('url')
# Route to appropriate download handler based on source type
if source_name == 'YouTube':
# Use YouTubeDownloadService for all YouTube downloads
# This ensures consistent handling whether remote is configured or not
youtube_service = get_youtube_download_service()
if not source_id and download_url:
# Extract video ID from URL if not in source_id
source_id = youtube_service._extract_video_id(download_url)
if not source_id:
raise DownloadError(f"No video ID available for YouTube download")
logger.info(f"YouTube download via YouTubeDownloadService: video_id={source_id}")
try:
audio_gcs_path = await youtube_service.download_by_id(
video_id=source_id,
job_id=job_id,
artist=selected.get('artist'),
title=selected.get('title'),
)
filename = os.path.basename(audio_gcs_path)
except YouTubeDownloadError as e:
raise DownloadError(f"YouTube download failed: {e}")
elif source_name in ['RED', 'OPS']:
# Torrent sources - must use remote flacfetch
if not audio_search_service.is_remote_enabled():
raise DownloadError(
f"Cannot download from {source_name} without remote flacfetch service. "
"Configure FLACFETCH_API_URL."
)
# For remote torrent downloads, have flacfetch VM upload directly to GCS
if is_torrent_source and is_remote_enabled:
# Generate GCS path for remote upload
gcs_destination = f"uploads/{job_id}/audio/"
if source_id:
logger.info(f"Torrent download via download_by_id: {source_name} ID={source_id}")
# Use download_by_id if we have source_id (preferred - no re-search needed)
if source_id and source_name:
logger.info(f"Using download_by_id for {source_name} ID={source_id} with GCS upload to: {gcs_destination}")
result = audio_search_service.download_by_id(
source_name=source_name,
source_id=source_id,
output_dir="",
output_dir="", # Not used for remote
target_file=target_file,
download_url=download_url,
gcs_path=gcs_destination,
)
else:
logger.info(f"Torrent download via search-based download")
# Fallback to search-based download (requires re-search)
logger.info(f"No source_id available, falling back to search-based download to: {gcs_destination}")
result = audio_search_service.download(
result_index=selection_index,
output_dir="",
output_dir="", # Not used for remote
gcs_path=gcs_destination,
remote_search_id=remote_search_id,
)
# Extract GCS path from result
audio_gcs_path = _extract_gcs_path(result.filepath)
filename = os.path.basename(result.filepath)
logger.info(f"Torrent download complete: {audio_gcs_path}")
elif source_name == 'Spotify':
# Spotify downloads - use audio_search_service
if not audio_search_service.is_remote_enabled():
raise DownloadError(
f"Cannot download from Spotify without remote flacfetch service. "
"Configure FLACFETCH_API_URL."
)
gcs_destination = f"uploads/{job_id}/audio/"
logger.info(f"Spotify download: source_id={source_id}")
result = audio_search_service.download_by_id(
source_name=source_name,
source_id=source_id,
output_dir="",
download_url=download_url,
gcs_path=gcs_destination,
)
audio_gcs_path = _extract_gcs_path(result.filepath)
filename = os.path.basename(result.filepath)
logger.info(f"Spotify download complete: {audio_gcs_path}")
else:
# Unknown source - try generic download
logger.warning(f"Unknown source type: {source_name}, attempting generic download")
temp_dir = tempfile.mkdtemp(prefix=f"audio_download_{job_id}_")
try:
result = audio_search_service.download(
result_index=selection_index,
output_dir=temp_dir,
remote_search_id=remote_search_id,
)
# For remote downloads, filepath is already the GCS path
if result.filepath.startswith("gs://"):
# Extract the path portion after the bucket name
# Format: gs://bucket/uploads/job_id/audio/filename.flac
parts = result.filepath.replace("gs://", "").split("/", 1)
if len(parts) == 2:
audio_gcs_path = parts[1]
else:
audio_gcs_path = result.filepath
filename = os.path.basename(result.filepath)
else:
# Fallback: treat as local path (shouldn't happen for remote)
filename = os.path.basename(result.filepath)
audio_gcs_path = f"uploads/{job_id}/audio/{filename}"
content_type, _ = mimetypes.guess_type(result.filepath)
logger.warning(f"Remote download returned local path: {result.filepath}, uploading manually")
with open(result.filepath, 'rb') as f:
storage_service.upload_fileobj(
f,
audio_gcs_path,
content_type=content_type or 'application/octet-stream'
)
storage_service.upload_fileobj(f, audio_gcs_path, content_type='audio/flac')
logger.info(f"Generic download uploaded to GCS: {audio_gcs_path}")
finally:
# Clean up temp files
try:
if os.path.exists(temp_dir):
import shutil
shutil.rmtree(temp_dir)
except Exception as e:
logger.warning(f"Failed to clean up temp files: {e}")
logger.info(f"Remote download complete, GCS path: {audio_gcs_path}")
else:
# Local download (YouTube or fallback)
temp_dir = tempfile.mkdtemp(prefix=f"audio_download_{job_id}_")
# Use download_by_id if we have source_id and remote is enabled
if source_id and source_name and is_remote_enabled:
logger.info(f"Using download_by_id for local download: {source_name} ID={source_id}")
result = audio_search_service.download_by_id(
source_name=source_name,
source_id=source_id,
output_dir=temp_dir,
target_file=target_file,
download_url=download_url,
)
elif source_name == 'YouTube' and download_url:
# For YouTube without remote, download directly using URL from state_data
# This avoids relying on in-memory cache which doesn't persist across instances
logger.info(f"Downloading YouTube audio directly from URL: {download_url}")
from backend.workers.audio_worker import download_from_url
local_path = await download_from_url(
download_url,
temp_dir,
selected.get('artist'),
selected.get('title')
)
if not local_path or not os.path.exists(local_path):
raise DownloadError(f"Failed to download from YouTube: {download_url}")
# Create a result-like object
class DownloadResult:
def __init__(self, filepath):
self.filepath = filepath
result = DownloadResult(local_path)
else:
# Fallback to search-based download (may fail if cache is empty)
logger.warning(f"Falling back to cache-based download for {source_name} - this may fail on multi-instance deployments")
result = audio_search_service.download(
result_index=selection_index,
output_dir=temp_dir,
remote_search_id=remote_search_id, # Pass for potential fallback scenarios
)
# Upload to GCS
filename = os.path.basename(result.filepath)
audio_gcs_path = f"uploads/{job_id}/audio/{filename}"
with open(result.filepath, 'rb') as f:
storage_service.upload_fileobj(
f,
audio_gcs_path,
content_type='audio/flac' # flacfetch typically returns FLAC
)
logger.info(f"Uploaded audio to GCS: {audio_gcs_path}")
# Clean up temp file
try:
os.remove(result.filepath)
os.rmdir(temp_dir)
except Exception as e:
logger.warning(f"Failed to clean up temp files: {e}")
# Update job with GCS path and transition to DOWNLOADING
job_manager.update_job(job_id, {
+7 -59
View File
@@ -39,19 +39,8 @@ from backend.services.metrics import metrics
from backend.api.dependencies import require_auth
from backend.services.auth_service import UserType, AuthResult
from backend.middleware.tenant import get_tenant_config_from_request
from backend.services.youtube_download_service import (
get_youtube_download_service,
YouTubeDownloadError,
)
logger = logging.getLogger(__name__)
def _is_youtube_url(url: str) -> bool:
"""Check if a URL is a YouTube URL."""
return any(domain in url.lower() for domain in [
'youtube.com', 'youtu.be', 'youtube-nocookie.com'
])
router = APIRouter(tags=["jobs"])
@@ -1587,7 +1576,7 @@ async def create_job_from_url(
logger.info(f" Artist: {artist}")
if title:
logger.info(f" Title: {title}")
# Transition job to DOWNLOADING state
job_manager.transition_to_state(
job_id=job_id,
@@ -1596,48 +1585,11 @@ async def create_job_from_url(
message="Starting audio download from URL"
)
# For YouTube URLs, download audio NOW using YouTubeDownloadService
# This uses remote flacfetch (if configured) to avoid bot detection on Cloud Run
if _is_youtube_url(body.url):
logger.info(f"YouTube URL detected, downloading via YouTubeDownloadService")
try:
youtube_service = get_youtube_download_service()
audio_gcs_path = await youtube_service.download(
url=body.url,
job_id=job_id,
artist=artist,
title=title,
)
# Update job with the downloaded audio path
job_manager.update_job(job_id, {
'input_media_gcs_path': audio_gcs_path,
'filename': os.path.basename(audio_gcs_path),
})
logger.info(f"YouTube audio downloaded to GCS: {audio_gcs_path}")
# Now trigger both workers in parallel (audio already downloaded)
job_manager.transition_to_state(
job_id=job_id,
new_status=JobStatus.DOWNLOADING,
progress=10,
message="Audio downloaded, starting processing"
)
background_tasks.add_task(_trigger_workers_parallel, job_id)
except YouTubeDownloadError as e:
logger.error(f"YouTube download failed: {e}")
job_manager.fail_job(job_id, f"YouTube download failed: {e}")
raise HTTPException(
status_code=500,
detail=f"YouTube download failed: {e}"
)
else:
# For non-YouTube URLs, trigger ONLY audio worker first
# The audio worker will download the URL and trigger lyrics worker after
background_tasks.add_task(_trigger_audio_worker_only, job_id)
# For URL jobs, trigger ONLY audio worker first
# The audio worker will trigger lyrics worker after download completes
# This prevents race condition where lyrics worker times out waiting for audio
background_tasks.add_task(_trigger_audio_worker_only, job_id)
return CreateJobFromUrlResponse(
status="success",
job_id=job_id,
@@ -1927,13 +1879,9 @@ async def mark_finalise_uploads_complete(
):
"""
Mark finalise-only file uploads as complete and start video generation.
This is called after uploading prep outputs for a finalise-only job.
The job will transition directly to AWAITING_INSTRUMENTAL_SELECTION.
Note: Finalise-only jobs are a special case where users upload pre-rendered
video and only need to select instrumental. For normal jobs, instrumental
selection is combined with lyrics review in a single step.
"""
try:
# Get job and verify it exists
-31
View File
@@ -274,37 +274,6 @@ async def encoding_worker_health() -> Dict[str, Any]:
}
@router.get("/health/flacfetch")
async def flacfetch_health() -> Dict[str, Any]:
"""
Lightweight endpoint to check flacfetch service status.
Returns minimal info for frontend footer display.
No authentication required.
"""
status = await check_flacfetch_service_status()
# Return simplified response for frontend
if not status.get("configured"):
return {
"available": False,
"status": "not_configured",
}
if not status.get("available"):
return {
"available": False,
"status": "offline",
"error": status.get("error"),
}
return {
"available": True,
"status": "ok",
"version": status.get("version"),
}
@router.get("/health/detailed")
async def detailed_health_check() -> Dict[str, Any]:
"""
+2 -197
View File
@@ -288,7 +288,7 @@ async def trigger_render_video_worker(
Idempotency: If worker is already running or complete, returns early.
Output: with_vocals.mkv in GCS
Next state: INSTRUMENTAL_SELECTED (instrumental is now selected during review)
Next state: AWAITING_INSTRUMENTAL_SELECTION
"""
job_id = request.job_id
@@ -326,7 +326,7 @@ async def check_idle_reminder(
Check if a job needs an idle reminder email.
This endpoint is called by a Cloud Tasks scheduled task 5 minutes after
a job enters AWAITING_REVIEW state.
a job enters a blocking state (AWAITING_REVIEW or AWAITING_INSTRUMENTAL_SELECTION).
If the job is still in the blocking state and no reminder has been sent yet,
sends a reminder email to the user.
@@ -352,7 +352,6 @@ async def check_idle_reminder(
return {"status": "not_found", "job_id": job_id, "message": "Job not found"}
# Check if job is still in a blocking state
# Note: AWAITING_INSTRUMENTAL_SELECTION is LEGACY - kept for historical jobs only
blocking_states = [JobStatus.AWAITING_REVIEW, JobStatus.AWAITING_INSTRUMENTAL_SELECTION]
if job.status not in [s.value for s in blocking_states]:
logger.info(f"[job:{job_id}] Job no longer in blocking state ({job.status}), skipping reminder")
@@ -440,197 +439,3 @@ async def internal_health(
"""
return {"status": "healthy", "service": "karaoke-backend-internal"}
# =============================================================================
# Test Webhook Endpoint (for E2E testing)
# =============================================================================
class TestWebhookRequest(BaseModel):
"""
Request to simulate a Stripe webhook event for E2E testing.
This allows E2E tests to trigger payment flow logic without requiring
actual Stripe checkout sessions or valid webhook signatures.
"""
event_type: str # e.g., "checkout.session.completed"
session_id: str # Must start with "e2e-test-" prefix
customer_email: str
metadata: dict # order_type, package_id, credits, artist, title, etc.
class TestWebhookResponse(BaseModel):
"""Response from test webhook processing."""
status: str # "processed", "already_processed", "error"
job_id: Optional[str] = None # For made-for-you orders
credits_added: Optional[int] = None # For credit purchases
new_balance: Optional[int] = None # For credit purchases
message: str
@router.post("/test-webhook", response_model=TestWebhookResponse)
async def test_webhook(
request: TestWebhookRequest,
auth_data: Tuple[str, UserType, int] = Depends(require_admin)
):
"""
Test endpoint that simulates Stripe webhook events for E2E testing.
SECURITY:
- Protected by admin authentication (X-Admin-Token header)
- Session IDs must start with "e2e-test-" prefix to prevent collision
with real Stripe sessions
- Only for E2E testing - bypasses Stripe signature verification
This endpoint reuses the same handler logic as the real webhook endpoint,
ensuring E2E tests validate actual business logic.
Supported event types:
- checkout.session.completed: Handles credit purchases and made-for-you orders
For credit purchases, metadata must include:
- package_id: e.g., "1_credit"
- credits: e.g., "1"
- user_email: Email of user to credit
For made-for-you orders, metadata must include:
- order_type: "made_for_you"
- customer_email: Customer email for delivery
- artist: Song artist
- title: Song title
- source_type: "search" or "youtube"
- youtube_url: (optional) If source_type is "youtube"
- notes: (optional) Customer notes
"""
from backend.services.user_service import get_user_service
from backend.services.email_service import get_email_service
from backend.services.stripe_service import get_stripe_service
from backend.api.routes.users import _handle_made_for_you_order
# Validate session_id prefix for safety
if not request.session_id.startswith("e2e-test-"):
logger.warning(f"Test webhook rejected: session_id '{request.session_id}' missing required prefix")
raise HTTPException(
status_code=400,
detail="Session ID must start with 'e2e-test-' prefix for test webhooks"
)
logger.info(f"TEST_WEBHOOK event_type={request.event_type} session_id={request.session_id}")
add_span_attribute("event_type", request.event_type)
add_span_attribute("session_id", request.session_id)
add_span_attribute("is_test_webhook", True)
user_service = get_user_service()
email_service = get_email_service()
stripe_service = get_stripe_service()
if request.event_type == "checkout.session.completed":
session_id = request.session_id
metadata = request.metadata
# Idempotency check: Skip if this session was already processed
if user_service.is_stripe_session_processed(session_id):
logger.info(f"Test webhook: session {session_id} already processed")
return TestWebhookResponse(
status="already_processed",
message=f"Session {session_id} was already processed"
)
# Check if this is a made-for-you order
if metadata.get("order_type") == "made_for_you":
try:
# Call the same handler used by the real webhook
await _handle_made_for_you_order(
session_id=session_id,
metadata=metadata,
user_service=user_service,
email_service=email_service,
)
# Get the job ID from the most recent job for this customer
# The handler creates a job, so we need to find it
from google.cloud import firestore
from google.cloud.firestore_v1 import FieldFilter
db = user_service.db
# Look for the job by session_id pattern in state_data or by customer_email
# Since the job was just created, query by customer_email and made_for_you flag
customer_email = metadata.get("customer_email", "")
jobs_query = db.collection("jobs").where(
filter=FieldFilter("customer_email", "==", customer_email)
).where(
filter=FieldFilter("made_for_you", "==", True)
).order_by("created_at", direction=firestore.Query.DESCENDING).limit(1)
jobs = list(jobs_query.stream())
job_id = jobs[0].to_dict().get("job_id") if jobs else None
logger.info(f"Test webhook: made-for-you order processed, job_id={job_id}")
return TestWebhookResponse(
status="processed",
job_id=job_id,
message=f"Made-for-you order created successfully"
)
except Exception as e:
logger.exception(f"Test webhook: error processing made-for-you order: {e}")
return TestWebhookResponse(
status="error",
message=f"Error processing made-for-you order: {str(e)}"
)
else:
# Handle regular credit purchase
# Build a synthetic session object that matches Stripe's format
synthetic_session = {
"id": session_id,
"customer_email": request.customer_email,
"metadata": metadata,
}
success, user_email, credits, msg = stripe_service.handle_checkout_completed(
synthetic_session
)
if not success:
logger.warning(f"Test webhook: credit purchase validation failed: {msg}")
return TestWebhookResponse(
status="error",
message=msg
)
if user_email and credits > 0:
# Add credits to user account
ok, new_balance, credit_msg = user_service.add_credits(
email=user_email,
amount=credits,
reason="stripe_purchase",
stripe_session_id=session_id,
)
if ok:
# Send confirmation email (same as real webhook)
email_service.send_credits_added(user_email, credits, new_balance)
logger.info(f"Test webhook: added {credits} credits to {user_email}, new balance: {new_balance}")
return TestWebhookResponse(
status="processed",
credits_added=credits,
new_balance=new_balance,
message=f"Added {credits} credits to {user_email}"
)
else:
logger.error(f"Test webhook: failed to add credits: {credit_msg}")
return TestWebhookResponse(
status="error",
message=f"Failed to add credits: {credit_msg}"
)
return TestWebhookResponse(
status="error",
message="Invalid credit purchase data"
)
else:
# Unsupported event type
logger.warning(f"Test webhook: unsupported event type '{request.event_type}'")
return TestWebhookResponse(
status="error",
message=f"Unsupported event type: {request.event_type}"
)
+422 -60
View File
@@ -4,30 +4,31 @@ Job management routes.
Handles job lifecycle endpoints including:
- Job creation and submission
- Status polling
- Human-in-the-loop interactions (combined lyrics + instrumental review)
- Human-in-the-loop interactions (lyrics review, instrumental selection)
- Job deletion and cancellation
"""
import asyncio
import logging
from typing import List, Optional, Dict, Any
import httpx
from typing import List, Optional, Dict, Any, Tuple
from fastapi import APIRouter, HTTPException, BackgroundTasks, Depends, Request
from backend.models.job import Job, JobCreate, JobResponse, JobStatus
from backend.models.requests import (
URLSubmissionRequest,
CorrectionsSubmission,
InstrumentalSelection,
StartReviewRequest,
CancelJobRequest,
InstrumentalSelection,
CompleteReviewRequest,
CreateCustomInstrumentalRequest,
)
from backend.services.job_manager import JobManager
from backend.services.worker_service import get_worker_service
from backend.services.storage_service import StorageService
from backend.services.theme_service import get_theme_service
from backend.config import get_settings
from backend.api.dependencies import require_admin, require_auth
from backend.services.auth_service import AuthResult
from backend.api.dependencies import require_admin, require_auth, require_instrumental_auth
from backend.services.auth_service import UserType, AuthResult
from backend.services.metrics import metrics
from backend.middleware.tenant import get_tenant_from_request
from backend.utils.test_data import is_test_email
@@ -574,20 +575,17 @@ async def submit_corrections(
async def complete_review(
job_id: str,
background_tasks: BackgroundTasks,
auth_result: AuthResult = Depends(require_auth),
body: Optional[CompleteReviewRequest] = None
auth_result: AuthResult = Depends(require_auth)
) -> dict:
"""
Complete the human review and trigger video rendering.
Supports both legacy flows (no body) and combined review flow (with instrumental_selection).
When instrumental_selection is provided, it's stored in state_data for the render worker.
This is the FIRST critical human-in-the-loop completion point.
After this:
1. Job transitions to REVIEW_COMPLETE
2. Render video worker is triggered
3. Worker uses OutputGenerator to create with_vocals.mkv
4. Job transitions to INSTRUMENTAL_SELECTED then GENERATING_VIDEO
4. Job transitions to AWAITING_INSTRUMENTAL_SELECTION
"""
job = job_manager.get_job(job_id)
if not job:
@@ -602,102 +600,466 @@ async def complete_review(
status_code=400,
detail=f"Job not in review state (current status: {job.status})"
)
try:
# Store instrumental selection if provided (combined review flow)
instrumental_selection = body.instrumental_selection if body else None
if instrumental_selection:
job_manager.update_state_data(job_id, 'instrumental_selection', instrumental_selection)
logger.info(f"Job {job_id}: Stored instrumental selection: {instrumental_selection}")
# Transition to REVIEW_COMPLETE
message = f"Review complete (instrumental: {instrumental_selection})" if instrumental_selection else "Review complete"
job_manager.transition_to_state(
job_id=job_id,
new_status=JobStatus.REVIEW_COMPLETE,
progress=70,
message=f"{message}, rendering video with corrected lyrics"
message="Review complete, rendering video with corrected lyrics"
)
# Trigger render video worker
background_tasks.add_task(worker_service.trigger_render_video_worker, job_id)
logger.info(f"Job {job_id}: Review complete, triggering render video worker")
return {
"status": "success",
"job_status": "review_complete",
"message": "Review complete. Video rendering started."
}
except Exception as e:
logger.error(f"Error completing review for job {job_id}: {e}", exc_info=True)
raise HTTPException(status_code=500, detail=str(e))
@router.get("/{job_id}/instrumental-options")
async def get_instrumental_options(
job_id: str,
auth_info: Tuple[str, str] = Depends(require_instrumental_auth)
) -> Dict[str, Any]:
"""
Get instrumental audio options for user selection.
Returns signed URLs for both options:
1. Clean instrumental (no backing vocals)
2. Instrumental with backing vocals
Accepts either full auth token or job-specific instrumental_token.
"""
job = job_manager.get_job(job_id)
if not job:
raise HTTPException(status_code=404, detail="Job not found")
if job.status != JobStatus.AWAITING_INSTRUMENTAL_SELECTION:
raise HTTPException(
status_code=400,
detail=f"Job not ready for instrumental selection (current status: {job.status})"
)
# Get stem URLs
stems = job.file_urls.get('stems', {})
clean_url = stems.get('instrumental_clean')
backing_url = stems.get('instrumental_with_backing')
if not clean_url or not backing_url:
raise HTTPException(
status_code=500,
detail="Instrumental options not available"
)
# Generate signed URLs
from backend.services.storage_service import StorageService
storage = StorageService()
return {
"options": [
{
"id": "clean",
"label": "Clean Instrumental (no backing vocals)",
"audio_url": storage.generate_signed_url(clean_url, expiration_minutes=120),
"duration_seconds": None # TODO: Extract from audio file
},
{
"id": "with_backing",
"label": "Instrumental with Backing Vocals",
"audio_url": storage.generate_signed_url(backing_url, expiration_minutes=120),
"duration_seconds": None # TODO: Extract from audio file
}
],
"status": job.status,
"artist": job.artist,
"title": job.title
}
@router.get("/{job_id}/instrumental-analysis")
async def get_instrumental_analysis(
job_id: str,
auth_info: Tuple[str, str] = Depends(require_instrumental_auth)
) -> Dict[str, Any]:
"""
Get audio analysis data for instrumental selection.
Returns:
- Analysis of backing vocals (audible segments, recommendation)
- Waveform image URL
- Audio stream URLs for playback
This endpoint enables intelligent instrumental selection by providing
detailed analysis of the backing vocals track.
Accepts either full auth token or job-specific instrumental_token.
"""
job = job_manager.get_job(job_id)
if not job:
raise HTTPException(status_code=404, detail="Job not found")
if job.status != JobStatus.AWAITING_INSTRUMENTAL_SELECTION:
raise HTTPException(
status_code=400,
detail=f"Job not ready for instrumental analysis (current status: {job.status})"
)
storage = StorageService()
# Get analysis from state_data (populated by render_video_worker)
analysis_data = job.state_data.get('backing_vocals_analysis', {})
# Get URLs
stems = job.file_urls.get('stems', {})
analysis_files = job.file_urls.get('analysis', {})
clean_url = stems.get('instrumental_clean')
backing_vocals_url = stems.get('backing_vocals')
with_backing_url = stems.get('instrumental_with_backing')
waveform_url = analysis_files.get('backing_vocals_waveform')
custom_instrumental_url = stems.get('instrumental_custom')
# Build response
response = {
"job_id": job_id,
"artist": job.artist,
"title": job.title,
"status": job.status,
# Analysis results
"analysis": {
"has_audible_content": analysis_data.get('has_audible_content', False),
"total_duration_seconds": analysis_data.get('total_duration_seconds', 0),
"audible_segments": analysis_data.get('audible_segments', []),
"recommended_selection": analysis_data.get('recommended_selection', 'review_needed'),
"total_audible_duration_seconds": analysis_data.get('total_audible_duration_seconds', 0),
"audible_percentage": analysis_data.get('audible_percentage', 0),
"silence_threshold_db": analysis_data.get('silence_threshold_db', -40.0),
},
# Audio URLs for playback
"audio_urls": {
"clean_instrumental": storage.generate_signed_url(clean_url, expiration_minutes=120) if clean_url else None,
"backing_vocals": storage.generate_signed_url(backing_vocals_url, expiration_minutes=120) if backing_vocals_url else None,
"with_backing": storage.generate_signed_url(with_backing_url, expiration_minutes=120) if with_backing_url else None,
"custom_instrumental": storage.generate_signed_url(custom_instrumental_url, expiration_minutes=120) if custom_instrumental_url else None,
},
# Waveform image URL
"waveform_url": storage.generate_signed_url(waveform_url, expiration_minutes=120) if waveform_url else None,
# Whether a custom instrumental has been created
"has_custom_instrumental": custom_instrumental_url is not None,
}
return response
@router.get("/{job_id}/audio-stream/{stem_type}")
async def stream_audio(
job_id: str,
stem_type: str,
auth_info: Tuple[str, str] = Depends(require_instrumental_auth)
):
"""
Stream an audio file for playback in the browser.
Supported stem_type values:
- clean_instrumental: Clean instrumental (no backing vocals)
- backing_vocals: Backing vocals only
- with_backing: Instrumental with backing vocals
- custom_instrumental: Custom instrumental (if created)
Returns audio as a streaming response with proper headers for
browser audio playback.
Accepts either full auth token or job-specific instrumental_token.
"""
from fastapi.responses import StreamingResponse
import tempfile
job = job_manager.get_job(job_id)
if not job:
raise HTTPException(status_code=404, detail="Job not found")
# Map stem_type to file_urls key
stem_map = {
'clean_instrumental': ('stems', 'instrumental_clean'),
'backing_vocals': ('stems', 'backing_vocals'),
'with_backing': ('stems', 'instrumental_with_backing'),
'custom_instrumental': ('stems', 'instrumental_custom'),
}
if stem_type not in stem_map:
raise HTTPException(
status_code=400,
detail=f"Invalid stem_type. Must be one of: {list(stem_map.keys())}"
)
category, key = stem_map[stem_type]
gcs_path = job.file_urls.get(category, {}).get(key)
if not gcs_path:
raise HTTPException(
status_code=404,
detail=f"Audio file not available: {stem_type}"
)
# Determine content type
ext = gcs_path.split('.')[-1].lower()
content_types = {
'flac': 'audio/flac',
'mp3': 'audio/mpeg',
'wav': 'audio/wav',
'm4a': 'audio/mp4',
}
content_type = content_types.get(ext, 'audio/flac')
try:
storage = StorageService()
with tempfile.NamedTemporaryFile(delete=False, suffix=f'.{ext}') as tmp:
tmp_path = tmp.name
storage.download_file(gcs_path, tmp_path)
def file_iterator():
try:
with open(tmp_path, 'rb') as f:
while chunk := f.read(8192):
yield chunk
finally:
import os
os.unlink(tmp_path)
filename = gcs_path.split('/')[-1]
return StreamingResponse(
file_iterator(),
media_type=content_type,
headers={
'Content-Disposition': f'inline; filename="{filename}"',
'Accept-Ranges': 'bytes',
}
)
except Exception as e:
logger.exception(f"Error streaming audio {gcs_path}: {e}")
raise HTTPException(status_code=500, detail=f"Error streaming audio: {e}") from e
@router.post("/{job_id}/create-custom-instrumental")
async def create_custom_instrumental(
job_id: str,
request: CreateCustomInstrumentalRequest,
auth_info: Tuple[str, str] = Depends(require_instrumental_auth)
) -> Dict[str, Any]:
"""
Create a custom instrumental by muting regions of backing vocals.
This endpoint:
1. Takes a list of time ranges to mute in the backing vocals
2. Creates a custom instrumental (clean + muted backing vocals)
3. Stores the result and makes it available for selection
After calling this endpoint, the user can select "custom" in the
select-instrumental endpoint.
Accepts either full auth token or job-specific instrumental_token.
"""
from backend.services.audio_editing_service import AudioEditingService
from karaoke_gen.instrumental_review import MuteRegion
job = job_manager.get_job(job_id)
if not job:
raise HTTPException(status_code=404, detail="Job not found")
if job.status != JobStatus.AWAITING_INSTRUMENTAL_SELECTION:
raise HTTPException(
status_code=400,
detail=f"Job not ready for custom instrumental creation (current status: {job.status})"
)
# Get stem paths
stems = job.file_urls.get('stems', {})
clean_path = stems.get('instrumental_clean')
backing_path = stems.get('backing_vocals')
if not clean_path or not backing_path:
raise HTTPException(
status_code=500,
detail="Required audio files not available"
)
try:
# Convert request mute regions to model objects
mute_regions = [
MuteRegion(
start_seconds=r.start_seconds,
end_seconds=r.end_seconds
)
for r in request.mute_regions
]
# Create custom instrumental
editing_service = AudioEditingService()
# Determine output path
output_path = f"jobs/{job_id}/stems/instrumental_custom.flac"
result = editing_service.create_custom_instrumental(
gcs_clean_instrumental_path=clean_path,
gcs_backing_vocals_path=backing_path,
mute_regions=mute_regions,
gcs_output_path=output_path,
job_id=job_id,
)
# Update job file_urls with custom instrumental
job_manager.update_file_url(job_id, 'stems', 'instrumental_custom', output_path)
# Store mute regions in state_data for reference
job_manager.update_state_data(job_id, 'custom_instrumental_mute_regions', [
{"start_seconds": r.start_seconds, "end_seconds": r.end_seconds}
for r in mute_regions
])
logger.info(f"Job {job_id}: Custom instrumental created with {len(mute_regions)} mute regions")
# Generate signed URL for the new file
storage = StorageService()
return {
"status": "success",
"message": "Custom instrumental created successfully",
"custom_instrumental_url": storage.generate_signed_url(output_path, expiration_minutes=120),
"mute_regions_applied": len(result.mute_regions_applied),
"total_muted_duration_seconds": result.total_muted_duration_seconds,
"output_duration_seconds": result.output_duration_seconds,
}
except Exception as e:
logger.exception(f"Error creating custom instrumental for job {job_id}: {e}")
raise HTTPException(status_code=500, detail=str(e)) from e
@router.get("/{job_id}/waveform-data")
async def get_waveform_data(
job_id: str,
num_points: int = 500,
auth_info: Tuple[str, str] = Depends(require_instrumental_auth)
) -> Dict[str, Any]:
"""
Get waveform amplitude data for client-side rendering.
This endpoint returns raw amplitude data that the frontend can use
to render a waveform using Canvas or SVG, enabling interactive
features like click-to-seek.
Args:
num_points: Number of data points to return (default 500)
Returns:
{
"amplitudes": [0.1, 0.2, ...], # Normalized 0-1 values
"duration_seconds": 180.5,
"num_points": 500
}
Accepts either full auth token or job-specific instrumental_token.
"""
from backend.services.audio_analysis_service import AudioAnalysisService
job = job_manager.get_job(job_id)
if not job:
raise HTTPException(status_code=404, detail="Job not found")
if job.status != JobStatus.AWAITING_INSTRUMENTAL_SELECTION:
raise HTTPException(
status_code=400,
detail=f"Job not ready for waveform data (current status: {job.status})"
)
backing_vocals_path = job.file_urls.get('stems', {}).get('backing_vocals')
if not backing_vocals_path:
raise HTTPException(status_code=404, detail="Backing vocals file not found")
try:
analysis_service = AudioAnalysisService()
amplitudes, duration = analysis_service.get_waveform_data(
gcs_audio_path=backing_vocals_path,
job_id=job_id,
num_points=num_points,
)
return {
"amplitudes": amplitudes,
"duration_seconds": duration,
"num_points": len(amplitudes),
}
except Exception as e:
logger.exception(f"Error getting waveform data for job {job_id}: {e}")
raise HTTPException(status_code=500, detail=str(e)) from e
@router.post("/{job_id}/select-instrumental")
async def select_instrumental(
job_id: str,
selection: InstrumentalSelection,
background_tasks: BackgroundTasks,
auth_result: AuthResult = Depends(require_auth)
auth_info: Tuple[str, str] = Depends(require_instrumental_auth)
) -> dict:
"""
Select instrumental audio option for finalise-only jobs.
This endpoint is used for jobs that enter AWAITING_INSTRUMENTAL_SELECTION
state directly (finalise-only jobs where users upload pre-rendered video).
For normal jobs, instrumental selection happens during the combined review
flow via the complete-review endpoint.
Args:
job_id: Job identifier
selection: Instrumental selection (clean, with_backing, custom, uploaded)
Returns:
Status and confirmation of selection
Submit instrumental selection.
This is the SECOND critical human-in-the-loop interaction point.
After selection, the job proceeds to video generation.
Accepts either full auth token or job-specific instrumental_token.
"""
job = job_manager.get_job(job_id)
if not job:
raise HTTPException(status_code=404, detail="Job not found")
# Check ownership
if not _check_job_ownership(job, auth_result):
raise HTTPException(status_code=403, detail="You don't have permission to access this job")
if job.status != JobStatus.AWAITING_INSTRUMENTAL_SELECTION:
raise HTTPException(
status_code=400,
detail=f"Job not awaiting instrumental selection (current status: {job.status})"
detail=f"Job not ready for instrumental selection (current status: {job.status})"
)
try:
# Store selection in state_data
state_data = job.state_data or {}
state_data["instrumental_selection"] = selection.selection
job_manager.update_job(job_id, state_data=state_data)
job_manager.update_state_data(job_id, 'instrumental_selection', selection.selection)
# Transition to INSTRUMENTAL_SELECTED
job_manager.transition_to_state(
job_id=job_id,
new_status=JobStatus.INSTRUMENTAL_SELECTED,
progress=80,
message=f"Instrumental selection: {selection.selection}"
progress=65,
message=f"Instrumental selected: {selection.selection}"
)
# Trigger video worker
# Trigger video generation worker
background_tasks.add_task(worker_service.trigger_video_worker, job_id)
logger.info(f"Job {job_id}: Instrumental selected ({selection.selection}), triggering video worker")
logger.info(f"Job {job_id}: Instrumental selected ({selection.selection}), triggering video generation")
return {
"status": "success",
"job_status": "instrumental_selected",
"selection": selection.selection,
"message": "Instrumental selected. Video generation started."
"message": "Selection accepted, starting video generation"
}
except Exception as e:
logger.error(f"Error selecting instrumental for job {job_id}: {e}", exc_info=True)
raise HTTPException(status_code=500, detail=str(e))
+47 -261
View File
@@ -1,15 +1,12 @@
"""
Review API routes - Combined lyrics + instrumental review.
Review API routes - Compatible with LyricsTranscriber frontend.
These endpoints support the combined review flow where users review lyrics
AND select their instrumental in a single session.
The correction-data endpoint now includes instrumental options and backing
vocals analysis, and the complete endpoint requires instrumental selection.
These endpoints match the API that the LyricsTranscriber review frontend expects,
allowing us to use the existing React review UI with our cloud backend.
Usage:
Frontend URL: http://localhost:5173/?baseApiUrl=http://localhost:8000/api/review/{job_id}
The baseApiUrl includes the job_id, and all endpoints are relative to that.
"""
import asyncio
@@ -75,54 +72,39 @@ async def get_correction_data(
auth_info: Tuple[str, str] = Depends(require_review_auth)
):
"""
Get correction data for the combined review interface.
Returns the CorrectionResult data plus instrumental options and analysis
for the combined lyrics + instrumental review UI.
Response includes:
- All lyrics correction data (segments, reference lyrics, anchors, etc.)
- instrumental_options: List of available instrumental tracks with audio URLs
- backing_vocals_analysis: Analysis data to help users choose instrumental
Get correction data for the review interface.
Returns the CorrectionResult data that the frontend needs to render
the lyrics review UI.
"""
job_manager = JobManager()
storage = StorageService()
job = job_manager.get_job(job_id)
if not job:
raise HTTPException(status_code=404, detail="Job not found")
if job.status not in [JobStatus.AWAITING_REVIEW, JobStatus.IN_REVIEW]:
raise HTTPException(
status_code=400,
detail=f"Job not ready for review (current status: {job.status})"
)
# Check for updated corrections first (from previous review sessions where user edited lyrics)
# This mirrors the logic in render_video_worker.py to ensure consistency
corrections_updated_gcs = job.file_urls.get('lyrics', {}).get('corrections_updated')
if not corrections_updated_gcs:
# Try direct path for updated corrections
corrections_updated_gcs = f"jobs/{job_id}/lyrics/corrections_updated.json"
if corrections_updated_gcs and storage.file_exists(corrections_updated_gcs):
corrections_gcs = corrections_updated_gcs
logger.info(f"Job {job_id}: Using updated corrections from previous review")
else:
# Fall back to original corrections
corrections_gcs = job.file_urls.get('lyrics', {}).get('corrections')
if not corrections_gcs:
corrections_gcs = f"jobs/{job_id}/lyrics/corrections.json"
if not storage.file_exists(corrections_gcs):
raise HTTPException(
status_code=404,
detail="Corrections data not found. Lyrics processing may not be complete."
)
# Get corrections URL from file_urls
corrections_gcs = job.file_urls.get('lyrics', {}).get('corrections')
if not corrections_gcs:
# Try direct path
corrections_gcs = f"jobs/{job_id}/lyrics/corrections.json"
if not storage.file_exists(corrections_gcs):
raise HTTPException(
status_code=404,
detail="Corrections data not found. Lyrics processing may not be complete."
)
# Download and return corrections data
try:
corrections_data = storage.download_json(corrections_gcs)
# Add audio hash for the frontend
audio_hash = _get_audio_hash(job_id)
if 'metadata' not in corrections_data:
@@ -130,62 +112,24 @@ async def get_correction_data(
corrections_data['metadata']['audio_hash'] = audio_hash
corrections_data['metadata']['artist'] = job.artist
corrections_data['metadata']['title'] = job.title
# Store context for audio serving
_job_contexts[job_id] = {
'audio_hash': audio_hash,
'audio_gcs_path': job.input_media_gcs_path
}
# === Add instrumental data for combined review ===
# Get instrumental stem URLs
stems = job.file_urls.get('stems', {})
clean_url = stems.get('instrumental_clean')
backing_url = stems.get('instrumental_with_backing')
# Build instrumental options with signed URLs
instrumental_options = []
if clean_url:
instrumental_options.append({
"id": "clean",
"label": "Clean Instrumental",
"description": "No backing vocals - just the music",
"audio_url": storage.generate_signed_url(clean_url, expiration_minutes=120),
})
if backing_url:
instrumental_options.append({
"id": "with_backing",
"label": "Instrumental with Backing Vocals",
"description": "Includes harmonies and background vocals",
"audio_url": storage.generate_signed_url(backing_url, expiration_minutes=120),
})
corrections_data['instrumental_options'] = instrumental_options
# Get backing vocals analysis from state_data (populated by screens_worker)
backing_vocals_analysis = job.state_data.get('backing_vocals_analysis', {})
corrections_data['backing_vocals_analysis'] = backing_vocals_analysis
# Get waveform URL if available
analysis_files = job.file_urls.get('analysis', {})
waveform_url = analysis_files.get('backing_vocals_waveform')
if waveform_url:
corrections_data['backing_vocals_waveform_url'] = storage.generate_signed_url(
waveform_url, expiration_minutes=120
)
# Transition to IN_REVIEW if not already
if job.status == JobStatus.AWAITING_REVIEW:
job_manager.transition_to_state(
job_id=job_id,
new_status=JobStatus.IN_REVIEW,
message="User opened combined review interface"
message="User opened review interface"
)
logger.info(f"Job {job_id}: Serving correction data with instrumental options for combined review")
logger.info(f"Job {job_id}: Serving correction data for review")
return corrections_data
except Exception as e:
logger.error(f"Job {job_id}: Error loading corrections: {e}", exc_info=True)
raise HTTPException(status_code=500, detail=f"Error loading corrections: {str(e)}")
@@ -265,99 +209,53 @@ async def complete_review(
auth_info: Tuple[str, str] = Depends(require_review_auth)
):
"""
Complete the combined review - save corrected lyrics AND instrumental selection.
This endpoint receives:
- Updated correction data (lyrics corrections)
- instrumental_selection: "clean" or "with_backing" (REQUIRED)
After saving, triggers the render video worker which will use the
pre-selected instrumental (no separate instrumental selection step).
Complete the review and save corrected lyrics.
This endpoint receives the updated correction data from the frontend
and saves it, then triggers the render video worker.
"""
job_manager = JobManager()
storage = StorageService()
job = job_manager.get_job(job_id)
if not job:
raise HTTPException(status_code=404, detail="Job not found")
if job.status not in [JobStatus.AWAITING_REVIEW, JobStatus.IN_REVIEW]:
raise HTTPException(
status_code=400,
detail=f"Job not in review state (current status: {job.status})"
)
# === Require instrumental selection ===
instrumental_selection = updated_data.get("instrumental_selection")
if not instrumental_selection:
raise HTTPException(
status_code=400,
detail="instrumental_selection is required. Must be 'clean' or 'with_backing'."
)
valid_selections = ["clean", "with_backing"]
# Also allow "custom" for user-provided instrumentals
if job.existing_instrumental_gcs_path:
valid_selections.append("custom")
if instrumental_selection not in valid_selections:
raise HTTPException(
status_code=400,
detail=f"Invalid instrumental_selection. Must be one of: {valid_selections}"
)
try:
# Remove instrumental_selection from updated_data before saving to corrections
# (it's stored separately in state_data)
corrections_to_save = {k: v for k, v in updated_data.items() if k != "instrumental_selection"}
# Save updated corrections to GCS
corrections_gcs_path = f"jobs/{job_id}/lyrics/corrections_updated.json"
storage.upload_json(corrections_gcs_path, corrections_to_save)
storage.upload_json(corrections_gcs_path, updated_data)
job_manager.update_file_url(job_id, 'lyrics', 'corrections_updated', corrections_gcs_path)
logger.info(f"Job {job_id}: Saved updated corrections")
# === Store instrumental selection in state_data ===
# This will be used by render_video_worker to skip AWAITING_INSTRUMENTAL_SELECTION
job_manager.update_state_data(job_id, 'instrumental_selection', instrumental_selection)
logger.info(f"Job {job_id}: Stored instrumental selection: {instrumental_selection}")
# Clear worker progress keys to ensure workers will run fresh (not skip due to idempotency)
# This handles cases where a job is re-reviewed after completion or where state was inconsistent.
# The idempotency check in workers looks at {worker}_progress.stage == 'complete' to skip,
# so we must clear these before triggering the render-video worker.
progress_keys = ['render_progress', 'screens_progress', 'video_progress', 'encoding_progress']
cleared = job_manager.delete_state_data_keys(job_id, progress_keys)
if cleared:
logger.info(f"Job {job_id}: Cleared worker progress keys for re-processing: {cleared}")
# Transition to REVIEW_COMPLETE
job_manager.transition_to_state(
job_id=job_id,
new_status=JobStatus.REVIEW_COMPLETE,
progress=70,
message=f"Review complete (instrumental: {instrumental_selection}), rendering video"
message="Review complete, rendering video with corrected lyrics"
)
# Trigger render video worker
from backend.services.worker_service import get_worker_service
worker_service = get_worker_service()
# Run in background, keep reference to prevent garbage collection
task = asyncio.create_task(worker_service.trigger_render_video_worker(job_id))
_background_tasks.add(task)
task.add_done_callback(_background_tasks.discard)
logger.info(f"Job {job_id}: Combined review complete, triggered render video worker")
return {
"status": "success",
"instrumental_selection": instrumental_selection,
}
except HTTPException:
raise
logger.info(f"Job {job_id}: Review complete, triggered render video worker")
return {"status": "success"}
except Exception as e:
logger.error(f"Job {job_id}: Error completing review: {e}", exc_info=True)
raise HTTPException(status_code=500, detail=f"Error completing review: {str(e)}")
@@ -759,115 +657,3 @@ async def get_annotation_stats(
"by_type": {},
"message": "Annotation stats not yet implemented"
}
@router.get("/{job_id}/instrumental-analysis")
async def get_instrumental_analysis(
job_id: str,
auth_info: Tuple[str, str] = Depends(require_review_auth)
):
"""
Get instrumental analysis data for the review interface.
Returns analysis data in the format expected by InstrumentalSelector component:
- analysis: backing vocals analysis with audible segments
- audio_urls: signed URLs for instrumental stems
- has_original: whether original audio is available
"""
job_manager = JobManager()
storage = StorageService()
job = job_manager.get_job(job_id)
if not job:
raise HTTPException(status_code=404, detail="Job not found")
# Get backing vocals analysis from state_data
backing_analysis = job.state_data.get('backing_vocals_analysis', {})
# Get stem URLs
stems = job.file_urls.get('stems', {})
clean_url = stems.get('instrumental_clean')
backing_url = stems.get('instrumental_with_backing')
# Build audio URLs with signed access
audio_urls = {}
if clean_url:
audio_urls['clean'] = storage.generate_signed_url(clean_url, expiration_minutes=120)
if backing_url:
audio_urls['with_backing'] = storage.generate_signed_url(backing_url, expiration_minutes=120)
if job.input_media_gcs_path:
audio_urls['original'] = storage.generate_signed_url(job.input_media_gcs_path, expiration_minutes=120)
# Check for backing vocals stem for playback
backing_vocals_path = stems.get('backing_vocals')
if backing_vocals_path:
audio_urls['backing_vocals'] = storage.generate_signed_url(backing_vocals_path, expiration_minutes=120)
# Format response to match InstrumentalAnalysis type
return {
"job_id": job_id,
"artist": job.artist,
"title": job.title,
"duration_seconds": backing_analysis.get('total_duration_seconds'),
"analysis": {
"has_audible_content": backing_analysis.get('has_audible_content', False),
"total_duration_seconds": backing_analysis.get('total_duration_seconds', 0),
"audible_segments": backing_analysis.get('audible_segments', []),
"recommended_selection": backing_analysis.get('recommended_selection', 'with_backing'),
"total_audible_duration_seconds": backing_analysis.get('total_audible_duration_seconds', 0),
"audible_percentage": backing_analysis.get('audible_percentage', 0),
"silence_threshold_db": backing_analysis.get('silence_threshold_db', -40),
},
"audio_urls": audio_urls,
"has_original": bool(job.input_media_gcs_path),
"has_uploaded_instrumental": bool(job.existing_instrumental_gcs_path),
}
@router.get("/{job_id}/waveform-data")
async def get_waveform_data(
job_id: str,
num_points: int = 1000,
auth_info: Tuple[str, str] = Depends(require_review_auth)
):
"""
Get waveform amplitude data for client-side rendering.
Returns amplitude values for drawing the waveform in the frontend.
"""
from backend.services.audio_analysis_service import AudioAnalysisService
job_manager = JobManager()
job = job_manager.get_job(job_id)
if not job:
raise HTTPException(status_code=404, detail="Job not found")
# Get backing vocals path
stems = job.file_urls.get('stems', {})
backing_vocals_path = stems.get('backing_vocals')
if not backing_vocals_path:
# Fallback to input audio if no backing vocals
backing_vocals_path = job.input_media_gcs_path
if not backing_vocals_path:
raise HTTPException(status_code=404, detail="No audio available for waveform")
try:
analysis_service = AudioAnalysisService()
amplitudes, duration = analysis_service.get_waveform_data(
gcs_audio_path=backing_vocals_path,
job_id=job_id,
num_points=num_points,
)
return {
"amplitudes": amplitudes,
"duration_seconds": duration,
"duration": duration, # Legacy field name
"sample_rate": num_points,
}
except Exception as e:
logger.error(f"Job {job_id}: Error generating waveform data: {e}", exc_info=True)
raise HTTPException(status_code=500, detail=f"Error generating waveform: {str(e)}")
+6 -41
View File
@@ -61,10 +61,6 @@ from backend.api.dependencies import require_admin
from backend.api.routes.file_upload import _prepare_theme_for_job
from backend.services.auth_service import UserType
from backend.utils.test_data import is_test_email
from backend.services.youtube_download_service import (
get_youtube_download_service,
YouTubeDownloadError,
)
logger = logging.getLogger(__name__)
@@ -582,43 +578,12 @@ async def _handle_made_for_you_order(
# Handle based on whether we have a YouTube URL or need to search
if youtube_url:
# URL provided - download audio first, then trigger workers
# CRITICAL: Workers require input_media_gcs_path to be set before they can process
# See docs/LESSONS-LEARNED.md "YouTube URL Downloads Must Happen Before Workers"
logger.info(f"Job {job_id}: YouTube URL provided, downloading audio first")
try:
youtube_service = get_youtube_download_service()
audio_gcs_path = await youtube_service.download(
url=youtube_url,
job_id=job_id,
artist=artist,
title=title,
)
# Update job with the downloaded audio path BEFORE triggering workers
job_manager.update_job(job_id, {
'input_media_gcs_path': audio_gcs_path,
'filename': os.path.basename(audio_gcs_path),
})
logger.info(f"Job {job_id}: YouTube audio downloaded to {audio_gcs_path}, triggering workers")
# Now trigger both workers in parallel (audio already downloaded)
await asyncio.gather(
worker_service.trigger_audio_worker(job_id),
worker_service.trigger_lyrics_worker(job_id)
)
except YouTubeDownloadError as e:
logger.error(f"Job {job_id}: YouTube download failed: {e}")
job_manager.transition_to_state(
job_id=job_id,
new_status=JobStatus.FAILED,
progress=0,
message=f"YouTube download failed: {str(e)}"
)
raise
# URL provided - trigger workers directly (no audio selection needed)
logger.info(f"Job {job_id}: YouTube URL provided, triggering workers")
await asyncio.gather(
worker_service.trigger_audio_worker(job_id),
worker_service.trigger_lyrics_worker(job_id)
)
else:
# No URL - use audio search flow, pause for admin selection
# Made-for-you jobs require admin to select audio source
-8
View File
@@ -41,15 +41,7 @@ class Settings(BaseSettings):
max_concurrent_jobs: int = int(os.getenv("MAX_CONCURRENT_JOBS", "5"))
job_timeout_seconds: int = int(os.getenv("JOB_TIMEOUT_SECONDS", "3600"))
# Lyrics Correction Settings
# Skip all auto-correction (both agentic AI and heuristic handlers).
# When true, raw transcription goes directly to human review without any automatic fixes.
# This is the default because auto-correction currently creates more work for reviewers.
# Set SKIP_CORRECTION=false to re-enable auto-correction if quality improves.
skip_correction: bool = os.getenv("SKIP_CORRECTION", "true").lower() in ("true", "1", "yes")
# Agentic AI Correction (for lyrics correction via LLM)
# Only used when skip_correction=false
# When enabled, uses Gemini via Vertex AI for intelligent lyrics correction
use_agentic_ai: bool = os.getenv("USE_AGENTIC_AI", "true").lower() in ("true", "1", "yes")
agentic_ai_model: str = os.getenv("AGENTIC_AI_MODEL", "vertexai/gemini-3-flash-preview")
+46 -53
View File
@@ -15,58 +15,54 @@ from karaoke_gen.utils import normalize_text
class JobStatus(str, Enum):
"""
Job status enumeration - Complete state machine.
The workflow has 7 main stages with 2 human interaction points:
The workflow has 8 main stages with 3 human interaction points:
1. Input & Setup (may include audio source selection)
2. Parallel Processing (audio + lyrics)
3. Title/End Screen Generation + Backing Vocals Analysis
3. Title/End Screen Generation
4. Countdown Padding Synchronization
5. Combined Human Review (BLOCKING) - lyrics review + instrumental selection
6. Video Finalization
7. Distribution
Note: As of 2026-01, lyrics review and instrumental selection are combined
into a single human review step. The user selects their instrumental
during the lyrics review phase, before the video is rendered.
5. Human Review (BLOCKING)
6. Instrumental Selection (BLOCKING)
7. Video Finalization
8. Distribution
"""
# Initial states
PENDING = "pending" # Job created, queued for processing
# Audio search states (for artist+title search mode)
SEARCHING_AUDIO = "searching_audio" # Searching for audio sources via flacfetch
AWAITING_AUDIO_SELECTION = "awaiting_audio_selection" # ⚠️ WAITING FOR USER - select audio source
DOWNLOADING_AUDIO = "downloading_audio" # Downloading selected audio from source
DOWNLOADING = "downloading" # Downloading from URL or processing upload
# Stage 2a: Audio separation (parallel track 1)
SEPARATING_STAGE1 = "separating_stage1" # Clean instrumental separation (Modal API)
SEPARATING_STAGE2 = "separating_stage2" # Backing vocals separation (Modal API)
AUDIO_COMPLETE = "audio_complete" # All audio stems ready
# Stage 2b: Lyrics processing (parallel track 2)
TRANSCRIBING = "transcribing" # AudioShake API transcription
CORRECTING = "correcting" # Automatic lyrics correction
LYRICS_COMPLETE = "lyrics_complete" # Corrections JSON ready
# Stage 3: Title/End screens + Backing Vocals Analysis
# Stage 3: Title/End screens
GENERATING_SCREENS = "generating_screens" # Creating title and end screen videos
# Stage 4: Countdown padding (automatic)
APPLYING_PADDING = "applying_padding" # Synchronizing countdown padding
# Stage 5: Combined Human Review (BLOCKING) - lyrics + instrumental selection
AWAITING_REVIEW = "awaiting_review" # ⚠️ WAITING FOR USER - combined review needed
IN_REVIEW = "in_review" # User is actively reviewing
REVIEW_COMPLETE = "review_complete" # User submitted lyrics + instrumental selection
# Stage 5: Human review (BLOCKING)
AWAITING_REVIEW = "awaiting_review" # ⚠️ WAITING FOR USER - lyrics review needed
IN_REVIEW = "in_review" # User is actively reviewing lyrics
REVIEW_COMPLETE = "review_complete" # User submitted corrected lyrics
# Stage 5.5: Render video with corrected lyrics (post-review)
RENDERING_VIDEO = "rendering_video" # Using OutputGenerator to create with_vocals.mkv
# Stage 6: Instrumental already selected (during combined review)
# Note: AWAITING_INSTRUMENTAL_SELECTION kept for DB compatibility with historical jobs only
AWAITING_INSTRUMENTAL_SELECTION = "awaiting_instrumental_selection" # LEGACY - no longer used
INSTRUMENTAL_SELECTED = "instrumental_selected" # Instrumental was selected during combined review
# Stage 6: Instrumental selection (BLOCKING)
AWAITING_INSTRUMENTAL_SELECTION = "awaiting_instrumental_selection" # ⚠️ WAITING FOR USER
INSTRUMENTAL_SELECTED = "instrumental_selected" # User made selection
# Stage 7: Video generation and finalization
GENERATING_VIDEO = "generating_video" # Creating initial karaoke video
@@ -93,74 +89,71 @@ class JobStatus(str, Enum):
# Valid state transitions
STATE_TRANSITIONS = {
# PENDING can go to DOWNLOADING (file upload) or SEARCHING_AUDIO (artist+title search)
JobStatus.PENDING: [JobStatus.DOWNLOADING, JobStatus.SEARCHING_AUDIO, JobStatus.FAILED, JobStatus.CANCELLED],
# PENDING can go to DOWNLOADING (file upload), SEARCHING_AUDIO (artist+title search), or AWAITING_INSTRUMENTAL_SELECTION (finalise-only)
JobStatus.PENDING: [JobStatus.DOWNLOADING, JobStatus.SEARCHING_AUDIO, JobStatus.AWAITING_INSTRUMENTAL_SELECTION, JobStatus.FAILED, JobStatus.CANCELLED],
# Audio search flow (for artist+title search mode)
JobStatus.SEARCHING_AUDIO: [JobStatus.AWAITING_AUDIO_SELECTION, JobStatus.DOWNLOADING_AUDIO, JobStatus.FAILED],
JobStatus.AWAITING_AUDIO_SELECTION: [JobStatus.DOWNLOADING_AUDIO, JobStatus.FAILED, JobStatus.CANCELLED],
JobStatus.DOWNLOADING_AUDIO: [JobStatus.DOWNLOADING, JobStatus.FAILED],
# DOWNLOADING allows parallel processing (audio + lyrics) and then screens when both complete
JobStatus.DOWNLOADING: [JobStatus.SEPARATING_STAGE1, JobStatus.TRANSCRIBING, JobStatus.GENERATING_SCREENS, JobStatus.FAILED],
# Audio separation flow
JobStatus.SEPARATING_STAGE1: [JobStatus.SEPARATING_STAGE2, JobStatus.FAILED],
JobStatus.SEPARATING_STAGE2: [JobStatus.AUDIO_COMPLETE, JobStatus.FAILED],
JobStatus.AUDIO_COMPLETE: [JobStatus.GENERATING_SCREENS, JobStatus.FAILED],
# Lyrics flow
JobStatus.TRANSCRIBING: [JobStatus.CORRECTING, JobStatus.FAILED],
JobStatus.CORRECTING: [JobStatus.LYRICS_COMPLETE, JobStatus.FAILED],
JobStatus.LYRICS_COMPLETE: [JobStatus.GENERATING_SCREENS, JobStatus.FAILED],
# Post-parallel processing (screens + backing vocals analysis)
JobStatus.GENERATING_SCREENS: [JobStatus.APPLYING_PADDING, JobStatus.AWAITING_REVIEW, JobStatus.FAILED],
# Post-parallel processing
JobStatus.GENERATING_SCREENS: [JobStatus.APPLYING_PADDING, JobStatus.AWAITING_REVIEW, JobStatus.AWAITING_INSTRUMENTAL_SELECTION, JobStatus.FAILED],
JobStatus.APPLYING_PADDING: [JobStatus.AWAITING_REVIEW, JobStatus.FAILED],
# Combined human review flow (lyrics + instrumental selection)
# Human review flow
# AWAITING_REVIEW can go directly to REVIEW_COMPLETE (quick review) or to IN_REVIEW (editing)
JobStatus.AWAITING_REVIEW: [JobStatus.IN_REVIEW, JobStatus.REVIEW_COMPLETE, JobStatus.FAILED, JobStatus.CANCELLED],
JobStatus.IN_REVIEW: [JobStatus.REVIEW_COMPLETE, JobStatus.AWAITING_REVIEW, JobStatus.FAILED],
JobStatus.REVIEW_COMPLETE: [JobStatus.RENDERING_VIDEO, JobStatus.PREP_COMPLETE, JobStatus.FAILED], # PREP_COMPLETE for prep-only jobs
# Video rendering (post-review) - instrumental was already selected during combined review
JobStatus.RENDERING_VIDEO: [JobStatus.INSTRUMENTAL_SELECTED, JobStatus.PREP_COMPLETE, JobStatus.FAILED],
# AWAITING_INSTRUMENTAL_SELECTION is LEGACY - kept for historical jobs in DB
# New jobs never enter this state; they go directly to INSTRUMENTAL_SELECTED after render
# Video rendering (post-review)
JobStatus.RENDERING_VIDEO: [JobStatus.AWAITING_INSTRUMENTAL_SELECTION, JobStatus.PREP_COMPLETE, JobStatus.FAILED],
# Instrumental selection flow
JobStatus.AWAITING_INSTRUMENTAL_SELECTION: [JobStatus.INSTRUMENTAL_SELECTED, JobStatus.FAILED, JobStatus.CANCELLED],
JobStatus.INSTRUMENTAL_SELECTED: [JobStatus.GENERATING_VIDEO, JobStatus.FAILED],
# Video generation flow
JobStatus.GENERATING_VIDEO: [JobStatus.ENCODING, JobStatus.FAILED],
JobStatus.ENCODING: [JobStatus.PACKAGING, JobStatus.COMPLETE, JobStatus.FAILED],
JobStatus.PACKAGING: [JobStatus.UPLOADING, JobStatus.COMPLETE, JobStatus.FAILED],
# Distribution flow
JobStatus.UPLOADING: [JobStatus.NOTIFYING, JobStatus.COMPLETE, JobStatus.FAILED],
JobStatus.NOTIFYING: [JobStatus.COMPLETE, JobStatus.FAILED],
# Terminal states - COMPLETE, PREP_COMPLETE have no transitions
# FAILED and CANCELLED allow retry transitions to resume from checkpoints
# PREP_COMPLETE allows continuation from combined review
# PREP_COMPLETE allows finalise-only continuation
JobStatus.COMPLETE: [],
JobStatus.PREP_COMPLETE: [JobStatus.AWAITING_REVIEW, JobStatus.FAILED], # Continue from combined review
JobStatus.PREP_COMPLETE: [JobStatus.AWAITING_INSTRUMENTAL_SELECTION, JobStatus.FAILED], # Finalise-only continues from here
JobStatus.FAILED: [
JobStatus.DOWNLOADING, # Retry from beginning (if input audio exists)
JobStatus.INSTRUMENTAL_SELECTED, # Retry from video generation
JobStatus.REVIEW_COMPLETE, # Retry from render stage
JobStatus.LYRICS_COMPLETE, # Retry from screens generation
JobStatus.AWAITING_REVIEW, # Retry from combined review
],
JobStatus.CANCELLED: [
JobStatus.DOWNLOADING, # Retry from beginning (if input audio exists)
JobStatus.INSTRUMENTAL_SELECTED, # Retry from video generation
JobStatus.REVIEW_COMPLETE, # Retry from render stage
JobStatus.LYRICS_COMPLETE, # Retry from screens generation
JobStatus.AWAITING_REVIEW, # Retry from combined review
],
# Legacy states (for backward compatibility)
JobStatus.QUEUED: [JobStatus.PENDING],
JobStatus.PROCESSING: [JobStatus.SEPARATING_STAGE1, JobStatus.TRANSCRIBING],
+2 -22
View File
@@ -57,13 +57,13 @@ class CorrectionsSubmission(BaseModel):
class InstrumentalSelection(BaseModel):
"""
Request to select instrumental audio option.
This is the second critical human-in-the-loop interaction point.
User chooses between clean instrumental, instrumental with backing vocals,
or a custom instrumental (created via create-custom-instrumental endpoint).
"""
selection: str # "clean", "with_backing", or "custom"
@validator('selection')
def validate_selection(cls, v):
"""Validate selection is a valid option."""
@@ -73,26 +73,6 @@ class InstrumentalSelection(BaseModel):
return v
class CompleteReviewRequest(BaseModel):
"""
Request to complete the review with optional instrumental selection.
For the combined review flow, this includes the instrumental selection.
For legacy flows, the instrumental_selection can be omitted.
"""
instrumental_selection: Optional[str] = None # "clean", "with_backing", or "custom"
@validator('instrumental_selection')
def validate_instrumental_selection(cls, v):
"""Validate instrumental selection if provided."""
if v is None:
return v
valid_options = ['clean', 'with_backing', 'custom']
if v not in valid_options:
raise ValueError(f"instrumental_selection must be one of: {valid_options}")
return v
class MuteRegionRequest(BaseModel):
"""A region to mute in the backing vocals."""
start_seconds: float
+1 -3
View File
@@ -204,9 +204,7 @@ class AudioSearchService:
provider=item.get("provider", "Unknown"),
duration=item.get("duration_seconds"),
quality=item.get("quality", ""),
# source_id: use explicit source_id field first (YouTube video ID, Spotify track ID),
# fall back to info_hash for torrent sources (RED/OPS)
source_id=item.get("source_id") or item.get("info_hash"),
source_id=item.get("info_hash"),
index=item.get("index", 0),
seeders=item.get("seeders"),
target_file=item.get("target_file"),
+8 -20
View File
@@ -109,42 +109,30 @@ class DropboxService:
def get_next_brand_code(self, path: str, brand_prefix: str) -> str:
"""
Calculate next brand code from existing folders, filling gaps in the sequence.
Calculate next brand code from existing folders.
Scans the folder for existing brand codes like "NOMAD-0001" and
returns the first available code, filling any gaps left by deleted outputs.
returns the next sequential code.
Args:
path: Dropbox path containing organized folders
brand_prefix: Brand prefix (e.g., "NOMAD")
Returns:
Next brand code (e.g., "NOMAD-0003" if 0003 is the first gap)
Next brand code (e.g., "NOMAD-1164")
"""
folders = self.list_folders(path)
pattern = re.compile(rf"^{re.escape(brand_prefix)}-(\d{{4}})")
existing_nums: set[int] = set()
max_num = 0
for folder in folders:
match = pattern.match(folder)
if match:
existing_nums.add(int(match.group(1)))
num = int(match.group(1))
max_num = max(max_num, num)
max_existing = max(existing_nums) if existing_nums else 0
# Only fill gaps for numbers >= 1001 (preserve legacy gaps below 1000)
if max_existing >= 1000:
next_num = 1001
while next_num in existing_nums:
next_num += 1
else:
next_num = max_existing + 1
next_code = f"{brand_prefix}-{next_num:04d}"
if max_existing >= 1000 and next_num <= max_existing:
logger.info(f"Next brand code for {brand_prefix}: {next_code} (filling gap, max existing: {max_existing})")
else:
logger.info(f"Next brand code for {brand_prefix}: {next_code} (max existing: {max_existing})")
next_code = f"{brand_prefix}-{max_num + 1:04d}"
logger.info(f"Next brand code for {brand_prefix}: {next_code} (max existing: {max_num})")
return next_code
def upload_file(self, local_path: str, remote_path: str) -> None:
-7
View File
@@ -200,7 +200,6 @@ class LocalEncodingBackend(EncodingBackend):
output_lossy_4k_mp4=os.path.join(output_dir, f"{base_name} (Final Karaoke Lossy 4k).mp4"),
output_lossless_mkv=os.path.join(output_dir, f"{base_name} (Final Karaoke Lossless 4k).mkv"),
output_720p_mp4=os.path.join(output_dir, f"{base_name} (Final Karaoke Lossy 720p).mp4"),
countdown_padding_seconds=input_config.options.get("countdown_padding_seconds"),
)
# Run encoding in thread pool to avoid blocking
@@ -335,12 +334,6 @@ class GCEEncodingBackend(EncodingBackend):
"instrumental_selection": input_config.instrumental_selection,
}
# Add countdown padding if present (for audio sync with countdown-padded vocals)
countdown_padding = input_config.options.get("countdown_padding_seconds")
if countdown_padding:
encoding_config["countdown_padding_seconds"] = countdown_padding
self.logger.info(f"GCE encoding with countdown_padding_seconds={countdown_padding}")
# Submit and wait for completion
result = await service.encode_videos(
job_id=job_id,
+4 -28
View File
@@ -325,35 +325,18 @@ def run_encoding(job_id: str, work_dir: Path, config: dict):
end_video = find_file(work_dir, "screens/end.mov", "*End*.mov", "*end*.mov")
# Karaoke video - search for With Vocals or main karaoke video
# IMPORTANT: Search order matters! Search specific paths first to avoid picking up
# old encoding outputs from finals/ directory when re-encoding after reset
karaoke_video = find_file(
work_dir,
# First: Look in videos/ subdirectory (where render_video_worker puts output)
"videos/with_vocals.mkv", "videos/with_vocals.mov",
"videos/*With Vocals*.mkv", "videos/*With Vocals*.mov",
# Second: Look for specific filename patterns (case-insensitive variants)
"*with_vocals*.mkv", "*with_vocals*.mov",
"*With Vocals*.mov", "*With Vocals*.mkv",
"*vocals*.mkv", "*vocals*.mov",
"*Vocals*.mov", "*Vocals*.mkv",
# Last resort: any video file (but excluding finals/outputs)
"*.mkv", "*.mov"
)
# Exclude title/end/output/finals videos to avoid re-encoding old outputs
# Exclude title/end/output videos
if karaoke_video:
path_str = str(karaoke_video).lower()
name_lower = karaoke_video.name.lower()
if ("title" in name_lower or "end" in name_lower or
"outputs" in path_str or "finals" in path_str or
"final karaoke" in name_lower or "lossless" in name_lower or
"lossy" in name_lower or "720p" in name_lower):
# Search more specifically for karaoke video in videos/ directory only
karaoke_video = find_file(
work_dir,
"videos/with_vocals.mkv", "videos/with_vocals.mov",
"videos/*vocals*.mkv", "videos/*vocals*.mov"
)
if "title" in name_lower or "end" in name_lower or "outputs" in str(karaoke_video):
# Search more specifically for karaoke video
karaoke_video = find_file(work_dir, "*Karaoke*.mkv", "*Karaoke*.mov", "*vocals*.mkv")
# Instrumental audio - respect user's selection from encoding config
instrumental_selection = config.get("instrumental_selection", "clean")
@@ -383,11 +366,6 @@ def run_encoding(job_id: str, work_dir: Path, config: dict):
logger.info(f" End video: {end_video}")
logger.info(f" Instrumental ({instrumental_selection}): {instrumental}")
# Check for countdown padding - if vocals were padded, instrumental must match
countdown_padding_seconds = config.get("countdown_padding_seconds")
if countdown_padding_seconds:
logger.info(f" Countdown padding: {countdown_padding_seconds}s - will be handled by LocalEncodingService")
# Validate required files
if not title_video:
raise ValueError(f"No title video found in {work_dir}. Check screens/ subdirectory.")
@@ -402,7 +380,6 @@ def run_encoding(job_id: str, work_dir: Path, config: dict):
jobs[job_id]["progress"] = 20
# Build encoding config with proper file names
# Note: countdown_padding_seconds is passed to LocalEncodingService which handles padding
encoding_config = EncodingConfig(
title_video=str(title_video),
karaoke_video=str(karaoke_video),
@@ -414,7 +391,6 @@ def run_encoding(job_id: str, work_dir: Path, config: dict):
output_lossy_4k_mp4=str(output_dir / f"{base_name} (Final Karaoke Lossy 4k).mp4"),
output_lossless_mkv=str(output_dir / f"{base_name} (Final Karaoke Lossless 4k).mkv"),
output_720p_mp4=str(output_dir / f"{base_name} (Final Karaoke Lossy 720p).mp4"),
countdown_padding_seconds=countdown_padding_seconds,
)
# Create service and run encoding
+19 -57
View File
@@ -381,6 +381,15 @@ class JobManager:
updates['review_token'] = review_token
updates['review_token_expires_at'] = None # No expiry - token is job-scoped
logger.info(f"Generated review token for job {job_id} (no expiry)")
# Generate instrumental token when entering AWAITING_INSTRUMENTAL_SELECTION state
# Tokens don't expire - they're job-scoped so low risk, and natural expiry happens when job completes
if new_status == JobStatus.AWAITING_INSTRUMENTAL_SELECTION:
from backend.api.dependencies import generate_review_token
instrumental_token = generate_review_token() # Reuse same token generator
updates['instrumental_token'] = instrumental_token
updates['instrumental_token_expires_at'] = None # No expiry - token is job-scoped
logger.info(f"Generated instrumental token for job {job_id} (no expiry)")
# If we have state_data_updates, merge them with existing state_data
merged_state_data = None
@@ -412,6 +421,13 @@ class JobManager:
'review_token': updates['review_token'],
'review_token_expires_at': updates['review_token_expires_at']
})
# Apply instrumental token update separately if generated
if new_status == JobStatus.AWAITING_INSTRUMENTAL_SELECTION and 'instrumental_token' in updates:
self.firestore.update_job(job_id, {
'instrumental_token': updates['instrumental_token'],
'instrumental_token_expires_at': updates['instrumental_token_expires_at']
})
logger.info(f"Job {job_id} transitioned to {new_status}")
@@ -626,73 +642,19 @@ class JobManager:
def update_state_data(self, job_id: str, key: str, value: Any) -> None:
"""
Update a specific key in the job's state_data field.
This is used by workers to store stage-specific metadata.
"""
job = self.get_job(job_id)
if not job:
logger.error(f"Job {job_id} not found")
return
state_data = job.state_data.copy()
state_data[key] = value
self.update_job(job_id, {'state_data': state_data})
logger.debug(f"Job {job_id} state_data updated: {key} = {value}")
def delete_state_data_key(self, job_id: str, key: str) -> bool:
"""
Delete a specific key from the job's state_data field.
Uses Firestore DELETE_FIELD to atomically remove the key.
This is used to clear worker progress markers, allowing workers
to re-run without skipping due to idempotency checks.
Args:
job_id: Job ID
key: The state_data key to delete (e.g., "render_progress")
Returns:
True if successful, False otherwise
"""
from google.cloud.firestore_v1 import DELETE_FIELD
try:
job_ref = self.firestore.db.collection("jobs").document(job_id)
job_ref.update({f"state_data.{key}": DELETE_FIELD})
logger.debug(f"Job {job_id}: Deleted state_data key '{key}'")
return True
except Exception as e:
logger.error(f"Job {job_id}: Failed to delete state_data key '{key}': {e}")
return False
def delete_state_data_keys(self, job_id: str, keys: List[str]) -> List[str]:
"""
Delete multiple keys from the job's state_data field in a single operation.
Uses Firestore DELETE_FIELD to atomically remove the keys.
Args:
job_id: Job ID
keys: List of state_data keys to delete
Returns:
List of keys that were successfully deleted
"""
from google.cloud.firestore_v1 import DELETE_FIELD
if not keys:
return []
try:
job_ref = self.firestore.db.collection("jobs").document(job_id)
update_payload = {f"state_data.{key}": DELETE_FIELD for key in keys}
job_ref.update(update_payload)
logger.info(f"Job {job_id}: Deleted state_data keys: {keys}")
return keys
except Exception as e:
logger.error(f"Job {job_id}: Failed to delete state_data keys {keys}: {e}")
return []
def fail_job(self, job_id: str, error_message: str, error_details: Optional[Dict[str, Any]] = None) -> bool:
"""
+4 -6
View File
@@ -147,15 +147,13 @@ class JobNotificationService:
Args:
job_id: Job ID
user_email: User's email address
action_type: Type of action needed:
- "lyrics": Combined review (lyrics + instrumental selection) for normal jobs
- "instrumental": Instrumental-only selection for finalise-only jobs
action_type: Type of action needed ("lyrics" or "instrumental")
user_name: User's display name
artist: Artist name
title: Song title
audio_hash: Audio hash for review URL (unused, kept for compatibility)
review_token: Review token for URL (unused, kept for compatibility)
instrumental_token: Instrumental token for URL (unused, kept for compatibility)
audio_hash: Audio hash for review URL
review_token: Review token for URL
instrumental_token: Instrumental token for URL
Returns:
True if email was sent successfully
+6 -38
View File
@@ -5,19 +5,7 @@ slow initialization during request processing. The CallbackHandler()
constructor makes blocking network calls to the Langfuse API, which can
take 3+ minutes on Cloud Run cold starts.
IMPORTANT: Langfuse v3 is built on OpenTelemetry. If not configured carefully,
it will install itself as the GLOBAL OpenTelemetry tracer provider, capturing
ALL spans from every library (FastAPI, HTTP clients, etc.) - not just LLM calls.
This caused us to exceed our Langfuse hobby tier monthly event limit with
thousands of "(empty)" HTTP request traces.
The fix: Initialize the Langfuse client with an ISOLATED TracerProvider before
creating the CallbackHandler. This ensures only LangChain/LLM traces are sent
to Langfuse, while other OTEL instrumentation remains separate.
See:
- docs/archive/2026-01-08-performance-investigation.md for background
- https://github.com/orgs/langfuse/discussions/9136 for the OTEL issue
See docs/archive/2026-01-08-performance-investigation.md for background.
"""
import logging
@@ -38,10 +26,6 @@ def preload_langfuse_handler() -> Optional[Any]:
Only initializes if LANGFUSE_PUBLIC_KEY and LANGFUSE_SECRET_KEY
environment variables are set.
CRITICAL: We initialize the Langfuse client with an isolated TracerProvider
BEFORE creating the CallbackHandler. This prevents Langfuse from hijacking
the global OpenTelemetry provider and capturing all HTTP request spans.
Returns:
The preloaded CallbackHandler, or None if not configured
"""
@@ -56,44 +40,28 @@ def preload_langfuse_handler() -> Optional[Any]:
# Check if Langfuse is configured
public_key = os.getenv("LANGFUSE_PUBLIC_KEY")
secret_key = os.getenv("LANGFUSE_SECRET_KEY")
host = os.getenv("LANGFUSE_HOST", "https://us.cloud.langfuse.com")
if not (public_key and secret_key):
logger.info("Langfuse not configured (missing keys), skipping preload")
return None
logger.info("Preloading Langfuse callback handler with isolated tracer...")
logger.info("Preloading Langfuse callback handler...")
start_time = time.time()
try:
# Import OpenTelemetry TracerProvider for isolation
from opentelemetry.sdk.trace import TracerProvider
from langfuse import Langfuse
from langfuse.langchain import CallbackHandler
# CRITICAL: Initialize Langfuse client with an ISOLATED TracerProvider
# This prevents Langfuse from setting itself as the global OTEL provider,
# which would cause ALL HTTP requests to be traced (not just LLM calls).
# See: https://github.com/orgs/langfuse/discussions/9136
Langfuse(
public_key=public_key,
secret_key=secret_key,
host=host,
tracer_provider=TracerProvider(), # Isolated - won't touch global OTEL
)
logger.info("Langfuse client initialized with isolated TracerProvider")
# Now create the CallbackHandler - it will reuse the existing client
# instead of creating a new one that hijacks global OTEL
# Initialize the handler - this is the slow part that makes network calls
_preloaded_handler = CallbackHandler()
elapsed = time.time() - start_time
host = os.getenv("LANGFUSE_HOST", "cloud.langfuse.com")
logger.info(f"Langfuse handler preloaded in {elapsed:.2f}s (host: {host})")
return _preloaded_handler
except ImportError as e:
logger.warning(f"langfuse or opentelemetry package not installed, skipping preload: {e}")
except ImportError:
logger.warning("langfuse package not installed, skipping preload")
return None
except Exception as e:
elapsed = time.time() - start_time
+1 -51
View File
@@ -38,9 +38,6 @@ class EncodingConfig:
output_lossless_mkv: Optional[str] = None
output_720p_mp4: Optional[str] = None
# Audio synchronization
countdown_padding_seconds: Optional[float] = None # Pad instrumental to match countdown-padded vocals
@dataclass
class EncodingResult:
@@ -263,38 +260,6 @@ class LocalEncodingService:
return self._execute_command(cpu_command, description)
def _pad_audio_file(self, input_audio: str, output_audio: str, padding_seconds: float) -> str:
"""Pad an audio file by prepending silence at the beginning.
Uses FFmpeg adelay filter - same approach as KaraokeFinalise._pad_audio_file().
This ensures the instrumental audio is synchronized with countdown-padded vocals.
Args:
input_audio: Path to input audio file
output_audio: Path for the padded output file
padding_seconds: Amount of silence to prepend (in seconds)
Returns:
Path to the output audio file
"""
delay_ms = int(padding_seconds * 1000)
cmd = [
"ffmpeg", "-y", "-hide_banner", "-loglevel", "error",
"-i", input_audio,
"-af", f"adelay={delay_ms}|{delay_ms}",
output_audio
]
self.logger.info(f"Padding audio with {padding_seconds}s silence: {os.path.basename(input_audio)}")
if self.dry_run:
self.logger.info(f"DRY RUN: Would pad audio file")
return output_audio
result = subprocess.run(cmd, capture_output=True, text=True)
if result.returncode != 0:
raise RuntimeError(f"FFmpeg padding failed: {result.stderr}")
return output_audio
def remux_with_instrumental(
self,
input_video: str,
@@ -500,27 +465,12 @@ class LocalEncodingService:
output_files = {}
try:
# Pad instrumental if countdown padding was applied to vocals
actual_instrumental = config.instrumental_audio
if config.countdown_padding_seconds and config.countdown_padding_seconds > 0:
if "(Padded)" not in config.instrumental_audio:
base, ext = os.path.splitext(config.instrumental_audio)
padded_path = f"{base} (Padded){ext}"
if not os.path.exists(padded_path) or self.dry_run:
self._pad_audio_file(config.instrumental_audio, padded_path, config.countdown_padding_seconds)
self.logger.info(f"Created padded instrumental: {padded_path}")
else:
self.logger.info(f"Using existing padded instrumental: {padded_path}")
actual_instrumental = padded_path
else:
self.logger.info(f"Instrumental already padded: {config.instrumental_audio}")
# Step 1: Remux with instrumental audio
if config.output_karaoke_mp4:
self.logger.info("[Step 1/6] Remuxing video with instrumental audio...")
if not self.remux_with_instrumental(
config.karaoke_video,
actual_instrumental,
config.instrumental_audio,
config.output_karaoke_mp4
):
return EncodingResult(
+7 -14
View File
@@ -6,8 +6,8 @@ This allows updating email content without code deployment.
Template locations:
- gs://{bucket}/templates/job-completion.txt - Job completion email (plain text)
- gs://{bucket}/templates/action-needed-lyrics.txt - Combined review reminder (lyrics + instrumental)
- gs://{bucket}/templates/action-needed-instrumental.txt - Instrumental selection reminder (finalise-only jobs)
- gs://{bucket}/templates/action-needed-lyrics.txt - Lyrics review reminder
- gs://{bucket}/templates/action-needed-instrumental.txt - Instrumental selection reminder
"""
import logging
import re
@@ -52,11 +52,11 @@ Thanks again and have a great day!
DEFAULT_ACTION_NEEDED_LYRICS_TEMPLATE = """Hi {name},
Your karaoke video for "{artist} - {title}" is ready for review!
Your karaoke video for "{artist} - {title}" is ready for lyrics review!
Our system has transcribed and synchronized the lyrics, but they may need some corrections. You'll also be able to select your preferred instrumental track (with or without backing vocals).
Our system has transcribed and synchronized the lyrics, but they may need some corrections. Please review and make any corrections necessary so we can finish generating your video.
Please review your lyrics and select an instrumental:
Review your lyrics here:
{review_url}
This usually takes just a few minutes.
@@ -248,16 +248,13 @@ class TemplateService:
review_url: str = "",
) -> str:
"""
Render the combined review reminder template.
This is used for the combined lyrics + instrumental review flow.
Users review lyrics and select their instrumental in a single session.
Render the lyrics review reminder template.
Args:
name: User's display name
artist: Artist name
title: Song title
review_url: Combined review URL
review_url: Lyrics review URL
Returns:
Rendered email content
@@ -281,10 +278,6 @@ class TemplateService:
"""
Render the instrumental selection reminder template.
This is only used for finalise-only jobs where users upload pre-rendered
video and only need to select the instrumental track. For normal jobs,
instrumental selection is combined with lyrics review (see render_action_needed_lyrics).
Args:
name: User's display name
artist: Artist name
+18 -72
View File
@@ -357,33 +357,12 @@ class WorkerService:
# These provide a cleaner API and better IDE autocomplete
async def trigger_audio_worker(self, job_id: str) -> bool:
"""
Trigger audio separation worker.
Always uses Cloud Run Jobs to avoid instance termination during long-running
Modal API calls. The BackgroundTasks approach caused Cloud Run to terminate
instances mid-processing when no HTTP requests were active.
"""
return await self._trigger_worker_cloud_run_job(
job_id=job_id,
cloud_run_job_name="audio-separation-job",
worker_module="audio_worker",
)
"""Trigger audio separation worker."""
return await self.trigger_worker("audio", job_id)
async def trigger_lyrics_worker(self, job_id: str) -> bool:
"""
Trigger lyrics transcription worker.
Always uses Cloud Run Jobs to avoid instance termination during long-running
AudioShake transcription. The BackgroundTasks approach caused Cloud Run to
terminate instances mid-processing (e.g., job c94cc9d6) when no HTTP requests
were active, even though the worker was still processing.
"""
return await self._trigger_worker_cloud_run_job(
job_id=job_id,
cloud_run_job_name="lyrics-transcription-job",
worker_module="lyrics_worker",
)
"""Trigger lyrics transcription worker."""
return await self.trigger_worker("lyrics", job_id)
async def trigger_screens_worker(self, job_id: str) -> bool:
"""Trigger screen generation worker."""
@@ -404,83 +383,50 @@ class WorkerService:
async def _trigger_cloud_run_job(self, job_id: str) -> bool:
"""
Trigger video encoding as a Cloud Run Job.
Cloud Run Jobs support up to 24 hours of execution time,
making them suitable for very long video encoding tasks.
Args:
job_id: Job ID to process
Returns:
True if job was triggered successfully, False otherwise
"""
return await self._trigger_worker_cloud_run_job(
job_id=job_id,
cloud_run_job_name="video-encoding-job",
worker_module="video_worker",
)
async def _trigger_worker_cloud_run_job(
self,
job_id: str,
cloud_run_job_name: str,
worker_module: str,
) -> bool:
"""
Trigger a worker as a Cloud Run Job.
Cloud Run Jobs run to completion without HTTP request lifecycle concerns,
avoiding the instance termination issue where Cloud Run would shut down
instances mid-processing when using BackgroundTasks.
Args:
job_id: Job ID to process
cloud_run_job_name: Name of the Cloud Run Job (e.g., "lyrics-transcription-job")
worker_module: Worker module name (e.g., "lyrics_worker")
Returns:
True if job was triggered successfully, False otherwise
"""
try:
from google.cloud import run_v2
project = self.settings.google_cloud_project
if not project:
logger.error("GOOGLE_CLOUD_PROJECT not set, cannot trigger Cloud Run Job")
return False
location = self.settings.gcp_region
job_name = f"projects/{project}/locations/{location}/jobs/{cloud_run_job_name}"
job_name = f"projects/{project}/locations/{location}/jobs/video-encoding-job"
# Create Cloud Run Jobs client
client = run_v2.JobsClient()
# Run the job with overrides for the specific job_id
request = run_v2.RunJobRequest(
name=job_name,
overrides=run_v2.RunJobRequest.Overrides(
container_overrides=[
run_v2.RunJobRequest.Overrides.ContainerOverride(
args=[
"python", "-m", f"backend.workers.{worker_module}",
"--job-id", job_id,
],
args=["python", "-m", "backend.workers.video_worker", "--job-id", job_id],
)
]
)
)
# Run the job (async operation)
operation = client.run_job(request=request)
logger.info(
f"[job:{job_id}] Started Cloud Run Job {cloud_run_job_name}: "
f"{operation.metadata}"
)
logger.info(f"Started Cloud Run Job for video encoding, job {job_id}: {operation.metadata}")
return True
except Exception as e:
logger.error(
f"[job:{job_id}] Failed to trigger Cloud Run Job {cloud_run_job_name}: {e}",
f"Failed to trigger Cloud Run Job for video/{job_id}: {e}",
exc_info=True
)
return False
@@ -1,349 +0,0 @@
"""
YouTube download service for cloud backend.
This service provides a single entry point for all YouTube downloads in the cloud.
It automatically routes downloads through the remote flacfetch service when configured,
avoiding YouTube bot detection issues on Cloud Run IPs.
The flow is:
1. Check if remote flacfetch is configured (FLACFETCH_API_URL)
2. If yes: Use FlacfetchClient.download_by_id() - downloads on VM, uploads to GCS
3. If no: Use local yt_dlp (with warning about bot detection risk)
All entry points (audio search selection, direct URL submission) should use this
service for YouTube downloads to ensure consistent behavior.
"""
import asyncio
import logging
import mimetypes
import os
import re
import tempfile
from typing import Optional
from .flacfetch_client import get_flacfetch_client, FlacfetchServiceError
from .storage_service import StorageService
logger = logging.getLogger(__name__)
class YouTubeDownloadError(Exception):
"""Error downloading from YouTube."""
pass
class YouTubeDownloadService:
"""
Single point of entry for all YouTube downloads in the cloud backend.
When remote flacfetch is configured (FLACFETCH_API_URL), downloads happen
on the flacfetch VM which has YouTube cookies and avoids bot detection.
When remote is not configured, falls back to local yt_dlp with a warning
that downloads may be blocked on Cloud Run IPs.
Usage:
service = get_youtube_download_service()
gcs_path = await service.download(
url="https://www.youtube.com/watch?v=dQw4w9WgXcQ",
job_id="job123",
artist="Rick Astley",
title="Never Gonna Give You Up",
)
"""
def __init__(self):
self._flacfetch_client = get_flacfetch_client()
self._storage_service = StorageService()
if self._flacfetch_client:
logger.info("YouTubeDownloadService using REMOTE flacfetch (recommended)")
else:
logger.warning(
"YouTubeDownloadService using LOCAL yt_dlp - "
"downloads may fail due to YouTube bot detection on Cloud Run IPs"
)
def is_remote_enabled(self) -> bool:
"""Check if remote flacfetch is configured."""
return self._flacfetch_client is not None
async def download(
self,
url: str,
job_id: str,
artist: Optional[str] = None,
title: Optional[str] = None,
) -> str:
"""
Download YouTube audio and upload to GCS.
Args:
url: YouTube URL (any format - watch, youtu.be, shorts, etc.)
job_id: Job ID for GCS path
artist: Optional artist name for filename
title: Optional title for filename
Returns:
GCS path (not gs:// prefix, just the path portion)
Example: "uploads/job123/audio/Artist - Title.flac"
Raises:
YouTubeDownloadError: If download fails
"""
video_id = self._extract_video_id(url)
if not video_id:
raise YouTubeDownloadError(f"Could not extract video ID from URL: {url}")
logger.info(f"YouTube download: video_id={video_id}, job_id={job_id}")
if self._flacfetch_client:
return await self._download_remote(video_id, job_id, artist, title)
else:
logger.warning(
"Remote flacfetch not configured. Attempting local download, "
"but this may fail due to YouTube bot detection on Cloud Run IPs."
)
return await self._download_local(url, job_id, artist, title)
async def download_by_id(
self,
video_id: str,
job_id: str,
artist: Optional[str] = None,
title: Optional[str] = None,
) -> str:
"""
Download YouTube audio by video ID.
Same as download() but takes a video ID directly instead of URL.
Args:
video_id: YouTube video ID (e.g., "dQw4w9WgXcQ")
job_id: Job ID for GCS path
artist: Optional artist name for filename
title: Optional title for filename
Returns:
GCS path (path portion only, no gs:// prefix)
Raises:
YouTubeDownloadError: If download fails
"""
url = f"https://www.youtube.com/watch?v={video_id}"
return await self.download(url, job_id, artist, title)
def _extract_video_id(self, url: str) -> Optional[str]:
"""
Extract YouTube video ID from various URL formats.
Supports:
- https://www.youtube.com/watch?v=VIDEO_ID
- https://youtu.be/VIDEO_ID
- https://www.youtube.com/shorts/VIDEO_ID
- https://www.youtube.com/embed/VIDEO_ID
- https://youtube.com/v/VIDEO_ID
Returns:
Video ID string, or None if extraction fails
"""
patterns = [
# Standard watch URL
r'(?:youtube\.com/watch\?v=|youtube\.com/watch\?.*&v=)([a-zA-Z0-9_-]{11})',
# Short URL
r'youtu\.be/([a-zA-Z0-9_-]{11})',
# Shorts URL
r'youtube\.com/shorts/([a-zA-Z0-9_-]{11})',
# Embed URL
r'youtube\.com/embed/([a-zA-Z0-9_-]{11})',
# Old-style URL
r'youtube\.com/v/([a-zA-Z0-9_-]{11})',
]
for pattern in patterns:
match = re.search(pattern, url)
if match:
return match.group(1)
return None
async def _download_remote(
self,
video_id: str,
job_id: str,
artist: Optional[str] = None,
title: Optional[str] = None,
) -> str:
"""Download using remote flacfetch service."""
gcs_destination = f"uploads/{job_id}/audio/"
# Build output filename if artist/title provided
output_filename = None
if artist and title:
from karaoke_gen.utils import sanitize_filename
safe_artist = sanitize_filename(artist)
safe_title = sanitize_filename(title)
output_filename = f"{safe_artist} - {safe_title}"
logger.info(
f"Remote YouTube download: video_id={video_id}, "
f"gcs_path={gcs_destination}, filename={output_filename}"
)
try:
# Start download
download_id = await self._flacfetch_client.download_by_id(
source_name="YouTube",
source_id=video_id,
output_filename=output_filename,
gcs_path=gcs_destination,
)
logger.info(f"Remote download started: {download_id}")
# Wait for completion
def log_progress(status):
progress = status.get("progress", 0)
speed = status.get("download_speed_kbps", 0)
logger.debug(f"Download progress: {progress:.1f}% ({speed:.1f} KB/s)")
final_status = await self._flacfetch_client.wait_for_download(
download_id,
timeout=300, # 5 minute timeout for YouTube downloads
progress_callback=log_progress,
)
# Extract GCS path from response
gcs_path = final_status.get("gcs_path")
if not gcs_path:
raise YouTubeDownloadError(
"Remote download completed but no GCS path returned"
)
# Convert gs:// URL to path portion
if gcs_path.startswith("gs://"):
parts = gcs_path.replace("gs://", "").split("/", 1)
if len(parts) == 2:
gcs_path = parts[1]
logger.info(f"Remote YouTube download complete: {gcs_path}")
return gcs_path
except FlacfetchServiceError as e:
raise YouTubeDownloadError(f"Remote download failed: {e}") from e
except Exception as e:
logger.error(f"Remote YouTube download error: {e}", exc_info=True)
raise YouTubeDownloadError(f"Remote download failed: {e}") from e
async def _download_local(
self,
url: str,
job_id: str,
artist: Optional[str] = None,
title: Optional[str] = None,
) -> str:
"""
Download using local yt_dlp (fallback when remote not configured).
Warning: This may fail on Cloud Run due to YouTube bot detection.
"""
temp_dir = tempfile.mkdtemp(prefix=f"youtube_{job_id}_")
try:
from karaoke_gen.file_handler import FileHandler
from karaoke_gen.utils import sanitize_filename
# Create FileHandler
file_handler = FileHandler(
logger=logger,
ffmpeg_base_command="ffmpeg -hide_banner -loglevel error -nostats -y",
create_track_subfolders=False,
dry_run=False
)
# Build output filename
safe_artist = sanitize_filename(artist) if artist else "Unknown"
safe_title = sanitize_filename(title) if title else "Unknown"
output_filename_no_extension = os.path.join(
temp_dir, f"{safe_artist} - {safe_title}"
)
# Get cookies from environment
cookies_str = os.environ.get("YOUTUBE_COOKIES")
if cookies_str:
logger.info("Using YouTube cookies for local download")
# Download
logger.info(f"Local YouTube download: {url}")
downloaded_file = file_handler.download_video(
url=url,
output_filename_no_extension=output_filename_no_extension,
cookies_str=cookies_str
)
if not downloaded_file or not os.path.exists(downloaded_file):
raise YouTubeDownloadError(
f"Local download failed - no file returned. "
"This is likely due to YouTube bot detection on Cloud Run IPs. "
"Configure FLACFETCH_API_URL for remote downloads."
)
logger.info(f"Downloaded video: {downloaded_file}")
# Convert to WAV for processing
wav_file = file_handler.convert_to_wav(
input_filename=downloaded_file,
output_filename_no_extension=output_filename_no_extension
)
if not wav_file or not os.path.exists(wav_file):
raise YouTubeDownloadError("WAV conversion failed")
logger.info(f"Converted to WAV: {wav_file}")
# Upload to GCS
filename = os.path.basename(wav_file)
gcs_path = f"uploads/{job_id}/audio/{filename}"
content_type, _ = mimetypes.guess_type(wav_file)
with open(wav_file, 'rb') as f:
self._storage_service.upload_fileobj(
f,
gcs_path,
content_type=content_type or 'audio/wav'
)
logger.info(f"Uploaded to GCS: {gcs_path}")
return gcs_path
except YouTubeDownloadError:
raise
except Exception as e:
logger.error(f"Local YouTube download error: {e}", exc_info=True)
raise YouTubeDownloadError(f"Local download failed: {e}") from e
finally:
# Cleanup temp directory
try:
import shutil
if os.path.exists(temp_dir):
shutil.rmtree(temp_dir)
except Exception as cleanup_error:
logger.warning(f"Failed to cleanup temp dir: {cleanup_error}")
# Singleton instance
_youtube_download_service: Optional[YouTubeDownloadService] = None
def get_youtube_download_service() -> YouTubeDownloadService:
"""Get the singleton YouTubeDownloadService instance."""
global _youtube_download_service
if _youtube_download_service is None:
_youtube_download_service = YouTubeDownloadService()
return _youtube_download_service
def reset_youtube_download_service():
"""Reset the singleton (for testing)."""
global _youtube_download_service
_youtube_download_service = None
+14 -21
View File
@@ -105,39 +105,32 @@ def mock_auth_dependency(request):
yield
return
from backend.services.auth_service import UserType, AuthResult
from backend.api.dependencies import require_auth, require_admin, require_review_auth
from backend.services.auth_service import UserType
from backend.api.dependencies import require_auth, require_admin, require_review_auth, require_instrumental_auth
from backend.main import app
# Create mock auth functions that return AuthResult objects
# Create mock auth functions
async def mock_require_auth():
"""Mock require_auth to always return valid admin credentials."""
return AuthResult(
is_valid=True,
user_type=UserType.ADMIN,
remaining_uses=999,
message="Test admin token",
is_admin=True
)
return ("test-admin-token", UserType.ADMIN, 999)
async def mock_require_admin():
"""Mock require_admin to always return valid admin credentials."""
return AuthResult(
is_valid=True,
user_type=UserType.ADMIN,
remaining_uses=999,
message="Test admin token",
is_admin=True
)
return ("test-admin-token", UserType.ADMIN, 999)
async def mock_require_review_auth(job_id: str = "test123"):
"""Mock require_review_auth to always return valid review access."""
return (job_id, "full")
async def mock_require_instrumental_auth(job_id: str = "test123"):
"""Mock require_instrumental_auth to always return valid instrumental review access."""
return (job_id, "full")
# Use FastAPI's dependency override system
app.dependency_overrides[require_auth] = mock_require_auth
app.dependency_overrides[require_admin] = mock_require_admin
app.dependency_overrides[require_review_auth] = mock_require_review_auth
app.dependency_overrides[require_instrumental_auth] = mock_require_instrumental_auth
yield
-1
View File
@@ -1 +0,0 @@
"""E2E tests for backend functionality."""
@@ -1,360 +0,0 @@
"""
E2E tests that verify actual audio synchronization in generated videos.
These tests generate real (short) videos and verify the audio offset.
Uses ffprobe to analyze video/audio streams.
These tests are SLOW and should be run on-demand, not on every PR.
"""
import pytest
import subprocess
import tempfile
import os
import json
from pathlib import Path
from unittest.mock import MagicMock, patch
def ffprobe_available():
"""Check if ffprobe is available."""
try:
result = subprocess.run(
['ffprobe', '-version'],
capture_output=True,
text=True
)
return result.returncode == 0
except FileNotFoundError:
return False
# Skip all tests if ffprobe not available
pytestmark = pytest.mark.skipif(
not ffprobe_available(),
reason="ffprobe not available. Install ffmpeg to run these tests."
)
class TestAudioFileAnalysis:
"""Helper tests for audio file analysis functions."""
def test_ffprobe_can_analyze_audio(self, tmp_path):
"""Verify ffprobe can analyze audio files."""
# Create a silent audio file for testing
test_audio = tmp_path / "test.wav"
# Generate 5 seconds of silence using ffmpeg
result = subprocess.run([
'ffmpeg', '-y', '-f', 'lavfi',
'-i', 'anullsrc=r=44100:cl=stereo',
'-t', '5',
str(test_audio)
], capture_output=True)
if result.returncode != 0:
pytest.skip("Could not create test audio file")
# Analyze with ffprobe
probe_result = subprocess.run([
'ffprobe', '-v', 'quiet',
'-print_format', 'json',
'-show_format',
str(test_audio)
], capture_output=True, text=True)
assert probe_result.returncode == 0
data = json.loads(probe_result.stdout)
assert 'format' in data
duration = float(data['format']['duration'])
assert 4.9 < duration < 5.1 # ~5 seconds
class TestCountdownAudioPadding:
"""Tests for audio padding functionality."""
@pytest.fixture
def source_audio(self, tmp_path):
"""Create a short source audio file."""
audio_path = tmp_path / "source.flac"
# Create a 5-second audio file with a tone at the start
# This simulates audio content that starts immediately
result = subprocess.run([
'ffmpeg', '-y', '-f', 'lavfi',
'-i', 'sine=frequency=440:duration=5',
'-c:a', 'flac',
str(audio_path)
], capture_output=True)
if result.returncode != 0:
raise RuntimeError(f"FFmpeg failed to create source audio: {result.stderr.decode()}")
return audio_path
def test_padding_adds_correct_duration(self, source_audio, tmp_path):
"""Verify that padding adds exactly the specified duration."""
padded_audio = tmp_path / "padded.flac"
padding_seconds = 3.0
# Apply padding using the same method as CountdownProcessor
# adelay filter adds delay in milliseconds
delay_ms = int(padding_seconds * 1000)
result = subprocess.run([
'ffmpeg', '-y',
'-i', str(source_audio),
'-af', f'adelay={delay_ms}|{delay_ms}',
'-c:a', 'flac',
str(padded_audio)
], capture_output=True)
assert result.returncode == 0, f"FFmpeg failed: {result.stderr}"
# Analyze both files
def get_duration(path):
probe = subprocess.run([
'ffprobe', '-v', 'quiet',
'-print_format', 'json',
'-show_format',
str(path)
], capture_output=True, text=True)
data = json.loads(probe.stdout)
return float(data['format']['duration'])
source_duration = get_duration(source_audio)
padded_duration = get_duration(padded_audio)
# Padded should be source + padding
expected_duration = source_duration + padding_seconds
assert abs(padded_duration - expected_duration) < 0.1, (
f"Expected duration ~{expected_duration}s, got {padded_duration}s"
)
def test_karaoke_finalise_pad_audio_file(self, source_audio, tmp_path):
"""Test KaraokeFinalise._pad_audio_file creates correctly padded audio."""
from karaoke_gen.karaoke_finalise.karaoke_finalise import KaraokeFinalise
padded_audio = tmp_path / "padded_via_finalise.flac"
padding_seconds = 3.0
finalise = KaraokeFinalise(non_interactive=True)
finalise._pad_audio_file(str(source_audio), str(padded_audio), padding_seconds)
assert padded_audio.exists(), "Padded audio file was not created"
# Verify duration
probe = subprocess.run([
'ffprobe', '-v', 'quiet',
'-print_format', 'json',
'-show_format',
str(padded_audio)
], capture_output=True, text=True)
data = json.loads(probe.stdout)
padded_duration = float(data['format']['duration'])
# Get source duration
source_probe = subprocess.run([
'ffprobe', '-v', 'quiet',
'-print_format', 'json',
'-show_format',
str(source_audio)
], capture_output=True, text=True)
source_data = json.loads(source_probe.stdout)
source_duration = float(source_data['format']['duration'])
expected = source_duration + padding_seconds
assert abs(padded_duration - expected) < 0.1, (
f"Expected ~{expected}s, got {padded_duration}s"
)
class TestAudioSyncVerification:
"""
Tests that verify the final audio sync between vocals and instrumental.
These tests create test audio files and verify that when countdown is added:
1. Vocals are padded by countdown duration
2. Instrumental is padded by same duration
3. Relative timing is preserved
"""
@pytest.fixture
def vocals_audio(self, tmp_path):
"""Create vocals audio with a distinctive marker at 0.5s."""
audio_path = tmp_path / "vocals.flac"
# Create audio: 0.5s silence, then 1s tone, then silence
# Total 5 seconds
filter_complex = (
'aevalsrc=0:d=0.5[silence1];'
'sine=frequency=880:duration=1[tone];'
'aevalsrc=0:d=3.5[silence2];'
'[silence1][tone][silence2]concat=n=3:v=0:a=1'
)
subprocess.run([
'ffmpeg', '-y', '-f', 'lavfi',
'-i', filter_complex,
'-c:a', 'flac',
str(audio_path)
], capture_output=True)
return audio_path
@pytest.fixture
def instrumental_audio(self, tmp_path):
"""Create instrumental audio with marker at 0.5s."""
audio_path = tmp_path / "instrumental.flac"
filter_complex = (
'aevalsrc=0:d=0.5[silence1];'
'sine=frequency=440:duration=1[tone];'
'aevalsrc=0:d=3.5[silence2];'
'[silence1][tone][silence2]concat=n=3:v=0:a=1'
)
subprocess.run([
'ffmpeg', '-y', '-f', 'lavfi',
'-i', filter_complex,
'-c:a', 'flac',
str(audio_path)
], capture_output=True)
return audio_path
def test_both_tracks_padded_equally(self, vocals_audio, instrumental_audio, tmp_path):
"""
When countdown is applied, both vocals and instrumental must be
padded by the same amount to maintain sync.
"""
padding_seconds = 3.0
delay_ms = int(padding_seconds * 1000)
padded_vocals = tmp_path / "padded_vocals.flac"
padded_instrumental = tmp_path / "padded_instrumental.flac"
# Pad both
for src, dst in [(vocals_audio, padded_vocals), (instrumental_audio, padded_instrumental)]:
subprocess.run([
'ffmpeg', '-y',
'-i', str(src),
'-af', f'adelay={delay_ms}|{delay_ms}',
'-c:a', 'flac',
str(dst)
], capture_output=True)
# Get durations
def get_duration(path):
probe = subprocess.run([
'ffprobe', '-v', 'quiet',
'-print_format', 'json',
'-show_format',
str(path)
], capture_output=True, text=True)
data = json.loads(probe.stdout)
return float(data['format']['duration'])
vocals_dur = get_duration(padded_vocals)
inst_dur = get_duration(padded_instrumental)
# Both should have same duration (sync maintained)
assert abs(vocals_dur - inst_dur) < 0.1, (
f"Padded vocals ({vocals_dur}s) and instrumental ({inst_dur}s) "
"have different durations - sync will be off"
)
class TestCountdownTimingInvariant:
"""
Tests for the critical audio timing invariant.
INVARIANT: If vocals start at time T (after countdown),
instrumental must also start at time T (also padded).
Violation of this invariant causes audio desync.
"""
def test_relative_timing_preserved_after_padding(self):
"""
The relative timing between vocals and instrumental must be preserved.
If vocals originally start at 0.5s with content at 0.5s,
after 3s padding, vocals content is at 3.5s.
Instrumental must also be padded so its content is at 3.5s.
"""
original_content_start = 0.5 # seconds
padding_seconds = 3.0
# After padding
vocals_content_after_padding = original_content_start + padding_seconds
instrumental_start_after_padding = 0 + padding_seconds # instrumental starts at 0
# The relative offset should be preserved
original_relative_offset = original_content_start - 0 # vocals relative to instrumental start
new_relative_offset = vocals_content_after_padding - instrumental_start_after_padding
assert original_relative_offset == new_relative_offset, (
"INVARIANT VIOLATION: Padding changed the relative timing "
"between vocals and instrumental"
)
def test_sync_check_formula(self):
"""
Formula for checking if audio is in sync:
If has_countdown_padding=True:
expected_instrumental_start = countdown_padding_seconds
actual_instrumental_start should equal expected
"""
countdown_padding_seconds = 3.0
has_countdown_padding = True
if has_countdown_padding:
expected_instrumental_start = countdown_padding_seconds
else:
expected_instrumental_start = 0
# If instrumental is correctly padded
actual_instrumental_start = 3.0 # (padded)
assert actual_instrumental_start == expected_instrumental_start, (
"Audio sync check failed"
)
class TestAudioSyncDriftAcceptance:
"""
Tests for acceptable audio drift tolerance.
Small amounts of drift (<100ms) are acceptable in karaoke videos.
Larger drift (>100ms) is noticeable and indicates a bug.
"""
def test_acceptable_drift_threshold(self):
"""Define the acceptable drift threshold."""
acceptable_drift_ms = 100 # 100ms
acceptable_drift_s = acceptable_drift_ms / 1000
# A 50ms drift is acceptable
actual_drift_s = 0.050
assert actual_drift_s <= acceptable_drift_s, "50ms drift should be acceptable"
# A 200ms drift is NOT acceptable
actual_drift_s = 0.200
assert actual_drift_s > acceptable_drift_s, "200ms drift is too much"
def test_countdown_sync_within_tolerance(self):
"""
After countdown processing, drift should be within tolerance.
"""
# Ideal case: no drift
expected_sync = 3.0 # seconds
actual_sync = 3.0 # seconds
drift = abs(actual_sync - expected_sync)
acceptable_drift = 0.1 # 100ms
assert drift <= acceptable_drift, (
f"Drift of {drift*1000}ms exceeds tolerance of {acceptable_drift*1000}ms"
)
@@ -1,361 +0,0 @@
"""
Integration tests for countdown padding across the full worker pipeline.
These tests use Firestore emulator to verify real state updates flow correctly
between workers. They mock only external services (Modal, AudioShake, etc.)
but use real Firestore for state persistence.
Run with: scripts/run-emulator-tests.sh
"""
import pytest
import os
from datetime import datetime, UTC
from unittest.mock import MagicMock, AsyncMock, patch
import tempfile
# Import base fixtures from conftest
from backend.tests.emulator.conftest import emulators_running
# Skip all tests if emulators aren't running
pytestmark = pytest.mark.skipif(
not emulators_running(),
reason="GCP emulators not running. Start with: scripts/start-emulators.sh"
)
# Only import if emulators are running
if emulators_running():
from backend.models.job import Job, JobStatus
from backend.services.job_manager import JobManager
from backend.services.storage_service import StorageService
from backend.services.firestore_service import FirestoreService
class TestCountdownStateFlowThroughPipeline:
"""
Integration tests verifying countdown state flows correctly through
the full worker pipeline: lyrics -> render_video -> video.
"""
@pytest.fixture
def job_manager(self):
"""Create a real JobManager using emulator."""
return JobManager()
@pytest.fixture
def firestore_service(self):
"""Create a real FirestoreService using emulator for direct job operations."""
return FirestoreService()
@pytest.fixture
def storage_service(self):
"""Create a real StorageService using emulator."""
return StorageService()
@pytest.fixture
def job_with_early_start_song(self, job_manager, firestore_service):
"""
Create a job where lyrics start within 3 seconds (needs countdown).
This simulates a song where the first word appears at 0.5 seconds,
which triggers countdown processing.
"""
job_id = f"countdown-test-early-{datetime.now(UTC).timestamp()}"
job = Job(
job_id=job_id,
status=JobStatus.PENDING,
created_at=datetime.now(UTC),
updated_at=datetime.now(UTC),
artist="Test Artist",
title="Early Start Song",
state_data={
'lyrics_metadata': {
'has_countdown_padding': False,
'countdown_padding_seconds': 0.0,
'first_word_start_time': 0.5, # Within 3s threshold
}
}
)
# Create job in Firestore directly
firestore_service.create_job(job)
yield job
# Cleanup
try:
firestore_service.delete_job(job_id)
except Exception:
pass
@pytest.fixture
def job_with_late_start_song(self, job_manager, firestore_service):
"""
Create a job where lyrics start after 3 seconds (no countdown needed).
This simulates a song where the first word appears at 5.0 seconds,
which does NOT trigger countdown processing.
"""
job_id = f"countdown-test-late-{datetime.now(UTC).timestamp()}"
job = Job(
job_id=job_id,
status=JobStatus.PENDING,
created_at=datetime.now(UTC),
updated_at=datetime.now(UTC),
artist="Test Artist",
title="Late Start Song",
state_data={
'lyrics_metadata': {
'has_countdown_padding': False,
'countdown_padding_seconds': 0.0,
'first_word_start_time': 5.0, # After 3s threshold
}
}
)
firestore_service.create_job(job)
yield job
try:
firestore_service.delete_job(job_id)
except Exception:
pass
@pytest.mark.asyncio
async def test_state_update_persists_to_firestore(self, job_with_early_start_song, job_manager):
"""
Verify that state updates via update_state_data() persist to Firestore.
This is the foundation of the countdown state flow - if updates don't
persist, video_worker can't read the countdown state.
"""
job = job_with_early_start_song
# Simulate render_video_worker updating countdown state
new_lyrics_metadata = {
'has_countdown_padding': True,
'countdown_padding_seconds': 3.0,
'first_word_start_time': 0.5,
}
job_manager.update_state_data(job.job_id, 'lyrics_metadata', new_lyrics_metadata)
# Read job back from Firestore
refreshed_job = job_manager.get_job(job.job_id)
assert refreshed_job is not None
lyrics_metadata = refreshed_job.state_data.get('lyrics_metadata', {})
assert lyrics_metadata.get('has_countdown_padding') is True
assert lyrics_metadata.get('countdown_padding_seconds') == 3.0
@pytest.mark.asyncio
async def test_video_worker_reads_updated_state(self, job_with_early_start_song, job_manager):
"""
Verify that video_worker can read countdown state after render_video_worker sets it.
This simulates the full flow:
1. render_video_worker sets has_countdown_padding=True
2. video_worker reads the state and extracts countdown_padding_seconds
"""
job = job_with_early_start_song
# Step 1: render_video_worker updates state
new_lyrics_metadata = {
'has_countdown_padding': True,
'countdown_padding_seconds': 3.0,
}
job_manager.update_state_data(job.job_id, 'lyrics_metadata', new_lyrics_metadata)
# Step 2: video_worker reads state (simulated)
refreshed_job = job_manager.get_job(job.job_id)
# video_worker extraction logic
lyrics_metadata = refreshed_job.state_data.get('lyrics_metadata', {})
countdown_padding_seconds = None
if lyrics_metadata.get('has_countdown_padding'):
countdown_padding_seconds = lyrics_metadata.get('countdown_padding_seconds', 3.0)
assert countdown_padding_seconds == 3.0
@pytest.mark.asyncio
async def test_no_countdown_state_when_not_needed(self, job_with_late_start_song, job_manager):
"""
Verify that when song doesn't need countdown, state remains False.
"""
job = job_with_late_start_song
# Read initial state
current_job = job_manager.get_job(job.job_id)
lyrics_metadata = current_job.state_data.get('lyrics_metadata', {})
assert lyrics_metadata.get('has_countdown_padding') is False
assert lyrics_metadata.get('countdown_padding_seconds') == 0.0
# video_worker extraction logic
countdown_padding_seconds = None
if lyrics_metadata.get('has_countdown_padding'):
countdown_padding_seconds = lyrics_metadata.get('countdown_padding_seconds', 3.0)
assert countdown_padding_seconds is None
class TestCountdownStateTransitions:
"""Tests for state transitions during countdown processing."""
@pytest.fixture
def job_manager(self):
return JobManager()
@pytest.fixture
def firestore_service(self):
return FirestoreService()
@pytest.mark.asyncio
async def test_state_transitions_preserve_countdown_info(self, job_manager, firestore_service):
"""
Countdown state should survive job state transitions.
When job moves from RENDERING_VIDEO to ENCODING,
the lyrics_metadata should remain intact.
"""
job_id = f"transition-test-{datetime.now(UTC).timestamp()}"
try:
# Create job in RENDERING_VIDEO state
job = Job(
job_id=job_id,
status=JobStatus.RENDERING_VIDEO,
created_at=datetime.now(UTC),
updated_at=datetime.now(UTC),
artist="Test",
title="Test",
state_data={
'lyrics_metadata': {
'has_countdown_padding': True,
'countdown_padding_seconds': 3.0,
}
}
)
firestore_service.create_job(job)
# Transition to ENCODING
job_manager.transition_to_state(
job_id=job_id,
new_status=JobStatus.ENCODING,
progress=75,
message="Starting encoding"
)
# Read job and verify countdown state preserved
refreshed_job = job_manager.get_job(job_id)
lyrics_metadata = refreshed_job.state_data.get('lyrics_metadata', {})
assert lyrics_metadata.get('has_countdown_padding') is True
assert lyrics_metadata.get('countdown_padding_seconds') == 3.0
finally:
try:
firestore_service.delete_job(job_id)
except Exception:
pass
class TestWorkerStateDataIsolation:
"""Tests that different workers don't overwrite each other's state data."""
@pytest.fixture
def job_manager(self):
return JobManager()
@pytest.fixture
def firestore_service(self):
return FirestoreService()
@pytest.mark.asyncio
async def test_lyrics_metadata_not_overwritten_by_other_updates(self, job_manager, firestore_service):
"""
Updating video_worker state should NOT overwrite lyrics_metadata.
Each worker stores its state in separate keys within state_data.
"""
job_id = f"isolation-test-{datetime.now(UTC).timestamp()}"
try:
# Create job with countdown state
job = Job(
job_id=job_id,
status=JobStatus.ENCODING,
created_at=datetime.now(UTC),
updated_at=datetime.now(UTC),
artist="Test",
title="Test",
state_data={
'lyrics_metadata': {
'has_countdown_padding': True,
'countdown_padding_seconds': 3.0,
}
}
)
firestore_service.create_job(job)
# Update video_worker state (different key)
job_manager.update_state_data(job_id, 'video_worker', {
'instrumental_source': 'clean',
'encoding_started': True,
})
# Verify lyrics_metadata preserved
refreshed_job = job_manager.get_job(job_id)
lyrics_metadata = refreshed_job.state_data.get('lyrics_metadata', {})
assert lyrics_metadata.get('has_countdown_padding') is True
assert lyrics_metadata.get('countdown_padding_seconds') == 3.0
# Verify video_worker state also present
video_worker_state = refreshed_job.state_data.get('video_worker', {})
assert video_worker_state.get('instrumental_source') == 'clean'
finally:
try:
firestore_service.delete_job(job_id)
except Exception:
pass
class TestCountdownStateUpdateCodeExists:
"""
Verify the render_video_worker code contains the state update logic.
This is a code inspection test rather than an execution test, since running
the actual worker requires too much complex setup (audio files, corrections, etc.).
The actual state update behavior is verified by the unit tests.
"""
def test_render_video_worker_has_state_update_code(self):
"""
Verify render_video_worker.py contains the code to update lyrics_metadata
when countdown padding is added.
This ensures the fix hasn't been accidentally removed.
"""
import inspect
from backend.workers import render_video_worker
source = inspect.getsource(render_video_worker)
# The fix should include updating lyrics_metadata when padding_added is True
assert "update_state_data" in source, (
"render_video_worker must call update_state_data"
)
assert "has_countdown_padding" in source, (
"render_video_worker must set has_countdown_padding"
)
assert "countdown_padding_seconds" in source, (
"render_video_worker must set countdown_padding_seconds"
)
assert "lyrics_metadata" in source, (
"render_video_worker must update lyrics_metadata"
)
@@ -1,380 +0,0 @@
"""
Tests that countdown state survives worker boundaries and job reloads.
These tests verify that countdown state persists correctly in Firestore
and can be read across different worker invocations.
Run with: scripts/run-emulator-tests.sh
"""
import pytest
import os
from datetime import datetime, UTC
from unittest.mock import MagicMock, AsyncMock, patch
import time
# Import base fixtures from conftest
from backend.tests.emulator.conftest import emulators_running
# Skip all tests if emulators aren't running
pytestmark = pytest.mark.skipif(
not emulators_running(),
reason="GCP emulators not running. Start with: scripts/start-emulators.sh"
)
# Only import if emulators are running
if emulators_running():
from backend.models.job import Job, JobStatus
from backend.services.job_manager import JobManager
from backend.services.firestore_service import FirestoreService
class TestCountdownStatePersistence:
"""Tests that countdown state persists between worker boundaries."""
@pytest.fixture
def job_manager(self):
"""Create a real JobManager using emulator."""
return JobManager()
@pytest.fixture
def firestore_service(self):
"""Create a real FirestoreService using emulator for direct job operations."""
return FirestoreService()
@pytest.mark.asyncio
async def test_state_persists_between_render_and_video_workers(self, job_manager, firestore_service):
"""
State set by render_video_worker is readable by video_worker.
Simulates:
1. render_video_worker completes, sets has_countdown_padding=True
2. Some time passes (worker boundary)
3. video_worker starts, reads has_countdown_padding
"""
job_id = f"persist-test-{datetime.now(UTC).timestamp()}"
try:
# Create initial job (as lyrics_worker would leave it)
job = Job(
job_id=job_id,
status=JobStatus.RENDERING_VIDEO,
created_at=datetime.now(UTC),
updated_at=datetime.now(UTC),
artist="Test",
title="Test",
state_data={
'lyrics_metadata': {
'has_countdown_padding': False,
'countdown_padding_seconds': 0.0,
}
}
)
firestore_service.create_job(job)
# === render_video_worker completes ===
job_manager.update_state_data(job_id, 'lyrics_metadata', {
'has_countdown_padding': True,
'countdown_padding_seconds': 3.0,
})
# Simulate worker boundary (job is re-fetched from Firestore)
# Create a NEW JobManager instance to simulate video_worker starting
video_job_manager = JobManager()
# === video_worker starts ===
job_from_video_worker = video_job_manager.get_job(job_id)
# video_worker reads countdown state
lyrics_metadata = job_from_video_worker.state_data.get('lyrics_metadata', {})
countdown_padding_seconds = None
if lyrics_metadata.get('has_countdown_padding'):
countdown_padding_seconds = lyrics_metadata.get('countdown_padding_seconds', 3.0)
assert countdown_padding_seconds == 3.0, (
"Countdown state did not persist between workers"
)
finally:
try:
firestore_service.delete_job(job_id)
except Exception:
pass
@pytest.mark.asyncio
async def test_state_survives_job_reload(self, job_manager, firestore_service):
"""
State persists when job is reloaded from Firestore multiple times.
"""
job_id = f"reload-test-{datetime.now(UTC).timestamp()}"
try:
# Create job with countdown state
job = Job(
job_id=job_id,
status=JobStatus.ENCODING,
created_at=datetime.now(UTC),
updated_at=datetime.now(UTC),
artist="Test",
title="Test",
state_data={
'lyrics_metadata': {
'has_countdown_padding': True,
'countdown_padding_seconds': 3.0,
}
}
)
firestore_service.create_job(job)
# Reload job multiple times
for i in range(3):
reloaded_job = job_manager.get_job(job_id)
lyrics_metadata = reloaded_job.state_data.get('lyrics_metadata', {})
assert lyrics_metadata.get('has_countdown_padding') is True, (
f"Countdown state lost after reload #{i+1}"
)
assert lyrics_metadata.get('countdown_padding_seconds') == 3.0, (
f"Countdown duration changed after reload #{i+1}"
)
finally:
try:
firestore_service.delete_job(job_id)
except Exception:
pass
@pytest.mark.asyncio
async def test_partial_state_update_preserves_other_fields(self, job_manager, firestore_service):
"""
Updating one field in state_data doesn't affect other fields.
"""
job_id = f"partial-test-{datetime.now(UTC).timestamp()}"
try:
# Create job with multiple state fields
job = Job(
job_id=job_id,
status=JobStatus.ENCODING,
created_at=datetime.now(UTC),
updated_at=datetime.now(UTC),
artist="Test",
title="Test",
state_data={
'lyrics_metadata': {
'has_countdown_padding': True,
'countdown_padding_seconds': 3.0,
'original_field': 'should_be_preserved',
},
'audio_complete': True,
}
)
firestore_service.create_job(job)
# Update video_worker state (different key)
job_manager.update_state_data(job_id, 'video_worker', {
'encoding_started': True,
})
# Verify lyrics_metadata is completely preserved
reloaded_job = job_manager.get_job(job_id)
lyrics_metadata = reloaded_job.state_data.get('lyrics_metadata', {})
assert lyrics_metadata.get('has_countdown_padding') is True
assert lyrics_metadata.get('countdown_padding_seconds') == 3.0
assert lyrics_metadata.get('original_field') == 'should_be_preserved'
assert reloaded_job.state_data.get('audio_complete') is True
finally:
try:
firestore_service.delete_job(job_id)
except Exception:
pass
class TestCountdownStateConsistency:
"""Tests for state consistency under concurrent access patterns."""
@pytest.fixture
def job_manager(self):
return JobManager()
@pytest.fixture
def firestore_service(self):
return FirestoreService()
@pytest.mark.asyncio
async def test_concurrent_reads_see_same_state(self, job_manager, firestore_service):
"""
Multiple concurrent reads should see the same countdown state.
"""
job_id = f"concurrent-test-{datetime.now(UTC).timestamp()}"
try:
# Create job with countdown
job = Job(
job_id=job_id,
status=JobStatus.ENCODING,
created_at=datetime.now(UTC),
updated_at=datetime.now(UTC),
artist="Test",
title="Test",
state_data={
'lyrics_metadata': {
'has_countdown_padding': True,
'countdown_padding_seconds': 3.0,
}
}
)
firestore_service.create_job(job)
# Simulate concurrent reads
reads = []
for _ in range(5):
reloaded = job_manager.get_job(job_id)
lyrics_metadata = reloaded.state_data.get('lyrics_metadata', {})
reads.append({
'has_countdown_padding': lyrics_metadata.get('has_countdown_padding'),
'countdown_padding_seconds': lyrics_metadata.get('countdown_padding_seconds'),
})
# All reads should be consistent
for read in reads:
assert read['has_countdown_padding'] is True
assert read['countdown_padding_seconds'] == 3.0
finally:
try:
firestore_service.delete_job(job_id)
except Exception:
pass
class TestCountdownStateMigration:
"""Tests for handling jobs with missing or old countdown state format."""
@pytest.fixture
def job_manager(self):
return JobManager()
@pytest.fixture
def firestore_service(self):
return FirestoreService()
@pytest.mark.asyncio
async def test_handles_missing_lyrics_metadata(self, job_manager, firestore_service):
"""
Jobs without lyrics_metadata should be handled gracefully.
This can happen for jobs created before the countdown feature.
"""
job_id = f"migration-test-{datetime.now(UTC).timestamp()}"
try:
# Create job WITHOUT lyrics_metadata (old format)
job = Job(
job_id=job_id,
status=JobStatus.ENCODING,
created_at=datetime.now(UTC),
updated_at=datetime.now(UTC),
artist="Test",
title="Test",
state_data={} # No lyrics_metadata
)
firestore_service.create_job(job)
# video_worker logic should handle this
reloaded = job_manager.get_job(job_id)
lyrics_metadata = reloaded.state_data.get('lyrics_metadata', {})
countdown_padding_seconds = None
if lyrics_metadata.get('has_countdown_padding'):
countdown_padding_seconds = lyrics_metadata.get('countdown_padding_seconds', 3.0)
# Should return None, not crash
assert countdown_padding_seconds is None
finally:
try:
firestore_service.delete_job(job_id)
except Exception:
pass
@pytest.mark.asyncio
async def test_handles_missing_countdown_fields(self, job_manager, firestore_service):
"""
Jobs with lyrics_metadata but missing countdown fields handled gracefully.
"""
job_id = f"fields-test-{datetime.now(UTC).timestamp()}"
try:
# Create job with lyrics_metadata but no countdown fields
job = Job(
job_id=job_id,
status=JobStatus.ENCODING,
created_at=datetime.now(UTC),
updated_at=datetime.now(UTC),
artist="Test",
title="Test",
state_data={
'lyrics_metadata': {
'some_other_field': 'value',
# No has_countdown_padding
# No countdown_padding_seconds
}
}
)
firestore_service.create_job(job)
reloaded = job_manager.get_job(job_id)
lyrics_metadata = reloaded.state_data.get('lyrics_metadata', {})
countdown_padding_seconds = None
if lyrics_metadata.get('has_countdown_padding'):
countdown_padding_seconds = lyrics_metadata.get('countdown_padding_seconds', 3.0)
# Should return None when has_countdown_padding is missing/falsy
assert countdown_padding_seconds is None
finally:
try:
firestore_service.delete_job(job_id)
except Exception:
pass
@pytest.mark.asyncio
async def test_handles_countdown_true_without_duration(self, job_manager, firestore_service):
"""
If has_countdown_padding=True but duration missing, default to 3.0.
"""
job_id = f"default-duration-test-{datetime.now(UTC).timestamp()}"
try:
job = Job(
job_id=job_id,
status=JobStatus.ENCODING,
created_at=datetime.now(UTC),
updated_at=datetime.now(UTC),
artist="Test",
title="Test",
state_data={
'lyrics_metadata': {
'has_countdown_padding': True,
# countdown_padding_seconds intentionally missing
}
}
)
firestore_service.create_job(job)
reloaded = job_manager.get_job(job_id)
lyrics_metadata = reloaded.state_data.get('lyrics_metadata', {})
countdown_padding_seconds = None
if lyrics_metadata.get('has_countdown_padding'):
countdown_padding_seconds = lyrics_metadata.get('countdown_padding_seconds', 3.0)
# Should default to 3.0
assert countdown_padding_seconds == 3.0
finally:
try:
firestore_service.delete_job(job_id)
except Exception:
pass
@@ -1,698 +0,0 @@
"""
Emulator integration tests for made-for-you orders with YouTube URLs.
These tests use REAL Firestore and GCS emulators to test the full API flow,
mocking only external services (YouTube download, email, Stripe).
Tests cover:
1. Job creation with YouTube URL metadata
2. YouTube audio download and GCS storage
3. Worker trigger sequencing (download before workers)
4. Error handling for download failures
5. Job state persistence and consistency
Run with: scripts/run-emulator-tests.sh
Or directly: make test-e2e
"""
import pytest
import time
import asyncio
from unittest.mock import MagicMock, AsyncMock, patch
from .conftest import emulators_running
# Skip all tests if emulators not running
pytestmark = pytest.mark.skipif(
not emulators_running(),
reason="GCP emulators not running. Start with: scripts/start-emulators.sh"
)
# =============================================================================
# Fixtures
# =============================================================================
@pytest.fixture
def youtube_order_metadata():
"""Standard metadata for YouTube URL made-for-you order."""
return {
"order_type": "made_for_you",
"customer_email": "customer@example.com",
"artist": "Test Artist",
"title": "Test Song",
"source_type": "youtube",
"youtube_url": "https://youtube.com/watch?v=abc123",
}
@pytest.fixture
def youtube_order_metadata_with_notes():
"""YouTube order metadata with customer notes."""
return {
"order_type": "made_for_you",
"customer_email": "customer@example.com",
"artist": "Test Artist",
"title": "Test Song",
"source_type": "youtube",
"youtube_url": "https://youtube.com/watch?v=abc123",
"notes": "Please make it extra special!",
}
@pytest.fixture
def mock_stripe_service():
"""Mock Stripe service for webhook verification."""
service = MagicMock()
service.is_configured.return_value = True
service.verify_webhook_signature.return_value = (True, {}, "OK")
return service
@pytest.fixture
def mock_email_service():
"""Mock email service for notification verification."""
service = MagicMock()
service.is_configured.return_value = True
service.send_email.return_value = True
service.send_made_for_you_order_confirmation.return_value = True
service.send_made_for_you_admin_notification.return_value = True
return service
@pytest.fixture
def mock_user_service_for_webhook():
"""Mock user service for webhook processing."""
service = MagicMock()
service.is_stripe_session_processed.return_value = False
service._mark_stripe_session_processed.return_value = None
# Mock admin login token creation
mock_admin_login = MagicMock()
mock_admin_login.token = "test-admin-login-token"
service.create_admin_login_token.return_value = mock_admin_login
return service
@pytest.fixture
def mock_youtube_download_service():
"""Mock YouTube download service returning valid GCS path."""
service = MagicMock()
service.download = AsyncMock(return_value="uploads/test-job-123/audio/test.webm")
service.is_remote_enabled.return_value = True
return service
@pytest.fixture
def mock_worker_service_local():
"""Mock worker service to track trigger calls (local scope, different name to avoid conftest collision)."""
service = MagicMock()
service.trigger_audio_worker = AsyncMock(return_value=True)
service.trigger_lyrics_worker = AsyncMock(return_value=True)
service.trigger_screens_worker = AsyncMock(return_value=True)
service.trigger_video_worker = AsyncMock(return_value=True)
return service
@pytest.fixture
def mock_theme_service():
"""Mock theme service to avoid real Firestore calls."""
service = MagicMock()
service.get_default_theme_id.return_value = "nomad"
service.get_theme.return_value = MagicMock(
theme_id="nomad",
name="Nomad Karaoke",
style_params={}
)
return service
@pytest.fixture
def mock_distribution_settings():
"""Mock distribution settings for made-for-you orders."""
settings = MagicMock()
settings.enable_youtube_upload = True
settings.dropbox_path = "/Nomad Karaoke"
settings.gdrive_folder_id = "test-folder-id"
settings.brand_prefix = "Nomad Karaoke"
settings.discord_webhook_url = None
settings.youtube_description = None
return settings
# =============================================================================
# Test Classes
# =============================================================================
class TestMadeForYouYouTubeOrderFlow:
"""
Integration tests for made-for-you orders with YouTube URLs.
Uses real Firestore/GCS emulators to test the full flow:
1. Webhook creates job with YouTube URL
2. YouTube audio is downloaded to GCS
3. Job is updated with input_media_gcs_path
4. Workers are triggered with correct job state
"""
@pytest.mark.asyncio
async def test_youtube_url_order_creates_job_with_correct_fields(
self, mock_email_service, mock_user_service_for_webhook,
mock_worker_service_local, mock_theme_service, mock_distribution_settings,
mock_youtube_download_service, youtube_order_metadata
):
"""
Test that a YouTube URL order creates a job with the correct fields.
Verifies:
- Job is created with made_for_you=True
- customer_email is set correctly
- url is set to the YouTube URL
- theme_id defaults to "nomad"
"""
from backend.api.routes.users import _handle_made_for_you_order, ADMIN_EMAIL
mock_job_manager = MagicMock()
mock_job = MagicMock()
mock_job.job_id = "test-job-youtube-fields"
mock_job_manager.create_job.return_value = mock_job
with patch('backend.services.job_manager.JobManager', return_value=mock_job_manager), \
patch('backend.services.worker_service.get_worker_service', return_value=mock_worker_service_local), \
patch('backend.services.storage_service.StorageService'), \
patch('backend.api.routes.users.get_theme_service', return_value=mock_theme_service), \
patch('backend.api.routes.users.get_effective_distribution_settings', return_value=mock_distribution_settings), \
patch('backend.api.routes.users.resolve_cdg_txt_defaults', return_value=(False, False)), \
patch('backend.api.routes.users._prepare_theme_for_job', return_value=(None, None, None)), \
patch('backend.api.routes.users.get_youtube_download_service', return_value=mock_youtube_download_service):
await _handle_made_for_you_order(
session_id="sess_test_fields",
metadata=youtube_order_metadata,
user_service=mock_user_service_for_webhook,
email_service=mock_email_service,
)
# Verify job creation was called
assert mock_job_manager.create_job.called
# Get the JobCreate argument
job_create_arg = mock_job_manager.create_job.call_args[0][0]
# Verify made_for_you fields
assert job_create_arg.made_for_you is True, \
"Job must be marked as made_for_you"
assert job_create_arg.customer_email == youtube_order_metadata["customer_email"], \
"Customer email must be set for delivery"
assert job_create_arg.url == youtube_order_metadata["youtube_url"], \
"YouTube URL must be set on job"
assert job_create_arg.user_email == ADMIN_EMAIL, \
"Admin must own job during processing"
assert job_create_arg.theme_id == "nomad", \
"Default nomad theme should be applied"
@pytest.mark.asyncio
async def test_youtube_url_order_triggers_workers(
self, mock_email_service, mock_user_service_for_webhook,
mock_worker_service_local, mock_theme_service, mock_distribution_settings,
mock_youtube_download_service, youtube_order_metadata
):
"""
Test that YouTube URL orders trigger workers after downloading audio.
Verifies workers are called after YouTube download completes successfully.
"""
from backend.api.routes.users import _handle_made_for_you_order
mock_job_manager = MagicMock()
mock_job = MagicMock()
mock_job.job_id = "test-job-workers"
mock_job_manager.create_job.return_value = mock_job
with patch('backend.services.job_manager.JobManager', return_value=mock_job_manager), \
patch('backend.services.worker_service.get_worker_service', return_value=mock_worker_service_local), \
patch('backend.services.storage_service.StorageService'), \
patch('backend.api.routes.users.get_theme_service', return_value=mock_theme_service), \
patch('backend.api.routes.users.get_effective_distribution_settings', return_value=mock_distribution_settings), \
patch('backend.api.routes.users.resolve_cdg_txt_defaults', return_value=(False, False)), \
patch('backend.api.routes.users._prepare_theme_for_job', return_value=(None, None, None)), \
patch('backend.api.routes.users.get_youtube_download_service', return_value=mock_youtube_download_service):
await _handle_made_for_you_order(
session_id="sess_test_workers",
metadata=youtube_order_metadata,
user_service=mock_user_service_for_webhook,
email_service=mock_email_service,
)
# Verify workers were triggered
mock_worker_service_local.trigger_audio_worker.assert_called_once_with(mock_job.job_id)
mock_worker_service_local.trigger_lyrics_worker.assert_called_once_with(mock_job.job_id)
@pytest.mark.asyncio
async def test_youtube_url_order_sends_notifications(
self, mock_email_service, mock_user_service_for_webhook,
mock_worker_service_local, mock_theme_service, mock_distribution_settings,
mock_youtube_download_service, youtube_order_metadata
):
"""
Test that YouTube URL orders send notification emails.
Verifies:
- Customer receives order confirmation
- Admin receives notification with job details
"""
from backend.api.routes.users import _handle_made_for_you_order, ADMIN_EMAIL
mock_job_manager = MagicMock()
mock_job = MagicMock()
mock_job.job_id = "test-job-notifications"
mock_job_manager.create_job.return_value = mock_job
with patch('backend.services.job_manager.JobManager', return_value=mock_job_manager), \
patch('backend.services.worker_service.get_worker_service', return_value=mock_worker_service_local), \
patch('backend.services.storage_service.StorageService'), \
patch('backend.api.routes.users.get_theme_service', return_value=mock_theme_service), \
patch('backend.api.routes.users.get_effective_distribution_settings', return_value=mock_distribution_settings), \
patch('backend.api.routes.users.resolve_cdg_txt_defaults', return_value=(False, False)), \
patch('backend.api.routes.users._prepare_theme_for_job', return_value=(None, None, None)), \
patch('backend.api.routes.users.get_youtube_download_service', return_value=mock_youtube_download_service):
await _handle_made_for_you_order(
session_id="sess_test_notifications",
metadata=youtube_order_metadata,
user_service=mock_user_service_for_webhook,
email_service=mock_email_service,
)
# Verify customer confirmation email
mock_email_service.send_made_for_you_order_confirmation.assert_called_once()
confirm_call = mock_email_service.send_made_for_you_order_confirmation.call_args
assert confirm_call.kwargs['to_email'] == youtube_order_metadata["customer_email"]
assert confirm_call.kwargs['artist'] == youtube_order_metadata["artist"]
assert confirm_call.kwargs['title'] == youtube_order_metadata["title"]
# Verify admin notification email
mock_email_service.send_made_for_you_admin_notification.assert_called_once()
admin_call = mock_email_service.send_made_for_you_admin_notification.call_args
assert admin_call.kwargs['to_email'] == ADMIN_EMAIL
assert admin_call.kwargs['job_id'] == mock_job.job_id
assert 'admin_login_token' in admin_call.kwargs
@pytest.mark.asyncio
async def test_youtube_url_order_marks_session_processed(
self, mock_email_service, mock_user_service_for_webhook,
mock_worker_service_local, mock_theme_service, mock_distribution_settings,
mock_youtube_download_service, youtube_order_metadata
):
"""
Test that Stripe session is marked as processed for idempotency.
"""
from backend.api.routes.users import _handle_made_for_you_order
mock_job_manager = MagicMock()
mock_job = MagicMock()
mock_job.job_id = "test-job-idempotency"
mock_job_manager.create_job.return_value = mock_job
with patch('backend.services.job_manager.JobManager', return_value=mock_job_manager), \
patch('backend.services.worker_service.get_worker_service', return_value=mock_worker_service_local), \
patch('backend.services.storage_service.StorageService'), \
patch('backend.api.routes.users.get_theme_service', return_value=mock_theme_service), \
patch('backend.api.routes.users.get_effective_distribution_settings', return_value=mock_distribution_settings), \
patch('backend.api.routes.users.resolve_cdg_txt_defaults', return_value=(False, False)), \
patch('backend.api.routes.users._prepare_theme_for_job', return_value=(None, None, None)), \
patch('backend.api.routes.users.get_youtube_download_service', return_value=mock_youtube_download_service):
await _handle_made_for_you_order(
session_id="sess_test_idempotency",
metadata=youtube_order_metadata,
user_service=mock_user_service_for_webhook,
email_service=mock_email_service,
)
# Verify session was marked processed
mock_user_service_for_webhook._mark_stripe_session_processed.assert_called_once()
mark_call = mock_user_service_for_webhook._mark_stripe_session_processed.call_args
assert mark_call.kwargs['stripe_session_id'] == "sess_test_idempotency"
assert mark_call.kwargs['email'] == youtube_order_metadata["customer_email"]
@pytest.mark.asyncio
async def test_youtube_url_order_with_customer_notes(
self, mock_email_service, mock_user_service_for_webhook,
mock_worker_service_local, mock_theme_service, mock_distribution_settings,
mock_youtube_download_service, youtube_order_metadata_with_notes
):
"""
Test that customer notes are preserved on the job.
"""
from backend.api.routes.users import _handle_made_for_you_order
mock_job_manager = MagicMock()
mock_job = MagicMock()
mock_job.job_id = "test-job-notes"
mock_job_manager.create_job.return_value = mock_job
with patch('backend.services.job_manager.JobManager', return_value=mock_job_manager), \
patch('backend.services.worker_service.get_worker_service', return_value=mock_worker_service_local), \
patch('backend.services.storage_service.StorageService'), \
patch('backend.api.routes.users.get_theme_service', return_value=mock_theme_service), \
patch('backend.api.routes.users.get_effective_distribution_settings', return_value=mock_distribution_settings), \
patch('backend.api.routes.users.resolve_cdg_txt_defaults', return_value=(False, False)), \
patch('backend.api.routes.users._prepare_theme_for_job', return_value=(None, None, None)), \
patch('backend.api.routes.users.get_youtube_download_service', return_value=mock_youtube_download_service):
await _handle_made_for_you_order(
session_id="sess_test_notes",
metadata=youtube_order_metadata_with_notes,
user_service=mock_user_service_for_webhook,
email_service=mock_email_service,
)
job_create_arg = mock_job_manager.create_job.call_args[0][0]
assert job_create_arg.customer_notes == youtube_order_metadata_with_notes["notes"], \
"Customer notes must be preserved on job"
@pytest.mark.asyncio
async def test_youtube_url_order_skips_audio_search(
self, mock_email_service, mock_user_service_for_webhook,
mock_worker_service_local, mock_theme_service, mock_distribution_settings,
mock_youtube_download_service, youtube_order_metadata
):
"""
Test that YouTube URL orders skip audio search (download + trigger workers directly).
"""
from backend.api.routes.users import _handle_made_for_you_order
mock_job_manager = MagicMock()
mock_job = MagicMock()
mock_job.job_id = "test-job-no-search"
mock_job_manager.create_job.return_value = mock_job
with patch('backend.services.job_manager.JobManager', return_value=mock_job_manager), \
patch('backend.services.worker_service.get_worker_service', return_value=mock_worker_service_local), \
patch('backend.services.audio_search_service.get_audio_search_service') as mock_get_audio, \
patch('backend.services.storage_service.StorageService'), \
patch('backend.api.routes.users.get_theme_service', return_value=mock_theme_service), \
patch('backend.api.routes.users.get_effective_distribution_settings', return_value=mock_distribution_settings), \
patch('backend.api.routes.users.resolve_cdg_txt_defaults', return_value=(False, False)), \
patch('backend.api.routes.users._prepare_theme_for_job', return_value=(None, None, None)), \
patch('backend.api.routes.users.get_youtube_download_service', return_value=mock_youtube_download_service):
mock_audio_service = MagicMock()
mock_get_audio.return_value = mock_audio_service
await _handle_made_for_you_order(
session_id="sess_test_no_search",
metadata=youtube_order_metadata,
user_service=mock_user_service_for_webhook,
email_service=mock_email_service,
)
# Audio search should NOT be called for YouTube URL orders
mock_audio_service.search.assert_not_called()
# Workers should be triggered directly
mock_worker_service_local.trigger_audio_worker.assert_called_once()
class TestMadeForYouYouTubeJobState:
"""
Tests for job state consistency during YouTube URL processing.
These tests verify job state is set correctly when jobs are created
through the made-for-you webhook handler.
"""
@pytest.mark.asyncio
async def test_job_state_set_correctly_for_youtube_url_order(
self, mock_email_service, mock_user_service_for_webhook,
mock_worker_service_local, mock_theme_service, mock_distribution_settings,
mock_youtube_download_service, youtube_order_metadata
):
"""
Test that job state is set correctly for YouTube URL orders.
Verifies the JobCreate object has all required fields for proper
job state persistence.
"""
from backend.api.routes.users import _handle_made_for_you_order, ADMIN_EMAIL
mock_job_manager = MagicMock()
mock_job = MagicMock()
mock_job.job_id = "test-job-state"
mock_job_manager.create_job.return_value = mock_job
with patch('backend.services.job_manager.JobManager', return_value=mock_job_manager), \
patch('backend.services.worker_service.get_worker_service', return_value=mock_worker_service_local), \
patch('backend.services.storage_service.StorageService'), \
patch('backend.api.routes.users.get_theme_service', return_value=mock_theme_service), \
patch('backend.api.routes.users.get_effective_distribution_settings', return_value=mock_distribution_settings), \
patch('backend.api.routes.users.resolve_cdg_txt_defaults', return_value=(False, False)), \
patch('backend.api.routes.users._prepare_theme_for_job', return_value=(None, None, None)), \
patch('backend.api.routes.users.get_youtube_download_service', return_value=mock_youtube_download_service):
await _handle_made_for_you_order(
session_id="sess_test_state",
metadata=youtube_order_metadata,
user_service=mock_user_service_for_webhook,
email_service=mock_email_service,
)
# Verify job was created with correct state
job_create_arg = mock_job_manager.create_job.call_args[0][0]
# Core made-for-you fields
assert job_create_arg.made_for_you is True
assert job_create_arg.customer_email == youtube_order_metadata["customer_email"]
assert job_create_arg.url == youtube_order_metadata["youtube_url"]
assert job_create_arg.artist == youtube_order_metadata["artist"]
assert job_create_arg.title == youtube_order_metadata["title"]
assert job_create_arg.user_email == ADMIN_EMAIL
# Job configuration
assert job_create_arg.non_interactive is False # Admin will review
assert job_create_arg.auto_download is False # Pause at audio selection
class TestMadeForYouYouTubeErrorHandling:
"""
Tests for error handling scenarios in the YouTube URL flow.
"""
@pytest.mark.asyncio
async def test_exception_sends_failure_notification(
self, mock_email_service, mock_user_service_for_webhook,
mock_theme_service, mock_distribution_settings,
youtube_order_metadata
):
"""
Test that exceptions during processing send failure notification to admin.
"""
from backend.api.routes.users import _handle_made_for_you_order, ADMIN_EMAIL
# Make job creation fail
mock_job_manager = MagicMock()
mock_job_manager.create_job.side_effect = Exception("Database error")
with patch('backend.services.job_manager.JobManager', return_value=mock_job_manager), \
patch('backend.services.storage_service.StorageService'), \
patch('backend.api.routes.users.get_theme_service', return_value=mock_theme_service), \
patch('backend.api.routes.users.get_effective_distribution_settings', return_value=mock_distribution_settings), \
patch('backend.api.routes.users.resolve_cdg_txt_defaults', return_value=(False, False)):
# Should not raise - error is handled internally
await _handle_made_for_you_order(
session_id="sess_test_error",
metadata=youtube_order_metadata,
user_service=mock_user_service_for_webhook,
email_service=mock_email_service,
)
# Verify failure email was sent to admin
mock_email_service.send_email.assert_called()
call_args = mock_email_service.send_email.call_args
assert call_args.kwargs['to_email'] == ADMIN_EMAIL
assert "[FAILED]" in call_args.kwargs['subject']
assert "Database error" in call_args.kwargs['html_content']
class TestMadeForYouWebhookIntegration:
"""
Tests for the Stripe webhook handling with made-for-you orders.
These tests verify the webhook handler processes made-for-you orders correctly.
"""
@pytest.mark.asyncio
async def test_handle_made_for_you_order_full_flow(
self, mock_email_service, mock_user_service_for_webhook,
mock_worker_service_local, mock_theme_service, mock_distribution_settings,
mock_youtube_download_service, youtube_order_metadata
):
"""
Test the full made-for-you order handler flow.
This is what gets called when a Stripe webhook fires for a
made-for-you order. Verifies:
- Job is created with correct metadata
- YouTube audio is downloaded
- Workers are triggered
- Emails are sent
- Session is marked processed
"""
from backend.api.routes.users import _handle_made_for_you_order, ADMIN_EMAIL
mock_job_manager = MagicMock()
mock_job = MagicMock()
mock_job.job_id = "test-job-webhook-flow"
mock_job_manager.create_job.return_value = mock_job
with patch('backend.services.job_manager.JobManager', return_value=mock_job_manager), \
patch('backend.services.worker_service.get_worker_service', return_value=mock_worker_service_local), \
patch('backend.services.storage_service.StorageService'), \
patch('backend.api.routes.users.get_theme_service', return_value=mock_theme_service), \
patch('backend.api.routes.users.get_effective_distribution_settings', return_value=mock_distribution_settings), \
patch('backend.api.routes.users.resolve_cdg_txt_defaults', return_value=(False, False)), \
patch('backend.api.routes.users._prepare_theme_for_job', return_value=(None, None, None)), \
patch('backend.api.routes.users.get_youtube_download_service', return_value=mock_youtube_download_service):
await _handle_made_for_you_order(
session_id="cs_test_webhook_flow",
metadata=youtube_order_metadata,
user_service=mock_user_service_for_webhook,
email_service=mock_email_service,
)
# 1. Verify job was created
assert mock_job_manager.create_job.called
job_create_arg = mock_job_manager.create_job.call_args[0][0]
assert job_create_arg.made_for_you is True
assert job_create_arg.url == youtube_order_metadata["youtube_url"]
# 2. Verify YouTube download was called
mock_youtube_download_service.download.assert_called_once()
# 3. Verify workers were triggered
mock_worker_service_local.trigger_audio_worker.assert_called_once()
mock_worker_service_local.trigger_lyrics_worker.assert_called_once()
# 4. Verify emails were sent
mock_email_service.send_made_for_you_order_confirmation.assert_called_once()
mock_email_service.send_made_for_you_admin_notification.assert_called_once()
# 5. Verify session marked processed
mock_user_service_for_webhook._mark_stripe_session_processed.assert_called_once()
assert mock_user_service_for_webhook._mark_stripe_session_processed.call_args.kwargs['stripe_session_id'] == "cs_test_webhook_flow"
class TestMadeForYouYouTubeDistributionSettings:
"""
Tests for distribution settings on made-for-you YouTube orders.
"""
@pytest.mark.asyncio
async def test_distribution_defaults_applied(
self, mock_email_service, mock_user_service_for_webhook,
mock_worker_service_local, mock_theme_service, mock_youtube_download_service,
youtube_order_metadata
):
"""
Test that distribution defaults are applied to made-for-you jobs.
Made-for-you jobs should have automatic distribution enabled.
"""
from backend.api.routes.users import _handle_made_for_you_order
mock_job_manager = MagicMock()
mock_job = MagicMock()
mock_job.job_id = "test-job-distribution"
mock_job_manager.create_job.return_value = mock_job
# Mock distribution settings
mock_settings = MagicMock()
mock_settings.enable_youtube_upload = True
mock_settings.dropbox_path = "/Nomad Karaoke"
mock_settings.gdrive_folder_id = "test-folder-id"
mock_settings.brand_prefix = "Nomad Karaoke"
mock_settings.discord_webhook_url = None
mock_settings.youtube_description = None
with patch('backend.services.job_manager.JobManager', return_value=mock_job_manager), \
patch('backend.services.worker_service.get_worker_service', return_value=mock_worker_service_local), \
patch('backend.services.storage_service.StorageService'), \
patch('backend.api.routes.users.get_theme_service', return_value=mock_theme_service), \
patch('backend.api.routes.users.get_youtube_download_service', return_value=mock_youtube_download_service), \
patch('backend.api.routes.users.get_effective_distribution_settings', return_value=mock_settings), \
patch('backend.api.routes.users.resolve_cdg_txt_defaults', return_value=(False, False)), \
patch('backend.api.routes.users._prepare_theme_for_job', return_value=(None, None, None)):
await _handle_made_for_you_order(
session_id="sess_test_distribution",
metadata=youtube_order_metadata,
user_service=mock_user_service_for_webhook,
email_service=mock_email_service,
)
# Verify distribution settings were applied
job_create_arg = mock_job_manager.create_job.call_args[0][0]
# These should match the mocked distribution settings
assert job_create_arg.enable_youtube_upload == mock_settings.enable_youtube_upload
assert job_create_arg.dropbox_path == mock_settings.dropbox_path
assert job_create_arg.gdrive_folder_id == mock_settings.gdrive_folder_id
assert job_create_arg.brand_prefix == mock_settings.brand_prefix
class TestMadeForYouYouTubeGCSPath:
"""
Tests for GCS path format consistency in YouTube URL orders.
"""
@pytest.mark.asyncio
async def test_job_url_set_correctly(
self, mock_email_service, mock_user_service_for_webhook,
mock_worker_service_local, mock_theme_service, mock_distribution_settings,
mock_youtube_download_service
):
"""
Test that the YouTube URL is stored correctly on the job.
"""
from backend.api.routes.users import _handle_made_for_you_order
test_url = "https://www.youtube.com/watch?v=dQw4w9WgXcQ"
metadata = {
"order_type": "made_for_you",
"customer_email": "customer@example.com",
"artist": "Rick Astley",
"title": "Never Gonna Give You Up",
"source_type": "youtube",
"youtube_url": test_url,
}
mock_job_manager = MagicMock()
mock_job = MagicMock()
mock_job.job_id = "test-job-url"
mock_job_manager.create_job.return_value = mock_job
with patch('backend.services.job_manager.JobManager', return_value=mock_job_manager), \
patch('backend.services.worker_service.get_worker_service', return_value=mock_worker_service_local), \
patch('backend.services.storage_service.StorageService'), \
patch('backend.api.routes.users.get_theme_service', return_value=mock_theme_service), \
patch('backend.api.routes.users.get_effective_distribution_settings', return_value=mock_distribution_settings), \
patch('backend.api.routes.users.resolve_cdg_txt_defaults', return_value=(False, False)), \
patch('backend.api.routes.users._prepare_theme_for_job', return_value=(None, None, None)), \
patch('backend.api.routes.users.get_youtube_download_service', return_value=mock_youtube_download_service):
await _handle_made_for_you_order(
session_id="sess_test_url",
metadata=metadata,
user_service=mock_user_service_for_webhook,
email_service=mock_email_service,
)
job_create_arg = mock_job_manager.create_job.call_args[0][0]
assert job_create_arg.url == test_url
print("Made-for-you YouTube integration tests ready to run")
@@ -1,332 +0,0 @@
"""
Integration tests for YouTube download flow.
Tests the full flow from API endpoint to download completion, verifying that
both entry points (audio search selection and direct URL submission) correctly
use the YouTubeDownloadService.
"""
import pytest
from unittest.mock import AsyncMock, MagicMock, patch
from fastapi.testclient import TestClient
@pytest.fixture
def mock_youtube_service():
"""Create a mock YouTubeDownloadService."""
mock = MagicMock()
mock.download = AsyncMock(return_value="uploads/test_job/audio/test.flac")
mock.download_by_id = AsyncMock(return_value="uploads/test_job/audio/test.flac")
mock.is_remote_enabled = MagicMock(return_value=True)
mock._extract_video_id = MagicMock(return_value="test_video_id")
return mock
@pytest.fixture
def mock_services():
"""Mock common services used across tests."""
mocks = {
'job_manager': MagicMock(),
'storage_service': MagicMock(),
'worker_service': MagicMock(),
'auth_service': MagicMock(),
}
# Configure job_manager mock
mock_job = MagicMock()
mock_job.job_id = "test_job_123"
mock_job.status = "pending"
mock_job.state_data = {}
mocks['job_manager'].create_job = MagicMock(return_value=mock_job)
mocks['job_manager'].get_job = MagicMock(return_value=mock_job)
mocks['job_manager'].transition_to_state = MagicMock()
mocks['job_manager'].update_job = MagicMock()
mocks['job_manager'].fail_job = MagicMock()
# Configure worker service mock
mocks['worker_service'].trigger_audio_worker = AsyncMock()
mocks['worker_service'].trigger_lyrics_worker = AsyncMock()
return mocks
class TestAudioSearchYouTubeFlow:
"""Tests for YouTube downloads via audio search selection."""
@pytest.mark.asyncio
async def test_audio_search_select_youtube_uses_youtube_service(self, mock_youtube_service, mock_services):
"""
When selecting a YouTube result from audio search,
should use YouTubeDownloadService instead of direct download.
"""
# Configure mock job with search results
mock_job = mock_services['job_manager'].get_job.return_value
mock_job.status = "awaiting_audio_selection"
mock_job.state_data = {
'audio_search_results': [{
'index': 0,
'title': 'Test Song',
'artist': 'Test Artist',
'provider': 'YouTube',
'source_id': 'dQw4w9WgXcQ',
'url': 'https://www.youtube.com/watch?v=dQw4w9WgXcQ',
}],
'audio_search_count': 1,
}
# Import the route function
from backend.api.routes.audio_search import _download_and_start_processing
# Patch dependencies
with patch(
'backend.api.routes.audio_search.get_youtube_download_service',
return_value=mock_youtube_service
), patch(
'backend.api.routes.audio_search.job_manager',
mock_services['job_manager']
), patch(
'backend.api.routes.audio_search.storage_service',
mock_services['storage_service']
):
# Create a mock audio_search_service
mock_audio_search_service = MagicMock()
mock_audio_search_service.is_remote_enabled = MagicMock(return_value=True)
# Create a mock background_tasks
mock_background_tasks = MagicMock()
# Call the download function
result = await _download_and_start_processing(
job_id="test_job_123",
selection_index=0,
audio_search_service=mock_audio_search_service,
background_tasks=mock_background_tasks,
)
# Verify YouTubeDownloadService was used
mock_youtube_service.download_by_id.assert_called_once()
call_args = mock_youtube_service.download_by_id.call_args
assert call_args.kwargs['video_id'] == 'dQw4w9WgXcQ'
assert call_args.kwargs['job_id'] == 'test_job_123'
assert call_args.kwargs['artist'] == 'Test Artist'
assert call_args.kwargs['title'] == 'Test Song'
@pytest.mark.asyncio
async def test_torrent_download_does_not_use_youtube_service(self, mock_youtube_service, mock_services):
"""
When selecting a torrent result (RED/OPS), should NOT use YouTubeDownloadService.
"""
mock_job = mock_services['job_manager'].get_job.return_value
mock_job.status = "awaiting_audio_selection"
mock_job.state_data = {
'audio_search_results': [{
'index': 0,
'title': 'Test Song',
'artist': 'Test Artist',
'provider': 'RED',
'source_id': '12345',
'target_file': 'test.flac',
}],
'audio_search_count': 1,
}
from backend.api.routes.audio_search import _download_and_start_processing
# Mock audio_search_service for torrent download
mock_audio_search_service = MagicMock()
mock_audio_search_service.is_remote_enabled = MagicMock(return_value=True)
mock_result = MagicMock()
mock_result.filepath = "gs://bucket/uploads/test/audio/test.flac"
mock_audio_search_service.download_by_id = MagicMock(return_value=mock_result)
with patch(
'backend.api.routes.audio_search.get_youtube_download_service',
return_value=mock_youtube_service
), patch(
'backend.api.routes.audio_search.job_manager',
mock_services['job_manager']
), patch(
'backend.api.routes.audio_search.storage_service',
mock_services['storage_service']
):
mock_background_tasks = MagicMock()
result = await _download_and_start_processing(
job_id="test_job_123",
selection_index=0,
audio_search_service=mock_audio_search_service,
background_tasks=mock_background_tasks,
)
# YouTubeDownloadService should NOT be called for torrent
mock_youtube_service.download_by_id.assert_not_called()
# audio_search_service.download_by_id should be called instead
mock_audio_search_service.download_by_id.assert_called_once()
class TestDirectUrlSubmissionFlow:
"""Tests for direct YouTube URL submission via /api/jobs/url endpoint."""
@pytest.mark.asyncio
async def test_create_job_from_youtube_url_downloads_immediately(
self, mock_youtube_service, mock_services
):
"""
When creating a job from a YouTube URL, should download audio
immediately (before triggering workers) using YouTubeDownloadService.
"""
from backend.api.routes.file_upload import _is_youtube_url
# Verify helper function works
assert _is_youtube_url("https://www.youtube.com/watch?v=test") is True
assert _is_youtube_url("https://youtu.be/test") is True
assert _is_youtube_url("https://vimeo.com/test") is False
@pytest.mark.asyncio
async def test_non_youtube_url_triggers_worker_download(self, mock_services):
"""
When creating a job from a non-YouTube URL (e.g., Vimeo),
should trigger audio worker to handle the download.
"""
from backend.api.routes.file_upload import _is_youtube_url
# Non-YouTube URLs should not match
assert _is_youtube_url("https://vimeo.com/123456") is False
assert _is_youtube_url("https://soundcloud.com/artist/track") is False
class TestYouTubeDownloadErrorHandling:
"""Tests for error handling in YouTube download flow."""
@pytest.mark.asyncio
async def test_youtube_download_failure_fails_job(self, mock_services):
"""
When YouTube download fails, should properly fail the job
with a clear error message.
"""
from backend.services.youtube_download_service import (
YouTubeDownloadService,
YouTubeDownloadError,
)
# Create a service that will fail
mock_client = MagicMock()
mock_client.download_by_id = AsyncMock(
side_effect=Exception("Bot detection triggered")
)
with patch(
'backend.services.youtube_download_service.get_flacfetch_client',
return_value=mock_client
):
from backend.services.youtube_download_service import reset_youtube_download_service
reset_youtube_download_service()
service = YouTubeDownloadService()
with pytest.raises(YouTubeDownloadError) as exc:
await service.download(
url="https://youtu.be/abcDEF12345",
job_id="job123",
)
assert "Remote download failed" in str(exc.value)
@pytest.mark.asyncio
async def test_invalid_youtube_url_raises_clear_error(self):
"""
When given an invalid YouTube URL (no video ID extractable),
should raise a clear error.
"""
from backend.services.youtube_download_service import (
YouTubeDownloadService,
YouTubeDownloadError,
reset_youtube_download_service,
)
with patch(
'backend.services.youtube_download_service.get_flacfetch_client',
return_value=None # No remote client
):
reset_youtube_download_service()
service = YouTubeDownloadService()
with pytest.raises(YouTubeDownloadError) as exc:
await service.download(
url="not-a-valid-youtube-url",
job_id="job123",
)
assert "Could not extract video ID" in str(exc.value)
class TestRemoteVsLocalDownload:
"""Tests verifying correct routing between remote and local download."""
@pytest.mark.asyncio
async def test_remote_download_when_configured(self):
"""
When FLACFETCH_API_URL is configured, should use remote download.
"""
mock_client = MagicMock()
mock_client.download_by_id = AsyncMock(return_value="download_123")
mock_client.wait_for_download = AsyncMock(return_value={
"status": "complete",
"gcs_path": "gs://bucket/uploads/job_test/audio/test.flac",
})
with patch(
'backend.services.youtube_download_service.get_flacfetch_client',
return_value=mock_client
):
from backend.services.youtube_download_service import (
YouTubeDownloadService,
reset_youtube_download_service,
)
reset_youtube_download_service()
service = YouTubeDownloadService()
assert service.is_remote_enabled() is True
result = await service.download(
url="https://youtu.be/abcDEF12345",
job_id="job_test",
)
# Should have called remote client
mock_client.download_by_id.assert_called_once()
assert "uploads/job_test/audio/" in result
@pytest.mark.asyncio
async def test_local_download_when_remote_not_configured(self):
"""
When FLACFETCH_API_URL is not configured, should fall back to local.
"""
with patch(
'backend.services.youtube_download_service.get_flacfetch_client',
return_value=None # No remote client
):
from backend.services.youtube_download_service import (
YouTubeDownloadService,
reset_youtube_download_service,
)
reset_youtube_download_service()
service = YouTubeDownloadService()
assert service.is_remote_enabled() is False
# Mock the local download to avoid actually calling yt_dlp
with patch.object(
service, '_download_local',
new_callable=AsyncMock,
return_value="uploads/job/audio/test.wav"
) as mock_local:
result = await service.download(
url="https://youtu.be/abcDEF12345",
job_id="job_test",
)
mock_local.assert_called_once()
assert result == "uploads/job/audio/test.wav"
-120
View File
@@ -350,123 +350,3 @@ class TestDeleteOutputsAuthorization:
app.dependency_overrides[require_admin] = original_override
else:
app.dependency_overrides[require_admin] = get_mock_admin
class TestDeleteOutputsGCSFinals:
"""Tests for GCS finals folder deletion."""
def test_delete_outputs_deletes_gcs_finals(self, client, mock_complete_job):
"""Test that GCS finals folder is deleted when outputs are deleted."""
with patch('backend.api.routes.admin.JobManager') as mock_jm_class, \
patch('backend.api.routes.admin.get_user_service') as mock_user_service, \
patch('backend.services.storage_service.StorageService') as mock_storage_class:
mock_jm = Mock()
mock_jm.get_job.return_value = mock_complete_job
mock_jm_class.return_value = mock_jm
# Mock storage service
mock_storage = Mock()
mock_storage.delete_folder.return_value = 4 # 4 files deleted
mock_storage_class.return_value = mock_storage
# Mock Firestore
mock_db = Mock()
mock_job_ref = Mock()
mock_db.collection.return_value.document.return_value = mock_job_ref
mock_user_service.return_value.db = mock_db
response = client.post("/api/admin/jobs/test-job-123/delete-outputs")
assert response.status_code == 200
data = response.json()
# Verify GCS finals was deleted
mock_storage.delete_folder.assert_called_once_with("jobs/test-job-123/finals/")
assert "gcs_finals" in data["deleted_services"]
assert data["deleted_services"]["gcs_finals"]["status"] == "success"
assert data["deleted_services"]["gcs_finals"]["deleted_count"] == 4
assert data["deleted_services"]["gcs_finals"]["path"] == "jobs/test-job-123/finals/"
def test_delete_outputs_gcs_success_with_zero_files(self, client, mock_complete_job):
"""Test that GCS deletion reports success even when no files exist."""
with patch('backend.api.routes.admin.JobManager') as mock_jm_class, \
patch('backend.api.routes.admin.get_user_service') as mock_user_service, \
patch('backend.services.storage_service.StorageService') as mock_storage_class:
mock_jm = Mock()
mock_jm.get_job.return_value = mock_complete_job
mock_jm_class.return_value = mock_jm
# Mock storage service - no files to delete
mock_storage = Mock()
mock_storage.delete_folder.return_value = 0
mock_storage_class.return_value = mock_storage
# Mock Firestore
mock_db = Mock()
mock_job_ref = Mock()
mock_db.collection.return_value.document.return_value = mock_job_ref
mock_user_service.return_value.db = mock_db
response = client.post("/api/admin/jobs/test-job-123/delete-outputs")
assert response.status_code == 200
data = response.json()
# Still success even with zero files deleted
assert data["deleted_services"]["gcs_finals"]["status"] == "success"
assert data["deleted_services"]["gcs_finals"]["deleted_count"] == 0
def test_delete_outputs_continues_if_gcs_fails(self, client, mock_complete_job):
"""Test that other deletions continue even if GCS deletion fails."""
with patch('backend.api.routes.admin.JobManager') as mock_jm_class, \
patch('backend.api.routes.admin.get_user_service') as mock_user_service, \
patch('backend.services.storage_service.StorageService') as mock_storage_class:
mock_jm = Mock()
mock_jm.get_job.return_value = mock_complete_job
mock_jm_class.return_value = mock_jm
# Mock storage service to fail
mock_storage = Mock()
mock_storage.delete_folder.side_effect = Exception("GCS error")
mock_storage_class.return_value = mock_storage
# Mock Firestore
mock_db = Mock()
mock_job_ref = Mock()
mock_db.collection.return_value.document.return_value = mock_job_ref
mock_user_service.return_value.db = mock_db
response = client.post("/api/admin/jobs/test-job-123/delete-outputs")
# Should still succeed (partial success) with error for GCS
assert response.status_code == 200
data = response.json()
assert "gcs_finals" in data["deleted_services"]
assert data["deleted_services"]["gcs_finals"]["status"] == "error"
assert "GCS error" in data["deleted_services"]["gcs_finals"]["error"]
def test_delete_outputs_gcs_uses_correct_path(self, client, mock_complete_job):
"""Test that GCS deletion uses the correct path format."""
mock_complete_job.job_id = "abc-123-xyz"
with patch('backend.api.routes.admin.JobManager') as mock_jm_class, \
patch('backend.api.routes.admin.get_user_service') as mock_user_service, \
patch('backend.services.storage_service.StorageService') as mock_storage_class:
mock_jm = Mock()
mock_jm.get_job.return_value = mock_complete_job
mock_jm_class.return_value = mock_jm
mock_storage = Mock()
mock_storage.delete_folder.return_value = 2
mock_storage_class.return_value = mock_storage
mock_db = Mock()
mock_job_ref = Mock()
mock_db.collection.return_value.document.return_value = mock_job_ref
mock_user_service.return_value.db = mock_db
response = client.post("/api/admin/jobs/abc-123-xyz/delete-outputs")
assert response.status_code == 200
# Verify path format is jobs/{job_id}/finals/
mock_storage.delete_folder.assert_called_once_with("jobs/abc-123-xyz/finals/")
-551
View File
@@ -382,554 +382,3 @@ class TestResetJobClearsStateData:
)
assert response.status_code == 200
class TestResetJobToInstrumentalSelected:
"""Tests for resetting job to INSTRUMENTAL_SELECTED state (reprocess video)."""
def test_reset_to_instrumental_selected_success(self, client, mock_job):
"""Test resetting a job to INSTRUMENTAL_SELECTED state."""
import asyncio
from unittest.mock import AsyncMock
with patch('backend.api.routes.admin.JobManager') as mock_jm_class, \
patch('backend.services.worker_service.get_worker_service') as mock_ws:
mock_jm = Mock()
mock_jm.get_job.return_value = mock_job
mock_jm.update_job.return_value = True
mock_jm_class.return_value = mock_jm
# Mock worker service with async method
mock_worker_service = Mock()
mock_worker_service.trigger_video_worker = AsyncMock(return_value=True)
mock_ws.return_value = mock_worker_service
response = client.post(
"/api/admin/jobs/test-job-123/reset",
json={"target_state": "instrumental_selected"},
)
assert response.status_code == 200
data = response.json()
assert data["new_status"] == "instrumental_selected"
def test_reset_to_instrumental_selected_triggers_video_worker(self, client, mock_job):
"""Test that resetting to INSTRUMENTAL_SELECTED triggers video worker."""
from unittest.mock import AsyncMock
with patch('backend.api.routes.admin.JobManager') as mock_jm_class, \
patch('backend.services.worker_service.get_worker_service') as mock_ws:
mock_jm = Mock()
mock_jm.get_job.return_value = mock_job
mock_jm.update_job.return_value = True
mock_jm_class.return_value = mock_jm
# Mock worker service
mock_worker_service = Mock()
mock_worker_service.trigger_video_worker = AsyncMock(return_value=True)
mock_ws.return_value = mock_worker_service
response = client.post(
"/api/admin/jobs/test-job-123/reset",
json={"target_state": "instrumental_selected"},
)
assert response.status_code == 200
# Verify worker was triggered
mock_worker_service.trigger_video_worker.assert_called_once_with("test-job-123")
# Verify message indicates worker was triggered
data = response.json()
assert "video worker triggered" in data["message"]
def test_reset_to_instrumental_selected_message_on_worker_failure(self, client, mock_job):
"""Test message when video worker trigger fails."""
from unittest.mock import AsyncMock
with patch('backend.api.routes.admin.JobManager') as mock_jm_class, \
patch('backend.services.worker_service.get_worker_service') as mock_ws:
mock_jm = Mock()
mock_jm.get_job.return_value = mock_job
mock_jm.update_job.return_value = True
mock_jm_class.return_value = mock_jm
# Mock worker service failure
mock_worker_service = Mock()
mock_worker_service.trigger_video_worker = AsyncMock(return_value=False)
mock_ws.return_value = mock_worker_service
response = client.post(
"/api/admin/jobs/test-job-123/reset",
json={"target_state": "instrumental_selected"},
)
assert response.status_code == 200
# Reset still succeeds even if worker trigger fails
data = response.json()
assert data["new_status"] == "instrumental_selected"
# Message should NOT say worker was triggered
assert "video worker triggered" not in data["message"]
def test_reset_to_instrumental_selected_clears_video_state(self, client, mock_job):
"""Test that INSTRUMENTAL_SELECTED reset clears video/encoding/distribution state."""
from unittest.mock import AsyncMock
mock_job.state_data = {
"instrumental_selection": "clean",
"lyrics_metadata": {"has_countdown_padding": True},
"video_progress": {"stage": "running"},
"brand_code": "NOMAD-1234",
"youtube_url": "https://youtube.com/watch?v=xxx",
"dropbox_link": "https://dropbox.com/xxx",
}
with patch('backend.api.routes.admin.JobManager') as mock_jm_class, \
patch('backend.services.worker_service.get_worker_service') as mock_ws:
mock_jm = Mock()
mock_jm.get_job.return_value = mock_job
mock_jm.update_job.return_value = True
mock_jm_class.return_value = mock_jm
mock_worker_service = Mock()
mock_worker_service.trigger_video_worker = AsyncMock(return_value=True)
mock_ws.return_value = mock_worker_service
response = client.post(
"/api/admin/jobs/test-job-123/reset",
json={"target_state": "instrumental_selected"},
)
assert response.status_code == 200
data = response.json()
# These keys should be marked as cleared
cleared = data.get("cleared_data", [])
# At minimum, video_progress and brand_code should be cleared
# (they exist in the mock job's state_data)
assert any(key in ["video_progress", "brand_code", "youtube_url", "dropbox_link"]
for key in cleared)
class TestResetJobClearsErrorState:
"""Tests that reset clears error_message and error_details."""
def test_reset_clears_error_message(self, client, mock_job):
"""Test that resetting a job clears error_message field."""
# Set job to failed state with error message
mock_job.status = JobStatus.FAILED
mock_job.error_message = "Previous error message"
mock_job.error_details = {"code": "TEST_ERROR"}
with patch('backend.api.routes.admin.JobManager') as mock_jm_class:
mock_jm = Mock()
mock_jm.get_job.return_value = mock_job
mock_jm.update_job.return_value = True
mock_jm_class.return_value = mock_jm
response = client.post(
"/api/admin/jobs/test-job-123/reset",
json={"target_state": "pending"},
)
assert response.status_code == 200
# The endpoint should issue an update that includes DELETE_FIELD for error fields
# We verify this indirectly through the success of the operation
class TestClearWorkersEndpoint:
"""Tests for the POST /api/admin/jobs/{job_id}/clear-workers endpoint."""
def test_clear_workers_success(self, client, mock_job):
"""Test clearing worker progress keys."""
mock_job.state_data = {
"render_progress": {"stage": "complete"},
"video_progress": {"stage": "complete"},
"encoding_progress": {"stage": "complete"},
}
with patch('backend.api.routes.admin.JobManager') as mock_jm_class:
mock_jm = Mock()
mock_jm.get_job.return_value = mock_job
mock_jm_class.return_value = mock_jm
response = client.post("/api/admin/jobs/test-job-123/clear-workers")
assert response.status_code == 200
data = response.json()
assert data["status"] == "success"
assert "cleared_keys" in data
# Should have cleared the progress keys that exist
assert any(k in data["cleared_keys"] for k in
["render_progress", "video_progress", "encoding_progress"])
def test_clear_workers_no_progress_keys(self, client, mock_job):
"""Test clearing workers when no progress keys exist."""
mock_job.state_data = {"other_data": "value"}
with patch('backend.api.routes.admin.JobManager') as mock_jm_class:
mock_jm = Mock()
mock_jm.get_job.return_value = mock_job
mock_jm_class.return_value = mock_jm
response = client.post("/api/admin/jobs/test-job-123/clear-workers")
assert response.status_code == 200
data = response.json()
assert data["status"] == "success"
assert len(data["cleared_keys"]) == 0
def test_clear_workers_job_not_found(self, client):
"""Test clearing workers for non-existent job."""
with patch('backend.api.routes.admin.JobManager') as mock_jm_class:
mock_jm = Mock()
mock_jm.get_job.return_value = None
mock_jm_class.return_value = mock_jm
response = client.post("/api/admin/jobs/nonexistent/clear-workers")
assert response.status_code == 404
def test_clear_workers_clears_all_worker_types(self, client, mock_job):
"""Test that all defined worker progress keys can be cleared."""
# Set all possible worker progress keys
mock_job.state_data = {
"audio_progress": {"stage": "complete"},
"lyrics_progress": {"stage": "complete"},
"render_progress": {"stage": "complete"},
"screens_progress": {"stage": "complete"},
"video_progress": {"stage": "complete"},
"encoding_progress": {"stage": "complete"},
}
with patch('backend.api.routes.admin.JobManager') as mock_jm_class:
mock_jm = Mock()
mock_jm.get_job.return_value = mock_job
mock_jm_class.return_value = mock_jm
response = client.post("/api/admin/jobs/test-job-123/clear-workers")
assert response.status_code == 200
data = response.json()
# Should have cleared all 6 progress keys
assert len(data["cleared_keys"]) == 6
class TestResetIncludesEncodingProgress:
"""Tests that reset includes encoding_progress in cleared keys."""
def test_reset_to_awaiting_review_clears_encoding_progress(self, client, mock_job):
"""Test that resetting to awaiting_review clears encoding_progress."""
mock_job.state_data = {
"encoding_progress": {"stage": "complete"},
"video_progress": {"stage": "complete"},
}
with patch('backend.api.routes.admin.JobManager') as mock_jm_class:
mock_jm = Mock()
mock_jm.get_job.return_value = mock_job
mock_jm.update_job.return_value = True
mock_jm_class.return_value = mock_jm
response = client.post(
"/api/admin/jobs/test-job-123/reset",
json={"target_state": "awaiting_review"},
)
assert response.status_code == 200
data = response.json()
# encoding_progress should be in cleared_data
assert "encoding_progress" in data.get("cleared_data", [])
class TestResetWorkerTriggerResponseFields:
"""Tests that reset response includes worker_triggered and worker_trigger_error fields."""
def test_reset_to_instrumental_selected_includes_worker_triggered_true(self, client, mock_job):
"""Test response includes worker_triggered=True when trigger succeeds."""
from unittest.mock import AsyncMock
with patch('backend.api.routes.admin.JobManager') as mock_jm_class, \
patch('backend.services.worker_service.get_worker_service') as mock_ws:
mock_jm = Mock()
mock_jm.get_job.return_value = mock_job
mock_jm.update_job.return_value = True
mock_jm_class.return_value = mock_jm
mock_worker_service = Mock()
mock_worker_service.trigger_video_worker = AsyncMock(return_value=True)
mock_ws.return_value = mock_worker_service
response = client.post(
"/api/admin/jobs/test-job-123/reset",
json={"target_state": "instrumental_selected"},
)
assert response.status_code == 200
data = response.json()
assert data["worker_triggered"] is True
assert data.get("worker_trigger_error") is None
def test_reset_to_instrumental_selected_includes_worker_triggered_false_on_failure(self, client, mock_job):
"""Test response includes worker_triggered=False and error when trigger fails."""
from unittest.mock import AsyncMock
with patch('backend.api.routes.admin.JobManager') as mock_jm_class, \
patch('backend.services.worker_service.get_worker_service') as mock_ws:
mock_jm = Mock()
mock_jm.get_job.return_value = mock_job
mock_jm.update_job.return_value = True
mock_jm_class.return_value = mock_jm
mock_worker_service = Mock()
mock_worker_service.trigger_video_worker = AsyncMock(return_value=False)
mock_ws.return_value = mock_worker_service
response = client.post(
"/api/admin/jobs/test-job-123/reset",
json={"target_state": "instrumental_selected"},
)
assert response.status_code == 200
data = response.json()
assert data["worker_triggered"] is False
assert data["worker_trigger_error"] is not None
assert "manual trigger" in data["message"].lower()
def test_reset_to_instrumental_selected_includes_error_on_exception(self, client, mock_job):
"""Test response includes worker_trigger_error when exception occurs."""
from unittest.mock import AsyncMock
with patch('backend.api.routes.admin.JobManager') as mock_jm_class, \
patch('backend.services.worker_service.get_worker_service') as mock_ws:
mock_jm = Mock()
mock_jm.get_job.return_value = mock_job
mock_jm.update_job.return_value = True
mock_jm_class.return_value = mock_jm
mock_worker_service = Mock()
mock_worker_service.trigger_video_worker = AsyncMock(
side_effect=Exception("Connection timeout")
)
mock_ws.return_value = mock_worker_service
response = client.post(
"/api/admin/jobs/test-job-123/reset",
json={"target_state": "instrumental_selected"},
)
assert response.status_code == 200
data = response.json()
assert data["worker_triggered"] is False
assert "Connection timeout" in data["worker_trigger_error"]
def test_reset_to_pending_has_null_worker_fields(self, client, mock_job):
"""Test response has null worker fields for non-instrumental_selected resets."""
with patch('backend.api.routes.admin.JobManager') as mock_jm_class:
mock_jm = Mock()
mock_jm.get_job.return_value = mock_job
mock_jm.update_job.return_value = True
mock_jm_class.return_value = mock_jm
response = client.post(
"/api/admin/jobs/test-job-123/reset",
json={"target_state": "pending"},
)
assert response.status_code == 200
data = response.json()
assert data.get("worker_triggered") is None
assert data.get("worker_trigger_error") is None
class TestTriggerWorkerEndpoint:
"""Tests for POST /api/admin/jobs/{job_id}/trigger-worker endpoint."""
def test_trigger_video_worker_success(self, client, mock_job):
"""Test successfully triggering video worker."""
from unittest.mock import AsyncMock
mock_job.status = "instrumental_selected"
with patch('backend.api.routes.admin.JobManager') as mock_jm_class, \
patch('backend.services.worker_service.get_worker_service') as mock_ws:
mock_jm = Mock()
mock_jm.get_job.return_value = mock_job
mock_jm_class.return_value = mock_jm
mock_worker_service = Mock()
mock_worker_service.trigger_video_worker = AsyncMock(return_value=True)
mock_ws.return_value = mock_worker_service
response = client.post(
"/api/admin/jobs/test-job-123/trigger-worker",
json={"worker_type": "video"},
)
assert response.status_code == 200
data = response.json()
assert data["status"] == "success"
assert data["job_id"] == "test-job-123"
assert data["worker_type"] == "video"
assert data["triggered"] is True
assert data.get("error") is None
mock_worker_service.trigger_video_worker.assert_called_once_with("test-job-123")
def test_trigger_worker_default_type_is_video(self, client, mock_job):
"""Test that default worker_type is video."""
from unittest.mock import AsyncMock
mock_job.status = "instrumental_selected"
with patch('backend.api.routes.admin.JobManager') as mock_jm_class, \
patch('backend.services.worker_service.get_worker_service') as mock_ws:
mock_jm = Mock()
mock_jm.get_job.return_value = mock_job
mock_jm_class.return_value = mock_jm
mock_worker_service = Mock()
mock_worker_service.trigger_video_worker = AsyncMock(return_value=True)
mock_ws.return_value = mock_worker_service
# Send empty body - should default to video
response = client.post(
"/api/admin/jobs/test-job-123/trigger-worker",
json={},
)
assert response.status_code == 200
data = response.json()
assert data["worker_type"] == "video"
def test_trigger_worker_failure_returns_error(self, client, mock_job):
"""Test response when worker trigger returns False."""
from unittest.mock import AsyncMock
mock_job.status = "instrumental_selected"
with patch('backend.api.routes.admin.JobManager') as mock_jm_class, \
patch('backend.services.worker_service.get_worker_service') as mock_ws:
mock_jm = Mock()
mock_jm.get_job.return_value = mock_job
mock_jm_class.return_value = mock_jm
mock_worker_service = Mock()
mock_worker_service.trigger_video_worker = AsyncMock(return_value=False)
mock_ws.return_value = mock_worker_service
response = client.post(
"/api/admin/jobs/test-job-123/trigger-worker",
json={"worker_type": "video"},
)
assert response.status_code == 200
data = response.json()
assert data["status"] == "error"
assert data["triggered"] is False
assert data["error"] is not None
def test_trigger_worker_exception_returns_error(self, client, mock_job):
"""Test response when worker trigger throws exception."""
from unittest.mock import AsyncMock
mock_job.status = "instrumental_selected"
with patch('backend.api.routes.admin.JobManager') as mock_jm_class, \
patch('backend.services.worker_service.get_worker_service') as mock_ws:
mock_jm = Mock()
mock_jm.get_job.return_value = mock_job
mock_jm_class.return_value = mock_jm
mock_worker_service = Mock()
mock_worker_service.trigger_video_worker = AsyncMock(
side_effect=Exception("Network error")
)
mock_ws.return_value = mock_worker_service
response = client.post(
"/api/admin/jobs/test-job-123/trigger-worker",
json={"worker_type": "video"},
)
assert response.status_code == 200
data = response.json()
assert data["status"] == "error"
assert data["triggered"] is False
assert "Network error" in data["error"]
def test_trigger_worker_invalid_type_rejected(self, client, mock_job):
"""Test that invalid worker_type is rejected."""
with patch('backend.api.routes.admin.JobManager') as mock_jm_class:
mock_jm = Mock()
mock_jm.get_job.return_value = mock_job
mock_jm_class.return_value = mock_jm
response = client.post(
"/api/admin/jobs/test-job-123/trigger-worker",
json={"worker_type": "invalid"},
)
assert response.status_code == 400
assert "unsupported" in response.json()["detail"].lower()
def test_trigger_worker_job_not_found(self, client):
"""Test 404 when job doesn't exist."""
with patch('backend.api.routes.admin.JobManager') as mock_jm_class:
mock_jm = Mock()
mock_jm.get_job.return_value = None
mock_jm_class.return_value = mock_jm
response = client.post(
"/api/admin/jobs/nonexistent/trigger-worker",
json={"worker_type": "video"},
)
assert response.status_code == 404
def test_trigger_worker_logs_warning_for_unexpected_status(self, client, mock_job):
"""Test that warning is logged when job is not in expected status."""
from unittest.mock import AsyncMock
mock_job.status = "complete" # Not instrumental_selected
with patch('backend.api.routes.admin.JobManager') as mock_jm_class, \
patch('backend.services.worker_service.get_worker_service') as mock_ws, \
patch('backend.api.routes.admin.logger') as mock_logger:
mock_jm = Mock()
mock_jm.get_job.return_value = mock_job
mock_jm_class.return_value = mock_jm
mock_worker_service = Mock()
mock_worker_service.trigger_video_worker = AsyncMock(return_value=True)
mock_ws.return_value = mock_worker_service
response = client.post(
"/api/admin/jobs/test-job-123/trigger-worker",
json={"worker_type": "video"},
)
# Should still succeed - we don't block based on status
assert response.status_code == 200
# But should log a warning
mock_logger.warning.assert_called()
warning_msg = mock_logger.warning.call_args[0][0]
assert "unexpected status" in warning_msg.lower()
def test_trigger_worker_requires_admin(self, client, mock_job):
"""Test that endpoint requires admin access."""
original_override = app.dependency_overrides.get(require_admin)
def get_non_admin():
from fastapi import HTTPException
raise HTTPException(status_code=403, detail="Admin access required")
app.dependency_overrides[require_admin] = get_non_admin
try:
response = client.post(
"/api/admin/jobs/test-job-123/trigger-worker",
json={"worker_type": "video"},
)
assert response.status_code == 403
finally:
if original_override:
app.dependency_overrides[require_admin] = original_override
else:
app.dependency_overrides[require_admin] = get_mock_admin
-86
View File
@@ -44,92 +44,6 @@ class TestHealthRoutes:
response = client.get("/", )
assert response.status_code == 200
@patch('backend.api.routes.health.check_flacfetch_service_status')
def test_flacfetch_health_not_configured(self, mock_check, client):
"""Test /api/health/flacfetch when service is not configured."""
mock_check.return_value = {"configured": False}
response = client.get("/api/health/flacfetch")
assert response.status_code == 200
data = response.json()
assert data["available"] is False
assert data["status"] == "not_configured"
@patch('backend.api.routes.health.check_flacfetch_service_status')
def test_flacfetch_health_offline(self, mock_check, client):
"""Test /api/health/flacfetch when service is offline."""
mock_check.return_value = {
"configured": True,
"available": False,
"error": "Connection timeout"
}
response = client.get("/api/health/flacfetch")
assert response.status_code == 200
data = response.json()
assert data["available"] is False
assert data["status"] == "offline"
assert data["error"] == "Connection timeout"
@patch('backend.api.routes.health.check_flacfetch_service_status')
def test_flacfetch_health_ok(self, mock_check, client):
"""Test /api/health/flacfetch when service is healthy."""
mock_check.return_value = {
"configured": True,
"available": True,
"status": "healthy",
"version": "1.2.3"
}
response = client.get("/api/health/flacfetch")
assert response.status_code == 200
data = response.json()
assert data["available"] is True
assert data["status"] == "ok"
assert data["version"] == "1.2.3"
@patch('backend.api.routes.health.check_encoding_worker_status')
def test_encoding_worker_health_not_configured(self, mock_check, client):
"""Test /api/health/encoding-worker when not configured."""
mock_check.return_value = {"configured": False}
response = client.get("/api/health/encoding-worker")
assert response.status_code == 200
data = response.json()
assert data["available"] is False
assert data["status"] == "not_configured"
@patch('backend.api.routes.health.check_encoding_worker_status')
def test_encoding_worker_health_offline(self, mock_check, client):
"""Test /api/health/encoding-worker when offline."""
mock_check.return_value = {
"configured": True,
"available": False,
"error": "Worker unreachable"
}
response = client.get("/api/health/encoding-worker")
assert response.status_code == 200
data = response.json()
assert data["available"] is False
assert data["status"] == "offline"
assert data["error"] == "Worker unreachable"
@patch('backend.api.routes.health.check_encoding_worker_status')
def test_encoding_worker_health_ok(self, mock_check, client):
"""Test /api/health/encoding-worker when healthy."""
mock_check.return_value = {
"configured": True,
"available": True,
"status": "ok",
"wheel_version": "0.123.0",
"active_jobs": 2,
"queue_length": 5
}
response = client.get("/api/health/encoding-worker")
assert response.status_code == 200
data = response.json()
assert data["available"] is True
assert data["status"] == "ok"
assert data["version"] == "0.123.0"
assert data["active_jobs"] == 2
assert data["queue_length"] == 5
class TestJobRoutes:
"""Tests for jobs.py routes.
+80 -128
View File
@@ -5,7 +5,7 @@ Tests the audio search service, API routes, and job model fields.
"""
import pytest
import os
from unittest.mock import Mock, patch, MagicMock, AsyncMock
from unittest.mock import Mock, patch, MagicMock
from datetime import datetime
from backend.models.job import Job, JobCreate, JobStatus, STATE_TRANSITIONS
@@ -1219,48 +1219,44 @@ class TestAudioSearchApiRouteDownload:
assert 'gcs_path' in call_kwargs
assert call_kwargs['gcs_path'] == "uploads/job123/audio/"
def test_youtube_remote_download_uses_youtube_service(self, mock_job_manager, mock_storage_service):
"""
Test that YouTube downloads use YouTubeDownloadService.
After the consolidation refactor, all YouTube downloads go through
YouTubeDownloadService instead of audio_search_service.download_by_id().
"""
def test_local_youtube_download_does_not_pass_gcs_path(self, mock_job_manager, mock_storage_service):
"""Test that YouTube downloads don't use gcs_path (download locally, upload manually)."""
from backend.api.routes.audio_search import _download_and_start_processing
import asyncio
# Setup mock job with YouTube search result that has source_id
import tempfile
# Setup mock job with YouTube search result
mock_job = Mock()
mock_job.state_data = {
'audio_search_results': [{
'title': 'Demons',
'artist': 'Kenny Chesney',
'provider': 'YouTube',
'title': 'Unwanted',
'artist': 'Avril Lavigne',
'provider': 'YouTube', # NOT a torrent source
'quality': 'Opus 128kbps',
'source_id': 'qmTjyGH9ro4', # YouTube video ID (11 chars)
'url': 'https://www.youtube.com/watch?v=qmTjyGH9ro4',
}]
}
mock_job.audio_search_artist = 'Kenny Chesney'
mock_job.audio_search_title = 'Demons'
mock_job.audio_search_artist = 'Avril Lavigne'
mock_job.audio_search_title = 'Unwanted'
mock_job_manager.get_job.return_value = mock_job
# Create mock audio search service (not used for YouTube anymore)
# Create mock audio search service with remote client (but YouTube doesn't use it)
mock_audio_service = Mock()
mock_audio_service.is_remote_enabled.return_value = True
# Create mock YouTubeDownloadService
mock_youtube_service = Mock()
mock_youtube_service.download_by_id = AsyncMock(
return_value="uploads/job456/audio/Kenny Chesney - Demons.webm"
)
mock_youtube_service._extract_video_id = Mock(return_value="qmTjyGH9ro4")
# Create mock background tasks
mock_bg_tasks = Mock()
# Patch get_youtube_download_service to return our mock
with patch('backend.api.routes.audio_search.get_youtube_download_service', return_value=mock_youtube_service):
# Create a temp file to simulate downloaded file
temp_file = tempfile.NamedTemporaryFile(suffix='.opus', delete=False)
temp_file.write(b'fake audio data')
temp_file.close()
try:
# Mock download to return local path
mock_download_result = Mock()
mock_download_result.filepath = temp_file.name
mock_audio_service.download.return_value = mock_download_result
# Create mock background tasks
mock_bg_tasks = Mock()
# Run the async function
loop = asyncio.new_event_loop()
try:
@@ -1274,76 +1270,22 @@ class TestAudioSearchApiRouteDownload:
)
finally:
loop.close()
# CRITICAL: Verify YouTubeDownloadService.download_by_id was called
mock_youtube_service.download_by_id.assert_called_once()
call_kwargs = mock_youtube_service.download_by_id.call_args.kwargs
# Verify correct arguments
assert call_kwargs['video_id'] == 'qmTjyGH9ro4'
assert call_kwargs['job_id'] == 'job456'
assert call_kwargs['artist'] == 'Kenny Chesney'
assert call_kwargs['title'] == 'Demons'
# audio_search_service.download_by_id should NOT be called for YouTube
mock_audio_service.download_by_id.assert_not_called()
def test_youtube_download_always_uses_youtube_service(self, mock_job_manager, mock_storage_service):
"""
Test that YouTube downloads always use YouTubeDownloadService,
regardless of remote enabled status. The service handles fallback internally.
"""
from backend.api.routes.audio_search import _download_and_start_processing
import asyncio
# Setup mock job with YouTube search result (use valid 11-char video ID)
mock_job = Mock()
mock_job.state_data = {
'audio_search_results': [{
'title': 'Unwanted',
'artist': 'Avril Lavigne',
'provider': 'YouTube',
'quality': 'Opus 128kbps',
'source_id': 'abc123xyz45', # Valid 11-char ID
'url': 'https://www.youtube.com/watch?v=abc123xyz45',
}]
}
mock_job.audio_search_artist = 'Avril Lavigne'
mock_job.audio_search_title = 'Unwanted'
mock_job_manager.get_job.return_value = mock_job
# Create mock audio search service (not used for YouTube)
mock_audio_service = Mock()
mock_audio_service.is_remote_enabled.return_value = False
# Create mock YouTubeDownloadService
mock_youtube_service = Mock()
mock_youtube_service.download_by_id = AsyncMock(
return_value="uploads/job789/audio/Avril Lavigne - Unwanted.opus"
)
mock_youtube_service._extract_video_id = Mock(return_value="abc123xyz45")
# Create mock background tasks
mock_bg_tasks = Mock()
# Patch get_youtube_download_service to return our mock
with patch('backend.api.routes.audio_search.get_youtube_download_service', return_value=mock_youtube_service):
# Run the async function
loop = asyncio.new_event_loop()
# YouTube should NOT have gcs_path in kwargs
mock_audio_service.download.assert_called_once()
call_kwargs = mock_audio_service.download.call_args.kwargs
# For YouTube, gcs_path should NOT be set (or should be None)
assert call_kwargs.get('gcs_path') is None
# Instead, storage service should be called to upload
mock_storage_service.upload_fileobj.assert_called_once()
finally:
import os
try:
result = loop.run_until_complete(
_download_and_start_processing(
job_id="job789",
selection_index=0,
audio_search_service=mock_audio_service,
background_tasks=mock_bg_tasks,
)
)
finally:
loop.close()
# YouTubeDownloadService.download_by_id should always be called for YouTube
mock_youtube_service.download_by_id.assert_called_once()
os.unlink(temp_file.name)
except:
pass
def test_handles_gcs_path_response_correctly(self, mock_job_manager, mock_storage_service):
"""Test that GCS path responses are parsed correctly."""
@@ -1400,43 +1342,41 @@ class TestAudioSearchApiRouteDownload:
assert gcs_path_set, "input_media_gcs_path was not set in job update"
def test_torrent_download_requires_remote(self, mock_job_manager, mock_storage_service):
"""
Test that torrent sources (RED/OPS) require remote flacfetch.
Torrent downloads can only happen on the flacfetch VM which has:
- BitTorrent peer connectivity
- Private tracker credentials
- Long-running seeding capability
Cloud Run cannot download torrents, so remote must be enabled.
"""
def test_remote_disabled_always_uses_local(self, mock_job_manager, mock_storage_service):
"""Test that when remote is disabled, even torrent sources use local download."""
from backend.api.routes.audio_search import _download_and_start_processing
from backend.services.audio_search_service import DownloadError
import asyncio
import pytest
import tempfile
# Setup mock job with RED search result
mock_job = Mock()
mock_job.state_data = {
'audio_search_results': [{
'title': 'Test',
'artist': 'Test',
'provider': 'RED', # Torrent source requires remote
'provider': 'RED', # Torrent source, but remote is disabled
'quality': 'FLAC',
}]
}
mock_job_manager.get_job.return_value = mock_job
# Create mock audio search service WITHOUT remote client
mock_audio_service = Mock()
mock_audio_service.is_remote_enabled.return_value = False # REMOTE DISABLED
loop = asyncio.new_event_loop()
# Create temp file
temp_file = tempfile.NamedTemporaryFile(suffix='.flac', delete=False)
temp_file.write(b'fake')
temp_file.close()
try:
# Should raise an error because RED requires remote
with pytest.raises(Exception) as exc_info:
loop.run_until_complete(
# Create mock audio search service WITHOUT remote client
mock_audio_service = Mock()
mock_audio_service.is_remote_enabled.return_value = False # REMOTE DISABLED
mock_download_result = Mock()
mock_download_result.filepath = temp_file.name
mock_audio_service.download.return_value = mock_download_result
loop = asyncio.new_event_loop()
try:
result = loop.run_until_complete(
_download_and_start_processing(
job_id="job_no_remote",
selection_index=0,
@@ -1444,7 +1384,19 @@ class TestAudioSearchApiRouteDownload:
background_tasks=Mock(),
)
)
# Verify the error message mentions remote flacfetch
assert "remote flacfetch" in str(exc_info.value).lower() or "flacfetch" in str(exc_info.value).lower()
finally:
loop.close()
# When remote is disabled, gcs_path should NOT be passed
mock_audio_service.download.assert_called_once()
call_kwargs = mock_audio_service.download.call_args.kwargs
assert call_kwargs.get('gcs_path') is None
# And storage should upload manually
mock_storage_service.upload_fileobj.assert_called_once()
finally:
loop.close()
import os
try:
os.unlink(temp_file.name)
except:
pass
-321
View File
@@ -1,321 +0,0 @@
"""
Integration tests for the combined review flow.
These tests verify that corrections made during human review survive the full flow:
1. User edits lyrics in LyricsAnalyzer
2. Corrections saved via POST /api/jobs/{job_id}/corrections
3. User navigates to InstrumentalSelector
4. InstrumentalSelector fetches via GET /api/review/{job_id}/correction-data
5. Corrections are included in final submission
6. render_video_worker uses the corrected data
The key invariant: If corrections_updated.json exists, the API must return its
contents, not the original corrections.json.
"""
import pytest
import json
from unittest.mock import Mock, patch, MagicMock
from datetime import datetime
from backend.models.job import Job, JobStatus
class TestGetCorrectionDataReturnsUpdatedCorrections:
"""Test that get_correction_data returns updated corrections when available.
This is the critical fix for the bug where user's lyrics corrections were
being ignored because InstrumentalSelector was fetching original corrections.
"""
@pytest.fixture
def mock_job(self):
"""Create a mock job in AWAITING_REVIEW state."""
job = Mock(spec=Job)
job.job_id = "test-job-123"
job.status = JobStatus.AWAITING_REVIEW
job.artist = "Test Artist"
job.title = "Test Title"
job.input_media_gcs_path = "jobs/test-job-123/input/audio.flac"
job.file_urls = {
"lyrics": {
"corrections": "jobs/test-job-123/lyrics/corrections.json",
},
"stems": {},
"analysis": {},
}
job.state_data = {}
job.style_params_gcs_path = None
job.style_assets = {}
return job
@pytest.fixture
def original_corrections(self):
"""Original corrections from lyrics processing (before user edits)."""
return {
"corrected_segments": [
{"id": 1, "text": "Hello world"}, # Original: "Hello world"
],
"corrections": [],
"metadata": {"source": "original"},
}
@pytest.fixture
def updated_corrections(self):
"""Updated corrections after user edits."""
return {
"corrected_segments": [
{"id": 1, "text": "Hello beautiful world"}, # User changed this!
],
"corrections": [
{"id": 1, "type": "edit", "original": "Hello world", "corrected": "Hello beautiful world"}
],
"metadata": {"source": "updated"},
}
def test_returns_updated_corrections_when_file_exists(
self, mock_job, original_corrections, updated_corrections
):
"""Verify endpoint returns updated corrections when corrections_updated.json exists.
This is the main test for the bug fix.
"""
from backend.api.routes.review import get_correction_data
# Add corrections_updated to file_urls
mock_job.file_urls["lyrics"]["corrections_updated"] = "jobs/test-job-123/lyrics/corrections_updated.json"
with patch("backend.api.routes.review.JobManager") as MockJobManager, \
patch("backend.api.routes.review.StorageService") as MockStorageService:
# Setup mocks
job_manager = MockJobManager.return_value
job_manager.get_job.return_value = mock_job
storage = MockStorageService.return_value
# Return True for corrections_updated.json
storage.file_exists.return_value = True
# Return updated corrections (not original)
storage.download_json.return_value = updated_corrections.copy()
storage.generate_signed_url.return_value = "https://signed-url.com"
# Call the endpoint (synchronously for testing)
import asyncio
result = asyncio.get_event_loop().run_until_complete(
get_correction_data("test-job-123", ("user@test.com", "job_owner"))
)
# Verify it downloaded from corrections_updated, not corrections
storage.download_json.assert_called_once()
downloaded_path = storage.download_json.call_args[0][0]
assert "corrections_updated" in downloaded_path, \
f"Expected to download corrections_updated.json, got: {downloaded_path}"
# Verify the returned data has the user's edits
assert result["corrected_segments"][0]["text"] == "Hello beautiful world"
assert result["metadata"]["source"] == "updated"
def test_falls_back_to_original_when_no_updates(
self, mock_job, original_corrections
):
"""Verify endpoint falls back to original corrections when no updates exist."""
from backend.api.routes.review import get_correction_data
with patch("backend.api.routes.review.JobManager") as MockJobManager, \
patch("backend.api.routes.review.StorageService") as MockStorageService:
# Setup mocks
job_manager = MockJobManager.return_value
job_manager.get_job.return_value = mock_job
storage = MockStorageService.return_value
# Return False for corrections_updated.json (doesn't exist)
storage.file_exists.side_effect = lambda path: "corrections_updated" not in path
# Return original corrections
storage.download_json.return_value = original_corrections.copy()
storage.generate_signed_url.return_value = "https://signed-url.com"
# Call the endpoint
import asyncio
result = asyncio.get_event_loop().run_until_complete(
get_correction_data("test-job-123", ("user@test.com", "job_owner"))
)
# Verify it downloaded from corrections.json (original)
storage.download_json.assert_called_once()
downloaded_path = storage.download_json.call_args[0][0]
assert "corrections.json" in downloaded_path
# Verify the returned data is the original
assert result["corrected_segments"][0]["text"] == "Hello world"
def test_checks_direct_gcs_path_when_not_in_file_urls(
self, mock_job, updated_corrections
):
"""Verify endpoint checks direct GCS path for corrections_updated.json."""
from backend.api.routes.review import get_correction_data
# Remove corrections_updated from file_urls (simulates older job)
# But the file exists in GCS from a previous save
with patch("backend.api.routes.review.JobManager") as MockJobManager, \
patch("backend.api.routes.review.StorageService") as MockStorageService:
# Setup mocks
job_manager = MockJobManager.return_value
job_manager.get_job.return_value = mock_job
storage = MockStorageService.return_value
# Return True for direct path check of corrections_updated.json
storage.file_exists.return_value = True
storage.download_json.return_value = updated_corrections.copy()
storage.generate_signed_url.return_value = "https://signed-url.com"
# Call the endpoint
import asyncio
result = asyncio.get_event_loop().run_until_complete(
get_correction_data("test-job-123", ("user@test.com", "job_owner"))
)
# Verify it checked for corrections_updated.json via direct path
file_exists_calls = storage.file_exists.call_args_list
checked_paths = [call[0][0] for call in file_exists_calls]
assert any("corrections_updated" in path for path in checked_paths), \
f"Expected to check for corrections_updated.json, checked: {checked_paths}"
def test_falls_back_when_file_urls_has_stale_corrections_updated(
self, mock_job, original_corrections
):
"""Verify endpoint falls back to original when corrections_updated in file_urls but file deleted.
This handles the edge case where:
1. User saved corrections (corrections_updated added to file_urls)
2. Admin reset job or file was deleted
3. corrections_updated.json no longer exists in GCS
4. Endpoint should fall back to original corrections.json
"""
from backend.api.routes.review import get_correction_data
# Add corrections_updated to file_urls (stale reference)
mock_job.file_urls["lyrics"]["corrections_updated"] = "jobs/test-job-123/lyrics/corrections_updated.json"
with patch("backend.api.routes.review.JobManager") as MockJobManager, \
patch("backend.api.routes.review.StorageService") as MockStorageService:
job_manager = MockJobManager.return_value
job_manager.get_job.return_value = mock_job
storage = MockStorageService.return_value
# corrections_updated.json does NOT exist (stale reference)
# but corrections.json does
def file_exists_check(path):
return "corrections_updated" not in path
storage.file_exists.side_effect = file_exists_check
storage.download_json.return_value = original_corrections.copy()
storage.generate_signed_url.return_value = "https://signed-url.com"
import asyncio
result = asyncio.get_event_loop().run_until_complete(
get_correction_data("test-job-123", ("user@test.com", "job_owner"))
)
# Should fall back to original corrections
downloaded_path = storage.download_json.call_args[0][0]
assert "corrections.json" in downloaded_path or "corrections_updated" not in downloaded_path
# Verify original data was returned
assert result["corrected_segments"][0]["text"] == "Hello world"
class TestCorrectionDataPreservation:
"""Test that correction data is preserved through the full flow."""
def test_post_corrections_saves_to_corrections_updated(self):
"""Verify POST /api/jobs/{job_id}/corrections saves to corrections_updated.json."""
# This documents the expected behavior of the jobs.py endpoint
# The actual implementation saves to f"jobs/{job_id}/lyrics/corrections_updated.json"
job_id = "test-job-123"
expected_gcs_path = f"jobs/{job_id}/lyrics/corrections_updated.json"
assert "corrections_updated" in expected_gcs_path
assert job_id in expected_gcs_path
def test_complete_review_submits_correction_data(self):
"""Verify POST /api/review/{job_id}/complete receives and saves corrections."""
# This documents what data complete_review expects
expected_payload = {
"corrections": [{"id": 1, "type": "edit"}],
"corrected_segments": [{"id": 1, "text": "corrected"}],
"instrumental_selection": "with_backing", # Required
}
# instrumental_selection is required
assert "instrumental_selection" in expected_payload
# corrections data should be included
assert "corrections" in expected_payload
assert "corrected_segments" in expected_payload
class TestRenderVideoWorkerUsesCorrectedData:
"""Test that render_video_worker uses the corrected data.
This verifies that the fix is consistent with render_video_worker's behavior.
"""
def test_render_worker_checks_corrections_updated_first(self):
"""Document that render_video_worker checks corrections_updated first.
From render_video_worker.py lines 128-142, the worker:
1. Checks file_urls.lyrics.corrections_updated
2. Falls back to file_urls.lyrics.corrections
3. Falls back to direct GCS paths
This is the same pattern the API endpoint should use.
"""
# Document the expected check order
check_order = [
"file_urls.lyrics.corrections_updated",
"file_urls.lyrics.corrections",
"jobs/{job_id}/lyrics/corrections_updated.json",
"jobs/{job_id}/lyrics/corrections.json",
]
# corrections_updated must be checked before corrections
updated_index = next(i for i, p in enumerate(check_order) if "corrections_updated" in p)
original_index = next(i for i, p in enumerate(check_order) if "corrections.json" in p or
(p == "file_urls.lyrics.corrections"))
# Only check the file_urls entries (indices 0 and 1)
assert updated_index < 2, "corrections_updated should be in top 2 checks"
def test_render_worker_merges_corrections(self):
"""Document that render_video_worker merges corrections.
From render_video_worker.py lines 159-175:
- Downloads original corrections.json
- If corrections_updated.json exists, downloads and merges
- Merged data has user's corrections with original metadata
"""
original_data = {
"original_segments": [{"id": 1}],
"corrected_segments": [{"id": 1, "text": "original"}],
"corrections": [],
"metadata": {"preserved": True},
}
updated_data = {
"corrected_segments": [{"id": 1, "text": "user-edited"}],
"corrections": [{"id": 1, "type": "edit"}],
}
# Apply merge logic (same as render_video_worker.py lines 169-172)
if 'corrections' in updated_data:
original_data['corrections'] = updated_data['corrections']
if 'corrected_segments' in updated_data:
original_data['corrected_segments'] = updated_data['corrected_segments']
# Verify merge result
assert original_data['corrected_segments'][0]['text'] == "user-edited"
assert original_data['corrections'][0]['type'] == "edit"
# Metadata and original_segments preserved
assert original_data['metadata']['preserved'] is True
assert original_data['original_segments'][0]['id'] == 1
@@ -1,409 +0,0 @@
"""
Contract tests for countdown padding state synchronization.
These tests verify the data contract between:
- render_video_worker (WRITER) - must set has_countdown_padding after adding countdown
- video_worker (READER) - must read and use has_countdown_padding
The bug this test suite catches:
- render_video_worker adds countdown (deferred from lyrics phase) but doesn't update state
- video_worker reads lyrics_metadata.has_countdown_padding which is still False
- Result: instrumental is not padded, causing audio desync
These tests will FAIL if either side of the contract is broken.
"""
import pytest
from datetime import datetime, UTC
from unittest.mock import MagicMock, AsyncMock, patch
import tempfile
import os
from backend.models.job import Job, JobStatus
class TestCountdownStateSchema:
"""Tests that the lyrics_metadata schema supports countdown padding fields."""
def test_lyrics_metadata_can_store_has_countdown_padding(self):
"""lyrics_metadata must support has_countdown_padding (bool) field."""
job = Job(
job_id="test123",
status=JobStatus.RENDERING_VIDEO,
created_at=datetime.now(UTC),
updated_at=datetime.now(UTC),
artist="Test",
title="Test",
state_data={
'lyrics_metadata': {
'has_countdown_padding': True
}
}
)
lyrics_metadata = job.state_data.get('lyrics_metadata', {})
assert lyrics_metadata.get('has_countdown_padding') is True
def test_lyrics_metadata_can_store_countdown_padding_seconds(self):
"""lyrics_metadata must support countdown_padding_seconds (float) field."""
job = Job(
job_id="test123",
status=JobStatus.RENDERING_VIDEO,
created_at=datetime.now(UTC),
updated_at=datetime.now(UTC),
artist="Test",
title="Test",
state_data={
'lyrics_metadata': {
'countdown_padding_seconds': 3.0
}
}
)
lyrics_metadata = job.state_data.get('lyrics_metadata', {})
assert lyrics_metadata.get('countdown_padding_seconds') == 3.0
def test_lyrics_metadata_defaults_to_no_countdown(self):
"""Default state should have has_countdown_padding=False."""
job = Job(
job_id="test123",
status=JobStatus.RENDERING_VIDEO,
created_at=datetime.now(UTC),
updated_at=datetime.now(UTC),
artist="Test",
title="Test",
)
lyrics_metadata = job.state_data.get('lyrics_metadata', {})
assert lyrics_metadata.get('has_countdown_padding', False) is False
class TestRenderVideoWorkerWriterContract:
"""
WRITER CONTRACT: render_video_worker must update job state when countdown is added.
When CountdownProcessor.process() returns padding_added=True:
- render_video_worker MUST call job_manager.update_state_data to set
lyrics_metadata.has_countdown_padding = True
lyrics_metadata.countdown_padding_seconds = <padding_seconds>
"""
def test_render_video_updates_state_when_countdown_added(self):
"""
WRITER CONTRACT TEST: When countdown is added, state MUST be updated.
This test verifies the state update logic that should execute in render_video_worker
after CountdownProcessor adds padding. We test the logic directly to avoid the
complexity of mocking the full worker function.
The actual code in render_video_worker.py should look like:
if padding_added:
existing_lyrics_metadata = job.state_data.get('lyrics_metadata', {})
existing_lyrics_metadata['has_countdown_padding'] = True
existing_lyrics_metadata['countdown_padding_seconds'] = padding_seconds
job_manager.update_state_data(job_id, 'lyrics_metadata', existing_lyrics_metadata)
"""
# Simulate the inputs from CountdownProcessor
padding_added = True
padding_seconds = 3.0
# Simulate existing job state
existing_state_data = {
'lyrics_metadata': {
'has_countdown_padding': False,
'countdown_padding_seconds': 0.0
}
}
# This is the exact logic that should be in render_video_worker.py
if padding_added:
existing_lyrics_metadata = existing_state_data.get('lyrics_metadata', {})
existing_lyrics_metadata['has_countdown_padding'] = True
existing_lyrics_metadata['countdown_padding_seconds'] = padding_seconds
# Verify the state was updated correctly
assert existing_lyrics_metadata['has_countdown_padding'] is True, (
"WRITER CONTRACT: has_countdown_padding must be True when countdown added"
)
assert existing_lyrics_metadata['countdown_padding_seconds'] == 3.0, (
f"WRITER CONTRACT: countdown_padding_seconds must match padding_seconds"
)
def test_render_video_worker_source_contains_state_update(self):
"""
WRITER CONTRACT TEST: Verify the state update code exists in render_video_worker.py.
This is a source code inspection test that ensures the critical state update
code is present and hasn't been accidentally removed or refactored incorrectly.
"""
import inspect
from backend.workers import render_video_worker
# Get the source code of the module
source = inspect.getsource(render_video_worker)
# Verify the critical state update patterns exist
assert "existing_lyrics_metadata['has_countdown_padding'] = True" in source, (
"WRITER CONTRACT VIOLATION: render_video_worker.py must contain "
"code to set has_countdown_padding=True when countdown is added"
)
assert "existing_lyrics_metadata['countdown_padding_seconds']" in source, (
"WRITER CONTRACT VIOLATION: render_video_worker.py must contain "
"code to set countdown_padding_seconds when countdown is added"
)
assert "job_manager.update_state_data(job_id, 'lyrics_metadata'" in source, (
"WRITER CONTRACT VIOLATION: render_video_worker.py must call "
"job_manager.update_state_data to persist countdown state"
)
def test_render_video_no_state_update_when_countdown_not_needed(self):
"""
When countdown is NOT needed, state should NOT be updated to True.
This ensures we don't incorrectly set has_countdown_padding when
the song starts after 3 seconds and doesn't need countdown.
"""
# Simulate the inputs when countdown is NOT needed
padding_added = False
padding_seconds = 0.0
# Simulate existing job state
existing_state_data = {
'lyrics_metadata': {
'has_countdown_padding': False,
'countdown_padding_seconds': 0.0
}
}
# The logic should NOT update state when padding_added is False
if padding_added:
existing_lyrics_metadata = existing_state_data.get('lyrics_metadata', {})
existing_lyrics_metadata['has_countdown_padding'] = True
existing_lyrics_metadata['countdown_padding_seconds'] = padding_seconds
# Verify state was NOT changed
lyrics_metadata = existing_state_data.get('lyrics_metadata', {})
assert lyrics_metadata.get('has_countdown_padding') is False, (
"State should remain False when countdown not added"
)
assert lyrics_metadata.get('countdown_padding_seconds') == 0.0, (
"countdown_padding_seconds should remain 0 when countdown not added"
)
class TestVideoWorkerReaderContract:
"""
READER CONTRACT: video_worker must read and use countdown state for instrumental padding.
When lyrics_metadata.has_countdown_padding = True:
- video_worker MUST extract countdown_padding_seconds
- video_worker MUST pass countdown_padding_seconds to encoding backend
"""
@pytest.fixture
def job_with_countdown(self):
"""Create a job where countdown padding was added."""
return Job(
job_id="test123",
status=JobStatus.ENCODING,
created_at=datetime.now(UTC),
updated_at=datetime.now(UTC),
artist="Test",
title="Test",
file_urls={
'instrumental_clean': 'gs://bucket/instrumental.flac',
'video_output': 'gs://bucket/video.mp4',
},
state_data={
'lyrics_metadata': {
'has_countdown_padding': True,
'countdown_padding_seconds': 3.0,
}
}
)
@pytest.fixture
def job_without_countdown(self):
"""Create a job where countdown padding was NOT added."""
return Job(
job_id="test123",
status=JobStatus.ENCODING,
created_at=datetime.now(UTC),
updated_at=datetime.now(UTC),
artist="Test",
title="Test",
file_urls={
'instrumental_clean': 'gs://bucket/instrumental.flac',
'video_output': 'gs://bucket/video.mp4',
},
state_data={
'lyrics_metadata': {
'has_countdown_padding': False,
'countdown_padding_seconds': 0.0,
}
}
)
def test_video_worker_reads_countdown_padding_from_state(self, job_with_countdown):
"""
READER CONTRACT: video_worker must read has_countdown_padding from lyrics_metadata.
"""
# This mirrors the logic in video_worker.py
lyrics_metadata = job_with_countdown.state_data.get('lyrics_metadata', {})
countdown_padding_seconds = None
if lyrics_metadata.get('has_countdown_padding'):
countdown_padding_seconds = lyrics_metadata.get('countdown_padding_seconds', 3.0)
assert countdown_padding_seconds == 3.0
def test_video_worker_returns_none_when_no_countdown(self, job_without_countdown):
"""
READER CONTRACT: video_worker returns None when has_countdown_padding=False.
"""
lyrics_metadata = job_without_countdown.state_data.get('lyrics_metadata', {})
countdown_padding_seconds = None
if lyrics_metadata.get('has_countdown_padding'):
countdown_padding_seconds = lyrics_metadata.get('countdown_padding_seconds', 3.0)
assert countdown_padding_seconds is None
def test_video_worker_passes_countdown_to_karaoke_finalise(self, job_with_countdown):
"""
READER CONTRACT: video_worker must pass countdown_padding_seconds to KaraokeFinalise.
This verifies the standard encoding path uses countdown info by testing the
extraction logic that would be used to pass the value to KaraokeFinalise.
Note: This is a focused unit test that verifies the contract without running
the full complex worker function. The actual integration is tested in
emulator tests.
"""
# Simulate the exact code path from video_worker.py lines 541-544
lyrics_metadata = job_with_countdown.state_data.get('lyrics_metadata', {})
countdown_padding_seconds = None
if lyrics_metadata.get('has_countdown_padding'):
countdown_padding_seconds = lyrics_metadata.get('countdown_padding_seconds', 3.0)
# CRITICAL ASSERTION: The value that would be passed to KaraokeFinalise
assert countdown_padding_seconds is not None, (
"READER CONTRACT VIOLATION: countdown_padding_seconds should be extracted "
"when has_countdown_padding is True"
)
assert countdown_padding_seconds == 3.0, (
f"READER CONTRACT VIOLATION: Expected countdown_padding_seconds=3.0, "
f"got {countdown_padding_seconds}"
)
def test_karaoke_finalise_accepts_countdown_parameter(self):
"""
READER CONTRACT: KaraokeFinalise constructor must accept countdown_padding_seconds.
This verifies that the encoding layer can receive the countdown info.
"""
from karaoke_gen.karaoke_finalise.karaoke_finalise import KaraokeFinalise
import inspect
# Get the signature of KaraokeFinalise.__init__
sig = inspect.signature(KaraokeFinalise.__init__)
param_names = list(sig.parameters.keys())
assert 'countdown_padding_seconds' in param_names, (
"READER CONTRACT VIOLATION: KaraokeFinalise does not accept "
"countdown_padding_seconds parameter. This is required for audio sync."
)
class TestEndToEndStateFlow:
"""
Tests that verify the complete state flow from render_video_worker to video_worker.
"""
def test_state_written_by_render_can_be_read_by_video(self):
"""
State written by render_video_worker format can be read by video_worker logic.
This ensures both sides use the same state format.
"""
# Simulate state written by render_video_worker
state_from_render = {
'lyrics_metadata': {
'has_countdown_padding': True,
'countdown_padding_seconds': 3.0,
}
}
# Create job with this state
job = Job(
job_id="test123",
status=JobStatus.ENCODING,
created_at=datetime.now(UTC),
updated_at=datetime.now(UTC),
artist="Test",
title="Test",
state_data=state_from_render
)
# Simulate video_worker reading logic (from video_worker.py lines 539-545)
lyrics_metadata = job.state_data.get('lyrics_metadata', {})
countdown_padding_seconds = None
if lyrics_metadata.get('has_countdown_padding'):
countdown_padding_seconds = lyrics_metadata.get('countdown_padding_seconds', 3.0)
# Verify the state can be correctly read
assert countdown_padding_seconds == 3.0, (
"State format mismatch: render_video_worker writes in a format "
"that video_worker cannot correctly read"
)
def test_missing_lyrics_metadata_handled_gracefully(self):
"""
video_worker should handle missing lyrics_metadata gracefully.
"""
job = Job(
job_id="test123",
status=JobStatus.ENCODING,
created_at=datetime.now(UTC),
updated_at=datetime.now(UTC),
artist="Test",
title="Test",
state_data={} # No lyrics_metadata at all
)
# This should not raise an exception
lyrics_metadata = job.state_data.get('lyrics_metadata', {})
countdown_padding_seconds = None
if lyrics_metadata.get('has_countdown_padding'):
countdown_padding_seconds = lyrics_metadata.get('countdown_padding_seconds', 3.0)
assert countdown_padding_seconds is None
def test_countdown_padding_seconds_defaults_to_3_when_missing(self):
"""
If has_countdown_padding=True but countdown_padding_seconds is missing,
default to 3.0 seconds (the standard countdown duration).
"""
job = Job(
job_id="test123",
status=JobStatus.ENCODING,
created_at=datetime.now(UTC),
updated_at=datetime.now(UTC),
artist="Test",
title="Test",
state_data={
'lyrics_metadata': {
'has_countdown_padding': True,
# countdown_padding_seconds intentionally missing
}
}
)
lyrics_metadata = job.state_data.get('lyrics_metadata', {})
countdown_padding_seconds = None
if lyrics_metadata.get('has_countdown_padding'):
countdown_padding_seconds = lyrics_metadata.get('countdown_padding_seconds', 3.0)
assert countdown_padding_seconds == 3.0
+10 -58
View File
@@ -65,87 +65,39 @@ class TestDropboxService:
def test_get_next_brand_code_first_track(self):
"""Test brand code calculation when no existing tracks."""
from backend.services.dropbox_service import DropboxService
service = DropboxService()
with patch.object(service, 'list_folders', return_value=[]):
brand_code = service.get_next_brand_code("/test/path", "NOMAD")
assert brand_code == "NOMAD-0001"
def test_get_next_brand_code_legacy_gaps_preserved(self):
"""Test that gaps below 1000 are NOT filled (legacy behavior)."""
def test_get_next_brand_code_sequential(self):
"""Test brand code calculation with existing tracks."""
from backend.services.dropbox_service import DropboxService
service = DropboxService()
existing_folders = [
"NOMAD-0001 - Artist1 - Song1",
"NOMAD-0002 - Artist2 - Song2",
"NOMAD-0005 - Artist3 - Song3", # Gap at 0003, 0004 - should be preserved
"NOMAD-0005 - Artist3 - Song3", # Gap in sequence
"OTHER-0001 - Different Brand", # Different brand
]
with patch.object(service, 'list_folders', return_value=existing_folders):
brand_code = service.get_next_brand_code("/test/path", "NOMAD")
assert brand_code == "NOMAD-0006" # Does NOT fill legacy gap
def test_get_next_brand_code_fills_gap_above_1000(self):
"""Test that gaps at 1001+ ARE filled."""
from backend.services.dropbox_service import DropboxService
service = DropboxService()
existing_folders = [
"NOMAD-1001 - Artist1 - Song1",
"NOMAD-1002 - Artist2 - Song2",
"NOMAD-1005 - Artist3 - Song3", # Gap at 1003, 1004
]
with patch.object(service, 'list_folders', return_value=existing_folders):
brand_code = service.get_next_brand_code("/test/path", "NOMAD")
assert brand_code == "NOMAD-1003" # Fills gap at 1003
def test_get_next_brand_code_no_gaps_above_1000(self):
"""Test brand code calculation with no gaps above 1000 returns max+1."""
from backend.services.dropbox_service import DropboxService
service = DropboxService()
existing_folders = [
"NOMAD-1001 - Artist1 - Song1",
"NOMAD-1002 - Artist2 - Song2",
"NOMAD-1003 - Artist3 - Song3",
]
with patch.object(service, 'list_folders', return_value=existing_folders):
brand_code = service.get_next_brand_code("/test/path", "NOMAD")
assert brand_code == "NOMAD-1004"
def test_get_next_brand_code_mixed_legacy_and_new(self):
"""Test with both legacy (<1000) and new (>=1000) codes."""
from backend.services.dropbox_service import DropboxService
service = DropboxService()
existing_folders = [
"NOMAD-0001 - Artist1 - Song1",
"NOMAD-0005 - Artist2 - Song2", # Legacy gap - preserved
"NOMAD-1001 - Artist3 - Song3",
"NOMAD-1003 - Artist4 - Song4", # Gap at 1002 - should be filled
]
with patch.object(service, 'list_folders', return_value=existing_folders):
brand_code = service.get_next_brand_code("/test/path", "NOMAD")
assert brand_code == "NOMAD-1002" # Fills gap at 1002, ignores legacy gap
assert brand_code == "NOMAD-0006"
def test_get_next_brand_code_different_prefix(self):
"""Test brand code calculation with different brand prefix."""
from backend.services.dropbox_service import DropboxService
service = DropboxService()
existing_folders = [
"NOMAD-0001 - Artist1 - Song1",
"NOMAD-0002 - Artist2 - Song2",
"TEST-0001 - Artist3 - Song3",
"TEST-0002 - Artist4 - Song4",
"TEST-0010 - Artist2 - Song2",
]
with patch.object(service, 'list_folders', return_value=existing_folders):
# NOMAD sequence: 1,2 (below 1000) -> next is 3
brand_code = service.get_next_brand_code("/test/path", "NOMAD")
assert brand_code == "NOMAD-0003"
# TEST sequence: 1,2 (below 1000) -> next is 3
brand_code = service.get_next_brand_code("/test/path", "TEST")
assert brand_code == "TEST-0003"
assert brand_code == "TEST-0011"
def test_factory_function_returns_instance(self):
"""Test that get_dropbox_service returns a service instance."""
+12 -13
View File
@@ -222,31 +222,30 @@ class TestGetNextBrandCode:
@patch("backend.services.dropbox_service.secretmanager.SecretManagerServiceClient")
def test_get_next_brand_code(self, mock_sm_client_class):
"""Test calculating next brand code - fills gaps only above 1000."""
"""Test calculating next brand code from existing folders."""
from backend.services.dropbox_service import DropboxService
mock_sm_client = Mock()
mock_response = Mock()
mock_response.payload.data = json.dumps({"access_token": "token"}).encode()
mock_sm_client.access_secret_version.return_value = mock_response
mock_sm_client_class.return_value = mock_sm_client
service = DropboxService()
# Mock list_folders to return existing codes with a gap at 1003
# Mock list_folders to return existing codes
with patch.object(service, "list_folders") as mock_list:
mock_list.return_value = [
"NOMAD-1001",
"NOMAD-1002",
"NOMAD-1004", # Gap at 1003 - should be filled
"NOMAD-1005",
"NOMAD-1161",
"NOMAD-1162",
"NOMAD-1163",
"Other Folder",
"NOMAD-0001",
]
next_code = service.get_next_brand_code("/path", "NOMAD")
# Should fill the gap at 1003 (gaps >= 1001 are filled)
assert next_code == "NOMAD-1003"
assert next_code == "NOMAD-1164"
@patch("backend.services.dropbox_service.secretmanager.SecretManagerServiceClient")
def test_get_next_brand_code_empty_folder(self, mock_sm_client_class):
-631
View File
@@ -1,631 +0,0 @@
"""
Unit tests for countdown padding in all encoding paths.
These tests verify that all encoding backends correctly handle countdown padding
to ensure instrumental audio is synchronized with the countdown-padded vocals.
The tests cover:
1. KaraokeFinalise - standard encoding path
2. Local encoding backend
3. GCE encoding backend
"""
import pytest
from datetime import datetime, UTC
from unittest.mock import MagicMock, AsyncMock, patch, call
import tempfile
import os
from backend.models.job import Job, JobStatus
class TestKaraokeFinaliseCountdownPadding:
"""Tests for KaraokeFinalise countdown padding logic."""
def test_countdown_padding_stored_in_instance(self):
"""countdown_padding_seconds is stored when passed to constructor."""
from karaoke_gen.karaoke_finalise.karaoke_finalise import KaraokeFinalise
finalise = KaraokeFinalise(
countdown_padding_seconds=3.0,
non_interactive=True,
)
assert finalise.countdown_padding_seconds == 3.0
def test_pad_audio_file_method_signature(self):
"""_pad_audio_file accepts input, output, and padding_seconds."""
from karaoke_gen.karaoke_finalise.karaoke_finalise import KaraokeFinalise
import inspect
sig = inspect.signature(KaraokeFinalise._pad_audio_file)
params = list(sig.parameters.keys())
# Should have self, input_audio, output_audio, padding_seconds
assert 'input_audio' in params
assert 'output_audio' in params
assert 'padding_seconds' in params
@patch('subprocess.run')
def test_pad_audio_file_calls_ffmpeg(self, mock_subprocess):
"""_pad_audio_file uses ffmpeg to add silence padding."""
from karaoke_gen.karaoke_finalise.karaoke_finalise import KaraokeFinalise
mock_subprocess.return_value = MagicMock(returncode=0)
finalise = KaraokeFinalise(non_interactive=True)
with tempfile.TemporaryDirectory() as temp_dir:
input_audio = os.path.join(temp_dir, "input.flac")
output_audio = os.path.join(temp_dir, "output.flac")
# Create a fake input file
with open(input_audio, 'w') as f:
f.write("fake")
finalise._pad_audio_file(input_audio, output_audio, 3.0)
# Verify subprocess was called (ffmpeg)
assert mock_subprocess.called
call_args = str(mock_subprocess.call_args)
# Should use adelay filter for padding
assert 'adelay' in call_args
def test_remux_creates_padded_instrumental_when_needed(self):
"""remux_with_instrumental creates padded instrumental if countdown active."""
from karaoke_gen.karaoke_finalise.karaoke_finalise import KaraokeFinalise
finalise = KaraokeFinalise(
countdown_padding_seconds=3.0,
non_interactive=True,
)
# Mock the _pad_audio_file and execute_command methods
finalise._pad_audio_file = MagicMock()
finalise.execute_command = MagicMock()
with tempfile.TemporaryDirectory() as temp_dir:
with_vocals = os.path.join(temp_dir, "with_vocals.mp4")
instrumental = os.path.join(temp_dir, "instrumental.flac")
output = os.path.join(temp_dir, "output.mp4")
# Create fake files
for f in [with_vocals, instrumental]:
with open(f, 'w') as fp:
fp.write("fake")
finalise.remux_with_instrumental(with_vocals, instrumental, output)
# Should have called _pad_audio_file because instrumental isn't "(Padded)"
finalise._pad_audio_file.assert_called_once()
call_args = finalise._pad_audio_file.call_args
# Verify padding amount
assert call_args[0][2] == 3.0 # Third arg is padding_seconds
def test_remux_skips_padding_when_already_padded(self):
"""remux_with_instrumental skips padding if file already has (Padded)."""
from karaoke_gen.karaoke_finalise.karaoke_finalise import KaraokeFinalise
finalise = KaraokeFinalise(
countdown_padding_seconds=3.0,
non_interactive=True,
)
finalise._pad_audio_file = MagicMock()
finalise.execute_command = MagicMock()
with tempfile.TemporaryDirectory() as temp_dir:
with_vocals = os.path.join(temp_dir, "with_vocals.mp4")
# Note: "(Padded)" in the filename
instrumental = os.path.join(temp_dir, "instrumental (Padded).flac")
output = os.path.join(temp_dir, "output.mp4")
for f in [with_vocals, instrumental]:
with open(f, 'w') as fp:
fp.write("fake")
finalise.remux_with_instrumental(with_vocals, instrumental, output)
# Should NOT have called _pad_audio_file
finalise._pad_audio_file.assert_not_called()
def test_remux_no_padding_when_countdown_is_none(self):
"""remux_with_instrumental doesn't pad when countdown_padding_seconds is None."""
from karaoke_gen.karaoke_finalise.karaoke_finalise import KaraokeFinalise
finalise = KaraokeFinalise(
countdown_padding_seconds=None,
non_interactive=True,
)
finalise._pad_audio_file = MagicMock()
finalise.execute_command = MagicMock()
with tempfile.TemporaryDirectory() as temp_dir:
with_vocals = os.path.join(temp_dir, "with_vocals.mp4")
instrumental = os.path.join(temp_dir, "instrumental.flac")
output = os.path.join(temp_dir, "output.mp4")
for f in [with_vocals, instrumental]:
with open(f, 'w') as fp:
fp.write("fake")
finalise.remux_with_instrumental(with_vocals, instrumental, output)
# Should NOT have called _pad_audio_file
finalise._pad_audio_file.assert_not_called()
def test_remux_no_padding_when_countdown_is_zero(self):
"""remux_with_instrumental doesn't pad when countdown_padding_seconds is 0."""
from karaoke_gen.karaoke_finalise.karaoke_finalise import KaraokeFinalise
finalise = KaraokeFinalise(
countdown_padding_seconds=0,
non_interactive=True,
)
finalise._pad_audio_file = MagicMock()
finalise.execute_command = MagicMock()
with tempfile.TemporaryDirectory() as temp_dir:
with_vocals = os.path.join(temp_dir, "with_vocals.mp4")
instrumental = os.path.join(temp_dir, "instrumental.flac")
output = os.path.join(temp_dir, "output.mp4")
for f in [with_vocals, instrumental]:
with open(f, 'w') as fp:
fp.write("fake")
finalise.remux_with_instrumental(with_vocals, instrumental, output)
# Should NOT have called _pad_audio_file
finalise._pad_audio_file.assert_not_called()
class TestCountdownProcessorAudioPadding:
"""Tests for CountdownProcessor audio padding logic."""
def test_countdown_processor_constants(self):
"""CountdownProcessor has expected padding constants."""
from karaoke_gen.lyrics_transcriber.output.countdown_processor import CountdownProcessor
assert hasattr(CountdownProcessor, 'COUNTDOWN_PADDING_SECONDS')
assert CountdownProcessor.COUNTDOWN_PADDING_SECONDS == 3.0
assert hasattr(CountdownProcessor, 'COUNTDOWN_THRESHOLD_SECONDS')
assert CountdownProcessor.COUNTDOWN_THRESHOLD_SECONDS == 3.0
def test_countdown_processor_process_signature(self):
"""CountdownProcessor.process() has the correct signature for countdown handling.
This test verifies the process() method returns the expected types:
- CorrectionResult (modified with shifted timestamps)
- str (path to audio file, possibly padded)
- bool (padding_added)
- float (padding_seconds)
"""
from karaoke_gen.lyrics_transcriber.output.countdown_processor import CountdownProcessor
import inspect
sig = inspect.signature(CountdownProcessor.process)
# Check parameters
params = list(sig.parameters.keys())
assert 'correction_result' in params, "process() must accept correction_result"
assert 'audio_filepath' in params, "process() must accept audio_filepath"
def test_countdown_processor_determines_padding_need(self):
"""CountdownProcessor correctly determines if padding is needed.
Songs that start within 3 seconds should trigger countdown padding.
This tests the decision logic without running actual ffmpeg.
"""
from karaoke_gen.lyrics_transcriber.output.countdown_processor import CountdownProcessor
# Verify the CountdownProcessor has the padding threshold logic
processor = CountdownProcessor.__new__(CountdownProcessor)
# Check the default padding is 3.0 seconds
# Note: CountdownProcessor should have COUNTDOWN_DURATION constant
import inspect
source = inspect.getsource(CountdownProcessor)
# Verify countdown duration is defined
assert 'COUNTDOWN_DURATION' in source or '3.0' in source, (
"CountdownProcessor should define countdown duration (typically 3.0 seconds)"
)
class TestVideoWorkerCountdownPassthrough:
"""Tests that video_worker passes countdown to encoding correctly."""
@pytest.fixture
def job_with_countdown(self):
"""Job with countdown padding enabled."""
return Job(
job_id="test123",
status=JobStatus.ENCODING,
created_at=datetime.now(UTC),
updated_at=datetime.now(UTC),
artist="Test",
title="Test",
file_urls={
'instrumental_clean': 'gs://bucket/instrumental.flac',
'video_output': 'gs://bucket/video.mp4',
},
state_data={
'lyrics_metadata': {
'has_countdown_padding': True,
'countdown_padding_seconds': 3.0,
},
'video_worker': {
'instrumental_source': 'clean',
}
}
)
@pytest.fixture
def job_without_countdown(self):
"""Job without countdown padding."""
return Job(
job_id="test123",
status=JobStatus.ENCODING,
created_at=datetime.now(UTC),
updated_at=datetime.now(UTC),
artist="Test",
title="Test",
file_urls={
'instrumental_clean': 'gs://bucket/instrumental.flac',
'video_output': 'gs://bucket/video.mp4',
},
state_data={
'lyrics_metadata': {
'has_countdown_padding': False,
'countdown_padding_seconds': 0.0,
},
'video_worker': {
'instrumental_source': 'clean',
}
}
)
def test_extract_countdown_with_padding(self, job_with_countdown):
"""video_worker extracts countdown_padding_seconds when padding enabled."""
job = job_with_countdown
# This mirrors the logic in video_worker.py
lyrics_metadata = job.state_data.get('lyrics_metadata', {})
countdown_padding_seconds = None
if lyrics_metadata.get('has_countdown_padding'):
countdown_padding_seconds = lyrics_metadata.get('countdown_padding_seconds', 3.0)
assert countdown_padding_seconds == 3.0
def test_extract_countdown_without_padding(self, job_without_countdown):
"""video_worker returns None when no countdown padding."""
job = job_without_countdown
lyrics_metadata = job.state_data.get('lyrics_metadata', {})
countdown_padding_seconds = None
if lyrics_metadata.get('has_countdown_padding'):
countdown_padding_seconds = lyrics_metadata.get('countdown_padding_seconds', 3.0)
assert countdown_padding_seconds is None
def test_karaoke_finalise_receives_countdown(self, job_with_countdown):
"""KaraokeFinalise constructor receives countdown_padding_seconds.
This test verifies that:
1. KaraokeFinalise accepts the countdown_padding_seconds parameter
2. The video_worker code passes the extracted value to KaraokeFinalise
"""
# Verify the value that would be passed to KaraokeFinalise
job = job_with_countdown
lyrics_metadata = job.state_data.get('lyrics_metadata', {})
countdown_padding_seconds = None
if lyrics_metadata.get('has_countdown_padding'):
countdown_padding_seconds = lyrics_metadata.get('countdown_padding_seconds', 3.0)
assert countdown_padding_seconds == 3.0, (
"countdown_padding_seconds should be 3.0 when has_countdown_padding is True"
)
# Verify KaraokeFinalise accepts the parameter
from karaoke_gen.karaoke_finalise.karaoke_finalise import KaraokeFinalise
import inspect
sig = inspect.signature(KaraokeFinalise.__init__)
assert 'countdown_padding_seconds' in sig.parameters, (
"KaraokeFinalise must accept countdown_padding_seconds parameter"
)
def test_karaoke_finalise_receives_none_when_no_countdown(self, job_without_countdown):
"""KaraokeFinalise receives None for countdown when no padding needed.
This test verifies that when has_countdown_padding is False,
the countdown_padding_seconds remains None (not passed to KaraokeFinalise).
"""
job = job_without_countdown
lyrics_metadata = job.state_data.get('lyrics_metadata', {})
countdown_padding_seconds = None
if lyrics_metadata.get('has_countdown_padding'):
countdown_padding_seconds = lyrics_metadata.get('countdown_padding_seconds', 3.0)
# Should be None when no countdown
assert countdown_padding_seconds is None, (
"countdown_padding_seconds should be None when has_countdown_padding is False"
)
def test_video_worker_source_passes_countdown_to_karaoke_finalise(self):
"""Verify video_worker.py source code passes countdown to KaraokeFinalise.
This is a source code inspection test to ensure the parameter is actually
being passed in the KaraokeFinalise constructor call.
"""
import inspect
from backend.workers import video_worker
source = inspect.getsource(video_worker)
# Verify the KaraokeFinalise instantiation includes countdown_padding_seconds
assert 'countdown_padding_seconds=countdown_padding_seconds' in source, (
"video_worker.py must pass countdown_padding_seconds to KaraokeFinalise"
)
class TestGCEEncodingCountdownPassthrough:
"""Tests for GCE encoding countdown handling."""
def test_gce_encoding_config_structure(self):
"""GCE encoding config can include countdown_padding_seconds."""
encoding_config = {
"formats": ["mp4_4k_lossless", "mp4_4k_lossy", "mkv_4k", "mp4_720p"],
"artist": "Test",
"title": "Song",
"countdown_padding_seconds": 3.0,
}
assert encoding_config.get('countdown_padding_seconds') == 3.0
@pytest.mark.asyncio
async def test_gce_backend_encodes_with_options(self):
"""GCE backend passes options through to encoding service."""
from backend.services.encoding_interface import GCEEncodingBackend, EncodingInput
backend = GCEEncodingBackend()
encoding_input = EncodingInput(
title_video_path="/path/title.mov",
karaoke_video_path="/path/karaoke.mov",
instrumental_audio_path="/path/instrumental.flac",
artist="Test",
title="Song",
output_dir="/output",
options={
'job_id': 'test123',
'countdown_padding_seconds': 3.0,
}
)
# The options field should be accessible
assert encoding_input.options.get('countdown_padding_seconds') == 3.0
class TestLocalEncodingCountdownPassthrough:
"""Tests for local encoding countdown handling."""
def test_local_encoding_input_with_countdown(self):
"""Local encoding receives EncodingInput with countdown in options."""
from backend.services.encoding_interface import LocalEncodingBackend, EncodingInput
encoding_input = EncodingInput(
title_video_path="/path/title.mov",
karaoke_video_path="/path/karaoke.mov",
instrumental_audio_path="/path/instrumental.flac",
artist="Test",
title="Song",
output_dir="/output",
options={
'countdown_padding_seconds': 3.0,
}
)
backend = LocalEncodingBackend()
# Backend should be able to access countdown from options
assert encoding_input.options.get('countdown_padding_seconds') == 3.0
class TestCountdownTimingInvariant:
"""Tests for the audio timing invariant."""
def test_padding_preserves_timing_relationship(self):
"""
INVARIANT: If vocals are padded by N seconds, instrumental must also
be padded by N seconds to maintain sync.
Final timing relationship:
- Original first word at T seconds
- After padding: first word at T + N seconds
- Instrumental must also start N seconds later (via padding)
"""
vocal_first_word_original = 0.5 # seconds
padding_seconds = 3.0
vocal_first_word_after_padding = vocal_first_word_original + padding_seconds
# Instrumental should be padded by same amount
instrumental_start_original = 0.0
instrumental_start_after_padding = instrumental_start_original + padding_seconds
# The relative timing should be preserved
relative_timing_before = vocal_first_word_original - instrumental_start_original
relative_timing_after = vocal_first_word_after_padding - instrumental_start_after_padding
assert relative_timing_before == relative_timing_after, (
"INVARIANT VIOLATION: Padding must preserve relative timing between "
"vocals and instrumental"
)
def test_countdown_threshold_makes_sense(self):
"""
Songs starting before threshold need countdown.
Songs starting after threshold don't need countdown.
"""
from karaoke_gen.lyrics_transcriber.output.countdown_processor import CountdownProcessor
threshold = CountdownProcessor.COUNTDOWN_THRESHOLD_SECONDS
# Song starting at 0.5s needs countdown
early_start = 0.5
assert early_start < threshold, "Early start should be before threshold"
# Song starting at 5.0s doesn't need countdown
late_start = 5.0
assert late_start > threshold, "Late start should be after threshold"
# Padding should match threshold
padding = CountdownProcessor.COUNTDOWN_PADDING_SECONDS
assert padding == threshold, (
"Padding duration should match threshold to give singer exactly "
"the countdown time they need"
)
class TestGCEEncodingFileFinderBugRegression:
"""Regression tests for GCE encoding file finder bug (fixed 2026-01-24).
Bug description:
When re-encoding a job after reset, the GCE encoding worker would pick up
old finals/*.mkv files instead of the fresh videos/with_vocals.mkv source.
This caused:
1. The file finder patterns didn't match 'with_vocals.mkv' (case-sensitivity)
2. The fallback '*.mkv' matched old finals that already had title/end concatenated
3. Re-encoding these files caused doubled title/end screens and audio desync
The fix was to:
1. Add lowercase patterns ('*with_vocals*', '*vocals*')
2. Prioritize videos/ subdirectory in search order
3. Exclude finals/ and encoded output files from file finder
"""
def test_find_file_patterns_match_with_vocals_mkv(self):
"""File finder patterns must match 'with_vocals.mkv' (lowercase)."""
import fnmatch
actual_filename = "with_vocals.mkv"
# These patterns MUST match the actual filename
matching_patterns = [
"*with_vocals*.mkv",
"*vocals*.mkv",
]
for pattern in matching_patterns:
assert fnmatch.fnmatch(actual_filename, pattern), (
f"Pattern '{pattern}' MUST match '{actual_filename}'"
)
# These patterns should NOT match (wrong case)
non_matching_patterns = [
"*With Vocals*.mkv", # Wrong case
"*Vocals*.mkv", # Wrong case
]
for pattern in non_matching_patterns:
assert not fnmatch.fnmatch(actual_filename, pattern), (
f"Pattern '{pattern}' should NOT match '{actual_filename}' (case mismatch)"
)
def test_find_file_excludes_finals_directory(self):
"""File finder must exclude files from finals/ directory.
This prevents picking up old encoding outputs when re-encoding.
"""
from pathlib import Path
# Paths that should be EXCLUDED
excluded_paths = [
Path("finals/Regina Spektor - Hotel Song (Final Karaoke Lossless 4k).mkv"),
Path("finals/lossless_4k_mkv.mkv"),
Path("outputs/something.mkv"),
]
for path in excluded_paths:
path_str = str(path).lower()
name_lower = path.name.lower()
# Apply the exclusion logic from the fix
should_exclude = (
"title" in name_lower or "end" in name_lower or
"outputs" in path_str or "finals" in path_str or
"final karaoke" in name_lower or "lossless" in name_lower or
"lossy" in name_lower or "720p" in name_lower
)
assert should_exclude, (
f"Path '{path}' MUST be excluded from file finder"
)
def test_find_file_prioritizes_videos_directory(self):
"""File finder must check videos/ subdirectory before other locations.
This ensures the fresh source video is found before any old outputs.
"""
from pathlib import Path
import tempfile
def find_file(work_dir: Path, *patterns):
'''Find a file matching any of the given glob patterns.'''
for pattern in patterns:
matches = list(work_dir.glob(f"**/{pattern}"))
if matches:
return matches[0]
return None
with tempfile.TemporaryDirectory() as td:
work_dir = Path(td)
# Create both source and old output
(work_dir / "videos").mkdir()
source_file = work_dir / "videos" / "with_vocals.mkv"
source_file.touch()
(work_dir / "finals").mkdir()
old_output = work_dir / "finals" / "lossless_4k_mkv.mkv"
old_output.touch()
# The fixed patterns from the fix
result = find_file(
work_dir,
"videos/with_vocals.mkv", # Must be first!
"*with_vocals*.mkv",
"*vocals*.mkv",
"*.mkv"
)
assert result == source_file, (
f"File finder must return source file from videos/, got {result}"
)
def test_gce_encoding_main_has_fixed_patterns(self):
"""Verify the GCE encoding main.py has the fixed file finder patterns.
This is a source code inspection test to ensure the fix is in place.
"""
from pathlib import Path
gce_main_path = Path(__file__).parent.parent.parent / "backend" / "services" / "gce_encoding" / "main.py"
source = gce_main_path.read_text()
# Check for the key patterns that were added in the fix
assert "videos/with_vocals.mkv" in source, (
"GCE encoding main.py must prioritize videos/ subdirectory"
)
assert '*with_vocals*' in source or "*with_vocals*" in source, (
"GCE encoding main.py must have lowercase pattern for with_vocals"
)
assert '"finals"' in source.lower() or "'finals'" in source.lower(), (
"GCE encoding main.py must exclude finals/ directory"
)
@@ -1,365 +0,0 @@
"""
Contract tests for encoding interface countdown support.
These tests verify that all encoding backends correctly handle countdown padding.
The encoding backends must:
1. Accept countdown_padding_seconds parameter
2. Pad the instrumental audio when countdown_padding_seconds > 0
3. Ensure audio sync between padded vocals and instrumental
"""
import pytest
from datetime import datetime, UTC
from unittest.mock import MagicMock, AsyncMock, patch
import tempfile
import os
from backend.models.job import Job, JobStatus
class TestEncodingInputCountdownSupport:
"""Tests that EncodingInput can carry countdown information."""
def test_encoding_input_options_can_contain_countdown(self):
"""EncodingInput.options dict can carry countdown_padding_seconds."""
from backend.services.encoding_interface import EncodingInput
encoding_input = EncodingInput(
title_video_path="/path/to/title.mov",
karaoke_video_path="/path/to/karaoke.mov",
instrumental_audio_path="/path/to/instrumental.flac",
artist="Test",
title="Song",
output_dir="/output",
options={
'countdown_padding_seconds': 3.0
}
)
assert encoding_input.options.get('countdown_padding_seconds') == 3.0
def test_encoding_input_options_default_to_empty(self):
"""EncodingInput.options defaults to empty dict (no countdown)."""
from backend.services.encoding_interface import EncodingInput
encoding_input = EncodingInput(
title_video_path="/path/to/title.mov",
karaoke_video_path="/path/to/karaoke.mov",
instrumental_audio_path="/path/to/instrumental.flac",
artist="Test",
title="Song",
output_dir="/output",
)
assert encoding_input.options == {}
assert encoding_input.options.get('countdown_padding_seconds') is None
class TestKaraokeFinaliseCountdownContract:
"""Contract tests for KaraokeFinalise countdown padding support."""
def test_karaoke_finalise_accepts_countdown_parameter(self):
"""KaraokeFinalise __init__ must accept countdown_padding_seconds."""
from karaoke_gen.karaoke_finalise.karaoke_finalise import KaraokeFinalise
import inspect
sig = inspect.signature(KaraokeFinalise.__init__)
params = list(sig.parameters.keys())
assert 'countdown_padding_seconds' in params, (
"CONTRACT VIOLATION: KaraokeFinalise must accept countdown_padding_seconds parameter"
)
def test_karaoke_finalise_stores_countdown_value(self):
"""KaraokeFinalise stores countdown_padding_seconds for later use."""
from karaoke_gen.karaoke_finalise.karaoke_finalise import KaraokeFinalise
finalise = KaraokeFinalise(
countdown_padding_seconds=3.0,
non_interactive=True,
)
assert finalise.countdown_padding_seconds == 3.0
def test_karaoke_finalise_handles_none_countdown(self):
"""KaraokeFinalise handles None countdown gracefully."""
from karaoke_gen.karaoke_finalise.karaoke_finalise import KaraokeFinalise
finalise = KaraokeFinalise(
countdown_padding_seconds=None,
non_interactive=True,
)
assert finalise.countdown_padding_seconds is None
def test_karaoke_finalise_handles_zero_countdown(self):
"""KaraokeFinalise handles zero countdown (no padding needed)."""
from karaoke_gen.karaoke_finalise.karaoke_finalise import KaraokeFinalise
finalise = KaraokeFinalise(
countdown_padding_seconds=0,
non_interactive=True,
)
assert finalise.countdown_padding_seconds == 0
class TestGCEEncodingCountdownContract:
"""Contract tests for GCE encoding countdown support."""
def test_gce_encoding_config_can_include_countdown(self):
"""GCE encoding config dict structure supports countdown_padding_seconds."""
# This mirrors how video_worker builds the config for GCE
encoding_config = {
"formats": ["mp4_4k_lossless", "mp4_4k_lossy", "mkv_4k", "mp4_720p"],
"artist": "Test Artist",
"title": "Test Song",
"countdown_padding_seconds": 3.0, # MUST be supported
}
assert 'countdown_padding_seconds' in encoding_config
assert encoding_config['countdown_padding_seconds'] == 3.0
def test_gce_backend_receives_countdown_in_config(self):
"""GCE encoding backend receives countdown_padding_seconds in config.
This test verifies that the GCEEncodingBackend can receive and process
countdown_padding_seconds through the EncodingInput options.
"""
from backend.services.encoding_interface import GCEEncodingBackend, EncodingInput
# Create an encoding input with countdown padding
encoding_input = EncodingInput(
title_video_path="/path/title.mov",
karaoke_video_path="/path/karaoke.mov",
instrumental_audio_path="/path/instrumental.flac",
artist="Test",
title="Song",
output_dir="/output",
options={
'job_id': 'test123',
'countdown_padding_seconds': 3.0,
}
)
# Verify the options can store countdown_padding_seconds
assert encoding_input.options.get('countdown_padding_seconds') == 3.0, (
"EncodingInput.options must support countdown_padding_seconds"
)
# Verify GCEEncodingBackend can be instantiated
backend = GCEEncodingBackend()
assert backend is not None
def test_gce_encoding_path_documented(self):
"""Document the GCE encoding path's countdown handling.
NOTE: The encoding_interface.py is a general abstraction that doesn't
directly handle countdown_padding_seconds. The countdown handling for
GCE encoding is managed in video_worker.py which passes it in the
encoding_config to the GCE worker.
This test verifies the encoding interface structure is correct.
"""
from backend.services.encoding_interface import GCEEncodingBackend, EncodingInput
# Verify the interface supports options which can carry countdown info
encoding_input = EncodingInput(
title_video_path="/path/title.mov",
karaoke_video_path="/path/karaoke.mov",
instrumental_audio_path="/path/instrumental.flac",
artist="Test",
title="Song",
output_dir="/output",
options={'countdown_padding_seconds': 3.0} # Can be passed via options
)
assert 'countdown_padding_seconds' in encoding_input.options, (
"EncodingInput.options should support countdown_padding_seconds"
)
class TestLocalEncodingCountdownContract:
"""Contract tests for local encoding countdown support."""
def test_local_encoding_service_exists(self):
"""LocalEncodingService class exists and can be imported."""
from backend.services.encoding_interface import LocalEncodingBackend
assert LocalEncodingBackend is not None
@pytest.mark.asyncio
async def test_local_backend_uses_encoding_input_options(self):
"""Local encoding backend should use options from EncodingInput."""
from backend.services.encoding_interface import LocalEncodingBackend, EncodingInput
backend = LocalEncodingBackend()
encoding_input = EncodingInput(
title_video_path="/path/title.mov",
karaoke_video_path="/path/karaoke.mov",
instrumental_audio_path="/path/instrumental.flac",
artist="Test",
title="Song",
output_dir="/output",
options={
'countdown_padding_seconds': 3.0,
}
)
# Verify options are accessible
assert encoding_input.options.get('countdown_padding_seconds') == 3.0
class TestVideoWorkerEncodingPathContracts:
"""Tests that video_worker passes countdown to all encoding paths."""
@pytest.fixture
def mock_job_with_countdown(self):
"""Create a job with countdown padding."""
return Job(
job_id="test123",
status=JobStatus.ENCODING,
created_at=datetime.now(UTC),
updated_at=datetime.now(UTC),
artist="Test",
title="Test",
file_urls={
'instrumental_clean': 'gs://bucket/instrumental.flac',
'video_output': 'gs://bucket/video.mp4',
},
state_data={
'lyrics_metadata': {
'has_countdown_padding': True,
'countdown_padding_seconds': 3.0,
},
'video_worker': {
'instrumental_source': 'clean',
}
}
)
def test_countdown_extraction_logic(self, mock_job_with_countdown):
"""video_worker countdown extraction logic works correctly."""
job = mock_job_with_countdown
# This is the extraction logic from video_worker.py
lyrics_metadata = job.state_data.get('lyrics_metadata', {})
countdown_padding_seconds = None
if lyrics_metadata.get('has_countdown_padding'):
countdown_padding_seconds = lyrics_metadata.get('countdown_padding_seconds', 3.0)
assert countdown_padding_seconds == 3.0
def test_karaoke_finalise_path_receives_countdown(self, mock_job_with_countdown):
"""Standard encoding path passes countdown to KaraokeFinalise.
This test verifies that:
1. The job state contains countdown info
2. KaraokeFinalise accepts countdown_padding_seconds
3. video_worker.py source passes countdown to KaraokeFinalise
"""
job = mock_job_with_countdown
# Verify job has countdown state
lyrics_metadata = job.state_data.get('lyrics_metadata', {})
assert lyrics_metadata.get('has_countdown_padding') is True
assert lyrics_metadata.get('countdown_padding_seconds') == 3.0
# Verify KaraokeFinalise accepts the parameter
from karaoke_gen.karaoke_finalise.karaoke_finalise import KaraokeFinalise
import inspect
sig = inspect.signature(KaraokeFinalise.__init__)
assert 'countdown_padding_seconds' in sig.parameters, (
"KaraokeFinalise must accept countdown_padding_seconds parameter"
)
# Verify video_worker.py passes the parameter
source = inspect.getsource(inspect.getmodule(KaraokeFinalise))
# Note: The actual source check is in the video_worker tests
class TestCountdownPaddingInvariant:
"""
Tests the critical invariant:
If countdown was added to vocals, instrumental MUST be padded to match.
"""
def test_invariant_countdown_implies_padding(self):
"""
INVARIANT: has_countdown_padding=True implies countdown_padding_seconds > 0
"""
# Valid states
valid_state_with_countdown = {
'has_countdown_padding': True,
'countdown_padding_seconds': 3.0,
}
valid_state_without_countdown = {
'has_countdown_padding': False,
'countdown_padding_seconds': 0.0,
}
# Check invariant
if valid_state_with_countdown['has_countdown_padding']:
assert valid_state_with_countdown['countdown_padding_seconds'] > 0, (
"INVARIANT VIOLATION: has_countdown_padding=True but padding_seconds=0"
)
if not valid_state_without_countdown['has_countdown_padding']:
assert valid_state_without_countdown['countdown_padding_seconds'] == 0.0, (
"INVARIANT VIOLATION: has_countdown_padding=False but padding_seconds > 0"
)
def test_render_video_worker_sets_valid_state(self):
"""render_video_worker should never set invalid state (countdown=True but seconds=0).
This test verifies that the fix in render_video_worker.py sets both
has_countdown_padding and countdown_padding_seconds together.
"""
# Simulate the state update from render_video_worker
# When padding_added=True, BOTH fields must be set correctly
padding_added = True
padding_seconds = 3.0 # Always 3.0 from CountdownProcessor
state_data = {'lyrics_metadata': {}}
# This is the expected behavior from render_video_worker
if padding_added:
state_data['lyrics_metadata']['has_countdown_padding'] = True
state_data['lyrics_metadata']['countdown_padding_seconds'] = padding_seconds
# Verify the invariant holds
lyrics_metadata = state_data.get('lyrics_metadata', {})
if lyrics_metadata.get('has_countdown_padding'):
assert lyrics_metadata.get('countdown_padding_seconds', 0) > 0, (
"INVARIANT VIOLATION: render_video_worker must set both "
"has_countdown_padding and countdown_padding_seconds together"
)
class TestAudioPaddingFunction:
"""Tests for the _pad_audio_file function in KaraokeFinalise."""
def test_pad_audio_file_function_exists(self):
"""KaraokeFinalise has _pad_audio_file method."""
from karaoke_gen.karaoke_finalise.karaoke_finalise import KaraokeFinalise
finalise = KaraokeFinalise(non_interactive=True)
assert hasattr(finalise, '_pad_audio_file'), (
"KaraokeFinalise must have _pad_audio_file method for instrumental padding"
)
def test_remux_with_instrumental_checks_padding(self):
"""remux_with_instrumental method handles countdown padding."""
from karaoke_gen.karaoke_finalise.karaoke_finalise import KaraokeFinalise
finalise = KaraokeFinalise(
countdown_padding_seconds=3.0,
non_interactive=True,
)
# The method should exist and check countdown_padding_seconds
assert hasattr(finalise, 'remux_with_instrumental'), (
"KaraokeFinalise must have remux_with_instrumental method"
)
+371 -33
View File
@@ -1,93 +1,431 @@
"""
API endpoint tests for instrumental review functionality.
Note: Most standalone instrumental endpoints were removed in the combined review flow.
This file now only tests request models that are still used.
Tests for:
- GET /api/jobs/{job_id}/instrumental-analysis
- GET /api/jobs/{job_id}/audio-stream/{stem_type}
- POST /api/jobs/{job_id}/create-custom-instrumental
- GET /api/jobs/{job_id}/waveform-data
"""
import pytest
from pydantic import ValidationError
from datetime import datetime
from unittest.mock import MagicMock, patch, AsyncMock
from backend.models.job import Job, JobStatus
@pytest.fixture
def mock_job_manager():
"""Create a mock job manager."""
with patch('backend.api.routes.jobs.job_manager') as mock:
yield mock
@pytest.fixture
def mock_storage_service():
"""Create a mock storage service."""
with patch('backend.api.routes.jobs.StorageService') as mock:
mock_instance = MagicMock()
mock_instance.generate_signed_url.return_value = "https://storage.googleapis.com/signed-url"
mock.return_value = mock_instance
yield mock_instance
@pytest.fixture
def sample_job():
"""Create a sample job for testing."""
return Job(
job_id="test-job-123",
artist="Test Artist",
title="Test Song",
status=JobStatus.AWAITING_INSTRUMENTAL_SELECTION,
created_at=datetime.now(),
updated_at=datetime.now(),
file_urls={
"stems": {
"instrumental_clean": "jobs/test/stems/instrumental_clean.flac",
"backing_vocals": "jobs/test/stems/backing_vocals.flac",
"instrumental_with_backing": "jobs/test/stems/instrumental_with_backing.flac",
},
"analysis": {
"backing_vocals_waveform": "jobs/test/analysis/waveform.png",
},
},
state_data={
"backing_vocals_analysis": {
"has_audible_content": True,
"total_duration_seconds": 180.0,
"audible_segments": [
{
"start_seconds": 10.0,
"end_seconds": 20.0,
"duration_seconds": 10.0,
"avg_amplitude_db": -25.0,
},
{
"start_seconds": 60.0,
"end_seconds": 80.0,
"duration_seconds": 20.0,
"avg_amplitude_db": -30.0,
},
],
"recommended_selection": "review_needed",
"total_audible_duration_seconds": 30.0,
"audible_percentage": 16.67,
"silence_threshold_db": -40.0,
}
},
)
@pytest.fixture
def silent_job():
"""Create a job with silent backing vocals."""
return Job(
job_id="test-job-silent",
artist="Test Artist",
title="Test Song",
status=JobStatus.AWAITING_INSTRUMENTAL_SELECTION,
created_at=datetime.now(),
updated_at=datetime.now(),
file_urls={
"stems": {
"instrumental_clean": "jobs/test/stems/instrumental_clean.flac",
"backing_vocals": "jobs/test/stems/backing_vocals.flac",
"instrumental_with_backing": "jobs/test/stems/instrumental_with_backing.flac",
},
"analysis": {
"backing_vocals_waveform": "jobs/test/analysis/waveform.png",
},
},
state_data={
"backing_vocals_analysis": {
"has_audible_content": False,
"total_duration_seconds": 180.0,
"audible_segments": [],
"recommended_selection": "clean",
"total_audible_duration_seconds": 0.0,
"audible_percentage": 0.0,
"silence_threshold_db": -40.0,
}
},
)
class TestGetInstrumentalAnalysis:
"""Tests for GET /api/jobs/{job_id}/instrumental-analysis."""
@pytest.mark.asyncio
async def test_get_analysis_returns_data(self, mock_job_manager, mock_storage_service, sample_job):
"""GET /instrumental-analysis should return analysis data."""
mock_job_manager.get_job.return_value = sample_job
from backend.api.routes.jobs import get_instrumental_analysis
result = await get_instrumental_analysis("test-job-123")
assert result["job_id"] == "test-job-123"
assert result["artist"] == "Test Artist"
assert result["title"] == "Test Song"
assert result["analysis"]["has_audible_content"] is True
assert len(result["analysis"]["audible_segments"]) == 2
@pytest.mark.asyncio
async def test_get_analysis_includes_audio_urls(self, mock_job_manager, mock_storage_service, sample_job):
"""GET /instrumental-analysis should include audio URLs."""
mock_job_manager.get_job.return_value = sample_job
from backend.api.routes.jobs import get_instrumental_analysis
result = await get_instrumental_analysis("test-job-123")
assert "audio_urls" in result
assert "clean_instrumental" in result["audio_urls"]
assert "backing_vocals" in result["audio_urls"]
assert "with_backing" in result["audio_urls"]
@pytest.mark.asyncio
async def test_get_analysis_includes_waveform_url(self, mock_job_manager, sample_job):
"""GET /instrumental-analysis should include waveform URL."""
mock_job_manager.get_job.return_value = sample_job
# Need to mock StorageService at module level for the signed URL
with patch('backend.api.routes.jobs.StorageService') as mock_storage_cls:
mock_storage = MagicMock()
mock_storage.generate_signed_url.return_value = "https://storage.googleapis.com/signed-url"
mock_storage_cls.return_value = mock_storage
from backend.api.routes.jobs import get_instrumental_analysis
result = await get_instrumental_analysis("test-job-123")
assert "waveform_url" in result
# Waveform URL should be generated from the file_urls
assert result["waveform_url"] == "https://storage.googleapis.com/signed-url"
@pytest.mark.asyncio
async def test_get_analysis_silent_audio(self, mock_job_manager, mock_storage_service, silent_job):
"""GET /instrumental-analysis should correctly report silent audio."""
mock_job_manager.get_job.return_value = silent_job
from backend.api.routes.jobs import get_instrumental_analysis
result = await get_instrumental_analysis("test-job-silent")
assert result["analysis"]["has_audible_content"] is False
assert result["analysis"]["recommended_selection"] == "clean"
assert len(result["analysis"]["audible_segments"]) == 0
@pytest.mark.asyncio
async def test_get_analysis_job_not_found(self, mock_job_manager):
"""GET /instrumental-analysis should return 404 for non-existent job."""
mock_job_manager.get_job.return_value = None
from backend.api.routes.jobs import get_instrumental_analysis
from fastapi import HTTPException
with pytest.raises(HTTPException) as exc_info:
await get_instrumental_analysis("non-existent")
assert exc_info.value.status_code == 404
@pytest.mark.asyncio
async def test_get_analysis_wrong_status(self, mock_job_manager, sample_job):
"""GET /instrumental-analysis should return 400 for wrong job status."""
sample_job.status = JobStatus.PENDING
mock_job_manager.get_job.return_value = sample_job
from backend.api.routes.jobs import get_instrumental_analysis
from fastapi import HTTPException
with pytest.raises(HTTPException) as exc_info:
await get_instrumental_analysis("test-job-123")
assert exc_info.value.status_code == 400
class TestAudioStream:
"""Tests for GET /api/jobs/{job_id}/audio-stream/{stem_type}."""
def test_stream_backing_vocals_valid(self, mock_job_manager, sample_job):
"""Should return redirect for backing vocals stream."""
mock_job_manager.get_job.return_value = sample_job
from backend.api.routes.jobs import stream_audio
# This will try to stream from GCS which we can't test easily
# Instead, verify the function exists and has correct signature
assert callable(stream_audio)
@pytest.mark.asyncio
async def test_stream_invalid_stem_type(self, mock_job_manager, sample_job):
"""Should return 400 for invalid stem type."""
mock_job_manager.get_job.return_value = sample_job
from backend.api.routes.jobs import stream_audio
from fastapi import HTTPException
with pytest.raises(HTTPException) as exc_info:
await stream_audio("test-job-123", "invalid_stem")
assert exc_info.value.status_code == 400
@pytest.mark.asyncio
async def test_stream_job_not_found(self, mock_job_manager):
"""Should return 404 for non-existent job."""
mock_job_manager.get_job.return_value = None
from backend.api.routes.jobs import stream_audio
from fastapi import HTTPException
with pytest.raises(HTTPException) as exc_info:
await stream_audio("non-existent", "backing_vocals")
assert exc_info.value.status_code == 404
class TestCreateCustomInstrumental:
"""Tests for POST /api/jobs/{job_id}/create-custom-instrumental."""
@pytest.fixture
def mock_audio_editing_service(self):
"""Mock the audio editing service."""
with patch('backend.api.routes.jobs.AudioEditingService') as mock:
mock_instance = MagicMock()
mock_instance.create_custom_instrumental.return_value = MagicMock(
output_path="jobs/test/stems/custom_instrumental.flac",
mute_regions_applied=[
MagicMock(start_seconds=10.0, end_seconds=20.0),
],
total_muted_duration_seconds=10.0,
output_duration_seconds=180.0,
)
mock.return_value = mock_instance
yield mock_instance
def test_create_custom_validates_mute_regions(self):
"""Should validate mute regions at model level."""
from backend.models.requests import CreateCustomInstrumentalRequest, MuteRegionRequest
from pydantic import ValidationError
# Empty mute regions should fail at model validation
with pytest.raises(ValidationError) as exc_info:
CreateCustomInstrumentalRequest(mute_regions=[])
assert "At least one mute region is required" in str(exc_info.value)
@pytest.mark.asyncio
async def test_create_custom_job_not_found(self, mock_job_manager):
"""Should return 404 for non-existent job."""
mock_job_manager.get_job.return_value = None
from backend.api.routes.jobs import create_custom_instrumental
from backend.models.requests import CreateCustomInstrumentalRequest, MuteRegionRequest
from fastapi import HTTPException
request = CreateCustomInstrumentalRequest(
mute_regions=[MuteRegionRequest(start_seconds=10.0, end_seconds=20.0)]
)
with pytest.raises(HTTPException) as exc_info:
await create_custom_instrumental("non-existent", request)
assert exc_info.value.status_code == 404
@pytest.mark.asyncio
async def test_create_custom_wrong_status(self, mock_job_manager, sample_job):
"""Should return 400 for wrong job status."""
sample_job.status = JobStatus.PENDING
mock_job_manager.get_job.return_value = sample_job
from backend.api.routes.jobs import create_custom_instrumental
from backend.models.requests import CreateCustomInstrumentalRequest, MuteRegionRequest
from fastapi import HTTPException
request = CreateCustomInstrumentalRequest(
mute_regions=[MuteRegionRequest(start_seconds=10.0, end_seconds=20.0)]
)
with pytest.raises(HTTPException) as exc_info:
await create_custom_instrumental("test-job-123", request)
assert exc_info.value.status_code == 400
class TestGetWaveformData:
"""Tests for GET /api/jobs/{job_id}/waveform-data."""
@pytest.mark.asyncio
async def test_get_waveform_data_job_not_found(self, mock_job_manager):
"""Should return 404 for non-existent job."""
mock_job_manager.get_job.return_value = None
from backend.api.routes.jobs import get_waveform_data
from fastapi import HTTPException
with pytest.raises(HTTPException) as exc_info:
await get_waveform_data("non-existent")
assert exc_info.value.status_code == 404
@pytest.mark.asyncio
async def test_get_waveform_data_wrong_status(self, mock_job_manager, sample_job):
"""Should return 400 for wrong job status."""
sample_job.status = JobStatus.PENDING
mock_job_manager.get_job.return_value = sample_job
from backend.api.routes.jobs import get_waveform_data
from fastapi import HTTPException
with pytest.raises(HTTPException) as exc_info:
await get_waveform_data("test-job-123")
assert exc_info.value.status_code == 400
def test_waveform_data_endpoint_exists(self):
"""Verify the endpoint function exists and is callable."""
from backend.api.routes.jobs import get_waveform_data
assert callable(get_waveform_data)
# Verify it's an async function
import inspect
assert inspect.iscoroutinefunction(get_waveform_data)
class TestRequestModels:
"""Tests for API request models used in instrumental selection."""
"""Tests for API request models."""
def test_mute_region_request_valid(self):
"""MuteRegionRequest should accept valid values."""
from backend.models.requests import MuteRegionRequest
region = MuteRegionRequest(start_seconds=10.0, end_seconds=20.0)
assert region.start_seconds == 10.0
assert region.end_seconds == 20.0
def test_mute_region_request_invalid_start(self):
"""MuteRegionRequest should reject negative start."""
from backend.models.requests import MuteRegionRequest
from pydantic import ValidationError
with pytest.raises(ValidationError):
MuteRegionRequest(start_seconds=-1.0, end_seconds=20.0)
def test_mute_region_request_invalid_order(self):
"""MuteRegionRequest should reject end before start."""
from backend.models.requests import MuteRegionRequest
from pydantic import ValidationError
with pytest.raises(ValidationError):
MuteRegionRequest(start_seconds=20.0, end_seconds=10.0)
def test_create_custom_instrumental_request_valid(self):
"""CreateCustomInstrumentalRequest should accept valid mute regions."""
from backend.models.requests import (
CreateCustomInstrumentalRequest,
MuteRegionRequest,
)
request = CreateCustomInstrumentalRequest(
mute_regions=[
MuteRegionRequest(start_seconds=10.0, end_seconds=20.0),
MuteRegionRequest(start_seconds=60.0, end_seconds=80.0),
]
)
assert len(request.mute_regions) == 2
def test_create_custom_instrumental_request_empty_regions(self):
"""CreateCustomInstrumentalRequest should reject empty mute regions."""
from backend.models.requests import CreateCustomInstrumentalRequest
with pytest.raises(ValidationError) as exc_info:
CreateCustomInstrumentalRequest(mute_regions=[])
assert "At least one mute region is required" in str(exc_info.value)
class TestInstrumentalSelectionModel:
"""Tests for InstrumentalSelection request model."""
class TestInstrumentalSelectionExtension:
"""Tests for extended instrumental selection (including 'custom')."""
def test_instrumental_selection_accepts_custom(self):
"""InstrumentalSelection should accept 'custom' as valid selection."""
from backend.models.requests import InstrumentalSelection
selection = InstrumentalSelection(selection="custom")
assert selection.selection == "custom"
def test_instrumental_selection_accepts_clean(self):
"""InstrumentalSelection should accept 'clean' as valid selection."""
from backend.models.requests import InstrumentalSelection
selection = InstrumentalSelection(selection="clean")
assert selection.selection == "clean"
def test_instrumental_selection_accepts_with_backing(self):
"""InstrumentalSelection should accept 'with_backing' as valid selection."""
from backend.models.requests import InstrumentalSelection
selection = InstrumentalSelection(selection="with_backing")
assert selection.selection == "with_backing"
def test_instrumental_selection_rejects_invalid(self):
"""InstrumentalSelection should reject invalid values."""
from backend.models.requests import InstrumentalSelection
from pydantic import ValidationError
with pytest.raises(ValidationError):
InstrumentalSelection(selection="invalid_option")
-337
View File
@@ -1,337 +0,0 @@
"""
Unit tests for the test-webhook endpoint in internal.py.
The test-webhook endpoint allows E2E tests to simulate Stripe webhook events
without requiring valid Stripe signatures. It's admin-protected and requires
session IDs to start with "e2e-test-" prefix for safety.
Tests cover:
- Credit purchase flow simulation
- Made-for-you order flow simulation
- Session ID prefix validation
- Idempotency checks
- Admin authentication requirement
"""
import os
import pytest
from datetime import datetime, UTC
from unittest.mock import MagicMock, AsyncMock, patch
from fastapi.testclient import TestClient
from backend.models.job import Job, JobStatus
# Set test admin token for auth
os.environ.setdefault('ADMIN_TOKENS', 'test-admin-token')
@pytest.fixture
def mock_services():
"""Create mock services for testing."""
user_service = MagicMock()
user_service.is_stripe_session_processed.return_value = False
user_service.add_credits.return_value = (True, 5, "Added 1 credits")
user_service.db = MagicMock()
email_service = MagicMock()
email_service.send_credits_added.return_value = True
email_service.send_made_for_you_order_confirmation.return_value = True
email_service.send_made_for_you_admin_notification.return_value = True
stripe_service = MagicMock()
stripe_service.handle_checkout_completed.return_value = (True, "test@example.com", 1, "Success")
return {
'user_service': user_service,
'email_service': email_service,
'stripe_service': stripe_service,
}
@pytest.fixture
def client(mock_services):
"""Create TestClient with mocked services."""
mock_creds = MagicMock()
mock_creds.universe_domain = 'googleapis.com'
# Patch at the service module level where get_*_service functions are defined
with patch('backend.services.user_service.get_user_service', return_value=mock_services['user_service']), \
patch('backend.services.email_service.get_email_service', return_value=mock_services['email_service']), \
patch('backend.services.stripe_service.get_stripe_service', return_value=mock_services['stripe_service']), \
patch('backend.services.firestore_service.firestore'), \
patch('backend.services.storage_service.storage'), \
patch('google.auth.default', return_value=(mock_creds, 'test-project')):
from backend.main import app
yield TestClient(app)
@pytest.fixture
def auth_headers():
"""Auth headers for internal API testing."""
return {"Authorization": "Bearer test-admin-token"}
class TestTestWebhookSessionIdValidation:
"""Tests for session ID prefix validation."""
def test_rejects_session_id_without_prefix(self, client, auth_headers):
"""Test webhook rejects session IDs not starting with e2e-test-."""
response = client.post(
"/api/internal/test-webhook",
headers=auth_headers,
json={
"event_type": "checkout.session.completed",
"session_id": "cs_live_abc123", # Real Stripe format
"customer_email": "test@example.com",
"metadata": {"package_id": "1_credit", "credits": "1", "user_email": "test@example.com"}
}
)
assert response.status_code == 400
assert "e2e-test-" in response.json()["detail"]
def test_accepts_session_id_with_prefix(self, client, auth_headers):
"""Test webhook accepts session IDs starting with e2e-test-."""
response = client.post(
"/api/internal/test-webhook",
headers=auth_headers,
json={
"event_type": "checkout.session.completed",
"session_id": "e2e-test-12345",
"customer_email": "test@example.com",
"metadata": {"package_id": "1_credit", "credits": "1", "user_email": "test@example.com"}
}
)
assert response.status_code == 200
class TestTestWebhookAuthentication:
"""Tests for admin authentication requirement.
Note: Admin auth is already covered by test_internal_api.py for other endpoints.
The E2E security tests in stripe-webhook-security.spec.ts verify auth against
the real production endpoint. These unit tests focus on the endpoint-specific
logic rather than auth mechanics.
"""
@pytest.mark.skip(reason="Auth covered by test_internal_api.py and E2E tests - client fixture has mock auth")
def test_requires_auth(self, client):
"""Test webhook requires authentication."""
response = client.post(
"/api/internal/test-webhook",
json={
"event_type": "checkout.session.completed",
"session_id": "e2e-test-12345",
"customer_email": "test@example.com",
"metadata": {"package_id": "1_credit", "credits": "1"}
}
)
assert response.status_code == 401
@pytest.mark.skip(reason="Auth covered by test_internal_api.py and E2E tests - client fixture has mock auth")
def test_rejects_invalid_token(self, client):
"""Test webhook rejects invalid admin token."""
response = client.post(
"/api/internal/test-webhook",
headers={"Authorization": "Bearer invalid-token"},
json={
"event_type": "checkout.session.completed",
"session_id": "e2e-test-12345",
"customer_email": "test@example.com",
"metadata": {"package_id": "1_credit", "credits": "1"}
}
)
assert response.status_code == 401
class TestTestWebhookCreditPurchase:
"""Tests for credit purchase flow simulation."""
def test_credit_purchase_adds_credits(self, client, auth_headers, mock_services):
"""Test credit purchase adds credits to user account."""
response = client.post(
"/api/internal/test-webhook",
headers=auth_headers,
json={
"event_type": "checkout.session.completed",
"session_id": "e2e-test-credit-purchase-1",
"customer_email": "test@example.com",
"metadata": {
"package_id": "1_credit",
"credits": "1",
"user_email": "test@example.com"
}
}
)
assert response.status_code == 200
data = response.json()
assert data["status"] == "processed"
assert data["credits_added"] == 1
assert data["new_balance"] == 5
def test_credit_purchase_response_format(self, client, auth_headers):
"""Test credit purchase returns expected response format."""
response = client.post(
"/api/internal/test-webhook",
headers=auth_headers,
json={
"event_type": "checkout.session.completed",
"session_id": "e2e-test-format-check",
"customer_email": "test@example.com",
"metadata": {
"package_id": "1_credit",
"credits": "1",
"user_email": "test@example.com"
}
}
)
assert response.status_code == 200
data = response.json()
# Check required fields
assert "status" in data
assert "message" in data
# Credits-specific fields
assert "credits_added" in data or data["status"] == "error"
assert "new_balance" in data or data["status"] == "error"
class TestTestWebhookIdempotency:
"""Tests for idempotency checks."""
def test_duplicate_session_returns_already_processed(self, auth_headers, mock_services):
"""Test duplicate session IDs return already_processed status."""
mock_creds = MagicMock()
mock_creds.universe_domain = 'googleapis.com'
# Set up user_service to return already processed
mock_services['user_service'].is_stripe_session_processed.return_value = True
with patch('backend.services.user_service.get_user_service', return_value=mock_services['user_service']), \
patch('backend.services.email_service.get_email_service', return_value=mock_services['email_service']), \
patch('backend.services.stripe_service.get_stripe_service', return_value=mock_services['stripe_service']), \
patch('backend.services.firestore_service.firestore'), \
patch('backend.services.storage_service.storage'), \
patch('google.auth.default', return_value=(mock_creds, 'test-project')):
from backend.main import app
client = TestClient(app)
response = client.post(
"/api/internal/test-webhook",
headers=auth_headers,
json={
"event_type": "checkout.session.completed",
"session_id": "e2e-test-duplicate",
"customer_email": "test@example.com",
"metadata": {
"package_id": "1_credit",
"credits": "1",
"user_email": "test@example.com"
}
}
)
assert response.status_code == 200
data = response.json()
assert data["status"] == "already_processed"
class TestTestWebhookEventTypes:
"""Tests for different event types."""
def test_unsupported_event_type_returns_error(self, client, auth_headers):
"""Test unsupported event types return error status."""
response = client.post(
"/api/internal/test-webhook",
headers=auth_headers,
json={
"event_type": "customer.subscription.created", # Not supported
"session_id": "e2e-test-unsupported",
"customer_email": "test@example.com",
"metadata": {}
}
)
assert response.status_code == 200
data = response.json()
assert data["status"] == "error"
assert "unsupported" in data["message"].lower()
def test_checkout_session_completed_is_supported(self, client, auth_headers):
"""Test checkout.session.completed event type is supported."""
response = client.post(
"/api/internal/test-webhook",
headers=auth_headers,
json={
"event_type": "checkout.session.completed",
"session_id": "e2e-test-supported",
"customer_email": "test@example.com",
"metadata": {
"package_id": "1_credit",
"credits": "1",
"user_email": "test@example.com"
}
}
)
assert response.status_code == 200
data = response.json()
# Should not return unsupported error
assert data["status"] != "error" or "unsupported" not in data["message"].lower()
class TestTestWebhookRequestValidation:
"""Tests for request validation."""
def test_requires_event_type(self, client, auth_headers):
"""Test webhook requires event_type field."""
response = client.post(
"/api/internal/test-webhook",
headers=auth_headers,
json={
"session_id": "e2e-test-no-event-type",
"customer_email": "test@example.com",
"metadata": {}
}
)
assert response.status_code == 422
def test_requires_session_id(self, client, auth_headers):
"""Test webhook requires session_id field."""
response = client.post(
"/api/internal/test-webhook",
headers=auth_headers,
json={
"event_type": "checkout.session.completed",
"customer_email": "test@example.com",
"metadata": {}
}
)
assert response.status_code == 422
def test_requires_customer_email(self, client, auth_headers):
"""Test webhook requires customer_email field."""
response = client.post(
"/api/internal/test-webhook",
headers=auth_headers,
json={
"event_type": "checkout.session.completed",
"session_id": "e2e-test-no-email",
"metadata": {}
}
)
assert response.status_code == 422
def test_requires_metadata(self, client, auth_headers):
"""Test webhook requires metadata field."""
response = client.post(
"/api/internal/test-webhook",
headers=auth_headers,
json={
"event_type": "checkout.session.completed",
"session_id": "e2e-test-no-metadata",
"customer_email": "test@example.com",
}
)
assert response.status_code == 422
+23 -26
View File
@@ -140,20 +140,13 @@ class TestUserEmailExtraction:
with patch('backend.api.routes.file_upload.job_manager') as mock_jm, \
patch('backend.api.routes.file_upload.worker_service') as mock_worker, \
patch('backend.api.routes.file_upload.get_credential_manager') as mock_cred, \
patch('backend.api.routes.file_upload.get_youtube_download_service') as mock_yt:
patch('backend.api.routes.file_upload.get_credential_manager') as mock_cred:
mock_job = Mock()
mock_job.job_id = "test-job-456"
mock_jm.create_job.return_value = mock_job
mock_cred.return_value.check_youtube_credentials.return_value = Mock(status=Mock(value="valid"))
# Mock YouTube download service to return a GCS path
from unittest.mock import AsyncMock
mock_yt_service = Mock()
mock_yt_service.download = AsyncMock(return_value="uploads/test-job-456/audio/test.webm")
mock_yt.return_value = mock_yt_service
await create_job_from_url(
request=mock_request,
background_tasks=mock_background_tasks,
@@ -377,13 +370,12 @@ class TestAudioSearchCacheIndependence:
assert 'state_data' in source, \
"search_audio must use state_data for persistence"
def test_download_code_uses_youtube_download_service(self):
def test_download_code_has_youtube_direct_download_branch(self):
"""
Verify that _download_and_start_processing uses YouTubeDownloadService.
Verify that _download_and_start_processing has the YouTube direct download path.
After the consolidation refactor (2025-01), all YouTube downloads go through
YouTubeDownloadService instead of using download_from_url directly. This
ensures consistent handling with remote flacfetch support.
This checks that the code path exists for downloading YouTube audio directly
using the URL from state_data, avoiding the in-memory cache dependency.
"""
from backend.api.routes import audio_search
import inspect
@@ -391,28 +383,33 @@ class TestAudioSearchCacheIndependence:
# Get the source of _download_and_start_processing
source = inspect.getsource(audio_search._download_and_start_processing)
# Verify YouTubeDownloadService is used for YouTube
# Verify the YouTube direct download branch exists
assert "source_name == 'YouTube'" in source, \
"_download_and_start_processing must check for YouTube source"
assert 'youtube_service' in source or 'get_youtube_download_service' in source, \
"_download_and_start_processing must use YouTubeDownloadService for YouTube"
assert 'download_url' in source, \
"_download_and_start_processing must use download_url from state_data"
assert 'download_from_url' in source, \
"_download_and_start_processing must call download_from_url for YouTube"
def test_youtube_download_service_import_exists(self):
def test_youtube_download_branch_avoids_cache_dependency(self):
"""
Verify that audio_search imports YouTubeDownloadService.
This ensures the consolidated YouTube download path is properly wired.
Verify that the YouTube download path uses URL from state_data,
not the in-memory cache that doesn't persist across Cloud Run instances.
"""
from backend.api.routes import audio_search
import inspect
source = inspect.getsource(audio_search)
source = inspect.getsource(audio_search._download_and_start_processing)
# Verify YouTubeDownloadService is imported
assert 'get_youtube_download_service' in source, \
"audio_search must import get_youtube_download_service"
assert 'YouTubeDownloadError' in source, \
"audio_search must import YouTubeDownloadError"
# The fix specifically uses selected.get('url') which comes from state_data
# rather than audio_search_service cache
assert "selected.get('url')" in source or "download_url" in source, \
"YouTube download must use URL from selected result (state_data)"
# Verify it doesn't rely on cache for YouTube
# The branch is: elif source_name == 'YouTube' and download_url:
assert "YouTube" in source and "download_url" in source, \
"YouTube branch must check for download_url availability"
class TestTranscriptionTimeout:
-72
View File
@@ -496,78 +496,6 @@ class TestJobDeletion:
mock_firestore_service.delete_job.assert_called_once_with("test123")
class TestDeleteStateDataKeys:
"""Tests for delete_state_data_key and delete_state_data_keys methods.
These methods are used to clear worker progress keys from state_data,
which is critical for allowing workers to re-run after admin resets.
"""
def test_delete_state_data_key_success(self, job_manager, mock_firestore_service):
"""Test deleting a single key from state_data."""
# Setup mock to return success
mock_firestore_service.db = Mock()
mock_job_ref = Mock()
mock_firestore_service.db.collection.return_value.document.return_value = mock_job_ref
result = job_manager.delete_state_data_key("test123", "render_progress")
assert result is True
# Verify update was called with DELETE_FIELD for the key
mock_job_ref.update.assert_called_once()
call_args = mock_job_ref.update.call_args[0][0]
assert "state_data.render_progress" in call_args
def test_delete_state_data_key_handles_exception(self, job_manager, mock_firestore_service):
"""Test that delete_state_data_key handles exceptions gracefully."""
mock_firestore_service.db = Mock()
mock_job_ref = Mock()
mock_job_ref.update.side_effect = Exception("Firestore error")
mock_firestore_service.db.collection.return_value.document.return_value = mock_job_ref
result = job_manager.delete_state_data_key("test123", "render_progress")
assert result is False
def test_delete_state_data_keys_success(self, job_manager, mock_firestore_service):
"""Test deleting multiple keys from state_data in one operation."""
mock_firestore_service.db = Mock()
mock_job_ref = Mock()
mock_firestore_service.db.collection.return_value.document.return_value = mock_job_ref
keys_to_delete = ["render_progress", "video_progress", "encoding_progress"]
result = job_manager.delete_state_data_keys("test123", keys_to_delete)
assert result == keys_to_delete
# Verify single update was called with all keys
mock_job_ref.update.assert_called_once()
call_args = mock_job_ref.update.call_args[0][0]
assert "state_data.render_progress" in call_args
assert "state_data.video_progress" in call_args
assert "state_data.encoding_progress" in call_args
def test_delete_state_data_keys_empty_list(self, job_manager, mock_firestore_service):
"""Test that empty list returns empty without calling Firestore."""
mock_firestore_service.db = Mock()
result = job_manager.delete_state_data_keys("test123", [])
assert result == []
# Should NOT call Firestore for empty list
mock_firestore_service.db.collection.assert_not_called()
def test_delete_state_data_keys_handles_exception(self, job_manager, mock_firestore_service):
"""Test that delete_state_data_keys handles exceptions gracefully."""
mock_firestore_service.db = Mock()
mock_job_ref = Mock()
mock_job_ref.update.side_effect = Exception("Firestore error")
mock_firestore_service.db.collection.return_value.document.return_value = mock_job_ref
result = job_manager.delete_state_data_keys("test123", ["render_progress"])
assert result == []
if __name__ == "__main__":
pytest.main([__file__, "-v"])
@@ -421,199 +421,3 @@ class TestGetLocalEncodingService:
service = get_local_encoding_service(dry_run=True)
assert service.dry_run is True
class TestLocalEncodingServiceCountdownPadding:
"""Tests for countdown padding in LocalEncodingService."""
def test_encoding_config_accepts_countdown_padding(self):
"""EncodingConfig dataclass accepts countdown_padding_seconds."""
config = EncodingConfig(
title_video="/path/title.mov",
karaoke_video="/path/karaoke.mov",
instrumental_audio="/path/audio.flac",
countdown_padding_seconds=3.0,
)
assert config.countdown_padding_seconds == 3.0
def test_encoding_config_countdown_padding_default_none(self):
"""EncodingConfig countdown_padding_seconds defaults to None."""
config = EncodingConfig(
title_video="/path/title.mov",
karaoke_video="/path/karaoke.mov",
instrumental_audio="/path/audio.flac",
)
assert config.countdown_padding_seconds is None
def test_pad_audio_file_method_exists(self):
"""LocalEncodingService has _pad_audio_file method."""
service = LocalEncodingService()
assert hasattr(service, "_pad_audio_file")
assert callable(service._pad_audio_file)
@patch("subprocess.run")
def test_pad_audio_file_calls_ffmpeg_with_adelay(self, mock_run):
"""_pad_audio_file uses ffmpeg adelay filter."""
mock_run.return_value = MagicMock(returncode=0)
service = LocalEncodingService()
result = service._pad_audio_file(
"/input/audio.flac",
"/output/audio_padded.flac",
3.0
)
assert result == "/output/audio_padded.flac"
mock_run.assert_called_once()
call_args = mock_run.call_args
cmd = call_args[0][0]
assert "ffmpeg" in cmd
assert "-af" in cmd
# adelay should be 3000ms (3.0 * 1000) for both channels
af_index = cmd.index("-af")
assert "adelay=3000|3000" in cmd[af_index + 1]
def test_pad_audio_file_dry_run_skips_execution(self):
"""_pad_audio_file in dry run mode doesn't execute ffmpeg."""
service = LocalEncodingService(dry_run=True)
result = service._pad_audio_file(
"/input/audio.flac",
"/output/audio_padded.flac",
3.0
)
# Should return output path without error (no subprocess called)
assert result == "/output/audio_padded.flac"
@patch("subprocess.run")
def test_pad_audio_file_raises_on_ffmpeg_error(self, mock_run):
"""_pad_audio_file raises RuntimeError on FFmpeg failure."""
mock_run.return_value = MagicMock(returncode=1, stderr="ffmpeg: error processing audio")
service = LocalEncodingService()
with pytest.raises(RuntimeError, match="FFmpeg padding failed"):
service._pad_audio_file("/input/audio.flac", "/output/audio_padded.flac", 3.0)
@patch.object(LocalEncodingService, "_pad_audio_file")
@patch.object(LocalEncodingService, "remux_with_instrumental")
@patch.object(LocalEncodingService, "encode_lossless_mp4")
@patch("os.path.exists")
def test_encode_all_formats_pads_when_countdown_set(
self, mock_exists, mock_lossless, mock_remux, mock_pad
):
"""encode_all_formats pads instrumental when countdown_padding_seconds > 0."""
mock_exists.return_value = False # Padded file doesn't exist
mock_pad.return_value = "/output/instrumental (Padded).flac"
mock_remux.return_value = True
mock_lossless.return_value = True
service = LocalEncodingService()
config = EncodingConfig(
title_video="/input/title.mov",
karaoke_video="/input/karaoke.mov",
instrumental_audio="/input/instrumental.flac",
output_karaoke_mp4="/output/karaoke.mp4",
output_lossless_4k_mp4="/output/lossless_4k.mp4",
countdown_padding_seconds=3.0,
)
result = service.encode_all_formats(config)
assert result.success is True
# Should have called _pad_audio_file with 3.0 seconds
mock_pad.assert_called_once()
call_args = mock_pad.call_args
assert call_args[0][0] == "/input/instrumental.flac"
assert call_args[0][2] == 3.0
# remux should use the padded path
mock_remux.assert_called_once()
remux_args = mock_remux.call_args
assert "(Padded)" in remux_args[0][1]
@patch.object(LocalEncodingService, "_pad_audio_file")
@patch.object(LocalEncodingService, "remux_with_instrumental")
@patch.object(LocalEncodingService, "encode_lossless_mp4")
def test_encode_all_formats_skips_padding_when_no_countdown(
self, mock_lossless, mock_remux, mock_pad
):
"""encode_all_formats skips padding when countdown_padding_seconds is None."""
mock_remux.return_value = True
mock_lossless.return_value = True
service = LocalEncodingService()
config = EncodingConfig(
title_video="/input/title.mov",
karaoke_video="/input/karaoke.mov",
instrumental_audio="/input/instrumental.flac",
output_karaoke_mp4="/output/karaoke.mp4",
output_lossless_4k_mp4="/output/lossless_4k.mp4",
# No countdown_padding_seconds
)
result = service.encode_all_formats(config)
assert result.success is True
# Should NOT have called _pad_audio_file
mock_pad.assert_not_called()
# remux should use the original path
mock_remux.assert_called_once()
remux_args = mock_remux.call_args
assert remux_args[0][1] == "/input/instrumental.flac"
@patch.object(LocalEncodingService, "_pad_audio_file")
@patch.object(LocalEncodingService, "remux_with_instrumental")
@patch.object(LocalEncodingService, "encode_lossless_mp4")
def test_encode_all_formats_skips_already_padded(
self, mock_lossless, mock_remux, mock_pad
):
"""encode_all_formats skips padding if file has (Padded) in name."""
mock_remux.return_value = True
mock_lossless.return_value = True
service = LocalEncodingService()
config = EncodingConfig(
title_video="/input/title.mov",
karaoke_video="/input/karaoke.mov",
instrumental_audio="/input/instrumental (Padded).flac", # Already padded
output_karaoke_mp4="/output/karaoke.mp4",
output_lossless_4k_mp4="/output/lossless_4k.mp4",
countdown_padding_seconds=3.0,
)
result = service.encode_all_formats(config)
assert result.success is True
# Should NOT have called _pad_audio_file (already padded)
mock_pad.assert_not_called()
# remux should use the original path (which is already padded)
mock_remux.assert_called_once()
remux_args = mock_remux.call_args
assert remux_args[0][1] == "/input/instrumental (Padded).flac"
@patch.object(LocalEncodingService, "_pad_audio_file")
@patch.object(LocalEncodingService, "remux_with_instrumental")
@patch.object(LocalEncodingService, "encode_lossless_mp4")
def test_encode_all_formats_skips_padding_when_zero(
self, mock_lossless, mock_remux, mock_pad
):
"""encode_all_formats skips padding when countdown_padding_seconds is 0."""
mock_remux.return_value = True
mock_lossless.return_value = True
service = LocalEncodingService()
config = EncodingConfig(
title_video="/input/title.mov",
karaoke_video="/input/karaoke.mov",
instrumental_audio="/input/instrumental.flac",
output_karaoke_mp4="/output/karaoke.mp4",
output_lossless_4k_mp4="/output/lossless_4k.mp4",
countdown_padding_seconds=0, # Zero, should skip padding
)
result = service.encode_all_formats(config)
assert result.success is True
# Should NOT have called _pad_audio_file (zero padding)
mock_pad.assert_not_called()
+3 -322
View File
@@ -1332,10 +1332,7 @@ class TestHandleMadeForYouOrder:
mock_theme_service
):
"""
Orders with YouTube URL should download audio then trigger workers (no search).
Note: The download step was added in Jan 2026 to fix job 811ec4e5 failure.
See docs/LESSONS-LEARNED.md "YouTube URL Downloads Must Happen Before Workers".
Orders with YouTube URL should trigger workers directly, not search.
"""
from backend.api.routes.users import _handle_made_for_you_order
@@ -1352,18 +1349,11 @@ class TestHandleMadeForYouOrder:
mock_worker_service.trigger_audio_worker = AsyncMock()
mock_worker_service.trigger_lyrics_worker = AsyncMock()
# Mock YouTube download service to return a GCS path
mock_youtube_service = MagicMock()
mock_youtube_service.download = AsyncMock(
return_value="uploads/test-job-123/audio/test.webm"
)
with patch('backend.services.job_manager.JobManager', return_value=mock_job_manager), \
patch('backend.services.worker_service.get_worker_service', return_value=mock_worker_service), \
patch('backend.services.audio_search_service.get_audio_search_service') as mock_get_audio, \
patch('backend.services.storage_service.StorageService'), \
patch('backend.api.routes.users.get_theme_service', return_value=mock_theme_service), \
patch('backend.api.routes.users.get_youtube_download_service', return_value=mock_youtube_service):
patch('backend.api.routes.users.get_theme_service', return_value=mock_theme_service):
await _handle_made_for_you_order(
session_id="sess_123",
@@ -1377,10 +1367,7 @@ class TestHandleMadeForYouOrder:
# URL should be set
assert job_create_arg.url == youtube_metadata["youtube_url"]
# Audio should be downloaded first
mock_youtube_service.download.assert_called_once()
# Workers should be triggered after download
# Workers should be triggered directly
mock_worker_service.trigger_audio_worker.assert_called_once()
mock_worker_service.trigger_lyrics_worker.assert_called_once()
@@ -1448,312 +1435,6 @@ class TestHandleMadeForYouOrder:
"Default nomad theme should be applied to made-for-you jobs"
class TestMadeForYouYouTubeDownloadContract:
"""
CRITICAL: Contract tests for YouTube URL handling in made-for-you orders.
Bug context (2026-01-28): Job 811ec4e5 failed because the webhook handler
triggered workers directly for YouTube URLs without downloading audio first.
Workers require input_media_gcs_path to be set, which only happens after
YouTubeDownloadService.download() completes.
This test ensures the correct pattern:
1. YouTubeDownloadService.download() is called BEFORE workers
2. Job is updated with input_media_gcs_path BEFORE workers
3. THEN workers are triggered
See docs/LESSONS-LEARNED.md "YouTube URL Downloads Must Happen Before Workers"
"""
@pytest.fixture
def mock_job_manager(self):
"""Create mock JobManager with update tracking."""
manager = MagicMock()
mock_job = MagicMock()
mock_job.job_id = "test-job-123"
manager.create_job.return_value = mock_job
return manager
@pytest.fixture
def mock_email_service(self):
"""Create mock email service."""
service = MagicMock()
service.send_email.return_value = True
return service
@pytest.fixture
def mock_user_service(self):
"""Create mock user service."""
service = MagicMock()
service._mark_stripe_session_processed.return_value = None
mock_admin_login = MagicMock()
mock_admin_login.token = "test-admin-login-token-123"
service.create_admin_login_token.return_value = mock_admin_login
return service
@pytest.fixture
def mock_theme_service(self):
"""Create mock theme service."""
service = MagicMock()
service.get_default_theme_id.return_value = "nomad"
return service
@pytest.fixture
def youtube_order_metadata(self):
"""Order metadata with YouTube URL."""
return {
"order_type": "made_for_you",
"customer_email": "customer@example.com",
"artist": "Test Artist",
"title": "Test Song",
"source_type": "youtube",
"youtube_url": "https://youtube.com/watch?v=abc123",
}
@pytest.mark.asyncio
async def test_youtube_url_downloads_before_workers(
self, mock_job_manager, mock_email_service, mock_user_service,
mock_theme_service, youtube_order_metadata
):
"""
CRITICAL: YouTube audio MUST be downloaded before workers are triggered.
The correct flow is:
1. YouTubeDownloadService.download() returns GCS path
2. Job is updated with input_media_gcs_path
3. Workers are triggered
Without this, workers fail with "Failed to download audio file" because
input_media_gcs_path is null.
"""
from backend.api.routes.users import _handle_made_for_you_order
# Track call order to verify sequence
call_order = []
# Mock YouTubeDownloadService to return a GCS path
mock_youtube_service = MagicMock()
mock_youtube_service.download = AsyncMock(
return_value="uploads/test-job-123/audio/test.webm",
side_effect=lambda **kwargs: (
call_order.append('youtube_download'),
"uploads/test-job-123/audio/test.webm"
)[1]
)
# Mock worker service and track when workers are triggered
mock_worker_service = MagicMock()
mock_worker_service.trigger_audio_worker = AsyncMock(
side_effect=lambda x: call_order.append('audio_worker')
)
mock_worker_service.trigger_lyrics_worker = AsyncMock(
side_effect=lambda x: call_order.append('lyrics_worker')
)
# Track when input_media_gcs_path is set
original_update_job = mock_job_manager.update_job
def tracking_update_job(job_id, data):
if isinstance(data, dict) and 'input_media_gcs_path' in data:
call_order.append('gcs_path_set')
return original_update_job(job_id, data) if callable(original_update_job) else None
mock_job_manager.update_job = MagicMock(side_effect=tracking_update_job)
with patch('backend.services.job_manager.JobManager', return_value=mock_job_manager), \
patch('backend.services.worker_service.get_worker_service', return_value=mock_worker_service), \
patch('backend.services.storage_service.StorageService'), \
patch('backend.api.routes.users.get_theme_service', return_value=mock_theme_service), \
patch('backend.api.routes.users.get_youtube_download_service', return_value=mock_youtube_service):
await _handle_made_for_you_order(
session_id="sess_123",
metadata=youtube_order_metadata,
user_service=mock_user_service,
email_service=mock_email_service,
)
# CRITICAL: YouTubeDownloadService.download MUST be called for YouTube URLs
mock_youtube_service.download.assert_called_once()
# CRITICAL: GCS path MUST be set BEFORE workers are triggered
assert 'gcs_path_set' in call_order, \
"input_media_gcs_path must be set for YouTube URL orders. " \
f"Call order was: {call_order}"
gcs_path_index = call_order.index('gcs_path_set')
worker_indices = [i for i, x in enumerate(call_order)
if x in ('audio_worker', 'lyrics_worker')]
for worker_index in worker_indices:
assert gcs_path_index < worker_index, \
f"input_media_gcs_path must be set BEFORE workers are triggered. " \
f"Call order was: {call_order}"
@pytest.mark.asyncio
async def test_youtube_download_called_with_correct_params(
self, mock_job_manager, mock_email_service, mock_user_service,
mock_theme_service, youtube_order_metadata
):
"""
YouTubeDownloadService.download must be called with job_id, url, artist, and title.
"""
from backend.api.routes.users import _handle_made_for_you_order
mock_youtube_service = MagicMock()
mock_youtube_service.download = AsyncMock(
return_value="uploads/test-job-123/audio/test.webm"
)
mock_worker_service = MagicMock()
mock_worker_service.trigger_audio_worker = AsyncMock()
mock_worker_service.trigger_lyrics_worker = AsyncMock()
with patch('backend.services.job_manager.JobManager', return_value=mock_job_manager), \
patch('backend.services.worker_service.get_worker_service', return_value=mock_worker_service), \
patch('backend.services.storage_service.StorageService'), \
patch('backend.api.routes.users.get_theme_service', return_value=mock_theme_service), \
patch('backend.api.routes.users.get_youtube_download_service', return_value=mock_youtube_service):
await _handle_made_for_you_order(
session_id="sess_123",
metadata=youtube_order_metadata,
user_service=mock_user_service,
email_service=mock_email_service,
)
# Verify download was called with expected parameters
mock_youtube_service.download.assert_called_once()
call_kwargs = mock_youtube_service.download.call_args.kwargs
assert call_kwargs.get('url') == youtube_order_metadata['youtube_url'], \
"download must be called with the YouTube URL"
assert call_kwargs.get('job_id') == "test-job-123", \
"download must be called with the job_id"
assert call_kwargs.get('artist') == youtube_order_metadata['artist'], \
"download must be called with the artist"
assert call_kwargs.get('title') == youtube_order_metadata['title'], \
"download must be called with the title"
@pytest.mark.asyncio
async def test_youtube_download_failure_marks_job_failed(
self, mock_job_manager, mock_email_service, mock_user_service,
mock_theme_service, youtube_order_metadata
):
"""
When YouTube download fails, job should be transitioned to FAILED status.
Flow in _handle_made_for_you_order:
1. Inner except catches YouTubeDownloadError
2. Transitions job to FAILED status
3. Re-raises the exception
4. Outer except catches it and sends admin notification email
5. Outer handler swallows the exception (doesn't propagate to caller)
"""
from backend.api.routes.users import _handle_made_for_you_order
from backend.services.youtube_download_service import YouTubeDownloadError
from backend.models.job import JobStatus
# Mock YouTubeDownloadService to raise an error
mock_youtube_service = MagicMock()
mock_youtube_service.download = AsyncMock(
side_effect=YouTubeDownloadError("Video unavailable")
)
mock_worker_service = MagicMock()
mock_worker_service.trigger_audio_worker = AsyncMock()
mock_worker_service.trigger_lyrics_worker = AsyncMock()
with patch('backend.services.job_manager.JobManager', return_value=mock_job_manager), \
patch('backend.services.worker_service.get_worker_service', return_value=mock_worker_service), \
patch('backend.services.storage_service.StorageService'), \
patch('backend.api.routes.users.get_theme_service', return_value=mock_theme_service), \
patch('backend.api.routes.users.get_youtube_download_service', return_value=mock_youtube_service):
# Inner handler marks job failed and re-raises; outer handler sends notification
await _handle_made_for_you_order(
session_id="sess_123",
metadata=youtube_order_metadata,
user_service=mock_user_service,
email_service=mock_email_service,
)
# Verify job was transitioned to FAILED
mock_job_manager.transition_to_state.assert_called()
transition_call = mock_job_manager.transition_to_state.call_args
assert transition_call.kwargs.get('new_status') == JobStatus.FAILED, \
"Job should be transitioned to FAILED on download error"
assert "YouTube download failed" in transition_call.kwargs.get('message', ''), \
"Error message should indicate YouTube download failure"
# Workers should NOT be triggered when download fails
mock_worker_service.trigger_audio_worker.assert_not_called()
mock_worker_service.trigger_lyrics_worker.assert_not_called()
# Admin should receive error notification email
mock_email_service.send_email.assert_called_once()
email_call = mock_email_service.send_email.call_args
assert "[FAILED]" in email_call.kwargs.get('subject', ''), \
"Error notification email should have [FAILED] in subject"
class TestMadeForYouYouTubeImports:
"""
AST-based import validation to prevent missing dependency bugs.
Verifies that _handle_made_for_you_order imports and uses the correct
services for YouTube URL handling.
"""
def test_handler_imports_youtube_download_service(self):
"""
Verify _handle_made_for_you_order uses YouTubeDownloadService for YouTube URLs.
This is a regression guard that uses source inspection to verify the
handler imports the required service. If this test fails, YouTube URL
orders will fail at runtime because the download step is missing.
"""
import inspect
from backend.api.routes.users import _handle_made_for_you_order
source = inspect.getsource(_handle_made_for_you_order)
# Must use YouTubeDownloadService for YouTube URLs
assert 'get_youtube_download_service' in source or 'YouTubeDownloadService' in source, \
"_handle_made_for_you_order must use YouTubeDownloadService for YouTube URLs. " \
"Without this, workers are triggered without downloading audio first, " \
"causing 'Failed to download audio file' errors."
def test_handler_calls_download_for_youtube_urls(self):
"""
Verify the handler has a code path that calls download() for YouTube URLs.
"""
import inspect
from backend.api.routes.users import _handle_made_for_you_order
source = inspect.getsource(_handle_made_for_you_order)
# The handler must have download logic for YouTube URLs
# Either calling .download() directly or await youtube_service.download()
has_download_call = '.download(' in source or 'download(' in source
assert has_download_call, \
"_handle_made_for_you_order must call download() for YouTube URLs. " \
"The current code triggers workers directly without downloading."
def test_users_module_imports_youtube_service(self):
"""
Verify the users module imports YouTubeDownloadService.
"""
import inspect
from backend.api.routes import users
source = inspect.getsource(users)
# Module must import the YouTube download service
assert 'get_youtube_download_service' in source, \
"backend.api.routes.users must import get_youtube_download_service. " \
"This is required for handling YouTube URL made-for-you orders."
class TestMadeForYouDistributionSettings:
"""
Tests for distribution settings on made-for-you jobs.
+10 -12
View File
@@ -812,21 +812,19 @@ class TestPrepCompleteStatus:
valid_transitions = STATE_TRANSITIONS.get(JobStatus.REVIEW_COMPLETE, [])
assert JobStatus.PREP_COMPLETE in valid_transitions
def test_state_transitions_prep_complete_to_awaiting_review(self):
"""Test that PREP_COMPLETE can transition to AWAITING_REVIEW (to continue combined review)."""
def test_state_transitions_prep_complete_to_awaiting_instrumental(self):
"""Test that PREP_COMPLETE can transition to AWAITING_INSTRUMENTAL_SELECTION."""
from backend.models.job import STATE_TRANSITIONS
valid_transitions = STATE_TRANSITIONS.get(JobStatus.PREP_COMPLETE, [])
assert JobStatus.AWAITING_REVIEW in valid_transitions
def test_state_transitions_rendering_video_to_instrumental_selected(self):
"""Test that RENDERING_VIDEO transitions directly to INSTRUMENTAL_SELECTED (combined review flow)."""
assert JobStatus.AWAITING_INSTRUMENTAL_SELECTION in valid_transitions
def test_state_transitions_pending_to_awaiting_instrumental(self):
"""Test that PENDING can transition to AWAITING_INSTRUMENTAL_SELECTION (for finalise-only)."""
from backend.models.job import STATE_TRANSITIONS
valid_transitions = STATE_TRANSITIONS.get(JobStatus.RENDERING_VIDEO, [])
assert JobStatus.INSTRUMENTAL_SELECTED in valid_transitions
# AWAITING_INSTRUMENTAL_SELECTION is no longer a valid destination
assert JobStatus.AWAITING_INSTRUMENTAL_SELECTION not in valid_transitions
valid_transitions = STATE_TRANSITIONS.get(JobStatus.PENDING, [])
assert JobStatus.AWAITING_INSTRUMENTAL_SELECTION in valid_transitions
def test_critical_statuses_include_prep_complete(self):
"""Test that PREP_COMPLETE is in the list of job statuses."""
@@ -1,89 +0,0 @@
"""
Integration tests for render_video_worker that exercise actual code paths.
These tests verify the worker can execute without import errors and follows
the correct flow, catching bugs that mocked unit tests miss.
Key difference from test_internal_api.py:
- test_internal_api.py mocks process_render_video entirely (never executes real code)
- These tests call the REAL process_render_video function with minimal mocking
"""
import pytest
import os
import json
import tempfile
from datetime import datetime, UTC
from unittest.mock import MagicMock, AsyncMock, patch, Mock
from pathlib import Path
from backend.models.job import Job, JobStatus
class TestRenderVideoWorkerImports:
"""Test that render_video_worker can import all its dependencies."""
def test_can_import_render_video_worker_module(self):
"""Test that the render_video_worker module imports without errors."""
# This catches import-time errors
from backend.workers import render_video_worker
assert render_video_worker is not None
def test_can_import_process_render_video_function(self):
"""Test that process_render_video function exists and is importable."""
from backend.workers.render_video_worker import process_render_video
assert callable(process_render_video)
def test_render_video_worker_has_no_undefined_imports(self):
"""
Test that all imports used in render_video_worker exist.
This test catches errors like:
- from backend.workers.video_worker import run_video_worker
when run_video_worker doesn't exist.
"""
import ast
import inspect
from backend.workers import render_video_worker
# Get the source code
source = inspect.getsource(render_video_worker)
tree = ast.parse(source)
# Find all imports
imports = []
for node in ast.walk(tree):
if isinstance(node, ast.ImportFrom):
module = node.module
for alias in node.names:
imports.append((module, alias.name))
# Try to actually import each one
failed_imports = []
for module, name in imports:
if module and not module.startswith('backend'):
# Skip non-backend modules (standard library, third-party)
continue
try:
if module:
mod = __import__(module, fromlist=[name])
if not hasattr(mod, name):
failed_imports.append(f"from {module} import {name} - {name} not found in module")
except (ImportError, AttributeError) as e:
failed_imports.append(f"from {module} import {name} - {e}")
if failed_imports:
pytest.fail(
"render_video_worker.py has invalid imports:\n" +
"\n".join(failed_imports)
)
# NOTE: Flow tests removed - they require perfect mocking of complex data structures
# which is fragile and doesn't provide much value over the AST-based import validation.
#
# The import validation test above is sufficient to catch import errors like:
# - from backend.workers.video_worker import run_video_worker
#
# For end-to-end testing of the full worker flow, use the emulator integration tests
# in backend/tests/emulator/ which use real Firestore/GCS emulators.
-160
View File
@@ -280,163 +280,3 @@ class TestRetryEndpoint:
has_input = job.input_media_gcs_path or job.url
assert not has_input
class TestCompleteReviewWithInstrumentalSelection:
"""Tests for complete_review endpoint with combined review flow.
The combined review flow allows users to submit both lyrics corrections
AND instrumental selection in a single request. This tests:
1. instrumental_selection is properly stored in state_data
2. Backward compatibility (no body still works)
3. Request validation
"""
def test_complete_review_request_model_accepts_valid_selections(self):
"""Test CompleteReviewRequest accepts valid instrumental_selection values."""
from backend.models.requests import CompleteReviewRequest
# Valid selections
for selection in ['clean', 'with_backing', 'custom']:
req = CompleteReviewRequest(instrumental_selection=selection)
assert req.instrumental_selection == selection
def test_complete_review_request_model_accepts_none(self):
"""Test CompleteReviewRequest accepts None for backward compatibility."""
from backend.models.requests import CompleteReviewRequest
req = CompleteReviewRequest()
assert req.instrumental_selection is None
req = CompleteReviewRequest(instrumental_selection=None)
assert req.instrumental_selection is None
def test_complete_review_request_model_rejects_invalid_selection(self):
"""Test CompleteReviewRequest rejects invalid instrumental_selection values."""
from backend.models.requests import CompleteReviewRequest
from pydantic import ValidationError
with pytest.raises(ValidationError) as exc_info:
CompleteReviewRequest(instrumental_selection='invalid_option')
assert 'instrumental_selection' in str(exc_info.value)
def test_instrumental_selection_stored_in_state_data(self):
"""Test that instrumental_selection is stored in job state_data.
This is critical for the combined review flow - the selection must
be persisted so render_video_worker can use it.
"""
# Create a sample job in AWAITING_REVIEW state
job = Job(
job_id="test-combined-review",
status=JobStatus.AWAITING_REVIEW,
created_at=datetime.now(UTC),
updated_at=datetime.now(UTC),
artist="Test Artist",
title="Test Song",
state_data={}
)
# Simulate what the endpoint does when instrumental_selection is provided
instrumental_selection = 'clean'
job.state_data['instrumental_selection'] = instrumental_selection
# Verify it's stored correctly
assert job.state_data.get('instrumental_selection') == 'clean'
def test_state_data_instrumental_selection_used_by_render_worker(self):
"""Document that render_video_worker reads instrumental_selection from state_data.
The render_video_worker no longer waits for AWAITING_INSTRUMENTAL_SELECTION.
Instead, it reads the selection from state_data['instrumental_selection']
which was set during complete_review.
"""
# This documents the expected flow:
# 1. User completes combined review with instrumental_selection='with_backing'
# 2. complete_review stores it in state_data
# 3. render_video_worker reads it and uses it for final video
job = Job(
job_id="test-flow",
status=JobStatus.REVIEW_COMPLETE,
created_at=datetime.now(UTC),
updated_at=datetime.now(UTC),
artist="Test",
title="Song",
state_data={
'instrumental_selection': 'with_backing'
}
)
# The worker should be able to read the selection
selection = job.state_data.get('instrumental_selection')
assert selection == 'with_backing'
assert selection in ['clean', 'with_backing', 'custom']
class TestScreensWorkerBackingVocalsAnalysis:
"""Tests documenting that backing vocals analysis now runs in screens_worker.
Previously, backing vocals analysis ran in render_video_worker AFTER review.
Now it runs in screens_worker BEFORE review, so the analysis data is available
when the user opens the combined review UI.
"""
def test_analysis_data_structure(self):
"""Document the expected backing_vocals_analysis structure in state_data."""
# This is the structure stored by screens_worker._analyze_backing_vocals()
expected_structure = {
'has_audible_content': True, # or False
'total_duration_seconds': 180.0,
'audible_segments': [
{
'start_seconds': 10.0,
'end_seconds': 20.0,
'duration_seconds': 10.0,
'avg_amplitude_db': -25.0,
'peak_amplitude_db': -20.0,
}
],
'recommended_selection': 'with_backing', # or 'clean' or 'review_needed'
'total_audible_duration_seconds': 30.0,
'audible_percentage': 16.67,
'silence_threshold_db': -40.0,
}
# Verify structure has expected keys
assert 'has_audible_content' in expected_structure
assert 'recommended_selection' in expected_structure
assert 'audible_segments' in expected_structure
def test_analysis_available_before_review(self):
"""Document that analysis is available when job enters AWAITING_REVIEW.
The screens_worker runs analysis before transitioning to AWAITING_REVIEW,
so the frontend can show the analysis in the combined review UI.
"""
# Job in AWAITING_REVIEW should have analysis data if stems exist
job = Job(
job_id="test-analysis",
status=JobStatus.AWAITING_REVIEW,
created_at=datetime.now(UTC),
updated_at=datetime.now(UTC),
artist="Test",
title="Song",
file_urls={
'stems': {
'backing_vocals': 'jobs/test/stems/backing_vocals.flac',
'instrumental_clean': 'jobs/test/stems/clean.flac',
}
},
state_data={
'backing_vocals_analysis': {
'has_audible_content': True,
'recommended_selection': 'with_backing',
}
}
)
# Frontend can read analysis from state_data
analysis = job.state_data.get('backing_vocals_analysis', {})
assert analysis.get('recommended_selection') is not None
+24 -302
View File
@@ -44,13 +44,11 @@ class TestJobStatusTransitionsForReview:
assert hasattr(JobStatus, 'RENDERING_VIDEO')
assert JobStatus.RENDERING_VIDEO.value == "rendering_video"
def test_rendering_video_transitions_to_instrumental_selected(self):
"""Test RENDERING_VIDEO -> INSTRUMENTAL_SELECTED is valid (combined review flow)."""
def test_rendering_video_transitions_to_instrumental(self):
"""Test RENDERING_VIDEO -> AWAITING_INSTRUMENTAL_SELECTION is valid."""
from backend.models.job import STATE_TRANSITIONS, JobStatus
valid_transitions = STATE_TRANSITIONS.get(JobStatus.RENDERING_VIDEO, [])
assert JobStatus.INSTRUMENTAL_SELECTED in valid_transitions
# AWAITING_INSTRUMENTAL_SELECTION is no longer a valid transition from RENDERING_VIDEO
assert JobStatus.AWAITING_INSTRUMENTAL_SELECTION not in valid_transitions
assert JobStatus.AWAITING_INSTRUMENTAL_SELECTION in valid_transitions
class TestStylesConfigRequirements:
@@ -218,65 +216,6 @@ class TestAddLyricsEndpoint:
assert callable(method)
class TestReviewResubmissionClearsWorkerProgress:
"""Tests for ensuring review submission clears worker progress keys.
When a job is re-reviewed (e.g., after a reset to awaiting_review), the worker
progress keys must be cleared to prevent workers from skipping due to
idempotency checks (checking {worker}_progress.stage == 'complete').
"""
def test_complete_review_clears_progress_keys(self):
"""Document that complete_review should clear worker progress keys.
The complete_review endpoint calls delete_state_data_keys with:
['render_progress', 'screens_progress', 'video_progress', 'encoding_progress']
This ensures that when a review is submitted, even if the job was
previously completed, workers will run fresh instead of skipping.
"""
# This test documents the expected behavior.
# The actual integration test is in test_emulator_integration.py.
expected_keys_to_clear = [
'render_progress',
'screens_progress',
'video_progress',
'encoding_progress',
]
# Verify all worker progress keys that affect idempotency are included
assert 'render_progress' in expected_keys_to_clear
assert 'video_progress' in expected_keys_to_clear
assert 'encoding_progress' in expected_keys_to_clear
def test_idempotency_check_examines_progress_stage(self):
"""Document the idempotency check pattern used by workers.
Workers check state_data.{worker}_progress.stage == 'complete' to skip.
This is why clearing these keys is critical for re-processing.
"""
# Example of what stale state looks like (causes skip)
stale_state_data = {
'render_progress': {'stage': 'complete'}, # Worker will skip!
'video_progress': {'stage': 'complete'},
}
# After clearing, the keys should not exist
clean_state_data = {}
# Verify worker would NOT skip after clearing
render_progress = clean_state_data.get('render_progress', {})
assert render_progress.get('stage') != 'complete'
def test_job_manager_has_delete_state_data_keys_method(self):
"""Verify JobManager has the method needed for clearing keys."""
from backend.services.job_manager import JobManager
jm = JobManager.__new__(JobManager) # Create without __init__
assert hasattr(jm, 'delete_state_data_keys')
assert callable(getattr(jm, 'delete_state_data_keys'))
class TestPreviewStyleLoading:
"""Tests for the unified style loader used in preview video generation.
@@ -350,18 +289,28 @@ class TestPreviewStyleLoading:
assert result_styles["karaoke"]["font_path"] != "/original/path/font.ttf"
assert "font.ttf" in result_styles["karaoke"]["font_path"]
def test_load_styles_from_gcs_requires_explicit_theme(self, tmp_path):
"""Test that load_styles_from_gcs requires explicit theme (Phase 2)."""
def test_load_styles_from_gcs_falls_back_to_minimal(self, tmp_path):
"""Test that minimal styles are used when job has no custom styles."""
import os
from karaoke_gen.style_loader import load_styles_from_gcs
# Phase 2: Calling with no custom styles should raise an error
with pytest.raises(ValueError, match="style_params_gcs_path is required"):
load_styles_from_gcs(
style_params_gcs_path=None, # No custom styles
style_assets={},
temp_dir=str(tmp_path),
download_func=lambda gcs_path, local_path: None, # Won't be called
)
# Call with no custom styles
styles_path, result_styles = load_styles_from_gcs(
style_params_gcs_path=None, # No custom styles
style_assets={},
temp_dir=str(tmp_path),
download_func=lambda gcs_path, local_path: None, # Won't be called
)
# Verify styles file was created
assert os.path.exists(styles_path)
# Should have karaoke section with minimal/default values
assert "karaoke" in result_styles
assert result_styles["karaoke"]["background_color"] == "#000000"
assert result_styles["karaoke"]["font"] == "Noto Sans"
# Minimal styles have background_image as None (default)
assert result_styles["karaoke"].get("background_image") is None
def test_asset_mapping_is_complete(self):
"""Verify all required asset mappings are defined in the unified style loader."""
@@ -386,230 +335,3 @@ class TestPreviewStyleLoading:
font_mappings = ASSET_KEY_MAPPINGS["font"]
assert isinstance(font_mappings, list)
assert ("karaoke", "font_path") in font_mappings
class TestInstrumentalAnalysisAudioUrls:
"""Tests for the audio_urls response from get_instrumental_analysis endpoint.
The frontend expects specific field names in the audio_urls dict.
This test documents and enforces the contract.
"""
def test_audio_url_field_names_match_frontend_expectations(self):
"""Verify audio_urls uses 'backing_vocals' not 'backing'.
The frontend InstrumentalSelector.tsx expects:
- audio_urls.clean
- audio_urls.with_backing
- audio_urls.original
- audio_urls.backing_vocals <-- NOT 'backing'
This field name mismatch caused backing vocals audio to never load
in cloud mode because the frontend looked for 'backing_vocals' but
the backend returned 'backing'.
"""
# Document the expected field names that frontend uses
expected_audio_url_keys = [
'clean', # Clean instrumental (vocals removed)
'with_backing', # Instrumental with backing vocals
'original', # Original audio file
'backing_vocals', # Backing vocals only stem (NOT 'backing')
]
# Verify 'backing' is NOT in the expected keys
assert 'backing' not in expected_audio_url_keys
assert 'backing_vocals' in expected_audio_url_keys
def test_frontend_stem_type_map_uses_backing_vocals(self):
"""Document that frontend maps 'backing' AudioType to 'backing_vocals' API field.
In InstrumentalSelector.tsx:
- STEM_TYPE_MAP['backing'] = 'backing_vocals'
- getAudioUrl() looks for urls.backing_vocals
- handleAudioChange() looks for urls.backing_vocals
The backend must return 'backing_vocals' to match.
"""
# Frontend expects these audio_urls keys in cloud mode
# (In local mode, it uses STEM_TYPE_MAP which has different values)
expected_cloud_mode_keys = {
'original': 'original',
'backing': 'backing_vocals', # AudioType 'backing' -> API field 'backing_vocals'
'clean': 'clean', # AudioType 'clean' -> API field 'clean'
'with_backing': 'with_backing',
}
# Verify the mapping for backing is 'backing_vocals'
assert expected_cloud_mode_keys['backing'] == 'backing_vocals'
assert expected_cloud_mode_keys['clean'] == 'clean'
class TestGetCorrectionDataPrioritizesUpdatedCorrections:
"""Tests for the correction data endpoint's handling of updated vs original corrections.
The bug: When a user edits lyrics in LyricsAnalyzer, the edits are saved to
corrections_updated.json. However, when InstrumentalSelector fetches correction
data via GET /api/review/{job_id}/correction-data, it was returning the ORIGINAL
corrections.json, ignoring the user's edits.
The fix: The endpoint now checks for corrections_updated.json first and returns
that if it exists, matching the behavior of render_video_worker.py.
"""
def test_endpoint_should_check_updated_corrections_first(self):
"""Document that get_correction_data should check for corrections_updated first.
The endpoint logic should be:
1. Check for corrections_updated in file_urls or direct GCS path
2. If exists, use it
3. Otherwise, fall back to original corrections.json
This matches the pattern in render_video_worker.py:128-142.
"""
# Document the expected file priority
file_priority = [
("corrections_updated", "jobs/{job_id}/lyrics/corrections_updated.json"),
("corrections", "jobs/{job_id}/lyrics/corrections.json"),
]
# Verify corrections_updated comes first
assert file_priority[0][0] == "corrections_updated"
assert file_priority[1][0] == "corrections"
def test_updated_corrections_file_path_format(self):
"""Document the expected file path for updated corrections."""
job_id = "test-job-123"
# The updated corrections path format
updated_path = f"jobs/{job_id}/lyrics/corrections_updated.json"
assert "corrections_updated.json" in updated_path
assert job_id in updated_path
def test_file_urls_key_for_updated_corrections(self):
"""Document the file_urls key used for updated corrections."""
# The file_urls structure for a job with updated corrections
file_urls = {
"lyrics": {
"corrections": "jobs/test-job/lyrics/corrections.json",
"corrections_updated": "jobs/test-job/lyrics/corrections_updated.json",
}
}
# The key for updated corrections is 'corrections_updated'
assert "corrections_updated" in file_urls["lyrics"]
# This is different from 'corrections' (the original)
assert "corrections" in file_urls["lyrics"]
assert file_urls["lyrics"]["corrections"] != file_urls["lyrics"]["corrections_updated"]
def test_render_video_worker_and_api_use_same_pattern(self):
"""Verify render_video_worker and get_correction_data use consistent logic.
Both should:
1. Try corrections_updated from file_urls first
2. Try direct GCS path for corrections_updated
3. Fall back to corrections from file_urls
4. Fall back to direct GCS path for corrections.json
This ensures the user's edits are used consistently.
"""
# render_video_worker.py pattern (lines 128-142):
render_worker_priority = [
"file_urls.lyrics.corrections_updated",
"file_urls.lyrics.corrections",
"jobs/{job_id}/lyrics/corrections_updated.json (GCS)",
"jobs/{job_id}/lyrics/corrections.json (GCS)",
]
# Verify corrections_updated is checked before corrections
assert "corrections_updated" in render_worker_priority[0]
assert "corrections_updated" not in render_worker_priority[1]
def test_correction_data_keys_that_frontend_edits(self):
"""Document which keys the frontend modifies during review.
The frontend (LyricsAnalyzer) sends only partial data when saving:
- corrections: List of corrections made
- corrected_segments: Updated segment data with user edits
The backend merges these with the original to preserve metadata,
original_segments, etc.
"""
# Keys that the frontend modifies
frontend_edited_keys = ["corrections", "corrected_segments"]
# Keys that are preserved from original
preserved_keys = [
"original_segments",
"metadata",
"reference_lyrics",
"anchor_sequences",
"gap_sequences",
]
# These should not overlap
assert not set(frontend_edited_keys) & set(preserved_keys)
class TestCorrectionFlowIntegration:
"""Integration tests documenting the full correction data flow.
These tests document the expected behavior of the combined review flow
where corrections must survive from LyricsAnalyzer through InstrumentalSelector
to the final video render.
"""
def test_correction_flow_endpoints(self):
"""Document the endpoints involved in the correction flow."""
# The flow uses these endpoints in order:
endpoints = [
("GET", "/api/review/{job_id}/correction-data", "Fetch corrections for LyricsAnalyzer"),
("POST", "/api/jobs/{job_id}/corrections", "Save user's edits from LyricsAnalyzer"),
("GET", "/api/review/{job_id}/correction-data", "Fetch corrections for InstrumentalSelector"),
("POST", "/api/review/{job_id}/complete", "Submit final review with instrumental selection"),
]
# The second GET must return the UPDATED corrections, not the original
assert endpoints[2][0] == "GET"
assert "correction-data" in endpoints[2][1]
def test_gcs_file_lifecycle_during_review(self):
"""Document the GCS file lifecycle during the review flow."""
job_id = "test-job-123"
# Initial state (after lyrics processing)
initial_files = {
f"jobs/{job_id}/lyrics/corrections.json": "Created by lyrics worker",
}
# After LyricsAnalyzer saves edits
after_lyrics_review = {
f"jobs/{job_id}/lyrics/corrections.json": "Still exists (original)",
f"jobs/{job_id}/lyrics/corrections_updated.json": "Created with user's edits",
}
# Verify corrections_updated.json is created during review
assert f"jobs/{job_id}/lyrics/corrections_updated.json" in after_lyrics_review
assert f"jobs/{job_id}/lyrics/corrections_updated.json" not in initial_files
def test_job_file_urls_updated_after_saving_corrections(self):
"""Document that file_urls is updated when corrections are saved."""
# Before saving corrections
file_urls_before = {
"lyrics": {
"corrections": "jobs/test-job/lyrics/corrections.json",
}
}
# After POST /api/jobs/{job_id}/corrections
file_urls_after = {
"lyrics": {
"corrections": "jobs/test-job/lyrics/corrections.json",
"corrections_updated": "jobs/test-job/lyrics/corrections_updated.json",
}
}
# The corrections_updated key should be added
assert "corrections_updated" not in file_urls_before["lyrics"]
assert "corrections_updated" in file_urls_after["lyrics"]
+37 -73
View File
@@ -157,83 +157,47 @@ class TestStorageService:
class TestWorkerService:
"""Test WorkerService worker triggering."""
"""Test WorkerService internal HTTP calls."""
@pytest.mark.asyncio
async def test_trigger_audio_worker(self):
"""Test triggering audio worker via Cloud Run Jobs."""
from backend.services.worker_service import WorkerService, reset_worker_service
reset_worker_service()
with patch('backend.services.worker_service.get_settings') as mock_settings:
mock_settings.return_value.admin_tokens = "test-token"
mock_settings.return_value.google_cloud_project = "test-project"
mock_settings.return_value.enable_cloud_tasks = False
mock_settings.return_value.gcp_region = "us-central1"
mock_settings.return_value.use_cloud_run_jobs_for_video = False
# Mock Cloud Run v2 Jobs client
mock_jobs_client = MagicMock()
mock_operation = MagicMock()
mock_operation.metadata = "test-metadata"
mock_jobs_client.run_job.return_value = mock_operation
mock_run_v2 = MagicMock()
mock_run_v2.JobsClient.return_value = mock_jobs_client
mock_run_v2.RunJobRequest = MagicMock()
mock_run_v2.RunJobRequest.Overrides = MagicMock()
mock_run_v2.RunJobRequest.Overrides.ContainerOverride = MagicMock()
with patch.dict('sys.modules', {'google.cloud.run_v2': mock_run_v2}):
service = WorkerService()
result = await service.trigger_audio_worker("test123")
# Verify Cloud Run Job was triggered
assert result is True
mock_jobs_client.run_job.assert_called_once()
# Verify correct job name was used
call_args = mock_run_v2.RunJobRequest.call_args
assert call_args is not None
assert "audio-separation-job" in str(call_args)
"""Test triggering audio worker via internal HTTP."""
with patch('backend.services.worker_service.httpx.AsyncClient') as mock_client_cls:
mock_client = AsyncMock()
mock_client_cls.return_value.__aenter__.return_value = mock_client
mock_response = Mock()
mock_response.status_code = 200
mock_client.post.return_value = mock_response
worker_service = get_worker_service()
await worker_service.trigger_audio_worker("test123")
# Verify HTTP POST was made
mock_client.post.assert_called_once()
call_args = mock_client.post.call_args
assert "/api/internal/workers/audio" in str(call_args)
@pytest.mark.asyncio
async def test_trigger_lyrics_worker(self):
"""Test triggering lyrics worker via Cloud Run Jobs."""
from backend.services.worker_service import WorkerService, reset_worker_service
reset_worker_service()
with patch('backend.services.worker_service.get_settings') as mock_settings:
mock_settings.return_value.admin_tokens = "test-token"
mock_settings.return_value.google_cloud_project = "test-project"
mock_settings.return_value.enable_cloud_tasks = False
mock_settings.return_value.gcp_region = "us-central1"
mock_settings.return_value.use_cloud_run_jobs_for_video = False
# Mock Cloud Run v2 Jobs client
mock_jobs_client = MagicMock()
mock_operation = MagicMock()
mock_operation.metadata = "test-metadata"
mock_jobs_client.run_job.return_value = mock_operation
mock_run_v2 = MagicMock()
mock_run_v2.JobsClient.return_value = mock_jobs_client
mock_run_v2.RunJobRequest = MagicMock()
mock_run_v2.RunJobRequest.Overrides = MagicMock()
mock_run_v2.RunJobRequest.Overrides.ContainerOverride = MagicMock()
with patch.dict('sys.modules', {'google.cloud.run_v2': mock_run_v2}):
service = WorkerService()
result = await service.trigger_lyrics_worker("test123")
# Verify Cloud Run Job was triggered
assert result is True
mock_jobs_client.run_job.assert_called_once()
# Verify correct job name was used
call_args = mock_run_v2.RunJobRequest.call_args
assert call_args is not None
assert "lyrics-transcription-job" in str(call_args)
"""Test triggering lyrics worker via internal HTTP."""
with patch('backend.services.worker_service.httpx.AsyncClient') as mock_client_cls:
mock_client = AsyncMock()
mock_client_cls.return_value.__aenter__.return_value = mock_client
mock_response = Mock()
mock_response.status_code = 200
mock_client.post.return_value = mock_response
worker_service = get_worker_service()
await worker_service.trigger_lyrics_worker("test123")
# Verify HTTP POST was made
mock_client.post.assert_called_once()
call_args = mock_client.post.call_args
assert "/api/internal/workers/lyrics" in str(call_args)
@pytest.mark.asyncio
async def test_trigger_screens_worker(self):
+31 -39
View File
@@ -119,70 +119,62 @@ class TestStyleHelper:
"""Test the backend style helper module."""
@pytest.mark.asyncio
async def test_style_config_raises_error_when_no_theme(self):
"""Test that StyleConfig raises error when no theme provided (Phase 2)."""
async def test_style_config_loads_defaults_when_no_custom_styles(self):
"""Test that StyleConfig returns defaults when no custom styles."""
pytest.importorskip("google.cloud.storage", reason="GCP libraries not available")
from backend.workers.style_helper import StyleConfig
from backend.workers.style_helper import StyleConfig, DEFAULT_INTRO_FORMAT
# Mock job with no style assets and no style_params_gcs_path
job = Mock()
job.job_id = "test-123"
job.style_assets = {}
job.style_params_gcs_path = None # Explicitly set to None
storage = Mock()
with tempfile.TemporaryDirectory() as temp_dir:
config = StyleConfig(job, storage, temp_dir)
# Phase 2: Should raise error when loading without theme
with pytest.raises(ValueError, match="style_params_gcs_path is required"):
await config.load()
await config.load()
assert not config.has_custom_styles()
intro_format = config.get_intro_format()
assert intro_format == DEFAULT_INTRO_FORMAT
@pytest.mark.asyncio
async def test_style_config_loads_complete_theme(self):
"""Test that StyleConfig loads and parses complete themes (Phase 2)."""
async def test_style_config_loads_custom_styles(self):
"""Test that StyleConfig loads and parses custom styles."""
pytest.importorskip("google.cloud.storage", reason="GCP libraries not available")
from backend.workers.style_helper import StyleConfig
from karaoke_gen.style_loader import (
DEFAULT_INTRO_STYLE,
DEFAULT_END_STYLE,
DEFAULT_KARAOKE_STYLE,
DEFAULT_CDG_STYLE,
)
with tempfile.TemporaryDirectory() as temp_dir:
# Create a SEPARATE source directory for the source style file
# (StyleConfig will use temp_dir/style/ for downloads, so we use source/)
source_dir = os.path.join(temp_dir, "source")
os.makedirs(source_dir, exist_ok=True)
# Create a COMPLETE theme (Phase 2 requirement)
# Create a mock style_params.json in the source directory
style_params = {
"intro": DEFAULT_INTRO_STYLE.copy(),
"end": DEFAULT_END_STYLE.copy(),
"karaoke": DEFAULT_KARAOKE_STYLE.copy(),
"cdg": DEFAULT_CDG_STYLE.copy(),
"intro": {
"background_color": "#FF0000",
"title_color": "#00FF00",
},
"cdg": {
"background_color": "#0000FF",
}
}
# Customize some values
style_params["intro"]["background_color"] = "#FF0000"
style_params["intro"]["title_color"] = "#00FF00"
style_params["cdg"]["instrumental_background"] = "custom_bg.png"
style_json_path = os.path.join(source_dir, "style_params.json")
with open(style_json_path, 'w') as f:
json.dump(style_params, f)
# Create a work directory for StyleConfig (separate from source)
work_dir = os.path.join(temp_dir, "work")
os.makedirs(work_dir, exist_ok=True)
# Mock job with style assets
job = Mock()
job.job_id = "test-123"
job.style_assets = {"style_params": "uploads/test/style_params.json"}
job.style_params_gcs_path = "uploads/test/style_params.json"
# Mock storage to "download" the file
storage = Mock()
def mock_download(gcs_path, local_path):
@@ -191,18 +183,18 @@ class TestStyleHelper:
os.makedirs(os.path.dirname(local_path), exist_ok=True)
shutil.copy(style_json_path, local_path)
storage.download_file = mock_download
config = StyleConfig(job, storage, work_dir)
await config.load()
assert config.has_custom_styles()
intro_format = config.get_intro_format()
assert intro_format['background_color'] == "#FF0000"
assert intro_format['title_color'] == "#00FF00"
cdg_styles = config.get_cdg_styles()
assert cdg_styles['instrumental_background'] == "custom_bg.png"
assert cdg_styles['background_color'] == "#0000FF"
class TestFileUploadEndpoint:
@@ -1034,196 +1034,3 @@ class TestInstrumentalSelectionEndToEnd:
assert captured_input is not None, "encode() should have been called"
assert captured_input.instrumental_selection == "with_backing", \
"Orchestrator must pass instrumental_selection to EncodingInput"
class TestOrchestratorCountdownPadding:
"""Tests for countdown padding flow through orchestrator.
These tests verify that countdown_padding_seconds flows correctly from:
job.state_data.lyrics_metadata -> OrchestratorConfig -> EncodingInput.options
This test class was added after discovering that the orchestrator path
did not propagate countdown_padding_seconds to the GCE encoding backend,
causing instrumental audio to not be padded for songs with countdown intros.
"""
def test_orchestrator_config_has_countdown_field(self):
"""Test that OrchestratorConfig dataclass includes countdown_padding_seconds."""
config = OrchestratorConfig(
job_id="test-job",
artist="Test Artist",
title="Test Title",
title_video_path="/path/title.mov",
karaoke_video_path="/path/karaoke.mov",
instrumental_audio_path="/path/audio.flac",
countdown_padding_seconds=3.0,
)
assert config.countdown_padding_seconds == 3.0
def test_orchestrator_config_countdown_defaults_to_none(self):
"""Test that countdown_padding_seconds defaults to None."""
config = OrchestratorConfig(
job_id="test-job",
artist="Test Artist",
title="Test Title",
title_video_path="/path/title.mov",
karaoke_video_path="/path/karaoke.mov",
instrumental_audio_path="/path/audio.flac",
)
assert config.countdown_padding_seconds is None
def test_create_orchestrator_config_reads_countdown_from_lyrics_metadata(self):
"""Test that create_orchestrator_config_from_job reads countdown from lyrics_metadata.
This is the CRITICAL test that would have caught the bug where the orchestrator
path didn't read countdown state from lyrics_metadata.
"""
job = MagicMock()
job.job_id = "test-job"
job.artist = "Test Artist"
job.title = "Test Title"
job.state_data = {
'instrumental_selection': 'clean',
'lyrics_metadata': {
'has_countdown_padding': True,
'countdown_padding_seconds': 3.0,
}
}
job.enable_cdg = False
job.enable_txt = False
job.enable_youtube_upload = False
job.brand_prefix = None
job.discord_webhook_url = None
job.youtube_description_template = None
job.dropbox_path = None
job.gdrive_folder_id = None
job.keep_brand_code = None
job.existing_instrumental_gcs_path = None
config = create_orchestrator_config_from_job(
job=job,
temp_dir="/tmp/test",
)
# CRITICAL: countdown_padding_seconds must be passed to OrchestratorConfig
# If this fails, the GCE worker will not pad the instrumental audio
assert config.countdown_padding_seconds == 3.0, \
"countdown_padding_seconds must be read from job.state_data.lyrics_metadata"
def test_create_orchestrator_config_handles_missing_countdown(self):
"""Test graceful handling when lyrics_metadata has no countdown fields."""
job = MagicMock()
job.job_id = "test-job"
job.artist = "Test Artist"
job.title = "Test Title"
job.state_data = {
'instrumental_selection': 'clean',
'lyrics_metadata': {
'has_corrections': True, # No countdown fields
}
}
job.enable_cdg = False
job.enable_txt = False
job.enable_youtube_upload = False
job.brand_prefix = None
job.discord_webhook_url = None
job.youtube_description_template = None
job.dropbox_path = None
job.gdrive_folder_id = None
job.keep_brand_code = None
job.existing_instrumental_gcs_path = None
config = create_orchestrator_config_from_job(
job=job,
temp_dir="/tmp/test",
)
# Should default to None when not present
assert config.countdown_padding_seconds is None
def test_create_orchestrator_config_handles_false_countdown(self):
"""Test that countdown is None when has_countdown_padding is False."""
job = MagicMock()
job.job_id = "test-job"
job.artist = "Test Artist"
job.title = "Test Title"
job.state_data = {
'instrumental_selection': 'clean',
'lyrics_metadata': {
'has_countdown_padding': False, # Explicitly False
'countdown_padding_seconds': 0.0,
}
}
job.enable_cdg = False
job.enable_txt = False
job.enable_youtube_upload = False
job.brand_prefix = None
job.discord_webhook_url = None
job.youtube_description_template = None
job.dropbox_path = None
job.gdrive_folder_id = None
job.keep_brand_code = None
job.existing_instrumental_gcs_path = None
config = create_orchestrator_config_from_job(
job=job,
temp_dir="/tmp/test",
)
# Should be None because has_countdown_padding is False
assert config.countdown_padding_seconds is None
@pytest.mark.asyncio
async def test_orchestrator_passes_countdown_to_encoding_options(self):
"""Test full flow: orchestrator passes countdown_padding_seconds through EncodingInput.options.
This verifies the complete flow from OrchestratorConfig to the encoding backend.
"""
config = OrchestratorConfig(
job_id="test-job",
artist="Test Artist",
title="Test Title",
title_video_path="/path/title.mov",
karaoke_video_path="/path/karaoke.mov",
instrumental_audio_path="/path/audio.flac",
output_dir="/output",
countdown_padding_seconds=3.0, # Countdown padding enabled
)
job_manager = MagicMock()
storage = MagicMock()
job_logger = MagicMock()
orchestrator = VideoWorkerOrchestrator(
config=config,
job_manager=job_manager,
storage=storage,
job_logger=job_logger,
)
# Capture the EncodingInput that gets passed to the backend
captured_input = None
async def capture_encode(encoding_input):
nonlocal captured_input
captured_input = encoding_input
from backend.services.encoding_interface import EncodingOutput
return EncodingOutput(
success=True,
lossless_4k_mp4_path="/output/lossless.mp4",
encoding_time_seconds=1.0,
encoding_backend="mock",
)
with patch.object(orchestrator, "_get_encoding_backend") as mock_get:
mock_backend = MagicMock()
mock_backend.name = "mock"
mock_backend.encode = capture_encode
mock_get.return_value = mock_backend
await orchestrator._run_encoding()
# CRITICAL ASSERTION: countdown_padding_seconds must be passed through options
assert captured_input is not None, "encode() should have been called"
assert captured_input.options.get("countdown_padding_seconds") == 3.0, \
"Orchestrator must pass countdown_padding_seconds in EncodingInput.options"
+22 -303
View File
@@ -901,71 +901,67 @@ class TestDownloadAudioWithUrl:
class TestBackingVocalsAnalysis:
"""Tests for backing vocals analysis in screens_worker.
Note: Analysis was moved from render_video_worker to screens_worker as part of
the combined review flow - analysis now runs before review, not after.
"""
"""Tests for backing vocals analysis in render_video_worker."""
def test_analyze_backing_vocals_function_exists(self):
"""Test that _analyze_backing_vocals function exists in screens_worker."""
from backend.workers.screens_worker import _analyze_backing_vocals
"""Test that _analyze_backing_vocals function exists in render_video_worker."""
from backend.workers.render_video_worker import _analyze_backing_vocals
assert callable(_analyze_backing_vocals)
def test_analyze_backing_vocals_is_async(self):
"""Test that _analyze_backing_vocals is an async function."""
import inspect
from backend.workers.screens_worker import _analyze_backing_vocals
from backend.workers.render_video_worker import _analyze_backing_vocals
assert inspect.iscoroutinefunction(_analyze_backing_vocals)
def test_screens_worker_imports_analysis_service(self):
"""Test that screens_worker can import AudioAnalysisService."""
def test_render_video_worker_imports_analysis_service(self):
"""Test that render_video_worker can import AudioAnalysisService."""
# This verifies the import path is correct
from backend.services.audio_analysis_service import AudioAnalysisService
assert AudioAnalysisService is not None
@pytest.mark.asyncio
async def test_analyze_backing_vocals_handles_missing_job(self):
"""Test that analysis returns early when job not found (no error stored)."""
mock_job_manager = MagicMock()
mock_job_manager.get_job.return_value = None # No job found
mock_job_manager.update_state_data = MagicMock()
mock_storage = MagicMock()
mock_logger = MagicMock()
from backend.workers.screens_worker import _analyze_backing_vocals
from backend.workers.render_video_worker import _analyze_backing_vocals
# Should not raise when job not found, just log warning and return
await _analyze_backing_vocals(
"nonexistent", mock_job_manager, mock_storage, mock_logger
)
# Should log warning but not update state (early return)
mock_logger.warning.assert_called()
# No state_data update since we return early
mock_job_manager.update_state_data.assert_not_called()
@pytest.mark.asyncio
async def test_analyze_backing_vocals_handles_missing_stems(self):
"""Test that analysis returns early when stems not found (no error stored)."""
mock_job = MagicMock()
mock_job.file_urls = {} # No stems
mock_job_manager = MagicMock()
mock_job_manager.get_job.return_value = mock_job
mock_job_manager.update_state_data = MagicMock()
mock_storage = MagicMock()
mock_logger = MagicMock()
from backend.workers.screens_worker import _analyze_backing_vocals
from backend.workers.render_video_worker import _analyze_backing_vocals
# Should not raise when stems not found, just log warning and return
await _analyze_backing_vocals(
"test123", mock_job_manager, mock_storage, mock_logger
)
# Should log warning but not update state (early return)
mock_logger.warning.assert_called()
# No state_data update since we return early
@@ -1118,280 +1114,3 @@ class TestDistributionDirectoryPreparation:
assert not simplified_clean.exists()
assert not simplified_backing.exists()
class TestRenderVideoWorkerCountdownStateSync:
"""
Tests that render_video_worker correctly updates job state when countdown is added.
This is the critical test for the countdown audio sync bug. The bug occurs when:
1. render_video_worker adds countdown via CountdownProcessor
2. But doesn't update job.state_data.lyrics_metadata.has_countdown_padding
3. So video_worker doesn't know to pad the instrumental
These tests verify the state sync behavior.
"""
@pytest.fixture
def mock_job_for_render(self):
"""Create a mock job in RENDERING_VIDEO state."""
return Job(
job_id="test123",
status=JobStatus.RENDERING_VIDEO,
created_at=datetime.now(UTC),
updated_at=datetime.now(UTC),
artist="Test Artist",
title="Early Start Song",
file_urls={
'vocals': 'gs://bucket/vocals.flac',
},
state_data={
'lyrics_metadata': {
'has_countdown_padding': False,
'countdown_padding_seconds': 0.0,
}
}
)
@pytest.fixture
def mock_job_manager_for_render(self, mock_job_for_render):
"""Create a mock JobManager for render tests."""
manager = MagicMock()
manager.get_job.return_value = mock_job_for_render
manager.update_state_data = MagicMock()
manager.update_job_status = MagicMock()
manager.update_file_url = MagicMock()
manager.transition_to_state = MagicMock()
return manager
def test_countdown_added_updates_state_data(self):
"""When countdown is added, state_data MUST be updated with countdown info.
This test verifies the contract: render_video_worker must call
update_state_data('lyrics_metadata', {..., 'has_countdown_padding': True})
when CountdownProcessor returns padding_added=True.
"""
# This is a unit test for the expected behavior
# The actual implementation test is in test_countdown_state_contract.py
# When countdown is added:
padding_added = True
padding_seconds = 3.0
# State should be updated to:
expected_state = {
'has_countdown_padding': True,
'countdown_padding_seconds': 3.0,
}
# Verify logic
if padding_added:
new_lyrics_metadata = {
'has_countdown_padding': True,
'countdown_padding_seconds': padding_seconds,
}
assert new_lyrics_metadata == expected_state
def test_no_countdown_keeps_state_false(self):
"""When countdown is NOT added, state should remain False.
This ensures we don't incorrectly set has_countdown_padding=True
when the song doesn't need a countdown.
"""
padding_added = False
padding_seconds = 0.0
# State should remain:
expected_state = {
'has_countdown_padding': False,
'countdown_padding_seconds': 0.0,
}
if not padding_added:
lyrics_metadata = {
'has_countdown_padding': False,
'countdown_padding_seconds': 0.0,
}
assert lyrics_metadata == expected_state
class TestVideoWorkerCountdownStateRead:
"""
Tests that video_worker correctly reads countdown state from job.state_data.
This verifies the reader side of the countdown state contract.
"""
@pytest.fixture
def mock_job_with_countdown_state(self):
"""Create a job where render_video_worker set countdown state."""
return Job(
job_id="test123",
status=JobStatus.ENCODING,
created_at=datetime.now(UTC),
updated_at=datetime.now(UTC),
artist="Test",
title="Test",
file_urls={
'instrumental_clean': 'gs://bucket/instrumental.flac',
'video_output': 'gs://bucket/video.mp4',
},
state_data={
'lyrics_metadata': {
'has_countdown_padding': True,
'countdown_padding_seconds': 3.0,
},
'video_worker': {
'instrumental_source': 'clean',
}
}
)
@pytest.fixture
def mock_job_without_countdown_state(self):
"""Create a job where no countdown was needed."""
return Job(
job_id="test123",
status=JobStatus.ENCODING,
created_at=datetime.now(UTC),
updated_at=datetime.now(UTC),
artist="Test",
title="Test",
file_urls={
'instrumental_clean': 'gs://bucket/instrumental.flac',
'video_output': 'gs://bucket/video.mp4',
},
state_data={
'lyrics_metadata': {
'has_countdown_padding': False,
'countdown_padding_seconds': 0.0,
},
'video_worker': {
'instrumental_source': 'clean',
}
}
)
def test_reads_countdown_from_state(self, mock_job_with_countdown_state):
"""video_worker reads countdown_padding_seconds from lyrics_metadata."""
job = mock_job_with_countdown_state
# This mirrors video_worker.py logic
lyrics_metadata = job.state_data.get('lyrics_metadata', {})
countdown_padding_seconds = None
if lyrics_metadata.get('has_countdown_padding'):
countdown_padding_seconds = lyrics_metadata.get('countdown_padding_seconds', 3.0)
assert countdown_padding_seconds == 3.0
def test_reads_none_when_no_countdown(self, mock_job_without_countdown_state):
"""video_worker returns None when has_countdown_padding=False."""
job = mock_job_without_countdown_state
lyrics_metadata = job.state_data.get('lyrics_metadata', {})
countdown_padding_seconds = None
if lyrics_metadata.get('has_countdown_padding'):
countdown_padding_seconds = lyrics_metadata.get('countdown_padding_seconds', 3.0)
assert countdown_padding_seconds is None
def test_handles_missing_lyrics_metadata(self):
"""video_worker handles missing lyrics_metadata gracefully."""
job = Job(
job_id="test123",
status=JobStatus.ENCODING,
created_at=datetime.now(UTC),
updated_at=datetime.now(UTC),
artist="Test",
title="Test",
state_data={} # No lyrics_metadata
)
lyrics_metadata = job.state_data.get('lyrics_metadata', {})
countdown_padding_seconds = None
if lyrics_metadata.get('has_countdown_padding'):
countdown_padding_seconds = lyrics_metadata.get('countdown_padding_seconds', 3.0)
# Should not raise, should return None
assert countdown_padding_seconds is None
def test_defaults_to_3_seconds_when_duration_missing(self):
"""If has_countdown_padding=True but duration missing, default to 3.0."""
job = Job(
job_id="test123",
status=JobStatus.ENCODING,
created_at=datetime.now(UTC),
updated_at=datetime.now(UTC),
artist="Test",
title="Test",
state_data={
'lyrics_metadata': {
'has_countdown_padding': True,
# countdown_padding_seconds intentionally missing
}
}
)
lyrics_metadata = job.state_data.get('lyrics_metadata', {})
countdown_padding_seconds = None
if lyrics_metadata.get('has_countdown_padding'):
countdown_padding_seconds = lyrics_metadata.get('countdown_padding_seconds', 3.0)
assert countdown_padding_seconds == 3.0
class TestLyricsWorkerCountdownState:
"""
Tests for lyrics_worker countdown state handling.
Note: Since countdown processing is now deferred to render_video_worker,
lyrics_worker should set has_countdown_padding=False in all cases.
The actual countdown state is set by render_video_worker.
"""
def test_lyrics_worker_sets_countdown_false_by_default(self):
"""lyrics_worker sets has_countdown_padding=False since countdown is deferred."""
# When transcription_result doesn't have countdown_padding_added
transcription_result = {
'lrc_filepath': '/tmp/test.lrc',
}
# lyrics_worker logic
lyrics_metadata = {}
if transcription_result.get("countdown_padding_added"):
lyrics_metadata['has_countdown_padding'] = True
lyrics_metadata['countdown_padding_seconds'] = transcription_result.get(
"countdown_padding_seconds", 3.0
)
else:
lyrics_metadata['has_countdown_padding'] = False
lyrics_metadata['countdown_padding_seconds'] = 0.0
assert lyrics_metadata['has_countdown_padding'] is False
assert lyrics_metadata['countdown_padding_seconds'] == 0.0
def test_lyrics_worker_would_set_true_if_countdown_in_result(self):
"""If transcription_result had countdown info, it would be set.
This tests the code path for when countdown WAS done during lyrics phase.
Currently this path isn't used (countdown is deferred), but the code exists.
"""
# If countdown_padding_added was True
transcription_result = {
'lrc_filepath': '/tmp/test.lrc',
'countdown_padding_added': True,
'countdown_padding_seconds': 3.0,
}
lyrics_metadata = {}
if transcription_result.get("countdown_padding_added"):
lyrics_metadata['has_countdown_padding'] = True
lyrics_metadata['countdown_padding_seconds'] = transcription_result.get(
"countdown_padding_seconds", 3.0
)
else:
lyrics_metadata['has_countdown_padding'] = False
lyrics_metadata['countdown_padding_seconds'] = 0.0
assert lyrics_metadata['has_countdown_padding'] is True
assert lyrics_metadata['countdown_padding_seconds'] == 3.0
@@ -1,377 +0,0 @@
"""
Unit tests for YouTubeDownloadService.
Tests the consolidated YouTube download service that handles all YouTube
downloads in the cloud backend.
"""
import os
import pytest
from unittest.mock import AsyncMock, MagicMock, patch
from backend.services.youtube_download_service import (
YouTubeDownloadService,
YouTubeDownloadError,
get_youtube_download_service,
reset_youtube_download_service,
)
class TestYouTubeDownloadService:
"""Tests for YouTubeDownloadService."""
def setup_method(self):
"""Reset singleton before each test."""
reset_youtube_download_service()
# =========================================================================
# Video ID Extraction Tests
# =========================================================================
def test_extract_video_id_standard_watch_url(self):
"""Should extract video ID from standard watch URL."""
service = YouTubeDownloadService()
url = "https://www.youtube.com/watch?v=dQw4w9WgXcQ"
assert service._extract_video_id(url) == "dQw4w9WgXcQ"
def test_extract_video_id_short_url(self):
"""Should extract video ID from youtu.be short URL."""
service = YouTubeDownloadService()
url = "https://youtu.be/dQw4w9WgXcQ"
assert service._extract_video_id(url) == "dQw4w9WgXcQ"
def test_extract_video_id_shorts_url(self):
"""Should extract video ID from YouTube Shorts URL."""
service = YouTubeDownloadService()
url = "https://www.youtube.com/shorts/dQw4w9WgXcQ"
assert service._extract_video_id(url) == "dQw4w9WgXcQ"
def test_extract_video_id_embed_url(self):
"""Should extract video ID from embed URL."""
service = YouTubeDownloadService()
url = "https://www.youtube.com/embed/dQw4w9WgXcQ"
assert service._extract_video_id(url) == "dQw4w9WgXcQ"
def test_extract_video_id_with_extra_params(self):
"""Should extract video ID when URL has extra parameters."""
service = YouTubeDownloadService()
url = "https://www.youtube.com/watch?v=dQw4w9WgXcQ&list=PLtest&t=120"
assert service._extract_video_id(url) == "dQw4w9WgXcQ"
def test_extract_video_id_v_before_other_params(self):
"""Should extract video ID when v is not the first param."""
service = YouTubeDownloadService()
url = "https://www.youtube.com/watch?list=PLtest&v=dQw4w9WgXcQ"
assert service._extract_video_id(url) == "dQw4w9WgXcQ"
def test_extract_video_id_old_style_url(self):
"""Should extract video ID from old-style /v/ URL."""
service = YouTubeDownloadService()
url = "https://www.youtube.com/v/dQw4w9WgXcQ"
assert service._extract_video_id(url) == "dQw4w9WgXcQ"
def test_extract_video_id_invalid_url(self):
"""Should return None for non-YouTube URLs."""
service = YouTubeDownloadService()
assert service._extract_video_id("https://vimeo.com/123456") is None
assert service._extract_video_id("not a url") is None
# =========================================================================
# Remote Download Tests
# =========================================================================
@pytest.mark.asyncio
async def test_download_uses_remote_when_configured(self):
"""When FLACFETCH_API_URL is set, should use remote client."""
mock_client = MagicMock()
mock_client.download_by_id = AsyncMock(return_value="download_123")
mock_client.wait_for_download = AsyncMock(return_value={
"status": "complete",
"gcs_path": "gs://bucket/uploads/job123/audio/Artist - Title.flac",
})
with patch(
'backend.services.youtube_download_service.get_flacfetch_client',
return_value=mock_client
):
service = YouTubeDownloadService()
result = await service.download(
url="https://www.youtube.com/watch?v=dQw4w9WgXcQ",
job_id="job123",
artist="Rick Astley",
title="Never Gonna Give You Up",
)
assert result == "uploads/job123/audio/Artist - Title.flac"
mock_client.download_by_id.assert_called_once()
mock_client.wait_for_download.assert_called_once()
@pytest.mark.asyncio
async def test_download_by_id_constructs_url(self):
"""download_by_id should construct URL from video ID."""
mock_client = MagicMock()
mock_client.download_by_id = AsyncMock(return_value="download_123")
mock_client.wait_for_download = AsyncMock(return_value={
"status": "complete",
"gcs_path": "gs://bucket/uploads/job123/audio/file.flac",
})
with patch(
'backend.services.youtube_download_service.get_flacfetch_client',
return_value=mock_client
):
service = YouTubeDownloadService()
result = await service.download_by_id(
video_id="dQw4w9WgXcQ",
job_id="job123",
)
assert result == "uploads/job123/audio/file.flac"
@pytest.mark.asyncio
async def test_download_extracts_gcs_path_from_gs_url(self):
"""Should correctly extract path portion from gs:// URL."""
mock_client = MagicMock()
mock_client.download_by_id = AsyncMock(return_value="download_123")
mock_client.wait_for_download = AsyncMock(return_value={
"status": "complete",
"gcs_path": "gs://my-bucket/uploads/job123/audio/test.flac",
})
with patch(
'backend.services.youtube_download_service.get_flacfetch_client',
return_value=mock_client
):
service = YouTubeDownloadService()
result = await service.download(
url="https://youtu.be/dQw4w9WgXcQ",
job_id="job123",
)
assert result == "uploads/job123/audio/test.flac"
assert not result.startswith("gs://")
@pytest.mark.asyncio
async def test_download_handles_remote_failure(self):
"""Should raise YouTubeDownloadError on remote failure."""
from backend.services.flacfetch_client import FlacfetchServiceError
mock_client = MagicMock()
mock_client.download_by_id = AsyncMock(
side_effect=FlacfetchServiceError("Connection failed")
)
with patch(
'backend.services.youtube_download_service.get_flacfetch_client',
return_value=mock_client
):
service = YouTubeDownloadService()
with pytest.raises(YouTubeDownloadError) as exc:
await service.download(
url="https://youtu.be/dQw4w9WgXcQ",
job_id="job123",
)
assert "Remote download failed" in str(exc.value)
# =========================================================================
# Local Fallback Tests
# =========================================================================
@pytest.mark.asyncio
async def test_download_falls_back_to_local_when_remote_disabled(self):
"""When remote not configured, should use local yt_dlp."""
mock_storage = MagicMock()
mock_storage.upload_fileobj = MagicMock()
mock_file_handler = MagicMock()
mock_file_handler.download_video = MagicMock(return_value="/tmp/test.webm")
mock_file_handler.convert_to_wav = MagicMock(return_value="/tmp/test.wav")
with patch(
'backend.services.youtube_download_service.get_flacfetch_client',
return_value=None # No remote client
), patch(
'backend.services.youtube_download_service.StorageService',
return_value=mock_storage
), patch(
'backend.services.youtube_download_service.YouTubeDownloadService._download_local',
new_callable=AsyncMock,
return_value="uploads/job123/audio/test.wav"
) as mock_local:
service = YouTubeDownloadService()
result = await service.download(
url="https://youtu.be/dQw4w9WgXcQ",
job_id="job123",
)
assert result == "uploads/job123/audio/test.wav"
mock_local.assert_called_once()
@pytest.mark.asyncio
async def test_download_invalid_url_raises_error(self):
"""Should raise error for URLs that can't extract video ID."""
with patch(
'backend.services.youtube_download_service.get_flacfetch_client',
return_value=None
):
service = YouTubeDownloadService()
with pytest.raises(YouTubeDownloadError) as exc:
await service.download(
url="https://vimeo.com/123456",
job_id="job123",
)
assert "Could not extract video ID" in str(exc.value)
# =========================================================================
# Singleton Tests
# =========================================================================
def test_get_youtube_download_service_returns_singleton(self):
"""get_youtube_download_service should return same instance."""
with patch(
'backend.services.youtube_download_service.get_flacfetch_client',
return_value=None
):
service1 = get_youtube_download_service()
service2 = get_youtube_download_service()
assert service1 is service2
def test_reset_youtube_download_service_clears_singleton(self):
"""reset_youtube_download_service should clear the singleton."""
with patch(
'backend.services.youtube_download_service.get_flacfetch_client',
return_value=None
):
service1 = get_youtube_download_service()
reset_youtube_download_service()
service2 = get_youtube_download_service()
assert service1 is not service2
# =========================================================================
# Remote Enabled Check
# =========================================================================
def test_is_remote_enabled_true_when_client_configured(self):
"""is_remote_enabled should return True when flacfetch client exists."""
mock_client = MagicMock()
with patch(
'backend.services.youtube_download_service.get_flacfetch_client',
return_value=mock_client
):
service = YouTubeDownloadService()
assert service.is_remote_enabled() is True
def test_is_remote_enabled_false_when_no_client(self):
"""is_remote_enabled should return False when no flacfetch client."""
with patch(
'backend.services.youtube_download_service.get_flacfetch_client',
return_value=None
):
service = YouTubeDownloadService()
assert service.is_remote_enabled() is False
class TestYouTubeDownloadServiceIntegration:
"""Integration tests that verify full download flow with mocked external services."""
def setup_method(self):
"""Reset singleton before each test."""
reset_youtube_download_service()
@pytest.mark.asyncio
async def test_remote_download_flow_complete(self):
"""Test complete remote download flow with proper state transitions."""
mock_client = MagicMock()
# Track call sequence
call_sequence = []
async def mock_download_by_id(**kwargs):
call_sequence.append(('download_by_id', kwargs))
return "download_abc123"
async def mock_wait_for_download(download_id, **kwargs):
call_sequence.append(('wait_for_download', download_id))
return {
"status": "complete",
"gcs_path": "gs://karaoke-bucket/uploads/job_test/audio/Artist - Song.opus",
}
mock_client.download_by_id = mock_download_by_id
mock_client.wait_for_download = mock_wait_for_download
with patch(
'backend.services.youtube_download_service.get_flacfetch_client',
return_value=mock_client
):
service = YouTubeDownloadService()
# Use a valid 11-character video ID (YouTube IDs are always 11 chars)
result = await service.download(
url="https://www.youtube.com/watch?v=abcDEF12345",
job_id="job_test",
artist="Test Artist",
title="Test Song",
)
# Verify result
assert result == "uploads/job_test/audio/Artist - Song.opus"
# Verify call sequence
assert len(call_sequence) == 2
assert call_sequence[0][0] == 'download_by_id'
assert call_sequence[0][1]['source_name'] == 'YouTube'
assert call_sequence[0][1]['source_id'] == 'abcDEF12345'
assert call_sequence[1][0] == 'wait_for_download'
assert call_sequence[1][1] == 'download_abc123'
@pytest.mark.asyncio
async def test_output_filename_sanitization(self):
"""Test that artist/title are sanitized for output filename."""
mock_client = MagicMock()
mock_client.download_by_id = AsyncMock(return_value="download_123")
mock_client.wait_for_download = AsyncMock(return_value={
"status": "complete",
"gcs_path": "gs://bucket/uploads/job/audio/file.flac",
})
with patch(
'backend.services.youtube_download_service.get_flacfetch_client',
return_value=mock_client
):
service = YouTubeDownloadService()
# Use a valid 11-character video ID
await service.download(
url="https://youtu.be/abcDEF12345",
job_id="job123",
artist="Artist's Name", # Contains apostrophe
title="Title: With/Symbols", # Contains colon and slash
)
# Check that download_by_id was called with output_filename
call_args = mock_client.download_by_id.call_args
assert call_args is not None
# Verify output_filename was passed and is sanitized
call_kwargs = call_args.kwargs
assert 'output_filename' in call_kwargs
output_filename = call_kwargs['output_filename']
# The filename should NOT contain special chars that break filenames
# (apostrophe, colon, slash are typically sanitized)
assert ':' not in output_filename
assert '/' not in output_filename
# Should contain artist and title parts
assert 'Artist' in output_filename
assert 'Title' in output_filename
+9 -93
View File
@@ -74,33 +74,24 @@ AUDIO_WORKER_LOGGERS = [
async def download_from_url(url: str, temp_dir: str, artist: str, title: str, job_manager: JobManager = None, job_id: str = None) -> Optional[str]:
"""
Download audio from a URL using local yt_dlp.
IMPORTANT: This is a LEGACY FALLBACK for non-YouTube URLs only!
For YouTube URLs, use YouTubeDownloadService instead, which:
- Uses remote flacfetch when configured (avoids Cloud Run bot detection)
- Has proper error handling and GCS upload
This function uses local yt_dlp and will likely FAIL for YouTube URLs
on Cloud Run due to bot detection (YouTube blocks Cloud Run IP ranges).
Uses the FileHandler from karaoke_gen which includes:
Download audio from a URL (YouTube, etc.) using karaoke_gen.FileHandler.
Uses the battle-tested FileHandler from karaoke_gen which includes:
- Anti-detection options (user agent, headers, delays)
- Cookie support for authenticated downloads
- Retry logic
If artist and/or title are not provided, attempts to extract them from
If artist and/or title are not provided, attempts to extract them from
the URL metadata.
Args:
url: URL to download from (NOT recommended for YouTube - use YouTubeDownloadService)
url: URL to download from
temp_dir: Temporary directory to save to
artist: Artist name for filename (can be None for auto-detection)
title: Song title for filename (can be None for auto-detection)
job_manager: Optional JobManager to update job with detected metadata
job_id: Optional job ID to update
Returns:
Path to downloaded audio file, or None if failed
"""
@@ -487,20 +478,8 @@ async def download_audio(
logger.info(f"Job {job_id}: Downloaded audio from GCS: {input_url}")
return local_path
# Case 3: Fresh URL that needs downloading (legacy fallback)
# NOTE: YouTube URLs should be downloaded by YouTubeDownloadService in file_upload.py
# BEFORE triggering this worker. If we reach here with a YouTube URL, it means
# the download was not done upfront, which will likely fail due to bot detection.
# Case 3: Fresh URL that needs downloading
if job.url:
# Check if this is a YouTube URL that should have been handled earlier
is_youtube = any(domain in job.url.lower() for domain in ['youtube.com', 'youtu.be'])
if is_youtube:
logger.warning(
f"Job {job_id}: YouTube URL reached download_audio() fallback. "
"This may fail due to bot detection. YouTube URLs should be "
"downloaded via YouTubeDownloadService before triggering workers."
)
logger.info(f"Job {job_id}: Downloading from URL: {job.url}")
# Use provided job_manager or create new one
@@ -645,66 +624,3 @@ async def upload_separation_results(
job_manager.update_state_data(job_id, 'instrumental_options', instrumental_options)
logger.info(f"Job {job_id}: Stored instrumental options: {list(instrumental_options.keys())}")
# ==================== CLI Entry Point for Cloud Run Jobs ====================
# This allows the audio worker to be run as a standalone Cloud Run Job.
# Usage: python -m backend.workers.audio_worker --job-id <job_id>
def main():
"""
CLI entry point for running audio worker as a Cloud Run Job.
Cloud Run Jobs run to completion without HTTP request lifecycle concerns,
avoiding the instance termination issue where Cloud Run would shut down
instances mid-processing when using BackgroundTasks.
Usage:
python -m backend.workers.audio_worker --job-id abc123
Environment Variables:
GOOGLE_CLOUD_PROJECT: GCP project ID (required)
GCS_BUCKET_NAME: Storage bucket name (required)
AUDIO_SEPARATOR_API_URL: Modal API URL (required)
"""
import argparse
import asyncio
import sys
# Set up argument parser
parser = argparse.ArgumentParser(
description="Audio separation worker for karaoke generation"
)
parser.add_argument(
"--job-id",
required=True,
help="Job ID to process"
)
args = parser.parse_args()
job_id = args.job_id
# Configure logging
logging.basicConfig(
level=logging.INFO,
format="%(asctime)s - %(name)s - %(levelname)s - %(message)s"
)
logger.info(f"Starting audio worker CLI for job {job_id}")
# Run the async worker function
try:
success = asyncio.run(process_audio_separation(job_id))
if success:
logger.info(f"Audio separation completed successfully for job {job_id}")
sys.exit(0)
else:
logger.error(f"Audio separation failed for job {job_id}")
sys.exit(1)
except Exception as e:
logger.error(f"Audio worker crashed: {e}", exc_info=True)
sys.exit(1)
if __name__ == "__main__":
main()
-62
View File
@@ -689,65 +689,3 @@ async def upload_lyrics_results(
logger.info(f"Job {job_id}: All lyrics results uploaded successfully")
# ==================== CLI Entry Point for Cloud Run Jobs ====================
# This allows the lyrics worker to be run as a standalone Cloud Run Job.
# Usage: python -m backend.workers.lyrics_worker --job-id <job_id>
def main():
"""
CLI entry point for running lyrics worker as a Cloud Run Job.
Cloud Run Jobs run to completion without HTTP request lifecycle concerns,
avoiding the instance termination issue where Cloud Run would shut down
instances mid-processing when using BackgroundTasks.
Usage:
python -m backend.workers.lyrics_worker --job-id abc123
Environment Variables:
GOOGLE_CLOUD_PROJECT: GCP project ID (required)
GCS_BUCKET_NAME: Storage bucket name (required)
AUDIOSHAKE_API_TOKEN: AudioShake API key (required)
"""
import argparse
import sys
# Set up argument parser
parser = argparse.ArgumentParser(
description="Lyrics transcription worker for karaoke generation"
)
parser.add_argument(
"--job-id",
required=True,
help="Job ID to process"
)
args = parser.parse_args()
job_id = args.job_id
# Configure logging
logging.basicConfig(
level=logging.INFO,
format="%(asctime)s - %(name)s - %(levelname)s - %(message)s"
)
logger.info(f"Starting lyrics worker CLI for job {job_id}")
# Run the async worker function
try:
success = asyncio.run(process_lyrics_transcription(job_id))
if success:
logger.info(f"Lyrics transcription completed successfully for job {job_id}")
sys.exit(0)
else:
logger.error(f"Lyrics transcription failed for job {job_id}")
sys.exit(1)
except Exception as e:
logger.error(f"Lyrics worker crashed: {e}", exc_info=True)
sys.exit(1)
if __name__ == "__main__":
main()
+137 -35
View File
@@ -9,12 +9,7 @@ This worker:
3. Downloads style assets from GCS
4. Uses LyricsTranscriber's OutputGenerator to render video
5. Uploads the with_vocals.mkv to GCS
6. Transitions to INSTRUMENTAL_SELECTED (user already selected during combined review)
7. Triggers video worker for final encoding
Note: Instrumental selection now happens during the combined lyrics + instrumental
review phase (before this worker runs). The selection is stored in
state_data['instrumental_selection'] by the review completion endpoint.
6. Transitions to AWAITING_INSTRUMENTAL_SELECTION
Key insight: We use OutputGenerator from lyrics_transcriber library
WITHOUT using its blocking ReviewServer. This allows async operation
@@ -204,16 +199,7 @@ async def process_render_video(job_id: str) -> bool:
job_log.info(f"Countdown added: {padding_seconds}s padding applied to audio and timestamps shifted")
else:
job_log.info("No countdown needed - song starts after 3 seconds")
# Update lyrics_metadata with countdown info so video_worker knows to pad instrumental
# This is critical for audio sync - without this, instrumental won't match padded vocals
if padding_added:
existing_lyrics_metadata = job.state_data.get('lyrics_metadata', {})
existing_lyrics_metadata['has_countdown_padding'] = True
existing_lyrics_metadata['countdown_padding_seconds'] = padding_seconds
job_manager.update_state_data(job_id, 'lyrics_metadata', existing_lyrics_metadata)
job_log.info(f"Updated lyrics_metadata: has_countdown_padding=True, countdown_padding_seconds={padding_seconds}")
# 6. Get or create styles using the unified style loader
job_log.info("Loading style configuration...")
job_log.info(f" job.style_params_gcs_path: {job.style_params_gcs_path}")
@@ -309,10 +295,12 @@ async def process_render_video(job_id: str) -> bool:
job_manager.update_file_url(job_id, 'lyrics', 'corrected_txt', txt_url)
job_log.info("Uploaded corrected.txt")
logger.info(f"Job {job_id}: Uploaded corrected.txt")
# 13. Transition based on prep_only flag
# Note: Instrumental selection was already made during combined review
# (stored in state_data['instrumental_selection'])
# 13. Analyze backing vocals for intelligent instrumental selection
job_log.info("Analyzing backing vocals for instrumental selection...")
await _analyze_backing_vocals(job_id, job_manager, storage, job_log)
# 14. Transition based on prep_only flag and existing instrumental
if getattr(job, 'prep_only', False):
# Prep-only mode: stop here and mark as prep complete
job_manager.transition_to_state(
@@ -326,30 +314,40 @@ async def process_render_video(job_id: str) -> bool:
root_span.set_attribute("duration_seconds", duration)
root_span.set_attribute("prep_only", True)
logger.info(f"[job:{job_id}] WORKER_END worker=render-video status=success duration={duration:.1f}s prep_only=true")
else:
# Normal mode: instrumental was selected during combined review
# Transition directly to INSTRUMENTAL_SELECTED and trigger video worker
elif getattr(job, 'existing_instrumental_gcs_path', None):
# Existing instrumental provided - skip selection, auto-use it
# Store selection as 'custom' to indicate user-provided instrumental
job_manager.update_state_data(job_id, 'instrumental_selection', 'custom')
job_log.info("Existing instrumental provided - skipping selection, using user-provided file")
job_manager.transition_to_state(
job_id=job_id,
new_status=JobStatus.INSTRUMENTAL_SELECTED,
progress=82,
message="Video rendered, starting final encoding"
message="Using user-provided instrumental"
)
job_log.info("=== RENDER VIDEO WORKER COMPLETE (EXISTING INSTRUMENTAL) ===")
duration = time.time() - start_time
root_span.set_attribute("duration_seconds", duration)
root_span.set_attribute("existing_instrumental", True)
logger.info(f"[job:{job_id}] WORKER_END worker=render-video status=success duration={duration:.1f}s existing_instrumental=true")
# Trigger video worker directly since no user selection needed
from backend.workers.video_worker import run_video_worker
job_log.info("Triggering video worker for final encoding...")
await run_video_worker(job_id, job_manager, storage)
else:
# Normal mode: proceed to instrumental selection
job_manager.transition_to_state(
job_id=job_id,
new_status=JobStatus.AWAITING_INSTRUMENTAL_SELECTION,
progress=80,
message="Video rendered - select your instrumental"
)
job_log.info("=== RENDER VIDEO WORKER COMPLETE ===")
duration = time.time() - start_time
root_span.set_attribute("duration_seconds", duration)
logger.info(f"[job:{job_id}] WORKER_END worker=render-video status=success duration={duration:.1f}s")
# Trigger video worker for final encoding
from backend.services.worker_service import get_worker_service
worker_service = get_worker_service()
job_log.info("Triggering video worker for final encoding...")
await worker_service.trigger_video_worker(job_id)
# Mark render progress as complete for idempotency
# This allows the worker to be re-triggered after admin reset
job_manager.update_state_data(job_id, 'render_progress', {'stage': 'complete'})
return True
except Exception as e:
@@ -377,5 +375,109 @@ def _extract_gcs_path(url: str) -> str:
return url
async def _analyze_backing_vocals(
job_id: str,
job_manager: JobManager,
storage: StorageService,
job_log: logging.Logger,
) -> None:
"""
Analyze backing vocals to help with intelligent instrumental selection.
This function:
1. Downloads the backing vocals stem from GCS
2. Runs audio analysis to detect audible content
3. Generates a waveform visualization image
4. Stores analysis results and waveform URL in job state
The analysis data is then used by the frontend to provide an
intelligent instrumental selection experience.
"""
from backend.services.audio_analysis_service import AudioAnalysisService
try:
# Get the job to access file URLs
job = job_manager.get_job(job_id)
if not job:
job_log.warning(f"Could not get job {job_id} for backing vocals analysis")
return
# Get backing vocals path
backing_vocals_path = job.file_urls.get('stems', {}).get('backing_vocals')
if not backing_vocals_path:
job_log.warning("No backing vocals file found - skipping analysis")
return
job_log.info(f"Analyzing backing vocals: {backing_vocals_path}")
# Create analysis service and run analysis
analysis_service = AudioAnalysisService()
# Define output path for waveform
waveform_gcs_path = f"jobs/{job_id}/analysis/backing_vocals_waveform.png"
# Run analysis and generate waveform
result, waveform_path = analysis_service.analyze_and_generate_waveform(
gcs_audio_path=backing_vocals_path,
job_id=job_id,
gcs_waveform_destination=waveform_gcs_path,
)
# Store analysis results in job state_data
analysis_data = {
'has_audible_content': result.has_audible_content,
'total_duration_seconds': result.total_duration_seconds,
'audible_segments': [
{
'start_seconds': seg.start_seconds,
'end_seconds': seg.end_seconds,
'duration_seconds': seg.duration_seconds,
'avg_amplitude_db': seg.avg_amplitude_db,
'peak_amplitude_db': seg.peak_amplitude_db,
}
for seg in result.audible_segments
],
'recommended_selection': result.recommended_selection.value,
'total_audible_duration_seconds': result.total_audible_duration_seconds,
'audible_percentage': result.audible_percentage,
'silence_threshold_db': result.silence_threshold_db,
}
job_manager.update_state_data(job_id, 'backing_vocals_analysis', analysis_data)
# Store waveform URL
job_manager.update_file_url(job_id, 'analysis', 'backing_vocals_waveform', waveform_path)
job_log.info(
f"Backing vocals analysis complete: "
f"has_audible={result.has_audible_content}, "
f"segments={result.segment_count}, "
f"recommendation={result.recommended_selection.value}"
)
# Log segment details if there are any
if result.audible_segments:
job_log.info(f"Audible segments ({len(result.audible_segments)}):")
for i, seg in enumerate(result.audible_segments[:5]): # Log first 5
job_log.info(
f" [{i+1}] {seg.start_seconds:.1f}s - {seg.end_seconds:.1f}s "
f"({seg.duration_seconds:.1f}s, avg: {seg.avg_amplitude_db:.1f}dB)"
)
if len(result.audible_segments) > 5:
job_log.info(f" ... and {len(result.audible_segments) - 5} more")
except Exception as e:
# Log the error but don't fail the job - analysis is a nice-to-have
job_log.warning(f"Backing vocals analysis failed (non-fatal): {e}")
logger.warning(f"Job {job_id}: Backing vocals analysis failed: {e}")
# Store empty analysis so the frontend knows analysis was attempted
job_manager.update_state_data(job_id, 'backing_vocals_analysis', {
'has_audible_content': None,
'analysis_error': str(e),
'recommended_selection': 'review_needed',
})
# For compatibility with worker service
render_video_worker = process_render_video
+9 -127
View File
@@ -5,12 +5,11 @@ Handles screen generation after parallel processing completes:
1. Generate title screen with artist/song info
2. Generate end screen ("Thank you for singing!")
3. Upload both screens to GCS
4. Analyze backing vocals for instrumental selection
5. Transition to AWAITING_REVIEW (combined lyrics + instrumental review)
4. Transition to AWAITING_REVIEW (human must review lyrics)
After review:
6. Render video worker generates with_vocals.mkv
7. Then INSTRUMENTAL_SELECTED (user selected during review)
5. Render video worker generates with_vocals.mkv
6. Then AWAITING_INSTRUMENTAL_SELECTION
This worker is triggered automatically when both audio and lyrics
processing complete (via mark_audio_complete/mark_lyrics_complete coordination).
@@ -176,32 +175,22 @@ async def generate_screens(job_id: str) -> bool:
# Apply countdown padding if needed
await _apply_countdown_padding_if_needed(job_id, job_manager, job)
# Analyze backing vocals for combined review
# This runs BEFORE review so user can select instrumental during lyrics review
with job_span("analyze-backing-vocals", job_id):
job_log.info("Analyzing backing vocals for instrumental selection...")
await _analyze_backing_vocals(job_id, job_manager, storage, job_log)
# Transition to AWAITING_REVIEW (combined lyrics + instrumental review)
# Human must review lyrics AND select instrumental before video can be rendered
logger.info(f"[job:{job_id}] Screens generated, awaiting combined review")
# Transition to AWAITING_REVIEW
# Human must review lyrics before video can be rendered
logger.info(f"[job:{job_id}] Screens generated, awaiting lyrics review")
job_manager.transition_to_state(
job_id=job_id,
new_status=JobStatus.AWAITING_REVIEW,
progress=55,
message="Ready for review. Please review lyrics and select your instrumental."
message="Ready for lyrics review. Please review and correct lyrics."
)
duration = time.time() - start_time
root_span.set_attribute("duration_seconds", duration)
logger.info(f"[job:{job_id}] WORKER_END worker=screens status=success duration={duration:.1f}s")
# Mark screens progress as complete for idempotency
# This allows the worker to be re-triggered after admin reset
job_manager.update_state_data(job_id, 'screens_progress', {'stage': 'complete'})
return True
except Exception as e:
duration = time.time() - start_time
logger.error(f"[job:{job_id}] WORKER_END worker=screens status=error duration={duration:.1f}s error={e}")
@@ -499,113 +488,6 @@ async def _upload_screens(
logger.info(f"Job {job_id}: Uploaded end screen image ({ext})")
async def _analyze_backing_vocals(
job_id: str,
job_manager: JobManager,
storage: StorageService,
job_log: logging.Logger,
) -> None:
"""
Analyze backing vocals to help with intelligent instrumental selection.
This function:
1. Downloads the backing vocals stem from GCS
2. Runs audio analysis to detect audible content
3. Generates a waveform visualization image
4. Stores analysis results and waveform URL in job state
The analysis data is used by the combined review UI to help users
make an informed instrumental selection.
Note: This runs BEFORE the review phase so data is available when
the user opens the combined lyrics + instrumental review page.
"""
from backend.services.audio_analysis_service import AudioAnalysisService
try:
# Get the job to access file URLs
job = job_manager.get_job(job_id)
if not job:
job_log.warning(f"Could not get job {job_id} for backing vocals analysis")
return
# Get backing vocals path
backing_vocals_path = job.file_urls.get('stems', {}).get('backing_vocals')
if not backing_vocals_path:
job_log.warning("No backing vocals file found - skipping analysis")
return
job_log.info(f"Analyzing backing vocals: {backing_vocals_path}")
# Create analysis service and run analysis
analysis_service = AudioAnalysisService()
# Define output path for waveform
waveform_gcs_path = f"jobs/{job_id}/analysis/backing_vocals_waveform.png"
# Run analysis and generate waveform
result, waveform_path = analysis_service.analyze_and_generate_waveform(
gcs_audio_path=backing_vocals_path,
job_id=job_id,
gcs_waveform_destination=waveform_gcs_path,
)
# Store analysis results in job state_data
analysis_data = {
'has_audible_content': result.has_audible_content,
'total_duration_seconds': result.total_duration_seconds,
'audible_segments': [
{
'start_seconds': seg.start_seconds,
'end_seconds': seg.end_seconds,
'duration_seconds': seg.duration_seconds,
'avg_amplitude_db': seg.avg_amplitude_db,
'peak_amplitude_db': seg.peak_amplitude_db,
}
for seg in result.audible_segments
],
'recommended_selection': result.recommended_selection.value,
'total_audible_duration_seconds': result.total_audible_duration_seconds,
'audible_percentage': result.audible_percentage,
'silence_threshold_db': result.silence_threshold_db,
}
job_manager.update_state_data(job_id, 'backing_vocals_analysis', analysis_data)
# Store waveform URL
job_manager.update_file_url(job_id, 'analysis', 'backing_vocals_waveform', waveform_path)
job_log.info(
f"Backing vocals analysis complete: "
f"has_audible={result.has_audible_content}, "
f"segments={result.segment_count}, "
f"recommendation={result.recommended_selection.value}"
)
# Log segment details if there are any
if result.audible_segments:
job_log.info(f"Audible segments ({len(result.audible_segments)}):")
for i, seg in enumerate(result.audible_segments[:5]): # Log first 5
job_log.info(
f" [{i+1}] {seg.start_seconds:.1f}s - {seg.end_seconds:.1f}s "
f"({seg.duration_seconds:.1f}s, avg: {seg.avg_amplitude_db:.1f}dB)"
)
if len(result.audible_segments) > 5:
job_log.info(f" ... and {len(result.audible_segments) - 5} more")
except Exception as e:
# Log the error but don't fail the job - analysis is a nice-to-have
job_log.warning(f"Backing vocals analysis failed (non-fatal): {e}")
logger.warning(f"Job {job_id}: Backing vocals analysis failed: {e}")
# Store empty analysis so the frontend knows analysis was attempted
job_manager.update_state_data(job_id, 'backing_vocals_analysis', {
'has_audible_content': None,
'analysis_error': str(e),
'recommended_selection': 'review_needed',
})
async def _apply_countdown_padding_if_needed(
job_id: str,
job_manager: JobManager,
+31 -68
View File
@@ -124,50 +124,31 @@ class StyleConfig:
logger.info(f"Successfully loaded style configuration")
def get_intro_format(self) -> Dict[str, Any]:
"""
Get title/intro screen format from loaded theme.
Raises:
ValueError: If no style params loaded (call load() first) or theme incomplete
"""
if not self._style_params:
raise ValueError("No style parameters loaded. Call load() first and ensure job has a theme.")
return get_intro_format(self._style_params)
"""Get title/intro screen format, with custom styles if available."""
if self._style_params:
return get_intro_format(self._style_params)
return DEFAULT_INTRO_STYLE.copy()
def get_end_format(self) -> Dict[str, Any]:
"""
Get end screen format from loaded theme.
Raises:
ValueError: If no style params loaded (call load() first) or theme incomplete
"""
if not self._style_params:
raise ValueError("No style parameters loaded. Call load() first and ensure job has a theme.")
return get_end_format(self._style_params)
"""Get end screen format, with custom styles if available."""
if self._style_params:
return get_end_format(self._style_params)
return DEFAULT_END_STYLE.copy()
def get_karaoke_format(self) -> Dict[str, Any]:
"""
Get karaoke video format from loaded theme.
Raises:
ValueError: If no style params loaded (call load() first) or theme incomplete
"""
if not self._style_params:
raise ValueError("No style parameters loaded. Call load() first and ensure job has a theme.")
return get_karaoke_format(self._style_params)
"""Get karaoke video format, with custom styles if available."""
if self._style_params:
return get_karaoke_format(self._style_params)
return DEFAULT_KARAOKE_STYLE.copy()
def get_cdg_styles(self) -> Optional[Dict[str, Any]]:
"""
Get CDG generation styles from loaded theme.
Returns None if CDG section is not present in theme.
Raises:
ValueError: If no style params loaded (call load() first) or CDG section incomplete
"""
if not self._style_params:
raise ValueError("No style parameters loaded. Call load() first and ensure job has a theme.")
return get_cdg_format(self._style_params)
"""Get CDG generation styles if available, falling back to defaults."""
if self._style_params:
cdg_styles = get_cdg_format(self._style_params)
if cdg_styles:
return cdg_styles
# Return default CDG styles if no custom styles
return DEFAULT_CDG_STYLE.copy()
def get_style_params_path(self) -> Optional[str]:
"""
@@ -187,35 +168,17 @@ class StyleConfig:
@property
def intro_video_duration(self) -> int:
"""
Get intro video duration in seconds.
Raises:
ValueError: If no style params loaded, intro section missing, or video_duration missing
"""
if not self._style_params:
raise ValueError("No style parameters loaded. Call load() first and ensure job has a theme.")
if 'intro' not in self._style_params:
raise ValueError("Missing 'intro' section in theme.")
if 'video_duration' not in self._style_params['intro']:
raise ValueError("Missing 'video_duration' in intro section.")
return self._style_params['intro']['video_duration']
"""Get intro video duration in seconds."""
if self._style_params and 'intro' in self._style_params:
return self._style_params['intro'].get('video_duration', 5)
return 5
@property
def end_video_duration(self) -> int:
"""
Get end video duration in seconds.
Raises:
ValueError: If no style params loaded, end section missing, or video_duration missing
"""
if not self._style_params:
raise ValueError("No style parameters loaded. Call load() first and ensure job has a theme.")
if 'end' not in self._style_params:
raise ValueError("Missing 'end' section in theme.")
if 'video_duration' not in self._style_params['end']:
raise ValueError("Missing 'video_duration' in end section.")
return self._style_params['end']['video_duration']
"""Get end video duration in seconds."""
if self._style_params and 'end' in self._style_params:
return self._style_params['end'].get('video_duration', 5)
return 5
async def load_style_config(job, storage: StorageService, temp_dir: str) -> StyleConfig:
+1 -9
View File
@@ -358,10 +358,6 @@ async def generate_video_orchestrated(job_id: str) -> bool:
root_span.set_attribute("duration_seconds", duration)
root_span.set_attribute("brand_code", result.brand_code or '')
logger.info(f"[job:{job_id}] WORKER_END worker=video orchestrator=true status=success duration={duration:.1f}s")
# Mark video progress as complete for idempotency
# This allows the worker to be re-triggered after admin reset
job_manager.update_state_data(job_id, 'video_progress', {'stage': 'complete'})
return True
except Exception as e:
@@ -699,12 +695,8 @@ async def generate_video_legacy(job_id: str) -> bool:
root_span.set_attribute("duration_seconds", duration)
root_span.set_attribute("brand_code", result.get('brand_code', ''))
logger.info(f"[job:{job_id}] WORKER_END worker=video status=success duration={duration:.1f}s")
# Mark video progress as complete for idempotency
# This allows the worker to be re-triggered after admin reset
job_manager.update_state_data(job_id, 'video_progress', {'stage': 'complete'})
return True
except Exception as e:
duration = time.time() - start_time
logger.error(f"[job:{job_id}] WORKER_END worker=video status=error duration={duration:.1f}s error={e}")
@@ -72,9 +72,6 @@ class OrchestratorConfig:
# Instrumental selection (clean, with_backing, or custom)
instrumental_selection: str = "clean"
# Audio synchronization - pad instrumental to match countdown-padded vocals
countdown_padding_seconds: Optional[float] = None
# Encoding backend preference
encoding_backend: str = "auto" # "auto", "local", "gce"
@@ -358,14 +355,9 @@ class VideoWorkerOrchestrator:
"job_id": self.config.job_id,
"input_gcs_path": input_gcs_path,
"output_gcs_path": output_gcs_path,
"countdown_padding_seconds": self.config.countdown_padding_seconds,
},
)
# Log countdown info for debugging
if self.config.countdown_padding_seconds:
self.job_log.info(f"Countdown padding: {self.config.countdown_padding_seconds}s - instrumental will be padded")
# Run encoding
with job_span("encoding", self.config.job_id) as span:
add_span_event("encoding_started", {"backend": encoding_backend.name})
@@ -704,13 +696,6 @@ def create_orchestrator_config_from_job(
instrumental_selection = job.state_data.get('instrumental_selection', 'clean')
existing_instrumental = getattr(job, 'existing_instrumental_gcs_path', None)
# Get countdown padding info from lyrics metadata
# This ensures instrumental is padded to match vocals if countdown was added
lyrics_metadata = job.state_data.get('lyrics_metadata', {})
countdown_padding_seconds = None
if lyrics_metadata.get('has_countdown_padding'):
countdown_padding_seconds = lyrics_metadata.get('countdown_padding_seconds', 3.0)
if existing_instrumental:
ext = Path(existing_instrumental).suffix.lower()
instrumental_path = os.path.join(temp_dir, f"{base_name} (Instrumental User){ext}")
@@ -756,9 +741,6 @@ def create_orchestrator_config_from_job(
# Instrumental selection (for GCE encoding)
instrumental_selection=instrumental_selection,
# Audio synchronization - pad instrumental to match countdown-padded vocals
countdown_padding_seconds=countdown_padding_seconds,
# Encoding backend - auto selects GCE if available
encoding_backend="auto",
+57
View File
@@ -0,0 +1,57 @@
# Cloud Build configuration for base image
#
# This builds the base image with all Python dependencies pre-installed.
# Run this manually when poetry.lock OR Dockerfile.base changes:
# gcloud builds submit --config=cloudbuild-base.yaml --project=nomadkaraoke
#
# The base image is used by the main Dockerfile for fast app builds.
# A combined hash of poetry.lock + Dockerfile.base is stored as a label for change detection.
steps:
# Calculate combined hash for change detection (poetry.lock + Dockerfile.base)
- name: 'gcr.io/cloud-builders/docker'
entrypoint: 'bash'
args:
- '-c'
- |
# Hash both files to detect changes in either
CONTENT_HASH=$$(cat poetry.lock backend/Dockerfile.base | sha256sum | cut -d' ' -f1)
echo "Combined content hash (poetry.lock + Dockerfile.base): $$CONTENT_HASH"
echo "$$CONTENT_HASH" > /workspace/CONTENT_HASH
# Build base image with all dependencies
- name: 'gcr.io/cloud-builders/docker'
entrypoint: 'bash'
args:
- '-c'
- |
CONTENT_HASH=$$(cat /workspace/CONTENT_HASH)
docker build \
--build-arg BUILD_DATE=$BUILD_ID \
--build-arg BUILD_ID=$BUILD_ID \
--build-arg BASE_CONTENT_HASH=$$CONTENT_HASH \
-t us-central1-docker.pkg.dev/$PROJECT_ID/karaoke-repo/karaoke-backend-base:latest \
-t us-central1-docker.pkg.dev/$PROJECT_ID/karaoke-repo/karaoke-backend-base:$BUILD_ID \
-f backend/Dockerfile.base \
.
# Push base image
- name: 'gcr.io/cloud-builders/docker'
args:
- 'push'
- 'us-central1-docker.pkg.dev/$PROJECT_ID/karaoke-repo/karaoke-backend-base:latest'
- name: 'gcr.io/cloud-builders/docker'
args:
- 'push'
- 'us-central1-docker.pkg.dev/$PROJECT_ID/karaoke-repo/karaoke-backend-base:$BUILD_ID'
images:
- 'us-central1-docker.pkg.dev/$PROJECT_ID/karaoke-repo/karaoke-backend-base:latest'
- 'us-central1-docker.pkg.dev/$PROJECT_ID/karaoke-repo/karaoke-backend-base:$BUILD_ID'
options:
machineType: 'N1_HIGHCPU_32'
diskSizeGb: 100
timeout: 1800s
+89
View File
@@ -0,0 +1,89 @@
# Cloud Build configuration for karaoke-backend
#
# This builds and deploys the application using the pre-built base image.
# Build time: ~1-2 minutes (vs ~8 minutes without base image)
#
# Prerequisites:
# - Base image must exist: gcloud builds submit --config=cloudbuild-base.yaml
#
# Trigger manually:
# gcloud builds submit --config=cloudbuild.yaml --project=nomadkaraoke
steps:
# Extract version from pyproject.toml and save to file for later steps
- name: 'gcr.io/cloud-builders/docker'
entrypoint: 'bash'
args:
- '-c'
- |
VERSION=$$(grep -m1 'version = ' pyproject.toml | sed 's/version = "\(.*\)"/\1/')
echo "Extracted version: $$VERSION"
echo "$$VERSION" > /workspace/VERSION
# Pull the previous app image for layer caching
- name: 'gcr.io/cloud-builders/docker'
entrypoint: 'bash'
args:
- '-c'
- |
docker pull us-central1-docker.pkg.dev/$PROJECT_ID/karaoke-repo/karaoke-backend:latest || exit 0
# Build app image (fast - just copies backend code onto base image)
- name: 'gcr.io/cloud-builders/docker'
args:
- 'build'
- '--cache-from'
- 'us-central1-docker.pkg.dev/$PROJECT_ID/karaoke-repo/karaoke-backend:latest'
- '-t'
- 'us-central1-docker.pkg.dev/$PROJECT_ID/karaoke-repo/karaoke-backend:latest'
- '-t'
- 'us-central1-docker.pkg.dev/$PROJECT_ID/karaoke-repo/karaoke-backend:$BUILD_ID'
- '-f'
- 'backend/Dockerfile'
- '.'
# Push images to Artifact Registry
- name: 'gcr.io/cloud-builders/docker'
args:
- 'push'
- 'us-central1-docker.pkg.dev/$PROJECT_ID/karaoke-repo/karaoke-backend:$BUILD_ID'
- name: 'gcr.io/cloud-builders/docker'
args:
- 'push'
- 'us-central1-docker.pkg.dev/$PROJECT_ID/karaoke-repo/karaoke-backend:latest'
# Deploy to Cloud Run with version from pyproject.toml
# Uses Cloud Tasks for worker coordination (scalable architecture)
- name: 'gcr.io/google.com/cloudsdktool/cloud-sdk'
entrypoint: 'bash'
args:
- '-c'
- |
VERSION=$$(cat /workspace/VERSION)
echo "Deploying version: $$VERSION"
gcloud run deploy karaoke-backend \
--image us-central1-docker.pkg.dev/$PROJECT_ID/karaoke-repo/karaoke-backend:$BUILD_ID \
--region us-central1 \
--platform managed \
--allow-unauthenticated \
--memory 16Gi \
--cpu 8 \
--cpu-boost \
--execution-environment gen2 \
--timeout 1800 \
--concurrency 1 \
--max-instances 50 \
--min-instances 0 \
--set-env-vars "GOOGLE_CLOUD_PROJECT=$PROJECT_ID,GCS_BUCKET_NAME=karaoke-gen-storage-$PROJECT_ID,FIRESTORE_COLLECTION=jobs,ENVIRONMENT=production,AUDIO_SEPARATOR_API_URL=https://nomadkaraoke--audio-separator-api.modal.run,KARAOKE_GEN_VERSION=$$VERSION,DEFAULT_DROPBOX_PATH=/MediaUnsynced/Karaoke/Tracks-Organized,DEFAULT_GDRIVE_FOLDER_ID=1laRKAyxo0v817SstfM5XkpbWiNKNAMSX,ENABLE_CLOUD_TASKS=true,CLOUD_RUN_SERVICE_URL=https://api.nomadkaraoke.com,GCP_REGION=us-central1,DEFAULT_ENABLE_YOUTUBE_UPLOAD=true,DEFAULT_BRAND_PREFIX=NOMAD,USE_CLOUD_RUN_JOBS_FOR_VIDEO=true" \
--set-secrets "AUDIOSHAKE_API_TOKEN=audioshake-api-key:latest,GENIUS_API_TOKEN=genius-api-key:latest,SPOTIFY_COOKIE_SP_DC=spotify-cookie:latest,RAPIDAPI_KEY=rapidapi-key:latest,DEFAULT_DISCORD_WEBHOOK_URL=discord-releases-webhook:latest,ADMIN_TOKENS=admin-tokens:latest,RED_API_KEY=red-api-key:latest,RED_API_URL=red-api-url:latest,OPS_API_KEY=ops-api-key:latest,OPS_API_URL=ops-api-url:latest,FLACFETCH_API_URL=flacfetch-api-url:latest,FLACFETCH_API_KEY=flacfetch-api-key:latest"
images:
- 'us-central1-docker.pkg.dev/$PROJECT_ID/karaoke-repo/karaoke-backend:latest'
- 'us-central1-docker.pkg.dev/$PROJECT_ID/karaoke-repo/karaoke-backend:$BUILD_ID'
options:
machineType: 'N1_HIGHCPU_32'
diskSizeGb: 100
timeout: 1200s
+13 -145
View File
@@ -44,28 +44,6 @@ Status values:
- `offline` - Worker is unavailable (includes `error` field)
- `not_configured` - Encoding worker not configured on backend
#### Flacfetch Service Health
```http
GET /health/flacfetch
```
No auth required. Returns flacfetch service status for frontend display.
Response:
```json
{
"available": true,
"status": "ok",
"version": "1.2.3"
}
```
Status values:
- `ok` - Flacfetch service is healthy and available
- `offline` - Service is unavailable (includes `error` field)
- `not_configured` - Flacfetch service not configured on backend
### Jobs
#### Create Job with Upload
@@ -115,20 +93,13 @@ DELETE /api/jobs/{job_id}
### Review
The combined review flow allows users to review lyrics AND select instrumental track in a single session.
#### Get Correction Data
```http
GET /api/review/{job_id}/correction-data
```
Returns lyrics correction data plus instrumental options for the review UI.
Response includes:
- `correction_data` - Lyrics and segments for editing
- `instrumental_options` - Available instrumental tracks (`clean`, `with_backing`)
- `backing_vocals_analysis` - Analysis data to help with selection
Returns lyrics correction data for the review UI.
#### Complete Review
@@ -138,12 +109,11 @@ Content-Type: application/json
{
"corrections": [...],
"corrected_segments": [...],
"instrumental_selection": "clean" // Required: "clean" or "with_backing"
"corrected_segments": [...]
}
```
Saves corrections, stores instrumental selection, and triggers video rendering. The instrumental selection is now required as part of the combined review flow.
Saves corrections and triggers video rendering.
#### Generate Preview
@@ -168,9 +138,7 @@ GET /api/review/{job_id}/audio/{stem_type}
Streams audio for review playback.
### Instrumental Selection (Finalise-Only Jobs)
For finalise-only jobs (where audio prep was done externally), instrumental selection is handled separately:
### Instrumental Selection
```http
POST /api/jobs/{job_id}/select-instrumental
@@ -181,8 +149,6 @@ Content-Type: application/json
}
```
**Note**: For normal jobs, instrumental selection is now part of the combined review flow (see `/api/review/{job_id}/complete` above).
### Audio Search
#### Search for Audio
@@ -325,18 +291,17 @@ POST /api/internal/jobs/{job_id}/trigger-video
| `separating_stage1` | Audio separation (1/2) |
| `separating_stage2` | Audio separation (2/2) |
| `transcribing` | Lyrics transcription |
| `generating_screens` | Title/end screens + backing vocals analysis |
| `awaiting_review` | Waiting for combined human review (lyrics + instrumental) |
| `generating_screens` | Title/end screens |
| `awaiting_review` | Waiting for human review |
| `in_review` | Human reviewing |
| `review_complete` | Review submitted with instrumental selection |
| `review_complete` | Review submitted |
| `rendering_video` | Generating karaoke video |
| `instrumental_selected` | Instrumental selection confirmed, ready for video |
| `awaiting_instrumental_selection` | Waiting for selection |
| `instrumental_selected` | Selection made |
| `generating_video` | Final encoding |
| `complete` | Done |
| `failed` | Error |
**Note**: `awaiting_instrumental_selection` exists for backwards compatibility with historical jobs but is no longer used for new jobs. Instrumental selection is now part of the combined review flow.
## Error Responses
```json
@@ -889,15 +854,8 @@ Resets a job to a specific workflow checkpoint for re-processing.
Allowed target states:
- `pending` - Restart from beginning (clears all processing data)
- `awaiting_audio_selection` - Re-select audio source
- `awaiting_review` - Re-do combined review (lyrics + instrumental selection)
- `instrumental_selected` - **Reprocess video** (preserves all settings, triggers video worker automatically)
**Note**: With the combined review flow, `awaiting_instrumental_selection` is no longer a valid reset target for new jobs. Use `awaiting_review` instead to re-select the instrumental.
The `instrumental_selected` state is useful for re-encoding and re-distributing a job after using "Delete Outputs":
1. Delete outputs → Removes YouTube/Dropbox/GDrive files, frees brand code
2. Reset to `instrumental_selected` → Clears video state, keeps settings, auto-triggers video worker
3. Job re-processes with same instrumental, lyrics, and distribution settings
- `awaiting_review` - Re-review lyrics (preserves audio stems)
- `awaiting_instrumental_selection` - Re-select instrumental (preserves review)
Response:
```json
@@ -907,66 +865,7 @@ Response:
"previous_status": "complete",
"new_status": "awaiting_review",
"message": "Job reset from complete to awaiting_review",
"cleared_data": ["review_complete", "corrected_lyrics", "instrumental_selection"],
"worker_triggered": null,
"worker_trigger_error": null
}
```
When resetting to `instrumental_selected`, the response includes worker trigger status:
```json
{
"status": "success",
"job_id": "abc123",
"previous_status": "complete",
"new_status": "instrumental_selected",
"message": "Job reset from complete to instrumental_selected",
"cleared_data": ["video_progress", "render_progress", "screens_progress", "encoding_progress", "distribution", "brand_code", "youtube_url", "youtube_video_id", "dropbox_link", "gdrive_files"],
"worker_triggered": true,
"worker_trigger_error": null
}
```
If the worker trigger fails, `worker_triggered` will be `false` and `worker_trigger_error` will contain an error message. The job reset still succeeds, but you'll need to manually trigger the worker using the endpoint below.
#### Trigger Worker (Manual)
```http
POST /api/admin/jobs/{job_id}/trigger-worker
Authorization: Bearer ADMIN_TOKEN
Content-Type: application/json
{
"worker_type": "video"
}
```
Manually triggers a worker for a job. Use this when the auto-trigger fails after a reset, or to re-run processing without resetting state.
Supported worker types:
- `video` - Video processing worker (for jobs in `instrumental_selected` state)
Response:
```json
{
"status": "success",
"job_id": "abc123",
"worker_type": "video",
"triggered": true,
"message": "Video worker triggered successfully for job abc123",
"error": null
}
```
If triggering fails:
```json
{
"status": "error",
"job_id": "abc123",
"worker_type": "video",
"triggered": false,
"message": "Failed to trigger video worker for job abc123",
"error": "Job is in wrong state for video processing"
"cleared_data": ["review_complete", "corrected_lyrics", "instrumental_selection"]
}
```
@@ -1003,35 +902,6 @@ Service statuses: `success`, `failed`, `skipped` (not configured or no data), `p
The brand code is freed for reuse when the Dropbox folder is deleted. When the job is re-processed and outputs are re-uploaded, the `outputs_deleted_at` flag is automatically cleared.
#### Clear Worker Progress
```http
POST /api/admin/jobs/{job_id}/clear-workers
Authorization: Bearer ADMIN_TOKEN
```
Clears all worker completion markers from `state_data` to allow re-execution. Use when a job has stale progress keys that prevent workers from running (e.g., after a reset that didn't fully clear state).
Worker progress keys cleared:
- `audio_progress`
- `lyrics_progress`
- `render_progress`
- `screens_progress`
- `video_progress`
- `encoding_progress`
Response:
```json
{
"status": "success",
"job_id": "abc123",
"message": "Cleared 3 worker progress keys",
"cleared_keys": ["render_progress", "video_progress", "encoding_progress"]
}
```
**Why this is needed**: Workers check `state_data.{worker}_progress.stage == 'complete'` for idempotency - if this key exists from a previous run, the worker will skip execution. When re-reviewing or resetting jobs, these keys must be cleared.
### Internal Email Endpoints
These endpoints are used by Cloud Tasks for automated notifications.
@@ -1043,9 +913,7 @@ POST /api/internal/jobs/{job_id}/check-idle-reminder
Authorization: Bearer ADMIN_TOKEN
```
Called by Cloud Tasks 5 minutes after a job enters a blocking state (primarily `awaiting_review` for combined review). Sends a reminder email if the user is still idle and no reminder has been sent yet.
**Note**: With the combined review flow, users complete both lyrics review and instrumental selection in one session. The `awaiting_instrumental_selection` state is only used for finalise-only jobs.
Called by Cloud Tasks 5 minutes after a job enters a blocking state (awaiting_review or awaiting_instrumental_selection). Sends a reminder email if the user is still idle and no reminder has been sent yet.
## Rate Limits
+27 -101
View File
@@ -33,7 +33,7 @@
(*) GCE Encoding Worker: c4d-highcpu-32 VM with AMD EPYC 9B45 (Turin) for
high-performance FFmpeg encoding (4.92x faster than previous c4-standard-8).
Used for both final video encoding and preview video generation.
Uses immutable deployment pattern - see infrastructure/encoding-worker/README.md.
Uses LocalEncodingService via GCS wheel deployment.
```
## Processing Pipeline
@@ -55,32 +55,31 @@
└────────┬────────┘ └────────┬────────┘
└──────────┬───────────┘
▼
3. SCREENS GENERATION + BACKING VOCALS ANALYSIS
3. SCREENS GENERATION
• Title screen (5s video)
• End screen (5s video)
• Analyze backing vocals track (for instrumental selection)
▼
4. COMBINED HUMAN REVIEW (AWAITING_REVIEW)
4. HUMAN REVIEW (AWAITING_REVIEW)
• React UI for lyrics correction
• Preview video generation (optionally via GCE for speed)
• Instrumental track selection (clean vs with_backing)
• Single submit saves both lyrics + instrumental choice
• Submit corrections
▼
5. VIDEO RENDERING
• Merge corrections with original data
• Generate karaoke video with lyrics
▼
6. FINAL VIDEO (INSTRUMENTAL_SELECTED)
6. INSTRUMENTAL SELECTION
• User chooses: clean or with_backing
▼
7. FINAL VIDEO
• Remux with selected instrumental
• Concatenate: title + karaoke + end
• Encode to 4 formats
▼
7. COMPLETE
8. COMPLETE
• User downloads via signed URLs
```
**Note**: As of 2026-01, instrumental selection is combined with lyrics review. Users complete both in a single session rather than in separate steps.
## Job States
| State | Description | Human Action |
@@ -90,18 +89,17 @@
| `separating_stage1` | Audio separation stage 1 | - |
| `separating_stage2` | Audio separation stage 2 | - |
| `transcribing` | Lyrics transcription | - |
| `generating_screens` | Creating title/end screens + backing vocals analysis | - |
| `awaiting_review` | **Waiting for human** | Review lyrics + select instrumental |
| `in_review` | Human actively reviewing | Edit lyrics + select instrumental |
| `review_complete` | Review submitted with instrumental selection | - |
| `generating_screens` | Creating title/end screens | - |
| `awaiting_review` | **Waiting for human** | Review lyrics |
| `in_review` | Human actively reviewing | Edit lyrics |
| `review_complete` | Review submitted | - |
| `rendering_video` | Generating karaoke video | - |
| `instrumental_selected` | Instrumental confirmed, ready for final video | - |
| `awaiting_instrumental_selection` | **Waiting for human** | Choose audio |
| `instrumental_selected` | Selection made | - |
| `generating_video` | Final encoding | - |
| `complete` | All done | Download files |
| `failed` | Error occurred | - |
**Note**: `awaiting_instrumental_selection` exists for backwards compatibility with historical jobs but is no longer used. Instrumental selection is now part of the combined review (`awaiting_review` → `in_review` → `review_complete`).
## GCS File Structure
```text
@@ -157,15 +155,12 @@ LyricsTranscriber LyricsTranscriber
|---------|---------|----------|
| Modal | GPU audio separation | Yes |
| AudioShake | Lyrics transcription | Yes |
| Vertex AI | Agentic AI correction (Gemini 3 Flash) | Default off (SKIP_CORRECTION=false to enable) |
| Vertex AI | Agentic AI correction (Gemini 3 Flash) | Default on |
| Genius | Reference lyrics | Yes |
| Flacfetch | Audio downloads (YouTube, torrents) | Recommended* |
| YouTube API | Video upload | Optional |
| SendGrid | Email notifications | Optional |
| Cloud Tasks | Delayed task scheduling (idle reminders) | Optional |
*Flacfetch runs on a dedicated GCE VM with YouTube cookies and tracker access. Without it, YouTube downloads will fail due to bot detection on Cloud Run IPs.
## Firestore Collections
karaoke-gen shares a GCP project (`nomadkaraoke`) with karaoke-decide, but uses separate Firestore collections:
@@ -214,9 +209,6 @@ The Video Worker uses an orchestrator pattern to ensure all features work regard
**Service Modules**:
| Module | Purpose |
|--------|---------|
| `youtube_download_service.py` | YouTube audio downloads (via remote flacfetch) |
| `flacfetch_client.py` | Client for remote flacfetch HTTP API |
| `audio_search_service.py` | Audio source search (YouTube, RED, OPS, Spotify) |
| `youtube_upload_service.py` | YouTube OAuth flow and upload |
| `discord_service.py` | Discord webhook notifications |
| `packaging_service.py` | CDG/TXT package generation |
@@ -227,100 +219,34 @@ The Video Worker uses an orchestrator pattern to ensure all features work regard
| `template_service.py` | GCS-backed email templates |
| `job_notification_service.py` | Email orchestration (completion, reminders) |
## Audio Source Download Flow
## Worker Coordination
Audio can come from file upload or remote search (YouTube, torrents). All YouTube downloads route through `YouTubeDownloadService` to ensure consistent handling:
```text
┌──────────────────────────────────────────────────────────────────────┐
│ Audio Source Entry Points │
│ │
│ 1. File Upload (/api/jobs/upload) │
│ └─ Direct to GCS → triggers workers │
│ │
│ 2. YouTube URL (/api/jobs/create-from-url) │
│ └─ YouTubeDownloadService → GCS → triggers workers │
│ │
│ 3. Audio Search + Select (/api/audio-search/...) │
│ ├─ YouTube → YouTubeDownloadService → GCS → triggers workers │
│ ├─ RED/OPS → FlacfetchClient (torrent) → GCS → triggers workers │
│ └─ Spotify → FlacfetchClient (spotify) → GCS → triggers workers │
└──────────────────────────────────────────────────────────────────────┘
│
▼
┌──────────────────────────────────────────────────────────────────────┐
│ YouTubeDownloadService │
│ │
│ • Single entry point for ALL YouTube downloads in cloud │
│ • Uses remote flacfetch when FLACFETCH_API_URL is set (recommended) │
│ • Falls back to local yt_dlp with warning (usually fails on CR) │
│ • Handles all URL formats (watch, youtu.be, shorts, embed) │
└──────────────────────────────────────────────────────────────────────┘
│
▼
┌──────────────────────────────────────────────────────────────────────┐
│ Flacfetch VM (GCE) │
│ │
│ • Has YouTube cookies (avoids bot detection) │
│ • Has tracker credentials (RED, OPS private trackers) │
│ • Runs BitTorrent client for seeding │
│ • Uploads directly to GCS (no data transfer through Cloud Run) │
└──────────────────────────────────────────────────────────────────────┘
```
**Key environment variables**:
- `FLACFETCH_API_URL` - URL of flacfetch VM (e.g., `http://10.x.x.x:8080`)
- `FLACFETCH_API_KEY` - API key for authentication
## Worker Execution
Workers run via two mechanisms depending on their processing duration:
### Cloud Run Jobs (Long-Running Workers)
Audio and lyrics workers run as **Cloud Run Jobs** - standalone batch containers that run to completion without HTTP request lifecycle concerns. This eliminates the instance termination issue where Cloud Run would shut down instances mid-processing.
```text
┌─────────────────────────────────────────────────────────────────┐
│ Cloud Run Jobs (via WorkerService) │
│ │
│ lyrics-transcription-job - 5-15 min (AudioShake + correction)│
│ audio-separation-job - 10-20 min (Modal API) │
│ video-encoding-job - up to 60 min (optional) │
│ │
│ Triggered via: google.cloud.run_v2.JobsClient.run_job() │
│ Job ID passed as: --job-id argument │
└─────────────────────────────────────────────────────────────────┘
```
**Why Cloud Run Jobs**: When using FastAPI BackgroundTasks, Cloud Run would terminate instances when HTTP requests completed, even if background work was still running. Job c94cc9d6 failed because the lyrics worker was killed mid-processing after AudioShake completed. Cloud Run Jobs solve this by running workers as standalone processes that complete naturally.
**CLI Entry Points**: Each worker has a `main()` function for Cloud Run Job execution:
- `python -m backend.workers.lyrics_worker --job-id abc123`
- `python -m backend.workers.audio_worker --job-id abc123`
- `python -m backend.workers.video_worker --job-id abc123`
### HTTP Workers (Fast Workers)
Screens and render workers run via internal HTTP endpoints with BackgroundTasks, as they complete quickly (~30 seconds). They still use the WorkerRegistry for coordination:
Workers run in parallel via FastAPI BackgroundTasks. To prevent Cloud Run from terminating the container when one worker completes while another is still running, workers register with a global `WorkerRegistry`.
```text
┌─────────────────────────────────────────────────────────────────┐
│ WorkerRegistry │
│ │
│ register(job_id, "screens") unregister(job_id, "screens") │
│ register(job_id, "render") unregister(job_id, "render") │
│ register(job_id, "audio") unregister(job_id, "audio") │
│ register(job_id, "lyrics") unregister(job_id, "lyrics") │
│ │
│ has_active_workers() → bool │
│ wait_for_completion(timeout) → bool │
└─────────────────────────────────────────────────────────────────┘
```
**Flow**:
1. Worker starts → `await worker_registry.register(job_id, "audio")`
2. Worker does work in try block
3. Worker completes → `await worker_registry.unregister(job_id, "audio")` in finally block
**Shutdown Handler** (in `main.py` lifespan):
- On shutdown signal, checks `worker_registry.has_active_workers()`
- If workers active, calls `wait_for_completion(timeout=600)` (10 min max)
- Logs which workers are still running for debugging
**Why needed**: Without this, when the audio worker's BackgroundTask completes, Cloud Run sees the container as idle (no active requests) and may terminate it, killing the still-running lyrics worker.
## Multitenancy
The platform supports white-label B2B portals where business customers get their own branded karaoke generation experience.
+3 -207
View File
@@ -23,16 +23,10 @@ Audio separation and lyrics transcription run in parallel. Both can fail indepen
### GCE Instance Selection for FFmpeg
AMD EPYC (C4D series) significantly outperforms Intel Xeon (C4 series) for CPU-bound FFmpeg encoding with libass. C4D-highcpu-32 is ~5x faster than C4-standard-8. C4D requires `hyperdisk-balanced` disk type.
### Combined Review Flow (Jan 2026)
When two sequential human review steps can be combined into one, do it. We originally had separate steps: (1) lyrics review, (2) instrumental selection. This doubled user friction and email notifications. By the time users enter lyrics review, audio separation is already complete - the instrumental stems are ready. We combined both into a single "Combined Review" session: users review lyrics AND select instrumental track on the same page, submit once. **Benefits**: Better UX (one interaction vs two), faster turnaround, reduced drop-off, simpler email notifications, cleaner codebase. **Implementation pattern**: Move analysis that was done after review (backing vocals analysis) to before review (in screens_worker). Include instrumental options in the correction-data endpoint. Require instrumental selection in the review completion request body. Remove the separate `AWAITING_INSTRUMENTAL_SELECTION` state from normal flow (keep for edge cases like finalise-only jobs for DB compatibility).
---
## Common Gotchas
### Fail Fast, Don't Fall Back
Silent fallbacks hide configuration errors. When critical configuration is missing (themes, credentials, etc.), raise clear errors instead of falling back to defaults. Better to fail loudly during testing than silently produce incorrect output in production. Example: Theme validation (v0.109.0) now raises `ValueError` on incomplete themes instead of merging with defaults. This ensures all cloud jobs use complete, explicit themes with no silent degradation.
### Verify Active Worktree
Before making changes in a multi-worktree setup, verify which directory the user's dev server is running from.
@@ -48,44 +42,6 @@ When multiple code paths create jobs (file upload, audio search, webhooks), use
### Fix Both Sides of Dual Code Paths
When fixing a bug in a system with multiple code paths (e.g., legacy vs orchestrator, local vs cloud), verify ALL paths are fixed. PR #271 fixed the GCE worker to READ `instrumental_selection` but only checked the legacy path which was already SENDING it. The orchestrator path (production default) wasn't sending it. **Pattern**: If a component receives config from multiple callers, check ALL callers when fixing the receiving side. Write integration tests that cover each path.
### Test Real Code, Not Just Mocked Endpoints (Jan 2026)
**What happened**: Job `5a69afd1` failed in production with `ImportError: cannot import name 'run_video_worker' from 'backend.workers.video_worker'`. This function never existed - the import was incorrect from the start. The bug was introduced in commit 29be83cb (Jan 25, 2026) and reached production the same day.
**Why tests didn't catch it**: `test_internal_api.py` mocked the entire worker function, so the import statement inside the function was never executed:
```python
patch('backend.api.routes.internal.process_render_video', mock_render)
```
This verified the HTTP endpoint worked but bypassed all real worker code, including imports.
**The fix**: Added `test_render_video_worker_integration.py` with AST-based import validation that parses the worker source code and verifies every import actually exists. This test runs on every CI build and would have caught the bug before deployment.
**Pattern for preventing similar bugs**:
1. **Mock minimally**: Don't mock entire worker functions - mock only external dependencies (Firestore, GCS, APIs)
2. **Use AST validation**: For critical worker modules, add tests that parse the source and validate all imports exist
3. **Integration tests**: Use emulator-based integration tests (in `backend/tests/emulator/`) that execute real worker code paths
**Correct worker coordination pattern**:
```python
# ✅ CORRECT: Use WorkerService
from backend.services.worker_service import get_worker_service
worker_service = get_worker_service()
await worker_service.trigger_video_worker(job_id)
# ❌ WRONG: Import non-existent function directly
from backend.workers.video_worker import run_video_worker
await run_video_worker(job_id, job_manager, storage)
```
**Example 1:** The `gcs_path` parameter bug for remote flacfetch downloads was fixed for RED/OPS torrent sources in December 2025, but the same bug existed for YouTube sources. The fix only addressed one branch of the conditional, leaving YouTube downloads broken when remote was enabled. Always search for ALL code paths that might need the same fix.
**Example 2:** The countdown audio sync fix (PR #328) only fixed the writer side (`render_video_worker` updating `lyrics_metadata.has_countdown_padding`). The orchestrator path that READS that state and pads the instrumental wasn't updated. PR #338 added the reader-side fix: `OrchestratorConfig` now has `countdown_padding_seconds`, `create_orchestrator_config_from_job` reads from lyrics_metadata, and GCE worker pads instrumental audio. **Lesson**: When adding cross-worker state, trace the data flow end-to-end through ALL code paths (legacy KaraokeFinalise path AND orchestrator path).
### Worker Idempotency Must Complete the Lifecycle
When implementing idempotency checks that set `stage='running'` at start, workers MUST also set `stage='complete'` on success. Without the completion update, retries or reprocessing attempts will be blocked because the stage is permanently stuck at `'running'`. The fix in v0.108.14 added completion markers to render_video, video, and screens workers.
### Clear Worker Progress Keys When Reprocessing
When resetting a job or re-reviewing a completed job, all worker progress keys (`*_progress`) must be cleared from `state_data`. Workers check `state_data.{worker}_progress.stage == 'complete'` for idempotency - if stale keys exist from a previous run, workers will skip execution even though the job needs reprocessing. **Pattern**: Any operation that intends to re-run workers (admin reset, review resubmission) must explicitly clear progress keys using `job_manager.delete_state_data_keys()`. See `backend/api/routes/review.py:complete_review()` and `backend/api/routes/admin.py:clear_worker_state()`.
### Defense in Depth
Enforce critical requirements at multiple layers (e.g., reject at creation in JobManager + safety net at processing time).
@@ -98,9 +54,6 @@ Auth tokens in localStorage are domain-isolated. Keep auth on a single domain or
### Standalone HTML Pages Need Auth Fallback
Standalone HTML pages (like instrumental review) that use magic link tokens should also check localStorage for user auth tokens. Priority: full auth token (doesn't expire) > magic link token (expires). This prevents logged-in users from being blocked when their magic link expires.
### Local Mode Detection Must Check Routing Pattern
When detecting local CLI mode on localhost, check both port AND routing pattern. Hash-based routing (`#/{jobId}/route`) indicates cloud mode even on localhost:3000, while path-based routing (`/local/route`) indicates local CLI mode. This allows E2E tests to run in cloud mode on localhost without being misdetected as local CLI mode. **Implementation**: `isLocalMode()` should return false when it detects the hash-based routing pattern `#/?[^/]+/(review|instrumental)`, regardless of hostname/port.
### Tab Visibility Refresh
Use `visibilitychange` event for immediate refresh when user returns to tab, not just polling intervals.
@@ -113,9 +66,6 @@ Update job, then verify update is visible before triggering workers to avoid rea
### Stale Object References
After calling async functions that update the database, re-fetch objects before making additional updates to avoid overwriting changes.
### Cross-Worker State Communication
When deferring processing from one worker to another, you MUST update the job state to communicate what was done. The countdown padding bug regressed multiple times because `render_video_worker` added countdown padding to vocals/audio but didn't update `lyrics_metadata.has_countdown_padding`, so `video_worker` didn't know to pad the instrumental, causing audio desync. **Pattern**: If Worker A performs an action that Worker B must know about, Worker A MUST write that info to `job.state_data` immediately after the action. Don't rely on implicit assumptions between workers.
### Google Drive Query Escaping
Escape special chars in Google Drive API queries: `'` → `\'`, `\` → `\\\\`.
@@ -128,17 +78,11 @@ Sanitize user input (artist/title) to ASCII for HTTP headers. Smart quotes, em d
### Fonts in Docker
Base Docker images have no fonts. Install `fonts-noto-core`, `fonts-noto-cjk` for video rendering.
### Validate External API Format Support
Don't assume external APIs support all common formats. AudioShake only supports `.wav`, `.mp3`, `.aac`, `.flac`, `.aiff`, `.mp4`, `.mov` - NOT `.webm`, `.ogg`, `.m4a`, `.opus`. When remote flacfetch downloads YouTube audio as `.webm`, the lyrics worker must convert to FLAC before uploading. **Pattern**: Use a whitelist of known-supported formats and convert everything else, rather than trying to upload directly and hoping it works. Always check the API's supported format documentation.
### Cloud Run CPU Throttling Kills Background Tasks
Cloud Run throttles CPU to near-zero when the main request handler returns, even if background tasks are running. This caused lyrics processing (running as a FastAPI background task) to slow from 17-52 seconds to 8+ minutes, and instances being terminated mid-processing. **Fix**: Add `--no-cpu-throttling` flag to `gcloud run deploy`. Keep `--cpu-boost` for faster cold starts. **Diagnosis**: Look for "Application shutdown" in logs during long operations, and compare processing times (27x slowdown is a telltale sign).
### Cloud Run Premature Shutdown with Parallel Workers
When multiple workers run in parallel via BackgroundTasks, Cloud Run can shut down when one worker completes, killing others mid-processing. This happens because each worker endpoint returns quickly after spawning its BackgroundTask - when the audio worker's task completes, Cloud Run sees the container as idle. **Initial Fix**: Implement a `WorkerRegistry` that tracks active workers per job. Register at worker start, unregister in finally block. Add shutdown handler in FastAPI lifespan that calls `worker_registry.wait_for_completion(timeout=600)` before allowing shutdown. **Pattern**: `backend/workers/registry.py` provides the global registry; workers import and call `await worker_registry.register(job_id, "audio")` / `unregister()`.
### Cloud Run Jobs for Long-Running Workers (Better Solution)
The WorkerRegistry approach helped but didn't fully solve the problem - job c94cc9d6 was still killed mid-processing because Cloud Run sent "Shutting down user disabled instance" while lyrics worker was still running. **Better fix**: Use Cloud Run Jobs for long-running workers (lyrics, audio). Cloud Run Jobs run to completion without HTTP request lifecycle concerns. Workers have CLI entry points (`python -m backend.workers.lyrics_worker --job-id abc123`) and are triggered via `google.cloud.run_v2.JobsClient.run_job()`. Jobs can run up to 24 hours vs 30-minute HTTP timeout. Keep fast workers (screens, render) on HTTP with WorkerRegistry. **Pattern**: See `infrastructure/modules/cloud_run.py` for job definitions and `backend/services/worker_service.py` for triggering logic.
When multiple workers run in parallel via BackgroundTasks, Cloud Run can shut down when one worker completes, killing others mid-processing. This happens because each worker endpoint returns quickly after spawning its BackgroundTask - when the audio worker's task completes, Cloud Run sees the container as idle. **Fix**: Implement a `WorkerRegistry` that tracks active workers per job. Register at worker start, unregister in finally block. Add shutdown handler in FastAPI lifespan that calls `worker_registry.wait_for_completion(timeout=600)` before allowing shutdown. **Pattern**: `backend/workers/registry.py` provides the global registry; workers import and call `await worker_registry.register(job_id, "audio")` / `unregister()`.
### Add Progress Logging for Long Operations
Operations over ~30 seconds should log progress periodically (time-based, not count-based). Count-based logging like "log every 20 items" can miss entirely if items complete slowly. Time-based logging (every 30s) ensures visibility regardless of processing speed.
@@ -176,98 +120,9 @@ Testing webhook creates correct DTO isn't enough. Also test that manager/service
### E2E Mocks Must Match Types
Mock responses must match TypeScript interfaces exactly. Silent failures from type mismatches are hard to debug.
### Mocks Hide API Contract Mismatches
When E2E tests use mocked API endpoints, the mocks define what you *think* the API is, not what it *actually* is. If frontend code calls a non-existent endpoint (`/api/jobs/{id}/lyrics`), a mock responding to that path will pass—but production will 404. **Pattern**: (1) Add unit tests that verify exact endpoint URLs, (2) Add at least one integration test that calls the real backend, (3) When mocks fail in ways that don't match prod, suspect a contract mismatch. Example: The "Add Reference Lyrics" feature had frontend calling `/api/jobs/{id}/lyrics` while backend expected `/api/review/{id}/add-lyrics`. Mocks used the wrong path too, so regression tests passed while production failed with 404.
### YouTube URL Downloads Must Happen Before Workers (Jan 2026)
**Problem**: Job 811ec4e5 (made_for_you order with YouTube URL) failed with "Failed to download audio file". Workers were triggered but `input_media_gcs_path` was null.
**Root cause**: The `_handle_made_for_you_order` webhook handler triggered workers directly for YouTube URLs without downloading audio first via `YouTubeDownloadService`. The correct flow (used in `file_upload.py` and `audio_search.py`) is:
1. Detect YouTube URL
2. Use `YouTubeDownloadService.download()` to download audio to GCS
3. Update job with `input_media_gcs_path`
4. THEN trigger workers
**Why tests didn't catch it**: The test `test_youtube_url_order_skips_search` mocked the entire `worker_service`, verifying only that `trigger_audio_worker.assert_called_once()`. The mock hid the fact that workers require `input_media_gcs_path` to be set, which only happens after `YouTubeDownloadService.download()` completes.
**Pattern for preventing similar bugs**:
1. **Test prerequisites, not just calls**: When testing worker triggers, verify all fields workers need are set BEFORE triggers
2. **Use call-order tracking**: Track sequence of operations to verify `download → update job → trigger workers`
3. **AST import validation**: Verify the handler imports and uses required services (same pattern as `test_job_creation_regression.py`)
**Example test pattern**:
```python
call_order = []
mock_youtube_service.download = AsyncMock(
side_effect=lambda **kwargs: call_order.append('download')
)
mock_worker_service.trigger_audio_worker = AsyncMock(
side_effect=lambda x: call_order.append('audio_worker')
)
# After calling handler...
assert 'download' in call_order, "YouTube download must be called"
download_index = call_order.index('download')
worker_index = call_order.index('audio_worker')
assert download_index < worker_index, "Download must happen BEFORE workers"
```
### Use data-testid for E2E
Prefer `data-testid` over label/text selectors. They're immune to label changes and won't break when similar fields are added.
### Debounced Inputs for Formatted Controlled Components
**Problem**: Controlled number inputs with immediate formatting (e.g., `.toFixed(2)`) on every keystroke cause cursor jumping. When user types "2", the value updates to "2.00", and the cursor jumps to the end. User can't type multi-digit numbers like "12.34" smoothly.
**Root cause**: React's controlled input pattern updates `value` prop on every `onChange`, triggering re-render. When the value changes from "2" to "2.00", React resets the cursor position to the end.
**Solution**: Create a debounced input component with local state:
1. Maintain local string state while input is focused
2. Debounce updates to parent (e.g., 300ms) for live preview updates
3. Format and sync immediately on blur
4. Ignore parent updates while focused (user has control)
**Implementation pattern**:
```tsx
const [localValue, setLocalValue] = useState('')
const [isFocused, setIsFocused] = useState(false)
// Only sync from parent when not focused
useEffect(() => {
if (!isFocused) {
setLocalValue(value?.toFixed(2) ?? '')
}
}, [value, isFocused])
// Debounce updates to parent
const debouncedUpdate = useCallback((val: string) => {
if (debounceRef.current) clearTimeout(debounceRef.current)
debounceRef.current = setTimeout(() => {
onChange(parseFloat(val) || null)
}, 300)
}, [onChange])
// Show local value when focused, formatted parent value when not
<Input
value={isFocused ? localValue : (value?.toFixed(2) ?? '')}
onFocus={(e) => { setIsFocused(true); e.target.select() }}
onChange={(e) => { setLocalValue(e.target.value); debouncedUpdate(e.target.value) }}
onBlur={() => { setIsFocused(false); onChange(parseFloat(localValue) || null) }}
/>
```
**Benefits**:
- Smooth typing experience (no cursor jumping)
- Live preview updates after debounce delay
- Formatted display when not editing
- Immediate sync on blur (no lost edits)
**When to use**: Any controlled input that needs formatting but must accept intermediate invalid states (partial numbers, decimals being typed, etc.). Common for number inputs with decimal formatting, currency inputs, percentage inputs.
**Testing**: Verify (1) typing multi-digit values doesn't jump cursor, (2) debounce fires after delay, (3) blur syncs immediately, (4) external updates respected when not focused. See `frontend/components/lyrics-review/__tests__/TimeInput.test.tsx` for comprehensive test suite.
### Emulator Tests Catch Real Bugs
Firestore emulator tests catch issues (like missing indexes) that unit tests with mocks miss.
@@ -277,37 +132,6 @@ Use `expect(locator).toBeVisible()` instead of `page.waitForTimeout()`. Assertio
### Playwright: 'load' Not 'networkidle'
Use `waitUntil: 'load'` for audio-heavy pages. `'networkidle'` can timeout waiting for streaming audio.
### Always Test the Full Data Flow, Not Just Individual Components
**Problem (Jan 2026)**: Lyrics corrections made during human review were being silently discarded. Final videos rendered with raw transcription instead of user-corrected lyrics.
**Why tests didn't catch it**:
1. **Component tests were isolated**: LyricsAnalyzer tests verified corrections saved to server. InstrumentalSelector tests verified submission worked. But no test verified the *connection* between them.
2. **E2E tests used mocks**: Mock API responses always returned consistent data, hiding the fact that the real API returned different data than what was saved.
3. **Backend tests checked structure, not behavior**: The `test_routes_review.py` tests documented expected behavior but didn't actually test the endpoint logic.
**Root cause**:
- `POST /api/jobs/{job_id}/corrections` saved to `corrections_updated.json` ✓
- `GET /api/review/{job_id}/correction-data` returned `corrections.json` (original) ✗
- `POST /api/review/{job_id}/complete` received original data and overwrote the user's edits ✗
**The fix**: Make `get_correction_data` check for `corrections_updated.json` first (same pattern as `render_video_worker.py`).
**Prevention pattern**:
1. **Integration tests for data round-trips**: Test that data saved by component A is retrievable by component B
2. **Contract tests**: Verify endpoints return what other endpoints expect
3. **Real backend E2E tests**: At least one test per flow that hits the actual backend, not mocks
4. **When two workers use the same data, verify they use it consistently**: Both `get_correction_data` and `render_video_worker` needed to check `corrections_updated.json` first
**Test added**:
```python
def test_corrections_preserved_through_combined_review():
# 1. Create job with original corrections
# 2. Submit edited corrections (saves to corrections_updated.json)
# 3. Fetch correction data (should return corrections_updated content!)
# 4. Verify returned data contains the EDITED corrections
```
---
## Deployment Notes
@@ -339,33 +163,8 @@ Use `pulumi up --skip-preview --yes` when CI service account can't run preview.
### Docker Disk Management
Self-hosted runners need aggressive disk cleanup. Use threshold-based (70%) not age-based cleanup.
### Version Sort for Artifact Selection (SUPERSEDED)
**Note:** This lesson is superseded by the "Immutable Deployment Pattern" below, which eliminates version sorting entirely.
Original lesson: When downloading multiple versioned files (wheels, tarballs), use `sort -V | tail -1` not `ls -t | head -1`. Files downloaded simultaneously have the same timestamp, so time-based sorting picks arbitrarily.
### Immutable Deployment Pattern for GCE Workers
When deploying to long-running VMs (like the encoding worker), avoid baking deployment logic into Packer images:
**Problem:** The encoding worker had recurring version mismatch issues because:
1. `startup.sh` was baked into the Packer image
2. Fixing logic required image rebuild + VM recreation
3. CI only warned on version mismatch, didn't fail
4. Multiple deployment vectors (CI, Pulumi, manual SSH) caused drift
**Solution:** Self-updating startup script from GCS:
1. Packer image contains minimal `bootstrap.sh` (~15 lines, rarely changes)
2. `bootstrap.sh` downloads real `startup.sh` from GCS on every service start
3. CI uploads `startup.sh`, wheel, and version manifest to GCS
4. CI **strictly verifies** deployed version - fails on mismatch
**Benefits:**
- Logic changes via CI, no Packer rebuild needed
- Fixed wheel path (`karaoke_gen-current.whl`) eliminates sorting bugs
- Version verification is mandatory, not optional
- Single source of truth (GCS), no drift from manual changes
See `infrastructure/encoding-worker/README.md` for implementation details.
### Version Sort for Artifact Selection
When downloading multiple versioned files (wheels, tarballs), use `sort -V | tail -1` not `ls -t | head -1`. Files downloaded simultaneously have the same timestamp, so time-based sorting picks arbitrarily. Version sorting (`sort -V`) correctly identifies the highest semantic version.
---
@@ -380,9 +179,6 @@ Use double-checked locking (`if not X: with lock: if not X: init()`) for shared
### Preload at Startup
Load SpaCy models, NLTK data, Langfuse handlers at container startup, not lazily during requests. Saves 60-200s on cold starts.
### Langfuse v3 OTEL Isolation
Langfuse v3 is built on OpenTelemetry. If `CallbackHandler()` is created without a pre-configured `Langfuse` client, it installs itself as the **global** OTEL tracer provider, capturing ALL spans (FastAPI requests, HTTP clients, etc.) - not just LLM calls. Fix: Initialize `Langfuse(tracer_provider=TracerProvider())` with an isolated provider BEFORE creating `CallbackHandler()`. See: https://github.com/orgs/langfuse/discussions/9136
### Cold Start Mitigation
Set `min-instances > 0` for Cloud Run services with heavy initialization.
+4 -29
View File
@@ -17,9 +17,10 @@
|-----------|--------|
| Audio upload & separation | Working |
| Lyrics transcription | Working |
| Auto-correction (agentic + heuristic) | Disabled (raw transcription goes to human review) |
| Combined review (lyrics + instrumental) | Working |
| Agentic AI correction | Working (Gemini 2 Flash via Vertex AI) |
| Human lyrics review | Working |
| Preview video generation | Working |
| Instrumental selection | Working |
| Multi-format encoding | Working |
| Token-based auth | Working |
| Magic link auth | Working |
@@ -41,35 +42,9 @@
## Recent Changes
- **Made-For-You YouTube URL Download Fix** (2026-01-28): Fixed bug where made-for-you orders with YouTube URLs failed with "Failed to download audio file". **Root cause**: The webhook handler triggered workers directly without downloading the audio first via `YouTubeDownloadService`. Workers require `input_media_gcs_path` to be set, which only happens after download completes. **Fix**: Handler now downloads YouTube audio to GCS before triggering workers, matching the pattern in `file_upload.py` and `audio_search.py`. Added 4 new contract tests with call-order tracking and AST import validation to prevent regression. See [LESSONS-LEARNED.md](LESSONS-LEARNED.md#youtube-url-downloads-must-happen-before-workers-jan-2026).
- **Lyrics Corrections Now Applied to Video** (2026-01-26): Fixed critical bug where user's lyrics corrections made during human review were being ignored. Videos were rendering with raw transcription instead of corrected lyrics. **Root cause**: The `GET /api/review/{job_id}/correction-data` endpoint was returning the original `corrections.json` instead of `corrections_updated.json` (where user edits are saved). The combined review flow would then overwrite the user's edits when completing the review. **Fix**: The endpoint now checks for `corrections_updated.json` first, matching the behavior of `render_video_worker.py`. Added 35 new tests (backend unit + integration + frontend E2E) to prevent regression. See [LESSONS-LEARNED.md](LESSONS-LEARNED.md#always-test-the-full-data-flow-not-just-individual-components).
- **Instrumental Review UX Improvements** (2026-01-26): Fixed three UX issues in the instrumental review flow: (1) Loading spinner now visible when switching audio tabs (shows on active tab with disabled state on others), (2) Removed large gap under waveform by fixing flexbox layout, (3) Success screen now appears in cloud mode with countdown before redirect (previously only showed in local CLI mode). Also fixed `isLocalMode()` detection to correctly distinguish cloud mode (hash-based routing like `#/jobId/review`) from local CLI mode (path-based routing like `/local/review`) on localhost, allowing E2E tests to run in cloud mode. Added 17 unit tests for `local-mode.ts` and 12 E2E regression tests for instrumental review UX. Backend fix: Changed `audio_urls['backing']` to `audio_urls['backing_vocals']` to match frontend expectations.
- **Dashboard Performance & UX Improvements** (2026-01-25): Improved recent jobs list loading performance and usability. Changes: (1) Reduced default job limit from 20 to 5 for faster initial load (typical users only need to see recent jobs), (2) Added job limit dropdown (5/10/20/All options) for power users who want more history, (3) Implemented priority sorting - blocking jobs (awaiting user action) now appear first, followed by processing, completed, and failed jobs, (4) Made OutputLinks buttons responsive by hiding text labels on mobile (icons-only) to prevent horizontal overflow. Added 14 comprehensive unit tests for priority sorting logic. Frontend-only changes, no API modifications.
- **Combined Lyrics + Instrumental Review** (2026-01-24): Merged the two-step human review process (lyrics review, then instrumental selection) into a single unified flow. Users now review lyrics AND select instrumental track in one session, submitting both with a single click. **Benefits**: Better UX (one interaction vs two), reduced drop-off, single email notification instead of two. **Technical changes**: Backing vocals analysis moved to screens_worker (runs before review), instrumental options included in `/api/review/{job_id}/correction-data` response, `POST /api/review/{job_id}/complete` now requires `instrumental_selection` field. The `AWAITING_INSTRUMENTAL_SELECTION` state is retained for finalise-only jobs but no longer used in the normal flow. See [ARCHITECTURE.md](ARCHITECTURE.md#processing-pipeline) and [LESSONS-LEARNED.md](LESSONS-LEARNED.md#combined-review-flow-jan-2026).
- **Edit Segment Time Input UX Fix** (2026-01-24): Fixed cursor jumping issue when typing multi-digit time values in the edit segment modal. Replaced controlled number inputs with debounced `TimeInput` component that maintains local state while focused and only syncs to parent after 300ms or on blur. This allows users to smoothly type values like "12.34" without the cursor jumping to the end on each keystroke. Added 15 comprehensive unit tests and E2E regression test to prevent future regressions. See [LESSONS-LEARNED.md](LESSONS-LEARNED.md#debounced-inputs-for-formatted-controlled-components).
- **Cloud Run Jobs for Workers** (2026-01-24): Migrated lyrics and audio workers from FastAPI BackgroundTasks to Cloud Run Jobs. This fixes instance termination issues where Cloud Run would shut down containers mid-processing (e.g., job c94cc9d6 was killed after AudioShake completed but before anchor sequence processing finished). Workers now have CLI entry points (`python -m backend.workers.lyrics_worker --job-id abc123`) and run as standalone batch containers. Cloud Run Jobs run to completion without HTTP lifecycle concerns, with up to 24-hour execution time. Infrastructure changes via Pulumi define `lyrics-transcription-job` and `audio-separation-job`. See [ARCHITECTURE.md](ARCHITECTURE.md#worker-execution) and [LESSONS-LEARNED.md](LESSONS-LEARNED.md#cloud-run-jobs-for-long-running-workers-better-solution).
- **Encoding Worker Immutable Deployment** (2026-01-22): Implemented robust deployment pattern for GCE encoding worker to eliminate version mismatch issues. Changes: (1) Fixed wheel path `karaoke_gen-current.whl` eliminates version sorting bugs, (2) Self-updating startup script downloaded from GCS on every service start (no Packer rebuild needed for logic changes), (3) Version manifest with strict CI verification (deployment fails on mismatch). Root cause: startup.sh was baked into Packer image, requiring image rebuild + VM recreation for fixes. New pattern allows CI to update deployment logic without infrastructure changes. See [infrastructure/encoding-worker/README.md](../infrastructure/encoding-worker/README.md) and [LESSONS-LEARNED.md](LESSONS-LEARNED.md#immutable-deployment-pattern-for-gce-workers).
- **Theme Validation & Default Removal** (2026-01-22): **BREAKING CHANGE in v0.109.0** - Removed all default style fallback logic to ensure cloud jobs always use complete themes. Style getter functions now raise `ValueError` if themes are incomplete instead of merging with defaults. Added `--list-themes` and `--validate-theme` CLI flags. All cloud jobs guaranteed to use complete nomad theme with no silent fallbacks. See [archive/2026-01-22-theme-validation-remove-defaults.md](archive/2026-01-22-theme-validation-remove-defaults.md).
- **Admin Job Reprocess** (2026-01-21, **improved 2026-01-24**): Added "Reprocess" button to admin job reset toolbar. Allows re-encoding and re-distributing a job with existing settings after deleting outputs. Resets job to `instrumental_selected` state and automatically triggers video worker. Clears video/encoding/distribution state while preserving instrumental selection and lyrics metadata. Use case: Fix distribution issues, re-upload to YouTube with different settings, or re-run encoding after infrastructure changes. **Improvements (2026-01-24)**: API response now includes `worker_triggered` and `worker_trigger_error` fields for transparency. Added new `/api/admin/jobs/{job_id}/trigger-worker` endpoint for manual worker triggering when auto-trigger fails. Frontend shows detailed result dialog after reset with status change, cleared data, and worker trigger status. Added "Trigger" button in admin toolbar for manual worker triggering. See [API.md](API.md#reset-job-state).
- **Countdown Audio Sync Fix** (2026-01-21, **completed 2026-01-24**): Fixed bug where songs starting within 3 seconds would have audio desync. PR #328 fixed the writer side: `render_video_worker` now updates `lyrics_metadata.has_countdown_padding`. PR #338 fixed the reader side: `video_worker_orchestrator` now reads the countdown state and passes it through to GCE encoding, which pads the instrumental audio to match. This was a "Fix Both Sides of Dual Code Paths" bug - see [LESSONS-LEARNED.md](LESSONS-LEARNED.md#fix-both-sides-of-dual-code-paths). Added comprehensive test suite (contract tests, pipeline integration, orchestrator countdown tests) to prevent regression. See [archive/2026-01-21-countdown-audio-sync-testing-strategy.md](archive/2026-01-21-countdown-audio-sync-testing-strategy.md).
- **Anchor Sequences Fix** (2026-01-21): Fixed anchor sequence calculation when auto-correction is disabled. Previously, disabling auto-correction (`SKIP_CORRECTION=true`) also stopped anchor/gap calculation, leaving reviewers without guidance on uncertain lyrics areas. Now anchor sequences are calculated independently, so reviewers always see highlighted matches against reference lyrics regardless of correction settings.
- **Auto-Correction Disabled** (2026-01-20): Disabled all auto-correction during lyrics transcription (both agentic AI via Gemini and heuristic rule-based handlers). Raw transcription now goes directly to human review without automatic fixes. This change was made because auto-correction was creating more work for reviewers by introducing errors that needed manual correction. Anchor sequences are still calculated to help reviewers identify uncertain areas. The correction feature can be re-enabled by setting `SKIP_CORRECTION=false` environment variable. See `backend/config.py` for configuration.
- **Worker Coordination Fix** (2026-01-20): Fixed bug where Cloud Run could shut down prematurely when one parallel worker completed while another was still running. Added `WorkerRegistry` to track active workers per job. Workers register at start and unregister in finally block. FastAPI lifespan shutdown handler now waits up to 10 minutes for active workers to complete before allowing container termination. Fixes job failures where lyrics worker was killed mid-processing after audio worker completed. See [ARCHITECTURE.md](ARCHITECTURE.md#worker-coordination) and [LESSONS-LEARNED.md](LESSONS-LEARNED.md#cloud-run-premature-shutdown-with-parallel-workers).
- **Admin Delete Outputs** (2026-01-10, **UX improved 2026-01-24**): Added admin button to delete all distributed outputs (YouTube, Dropbox, Google Drive) for a job while preserving the job record. Use case: fix quality issues by deleting outputs, resetting to `awaiting_review`, correcting lyrics, and re-generating. The brand code is freed for reuse when Dropbox folder is deleted. New endpoint: `POST /api/admin/jobs/{job_id}/delete-outputs`. UI: Delete Outputs button in admin job detail page, "Outputs Deleted" badge when flag is set. **UX improvements (2026-01-24)**: Download buttons (4K, 720p, CDG, TXT, etc.) are now hidden when outputs have been deleted, preventing broken links. Delete confirmation now shows a detailed result dialog with per-service status (YouTube video ID deleted, Dropbox folder path, Google Drive files) instead of a simple toast. See [API.md](API.md#delete-job-outputs).
- **Admin Delete Outputs** (2026-01-10): Added admin button to delete all distributed outputs (YouTube, Dropbox, Google Drive) for a job while preserving the job record. Use case: fix quality issues by deleting outputs, resetting to `awaiting_review`, correcting lyrics, and re-generating. The brand code is freed for reuse when Dropbox folder is deleted. New endpoint: `POST /api/admin/jobs/{job_id}/delete-outputs`. UI: Delete Outputs button in admin job detail page, "Outputs Deleted" badge when flag is set. See [API.md](API.md#delete-job-outputs).
- **Rate Limiting & Abuse Prevention** (2026-01-09): Added per-user rate limiting (5 jobs/day default), system-wide YouTube upload limits (10/day), and beta enrollment abuse prevention. Features: email normalization (Gmail alias detection), disposable domain blocklist (130+ domains), blocked email/IP lists, IP-based enrollment rate limiting (1 per 24h), and admin UI for managing blocklists and user overrides at `/admin/rate-limits`. Admins can grant users bypass permissions or custom limits. All limits configurable via environment variables. See [API.md](API.md#rate-limits).
@@ -1,410 +0,0 @@
# Countdown Audio Sync - Testing Strategy Plan
## Problem Summary
The countdown padding sync bug has been fixed and re-introduced multiple times:
- **PR #35** (Dec 19) - Fixed by storing `lyrics_metadata.has_countdown_padding` in job state
- **Commit 989abeaf + PR #64** (Dec 27) - Re-broken by deferring countdown to render phase without updating state sync
**Root cause:** `render_video_worker.py` adds countdown but doesn't update `job.state_data.lyrics_metadata.has_countdown_padding`, so `video_worker.py` doesn't know to pad the instrumental.
## Testing Strategy Overview
We need **multiple layers of tests** to catch this regression at different points:
```
┌─────────────────────────────────────────────────────────────────────┐
│ TEST PYRAMID │
├─────────────────────────────────────────────────────────────────────┤
│ │
│ Layer 4: E2E / Acceptance │
│ ───────────────────────── │
│ • Verify final video has correct audio sync (< 100ms drift) │
│ • Run against real backend or emulators │
│ • Slow but catches integration issues │
│ │
│ Layer 3: Pipeline Integration │
│ ───────────────────────────── │
│ • Test full worker sequence: lyrics → render → video │
│ • Verify state propagation between workers │
│ • Use emulators for Firestore │
│ │
│ Layer 2: Contract Tests │
│ ─────────────────────── │
│ • Verify render_video_worker WRITES countdown state │
│ • Verify video_worker READS and USES countdown state │
│ • Fast, mock-based, run on every PR │
│ │
│ Layer 1: Unit Tests │
│ ────────────────── │
│ • Test CountdownProcessor logic │
│ • Test state update function │
│ • Test each code path in isolation │
│ │
└─────────────────────────────────────────────────────────────────────┘
```
---
## Layer 1: Unit Tests
### 1.1 Test: render_video_worker updates countdown state
**File:** `backend/tests/test_workers.py` (extend `TestRenderVideoWorkerCountdownPadding`)
**Purpose:** Verify that when countdown is added, state is updated
```python
class TestRenderVideoWorkerCountdownStateSync:
"""Tests that render_video_worker correctly updates job state when countdown is added."""
def test_updates_lyrics_metadata_when_countdown_added(self):
"""When CountdownProcessor.process() returns padding_added=True,
job.state_data.lyrics_metadata.has_countdown_padding must be set to True."""
def test_updates_countdown_padding_seconds_in_state(self):
"""When countdown is added, the padding_seconds value must be stored."""
def test_no_state_update_when_countdown_not_needed(self):
"""When song starts after 3 seconds, has_countdown_padding stays False."""
```
### 1.2 Test: video_worker reads and uses countdown state
**File:** `backend/tests/test_workers.py` (extend `TestVideoWorker`)
**Purpose:** Verify video_worker correctly reads state and passes to encoding
```python
class TestVideoWorkerCountdownSync:
"""Tests that video_worker reads countdown state and applies instrumental padding."""
def test_reads_countdown_padding_from_lyrics_metadata(self):
"""When has_countdown_padding=True, countdown_padding_seconds is extracted."""
def test_passes_countdown_to_karaoke_finalise(self):
"""Legacy path: countdown_padding_seconds passed to KaraokeFinalise constructor."""
def test_passes_countdown_to_orchestrator(self):
"""Orchestrator path: countdown info passed through encoding input."""
def test_passes_countdown_to_gce_encoding(self):
"""GCE path: countdown_padding_seconds included in encoding_config."""
```
### 1.3 Test: All encoding paths handle countdown padding
**File:** `backend/tests/test_encoding_countdown.py` (new file)
```python
class TestEncodingCountdownPadding:
"""Tests that all encoding backends correctly pad instrumental when countdown present."""
def test_local_encoding_pads_instrumental(self):
"""LocalEncodingService pads instrumental when countdown_padding_seconds set."""
def test_gce_encoding_config_includes_countdown(self):
"""GCE encoding config dict includes countdown_padding_seconds field."""
def test_karaoke_finalise_pads_instrumental(self):
"""KaraokeFinalise.remux_with_instrumental() creates padded instrumental."""
```
---
## Layer 2: Contract Tests
**Purpose:** Ensure the data contract between workers is maintained. These tests explicitly verify the "handshake" between components.
### 2.1 State Contract Test
**File:** `backend/tests/test_countdown_state_contract.py` (new file)
```python
class TestCountdownStateContract:
"""
Contract tests for countdown padding state synchronization.
These tests verify the data contract between:
- render_video_worker (WRITER) - must set has_countdown_padding after adding countdown
- video_worker (READER) - must read and use has_countdown_padding
If either side changes its behavior, these tests will fail,
alerting us to a potential sync issue.
"""
# Contract: State schema
def test_lyrics_metadata_schema_has_countdown_fields(self):
"""lyrics_metadata must have has_countdown_padding (bool) and countdown_padding_seconds (float)."""
# Contract: Writer (render_video_worker)
def test_render_video_worker_sets_countdown_state_when_padding_added(self):
"""WRITER CONTRACT: render_video_worker MUST call update_state_data with countdown info."""
# Contract: Reader (video_worker)
def test_video_worker_reads_countdown_state(self):
"""READER CONTRACT: video_worker MUST read lyrics_metadata.has_countdown_padding."""
def test_video_worker_applies_padding_when_state_true(self):
"""READER CONTRACT: video_worker MUST pass countdown_padding_seconds when state is True."""
# Contract: End-to-end state flow
def test_state_written_by_render_can_be_read_by_video(self):
"""State written by render_video_worker can be correctly read by video_worker."""
```
### 2.2 Encoding Interface Contract
**File:** `backend/tests/test_encoding_interface_contract.py` (new file)
```python
class TestEncodingInterfaceContract:
"""Contract tests for encoding interface countdown support."""
def test_encoding_input_supports_countdown_field(self):
"""EncodingInput dataclass must have countdown_padding_seconds field."""
def test_all_backends_accept_countdown_parameter(self):
"""All encoding backends (local, GCE) must accept countdown parameter."""
```
---
## Layer 3: Pipeline Integration Tests
**Purpose:** Test the full worker sequence with real Firestore emulator to verify state flows correctly.
### 3.1 Worker Pipeline Test
**File:** `backend/tests/emulator/test_countdown_pipeline_integration.py` (new file)
```python
class TestCountdownPipelineIntegration:
"""
Integration tests for countdown padding across the full worker pipeline.
Uses Firestore emulator to test real state updates.
Mocks only external services (Modal, AudioShake, etc.)
"""
@pytest.fixture
def job_with_early_start_song(self, db_client):
"""Create a job where lyrics start within 3 seconds (needs countdown)."""
@pytest.fixture
def job_with_late_start_song(self, db_client):
"""Create a job where lyrics start after 3 seconds (no countdown needed)."""
async def test_countdown_state_flows_through_pipeline(self, job_with_early_start_song):
"""
Full pipeline test:
1. Run lyrics_worker (sets has_countdown_padding=False due to deferred countdown)
2. Run render_video_worker (adds countdown, MUST set has_countdown_padding=True)
3. Verify state_data has correct countdown info
4. Run video_worker (MUST read state and pad instrumental)
5. Verify countdown_padding_seconds was passed to encoding
"""
async def test_no_countdown_state_when_not_needed(self, job_with_late_start_song):
"""When song doesn't need countdown, state remains False throughout pipeline."""
async def test_orchestrator_path_receives_countdown(self, job_with_early_start_song):
"""Orchestrator path correctly receives and uses countdown info."""
async def test_gce_encoding_path_receives_countdown(self, job_with_early_start_song):
"""GCE encoding path correctly receives countdown in config."""
```
### 3.2 State Persistence Test
**File:** `backend/tests/emulator/test_countdown_state_persistence.py` (new file)
```python
class TestCountdownStatePersistence:
"""Tests that countdown state survives worker boundaries."""
async def test_state_persists_between_render_and_video_workers(self):
"""State set by render_video_worker is readable by video_worker."""
async def test_state_survives_job_reload(self):
"""State persists when job is reloaded from Firestore."""
```
---
## Layer 4: E2E / Acceptance Tests
**Purpose:** Verify the actual output - that the final video has correctly synced audio.
### 4.1 Audio Sync Verification Test
**File:** `backend/tests/e2e/test_countdown_audio_sync.py` (new file)
```python
class TestCountdownAudioSync:
"""
E2E tests that verify actual audio synchronization in generated videos.
These tests generate real (short) videos and verify the audio offset.
Uses ffprobe to analyze video/audio streams.
"""
@pytest.fixture
def short_test_audio_early_start(self, tmp_path):
"""Generate a short test audio file where 'lyrics' start at 0.5 seconds."""
# Use ffmpeg to create a test audio file with a distinctive marker
def test_countdown_video_has_correct_instrumental_offset(self, short_test_audio_early_start):
"""
Given: Audio where content starts at 0.5 seconds
When: Full pipeline runs with countdown enabled
Then: Final video has instrumental starting at 3.5 seconds (0.5 + 3.0 padding)
Verification:
- Use ffprobe to get audio stream start time
- Verify offset matches expected padding
"""
def test_no_countdown_video_has_zero_offset(self):
"""
Given: Audio where content starts at 4.0 seconds
When: Full pipeline runs (no countdown needed)
Then: Final video has instrumental starting at 0 seconds (no padding)
"""
```
### 4.2 CLI E2E Test
**File:** `tests/integration/test_countdown_cli_e2e.py` (new file)
```python
class TestCountdownCLIE2E:
"""E2E tests for countdown sync in CLI workflow."""
def test_cli_produces_synced_output(self, tmp_path):
"""Running full CLI pipeline produces video with correct sync."""
```
---
## Implementation Order
### Phase 1: Contract Tests (catches regression immediately)
1. `test_countdown_state_contract.py` - Verify writer/reader contract
2. `test_encoding_interface_contract.py` - Verify encoding interface
### Phase 2: Unit Tests (detailed coverage)
3. Extend `test_workers.py` with state sync tests
4. Create `test_encoding_countdown.py`
### Phase 3: Integration Tests (verify full flow)
5. `test_countdown_pipeline_integration.py` - Full worker pipeline
6. `test_countdown_state_persistence.py` - State survives boundaries
### Phase 4: E2E Tests (acceptance criteria)
7. `test_countdown_audio_sync.py` - Actual audio verification
---
## Files to Create/Modify
| File | Action | Purpose |
|------|--------|---------|
| `backend/tests/test_countdown_state_contract.py` | Create | Contract tests for state sync |
| `backend/tests/test_encoding_interface_contract.py` | Create | Contract tests for encoding |
| `backend/tests/test_encoding_countdown.py` | Create | Unit tests for encoding paths |
| `backend/tests/test_workers.py` | Extend | Add state sync unit tests |
| `backend/tests/emulator/test_countdown_pipeline_integration.py` | Create | Integration tests with emulator |
| `backend/tests/emulator/test_countdown_state_persistence.py` | Create | State persistence tests |
| `backend/tests/e2e/test_countdown_audio_sync.py` | Create | Audio sync verification |
| `tests/integration/test_countdown_cli_e2e.py` | Create | CLI E2E tests |
---
## Test Execution
```bash
# Run all countdown-related tests
pytest -v -k "countdown" backend/tests/ tests/
# Run contract tests only (fast, every PR)
pytest -v backend/tests/test_countdown_state_contract.py
# Run integration tests (requires emulators)
make emulators-start && pytest -v backend/tests/emulator/test_countdown_pipeline_integration.py
# Run E2E tests (slow, on-demand)
pytest -v backend/tests/e2e/test_countdown_audio_sync.py
```
---
## Key Test Assertions
The tests must verify these critical invariants:
1. **INVARIANT 1:** If `CountdownProcessor.process()` returns `padding_added=True`, then `job.state_data.lyrics_metadata.has_countdown_padding` MUST be set to `True`
2. **INVARIANT 2:** If `lyrics_metadata.has_countdown_padding=True`, then `video_worker` MUST pass `countdown_padding_seconds` to the encoding backend
3. **INVARIANT 3:** All encoding backends (KaraokeFinalise, Orchestrator, GCE) MUST pad instrumental when `countdown_padding_seconds > 0`
4. **INVARIANT 4:** Final video instrumental start time MUST equal original start time + countdown_padding_seconds
---
## Verification Strategy
After implementing tests:
1. **Run tests against current code** - Should FAIL (bug exists)
2. **Implement the fix** - Add state update to render_video_worker
3. **Run tests again** - Should PASS
4. **Create regression test script** that runs countdown tests before every merge
---
## CI Integration
Add to `.github/workflows/test.yml`:
```yaml
# Countdown sync tests (contract layer - every PR)
- name: Run countdown contract tests
run: pytest -v backend/tests/test_countdown_state_contract.py
# Countdown integration tests (weekly or on-demand)
- name: Run countdown integration tests
run: |
make emulators-start
pytest -v backend/tests/emulator/test_countdown_*
make emulators-stop
```
---
## Fix Implementation
The fix was implemented in `backend/workers/render_video_worker.py` after line 201.
After the existing countdown processing code:
```python
if padding_added:
job_log.info(f"Countdown added: {padding_seconds}s padding applied to audio and timestamps shifted")
else:
job_log.info("No countdown needed - song starts after 3 seconds")
```
Added state update to communicate countdown info to video_worker:
```python
# Update lyrics_metadata with countdown info so video_worker knows to pad instrumental
# This is critical for audio sync - without this, instrumental won't match padded vocals
if padding_added:
existing_lyrics_metadata = job.state_data.get('lyrics_metadata', {})
existing_lyrics_metadata['has_countdown_padding'] = True
existing_lyrics_metadata['countdown_padding_seconds'] = padding_seconds
job_manager.update_state_data(job_id, 'lyrics_metadata', existing_lyrics_metadata)
job_log.info(f"Updated lyrics_metadata: has_countdown_padding=True, countdown_padding_seconds={padding_seconds}")
```
This ensures **INVARIANT 1** is satisfied: If `CountdownProcessor.process()` returns `padding_added=True`, then `job.state_data.lyrics_metadata.has_countdown_padding` is set to `True`.
@@ -1,168 +0,0 @@
# Theme Validation & Removal of Default Fallbacks - 2026-01-22
## Summary
Removed all default style fallback logic to ensure cloud karaoke jobs always use complete, explicit themes (nomad). Implemented two-phase rollout: Phase 1 added validation with warnings (non-breaking), Phase 2 removed defaults and requires complete themes (breaking change).
## Problem
The system had multiple layers of fallback logic that would silently use incomplete default styles when themes were missing or incomplete:
- `load_styles_from_gcs()` would fall back to minimal styles if no theme path
- `load_styles_from_gcs()` would fall back to defaults on download failure
- Getter functions (`get_intro_format()`, etc.) would merge incomplete themes with DEFAULT_* constants
- `StyleConfig` class would return defaults if no theme loaded
This created risk of generating unstyled or partially-styled videos if theme configuration was missing.
## Solution
### Phase 1: Validation with Warnings (v0.108.15)
Added infrastructure to detect incomplete themes without breaking existing behavior:
**New Validation Function:**
```python
def validate_theme_completeness(style_params, logger) -> Tuple[bool, List[str]]
```
- Checks all required sections: intro, end, karaoke, cdg
- Checks all required fields within each section
- Returns (is_complete, list_of_missing_fields)
**Updated Getter Functions:**
- Added warnings when fields missing from themes
- Still merged with defaults (backward compatible)
- Warned: "In future versions, incomplete themes will be rejected"
**New CLI Flags:**
- `--list-themes`: Show available themes from GCS
- `--validate-theme`: Validate theme completeness and exit
**New Script:**
- `scripts/verify-gcs-themes.py`: Validate all GCS themes are complete
### Phase 2: Remove Defaults & Require Complete Themes (v0.109.0) - BREAKING
Removed all fallback logic and require complete themes everywhere:
**Style Getter Functions:**
```python
def get_intro_format(style_params) -> Dict[str, Any]:
"""Raises ValueError if intro section missing or incomplete."""
if "intro" not in style_params:
raise ValueError("Missing 'intro' section in theme...")
missing = [f for f in required if f not in intro_params]
if missing:
raise ValueError(f"Incomplete 'intro' section. Missing: {missing}...")
return intro_params.copy() # Return as-is, no merging
```
Applied same pattern to `get_end_format()`, `get_karaoke_format()`, `get_cdg_format()`.
**GCS Style Loader:**
```python
def load_styles_from_gcs(...):
if not style_params_gcs_path:
raise ValueError("style_params_gcs_path is required...")
# On download failure:
except Exception as e:
raise ValueError(f"Failed to download theme from GCS: {e}") from e
```
**Worker Style Helper:**
```python
class StyleConfig:
def get_intro_format(self) -> Dict[str, Any]:
if not self._style_params:
raise ValueError("No style parameters loaded...")
return get_intro_format(self._style_params)
```
## Multi-Layer Protection
The system now has 4 layers ensuring all cloud jobs use complete themes:
1. **API Layer** (`backend/api/routes/file_upload.py`): Auto-applies nomad theme if none specified
2. **Job Manager** (`backend/services/job_manager.py`): Enforces theme_id requirement at creation
3. **Workers** (`backend/workers/screens_worker.py`): Check theme presence at processing time
4. **Style Loader** (NEW): Fail fast if theme missing or incomplete
## Files Changed
### Core Logic
- `karaoke_gen/style_loader.py`: Added validation, removed merging, fail fast
- `backend/workers/style_helper.py`: Removed fallbacks, fail fast
### CLI
- `karaoke_gen/utils/cli_args.py`: Added --list-themes, --validate-theme flags
- `karaoke_gen/utils/gen_cli.py`: Added handlers for new flags
### Tests
- `tests/unit/conftest.py`: Added complete_theme fixture
- `tests/unit/test_style_validation.py`: New comprehensive test suite (19 tests)
- `tests/unit/test_style_loader.py`: Updated to expect errors instead of fallbacks
- `tests/unit/test_initialization.py`: Updated to use complete theme
- `tests/unit/test_gen_cli.py`: Added new CLI flags to mock fixtures
### Scripts
- `scripts/verify-gcs-themes.py`: New script to validate all GCS themes
## Breaking Changes in v0.109.0
**API Changes:**
- Style getter functions now raise `ValueError` instead of merging with defaults
- `load_styles_from_gcs()` raises `ValueError` instead of returning minimal styles
- `StyleConfig` methods raise `ValueError` instead of returning defaults
**Behavioral Changes:**
- No fallback to defaults if theme download fails
- No merging with defaults if theme incomplete
- All themes must have complete intro, end, karaoke, cdg sections
**Migration Path:**
- Ensure all GCS themes are complete (use `scripts/verify-gcs-themes.py`)
- Fix any incomplete themes before upgrading
- Local CLI users must specify `--theme` or `--style_params_json`
## Testing
- **1086 unit tests passing** (no regressions)
- New validation tests cover all error scenarios
- Tests updated to use complete themes via fixture
## Key Decisions
**Why two-phase rollout?**
- Phase 1 allowed us to validate that all GCS themes are complete before breaking changes
- Warnings in production logs would alert us to incomplete themes
- Non-breaking phase could be deployed and monitored safely
**Why fail fast instead of fallback?**
- Silent fallbacks hide configuration errors
- Better to fail loudly and fix the root cause
- Ensures video quality by preventing partially-styled outputs
- Clear error messages guide users to fix incomplete themes
**Why remove DEFAULT_* constants entirely?**
- They're still used for validation (checking required fields)
- But removed from runtime merging logic
- Could be deleted in future if validation uses schema instead
## Future Considerations
**Potential Improvements:**
- Schema-based validation instead of comparing to DEFAULT_* constants
- Theme versioning to handle breaking changes to theme structure
- Theme inheritance (e.g., themes that extend base themes)
**Monitoring:**
- Watch for `ValueError` exceptions in production logs
- Should be zero if all themes are complete
- Any errors indicate theme configuration issues that need fixing
## Related PRs
- Phase 1 (v0.108.15): Theme validation with warnings
- Phase 2 (v0.109.0): Remove defaults, require complete themes
@@ -1,220 +0,0 @@
# Instrumental Review UX Fixes - 2026-01-26
## Summary
Fixed three UX issues in the instrumental review flow and one testing infrastructure issue related to local mode detection. The changes improve user feedback during audio loading and provide clear post-submission confirmation.
## Issues Fixed
### 1. Loading Spinner Not Visible
**Problem:** When switching between audio stems (backing vocals, pure instrumental, etc.), there was no visual feedback that audio was loading.
**Root Cause:** The loading spinner was implemented but wasn't visible because:
- The wrapper div around WaveformViewer was breaking absolute positioning
- The overlay approach wasn't integrated with the tab component
**Solution:** The StemComparison component already had loading state support via the `isLoading` prop:
- Shows `Loader2` spinner icon on the active tab
- Disables all tabs during loading to prevent race conditions
- Applies `opacity-50` styling to inactive tabs
No additional overlay needed - the tab component handles loading state elegantly.
### 2. Large Gap Under Waveform
**Problem:** The waveform had excess vertical space below it, wasting screen real estate.
**Root Cause:** A wrapper `<div className="relative flex-1 min-h-0">` was added around WaveformViewer, breaking the flexbox layout. The WaveformViewer was originally designed as a direct child of the flex container.
**Solution:** Removed the wrapper div. The WaveformViewer is now a direct flex child again, filling available space correctly.
**Before:**
```tsx
<div className="relative flex-1 min-h-0">
<WaveformViewer ... />
{isAudioLoading && <overlay />}
</div>
```
**After:**
```tsx
<WaveformViewer ... />
```
### 3. No Success Screen in Cloud Mode
**Problem:** After submitting the instrumental selection in cloud mode, users saw a brief toast and were immediately redirected without clear confirmation.
**Root Cause:** The success screen logic only ran in local mode (`window.close()`). Cloud mode skipped it and used `router.push("/app")` immediately.
**Solution:** Extended the success screen to both modes with appropriate messaging:
- Cloud mode: "Redirecting in 3s..." then redirects to `/app`
- Local mode: "Closing in 2s..." then closes window
- Both modes show which instrumental option was selected
**Implementation:**
```tsx
// Show success screen in both modes
setShowSuccess(true)
setCountdown(isLocalMode ? 2 : 3)
// Countdown effect handles both modes
useEffect(() => {
if (!showSuccess) return
const interval = setInterval(() => {
setCountdown((prev) => {
if (prev <= 1) {
clearInterval(interval)
if (isLocalMode) {
window.close()
} else {
router.push("/app")
}
return 0
}
return prev - 1
})
}, 1000)
return () => clearInterval(interval)
}, [showSuccess, isLocalMode, router])
```
## Testing Infrastructure Fix
### Local Mode Detection for E2E Tests
**Problem:** E2E tests running on `localhost:3000` were being detected as local CLI mode, making it impossible to test cloud mode behavior.
**Root Cause:** The `isLocalMode()` function only checked hostname and port, not the routing pattern.
**Solution:** Enhanced `isLocalMode()` to check for hash-based routing:
- Hash pattern `#/jobId/review` or `#/jobId/instrumental` → cloud mode
- Path pattern `/local/review` or `/local/instrumental` → local CLI mode
- Allows E2E tests to run in cloud mode on localhost by using hash routing
**Implementation:**
```typescript
export function isLocalMode(): boolean {
if (typeof window === 'undefined') return false
const { hostname, port, hash, pathname } = window.location
// Must be on localhost
if (hostname !== 'localhost' && hostname !== '127.0.0.1') return false
// Must be on one of the known local server ports
if (!ALL_LOCAL_PORTS.includes(port)) return false
// If using hash-based routing (cloud mode pattern), it's NOT local mode
if (hash && hash.match(/^#\/?[^/]+\/(review|instrumental)/)) {
return false
}
// If pathname explicitly contains /local/, it's local mode
if (pathname.includes('/local/')) {
return true
}
// Default to local mode on localhost (backwards compatibility)
return true
}
```
## Backend Fix
### Backing Vocals Audio URL Field Name
**Problem:** Backing vocals audio wouldn't load in cloud mode.
**Root Cause:** Backend was returning `audio_urls['backing']` but frontend expected `audio_urls['backing_vocals']`.
**Solution:** Changed backend to use `'backing_vocals'` to match frontend expectations:
```python
# backend/api/routes/review.py
audio_urls = {
"clean": clean_flac_url,
"backing_vocals": backing_flac_url, # Changed from "backing"
"original": original_flac_url,
}
```
## Testing
### Unit Tests (29 new tests)
- **`local-mode.test.ts`** (17 tests): Comprehensive coverage of `isLocalMode()` logic
- Production hostname detection
- Localhost with non-local ports
- Hash-based routing (cloud mode)
- Path-based routing (local CLI mode)
- Backwards compatibility
- Helper functions (`getLocalJobId`, `createLocalModeJob`, etc.)
- **`StemComparison.test.tsx`** (11 tests): Loading state behavior
- Spinner visibility on active tab
- Tab disable during loading
- Opacity styling on inactive tabs
- **`test_routes_review.py`** (2 tests): Backend API contract
- Instrumental analysis returns correct audio_urls structure
### E2E Tests (12 new tests)
- **`instrumental-review-ux.spec.ts`** (12 tests):
- Loading spinner visibility when switching tabs
- Tab disable behavior during loading
- Waveform layout without excess gaps
- Success screen in cloud mode with redirect
- Success screen in local mode with window close
- Countdown timer behavior
- Selection options display
- Visual regression screenshots
All 276 unit tests pass. All 12 E2E tests pass.
## Files Modified
### Source Files
- `frontend/components/instrumental-review/InstrumentalSelector.tsx`
- Removed wrapper div around WaveformViewer
- Extended success screen to cloud mode with countdown
- Fixed dependency array (removed unused `router` from handleSubmit)
- `frontend/lib/local-mode.ts`
- Enhanced `isLocalMode()` with hash routing detection
- `backend/api/routes/review.py`
- Changed `audio_urls['backing']` to `audio_urls['backing_vocals']`
### Test Files (new)
- `frontend/lib/__tests__/local-mode.test.ts`
- `frontend/e2e/regression/instrumental-review-ux.spec.ts`
### Documentation
- `docs/LESSONS-LEARNED.md` - Added lesson about local mode detection
- `docs/README.md` - Added recent changes entry
## Lessons Learned
### Local Mode Detection for Testing
When detecting local CLI mode, always check the routing pattern in addition to hostname/port. Hash-based routing indicates cloud mode even on localhost, allowing E2E tests to exercise cloud behavior without a remote server.
### Loading States in Tab Components
Tab components with built-in loading state support (spinner on active tab, disabled state on others) provide better UX than generic overlays. The user can see which audio source they selected and that it's loading.
### Success Screens Should Be Explicit
Even if users will be redirected automatically, showing an explicit success screen with countdown:
- Confirms the action was successful
- Shows what was selected
- Gives users time to read the confirmation
- Prevents confusion about what just happened
Better to show 3 seconds of confirmation than immediately redirect and leave users wondering if it worked.
## Future Considerations
### InstrumentalSelector Unit Tests
The `InstrumentalSelector` component is complex but doesn't have direct unit tests. The E2E tests provide good coverage of user-facing behavior, but unit tests for individual functions (like `mergeAudibleSegments`) could improve maintainability.
### Loading State Consistency
Consider using the StemComparison loading pattern (spinner in tab) across other audio/video components for consistency.
+1 -139
View File
@@ -2,7 +2,7 @@
* Unit tests for job-status.ts utility functions
*/
import { getJobStep, formatStepIndicator, isBlockingStatus, getJobProgressPercent, getJobPriority, sortJobsByPriority, JobStep } from '../lib/job-status';
import { getJobStep, formatStepIndicator, isBlockingStatus, getJobProgressPercent, JobStep } from '../lib/job-status';
import type { Job } from '../lib/api';
// Helper to create a minimal Job object for testing
@@ -457,141 +457,3 @@ describe('getJobProgressPercent', () => {
expect(getJobProgressPercent(job)).toBe(0);
});
});
describe('getJobPriority', () => {
it('returns 0 for blocking statuses (user action needed)', () => {
expect(getJobPriority(createJob('awaiting_review'))).toBe(0);
expect(getJobPriority(createJob('awaiting_audio_selection'))).toBe(0);
expect(getJobPriority(createJob('awaiting_instrumental_selection'))).toBe(0);
expect(getJobPriority(createJob('in_review'))).toBe(0);
});
it('returns 1 for active/processing statuses', () => {
expect(getJobPriority(createJob('pending'))).toBe(1);
expect(getJobPriority(createJob('downloading'))).toBe(1);
expect(getJobPriority(createJob('rendering_video'))).toBe(1);
expect(getJobPriority(createJob('separating_stage1'))).toBe(1);
});
it('returns 2 for completed statuses', () => {
expect(getJobPriority(createJob('complete'))).toBe(2);
expect(getJobPriority(createJob('prep_complete'))).toBe(2);
});
it('returns 3 for failed/cancelled statuses', () => {
expect(getJobPriority(createJob('failed'))).toBe(3);
expect(getJobPriority(createJob('cancelled'))).toBe(3);
});
it('handles case insensitivity', () => {
expect(getJobPriority(createJob('AWAITING_REVIEW'))).toBe(0);
expect(getJobPriority(createJob('COMPLETE'))).toBe(2);
});
});
describe('sortJobsByPriority', () => {
// Helper to create job with specific date
function createJobWithDate(status: string, createdAt: string): Job {
return {
job_id: `job-${status}-${createdAt}`,
status,
progress: 0,
created_at: createdAt,
updated_at: createdAt,
};
}
it('sorts blocking jobs before completed jobs', () => {
const jobs = [
createJobWithDate('complete', '2024-01-01T12:00:00Z'),
createJobWithDate('awaiting_review', '2024-01-01T12:00:00Z'),
];
const sorted = sortJobsByPriority(jobs);
expect(sorted[0].status).toBe('awaiting_review');
expect(sorted[1].status).toBe('complete');
});
it('sorts blocking jobs before processing jobs', () => {
const jobs = [
createJobWithDate('rendering_video', '2024-01-01T12:00:00Z'),
createJobWithDate('awaiting_review', '2024-01-01T12:00:00Z'),
];
const sorted = sortJobsByPriority(jobs);
expect(sorted[0].status).toBe('awaiting_review');
expect(sorted[1].status).toBe('rendering_video');
});
it('sorts processing jobs before completed jobs', () => {
const jobs = [
createJobWithDate('complete', '2024-01-01T12:00:00Z'),
createJobWithDate('rendering_video', '2024-01-01T12:00:00Z'),
];
const sorted = sortJobsByPriority(jobs);
expect(sorted[0].status).toBe('rendering_video');
expect(sorted[1].status).toBe('complete');
});
it('sorts completed jobs before failed jobs', () => {
const jobs = [
createJobWithDate('failed', '2024-01-01T12:00:00Z'),
createJobWithDate('complete', '2024-01-01T12:00:00Z'),
];
const sorted = sortJobsByPriority(jobs);
expect(sorted[0].status).toBe('complete');
expect(sorted[1].status).toBe('failed');
});
it('sorts by created_at (newest first) within same priority', () => {
const jobs = [
createJobWithDate('complete', '2024-01-01T10:00:00Z'),
createJobWithDate('complete', '2024-01-01T14:00:00Z'),
createJobWithDate('complete', '2024-01-01T12:00:00Z'),
];
const sorted = sortJobsByPriority(jobs);
expect(sorted[0].created_at).toBe('2024-01-01T14:00:00Z');
expect(sorted[1].created_at).toBe('2024-01-01T12:00:00Z');
expect(sorted[2].created_at).toBe('2024-01-01T10:00:00Z');
});
it('does not mutate the original array', () => {
const jobs = [
createJobWithDate('complete', '2024-01-01T12:00:00Z'),
createJobWithDate('awaiting_review', '2024-01-01T12:00:00Z'),
];
const originalOrder = [...jobs];
sortJobsByPriority(jobs);
expect(jobs[0].status).toBe(originalOrder[0].status);
expect(jobs[1].status).toBe(originalOrder[1].status);
});
it('handles empty array', () => {
const sorted = sortJobsByPriority([]);
expect(sorted).toEqual([]);
});
it('handles single job', () => {
const jobs = [createJobWithDate('complete', '2024-01-01T12:00:00Z')];
const sorted = sortJobsByPriority(jobs);
expect(sorted.length).toBe(1);
expect(sorted[0].status).toBe('complete');
});
it('correctly orders a realistic job list', () => {
const jobs = [
createJobWithDate('complete', '2024-01-01T10:00:00Z'),
createJobWithDate('failed', '2024-01-01T11:00:00Z'),
createJobWithDate('awaiting_review', '2024-01-01T09:00:00Z'),
createJobWithDate('rendering_video', '2024-01-01T12:00:00Z'),
createJobWithDate('complete', '2024-01-01T14:00:00Z'),
];
const sorted = sortJobsByPriority(jobs);
// Priority order: blocking (0), processing (1), complete (2), failed (3)
expect(sorted[0].status).toBe('awaiting_review'); // blocking
expect(sorted[1].status).toBe('rendering_video'); // processing
expect(sorted[2].status).toBe('complete'); // complete, newer
expect(sorted[2].created_at).toBe('2024-01-01T14:00:00Z');
expect(sorted[3].status).toBe('complete'); // complete, older
expect(sorted[3].created_at).toBe('2024-01-01T10:00:00Z');
expect(sorted[4].status).toBe('failed'); // failed
});
});
+25 -364
View File
@@ -2,7 +2,7 @@
import { useEffect, useState, useCallback, Suspense } from "react"
import { useRouter, useSearchParams } from "next/navigation"
import { adminApi, api, Job, FileInfo, JobFilesResponse, JobUpdateRequest, JobResetResponse, DeleteOutputsResponse, ClearWorkersResponse, TriggerWorkerResponse } from "@/lib/api"
import { adminApi, api, Job, FileInfo, JobFilesResponse, JobUpdateRequest, JobResetResponse, DeleteOutputsResponse } from "@/lib/api"
import { useAdminSettings } from "@/lib/admin-settings"
import { Button } from "@/components/ui/button"
import { Input } from "@/components/ui/input"
@@ -82,7 +82,6 @@ import {
AlertTriangle,
ChevronRight,
Copy,
Play,
} from "lucide-react"
import { useToast } from "@/hooks/use-toast"
@@ -173,20 +172,10 @@ function AdminJobsPageContent() {
const [resetDialogOpen, setResetDialogOpen] = useState(false)
const [resetTarget, setResetTarget] = useState<string | null>(null)
const [resetting, setResetting] = useState(false)
const [resetResult, setResetResult] = useState<JobResetResponse | null>(null)
const [resetResultOpen, setResetResultOpen] = useState(false)
// Delete outputs state
const [deleteOutputsDialogOpen, setDeleteOutputsDialogOpen] = useState(false)
const [deletingOutputs, setDeletingOutputs] = useState(false)
const [deleteOutputsResult, setDeleteOutputsResult] = useState<DeleteOutputsResponse | null>(null)
const [deleteOutputsResultOpen, setDeleteOutputsResultOpen] = useState(false)
// Clear workers state
const [clearingWorkers, setClearingWorkers] = useState(false)
// Trigger worker state
const [triggeringWorker, setTriggeringWorker] = useState(false)
const loadJobs = useCallback(async () => {
try {
@@ -371,12 +360,12 @@ function AdminJobsPageContent() {
try {
setResetting(true)
const result = await adminApi.resetJob(selectedJobId, resetTarget)
// Store result and show detailed result dialog
setResetResult(result)
toast({
title: "Job Reset",
description: result.message,
})
setResetDialogOpen(false)
setResetResultOpen(true)
setResetTarget(null)
// Refresh job details
loadJobDetail(selectedJobId)
loadLogs(selectedJobId)
@@ -386,10 +375,8 @@ function AdminJobsPageContent() {
description: err.message || "Failed to reset job",
variant: "destructive",
})
setResetDialogOpen(false)
} finally {
setResetting(false)
setResetTarget(null)
}
}
@@ -401,11 +388,26 @@ function AdminJobsPageContent() {
setDeletingOutputs(true)
const result = await adminApi.deleteJobOutputs(selectedJobId)
// Store result and show detailed result dialog
setDeleteOutputsResult(result)
setDeleteOutputsDialogOpen(false)
setDeleteOutputsResultOpen(true)
// Check for errors in individual services
const hasErrors = result.status === "error" || result.status === "partial_success"
const serviceResults = Object.entries(result.deleted_services)
.map(([service, r]: [string, any]) => `${service}: ${r.status}${r.error ? ` (${r.error})` : ''}`)
.join(", ")
if (hasErrors) {
toast({
title: result.status === "error" ? "Delete Failed" : "Partial Success",
description: `${result.message}. Services: ${serviceResults}`,
variant: "destructive",
})
} else {
toast({
title: "Outputs Deleted",
description: `${result.message}. Cleared: ${result.cleared_state_data.join(", ") || "none"}`,
})
}
setDeleteOutputsDialogOpen(false)
// Refresh job details
loadJobDetail(selectedJobId)
loadLogs(selectedJobId)
@@ -420,77 +422,6 @@ function AdminJobsPageContent() {
}
}
// Handle clear workers - clears worker progress to allow re-execution
const handleClearWorkers = async () => {
if (!selectedJobId) return
try {
setClearingWorkers(true)
const result = await adminApi.clearWorkers(selectedJobId)
if (result.cleared_keys.length > 0) {
toast({
title: "Workers Cleared",
description: `${result.message}: ${result.cleared_keys.join(", ")}`,
})
} else {
toast({
title: "No Progress Keys",
description: result.message,
})
}
// Refresh job details
loadJobDetail(selectedJobId)
loadLogs(selectedJobId)
} catch (err: any) {
toast({
title: "Error",
description: err.message || "Failed to clear workers",
variant: "destructive",
})
} finally {
setClearingWorkers(false)
}
}
// Handle trigger worker - manually triggers a worker for processing
const handleTriggerWorker = async (workerType: string = "video") => {
if (!selectedJobId) return
try {
setTriggeringWorker(true)
const result = await adminApi.triggerWorker(selectedJobId, workerType)
if (result.triggered) {
toast({
title: `${workerType.charAt(0).toUpperCase() + workerType.slice(1)} Worker Triggered`,
description: `Job ${selectedJobId} is now processing. The ${workerType} worker will begin generating the video.`,
})
} else {
toast({
title: "Trigger Failed",
description: result.error
? `${result.message}: ${result.error}`
: result.message,
variant: "destructive",
})
}
// Refresh job details
loadJobDetail(selectedJobId)
loadLogs(selectedJobId)
} catch (err: any) {
toast({
title: "Error",
description: err.message || "Failed to trigger worker",
variant: "destructive",
})
} finally {
setTriggeringWorker(false)
}
}
// Get reset target display info
const getResetTargetInfo = (targetState: string) => {
const info: Record<string, { label: string; description: string; icon: any }> = {
@@ -514,11 +445,6 @@ function AdminJobsPageContent() {
description: "Re-select instrumental. Preserves lyrics review.",
icon: Sliders,
},
instrumental_selected: {
label: "Reprocess Video",
description: "Re-encode and re-distribute with current settings. Triggers video worker automatically.",
icon: RefreshCw,
},
}
return info[targetState] || { label: targetState, description: "", icon: Settings }
}
@@ -818,7 +744,6 @@ function AdminJobsPageContent() {
{ state: "awaiting_audio_selection", icon: Music, label: "Audio" },
{ state: "awaiting_review", icon: Mic, label: "Lyrics" },
{ state: "awaiting_instrumental_selection", icon: Sliders, label: "Inst." },
{ state: "instrumental_selected", icon: RefreshCw, label: "Reprocess" },
].map(({ state, icon: Icon, label }) => (
<Tooltip key={state}>
<TooltipTrigger asChild>
@@ -864,46 +789,6 @@ function AdminJobsPageContent() {
</TooltipContent>
</Tooltip>
<Tooltip>
<TooltipTrigger asChild>
<Button
variant="outline"
size="sm"
className="h-7 px-2 text-xs"
onClick={handleClearWorkers}
disabled={clearingWorkers}
>
<RefreshCw className={`w-3 h-3 mr-1 ${clearingWorkers ? "animate-spin" : ""}`} />
Clear Workers
</Button>
</TooltipTrigger>
<TooltipContent>
Clear worker progress markers to allow re-execution
</TooltipContent>
</Tooltip>
<Tooltip>
<TooltipTrigger asChild>
<Button
variant="outline"
size="sm"
className={`h-7 px-2 text-xs ${selectedJob.status === "instrumental_selected" ? "border-green-500 text-green-600 hover:bg-green-50 dark:hover:bg-green-950" : ""}`}
onClick={() => handleTriggerWorker("video")}
disabled={triggeringWorker}
>
{triggeringWorker ? (
<Loader2 className="w-3 h-3 mr-1 animate-spin" />
) : (
<Play className="w-3 h-3 mr-1" />
)}
Trigger
</Button>
</TooltipTrigger>
<TooltipContent>
Manually trigger video worker processing
</TooltipContent>
</Tooltip>
<Tooltip>
<TooltipTrigger asChild>
<Button
@@ -1443,230 +1328,6 @@ function AdminJobsPageContent() {
</AlertDialogFooter>
</AlertDialogContent>
</AlertDialog>
{/* Delete Outputs Result Dialog */}
<Dialog open={deleteOutputsResultOpen} onOpenChange={setDeleteOutputsResultOpen}>
<DialogContent className="sm:max-w-md">
<DialogHeader>
<DialogTitle className="flex items-center gap-2">
{deleteOutputsResult?.status === "success" ? (
<Check className="w-5 h-5 text-green-500" />
) : deleteOutputsResult?.status === "partial_success" ? (
<AlertTriangle className="w-5 h-5 text-yellow-500" />
) : (
<XCircle className="w-5 h-5 text-red-500" />
)}
{deleteOutputsResult?.status === "success"
? "Outputs Deleted Successfully"
: deleteOutputsResult?.status === "partial_success"
? "Partial Success"
: "Delete Failed"}
</DialogTitle>
</DialogHeader>
{deleteOutputsResult && (
<div className="space-y-4">
<div className="space-y-3">
{/* YouTube */}
<div className="flex items-start gap-3 p-3 rounded-lg bg-muted/50">
<div className={`w-2 h-2 rounded-full mt-1.5 ${
deleteOutputsResult.deleted_services.youtube.status === "deleted"
? "bg-green-500"
: deleteOutputsResult.deleted_services.youtube.status === "skipped"
? "bg-gray-400"
: "bg-red-500"
}`} />
<div className="flex-1 min-w-0">
<div className="font-medium text-sm">YouTube</div>
<div className="text-xs text-muted-foreground">
{deleteOutputsResult.deleted_services.youtube.status === "deleted" && (
<>Deleted video: <code className="text-xs bg-muted px-1 rounded">{deleteOutputsResult.deleted_services.youtube.video_id}</code></>
)}
{deleteOutputsResult.deleted_services.youtube.status === "skipped" && (
<>{deleteOutputsResult.deleted_services.youtube.reason}</>
)}
{deleteOutputsResult.deleted_services.youtube.status === "error" && (
<span className="text-red-500">{deleteOutputsResult.deleted_services.youtube.error}</span>
)}
</div>
</div>
</div>
{/* Dropbox */}
<div className="flex items-start gap-3 p-3 rounded-lg bg-muted/50">
<div className={`w-2 h-2 rounded-full mt-1.5 ${
deleteOutputsResult.deleted_services.dropbox.status === "deleted"
? "bg-green-500"
: deleteOutputsResult.deleted_services.dropbox.status === "skipped"
? "bg-gray-400"
: "bg-red-500"
}`} />
<div className="flex-1 min-w-0">
<div className="font-medium text-sm">Dropbox</div>
<div className="text-xs text-muted-foreground">
{deleteOutputsResult.deleted_services.dropbox.status === "deleted" && (
<>Deleted folder: <code className="text-xs bg-muted px-1 rounded break-all">{deleteOutputsResult.deleted_services.dropbox.path}</code></>
)}
{deleteOutputsResult.deleted_services.dropbox.status === "skipped" && (
<>{deleteOutputsResult.deleted_services.dropbox.reason}</>
)}
{deleteOutputsResult.deleted_services.dropbox.status === "error" && (
<span className="text-red-500">{deleteOutputsResult.deleted_services.dropbox.error}</span>
)}
</div>
</div>
</div>
{/* Google Drive */}
<div className="flex items-start gap-3 p-3 rounded-lg bg-muted/50">
<div className={`w-2 h-2 rounded-full mt-1.5 ${
deleteOutputsResult.deleted_services.gdrive.status === "deleted"
? "bg-green-500"
: deleteOutputsResult.deleted_services.gdrive.status === "skipped"
? "bg-gray-400"
: "bg-red-500"
}`} />
<div className="flex-1 min-w-0">
<div className="font-medium text-sm">Google Drive</div>
<div className="text-xs text-muted-foreground">
{deleteOutputsResult.deleted_services.gdrive.status === "deleted" && deleteOutputsResult.deleted_services.gdrive.files && (
<>Deleted {Object.keys(deleteOutputsResult.deleted_services.gdrive.files).length} file(s)</>
)}
{deleteOutputsResult.deleted_services.gdrive.status === "skipped" && (
<>{deleteOutputsResult.deleted_services.gdrive.reason}</>
)}
{deleteOutputsResult.deleted_services.gdrive.status === "error" && (
<span className="text-red-500">{deleteOutputsResult.deleted_services.gdrive.error}</span>
)}
</div>
</div>
</div>
</div>
{deleteOutputsResult.cleared_state_data.length > 0 && (
<div className="text-xs text-muted-foreground border-t pt-3">
<span className="font-medium">Cleared from job:</span>{" "}
{deleteOutputsResult.cleared_state_data.join(", ")}
</div>
)}
<div className="text-xs text-muted-foreground">
Deleted at: {new Date(deleteOutputsResult.outputs_deleted_at).toLocaleString()}
</div>
</div>
)}
<div className="flex justify-end">
<Button onClick={() => setDeleteOutputsResultOpen(false)}>
Close
</Button>
</div>
</DialogContent>
</Dialog>
{/* Reset Result Dialog */}
<Dialog open={resetResultOpen} onOpenChange={setResetResultOpen}>
<DialogContent className="sm:max-w-md">
<DialogHeader>
<DialogTitle className="flex items-center gap-2">
{resetResult?.worker_triggered === false && resetResult?.new_status === "instrumental_selected" ? (
<AlertTriangle className="w-5 h-5 text-yellow-500" />
) : (
<Check className="w-5 h-5 text-green-500" />
)}
Job Reset Complete
</DialogTitle>
</DialogHeader>
{resetResult && (
<div className="space-y-4">
{/* Status Change */}
<div className="flex items-start gap-3 p-3 rounded-lg bg-muted/50">
<RotateCcw className="w-4 h-4 text-blue-500 mt-0.5" />
<div className="flex-1 min-w-0">
<div className="font-medium text-sm">Status Changed</div>
<div className="text-xs text-muted-foreground">
<code className="bg-muted px-1 rounded">{resetResult.previous_status}</code>
{" → "}
<code className="bg-muted px-1 rounded">{resetResult.new_status}</code>
</div>
</div>
</div>
{/* Cleared Data */}
{resetResult.cleared_data.length > 0 && (
<div className="flex items-start gap-3 p-3 rounded-lg bg-muted/50">
<Trash2 className="w-4 h-4 text-orange-500 mt-0.5" />
<div className="flex-1 min-w-0">
<div className="font-medium text-sm">State Data Cleared</div>
<div className="text-xs text-muted-foreground flex flex-wrap gap-1 mt-1">
{resetResult.cleared_data.map((key) => (
<code key={key} className="bg-muted px-1 rounded">{key}</code>
))}
</div>
</div>
</div>
)}
{/* Worker Trigger Status (only for instrumental_selected) */}
{resetResult.new_status === "instrumental_selected" && (
<div className={`flex items-start gap-3 p-3 rounded-lg ${
resetResult.worker_triggered
? "bg-green-50 dark:bg-green-950/30"
: "bg-yellow-50 dark:bg-yellow-950/30"
}`}>
{resetResult.worker_triggered ? (
<Play className="w-4 h-4 text-green-600 mt-0.5" />
) : (
<AlertTriangle className="w-4 h-4 text-yellow-600 mt-0.5" />
)}
<div className="flex-1 min-w-0">
<div className={`font-medium text-sm ${
resetResult.worker_triggered ? "text-green-700 dark:text-green-400" : "text-yellow-700 dark:text-yellow-400"
}`}>
{resetResult.worker_triggered ? "Video Worker Triggered" : "Video Worker NOT Triggered"}
</div>
<div className="text-xs text-muted-foreground">
{resetResult.worker_triggered
? "Processing will begin automatically"
: resetResult.worker_trigger_error || "Use the Trigger button to start processing manually"
}
</div>
</div>
</div>
)}
{/* What Happens Next */}
<div className="text-xs text-muted-foreground border-t pt-3">
<span className="font-medium">What happens next:</span>{" "}
{resetResult.new_status === "pending" && "Job will need to go through audio search, download, and all processing steps."}
{resetResult.new_status === "awaiting_audio_selection" && "User can select from the cached audio search results."}
{resetResult.new_status === "awaiting_review" && "User can review and correct the lyrics."}
{resetResult.new_status === "awaiting_instrumental_selection" && "User can select the instrumental track."}
{resetResult.new_status === "instrumental_selected" && (
resetResult.worker_triggered
? "Video generation is now in progress."
: "Click the Trigger button to start video generation."
)}
</div>
</div>
)}
<div className="flex justify-end gap-2">
{resetResult?.new_status === "instrumental_selected" && !resetResult?.worker_triggered && (
<Button
variant="outline"
onClick={() => {
setResetResultOpen(false)
handleTriggerWorker("video")
}}
>
<Play className="w-4 h-4 mr-2" />
Trigger Now
</Button>
)}
<Button onClick={() => setResetResultOpen(false)}>
Close
</Button>
</div>
</DialogContent>
</Dialog>
</div>
</TooltipProvider>
)
+44 -50
View File
@@ -1,14 +1,15 @@
"use client"
import { useEffect, useState, useCallback } from "react"
import { useParams } from "next/navigation"
import { useAuth } from "@/lib/auth"
import { api, Job, createLyricsReviewApiClient, lyricsReviewApi } from "@/lib/api"
import { Spinner } from "@/components/ui/spinner"
import { Button } from "@/components/ui/button"
import { ArrowLeft, AlertCircle } from "lucide-react"
import Link from "next/link"
import { LyricsAnalyzer } from "@/components/lyrics-review"
import { InstrumentalSelector } from "@/components/instrumental-review"
import { LyricsAnalyzer } from "@/components/lyrics-review"
import { ThemeToggle } from "@/components/ThemeToggle"
import type { CorrectionData } from "@/lib/lyrics-review/types"
import { isLocalMode, createLocalModeJob } from "@/lib/local-mode"
@@ -25,20 +26,23 @@ type AccessState =
| { status: "authorized"; job: Job; routeType: RouteType }
| { status: "local_mode"; job: Job; routeType: RouteType }
// Parse route from window.location.pathname (used for local mode with path-based routing)
// For static exports, useParams() returns empty - we must parse the URL directly
function parseRouteFromPathname(pathname: string): { jobId: string | null; routeType: RouteType } {
if (!pathname) {
// Parse route from Next.js slug params (used for local mode with path-based routing)
function parseRouteFromSlug(slug: string[] | undefined): { jobId: string | null; routeType: RouteType } {
if (!slug || slug.length === 0) {
return { jobId: null, routeType: "unknown" }
}
// Expected format: /app/jobs/{jobId}/{action}
// e.g., /app/jobs/local/review or /app/jobs/local/instrumental
const match = pathname.match(/^\/app\/jobs\/([^/]+)\/(review|instrumental)\/?$/)
if (match) {
const [, jobId, action] = match
return { jobId, routeType: action as RouteType }
// Expected formats:
// [jobId, "review"] -> /app/jobs/{jobId}/review
// [jobId, "instrumental"] -> /app/jobs/{jobId}/instrumental
if (slug.length === 2) {
const [jobId, action] = slug
if (action === "review") {
return { jobId, routeType: "review" }
}
if (action === "instrumental") {
return { jobId, routeType: "instrumental" }
}
}
return { jobId: null, routeType: "unknown" }
@@ -53,7 +57,6 @@ function parseRouteFromHash(hash: string): { jobId: string | null; routeType: Ro
// Remove the leading '#' and parse
const hashPath = hash.substring(1)
// Support BOTH review and instrumental routes
const match = hashPath.match(/^\/?([^/]+)\/(review|instrumental)\/?$/)
if (match) {
@@ -68,20 +71,17 @@ function getExpectedStates(routeType: RouteType): string[] {
case "review":
return ["awaiting_review", "in_review"]
case "instrumental":
return ["awaiting_review", "in_review"] // Same states for now
return ["awaiting_instrumental_selection"]
default:
return []
}
}
export function JobRouterClient() {
// For static exports, useParams() returns empty object
// We need to parse the URL path directly instead
const [pathname, setPathname] = useState(() =>
typeof window !== 'undefined' ? window.location.pathname : ''
)
const params = useParams()
const slug = params.slug as string[] | undefined
// Track hash for re-renders when hash changes (cloud mode)
// Track hash for re-renders when hash changes
const [hash, setHash] = useState(() =>
typeof window !== 'undefined' ? window.location.hash : ''
)
@@ -95,17 +95,12 @@ export function JobRouterClient() {
return () => window.removeEventListener('hashchange', handleHashChange)
}, [])
// Update pathname on mount (needed for SSR hydration)
useEffect(() => {
setPathname(window.location.pathname)
}, [])
// Determine routing mode and parse route
// - Local mode: Use path-based routing parsed from window.location.pathname
// - Local mode: Use path-based routing from Next.js slug params
// - Cloud mode: Use hash-based routing (e.g., /app/jobs/#/{jobId}/review)
const inLocalMode = isLocalMode()
const { jobId, routeType } = inLocalMode
? parseRouteFromPathname(pathname)
? parseRouteFromSlug(slug)
: parseRouteFromHash(hash)
const { user, isLoading: authLoading, hasHydrated } = useAuth()
@@ -115,8 +110,18 @@ export function JobRouterClient() {
async function checkAccess() {
// Handle local mode (skip all auth checks)
if (inLocalMode) {
// In local mode, always use review (combined lyrics + instrumental)
const localRouteType: RouteType = routeType === "unknown" ? "review" : routeType
// In local mode, determine route type from URL or default to review
let localRouteType: RouteType = routeType
// If route is unknown but we're in local mode, try to determine from path
if (localRouteType === "unknown") {
const path = typeof window !== 'undefined' ? window.location.pathname : ''
if (path.includes('instrumental')) {
localRouteType = "instrumental"
} else {
localRouteType = "review" // Default to review in local mode
}
}
// Create mock job for local mode
const localJob = createLocalModeJob({ routeType: localRouteType }) as Job
@@ -278,13 +283,14 @@ export function JobRouterClient() {
// Wrong state
if (accessState.status === "wrong_state") {
const actionName = routeType === "review" ? "lyrics review" : "instrumental selection"
return (
<div className="flex items-center justify-center min-h-screen">
<div className="text-center max-w-md p-6">
<AlertCircle className="w-12 h-12 mx-auto mb-4 text-amber-500" />
<h1 className="text-xl font-semibold mb-2">Not available</h1>
<p className="text-muted-foreground mb-4">
This job is currently in &quot;{accessState.currentState}&quot; state and is not ready for review.
This job is currently in &quot;{accessState.currentState}&quot; state and is not ready for {actionName}.
</p>
<Button variant="outline" asChild>
<Link href="/app">
@@ -298,18 +304,17 @@ export function JobRouterClient() {
}
// Authorized or local mode - render the appropriate UI
const { job } = accessState
const { job, routeType: authorizedRouteType } = accessState
// Get route type from access state (which validated and stored it during checkAccess)
const currentRouteType = accessState.status === "authorized" || accessState.status === "local_mode"
? accessState.routeType
: "review" // Fallback (shouldn't happen since we return early for other statuses)
if (currentRouteType === "instrumental") {
return <InstrumentalReviewWrapper job={job} isLocalMode={inLocalMode} />
if (authorizedRouteType === "review") {
return <LyricsReviewWrapper job={job} isLocalMode={inLocalMode} />
}
return <LyricsReviewWrapper job={job} isLocalMode={inLocalMode} />
if (authorizedRouteType === "instrumental") {
return <InstrumentalSelector job={job} isLocalMode={inLocalMode} />
}
return null
}
// Lyrics Review Component Wrapper
@@ -419,20 +424,9 @@ function LyricsReviewWrapper({ job, isLocalMode = false }: { job: Job; isLocalMo
apiClient={apiClient}
isReadOnly={false}
audioHash={correctionData.metadata?.audio_hash || job.audio_hash || job.job_id}
isLocalMode={isLocalMode}
jobId={job.job_id}
/>
</main>
</div>
)
}
// Instrumental Review Component Wrapper
// The InstrumentalSelector component handles its own data fetching and submission
// It uses the appropriate API based on isLocalMode
function InstrumentalReviewWrapper({ job, isLocalMode = false }: { job: Job; isLocalMode?: boolean }) {
return (
<InstrumentalSelector job={job} isLocalMode={isLocalMode} />
)
}
+12 -35
View File
@@ -8,15 +8,7 @@ import { useAdminSettings } from "@/lib/admin-settings"
import { useJobNotifications, useVisibilityRefresh } from "@/hooks/use-notifications"
import { Button } from "@/components/ui/button"
import { Card, CardContent, CardDescription, CardHeader, CardTitle } from "@/components/ui/card"
import {
Select,
SelectContent,
SelectItem,
SelectTrigger,
SelectValue,
} from "@/components/ui/select"
import { Music2, RefreshCw, Loader2, Moon, Sun } from "lucide-react"
import { sortJobsByPriority } from "@/lib/job-status"
import { WarmingUpLoader } from "@/components/WarmingUpLoader"
import { JobCard } from "@/components/job"
import { JobSubmission } from "@/components/job/JobSubmission"
@@ -44,7 +36,6 @@ function AppPageContent() {
const { isDarkMode, toggleTheme, mounted } = useTheme()
const { user, fetchUser, verifyMagicLink } = useAuth()
const { showTestData } = useAdminSettings()
const [jobLimit, setJobLimit] = useState<number>(5)
// Check if user is admin (for exclude_test parameter)
const isAdmin = user?.role === "admin" || user?.email?.endsWith("@nomadkaraoke.com")
@@ -59,11 +50,10 @@ function AppPageContent() {
try {
// Only admins can filter test jobs, and by default we hide them
const data = await api.listJobs({
limit: jobLimit === -1 ? 100 : jobLimit,
limit: 20,
exclude_test: isAdmin ? !showTestData : undefined
})
// Sort: blocking jobs first, then processing, then completed
setJobs(sortJobsByPriority(data))
setJobs(data)
} catch (err: any) {
// Don't log auth errors - user just needs to authenticate
if (err?.status !== 401) {
@@ -73,7 +63,7 @@ function AppPageContent() {
setIsLoadingJobs(false)
setIsInitialLoad(false)
}
}, [isAdmin, showTestData, jobLimit])
}, [isAdmin, showTestData])
// Enable notifications for job status changes (sound + title animation)
useJobNotifications(jobs)
@@ -216,7 +206,7 @@ function AppPageContent() {
<div className="grid lg:grid-cols-2 gap-8">
{/* Submit Job Card */}
<Card className="backdrop-blur min-w-0" style={{ borderColor: 'var(--card-border)', backgroundColor: 'var(--card)' }}>
<Card className="backdrop-blur" style={{ borderColor: 'var(--card-border)', backgroundColor: 'var(--card)' }}>
<CardHeader>
<CardTitle style={{ color: 'var(--text)' }}>Create Karaoke Video</CardTitle>
<CardDescription style={{ color: 'var(--text-muted)' }}>
@@ -229,30 +219,17 @@ function AppPageContent() {
</Card>
{/* Jobs List Card */}
<Card className="backdrop-blur min-w-0" style={{ borderColor: 'var(--card-border)', backgroundColor: 'var(--card)' }}>
<CardHeader className="px-3 sm:px-6">
<div className="flex items-center justify-between gap-2">
<CardTitle style={{ color: 'var(--text)' }}>Recent Jobs</CardTitle>
<div className="flex items-center gap-2">
{isLoadingJobs && <Loader2 className="w-4 h-4 animate-spin" style={{ color: 'var(--text-muted)' }} />}
<Select value={String(jobLimit)} onValueChange={(v) => setJobLimit(Number(v))}>
<SelectTrigger className="h-7 w-[80px] text-xs">
<SelectValue />
</SelectTrigger>
<SelectContent>
<SelectItem value="5">5</SelectItem>
<SelectItem value="10">10</SelectItem>
<SelectItem value="20">20</SelectItem>
<SelectItem value="-1">All</SelectItem>
</SelectContent>
</Select>
</div>
</div>
<Card className="backdrop-blur" style={{ borderColor: 'var(--card-border)', backgroundColor: 'var(--card)' }}>
<CardHeader>
<CardTitle className="flex items-center justify-between" style={{ color: 'var(--text)' }}>
Recent Jobs
{isLoadingJobs && <Loader2 className="w-4 h-4 animate-spin" style={{ color: 'var(--text-muted)' }} />}
</CardTitle>
<CardDescription style={{ color: 'var(--text-muted)' }}>
{jobs.length} job{jobs.length !== 1 ? 's' : ''} shown
{jobs.length} job{jobs.length !== 1 ? 's' : ''} found
</CardDescription>
</CardHeader>
<CardContent className="px-3 sm:px-6">
<CardContent>
{jobs.length === 0 ? (
<div className="text-center py-8" style={{ color: 'var(--text-muted)' }}>
<Music2 className="w-12 h-12 mx-auto mb-3 opacity-50" />
-2
View File
@@ -4,7 +4,6 @@ import { ThemeProvider } from "@/components/theme-provider"
import { TenantProvider } from "@/components/tenant-provider"
import { ImpersonationBannerWrapper } from "@/components/impersonation-banner-wrapper"
import { ServiceWorkerRegistration } from "@/components/service-worker-registration"
import { GoogleAnalytics } from "@/components/google-analytics"
import "./globals.css"
export const metadata: Metadata = {
@@ -29,7 +28,6 @@ export default function RootLayout({
return (
<html lang="en" suppressHydrationWarning>
<body className="font-sans antialiased" style={{ background: 'var(--bg)', color: 'var(--text)' }}>
<GoogleAnalytics />
<ServiceWorkerRegistration />
<ThemeProvider
attribute="class"
+5 -11
View File
@@ -39,20 +39,14 @@ export function AutoProcessor({ jobs, onJobsChanged }: AutoProcessorProps) {
try {
switch (action) {
case 'review': {
// Auto-complete lyrics review with instrumental selection
// Use backing vocals analyzer recommendation (default to clean if review needed or missing)
const analysis = job.state_data?.backing_vocals_analysis
const recommendation = analysis?.recommended_selection
// Only use 'with_backing' if explicitly recommended by analyzer, otherwise clean (safe default)
const selection = recommendation === 'with_backing' ? 'with_backing' : 'clean'
await api.completeReview(job.job_id, selection)
console.log(`[AutoProcessor] Auto-completed review for job ${job.job_id} with ${selection} instrumental (recommendation: ${recommendation || 'none'})`)
case 'review':
// Auto-complete lyrics review
await api.completeReview(job.job_id)
console.log(`[AutoProcessor] Auto-completed review for job ${job.job_id}`)
break
}
case 'instrumental': {
// For finalise-only jobs that still use the separate instrumental selection flow
// Use backing vocals analyzer recommendation (default to clean if review needed or missing)
const analysis = job.state_data?.backing_vocals_analysis
const recommendation = analysis?.recommended_selection
// Only use 'with_backing' if explicitly recommended by analyzer, otherwise clean (safe default)
@@ -1,177 +0,0 @@
/**
* Tests for OutputLinks component
*
* Tests the download buttons and external links behavior,
* including hiding links when outputs have been deleted.
*/
import { render, screen } from '@testing-library/react'
import { OutputLinks } from '../job/OutputLinks'
import { Job } from '@/lib/api'
// Mock the hooks
jest.mock('@/lib/auth', () => ({
useAuth: jest.fn(() => ({ user: { role: 'user' } })),
}))
jest.mock('@/lib/tenant', () => ({
useTenant: jest.fn(() => ({
features: {
youtube_upload: true,
dropbox_upload: true,
},
})),
}))
// Mock the api module
jest.mock('@/lib/api', () => ({
api: {
getDownloadUrl: jest.fn((jobId, category, key) => `https://example.com/${jobId}/${category}/${key}`),
},
adminApi: {
getCompletionMessage: jest.fn(),
sendCompletionEmail: jest.fn(),
},
}))
describe('OutputLinks', () => {
const baseJob: Job = {
job_id: 'test-123',
status: 'complete',
progress: 100,
created_at: '2026-01-01T00:00:00Z',
updated_at: '2026-01-01T00:00:00Z',
artist: 'Test Artist',
title: 'Test Song',
file_urls: {
finals: {
lossy_4k_mp4: 'gs://bucket/finals/4k.mp4',
lossy_720p_mp4: 'gs://bucket/finals/720p.mp4',
},
videos: {
with_vocals: 'gs://bucket/videos/with_vocals.mkv',
},
packages: {
cdg_zip: 'gs://bucket/packages/cdg.zip',
txt_zip: 'gs://bucket/packages/txt.zip',
},
},
state_data: {
youtube_url: 'https://youtube.com/watch?v=abc123',
dropbox_link: 'https://dropbox.com/s/xyz789',
},
}
beforeEach(() => {
jest.clearAllMocks()
// Reset to default non-admin user
const { useAuth } = require('@/lib/auth')
useAuth.mockReturnValue({ user: { role: 'user' } })
})
describe('when outputs are NOT deleted', () => {
it('shows download buttons when file_urls are present', () => {
render(<OutputLinks job={baseJob} />)
expect(screen.getByText('4K Video')).toBeInTheDocument()
expect(screen.getByText('720p Video')).toBeInTheDocument()
expect(screen.getByText('With Vocals')).toBeInTheDocument()
expect(screen.getByText('CDG')).toBeInTheDocument()
expect(screen.getByText('TXT')).toBeInTheDocument()
})
it('shows YouTube link when youtube_url is in state_data', () => {
render(<OutputLinks job={baseJob} />)
expect(screen.getByText('YouTube')).toBeInTheDocument()
})
it('shows Dropbox link when dropbox_link is in state_data', () => {
render(<OutputLinks job={baseJob} />)
expect(screen.getByText('Dropbox')).toBeInTheDocument()
})
})
describe('when outputs ARE deleted (outputs_deleted_at is set)', () => {
const deletedJob: Job = {
...baseJob,
outputs_deleted_at: '2026-01-15T10:00:00Z',
outputs_deleted_by: 'admin@example.com',
}
it('hides all download buttons', () => {
render(<OutputLinks job={deletedJob} />)
expect(screen.queryByText('4K Video')).not.toBeInTheDocument()
expect(screen.queryByText('720p Video')).not.toBeInTheDocument()
expect(screen.queryByText('With Vocals')).not.toBeInTheDocument()
expect(screen.queryByText('CDG')).not.toBeInTheDocument()
expect(screen.queryByText('TXT')).not.toBeInTheDocument()
})
it('hides YouTube link', () => {
render(<OutputLinks job={deletedJob} />)
expect(screen.queryByText('YouTube')).not.toBeInTheDocument()
})
it('hides Dropbox link', () => {
render(<OutputLinks job={deletedJob} />)
expect(screen.queryByText('Dropbox')).not.toBeInTheDocument()
})
it('still shows Admin link for admin users', () => {
// Mock admin user
const { useAuth } = require('@/lib/auth')
useAuth.mockReturnValue({ user: { role: 'admin' } })
render(<OutputLinks job={deletedJob} />)
// Admin link should still be visible even when outputs are deleted
expect(screen.getByText('Admin')).toBeInTheDocument()
})
})
describe('edge cases', () => {
it('shows nothing when job has no file_urls and no state_data (non-admin)', () => {
// Reset to non-admin user
const { useAuth } = require('@/lib/auth')
useAuth.mockReturnValue({ user: { role: 'user' } })
const emptyJob: Job = {
...baseJob,
file_urls: undefined,
state_data: undefined,
}
render(<OutputLinks job={emptyJob} />)
// Should show "No outputs available yet" for non-admin
expect(screen.getByText('No outputs available yet')).toBeInTheDocument()
})
it('handles partial file_urls gracefully', () => {
// Reset to non-admin user
const { useAuth } = require('@/lib/auth')
useAuth.mockReturnValue({ user: { role: 'user' } })
const partialJob: Job = {
...baseJob,
file_urls: {
finals: {
lossy_720p_mp4: 'gs://bucket/finals/720p.mp4',
},
},
state_data: undefined,
}
render(<OutputLinks job={partialJob} />)
expect(screen.getByText('720p Video')).toBeInTheDocument()
expect(screen.queryByText('4K Video')).not.toBeInTheDocument()
expect(screen.queryByText('YouTube')).not.toBeInTheDocument()
})
})
})
@@ -1,117 +0,0 @@
/**
* Tests for StemComparison component
*
* This component renders audio type selection tabs for the instrumental review UI.
* It supports showing a loading spinner when audio is loading.
*/
import { render, screen, fireEvent } from '@testing-library/react'
import { StemComparison, AudioType } from '../instrumental-review/StemComparison'
describe('StemComparison', () => {
const mockOnAudioChange = jest.fn()
const defaultProps = {
activeAudio: 'clean' as AudioType,
onAudioChange: mockOnAudioChange,
hasOriginal: true,
hasWithBacking: true,
hasCustom: false,
hasUploaded: false,
}
beforeEach(() => {
jest.clearAllMocks()
})
it('renders available audio options', () => {
render(<StemComparison {...defaultProps} />)
expect(screen.getByText('Original')).toBeInTheDocument()
expect(screen.getByText('Backing Vocals Only')).toBeInTheDocument()
expect(screen.getByText('Pure Instrumental')).toBeInTheDocument()
expect(screen.getByText('Instrumental + Backing')).toBeInTheDocument()
})
it('does not render unavailable options', () => {
render(<StemComparison {...defaultProps} hasOriginal={false} hasWithBacking={false} />)
expect(screen.queryByText('Original')).not.toBeInTheDocument()
expect(screen.queryByText('Instrumental + Backing')).not.toBeInTheDocument()
// These are always available
expect(screen.getByText('Backing Vocals Only')).toBeInTheDocument()
expect(screen.getByText('Pure Instrumental')).toBeInTheDocument()
})
it('shows custom option when available', () => {
render(<StemComparison {...defaultProps} hasCustom={true} />)
expect(screen.getByText('Custom')).toBeInTheDocument()
})
it('shows uploaded option when available', () => {
render(<StemComparison {...defaultProps} hasUploaded={true} uploadedFilename="my-instrumental.flac" />)
const uploadedButton = screen.getByText('Uploaded')
expect(uploadedButton).toBeInTheDocument()
expect(uploadedButton).toHaveAttribute('title', 'my-instrumental.flac')
})
it('calls onAudioChange when an option is clicked', () => {
render(<StemComparison {...defaultProps} />)
fireEvent.click(screen.getByText('Original'))
expect(mockOnAudioChange).toHaveBeenCalledWith('original')
fireEvent.click(screen.getByText('Backing Vocals Only'))
expect(mockOnAudioChange).toHaveBeenCalledWith('backing')
})
it('highlights the active audio option', () => {
render(<StemComparison {...defaultProps} activeAudio="backing" />)
const backingButton = screen.getByText('Backing Vocals Only')
expect(backingButton).toHaveClass('bg-primary')
})
describe('loading state', () => {
it('shows loading spinner on active tab when isLoading is true', () => {
render(<StemComparison {...defaultProps} activeAudio="backing" isLoading={true} />)
// The Loader2 icon should be present in the active tab
const backingButton = screen.getByText('Backing Vocals Only').closest('button')
expect(backingButton).toContainHTML('animate-spin')
})
it('does not show spinner on inactive tabs when loading', () => {
render(<StemComparison {...defaultProps} activeAudio="backing" isLoading={true} />)
const originalButton = screen.getByText('Original').closest('button')
expect(originalButton).not.toContainHTML('animate-spin')
})
it('disables all tabs when loading', () => {
render(<StemComparison {...defaultProps} isLoading={true} />)
const originalButton = screen.getByText('Original').closest('button')
const backingButton = screen.getByText('Backing Vocals Only').closest('button')
expect(originalButton).toBeDisabled()
expect(backingButton).toBeDisabled()
})
it('does not disable tabs when not loading', () => {
render(<StemComparison {...defaultProps} isLoading={false} />)
const originalButton = screen.getByText('Original').closest('button')
expect(originalButton).not.toBeDisabled()
})
it('applies opacity styling to inactive tabs when loading', () => {
render(<StemComparison {...defaultProps} activeAudio="backing" isLoading={true} />)
const originalButton = screen.getByText('Original').closest('button')
expect(originalButton).toHaveClass('opacity-50')
})
})
})
-24
View File
@@ -1,24 +0,0 @@
'use client'
import Script from 'next/script'
const GA_MEASUREMENT_ID = process.env.NEXT_PUBLIC_GA_MEASUREMENT_ID || 'G-D0E63LB9FY'
export function GoogleAnalytics() {
return (
<>
<Script
src={`https://www.googletagmanager.com/gtag/js?id=${GA_MEASUREMENT_ID}`}
strategy="afterInteractive"
/>
<Script id="google-analytics" strategy="afterInteractive">
{`
window.dataLayer = window.dataLayer || [];
function gtag(){dataLayer.push(arguments);}
gtag('js', new Date());
gtag('config', '${GA_MEASUREMENT_ID}');
`}
</Script>
</>
)
}
@@ -9,7 +9,6 @@ import Link from "next/link"
import { toast } from "sonner"
import {
api,
lyricsReviewApi,
type Job,
type InstrumentalAnalysis,
type WaveformData,
@@ -75,7 +74,6 @@ export function InstrumentalSelector({ job, isLocalMode = false }: InstrumentalS
const [isUploading, setIsUploading] = useState(false)
const [isSubmitting, setIsSubmitting] = useState(false)
const [isShiftHeld, setIsShiftHeld] = useState(false)
const [isAudioLoading, setIsAudioLoading] = useState(false)
// Derived state
const hasOriginal = analysisData?.has_original ?? false
@@ -101,16 +99,9 @@ export function InstrumentalSelector({ job, isLocalMode = false }: InstrumentalS
useEffect(() => {
async function fetchData() {
try {
// Use different API based on mode:
// - Local mode: uses /api/jobs/{id}/... endpoints
// - Cloud mode: uses /api/review/{id}/... endpoints
const [analysis, waveform] = await Promise.all([
isLocalMode
? api.getInstrumentalAnalysis(job.job_id)
: lyricsReviewApi.getInstrumentalAnalysis(job.job_id),
isLocalMode
? api.getWaveformData(job.job_id)
: lyricsReviewApi.getWaveformData(job.job_id),
api.getInstrumentalAnalysis(job.job_id),
api.getWaveformData(job.job_id),
])
setAnalysisData(analysis)
@@ -136,7 +127,7 @@ export function InstrumentalSelector({ job, isLocalMode = false }: InstrumentalS
}
fetchData()
}, [job.job_id, isLocalMode])
}, [job.job_id])
// Keyboard shortcuts
useEffect(() => {
@@ -222,22 +213,9 @@ export function InstrumentalSelector({ job, isLocalMode = false }: InstrumentalS
// Get audio URL for current selection
const getAudioUrl = useCallback(() => {
// For cloud mode, use signed URLs from analysis data
// For local mode, use the API stream endpoint
if (!isLocalMode && analysisData?.audio_urls) {
const urls = analysisData.audio_urls
switch (activeAudio) {
case "clean": return urls.clean || ""
case "with_backing": return urls.with_backing || ""
case "original": return urls.original || ""
case "backing": return urls.backing_vocals || ""
default: return urls.with_backing || urls.clean || ""
}
}
// Local mode: use API stream endpoint
const stemType = STEM_TYPE_MAP[activeAudio]
return api.getAudioStreamUrl(job.job_id, stemType)
}, [job.job_id, activeAudio, isLocalMode, analysisData])
}, [job.job_id, activeAudio])
// Handle audio type change
const handleAudioChange = useCallback(
@@ -250,47 +228,23 @@ export function InstrumentalSelector({ job, isLocalMode = false }: InstrumentalS
audio.pause()
setActiveAudio(type)
setIsAudioLoading(true)
// Update audio source
// For cloud mode, use signed URLs from analysis data
let audioUrl: string
if (!isLocalMode && analysisData?.audio_urls) {
const urls = analysisData.audio_urls
switch (type) {
case "clean": audioUrl = urls.clean || ""; break
case "with_backing": audioUrl = urls.with_backing || ""; break
case "original": audioUrl = urls.original || ""; break
case "backing": audioUrl = urls.backing_vocals || ""; break
default: audioUrl = urls.with_backing || urls.clean || ""
}
} else {
const stemType = STEM_TYPE_MAP[type]
audioUrl = api.getAudioStreamUrl(job.job_id, stemType)
}
audio.src = audioUrl
const stemType = STEM_TYPE_MAP[type]
audio.src = api.getAudioStreamUrl(job.job_id, stemType)
const handleLoaded = () => {
setIsAudioLoading(false)
audio.currentTime = time
if (wasPlaying) {
audio.play().catch(() => {})
}
}
const handleError = () => {
setIsAudioLoading(false)
toast.error("Failed to load audio")
// Restore previous playback state
if (wasPlaying) {
audio.play().catch(() => {})
}
}
audio.addEventListener("loadeddata", handleLoaded, { once: true })
audio.addEventListener("error", handleError, { once: true })
audio.addEventListener(
"loadeddata",
() => {
audio.currentTime = time
if (wasPlaying) {
audio.play().catch(() => {})
}
},
{ once: true }
)
},
[job.job_id, isLocalMode, analysisData]
[job.job_id]
)
// Seek to time
@@ -422,23 +376,28 @@ export function InstrumentalSelector({ job, isLocalMode = false }: InstrumentalS
const [showSuccess, setShowSuccess] = useState(false)
const [countdown, setCountdown] = useState(2)
// Submit selection (final submission: corrections + instrumental)
// Submit selection
const handleSubmit = useCallback(async () => {
setIsSubmitting(true)
try {
// Fetch current correction data and submit with instrumental selection
// Using the same review API for both local and cloud mode
const correctionData = await lyricsReviewApi.getCorrectionData(job.job_id)
await lyricsReviewApi.completeReview(job.job_id, correctionData, selectedOption)
await api.selectInstrumental(job.job_id, selectedOption)
// Show success screen in both modes
setShowSuccess(true)
setCountdown(isLocalMode ? 2 : 3)
if (isLocalMode) {
// In local mode, show success screen with countdown then close
setShowSuccess(true)
setCountdown(2)
} else {
// Redirect to dashboard after short delay (cloud mode)
toast.success("Selection submitted successfully")
setTimeout(() => {
router.push("/app")
}, 1500)
}
} catch (err) {
toast.error(err instanceof Error ? err.message : "Failed to complete review")
toast.error(err instanceof Error ? err.message : "Failed to submit selection")
setIsSubmitting(false)
}
}, [job.job_id, selectedOption, isLocalMode])
}, [job.job_id, selectedOption, router, isLocalMode])
// Countdown effect for success screen
useEffect(() => {
@@ -448,11 +407,7 @@ export function InstrumentalSelector({ job, isLocalMode = false }: InstrumentalS
setCountdown((prev) => {
if (prev <= 1) {
clearInterval(interval)
if (isLocalMode) {
window.close()
} else {
router.push("/app")
}
window.close()
return 0
}
return prev - 1
@@ -460,7 +415,7 @@ export function InstrumentalSelector({ job, isLocalMode = false }: InstrumentalS
}, 1000)
return () => clearInterval(interval)
}, [showSuccess, isLocalMode, router])
}, [showSuccess])
// Loading state
if (isLoading) {
@@ -493,7 +448,7 @@ export function InstrumentalSelector({ job, isLocalMode = false }: InstrumentalS
)
}
// Success screen
// Success screen (local mode)
if (showSuccess) {
const selectionLabels: Record<string, string> = {
clean: "Clean Instrumental",
@@ -516,12 +471,8 @@ export function InstrumentalSelector({ job, isLocalMode = false }: InstrumentalS
</p>
<p className="text-muted-foreground">
{countdown > 0
? isLocalMode
? `Closing in ${countdown}s...`
: `Redirecting in ${countdown}s...`
: isLocalMode
? "You can close this window now."
: "Redirecting..."}
? `Closing in ${countdown}s...`
: "You can close this window now."}
</p>
</div>
</div>
@@ -606,7 +557,6 @@ export function InstrumentalSelector({ job, isLocalMode = false }: InstrumentalS
hasCustom={hasCustom}
hasUploaded={hasUploaded}
uploadedFilename={uploadedFilename}
isLoading={isAudioLoading}
/>
<CustomUpload onUpload={handleUpload} isUploading={isUploading} />
</div>
@@ -641,27 +591,17 @@ export function InstrumentalSelector({ job, isLocalMode = false }: InstrumentalS
</div>
{/* Waveform */}
<div className="relative flex-1 min-h-0">
<WaveformViewer
amplitudes={amplitudes}
duration={duration}
currentTime={currentTime}
muteRegions={muteRegions}
audibleSegments={audibleSegments}
zoomLevel={zoomLevel}
onSeek={seekTo}
onRegionCreate={addRegion}
isShiftHeld={isShiftHeld}
/>
{isAudioLoading && (
<div className="absolute inset-0 flex items-center justify-center bg-background/50 backdrop-blur-sm">
<div className="flex items-center gap-2 text-sm text-muted-foreground">
<Spinner className="w-4 h-4" />
Loading audio...
</div>
</div>
)}
</div>
<WaveformViewer
amplitudes={amplitudes}
duration={duration}
currentTime={currentTime}
muteRegions={muteRegions}
audibleSegments={audibleSegments}
zoomLevel={zoomLevel}
onSeek={seekTo}
onRegionCreate={addRegion}
isShiftHeld={isShiftHeld}
/>
{/* Time axis */}
<div className="flex justify-between px-3 py-1 text-[10px] text-muted-foreground bg-black/40 flex-shrink-0">
@@ -1,212 +0,0 @@
"use client"
import { useState, useCallback, useMemo } from "react"
import { Card } from "@/components/ui/card"
import { Label } from "@/components/ui/label"
import { Button } from "@/components/ui/button"
import { Badge } from "@/components/ui/badge"
import { Play, Pause, Music } from "lucide-react"
import { cn } from "@/lib/utils"
import type { InstrumentalSelectionType } from "@/lib/api"
export interface InstrumentalOption {
id: InstrumentalSelectionType
label: string
audio_url: string | null
}
/** Analysis data for backing vocals - flexible type to match correction-data endpoint */
interface BackingVocalsAnalysisProps {
audible_segments?: Array<{ start_seconds: number; end_seconds: number }>
audible_percentage?: number
recommended_selection?: string
has_audible_content?: boolean
total_duration_seconds?: number
total_audible_duration_seconds?: number
silence_threshold_db?: number
}
interface InstrumentalSelectorEmbeddedProps {
/** Available instrumental options from the correction-data endpoint */
options: InstrumentalOption[]
/** Backing vocals analysis data from the correction-data endpoint */
analysis?: BackingVocalsAnalysisProps | null
/** Currently selected instrumental option */
value: InstrumentalSelectionType | null
/** Callback when selection changes */
onChange: (value: InstrumentalSelectionType) => void
/** Whether to show compact layout (for embedding in review modal) */
compact?: boolean
/** Disabled state */
disabled?: boolean
}
const OPTION_DESCRIPTIONS: Record<string, string> = {
clean: "No backing vocals - pure instrumental",
with_backing: "Includes backing vocals from original",
}
export function InstrumentalSelectorEmbedded({
options,
analysis,
value,
onChange,
compact = false,
disabled = false,
}: InstrumentalSelectorEmbeddedProps) {
const [playingId, setPlayingId] = useState<string | null>(null)
const [audioElement] = useState<HTMLAudioElement | null>(() =>
typeof window !== 'undefined' ? new Audio() : null
)
// Get recommendation from analysis
const recommendedOption = useMemo(() => {
if (!analysis) return null
return analysis.recommended_selection || null
}, [analysis])
// Handle audio playback
const handlePlayPause = useCallback((optionId: string, audioUrl: string | null) => {
if (!audioElement || !audioUrl) return
if (playingId === optionId) {
// Stop playing
audioElement.pause()
audioElement.currentTime = 0
setPlayingId(null)
} else {
// Start playing new audio
audioElement.src = audioUrl
audioElement.play().catch(() => {})
setPlayingId(optionId)
// Stop when ended
audioElement.onended = () => setPlayingId(null)
}
}, [audioElement, playingId])
// Clean up audio on unmount
const handleSelect = useCallback((optionId: InstrumentalSelectionType) => {
if (disabled) return
onChange(optionId)
}, [onChange, disabled])
if (options.length === 0) {
return (
<Card className="p-4">
<div className="text-sm text-muted-foreground">
No instrumental options available.
</div>
</Card>
)
}
return (
<Card className={cn("p-4", compact && "p-3")}>
<div className="flex items-center gap-2 mb-3">
<Music className="h-4 w-4 text-muted-foreground" />
<Label className={cn("font-semibold", compact ? "text-sm" : "text-base")}>
Select Instrumental Track
</Label>
{!value && (
<Badge variant="destructive" className="ml-auto text-xs">
Required
</Badge>
)}
</div>
{/* Backing vocals info */}
{analysis && (analysis.audible_percentage ?? 0) > 0 && (
<div className="mb-3 text-xs text-muted-foreground">
<span className="font-medium">
{(analysis.audible_percentage ?? 0).toFixed(0)}% backing vocals detected
</span>
{(analysis.audible_segments?.length ?? 0) > 0 && (
<span> ({analysis.audible_segments?.length ?? 0} segments)</span>
)}
</div>
)}
<div className={cn("flex flex-col gap-2", compact && "gap-1.5")}>
{options.map((option) => {
const isSelected = value === option.id
const isRecommended = recommendedOption === option.id
const isPlaying = playingId === option.id
return (
<div
key={option.id}
className={cn(
"flex items-center gap-3 p-3 rounded-lg transition-all cursor-pointer",
"bg-secondary/50 hover:bg-secondary/80",
"border-2",
isSelected ? "border-primary bg-primary/10" : "border-transparent",
disabled && "opacity-50 cursor-not-allowed",
compact && "p-2"
)}
onClick={() => handleSelect(option.id)}
role="button"
aria-pressed={isSelected}
tabIndex={disabled ? -1 : 0}
onKeyDown={(e) => {
if (e.key === "Enter" || e.key === " ") {
e.preventDefault()
handleSelect(option.id)
}
}}
>
{/* Radio indicator */}
<div
className={cn(
"w-4 h-4 rounded-full border-2 flex items-center justify-center flex-shrink-0",
isSelected ? "border-primary" : "border-muted-foreground"
)}
>
{isSelected && (
<div className="w-2 h-2 rounded-full bg-primary" />
)}
</div>
{/* Label and description */}
<div className="flex-1 min-w-0">
<div className={cn("font-medium flex items-center gap-2", compact ? "text-sm" : "text-sm")}>
{option.label}
{isRecommended && (
<Badge variant="secondary" className="text-xs bg-green-500/20 text-green-600">
Recommended
</Badge>
)}
</div>
<div className={cn("text-muted-foreground", compact ? "text-xs" : "text-xs")}>
{OPTION_DESCRIPTIONS[option.id] || `Instrumental option: ${option.id}`}
</div>
</div>
{/* Preview button */}
{option.audio_url && (
<Button
variant="ghost"
size="icon"
className="h-8 w-8 flex-shrink-0"
onClick={(e) => {
e.stopPropagation()
handlePlayPause(option.id, option.audio_url)
}}
disabled={disabled}
>
{isPlaying ? (
<Pause className="h-4 w-4" />
) : (
<Play className="h-4 w-4" />
)}
</Button>
)}
</div>
)
})}
</div>
</Card>
)
}
export default InstrumentalSelectorEmbedded
@@ -1,7 +1,6 @@
"use client"
import { cn } from "@/lib/utils"
import { Loader2 } from "lucide-react"
export type AudioType = "original" | "backing" | "clean" | "with_backing" | "custom" | "uploaded"
@@ -13,7 +12,6 @@ interface StemComparisonProps {
hasCustom: boolean
hasUploaded: boolean
uploadedFilename?: string
isLoading?: boolean
}
interface AudioOption {
@@ -31,7 +29,6 @@ export function StemComparison({
hasCustom,
hasUploaded,
uploadedFilename,
isLoading = false,
}: StemComparisonProps) {
const options: AudioOption[] = [
{ type: "original", label: "Original", available: hasOriginal },
@@ -46,29 +43,21 @@ export function StemComparison({
return (
<div className="flex flex-wrap gap-1 bg-background rounded-md p-0.5">
{availableOptions.map((option) => {
const isActive = activeAudio === option.type
const showSpinner = isActive && isLoading
return (
<button
key={option.type}
type="button"
onClick={() => onAudioChange(option.type)}
disabled={isLoading}
title={option.title}
className={cn(
"px-2.5 py-1.5 rounded text-xs font-medium transition-all flex items-center gap-1.5",
"text-muted-foreground hover:text-foreground",
isActive && "bg-primary text-primary-foreground",
isLoading && !isActive && "opacity-50 cursor-not-allowed"
)}
>
{showSpinner && <Loader2 className="w-3 h-3 animate-spin" />}
{option.label}
</button>
)
})}
{availableOptions.map((option) => (
<button
key={option.type}
type="button"
onClick={() => onAudioChange(option.type)}
title={option.title}
className={cn(
"px-2.5 py-1.5 rounded text-xs font-medium transition-all",
"text-muted-foreground hover:text-foreground",
activeAudio === option.type && "bg-primary text-primary-foreground"
)}
>
{option.label}
</button>
))}
</div>
)
}
@@ -1,6 +1,4 @@
export { InstrumentalSelector } from "./InstrumentalSelector"
export { InstrumentalSelectorEmbedded } from "./InstrumentalSelectorEmbedded"
export type { InstrumentalOption } from "./InstrumentalSelectorEmbedded"
export { SelectionOptions } from "./SelectionOptions"
export { StemComparison } from "./StemComparison"
export { WaveformViewer } from "./WaveformViewer"
+5 -9
View File
@@ -141,7 +141,7 @@ export function JobCard({ job, onRefresh }: JobCardProps) {
return (
<div
className={`rounded-lg border p-3 transition-colors overflow-hidden
className={`rounded-lg border p-3 transition-colors
${isComplete ? "border-green-500/30" : ""}
${isFailed ? "border-red-500/30" : ""}`}
style={{
@@ -151,7 +151,7 @@ export function JobCard({ job, onRefresh }: JobCardProps) {
>
{/* Header row with title and non-interactive badge */}
<div className="flex items-start justify-between gap-2">
<p className="font-medium truncate min-w-0" style={{ color: 'var(--text)' }}>
<p className="font-medium truncate" style={{ color: 'var(--text)' }}>
{job.state_data?.brand_code && `${job.state_data.brand_code}: `}
{job.artist || "Unknown"} - {job.title || "Unknown"}
</p>
@@ -163,13 +163,9 @@ export function JobCard({ job, onRefresh }: JobCardProps) {
)}
</div>
{/* Meta row: ID, date, status - wraps on mobile */}
<p className="text-xs mt-1 flex flex-wrap gap-x-1.5 gap-y-0.5" style={{ color: 'var(--text-muted)' }}>
<span><span style={{ opacity: 0.7 }}>ID:</span> {job.job_id.slice(0, 8)}</span>
<span style={{ opacity: 0.7 }}>•</span>
<span>{createdAt}</span>
<span style={{ opacity: 0.7 }}>•</span>
<StatusIndicator job={job} />
{/* Meta row: ID, date, status */}
<p className="text-xs mt-1" style={{ color: 'var(--text-muted)' }}>
<span style={{ opacity: 0.7 }}>ID:</span> {job.job_id} <span style={{ opacity: 0.7 }}>•</span> {createdAt} <span style={{ opacity: 0.7 }}>•</span> <StatusIndicator job={job} />
</p>
{/* Progress bar for active jobs */}
+5 -8
View File
@@ -52,12 +52,9 @@ export function OutputLinks({ job }: OutputLinksProps) {
const { features } = useTenant()
const isAdmin = user?.role === 'admin'
// Check if outputs have been deleted by admin
const outputsDeleted = !!job.outputs_deleted_at
// Filter external links based on tenant features (and outputs not deleted)
const showYoutubeLink = features.youtube_upload && youtubeUrl && !outputsDeleted
const showDropboxLink = features.dropbox_upload && dropboxUrl && !outputsDeleted
// Filter external links based on tenant features
const showYoutubeLink = features.youtube_upload && youtubeUrl
const showDropboxLink = features.dropbox_upload && dropboxUrl
const copyToClipboard = useCallback(async (url: string) => {
// Check if clipboard API is available
@@ -140,8 +137,8 @@ export function OutputLinks({ job }: OutputLinksProps) {
const hasOutputs = showYoutubeLink || showDropboxLink || (downloadUrls && Object.keys(downloadUrls).length > 0)
// Check if we have any downloads (and outputs haven't been deleted)
const hasDownloads = !outputsDeleted && downloadUrls && (
// Check if we have any downloads
const hasDownloads = downloadUrls && (
downloadUrls?.finals?.lossy_720p_mp4 ||
downloadUrls?.finals?.lossy_4k_mp4 ||
downloadUrls?.videos?.with_vocals ||
@@ -7,7 +7,6 @@ import { Label } from '@/components/ui/label'
import { Trash2, Split, Wand2 } from 'lucide-react'
import { Word } from '@/lib/lyrics-review/types'
import WordDivider from './WordDivider'
import TimeInput from './TimeInput'
interface EditWordListProps {
words: Word[]
@@ -98,18 +97,26 @@ const WordRow = memo(function WordRow({
<div
className={`flex gap-2 items-end ${isMobile ? 'justify-start' : 'justify-end'}`}
>
<TimeInput
value={word.start_time ?? null}
onChange={(val) => onWordUpdate(index, { start_time: val ?? undefined })}
label="Start"
widthClass={isMobile ? 'w-20' : 'w-24'}
/>
<TimeInput
value={word.end_time ?? null}
onChange={(val) => onWordUpdate(index, { end_time: val ?? undefined })}
label="End"
widthClass={isMobile ? 'w-20' : 'w-24'}
/>
<div className={`${isMobile ? 'w-20' : 'w-24'}`}>
<Label className="text-xs text-muted-foreground mb-1">Start</Label>
<Input
type="number"
value={word.start_time?.toFixed(2) ?? ''}
onChange={(e) => onWordUpdate(index, { start_time: parseFloat(e.target.value) })}
step={0.01}
className="h-8"
/>
</div>
<div className={`${isMobile ? 'w-20' : 'w-24'}`}>
<Label className="text-xs text-muted-foreground mb-1">End</Label>
<Input
type="number"
value={word.end_time?.toFixed(2) ?? ''}
onChange={(e) => onWordUpdate(index, { end_time: parseFloat(e.target.value) })}
step={0.01}
className="h-8"
/>
</div>
{/* Action buttons on desktop only */}
{!isMobile && (
<>
@@ -351,13 +358,6 @@ export default function EditWordList({
<Input
value={replacementText}
onChange={(e) => setReplacementText(e.target.value)}
onKeyDown={(e) => {
if (e.key === 'Enter' && replacementText.trim()) {
e.preventDefault()
e.stopPropagation()
handleReplaceAllWords()
}
}}
placeholder="Replace all words"
className="flex-grow"
/>
+2 -2
View File
@@ -191,7 +191,7 @@ export default function Header({
<TooltipTrigger asChild>
<Badge
variant={reviewMode ? 'default' : 'outline'}
className="cursor-pointer hover:opacity-80 transition-opacity"
className="cursor-pointer"
onClick={() => onReviewModeToggle(!reviewMode)}
>
<Eye className="h-3 w-3 mr-1" />
@@ -231,7 +231,7 @@ export default function Header({
className="scale-75"
/>
<Label
className="text-xs cursor-pointer hover:text-foreground hover:underline transition-colors"
className="text-xs cursor-pointer"
onClick={() => onHandlerClick?.(handler.id)}
>
{handler.name} ({handlerCounts[handler.id] || 0})
@@ -1,10 +1,8 @@
'use client'
import { useState, useEffect, useCallback, useMemo, memo } from 'react'
import { useRouter } from 'next/navigation'
import { Button } from '@/components/ui/button'
import { Video, Check } from 'lucide-react'
import { toast } from 'sonner'
import { Video } from 'lucide-react'
import {
AnchorSequence,
CorrectionData,
@@ -19,7 +17,6 @@ import {
WordClickInfo,
ModalContent,
} from '@/lib/lyrics-review/types'
import type { InstrumentalSelectionType } from '@/lib/api'
import { cn } from '@/lib/utils'
import ReferenceView from './ReferenceView'
import TranscriptionView from './TranscriptionView'
@@ -81,8 +78,6 @@ export interface LyricsAnalyzerProps {
apiClient: ApiClient | null
isReadOnly: boolean
audioHash: string
isLocalMode?: boolean
jobId?: string
}
export default function LyricsAnalyzer({
@@ -91,10 +86,7 @@ export default function LyricsAnalyzer({
apiClient,
isReadOnly,
audioHash,
isLocalMode = false,
jobId,
}: LyricsAnalyzerProps) {
const router = useRouter()
const isMobile = typeof window !== 'undefined' && window.innerWidth < 768
// Modal state
@@ -128,12 +120,6 @@ export default function LyricsAnalyzer({
const [isTimingOffsetModalOpen, setIsTimingOffsetModalOpen] = useState(false)
const [timingOffsetMs, setTimingOffsetMs] = useState(0)
const [reviewMode, setReviewMode] = useState(false)
const [isSubmitting, setIsSubmitting] = useState(false)
// Success screen state (for auto-close after submission)
const [showSuccess, setShowSuccess] = useState(false)
const [countdown, setCountdown] = useState(2)
const [showInstrumentalReview, setShowInstrumentalReview] = useState(false)
// Annotation state
const [annotations, setAnnotations] = useState<Omit<CorrectionAnnotation, 'annotation_id' | 'timestamp'>[]>([])
@@ -249,38 +235,6 @@ export default function LyricsAnalyzer({
isTimingOffsetModalOpen,
])
// Effect to handle local mode navigation to instrumental review
useEffect(() => {
if (showInstrumentalReview && isLocalMode) {
// In local mode, redirect to instrumental page served by ReviewServer
window.location.href = `/app/jobs/local/instrumental`
}
}, [showInstrumentalReview, isLocalMode])
// Countdown effect for success screen (auto-close/redirect after submission)
useEffect(() => {
if (!showSuccess) return
const interval = setInterval(() => {
setCountdown((prev) => {
if (prev <= 1) {
clearInterval(interval)
if (isLocalMode) {
// Note: window.close() may be blocked by browsers if the window wasn't opened
// by script. The success screen shows "You can close this window now" as fallback.
window.close()
} else {
router.push('/app')
}
return 0
}
return prev - 1
})
}, 1000)
return () => clearInterval(interval)
}, [showSuccess, isLocalMode, router])
// Calculate effective mode based on modifier key states
const effectiveMode = isCtrlPressed ? 'delete_word' : isShiftPressed ? 'highlight' : interactionMode
@@ -612,28 +566,21 @@ export default function LyricsAnalyzer({
updateDataWithHistory(newData, 'revert all corrections')
}, [data, updateDataWithHistory])
// Finish review - opens preview modal
// Finish review
const handleFinishReview = useCallback(() => {
// Pause audio if playing to avoid clash with preview video
if (typeof window !== 'undefined' && window.isAudioPlaying && window.toggleAudioPlayback) {
window.toggleAudioPlayback()
}
setIsReviewModalOpen(true)
}, [])
// Submit to server (save corrections, don't complete review yet)
// Submit to server
const handleSubmitToServer = useCallback(async () => {
if (!apiClient) return
setIsSubmitting(true)
try {
const dataToSubmit =
timingOffsetMs !== 0 ? applyOffsetToCorrectionData(data, timingOffsetMs) : data
// 1. Save corrections (not final submission yet)
await apiClient.submitCorrections(dataToSubmit)
// 2. Save annotations
if (annotations.length > 0) {
try {
await apiClient.submitAnnotations(annotations)
@@ -642,31 +589,17 @@ export default function LyricsAnalyzer({
}
}
// 3. Close modal
setIsReviewModalOpen(false)
setIsSubmitting(false)
// Complete review and trigger video rendering
await apiClient.completeReview()
// 4. Navigate to instrumental review
if (isLocalMode) {
// Local mode: Show transition message, navigation handled by browser/server
toast.success('Lyrics saved! Loading instrumental review...')
setShowInstrumentalReview(true)
} else {
// Cloud mode: Use hash-based navigation with jobId
if (!jobId) {
console.error('No jobId available for cloud mode navigation')
toast.error('Navigation error - please return to dashboard')
return
}
toast.success('Lyrics saved! Proceeding to instrumental review...')
window.location.hash = `/${jobId}/instrumental`
}
setIsReviewComplete(true)
setIsReviewModalOpen(false)
window.close()
} catch (error) {
console.error('Failed to submit corrections:', error)
toast.error('Failed to submit corrections. Please try again.')
setIsSubmitting(false) // Reset on error so user can retry
alert('Failed to submit corrections. Please try again.')
}
}, [apiClient, data, timingOffsetMs, annotations, isLocalMode, jobId])
}, [apiClient, data, timingOffsetMs, annotations])
// Play segment handler
const handlePlaySegment = useCallback(
@@ -873,35 +806,6 @@ export default function LyricsAnalyzer({
setTimingOffsetMs(offsetMs)
}, [])
// Success screen - shown after review submission
if (showSuccess) {
return (
<div className="flex items-center justify-center min-h-screen bg-background">
<div className="text-center max-w-md p-6">
<div className="mb-4">
<Check className="w-12 h-12 mx-auto text-green-500" />
</div>
<h2 className="text-2xl font-semibold text-green-500 mb-4">
Review Submitted
</h2>
<p className="text-lg mb-2">
Your lyrics corrections have been saved.
</p>
<p className="text-muted-foreground mb-4">
Video generation has started and you&apos;ll receive a notification when it&apos;s ready.
</p>
<p className="text-muted-foreground">
{countdown > 0
? `${isLocalMode ? 'Closing' : 'Redirecting'} in ${countdown}s...`
: isLocalMode
? 'You can close this window now.'
: 'Redirecting...'}
</p>
</div>
</div>
)
}
return (
<div className="max-w-full overflow-x-hidden">
<Header
@@ -1039,7 +943,6 @@ export default function LyricsAnalyzer({
onClose={() => setIsReviewModalOpen(false)}
data={data}
onSubmit={handleSubmitToServer}
isSubmitting={isSubmitting}
apiClient={apiClient}
timingOffsetMs={timingOffsetMs}
/>
@@ -1,82 +0,0 @@
'use client'
import { useState, useRef, useEffect, useCallback, memo } from 'react'
import { Input } from '@/components/ui/input'
import { Label } from '@/components/ui/label'
interface TimeInputProps {
value: number | null
onChange: (value: number | null) => void
label: string
widthClass?: string
debounceMs?: number
}
const TimeInput = memo(function TimeInput({
value,
onChange,
label,
widthClass = 'w-24',
debounceMs = 300
}: TimeInputProps) {
const [localValue, setLocalValue] = useState<string>('')
const [isFocused, setIsFocused] = useState(false)
const debounceRef = useRef<NodeJS.Timeout | null>(null)
// Sync local state when parent value changes (while not focused)
useEffect(() => {
if (!isFocused) {
setLocalValue(value?.toFixed(2) ?? '')
}
}, [value, isFocused])
const debouncedUpdate = useCallback((strValue: string) => {
if (debounceRef.current) clearTimeout(debounceRef.current)
debounceRef.current = setTimeout(() => {
const parsed = parseFloat(strValue)
onChange(isNaN(parsed) ? null : parsed)
}, debounceMs)
}, [onChange, debounceMs])
// Cleanup debounce timer on unmount
useEffect(() => {
return () => {
if (debounceRef.current) clearTimeout(debounceRef.current)
}
}, [])
const handleFocus = (e: React.FocusEvent<HTMLInputElement>) => {
setLocalValue(value?.toFixed(2) ?? '')
setIsFocused(true)
e.target.select()
}
const handleChange = (e: React.ChangeEvent<HTMLInputElement>) => {
setLocalValue(e.target.value)
debouncedUpdate(e.target.value)
}
const handleBlur = () => {
setIsFocused(false)
if (debounceRef.current) clearTimeout(debounceRef.current)
const parsed = parseFloat(localValue)
onChange(isNaN(parsed) ? null : parsed)
}
return (
<div className={widthClass}>
<Label className="text-xs text-muted-foreground mb-1">{label}</Label>
<Input
type="number"
value={isFocused ? localValue : (value?.toFixed(2) ?? '')}
onFocus={handleFocus}
onChange={handleChange}
onBlur={handleBlur}
step={0.01}
className="h-8"
/>
</div>
)
})
export default TimeInput
Loaded 100 of 519 files, more files were not shown because too many files have changed in this diff. Show more